Thank you for pointing it out. Added mention of it to the epic management check list - MR.
1 Like
Perfect! Thank you @maxim 
I just wanted to take a moment to summarize a few of the points here and both see what has been done and what might still be done based on the suggestions:
- We’ve gotten the automation fixed surrounding strategic planning tickets, which can help both with spotting new potential epics and for generating them.
- Trial projects are proving especially challenging to produce. The following ideas have been put forth for them:
- Keep them to internal projects and then only perform recruitment when our sustainability is in a good spot.
- Offer to perform rejected, discovered work at a discount in order to test a newcomer.
- There are some general issues with epic management:
- It’s difficult to predict the trajectory of some epic’s implementation enough consistently to always have tickets ready.
- Epic budgets are often well-set and don’t allow for more intake.
- Epic ticket creation may be pushed too far back in sprint planning to have enough breathing room to think ahead and create more tickets.
Unfortunately it’s hard to recommend a course of action that will work for all epics and epic owners, so we may not focus on this angle as well as we might hope. The automation in 1 may address some of this.
- Adding a few different backlogs may be a way to handle some of the planning and make it clear where future tasks stand.
OK! So. I wanted to add a few thoughts to these:
On 2-- I think the idea of constraining recruitment to sustainability and using internal projects is probably the best idea. While it’s conceivable that we could offer a discounted discovery implementation, it strikes me as having a few issues:
- Lag time. Having a staggered pool of projects to trawl through adds lag time (and stress) to the recruitment process. If you’re searching first from old discoveries, you have to add time for people to search for them and then if that doesn’t work, fall back on the internal projects. I’ve seen @tecoholic stressing to get a project while I’ve always had one available from Listaflow, and just withhold it in hope a more billable thing is available first.
- I’m pretty sure if we do this we’re going to have a client that picks up on the fact that if they say ‘no’ to discoveries they may get what they’re looking for anyway, and may see this as a way to haggle down our pricing in an unsustainable way, which also drive us to overrecruit when fully billable hours aren’t forthcoming.
- If our sustainability is bad, we shouldn’t be recruiting anyway. A certain sustainability level should unlock recruiting. While it’s conceivable that we may not be prepared for a sudden arrival of a large project if not always recruiting, other issues are going to be greater when this happens:
- A period of unsusatainability could be caused by not enough billable hours. If this is the case, adding more talent means more billable hours are required, not less.
- If the issue is we can’t take on more projects, then it means we have too many roles/projects assigned to people and we need to drop/reprioritize them. That may mean handing off clients whose projects have gotten into maintenance mode, merging two cells, or any number of things.
- If there roles and projects don’t seem to be too numerous, but we don’t have free hours and we’re not sustainable, then our processes need work because we’re spending too much time on overhead.
On 4, (I’m not spoilering this since I know everyone has had a chance to read Braden’s comments by now), multiple backlogs seem like a solid idea. One thing that’s occasionally an issue for me is that I’ll think ‘oh, we’ll get this thing done, then this thing, so let me put tickets in this sprint and the following one’. But then needs change by the time we get to the ‘following one’ and I forgot to take it out of that sprint.
Now, this could just be a ‘me’ issue since maybe I shouldn’t be planning more than one sprint ahead, but putting tickets in the monolithic backlog makes me lose track of them more easily. This multiple backlogs idea sounds like it would help me better frame how to think about the tickets in my epics that aren’t scheduled either now or next sprint.
One other tangental note-- it looks like our recent forum thread here has helped several people who were looking for more hours find assignments, so this comment from Jill:
…Is probably correct, for newcomers or otherwise. It seems that if the issue is one of tickets needing creation/reassignment, we can certainly do it, we just need to be prompted. And maybe part of the answer is, when we fall behind, we create a thread noting extra capacity helps us coordinate that work. That’s reactive rather than proactive (creating tickets forward), but perhaps creating tickets forward isn’t happening because in many cases (as demonstrated by the numerous issues in item 3) it could be maladaptive to do so.
to this – if anyone is struggling to find work, for themselves or for newcomers, please start a thread and ask for help. It helps everybody, including our clients, to get things done sooner.
I think this needs rephrasing:
When sustainability is good, that should trigger hiring and growth.
But it shouldn’t be the only time we hire.
When we get into a sustainability hole, which happens sometimes, the only way out of that hole is to take on new client projects. But if we’re already capacity-full and we can’t hire, then we’ll burn everyone out trying to do it all within the current team.
So I think it the policy needs to be more nuanced than this. If sustainability is bad, and we need to hire to get out of it, then one of the first recruitment steps should be finding/preparing trial projects for these potential hires to take up. If there is nothing for them to do, then we have a bigger problem, but it’s better to know that before we’ve got a great candidate starting next week.
3 Likes
Well, that’s what I’m getting at-- if we’re actually at full capacity and sustainability is still bad, it’s likely to mean our processes/organization are the current issue and that adding more people will compound the problem and not bring the relief we expected. That was the thrust of the issue last year/end of the year before that-- we were getting bogged down by our processes and we thought ‘oh, we need more people’, but the real issue was that we had too many roles on too few people, process debt, a few clients we needed to hand off, and cell reorganization to do. We were losing people as soon as we hired someone else because of this, churning instead of expanding.
In fact this would suggest that non-billable hours are the better indicator of whether it’s time to hire–we use non-billables for business overhead work, automation, and documentation, but they serve one more useful function as well: They’re a ‘more elastic’ source of hours for onboarding new projects. It’s easier to pause non-billable work to recruit those hours for billable projects, allowing us to keep spare capacity for taking them on.
Thus, when our non-billable hours are dropping and our sustainability is good, then it would mean we need to hire. But it’s hard for me to imagine a scenario, after our most recent process changes (especially re: role load) where we have everyone working heavily on billable projects and yet we need to hire to hit sustainability.
There is one situation where this might be possible: We could be far behind on sustainability for reasons that are no longer active, but sign a contract to take on a very large client in a few months. In that case hiring people even in our unsustainable state would be good. But this seems like a rather exceptional situation and maybe in such cases we’d be better off making an exception to the rules rather than making it our normal mode of operating.
Maybe that’s what you mean by nuance, though-- of course I don’t consider any rule written in the handbook to necessarily be without exception-- and I hope you don’t either! We’re people, not computers. But the rules should give us pause and point us in the direction of the best habits to form so we run into fewer exceptions rather than more of them. We could put a note of the possibility next to any rule we make to remind the reader to think about it.
3 Likes
Well… we’re engineers, which is somewhere in between, especially when under stress 
1 Like
Just a quick note to bump my comment above about this - if we don’t have a good client project, we should look at assigning a small contribution to Open edX. A byte-sized ticket from the Open edX org would be best - again this has worked well in the past.
On the other end, putting newcomers on internal projects has been tried, and with poor results - it degraded the quality of Ocim’s code for example, as we didn’t end up paying enough attention to the quality of the contributions. On Open edX, this wouldn’t be a risk.
I would also object to tie recruitment to sustainability - recruitment takes a lot of time, while sustainability will always be looking back as an average over past period, independently of the work coming ahead. The current situation is a good example of this - sustainability is bad, there is even a shortage of work, but there is a lot of work coming up. And if we don’t anticipate this and recruit ahead of time, we are going to end up overcommited. To be clear, we need to keep actively recruiting now!
It was a very conscious decision to not make recruitment part of cell budgets - so we don’t end up saving on this for sustainability reasons, to the detriment of future planning. Also, if we take some upstream tickets as I suggested above, trial projects doesn’t take work away from the current team.
1 Like
Making contributions to the project is a good idea as well-- I should have added that to the list. It does present some question of how to find and pick the right projects-- yes, there’s the roadmap. But those often require a great deal of outside coordination. But honestly I’m getting ahead of things here-- @tecoholic would likely have to look into it and see if there is a good way to select such contributions now or otherwise determine (or suggest a process for) how we could find good contributions to do.
On the matter of internal projects, I feel that the situation has greatly changed and that this is not the issue it once was. Our internal products did not have product owners. But now Grove has Gabor, and Listaflow has me/Ali. Recently we had a recruit complete a contribution to Listaflow and they did it well-- we were ready to onboard them, but at the last moment they took another opportunity. Still, I was able to work with them in the process to make sure their changes would work with the overall vision of the arch, and they did make the needed adjustments as they got feedback.
Hold on here-- because this is a different evaluation to how I understand things to be. And if this is true I need to be less aggressive in scheduling the discoveries that I’ve been doing (not stopping them, of course). My evaluation of the current situation is that we DO have large amounts of work coming up, but that the large amounts of work will mostly fill existing capacity, not give us more than we can handle, and that we might even be able to take a bit more than they’re going to give us even when in full swing.
@tikr could you weigh in here? Do you have a projection on just how full we’re going to be once our slated projects are running full speed?
Thank you for explaining the reasoning behind this, as I (and may be others) have found it hard to wrap my head around this.
3 Likes
@Fox We already talked about this at today’s sprint planning meeting but here are the main points again for reference:
- There is a “worst case scenario” where existing projects would have Bebop working at capacity for a couple months (i.e., where Bebop wouldn’t have room to take on any additional projects).
- However, if that happens it shouldn’t last too long: Based on what we know right now, some projects (namely BB-7102: Video XBlock improvements, BB-7103: Improve Common Cartridge Import, BB-7121: Redis Event Bus) will not be generating a lot of hours. So they will either be done after a relatively short period of time (or take a bit longer but require only small amounts of capacity each month).
- With the current set of projects we seem to have more of a say in terms of how much capacity to allocate to them than we may have had in the past, so there’s probably some wiggle room in terms of fitting in new projects.
- On the other hand, role load is still relatively high in Bebop so if we were to add too many projects at once it would likely slow down the pace on each individual one. (And it feels like we’re at a tipping point right now where a couple of Bebop’s bigger projects – HMS Learning Paths and ASU MoE – are actually starting to gain momentum and generate more substantial amounts of work.)
Overall, it’s probably fine to keep scheduling discoveries but it might make sense to be a bit careful about promising specific amounts of capacity to new prospects and new clients.
I’m expecting that by the time that this year’s conference and coworking week are over, we’ll have more clarity on how changing capacity requirements for existing projects are impacting Bebop’s total capacity.
CC @farhaan
2 Likes
I spent a bit of time re-reading the conversation. While there is a lot of ideas about tickets and epic management, which I find useful. I am responding mostly with the view of the recruitment manager and the trial projects.
I guess the answer here is we wouldn’t know until we attempt it. We have 2 candidates joining us for the Sprint BB.295 starting March 20th. Let me try out the following:
- Go through the roadmap and identify if there are Open edX contributions that we can pick up. Try to zero in on the one that might require less external dependencies/outside coordination, like the ones that any OSS contributor could pick up. We will add some buffer time for onboarding and context setting to the trial project estimation.
- If that doesn’t work out, we will fall back to Listaflow tickets. We already have one the previous candidate failed to work on, and I hope @Fox has another one.
- If we aren’t able to find good upstream tickets for both the trial projects, we could also, do a hybrid of with 1 on each. But this is mostly just a logistical adjustment, I am just mentioning it here to have the full plan in place.
Thank you for explaining this. 
This does take away a bit of stress that I had while working on recruitment work.
After spending about an hour going through the Open edX Roadmap of issues (usually each issue is an epic) that are “in-progress”, here are my observations:
- A lot of the tickets/issues on github is usually result of some discussion that happened elsewhere
- A big chunk of the “in progress” epics are in the planning stage
- For projects that are being worked on actively, there isn’t much in terms of “backlog of issues” for us to identify
- For those which have their own sub boards - like the Copy/Paste feature - the tickets are usually assigned by whoever is going to be working on them
- Drilling further down into the projects previously closed issues like this Survery App issue does provide some hope that the byte-sized contributions are possible with some help from core contributors who are directly involved in the programs
With the above observations, some of which might be very obvious to long term contributors, I would say, identifying such contributions by someone not directly involved the platform projects would be difficult. A lot context building in the projects, I assume, happen in the planning meetings, Slack conversations…etc., which would be a tough task to parse and sort through (if it’s even possible) for a single person.
However, I think we could come up with a process with the help of core contributors. While going through the epics on the roadmap, I was able to see @kshitij 's involvement in the MFEs, @braden 's involvement in the Copy/Paste Feature, and @Agrendalath’s involvement in the XBlock Descriptor/Module System refactoring, and @jill 's involvement in the OARS data project. I am hoping CCs who are involvement in the epics would have the insight and context to propose small improvements like adding a button to the admin page.
Now, I am not sure what would be the right process for it. I believe “creating tickets in advance that would fit the Trial Project requirements” shouldn’t be their responsibility. It should be the recruitment manager’s, as it rightfully should. However, as the recruitment manager doesn’t have the context to actually create a ticket, could we do something like this?
- Assuming candidate starts in Sprint X+1, recruitment manger creates a ticket for Sprint X assigned to all the CCs, blocking, say, 30 mins.
- The CCs use that time to create a byte sized task and schedule it for the sprint X+1 based on whatever they are currently involved in.
- The recruitment manager assigns the task to the candidate(s) for Sprint X+1
- Any extra/unused tickets are either retained for future trial projects if their aren’t time sensitive, otherwise they are assigned back to the CC or a team member for sprint X+1
There are other questions which we could think about like:
- what happens if none of the CCs are able to define a trial project sized ticket
- this would add extra review time for the CCs even if they are not the internal reviewer. The availability and comfort level of the CCs to do this …etc.,
I guess we can get to them, if we consider this proposed process seriously. The reason for the proposal is that the recruitment manager still owns the responsibility of handling the trial project scheduling, and takes the help from CCs who are involved in the Open edX projects just to create the tickets. Since everything is done within a span of 2 sprints, we don’t need to have a backlog of tickets waiting for the candidate multiple sprints in advance. Looking for comments and suggestions on this.
1 Like
Oh, I forgot to add. I also went through the list of github issues for edx-platform to see if there are community reported bugs that could be used as trial projects by asking the candidates to create fixes:
- There are very few community reported bugs and issues created by people who have better context.
- The ones that I could find like this one and this require additional verification from our side. We need to be able to reproduce and define the scope of the ticket, which might lead us into a rabbit hole and add significant overhead to the process.
So, I have discounted trawling through open tickets as a plausible solution. Kindly comment if you think otherwise.
I love this idea because it gives dedicated space for team members to create the needed tickets.
One of the projects was never completed by a recruit and so I’ve re-instantiated it with the trial-project label.
I don’t think we need to verify them at all. We can have the trial project team member go through the process of setting up their devstack and attempting to reproduce the issue, communicating with the reporter to gather more details if they can’t.
My trial project was a bug in the platform not dissimilar to the ones mentioned there. I could not reproduce the bug locally. I ended up having to walk through the code and hypothesize a way that it could fail that must have been true in production, but that I wouldn’t know how to reproduce locally. Once I’d found it, I asked the reporter to verify the conditions matched, and they did. Then I created a fix that matched their preferences and got it merged in.
Verification of the bug and communication with upstream are all part of the trial. Given that, I think both of these listed issues are likely good ones.
1 Like
That’s a great approach to take. 
My concern about letting the candidate verify the issue is that, what happens if the issue becomes non-reproducible and the candidate ends up with an non-issue. From what I see from the edx-platform’s list, that seems unlikely. The issue list is well tended and there is only one open issue older than 6 months. So, may be the risk here is fairly limited.
1 Like
Thanks for looking into test tasks! The approach that you have outlined @tecoholic sounds good - also note that to get small tasks it could also be worth reaching out to other community members, including tCRIL, when we can’t find a good byte-sized task - for example via the forums, or on github. They might have ideas, and it’s generally good to raise awareness that we use those and contribute this way. Another place to look at would be the core contributor checkins on the forum, every two weeks - there is a section “help needed” which can sometimes give some potential smaller tasks, and give you names of people who are potentially willing to create tasks to get help.
@Fox Note that I wouldn’t pause on the prospect side either - just like interested prospects don’t commit until they sign, it’s the same thing on our side. We need to keep actively working with prospects on their projects, regardless of the current availability - it often takes a lot of time for projects to pan out. If we start saying we might be full, we are going to scare off prospects and dry the pipeline down the line in a few months.
Also worth noting that part of the current problem with sustainability is that we have too much overhead for our current size - when we moved a part of the team to support roles, we both reduced our ability to take on billed work and increased the size of the team required to maintain the current structure. So we just don’t need to fill the billable hours of the current team, we also need to regrow to match our reorg, and that will take both filling the time of the current team with billable hours and of new recruits.
4 Likes
Thank you for the extra ideas for finding tickets @antoviaque.