Could you document this change in the handbook, e.g. in the Epic Management: epic template? If it’s already documented somewhere else, then even a link there would help make this feature more discoverable when people are creating new epics.
I wonder… if we’ve got a completed discovery that the client rejected due to budget constraints, could we offer to assign the work to a newcomer split the cost with the client? If we’re clear up front about what we’re promising, then we could balance the risks for everyone, e.g.
The client knows this is being done by a candidate, and so accepts that it may take longer than originally expected, and may require more back-and-forth to get the features right.
We could quote a fixed cost to the client but just a tentative timeline, so their risk is time/quality, not cost.
If the newcomer doesn’t end up completing the task, then OpenCraft bears the sustainability burden of finishing the work when our capacity allows.
The risk to our sustainability isn’t as high as it would be for a fully internal project.
Scope changes would need to be individually negotiated – newcomers will need a lot of help from their mentors/task reviewers to identify and manage these, since they’re under pressure to please all sides. But even that would also give us insight into the newcomer’s ability to manage shifting requirements and communication, all of which are good things to see during a trial.
@jill If this isn’t likely to scare off the client, why not yes
Also, for trial projects, remember that if nothing is available a fallback is to take a ticket from the Open edX project and get the candidate to contribute it upstream. It takes more unbilled time than a client project, but with recruitment speed is important - the more time between the moment a candidate applies and the actual hiring, the more likely they will have found something else (which we have seen apply to recent candidates btw). Also, contributing a task upstream used to be our main recruitment test, and it worked quite well.
I would like that It’d be great to have a place to dump ideas without having to fully spec them out.
That could be useful at times when we’re actively looking for tickets, but I don’t feel it’s worth making this a regular task. I’d leave it up to the epic owner oh how clean they want to keep their backlog of tickets.
Other than the reasons others have already mentioned, one of the problems is that tickets where priority and budget align perfectly for a trial project are very rare. Tickets for which we have budget are usually high enough priority that we can’t just keep them in the backlog for many sprints. It’s a bit better with internal projects, but I think this problem happens with internal projects as well.
There are some amazing ideas here and the problems that I am going to be pointing out about creating forward tickets could be just related to specific epic.
The Epic that I own has a severe budget restriction, so scheduling forward tickets is not something that I have the luxury for, at least for now. Having said that, we did get some budget to reduce the code drift, which can be a good candidate to try creating forward tickets for.
Having said that, with ongoing projects, there are times when you don’t really have much development work going. Under such Epics, creating Trial Projects/Forward tickets is not possible.
At times projects are time critical, and it can be unhealthy for the project to on-board a newcomer, I often have this dilemma that I shouldn’t be spending client’s time/money for our benefit. When I think through this, I do land on the premise that it could be because that tasks are not well detailed, or the discovery wasn’t able to do justice to the Epic Plan. Hence, it becomes a bit more difficult to give a task which has uncertainties to the newcomer.
I think most of what I have to say about this is pretty-much already covered by everyone else, but I decided to go through my own history of epic to analyse the situation with those.
For Yonkers, either we had project where all possible tickets were created in advance, or general support tickets, which were created when an issue was brought up, and scheduled when we were told to do so.
I think this has been a major reason for me. For many projects I’ve seen that creating tickets too early isn’t very helpful. Not all tasks are workable at all times, and sometimes the definition of a future task depends on work that’s yet to be done.
I’m not entirely convinced of budget as a reason, since that prevents scheduling tickets, not creating them, although creation also takes time.
I think for such tickets it would generally be OK to ping the assignee, and reassign, I’ve seen this happen a number of times, but it’s certainly an extra step.
I agree! I think aligning a trial project with a ready nice bundle of related tickets that a newcomer can take is going to be hard at best.
I’ve generally not coupled ticket creation with the end of the sprint, I’ve always thought of it as a deadline, so I try to create the tickets before that, whatever that day may be.
I just wanted to take a moment to summarize a few of the points here and both see what has been done and what might still be done based on the suggestions:
We’ve gotten the automation fixed surrounding strategic planning tickets, which can help both with spotting new potential epics and for generating them.
Trial projects are proving especially challenging to produce. The following ideas have been put forth for them:
Keep them to internal projects and then only perform recruitment when our sustainability is in a good spot.
Offer to perform rejected, discovered work at a discount in order to test a newcomer.
There are some general issues with epic management:
It’s difficult to predict the trajectory of some epic’s implementation enough consistently to always have tickets ready.
Epic budgets are often well-set and don’t allow for more intake.
Epic ticket creation may be pushed too far back in sprint planning to have enough breathing room to think ahead and create more tickets.
Unfortunately it’s hard to recommend a course of action that will work for all epics and epic owners, so we may not focus on this angle as well as we might hope. The automation in 1 may address some of this.
Adding a few different backlogs may be a way to handle some of the planning and make it clear where future tasks stand.
OK! So. I wanted to add a few thoughts to these:
On 2-- I think the idea of constraining recruitment to sustainability and using internal projects is probably the best idea. While it’s conceivable that we could offer a discounted discovery implementation, it strikes me as having a few issues:
Lag time. Having a staggered pool of projects to trawl through adds lag time (and stress) to the recruitment process. If you’re searching first from old discoveries, you have to add time for people to search for them and then if that doesn’t work, fall back on the internal projects. I’ve seen @tecoholic stressing to get a project while I’ve always had one available from Listaflow, and just withhold it in hope a more billable thing is available first.
I’m pretty sure if we do this we’re going to have a client that picks up on the fact that if they say ‘no’ to discoveries they may get what they’re looking for anyway, and may see this as a way to haggle down our pricing in an unsustainable way, which also drive us to overrecruit when fully billable hours aren’t forthcoming.
If our sustainability is bad, we shouldn’t be recruiting anyway. A certain sustainability level should unlock recruiting. While it’s conceivable that we may not be prepared for a sudden arrival of a large project if not always recruiting, other issues are going to be greater when this happens:
A period of unsusatainability could be caused by not enough billable hours. If this is the case, adding more talent means more billable hours are required, not less.
If the issue is we can’t take on more projects, then it means we have too many roles/projects assigned to people and we need to drop/reprioritize them. That may mean handing off clients whose projects have gotten into maintenance mode, merging two cells, or any number of things.
If there roles and projects don’t seem to be too numerous, but we don’t have free hours and we’re not sustainable, then our processes need work because we’re spending too much time on overhead.
On 4, (I’m not spoilering this since I know everyone has had a chance to read Braden’s comments by now), multiple backlogs seem like a solid idea. One thing that’s occasionally an issue for me is that I’ll think ‘oh, we’ll get this thing done, then this thing, so let me put tickets in this sprint and the following one’. But then needs change by the time we get to the ‘following one’ and I forgot to take it out of that sprint.
Now, this could just be a ‘me’ issue since maybe I shouldn’t be planning more than one sprint ahead, but putting tickets in the monolithic backlog makes me lose track of them more easily. This multiple backlogs idea sounds like it would help me better frame how to think about the tickets in my epics that aren’t scheduled either now or next sprint.
One other tangental note-- it looks like our recent forum thread here has helped several people who were looking for more hours find assignments, so this comment from Jill:
…Is probably correct, for newcomers or otherwise. It seems that if the issue is one of tickets needing creation/reassignment, we can certainly do it, we just need to be prompted. And maybe part of the answer is, when we fall behind, we create a thread noting extra capacity helps us coordinate that work. That’s reactive rather than proactive (creating tickets forward), but perhaps creating tickets forward isn’t happening because in many cases (as demonstrated by the numerous issues in item 3) it could be maladaptive to do so.
to this – if anyone is struggling to find work, for themselves or for newcomers, please start a thread and ask for help. It helps everybody, including our clients, to get things done sooner.
I think this needs rephrasing:
When sustainability is good, that should trigger hiring and growth.
But it shouldn’t be the only time we hire.
When we get into a sustainability hole, which happens sometimes, the only way out of that hole is to take on new client projects. But if we’re already capacity-full and we can’t hire, then we’ll burn everyone out trying to do it all within the current team.
So I think it the policy needs to be more nuanced than this. If sustainability is bad, and we need to hire to get out of it, then one of the first recruitment steps should be finding/preparing trial projects for these potential hires to take up. If there is nothing for them to do, then we have a bigger problem, but it’s better to know that before we’ve got a great candidate starting next week.
Well, that’s what I’m getting at-- if we’re actually at full capacity and sustainability is still bad, it’s likely to mean our processes/organization are the current issue and that adding more people will compound the problem and not bring the relief we expected. That was the thrust of the issue last year/end of the year before that-- we were getting bogged down by our processes and we thought ‘oh, we need more people’, but the real issue was that we had too many roles on too few people, process debt, a few clients we needed to hand off, and cell reorganization to do. We were losing people as soon as we hired someone else because of this, churning instead of expanding.
In fact this would suggest that non-billable hours are the better indicator of whether it’s time to hire–we use non-billables for business overhead work, automation, and documentation, but they serve one more useful function as well: They’re a ‘more elastic’ source of hours for onboarding new projects. It’s easier to pause non-billable work to recruit those hours for billable projects, allowing us to keep spare capacity for taking them on.
Thus, when our non-billable hours are dropping and our sustainability is good, then it would mean we need to hire. But it’s hard for me to imagine a scenario, after our most recent process changes (especially re: role load) where we have everyone working heavily on billable projects and yet we need to hire to hit sustainability.
There is one situation where this might be possible: We could be far behind on sustainability for reasons that are no longer active, but sign a contract to take on a very large client in a few months. In that case hiring people even in our unsustainable state would be good. But this seems like a rather exceptional situation and maybe in such cases we’d be better off making an exception to the rules rather than making it our normal mode of operating.
Maybe that’s what you mean by nuance, though-- of course I don’t consider any rule written in the handbook to necessarily be without exception-- and I hope you don’t either! We’re people, not computers. But the rules should give us pause and point us in the direction of the best habits to form so we run into fewer exceptions rather than more of them. We could put a note of the possibility next to any rule we make to remind the reader to think about it.
Just a quick note to bump my comment above about this - if we don’t have a good client project, we should look at assigning a small contribution to Open edX. A byte-sized ticket from the Open edX org would be best - again this has worked well in the past.
On the other end, putting newcomers on internal projects has been tried, and with poor results - it degraded the quality of Ocim’s code for example, as we didn’t end up paying enough attention to the quality of the contributions. On Open edX, this wouldn’t be a risk.
I would also object to tie recruitment to sustainability - recruitment takes a lot of time, while sustainability will always be looking back as an average over past period, independently of the work coming ahead. The current situation is a good example of this - sustainability is bad, there is even a shortage of work, but there is a lot of work coming up. And if we don’t anticipate this and recruit ahead of time, we are going to end up overcommited. To be clear, we need to keep actively recruiting now!
It was a very conscious decision to not make recruitment part of cell budgets - so we don’t end up saving on this for sustainability reasons, to the detriment of future planning. Also, if we take some upstream tickets as I suggested above, trial projects doesn’t take work away from the current team.
Making contributions to the project is a good idea as well-- I should have added that to the list. It does present some question of how to find and pick the right projects-- yes, there’s the roadmap. But those often require a great deal of outside coordination. But honestly I’m getting ahead of things here-- @tecoholic would likely have to look into it and see if there is a good way to select such contributions now or otherwise determine (or suggest a process for) how we could find good contributions to do.
On the matter of internal projects, I feel that the situation has greatly changed and that this is not the issue it once was. Our internal products did not have product owners. But now Grove has Gabor, and Listaflow has me/Ali. Recently we had a recruit complete a contribution to Listaflow and they did it well-- we were ready to onboard them, but at the last moment they took another opportunity. Still, I was able to work with them in the process to make sure their changes would work with the overall vision of the arch, and they did make the needed adjustments as they got feedback.
Hold on here-- because this is a different evaluation to how I understand things to be. And if this is true I need to be less aggressive in scheduling the discoveries that I’ve been doing (not stopping them, of course). My evaluation of the current situation is that we DO have large amounts of work coming up, but that the large amounts of work will mostly fill existing capacity, not give us more than we can handle, and that we might even be able to take a bit more than they’re going to give us even when in full swing.
@tikr could you weigh in here? Do you have a projection on just how full we’re going to be once our slated projects are running full speed?
@Fox We already talked about this at today’s sprint planning meeting but here are the main points again for reference:
There is a “worst case scenario” where existing projects would have Bebop working at capacity for a couple months (i.e., where Bebop wouldn’t have room to take on any additional projects).
However, if that happens it shouldn’t last too long: Based on what we know right now, some projects (namely BB-7102: Video XBlock improvements, BB-7103: Improve Common Cartridge Import, BB-7121: Redis Event Bus) will not be generating a lot of hours. So they will either be done after a relatively short period of time (or take a bit longer but require only small amounts of capacity each month).
With the current set of projects we seem to have more of a say in terms of how much capacity to allocate to them than we may have had in the past, so there’s probably some wiggle room in terms of fitting in new projects.
On the other hand, role load is still relatively high in Bebop so if we were to add too many projects at once it would likely slow down the pace on each individual one. (And it feels like we’re at a tipping point right now where a couple of Bebop’s bigger projects – HMS Learning Paths and ASU MoE – are actually starting to gain momentum and generate more substantial amounts of work.)
Overall, it’s probably fine to keep scheduling discoveries but it might make sense to be a bit careful about promising specific amounts of capacity to new prospects and new clients.
I’m expecting that by the time that this year’s conference and coworking week are over, we’ll have more clarity on how changing capacity requirements for existing projects are impacting Bebop’s total capacity.
I spent a bit of time re-reading the conversation. While there is a lot of ideas about tickets and epic management, which I find useful. I am responding mostly with the view of the recruitment manager and the trial projects.
I guess the answer here is we wouldn’t know until we attempt it. We have 2 candidates joining us for the Sprint BB.295 starting March 20th. Let me try out the following:
Go through the roadmap and identify if there are Open edX contributions that we can pick up. Try to zero in on the one that might require less external dependencies/outside coordination, like the ones that any OSS contributor could pick up. We will add some buffer time for onboarding and context setting to the trial project estimation.
If that doesn’t work out, we will fall back to Listaflow tickets. We already have one the previous candidate failed to work on, and I hope @Fox has another one.
If we aren’t able to find good upstream tickets for both the trial projects, we could also, do a hybrid of with 1 on each. But this is mostly just a logistical adjustment, I am just mentioning it here to have the full plan in place.
Thank you for explaining this. This does take away a bit of stress that I had while working on recruitment work.
After spending about an hour going through the Open edX Roadmap of issues (usually each issue is an epic) that are “in-progress”, here are my observations:
A lot of the tickets/issues on github is usually result of some discussion that happened elsewhere
A big chunk of the “in progress” epics are in the planning stage
For projects that are being worked on actively, there isn’t much in terms of “backlog of issues” for us to identify
For those which have their own sub boards - like the Copy/Paste feature - the tickets are usually assigned by whoever is going to be working on them
Drilling further down into the projects previously closed issues like this Survery App issue does provide some hope that the byte-sized contributions are possible with some help from core contributors who are directly involved in the programs
With the above observations, some of which might be very obvious to long term contributors, I would say, identifying such contributions by someone not directly involved the platform projects would be difficult. A lot context building in the projects, I assume, happen in the planning meetings, Slack conversations…etc., which would be a tough task to parse and sort through (if it’s even possible) for a single person.
However, I think we could come up with a process with the help of core contributors. While going through the epics on the roadmap, I was able to see @kshitij 's involvement in the MFEs, @braden 's involvement in the Copy/Paste Feature, and @Agrendalath’s involvement in the XBlock Descriptor/Module System refactoring, and @jill 's involvement in the OARS data project. I am hoping CCs who are involvement in the epics would have the insight and context to propose small improvements like adding a button to the admin page.
Now, I am not sure what would be the right process for it. I believe “creating tickets in advance that would fit the Trial Project requirements” shouldn’t be their responsibility. It should be the recruitment manager’s, as it rightfully should. However, as the recruitment manager doesn’t have the context to actually create a ticket, could we do something like this?
Assuming candidate starts in Sprint X+1, recruitment manger creates a ticket for Sprint X assigned to all the CCs, blocking, say, 30 mins.
The CCs use that time to create a byte sized task and schedule it for the sprint X+1 based on whatever they are currently involved in.
The recruitment manager assigns the task to the candidate(s) for Sprint X+1
Any extra/unused tickets are either retained for future trial projects if their aren’t time sensitive, otherwise they are assigned back to the CC or a team member for sprint X+1
There are other questions which we could think about like:
what happens if none of the CCs are able to define a trial project sized ticket
this would add extra review time for the CCs even if they are not the internal reviewer. The availability and comfort level of the CCs to do this …etc.,
I guess we can get to them, if we consider this proposed process seriously. The reason for the proposal is that the recruitment manager still owns the responsibility of handling the trial project scheduling, and takes the help from CCs who are involved in the Open edX projects just to create the tickets. Since everything is done within a span of 2 sprints, we don’t need to have a backlog of tickets waiting for the candidate multiple sprints in advance. Looking for comments and suggestions on this.
Oh, I forgot to add. I also went through the list of github issues for edx-platform to see if there are community reported bugs that could be used as trial projects by asking the candidates to create fixes:
There are very few community reported bugs and issues created by people who have better context.
The ones that I could find like this one and this require additional verification from our side. We need to be able to reproduce and define the scope of the ticket, which might lead us into a rabbit hole and add significant overhead to the process.
So, I have discounted trawling through open tickets as a plausible solution. Kindly comment if you think otherwise.
I love this idea because it gives dedicated space for team members to create the needed tickets.
One of the projects was never completed by a recruit and so I’ve re-instantiated it with the trial-project label.
I don’t think we need to verify them at all. We can have the trial project team member go through the process of setting up their devstack and attempting to reproduce the issue, communicating with the reporter to gather more details if they can’t.
My trial project was a bug in the platform not dissimilar to the ones mentioned there. I could not reproduce the bug locally. I ended up having to walk through the code and hypothesize a way that it could fail that must have been true in production, but that I wouldn’t know how to reproduce locally. Once I’d found it, I asked the reporter to verify the conditions matched, and they did. Then I created a fix that matched their preferences and got it merged in.
Verification of the bug and communication with upstream are all part of the trial. Given that, I think both of these listed issues are likely good ones.
My concern about letting the candidate verify the issue is that, what happens if the issue becomes non-reproducible and the candidate ends up with an non-issue. From what I see from the edx-platform’s list, that seems unlikely. The issue list is well tended and there is only one open issue older than 6 months. So, may be the risk here is fairly limited.