Proposition: A More Scalable OpenCraft — Improving Self Management

Hi all!

As promised, I have completed a revised proposal to address our scaling issues. This proposal’s suggestions are intense and far-reaching. It will take some time to read and chew through the ideas I’m proposing. It also contains a ‘Quick Wins’ section that we could begin almost immediately, irrespective of the larger discussion, which may take some time to complete. (Please let me know your thoughts on this so we can move forward on those, first of all.)

I expect there to be questions. Let’s dig into them!

5 Likes

Thanks, Fox. I scheduled a task to read/discuss this updated proposal in the coming week. Looking forward!

@Fox thanks for bringing this forward. It was a pleasure reading.

First of all, I want to reply to this thread mentioning my proposition wasn’t to propose current actions right now. My initial proposition was to bring forth new thoughts in addressing the problem we’re mainly suffering from at the moment, the burn-out.

The burn-out’s effects are showcased wonderfully in Fox’s proposal.

Accordingly, things such as the documentation cell and the devops cell or the recruitment cell were just examples of the idea of departmentalization, to an extent.

My goal wasn’t to achieve change from this single proposal, but to introduce the new ideas, for them to be discussed and argued (with or against). The goal was to throw something and receive the necessary feedback to take those ideas from an unpolished perspective to a more polished one, with your help.

Luckily, although my proposal was misunderstood (my bad), the next step after that was achieved. The next step was to take those argued ideas and present them into something that’s more feasible and accepted, which Fox did wonderfully.

I think the concerns I had and discussed with you in calls, Fox, were addressed in your proposal. You also brought forth ideas which I’ve always wanted to talk about, but didn’t have a good idea on how to do so.

Sure, there’s a lot of feasible suggestions and other far-reaching ones, but just as much sound. The goal of these proposals isn’t to get all the ideas accepted, but at least a good share, in order to start progressing forward again. And even if some of the ideas you raised today seem far-reaching, I have no doubt that they will rise again in the future.

It’s nice to see the main problem receiving more attention.
I look forward to seeing how your ideas will improve the state of things at OpenCraft.

5 Likes

Great to read ideas from Fox Piacenti and review thoughts from the original work of Nizar Mahmoud. Such an interesting and complicated discussion that I’ll just offer a couple of thoughts below that are limited to business development and don’t address the capacity planning or the many other issues raised.

I have observed that burnout results as much from the quality of the work being performed (too many unfulfilling, repetitive tasks) as the quantity (being busy is great if we’re doing fun work). I think two of the important functions of business development that are essential for OpenCraft to scale sustainably and maybe even minimize burnout include:

  • Prospecting and pursuing clients that fit an ideal profile (organizations oriented to open source, committed to Open edX innovation, etc.) and whom we are likely to be able to help
  • Qualifying and selecting projects, through sales discovery, that are likely to be both rewarding for OpenCraft, fun to work on and have a high likelihood of closing

If performed properly, these business development tasks reduce burnout by ensuring we’re working with the right clients and on the most attractive projects.

Regarding distributing bizdev across the cells, I’m not sure developers should assume these tasks. First, is it really their core competency? Second, wouldn’t it seem like piling on another meta task with more context switching, etc.?

I believe a dedicated bizdev team is better able to broadly consider the portfolio of current clients and projects in the pipeline to assess which new prospects represent the best possible additions, both in terms of diversification of the client base as well as the availability of sufficient resources. Distributing business development responsibilities across the cells may result in the loss of this broader perspective needed to shape the future client base and avoid suboptimal projects.

As mentioned, I don’t know the answer to the capacity planning question, whether it’s some central clearing function or having it distributed among the cells. However, I don’t think decentralizing business development is the best solution to this challenge.

1 Like

I’ve had a couple of people raise this question now, which tells me I’m not communicating my idea as effectively as I’d like. My goal is not necessarily to create new roles for cells to take on at this time. In fact, I think that would be a bad idea, especially for sales and administrative work. The example given was there to show how sales might have evolved if it had originated from the cell.

The primary thing I’m trying to illustrate is that roles designed by the cell for the cell’s needs will work better than those created by management independently. If we want to design a role we don’t ever want to hand to the cell, we’ll still want to imagine what it would look like if the role originated within the cell and use that to inform our design of the role.

To your point about qualifying leads-- I’ve placed a comment in the document I’d like your thoughts on. I’d like you to weigh in there and see what we can do re: Keeping the inflowing work stable and consistent.

1 Like

My thoughts here are copied from @Fox document to keep them in one place. With respect to the ideas from @braden@opencraft.com on Jan 28, I like his notion of building in flexibility by dividing an individual’s efforts into “active” projects (with clients) and “ongoing” (alternative name could be “building the franchise” as it’s work on the work). However, I wouldn’t lose the formal processes of planning, forecasting and estimation. Yes, the purchase cycle is lengthy, but that doesn’t mean that, over time, we can become more accurate in predicting demand for development. It just takes focus, but over time it is possible to assign probabilities to close as well as deal close dates that are close to reflecting what will actually happen. I also think these predictions are a specialized skill that is based on the observations and data an organization collects over time. For this reason it is a function that should be centralized and one that ideally rests with bizdev/sales.
So I like @braden@opencraft.com 's idea of built-in flexibility but combined with the rigor of some basic predictive analytics. Interested in your thoughts on this Fox and those of @gabriel@opencraft.com and @tim@opencraft.com.

Show less

@Fox Thanks for adding your ideas and proposals. There are several proposals, from documentation, to cells, to salaries, to sales. Maybe they could be discussed in separate places?, or maybe we could try the approach of opening MRs to the handbook.

I feel that every person has an opinion about the issues and the Google Docs right side is too small to hold large discussions. We could use the forum and separate forum threads for each topic.
To advance towards conclusions, each forum thread can have a corresponding MR.

I left some comments. From my side, I’m more interested in the style of how we fix things. The problems described in the document should be fixed but I don’t mind how they’re fixed as long as they’re fixed. I guess that not everyone wants to be involved in all discussions about processes, but they all prefer processes that work well.

My main OpenCraft wish is to have someone fixing the problems that people find. That person (or a group of people) should have power to try changes, and time, and budget, and will probably have to deal with burnout without burning out themselves.
The developer advocate role is a good way of doing of helping people if it’s implemented as originally intended :heavy_check_mark:, and if it’s kept that way.
In the document you also mention that we should go back to doing sprint retrospectives, and commenting about the problems experienced each sprint (like we did in the biweekly checkup form / stress report). These processes must work, but in the past they stopped working for several reasons, so a next step is to research why and to fix them (maybe by trying new ways of doing them).

This is all metawork because we’re trying to fix problems in the problem-fixing process in OpenCraft.

5 Likes

:sweat_smile: It ain’t easy, that’s for sure!

:+1: to this suggestion. The background/context information in @Fox 's doc provides a great perspective, but I think we’re getting caught up in discussing that, without addressing the concrete suggestions he’s made. And they’re really good! Some MRs to enact them would help move this along.

1 Like

Thank you, @daniel and @jill – you’re right that we’re starting to get the conversation fragmented. I’ve scheduled a ticket for my next sprint (next week, which coincides with the dev sprint rollover, too) to create several follow-up tasks. I’ll use these either for creating relevant MRs or else starting dedicated follow-up discussions.

In the meantime I’m going to let team members continue to trickle in and read. I’d especially be happy to see @antoviaque 's take on things. I’m out Friday, and have some things to deliver Thursday, so I know there are some threads I haven’t responded to, but I will–if not directly on the document, in the follow-up tasks.

By the way — this is a long read, a huge chunk to process, and something that will have potentially big impacts on how we work, so I definitely advise that people schedule a task to read and comment. It took me ~3h to read, process, and comment.

1 Like

Sorry for replying to this thread only now, but last month was tricky, and I wanted to make sure to have the time and attention to do justice to the work done here. Thanks a lot for that @nizarmah, @Fox and everyone else who participates to the discussions – there are lots of inspiring proposals in there, and it’s great to be able to think about those topics from new perspectives. Hopefully good will come out of this!

Cell types & priorities

It looks really useful to define types of cells according to their prioritization scheme: client-based cells correspond to the current prioritization we use across all of OpenCraft, ie “client first”. But this also means that we don’t handle our own projects well, since they always come after the client priorities - so having project-based cells who prioritize their project above all else, including internal ones, removes the interference of conflicting client priorities.

Devops cell

We definitely need more specialization and better assignation of work for DevOps, and experience has proven that the “DevOps specialist” role didn’t really solve our issues there. The risk with a “DevOps” cell though is that it makes it look like DevOps is something that is the sole responsibility of that cell… Maybe it would be less ambiguous to have that cell formed more around specific projects? Like the Grove/Ocim cell, which would be in charge of maintaining the deployment/hosting architecture – which then would in turn be used by other cells, to deploy their own clients/projects instances. All cells would still do the DevOps from their projects, and even contribute to Grove/Ocim if that serves their purposes – but there would be more specialization on the individual projects, a better sense of ownership, and less context switching.

+1 – that could be made part of the responsibilities of the same cell. I think it’s still worth having a cell specifically responsible for moving Grove/Ocim forward (as a type of project cell), but this same cell could also be responsible for establishing the best practices accross all OpenCraft DevOps.

Documentation cell

I agree with the current consensus that a whole cell for this might be overkill. However, it’s also true that maintaining and improving the documentation of OpenCraft is something that has become large and important enough to warrant affecting specialized and dedicated attention. We rely on it so much for sharing information and building our processes that we’d all benefit daily from this.

If this isn’t a full cell, maybe be a support cell role? One person assigned to it, possibly full time if the workload justifies it - and we could recruit specifically for it. Also I agree that it shouldn’t be the person writing everything - we still all need to keep writing documentation, that’s how we share our knowledge and experience. But that could be more the reviewer on all documentation, as well as the editor / refactorer of the documentation, to keep it consistent and well structured?

Recruitment

I’ll be interested in what the recruitment managers & @gabriel think about this. Given how the recruitment works, it could make sense to assign recruitment to a more specialized support role or cell, to help with reducing context switching, and providing more focus/accountability.

That said, the volume of recruitment can vary over time - I don’t think it’s ever a full time currently, and it can sometimes go back to zero. Over the past year, we have been recruiting almost non-stop, but we have often had times where we didn’t in the past, and that might come back. We might be able to pass on more work than what the recruitment managers do, by adding some of @gabriel 's work for recruitment/onboarding, and maybe some of the additional improvement work you mentioned @nizarmah (analyzing recruitment stats, improving the process, finding the best sources of candidatures, etc.) – but we need to keep in mind that there could be periods where there would not be any recruitment to do.

Project-based cells & sustainability

That’s definitely a point that needs to be considered, the consequences on how to calculate sustainability can be tricky. Maybe the cells could “buy” services from each other, through billed hours budgets? For example, the Grove/Ocim cell could “sell” its maintenance services to the cells who use hosted Open edX instances, by passing on X unbilled hours to each of those cells? Though, to a later point made by @Fox , it would add complexity to an already complex system for sustainability, and it might be worth refactoring the whole thing, taking project-based cells into account. (cf below)

Role based cells

It does look like this is similar to the support cell / deathstar.

Pursuer

I also like the fact that this offers to define a type of initiative/person that already happen, and is key to solving issues and keeping the iterative approach alive. It allows to define and provide support to those who start these initiatives – like more dedicated time, moving out of the development flow and to the support cell. This is be a disruptive change though, which impacts the responsibilities within the cell, and require non-cell budget. So to start with, I’ll want to review each new pursuer designation, based on a specific scope/mission & time budget.

Redefined sales role/process

It does makes sense that the decision to accept new projects come from each cell, it’s similar to how each cell decides on its own recruitment, based on the epic & sustainability manager predictions. I also agree that the sales role itself probably make sense to keep separate – as a delegation of each cell to a support cell, to keep it specialized. Within each cell we still need someone responsible for figuring out when sales should resume/stop; the natural role for this is likely the epic & sustainability manager?

Btw, I liked the approach to make epic/capacity planning more reliable @braden in [STAR-2224] Revisiting Approaches to Self-Management and Scaling - Google Documenten – it could actually be worth investigating on its own. Maybe you, @Fox @DouglasDraper and @tikr could take a task specifically to discuss how that could work, and create a corresponding handbook PR to experiment with it?

Removing managers

Making an organization fully decentralized, without me or Braden as backup managers, would require a lot more processes and group decisions. Just look at what happened to Zappos, Gitlab, etc. when they tried it – they ended up having to revert to a traditional hierarchy, because it was too difficult to handle edge cases, the ones the processes were not taking into account, or effects like the tragedy of the commons. And required more processes to handle that. I think our intermediate version is already providing too many processes, and this would only add to it. I’m keeping the BDFL status, because it’s an important failsafe.

I do agree though that there are different parts of self-management, and I definitely want to keep going in the direction of giving more latitude, decision power and autonomy to the person who do the work. It always feels like a win when a topic has been fully delegated to the team as a whole, and I don’t need to intervene or take care of it. It’s probably true that we (I?) have erred a bit too much on the side of ensuring that things keep happening, by adding too many rules or checklist items, so relaxing rules to replace them by personal judgement seem like an important next step. I’ve also replied inline to some of the elements related to this in the document.

Handbook - Separating core & cell-specific rules

+1 to this, it would be good to allow for more differentiation/experimentation within local contexts. Actually the original version of the handbook contained sections for cell-specific rules, but those were never used. It’s likely right that it’s a result of the centralized way we have come up with processes yes, and the fact that until fairly recently I was often the one driving most of the large processes changes.

To keep things sane though, rather than having multiples branches of the handbook for each cell, I’d simply have sections in the handbook describing the special rules of each cell. I would also consider the “core” rules as a sort of higher priority legal text, like a constitution - the “local” rules can’t contradict the core rules, which would be the part kept consistent across OpenCraft; this would mean that we need to keep the core rules minimal, and that doesn’t over-specify things, to keep that flexibility at the cell level.

We should also use the opportunity to differentiate parts of the handbook which are mandatory rules, and the ones which are advice - that’s a point @daniel brought up in the past that would be helpful to make the “core” set lighter, especially.

Sustainability

It’s true that sustainability calculations are tricky, and currently have many issues, including many blocked projects because of it. It would be good to get an idea of what would replace it, exactly - because that’s something that is hard to delegate the responsibility for to the cells in the first place. That seem like a topic big enough for its own thread and project though. :)

Exposing more of the finances for this seem overkill and going to the opposite of our goals - it would create even more metawork, and won’t make it easier to take decisions for budgets - on the contrary. I would keep the approach of using hours as a proxy for $, and find a simpler way to make it apparent when the number of hours billed allow us to afford the internal projects that we want.

Compensation

We already had a pretty lengthy conversation about this recently, I don’t want to reopen this for some time. The solution might not be perfect for everyone, but no solution is ever going to be perfect, and we are always going to have some people who want or need more money - and we can’t always say yes, or fix the processes to keep them. That’s life, and trying to keep tuning this will just remain a large distraction.

Task force

That also sounds like a project cell to me. I think what you’re saying is that the project cell could be made lighter - and we can indeed look at ways to do that, some of the proposals above help with this (by pushing out things like recruitment). But it’s also not a coincidence that the project cells still have a certain amount of process and responsibilities – someone still need to plan for capacity, deal with the client if there is one, etc. Maybe the best on this would be to look concretely at the responsibilities/processes that we want to remove from project cells, and see if that doable & how.

Quick wins

  • I agree that resuming the retrospectives is important, +1
  • We do need to do something about the number of core developers in Serenity; I’m not a fan of moving people between cells, which is really disruptive, but it is an extreme case - unless we will soon have newcomers confirmed?
  • For sales, cf above, it makes sense to move forward in that direction, as long as we don’t add new roles/metawork to cells

Next steps

There are LOTS of things being discussed here, and if we try to pursue everything at once, it will be overwhelming. A good next step would indeed be to start discussing on concrete merge requests. I would pick 2-3 of these points max (maybe even one to start?), see it through, and then come back to this list.

Which ones would that be?

6 Likes

I’m really glad to see such positive replies on this thread. I just want to post a small request; please do your best to keep the split discussions public on the forum. I hope I’ll have the opportunity to see how things progress. :crossed_fingers:

Also, massive thanks to anyone who put time towards this thread or is planning to. :hugs:

4 Likes
My choices (hidden so you can think of your own priorities before reading mine)

I would personally suggest, in order:

  1. DevOps cell - establish devops best practices for the team and proactively solve problems (upgrade Ubuntu 16.04!)
  2. Make capacity planning more reliable
  3. Simplify/refactor sustainability calculations
3 Likes

First of all, Thank you, @antoviaque for your thoughtful responses here. After reading Reinventing Organizations, it put in front of me a model that was both reminiscent and also distinct from what we’re currently doing.

I wonder if long-term we can ever embody the ideals presented in the book fully. The ‘obvious’ solution after reading it was to dive in as deep as I could and see where that took me. So, I wrote the proposal around that.

I knew that even if we couldn’t fully implement the archetypal model it would set the stage for useful discussion and give us a chance to re-examine much. And boy, has it. I feel both that I better understand the boundaries of what we can do and what we are willing to do, and what is most important to the team. I don’t think I could have done that without an idealized target to start with.

Aside from the quick wins, the items that Braden mentioned would also be my priorities. During my sprint this week I’ll perform the prepwork here for moving these forward, especially on the front of creating follow-up tasks and getting them scheduled.

4 Likes

@Fox Yes, we took inspiration from Reinventing Organizations, but the goal isn’t to implement it literally – more to take the pieces that are useful, but still come up with our own sauce. The literal implementation seem to be working for some, but not for others, and it looks like there is a quite significant amount of cases where it didn’t. I suspect some of it is related to the type of industry, and maybe on the predictability of the work & processes.

Thanks btw for moving the overall discussion forward – and the next steps @braden and you have prioritized look good to me :+1:

I’m a bit late to the party, but I’ve read through the documents and thread and left a few comments.


My choices
  1. Simplify sustainability

See my comments on:
[STAR-2224] Revisiting Approaches to Self-Management and Scaling - Google Documenten

  1. DevOps Task Force

Have a dedicated group of people responsible for maintaining and pushing DevOps tasks forward, until we catch up with the tech debt and then maybe bring back the DevOps Assignee (to keep the responsibility in one place).

3 Likes

I have created and scheduled:

  • Log in - OpenCraft For revising the retrospective process
  • Log in - OpenCraft For revising the sales process
  • Log in - OpenCraft For pre-work/discussion on a suggested solution for the DevOps cell brought up by @farhaan (Merging Bebop and Serenity, and splitting off the DevOps cell in the process). Forum thread here. Currently private since we might need to discuss client details for handoffs or what have you, though maybe that won’t happen and we’ll be able to make it public.
  • Log in - OpenCraft has been created and scheduled for refactoring sustainability/capacity planning.

I think that’s enough to get us started. Yay for progress!

6 Likes

@Fox Thanks!

What about “2. Make capacity planning more reliable” – is that something that would be handled separately?

Also, including that change, that would get us to 5 changes handled simultaneously – even maybe 6 considering that we are discussing implementing the recruitment changes in https://forum.opencraft.com/t/hasta-la-vista-leaving-in-february/1234/17 . That might be a bit too much to chew at once, especially given the size and impact of some of these changes? Prioritizing those is hard, as they are all good changes, but like with code doing too much at once runs the risk of taking longer with everything. Could we put a hard limit to 3 at once? As soon as one is done, we can start another one, but I think it will quickly become overwhelming otherwise.

2 Likes

Log in - OpenCraft is required for doing that, I think, since it’s going to change what figures we use for planning. That alone will likely improve our forecasting some, but I figure follow-up should wait until we know what we’re dealing with.

I’ve already staggered the tickets a few sprints, but I could, say, kick revising the sales process back since we’re not doing much selling right now anyway. I do think the research for cell split/merge for DevOps needs to be top priority and the Sprint retrospective is an easy thing to change/add to the handbook. Those can be done next week-- which is why I’ve scheduled them then.

Tim wrote up a timeline of the tickets I created so you can see how they’re spaced. If you still think these are a bit too tight together, I’ll kick the sales process and sustainability changes forward a bit more.

@Fox Alright – sounds good then. Let’s just remain mindful of this too then, and reconsider if it starts being too many changes at once.

2 Likes