Recruitment: Trial projects & bounties

Discussions - Public
1

Trial projects

As part of the changes to the trial process for newcomers, we are introducing the concept of “trial projects”, meant to replace the trial period, and allow us (and the candidate) to conclude the evaluation sooner. This is important to accelerate the recruitment process, but can only work if we are actually able to properly evaluate a newcomer once the trial project is over, with a similar level of accuracy as what we have now for the end of trial review. Which in turns means that we need good trial projects - so it’s worth spending some time brainstorming this together.

For trial projects to work, as noted on the merge request review for the recruitment process change, the project can’t just be a “big newcomer friendly task”, or we won’t have enough to judge during review:

The project specifically shouldn’t be “newcomer friendly”. That’s a part I liked from @bradenm 's proposal - we used to test newcomers on something difficult immediately. Good senior developers who will fit within the team are able to take this on – we regularly have good examples of this. On the other side, the “newcomer friendly” category we came up with helped some newcomers being less lost at the beginning, but that tends to only delay finding out they can’t take on harder work later on. If we want to be able to provide the same level of judgement on a 20h trial project as we do after 2 months of work, this trial project needs to be at least as hard as what we ask of newcomers now to approve them as core team members.

So we need large tasks that require significant work and skill from the candidate – including managing to acquire context in a short amount of time – ie a 8 point ticket? The projects also need to be real work, but work that doesn’t have a client deadline during the current sprint – this way, if the candidate fumbles, it can still be assigned to someone else the following sprint, without anyone having to suddenly take over a lot of work as a sprint injection.

Not easy, isn’t it? What do you think some good projects or topics for these types of tasks can be?

In any case, that would likely require thoughts from epic owners to find the right tasks - possibly including more advance planning with clients, to have more work lined up earlier, which we could then safely pass on to candidates. On the plus side, epics who are currently having trouble completing enough work due to capacity issues could get a boost, and in a way that’s safe budget-wise (since if the candidate doesn’t work out and the work has to be taken over, we’ll simply log the time on a separate recruitment budget).

What else? Are there epics owners willing to help feeding some trial projects?

@shimulch @jvdm @raul since you have tasks about this for this sprint, could you coordinate figuring out the epics & projects we’ll try out from your cell, within the current thread? CC @braden since you’re reviewing those.

Bounties

Another related topic is the follow-up on the bounties we discussed some time ago. It was within the context of providing a way for candidates without contribution experience to satisfy the requirement to have a merged upstream contribution before being considered. It could potentially be another source of trial projects.

Another application for bounties could be to provide a way to open up to work with more junior developers. A lot of the tickets generated by dependencies & release upgrades upstream are small and simple. Currently we contribute our share through the release upgrade epics & core committer time, but that’s probably overkill – it could maybe be better to offer those to candidates willing to spend some (paid) time becoming familiar with the platform, contributing open source code, and generally improving their skills. More junior developers could take those tickets on an ongoing basis, until they have acquired enough experience to consider tackling bigger tasks – like our trial projects?

CC @nizar since this could end up being linked to the community liaison role, to list upstream issues for bounties & coordinating upstream reviews (which we should participate in as core contributors).

[ Ticket to log time discussing in the current thread ]

3 Likes
2

Regarding the bug bounties, sounds awesome :+1:

I’d be happy to create a ticket to follow up on bounties. I’ll give a chance for the discussions to unfold, first, though.

I believe there are many boards, already, in the community that contain the different bugs available.

1 Like
3

My original suggestion was this: We create a queue of trial projects per cell. Every recruitment round, the Hiring Manager pop projects from this queue as needed by his recruitment activity (in the onboarding phase, after the candidate passed the interview). After the round, the Hiring Manager reports back projects that were not used, so they can be assigned and finished in the next sprint.

Primarily, it would be the cell Epic Owners’ responsibility to identify and assign trial projects to the trial project queue. Epic Owners are the individuals closest to the client, aware of the scope and planning, and have the best information to make the distinction of what is a good trial project and what is not, that is, projects that (a.) satisfy scope, (b.) can safely spillover to the next sprint, and (c.) can fail on budget (recruiting would be the buffer). Maybe (a.) is something that every Opencraft member can do, but Epic Owner will probably know better since he has the technical background, knows the project and the client. If we don’t have a clear definition of what is a good trial project, I think we should immediately create one before anything else.

Now, at the beginning of the recruiting round, the Hiring Manager determines the desired number of newcomers (the same process as today) minus the number of available trial projects. Once a newcomer is successful at the interviews, the first project in the queue is assigned to the newcomer, as part of his onboarding. The hiring manager finds a reviewer, which is the equivalent of the mentor. Any successful candidate, that is, candidates that have passed the interview but don’t have a trial project yet for them, wait in the Hiring Manager queue until there are more trial projects available. It’s the Hiring Manager’s responsibility to prioritize and manage the queue of candidates. It’s the Epic Owner’s responsibility to populate the cell’s Trial Project queue.

At the end of the round, I think it would be a good idea for the Epic Owner and Hiring Manager to coordinate on what projects were assigned. That means if a project was pushed to the trial project’s queue but not assigned in this round, it should be assigned to a regular core member so the customer won’t be impacted. This brings us back to the point that, trial projects are OK to fail or delay for one sprint. It is guaranteed that they will be done in the second since a core member will grab it out of the queue if not used.

(Oh, and it sounds like Epic Owners will need some budget on their own to sort out Trial Projects and assign them to the queue. I believe recruiting should be the account to log that and not use the customer’s budget.)

4

There is an initial definition in the PR:

A trial project should have following characteristics:

  • Trial project should take ~20h to complete for a candidate.
  • Client tasks
    • Low priority client tasks that have enough buffer between deadline and the start of the recruitment round (at least 2 sprints).
    • Should not require any extra client specific access. Access should be limited to client fork/template/app repositories as much as possible.
  • A newcomer in onboarding period should not be the reviewer of a trial project.

For the record, here was my test task back in 2014 :) https://github.com/edx/edx-platform/pull/4868 . I’d say it was a little too easy though, it took me 3.5 hours to setup and learn the devstack (simpler in those days) and 5 hours to do the fix + revisions + two follow up PRs to fix other bugs.

I know what you’re saying, but we can’t have something like this fall to “epic owners”, because that essentially means “everyone on the cell” as everyone owns an epic. For now we need the single responsibility principle, so could each of @shimulch @jvdm @raul be responsible for [talking to the epic owners and] selecting one or two initial trial projects here for your cell? We just need someone to chase people down and give suggestions and narrow down the list until we have some options.

5

It’s is essential to define what you mean by “something like this”. Although we want a single responsibility, there are different types of responsibility and the best people to take them.

One responsibility is evaluating the epic scope, planning, budget to identify trial projects for the next sprint (which can be none, of course). The other responsibility is to capture the available projects and use them in the Hiring Process. My point is, Epic Owners are the right people for taking care of the first. Hiring Magers the second. Now, if you want Hiring Managers to make sure Epic Owners are doing the first responsibility, that’s OK. I mean, I would rather mechanize this to a maximum because, honestly, a per-sprint reminder of some sort could do the job. But yeah, Hiring Managers can ping Epic Owners that did not respond to the call for Trial Projects.

In contrast, if you want Hiring Managers to take the first responsibility, there is a risk that they will not select the most appropriate projects by themselves. I admit I cannot measure what is the size of the risk right now. But, in any case, we would have one person (the HM) taking on this responsibility, but they will probably put the Epic Owners in the loop anyway to get feedback on their choices. That is a duplication of effort and goes against the principle you are attaining for. It would be more effective, in my opinion, to have Epic Owners select tasks that a suitable for Trial Projects, send them to the Hiring Manager, and let Hiring Managers process them in the Hiring Process, notifying Epic Owners which tasks are going to be used or not.

6

Glad to hear that. Thank you for posting the reference.

7

For Serenity, I’ve created this task and I will invite Epic Owners to identify Trial Projects on their epics roadmap in the next Sprint.

8

Would this be resolved by the new project-based trials? They are supposed to be projects we’d give to full fledged core members as a one-off evaluation.

9

I guess, I just anticipate a lot of meta-work in getting a project out there that is self-contained and can be taken up by a newcomers.

I can give entire discussions milestones, that would be great, but that needs knowledge of:

  • Mongo
  • Ruby
  • edx-platform
  • react and Open edX MFE

On one hand a pretty broad project, but also one that might go beyond a lot of newcomers’ skill-sets.

10

+1 to give this type of project as trial projects. As noted in Recruitment: Trial projects & bounties we want projects which will go beyond the capabilities of some newcomers – that’s precisely what we want to test them on: can they take it? If not, then we need to find someone else. The key to not jeopardize your epic’s work is to have them work on it in advance, and to not hesitate to discard the work & reassign it to someone else in a following sprint if the result isn’t there (and to move the worklogs to the recruitment budget in that case).

@kshitij Could you post about this in Recruitment: Trial projects & bounties to get the ball rolling? Maybe see if there is one or two projects like that that you can give to some of the current newcomers?

11

I did try this, but the issue I’m facing is that it isn’t a 20-hour project. It’s larger. Milestones 1.11 and 1.12 are pretty self-contained, they touch a lot of stuff, but I also don’t want to discard a good developer just because they aren’t familiar with Ruby, most people here aren’t! So what do I do in that case? Do I give the ruby task to someone else, or expect the developer to learn ruby and complete the task in 20hrs? Which is very optimistic.

12

@kshitij What volume of work are we talking about? Ie what’s the smallest project like that you would be able to design? We are testing things out, so we can try various sizes and see what we think of the result – especially since for now it would be done by newcomers, rather than candidates with potentially already another job. And even then, it just has to be reasonable enough to be doable by someone who already has a job.

NB: I’ve moved the discussion to the current thread, since it’s more related to the discussion here.

13

I’ll chalk out some time next sprint so create a project like this. I think the idea is to keep it flexible. By which I mean:

  • If we have a newcomer who knows ruby, then the ruby + edx-platform change could be roughly 20 hrs.
  • If the newcomer doesn’t know ruby, but knows enough about the frontend, then the edx-platform change + some frontend change could be 20 hrs.
  • If the newcomer doesn’t have enough knowledge of the frontend or ruby, then this is hard to fit in 20hrs.
1 Like
14

Yes, that’s all I’m asking. If I just post on the forums and say “epic owners, please suggest trial projects”, it’s unlikely we’ll get many good responses. So I need each hiring manager to chase those down :) But still deferring to the epic owners themselves to decide on those other factors you mention. See Xavier’s post above which outlines this in a bit more detail (“would likely require thoughts from epic owners to find the right task…”)

Eventually yes, of course we’ll automate it. (But we need to avoid too much process creep.) But for now we are just testing this process out and that means things will be more manual.

Thanks, though did you see @antoviaque 's request that those be done in the current sprint? Log in - OpenCraft
Oh actually, I see you have already started pinging epic owners. Thanks!

15

FYI some of the cell-specific discussions for trial projects are happening at:

16

I inquired Xavier Antoviaque about using Accelerated Epics as a source of trial projects:

[…] Our experience with putting newcomers mainly on purely internal work like Ocim is that we tend to pay less attention to the quality of the work (aka the “poorly shod shoemaker” effect) - though I’m hoping we’ll eventually find better ways to treat our own internal work with the same standards we have for upstream & clients – but we aren’t there yet.

Thanks for providing the context.

Thinking a little bit further about it, I realized that making an effort to put OpenCraft owned projects under fully-featured open source governance and eventually use them as the default source of Trial Projects is an excellent option with many benefits in the short and long term:

  • It gives us the ability to evaluate software engineers swiftly and with high confidence; We deeply know the work, its technical challenges, and we own the review timeline.
  • It reduces the risk on clients that trial projects might cause; The Impact on these projects is on OpenCraft’s own account.
  • On the other side of the above’s coin, it reduces the risk of blocking hiring because we cannot risk client work.
  • It moves the revenue hiring work generates (newcomer’s trial projects are not paid as far as I noticed) into accelerated work, which is a priority compared to client work (maybe I am wrong here, but that is what I understood from other discussions in the forum);
  • It makes the hiring process lighter for Hiring Managers and Epic Owners; It’s a proven challenge to scrap for Trial Projects in the client’s epics for them.
  • It can help with hiring pre-screening; great candidates sometimes don’t have Open Source contribs, and they now have an immediate route to raise the bar on that aspect by tackling any issue in our queue;
  • All the other benefits of Open Source in general – Quality, fast iteration, visibility, community building, portfolio, marketing, knowledge spreading, etc.

What is the cost involved in moving Accelerated Projects to become fully Open-Source – I mean, all the other stuff beyond making changes to the git repo using Github? In other words, why are we not there yet? Is it the lack of the right culture or knowledge of how to do it? Like, we do have a lot of expertise in contributing to open source, but we don’t have it on driving a good open source project? Is it something that we can start doing in parallel now and eventually meet with the hiring needs in the next month or so?

I have a feeling that putting all the costs and profits together, the net return of focusing on accelerated projects is higher compared to scrapping client work for trial projects, which seems to have a low benefit ratio, as we can see from this ticket and forum discussions.

I guess this discussion is skewing from the main point of the post, and maybe it’s a bit too late to mix up with the current Trial Project Experiment. But I wanted to capture my thoughts here, especially because we might need to return to this sooner rather than later since up until now we have 0 potential trial projects from all Serenity’s epic owners.

17

@jvdm Agreed, this is important! And that’s actually exactly what we are trying to push currently with the accelerated epics, see 2020 Retrospective & 2021 Outlook - Google Documenten . That’s part of what I mentioned in terms of hoping we’ll get better at this.

But it will take time, especially to get other third parties interested in contributing – and in the meantime, the “poorly shod shoemaker” effect comes more from priorities: the fact is that when we have to choose between paying attention and spending time on a client project and an internal one, we tend to prioritize the client (or upstream). Involving others and having clients & community for our own projects is part of the solution here, indeed. But until we have that, I wouldn’t give trial projects which don’t have a client or an upstream involved - the experience and result on Ocim was really not good. It’s a very attractive solution – but that’s partially attractive because we’d get less constraints & effort in it than in client or upstreaming projects…

18

@antoviaque on SE-4973 you mentioned:

The one criteria that I would relax first is “It should not require any extra client-specific access.” Candidates will be under contract, so we can provide them access to client instances – we have to be able to revoke them afterwards, but if we really want to see how they work, especially with DevOps, it seem difficult not to give them access.

Yes, this would help with my projects. LabXchange has a nice backlog of tasks, and so we can plan a couple of sprints in advance, just in case tasks don’t get completed.

19

+100, including to all the reasons you list. But as Xavier put it. there’s a problem:

I’ve given this a lot of thought over the past year, and the way I’d go about fixing it is via
Global Sustainability and Project Cells. In contrast to the current paradigm where cells inevitably prioritize client work, I imagine project cells being created to focus on internal projects - such as Ocim - and getting their budget from a company-wide sustainability surplus. I believe this would provide the required momentum to these projects.

The devil’s in the details, though. I snuck a mechanism for Global Sustainability into the initial Project Cells proposal I’m ironing out with Xavier, but we judged it too big for the first step. In other words, we’ll soon get a first project cell, but it’ll be focused on client work so that it can be traditionally sustainable.

Once the Project Cell concept is refined, though, I would like to see Global Sustainability revisited as a way to fix this problem.

1 Like
20

This point was not discussed up until now, and it was the main point of what I was trying to gauge. I personally don’t think the route to using accelerated epics is attractive because it is easier and has fewer constraints, on the contrary, I think it’s harder: There is a cost involved in pushing our accelerated projects to reach the bar required to accept contributions from potential candidates or candidates on Trial. I thought it was attractive because of the bullet points in the previous post. Also, I like to think that past experiences should not block us from trying to reach a goal, on the contrary, they should serve as a platform for us to understand what went wrong, identify the next steps, and improve. Finally, building community for those projects is not critical for using them for Trial Projects, is it? It’s how we drive the contribution process, review, test, and accept changes that will make a difference in the quality of the contributions, and in the quality of projects.

So, in essence, there is not enough sustainability for us to prioritize our own projects to the point where we can rely on them to be a reliable source of Trial Projects. It means the cost is too high right now and we can’t make it. It’s exciting to know that we are working on it already, though.

Is it possible that we can start with smaller steps, then? For example, can we organize our internal projects backlog, put them in the OSS repositories, and offer them in the initial tech screening? Some candidates without solid open-source experience could grab some of those items in the backlog and submit a patch as part of the tech screen. Or we could provide the option for candidates to submit PRs in the tech screen process. Do you think that could help our internal projects to “evolve” into an actual solid source of Trial Projects eventually?