Moving from JIRA to Huly - Planning

Discussions - Public
1

Ticket to log time

In our discovery and the surrounding discussion, the most promising replacement for JIRA was Huly, so I’ve been looking into what that migration would look like and I will share my thoughts here so that everyone can provide feedback and discovery potential pitfalls in my approach.

Huly is structured differently than Jira, so we should re-evaluate how we structure our data:

  • First of all, Huly supports workspaces, which is a container above issues, so rather than having a different project for Serenity, Bebop, Falcon etc, we can give each team its own workspace.
  • Now instead of having epics for each project we have, we can use projects. We regularly have tickets going off in thousands and Bebop is close to 10000. Just looking at an issue like BB-1234 or BB-3456 doesn’t tell me what it’s about, what project it’s related to etc. Instead, if we use a different key for each project we will be able to immediately get a lot more info.
  • Huly doesn’t have accounts. We can investigate adding them, use custom fields or we can use labels, milestones or projects to determine account. For Clients, Huly has company records as part of it CRMish features.
  • For reviewers, we can add a custom field, either two for Reviewer 1 and 2 or, we can create a Reviewer field that is an array of users that can support 2 or even more reviewers.
  • Point scores can be added as a custom field.
  • Instead of watchers you can use Collaborators in Huly, though this is a manual process.
  • Huly has no sprints yet, so we need to work with upstream on that before a proper migration.

I see this migration working in the following potentially overlapping stages:

  • Initial Discovery: The current stage
  • Exploration: Set up staging instances, and perform initial test migrations
  • Development: Upstream features we definitely need before a migration
  • Huly Phase-in: All new projects created in Huly. At this stage we use both but all new projects “epics” etc are on Huly.
  • Jira Phase out: All new tickets and activity on Huly, Jira only contains old data.
  • Final Migration: All remaining old data will be moved over to an archived workspace retaining the old ticket ids if possible.

Given the above here is my rough estimation:

  1. Discovery on Huly hosting 20h

  2. Upstream changes to Huly self-hosting to adapt for our needs TBD

  3. Discovery: Support for Sprints in Huly 40h

    This is the biggest missing piece and in my conversation with them, I learned that they are working on a process management module that should support our needs. However, we should probably see if we can support them in prioritising this. They are also working on import/export, which again should be prioritised.

  4. Create a stage deployment of Huly with a limited set of users 10h

    This is a simple deployment after everything is already working.

  5. Discovery: Export Data from JIRA 20h

    • There are docs for exporting from JIRA Cloud, they might work with our instance as well.
    • If the above docs work, we can simply take a dump of the data and start working on scripts.

  6. Discovery: Import Data to Huly 20h

    Huly supports importing from a “Uniform Format”, we can investigate creating scripts to convert from JIRA’s export format to Huly’s import format.

  7. Create migration scripts 50h

    • While a proper estimate will come with the above discovery, the above is a rough estimate that includes a test migration of limited data from JIRA → Export File → conversion script → Huly
    • Huly doesn’t seem to have a traditional API. Instead it has a websocket-based API where all actions, events etc happen over a websocket.

  8. Create user accounts, workspaces etc. 40h

    • This will involve creating accounts for all users, and workspaces for each team.
    • It will also involve creating Companies accounts etc.

  9. Migrate active data 40h

    This will involve migrating all data for active epics and tickets.

  10. Migrate inactive data in archived workspace 40h

I’d appreciate if everyone could go through this when they find time and give feedback.

CC: @farhaan as reviewer and @gabor as epic owner.

11 Likes
2

Thank you for this very informative post, @kshitij! Is there a ticket to log time?

Can this be implemented as a plugin?

Also, is it safe to assume that Huly supports logging time and has an API that we can use for it? I’m asking since it isn’t mentioned here, and I think, some of us use external tools to track time, and then automatically sync it with Jira (e.g. I use timewarrior to track and tempoit to sync; I dread the days I synced it manually). So for those of us who do use external tools to track and log time would be necessary to update our tools to be compatible with Huly. I think, it’s totally ok if we have to update our tools ourselves, but e.g. if there is no API for that, can OpenCraft cover implementing it?

3

Thank you for putting this together, @kshitij!

  • Discovery: Support for Sprints in Huly 40h
    This is the biggest missing piece and in my conversation with them, I learned that they are working on a process management module that should support our needs. However, we should probably see if we can support them in prioritising this. They are also working on import/export, which again should be prioritised.

The Slack link is not public, so would you mind providing more context about its content so we can see it without logging in? Is there any public architectural documentation of this functionality? Are there plans for some custom pipelines when completing a sprint? Will we still need SprintCraft? My biggest concern that I posted here is the following:

Another missing piece is the API documentation. While it is not essential, it seems like an odd choice not to have one. For example, if we want to use SprintCraft with Huly, knowing how to do this without digging into the code would be good.

It’s also odd how they closed the issue about missing API docs ref. Providing two basic examples in TypeScript ref is not enough to call such a thing done.

They are currently linking these examples as their API docs here, but these scripts rely on an npm package for an API client (available only in their develop branch). Therefore, if we need to update SprintCraft or use any external services that need to query the API (like our worklog syncing scripts), we must account for reverse-engineering their API. We should probably include this in the list of follow-up discoveries.

cc: @maxim, as this should partially answer your question about the API.

1 Like
4

Most of that sounds really great! Workspaces in particular would be a nice improvement.

What’s the ticket to log time for this post - SE-6407?

This might be more a question for @tikr, but could projects play the role of accounts in this scenario? It’s actually quite annoying in Jira today that we have accounts and epics, and most of the time they are set to the same thing, so having an “account” field mostly feels redundant to me if we could just do billing by epic/project instead. Though I guess there are a few use cases for having them separate.

I have the same question. From this video (in particular “Manually submitting a time report” and “Viewing time reports as a manager”), it looks like they do have time tracking. But I also would like to know about the API - I use Toggl and toggl-tempo-worklog-transfer.

5

I’ve added a link to the original post, but you can find it here.

There are, though I wonder if the exceptions are that critical to maintain. For instance, there was once a ticket for a Listaflow-lead project that adds a change to SprintCraft to improve their integration. Notionally, that change might be in SprintCraft’s scope, but coming from Listaflow’s budget. In this case I don’t think it’s that big of a deal if we require the ticket be in the funder’s workspace.

More weird might be the fact that there’d need to be a minimum two workspaces for most clients, one for their maintenance epic and one for their support epic. This abstraction could get in the way for such cases.

6

Looking into Huly plugins will need further discovery, but they are interested in sprints, I just don’t know how they’re prioritising it.

Huly supports logging time, but their “API” is web-socket based, so major changes will be needed. Either we help add a proper REST API to Huly or adapt tools to their mechanisms. It’s very poorly documented so I anticipate some challenges here.

Here is their quote:

AFAIK No. I don’t see much evidence of it being implemented. I hope we can consolidate SprintCraft into Huly, either as a plugin or native features, but more disovery and communication is definitely needed here. I’ve asked them about it.

Since the code is open source, I think I’m less worried about reverse-engineering than I am about having a stable API we can rely on.

I don’t see the need for two workspaces here. My proposal is to map workspaces to internal teams like Bebop and Serenity. We’d have two projects. So tickets like HMXM-1234 is for maintenance and HMXS-1234 would be support, for example.

We might need a workspace for clients to have access to our JIRA, and sync tickets between workspaces.

7

That’s an interesting idea! I’m definitely in favor of using the migration to Huly as an opportunity to simplify existing processes and workflows where possible :rocket:

As far as this particular question is concerned, I’ll have to take some time to think about it in a bit more detail. Scheduling that for next sprint.

1 Like
8

@kshitij Thanks for the detailed post and answering the follow ups. I have nothing to add to at this point.

9

@kshitij Thanks for the continued discovery! Great work. After digging some more based on your recap, and testing on huly.app, I’m pretty eager to switch! It looks like suuuuch an improvement over Jira.

However, with this type of app, the devil is in the details - so some comments:

First of all, Huly supports workspaces, which is a container above issues, so rather than having a different project for Serenity, Bebop, Falcon etc, we can give each team its own workspace.

Note that the documentation says: “In the near future, we will be adding capabilities for interconnectivity between workspaces, allowing you to manage shared tasks, resources and communication with workspaces operated by different individuals or teams.” - so it looks like we would not be able to share tasks between teams like we currently do, for example? There are likely other things this would be limiting, which would be good to understand, and have solutions/workarounds for.

We regularly have tickets going off in thousands and Bebop is close to 10000. Just looking at an issue like BB-1234 or BB-3456 doesn’t tell me what it’s about, what project it’s related to etc. Instead, if we use a different key for each project we will be able to immediately get a lot more info.

How do you do that? When I created a project inside my workspace “perso”, the URL for the issue ended up being Huly - hardly an improvement over the Jira identifiers?

Sounds like this will require a careful discovery task on its own to figure this out, with @tikr and @gabriel as reviewers, as well as someone familiar with the accounting system like @kaustav . This is so central to how we function, we need to have a proper solution from the get-go.

What do you mean by “manual process”? That we need to add people manually to the collaborators to receive all notifications, we don’t get added by just replying somewhere for example? That seems fine, as to ensure someone sees a specific reply, an explicit mention is better anyway.

+1

Rather than making all new projects in huly, we could start with one or two that are hand-picked. Maybe the very first one could be purely internal, like Listaflow. Then one with a client - maybe one that needs to be accessed by a client, willing to help test it? Then once we have ironed out the quirks with the different types of projects we have, we could migrate an entire cell. Then once that works for a cell, everyone?

Sorry, not “if possible” - we have to migrate all the previous data, otherwise we will lose it when Jira is stopped, as we would never go look for it in the old DB dumps. And ideally, it wouldn’t be on separate workspaces, but imported into the actual workspaces that we use, so that we can reference it properly (cf inter-workspace interaction limitations mentioned above).

We would use PostgreSQL, correct?

Why are those two different discoveries? The main goal being to convert to Huly’s format, it looks important to be matching one to the other, when exporting from Jira? There is no point in just exporting the data to an intermediary format imho, especially without consideration for what Huly will take in. Also rather than trying to do a global export to weird Jira XML formats, which might or might not include all the data, it might be worth querying the database directly.

Yes, it’s important to test the interaction with upstream, and see how willing they are to take in external contributions, and how our relationship with them develops. So I would definitely go for an upstream contribution here. And I wouldn’t migrate before we have at least some non-trivial contributions merged upstream (like the sprints feature), to make sure we have fully explored these aspects.

Besides the blockchain bullshit that was already pointed out as a worrying sign, this blog post also makes me wonder if the “next generation of Huly” will still be fully open source? Or will it require to depend on Huly’s own infrastructure, without the ability to host & control our setup and data ourselves? What if we don’t want our data on the blockchain, or want to develop features upstream doesn’t want to merge? It could be worth asking more details about this.

Also, can we keep the forum threads public? I don’t think there should be anything here that is private to clients?

1 Like
10

I wasn’t as concerned about the URL, but the ticket ids which we more commonly pass around.

The particular URL you’ve shared seems to be longer because it’s on the shared huly server, so they probably append a UUID in case of a name conflict. When I set up Huly locally, the URL for a “bebop” account just had “bebop” in the URL.

Yes, I meant that you’d need to manually add yourself as a collaborator.

I think this definitely makes more sense. I will update my approach to match.

I was mainly talking about the ticket ids, not the data itself.

Yes, I meant that we’d probably want to adapt it ot PostgreSQL.

They can be combined, however my reason for keeping them separate was to explicitly consider the export as a kind of full backup dump. It seemed like a more useful OS contribution since it could be used by someone to migrate from Jira to some other tracker without adding much effort for us.

I didn’t get the indication from them that plan to close things down. If anything, they seem to see this as a way of going even more “open”.

Here is a relevant conversation from their Slack:

Response from someone at Huly:

4 Likes
11

@kshitij Your replies sound good to me, my only comments left are with:

Looks like they are built the same way no? I had “WB-1” automatically as an ID on that ticket, and only the workspace had a custom id (“perso”). But maybe I missed something, or it can be set manually and I just didn’t see it.

I wouldn’t both with this - Jira has been deprecated for years now, it’s not open source in the first place, and I think the intermediate step would definitely add work on our side. Unless there is a compelling reason not to, I would treat the database itself as the dump to import into Huly.

Great! The answer is indeed reassuring. They are free to explore the blockchain stuff as long as we don’t have to, and can still self-host truly free software. :+1:

Btw what about my last remark:

13

When you create a new project, you get the option to select a key. For example the default key it selected when I tried to project called Serenity was SERE, but I changed that to SE so now I have a URL like:

https://huly.app/workbench/decorative/tracker/SE-1

Here, “decorative” is the name of the workspace. I picked a random long word that wouldn’t conflict with an existing workspace on Huly. For our use case, such as URL could be:

https://huly.app/workbench/bebop/tracker/SUMAC-1

I forgot to respond to this, but the reason for using an archived namespace was to reuse the same ticket IDs. i.e. SE, BB, OC, FAL, etc. We could technically migrate them to the team workspace, however that would make updating links harder. We could just regex replace old links referencing JIRA tickets, but if the URL workspace will also change depending on the ticket it would be harder.

It might also make sense to implement some kind of redirect in Huly so old tickets keep working, in which case this won’t matter as much.

Sure, my concern was that navigating the database structure of Jira would be more complex than using its tools to export data.

I’ve made this discussion public now.

1 Like
14

@team I’d love it if everyone could go through the plan and raise any issues or concerns.

We can start carving this into actionable tasks from next sprint.

1 Like
15

Most of the things sounds good to me, and many of my initial questions are already answered in the thread.

However, the lack of proper REST or GraphQL API and related documentation bugs me. I have the feeling that this will hinder us a lot during the integration of SprintCraft or any other tooling (like Crafty). Although I didn’t check for it, I really hope they have at least some kind of developer documentation. Otherwise, it would be required to discover most of the codebase to do even small changes/features.

16

@gabor That is a good point. Contributing documentation can be a fantastic first type of contribution though, and something that is highly valuable and often valued upstream. Even (maybe, especially) when the maintainers themselves don’t enjoy writing documentation…

If we do have to go through that effort of making sense of the codebase, we should definitely use the opportunity to contribute a good documentation upstream - both for ourselves later (we will then have the documentation, and others might keep maintaining it), and for the project itself (it is indeed an important piece for an open source project). Worth reaching out in advance though, to know what actually currently exists, and what plans/preference upstream has for documentation.

17

Could we apply for funding from Veitch foundation or Sovereign Tech Fund to support our contributions to Huly?

1 Like
18

I don’t think it would fit with the Veitch foundation. It might fit in the case of the Sovereign Tech Fund, possibly though its ’ Improve FOSS Developer Tooling’. However, looking at the projects they have/do fund, I don’t think Huly fits well in that set, and so I’m doubtful. The existing angle we’re taking with Superset seems like it has a better chance of success.

19

@kshitij I think this is in a spot where you could start carving out tasks-- looks to me like everyone who had issues has raised them by now. By the way, I’ve got some bandwidth in the coming sprints and would be happy to help here :slight_smile:

1 Like
20

I think it’s definitely time to start carving out tasks. I think the first task we can start looking into is the Huly Hosting Discovery. I’ve created a task for that: SE-6434. I think someone who has worked with Kubernetes or wants to should pick that up. CC: @gabor @maxim @kaustav

I’ve also been thinking of having another small discovery as a precursor to the Sprints discovery, which is a kind of technical dive and document on Huly that will help other developers quickly get up to date on Huly, how it’s structured, and how we can start hacking away at it. This can be a public document that could guide anyone, but the idea is that it will help OpenCraft devs get familiar with it quicker. I think you would be perfect for taking this one on @Fox. Feel free to pick up SE-6435.

3 Likes
21

@kshitij Great! I’ll take it up :)

2 Likes