Mail server migration updates

FAL-2014 :email:

@team, our mail handling for addresses at opencraft.com and all the mailing lists have now been migrated from mail.plebia.org to mail.opencraft.com. This is still a work in progress, but if you suspect any issues with mail delivery, please comment on this thread or on FAL-2014 asap!

Status

What has been migrated:

  • MX record: all mail sent to addresses at opencraft.com is now processed by mail.opencraft.com. The previous server is still listed as a secondary MX, so if there are issues with the new server, we can quickly and gracefully fall back.
  • Mailing lists: configuration for these is now at List Index - OpenCraft Mail (you will need to make an account with your opencraft email to log in and adjust settings). The emails subscribed should be the same, but any custom / non default configuration will need to be manually updated.
  • Mailing list archives (from now): new emails from now will be archived to the new server. For example, https://mail.opencraft.com/hyperkitty/list/help@opencraft.com/ for the help@ mailing list. We’ll follow up with updated docs and list of links soon.

What is still in progress:

  • Old mailing list archives: these are still at the plebia.org addresses as usual. They will be migrated to the new server later in the sprint.
  • Updating docs. These are almost done, so if you need to see the latest changes or wish to comment, please see the linked PRs on FAL-2014.

Team actionable items

What you may need to do:

  • Check your local email filters; some may need to be updated to match the new mailing lists, depending on how the filters were defined.
  • Create an account on mail.opencraft.com to check your subscriptions, update settings, and view archives.
  • Provide feedback on the new setup - did we miss anything? Do we want custom mailing list prefixes or not for the various lists? Is mail delivery working for you?

Please log time to FAL-2014

For emergency use only

If something is terribly wrong, and we need to revert to the old setup asap, navigate to the opencraft.com domain dns records in gandi and delete the 10 mail.opencraft.com. MX record. (also check that the mail.plebia.org MX record is still present) Then shut down the server at mail.opencraft.com. This should immediately revert everything back.

Kudos to @toxinu for helping with the migration and testing! :smiley:

That’s it for now. :wave: More updates to come soon as we update docs, migrate archives, etc.

7 Likes

@anon46505572, thanks for your work on this.

It seems to be the case, but can you please confirm that I don’t have to resubscribe to any lists? In other words, that I’m not missing any emails by doing nothing?

1 Like

Correct :+1:

2 Likes

@anon46505572 I think I’m receiving the emails correctly on Gmail, but HiperKitty shows this:


Is there something wrong with my setup? Can’t I use the web UI to navigate the mailing lists?

@giovannicimolin HyperKitty is reachable at: Available lists - Opencraft Mail. Can you see the mailing lists?

This is the direct link to the help mailing list: https://mail.opencraft.com/hyperkitty/list/help@opencraft.com/

1 Like

Is there a tree-like/threaded view like in the old system? The messages I saw are in a plain list.

@anon46505572, this change breaks sending activation emails from Sprints. It’s getting SMTPAuthenticationError with these settings.

@anon46505572 I think you commented on this before, but will contact@ be moved here? I don’t see it in the available lists and I’m not subscribed to it.

Yes, contact@ is moved here. You’re on the list subscribers, so if you’re not receiving emails from it, then there’s a bug. Can you login or create an account and view Sign In ?

Nothing has touched relay.net.opencraft.hosting as part of mail migration, so that sounds like an unrelated issue. I don’t see those credentials defined in POSTFIX_SASL_USERS in ansible-secrets, so probably a recent run of that playbook has overwritten any temporarily added credentials.

There should be; I don’t know what’s happening there… :thinking:

Yes, you should be able to use the web ui. I think that main mailing lists page is only displayed for admins. I think this page should work: Sign In

2 Likes

While we’re here, is there any particular mailing list configuration that we should or shouldn’t implement?

In particular, should any/all mailing lists have a custom subject prefix (eg. [Ops] Incident 1234567 OPENED...)?

Yep, I can see that, but it’s pretty non-intuitive to find.

List Index - OpenCraft Mail shows no lists, and clicking Archives shows no lists (defaults to “Most popular” on the right). Most active shows nothing too. It wasn’t until I clicked the By name link that I saw anything – Newest also shows our lists.

Can anything be done about this to either remove those “no lists” links, or change them to something that shows something?

Search doesn’t seem to work either – can we add this? It would be very useful.

:+1: to this too.

@anon46505572 @toxinu Kudos for the great work moving this to OpenCraft’s infra – and for taking over the work, it removes one of the last pieces where the bus factor was quite high. :slight_smile:

I have questions about newcomers on/offboarding, but I’ll ask them inline in the onboarding checklist

5 Likes

:/ Unfortunately not unless we contribute to the mailman project. However, the version installed is somewhat old (it’s the newest that is available in the ubuntu 18.04 repos). A lot of bugs were fixed between then and now. We can upgrade as soon as the playbooks work with ubuntu 20.04. :slight_smile:

Search certainly should work. :thinking: Will do some debugging on this. :+1:

2 Likes

The docs and onboarding course have been updated:

1 Like

Please note that we’re in a weird transition period, where many applications have the previous MX records still cached, so mail.plebia.org is still handling a lot of email for opencraft.com. If you don’t see an email in the new mail.opencraft.com archives, you may need to check the old archives address on plebia.org. This could go for several days.

2 Likes

That’s great news :slight_smile: I’m glad there’s a possibility of a low-friction fix, and I can deal with a little inconvenience until then.

And your updated docs make this even easier, thank you!

That would have been me… added a postfix user to the relay server as part of ansible-secrets#283. The good news is, you should be able to follow the instructions on that PR to add your missing credentials for Sprints.

1 Like

Thanks, @jill! I’ve tried running this playbook before, but it was failing for an unrelated reason. The postfix tag did the trick here.

Btw, does this thread need to be private? If not, can we make it public?

@anon46505572 @toxinu It looks like SoYouStart 2FA code emails are not being forwarded from billing@ to ops@ anymore. I’m not able to access the SoYouStart dashboard.

I don’t know where/if that was configured. billing@opencraft.com was previously an alias that pointed to xavier@ and gabriel@. Now it’s an unarchived mailing list with only gabriel and xavier subscribed. Perhaps @antoviaque or @gabriel had a filter in their mail software (gmail, fastmail) that did the forwarding?

I created it as private because we’re discussing a lot of internal/private details - private docs, private mailing lists, details about email addresses, etc. If you think that’s ok to be public, then sure, but I felt it safer to be private to avoid leaking anything sensitive.