All of their data including all of their backups across two different regions was accidentally deleted by Google. If they hadn’t had a backup with another provider, they would have been in serious trouble.
Though this sort of thing is exceptionally rare, it does make me worry about clients who only backup RDS databases using Amazon’s own snapshots. I think we need to set up at least a monthly encrypted backup for them that goes to another cloud provider / backup provider. @gabor@mtyaka thoughts?
A similar thing happened to AWS back in ~2016/7 when an Amazon employee accidentally hit a wrong command and wiped out an S3 region. I didn’t read the whole article again, but I think this is it. It happens from time to time with providers.
We were thinking about this, so Restic is installed for every Grove cluster. However, the integration should be reviewed from installation and configuration point of view. We could start promoting to clients that it would be a great idea to have a secondary backup location of their choice on an S3-like bucket. Though for privacy reasons most of them would want to have it hosted by themselves. Which is not an issue, but makes the initial configuration/setup slower/harder due to collaboration requirements.