Correct - this monitoring is reliable for HMS instances and has been useful many times, so I’d be cautious about replacing it.
@gabor, Other than APM you already linked, I’m also using the Errors (error inbox) to check exceptions without logging into every instance from the ASG separately and checking individual logs for each service we have in the supervisor (we currently run 23 services). I used to use our ELK stack for this, but we removed it some time ago.
Since we last considered self-hosting Sentry, it has become much more challenging. To replace New Relic fully, we should use a service that provides APM and error tracking, like GlitchTip. Alternatively (e.g., if OpenTelemetry is already supported), we could use Bugsink for error tracking (though this one does not expose a REST API).
That said, Uptime Kuma looks nice. I like that we can specify a proxy for the monitors, allowing us to set up multi-regional uptime checks. What slightly worries me is the lack of the API - it is an SPA app that uses websockets for all interactions. We would need to manage all monitors manually.
If this is problematic, and we would like to use GlichTip, we could look into its built-in uptime monitoring. It has fewer features, but the backend runs on Django, so we could implement and contribute things like configurable retries, if needed.
It should be possible. I have these monitors terraformed, so they are configured via the New Relic API.