If you are begging your engineers to be on call, you have built the wrong engineering culture. Your core engineering team should feel ownership of the app, it should be their baby. What happens in this case is EVERYONE shows up when there is a site issue, and your best engineers burnt out. Instituting an on-call schedule is what a good engineering manager does to force engineers to NOT be on-call 24/7, and instead institute an equitable distribution of responsibility. I think all small eng teams should have on-call rotations, and it's the reason behind why I built https://heiioncall.com/.
My pet peeve is these $PROCESS notifications that go to slack channels. I worked at a company that had an #engineering_humans slack channel because we got chased out of #engineering by bots.
Yes! This. This has happened to me at least two previous companies I have worked at. Everybody sets up thresholds on every possible Datadog metric and alerts become useless. That's part of the ethos of monitoring at my current company. We only set up alerts through https://heiioncall.com/ that we are convinced you absolutely need to look at right now. Anything that is not that gets shoved to a slack channel (that I have long since muted).