Show HN: I built an open-source tool to make on-call suck less

Or maybe page your managers, such that they can escalate the situation. They will be more aligned on solving the cultural problems if they get waked up too.

I work on a team which runs hyper critical infra on all production machines at BigCo and have the same experience as you.

The problem are not the alerts — the alerts actually are catching real problems — the problem is the following:

1. The team is understaffed so sometimes spending a few days root causing an alert is not prioritized 2. When alerts are root caused sometimes the work to fix the root cause is not prioritized 3. A culture on the team which allows alerts to go untriaged due to desensitization.

Our headcount got reduced by ~40% and — surprise surprise — reliability and on-call got much worse. Senior leadership has made the decision that the cost cuts are worth the decreased reliability so nothing is going to change.

The job market is rough so people put up with this for now.

That’s true. But technical tools can help you highlight culture problems so that they’re easier to to discuss and fix. It’s been a minute since I’ve had to process exactly the kind of on-call/alert problem we’re discussing here, but this does feel like the kind of tool that would help sell the kinds of management/culture changes necessary to really improve things, if not fix all of them.

Running on-call well is a culture problem. You need management to prioritize observability (you can't fix what you can't show as being broken), then you need management to build a no-broken-windows culture (feature development stops if anything is broken).

I was lucky enough to join a company where management does this. The managers were made to do this by experienced engineers who explained to them in no uncertain terms that stuff was broken and nothing was being shipped until things stopped being broken. Unless you have good managers this won’t happen without a fight and it’s a fight I think we as engineers need to take.

Some managers in other teams played the “oh it’s not super high impact it’s not prioritized” game, and those teams now own a bunch of broken stuff and make very slow progress because their developers are tiptoeing around broken glass, and end up building even more broken stuff because nothing they own is robust. Those managers played themselves.

Communication with management is bidirectional, sometimes they need a lot of persuasion.

I completely agree that technical tools cannot fix culture problems.

However, one of the things that I noticed in my previous companies was that my management chain wasn't even aware that the problem was this bad.

We also wanted to add better reporting (like the alert analytics) so that people have more visibility into the state of alerts + on-call load on engineers.

What strategies have worked well for you when it comes to management prioritizing these problems?

Obviously, the best way to get management's attention is to start a stop and frisk customer engagement plan.

enabling bad cultural practices

I strongly disagree. There is nothing culturally bad in a system issuing an error if there is an error. Sometimes systems issue errors that are considered noise by supporters because they are not actionable, but forcing a system to not issue an error just because your support team cannot directly take action on it is an extremely odd leakage of team responsibilities and bound to have unintended consequences. Imagine a developer telling management that they didn’t implement error checking on some edge case because the support team told them they didn’t have documentation about how to take action for instance. The appropriate response there would be “why on earth are you asking support permission to add error messages for a known error?”. On the other hand, if a support team is drowning in noisy error messages they need tooling to make it easy to distinguish between those and other messages that need to be reviewed of have action taken.

I agree with your intent and desire, but the fact is that this problem _keeps happening,_ and we can't fix it by advocating "well, just do alerts better." There's a lot of cultural inertia at a lot of places that leads to creating too many, too low-signal alerts, and fixing that across an entire company -- or, hell, industry -- is a magnificently tall order.

However, installing a tool to specifically reign in those garbage noisy alerts is a potentially easy, significant win for the time and mental health of on-call engineers.

I mean, it sounds like you can then afterwards go in and identify the alerts that are just noise, and having that data means you can take action. Maybe contact the teams that are writing the noisiest alerts, or prepare some data-driven engineering standards for the company, whatever. But that still falls into "fix the culture", which is famously hard to do by fiat.

which upon investigation might not even be “just” noise

My company (like so many) is struggling a bit with culture around noisy alarm. Not only is noise tolerated, but when some closes an alarm because it's "known to be noise" and I prod them, it turns out that there is a very real impact on the user, it's just that nobody bothered to look into it. The alarm rings, the on-call hopes that it closes itself soon enough, it does so they consider it "false positive and noisy" even though there was impact on the user during these few minutes.

The only way to fight that is a zero-tolerance culture on alarms, which means no false positive is ever tolerated: fix it.

If the tool works properly, then the value proposition is:

Sure, you could spend months trying to fix or tune out every useless alarm type, or try to hack your alert manager/email inbox with filters for things you know will get fixed in a few months - OR, you can use this tool that can quickly classify things as important or not.

If I may try to counter this point.

You are right, but there's ideal state and there's the real world. When on call most of my time is spent trying to make on call better. Reducing noise, providing more context when the alert and logs are lacking, and of course fixing the real issues that alerts have identified. That said there is a period of time in between receiving non-actionable alerts and classifying them as such, and more context without using brain power is always welcome. I think I'll give it a shot.

Agreed. This doesn't make the problem better, it's a bandaid solution that can make the problem worse by allowing you to ignore it for longer.

Iterating on your alarms is super informative about the underlying product. It'll point to how you might improve your KPI measurements, or find bugs you didn't know were there.

Missing from the context of sibling replies is the (in my experience as an SRE, quite large) category of alerts that are mostly-but-not-entirely noise and high-effort-duration to properly improve.

Consider a not-that-hypothetical example: "host computer is unreachable" alerts that page oncall when they arrive for members of a fleet of critical database servers or replicas.

The alerts have proven their usefulness (they tend to arrive several minutes before application-level error spikes when a database is e.g. so overloaded the monitoring agent can't function or a replica is gone so changelogs are overflowing) ... when they're genuine. However, they're mostly not genuine: alerting agents crash and automatic-restart-service init scripts bug out or give up; per-database-owner customizations in hosts' available file descriptor numbers are propagated incorrectly to non-database services and prevent the alerting agent from running, databases that serve infrequent-but-critical on-demand reporting loads are subjected to tens-of-minutes-long load spikes during which the host is doing what it's supposed to but so pegged that the alerting agent won't work, and so on.

What do you do with those alerts?

"Just fix the problems causing the false positives!" Fine, but that takes a lot of time and coordinated effort, even if the oncall folks are empowered to prioritize the work getting done (which is far from a given at many companies, for reasons both good and bad): auto-restart-agent scripts can be replaced with better scripts (time, effort, debugging of some hokey bash that needs to run on a wide variety of environments) or systemd (time, effort, maintenance windows, and approval/retraining to update ancient linux distributions running critical databases). File descriptor/per-database tunings can be unified and continually audited for/invalidated before configs are pushed (developer effort, coordination with teams writing configs). Reporting databases can be upsized (money, maitenance windows) or the database processes can be moved into a cgroup to leave some resources to spare (effort, distro upgrades, maintenance windows).

That's going to take awhile, if it ever happens to completion.

Meanwhile, this "host unreachable" alert is useless 90% of the time and very useful (as in: it can be leveraged to prevent downtime for customers entirely) the remaining 10% of the time.

Like, sure, some of those issues are stupid. But none are hypothetical, all are younger than 5y, and I bet this kind of struggle is common and representative even at companies who are invested in operations and operations staff.

That's not an "inability to create useful observability", that's a genuinely hard problem resulting in noisy, spurious alerts that, depending on the rate-of-change/regulatory space of the company, might persist for months or years. What's more, alert management is an ongoing process. Even if one family of noisy alerts is addressed, another one will emerge as new behaviors and technologies are adopted.

I guess this is all to say that I don't think tools like Opslane (which I have not used) are "enabling bad cultural practices". Organizations that don't give a shit about operations will continue to suck at operations no matter what tools they use. But products Opslane are valuable even (especially?) in capable, operations-focused organizations as well.

To give a more charitable take, in practice it's easy for noisy alerts to creep in and a tool like this could be a good nudge to dig a bit deeper on that alert and why it's gone from useful to noise.

As someone who has worked in non-tech startup transitioning into enterprise, or enterprise organisarions for decades… I can tell you that non of these companies or organisations are capable of making meaningful interactions with operations. Disclaimer I haven’t worked operations, but I’ve been in relative close proximity with them work wise for natural reasons or using their infrastructure and sometimes making their tools or helping them with things like automation and Powershell.

Anyway in lot of places management see IT as a necessary evil. Like a service center similar to HR but less popular because management genuinely don’t understand it and most IT departments lack HRs political shrewdness and communication abilities. At the same time it’s not uncommon for users to be unable to tell support if they’re on an Android or iOS device (yes, I’m serious). Sometimes employees won’t even differentiate between their professional and personal IT issues on their work devices. Which means that sometimes they’ll raise hell over things that might not warrant full alert systems for on-site support.

What might be challenging here is that you’ll still need someone to actually use the authors tool correctly. Though that is probably going to be a lot easier than making any sort of change management to was organisations relationship with IT.

Yeah, thats fair feedback. The main aim was to reduce the alert fatigue for on-call engineers and provide a way to get insight into the alerts at the end of the on-call shift.

This way there is data to make a case that certain alerts are noisy (for various reasons) and we should strive to reduce the time spent dealing with these alerts. Fixing some of them might be as easy as deleting them but for others might need dedicated time working on them.

100x this. Garbage in = Garbage out.

In similar mindset, I've seen attempts to "fix" flaky test suites by retrying failing tests 5 times until they pass. What happens: You just set a new baseline of shit allowed. This allows even more noise to enter the system and you have to rerun them even more or need increasingly more advanced tools to filter out the noise.

Once the new baseline is anchored, you become dependent on the filter. Now every tool that interacts with the metrics need to be aware of the additional filter, there may be more than just the slack messages. Should your dashboard show the raw or the filtered metrics?

Devils advocate: consider an alert with 99/100 false positives. The LLM may be good at classifying it as noisy but will it do a better job than a human to react to the 1 true positive? Maybe, but at the same time it allows more such noise to accumulate in the system, in effect a net negative. It's better to remove such an alert instead. Even if the numbers were turned around in favor, that's a lot of complexity added.

The additional context this product provides may of course still be useful and i applaud the effort. This product space does have a lot of potential for growth and is a real pain for operators. Be careful with using it as a substitute for proper alert hygiene and culture.

A candy bar cell phone, paid for by your employer and handed to whoever is on call. People who don't want it can just forward it to their phone.

This really sounds like a _you_ problem and not something you need hardware to fix.

You can already enable silence/focus time bypass modes for apps like PagerDuty and such…

If you can’t develop some sense of responsibility to check if you’re on-call, frankly you have no business being in an on-call role.

No hardware will make you or your engineers more diligent. The only reason pagers made more sense than phones is/was because of protocol reasons NOT because it’s some separate device.

There are phone apps that can pierce through all silent or DND settings. Get one of those. If the same app could buzz on less than 50% battery to remind to charge that would help. Also same app could request to confirm on call status so the don't forget. If they don't confirm someone else gets the shift.

yeah, thats the goal of adding the context and the report - to hopefully bring awareness to the team that this alert should be removed.

My rationale for flagging the alert was to help prioritization for the on-call (lets say there are multiple alerts going off at the same time)

I like incident.io's take on LLMs with incident management: which is essentially assist, don't decide. [1]

1: https://5x9s.svix.com/p/evolution-of-incident-management

There is a lot to be said for "smoke test" metrics. Things you expect to have frequent false positives, but are sometimes early indicators of larger problems or indicators of where to look deeper if something else goes sideways. They're not things that should wake you up in the middle of the night, but they're a damn valuable tool to quickly figure out what's actually wrong when a "real" alert triggers.

Many of these lend themselves well to dashboards instead of alerts, but not everything is "dashboardable". Sometimes it's good to have a set of low-priority alerts that are treated differently than others.

E.g. "we're not receiving any data/requests". Sometimes that's just a lull in activity. Maybe a holiday. Sometimes it's because everything _else_ is broken and nothing is getting in (e.g. DNS issues).

With that said, I do think that classification should be made manually and not automatically.

Why? We send alerts to Slack and Pagerduty. Slack is to help everyone who might be working, PagerDuty alerts the persons who are actually in charge of working on it.

Yeah, I agree that slack is not the best medium for alerts. I think we it has somewhat become the default in teams is that it makes it easy to collaborate while debugging. I don't know a good way to substitute that and share information.

What strategies have you seen work well?

I don't think Slack (or similar) should be a primary alert mechanism.

But, if alarms are configured in a clean way, ideally your team is getting some warnings and such there and then if there's an alert that needs to actually page, it sends that to PagerDuty or whatever platform you use along with another message to Slack.

THIS. Whispering into a slack channel off hours isn’t a way to get on-call support help nor is dropping alerts in one. If it’s a critical issue I’m going to need a page of some kind. Either from something like PagerDuty or directly wired up SMS messaging.

I would expect anything notifying via Slack or text would have an accompanying incident ticket in the system of record.

We had a rule in my team (before a management change that blew it all to shit) that we don’t use email or messaging for monitoring. Everything goes into the SOR. Once it’s in the SOR, if people want emails, texts, or whatever, it let them know there is work to do, that’s up to the team. Others would make dashboards… lots of options once it’s in the system, and nothing gets lost.

For example, I went from a team that looked at tickets all day to one that mainly worked on user stories in Jira. Because no one was looking at the incidents in the SOR, things were getting missed. I wrote something to check for incident tickets assigned to our team every hour, and it would post them in our team chat so people knew there was work to do. Then once per day, it would post everything still unassigned, so if something was lost on that hourly post, it would annoy everyone once per day until it was assigned/resolved. It worked out decently well. If there was a lot of stuff, it would post a message to have someone actually login to the SOR and look at all our tickets. I would sometimes use the standup to assign stuff out and get some attention on it, if things were getting bad.

Try Netherlands. We're Microsoft land over here. Pretty much everyone is on Azure and Teams. It's mostly startups and hip small companies that use Slack.

As Slack is not end to end encrypted we and I imagine many other companies cannot use it.

Thanks for the feedback. We want to get something out quickly and we had experience working with Slack so it made sense for us to start there.

However, the design is pretty flexible and we don't want to tie ourselves to a single platform either.

System goes down or degrades in some other way at night and important customers with a different timezone get angry, threatening to leave? (happened with us a few times)

But I would'nt use LLM for it due to hallucinations

On the Internet, with the war in Ukraine, that's entirely possible!

Why not use statistics? Been reading about xmr charts recently on commoncog. That might help for example.

I'm curious about incidental :) how are you going to compete with other, well established IM tools like rootly, incident.io, firehydrant.com?

Thanks for the feedback! I saw the incidental launch on HN and have been following your journey!

I wouldn’t say it’s particularly clever. It’s a fairly obvious idea to anyone who has worked with alerting through IMs. What it is, is /difficult/, because you really really need to avoid false positives. Probably lots of hard work involved. So kudos for making this work (if it works)!

Depending on the stakes this is a pretty dangerous attitude. The goal for oncall is to keep the website working, and if you're tuning for "never get paged" then you'll necessarily miss an incident eventually.

I wish everyone shared your philosophy! I once worked at a company where it was expected to get 10+ pages per day, and worse, a configuration error by a customer success team would trigger an engineering page because the error handling didn't distinguish between a config problem and an actual system issue. It was insane.

The alerts being sent to Slack are normally from one of those alert databases (such as Prometheus and AlertManager). Slack isn't the source of truth for them, just a notification channel.

The wikipedia for page for fault management has a "see also" for alarm management, which looks extremely relevant as well.

I agree - fixing the original problem is the main motivation.

We wanted to provide that awareness because a lot of teams arent fully aware how bad the problem might be (on-calls change weekly, there might be a bunch of other issues)

A central service might be in a better position to classify messages compared to lots of individual agents.

we've leveraged Clickhouse/S3 to build a cost effective alternative to Datadog at https://hyperdx.io (OSS, so you can self-host as well if you'd like)

You have plenty of options. some of them are open source https://signoz.io/ https://coroot.com/

Did you search tools that are cheaper than DD?

Yeah, unfortunately, I don't think these messaging tools are async. During oncall, I used to pretty much live on Slack. Incidents were on slack, customer tickets on slack, debugging on slack...

I guess 7-Eleven management trainees know that their company is just as replacable for their employees as their employees are to them.

https://github.com/keephq/keep (disclaimer - i'm the maintainer)

That's why it's always at least double time for call outs

I worked on a small team that covered a relatively big site where there were so many alerts it was simply hard to track... They were all sent over email to a group list and most would just delete.

I spent about 3 months, each day trying to triage into buckets based on activity and dealing with whatever was causing the most alerts each day. Some came down to just tamping out classes of 4xx errors that should never have been in the email/alert system to begin with. Others came down to indexes to reduce load/locking/contention on some db tables. Others still were much harder to dig into.

Will say at the end of the 3 months, there was only a trickle of emails a day and the notifications were taken much more seriously after not being so overwhelming as to being ignored altogether.

edit: This was just the first thing I did each day was deal with one problem, then moving to new feature work... It wasn't assigned, as the company would always prioritize new feature work, it was just something I did for my own sanity.

derp, thanks for catching. It has been fixed!

Could you please elaborate?