Oncall Bot and Frontend Monitoring

The problem

Silent drop-offs during third-party handoffs, such as NFC checks, were found days later through customer support. Our alerts were threshold-based and noisy, so people had learned to ignore them.

The approach

• Relative baselines, not static thresholds. The alerting service runs SQL against ClickHouse and compares each metric with the same hour last week. A 20% conversion drop means something at 3 a.m. in Tokyo and at noon in Berlin, and global traffic swings stop causing false alarms.
• An alert that arrives with context. When an alert fires it posts an incident card to the internal chat platform. A background worker then gathers recent frontend error stack traces from ClickHouse, the active config version from the configuration service and Kubernetes deployment logs, and sends them as a structured prompt to an enterprise LLM API.
• A summary in the thread. The bot replies with a probable root cause, for example: step 3 conversion down 32% in Japan, a null-pointer error in the NFC bridge wrapper, a config change deployed 15 minutes earlier and a vendor gateway timing out on the backend.

The result

Time to detect dropped from hours or days to under 5 minutes, time to mitigation fell by 65% and false-positive alerts by 80%. I also tracked alert signal-to-noise (the share of alerts that map to real incidents) and mean time between failures for frontend modules.

Why this way

A normal alert tells you something broke. Collecting the data automatically and summarizing it closes the gap between a metric and a person understanding it, which is where most of the response time was going.