Coder Legion - Recent posts tagged notification

Prometheus Alertmanager vs Grafana Alerting (2026): Architecture, Features, and When to Use Each

Mon, 13 Apr 2026 10:25:41 +0000

Most observability stacks that have been running in production for more than a year end up with alerting spread across two systems: Prometheus Alertmanager handling metric-based alerts and Grafana Alerting managing everything else. Engineers add a Slack integration in Grafana because it is convenient, then realize their Alertmanager routing tree already covers the same service. Before long, the on-call team receives duplicated pages, silencing rules live in two places, and nobody is confident which system is authoritative.

This is the alerting consolidation problem, and it affects teams of every size. The question is straightforward: should you standardize on Prometheus Alertmanager, move everything into Grafana Alerting, or deliberately run both? The answer depends on your datasource mix, your GitOps maturity, and how your organization manages on-call routing. This guide breaks down the architecture, features, and operational trade-offs of each system so you can make a deliberate choice instead of drifting into accidental complexity.

Architecture Overview

Before comparing features, you need to understand how each system fits into the alerting pipeline. They occupy the same logical space — “receive a condition, route a notification” — but they get there from fundamentally different starting points.

Prometheus Alertmanager: The Standalone Receiver

Alertmanager is a dedicated, standalone component in the Prometheus ecosystem. It does not evaluate alert rules itself. Instead, Prometheus (or any compatible sender like Thanos Ruler, Cortex, or Mimir Ruler) evaluates PromQL expressions and pushes firing alerts to the Alertmanager API. Alertmanager then handles deduplication, grouping, inhibition, silencing, and notification delivery.

# Simplified Prometheus → Alertmanager flow
#
# [Prometheus] --evaluates rules--> [firing alerts]
#        |
#        +--POST /api/v2/alerts--> [Alertmanager]
#                                      |
#                          +-----------+-----------+
#                          |           |           |
#                       [Slack]    [PagerDuty]  [Email]

The entire configuration lives in a single YAML file (alertmanager.yml). This includes the routing tree, receiver definitions, inhibition rules, and silence templates. There is no database, no UI-driven state — just a config file and an optional local storage directory for notification state and silences. This makes it trivially reproducible and ideal for GitOps workflows.

For high availability, you run multiple Alertmanager instances in a gossip-based cluster. They use a mesh protocol to share silence and notification state, ensuring that failover does not result in duplicate or lost notifications. The HA model is well-understood and has been stable for years.

Grafana Alerting: The Integrated Platform

Grafana Alerting (sometimes called “Grafana Unified Alerting,” introduced in Grafana 8 and significantly matured through Grafana 11 and 12) takes a different architectural approach. It embeds the entire alerting lifecycle — rule evaluation, state management, routing, and notification — inside the Grafana server process. Under the hood, it actually uses a fork of Alertmanager for the routing and notification layer, but this is an implementation detail that is invisible to users.

# Simplified Grafana Alerting flow
#
# [Grafana Server]
#   ├── Rule Evaluation Engine
#   │     ├── queries Prometheus
#   │     ├── queries Loki
#   │     ├── queries CloudWatch
#   │     └── queries any supported datasource
#   │
#   ├── Alert State Manager (internal)
#   │
#   └── Embedded Alertmanager (routing + notifications)
#           |
#           +-----------+-----------+
#           |           |           |
#        [Slack]    [PagerDuty]  [Email]

The critical distinction is that Grafana Alerting evaluates alert rules itself, querying any configured datasource — not just Prometheus. It can fire alerts based on Loki log queries, Elasticsearch searches, CloudWatch metrics, PostgreSQL queries, or any of the 100+ datasource plugins available in Grafana. Rule definitions, contact points, notification policies, and mute timings are stored in the Grafana database (or provisioned via YAML files and the Grafana API).

For high availability in self-hosted environments, Grafana Alerting relies on a shared database and a peer-discovery mechanism between Grafana instances. In Grafana Cloud, HA is fully managed by Grafana Labs.

Feature Comparison

The following table provides a side-by-side comparison of the capabilities that matter most in production alerting systems. Both systems are mature, but they prioritize different things.

Feature	Prometheus Alertmanager	Grafana Alerting
Datasources	Prometheus-compatible only (Prometheus, Thanos, Mimir, VictoriaMetrics)	Any Grafana datasource (Prometheus, Loki, Elasticsearch, CloudWatch, SQL databases, etc.)
Rule evaluation	External (Prometheus/Ruler evaluates rules and pushes alerts)	Built-in (Grafana evaluates rules directly)
Routing tree	Hierarchical YAML-based routing with match/match_re, continue, group_by	Notification policies with label matchers, nested policies, mute timings
Grouping	Full support via group_by, group_wait, group_interval	Full support via notification policies with equivalent controls
Inhibition	Native inhibition rules (suppress alerts when a related alert is firing)	Supported since Grafana 10.3 but less flexible than Alertmanager
Silencing	Label-based silences via API or UI, time-limited	Mute timings (recurring schedules) and silences (ad-hoc, label-based)
Notification channels	Email, Slack, PagerDuty, OpsGenie, VictoriaOps, webhook, WeChat, Telegram, SNS, Webex	All of the above plus Teams, Discord, Google Chat, LINE, Threema, Oncall, and more via contact points
Templating	Go templates in notification config	Go templates with access to Grafana template variables and functions
Multi-tenancy	Not built-in; achieved via separate instances or Mimir Alertmanager	Native multi-tenancy via Grafana organizations and RBAC
High availability	Gossip-based cluster (peer mesh, well-proven)	Database-backed HA with peer discovery between Grafana instances
Configuration model	Single YAML file, fully declarative	UI + API + provisioning YAML files, stored in database
GitOps compatibility	Excellent — config file lives in version control natively	Possible via provisioning files or Terraform provider, but requires extra tooling
External alert sources	Any system that can POST to the Alertmanager API	Supported via the Grafana Alerting API (external alerts can be pushed)
Managed service	Available via Grafana Cloud (as Mimir Alertmanager), Amazon Managed Prometheus	Available via Grafana Cloud

Alertmanager Strengths

Alertmanager has been a production staple since 2015. Over a decade of use across thousands of organizations has made it one of the most battle-tested components in the CNCF ecosystem. Here is where it genuinely excels.

Declarative, GitOps-Native Configuration

The entire Alertmanager configuration is a single YAML file. There is no hidden state in a database, no click-driven configuration that someone forgets to document. You check it into Git, review it in a pull request, and deploy it through your CI/CD pipeline like any other infrastructure code. This is a significant operational advantage for teams that have invested in GitOps.

# alertmanager.yml — everything in one file
global:
  resolve_timeout: 5m
  slack_api_url: "https://hooks.slack.com/services/T00/B00/XXX"

route:
  receiver: platform-team
  group_by: [alertname, cluster, namespace]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: pagerduty-oncall
      group_wait: 10s
    - match_re:
        team: "^(payments|checkout)$"
      receiver: payments-slack
      continue: true

receivers:
  - name: platform-team
    slack_configs:
      - channel: "#platform-alerts"
  - name: pagerduty-oncall
    pagerduty_configs:
      - service_key: ""
  - name: payments-slack
    slack_configs:
      - channel: "#payments-oncall"

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: [alertname, cluster]

Every change is auditable. Rollbacks are a git revert away. This matters enormously when you are debugging why an alert did not fire at 3 AM.

Lightweight and Single-Purpose

Alertmanager does one thing: route and deliver notifications. It has no dashboard, no query engine, no datasource plugins. This single-purpose design makes it operationally simple. Resource consumption is minimal — a small Alertmanager instance handles thousands of active alerts on a few hundred megabytes of memory. It starts in milliseconds and requires almost no maintenance.

Mature Inhibition and Routing

Alertmanager’s inhibition rules are first-class citizens. You can suppress downstream warnings when a critical alert is already firing, preventing alert storms from overwhelming your on-call team. The hierarchical routing tree with continue flags allows for nuanced delivery: send to the team channel AND escalate to PagerDuty simultaneously, with different grouping strategies at each level.

Proven High Availability

The gossip-based HA cluster has been stable for years. Running three Alertmanager replicas behind a load balancer (or using Kubernetes service discovery) gives you reliable notification delivery without shared storage. The protocol handles deduplication across instances automatically, which is the hardest part of distributed alerting.

Grafana Alerting Strengths

Grafana Alerting has matured considerably since its rocky introduction in Grafana 8. By Grafana 11 and 12, it has become a legitimate production alerting platform with capabilities that Alertmanager cannot match on its own.

Multi-Datasource Alert Rules

This is Grafana Alerting’s strongest differentiator. You can write alert rules that query Loki for error log spikes, CloudWatch for AWS resource utilization, Elasticsearch for application errors, or a PostgreSQL database for business metrics — all from the same alerting system. If your observability stack includes more than just Prometheus, this eliminates the need for separate alerting tools per datasource.

# Grafana alert rule provisioning example — alerting on Loki log errors
apiVersion: 1
groups:
  - orgId: 1
    name: application-errors
    folder: Production
    interval: 1m
    rules:
      - uid: loki-error-spike
        title: "High error rate in payment service"
        condition: C
        data:
          - refId: A
            datasourceUid: loki-prod
            model:
              expr: 'sum(rate({app="payment-service"} |= "ERROR" [5m]))'
          - refId: B
            datasourceUid: "__expr__"
            model:
              type: reduce
              expression: A
              reducer: last
          - refId: C
            datasourceUid: "__expr__"
            model:
              type: threshold
              expression: B
              conditions:
                - evaluator:
                    type: gt
                    params: [10]
        for: 5m
        labels:
          severity: warning
          team: payments

This is something Alertmanager simply cannot do. Alertmanager only receives pre-evaluated alerts — it has no concept of datasources or query execution.

Unified UI for Alert Management

Grafana provides a single pane of glass for alert rule creation, visualization, notification policy management, contact point configuration, and silence management. For teams where not every engineer is comfortable editing YAML routing trees, the visual notification policy editor significantly reduces the barrier to entry. You can see the state of every alert rule, its evaluation history, and the exact notification path it will take — all without leaving the browser.

Native Multi-Tenancy and RBAC

Grafana’s organization model and role-based access control extend naturally to alerting. Different teams can manage their own alert rules, contact points, and notification policies within their organization or folder scope, without seeing or interfering with other teams. Achieving this with standalone Alertmanager requires either running separate instances per tenant or using Mimir’s multi-tenant Alertmanager.

Mute Timings and Richer Scheduling

While Alertmanager supports silences (ad-hoc, time-limited suppressions), Grafana Alerting adds mute timings — recurring time-based windows where notifications are suppressed. This is useful for scheduled maintenance windows, business-hours-only alerting, or suppressing non-critical alerts on weekends. Alertmanager requires external tooling or manual silence creation for recurring windows.

Grafana Cloud as a Managed Option

For teams that want to avoid managing alerting infrastructure entirely, Grafana Cloud provides a fully managed Grafana Alerting stack. This includes HA, state persistence, and notification delivery without any self-hosted components. The Grafana Cloud alerting stack also includes a managed Mimir Alertmanager, which means you can use Prometheus-native alerting rules if you prefer that model while still benefiting from the managed infrastructure.

When to Use Prometheus Alertmanager

Alertmanager is the right choice when the following conditions describe your environment:

Your metrics stack is Prometheus-native. If all your alert rules are PromQL expressions evaluated by Prometheus, Thanos Ruler, or Mimir Ruler, Alertmanager is the natural fit. There is no added value in routing those alerts through Grafana.
GitOps is non-negotiable. If every infrastructure change must go through a pull request and be fully declarative, Alertmanager’s single-file configuration model is significantly easier to manage than Grafana’s database-backed state. Tools like amtool provide config validation in CI pipelines.
You need fine-grained routing with inhibition. Complex routing trees with multiple levels of grouping, inhibition rules, and continue flags are more naturally expressed in Alertmanager’s YAML format. The routing logic has been stable and well-documented for years.
You run microservices with per-team routing. If each team owns its routing subtree and the routing logic is complex, Alertmanager’s hierarchical model scales better than UI-driven configuration. Teams can own their section of the config file via CODEOWNERS in Git.
You want minimal operational overhead. Alertmanager is a single binary with minimal resource requirements. There is no database to back up, no migrations to run, and no UI framework to keep updated.

When to Use Grafana Alerting

Grafana Alerting is the right choice when these conditions apply:

You alert on more than just Prometheus metrics. If you need alert rules based on Loki logs, Elasticsearch queries, CloudWatch metrics, or database queries, Grafana Alerting is the only option that handles all of these natively. The alternative is running separate alerting tools per datasource, which is worse.
Your team prefers UI-driven configuration. Not every engineer wants to edit YAML routing trees. If your organization values a visual interface for managing alerts, contact points, and notification policies, Grafana’s UI is a major productivity advantage.
You are using Grafana Cloud. If you are already on Grafana Cloud, using its built-in alerting is the path of least resistance. You get HA, managed notification delive

A Tale of Accidental Architecture: How 50 Lines Became A Black Friday Disaster

Sat, 28 Feb 2026 03:46:54 +0000

Let me tell you about Sarah.

This is a fictional story. But I bet you’ll recognize it.

I’ve seen this pattern play out across different companies, different teams, different tech stacks. The details change. The progression doesn’t.

Week 1: The Perfect Start

Sarah’s building a notification system for an e-commerce platform.

First requirement: send an email when someone places an order.

Simple. She writes one function. Webhook comes in, format the email, hit SMTP, done.

The whole thing is maybe 50 lines. It works perfectly. Code review approves it. It ships.

Sarah’s thinking: “It’s just one notification type. I’ll add proper abstraction when we actually need it.”

You’ve thought this too. So have I.

Nothing wrong with it. Week 1, this is the right call.

Week 3: The First Copy-Paste

Product team loves the email notifications. Now they want SMS for order shipments.

Mike picks up the ticket.

He opens Sarah’s code. Sees the pattern. Makes sense. He follows it.

New handler. Receives the shipment webhook. Formats the SMS message. Connects to Twilio. Sends it.

He copies some of Sarah’s email formatting logic because customers should see consistent information. Has to adjust it for the 160-character SMS limit, but the core logic is the same.

Mike’s thinking: “There’s some duplication with the email code, but SMS is different enough that abstracting it would be premature. It’s only two notification types.”

Deadline is tomorrow. This ships.

Still nothing catastrophically wrong here. Two types, small duplication, it’s manageable.

Right?

Week 5: User Preferences

Customers start complaining.

“I don’t want SMS notifications.”

“Why am I getting emails for every status change?”

Sarah adds user preferences. Creates a database table. Updates her email handler to check if the user wants that particular notification before sending.

The handler triples in size.

Query the database. Check multiple preference flags. Handle the case where preferences don’t exist yet. Default values. Edge cases.

Sarah’s thinking: “This is getting messy, but the deadline is tomorrow and this works. I’ll refactor it next sprint.”

I cannot tell you how many times I’ve heard “next sprint.”

(Spoiler: next sprint never comes.)

Week 7: Two Ways to Do Everything

Mike needs to add notifications for order cancellations and delivery confirmations.

He realizes hardcoding email bodies isn’t going to scale.

So he builds a template system. Creates a templates directory. Writes a simple renderer. Updates his handlers to load templates, populate data, send.

It’s actually pretty clean.

Meanwhile, Sarah’s handlers still use string formatting. She doesn’t know Mike built a template system. Mike didn’t announce it in Slack. It just… exists now.

The codebase now has two different ways of generating notification content.

Sarah finds out later. Thinks: “I should probably switch to Mike’s templates… but my code is working and I’m slammed with other features.”

And she is. Three new features this sprint. No time to refactor working code.

Week 9: The Third Approach

Emma joins the team.

First task: add Slack notifications for the support team when high-value orders come in.

She opens the notification code. Finds Sarah’s inline approach. Finds Mike’s templates. Neither makes sense for Slack.

Slack needs structured JSON payloads, not formatted text.

So Emma does what any good engineer would do: she creates a “proper solution”.

Notification service class. Methods for each notification type. Handles destination-specific formatting internally. Clean. Testable. Well-designed.

She shows it to the team in standup.

Mike: “That’s nice, but I don’t have time to refactor my SMS code right now. Maybe later.”

Sarah: “I like it, but my code has been running in production for months. If it ain’t broke…”

Emma’s service class gets used for Slack notifications. Nothing else changes.

Now there are three ways to send notifications.

Week 12: The Chaos Compounds

Product wants:

Push notifications for the mobile app
Digest emails (daily order summaries)
Ability to snooze notifications

Three developers. Three features. Same week.

Each one discovers the existing fragmentation. Each one makes their own call.

Developer A tries to extend Sarah’s inline approach. Adds push notification logic directly in the handler.

Developer B uses Mike’s templates but creates a new template format because the existing one doesn’t support digest layouts.

Developer C tries to use Emma’s service class but realizes it doesn’t handle scheduling or snoozing. So they add that logic directly in their handler instead.

The notification preferences table is now being updated by five different code paths.

Each developer added their own columns because they didn’t realize others had added similar fields. One stores preferences as JSON. Another uses boolean columns. Another created a separate preferences table with foreign keys.

I’ve seen this code review happen. Every PR gets approved. Every piece of code works.

Nobody did anything wrong.

And yet.

“Every PR got approved. Every piece of code worked. Nobody did anything wrong. And yet.”

Week 15: Customer Complaints

Support tickets start flooding in.

“I’m getting duplicate notifications.”

“I disabled email but I’m still getting them.”

“I’m not getting notifications at all for important orders.”

Sarah investigates. Opens the codebase.

Six different code paths handle notifications. Some check preferences before sending. Some check during sending. Some don’t check at all because the developer assumed another layer was handling it.

She finds the bug. It’s in her original email handler. The preference check is wrong.

She fixes it. Deploys.

Three other notification types break. They were relying on her buggy behavior.

The team estimate to fix it properly: “We need to stop and refactor everything first, or we’ll just make it worse.”

Management: “We don’t have time for a refactor. Just fix the bugs.”

Week 17: The Template Nightmare

Marketing wants to update email designs. New brand guidelines.

The developer assigned to this opens the codebase.

Templates are everywhere.

Some in a /templates directory. Some hardcoded as strings. Some in the database. Some fetched from an external CMS that one developer integrated without telling anyone.

There’s no single source of truth.

Worse: the data passed to templates is completely inconsistent.

Email templates expect order objects with certain fields. SMS templates expect a flattened structure. Push notifications expect a completely different format.

One design change requires touching dozens of files.

The developer estimates: “Two weeks, maybe three.”

Marketing: “It’s just a design update. How is that two weeks?”

Week 20: Performance Crisis

Black Friday.

The system crashes.

Investigation reveals: notification handlers are opening new database connections for every single notification sent.

Some handlers properly close connections. Some don’t.

Connection pools exhausted. Some handlers retry failed sends immediately and indefinitely, amplifying the problem during the outage. One handler spawns a goroutine for each notification but never limits concurrency.

The server runs out of memory processing a batch of 10,000 order confirmations.

Different developers made different assumptions about error handling.

Some silently swallow errors and log them. Some retry with exponential backoff. Some fail fast. Some store failed notifications in one database table for retry. Others use a different table. One developer integrated a third-party queue system that nobody else knew existed.

Notifications are getting lost between these systems.

I’ve been on calls where the CTO asks: “How many notification systems do we have?”

Nobody can answer.

Week 24: The Audit

Compliance team asks a simple question:

“Can you show us a record of all notifications sent to customer X in the past 90 days?”

The team cannot answer this.

Notification logs are scattered everywhere.

Some handlers log to stdout. Some to files. Some to a database table. Some don’t log at all.

The log formats are completely different. Some include the full message content. Some just log “notification sent” without details. There’s no correlation between the notification and the triggering event.

The auditor asks: “How do you ensure notifications contain required legal disclosures?”

Each template was created independently. Some include required legal text. Some don’t. There’s no centralized enforcement.

I’ve seen this audit happen. Teams spend weeks reconstructing logs manually.

The Breaking Point

VP of Engineering asks for a simple feature:

“Add an unsubscribe link to all emails.”

The team estimates: Three weeks.

The VP is shocked.

“It’s just adding a link. How is that three weeks of work?”

The tech lead explains:

“We have seven different code paths that send emails. Each uses a different templating system. Some render templates on the server. Some fetch them from external systems. Some are hardcoded strings. We need to update each one individually, ensure the unsubscribe logic is consistent across all of them, add tracking for unsubscribe events, update the preferences system to handle unsubscribes properly, and test everything thoroughly because there’s no centralized testing strategy.”

Three weeks. For a link.

The VP asks the obvious question: “How did it get this bad?”

What Went Wrong?

Here’s the thing that kills me about this story.

Nobody made a catastrophically bad decision.

Sarah’s Week 1 implementation was appropriate. Mike’s template system was a reasonable improvement. Emma’s service class was a genuine attempt to bring order.

Every single developer was trying to do good work under deadline pressure.

The problem wasn’t the individual decisions.

It was the absence of a shared architectural vision.

Without clear boundaries and layers, each developer made reasonable local optimizations that created global chaos.

The “I’ll refactor it later” moments never came because there was never a good time to stop feature development.

The “let’s standardize this” conversations happened but never resulted in action because no one had time to migrate existing code.

The codebase evolved organically.

And organic growth without structure doesn’t produce a garden. It produces a weed-infested lot.

“But This Is Just a Communication Problem”

You might be thinking: the real issue was that developers didn’t communicate.

If Sarah and Mike had talked, they wouldn’t have built two different templating systems. If Emma had socialized her service class better, others would have adopted it.

Better standups. Better code reviews. Better documentation. That’s what was missing, not architecture.

This is seductive because it’s partially true.

But here’s why it misses the point: architecture IS communication.

“Architecture IS communication. It’s the most important form of communication for technical decisions.”

It’s the most important form of communication for technical decisions.

Think about what actually happened in the story.

The team DID communicate. Mike showed his template system in code review. Emma presented her service class and got positive feedback. They had a meeting in Week 11 trying to align on standards.

The communication happened.

What didn’t happen was turning those conversations into durable, enforceable decisions.

This is the key difference:

Conversation says “we should probably do X.”

Architecture says “X is how we do things here, and here’s where it lives.”

“Conversation is ephemeral. Architecture is the artifact that persists after the meeting ends.”

When a new developer joins and asks “where should notification logic go?”, the answer shouldn’t require scheduling a meeting or hunting through Slack history.

It should be obvious from looking at the codebase.

Communication without architecture leads to the problem Emma faced. She built something good. People agreed it was good. And then… nothing changed.

Without architectural decisions being explicitly made (“from now on, all notifications go through NotificationService”), the good idea just becomes another option in an increasingly fragmented codebase.

Good communication can prevent chaos. But it can’t survive bad processes.

When developers are under deadline pressure, working on different features, joining the team at different times, communication will have gaps.

Architecture is the safety net for when communication fails.

It’s the shared context that makes it possible to work somewhat independently without creating complete divergence.

So yes, the team in our story could have communicated better.

But the solution isn’t “communicate more.”

It’s “communicate the architecture and make it stick.”

Document where things belong. Make architectural decisions explicit. Enforce them in code review. Build structure that persists beyond any individual conversation.

Because at the end of the day, you can have all the Slack channels and standups and retros you want.

Without a shared architectural foundation, you’re just having the same conversations over and over while the codebase continues to fragment.

What Should Have Happened in Week 1

Sarah should have spent 30 minutes writing this:

# Notification System Architecture

## Where Things Live
- All notification logic → services/notification_service.py
- Templates → templates/ directory (Jinja2 format)
- Preference checks → services/preference_service.py
- Delivery logging → notification_log table

## How to Add a New Notification Type
1. Add template to templates/
2. Add method to NotificationService
3. Log delivery attempt (success or failure)
4. Add tests to test_notification_service.py

## Error Handling
- Retries: 3 attempts with exponential backoff (1s, 2s, 4s)
- Failed sends → dead_letter_queue table
- All errors logged with correlation ID

## Preferences
- Check preferences BEFORE sending (not during)
- Default: all notifications enabled
- Unsubscribe → set all preferences to false

That’s it. 30 minutes of work. Would have saved months of chaos.

<hr class="wp-block-separator has-alpha-channel-