Metrics and alerts

A region group reports two things you want to know before a failover, not during one:

• How far behind the passive members are. This lag is your RPO: the work a failover would lose right now.
• Which member accepts writes. It should be exactly one. None means the group has no writer and serves nothing. More than one means two databases are drifting apart.

Set up alerts for both before the group handles production traffic.

Metrics

Every member exports these on the Catalyst agent's /metrics endpoint. They sit next to the gateway probe metrics, which every region exports, whether it's in a group or not.

Metric	Type	Exported by	What it means
cra_region_replication_lag_seconds	Gauge, seconds	Passive members	How far this member's database is behind the one that accepts writes. If you promoted this member now, you'd lose about this much work.
cra_region_writable	Gauge, 1 or 0	Every member	1 when this member's database accepts writes. In a healthy group, exactly one member reports 1.
cra_gateway_probe_up	Gauge, 1 or 0	Every region	Whether this region's gateway responds. You get this signal even when the region has no customer traffic.
cra_gateway_probe_total	Counter, label result	Every region	Gateway probe results over time.

If a member reports no cra_region_writable at all, that's not the same as 0. It means the member's agent has stopped sending heartbeats, which is a separate problem that needs its own alert.

Scrape them from every member

You need these metrics from every member at once. Only passive members export cra_region_replication_lag_seconds, so a Prometheus that scrapes only the active region never sees it. And a Prometheus that scrapes only its own region can't tell you how many members accept writes.

Scrape every member into one Prometheus, or federate one Prometheus per region into a single view. Either way, keep a label that identifies each member region, so your alerts say which region they're about. See Observability to set up scraping and OpenTelemetry.

To check a member's values by hand, port-forward to the agent. You can't exec into it, because the agent image is distroless and has no shell.

kubectl -n cra-agent port-forward deploy/agent 9090:9090
curl -s localhost:9090/metrics | grep cra_region_

Alerts

These rules assume every member is scraped into one Prometheus, with a region label to tell them apart. The thresholds are starting points. Adjust them based on what you measure when you rehearse a failover.

Replication lag is higher than your RPO. Set the threshold to the amount of data loss you've agreed to accept, so the alert fires while you can still do something about it:

- alert: CatalystRegionReplicationLagHigh
  expr: cra_region_replication_lag_seconds > 30
  for: 5m
  annotations:
    summary: "Region {{ $labels.region }} is more than 30s behind. A failover now would lose that much work."

No member accepts writes. Either a failover is underway and you haven't promoted a replica yet, or the active member's database stopped accepting writes. Either way, the group is serving nothing:

- alert: CatalystRegionGroupNoWriter
  expr: max(cra_region_writable) == 0
  for: 2m
  annotations:
    summary: "No member of the region group accepts writes."

Two members accept writes. Both databases are taking writes and drifting apart, and you can't merge them later. This is expected for a few minutes during a planned failover, between promoting one member and rebuilding the other. Silence the alert for that window, and page on it any other time:

- alert: CatalystRegionGroupTwoWriters
  expr: sum(cra_region_writable) > 1
  for: 5m
  annotations:
    summary: "Two members accept writes. Their data is diverging."

A member stopped reporting. Its agent has stopped sending heartbeats, so you can't trust the other alerts for that region:

- alert: CatalystRegionNotReporting
  expr: absent_over_time(cra_region_writable[10m])
  for: 5m
  annotations:
    summary: "A region group member has stopped reporting whether it accepts writes."

Check the group without metrics

You can answer the same questions without a monitoring stack. This helps during an incident, or before your monitoring is set up.

Ask the control plane. Its view of the group includes which member accepts writes and any mismatched settings it has found:

diagrid region group get my-group --output json | jq '.status'

Or ask each member directly, using its region-specific hostname. That hostname works whether or not the region is receiving traffic:

curl -so /dev/null -w '%{http_code}\n' https://catalyst-west.catalyst.example.com/diagrid/region/writable

200 means the member accepts writes, 503 means it runs against a read-only replica, and 404 means it isn't in a group. The global load balancer health-checks this same endpoint.

Next steps

🌍

Multi-region high availability

What a region group is, what its members share, and how a failover works.

🔁

AWS failover and failback

The runbook to follow when these alerts fire, on AWS.

🔁

Azure failover and failback

The same runbook on Azure, where promotion works differently.

📊

Observability

Scrape metrics and export traces and logs from a self-managed region.