Multi-region high availability

Catalyst keeps your workflows and agents running when a pod restarts, a node fails, or an availability zone goes down. It can't do that when the whole cloud region goes down, because the cluster, its database, and its gateway all go down with it.

A region group covers that case. It connects several Catalyst regions to one replicated database. A project you create in the group runs in every member region and has one set of hostnames, so your applications use a single endpoint and never need to know which region is serving them.

A group is active/passive. The active member runs against the database that accepts writes, and it serves all traffic. The passive members run against read-only replicas. Their applications stay deployed and running, but they only take over after you promote their database.

At a glance

Property	What to expect
Model	Active/passive. One member accepts writes and serves all traffic.
Traffic failover	Automatic. The load balancer follows the health check within about 20 seconds.
Database failover	Manual. You promote the replica. See why it's manual.
RPO (data you lose)	The replication lag when you promote. Replication is asynchronous, so Catalyst doesn't cap it. Passive members report it in cra_region_replication_lag_seconds, so you can see it ahead of time.
RTO (time to recover)	Mostly the time you take to decide to promote. Moving traffic takes about 20 seconds.
What fails over	Catalyst's own state: projects, components, secrets, workflow history, jobs, and reminders.
What doesn't	Regional services your project uses, like a pub/sub broker or a rate-limit store. These need their own multi-region setup.
What your applications see	The same endpoint and hostnames before and after a failover. No redeploy and no configuration change.
Cost	About twice a single region. A passive member is a full-size region that serves no traffic.
Available on	Self-managed BYOC regions.

Plan the group before its regions hold anything

Every member must be the same type of region, and it must be empty when it joins. You can't move an existing project into a group or out of one. Today the documented setup is on self-managed BYOC regions, where you control the database, keys, and network a group depends on. For Dedicated Cloud or managed BYOC regions, contact Diagrid.

When to use a region group

Your goal	Use this
Keep a project running if a whole cloud region goes down	A region group
Survive losing a node or an availability zone within one region	High availability in that region
Serve users from closer locations, or add throughput	Separate regions
Separate environments, business units, or data residency areas	Separate regions

A region group is for disaster recovery, not performance. A passive member serves no traffic, so a group adds no throughput and doesn't make requests faster.

What survives what

Each column builds on the one before it. Only a region group survives losing a whole cloud region.

Failure	Single region, default install	Single region with HA	Region group
A pod restarts or crashes	Survives	Survives	Survives
A Kubernetes node is lost	Survives, with a short gap while single-replica pods move	Survives	Survives
An availability zone is lost	Down until the zone comes back	Survives	Survives
A cloud region is lost	Down until the region comes back	Down until the region comes back	Survives, after you promote the replica
A service you run yourself is lost, like a pub/sub broker or a rate-limit store	Down	Down	Down. A group fails over Catalyst's state, not your broker

In every column, workflows and agents are durable. Catalyst records them as they run, so they pick up where they left off when the platform comes back instead of starting over. The columns differ in how long the platform is unavailable.

"Single region with HA" means gateway.ha.enabled: true, with the external PostgreSQL and Kafka spread across availability zones, as described in Production planning. A region group doesn't replace this. Without HA in each member, losing one zone forces a full failover: you have to promote, and you lose whatever hadn't replicated. Most production setups use both.

Architecture

Each member's gateway has a health check endpoint at /diagrid/region/writable. It returns:

Response	Meaning
200	This region's database accepts writes. This is the active member.
503	This region runs against a read-only replica. This is a passive member.
404	This region isn't in a group.

You put a global load balancer in front of the members and point its health check at that path. It sends traffic to whichever member returns 200.

So the database decides which region is active. Catalyst reads what the database reports and publishes it, but never changes it.

What a failover does to running work

Your workflows, agents, jobs, and reminders live in the group's replicated database, so the passive member already has a copy. When you promote it, it continues from the history it has received so far.

• Workflows and agents continue from the last replicated step. The promoted member replays each one from its history and carries on. Nothing that replicated is lost, and you don't resubmit anything.
• Work that hadn't replicated is lost. Replication is asynchronous. If a workflow moved forward in the last few seconds before the failover, it resumes from its last replicated step and runs the activities after that step again.
• Activities, jobs, and reminders run at least once. If the old member started a job but hadn't recorded it as done, the promoted member runs it again. Catalyst isn't told that a promotion happened, so it can't prevent the repeat. Your handlers must be idempotent. This is the only thing a region group asks of your application code.
• Calls in progress fail. They hit a database that no longer accepts writes, or a region that traffic has already left. The SDKs retry them, and the retries succeed once the promoted member accepts writes.
• Resources you create during a failover wait. A project, component, or App ID created while no member accepts writes is set up once writes work again.
• Signed workflow history needs one PKI root for the whole group. If a project signs its workflow history and the members don't share a root CA, a workflow that resumes in another member fails signature verification. Catalyst then marks it as tampered and stops it instead of retrying it. You can't add a shared root after the regions join. See Requirements.

Throughout all of this, your applications keep using the same endpoint.

Failover: what is automatic and what is not

Catalyst never promotes a database, never demotes one, and never moves traffic between members.

• Moving traffic is automatic. The health check changes and the load balancer follows it.
• Promoting the database is not. You do that yourself.

This is on purpose, for three reasons:

• You can't undo a promotion. A database can promote a replica so it accepts writes, but it can't turn a writer back into a replica. To go back, you delete that region's database and rebuild it as a replica of the new one.
• Unreachable doesn't mean lost. A health check that has failed for two minutes might mean a gateway restart, an agent restart during a control plane outage, or a network problem between the checker and a region that is still serving its clients fine.
• Nothing stops the old writer. If you promote when you didn't need to, you end up with two databases that both accept writes and drift apart, and there's no way to merge them later. The risk isn't downtime. It's ending up with two conflicting copies of your data.

You can automate promotion, and you should if your RTO needs it. Make your automation wait longer than a simple health check would, require more than one signal, and refuse to promote a replica if it can't read the replication lag.

Requirements

A group needs more than two regions with similar settings. The settings below must match in every member. The control plane doesn't enforce any of them, and a group with a wrong setting looks healthy until a failover shows the problem.

Setting	Where you set it	What it must be	What happens if it differs
The database	agent.config.project.external_postgresql	One PostgreSQL cluster, replicated across the regions, with one writer. Each region connects to its local instance.	The members hold unrelated data, and a failover recovers nothing.
Database name and user	Your infrastructure	Exactly the same in every member.	Promoting the database replaces it instead.
The secrets provider	global.secrets.provider	postgresql in every member, so secrets are stored in the replicated database and not in one cluster's Kubernetes.	The promoted region can't read secrets the other region stored. Components fail to start, and there's nothing to recover them from.
The key encryption key	global.secrets.postgresql.kek_provider	The same key, named the same way, at the same key version.	The secret rows replicate, but the promoted region can't decrypt them.
The Dapr scheduler backend	agent.config.internal_dapr.scheduler	The PostgreSQL backend on the replicated database itself: postgresql.use_global: true, not a separate scheduler database. A separate database would be a second replication stream with a second writer to promote.	Every job and actor reminder is lost at failover.
The Dapr PKI root (optional)	agent.config.internal_dapr.pki	One root CA, with an intermediate per region. You only need it if a project in the group signs its workflow history, and you can't add it after the regions join.	A workflow that resumes in another member fails signature verification, and Catalyst stops it as tampered instead of retrying it.

The control plane checks three of these (the key encryption key, the Dapr trust anchors, and the scheduler backend) and warns you, but doesn't block you. Check the group's status after you create it, and before you create any projects in it:

diagrid region group get my-group --output json | jq '.status'

You want .status.messages to be empty. The status also shows which member currently accepts writes.

Deploy a region group on your cloud

A region group works on any cloud. Catalyst needs four things from the platform underneath it:

• PostgreSQL that replicates across regions, with a replica you can promote to accept writes.
• A global load balancer across the member regions that health-checks /diagrid/region/writable and sends traffic to the member that returns 200. Pick one whose address stays the same during a failover. The Dapr SDKs keep long-lived gRPC connections open, and those connections don't notice a DNS change.
• A key encryption key that every region can use, stored outside the clusters.
• One wildcard domain and certificate that every member serves, so the group's projects have one set of hostnames.

Any other regional service your project uses, like a pub/sub broker or a rate-limit store, needs its own multi-region setup. A group fails over Catalyst's state, not your message broker.

Platform	Status
AWS	Documented end to end, with Terraform. AWS multi-region deployment uses an RDS cross-region read replica, a KMS multi-region key, and AWS Global Accelerator.
Azure	Documented end to end, with Terraform. Azure multi-region deployment uses PostgreSQL Flexible Server cross-region read replicas and an Azure cross-region Load Balancer. The key encryption key is the one weaker spot: Azure has no managed provider yet, so the members share key material.
Other clouds and on-premises Kubernetes	Supported on any platform that provides the four pieces. Contact Diagrid if you want your design reviewed before you build it.

Manage the group

Use diagrid region group to manage groups. Each command acts on the whole group. The member regions are still managed on their own with diagrid region.

Create a group

diagrid region group create my-group --regions catalyst-west,catalyst-east

List all the members in --regions, at least two. Each one must be a region your organization owns, the same type as the others, empty, and not in another group. Names that start with diagrid- are reserved for Diagrid's own groups of public regions.

Create the group before you install Catalyst in the member regions. A region that joins after installing starts up as a standalone region, and its agent tries to create tables in a database that's read-only.

Inspect a group

diagrid region group list
diagrid region group get my-group

The table shows each group's regions, which member is writable, which members are draining, and the group's status and statusMessage. Add --output json to see details for each region, including any settings the control plane found don't match:

diagrid region group get my-group --output json | jq '.status'

Change the membership

diagrid region group update my-group --regions catalyst-west,catalyst-east,catalyst-north

--regions replaces the whole member list, so include every member you want to keep. A new member must meet the same rules as when you create a group: empty, the same type, and not in another group.

You can remove a region even if group projects still run in it. Catalyst removes those projects from the region but leaves their data in the shared PostgreSQL database alone. The group shows the region as draining until the last project is gone from it. You can also remove the member whose database accepts writes. The remaining members are then read-only until you promote one of their replicas. Catalyst doesn't promote one for you.

Delete a group

diagrid region group delete my-group

You can't delete a group that still has projects. The error names the first project it found. Delete the projects, wait for them to drain, and then delete the group. Deleting a group doesn't delete its regions. Once a region's last group project has drained, you can create projects in it again. Add --approve to skip the confirmation prompt.

Failover and failback

Every failover uses the same two actions:

• Promote the replica in the member that's taking over. It starts returning 200, and the load balancer starts sending it traffic.
• Rebuild the other member as a replica. Delete its database and rebuild it from the new writer. This is the only way to demote a member.

Planned failover

Drain the active member first, so writes stop before you move the database role. The order is: drain, promote, let traffic follow, rebuild the drained member as a replica, then undrain it.

Unplanned failover

There's nothing to drain, because traffic has already left the lost region when its health check stopped passing. But traffic hasn't gone anywhere else yet. Until you promote a replica, the group has no writer and no healthy endpoint.

When you promote:

• Work that hadn't replicated is lost.
• Writes in progress fail, and the client retries them.
• A job or reminder can run a second time, so handlers must be idempotent.

When the lost region comes back, its database still accepts writes, because nothing demoted it. Keep traffic away from it until you've rebuilt it as a replica.

Failback

To fail back, run the planned failover again with the members swapped. Neither member is special: whichever one holds the writer is the active one.

Recovery objectives (RPO and RTO)

Catalyst doesn't publish an RPO or RTO for region groups, because they depend on your setup, not on Catalyst. They come from your replication lag and your promotion process, so measure them yourself.

Objective	What sets it	How to measure it
RPO: how much data a failover loses	The replication lag when you promote. Cross-region replication is asynchronous, so Catalyst doesn't cap it.	cra_region_replication_lag_seconds, reported by every passive member. Set an alert on it, so you know your exposure before a failover.
RTO: how long recovery takes	Mostly how long you take to decide to promote. Moving traffic is the smaller part: the health check follows the database within about 20 seconds.	cra_region_writable shows which member accepts writes.

A worked example

Say your replication lag stays under 5 seconds, and your team takes 10 minutes to confirm an outage and start the promotion:

	Time	Where it goes
RPO	About 5 seconds of work	Whatever hadn't replicated when the region went down. Workflows resume from their last replicated step and run the activities after it again.
RTO	About 11 minutes	About 10 minutes to decide, under a minute to promote, and about 20 seconds for the health check to change and traffic to follow.

Most of the time goes into deciding, so improve your runbook before your infrastructure. Measure your own numbers when you rehearse: time how long it takes from "we decided to promote" to "the first write succeeded in the new region".

Rehearse a failover on a group that isn't serving production before you rely on it. A full failover and failback rebuilds both members' databases, so treat the rehearsal as planned maintenance. You can't undo it.

Limitations

• Only one member accepts writes. If a failover leaves two writers for a while, it's up to you to get back to one.
• Nothing promotes automatically, and nothing demotes at all. The only way to demote a member is to rebuild its database as a replica of the current writer.
• Every member is full size. A passive member runs the same data plane and must handle the same load, so size every member the same.
• Projects can't move. A project belongs to either a group or a single region from the moment you create it.
• A region must be empty to join a group, and a region in a group can't hold standalone projects. You can change the membership later (see Manage the group), but a region that already has projects can never join.
• Regional services don't fail over. Pub/sub brokers and rate-limit counters are regional. A project in a group needs its own multi-region broker.

Next steps

🌐

AWS multi-region deployment

Build a two-region group on AWS with Terraform.

🌐

Azure multi-region deployment

Build a two-region group on Azure with Terraform.

🔁

Failover and failback

The runbook: promote a region, move traffic, and rebuild the old one. Per cloud — AWS here, Azure alongside it.

📊

Metrics and alerts

Watch replication lag, and get alerted when no member, or more than one, accepts writes.

🌍

Regions

Create, inspect, and manage the regions a group is built from.