AWS failover and failback

This is the runbook for a two-region Catalyst region group on AWS, built with the AWS multi-region deployment guide. It covers what to do when a region goes down, how to fail over for planned maintenance, how to fail back, and what to check when something goes wrong.

Is a region down right now?

Go straight to Unplanned failover. You can read the rest of this page later.

How a failover works

Every failover on this page uses the same two actions:

• Promote the replica. It becomes a standalone primary, starts returning 200, and the accelerator starts sending it traffic.
• Rebuild the other region as a replica. Delete its databases and rebuild them from the new primary. This is the only way to demote a region.

You run both with failover.sh, which sits next to the Makefile in each region's working directory. The script plans with the same make target you used to deploy, checks the plan, and refuses to apply a plan that would delete a database you asked it to promote. It never calls the RDS API directly, so your Terraform state stays accurate and your next make plan is clean. The script doesn't make any decisions. You decide when to run it.

Promoting doesn't demote anything. RDS can promote a replica, but it can't turn a primary back into a replica. So between promoting the new primary and rebuilding the old one, both regions' databases accept writes, and both return 200. Only the traffic dial keeps clients away from the old one. That's why the drained region must stay drained until you've rebuilt it as a replica.

Unplanned failover

A region is down. Its Kubernetes API, gateway, and database are all unreachable, and nothing in it will respond until AWS brings it back.

Before you do anything, three things are already true:

• Traffic has left the lost region. Its load balancer stopped reporting healthy targets, so the accelerator stopped sending it connections. You didn't have to do anything.
• Traffic hasn't gone anywhere else. The other region is still a replica. It still returns 503, so the accelerator still treats it as unhealthy. Until you promote it, the group has no primary and no healthy endpoint.
• There's nothing to drain, and nothing to rebuild yet. You drain to stop writes before the database role moves, but here the region is already gone. And you can only rebuild a region as a replica once it responds again.

Recover

To recover, you run the promotion step from the planned procedure on its own.

1. Clear the replication source in the surviving region's terraform.tfvars:

   postgresql_replicate_source_db_arn = ""

2. Promote the replica, from that region's working directory:

   ./failover.sh promote --region us-east-1 --ingress-endpoint $INGRESS_DOMAIN

   The script shows what it's about to promote and waits for you to confirm. Then it applies the plan it showed you and checks the region until it reports that it accepts writes.

3. Set the lost region's traffic dial to 0 while it's still down. In region-group/terraform.tfvars:

   primary_traffic_dial_percentage = 0

   make group-plan
   make group-apply

   Do this before the lost region can get any traffic. This is why you set primary_gateway_lb_arn and secondary_gateway_lb_arn: with both set, the entry point stack doesn't need to contact either region, so you can apply it while one of them is down. Without them, the stack finds each load balancer by tag, and the lost region can't answer that lookup until it comes back. By then it might already be getting traffic.

What promoting costs you

Effect	Detail
Data loss	Whatever hadn't replicated is lost. That's the replication lag at the moment the region went down.
Writes in progress fail	They fail at the database, and the client retries them.
Jobs and reminders run at least once	If the old primary started a job but hadn't recorded it as done, the new primary can run it again. Catalyst isn't told that a promotion happened, so your handlers must be idempotent.
The lost region comes back as a second primary	Nothing demoted its database. When it comes back, it returns 200, its targets become healthy, and the accelerator sends it traffic alongside the promoted region. You then have two primaries whose data is drifting apart.

Once the lost region responds again, rebuild it as a replica with step 5 of the planned procedure. Keep its traffic dial at 0 until you're done.

Planned failover

Follow these five steps in order. You drain first so writes stop before the database role moves.

1. Confirm which region is active

./failover.sh status

primary:   catalyst-west.catalyst.example.com            writable     traffic dial 100%
secondary: catalyst-east.catalyst.example.com            replica      traffic dial 100%

This reads the entry point stack's state, so run it from the clone that holds that state. diagrid region group get my-group shows the control plane's view of the same thing.

2. Drain the active region

In region-group/terraform.tfvars, set that region's dial to 0, and apply:

primary_traffic_dial_percentage = 0

make group-plan
make group-apply

The accelerator stops sending new connections to that region, even though its load balancer is still healthy. New requests fail quickly instead of reaching a database that's about to change.

3. Promote the replica

In the passive region's terraform.tfvars, clear the replication source:

postgresql_replicate_source_db_arn = ""

Then, from that region's working directory:

./failover.sh promote --region us-east-1 --ingress-endpoint $INGRESS_DOMAIN

Clearing the replication source promotes the database in place. The script shows what it's about to promote and waits for you to confirm. Then it applies the plan it showed you and checks the region until it reports that it accepts writes.

If you usually pass other make variables to make apply in that directory, pass them after --, for example -- ENABLE_BASTION=false. If you leave one out, the plan tries to undo whatever that variable sets up, and the script refuses the plan.

If the script refuses to apply, the plan replaces a database instead of updating it. Replacing it would delete the data you're failing over to, so the script stops. Changing db_name or username forces a replacement, and at this step they go from unset to their actual values. So a refusal here means those two values are different in the two regions. Fix the variable that doesn't match. Don't work around the check.

A promotion also fails while the instance is in the backing-up state. That's temporary, so run it again.

4. Let traffic follow

The promoted region starts returning 200, its load balancer's targets become healthy, and the accelerator moves traffic to it. This takes less than a minute, and clients don't need to look up any new addresses.

Keep the old region drained. Its database is still a primary and still returns 200. If you undrain it now, traffic splits across two regions that both accept writes.

5. Rebuild the old region as a replica

This is the demotion. In the old region's terraform.tfvars, set postgresql_replicate_source_db_arn to the new primary's postgresql_arn output. Then, from its working directory, run:

./failover.sh follow --region us-west-2 --ingress-endpoint $INGRESS_DOMAIN

This deletes the region's database and rebuilds it as a replica. The old data is out of date by now, because the new primary has the current state, but the script still asks you to confirm. postgresql_skip_final_snapshot controls whether it keeps a snapshot first.

follow tells Terraform directly to replace the instance, instead of relying on the plan to do it. This matters from the second failover on. The AWS provider only creates a new instance for replicate_source_db if that setting wasn't in the previous state. So a database that has been a replica before would otherwise plan as an in-place update, which AWS can't do, because there's no API to demote a database.

If follow refuses because of deletion protection, the instance it's about to delete still has postgresql_deletion_protection = true. You can't clear that flag in the same apply as the rebuild. Pointing an instance at a replication source replaces it, so Terraform tries to delete it before it applies the new setting, and RDS refuses every time you retry. Clear the flag in a separate apply:

1. Leave postgresql_replicate_source_db_arn empty, and set postgresql_deletion_protection = false.
2. Run make apply REGION=us-west-2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN. This updates the instance in place and deletes nothing.
3. Set the replication source again, and run follow again.

That's why the deployment guide sets the flag to false from the start.

Once the old region is a replica, it returns 503 again. Set its dial back to 100 and apply the entry point stack. The group is back to normal and can fail over in either direction.

./failover.sh status

This confirms both parts: one primary, and no traffic dial left at 0.

Fail back

To fail back, run the planned failover again with the regions swapped: drain the current primary, promote the other region's replica, rebuild the drained region as a replica, and undrain it. There's no separate failback process, and neither region is special. Whichever region holds the primary is the active one.

After an unplanned failover, there's one extra step first. The region that went down comes back with a primary whose data has drifted, so rebuild it as a replica before you fail back to it.

Each failover rebuilds one region's databases from scratch, so a failover and failback rebuild both. Wait for the new replica to catch up before you fail back, because a promotion loses whatever hasn't replicated yet.

Recovery objectives (RPO and RTO)

See the region group page for what RPO and RTO mean for a group. On AWS:

Objective	On this setup
RPO	The RDS cross-region replication lag when you promote. This Terraform doesn't cap it. Set an alert on cra_region_replication_lag_seconds, which every passive member exports.
RTO	The time you take to decide to promote, plus the time to move traffic. Moving traffic is the smaller part. The health check changes within about 20 seconds of the database changing, and the accelerator's addresses stay the same, so there's no DNS change to wait for. Catalyst itself adds the time from the promotion to the first successful write.

Promotion is never automatic

No health check promotes a database, in this Terraform or in Catalyst. Moving traffic is automatic, but promoting the database isn't. That's on purpose.

You can automate promotion on AWS, and you should if your RTO needs it. For example, use a Route 53 health check on each region's own hostname, a CloudWatch alarm in the surviving region, and something that runs the promotion. Read what is automatic and what is not before you do. On RDS, an unneeded promotion is especially costly: the only fix is to rebuild a region as a replica, which deletes everything written to it since the promotion.

Troubleshooting

Symptom	What to check
Both regions time out, or the health check never passes	Check that healthcheck-port is the same port as httpsNodePort, and that the health check protocol is HTTPS, not HTTP.
A healthy region gets no traffic	A traffic_dial_percentage might still be at 0 from an earlier planned failover. ./failover.sh status shows both dials and which region is active.
Traffic doesn't move after a successful promotion	./failover.sh status checks each region by its own name. If the promoted region shows writable, the target group health hasn't caught up yet, so wait a minute. If it still shows replica, the promotion didn't take effect.
Both regions show writable	This is expected between a promotion and the rebuild after it. The drained region's dial keeps clients away from the old primary. At any other time, it means a rebuild never happened and the two databases are drifting apart.
The gateway load balancer was replaced	The accelerator stores the load balancer's ARN. Re-apply the entry point stack to pick up the new one. If that region's ARN is set in primary_gateway_lb_arn or secondary_gateway_lb_arn, update it first, because the stack doesn't look up a set ARN again.
A pod logs an AccessDeniedException from KMS	The service account isn't using the key encryption key (KEK) role. Check that global.serviceAccount.annotations includes eks.amazonaws.com/role-arn, and that the service account names in kek_kms_service_account_subjects match the ones the chart created. The names come from the Helm release name, so if you installed with a release name other than catalyst, they're different.
The group warns that the two regions don't resolve the same KEK	Compare aws_kms_key_id in both values files. It must be the key ID, and the same in both. The region-specific ARNs are different strings for the same key, so the check treats them as two keys.
Both regions return 503, and neither is a replica	This can happen for a short time when a Catalyst agent restarts during a control plane outage. The agent gets its group membership from the control plane and keeps it in memory, so after a restart it thinks it's standalone until the control plane sends the membership again.
The promoted region still shows it doesn't accept writes	The agent checks the database on each heartbeat, and keeps its previous answer through one failed check. Wait up to 60 seconds before you treat it as stuck.

Next steps

🌐

AWS multi-region deployment

Build the two-region group this runbook covers.

🌍

Multi-region high availability

What a region group is, what its members share, and how a failover works on any cloud.

📊

Metrics and alerts

Track your replication lag before a failover, not after.