AWS multi-region deployment

This guide builds a two-region region group on AWS, with one global load balancer in front of both regions. It uses the same Terraform as the AWS deployment guide.

Read Multi-region high availability first. It explains what a group is, what its members must share, and why you promote the database yourself. This page shows how to build it on AWS, with RDS cross-region read replicas, a KMS multi-region key, and AWS Global Accelerator as the global load balancer.

After you build the group, use AWS failover and failback to run it.

Architecture

Clients reach the group through the accelerator's two static IP addresses. These addresses never change, so a failover moves traffic without clients looking up a new address or waiting for a DNS TTL. This matters because Dapr sidecars and the SDKs keep long-lived gRPC connections open, and those connections don't notice a DNS change.

The accelerator sends traffic based on the health of each region's gateway load balancer. That health check asks the region whether its database accepts writes. So the region running against the primary database gets the traffic, and promoting the replica is what moves it.

It's up to you to make sure only one region holds the primary at a time. Nothing in this setup enforces it.

Both regions run a full Catalyst data plane. A passive region isn't smaller: size both the same, and budget for two.

Requirements checklist

The region group requirements list six settings the members must share. Five are always required, and one is optional. The control plane doesn't enforce any of them, so check all six before you create the group. This guide covers them like this:

Requirement	How this guide covers it	Where you set it
One replicated database	An RDS PostgreSQL instance in the first region, and a cross-region read replica of it in the second	`postgresql_replicate_source_db_arn` (Terraform)
The same database name and admin user	The same values in both regions. If they differ, promoting the database replaces it instead.	`postgresql_db_name`, `postgresql_username` (Terraform)
The PostgreSQL secrets provider	Secrets are stored in the replicated database, not in one cluster's Kubernetes	`global.secrets.provider` (Helm)
The same key encryption key	An AWS KMS multi-region key: a primary in the first region and a replica in the second. AWS treats them as one key.	`kek_kms_enabled` (Terraform), `global.secrets.postgresql.kek_provider` and `aws_kms_key_id` (Helm)
The PostgreSQL scheduler backend	Scheduler jobs and actor reminders are stored in the replicated database, so they fail over with it and there's only one writer to promote	`scheduler_postgresql_instances = []` (Terraform), `agent.config.internal_dapr.scheduler.postgresql.use_global` (Helm)
One Dapr PKI root (optional)	A root you create once for the group, with an intermediate for each region	`agent.config.internal_dapr.pki` (Helm)

The control plane warns you when two members aren't using the same key encryption key. It compares the value each region was configured with, not the key AWS resolves it to. If you point each region at the multi-region key by its own region-specific ARN, the control plane reports a mismatch even though it's the same key. That's why this guide configures both regions with the key ID instead.

Before you start

Read the AWS deployment guide and deploy a single region with it at least once. This page assumes you know those steps and only covers what's different.

You need:

Two AWS regions in one AWS account, and a wildcard domain you control.
Separate Terraform state for each region. You apply the same stack twice, once per region, from two working directories. If one configuration held both regions, Terraform couldn't plan while one region's API was down, and that's exactly when you need to promote.
Empty regions. A region that already has a project can't join a group, and a region in a group can't hold standalone projects. Create both regions, group them, and then create your projects.
A new project. You can't move an existing single-region project into a group. You need new regions, a new project, new hostnames and API keys, and you need to redeploy the applications that call it.
A decision about workflow history signing, which you make in step 1. Signing needs a shared PKI root installed in both regions before they first join. You can't add it later without Diagrid rejoining the regions for you.
Room for more Elastic IPs in both AWS regions. The stack uses four in each region (one for each NAT gateway, plus one for the bastion), and the AWS default limit is five. If anything else in the account uses Elastic IPs, check the quota before you apply and ask for an increase: aws service-quotas get-service-quota --service-code ec2 --quota-code L-0263D0A3 --region <region>.

If you run out of Elastic IPs, the error is easy to miss. Terraform allocates the addresses late in the apply, after it creates the cross-region database replicas. The AddressLimitExceeded errors scroll past, and the error you're left with is an EKS node group that never reaches ACTIVE, with Instances failed to join the kubernetes cluster. The real cause is that the NAT gateways never got addresses, so the private subnets had no route out. Check the quota instead of debugging the node group.

The Terraform is the same one the AWS deployment guide uses. It's in the guides/aws directory of the diagridio/charts repository. Every variable this page adds defaults to single-region behavior, so nothing here changes what an existing deployment builds.

Clone the repository once for each region:

git clone https://github.com/diagridio/charts.git catalyst-west
git clone https://github.com/diagridio/charts.git catalyst-east

catalyst-west/guides/aws and catalyst-east/guides/aws each hold one region's Terraform state. You also need a third state for the group's shared entry point, in guides/aws/terraform/region-group. You can keep it in either clone, as long as you always apply it from the same one.

1. Decide the shared values

You decide every value in this step once for the whole group, and both regions use the same values. Most of them can't be changed later without rebuilding something. Decide all of them before you apply anything: the values below, the key encryption key, and whether you need a shared PKI root.

# The wildcard domain both regions serve. One domain, one certificate, and one
# set of project hostnames, so your applications can't tell there are two regions.
export INGRESS_DOMAIN="catalyst.example.com"

# The database name and admin user. Use the same values in both regions. If
# they differ, promoting the database replaces it instead.
export PG_DB_NAME="catalyst"
export PG_USERNAME="postgres"

# The admin password. RDS can't create a read replica of an instance whose
# password it manages, so you set this yourself. Don't put it in
# terraform.tfvars.
export TF_VAR_postgresql_password="<choose one>"

The key encryption key

Catalyst encrypts every secret it stores in two layers. Each secret is encrypted with a data encryption key, and that key is encrypted with a key encryption key (KEK). The encrypted rows replicate between the two regions, so if the regions use different KEKs, the second region can read the rows but can't decrypt them.

The Terraform creates the KEK as an AWS KMS multi-region key. The first region creates the primary key, and the second region creates a replica of it. A replica is the same key with its own ARN in its own region. Each region encrypts with its local copy, and the key material never gets copied into a cluster. You turn this on with kek_kms_enabled in steps 3 and 5. There's nothing else to decide here.

You can't recover a lost KEK

Every secret Catalyst stores for a project in this group is encrypted with this key. If you lose the key, nobody can read those secrets in either region. If you delete it in one region, that region can't read them anymore. That's why the Terraform gives the key a 30-day deletion window.

Rotate the key in both regions together. If you rotate it in only one, the other region can't read the secrets encrypted with the new key.

If the key can't live in AWS, you can use the provider's local KEK instead. It's 32 random bytes, written as 64 hex characters, that you generate once and copy into both clusters:

export CATALYST_KEK=$(openssl rand -hex 32)

This works, but you lose what KMS gives you. You hold the key yourself, you copy it into both clusters by hand, and nothing checks that the two copies match. To use it, set kek_kms_enabled = false and use kek_provider: local in step 4.

Optional: one shared PKI root

You only need this if a project in this group will set enableWorkflowHistorySigning. Without a shared root, a workflow that starts in one region and resumes in the other fails signature verification. Catalyst then marks it as tampered and stops it instead of retrying it.

Decide now, because you can't add this later. The first time a region joins, it records its Dapr trust anchors, and the control plane rejects any later join with different anchors. To add a shared root to a region that has already joined, Diagrid has to rejoin it for you. If you skip this step, each region creates its own root, which is fine as long as nothing in the group signs its workflow history.

Create one root for the group and one intermediate for each region, before either cluster exists. Dapr Sentry requires ECDSA P-256 keys:

# Once for the group, on a machine you trust. root.key never goes on a cluster.
openssl ecparam -name prime256v1 -genkey -noout -out root.key
openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/CN=catalyst-group" \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,digitalSignature,keyCertSign,cRLSign" \
  -out root.crt

# Once per region.
for r in catalyst-west catalyst-east; do
  openssl ecparam -name prime256v1 -genkey -noout -out $r.key
  openssl req -new -key $r.key -subj "/CN=$r" -out $r.csr
  openssl x509 -req -in $r.csr -CA root.crt -CAkey root.key -CAcreateserial \
    -days 1825 -sha256 \
    -extfile <(printf "basicConstraints=critical,CA:TRUE,pathlen:0\nkeyUsage=critical,digitalSignature,keyCertSign,cRLSign\n") \
    -out $r.crt
  cat $r.crt root.crt > $r-chain.crt
  openssl verify -CAfile root.crt $r.crt   # must print: OK
done

Both regions use the same root. Each region installs its own intermediate, in steps 4 and 7.

You can rotate a region's intermediate later without affecting anything else. The root stays the same, so the other region still trusts whatever this region issues next, and history signed with the old intermediate still verifies. To rotate the root, contact Diagrid.

2. Create both regions in Diagrid Cloud

Create two regions, and keep both join tokens. You group these names in step 6. Use the same names for the EKS clusters, because the entry point gives each cluster its own DNS name.

diagrid login

export JOIN_TOKEN_WEST=$(diagrid region create catalyst-west \
  --ingress placeholder.example.com | jq -r .joinToken)

export JOIN_TOKEN_EAST=$(diagrid region create catalyst-east \
  --ingress placeholder.example.com | jq -r .joinToken)

The placeholder is there because your domain isn't delegated yet. You set both regions to the real ingress domain in steps 4 and 7.

3. Deploy the first region's AWS infrastructure

In catalyst-west/guides/aws, create this terraform/terraform.tfvars:

cluster_name = "catalyst-west"

postgresql_db_name = "catalyst"
postgresql_username = "postgres"

# RDS can't create a read replica of an instance whose password it manages,
# so a region group can't use an RDS-managed admin password.
postgresql_manage_master_user_password = false

# Both regions serve the same wildcard domain, so neither region owns its DNS
# record. The shared entry point owns it.
region_group_member = true

# Create the multi-region KMS key that encrypts the group's secrets, and the
# role the Catalyst pods use to access it. This region creates the primary key,
# and the second region replicates it.
kek_kms_enabled = true

# Don't create a separate database for the Dapr scheduler. Group members keep
# scheduler state in the shared database, so jobs and actor reminders replicate
# with everything else and there's only one writer to promote.
scheduler_postgresql_instances = []

# Failing back rebuilds this region's database as a replica of the other one,
# and RDS won't delete a protected instance. With deletion protection on, this
# region can't fail back.
postgresql_deletion_protection = false

Keep postgresql_backup_retention_period above zero. The default of 7 is fine. RDS can't replicate an instance that has automatic backups turned off.

Leave postgresql_version at its default, or set it to 16 or later. The passive region's scheduler reads changes from a read replica using logical decoding, and PostgreSQL only supports that on a replica from version 16. On an older version, the passive region's scheduler never starts. The Terraform refuses to plan a group with an older version.

Set postgresql_deletion_protection = false now, before the first apply. Turning it off later, when you rebuild a region as a replica, doesn't work in one step. Pointing an instance at a replication source replaces the instance. Terraform tries to delete it before it applies the new setting, and RDS refuses every time you retry. You'd have to clear the flag in a separate apply first, in the middle of a failover, with the group already running on one database.

Pass the AWS region and the domain on the make command line. make passes them as -var, so they override anything in terraform.tfvars:

make init
make plan  REGION=us-west-2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
make apply REGION=us-west-2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN

Then follow steps 3 to 7 of the AWS deployment guide as written: connect to the bastion host, configure kubectl, install the AWS Load Balancer Controller, install cert-manager and its Let's Encrypt ClusterIssuer, and install the monitoring tools.

Save these outputs for later:

make output postgresql_endpoint
make output postgresql_arn
make output kek_kms_key_id
make output kek_kms_key_arn
make output kek_kms_role_arn

kek_kms_key_id is the same in both regions, because a multi-region key and its replicas share one key ID. You configure both regions with it. kek_kms_key_arn is specific to this region, and the second region uses it to create the replica.

4. Install Catalyst in the first region

This is where you apply the shared settings from step 1. The values below are the single-region guide's values, plus everything a region group needs. The four blocks marked SHARED must be exactly the same in step 7.

Optional: this region's PKI intermediate

You only need this if you created a shared root in step 1. It must be in the cluster before you install Catalyst, because the agent reads it when it starts, and you can't add it after the region has joined.

The Helm install below creates the cra-agent namespace, but these two objects need to exist before that. Create the namespace yourself first:

kubectl create namespace cra-agent

kubectl -n cra-agent create secret generic catalyst-group-issuer \
  --from-file=tls.crt=catalyst-west-chain.crt \
  --from-file=tls.key=catalyst-west.key
kubectl -n cra-agent create configmap catalyst-group-trust-anchors \
  --from-file=ca.crt=root.crt

Then add this block to the values file below, next to the existing internal_dapr block. It's the same in both regions: the names match, and only the contents of the secret differ.

agent:
  config:
    internal_dapr:
      pki:
        issuer:
          secret:
            name: catalyst-group-issuer
            namespace: cra-agent
        trust:
          config_map:
            name: catalyst-group-trust-anchors
            namespace: cra-agent

The values file and the Helm install

export RDS_POSTGRESQL_ENDPOINT="<value-from-postgresql_endpoint-output>"
export RDS_POSTGRESQL_PASSWORD="<the password from step 1>"
export KEK_KEY_ID="<value-from-kek_kms_key_id-output>"
export KEK_ROLE_ARN="<value-from-kek_kms_role_arn-output>"
export JOIN_TOKEN="<value-of-JOIN_TOKEN_WEST>"

cat > catalyst-west-values.yaml << EOF
global:
  serviceAccount:
    annotations:
      # How the agent and the management service get access to KMS. They have
      # no access key, so the AWS SDK falls back to its credential chain, which
      # on EKS uses this role.
      eks.amazonaws.com/role-arn: "$KEK_ROLE_ARN"

  # SHARED: store secrets in the replicated database, not in this cluster's
  # Kubernetes, so the other region can read them after a promotion.
  secrets:
    provider: postgresql
    postgresql:
      connection_string: "postgres://postgres:$RDS_POSTGRESQL_PASSWORD@$RDS_POSTGRESQL_ENDPOINT:5432/catalyst"
      # SHARED: the key ID, exactly. A multi-region key and its replica share
      # one key ID. The control plane compares the value each region was
      # configured with, so region-specific ARNs look like two different keys.
      kek_provider: awskms
      aws_kms_key_id: "$KEK_KEY_ID"
      # NOT SHARED: each region uses its local copy of the key.
      aws_region: us-west-2
      primary_key_version: 1

agent:
  config:
    project:
      # SHARED: the replicated database. The host is this region's own
      # instance, and the user is the same in both regions. The database name
      # isn't set here. It comes from postgresql_db_name in Terraform.
      default_managed_state_store_type: postgresql-shared-external
      external_postgresql:
        enabled: true
        auth_type: connectionString
        namespace: postgresql
        connection_string_host: $RDS_POSTGRESQL_ENDPOINT
        connection_string_port: 5432
        connection_string_username: postgres
        connection_string_password: "$RDS_POSTGRESQL_PASSWORD"
    internal_dapr:
      scheduler:
        # SHARED: store jobs and actor reminders in the replicated database, so
        # they survive a failover. use_global keeps them in the database above
        # instead of a separate instance. On etcd, or on any database that
        # isn't replicated, they're lost when you fail over.
        backend_type: postgresql
        postgresql:
          use_global: true

gateway:
  tls:
    enabled: true
    secretName: "cert-wildcard"
  envoy:
    service:
      type: LoadBalancer
      httpsPort: 443
      httpsTargetPort: 8443
      # Fixed, so the health check below can use a port that doesn't change.
      httpsNodePort: 30443
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
        # This is how failover works. The targets are healthy while this
        # region's database accepts writes, and the accelerator sends traffic
        # to the region whose load balancer has healthy targets.
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "HTTPS"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/diagrid/region/writable"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "30443"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-success-codes: "200"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
EOF

helm install catalyst oci://public.ecr.aws/diagrid/catalyst \
     -n cra-agent \
     --create-namespace \
     -f catalyst-west-values.yaml \
     --set join_token="${JOIN_TOKEN}"

The example puts the database password in the values file so you can see all the shared settings in one place. In production, put the secrets provider's connection string in a Kubernetes Secret and set global.secrets.postgresql.existingSecret, as described in the Helm chart reference. The KMS key ID isn't a secret, because the role controls access to the key, not the ID. The database password is a secret.

Why the health check port is fixed. The gateway Service has two ports, and an AWS load balancer health check applies to every target group the Service creates. With traffic-port, the health check would also send an HTTPS request to the plain HTTP port, which fails. A target group with no healthy targets makes the accelerator treat the whole region as unhealthy. Fixing httpsNodePort gives the health check one port to use, so both target groups give the same answer: whether this region accepts writes. Any node port from 30000 to 32767 works, as long as each region uses it consistently.

The endpoint accepts any host name. A target group health check sends the target's own address as the Host header, and you can't change it. So the health check endpoint accepts any host name, and you don't need an extra DNS name or certificate for it. It also has no authentication, because a health check can't send credentials. The only thing it reveals is which region accepts writes.

Now finish setting up the region's ingress, following step 9 of the AWS deployment guide:

# Terraform creates the hosted zone after the gateway load balancer exists,
# which is why this apply runs after the Helm install.
make apply REGION=us-west-2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN

make output route53_zone_name_servers
make output route53_zone_id
make output region_ingress_endpoint
make output gateway_nlb_arn

diagrid region update catalyst-west \
  --ingress "<value-from-region_ingress_endpoint-output>"

These four outputs only exist once the gateway's load balancer does, which is why you read them here and not in step 3. If you ask for them before this apply, make output prints a Terraform error instead of an empty line. That means the output doesn't exist yet, not that the apply failed.

At your registrar, delegate the domain to the name servers in route53_zone_name_servers. Wait for the delegation to take effect, then issue the wildcard certificate the same way as step 9 of the single-region guide. The gateway pod isn't ready until the certificate exists.

Save route53_zone_id and gateway_nlb_arn. The second region uses this hosted zone instead of creating another one for the same domain, and the entry point stack writes the wildcard record here. The load balancer ARN lets you apply that stack later without contacting this region. See step 8.

5. Deploy the second region's AWS infrastructure

In catalyst-east/guides/aws, create this terraform/terraform.tfvars:

cluster_name = "catalyst-east"

# The same as region 1. If they differ, promoting the database replaces it.
postgresql_db_name = "catalyst"
postgresql_username = "postgres"

postgresql_manage_master_user_password = false

region_group_member = true

# Use region 1's hosted zone instead of creating a second zone for the same
# domain.
route53_zone_id = "Z0123456789ABCDEFGHIJ"

# Replicate region 1's KEK instead of creating a new key. A new key would be a
# different key, and each region could only read the secrets it stored itself.
kek_kms_enabled                = true
kek_kms_replica_source_key_arn = "arn:aws:kms:us-west-2:111122223333:key/mrk-abcdef"

# Build this region's database as a cross-region read replica of region 1's.
postgresql_replicate_source_db_arn = "arn:aws:rds:us-west-2:...:db:catalyst-west-postgresql"

# The same as region 1, for the same reasons.
scheduler_postgresql_instances = []
postgresql_deletion_protection = false

make init
make plan  REGION=us-east-1 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
make apply REGION=us-east-1 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN

REGION_INGRESS_ENDPOINT is the same as in region 1. Only REGION is different. If you leave REGION out, make builds this region in us-west-2, next to the first one, whatever the working directory is called.

A replica gets its admin password from its source through replication, so TF_VAR_postgresql_password has no effect while the database is a replica. Set it to the same value anyway, because it takes effect after a promotion.

Then prepare the cluster the same way as region 1: bastion, kubectl, AWS Load Balancer Controller, cert-manager and its ClusterIssuer, and monitoring. This region's cert-manager writes to the hosted zone that region 1 created. The IAM policy from the Terraform isn't limited to one zone, so you don't need to change anything.

6. Create the region group

Create the group now, while both regions are registered and still empty, and before you install Catalyst in the second region:

diagrid region group create my-group --regions catalyst-west,catalyst-east

The group only needs the two regions you created in step 2. A member doesn't need to be online, and the second region doesn't need Catalyst installed yet.

Why the group comes first. The second region's database is a read replica from the start, so its agent must set up without writing anything. The agent decides how to set up based on the group membership the control plane sends it when it joins. If the region joins before the group exists, the agent thinks it's a standalone region, tries to create tables in a read-only database, and exits:

level=fatal msg="error creating region mcp store: ... cannot execute CREATE TABLE
in a read-only transaction (SQLSTATE 25006)"

It fixes itself once you create the group, because the next restart picks up the membership. But if you create the group first, the region starts as a member from the beginning and never gets into a crash loop.

7. Install Catalyst in the second region

Create catalyst-east-values.yaml from the same heredoc as catalyst-west-values.yaml in step 4, with only these changes:

Set RDS_POSTGRESQL_ENDPOINT to this region's instance. Each region connects to the database instance in its own AWS region. Catalyst doesn't use a shared writer or reader endpoint.
Set aws_region to us-east-1, and KEK_ROLE_ARN to this region's own role. Each region uses its local copy of the key through a role in its own region.
Set JOIN_TOKEN to JOIN_TOKEN_EAST.
You can keep httpsNodePort at 30443.

If you created a shared root in step 1, install this region's intermediate first, the same way as in step 4, but with catalyst-east-chain.crt and catalyst-east.key. The secret and ConfigMap have the same names in both regions, so the values block stays the same.

Don't change aws_kms_key_id. The replica shares the primary key's ID, and both regions must use that ID. Before you install, run make output kek_kms_key_id in both working directories and check that they print the same value.

Everything marked SHARED stays exactly the same: the same user, the same key ID and key version, and the same scheduler backend. The database name must match too, but you set that in Terraform with postgresql_db_name, not in these values.

Compare the two values files before you install. Each cluster's API server is private, so you run kubectl and helm on each region's own bastion. The two bastions are in different VPCs and different AWS regions, so neither one has both files. Write both files on your own machine, compare them there, and then copy each one to its bastion. The mask function hides the three values that are allowed to differ (the database endpoint, the AWS region, and the role ARN), so any output left over is a mistake:

mask() {
  sed -E -e 's/[a-z0-9.-]+\.rds\.amazonaws\.com/RDS_ENDPOINT/g' \
         -e 's/(us|eu|ap)-[a-z]+-[0-9]/AWS_REGION/g' \
         -e 's#arn:aws:iam::[0-9]+:role/[A-Za-z0-9_+=,.@-]+#KEK_ROLE#g' "$1"
}
diff <(mask catalyst-west-values.yaml) <(mask catalyst-east-values.yaml)

The diff should print nothing.

On this region's bastion, install Catalyst with the same command as step 4, using this region's values file and join token:

export JOIN_TOKEN="<value-of-JOIN_TOKEN_EAST>"

helm install catalyst oci://public.ecr.aws/diagrid/catalyst \
     -n cra-agent \
     --create-namespace \
     -f catalyst-east-values.yaml \
     --set join_token="${JOIN_TOKEN}"

Then finish this region's ingress, as you did for region 1. This region doesn't create a hosted zone, because it uses region 1's. But you still run the apply after the Helm install, because the region's DNS record and the gateway_nlb_arn output both need the gateway load balancer, which the install creates.

make apply REGION=us-east-1 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN

make output region_ingress_endpoint
make output gateway_nlb_arn

diagrid region update catalyst-east \
  --ingress "<value-from-region_ingress_endpoint-output>"

This apply also creates catalyst-east.catalyst.example.com, a name that reaches only this region. Each region creates its own. A group member needs one because the wildcard name points to the shared entry point, not to the region. Save this region's gateway_nlb_arn too.

Both regions now use the same ingress domain. Issue this region's wildcard certificate the same way, from the shared hosted zone.

If you created a shared root in step 1, check that the root fingerprint is the same in both clusters:

kubectl -n root-dapr-system get configmap dapr-trust-bundle \
  -o jsonpath='{.data.ca\.crt}' | openssl x509 -noout -fingerprint -sha256

8. Create the shared entry point

Both regions are now running, each with its own gateway load balancer, but nothing resolves *.catalyst.example.com yet. The entry point is a third Terraform state. It holds one Global Accelerator in front of both load balancers, and the wildcard DNS record that points to it.

In catalyst-west/guides/aws (pick one clone and always use the same one), create terraform/region-group/terraform.tfvars:

region_ingress_endpoint = "catalyst.example.com"
route53_zone_id         = "Z0123456789ABCDEFGHIJ"

primary_aws_region   = "us-west-2"
primary_cluster_name = "catalyst-west"

secondary_aws_region   = "us-east-1"
secondary_cluster_name = "catalyst-east"

# Each region's gateway_nlb_arn output. With both set, this stack doesn't need
# to contact either region, so you can still apply it while one region is
# down. That's exactly when you need to change its traffic dial. If you leave
# them out, the stack finds each load balancer by tag, which only works while
# that region responds.
primary_gateway_lb_arn   = "arn:aws:elasticloadbalancing:us-west-2:111122223333:loadbalancer/net/k8s-craagent-gateway/abc123"
secondary_gateway_lb_arn = "arn:aws:elasticloadbalancing:us-east-1:111122223333:loadbalancer/net/k8s-craagent-gateway/def456"

make group-init
make group-plan
make group-apply

make group-output accelerator_ip_addresses
make group-output region_endpoints

This creates:

One accelerator with two static anycast addresses. It forwards TCP port 443 to both regions, and nothing else. The gateway handles TLS itself, so the accelerator passes connections through without decrypting them. accelerator_ip_addresses lists the addresses to allow in your firewall. They stay the same through every failover. If your gateway uses a different port, change listener_port.
*.catalyst.example.com, pointing to the accelerator. Every project hostname and region service name under the domain now resolves to it.

It doesn't create the per-region names. Each region already created its own name (catalyst-west.catalyst.example.com and catalyst-east.catalyst.example.com) in its earlier apply. These names point straight to the region's load balancer and skip the accelerator. They work whether or not the region is getting traffic, so you can use them to check a passive region. They live in each region's own state, so this stack needs nothing from a region except its load balancer ARN.

Check that each region answers for itself:

for r in catalyst-west catalyst-east; do
  printf '%s: ' "$r"
  curl -so /dev/null -w '%{http_code}\n' "https://$r.$INGRESS_DOMAIN/diagrid/region/writable"
done

Region 1 returns 200 and region 2 returns 503. Both are group members, region 1 has the database that accepts writes, and the passive region reports that it shouldn't get traffic. Step 10 checks this against what the control plane reports.

If either region returns 404, it isn't a group member. Check that it's in the group before you continue. If you get 000 or a TLS error, the wildcard certificate hasn't been issued yet, or the DNS name hasn't propagated.

9. Create a project

Only create projects after the group exists:

diagrid project create my-app --region-group my-group

diagrid project get my-app --output json | jq '.status.instances'

Each entry shows a region the project runs in and whether it's ready. Use jq '.status.regionEndpoints' to see the address for each region. A project in the group runs in each region and has one set of hostnames: http-prj<id>.catalyst.example.com and the matching grpc- name. They resolve through the accelerator to whichever region is active. Your applications use one endpoint and don't know when a failover happens.

If you create resources during a failover, they wait on the passive side until writes work again.

10. Verify the group

Run these four checks before the group handles anything important.

1. The control plane reports no mismatches. This is the first point where the control plane can compare the two members. It reports a mismatched KEK or an unshared PKI root here:

diagrid region group get my-group --output json | jq '.status.messages'

You want an empty list. If you see region catalyst-east does not resolve the KEK of region catalyst-west, the two regions are configured with different keys. Fix it now, before any project stores a secret. After that, each region holds rows the other can't read.

2. Exactly one member accepts writes.

diagrid region group get my-group --output json | jq '.status'

The status shows which member accepts writes. If a member reports nothing, that isn't the same as false. It means the member's agent has stopped sending heartbeats, which is a separate problem.

3. Each region agrees with the control plane. Run the loop from step 8 again. The region that check 2 showed as the writer returns 200, and the other returns 503. The passive region's load balancer targets become unhealthy within about 20 seconds, and the accelerator sends everything to the active region.

for r in catalyst-west catalyst-east; do
  printf '%s: ' "$r"
  curl -so /dev/null -w '%{http_code}\n' "https://$r.$INGRESS_DOMAIN/diagrid/region/writable"
done

4. Replication lag reaches your monitoring. Every passive member exports cra_region_replication_lag_seconds. That metric is your RPO, so check that your monitoring collects it, and set an alert on it before you go live.

11. Rehearse a failover

Do this before the group serves production.

A full failover and failback with AWS failover and failback rebuilds both regions' databases, so treat the rehearsal as planned maintenance. You can't undo it. Rehearse:

A planned failover from region 1 to region 2. Time each step.
A failback to region 1, so the group ends up where it started.
An unplanned failover. Simulate it by setting the active region's traffic dial to 0, then promote the other region without draining first.

Time how long it takes from "we decided to promote" to "the first write succeeded in the new region". That's your real RTO, and most of it is decision time, not machine time.

Tear the group down

For each region, follow the single-region guide's teardown. Run helm uninstall first, so the AWS Load Balancer Controller deletes the gateway's load balancer while it's still running. A group also needs things done in a specific order:

Delete the projects, then the group. Run diagrid project delete, then diagrid region group delete my-group. You can't delete a group that still has projects, and the error names the first one it finds. Deleting the group doesn't delete its regions. Once a region's last group project has drained, you can create projects in it again.
Destroy the shared entry point. In the working directory that holds its state, run make group-destroy. This removes the accelerator and the wildcard record. Every project hostname stops working at this point, so do it before you tear down either region.
Destroy the replica's region before the region it replicates from, so AWS doesn't promote an instance you're about to delete.
Destroy the region that created the hosted zone last, whichever region is active by then. Region 1 created the zone, and region 2 only added records to it. You can't delete the zone while region 2's record is still in it.

Steps 3 and 4 depend on what each region is, not on which one is active. If the group has failed over and not failed back, the roles are swapped: region 1 still owns the zone, but it's now the replica. Follow the steps in order and they still work.

Both regions need postgresql_deletion_protection = false, which the tfvars in step 3 and step 5 already set. Each region takes a final snapshot of its database when you destroy it, unless you set postgresql_skip_final_snapshot.

The group's KMS key is a multi-region key. When you destroy it, AWS schedules the primary and its replica for deletion after a 30-day waiting period instead of deleting them right away. That's intended: it means you can recover from an accidental destroy.

Limitations

These limits come from this Terraform, on top of the limitations of a region group:

It doesn't promote anything by itself. The only thing that reacts when a region stops accepting writes is the load balancer health check, and that only moves traffic.
It doesn't demote anything. RDS can't turn a primary back into a replica, so the only way to demote a region is to destroy its databases and rebuild them as replicas of the new primary.
It doesn't cap replication lag, but the Catalyst agent does measure it.
It doesn't fail over managed Kafka pub/sub or Redis rate-limit counters, which are regional. A project in a group needs its own pub/sub on a multi-region broker.
It can't size the two regions differently. Both regions use the same sizing variables, because a passive region is a full region.

Next steps

🔁

Architecture​

Requirements checklist​

Before you start​

1. Decide the shared values​

The key encryption key​

Optional: one shared PKI root​

2. Create both regions in Diagrid Cloud​

3. Deploy the first region's AWS infrastructure​

4. Install Catalyst in the first region​

Optional: this region's PKI intermediate​

The values file and the Helm install​

5. Deploy the second region's AWS infrastructure​

6. Create the region group​

7. Install Catalyst in the second region​

8. Create the shared entry point​

9. Create a project​

10. Verify the group​

11. Rehearse a failover​

Tear the group down​

Limitations​

Next steps​

AWS failover and failback

Multi-region high availability

Production planning