Skip to main content

Protect the regional PKI keys

When you bring your own PKI, two private keys decide who can mint a workload identity in your region:

  • The root CA key. It signs the issuer certificate. Its certificate is the trust anchor the region registers with the Catalyst control plane when it first joins, so it's long-lived, and changing it later needs a rejoin through Diagrid support.
  • The issuer key. Sentry signs every workload certificate with it, so it has to be in the cluster. Its certificate is an intermediate under the root, and you can renew it as often as you like.

This page shows how to keep the root key out of the cluster entirely, and how to limit who can read the issuer key, encrypt it and rotate it, with Kubernetes and cloud integrations.

Where the issuer key is stored​

The Catalyst agent reads the issuer certificate and key from the Secret you name in agent.config.internal_dapr.pki.issuer.secret, and installs the region's Dapr control plane with them. So the key exists in more than one Secret:

NamespaceSecretHolds
The one you choose, such as cert-managerThe one you name, such as dapr-trust-bundleThe issuer certificate and key you provide.
root-dapr-systemsh.helm.release.v1.*The Helm release of the Dapr control plane, including the issuer key, one Secret per kept revision.
root-dapr-systemdapr-trust-bundleThe issuer certificate and key that Sentry loads, as issuer.crt and issuer.key.

Everything below applies to all three. Two consequences follow:

  • Don't put the key in Helm values. The inline form, pki.issuer.cert_pem and pki.issuer.key_pem, adds the key to your values file, your CI system and the Catalyst Helm release. Use a Secret reference.
  • Moving the Secret to a locked-down namespace doesn't hide it from the agent. The agent has cluster-wide read access to Secrets, as the Helm reference explains. That namespace still keeps people and other workloads away from it.

Keep the root key out of the cluster​

The Dapr PKI guide and the AWS deployment create the root with cert-manager's self-signed issuer, so the root key sits in a Secret next to the issuer key. Anyone who can read that Secret can mint a new issuer the region trusts, for as long as the root lives.

Instead, keep the root in a managed certificate authority and let a cert-manager issuer for it sign the intermediate. The root key then never leaves the managed service, and the cluster holds only the issuer key.

AWS Private CA​

Use AWS Private CA with the aws-privateca-issuer plugin for cert-manager. Diagrid issues the intermediates for its own control plane the same way.

  1. Create a root CA in AWS Private CA, or a subordinate CA under it that is allowed to issue CA certificates. The CA's keys are generated and kept in FIPS-validated hardware security modules, and you can't export them.

  2. Install aws-privateca-issuer in the cluster. Give its service account an IAM role, with IAM Roles for Service Accounts or EKS Pod Identity, that allows acm-pca:DescribeCertificateAuthority, acm-pca:GetCertificate and acm-pca:IssueCertificate on that CA.

  3. Create the issuer and the Sentry certificate:

    apiVersion: awspca.cert-manager.io/v1beta1
    kind: AWSPCAIssuer
    metadata:
      name: dapr-pca
      namespace: cert-manager
    spec:
      arn: arn:aws:acm-pca:<region>:<account-id>:certificate-authority/<ca-id>
      region: <region>
    ---
    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: dapr-trust-bundle
      namespace: cert-manager
    spec:
      commonName: dapr-sentry
      uris:
        - "spiffe://prj-root.trust.diagrid.io/ns/root-dapr-system/dapr-sentry"
      isCA: true
      usages:
        - digital signature
        - key encipherment
        - cert sign
        - crl sign
      privateKey:
        algorithm: ECDSA
        size: 256
        rotationPolicy: Always
      duration: 2160h   # 90 days
      renewBefore: 720h # 30 days
      secretName: dapr-trust-bundle
      issuerRef:
        group: awspca.cert-manager.io
        kind: AWSPCAIssuer
        name: dapr-pca

    rotationPolicy: Always gives each renewal a new key, so a leaked key stops working when its certificate expires. A shorter duration than the one-year example in the Dapr PKI guide shortens that window.

  4. Publish the root certificate as the trust anchor. The control plane accepts only self-signed root certificates, so build the ConfigMap from the root itself, not from the ca.crt in the issued Secret, which can hold a subordinate CA:

    aws acm-pca get-certificate-authority-certificate \
      --certificate-authority-arn <root-ca-arn> \
      --query Certificate --output text > root-ca.pem
    kubectl -n cert-manager create configmap dapr-trust-anchors --from-file=ca.crt=root-ca.pem

  5. Point the agent at both, before the region first joins:

    agent:
      config:
        internal_dapr:
          pki:
            issuer:
              secret:
                name: dapr-trust-bundle
                namespace: cert-manager
            trust:
              config_map:
                name: dapr-trust-anchors
                namespace: cert-manager

Other certificate authorities​

The same pattern works with any cert-manager issuer that can sign a CA certificate, such as the Vault issuer or the Google CAS issuer: change the issuerRef, and publish the root you control as the trust anchor.

Azure Key Vault has no cert-manager issuer that signs CA certificates. On Azure, use one of the issuers above, or sign the issuer certificate outside the cluster with a root you keep offline, store it in Key Vault, and sync it into the Secret with External Secrets Operator. The Secrets Store CSI driver on its own doesn't work here: the agent reads a Kubernetes Secret through the API, not a file mounted into a pod.

Limit who can read the issuer key​

  • Keep the Secret in a namespace that only cert-manager and your platform team use. Don't grant get, list or watch on Secrets in that namespace or in root-dapr-system to application teams, and avoid ClusterRoles that grant them on all Secrets.

  • Check who has access. For example, to check a group:

    kubectl auth can-i get secrets -n cert-manager --as=dev-user --as-group=developers
    kubectl auth can-i get secrets -n root-dapr-system --as=dev-user --as-group=developers

  • Audit reads. Turn on the cluster's API server audit logs (control plane logging on EKS, kube-audit diagnostic settings on AKS, Cloud Audit Logs on GKE) and alert on reads of these Secrets by anyone other than the Catalyst agent, cert-manager and Sentry.

Encrypt the Secrets at rest​

Kubernetes stores Secrets in etcd. Encrypt them with a key in your cloud's key management service, so a copy of etcd or of a backup doesn't reveal the issuer key:

  • EKS: envelope encryption of Kubernetes Secrets with an AWS KMS key you manage.
  • AKS: KMS etcd encryption with a key in Azure Key Vault.
  • GKE: application-layer secrets encryption with a Cloud KMS key.
  • Other clusters: an EncryptionConfiguration with a KMS provider.

Back up the cluster with a tool that keeps Secrets encrypted, or leave these namespaces out of backups: cert-manager can issue a new issuer certificate from the root at any time.

Rotate the issuer​

cert-manager renews the issuer certificate in your Secret on schedule, but the agent copies it into the Dapr control plane only when the agent starts. After each renewal, restart the agent, well before the old certificate expires:

kubectl -n cra-agent rollout restart deployment agent

Then check that Sentry has the new certificate. The two commands print the same serial number when it does:

kubectl -n cert-manager get secret dapr-trust-bundle -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -serial -enddate
kubectl -n root-dapr-system get secret dapr-trust-bundle -o jsonpath='{.data.issuer\.crt}' \
  | base64 -d | openssl x509 -noout -serial -enddate

Workload certificates already issued stay valid until they expire, and Sentry signs new ones with the new key. With renewBefore: 720h you have 30 days to restart the agent, so a monthly scheduled restart, or one triggered by the renewal, is enough.

Renewing the issuer doesn't change the trust anchor. Changing the root does, and needs a region rejoin through Diagrid support.

Last updated on