Skip to main content

Apache Kafka pub/sub

Type: pubsub.kafka

Status: stable

Reference: https://docs.dapr.io/reference/components-reference/supported-pubsub/setup-apache-kafka/

Example​

apiVersion: cra.diagrid.io/v1beta1
kind: Component
metadata:
  name: <name>
spec:
  type: pubsub.kafka
  version: v1
  metadata:
    # Authentication type. This must be set to "password" for this authentication profile.
    - name: authType
      value: "password"
    # The SASL authentication mechanism to use.
    - name: saslMechanism
      value: "PLAINTEXT"
    # The SASL password.
    - name: saslPassword
      value: "mypassword"
    # The SASL username.
    - name: saslUsername
      value: "myuser"
    # A comma-separated list of Kafka brokers.
    - name: brokers
      value: "mycompany.com:9092,dapr-kafka.myapp.svc.cluster.local:9093"
    # The number of events to buffer in internal and external channels. (Optional)
    #- name: channelBufferSize
    #  value: "256"
    # The max amount of time for the client connection to be kept alive with the broker, as a Go duration, before closing the connection. A zero value (default) means keeping alive indefinitely. (Optional)
    #- name: clientConnectionKeepAliveInterval
    #  value: "0"
    # The interval for the client connection's topic metadata to be refreshed with the broker as a Go duration. (Optional)
    #- name: clientConnectionTopicMetadataRefreshInterval
    #  value: "9m"
    # A user-provided string sent with every request to the Kafka brokers for logging, debugging, and auditing purposes. (Optional)
    #- name: clientID
    #  value: "sarama"
    # Enables message compression. There are five types of compression available: none, gzip, snappy, lz4, and zstd. The default is none. (Optional)
    #- name: compression
    #  value: "none"
    # Disables consumer retry by setting this to "false". (Optional)
    #- name: consumeRetryEnabled
    #  value: "false"
    # The interval between retries when attempting to consume topics. (Optional)
    #- name: consumeRetryInterval
    #  value: "100ms"
    # The default number of message bytes to fetch from the broker in each request. (Optional)
    #- name: consumerFetchDefault
    #  value: "1048576"
    # The minimum number of message bytes to fetch in a request. (Optional)
    #- name: consumerFetchMin
    #  value: "1"
    # A kafka consumer group to listen on. Each record published to a topic is delivered to one consumer within each consumer group subscribed to the topic. (Optional)
    #- name: consumerGroup
    #  value: "group1"
    # The strategy to use for consumer group rebalancing. (Optional)
    #- name: consumerGroupRebalanceStrategy
    #  value: "range"
    # Isolation level for consumers. "read_uncommitted" (default) delivers all records; "read_committed" hides records belonging to open or aborted Kafka transactions and requires Kafka 0.11 or later. With consumerTransactionsEnabled the default does not apply: the level must be "read_committed" (leaving it unset selects it automatically). (Optional)
    #- name: consumerIsolationLevel
    #  value: "read_uncommitted"
    # When true, every delivery is processed inside a Kafka transaction (consume-transform-produce). The consumer isolation level must be "read_committed": leaving consumerIsolationLevel unset selects it automatically, while an explicit "read_uncommitted" fails validation. The component injects a "__txnToken" metadata entry, delivered to the app as a header. Publishes made during the handler that carry that token (HTTP: "metadata.__txnToken" query parameter; gRPC: "__txnToken" key in the publish metadata; bulk publish: request-level metadata only, not per entry) join the delivery's transaction, and the consumer offset commits with them atomically on handler success. IMPORTANT: token-carrying publishes must go through the SAME pubsub component the message was delivered by. The transaction is tracked per component instance: a publish through a different Kafka component fails loudly, while a non-Kafka component would ignore the token and publish outside any transaction. On handler failure the whole transaction aborts and the message is redelivered while consumeRetryEnabled is true (the pubsub default); with it disabled, a failed delivery is skipped once a later delivery commits a higher offset. Redelivery is governed by this component's backOff metadata (unbounded by default; set backOffMaxRetries to bound it) — Dapr resiliency policies bound the individual handler call, not the redelivery loop. The failed attempts' outputs are never visible to "read_committed" consumers. Non-Kafka side effects (databases, external calls) stay outside the transaction. With bulk subscribe the batch is all-or-nothing: any entry failure aborts and redelivers the whole batch. Requires a consumer group, producerRequiredAcks "all", producerRetryMax >= 1 and Kafka 2.5 or later (set "version" accordingly). The 2.5 floor is higher than the one for producerTransactionsEnabled because the offset commit only carries the consumer group member ID and generation from Kafka 2.5 on, and without them the broker cannot fence a revoked member that commits late. (Optional)
    #- name: consumerTransactionsEnabled
    #  value: "false"
    # The maximum duration to wait for the initial TCP connection to a broker to be established. Bounds how long Init() can block on an unreachable cluster. (Optional)
    #- name: dialTimeout
    #  value: "30s"
    # Disable TLS for transport security. This is potentially insecure and not recommended for use in production. (Optional)
    #- name: disableTls
    #  value: "false"
    # Enables URL escaping of the message header values. It allows sending headers with special characters that are usually not allowed in HTTP headers. (Optional)
    #- name: escapeHeaders
    #  value: "false"
    # A regular expression to exclude keys from being converted to/from headers from/to metadata to avoid unwanted downstream side effects. (Optional)
    #- name: excludeHeaderMetaRegex
    #  value: "^rawPayload|valueSchemaType$"
    # The interval between heartbeats to the consumer coordinator. (Optional)
    #- name: heartbeatInterval
    #  value: "3s"
    # The initial offset to use if no offset was previously committed. (Optional)
    #- name: initialOffset
    #  value: "newest"
    # The maximum size in bytes allowed for a single Kafka message. (Optional)
    #- name: maxMessageBytes
    #  value: "1024"
    # The per-request timeout for metadata refresh operations. When set to 0 (default) Sarama computes the effective timeout from the Net timeouts and retry count. (Optional)
    #- name: metadataTimeout
    #  value: "0"
    # The number of partition replicas that must acknowledge a produce request before it is considered successful. Accepted values: "all" (wait for all in-sync replicas, highest durability), "local" (wait for the partition leader only), "none" (no acknowledgement required, lowest durability). (Optional)
    #- name: producerRequiredAcks
    #  value: "all"
    # The maximum number of times to retry sending a message before giving up. (Optional)
    #- name: producerRetryMax
    #  value: "5"
    # When true, every publish (single or bulk) is wrapped in a Kafka transaction on an idempotent producer: an aborted publish is never visible to consumers reading with "read_committed", and bulk publishes become atomic (all entries commit or none do). Requires producerRequiredAcks "all", producerRetryMax >= 1 and Kafka 0.11 or later. Transactions serialize publishes and add broker round trips, lowering publish throughput. The producer's transactional.id gets a random per-instance suffix, so the guarantee is atomicity per publish; there is no transaction recovery or fencing across component restarts. (Optional)
    #- name: producerTransactionsEnabled
    #  value: "false"
    # The maximum duration to wait for a response from a broker after sending a request. (Optional)
    #- name: readTimeout
    #  value: "30s"
    # Enables caching for schemas. (Optional)
    #- name: schemaCachingEnabled
    #  value: "true"
    # The TTL for schema caching when publishing a message with latest schema available. (Optional)
    #- name: schemaLatestVersionCacheTTL
    #  value: "5m"
    # The Schema Registry credentials API Key. (Optional)
    #- name: schemaRegistryAPIKey
    #  value: "XYAXXAZ"
    # The Schema Registry credentials API Secret. (Optional)
    #- name: schemaRegistryAPISecret
    #  value: "ABCDEFGMEADFF"
    # The Schema Registry URL. (Optional)
    #- name: schemaRegistryURL
    #  value: "http://mycompany.com:8081"
    # The maximum time between heartbeats before the consumer is considered inactive and will timeout. (Optional)
    #- name: sessionTimeout
    #  value: "10s"
    # Skip TLS verification. This is potentially insecure and not recommended for use in production. (Optional)
    #- name: skipVerify
    #  value: "false"
    # Transaction timeout requested from the broker when transactions are enabled. With consumerTransactionsEnabled the transaction stays open for the whole handler invocation, so this must exceed the slowest expected handler. It must NOT exceed the broker's transaction.max.timeout.ms (15 minutes by default): the broker REJECTS larger values at producer initialization, which fails every publish and delivery. For handlers slower than the broker maximum, raise transaction.max.timeout.ms on the broker. (Optional)
    #- name: transactionTimeout
    #  value: "60s"
    # Prefix for the transactional.ids this component registers with the broker. The publish producer appends a random per-instance suffix (prefix defaults to clientID, then the consumer group, then "dapr"). With consumerTransactionsEnabled, each topic-partition uses the stable id "<prefix>-<digest>-<partition>", where digest is the first 8 bytes of SHA-256 over the consumer group, a NUL byte, and the topic, in lowercase hex (prefix defaults to clientID, then "dapr"). It must resolve to the same value on every replica for zombie fencing to work, so do not template a per-replica clientID in that case. The group and topic are digested rather than written out because "-" is legal in both, so joining them would let unrelated components register the same id and fence each other's producers indefinitely. The prefix is kept verbatim, so "<prefix>-*" broker ACLs are unaffected. To map an id seen at the broker back to a group and topic, recompute the digest for the candidates: on a shell, printf '%s\\0%s' "<consumerGroup>" "<topic>" | sha256sum | cut -c1-16. (Optional)
    #- name: transactionalIdPrefix
    #  value: "my-app"
    # Enables Avro JSON schema for serialization. Only applicable when the subscription uses valueSchemaType=Avro (Optional)
    #- name: useAvroJSON
    #  value: "false"
    # Kafka cluster version. Note that this must be set to "1.0.0" if you are using Azure Event Hubs with Kafka. (Optional)
    #- name: version
    #  value: "2.0.0.0"
    # The maximum duration to wait for a request to be transmitted to a broker. (Optional)
    #- name: writeTimeout
    #  value: "30s"

Authentication profiles​

Available authentication profiles:

  • OIDC Authentication

  • OIDC Private Key JWT Authentication

  • SASL Authentication

  • mTLS Authentication

  • No Authentication

  • AWS: Access Key ID and Secret Access Key

  • AWS: Assume IAM Role

  • AWS: IAM Roles Anywhere

OIDC Authentication​

Authenticate using OpenID Connect providing a client secret.

authType (string)​

Required - Authentication type. This must be set to "oidc" for this authentication profile.

Example value: oidc

Allowed values:

  • oidc

oidcClientID (string)​

Required - The OAuth2 client ID that has been provisioned in the identity provider.

Example value: my-client-id

oidcClientSecret (string)​

Required - The OAuth2 client secret that has been provisioned in the identity provider.

Example value: KeFg23!

oidcTokenEndpoint (string)​

Required - URL of the OAuth2 identity provider access token endpoint.

Example value: https://identity.example.com/v1/token

oidcExtensions (string)​

String containing a JSON-encoded dictionary of OAuth2/OIDC extensions to request with the access token.

Example value:

{"cluster":"kafka","poolid":"kafkapool"}

oidcScopes (string)​

Comma-delimited list of OAuth2/OIDC scopes to request with the access token. Although not required, this field is recommended.

Default value: openid

Example value: openid,kafka-prod

OIDC Private Key JWT Authentication​

Authenticate using OpenID Connect providing a client certificate and private key.

authType (string)​

Required - Authentication type. This must be set to "oidc_private_key_jwt" for this authentication profile.

Example value: oidc_private_key_jwt

Allowed values:

  • oidc_private_key_jwt

oidcClientAssertionCert (string)​

Required - PEM-encoded X.509 certificate used to advertise the client certificate in the x5c header.

Example value: -----BEGIN CERTIFICATE-----\n...

oidcClientAssertionKey (string)​

Required - PEM-encoded private key used to sign the client certificate.

Example value: -----BEGIN PRIVATE KEY-----\n...

oidcClientID (string)​

Required - The OAuth2 client ID that has been provisioned in the identity provider.

Example value: my-client-id

oidcTokenEndpoint (string)​

Required - URL of the OAuth2 identity provider access token endpoint.

Example value: https://identity.example.com/v1/token

oidcAudience (string)​

Overrides the JWT client assertion audience (aud). If not set, the component uses the issuer derived from the token endpoint URL when available; otherwise, it falls back to the token URL.

Example value: http://<idp-host>/realms/local

oidcExtensions (string)​

String containing a JSON-encoded dictionary of OAuth2/OIDC extensions to request with the access token.

Example value:

{"cluster":"kafka","poolid":"kafkapool"}

oidcKid (string)​

The JWT key ID (kid) to use for the client assertion.

Example value: 1234567890

oidcResource (string)​

Optional OAuth2 resource (audience) parameter to include in the token request when required by the identity provider.

Example value: api://kafka

oidcScopes (string)​

Comma-delimited list of OAuth2/OIDC scopes to request with the access token. Although not required, this field is recommended.

Default value: openid

Example value: openid,kafka-prod

SASL Authentication​

Authenticate using SASL.

authType (string)​

Required - Authentication type. This must be set to "password" for this authentication profile.

Example value: password

Allowed values:

  • password

saslMechanism (string)​

Required - The SASL authentication mechanism to use.

Default value: PLAINTEXT

Example value: SHA-512

Allowed values:

  • SHA-512

  • SHA-256

  • PLAINTEXT

saslPassword (string)​

Required - The SASL password.

Example value: mypassword

saslUsername (string)​

Required - The SASL username.

Example value: myuser

mTLS Authentication​

Authenticate using mTLS.

authType (string)​

Required - Authentication type. This must be set to "mtls" for this authentication profile.

Example value: mtls

Allowed values:

  • mtls

caCert (string)​

Required - Certificate authority certificate.

Example value:

-----BEGIN CERTIFICATE-----
<base64-encoded DER>
-----END CERTIFICATE-----

clientCert (string)​

Required - Client certificate.

Example value:

-----BEGIN CERTIFICATE-----
<base64-encoded DER>
-----END CERTIFICATE-----

clientKey (string)​

Required - Client key.

Example value:

-----BEGIN RSA PRIVATE KEY-----
<base64-encoded DER>
-----END RSA PRIVATE KEY-----

No Authentication​

Do not perform authentication.

authType (string)​

Required - Authentication type. This must be set to "none" for this authentication profile.

Example value: none

Allowed values:

  • none

AWS: Access Key ID and Secret Access Key​

Authenticate using an Access Key ID and Secret Access Key included in the metadata

authType (string)​

Required - Authentication type. This must be set to "awsiam" for this authentication profile.

Example value: awsiam

Allowed values:

  • awsiam

accessKey​

AWS access key associated with an IAM account

Example value: AKIAIOSFODNN7EXAMPLE

awsAccessKey (string)​

This maintains backwards compatibility with existing fields. It will be deprecated as of Dapr 1.17. Use 'accessKey' instead. If both fields are set, then 'accessKey' value will be used. AWS access key associated with an IAM account.

Example value: AKIAIOSFODNN7EXAMPLE

awsIamRoleArn (string)​

This maintains backwards compatibility with existing fields. It will be deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead. If both fields are set, then 'assumeRoleArn' value will be used. IAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.

Example value: arn:aws:iam::123456789:role/mskRole

awsRegion (string)​

This maintains backwards compatibility with existing fields. It will be deprecated as of Dapr 1.17. Use 'region' instead. The AWS Region where the AWS resource is deployed to.

Example value: us-east-1

awsSecretKey (string)​

This maintains backwards compatibility with existing fields. It will be deprecated as of Dapr 1.17. Use 'secretKey' instead. If both fields are set, then 'secretKey' value will be used. The secret key associated with the access key.

Example value: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

awsSessionToken (string)​

This maintains backwards compatibility with existing fields. It will be deprecated as of Dapr 1.17. Use 'sessionToken' instead. If both fields are set, then 'sessionToken' value will be used. AWS session token to use. A session token is only required if you are using temporary security credentials.

Example value: TOKEN

awsStsSessionName (string)​

This maintains backwards compatibility with existing fields. It will be deprecated as of Dapr 1.17. Use 'sessionName' instead. If both fields are set, then 'sessionName' value will be used. Represents the session name for assuming a role.

Default value: DaprDefaultSession

Example value: MyAppSession

region (string)​

The AWS Region where the AWS resource is deployed to. This will be marked required in Dapr 1.17.

Example value: us-east-1

secretKey​

The secret key associated with the access key

Example value: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

sessionToken (string)​

AWS session token to use. A session token is only required if you are using temporary security credentials.

Example value: TOKEN

AWS: Assume IAM Role​

Assume a specific IAM role. Note: This is only supported for Kafka and PostgreSQL.

authType (string)​

Required - Authentication type. This must be set to "awsiam" for this authentication profile.

Example value: awsiam

Allowed values:

  • awsiam

region (string)​

Required - The AWS Region where the AWS resource is deployed to.

Example value: us-east-1

assumeRoleArn (string)​

IAM role that has access to AWS resource. This is another option to authenticate with MSK and RDS Aurora aside from the AWS Credentials. This will be marked required in Dapr 1.17.

Example value: arn:aws:iam::123456789:role/mskRole

awsIamRoleArn (string)​

This maintains backwards compatibility with existing fields. It will be deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead. If both fields are set, then 'assumeRoleArn' value will be used. IAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.

Example value: arn:aws:iam::123456789:role/mskRole

awsStsSessionName (string)​

This maintains backwards compatibility with existing fields. It will be deprecated as of Dapr 1.17. Use 'sessionName' instead. If both fields are set, then 'sessionName' value will be used. Represents the session name for assuming a role.

Default value: DaprDefaultSession

Example value: MyAppSession

sessionName (string)​

The session name for assuming a role.

Default value: DaprDefaultSession

Example value: MyAppSession

AWS: IAM Roles Anywhere​

Use AWS IAM Roles Anywhere to establish trust between your AWS account and Diagrid.

assumeRoleArn​

Required - ARN of the AWS IAM role to assume in the trusting AWS account.

Example value: arn:aws:iam:012345678910:role/exampleIAMRoleName

authType (string)​

Required - Authentication type. This must be set to "awsiam" for this authentication profile.

Example value: awsiam

Allowed values:

  • awsiam

trustAnchorArn​

Required - ARN of the AWS Trust Anchor in the AWS account granting trust to the Dapr Certificate Authority.

Example value: arn:aws:rolesanywhere:us-west-1:012345678910:trust-anchor/01234568-0123-0123-0123-012345678901

trustProfileArn​

Required - ARN of the AWS IAM Profile in the trusting AWS account.

Example value: arn:aws:rolesanywhere:us-west-1:012345678910:profile/01234568-0123-0123-0123-012345678901

Metadata​

brokers (string)​

Required - A comma-separated list of Kafka brokers.

Example value: mycompany.com:9092,dapr-kafka.myapp.svc.cluster.local:9093

channelBufferSize (number)​

The number of events to buffer in internal and external channels.

Default value: 256

Example value: 128

clientConnectionKeepAliveInterval (duration)​

The max amount of time for the client connection to be kept alive with the broker, as a Go duration, before closing the connection. A zero value (default) means keeping alive indefinitely.

Default value: 0

Example value: 4m

clientConnectionTopicMetadataRefreshInterval (duration)​

The interval for the client connection's topic metadata to be refreshed with the broker as a Go duration.

Default value: 9m

Example value: 4m

clientID (string)​

A user-provided string sent with every request to the Kafka brokers for logging, debugging, and auditing purposes.

Default value: sarama

Example value: my-dapr-app

compression (string)​

Enables message compression. There are five types of compression available: none, gzip, snappy, lz4, and zstd. The default is none.

Default value: none

Example value: gzip

consumeRetryEnabled (bool)​

Disables consumer retry by setting this to "false".

Default value: false

Example value: true

consumeRetryInterval (duration)​

The interval between retries when attempting to consume topics.

Default value: 100ms

Example value: 200ms

consumerFetchDefault (number)​

The default number of message bytes to fetch from the broker in each request.

Default value: 1048576

Example value: 2097152

consumerFetchMin (number)​

The minimum number of message bytes to fetch in a request.

Default value: 1

Example value: 4

consumerGroup (string)​

A kafka consumer group to listen on. Each record published to a topic is delivered to one consumer within each consumer group subscribed to the topic.

Example value: group1

consumerGroupRebalanceStrategy (string)​

The strategy to use for consumer group rebalancing.

Default value: range

Example value: sticky

Allowed values:

  • range

  • sticky

  • roundrobin

consumerIsolationLevel (string)​

Isolation level for consumers. "read_uncommitted" (default) delivers all records; "read_committed" hides records belonging to open or aborted Kafka transactions and requires Kafka 0.11 or later. With consumerTransactionsEnabled the default does not apply: the level must be "read_committed" (leaving it unset selects it automatically).

Default value: read_uncommitted

Example value: read_committed

Allowed values:

  • read_uncommitted

  • read_committed

consumerTransactionsEnabled (bool)​

When true, every delivery is processed inside a Kafka transaction (consume-transform-produce). The consumer isolation level must be "read_committed": leaving consumerIsolationLevel unset selects it automatically, while an explicit "read_uncommitted" fails validation. The component injects a "__txnToken" metadata entry, delivered to the app as a header. Publishes made during the handler that carry that token (HTTP: "metadata.__txnToken" query parameter; gRPC: "__txnToken" key in the publish metadata; bulk publish: request-level metadata only, not per entry) join the delivery's transaction, and the consumer offset commits with them atomically on handler success. IMPORTANT: token-carrying publishes must go through the SAME pubsub component the message was delivered by. The transaction is tracked per component instance: a publish through a different Kafka component fails loudly, while a non-Kafka component would ignore the token and publish outside any transaction. On handler failure the whole transaction aborts and the message is redelivered while consumeRetryEnabled is true (the pubsub default); with it disabled, a failed delivery is skipped once a later delivery commits a higher offset. Redelivery is governed by this component's backOff metadata (unbounded by default; set backOffMaxRetries to bound it) — Dapr resiliency policies bound the individual handler call, not the redelivery loop. The failed attempts' outputs are never visible to "read_committed" consumers. Non-Kafka side effects (databases, external calls) stay outside the transaction. With bulk subscribe the batch is all-or-nothing: any entry failure aborts and redelivers the whole batch. Requires a consumer group, producerRequiredAcks "all", producerRetryMax &gt;= 1 and Kafka 2.5 or later (set "version" accordingly). The 2.5 floor is higher than the one for producerTransactionsEnabled because the offset commit only carries the consumer group member ID and generation from Kafka 2.5 on, and without them the broker cannot fence a revoked member that commits late.

Default value: false

Example value: true

dialTimeout (duration)​

The maximum duration to wait for the initial TCP connection to a broker to be established. Bounds how long Init() can block on an unreachable cluster.

Default value: 30s

Example value: 10s

disableTls (bool)​

Disable TLS for transport security. This is potentially insecure and not recommended for use in production.

Default value: false

Example value: true

escapeHeaders (bool)​

Enables URL escaping of the message header values. It allows sending headers with special characters that are usually not allowed in HTTP headers.

Default value: false

Example value: true

excludeHeaderMetaRegex (string)​

A regular expression to exclude keys from being converted to/from headers from/to metadata to avoid unwanted downstream side effects.

Example value: ^rawPayload|valueSchemaType$

heartbeatInterval (duration)​

The interval between heartbeats to the consumer coordinator.

Default value: 3s

Example value: 5s

initialOffset (string)​

The initial offset to use if no offset was previously committed.

Default value: newest

Example value: oldest

Allowed values:

  • newest

  • oldest

maxMessageBytes (number)​

The maximum size in bytes allowed for a single Kafka message.

Default value: 1024

Example value: 2048

metadataTimeout (duration)​

The per-request timeout for metadata refresh operations. When set to 0 (default) Sarama computes the effective timeout from the Net timeouts and retry count.

Default value: 0

Example value: 5s

producerRequiredAcks (string)​

The number of partition replicas that must acknowledge a produce request before it is considered successful. Accepted values: "all" (wait for all in-sync replicas, highest durability), "local" (wait for the partition leader only), "none" (no acknowledgement required, lowest durability).

Default value: all

Example value: local

producerRetryMax (number)​

The maximum number of times to retry sending a message before giving up.

Default value: 5

Example value: 3

producerTransactionsEnabled (bool)​

When true, every publish (single or bulk) is wrapped in a Kafka transaction on an idempotent producer: an aborted publish is never visible to consumers reading with "read_committed", and bulk publishes become atomic (all entries commit or none do). Requires producerRequiredAcks "all", producerRetryMax &gt;= 1 and Kafka 0.11 or later. Transactions serialize publishes and add broker round trips, lowering publish throughput. The producer's transactional.id gets a random per-instance suffix, so the guarantee is atomicity per publish; there is no transaction recovery or fencing across component restarts.

Default value: false

Example value: true

readTimeout (duration)​

The maximum duration to wait for a response from a broker after sending a request.

Default value: 30s

Example value: 10s

schemaCachingEnabled (bool)​

Enables caching for schemas.

Default value: true

Example value: true

schemaLatestVersionCacheTTL (duration)​

The TTL for schema caching when publishing a message with latest schema available.

Default value: 5m

Example value: 5m

schemaRegistryAPIKey (string)​

The Schema Registry credentials API Key.

Example value: XYAXXAZ

schemaRegistryAPISecret (string)​

The Schema Registry credentials API Secret.

Example value: ABCDEFGMEADFF

schemaRegistryURL (string)​

The Schema Registry URL.

Example value: http://mycompany.com:8081

sessionTimeout (duration)​

The maximum time between heartbeats before the consumer is considered inactive and will timeout.

Default value: 10s

Example value: 20s

skipVerify (bool)​

Skip TLS verification. This is potentially insecure and not recommended for use in production.

Default value: false

Example value: true

transactionTimeout (duration)​

Transaction timeout requested from the broker when transactions are enabled. With consumerTransactionsEnabled the transaction stays open for the whole handler invocation, so this must exceed the slowest expected handler. It must NOT exceed the broker's transaction.max.timeout.ms (15 minutes by default): the broker REJECTS larger values at producer initialization, which fails every publish and delivery. For handlers slower than the broker maximum, raise transaction.max.timeout.ms on the broker.

Default value: 60s

Example value: 90s

transactionalIdPrefix (string)​

Prefix for the transactional.ids this component registers with the broker. The publish producer appends a random per-instance suffix (prefix defaults to clientID, then the consumer group, then "dapr"). With consumerTransactionsEnabled, each topic-partition uses the stable id "<prefix>-<digest>-<partition>", where digest is the first 8 bytes of SHA-256 over the consumer group, a NUL byte, and the topic, in lowercase hex (prefix defaults to clientID, then "dapr"). It must resolve to the same value on every replica for zombie fencing to work, so do not template a per-replica clientID in that case. The group and topic are digested rather than written out because "-" is legal in both, so joining them would let unrelated components register the same id and fence each other's producers indefinitely. The prefix is kept verbatim, so "<prefix>-*" broker ACLs are unaffected. To map an id seen at the broker back to a group and topic, recompute the digest for the candidates: on a shell, printf '%s\0%s' "<consumerGroup>" "<topic>" | sha256sum | cut -c1-16.

Example value: my-app

useAvroJSON (bool)​

Enables Avro JSON schema for serialization. Only applicable when the subscription uses valueSchemaType=Avro

Default value: false

Example value: true

version (string)​

Kafka cluster version. Note that this must be set to "1.0.0" if you are using Azure Event Hubs with Kafka.

Default value: 2.0.0.0

Example value: 0.10.2.0

writeTimeout (duration)​

The maximum duration to wait for a request to be transmitted to a broker.

Default value: 30s

Example value: 10s

Last updated on