Skip to main content

Kafka Connection Security Configuration Guide

Note

Kafka is used by all three images: mingdaoyun-hap, mingdaoyun-hdp, and mingdaoyun-flink. Apply security configuration changes to all of them.

Connection Configuration Parameters​

Environment VariableDescriptionDefault Value
ENV_KAFKA_ENDPOINTSKafka Broker addresses, separated by commas127.0.0.1:9092
ENV_KAFKA_SECURITY_PROTOCOLKafka security protocol modeEmpty
ENV_KAFKA_SASL_MECHANISMSASL authentication mechanismEmpty
ENV_KAFKA_SASL_USERNAMESASL usernameEmpty
ENV_KAFKA_SASL_PASSWORDSASL passwordEmpty
ENV_KAFKA_SSL_CA_LOCATIONKafka CA certificate pathEmpty
ENV_KAFKA_SSL_CLIENT_CA_LOCATIONKafka client certificate pathEmpty
ENV_KAFKA_SSL_CLIENT_KEY_LOCATIONKafka client private key pathEmpty

Security Protocol Modes​

Kafka supports Plaintext, SaslPlaintext, Ssl, and SaslSsl security modes.

1. Plaintext (Default Mode)​

No security authentication mechanism is enabled. Communication uses plaintext, and no SASL or SSL parameters are required.

2. SaslPlaintext (SASL Authentication Enabled)​

SASL is used for authentication, but data is transmitted in plaintext.

Add the following environment variables to the configuration file:

Environment VariableDescription
ENV_KAFKA_SECURITY_PROTOCOLSet to SaslPlaintext
ENV_KAFKA_SASL_MECHANISMSASL authentication mechanism, fixed to PLAIN
ENV_KAFKA_SASL_USERNAMESASL username
ENV_KAFKA_SASL_PASSWORDSASL password

3. Ssl (TLS Encryption Enabled)​

TLS/SSL encrypts communication without enabling SASL authentication.

The following environment variables configure mTLS mutual authentication:

Environment VariableDescription
ENV_KAFKA_SECURITY_PROTOCOLSet to Ssl
ENV_KAFKA_SSL_CA_LOCATIONCA root certificate path
ENV_KAFKA_SSL_CLIENT_CA_LOCATIONmTLS client certificate path
ENV_KAFKA_SSL_CLIENT_KEY_LOCATIONmTLS client private key path

4. SaslSsl (SASL + TLS Enabled)​

Enables both SASL authentication and SSL-encrypted communication (recommended security mode).

The following includes environment variables required for mTLS mutual authentication. Whether a client certificate is used depends on the Kafka server configuration.

Environment VariableDescription
ENV_KAFKA_SECURITY_PROTOCOLSet to SaslSsl
ENV_KAFKA_SASL_MECHANISMSASL authentication mechanism, fixed to PLAIN
ENV_KAFKA_SASL_USERNAMESASL username
ENV_KAFKA_SASL_PASSWORDSASL password
ENV_KAFKA_SSL_CA_LOCATIONCA root certificate (PEM format)
ENV_KAFKA_SSL_CLIENT_CA_LOCATIONmTLS client certificate path (PEM format)
ENV_KAFKA_SSL_CLIENT_KEY_LOCATIONmTLS client private key path (PEM format)

Configure Client Certificates​

When using Ssl or SaslSsl, prepare certificates according to the Kafka server authentication requirements:

  • One-way TLS: The client verifies the Kafka server certificate and must trust the CA that issued it.
  • mTLS mutual authentication: In addition to trusting the server CA, the client must provide a client certificate and private key.

When the Kafka Broker setting ssl.client.auth is none, client certificates are not required; when it is required, client certificate verification is enforced.

Confirm the one-way TLS configuration and component support for the versions in use.

mTLS Certificate Files

Prepare the following PEM certificates and private key:

FileDescription
ca.crtCA certificate used to verify the Kafka server certificate
client.crtClient certificate
client.keyClient private key

Example directory:

volume/kafka/
├─ ca.crt
├─ client.crt
└─ client.key

Docker Certificate Mount Example

volumes:
- ./volume/kafka/ca.crt:/usr/local/kafka/ca.crt
- ./volume/kafka/client.crt:/usr/local/kafka/client.crt
- ./volume/kafka/client.key:/usr/local/kafka/client.key

The certificate paths in the corresponding environment variables must match the paths inside the container:

ENV_KAFKA_SSL_CA_LOCATION: '/usr/local/kafka/ca.crt'
ENV_KAFKA_SSL_CLIENT_CA_LOCATION: '/usr/local/kafka/client.crt'
ENV_KAFKA_SSL_CLIENT_KEY_LOCATION: '/usr/local/kafka/client.key'

mTLS Mutual Authentication Configuration Example​

The following example configures Kafka mTLS for microservices, dedicated computing resources, and Flink in HAP cluster mode.

Configure HAP Microservices​

Follow Mount Configuration Files to mount the CA certificate, client certificate, and private key separately in the container, then configure env-list in the microservices config.yaml:

ENV_KAFKA_SECURITY_PROTOCOL: 'Ssl'
ENV_KAFKA_SSL_CA_LOCATION: '/usr/local/kafka/ca.crt'
ENV_KAFKA_SSL_CLIENT_CA_LOCATION: '/usr/local/kafka/client.crt'
ENV_KAFKA_SSL_CLIENT_KEY_LOCATION: '/usr/local/kafka/client.key'

The paths in the environment variables must match the certificate paths inside the container.

Configure Dedicated Computing Resources​

If dedicated computing resources are deployed in the cluster, configure their Kafka connection and certificate mounts.

  1. In the computinginstance ConfigMap in computinginstance.yaml, add the following to common.kafka in data.config.yaml:

    securityProtocol: Ssl
    ssl:
    caLocation: /etc/kafka-mtls/ca.crt
    clientCaLocation: /etc/kafka-mtls/client.crt
    clientKeyLocation: /etc/kafka-mtls/client.key
  2. On the Kubernetes control node, create a Secret containing the three certificate files in the dedicated computing namespace:

    kubectl -n NAMESPACE create secret generic hap-kafka-computing-mtls \
    --from-file=ca.crt=PATH_TO_CA_CERT \
    --from-file=client.crt=PATH_TO_CLIENT_CERT \
    --from-file=client.key=PATH_TO_CLIENT_KEY
  3. In the Deployment in computinginstance.yaml, add volumeMounts to the corresponding container and add the Secret volume to the Pod volumes:

    volumeMounts:
    - name: kafka-mtls
    mountPath: /etc/kafka-mtls
    readOnly: true
    volumes:
    - name: kafka-mtls
    secret:
    secretName: hap-kafka-computing-mtls
    defaultMode: 0440

If Flink uses Kafka Reporter, configure the Reporter and mount the certificates for JobManager and TaskManager. These configurations apply to the listed HAP Flink image versions; verify the corresponding parameters for other versions.

Configure Kafka Reporter​

Configure Kafka Reporter in data.flink-conf.yaml of the flink-config ConfigMap according to the image version.

Use PEM files. The client private key must be unencrypted PKCS#8 format (starting with -----BEGIN PRIVATE KEY-----), and the client certificate must include the required intermediate certificate chain.

On the Kubernetes control node, create a Secret in the namespace where Flink runs:

kubectl -n NAMESPACE create secret generic hap-kafka-flink-mtls \
--from-file=ca.crt=PATH_TO_CA_CERT \
--from-file=client.crt=PATH_TO_CLIENT_CERT \
--from-file=client.key=PATH_TO_CLIENT_KEY

The following example uses kafka_reporter. Update the Kafka address and certificate paths for your environment:

metrics.reporter.kafka_reporter.bootstrap.servers: 'KAFKA_HOST:9092'
metrics.reporter.kafka_reporter.securityProtocol: SSL
metrics.reporter.kafka_reporter.sslCaCert: /etc/kafka-mtls/ca.crt
metrics.reporter.kafka_reporter.sslClientCert: /etc/kafka-mtls/client.crt
metrics.reporter.kafka_reporter.sslClientKey: /etc/kafka-mtls/client.key
metrics.reporter.kafka_reporter.ssl.endpoint.identification.algorithm: https

Configure the same parameters for any configured kafka_reporter_running, kafka_reporter2, and kafka_reporter_running2, replacing the Reporter name.

When hostname verification is enabled, the Kafka Broker connection address must match the domain name or IP address in its certificate.

Mount Certificates​

In the flink-jobmanager and flink-taskmanager Deployments in flink.yaml, add volumeMounts to each corresponding container and add the Secret volume to each Pod’s volumes:

volumeMounts:
- name: kafka-mtls
mountPath: /etc/kafka-mtls
readOnly: true
volumes:
- name: kafka-mtls
secret:
secretName: hap-kafka-flink-mtls
defaultMode: 0440

When the JobManager and TaskManager containers run as user 9999, set the following in spec.template.spec of each Deployment:

securityContext:
fsGroup: 9999

After configuring, apply the changes and restart the relevant services for them to take effect.