Kafka Connection Security Configuration Guide
Kafka is used by all three images: mingdaoyun-hap, mingdaoyun-hdp, and mingdaoyun-flink. Apply security configuration changes to all of them.
Connection Configuration Parameters
| Environment Variable | Description | Default Value |
|---|---|---|
ENV_KAFKA_ENDPOINTS | Kafka Broker addresses, separated by commas | 127.0.0.1:9092 |
ENV_KAFKA_SECURITY_PROTOCOL | Kafka security protocol mode | Empty |
ENV_KAFKA_SASL_MECHANISM | SASL authentication mechanism | Empty |
ENV_KAFKA_SASL_USERNAME | SASL username | Empty |
ENV_KAFKA_SASL_PASSWORD | SASL password | Empty |
ENV_KAFKA_SSL_CA_LOCATION | Kafka CA certificate path | Empty |
ENV_KAFKA_SSL_CLIENT_CA_LOCATION | Kafka client certificate path | Empty |
ENV_KAFKA_SSL_CLIENT_KEY_LOCATION | Kafka client private key path | Empty |
Security Protocol Modes
Kafka supports Plaintext, SaslPlaintext, Ssl, and SaslSsl security modes.
1. Plaintext (Default Mode)
No security authentication mechanism is enabled. Communication uses plaintext, and no SASL or SSL parameters are required.
2. SaslPlaintext (SASL Authentication Enabled)
SASL is used for authentication, but data is transmitted in plaintext.
Add the following environment variables to the configuration file:
| Environment Variable | Description |
|---|---|
ENV_KAFKA_SECURITY_PROTOCOL | Set to SaslPlaintext |
ENV_KAFKA_SASL_MECHANISM | SASL authentication mechanism, fixed to PLAIN |
ENV_KAFKA_SASL_USERNAME | SASL username |
ENV_KAFKA_SASL_PASSWORD | SASL password |
3. Ssl (TLS Encryption Enabled)
TLS/SSL encrypts communication without enabling SASL authentication.
The following environment variables configure mTLS mutual authentication:
| Environment Variable | Description |
|---|---|
ENV_KAFKA_SECURITY_PROTOCOL | Set to Ssl |
ENV_KAFKA_SSL_CA_LOCATION | CA root certificate path |
ENV_KAFKA_SSL_CLIENT_CA_LOCATION | mTLS client certificate path |
ENV_KAFKA_SSL_CLIENT_KEY_LOCATION | mTLS client private key path |
4. SaslSsl (SASL + TLS Enabled)
Enables both SASL authentication and SSL-encrypted communication (recommended security mode).
The following includes environment variables required for mTLS mutual authentication. Whether a client certificate is used depends on the Kafka server configuration.
| Environment Variable | Description |
|---|---|
ENV_KAFKA_SECURITY_PROTOCOL | Set to SaslSsl |
ENV_KAFKA_SASL_MECHANISM | SASL authentication mechanism, fixed to PLAIN |
ENV_KAFKA_SASL_USERNAME | SASL username |
ENV_KAFKA_SASL_PASSWORD | SASL password |
ENV_KAFKA_SSL_CA_LOCATION | CA root certificate (PEM format) |
ENV_KAFKA_SSL_CLIENT_CA_LOCATION | mTLS client certificate path (PEM format) |
ENV_KAFKA_SSL_CLIENT_KEY_LOCATION | mTLS client private key path (PEM format) |
Configure Client Certificates
When using Ssl or SaslSsl, prepare certificates according to the Kafka server authentication requirements:
- One-way TLS: The client verifies the Kafka server certificate and must trust the CA that issued it.
- mTLS mutual authentication: In addition to trusting the server CA, the client must provide a client certificate and private key.
When the Kafka Broker setting
ssl.client.authisnone, client certificates are not required; when it isrequired, client certificate verification is enforced.
Confirm the one-way TLS configuration and component support for the versions in use.
mTLS Certificate Files
Prepare the following PEM certificates and private key:
| File | Description |
|---|---|
ca.crt | CA certificate used to verify the Kafka server certificate |
client.crt | Client certificate |
client.key | Client private key |
Example directory:
volume/kafka/
├─ ca.crt
├─ client.crt
└─ client.key
Docker Certificate Mount Example
volumes:
- ./volume/kafka/ca.crt:/usr/local/kafka/ca.crt
- ./volume/kafka/client.crt:/usr/local/kafka/client.crt
- ./volume/kafka/client.key:/usr/local/kafka/client.key
The certificate paths in the corresponding environment variables must match the paths inside the container:
ENV_KAFKA_SSL_CA_LOCATION: '/usr/local/kafka/ca.crt'
ENV_KAFKA_SSL_CLIENT_CA_LOCATION: '/usr/local/kafka/client.crt'
ENV_KAFKA_SSL_CLIENT_KEY_LOCATION: '/usr/local/kafka/client.key'
mTLS Mutual Authentication Configuration Example
The following example configures Kafka mTLS for microservices, dedicated computing resources, and Flink in HAP cluster mode.
Configure HAP Microservices
Follow Mount Configuration Files to mount the CA certificate, client certificate, and private key separately in the container, then configure env-list in the microservices config.yaml:
ENV_KAFKA_SECURITY_PROTOCOL: 'Ssl'
ENV_KAFKA_SSL_CA_LOCATION: '/usr/local/kafka/ca.crt'
ENV_KAFKA_SSL_CLIENT_CA_LOCATION: '/usr/local/kafka/client.crt'
ENV_KAFKA_SSL_CLIENT_KEY_LOCATION: '/usr/local/kafka/client.key'
The paths in the environment variables must match the certificate paths inside the container.
Configure Dedicated Computing Resources
If dedicated computing resources are deployed in the cluster, configure their Kafka connection and certificate mounts.
-
In the
computinginstanceConfigMap incomputinginstance.yaml, add the following tocommon.kafkaindata.config.yaml:securityProtocol: Sslssl:caLocation: /etc/kafka-mtls/ca.crtclientCaLocation: /etc/kafka-mtls/client.crtclientKeyLocation: /etc/kafka-mtls/client.key -
On the Kubernetes control node, create a Secret containing the three certificate files in the dedicated computing namespace:
kubectl -n NAMESPACE create secret generic hap-kafka-computing-mtls \--from-file=ca.crt=PATH_TO_CA_CERT \--from-file=client.crt=PATH_TO_CLIENT_CERT \--from-file=client.key=PATH_TO_CLIENT_KEY -
In the Deployment in
computinginstance.yaml, addvolumeMountsto the corresponding container and add the Secret volume to the Podvolumes:volumeMounts:- name: kafka-mtlsmountPath: /etc/kafka-mtlsreadOnly: truevolumes:- name: kafka-mtlssecret:secretName: hap-kafka-computing-mtlsdefaultMode: 0440
Configure Flink
If Flink uses Kafka Reporter, configure the Reporter and mount the certificates for JobManager and TaskManager. These configurations apply to the listed HAP Flink image versions; verify the corresponding parameters for other versions.
Configure Kafka Reporter
Configure Kafka Reporter in data.flink-conf.yaml of the flink-config ConfigMap according to the image version.
- Flink 1.19.720
- Flink 1.17.1.530
Use PEM files. The client private key must be unencrypted PKCS#8 format (starting with -----BEGIN PRIVATE KEY-----), and the client certificate must include the required intermediate certificate chain.
On the Kubernetes control node, create a Secret in the namespace where Flink runs:
kubectl -n NAMESPACE create secret generic hap-kafka-flink-mtls \
--from-file=ca.crt=PATH_TO_CA_CERT \
--from-file=client.crt=PATH_TO_CLIENT_CERT \
--from-file=client.key=PATH_TO_CLIENT_KEY
The following example uses kafka_reporter. Update the Kafka address and certificate paths for your environment:
metrics.reporter.kafka_reporter.bootstrap.servers: 'KAFKA_HOST:9092'
metrics.reporter.kafka_reporter.securityProtocol: SSL
metrics.reporter.kafka_reporter.sslCaCert: /etc/kafka-mtls/ca.crt
metrics.reporter.kafka_reporter.sslClientCert: /etc/kafka-mtls/client.crt
metrics.reporter.kafka_reporter.sslClientKey: /etc/kafka-mtls/client.key
metrics.reporter.kafka_reporter.ssl.endpoint.identification.algorithm: https
Configure the same parameters for any configured kafka_reporter_running, kafka_reporter2, and kafka_reporter_running2, replacing the Reporter name.
On the server containing the certificate files, create a truststore using the CA certificate and a keystore using the client certificate and private key:
keytool -importcert -alias kafka-ca -file ca.crt \
-keystore kafka.truststore.p12 -storetype PKCS12
openssl pkcs12 -export -name kafka-client -in client.crt \
-inkey client.key -certfile ca.crt -out kafka.keystore.p12
Set passwords when prompted and include the client intermediate certificate chain if needed. Import multiple trusted CAs into the truststore separately using different aliases.
On the Kubernetes control node, create a Secret containing the certificates and PKCS12 files in the namespace where Flink runs:
kubectl -n NAMESPACE create secret generic hap-kafka-flink-mtls \
--from-file=ca.crt=PATH_TO_CA_CERT \
--from-file=client.crt=PATH_TO_CLIENT_CERT \
--from-file=client.key=PATH_TO_CLIENT_KEY \
--from-file=kafka.truststore.p12=PATH_TO_TRUSTSTORE \
--from-file=kafka.keystore.p12=PATH_TO_KEYSTORE
The following example uses kafka_reporter. The passwords must match those set when the files were created:
metrics.reporter.kafka_reporter.bootstrapServers: 'KAFKA_HOST:9092'
metrics.reporter.kafka_reporter.securityProtocol: SSL
metrics.reporter.kafka_reporter.ssl.truststore.location: /etc/kafka-mtls/kafka.truststore.p12
metrics.reporter.kafka_reporter.ssl.truststore.password: 'TRUSTSTORE_PASSWORD'
metrics.reporter.kafka_reporter.ssl.truststore.type: PKCS12
metrics.reporter.kafka_reporter.ssl.keystore.location: /etc/kafka-mtls/kafka.keystore.p12
metrics.reporter.kafka_reporter.ssl.keystore.password: 'KEYSTORE_PASSWORD'
metrics.reporter.kafka_reporter.ssl.keystore.type: PKCS12
metrics.reporter.kafka_reporter.ssl.endpoint.identification.algorithm: https
Configure the same parameters for kafka_reporter_running if configured, replacing the Reporter name.
When hostname verification is enabled, the Kafka Broker connection address must match the domain name or IP address in its certificate.
Mount Certificates
In the flink-jobmanager and flink-taskmanager Deployments in flink.yaml, add volumeMounts to each corresponding container and add the Secret volume to each Pod’s volumes:
volumeMounts:
- name: kafka-mtls
mountPath: /etc/kafka-mtls
readOnly: true
volumes:
- name: kafka-mtls
secret:
secretName: hap-kafka-flink-mtls
defaultMode: 0440
When the JobManager and TaskManager containers run as user 9999, set the following in spec.template.spec of each Deployment:
securityContext:
fsGroup: 9999
After configuring, apply the changes and restart the relevant services for them to take effect.