Kafka Connection Security Configuration Guide
Kafka is used in the two images of this product, mingdaoyun-hdp and mingdaoyun-flink. Adjusting the security configuration requires overall modification.
Description of connection configuration parameters
| Environment variables | Description | Default value |
|---|---|---|
ENV_KAFKA_ENDPOINTS | Kafka Broker address list, multiple addresses separated by commas | 127.0.0.1:9092 |
ENV_KAFKA_SECURITY_PROTOCOL | Kafka security protocol mode | empty |
ENV_KAFKA_SASL_MECHANISM | SASL authentication mechanism | empty |
ENV_KAFKA_SASL_USERNAME | SASL username | empty |
ENV_KAFKA_SASL_PASSWORD | SASL password | empty |
ENV_KAFKA_SSL_CA_LOCATION | Kafka CA certificate path | Empty |
ENV_KAFKA_SSL_CLIENT_CA_LOCATION | Kafka client certificate path | empty |
ENV_KAFKA_SSL_CLIENT_KEY_LOCATION | Kafka client private key path | Empty |
Security protocol mode description
Kafka supports multiple security authentication modes, including Plaintext, SaslPlaintext, Ssl, SaslSsl.
1. Plaintext (default mode)
No security authentication mechanism is enabled, only clear text communication is used, and no SASL or SSL related parameters need to be configured.
2. SaslPlaintext (enable SASL authentication)
Authentication is performed using the SASL mechanism, but data transmission remains clear text.
The configuration file needs to add the following environment variables:
| Environment variables | Description |
|---|---|
ENV_KAFKA_SECURITY_PROTOCOL | Set to SaslPlaintext |
ENV_KAFKA_SASL_MECHANISM | SASL authentication mechanism, fixed to PLAIN |
ENV_KAFKA_SASL_USERNAME | SASL username |
ENV_KAFKA_SASL_PASSWORD | SASL password |
3. Ssl (enable TLS encryption)
Encrypt communications using TLS/SSL, but do not enable SASL authentication.
The configuration file needs to add the following environment variables:
| Environment variables | Description |
|---|---|
ENV_KAFKA_SECURITY_PROTOCOL | Set to Ssl |
ENV_KAFKA_SSL_CA_LOCATION | CA root certificate path |
ENV_KAFKA_SSL_CLIENT_CA_LOCATION | Client certificate path |
ENV_KAFKA_SSL_CLIENT_KEY_LOCATION | Client private key path |
4. SaslSsl (enable SASL + TLS)
Enable both SASL authentication and SSL encrypted communication (recommended security mode).
The configuration file needs to add the following environment variables:
| Environment variables | Description |
|---|---|
ENV_KAFKA_SECURITY_PROTOCOL | Set to SaslSsl |
ENV_KAFKA_SASL_MECHANISM | SASL authentication mechanism, fixed to PLAIN |
ENV_KAFKA_SASL_USERNAME | SASL username |
ENV_KAFKA_SASL_PASSWORD | SASL password |
ENV_KAFKA_SSL_CA_LOCATION | CA root certificate (PEM format) |
ENV_KAFKA_SSL_CLIENT_CA_LOCATION | Client certificate (PEM format) |
ENV_KAFKA_SSL_CLIENT_KEY_LOCATION | Client private key (PEM format) |
TLS certificate configuration
When using Ssl or SaslSsl, a TLS certificate is required in the container.
Certificate File
The following PEM certificates need to be prepared:
| Documentation | Description |
|---|---|
ca.crt | Kafka CA root certificate |
client.crt | Client Certificate |
client.key | Client private key |
Directory example:
volume/kafka/
├─ ca.crt
├─ client.crt
└─ client.key
Docker mounting certificate example
The certificate needs to be mounted inside the container:
volumes:
- ./volume/kafka/ca.crt:/usr/local/kafka/ca.crt
- ./volume/kafka/client.crt:/usr/local/kafka/client.crt
- ./volume/kafka/client.key:/usr/local/kafka/client.key
The corresponding environment variable values are as follows (the environment variable path must be consistent with the internal path of the container):
ENV_KAFKA_SSL_CA_LOCATION: "/usr/local/kafka/ca.crt"
ENV_KAFKA_SSL_CLIENT_CA_LOCATION: "/usr/local/kafka/client.crt"
ENV_KAFKA_SSL_CLIENT_KEY_LOCATION: "/usr/local/kafka/client.key"