Skip to main content

How to Independently Deploy Flink Service

Flink is an extension module in the HAP system. Users can choose whether to enable it. For quick deployment, refer to Deploy Flink Service.

Quick deployment involves deploying the Flink service on the same server as the HAP microservices, which requires high hardware resource availability. If a single server cannot meet the requirements, you can refer to this document to independently deploy Flink service on a new server. The server configuration reference can be found at Single-Machine Flink Server Configuration.

Install Docker​

To install Docker, refer to the official instructions for different Linux versions at Docker Installation Guide or refer to the Docker Installation Section in the deployment example.

Check if the Current HAP Service Includes an Independent MinIO Service​

Execute the following command on the HAP server to check if the output contains a line with the keyword start minio.

docker logs $(docker ps | grep mingdaoyun-sc | awk '{print $1}') | grep minio
  • If output is present, it indicates the presence of an independent MinIO service.
  • If no output is present, it indicates the absence of an independent MinIO service.

HAP Microservice Adjustments​

The Flink service requires access to MinIO, Kafka, and MongoDB services. In this case, the local storage components' three service access ports (in standalone mode) need to be mapped. If HAP is in cluster mode, no adjustments are needed.

Refer to the following configuration example and modify the docker-compose.yaml file:

  1. Map the service ports of MinIO, Kafka, and MongoDB.

    The aggregation table feature requires Flink to access the MongoDB service. If the aggregation table feature is not enabled, MongoDB port mapping is not required.

    If MongoDB port mapping is necessary, note that the built-in MongoDB in standalone mode does not enable authentication by default. Read the documentation on Data Security carefully before making changes.

  2. Configure the environment variables for HAP to connect to the Flink service.

    app:
      environment:
        ENV_FLINK_URL: http://192.168.10.30:58081 # Added: Specify the address of the Flink service. Modify with the actual IP address.
      
    sc:
      ports:
        - 9010:9010 # New MinIO port mapping
        - 9092:9092 # New Kafka port mapping
        - 27017:27017 # New MongoDB port mapping
    Example docker-compose.yaml Configuration File
    version: '3'

    services:
      app:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-hap:7.1.0
        environment: &app-environment
          ENV_ADDRESS_MAIN: "https://hap.domain.com"
          ENV_APP_VERSION: "7.1.0"
          ENV_API_TOKEN: "******"
          ENV_FLINK_URL: http://192.168.10.30:58081 # Added: Specify the address of the Flink service. Modify with the actual IP address
        ports:
          - 8880:8880
        volumes:
          - ./volume/data/:/data/
          - ../data:/data/mingdao/data

      sc:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-sc:3.2.0
        environment:
          <<: *app-environment
        volumes:
          - ./volume/data/:/data/
        ports:
          - 9010:9010 # New MinIO port mapping
          - 9092:9092 # New Kafka port mapping
          - 27017:27017 # New MongoDB port mapping
        volumes:
          - ./volume/data/:/data/

After modifying, execute the following command from the management script directory:

bash service.sh restartall

This will restart the microservices for the changes to take effect.

  1. Initialize swarm environment:

    docker swarm init

  2. Create directories:

    mkdir -p /data/mingdao/script/volume/data

  3. Download the Flink image (offline package download):

    docker pull registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-flink:1.19.710

  4. Create the configuration file:

    cat > /data/mingdao/script/flink.yaml <<EOF
    version: '3'
    services:
      flink:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-flink:1.19.710
        entrypoint: ["/bin/bash"]
        command: ["/run.sh"]
        environment:
          ENV_FLINK_S3_ACCESSKEY: "mdstorage"
          ENV_FLINK_S3_SECRETKEY: "eBxExGQJNhGosgv5FQJiVNqH"
          ENV_FLINK_S3_SSL: "false"
          ENV_FLINK_S3_PATH_STYLE_ACCESS: "true"
          ENV_FLINK_S3_ENDPOINT: "sc:9010"
          ENV_FLINK_S3_BUCKET: "mdoc"
          ENV_FLINK_LOG_LEVEL: "INFO"
          ENV_FLINK_JOBMANAGER_MEMORY: "4096m"
          ENV_FLINK_TASKMANAGER_MEMORY: "16384m"
          ENV_FLINK_TASKMANAGER_SLOTS: "50"
          ENV_KAFKA_ENDPOINTS: "sc:9092"
        ports:
          - 58081:8081
        volumes:
          - ./volume/data/:/data/
        extra_hosts:
          - "sc:192.168.10.28" # Update this to the actual internal IP address of the HAP server.
    EOF

  5. Configure the startup script:

    cat > /data/mingdao/script/startflink.sh <<-EOF
    docker stack deploy -c /data/mingdao/script/flink.yaml flink
    EOF
    chmod +x  /data/mingdao/script/startflink.sh

  6. Start the Flink service:

    bash /data/mingdao/script/startflink.sh
    • After the Flink container starts, it may take about 5 minutes for the process to complete initialization.
    • To stop Flink, use: docker stack rm flink.