How to enable data pipeline

Data pipeline is an extension module in the HAP system that users can choose to enable independently. Before enabling, please make sure that Docker is of version 20.10.16+ and the server has at least 8GB of available physical memory.

For quick installation in standalone deployment mode, follow this article: (in cluster deployment mode it is more complex and may require assistance from the HAP team).

1. 1. Download the mirror (Offline Package)

   docker pull registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-flink:1.17.1.530

2. Modify docker-compose.yaml (default path is /data/mingdao/script/docker-compose.yaml).

   1. Add the flink service

      flink:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-flink:1.17.1.530
        entrypoint: ["/bin/bash"]
        command: ["/run.sh"]
        volumes:
          - ./volume/data/:/data/

      Note: If docker logs $(docker ps | grep '\-sc' | awk '{print $1}') | grep minio outputs results, then the ENV_FLINK_S3_ENDPOINT environment variable needs to be added in the flink service

      flink:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-flink:1.17.1.530
        entrypoint: ["/bin/bash"]
        command: ["/run.sh"]
        environment: 
          ENV_FLINK_S3_ENDPOINT: "sc:9010"
        volumes:
          - ./volume/data/:/data/

   2. Add the ENV_FLINK_URL environment variable in the app service

      app:
        environment:
          ENV_FLINK_URL: "http://flink:8081"
   Example

   version: '3'
   
   services:
     app:
       image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-community:5.8.1
       environment: &app-environment
         ENV_ADDRESS_MAIN: "https://hap.domain.com"
         ENV_APP_VERSION: "5.8.1"
         ENV_API_TOKEN: "******"
         ENV_FLINK_URL: "http://flink:8081" #Add
       ports:
         - 8880:8880
       volumes:
         - ./volume/data/:/data/
         - ../data:/data/mingdao/data
   
     sc:
       image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-sc:3.0.0
       environment:
         <<: *app-environment
       volumes:
         - ./volume/data/:/data/
   
     command:
       image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-command:node1018-python36
       environment:
         <<: *app-environment
   
     doc:
       image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-doc:1.2.0
       environment:
         ENV_FILE_INNER_URI: "app:8880"
   
     # Add
     flink:
       image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-flink:1.17.1.530
       entrypoint: ["/bin/bash"]
       command: ["/run.sh"]
       environment: 
         ENV_FLINK_S3_ENDPOINT: "sc:9010"
       volumes:
         - ./volume/data/:/data/

3. Restart the HAP microservices