Skip to main content

Configuring Independent Address Access for Files

description

In most browsers, the maximum number of concurrent requests to the same address is limited to 6. Therefore, if a sheet view contains a large number of image files, the user may cause a queue of browser requests when clicking into such a view. In this case, if the user switches to another page when the browser request is not fully loaded, the browsing speed may be slowed down or the browser may even freeze.

To address the above phenomenon, you first need to ensure that the file service instance resources and network bandwidth are sufficient. Secondly, you can refer to this document to configure an independent address for file requests to prevent file requests from blocking other interface requests.

Stand-alone environment​

Tips

If the independent file storage address uses the default 80/443 port of http/https, you do not need to specify the default 80/443 port in the address when configuring the relevant environment variables.

System single access address​

Here, the current system access address is http://example.domain.com and the new file independent address http://file.domain.com is taken as an example.

Configuration steps​

  1. Modify the microservice docker-compose.yaml configuration file

    version: '3'

    services:
      core:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-core:7.4.2
        environment: &app-environment
          ENV_ADDRESS_MAIN: "http://example.domain.com"
          ENV_APP_VERSION: "7.4.2"
          ENV_API_TOKEN: "******"
          ENV_FILE_DOMAIN: "http://file.domain.com" #Add a new variable to specify the independent address of the file
          ENV_FILE_ADDRESS_MAIN: "http://file.domain.com" #Add a new variable to specify the independent address of the file
          ENV_HAP_INTRANET_FILE_ENDPOINT: "core:8880/file" #New variable, fixed value
        ports:
          - 8880:8880
        volumes:
          - ./volume/data/:/data/
          - ../data:/data/mingdao/data

      sc:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-sc:3.2.0
        environment:
          <<: *app-environment
        volumes:
          - ./volume/data/:/data/
        ports:
          - 9000:9000 #Added port mapping to map port 9000 in the container to the host

      command:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-command:node1018-python36

      doc:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-doc:2.0.0

Added variable configuration instructions:

  • ENV_FILE_DOMAIN: The file storage service specifies an independent file address
  • ENV_FILE_ADDRESS_MAIN: Microservice specifies an independent file address
  • ENV_HAP_INTRANET_FILE_ENDPOINT fixed value "core:8880/file"
  • If the independent file address uses the default 80/443 port of http/https, you do not need to specify the default 80/443 port when configuring the above two variables.

Added port mapping instructions:

  • - 9000:9000 maps the file service port in the container to the host to allow the nginx agent to access the file service in the next step

doc service variable modification instructions:

  • ENV_FILE_INNER_URI, the default variable value is changed to sc:9000

  1. Restart the microservice

  2. Add nginx cross-domain proxy configuration for file independent address. For the proxy configuration file, please refer to the configuration below.

    upstream file {
        server 192.168.0.10:9000; #Specify the IP of the backend microservice and map it to the file storage port on the microservice host
    }

    server {
        listen 80;
        server_name file.domain.com; #Specify independent file address
        access_log /data/logs/weblogs/file.domain.com.log main; #Log path, customizable
        error_log /data/logs/weblogs/file.domain.com.error.log; #Log path, customizable

        location / {
            if ($request_method = OPTIONS) {
                return 204 "";
            }
            proxy_set_header HOST $http_host;
            proxy_pass http://file;
            proxy_hide_header Access-Control-Allow-Origin;
            add_header Access-Control-Allow-Headers authorization,content-type;
            add_header Access-Control-Allow-Origin "http://example.domain.com"; #Specify HDP system access address
        }
    }

  3. After the proxy configuration is completed, clear the browser cache. You can then use the browser developer tools to see whether file-related requests have gone to independent addresses in Network.

System multiple access addresses​

In a system with multiple access addresses, when configuring independent addresses for file services, you need to configure an independent file address for each system access address.

Here we take the current system main access address example-external.domain.com and extended address example-internal.domain.com as an example.

Add two independent file access addresses file-external.domain.com and file-internal.domain.com to the two access addresses.

The corresponding relationship between system access address and file independent access address is as follows:

Address typePrimary addressExtended address
System access addressexample-external.domain.comexample-internal.domain.com
File independent addressfile-external.domain.comfile-internal.domain.com

Configuration steps​

  1. Modify the microservice docker-compose.yaml file, add relevant configurations and restart the microservice

    version: '3'

    services:
      core:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-core:7.4.2
        environment:
          ENV_ADDRESS_MAIN: "http://example-external.domain.com"
          ENV_APP_VERSION: "7.4.2"
          ENV_FILE_DOMAIN: "http://file-external.domain.com,http://file-internal.domain.com" #Add new variables to specify multiple independent addresses of files
          ENV_FILE_ADDRESS_MAIN: "http://file-external.domain.com" #Add a new variable to specify the main independent address of the file
          ENV_HAP_INTRANET_FILE_ENDPOINT: "core:8880/file" #New variable, fixed value
        ports:
          - 8880:8880
          - 18880:18880
        volumes:
          - ./volume/data/:/data/
          - ../data:/data/mingdao/data

      sc:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-sc:3.2.0
        environment:
          <<: *app-environment
        volumes:
          - ./volume/data/:/data/
        ports:
          - 9000:9000 #Added port mapping to map port 9000 in the container to the host

      command:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-command:node1018-python36

      doc:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-doc:2.0.0

Added variable configuration instructions:

  • ENV_FILE_DOMAIN specifies multiple independent file access addresses for the file storage service
  • ENV_FILE_ADDRESS_MAIN specifies the primary independent file access address for microservices
  • ENV_HAP_INTRANET_FILE_ENDPOINT fixed value "core:8880/file"
  • If the independent file address uses the default 80/443 port of http/https, you do not need to specify the default 80/443 port when configuring the above two variables.

Added port mapping instructions:

  • - 9000:9000 maps the file service port in the container to the host to allow the nginx agent to access the file service in the next step

doc service variable modification instructions:

  • ENV_FILE_INNER_URI, the default variable value is changed to sc:9000

  1. Restart the microservice

  2. Add an nginx cross-origin proxy configuration for the primary independent file access address. You can refer to the following proxy configuration.

    upstream file {
        server 192.168.0.10:9000; #Specify the IP of the backend microservice and map it to the file storage port on the microservice host
    }

    server {
        listen 80;
        server_name file-external.domain.com; #Specify the main independent address of the file
        access_log /data/logs/weblogs/file-external.domain.com.log main; #Log path, customizable
        error_log /data/logs/weblogs/file-external.domain.com.error.log; #Log path, customizable

        location / {
            if ($request_method = OPTIONS) {
                return 204 "";
            }
            proxy_set_header HOST $http_host;
            proxy_pass http://file;
            proxy_hide_header Access-Control-Allow-Origin;
            add_header Access-Control-Allow-Headers authorization,content-type;
            add_header Access-Control-Allow-Origin "http://example-external.domain.com"; #Specify the HDP system primary access address
        }
    }

  3. Add nginx cross-domain proxy configuration for file extension independent address. For the proxy configuration file, please refer to the configuration below.

    upstream file {
        server 192.168.0.10:9000; #Specify the IP of the backend microservice and map it to the file storage port on the microservice host
    }

    server {
        listen 80;
        server_name file-internal.domain.com; #Specify file extension independent address
        access_log /data/logs/weblogs/file-internal.domain.com.log main; #Log path, customizable
        error_log /data/logs/weblogs/file-internal.domain.com.error.log; #Log path, customizable

        location / {
            if ($request_method = OPTIONS) {
                return 204 "";
            }
            proxy_set_header HOST $http_host;
            proxy_pass http://file;
            proxy_hide_header Access-Control-Allow-Origin;
            add_header Access-Control-Allow-Headers authorization,content-type;
            add_header Access-Control-Allow-Origin "http://example-internal.domain.com"; #HDP system extended access address
        }
    }

  4. Modify the agent of system extension address and add the request header pdfileaddr field under location / and location ~ /mds2 respectively to specify the file extension independent address, as shown below

    ......

        location / {
            set $real_ip '';
            if ($http_x_real_ip) {
                set $real_ip $http_x_real_ip;
            }
            if ($http_x_real_ip = '') {
                set $real_ip $remote_addr;
            }
            proxy_set_header X-Real-IP $real_ip;
            proxy_set_header Host $http_host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header pdaddr http://example-internal.domain.com;
            proxy_set_header pdfileaddr http://file-internal.domain.com; #Added request header pdfileaddr to specify the file extension independent address
            proxy_pass http://ext;
        }

        location ~ /mds2 {
            proxy_set_header Host $http_host;
            proxy_hide_header X-Powered-By;
            proxy_set_header X-NginX-Proxy true;
            proxy_pass http://ext;
            proxy_redirect off;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection upgrade;
            proxy_set_header pdaddr http://example-internal.domain.com;
            proxy_set_header pdfileaddr http://file-internal.domain.com; #Added request header pdfileaddr to specify the file extension independent address
        }

        location ~ ^/(agent|api/workflow/api/sse/chat) {
            set $md_real_ip '';
            if ($http_x_real_ip) {
                set $md_real_ip $http_x_real_ip;
            }
            if ($http_x_real_ip = '') {
                set $md_real_ip $remote_addr;
            }
            proxy_buffering off;
            proxy_read_timeout 30m;
            proxy_send_timeout 30m;
            client_max_body_size 16m;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $md_real_ip;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-HTTP-Method-Override "";
            proxy_set_header pdaddr http://example-internal.domain.com;
            proxy_set_header pdfileaddr http://file-internal.domain.com; #Added request header pdfileaddr to specify the file extension independent address
            proxy_pass http://ext;
            proxy_redirect default;
        }
    ......

  5. After the proxy configuration is completed, clear the browser cache. You can then use the browser developer tools to see whether file-related requests have gone to independent addresses in Network.

Cluster environment​

Tips

If the independent file storage address uses the default 80/443 port of http/https, you do not need to specify the default 80/443 port in the address when configuring the relevant environment variables.

System single access address​

Here, the current system access address is http://example.domain.com and the new file independent address http://file.domain.com is taken as an example.

Configuration steps​

  1. Modify microservice related configurations

    1. Modify microservice config.yaml and add environment variables:

      ENV_FILE_ADDRESS_MAIN: "http://file.domain.com"
      ENV_HAP_INTRANET_FILE_ENDPOINT: "www:8880/file"

  • ENV_FILE_ADDRESS_MAIN environment variable value is the independent access address of the file

    1. Modify the microservice service.yaml, find the ENV_FILE_INNER_URI variable of the doc service, and modify the variable value to the intranet address of the file storage

      env:
      - name: ENV_FILE_INNER_URI
        value: "192.168.0.10:9000" #Change to the intranet address of file storage

  1. Restart the microservice

  2. Modify the file storage service file.yaml configuration file, add relevant configurations and restart the file storage service

    1. Modify the file storage service file.yaml configuration file and add the environment variable ENV_FILE_DOMAIN to specify an independent file address for the file storage service.

      ENV_FILE_DOMAIN: "http://file.domain.com"

  3. Restart the file storage service

  4. Add nginx cross-domain proxy configuration for file independent address. For the proxy configuration file, please refer to the configuration below.

    upstream file {
        server 192.168.0.10:9000; #Specify the IP and port of the file storage service
    }

    server {
        listen 80;
        server_name file.domain.com; #Specify independent file address
        access_log /data/logs/weblogs/file.domain.com.log main; #Log path, customizable
        error_log /data/logs/weblogs/file.domain.com.error.log; #Log path, customizable

        location / {
            if ($request_method = OPTIONS) {
                return 204 "";
            }
            proxy_set_header HOST $http_host;
            proxy_pass http://file;
            proxy_hide_header Access-Control-Allow-Origin;
            add_header Access-Control-Allow-Headers authorization,content-type;
            add_header Access-Control-Allow-Origin "http://example.domain.com"; #Specify HDP system access address
        }
    }

  5. After the proxy configuration is completed, clear the browser cache. You can then use the browser developer tools to see whether file-related requests have gone to independent addresses in Network.

System multiple access addresses​

In a system with multiple access addresses, when configuring independent addresses for file services, you need to configure an independent file address for each system access address.

Here we take the current system main access address example-external.domain.com and extended address example-internal.domain.com as an example.

Add two independent file access addresses file-external.domain.com and file-internal.domain.com to the two access addresses.

The corresponding relationship between system access address and file independent access address is as follows:

Address typePrimary addressExtended address
System access addressexample-external.domain.comexample-internal.domain.com
File independent addressfile-external.domain.comfile-internal.domain.com

Configuration steps​

  1. Modify microservice related configurations

    1. Modify microservice config.yaml and add environment variables:

      ENV_FILE_ADDRESS_MAIN: "http://file-external.domain.com"
      ENV_HAP_INTRANET_FILE_ENDPOINT: "www:8880/file"

  • ENV_FILE_ADDRESS_MAIN environment variable value is the primary independent access address of the file

    1. Modify the microservice service.yaml, find the ENV_FILE_INNER_URI variable of the doc service, and modify the variable value to the intranet address of the file storage

      env:
      - name: ENV_FILE_INNER_URI
        value: "192.168.0.10:9000" #Change to the intranet address of file storage

  1. Restart the microservice

  2. Modify the file storage service file.yaml configuration file, add relevant configurations and restart the file storage service

    1. Modify the file storage service file.yaml configuration file and add the environment variable ENV_FILE_DOMAIN to specify multiple independent file addresses for the file storage service.

      ENV_FILE_DOMAIN: "http://file-external.domain.com，http://file-internal.domain.com"

  3. Restart the file storage service

  4. Add nginx cross-domain proxy configuration for file master independent address. For the proxy configuration file, please refer to the configuration below.

    upstream file {
        server 192.168.0.10:9000; #Specify the IP and port of the file storage service
    }

    server {
        listen 80;
        server_name file-external.domain.com; #Specify the main independent address of the file
        access_log /data/logs/weblogs/file-external.domain.com.log main; #Log path, customizable
        error_log /data/logs/weblogs/file-external.domain.com.error.log; #Log path, customizable

        location / {
            if ($request_method = OPTIONS) {
                return 204 "";
            }
            proxy_set_header HOST $http_host;
            proxy_pass http://file;
            proxy_hide_header Access-Control-Allow-Origin;
            add_header Access-Control-Allow-Headers authorization,content-type;
            add_header Access-Control-Allow-Origin "http://example-external.domain.com"; #Specify the HDP system primary access address
        }
    }

  5. Add nginx cross-domain proxy configuration for file extension independent address. For the proxy configuration file, please refer to the configuration below.

    upstream file {
        server 192.168.0.10:9000; #Specify the IP and port of the file storage service
    }

    server {
        listen 80;
        server_name file-internal.domain.com; #Specify file extension independent address
        access_log /data/logs/weblogs/file-internal.domain.com.log main; #Log path, customizable
        error_log /data/logs/weblogs/file-internal.domain.com.error.log; #Log path, customizable

        location / {
            if ($request_method = OPTIONS) {
                return 204 "";
            }
            proxy_set_header HOST $http_host;
            proxy_pass http://file;
            proxy_hide_header Access-Control-Allow-Origin;
            add_header Access-Control-Allow-Headers authorization,content-type;
            add_header Access-Control-Allow-Origin "http://example-internal.domain.com"; #HDP system extended access address
        }
    }

  6. Modify the agent of system extension address and add the request header pdfileaddr field under location / and location ~ /mds2 respectively to specify the file extension independent address, as shown below

    ......

        location / {
            set $real_ip '';
            if ($http_x_real_ip) {
                set $real_ip $http_x_real_ip;
            }
            if ($http_x_real_ip = '') {
                set $real_ip $remote_addr;
            }
            proxy_set_header X-Real-IP $real_ip;
            proxy_set_header Host $http_host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header pdaddr http://example-internal.domain.com;
            proxy_set_header pdfileaddr http://file-internal.domain.com; #Added request header pdfileaddr to specify the file extension independent address
            proxy_pass http://ext;
        }

        location ~ /mds2 {
            proxy_set_header Host $http_host;
            proxy_hide_header X-Powered-By;
            proxy_set_header X-NginX-Proxy true;
            proxy_pass http://ext;
            proxy_redirect off;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection upgrade;
            proxy_set_header pdaddr http://example-internal.domain.com;
            proxy_set_header pdfileaddr http://file-internal.domain.com; #Added request header pdfileaddr to specify the file extension independent address
        }

        location ~ ^/(agent|api/workflow/api/sse/chat) {
            set $md_real_ip '';
            if ($http_x_real_ip) {
                set $md_real_ip $http_x_real_ip;
            }
            if ($http_x_real_ip = '') {
                set $md_real_ip $remote_addr;
            }
            proxy_buffering off;
            proxy_read_timeout 30m;
            proxy_send_timeout 30m;
            client_max_body_size 16m;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $md_real_ip;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-HTTP-Method-Override "";
            proxy_set_header pdaddr http://example-internal.domain.com;
            proxy_set_header pdfileaddr http://file-internal.domain.com; #Added request header pdfileaddr to specify the file extension independent address
            proxy_pass http://ext;
            proxy_redirect default;
        }
    ......

  7. After the proxy configuration is completed, clear the browser cache. You can then use the browser developer tools to see whether file-related requests have gone to independent addresses in Network.