Skip to main content

Application Data Cleaning

Application data includes application information, application grouping, application roles, worksheet information, worksheet data, and so on. In order to ensure data security, the deletion of important data in the system is logical deletion. However, with the increase of deleted data, in addition to occupying disk storage space, it may also have some impact on database performance. Therefore, HAP provides application data cleaning service to physically delete the deleted data regularly according to the set rules.

  1. Download mirror

    docker pull registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-dataclean-app:1.0.0

  2. Docker Swarm initialization (skip this step if already initialized)

    docker swarm init

  3. Create the dataclean.yaml file in any directory with the following contents:

    version: '3'

    services:
      app:
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-dataclean-app:1.0.0
        volumes:
          - ./appsettings.json:/app/Config/appsettings.json

  4. Create the appsettings.json configuration file in the directory where the dataclean.yaml file is located, with the following contents:

    {
     "WorksheetConnStr": "mongodb://127.0.0.1:27017/mdworksheet",
     "MD_WSService1": "mongodb://127.0.0.1:27017/mdwsrows",
     "MD_WSService2": "mongodb://127.0.0.1:27017/mdwsrows",
     "MD_WSService3": "mongodb://127.0.0.1:27017/mdwsrows",
     "MD_WSService4": "mongodb://127.0.0.1:27017/mdwsrows",
     "MD_WSService5": "mongodb://127.0.0.1:27017/mdwsrows",
     "MD_WSService6": "mongodb://127.0.0.1:27017/mdwsrows",
     "MD_WSService7": "mongodb://127.0.0.1:27017/mdwsrows",
     "MD_WSService8": "mongodb://127.0.0.1:27017/mdwsrows",
     "MD_WSService9": "mongodb://127.0.0.1:27017/mdwsrows",
     "MD_WSService10": "mongodb://127.0.0.1:27017/mdwsrows",
     "MD_WSLogService1": "mongodb://127.0.0.1:27017/mdworksheetlog",
     "ApkConnStr": "mongodb://127.0.0.1:27017/mdapps", 
     "AppRoleConnStr": "mongodb://127.0.0.1:27017/mdapproles",
     "MD.Worksheet.DbName": "mdworksheet",
     "MD.WorksheetRow.DbName": "mdwsrows",
     "DeleteDay": 60, 
     "DeleteRowByNomal": 0,
     "DelateRowDay": 60, 
     "DelateRowTotal": 10000,
     "Corn": "0 0 2 * * ? "
    }

    Configuration Notes:

    • mongodb://127.0.0.1:27017 Replaced with the actual MongoDB connection address
    • DeleteDay Delete data from X day(s) ago
    • DeleteRowByNomal Whether to clean up deleted row records when the worksheet is in normal status (0: No cleaning, 1: Cleaning)
    • DelateRowDay When DeleteRowByNomal=1, delete row records from X days ago; when DeleteRowByNomal=0, the configuration is invalid.
    • DelateRowTotal When DeleteRowByNomal=1, delete data only when the total number of worksheet row records is greater than X; when DeleteRowByNomal=0, the configuration is invalid.
    • Corn Timed execution of the corn expression. See https://51tools.info/cron/ , 0 0 2 * * ? means execute at 2:00 every day.

  5. Start data cleaning service and execute it in the directory where dataclean.yaml is located

    docker stack deploy -c dataclean.yaml dataclean

    The execution logs of timed tasks can be viewed with docker logs $(docker ps | grep dataclean | awk '{print $1}') .

    If you need to remove the service, execute docker stack rm dataclean.