Skip to main content

Application Behavior Logs

To alleviate the continuous storage growth pressure in the mdservicedata database of MongoDB, you can choose to archive the application behavior logs and store them in a new MongoDB instance. Archived historical logs can also be selected and viewed on the page.

Configuration Steps for Archiving​

  1. Deploy a MongoDB instance in advance for storing archived data

  2. Download the mirror (offline package download)

    docker pull registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-archivetools:1.0.3

  3. Create a config.json configuration file with the following example content:

    [
        {
            "id": "1",
            "text": "Description",
            "start": "2022-12-31 16:00:00",
            "end": "2023-12-31 16:00:00",
            "src": "mongodb://root:password@192.168.1.20:27017/mdservicedata?authSource=admin",
            "archive": "mongodb://root:password@192.168.1.30:27017/mdservicedata_archive_2023?authSource=admin",
            "table": "al_actionlog*",
            "delete": true,
            "batchSize": 500,
            "retentionDays": 0
        }
    ]
    • Adjust the configuration content according to the above configuration file.

    Parameter Description:

    "id": "Task Identifier ID",
    "text": "Description",
    "start": "Specify the start time of the archived data, in UTC time zone (if the value of retentionDays is greater than 0, this configuration will automatically become invalid)",
    "end": "Specify the end time of the archived data, in UTC time zone (if the value of retentionDays is greater than 0, this configuration will automatically become invalid)",
    "src": "Connection address of the source database",
    "archive": "Connection address of the target database (if empty, no archiving will be done, only deletion according to the set rules)",
    "table": "Data table",
    "delete": "It is fixed to true; after the archiving task is completed, and the number of records verified is correct, clean up the archived data in the source database",
    "batchSize": "Number of entries and deletions in a single batch",
    "retentionDays": "It defaults to 0. If greater than 0, it means delete data X days ago and enable scheduled deletion mode, the dates specified in start and end will automatically become invalid, scheduled to run every 24 hours by default"

  4. Start the archiving service by executing the following in the directory where the config.json file is located

    docker run -d -it -v $(pwd)/config.json:/usr/local/MDArchiveTools/config.json  -v /usr/share/zoneinfo/Etc/GMT-8:/etc/localtime registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-archivetools:1.0.3

    Other:

    • Resource Usage: During program operation, there will be a certain amount of resource pressure on the source database, target database, and the device where the program is located. It is recommended to execute in the idle period of the business.

    • Viewing Logs:

      • Running in the background (default): Use docker ps -a to find the container ID, then execute docker logs container ID to view the logs.

      • Running in the foreground: Remove the -d parameter, and the logs will be output in real-time to the terminal for easy progress tracking.

    • In the example configuration file config.json, name the new database in the format of source database name_archive_date. Each time you execute, modify the target database name in archive.

      • Since the program will first count the amount of data in the target table after the archive is completed, if they are not equal, deletion will not occur; if the archive target database name is not modified in the second run, it may result in more data in the target table than in the current archive, preventing the deletion of the source data.

    • Reclaim Disk Space: After archiving is completed, the corresponding data in the source database will be deleted. The disk space occupied by the deleted data will not be immediately released, but it is typically reused by the same table.

Clean Up Historical Data in Elasticsearch​

  1. Enter the microservice container

    docker exec -it $(docker ps | grep community | awk '{print $1}') bash

  2. Clean up historical data in Elasticsearch

    source /entrypoint.sh && deleteActionlog "2022-12-31 16:00:00" "2023-12-31 16:00:00"
    • Please ensure the time range in the above command matches the time range configured in the corresponding archival task.

Configure Visualization of Archived Data​

  1. Create the application-www-ext.properties configuration file

    For example: /data/mingdao/script/volume/actionlog/application-www-ext.properties

    spring.data.mongodb.archive.group[0].id=0
    spring.data.mongodb.archive.group[0].text=\u5e94\u7528\u884c\u4e3a\u65e5\u5fd7-2023
    spring.data.mongodb.archive.group[0].uri=mongodb://root:password@192.168.1.30:27017/dservicedata_archive_2023?authSource=admin
    spring.data.mongodb.archive.group[0].start=2023-01-01
    spring.data.mongodb.archive.group[0].end=2023-12-31

    Parameter Description:

    • group[0]: It defaults to 0 and is incremented for multiple archived data
    • id: It defaults to 0 and is incremented for multiple archived data
    • text: Name displayed on the page, Unicode encoding is required if in Chinese
    • uri: Connection address of the archived database
    • start: Start date of the archived data
    • end: End date of the archived data

  2. Mount the configuration file

    Add the following to the docker-compose.yaml volumes corresponding to the microservice application:

    - ./volume/actionlog/application-www-ext.properties:/usr/local/MDPrivateDeployment/actionlog/application-www-ext.properties

  3. Restart the microservice