Skip to main content

Logs of Row Record in Worksheet

To alleviate the continuous storage growth pressure in the mdworksheetlog database of MongoDB, users can choose to archive the logs of row records in a worksheet and store them in a new MongoDB instance. The historical logs that have been archived can also be viewed on the page.

Configuration Steps for Archiving​

  1. Deploy a MongoDB instance in advance for storing archived data

  2. Download the mirror (offline package download)

    docker pull registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-archivetools:1.0.3

  3. Create a config.json configuration file with the following example content:

    [
        {
            "id": "1",
            "text": "Description",
            "start": "2022-12-31 16:00:00",
            "end": "2023-12-31 16:00:00",
            "src": "mongodb://root:password@192.168.1.20:27017/mdworksheetlog?authSource=admin",
            "archive": "mongodb://root:password@192.168.1.30:27017/mdworksheetlog_archive_2023?authSource=admin",
            "table": "wslog*",
            "delete": true,
            "batchSize": 500,
            "retentionDays": 0
        }
    ]
    • Adjust the configuration content according to the above configuration file.

    Parameter Description:

    "id": "Task Identifier ID",
    "text": "Description",
    "start": "Specify the start time of the archived data, in UTC time zone (if the value of retentionDays is greater than 0, this configuration will automatically become invalid)",
    "end": "Specify the end time of the archived data, in UTC time zone (if the value of retentionDays is greater than 0, this configuration will automatically become invalid)",
    "src": "Connection address of the source database",
    "archive": "Connection address of the target database (if empty, no archiving will be done, only deletion according to the set rules)",
    "table": "Data table",
    "delete": "It is fixed to true; after the archiving task is completed, and the number of records verified is correct, clean up the archived data in the source database",
    "batchSize": "Number of entries and deletions in a single batch",
    "retentionDays": "It defaults to 0. If greater than 0, it means delete data X days ago and enable scheduled deletion mode, the dates specified in start and end will automatically become invalid, scheduled to run every 24 hours by default"

  4. Start the archiving service by executing the following in the directory where the config.json file is located

    docker run -d -it -v $(pwd)/config.json:/usr/local/MDArchiveTools/config.json  -v /usr/share/zoneinfo/Etc/GMT-8:/etc/localtime registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-archivetools:1.0.3

    Other:

    • Resource Usage: During program operation, there will be a certain amount of resource pressure on the source database, target database, and the device where the program is located. It is recommended to execute in the idle period of the business.

    • Viewing Logs:

      • Running in the background (default): Use docker ps -a to find the container ID, then execute docker logs container ID to view the logs.

      • Running in the foreground: Remove the -d parameter, and the logs will be output in real-time to the terminal for easy progress tracking.

    • In the example configuration file config.json, name the new database in the format of source database name_archive_date. Each time you execute, modify the target database name in archive.

      • Since the program will first count the amount of data in the target table after the archive is completed, if they are not equal, deletion will not occur; if the archive target database name is not modified in the second run, it may result in more data in the target table than in the current archive, preventing the deletion of the source data.

    • Reclaim Disk Space: After archiving is completed, the corresponding data in the source database will be deleted. The disk space occupied by the deleted data will not be immediately released, but it is typically reused by the same table.

Configure Visualization of Archived Data​

  1. Create the appextensions.json configuration file

    For example: /data/mingdao/script/volume/worksheet/appextensions.json

    {
     "Log.Archive": [
       {
         "id": "1",
         "text": "Archive-2023",
         "uri": "mongodb://root:password@192.168.1.30:27017/mdworksheetlog_archive_2023?authSource=admin;socketTimeout=30s",
         "start": "2023-01-01",
         "end": "2023-12-31"
       }
     ]
    }

    Parameter Description:

    • id: It defaults to 1 and is incremented for multiple archived data
    • text: Name displayed on the page
    • uri: Connection address of the archived database
    • start: Start date of the archived data
    • end: End date of the archived data

  2. Mount the configuration file

    Add the following to the docker-compose.yaml volumes corresponding to the microservice application:

    - ./volume/worksheet/appextensions.json:/usr/local/MDPrivateDeployment/worksheet/Config/appextensions.json

  3. Restart the microservice