Data Management

Data Storage

In stand-lone deployment mode, the storage components are already built into the mirror, so users do not need to install them separately. The generated data is stored in a specified directory through mounting.

In HAP, the default data path is /data/mingdao, but if you have a custom data path, use the actual path.

The data generated by storage components is by default mounted in the /data/mingdao/script/volume/data/ directory on the server. The directory structure is as follows:

/data/mingdao/script/volume/data/
├── elasticsearch
├── elasticsearch-8
├── kafka
├── logs
├── mongodb
├── mysql
├── redis
├── storage
└── zookeeper

Data stored in each directory:

• elasticsearch, for elasticsearch data prior to version 3.9.0

• elasticsearch-8, for elasticsearch data starting from version 3.9.0, mainly supporting super search and application logs

• kafka, for storing message queue data, expired data is automatically cleaned up by default

• log, for logs of component service

• mongodb, for storing business data (such as applications, worksheets, workflows, etc.)

• mysql, for storing system core data such as users, organizations, etc.

• redis, for caching data

• storage, for storing attachment data

• zookeeper, to provide services for kafka

Data Backup

• View more details about backup

• If using a cloud server, you can utilize the disk snapshot feature provided by the cloud provider.

Data Cleaning

In general, the data in logs, mongodb, and storage occupy relatively large disk space.

• For logs stored in the logs directory for component services, if you do not need this log data, you can clean it up.

• For business data stored in mongodb, currently only log data archiving and cleaning deleted application data are supported.

• For attachment data stored in storage, if the disk capacity cannot be supported, it is recommended to limit the file size on the upload page.

  • If you are using a cloud server, go to enable object storage to reduce disk capacity pressure.

If your environment has been upgraded, mirrors of history versions that have not been cleaned up will also take up some disk space.

Example of Cleaning up History Mirrors

1. View the list of mirror files on the current server.

   [root@localhost ~]# docker images
   REPOSITORY                                                        TAG                   IMAGE ID       CREATED         SIZE
   registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-hap   4.4.2                 cacc79998cbd   5 days ago      9.33GB
   registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-hap   4.3.0                 e0c6e370de63   9 days ago      9.26GB
   registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-doc   1.2.0                 cc4d668f289b   4 months ago    2.66GB

   • For example, if the current version is 4.4.2, then the mirrors of version 4.3.0 can be cleaned up.

2. Clean up the mirrors of history versions.

   docker image rm registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-hap:4.3.0
   or
   docker image rm e0c6e370de63

If the above methods fail to free up more disk space, consider expanding the disk capacity.

Data Migration

Option 1: Backup and Restore

Back up data from the old environment using the Data Backup guide, then restore the data to the new environment using the Data Restoration guide. Use either dump or file copying.

Option 2: Directory Migration

Stop the services and transfer the entire data directory from the old environment to the new server, retaining the existing configuration and data.

Note

The following steps use the default data directory /data/mingdao. Both environments must use the same data path. Adjust custom paths as needed.

Before migrating, ensure there is enough disk space for the archive and extracted data. Migrate mounted files, proxy settings, and certificates outside the data directory separately.

1. Deploy the New Environment

   Install the same versions of HAP and its supporting components as in the old environment. The CPU architecture must also match.

2. Stop Both Environments

   On both servers, open the manager root directory and stop the manager and HAP services:

   bash ./service.sh stopall

   Confirm that services on both servers have stopped before creating the archive. Keep the old environment stopped during migration.

3. Archive the Old Environment Data

   On the old server, archive everything under /data/mingdao. Store the archive alongside the data directory:

   tar -zcpf /data/mingdao.tar.gz -C /data/mingdao .

4. Transfer the Data Archive

   Transfer /data/mingdao.tar.gz from the old server to the same path on the new server.

5. Move the Existing Data Directory on the New Server

   mv /data/mingdao /data/mingdao.bak-$(date +%Y%m%d%H%M%S)

6. Create the New Data Directory

   Re-create the data directory on the new server:

   mkdir -p /data/mingdao

7. Extract the Old Environment Data

   On the new server, extract the archive into /data/mingdao, preserving file permissions and ownership:

   tar -zxpf /data/mingdao.tar.gz -C /data/mingdao

8. Update the Access Addresses

   Edit the migrated configuration file on the new server:

   vi /data/mingdao/script/docker-compose.yaml

   Check access address settings such as ENV_ADDRESS_MAIN. Replace every reference to an old environment access address with the corresponding new address.

9. Start the New Environment

   On the new server, open the manager root directory and start the manager and HAP services:

   bash ./service.sh startall

10. Verify the Migration

    Access the system at its new address. Check login, business data, workflows, and attachment uploads and downloads.