Application Log and Usage Analysis Issues

Use the following steps when application logs stop updating or the Usage Analysis page is blank.

Cause

If Elasticsearch data disk usage has reached 95% or the disk has been full, index read-only protection may prevent new logs from being written. Check the read-only setting even if disk space is now available. Common errors include:

disk usage exceeded flood-stage watermark
index has read-only-allow-delete block

Remove the Read-Only Block

Expand the disk or remove unnecessary data to bring Elasticsearch data disk usage below 90%, then remove the read-only block from all indices.

Standalone Mode

1. Enter the Storage Component Container

   On the standalone server, enter the sc container:

   docker exec -it $(docker ps | grep mingdaoyun-sc | awk '{print $1}') bash

2. Remove the Block from All Indices

   Remove the read-only block inside the sc container:

   curl -sS -u 'md:ESPassWD1234' -X PUT 'http://127.0.0.1:9200/*/_settings?expand_wildcards=all' \
     -H 'Content-Type: application/json' \
     -d '{"index.blocks.read_only_allow_delete": null}'

3. Verify the Result

   Query the read-only setting inside the sc container:

   curl -sS -u 'md:ESPassWD1234' 'http://127.0.0.1:9200/_settings/index.blocks.read_only_allow_delete?expand_wildcards=all&flat_settings=true&pretty'

   After checking, run exit to leave the container.

Cluster Mode

Run on a server that can access Elasticsearch. Replace ES_HOST and ES_PASSWORD with the actual address and the password for the elastic account, and adjust the port to match your configuration.

1. Remove the Block from All Indices

   curl -sS -u 'elastic:ES_PASSWORD' -X PUT 'http://ES_HOST:9200/*/_settings?expand_wildcards=all' \
     -H 'Content-Type: application/json' \
     -d '{"index.blocks.read_only_allow_delete": null}'

2. Verify the Result

   curl -sS -u 'elastic:ES_PASSWORD' 'http://ES_HOST:9200/_settings/index.blocks.read_only_allow_delete?expand_wildcards=all&flat_settings=true&pretty'

The block is removed when the update returns "acknowledged": true and the query result shows that index.blocks.read_only_allow_delete is absent or false.

Perform an operation in HAP and check for new application logs. If the read-only block returns, check disk space on each Elasticsearch node.

Rebuild Elasticsearch Data

To recover historical logs, rebuild Elasticsearch data. The service will write the original logs retained in MongoDB back into Elasticsearch.

Standalone Mode

1. Stop the Services

   Stop the manager and HAP services from the manager root directory:

   bash ./service.sh stopall

2. Move the Elasticsearch Data Directory

   After confirming that the services have stopped, rename the Elasticsearch data directory on the host:

   mv /data/mingdao/script/volume/data/elasticsearch-8 /data/mingdao/script/volume/data/elasticsearch-8.bak-$(date +%Y%m%d%H%M%S)

3. Start the Services

   Start the services from the manager root directory:

   bash ./service.sh startall

Cluster Mode

1. Stop the HAP Microservices

   Stop the HAP microservices to prevent writes or index creation during cleanup.

2. Delete the Indices

   Run on the Elasticsearch server and replace 123456 with the actual password. If running from another server, replace both occurrences of 127.0.0.1:9200 with the actual Elasticsearch IP address and port.

   elastic_pwd=123456
   for i in $(curl -u elastic:$elastic_pwd http://127.0.0.1:9200/_cat/indices|awk '{print $3}'); do
       curl -XDELETE -u elastic:$elastic_pwd http://127.0.0.1:9200/$i
   done

   A response of "acknowledged": true means that the corresponding index was deleted. This command does not exclude system indices. Resolve any errors or authentication issues before starting the HAP microservices.

3. Start the HAP Microservices

   Start the HAP microservices to trigger data rebuilding.

Rebuild time depends on the amount of data. Larger datasets take longer.

• Application logs appear gradually as data is written. Check whether logs from the missing period are restored.
• Usage Analysis charts may be cached for 24 hours. If they do not update immediately after rebuilding, check again the next day.