Skip to main content

FAQs

The commands on this page mainly apply to Docker Compose deployments. For Kubernetes, use corresponding kubectl commands, such as kubectl -n hap-ops logs deploy/ops-gateway. The example access ports are 48881 for Docker Compose and 30881 for Kubernetes.

Startup and Access​

Containers Keep Restarting​

Check logs with docker compose -f ops.yaml logs 服务名. Common causes:

  • Incorrect connection information: Check the collection configuration on the Data Sources page.
  • Port conflict: If 48881, 59100, 4317, or 4318 is occupied, change the host port mapping or stop the process using the port.
  • Insufficient memory: Check usage with docker stats. It is recommended to reserve 8G or more of memory on the host.

The Page Keeps Loading​

If the page at :48881 keeps loading, check the following:

  • Run docker ps | grep gateway to confirm gateway has started.
  • If the browser console reports 401, log in with the correct ENV_OPS_TOKEN.
  • When accessing through a reverse proxy, confirm ENV_OPS_SUB_PATH is configured.

Grafana Panels Keep Loading or Return 404 After Login​

Check the reverse proxy configuration:

  • Reverse proxy: ENV_OPS_SUB_PATH must match the actual proxy path. For example, use /mdis when proxying /mdis.
  • Direct port access: The URL path must match ENV_OPS_SUB_PATH. With the deployment example's default value, access /mdis/web/.

gateway Starts Before ops-mongo​

If ops-server cannot connect to ops-mongo, it retries in the background with backoff and connects automatically when MongoDB is ready. Check logs with docker logs gateway容器 | grep 告警子系统. When [alert] 告警子系统已挂载 /api/alert/ appears, mounting succeeded; no manual container restart is required.

No Monitoring Data​

The Data Sources page has a Collection Status column showing whether each data source is being collected:

  • Collecting: Collection is normal.
  • No data: The collection process is running but has not obtained data, possibly due to incorrect credentials or insufficient permissions.
  • Stopped: The collection process is not running.
  • Not collected: Collection has not started for this data source, for example because View Charts is not selected under Purpose.

Hover over a status label to view the failure reason and retry information. Start troubleshooting collection issues from this column.

The Host Panel Has No Data​

  • Confirm nodeagent is Up.
  • Confirm the host data source IP on the Data Sources page is reachable from the prometheus container: docker exec prometheus容器 wget -O- http://IP:59100/metrics.

Middleware Panels Have No Data​

Troubleshoot according to the Collection Status column:

StatusMeaningAction
No dataConnection succeeded but no metrics were obtainedUsually a credentials or permissions issue, especially for ES. See below
StoppedThe exporter process is not runningHover to view the cause. The process retries automatically. If it stays Stopped while the retry count increases, the target remains unreachable due to an incorrect address, blocked port, or firewall
Not collectedCollection has not started for this data sourceCheck whether View Charts is selected under Purpose and the data source is enabled
Unknownagent has not reportedConfirm the agent container is Up

Elasticsearch Connection Test Succeeds but Shows No Data​

Check the username and password. The built-in Elasticsearch username in private deployments is md. Changes to credentials on the Data Sources page take effect immediately without restarting containers.

For underlying logs, run docker logs agent容器. Within 10–50 seconds of startup, agent prints a collection self-check table with ✓/✗ and a reason for each item. If an exporter exits unexpectedly, it also prints [supervise] xxx 退出(code=N) and the last few log lines.

A Panel Reports Datasource ${DS_PROMETHEUS} was not found​

This historical issue was fixed in the 1.4.0 single image. If it persists, confirm the image tag is 1.2.5 or later.

Create or Edit Data Sources​

The Connection Address Is Temporarily Unreachable​

Connection failure does not block saving the configuration. If the target is not deployed yet or the network is temporarily unreachable, the form displays the reason at the top and the button changes to Save Anyway. Confirm again to save. The Ops Platform keeps retrying and starts collection once connected. Track progress in the Collection Status column of the Data Sources list.

Connection Check Timing​

Checks run automatically about 0.6 seconds after changing connection fields. No manual trigger is needed. If you save before the check finishes, the Ops Platform completes it before deciding how to save; validation is not skipped. Host data sources also support automatic checks.

The Check Reports That the Connection Cannot Be Checked Temporarily​

This means the check service has not responded yet, rather than that the connection settings are incorrect. Save the configuration and check Collection Status later.

Collection Configuration Changes Do Not Take Effect​

Collection Does Not Change After Editing a Data Source​

Configuration normally takes effect within 30 seconds. If collection does not change, first check the agent container logs. Repeated [sd] 拉取 xxx 失败 messages mean agent cannot reach the service discovery API. Common causes include ENV_OPS_SD_URL pointing to gateway port 48881, or a Kubernetes Service exposing only 48881 without 8081. Port 48881 is used by nginx and does not provide /api/sd. Exporters cannot start in this case, leaving middleware monitoring empty.

Middleware Connection Changes in ops.yaml Do Not Take Effect​

Since 1.5.0, monitored targets are no longer configured through ops.yaml. ENV_KAFKA_ENDPOINTS/ENV_ELASTICSEARCH_*/ENV_REDIS_*/ENV_MYSQL_*/ENV_MONGODB_URI These variables are no longer read, as explicitly listed in the startup logs. Change all connection information on the UI Data Sources page.

Startup logs list each deprecated variable still in ops.yaml and indicate that it is ignored. Delete the related configuration lines.

Logs Are Not Visible​

HAP/HDP Microservice Logs Are Missing from Service Logs​

Check in this order:

  1. HAP/HDP services must configure ENV_LOKI_URL pointing to MDIS Loki, such as http://MDIS主机:3100. Without this variable, logservice does not write logs to Loki because the installer sets StoreInLoki to false; the Ops Platform cannot query service logs. Restart the corresponding product services after changing it.
  2. Confirm the product service containers can access this address. For deployment across hosts, check that port 3100 is allowed.
  3. Microservices write logs only when business calls occur. Trigger an action such as login or sending a verification code, then refresh.

The two ENV_LOKI_URL variables have different purposes:

  • HAP/HDP service-side ENV_LOKI_URL: Enables writing. When unset, no logs are written. Empty by default.
  • MDIS-side ENV_LOKI_URL: Query address, defaulting to http://ops-loki:3100. No configuration is needed in the standard topology.

Configure the former on the correct side.

Containers Are Missing from the Container Console​

Alloy collects only new container logs after startup; it does not backfill historical logs. A container with no recent stdout output may not appear. Trigger output or restart the container, then check again.

Object Storage (MinIO/COS)​

In standalone deployments, only Loki logs need object storage consideration. Standalone deployments do not support tracing, so ops-tempo generates no trace data; object storage is neither needed nor recommended for it. Tracing with object storage applies to cluster deployments.

ops-loki Cannot Start After Configuring Object Storage​

Check the first error in docker logs loki容器:

  • dial tcp: lookup xxx: no such host: The endpoint uses a name that cannot resolve in the container network. If MinIO is not on the same Compose network, for example because it runs in a separate stack or on the host, use the host IP and host-mapped port instead of its container name.
  • connection refused: Incorrect port. The host-mapped port may not be 9000; use the actual value shown by docker ps.
  • SignatureDoesNotMatch/AccessDenied: AK/SK or bucket permissions issue.
  • Path style: Use ENV_S3_FORCE_PATH_STYLE=true for MinIO. Tencent Cloud COS must use false because it supports only virtual-hosted style.

Historical Logs Disappear After Switching to Object Storage​

Loki does not migrate old data when switching storage backends. Historical logs remain in the original local volume. Newly generated logs can be viewed in the new backend.

Upgrade​

The Version Does Not Change After Upgrade​

Since 1.4.0, all components share the ops-allinone image. Change the tag in x-ops-image at the top of ops.yaml, run docker compose -f ops.yaml pull, then docker compose -f ops.yaml up -d. Add --force-recreate if forced container recreation is needed.

Historical Data After Upgrade​

Monitoring history and alert configuration remain if the Loki, Prometheus, Tempo, and ops-mongo storage volumes are unchanged. Unsaved manual edits in Grafana are lost; provisioned dashboards reload automatically.

Changing Data Volume Paths During Upgrade​

The volume/data/mdis/ directory layout in the new deployment guide applies only to new installations. Existing instances do not need to change volume paths. Directly changing paths in ops.yaml mounts empty directories, making monitoring history, alert rules, and data source configuration unavailable. Actual data remains at the original paths and becomes accessible after restoring the original configuration.

Collection Configuration After Upgrading from 1.4.x​

Data sources already configured in the UI remain unchanged, and monitoring is not interrupted. Middleware ENV_* variables in ops.yaml are no longer read since 1.5.0 and can be deleted. Keeping them does not affect operation; startup logs simply report that they are deprecated.

Post-Deployment Self-Check​

The delivery package includes an end-to-end regression script. Run it once after deployment to verify the main paths: login, resource pages, Grafana charts, data source creation/deletion and connection tests, PromQL preview, the alert flow, notification channels, and log/trace queries:

pip install playwright && playwright install chromium

# Direct port access
python3 mdis_regression.py --base http://<主机IP>:48881 --token <ENV_OPS_TOKEN>

# Reverse proxy sub-path
python3 mdis_regression.py --base http://<主机IP>:9080 --subpath /mdis --token <ENV_OPS_TOKEN>

Exit code 0 means all checks passed. A nonzero code means some checks failed; the script prints a failure list at the end.