Object Storage Integration (MinIO/COS)
Logs (Loki) and traces (Tempo) are stored in local volumes by default. Object storage can be configured when data volumes are large or data needs to be shared across nodes.
Scope:
- Standalone deployment: Only logs (Loki) need object storage consideration. Standalone deployment provides no tracing and generates no trace data; configuring object storage for
ops-tempois neither necessary nor recommended. - Cluster deployment: Both logs and traces can use object storage and must use two different buckets.
1. Prepare MinIO
If MinIO or cloud object storage is already available, skip to Create Buckets. The following command deploys an example MinIO instance:
docker run -d --name minio \
-p 9000:9000 -p 9001:9001 \
-e MINIO_ROOT_USER=minioadmin \
-e MINIO_ROOT_PASSWORD='<强随机密码>' \
-v /data/minio:/data \
--restart always \
minio/minio server /data --console-address ":9001"
9000is the S3 API port used by the Ops Platform;9001is the Web console.- In production, replace
MINIO_ROOT_PASSWORDwith a strong random password and expose port 9000 only on the internal network.
Open http://MinIO主机IP:9001 in a browser and log in with the account and password above.
2. Create Buckets
Loki and Tempo must use different buckets. Sharing a bucket causes metadata to overwrite each other.
In the console, select "Buckets → Create Bucket", or create them with:
docker run --rm --network host --entrypoint sh minio/mc -c "
mc alias set local http://<MinIO主机IP>:9000 minioadmin '<密码>' &&
mc mb local/mdis-loki &&
mc mb local/mdis-tempo &&
mc ls local"
Bucket names can be customized. This document uses mdis-loki and mdis-tempo.
3. Create Access Keys (AK/SK)
A dedicated account is recommended. In the console, select "Access Keys → Create access key" and save the generated Access Key and Secret Key. The Secret Key is shown only once.
Alternatively, create a dedicated user and assign the readwrite policy from the command line. Confirm the policy's permission scope against the object storage service configuration:
docker run --rm --network host --entrypoint sh minio/mc -c "
mc alias set local http://<MinIO主机IP>:9000 minioadmin '<密码>' &&
mc admin user add local mdis '<SK,8位以上>' &&
mc admin policy attach local readwrite --user mdis"
4. Fill in Configuration
Standalone (ops.yaml)
Add these environment variables to the ops-loki service:
ops-loki:
environment:
ENV_S3_ENDPOINT: 192.168.1.10:9000 # MinIO S3 API address; host:port is recommended
ENV_S3_BUCKET_LOKI: mdis-loki # Create this bucket in MinIO beforehand
ENV_S3_ACCESS_KEY: mdis
ENV_S3_SECRET_KEY: '<SK>'
ENV_S3_FORCE_PATH_STYLE: 'true'
After modification, run docker compose -f ops.yaml up -d ops-loki to recreate the container.
Kubernetes: hostPath Deployment
Add the object storage connection details and both bucket names to the ops-config ConfigMap in the deployment manifest ops.yaml:
data:
ENV_S3_ENDPOINT: "minio.default.svc.cluster.local:9000"
ENV_S3_ACCESS_KEY: "mdis"
ENV_S3_SECRET_KEY: "<SK>"
ENV_S3_FORCE_PATH_STYLE: "true"
ENV_S3_BUCKET_LOKI: "mdis-loki"
ENV_S3_BUCKET_TEMPO: "mdis-tempo"
Create both buckets beforehand and use different names. Apply the deployment manifest after configuration. If services are already running, restart ops-loki and ops-tempo for the environment variables to take effect. Restrict read access to the ConfigMap.
Kubernetes: PVC Deployment (10-secret.yaml)
stringData:
ENV_S3_ENDPOINT: "minio.default.svc.cluster.local:9000"
ENV_S3_ACCESS_KEY: "mdis"
ENV_S3_SECRET_KEY: "<SK>"
ENV_S3_FORCE_PATH_STYLE: "true"
In 30-stateful.yaml, configure ENV_S3_BUCKET_LOKI for ops-loki and
ENV_S3_BUCKET_TEMPO for ops-tempo. The two buckets must be different.
Do not put bucket names into the shared Secret or use the old generic ENV_S3_BUCKET variable. It is passed to both
components; if Loki writes the directory structure first, Tempo initialization fails and enters CrashLoopBackOff.
(ENV_S3_BUCKET is retained only for compatibility with existing deployments.)
Create both buckets before deployment. A missing bucket causes Loki to report NoSuchBucket and restart repeatedly.
5. Endpoint Format (Key Check)
ENV_S3_ENDPOINT is the MinIO S3 API address, not the console address, such as 192.168.1.10:9000
or http://192.168.1.10:9000. Both with and without http:// are accepted: Tempo uses minio-go, which only accepts
host:port, and the Ops Platform automatically strips the scheme when starting Tempo. Loki uses HTTP as fixed in configuration
(insecure: true). For this reason, object storage providing only HTTPS is currently unsupported.
| Deployment Relationship | endpoint |
|---|---|
| MinIO and Ops Platform in the same compose network | Container name, such as minio:9000 |
| MinIO in another compose stack or deployed on the host | Host IP + host mapped port, such as 192.168.1.10:9011 |
| MinIO in the same Kubernetes cluster | Service DNS, such as minio.default.svc.cluster.local:9000 |
| Cloud object storage | Endpoint domain provided by the vendor |
Container names cannot be resolved outside the same compose network. ops-loki fails to start and reports:
dial tcp: lookup minio: no such host
Use the host IP in this case. The mapped host port may differ from 9000; use the value actually shown by docker ps
(for example, use 9011 for a 9011:9000 mapping).
Path Style
| Object Storage | ENV_S3_FORCE_PATH_STYLE |
|---|---|
| MinIO | true |
| Tencent Cloud COS | false (only virtual-hosted style is supported; setting true causes connection failure) |
| UCloud US3 | true |
6. Verification
After configuration takes effect, check Loki logs. For Docker Compose deployment, run:
docker logs <loki 容器> 2>&1 | grep -iE "error|fail" | head
For Kubernetes deployment, run:
kubectl -n hap-ops logs -l app=ops-loki --tail=100
After triggering log writes, check whether objects appear in the bucket:
docker run --rm --network host --entrypoint sh minio/mc -c "
mc alias set local http://<MinIO主机IP>:9000 mdis '<SK>' &&
mc ls -r local/mdis-loki | head"
Objects starting with fake/ or index/ indicate success. This may take several minutes after configuration because Loki uploads only after batch writes.
7. Common Errors
| Error in Logs | Cause |
|---|---|
dial tcp: lookup xxx: no such host | endpoint uses a name that cannot be resolved in the container network. See Step 5 |
connection refused | Incorrect port configuration; use the mapping shown by docker ps |
SignatureDoesNotMatch | Incorrect AK/SK |
AccessDenied | AK/SK is correct but has no permission for the bucket; check the user policy |
NoSuchBucket | Bucket not created or incorrect bucket name |
Switching Loki storage backends does not migrate existing data; old logs remain in the original local volume. Generate new logs after switching to see content in the new backend. This does not mean configuration failed.