Skip to main content

Object Storage Integration (MinIO/COS)

Logs (Loki) and traces (Tempo) are stored in local volumes by default. Object storage can be configured when data volumes are large or data needs to be shared across nodes.

Scope:

  • Standalone deployment: Only logs (Loki) need object storage consideration. Standalone deployment provides no tracing and generates no trace data; configuring object storage for ops-tempo is neither necessary nor recommended.
  • Cluster deployment: Both logs and traces can use object storage and must use two different buckets.

1. Prepare MinIO​

If MinIO or cloud object storage is already available, skip to Create Buckets. The following command deploys an example MinIO instance:

docker run -d --name minio \
-p 9000:9000 -p 9001:9001 \
-e MINIO_ROOT_USER=minioadmin \
-e MINIO_ROOT_PASSWORD='<强随机密码>' \
-v /data/minio:/data \
--restart always \
minio/minio server /data --console-address ":9001"
  • 9000 is the S3 API port used by the Ops Platform; 9001 is the Web console.
  • In production, replace MINIO_ROOT_PASSWORD with a strong random password and expose port 9000 only on the internal network.

Open http://MinIO主机IP:9001 in a browser and log in with the account and password above.

2. Create Buckets​

Loki and Tempo must use different buckets. Sharing a bucket causes metadata to overwrite each other.

In the console, select "Buckets → Create Bucket", or create them with:

docker run --rm --network host --entrypoint sh minio/mc -c "
mc alias set local http://<MinIO主机IP>:9000 minioadmin '<密码>' &&
mc mb local/mdis-loki &&
mc mb local/mdis-tempo &&
mc ls local"

Bucket names can be customized. This document uses mdis-loki and mdis-tempo.

3. Create Access Keys (AK/SK)​

A dedicated account is recommended. In the console, select "Access Keys → Create access key" and save the generated Access Key and Secret Key. The Secret Key is shown only once.

Alternatively, create a dedicated user and assign the readwrite policy from the command line. Confirm the policy's permission scope against the object storage service configuration:

docker run --rm --network host --entrypoint sh minio/mc -c "
mc alias set local http://<MinIO主机IP>:9000 minioadmin '<密码>' &&
mc admin user add local mdis '<SK,8位以上>' &&
mc admin policy attach local readwrite --user mdis"

4. Fill in Configuration​

Standalone (ops.yaml)​

Add these environment variables to the ops-loki service:

ops-loki:
environment:
ENV_S3_ENDPOINT: 192.168.1.10:9000 # MinIO S3 API address; host:port is recommended
ENV_S3_BUCKET_LOKI: mdis-loki # Create this bucket in MinIO beforehand
ENV_S3_ACCESS_KEY: mdis
ENV_S3_SECRET_KEY: '<SK>'
ENV_S3_FORCE_PATH_STYLE: 'true'

After modification, run docker compose -f ops.yaml up -d ops-loki to recreate the container.

Kubernetes: hostPath Deployment​

Add the object storage connection details and both bucket names to the ops-config ConfigMap in the deployment manifest ops.yaml:

data:
ENV_S3_ENDPOINT: "minio.default.svc.cluster.local:9000"
ENV_S3_ACCESS_KEY: "mdis"
ENV_S3_SECRET_KEY: "<SK>"
ENV_S3_FORCE_PATH_STYLE: "true"
ENV_S3_BUCKET_LOKI: "mdis-loki"
ENV_S3_BUCKET_TEMPO: "mdis-tempo"

Create both buckets beforehand and use different names. Apply the deployment manifest after configuration. If services are already running, restart ops-loki and ops-tempo for the environment variables to take effect. Restrict read access to the ConfigMap.

Kubernetes: PVC Deployment (10-secret.yaml)​

stringData:
ENV_S3_ENDPOINT: "minio.default.svc.cluster.local:9000"
ENV_S3_ACCESS_KEY: "mdis"
ENV_S3_SECRET_KEY: "<SK>"
ENV_S3_FORCE_PATH_STYLE: "true"
Bucket Name Configuration

In 30-stateful.yaml, configure ENV_S3_BUCKET_LOKI for ops-loki and ENV_S3_BUCKET_TEMPO for ops-tempo. The two buckets must be different.

Do not put bucket names into the shared Secret or use the old generic ENV_S3_BUCKET variable. It is passed to both components; if Loki writes the directory structure first, Tempo initialization fails and enters CrashLoopBackOff. (ENV_S3_BUCKET is retained only for compatibility with existing deployments.)

Create both buckets before deployment. A missing bucket causes Loki to report NoSuchBucket and restart repeatedly.

5. Endpoint Format (Key Check)​

ENV_S3_ENDPOINT is the MinIO S3 API address, not the console address, such as 192.168.1.10:9000 or http://192.168.1.10:9000. Both with and without http:// are accepted: Tempo uses minio-go, which only accepts host:port, and the Ops Platform automatically strips the scheme when starting Tempo. Loki uses HTTP as fixed in configuration (insecure: true). For this reason, object storage providing only HTTPS is currently unsupported.

Deployment Relationshipendpoint
MinIO and Ops Platform in the same compose networkContainer name, such as minio:9000
MinIO in another compose stack or deployed on the hostHost IP + host mapped port, such as 192.168.1.10:9011
MinIO in the same Kubernetes clusterService DNS, such as minio.default.svc.cluster.local:9000
Cloud object storageEndpoint domain provided by the vendor

Container names cannot be resolved outside the same compose network. ops-loki fails to start and reports:

dial tcp: lookup minio: no such host

Use the host IP in this case. The mapped host port may differ from 9000; use the value actually shown by docker ps (for example, use 9011 for a 9011:9000 mapping).

Path Style​

Object StorageENV_S3_FORCE_PATH_STYLE
MinIOtrue
Tencent Cloud COSfalse (only virtual-hosted style is supported; setting true causes connection failure)
UCloud US3true

6. Verification​

After configuration takes effect, check Loki logs. For Docker Compose deployment, run:

docker logs <loki 容器> 2>&1 | grep -iE "error|fail" | head

For Kubernetes deployment, run:

kubectl -n hap-ops logs -l app=ops-loki --tail=100

After triggering log writes, check whether objects appear in the bucket:

docker run --rm --network host --entrypoint sh minio/mc -c "
mc alias set local http://<MinIO主机IP>:9000 mdis '<SK>' &&
mc ls -r local/mdis-loki | head"

Objects starting with fake/ or index/ indicate success. This may take several minutes after configuration because Loki uploads only after batch writes.

7. Common Errors​

Error in LogsCause
dial tcp: lookup xxx: no such hostendpoint uses a name that cannot be resolved in the container network. See Step 5
connection refusedIncorrect port configuration; use the mapping shown by docker ps
SignatureDoesNotMatchIncorrect AK/SK
AccessDeniedAK/SK is correct but has no permission for the bucket; check the user policy
NoSuchBucketBucket not created or incorrect bucket name
Existing Logs Are Not Migrated When Switching Storage

Switching Loki storage backends does not migrate existing data; old logs remain in the original local volume. Generate new logs after switching to see content in the new backend. This does not mean configuration failed.