Tracing Ingestion
Tracing shows the call path and latency of a request across multiple microservices. Business services report data through OTLP to the Ops Platform Alloy, which forwards it to Tempo for storage.
Tracing is intended for Kubernetes cluster deployment. Standalone (Docker Compose) deployment is currently unsupported.
Recommended Method: Istio Sidecar Automatic Reporting
When business services run inside an Istio mesh, sidecars can generate and report spans automatically without changes to business code. HAP/HDP cluster mode usually already uses the mesh.
After installing Istio and enabling istio-injection=enabled, configure the reporting address and sampling rate to generate spans. Missing configuration usually produces no error, but the tracing page has no data.
Self-check:
kubectl -n istio-system get cm istio \
-o jsonpath='{.data.mesh}' | grep -A3 extensionProviders
No output means configuration is missing. Complete the following steps.
Step 1: Configure the Istio Reporting Address
kubectl -n istio-system edit configmap istio
The value of mesh is YAML text. Add extensionProviders at the top level of this text, alongside defaultConfig and rootNamespace:
apiVersion: v1
kind: ConfigMap
metadata:
name: istio
namespace: istio-system
data:
mesh: |- # ← Note: this is a string block
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
defaultProviders:
metrics:
- prometheus
enablePrometheusMerge: true
rootNamespace: istio-system
trustDomain: cluster.local
extensionProviders: # ← Add this entire block
- name: mdis-otel
opentelemetry:
service: ops-alloy.<运维平台命名空间>.svc.cluster.local
port: 4317
meshNetworks: |-
networks: {}
Observe these configuration requirements:
-
extensionProvidersis a top-level mesh field; do not indent it underdefaultConfig. At the wrong level, istiod ignores the configuration and no spans are generated. -
servicemust be a full FQDN using the actual Ops Platform namespace. Default manifests usehap-ops, but actual deployments may differ. With an incorrect namespace, sidecars cannot resolve the service and usually report no error. Confirm the namespace with:
kubectl get svc -A | grep ops-alloy
- The
opentelemetryprovider uses OTLP gRPC on port 4317.
After modification, restart istiod to reload configuration:
kubectl -n istio-system rollout restart deployment/istiod
kubectl -n istio-system rollout status deployment/istiod
Read the configuration back and confirm that the added settings appear:
kubectl -n istio-system get cm istio -o jsonpath='{.data.mesh}' | grep -A4 extensionProviders
Step 2: Enable Sampling
The previous step only sets the Istio reporting address. Configure the sampling rate as well. Without a Telemetry resource, the sampling rate is 0 and no spans are generated.
kubectl apply -f - <<'EOF'
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
name: mdis-tracing
namespace: istio-system # Place in the Istio root namespace to apply mesh-wide
spec:
tracing:
- providers:
- name: mdis-otel # Must exactly match the name in Step 1
randomSamplingPercentage: 100
EOF
Confirm that the Telemetry resource was created:
kubectl get telemetry -A
We recommend setting randomSamplingPercentage to 100 during verification (with 10% sampling, a small number of requests may not be sampled,
which can be mistaken for configuration failure). After confirming connectivity, reduce it to 1~10. Higher sampling rates increase Tempo storage and write load.
Changing this value requires no istiod restart, but business Pods must restart for it to take effect.
The example uses telemetry.istio.io/v1 (Istio 1.22+, verified on 1.29). Earlier Istio versions use
v1alpha1 with the same remaining fields. Check the current version with:
kubectl api-resources | grep telemetry
Step 3: Restart Business Pods
Sidecars must reload configuration before reporting traces. Restart business Pods:
kubectl -n <业务命名空间> rollout restart deployment/<名字>
Confirm that injection is enabled for the business namespace:
kubectl get ns -L istio-injection
The corresponding row must show enabled to have sidecars.
Step 4: Confirm Spans Have Reached the Ops Platform
This step checks whether the Ops Platform Alloy has received spans:
kubectl -n <运维平台命名空间> exec deploy/ops-alloy -- \
curl -s localhost:12345/metrics | grep otelcol_receiver_accepted_spans_total
Trigger several cross-service calls, then check again. An increasing metric indicates that the reporting path is connected:
otelcol_receiver_accepted_spans_total{...,transport="grpc"} 83
If the metric remains 0 or cannot be found, troubleshoot with this table:
| Symptom | Likely Cause |
|---|---|
| Metric is 0 | Business Pods were not restarted (Step 3), or the sampling rate is too low and requests were not sampled |
| Metric does not exist | Alloy has not enabled OTLP receiving; check whether the ops-alloy Service exposes 4317 |
| Metric increases but page is empty | Alloy→Tempo path issue; check ENV_TEMPO_GRPC_URL and ops-tempo container logs |
Reporting Endpoints
Alloy OTLP receiving endpoints:
| Protocol | Endpoint |
|---|---|
| OTLP gRPC | 4317 |
| OTLP HTTP | 4318 |
For applications using an OpenTelemetry SDK, point the exporter to these endpoints. Inside the cluster, use Service DNS, such as
http://ops-alloy.<运维平台命名空间>.svc.cluster.local:4317 (replace the namespace with the actual Ops Platform namespace;
check it with kubectl get svc -A | grep ops-alloy).
ops-alloy is ClusterIP and resolves only inside the Ops Platform cluster. Cross-cluster reporting requires exposing 4317 first
through NodePort or Ingress. Trace data volumes can be large; plan exposure and rate limits according to the actual network environment.
The Alloy forwarding address for Tempo is controlled by ENV_TEMPO_GRPC_URL (default http://ops-tempo:4317); standard deployments require no change.
Data Retention and Storage
Trace data retention is controlled by ENV_TEMPO_RETENTION, default 30 days. See Environment Variables.
For long-term trace retention without node disk limitations, prepare a Tempo object storage bucket during deployment (recommended name mdis-tempo). See Object Storage.
Verification
After integration, trigger several cross-service calls and open Tracing. Integration is successful when the three overview charts at the top (request volume/error rate/P95 latency) show data and the Recent Traces list at the bottom displays trace details.