Skip to main content

Tracing

Tracing uses Tempo and Grafana Alloy to show request call paths and durations across multiple microservices, helping troubleshoot cross-service performance issues and exceptions. Open it from "Tracing → Trace Analysis" in the left sidebar.

Trace Analysis​

The top of the page contains three service-aggregated overview charts, and the bottom contains a searchable trace list:

Trace analysis

AreaDescription
Request Rate (req/s)Request rate by service, colored by service
Error Rate (5xx/s)Server-side error rate by service, useful for finding abnormal services
P95 Latency (P95 delay)95th percentile latency by service, used to identify slow services
Recent TracesRecent trace details (Trace ID/service/operation/start time/duration)
Top Service CallsMost frequent service-to-service calls and call rates, showing service dependencies intuitively

Search and Drill Down​

The top of the list supports multi-dimensional filters:

  • Service: Filter by service.
  • Operation Keyword: Filter by operation name keyword.
  • Min Duration (ms): Filter traces whose duration exceeds the specified value.
  • Error Filter: Show only traces with errors.
  • Trace ID: Search for a trace by Trace ID.

Click a Trace ID in the list to expand the call waterfall below and view span durations across services:

Trace waterfall and service call relationship

Connect Tracing​

Trace data is reported by the business side through OTLP. See Tracing Ingestion for the connection steps.

Tracing is intended for Kubernetes cluster deployment. Standalone (Docker Compose) deployment is currently not supported, so this page has no data in standalone scenarios.

Trace retention is controlled by ENV_TEMPO_RETENTION (30 days by default). See Environment Variables.