Kubernetes Cluster Monitoring
The Ops Platform collects Kubernetes node resources, container resources (cAdvisor), and cluster object status (kube-state-metrics), and displays node, Pod, Deployment, and container views in "Resource Monitoring → Kubernetes".
Confirm the Ops Platform Deployment Location
The Ops Platform deployment location determines how Kubernetes metrics are collected:
| Scenario | Configuration |
|---|---|
Ops Platform deployed inside the HAP/HDP Kubernetes cluster (reusing the existing cluster, either ops.yaml or k8s-cluster/) | No extra configuration required. Data from this cluster is collected automatically after deployment. Go directly to Verification |
| Ops Platform outside the target HAP/HDP Kubernetes cluster (for example, standalone Ops Platform deployment while another Kubernetes cluster needs monitoring) | Deploy collection components inside the monitored cluster |
| Monitoring multiple Kubernetes clusters | The HAP/HDP Kubernetes cluster where the Ops Platform runs is collected automatically; deploy one set of collection components in each other monitored cluster. See Monitoring Multiple Clusters |
The processes scraping kubelet, cAdvisor, and kube-state-metrics run inside the monitored cluster and access apiserver through a ServiceAccount. For cross-cluster collection, the collection components must also send metrics through the Ops Platform gateway and authenticate with the Ops Platform token.
Ops Platform Inside the HAP/HDP Kubernetes Cluster: Automatic Collection
Kubernetes deployment manifests already include the following configuration, so no extra action is required:
| Component | Location |
|---|---|
ENV_K8S_MONITOR_MODE: "incluster" | ConfigMap (ops.yaml top section/11-configmap.yaml) |
| Prometheus ServiceAccount + ClusterRole | End of the same ops.yaml/20-rbac.yaml |
| kube-state-metrics | End of the same ops.yaml / ops-ksm in 40-stateless.yaml |
The built-in Prometheus directly collects the HAP/HDP Kubernetes cluster where the Ops Platform runs and stores data in the local TSDB. This scenario requires no remote_write configuration, avoiding duplicate pushes back to itself.
When upgrading from a version before 1.5.7, the original single-file ops.yaml lacks RBAC and kube-state-metrics, so the Kubernetes panel may have no data. Download ops.yaml again and apply it over the existing deployment to add these configurations without affecting existing data. The additions include ServiceAccount, ClusterRole, kube-state-metrics, and three ConfigMap keys.
To disable the Kubernetes panel, set ENV_K8S_MONITOR_MODE in the ConfigMap to off and remove
kube-state-metrics related objects.
Collection Component Mode: Ops Platform Outside the Target Cluster or Monitoring Multiple Clusters
Deploy collection components inside the monitored cluster, including a Prometheus dedicated to Kubernetes metrics, RBAC, and kube-state-metrics. They push metrics to the Ops Platform through remote_write; reverse access from the Ops Platform to the monitored cluster is unnecessary.
This mode only requires outbound access from the monitored cluster to the Ops Platform and suits environments with restricted inbound access.
After extraction, check the image address and version in 30-prometheus-incluster.yaml and ensure that monitored cluster nodes can pull the image. Image versions in the download package do not update automatically with the site configuration. Then modify these three variables:
- name: ENV_PROMETHEUS_REMOTE_WRITE_URL
value: "http://运维平台地址:30881/mdis/prometheus/server/api/v1/write"
- name: ENV_PROMETHEUS_REMOTE_WRITE_TOKEN
value: "运维平台的 ENV_OPS_TOKEN"
- name: ENV_K8S_CLUSTER_NAME
value: "prod-k8s" # Cluster identifier used by dashboards; avoid duplicate names
Assemble the write address from all three parts below:
http://<运维平台网关地址>:<端口> + <子路径> + /prometheus/server/api/v1/write
↑ ↑ ↑
Kubernetes: ops-gateway Ops Platform Prometheus runs under
NodePort (default 30881) ENV_OPS_ --web.external-url=
Standalone: 48881 SUB_PATH /prometheus/server/
default /mdis
Observe these requirements when entering the address and token:
-
Do not use the Ops Platform
:9090.ops-prometheusis a ClusterIP accessible only inside the Ops Platform cluster; other clusters must use the gateway's external port. -
The address must include
/prometheus/server/; omitting it returns 404. -
Enter the Ops Platform
ENV_OPS_TOKEN. The gateway write endpoint usesBearerauthentication and returns 401 without a token. This is the same token used to log in to the Ops Platform.
Apply after modification:
kubectl apply \
-f k8s-addons/00-namespace.yaml \
-f k8s-addons/10-kube-state-metrics.yaml \
-f k8s-addons/30-prometheus-incluster.yaml
An incorrect address or token may cause continuous retries on the collector side and usually appears as "No Data" in the UI. After deployment, run these commands inside a collection component Pod to check the gateway endpoint:
# Without token: expect 401, confirming the endpoint exists and authentication works
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
http://运维平台地址:30881/mdis/prometheus/server/api/v1/write
# With token: expect 400 (the endpoint exists but rejects an empty request body)
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
-H 'Authorization: Bearer 运维平台的ENV_OPS_TOKEN' \
http://运维平台地址:30881/mdis/prometheus/server/api/v1/write
For 404, check the address. If both requests return 401, check the token.
Data flow:
┌─ Monitored cluster ─────────────────────────────┐
│ kubelet / cAdvisor ──┐ │
│ kube-state-metrics ──┼─→ Collection Prometheus │
│ apiserver ───────────┘ (uses local SA) │
└────────────────────────────────┬────────────────┘
│ remote_write (outbound only, with Bearer token)
↓
Ops Platform gateway :30881/mdis/prometheus/server/api/v1/write
↓
Ops Platform Prometheus (writes to local TSDB)
↓
Grafana "Resource Monitoring → Kubernetes" panel
The collector pushes only kube_*/container_*/kubelet_*/machine_*/node_* metrics back to the Ops Platform. Its own Prometheus runtime metrics are not pushed and do not consume storage there.
Collection Component Manifests
| File | Purpose | Required |
|---|---|---|
00-namespace.yaml | Namespace mdis-monitoring | Required |
10-kube-state-metrics.yaml | kube-state-metrics + RBAC + Service | Required |
20-node-exporter.yaml | node_exporter DaemonSet + Service | Optional (node/container metrics are already covered by kubelet+cAdvisor) |
30-prometheus-incluster.yaml | In-cluster ops-prometheus + RBAC + Service | Required |
40-alloy-logs.yaml | Container log collection DaemonSet | Deploy when container logs need to be collected. See Service Log Ingestion |
40-alloy-logs.yaml collects logs, not metrics, and is not included in the apply command above. Modify
ENV_LOKI_PUSH_URL before using it. Without this file, the Kubernetes panel still has metrics, but the "Container Console" has no data.
The Ops Platform already collects its own HAP/HDP Kubernetes cluster automatically. Another set of collection components causes duplicate collection: two copies of data are written to the same TSDB under different cluster names, creating confusing entries in the dashboard dropdown and additional storage usage.
Monitoring Multiple Clusters
Metrics from each cluster enter the Ops Platform with the origin_prometheus=<集群名> label. The "Resource Monitoring → Kubernetes" panel
has a "Cluster" dropdown at the top for switching clusters; the default All view shows combined data.
The following example configures the Ops Platform in cluster A to also monitor cluster B.
Step 1: Configure the Name of Cluster A
Cluster A is already collected automatically. Set an identifiable cluster name by modifying the Ops Platform ConfigMap:
kubectl -n hap-ops edit configmap ops-config
Change ENV_K8S_CLUSTER_NAME from default to a meaningful name:
ENV_K8S_CLUSTER_NAME: "main-k8s"
Restart Prometheus to apply the change (the value is written to Prometheus external_labels at startup). For hostPath deployment, run:
kubectl -n hap-ops rollout restart deployment/ops-prometheus
PVC deployment uses a StatefulSet. Run:
kubectl -n hap-ops rollout restart statefulset/ops-prometheus
origin_prometheus is a metric label. New data uses the new name after renaming, while historical data retains default,
so the dropdown shows both entries. Confirm the name early in rollout to avoid later changes affecting historical data identification.
Step 2: Deploy Collection Components in Cluster B
First confirm that B can access A's gateway port outbound (default NodePort 30881). A does not need reverse access to B:
# On any node in cluster B
curl -s -o /dev/null -w '%{http_code}\n' http://集群A地址:30881/mdis/web/
Then download k8s-addons.tar.gz and modify
these three variables in 30-prometheus-incluster.yaml:
- name: ENV_PROMETHEUS_REMOTE_WRITE_URL
value: "http://集群A地址:30881/mdis/prometheus/server/api/v1/write"
- name: ENV_PROMETHEUS_REMOTE_WRITE_TOKEN
value: "集群A运维平台的 ENV_OPS_TOKEN"
- name: ENV_K8S_CLUSTER_NAME
value: "prod-k8s"
Apply using cluster B's kubeconfig:
kubectl apply \
-f k8s-addons/00-namespace.yaml \
-f k8s-addons/10-kube-state-metrics.yaml \
-f k8s-addons/30-prometheus-incluster.yaml
Step 3: Confirm Successful Push
Check the collection component logs in cluster B. There should be no remote_write retry errors:
kubectl -n mdis-monitoring logs deploy/ops-prometheus | grep -iE '\[k8s\]|remote|error' | tail
Back in the Ops Platform, the "Cluster" dropdown in "Resource Monitoring → Kubernetes" should show both main-k8s and prod-k8s.
You can also query the Ops Platform Prometheus directly to see which clusters are reporting data:
count by (origin_prometheus) (kube_node_info)
Add Other Clusters
Repeat Step 2. Each cluster only needs a different ENV_K8S_CLUSTER_NAME; the write address and token point to the same Ops Platform:
| Cluster | Configuration | ENV_K8S_CLUSTER_NAME |
|---|---|---|
| A (Ops Platform location) | Already collected automatically; modify ConfigMap | main-k8s |
| B | One set of collection components | prod-k8s |
| C | One set of collection components | test-k8s |
| … | … | … |
Cluster names appear in the dashboard dropdown. Use identifiable names and avoid IPs or duplicates; metrics from clusters sharing a name are mixed and cannot be distinguished.
Metrics from all clusters are stored in the Ops Platform TSDB. As the number or size of clusters grows, monitor storage capacity and adjust ENV_PROMETHEUS_RETENTION_SIZE or expand storage as needed.
Verification
ServiceAccount + apiserver proxy automatically discovers kubelet, cAdvisor, kube-state-metrics, and pods. No kubelet port exposure or kubeconfig is required.
The Ops Platform automatically collects its own HAP/HDP Kubernetes cluster. For hostPath deployment, run:
kubectl -n hap-ops get deploy kube-state-metrics
kubectl -n hap-ops exec deploy/ops-prometheus -- \
curl -s localhost:9090/prometheus/server/api/v1/targets | grep -o '"job":"k8s-[^"]*"'
In PVC deployment, kube-state-metrics and Prometheus use the ops-ksm Deployment and ops-prometheus StatefulSet respectively:
kubectl -n hap-ops get deploy ops-ksm
kubectl -n hap-ops exec statefulset/ops-prometheus -- \
curl -s localhost:9090/prometheus/server/api/v1/targets | grep -o '"job":"k8s-[^"]*"'
Collection component mode (deployed in mdis-monitoring of the monitored cluster):
kubectl -n mdis-monitoring get pods
kubectl -n mdis-monitoring logs deploy/ops-prometheus | grep '\[k8s\]'
Startup logs should include [k8s] incluster 采集已启用,集群标识 origin_prometheus="...";
after configuring remote_write, logs also show the push target. An incorrect address prefix produces a warning in this log.
In both cases, the jobs k8s-kubelet/k8s-cadvisor/k8s-kube-state-metrics should all be UP.
Open "Resource Monitoring → Kubernetes" to view nodes and Pods.