Skip to main content

Kubernetes Cluster Monitoring

The Ops Platform collects Kubernetes node resources, container resources (cAdvisor), and cluster object status (kube-state-metrics), and displays node, Pod, Deployment, and container views in "Resource Monitoring → Kubernetes".

Confirm the Ops Platform Deployment Location​

The Ops Platform deployment location determines how Kubernetes metrics are collected:

ScenarioConfiguration
Ops Platform deployed inside the HAP/HDP Kubernetes cluster (reusing the existing cluster, either ops.yaml or k8s-cluster/)No extra configuration required. Data from this cluster is collected automatically after deployment. Go directly to Verification
Ops Platform outside the target HAP/HDP Kubernetes cluster (for example, standalone Ops Platform deployment while another Kubernetes cluster needs monitoring)Deploy collection components inside the monitored cluster
Monitoring multiple Kubernetes clustersThe HAP/HDP Kubernetes cluster where the Ops Platform runs is collected automatically; deploy one set of collection components in each other monitored cluster. See Monitoring Multiple Clusters

The processes scraping kubelet, cAdvisor, and kube-state-metrics run inside the monitored cluster and access apiserver through a ServiceAccount. For cross-cluster collection, the collection components must also send metrics through the Ops Platform gateway and authenticate with the Ops Platform token.

Ops Platform Inside the HAP/HDP Kubernetes Cluster: Automatic Collection​

Kubernetes deployment manifests already include the following configuration, so no extra action is required:

ComponentLocation
ENV_K8S_MONITOR_MODE: "incluster"ConfigMap (ops.yaml top section/11-configmap.yaml)
Prometheus ServiceAccount + ClusterRoleEnd of the same ops.yaml/20-rbac.yaml
kube-state-metricsEnd of the same ops.yaml / ops-ksm in 40-stateless.yaml

The built-in Prometheus directly collects the HAP/HDP Kubernetes cluster where the Ops Platform runs and stores data in the local TSDB. This scenario requires no remote_write configuration, avoiding duplicate pushes back to itself.

When upgrading from a version before 1.5.7, the original single-file ops.yaml lacks RBAC and kube-state-metrics, so the Kubernetes panel may have no data. Download ops.yaml again and apply it over the existing deployment to add these configurations without affecting existing data. The additions include ServiceAccount, ClusterRole, kube-state-metrics, and three ConfigMap keys.

To disable the Kubernetes panel, set ENV_K8S_MONITOR_MODE in the ConfigMap to off and remove kube-state-metrics related objects.

Collection Component Mode: Ops Platform Outside the Target Cluster or Monitoring Multiple Clusters​

Deploy collection components inside the monitored cluster, including a Prometheus dedicated to Kubernetes metrics, RBAC, and kube-state-metrics. They push metrics to the Ops Platform through remote_write; reverse access from the Ops Platform to the monitored cluster is unnecessary.

This mode only requires outbound access from the monitored cluster to the Ops Platform and suits environments with restricted inbound access.

Download k8s-addons.tar.gz

After extraction, check the image address and version in 30-prometheus-incluster.yaml and ensure that monitored cluster nodes can pull the image. Image versions in the download package do not update automatically with the site configuration. Then modify these three variables:

- name: ENV_PROMETHEUS_REMOTE_WRITE_URL
value: "http://运维平台地址:30881/mdis/prometheus/server/api/v1/write"
- name: ENV_PROMETHEUS_REMOTE_WRITE_TOKEN
value: "运维平台的 ENV_OPS_TOKEN"
- name: ENV_K8S_CLUSTER_NAME
value: "prod-k8s" # Cluster identifier used by dashboards; avoid duplicate names

Assemble the write address from all three parts below:

http://<运维平台网关地址>:<端口> + <子路径> + /prometheus/server/api/v1/write
↑ ↑ ↑
Kubernetes: ops-gateway Ops Platform Prometheus runs under
NodePort (default 30881) ENV_OPS_ --web.external-url=
Standalone: 48881 SUB_PATH /prometheus/server/
default /mdis

Observe these requirements when entering the address and token:

  1. Do not use the Ops Platform :9090. ops-prometheus is a ClusterIP accessible only inside the Ops Platform cluster; other clusters must use the gateway's external port.

  2. The address must include /prometheus/server/; omitting it returns 404.

  3. Enter the Ops Platform ENV_OPS_TOKEN. The gateway write endpoint uses Bearer authentication and returns 401 without a token. This is the same token used to log in to the Ops Platform.

Apply after modification:

kubectl apply \
-f k8s-addons/00-namespace.yaml \
-f k8s-addons/10-kube-state-metrics.yaml \
-f k8s-addons/30-prometheus-incluster.yaml

An incorrect address or token may cause continuous retries on the collector side and usually appears as "No Data" in the UI. After deployment, run these commands inside a collection component Pod to check the gateway endpoint:

# Without token: expect 401, confirming the endpoint exists and authentication works
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
http://运维平台地址:30881/mdis/prometheus/server/api/v1/write

# With token: expect 400 (the endpoint exists but rejects an empty request body)
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
-H 'Authorization: Bearer 运维平台的ENV_OPS_TOKEN' \
http://运维平台地址:30881/mdis/prometheus/server/api/v1/write

For 404, check the address. If both requests return 401, check the token.

Data flow:

┌─ Monitored cluster ─────────────────────────────┐
│ kubelet / cAdvisor ──┐ │
│ kube-state-metrics ──┼─→ Collection Prometheus │
│ apiserver ───────────┘ (uses local SA) │
└────────────────────────────────┬────────────────┘
│ remote_write (outbound only, with Bearer token)
↓
Ops Platform gateway :30881/mdis/prometheus/server/api/v1/write
↓
Ops Platform Prometheus (writes to local TSDB)
↓
Grafana "Resource Monitoring → Kubernetes" panel

The collector pushes only kube_*/container_*/kubelet_*/machine_*/node_* metrics back to the Ops Platform. Its own Prometheus runtime metrics are not pushed and do not consume storage there.

Collection Component Manifests​

FilePurposeRequired
00-namespace.yamlNamespace mdis-monitoringRequired
10-kube-state-metrics.yamlkube-state-metrics + RBAC + ServiceRequired
20-node-exporter.yamlnode_exporter DaemonSet + ServiceOptional (node/container metrics are already covered by kubelet+cAdvisor)
30-prometheus-incluster.yamlIn-cluster ops-prometheus + RBAC + ServiceRequired
40-alloy-logs.yamlContainer log collection DaemonSetDeploy when container logs need to be collected. See Service Log Ingestion

40-alloy-logs.yaml collects logs, not metrics, and is not included in the apply command above. Modify ENV_LOKI_PUSH_URL before using it. Without this file, the Kubernetes panel still has metrics, but the "Container Console" has no data.

Do not deploy collection components to the HAP/HDP Kubernetes cluster where the Ops Platform runs

The Ops Platform already collects its own HAP/HDP Kubernetes cluster automatically. Another set of collection components causes duplicate collection: two copies of data are written to the same TSDB under different cluster names, creating confusing entries in the dashboard dropdown and additional storage usage.

Monitoring Multiple Clusters​

Metrics from each cluster enter the Ops Platform with the origin_prometheus=<集群名> label. The "Resource Monitoring → Kubernetes" panel has a "Cluster" dropdown at the top for switching clusters; the default All view shows combined data.

The following example configures the Ops Platform in cluster A to also monitor cluster B.

Step 1: Configure the Name of Cluster A​

Cluster A is already collected automatically. Set an identifiable cluster name by modifying the Ops Platform ConfigMap:

kubectl -n hap-ops edit configmap ops-config

Change ENV_K8S_CLUSTER_NAME from default to a meaningful name:

ENV_K8S_CLUSTER_NAME: "main-k8s"

Restart Prometheus to apply the change (the value is written to Prometheus external_labels at startup). For hostPath deployment, run:

kubectl -n hap-ops rollout restart deployment/ops-prometheus

PVC deployment uses a StatefulSet. Run:

kubectl -n hap-ops rollout restart statefulset/ops-prometheus
Impact of Cluster Renaming on Historical Data

origin_prometheus is a metric label. New data uses the new name after renaming, while historical data retains default, so the dropdown shows both entries. Confirm the name early in rollout to avoid later changes affecting historical data identification.

Step 2: Deploy Collection Components in Cluster B​

First confirm that B can access A's gateway port outbound (default NodePort 30881). A does not need reverse access to B:

# On any node in cluster B
curl -s -o /dev/null -w '%{http_code}\n' http://集群A地址:30881/mdis/web/

Then download k8s-addons.tar.gz and modify these three variables in 30-prometheus-incluster.yaml:

- name: ENV_PROMETHEUS_REMOTE_WRITE_URL
value: "http://集群A地址:30881/mdis/prometheus/server/api/v1/write"
- name: ENV_PROMETHEUS_REMOTE_WRITE_TOKEN
value: "集群A运维平台的 ENV_OPS_TOKEN"
- name: ENV_K8S_CLUSTER_NAME
value: "prod-k8s"

Apply using cluster B's kubeconfig:

kubectl apply \
-f k8s-addons/00-namespace.yaml \
-f k8s-addons/10-kube-state-metrics.yaml \
-f k8s-addons/30-prometheus-incluster.yaml

Step 3: Confirm Successful Push​

Check the collection component logs in cluster B. There should be no remote_write retry errors:

kubectl -n mdis-monitoring logs deploy/ops-prometheus | grep -iE '\[k8s\]|remote|error' | tail

Back in the Ops Platform, the "Cluster" dropdown in "Resource Monitoring → Kubernetes" should show both main-k8s and prod-k8s. You can also query the Ops Platform Prometheus directly to see which clusters are reporting data:

count by (origin_prometheus) (kube_node_info)

Add Other Clusters​

Repeat Step 2. Each cluster only needs a different ENV_K8S_CLUSTER_NAME; the write address and token point to the same Ops Platform:

ClusterConfigurationENV_K8S_CLUSTER_NAME
A (Ops Platform location)Already collected automatically; modify ConfigMapmain-k8s
BOne set of collection componentsprod-k8s
COne set of collection componentstest-k8s
………

Cluster names appear in the dashboard dropdown. Use identifiable names and avoid IPs or duplicates; metrics from clusters sharing a name are mixed and cannot be distinguished.

Metrics from all clusters are stored in the Ops Platform TSDB. As the number or size of clusters grows, monitor storage capacity and adjust ENV_PROMETHEUS_RETENTION_SIZE or expand storage as needed.

Verification​

ServiceAccount + apiserver proxy automatically discovers kubelet, cAdvisor, kube-state-metrics, and pods. No kubelet port exposure or kubeconfig is required.

The Ops Platform automatically collects its own HAP/HDP Kubernetes cluster. For hostPath deployment, run:

kubectl -n hap-ops get deploy kube-state-metrics
kubectl -n hap-ops exec deploy/ops-prometheus -- \
curl -s localhost:9090/prometheus/server/api/v1/targets | grep -o '"job":"k8s-[^"]*"'

In PVC deployment, kube-state-metrics and Prometheus use the ops-ksm Deployment and ops-prometheus StatefulSet respectively:

kubectl -n hap-ops get deploy ops-ksm
kubectl -n hap-ops exec statefulset/ops-prometheus -- \
curl -s localhost:9090/prometheus/server/api/v1/targets | grep -o '"job":"k8s-[^"]*"'

Collection component mode (deployed in mdis-monitoring of the monitored cluster):

kubectl -n mdis-monitoring get pods
kubectl -n mdis-monitoring logs deploy/ops-prometheus | grep '\[k8s\]'

Startup logs should include [k8s] incluster 采集已启用,集群标识 origin_prometheus="..."; after configuring remote_write, logs also show the push target. An incorrect address prefix produces a warning in this log.

In both cases, the jobs k8s-kubelet/k8s-cadvisor/k8s-kube-state-metrics should all be UP. Open "Resource Monitoring → Kubernetes" to view nodes and Pods.