Kubernetes
This page describes a new Ops Platform deployment inside the HAP/HDP Kubernetes cluster. After deployment, the platform can collect metrics for that cluster. To monitor other clusters, deploy the collection components in each target cluster.
Prerequisites
- Ensure the HAP/HDP Kubernetes cluster runs normally and reserve the computing and storage resources required by the Ops Platform.
- Ensure
kubectlcan access the cluster with permissions to create namespaces, deploy components, and configure RBAC resources. - Prepare accounts for monitored middleware. For permissions, see Data Sources.
- For direct access, clients must reach port
30881on cluster nodes. When using a reverse proxy, the proxy server must reach that port.
Run the following kubectl commands on a server with kubectl configured and the required cluster permissions, such as a Kubernetes control node. Check node status:
kubectl get nodes
This guide uses the hap-ops namespace by default and provides two deployment methods:
| Storage Method | Data Location | Procedure |
|---|---|---|
| hostPath (default) | Local directory on a specified node | Follow steps 1–5 |
| StorageClass and PVC | Storage volumes provided by StorageClass | Complete step 1, then go to Deploy with StorageClass and PVC |
These methods use different configuration files. Do not deploy both in the same namespace. The capacities and directories below are examples for new deployments. Existing deployments should keep their current storage configuration and plan migration or expansion separately.
1. Obtain the Image
The image example below is for Linux AMD64 nodes. Before pulling it, ensure crictl on every node connects to the container runtime used by Kubernetes.
The Ops Platform deploys node collection components through DaemonSets:
ops-nodeagent: Collects node metrics.alloy-logs: Collects container logs. Included in the PVC deployment; deploy separately as needed for hostPath.
Prepare the image on every node for the collection components and other Ops Platform services. Actual node coverage also depends on scheduling conditions such as node taints.
In online environments, run on every node:
crictl pull registry.cn-hangzhou.aliyuncs.com/mdpublic/ops-allinone:1.5.12
In offline environments, download the offline image package matching the node architecture, upload it to every node, and import it on each node. The image address and version in the deployment configuration must match the imported image.
2. Select the Deployment Node
The Ops Platform uses hostPath by default, storing monitoring data, logs, traces, and MongoDB data in /data/mdis/ on the node.
Choose a node that can run the Ops Platform services, ensure the file system containing /data/mdis/ has at least 200 GB of free space, and label the node. The main Ops Platform services are scheduled there by label to access local data.
Find the node name with kubectl get nodes, then run:
kubectl label node <节点名称> hap-ops=true
Ensure only the selected node has this label, so services are not scheduled to another node where the existing data is unavailable.
hostPath data remains on the local node and does not move with Pods. If the node may be rebuilt or replaced, consider storage volumes that support mounting across nodes. PVCs backed by local storage also depend on their node.
Loki logs and Tempo traces are stored locally on the node by default. To use object storage, see Object Storage Configuration.
3. Prepare the Deployment Configuration File
On a Kubernetes control node or another server with kubectl configured and cluster permissions, create /data/mingdao/kubernetes/mdis/ for deployment configuration files:
mkdir -p /data/mingdao/kubernetes/mdis
This directory stores YAML configuration files. Service data is stored under /data/mdis/ on the node selected in step 2.
-
Download the Ops Platform deployment configuration and upload it to
/data/mingdao/kubernetes/mdis/on that server. -
Edit
/data/mingdao/kubernetes/mdis/ops.yamland replace the example values of the following environment variables:Environment Variable Description ENV_OPS_TOKENOps Platform login token. Use a strong password ENV_ALERT_CRYPTO_KEYCredential encryption key. Generate it separately and keep it safe. Changing it later makes existing encrypted credentials impossible to decrypt -
Check the following settings:
- The image address and version match the image used in step 1.
- The node label matches
hap-ops=trueconfigured in step 2.
If the target node has a
NoScheduletaint, also configure the corresponding toleration. See Dedicated Monitoring Nodes.
The configuration includes the hap-ops namespace, permissions for cluster monitoring, and kube-state-metrics.
To use another namespace, modify ops.yaml before deployment:
- Replace the Namespace name, each resource's
metadata.namespace, and the ServiceAccount namespace in ClusterRoleBinding with the actual namespace. - Update the namespace in Service addresses in ConfigMap and
ENV_K8S_KSM_NAMESPACE. If the cluster DNS suffix is notcluster.local, also adjust Service addresses accordingly. - Replace
-n hap-opsin the commands below with the actual namespace. Thehap-ops=truenode label selects the deployment node and does not need to change with the namespace.
The gateway Service contains both the access port and the internal API port. Kubernetes assigns a NodePort to each; restrict network access to these ports.
4. Deploy the Ops Platform
Deploy the Ops Platform:
kubectl apply -f /data/mingdao/kubernetes/mdis/ops.yaml
5. Access and Verify
Check the Ops Platform Pod and Service status:
kubectl get pods,svc -n hap-ops -o wide
Once the Ops Platform Pods are ready, open the following address in a browser:
http://节点IP:30881/mdis/web/
Log in with the configured ENV_OPS_TOKEN and confirm the overview page displays correctly. If ENV_OPS_SUB_PATH was changed, replace /mdis in the access URL with the actual path.
If a Pod is not ready, check the cause of scheduling, image pull, or mount failures:
kubectl describe pod <Pod名称> -n hap-ops
See FAQs for other troubleshooting methods.
Configure a Reverse Proxy
hostPath exposes port 30881 through the ops-gateway Service. PVC uses ops-gateway-ui with the same external port.
To access through a domain name, add upstream nodes in the Nginx http block:
upstream hap-ops {
server 192.168.1.11:30881;
server 192.168.1.12:30881;
server 192.168.1.13:30881;
}
Add the following to the domain's server block:
location /mdis/ {
# allow 192.168.1.100;
# deny all;
proxy_pass http://hap-ops;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
Adjust for your environment:
- Replace the example upstream IPs with cluster node IPs reachable from the proxy server. Add or remove entries for the actual node count.
- To restrict source addresses, enable both
allowanddenyand enter the actual operations egress IP. - When changing the access sub-path, update both
ENV_OPS_SUB_PATHin the deployment configuration and the proxy path.
After checking and reloading the Nginx configuration, access through the domain name. If HTTPS listening and certificates are configured, use https://域名/mdis/web/; otherwise use the actual HTTP protocol.
Next Steps
- Add monitored hosts, middleware, or custom pages in Data Sources.
- Configure the log push address on HAP/HDP services to ingest Service Logs.
- To collect container logs, deploy
alloy-logsseparately for hostPath; see Container Log Ingestion. The PVC deployment already includes this component. - Configure Tracing for the cluster environment.
- To monitor servers outside the cluster, deploy Node Exporter on each target server.
- To monitor other Kubernetes clusters, deploy the collection components in each target cluster.
Deploy with StorageClass and PVC
This method creates PVCs through StorageClass. After completing the prerequisites and obtaining images in step 1, follow this section. Skip hostPath steps 2–5.
Prepare PVC Storage
Check available StorageClasses:
kubectl get storageclass
The default StorageClass is marked (default) in the output. Choose according to the cluster's existing configuration:
- Use the default storage class: The example PVCs do not specify
storageClassName, so no additional changes are needed. - Use another existing storage class: Set
storageClassNameunderspecin all threevolumeClaimTemplatesin30-stateful.yaml. The cluster default does not need to change.
To create a StorageClass, refer to these examples:
StorageClass Configuration Reference
StorageClass definitions and parameters vary by storage service. Follow the service's documentation. These examples are for reference; check the storage type and reclaim policy before use.
The following example uses Tencent Cloud CBS. Ensure the corresponding CSI driver is installed. With reclaimPolicy: Delete, deleting a PVC may delete the associated persistent volume and backend storage. Choose the policy according to data retention requirements:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cbs-expandable
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: com.tencent.cloud.csi.cbs
parameters:
diskType: CLOUD_PREMIUM
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
Save the configuration as storageclass.yaml, then run:
kubectl apply -f storageclass.yaml
Changing the default StorageClass affects future PVCs that do not specify a storage class. If replacing the default is necessary, remove the previous default marker after configuring the new default:
kubectl patch sc <原默认SC名> -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
Test environments can also use local-path-provisioner to store data locally on the node:
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
kubectl patch sc local-path -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
For local storage, adjust PVC capacity according to actual free disk space. local-path-provisioner stores data locally on the node; it does not move with Pods to another node.
Once the StorageClass is available, plan storage using these default capacities:
| Component | Default Capacity |
|---|---|
| Prometheus | 200 GiB |
| MongoDB | 30 GiB |
| Grafana | 20 GiB |
The total is 250 GiB, adjustable in 30-stateful.yaml before the initial deployment. Existing PVCs do not expand automatically when example capacities change, and editing a StatefulSet's volumeClaimTemplates does not expand them directly. Handle expansion separately according to storage driver capabilities.
The PVC package sets Prometheus retention to 30d and the size limit to 80GB, with cleanup triggered by whichever limit is reached first. These are controlled by ENV_PROMETHEUS_RETENTION and ENV_PROMETHEUS_RETENTION_SIZE in 11-configmap.yaml. Increasing PVC capacity does not automatically increase the retention limits.
Loki and Tempo use emptyDir by default in the PVC deployment. Without object storage, logs and traces are stored in temporary directories and lost when the Pod is deleted. To retain them, configure object storage for both components or modify the deployment to use persistent volumes.
For object storage, create separate buckets for Loki and Tempo with different names:
| Configuration | Location |
|---|---|
| Object storage connection information | 10-secret.yaml |
| Loki bucket | ENV_S3_BUCKET_LOKI in 30-stateful.yaml |
| Tempo bucket | ENV_S3_BUCKET_TEMPO in 30-stateful.yaml |
See Object Storage Configuration for parameters.
Prepare PVC Deployment Configuration Files
-
Download the Ops Platform PVC deployment package, upload it to a server with
kubectlconfigured and cluster permissions, and extract it in the upload directory:tar -xzf k8s-cluster.tar.gzcd k8s-clusterSOURCE.jsonin the package records the source origin, image tags, and file checksums. For production, use the configuration package corresponding to the released image. Packages markedworking-tree-previeware for local review only. Before expanding an existing deployment, readstorage-expansion.mdin the package. -
Confirm the images in
30-stateful.yaml,40-stateless.yaml, and50-daemonsets.yamlmatch the image recorded in the package'sSOURCE.json, and prepare it on every node:grep -n 'image:' 30-stateful.yaml 40-stateless.yaml 50-daemonsets.yaml- For upgrades, obtain the configuration package for the target version. Do not only replace image tags in old configuration files.
- For an internal image registry, the registry address can change, but image content and version must remain unchanged.
-
Check PVC capacities in
30-stateful.yamlagainst the storage plan. For object storage, also check the Loki and Tempo bucket names. -
Copy the Secret example:
cp 10-secret.yaml.example 10-secret.yamlThe package uses
hap-opsby default. To change it, update namespaces, ServiceAccount bindings, Service addresses, andENV_K8S_KSM_NAMESPACEconsistently in all YAML files, including the newly created10-secret.yaml. Also replace-n hap-opsin subsequent commands with the actual namespace.After checking the configuration, create the namespace:
kubectl apply -f 00-namespace.yaml -
Edit
10-secret.yamland replace the example values of the following environment variables:vim 10-secret.yamlEnvironment Variable Description ENV_OPS_TOKENOps Platform login token. Use a strong password ENV_ALERT_CRYPTO_KEYCredential encryption key. Generate it separately and keep it safe. Changing it later makes existing encrypted credentials impossible to decrypt Run the following command twice to generate separate random values for the login token and credential encryption key, then enter them in the corresponding environment variables:
openssl rand -hex 16For object storage, also fill in
ENV_S3_ENDPOINT,ENV_S3_ACCESS_KEY,ENV_S3_SECRET_KEY, andENV_S3_FORCE_PATH_STYLE. -
Check the other environment variables in
11-configmap.yaml; see Environment Variables. The default access sub-path is/mdis. If changed, also update the gateway readiness probe path in40-stateless.yamland the reverse proxy configuration. -
Apply the Secret:
kubectl apply -f 10-secret.yaml
Deploy and Verify
Run in the k8s-cluster directory:
kubectl apply -f 11-configmap.yaml -f 20-rbac.yaml -f 30-stateful.yaml -f 40-stateless.yaml -f 50-daemonsets.yaml
Check Pod, Service, and PVC status:
kubectl get pods,svc,pvc -n hap-ops -o wide
Confirm all three PVCs are Bound and the related Pods are ready. With a StorageClass using delayed binding, PVCs may temporarily show Pending before Pod scheduling. If they remain unbound, check PVC events and the storage driver:
kubectl describe pvc <PVC名称> -n hap-ops
After verification, log in by following Access and Verify above. PVC exposes port 30881 through the ops-gateway-ui Service; the internal API uses the ops-gateway ClusterIP Service.
Appendix A: Multiple Nodes and Dedicated Monitoring Nodes
Deploying the main Ops Platform services on dedicated nodes requires both node selection and taint tolerations:
- Node selectors specify eligible nodes. hostPath already uses the
hap-ops=truelabel. PVC can use the same selector for main services as needed; confirm storage volumes can mount on the selected node. - Taints prevent Pods without matching tolerations from scheduling on the node. Adding a toleration alone does not pin services to it.
First label the target node:
kubectl label node <节点名称> hap-ops=true
Set the following under spec.template.spec in the main services' Deployments or StatefulSets:
nodeSelector:
hap-ops: "true"
To restrict other Pods from scheduling on the node, add a taint and configure the corresponding toleration:
kubectl taint nodes <节点名称> hap-ops=true:NoSchedule
Add the following under spec.template.spec for components that need to run on the node:
tolerations:
- key: hap-ops
operator: Equal
value: "true"
effect: NoSchedule
This toleration matches only hap-ops=true:NoSchedule. Handle any other taints separately. hostPath data must still reside on the single node carrying the label.
ops-nodeagent and alloy-logs collect data from all nodes; do not apply a nodeSelector that limits them to dedicated monitoring nodes. The hostPath configuration does not preset tolerations for ops-nodeagent; add matching tolerations to cover tainted nodes.
One ops-agent can collect multiple middleware instances added through the Data Sources page. ops-nodeagent runs as a DaemonSet to collect cluster node metrics. To monitor servers outside the cluster, deploy Node Exporter on each target server.
Appendix B: Cluster Deployment Configuration Notes
ops-agentuses a headless Service (clusterIP: None). The agent dynamically allocates exporter ports for data sources; Prometheus accesses them through Pod IPs.- The PVC deployment's
alloy-logsDaemonSet collects container logs through the Kubernetes API. In hostPath deployments, add this component according to Container Log Ingestion. - Stateful components use StatefulSets in PVC deployments. hostPath uses Deployments with the
Recreatestrategy for components using local data directories, preventing old and new Pods from using the same directory simultaneously.