Skip to main content

Kubernetes

This page describes a new Ops Platform deployment inside the HAP/HDP Kubernetes cluster. After deployment, the platform can collect metrics for that cluster. To monitor other clusters, deploy the collection components in each target cluster.

Prerequisites​

  • Ensure the HAP/HDP Kubernetes cluster runs normally and reserve the computing and storage resources required by the Ops Platform.
  • Ensure kubectl can access the cluster with permissions to create namespaces, deploy components, and configure RBAC resources.
  • Prepare accounts for monitored middleware. For permissions, see Data Sources.
  • For direct access, clients must reach port 30881 on cluster nodes. When using a reverse proxy, the proxy server must reach that port.

Run the following kubectl commands on a server with kubectl configured and the required cluster permissions, such as a Kubernetes control node. Check node status:

kubectl get nodes

This guide uses the hap-ops namespace by default and provides two deployment methods:

Storage MethodData LocationProcedure
hostPath (default)Local directory on a specified nodeFollow steps 1–5
StorageClass and PVCStorage volumes provided by StorageClassComplete step 1, then go to Deploy with StorageClass and PVC

These methods use different configuration files. Do not deploy both in the same namespace. The capacities and directories below are examples for new deployments. Existing deployments should keep their current storage configuration and plan migration or expansion separately.

1. Obtain the Image​

The image example below is for Linux AMD64 nodes. Before pulling it, ensure crictl on every node connects to the container runtime used by Kubernetes.

The Ops Platform deploys node collection components through DaemonSets:

  • ops-nodeagent: Collects node metrics.
  • alloy-logs: Collects container logs. Included in the PVC deployment; deploy separately as needed for hostPath.

Prepare the image on every node for the collection components and other Ops Platform services. Actual node coverage also depends on scheduling conditions such as node taints.

In online environments, run on every node:

crictl pull registry.cn-hangzhou.aliyuncs.com/mdpublic/ops-allinone:1.5.12

In offline environments, download the offline image package matching the node architecture, upload it to every node, and import it on each node. The image address and version in the deployment configuration must match the imported image.

2. Select the Deployment Node​

The Ops Platform uses hostPath by default, storing monitoring data, logs, traces, and MongoDB data in /data/mdis/ on the node.

Choose a node that can run the Ops Platform services, ensure the file system containing /data/mdis/ has at least 200 GB of free space, and label the node. The main Ops Platform services are scheduled there by label to access local data.

Find the node name with kubectl get nodes, then run:

kubectl label node <节点名称> hap-ops=true

Ensure only the selected node has this label, so services are not scheduled to another node where the existing data is unavailable.

hostPath data remains on the local node and does not move with Pods. If the node may be rebuilt or replaced, consider storage volumes that support mounting across nodes. PVCs backed by local storage also depend on their node.

Loki logs and Tempo traces are stored locally on the node by default. To use object storage, see Object Storage Configuration.

3. Prepare the Deployment Configuration File​

On a Kubernetes control node or another server with kubectl configured and cluster permissions, create /data/mingdao/kubernetes/mdis/ for deployment configuration files:

mkdir -p /data/mingdao/kubernetes/mdis

This directory stores YAML configuration files. Service data is stored under /data/mdis/ on the node selected in step 2.

  1. Download the Ops Platform deployment configuration and upload it to /data/mingdao/kubernetes/mdis/ on that server.

  2. Edit /data/mingdao/kubernetes/mdis/ops.yaml and replace the example values of the following environment variables:

    Environment VariableDescription
    ENV_OPS_TOKENOps Platform login token. Use a strong password
    ENV_ALERT_CRYPTO_KEYCredential encryption key. Generate it separately and keep it safe. Changing it later makes existing encrypted credentials impossible to decrypt
  3. Check the following settings:

    • The image address and version match the image used in step 1.
    • The node label matches hap-ops=true configured in step 2.

    If the target node has a NoSchedule taint, also configure the corresponding toleration. See Dedicated Monitoring Nodes.

The configuration includes the hap-ops namespace, permissions for cluster monitoring, and kube-state-metrics.

To use another namespace, modify ops.yaml before deployment:

  • Replace the Namespace name, each resource's metadata.namespace, and the ServiceAccount namespace in ClusterRoleBinding with the actual namespace.
  • Update the namespace in Service addresses in ConfigMap and ENV_K8S_KSM_NAMESPACE. If the cluster DNS suffix is not cluster.local, also adjust Service addresses accordingly.
  • Replace -n hap-ops in the commands below with the actual namespace. The hap-ops=true node label selects the deployment node and does not need to change with the namespace.

The gateway Service contains both the access port and the internal API port. Kubernetes assigns a NodePort to each; restrict network access to these ports.

4. Deploy the Ops Platform​

Deploy the Ops Platform:

kubectl apply -f /data/mingdao/kubernetes/mdis/ops.yaml

5. Access and Verify​

Check the Ops Platform Pod and Service status:

kubectl get pods,svc -n hap-ops -o wide

Once the Ops Platform Pods are ready, open the following address in a browser:

http://节点IP:30881/mdis/web/

Log in with the configured ENV_OPS_TOKEN and confirm the overview page displays correctly. If ENV_OPS_SUB_PATH was changed, replace /mdis in the access URL with the actual path.

If a Pod is not ready, check the cause of scheduling, image pull, or mount failures:

kubectl describe pod <Pod名称> -n hap-ops

See FAQs for other troubleshooting methods.

Configure a Reverse Proxy​

hostPath exposes port 30881 through the ops-gateway Service. PVC uses ops-gateway-ui with the same external port.

To access through a domain name, add upstream nodes in the Nginx http block:

upstream hap-ops {
server 192.168.1.11:30881;
server 192.168.1.12:30881;
server 192.168.1.13:30881;
}

Add the following to the domain's server block:

location /mdis/ {
# allow 192.168.1.100;
# deny all;
proxy_pass http://hap-ops;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Connection "";
}

Adjust for your environment:

  • Replace the example upstream IPs with cluster node IPs reachable from the proxy server. Add or remove entries for the actual node count.
  • To restrict source addresses, enable both allow and deny and enter the actual operations egress IP.
  • When changing the access sub-path, update both ENV_OPS_SUB_PATH in the deployment configuration and the proxy path.

After checking and reloading the Nginx configuration, access through the domain name. If HTTPS listening and certificates are configured, use https://域名/mdis/web/; otherwise use the actual HTTP protocol.

Next Steps​

  • Add monitored hosts, middleware, or custom pages in Data Sources.
  • Configure the log push address on HAP/HDP services to ingest Service Logs.
  • To collect container logs, deploy alloy-logs separately for hostPath; see Container Log Ingestion. The PVC deployment already includes this component.
  • Configure Tracing for the cluster environment.
  • To monitor servers outside the cluster, deploy Node Exporter on each target server.
  • To monitor other Kubernetes clusters, deploy the collection components in each target cluster.

Deploy with StorageClass and PVC​

This method creates PVCs through StorageClass. After completing the prerequisites and obtaining images in step 1, follow this section. Skip hostPath steps 2–5.

Prepare PVC Storage​

Check available StorageClasses:

kubectl get storageclass

The default StorageClass is marked (default) in the output. Choose according to the cluster's existing configuration:

  • Use the default storage class: The example PVCs do not specify storageClassName, so no additional changes are needed.
  • Use another existing storage class: Set storageClassName under spec in all three volumeClaimTemplates in 30-stateful.yaml. The cluster default does not need to change.

To create a StorageClass, refer to these examples:

StorageClass Configuration Reference

StorageClass definitions and parameters vary by storage service. Follow the service's documentation. These examples are for reference; check the storage type and reclaim policy before use.

The following example uses Tencent Cloud CBS. Ensure the corresponding CSI driver is installed. With reclaimPolicy: Delete, deleting a PVC may delete the associated persistent volume and backend storage. Choose the policy according to data retention requirements:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cbs-expandable
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: com.tencent.cloud.csi.cbs
parameters:
diskType: CLOUD_PREMIUM
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

Save the configuration as storageclass.yaml, then run:

kubectl apply -f storageclass.yaml

Changing the default StorageClass affects future PVCs that do not specify a storage class. If replacing the default is necessary, remove the previous default marker after configuring the new default:

kubectl patch sc <原默认SC名> -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Test environments can also use local-path-provisioner to store data locally on the node:

kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
kubectl patch sc local-path -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

For local storage, adjust PVC capacity according to actual free disk space. local-path-provisioner stores data locally on the node; it does not move with Pods to another node.

Once the StorageClass is available, plan storage using these default capacities:

ComponentDefault Capacity
Prometheus200 GiB
MongoDB30 GiB
Grafana20 GiB

The total is 250 GiB, adjustable in 30-stateful.yaml before the initial deployment. Existing PVCs do not expand automatically when example capacities change, and editing a StatefulSet's volumeClaimTemplates does not expand them directly. Handle expansion separately according to storage driver capabilities.

The PVC package sets Prometheus retention to 30d and the size limit to 80GB, with cleanup triggered by whichever limit is reached first. These are controlled by ENV_PROMETHEUS_RETENTION and ENV_PROMETHEUS_RETENTION_SIZE in 11-configmap.yaml. Increasing PVC capacity does not automatically increase the retention limits.

Loki and Tempo use emptyDir by default in the PVC deployment. Without object storage, logs and traces are stored in temporary directories and lost when the Pod is deleted. To retain them, configure object storage for both components or modify the deployment to use persistent volumes.

For object storage, create separate buckets for Loki and Tempo with different names:

ConfigurationLocation
Object storage connection information10-secret.yaml
Loki bucketENV_S3_BUCKET_LOKI in 30-stateful.yaml
Tempo bucketENV_S3_BUCKET_TEMPO in 30-stateful.yaml

See Object Storage Configuration for parameters.

Prepare PVC Deployment Configuration Files​

  1. Download the Ops Platform PVC deployment package, upload it to a server with kubectl configured and cluster permissions, and extract it in the upload directory:

    tar -xzf k8s-cluster.tar.gz
    cd k8s-cluster

    SOURCE.json in the package records the source origin, image tags, and file checksums. For production, use the configuration package corresponding to the released image. Packages marked working-tree-preview are for local review only. Before expanding an existing deployment, read storage-expansion.md in the package.

  2. Confirm the images in 30-stateful.yaml, 40-stateless.yaml, and 50-daemonsets.yaml match the image recorded in the package's SOURCE.json, and prepare it on every node:

    grep -n 'image:' 30-stateful.yaml 40-stateless.yaml 50-daemonsets.yaml
    • For upgrades, obtain the configuration package for the target version. Do not only replace image tags in old configuration files.
    • For an internal image registry, the registry address can change, but image content and version must remain unchanged.
  3. Check PVC capacities in 30-stateful.yaml against the storage plan. For object storage, also check the Loki and Tempo bucket names.

  4. Copy the Secret example:

    cp 10-secret.yaml.example 10-secret.yaml

    The package uses hap-ops by default. To change it, update namespaces, ServiceAccount bindings, Service addresses, and ENV_K8S_KSM_NAMESPACE consistently in all YAML files, including the newly created 10-secret.yaml. Also replace -n hap-ops in subsequent commands with the actual namespace.

    After checking the configuration, create the namespace:

    kubectl apply -f 00-namespace.yaml
  5. Edit 10-secret.yaml and replace the example values of the following environment variables:

    vim 10-secret.yaml
    Environment VariableDescription
    ENV_OPS_TOKENOps Platform login token. Use a strong password
    ENV_ALERT_CRYPTO_KEYCredential encryption key. Generate it separately and keep it safe. Changing it later makes existing encrypted credentials impossible to decrypt

    Run the following command twice to generate separate random values for the login token and credential encryption key, then enter them in the corresponding environment variables:

    openssl rand -hex 16

    For object storage, also fill in ENV_S3_ENDPOINT, ENV_S3_ACCESS_KEY, ENV_S3_SECRET_KEY, and ENV_S3_FORCE_PATH_STYLE.

  6. Check the other environment variables in 11-configmap.yaml; see Environment Variables. The default access sub-path is /mdis. If changed, also update the gateway readiness probe path in 40-stateless.yaml and the reverse proxy configuration.

  7. Apply the Secret:

    kubectl apply -f 10-secret.yaml

Deploy and Verify​

Run in the k8s-cluster directory:

kubectl apply -f 11-configmap.yaml -f 20-rbac.yaml -f 30-stateful.yaml -f 40-stateless.yaml -f 50-daemonsets.yaml

Check Pod, Service, and PVC status:

kubectl get pods,svc,pvc -n hap-ops -o wide

Confirm all three PVCs are Bound and the related Pods are ready. With a StorageClass using delayed binding, PVCs may temporarily show Pending before Pod scheduling. If they remain unbound, check PVC events and the storage driver:

kubectl describe pvc <PVC名称> -n hap-ops

After verification, log in by following Access and Verify above. PVC exposes port 30881 through the ops-gateway-ui Service; the internal API uses the ops-gateway ClusterIP Service.

Appendix A: Multiple Nodes and Dedicated Monitoring Nodes​

Deploying the main Ops Platform services on dedicated nodes requires both node selection and taint tolerations:

  • Node selectors specify eligible nodes. hostPath already uses the hap-ops=true label. PVC can use the same selector for main services as needed; confirm storage volumes can mount on the selected node.
  • Taints prevent Pods without matching tolerations from scheduling on the node. Adding a toleration alone does not pin services to it.

First label the target node:

kubectl label node <节点名称> hap-ops=true

Set the following under spec.template.spec in the main services' Deployments or StatefulSets:

nodeSelector:
hap-ops: "true"

To restrict other Pods from scheduling on the node, add a taint and configure the corresponding toleration:

kubectl taint nodes <节点名称> hap-ops=true:NoSchedule

Add the following under spec.template.spec for components that need to run on the node:

tolerations:
- key: hap-ops
operator: Equal
value: "true"
effect: NoSchedule

This toleration matches only hap-ops=true:NoSchedule. Handle any other taints separately. hostPath data must still reside on the single node carrying the label.

ops-nodeagent and alloy-logs collect data from all nodes; do not apply a nodeSelector that limits them to dedicated monitoring nodes. The hostPath configuration does not preset tolerations for ops-nodeagent; add matching tolerations to cover tainted nodes.

One ops-agent can collect multiple middleware instances added through the Data Sources page. ops-nodeagent runs as a DaemonSet to collect cluster node metrics. To monitor servers outside the cluster, deploy Node Exporter on each target server.

Appendix B: Cluster Deployment Configuration Notes​

  • ops-agent uses a headless Service (clusterIP: None). The agent dynamically allocates exporter ports for data sources; Prometheus accesses them through Pod IPs.
  • The PVC deployment's alloy-logs DaemonSet collects container logs through the Kubernetes API. In hostPath deployments, add this component according to Container Log Ingestion.
  • Stateful components use StatefulSets in PVC deployments. hostPath uses Deployments with the Recreate strategy for components using local data directories, preventing old and new Pods from using the same directory simultaneously.