Skip to main content

Workflow Parallelism Tuning

Workflow Processing Mechanism​

Workflows process tasks asynchronously through Kafka. After an event is triggered, the system writes a workflow message to the corresponding Kafka Topic. The workflow consumer service then reads the message and runs the workflow. During execution, the workflow may call business services such as worksheet services and read from or write to databases.

Kafka buffers tasks during traffic spikes, reducing the immediate load on business services and databases. If the message backlog continues to decrease and the tasks finish within an acceptable time, you do not need to adjust the parallelism.

The workflow consumer service uses the md-workflow-consumer consumer group to read the following Topics:

TopicPurpose
WorkFlowRun main workflows
WorkFlow-ProcessRun subprocesses
WorkFlow-RouterProcess the workflow slow queue
WorkFlow-BatchRun batch workflows
WorkFlow-ButtonRun workflows triggered by buttons
WorkSheetCheck whether worksheet rows trigger workflows
WorkSheet-BatchCheck worksheet rows in batches for workflow triggers
WorkSheet-RouterCheck worksheet rows for workflow triggers in the slow queue

A Kafka Topic contains one or more partitions. Within a consumer group, Kafka can assign a partition to only one consumer at a time. The number of partitions that workflows can consume in parallel depends on the Topic partition count, the number of threads in each consumer service instance, and the number of consumer service instances.

The partition count and the total number of consumer threads must match. If a Topic has 10 partitions and the service has 20 consumer threads, only 10 threads can receive partitions. If a Topic has 20 partitions but the service has only 5 consumer threads, it can consume only 5 partitions at a time.

In cluster mode, the total number of consumer threads equals the number of consumer threads in each instance multiplied by the number of instances. The number of threads that participate in consumption cannot exceed the Topic partition count.

Increasing the Topic partition count does not redistribute existing messages. Messages already queued in the original partitions remain there. Only messages written after the expansion can be assigned to the new partitions according to the partitioning strategy. The LAG of the original partitions therefore does not decrease immediately after the expansion.

Tuning Methods​

Before increasing workflow parallelism, confirm that the database can handle the additional concurrency. If database CPU or disk I/O usage is high, or if the database has many slow queries, adding Kafka partitions, consumer threads, and service instances increases the database load.

See MongoDB Slow Query Optimization to check slow queries, indexes, query conditions, and server resources. Resolve database bottlenecks before adjusting workflow parallelism.

Standalone Mode​

Standalone mode uses one workflow consumer process. Increase the number of Kafka Topic partitions and consumer threads to improve parallel processing capacity.

All services usually run on the same server. Check the server CPU, memory, and disk I/O before making changes. If resource usage is close to its limit, expand the server resources first.

By default, each related Topic has 10 partitions, and the workflow consumer service uses 10 consumer threads. The recommended range after tuning is 20 to 30. This example configures 20 partitions and 20 consumer threads.

  1. Enter the storage component container.

    docker exec -it $(docker ps | grep mingdaoyun-sc | awk '{print $1}') bash

  2. Check the current partition count and message backlog for each Topic.

    /usr/local/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --describe --group md-workflow-consumer

    In the command output, PARTITION identifies the partition, and LAG shows the number of queued messages in that partition.

  3. Increase the partition count based on the backlog of each Topic. This example sets the partition count to 20.

    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 20 --topic WorkFlow
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 20 --topic WorkFlow-Batch
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 20 --topic WorkFlow-Button
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 20 --topic WorkFlow-Process
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 20 --topic WorkFlow-Router
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 20 --topic WorkSheet
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 20 --topic WorkSheet-Batch
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 20 --topic WorkSheet-Router

    You can increase a Kafka Topic partition count, but you cannot decrease it. Increase the count in stages based on the backlog.

  4. Set the workflow consumer thread environment variables in docker-compose.yaml.

    ENV_WORKFLOW_CONSUMER_THREADS: '20'
    ENV_WORKFLOW_ROUTER_CONSUMER_THREADS: '20'

    ENV_WORKFLOW_CONSUMER_THREADS controls the consumer threads for regular workflow queues. ENV_WORKFLOW_ROUTER_CONSUMER_THREADS controls the consumer threads for the WorkFlow-Router slow queue. The consumer thread count usually matches the corresponding Topic partition count.

  5. Exit the storage component container, return to the manager installation directory, and restart the HAP services to apply the configuration.

    bash ./service.sh restartall

Cluster Mode​

In cluster mode, HAP microservices run on Kubernetes, and Kafka is deployed separately. Increase the Topic partition count and the number of consumer service replicas to improve parallel consumption capacity.

Before making changes, check the CPU, memory, and disk I/O of the cluster nodes, along with the CPU and memory usage of the Kubernetes Pods. If resource usage is close to its limit, resolve the resource bottleneck before increasing parallelism.

Each workflowconsumer and workflowrouterconsumer Pod uses 10 consumer threads by default. The recommended Topic partition range is 30 to 120. This example sets the partition count to 40 and configures four Pods for each consumer service.

  1. Log in to the Kafka server and check the current partition count and message backlog for each Topic.

    /usr/local/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --describe --group md-workflow-consumer

    In the command output, PARTITION identifies the partition, and LAG shows the number of queued messages in that partition.

  2. Increase the partition count based on the backlog of each Topic. This example sets the partition count to 40.

    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 40 --topic WorkFlow
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 40 --topic WorkFlow-Batch
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 40 --topic WorkFlow-Button
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 40 --topic WorkFlow-Process
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 40 --topic WorkFlow-Router
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 40 --topic WorkSheet
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 40 --topic WorkSheet-Batch
    /usr/local/kafka/bin/kafka-topics.sh --alter --bootstrap-server 127.0.0.1:9092 --partitions 40 --topic WorkSheet-Router

    You can increase a Kafka Topic partition count, but you cannot decrease it. Increase the count in stages based on the backlog.

  3. Scale the workflow consumer services to four Pods.

    kubectl scale deployment workflowconsumer --replicas=4
    kubectl scale deployment workflowrouterconsumer --replicas=4

    Update service.yaml as well, setting the replica count of workflowconsumer and workflowrouterconsumer to 4. This prevents the HAP services from reverting to the previous replica count after a restart.

    Example:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: workflowconsumer # <-- Service name
      namespace: default
    spec:
      replicas: 4 # <-- Number of replicas

    workflowconsumer consumes regular workflow queues, while workflowrouterconsumer consumes the WorkFlow-Router slow queue. If you change the consumer thread count of each Pod, recalculate the replica count based on the Topic partition count.

  4. After increasing workflow consumption concurrency, use Kubernetes cluster monitoring to check the Pod CPU usage of the microservices called by workflows. Increase the replica count of services with sustained high CPU usage. This example scales worksheetonlyworkflow, worksheetonlyworkflowr, and command to four Pods.

    kubectl scale deployment worksheetonlyworkflow --replicas=4
    kubectl scale deployment worksheetonlyworkflowr --replicas=4
    kubectl scale deployment command --replicas=4

    Update service.yaml as well, setting the replica count of these services to 4. This prevents the HAP services from reverting to the previous replica count after a restart.