Skip to main content

Sharded Cluster Storage for Log Data

In replica set mode, each node needs to store a complete copy of the data. As the system continues to run, log data keeps accumulating. The following data types in particular can easily grow to the TB level:

  • Worksheet record data change logs
  • Workflow execution history logs

When the log data scale continues to grow, relying only on a replica set brings higher storage costs and query pressure. In addition to regularly cleaning historical data and archiving data, MongoDB sharded clusters can also be used to horizontally split log data.

Applicable Scenarios​

Sharded cluster storage for log data applies to the following scenarios:

  • Log data grows quickly, and a single MongoDB replica set has high storage pressure
  • Worksheet change logs or workflow history logs reach the hundreds of GB or TB level
  • Query response time becomes significantly longer because a single table has too much data
  • Log data storage capacity needs to be gradually expanded by adding nodes

If the current log data scale is small, or existing cleanup and archiving policies already meet capacity requirements, prioritize cleanup and archiving to avoid introducing sharded cluster O&M complexity too early. Sharded clusters increase deployment, monitoring, backup, and failure handling complexity. Before enabling them, evaluate the log data scale, growth trend, query patterns, and O&M capability, and verify the shard key, indexes, and typical query performance in a test environment.

Architecture​

Sharded cluster storage for log data uses MongoDB Sharding to split large log collections across multiple shards. Each shard stores only part of the log data, and the system forwards read and write requests to the corresponding shard through routing nodes.

The typical architecture includes the following components:

  • Config Server: stores sharded cluster metadata and routing information
  • Mongos: acts as the query routing entry and distributes requests to target shards
  • Shard: stores the actual log data. Each shard usually uses a replica set to ensure availability

In this architecture, log data is no longer stored centrally in a single replica set. As data grows, Shards can be added to expand overall storage capacity and I/O capability.

Log data sharding cluster storage architecture

Main Advantages​

Horizontal Scaling​

Storage capacity for log data can be expanded by increasing the number of shards, avoiding continuous vertical scaling of a single MongoDB node. For long-term accumulated log data, sharded clusters can keep the data scale of each shard within a controllable range.

Query Performance​

When query conditions can hit the shard key, Mongos can route requests to the target shard and reduce unnecessary data scans. For cross-shard queries, multiple shards can also process requests in parallel, reducing the performance degradation caused by excessive data volume in a single collection.

Cost Control​

Sharded clusters support on-demand scaling. You can first plan the initial number of shards based on the current data scale, and then gradually add shards as logs grow, avoiding the need to configure excessive storage and compute resources for a single node at once.