DB-MongoDB-00002--Workload Generator for MongoDB

DB-MongoDB-00002–Workload Generator for MongoDB

1、介绍

Workload Generator for MongoDB was designed to help MongoDB users effortlessly generate data and simulate workloads for both sharded and non-sharded clusters. The generated workloads include standard CRUD operations, reflecting real-world usage patterns of a MongoDB environment.

Additionally, the tool supports the creation of realistic workloads by incorporating all CRUD operations through a set of queries that simulate common usage scenarios. Users can also define custom queries to run against collections created by the tool, further enhancing its flexibility and applicability.

While the tool provides extensive configuration capabilities, it requires minimal setup — only basic connection details are needed. It is user-friendly, does not require compilation, and offers a high level of flexibility.

The application was developed and tested using Python 3. As such, Python 3 is recommended for optimal compatibility. If upgrading is not feasible, modifications to the scripts may be necessary to support older Python versions.

The tool is optimized to leverage as many available CPU cores as you wish and supports configuration of an arbitrary number of threads among, enabling high parallelism and making it ideal for generating large-scale workloads and conducting effective stress tests on MongoDB clusters.

一句话：另一个YCSB

2、安装

[root@mongodb01 ~]# python --version
Python 3.9.18
[root@mongodb01 ~]# pip --version
-bash: pip: command not found
[root@mongodb01 ~]# pip3 --version
-bash: pip3: command not found
Error: Unable to find a match: install
[root@mongodb01 ~]# dnf install -y python3-pip

[root@mongodb01 ~]# pip3 install faker joblib pymongo
Collecting faker
  Downloading faker-37.3.0-py3-none-any.whl (1.9 MB)
     |████████████████████████████████| 1.9 MB 382 kB/s
Collecting joblib
  Downloading joblib-1.5.0-py3-none-any.whl (307 kB)
     |████████████████████████████████| 307 kB 10.2 MB/s
Collecting pymongo
  Downloading pymongo-4.13.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (937 kB)
     |████████████████████████████████| 937 kB 564 kB/s
Collecting tzdata
  Downloading tzdata-2025.2-py2.py3-none-any.whl (347 kB)
     |████████████████████████████████| 347 kB 11.2 MB/s
Collecting dnspython<3.0.0,>=1.16.0
  Downloading dnspython-2.7.0-py3-none-any.whl (313 kB)
     |████████████████████████████████| 313 kB 1.9 MB/s
Installing collected packages: tzdata, dnspython, pymongo, joblib, faker
Successfully installed dnspython-2.7.0 faker-37.3.0 joblib-1.5.0 pymongo-4.13.0 tzdata-2025.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead:                     https://pip.pypa.io/warnings/venv

[root@mongodb01 ~]# wget https://github.com/Percona-Lab/mongoDB-workload-generator/archive/refs/tags/0.1.zip
--2025-05-18 21:10:06--  https://github.com/Percona-Lab/mongoDB-workload-generator/archive/refs/tags/0.1.zip
Connecting to 192.168.5.18:7890... connected.
Proxy request sent, awaiting response... 302 Found
Location: https://codeload.github.com/Percona-Lab/mongoDB-workload-generator/zip/refs/tags/0.1 [following]
--2025-05-18 21:10:07--  https://codeload.github.com/Percona-Lab/mongoDB-workload-generator/zip/refs/tags/0.1
Connecting to 192.168.5.18:7890... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘0.1.zip’

0.1.zip                                            [ <=>                                                                                                ]  23.48K  --.-KB/s    in 0.06s

2025-05-18 21:10:09 (420 KB/s) - ‘0.1.zip’ saved [24040]

[root@mongodb01 ~]# mv 0.1.zip mongoDB-workload-generator-0.1.zip
[root@mongodb01 ~]# unzip mongoDB-workload-generator-0.1.zip
[root@mongodb01 ~]# cd mongoDB-workload-generator-0.1/

3、配置

基本配置就需要配置mongodbCreds.py这个文件就可以了

[root@mongodb01 mongoDB-workload-generator-0.1]# cat mongodbCreds.py
# Configuration for MongoDB connection
dbconfig = {
    "username": "login_here",
    "password": "password_here",
    "port": 27017,
    "hosts": [
        "mongos00",
        "mongos01"
    ],
    "serverSelectionTimeoutMS": 5000, # We need this to fail faster, otherwise the default is 30 seconds
    "connectTimeoutMS": 3000,  # Example timeout setting
    "maxPoolSize": 500, # Example pool setting
    # Leave replicaSet: None if connecting to mongos. Enter the appropriate replicaSet name if connecting to replicaSet instead of Mongos
    "replicaSet": None,
    "authSource": "admin",  # Adjust for authentication
    "tls": "false",  # Example tls setting
}[root@mongodb01 mongoDB-workload-generator-0.1]#

3.1、常见错误

如果连接信息配置错误或者在MongoDB没有开启认证的时候，就会出现这个错误

CRITICAL:root:Unable to connect to MongoDB, please make sure your configuration is correct.
Error: mongos01:27017: [Errno -2] Name or service not known (configured timeouts: socketTimeoutMS: 3000.0ms, connectTimeoutMS: 3000.0ms),mongos00:27017: [Errno -2] Name or service not known (configured timeouts: socketTimeoutMS: 3000.0ms, connectTimeoutMS: 3000.0ms), Timeout: 5.0s, Topology Description: <TopologyDescription id: 6829e152d0087848897bc0a8, topology_type: Unknown, servers: [<ServerDescription ('mongos00', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('mongos00:27017: [Errno -2] Name or service not known (configured timeouts: socketTimeoutMS: 3000.0ms, connectTimeoutMS: 3000.0ms)')>, <ServerDescription ('mongos01', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('mongos01:27017: [Errno -2] Name or service not known (configured timeouts: socketTimeoutMS: 3000.0ms, connectTimeoutMS: 3000.0ms)')>]>

3.2、配置无认证模式

如果MongoDB没有开启认证，如何连接数据库呢

3.2.1、配置mongodbCreds.py文件

旧文件

dbconfig = {
    "username": "login_here",
    "password": "password_here",
    "port": 27017,
    "hosts": [
        "mongos00",
        "mongos01"
    ],

新文件，把username/password删了，并且修改hosts

[root@mongodb01 mongoDB-workload-generator-0.1]# cat mongodbCreds.py
# Configuration for MongoDB connection
dbconfig = {
    "port": 27017,
    "hosts": [
        "mongodb01"
    ],
    "serverSelectionTimeoutMS": 5000, # We need this to fail faster, otherwise the default is 30 seconds
    "connectTimeoutMS": 3000,  # Example timeout setting
    "maxPoolSize": 500, # Example pool setting
    # Leave replicaSet: None if connecting to mongos. Enter the appropriate replicaSet name if connecting to replicaSet instead of Mongos
    "replicaSet": None,
    "authSource": "admin",  # Adjust for authentication
    "tls": "false",  # Example tls setting
}

3.2.2、配置app.py文件

旧连接串

[root@mongodb01 mongoDB-workload-generator-0.1]# more app.py
# Start building the base connection URI
connection_uri = f"mongodb://{dbconfig['username']}:{dbconfig['password']}@{hosts}"

新连接串，这个地方还可以加localhost

[root@mongodb01 mongoDB-workload-generator-0.1]# more app.py
# Start building the base connection URI
#connection_uri = f"mongodb://{dbconfig['username']}:{dbconfig['password']}@{hosts}"
connection_uri = f"mongodb://{hosts}"

3.3、负载测试

[root@mongodb01 mongoDB-workload-generator-0.1]# ./mongodbWorkload.py
2025-05-18 21:38:28 - INFO - Configuring Workload
2025-05-18 21:38:28 - INFO -

Duration: 60 seconds
CPUs: 1
Threads: (Per CPU: 4 | Total: 4)
Collections: 1
Configure Sharding: False
Insert batch size: 10
Optimized workload: False
Workload ratio: SELECTS: 60% | INSERTS: 10% | UPDATES: 20% | DELETES: 10%
Report frequency: 5 seconds
Report logfile: None

===================================================================================================================
                                                 Workload Started
===================================================================================================================

2025-05-18 21:38:38 - INFO - AVG Operations last 5s (1 CPUs): 264.20 (SELECTS: 154.20, INSERTS: 31.00, UPDATES: 50.40, DELETES: 28.60)
2025-05-18 21:38:43 - INFO - AVG Operations last 5s (1 CPUs): 303.40 (SELECTS: 187.40, INSERTS: 30.80, UPDATES: 54.80, DELETES: 30.40)
2025-05-18 21:38:48 - INFO - AVG Operations last 5s (1 CPUs): 243.00 (SELECTS: 138.00, INSERTS: 30.00, UPDATES: 53.00, DELETES: 22.00)
2025-05-18 21:38:53 - INFO - AVG Operations last 5s (1 CPUs): 326.00 (SELECTS: 201.40, INSERTS: 29.40, UPDATES: 60.40, DELETES: 34.80)
2025-05-18 21:38:58 - INFO - AVG Operations last 5s (1 CPUs): 271.20 (SELECTS: 162.60, INSERTS: 30.00, UPDATES: 53.20, DELETES: 25.40)
2025-05-18 21:39:03 - INFO - AVG Operations last 5s (1 CPUs): 322.80 (SELECTS: 191.80, INSERTS: 29.40, UPDATES: 67.60, DELETES: 34.00)
2025-05-18 21:39:08 - INFO - AVG Operations last 5s (1 CPUs): 276.20 (SELECTS: 167.60, INSERTS: 28.80, UPDATES: 49.20, DELETES: 30.60)
2025-05-18 21:39:13 - INFO - AVG Operations last 5s (1 CPUs): 260.40 (SELECTS: 149.20, INSERTS: 30.60, UPDATES: 51.20, DELETES: 29.40)
2025-05-18 21:39:18 - INFO - AVG Operations last 5s (1 CPUs): 290.20 (SELECTS: 172.20, INSERTS: 28.40, UPDATES: 59.60, DELETES: 30.00)
2025-05-18 21:39:23 - INFO - AVG Operations last 5s (1 CPUs): 282.60 (SELECTS: 168.00, INSERTS: 29.20, UPDATES: 54.60, DELETES: 30.80)
2025-05-18 21:39:28 - INFO - AVG Operations last 5s (1 CPUs): 245.80 (SELECTS: 145.00, INSERTS: 29.00, UPDATES: 48.40, DELETES: 23.40)
2025-05-18 21:39:33 - INFO - AVG Operations last 5s (1 CPUs): 266.60 (SELECTS: 158.20, INSERTS: 26.40, UPDATES: 54.40, DELETES: 27.60)
2025-05-18 21:39:38 - INFO - AVG Operations last 5s (1 CPUs): 266.60 (SELECTS: 158.20, INSERTS: 26.40, UPDATES: 54.40, DELETES: 27.60)
2025-05-18 21:39:38 - INFO -
===================================================================================================================
                                                Workload Finished
===================================================================================================================

2025-05-18 21:39:38 - INFO -
================================================================================
|                               Collection Stats                               |
================================================================================
|            Name         |     Sharded      |      Size      |    Documents   |
================================================================================
|         flights_1       |      False       |   115.40 MB    |      47407     |
================================================================================

2025-05-18 21:39:38 - INFO -
===================================================================================================================
                                        Workload Stats (All CPUs Combined)
===================================================================================================================
Workload Runtime: 1.17 minutes
CPUs Used: 1
Total Operations: 16762 (SELECT: 9978, INSERT: 1765, UPDATE: 3284, DELETE: 1735)
AVG QPS: 238.79 (SELECTS: 142.15, INSERTS: 25.14, UPDATES: 46.78, DELETES: 24.72)
Documents Inserted: 17650, Matching Documents Selected: 9865, Documents Updated: 2995, Documents Deleted: 1573
===================================================================================================================