[Bug]: bench_doc Deleting 10000 documents (in batches of 500) get stuck

[xiexiaoy]

$ ./bench_doc -eloqdoc-only
=== Database Benchmarking Tool ===
MongoDB URI: mongodb://127.0.0.1:17017
EloqDoc URI: mongodb://127.0.0.1:27017
Benchmark Configurations:
  1. Volume: 10000 documents, Sizes: [858993] bytes
Single Insert Count: 100
Batch Size: 100
Number of Batches: 100
Batch Insert Count: 10000
Query Iterations: 100000
Warmup Queries: 0
Loader Threads: 3
Output Format: table
Mode: EloqDoc Only


=== Testing with 10000 documents of size ~858993 bytes ===
Generating documents to disk...

  Target size: 858993 bytes, Actual size: 859680 bytes (diff: +687 bytes)
  Generating documents: 10000/10000 (100.0%)
  Generating documents: 10000/10000 (100.0%)
  ✓ Generated 10000 EloqDoc documents to benchmark_data/EloqDoc_10000_858993
Pre-generating query parameters...

Benchmarking EloqDoc...

  Setting up collection with 10000 documents...
  Loading and inserting documents from disk...
  Inserting documents: 10000/10000 (100.0%)
   Verified 4738 documents in collection

  Loading documents for insert benchmarks...
  ✓ Loaded 100 documents for single insert benchmark
  ✓ Loaded 10000 documents for batch insert benchmark

  === Single Insert Benchmark ===
  Deleting 100 documents (in batches of 500)...
  ✓ Deleted 100 documents
  ✓ Verified deletion completed
  Single insert: 100/100 (100.0%)
  Single insert: 100/100 (100.0%)
  ✓ Single Insert: p50=9.60ms, p99=14.32ms, mean=9.92ms, ops/sec=100.80 (100/100 successful)

  === Batch Insert Benchmark ===
  Deleting 10000 documents (in batches of 500)...
  ✓ Deleted 2500 documents
2025/12/16 17:51:49     Retry attempt 1/4 after 10s delay

--
I20251216 17:50:43.158427 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190743360, allocated 3189743872, frag ratio 0.031 , heap full: 1
I20251216 17:50:43.158474 339369 local_cc_shards.h:486] Table range memory report: allocated 1376576, committed 1575120, full: 0
I20251216 17:50:43.471378 339336 cc_shard.cpp:423] Abort cc request due to memory full. txn: 0, cc request type: N9txservice16FillStoreSliceCcE
E20251216 17:50:43.471426 339336 tx_execution.cpp:2966] ScanNextOperation failed for cc error: OUT_OF_MEMORY, txn: 9073, table: benchmark_db_11.benchmark_EloqDoc_10000_858993
I20251216 17:50:43.471684 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190743360, allocated 3189727312, frag ratio 0.032 , heap full: 1
I20251216 17:50:43.471714 339369 local_cc_shards.h:486] Table range memory report: allocated 1376592, committed 1575120, full: 0
I20251216 17:50:43.473019 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190743360, allocated 3189653568, frag ratio 0.034 , heap full: 1
--
I20251216 17:50:47.158561 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190743360, allocated 3189653568, frag ratio 0.034 , heap full: 1
I20251216 17:50:47.158605 339369 local_cc_shards.h:486] Table range memory report: allocated 1376624, committed 1575120, full: 0
I20251216 17:50:47.481410 339336 cc_shard.cpp:423] Abort cc request due to memory full. txn: 0, cc request type: N9txservice16FillStoreSliceCcE
E20251216 17:50:47.481468 339336 tx_execution.cpp:2966] ScanNextOperation failed for cc error: OUT_OF_MEMORY, txn: 9082, table: benchmark_db_11.benchmark_EloqDoc_10000_858993
I20251216 17:50:47.481792 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190718784, allocated 3189637008, frag ratio 0.034 , heap full: 1
I20251216 17:50:47.481828 339369 local_cc_shards.h:486] Table range memory report: allocated 1376640, committed 1575120, full: 0
I20251216 17:50:47.482470 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190718784, allocated 3189653568, frag ratio 0.033 , heap full: 1
--
I20251216 17:50:51.282554 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190731072, allocated 3189696544, frag ratio 0.032 , heap full: 1
I20251216 17:50:51.282583 339369 local_cc_shards.h:486] Table range memory report: allocated 1376688, committed 1575120, full: 0
I20251216 17:50:51.480000 339336 cc_shard.cpp:423] Abort cc request due to memory full. txn: 0, cc request type: N9txservice16FillStoreSliceCcE
E20251216 17:50:51.480046 339336 tx_execution.cpp:2966] ScanNextOperation failed for cc error: OUT_OF_MEMORY, txn: 9089, table: benchmark_db_11.benchmark_EloqDoc_10000_858993
I20251216 17:50:51.480352 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190743360, allocated 3189727312, frag ratio 0.032 , heap full: 1
I20251216 17:50:51.480377 339369 local_cc_shards.h:486] Table range memory report: allocated 1376704, committed 1575120, full: 0
I20251216 17:50:51.481566 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190743360, allocated 3189653568, frag ratio 0.034 , heap full: 1
--
I20251216 17:50:55.159320 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190743360, allocated 3189653568, frag ratio 0.034 , heap full: 1
I20251216 17:50:55.159353 339369 local_cc_shards.h:486] Table range memory report: allocated 1376736, committed 1575120, full: 0
I20251216 17:50:55.489215 339336 cc_shard.cpp:423] Abort cc request due to memory full. txn: 0, cc request type: N9txservice16FillStoreSliceCcE
E20251216 17:50:55.489252 339336 tx_execution.cpp:2966] ScanNextOperation failed for cc error: OUT_OF_MEMORY, txn: 9098, table: benchmark_db_11.benchmark_EloqDoc_10000_858993
I20251216 17:50:55.489622 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190718784, allocated 3189637008, frag ratio 0.034 , heap full: 1
I20251216 17:50:55.489645 339369 local_cc_shards.h:486] Table range memory report: allocated 1376752, committed 1575120, full: 0
I20251216 17:50:55.490351 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190718784, allocated 3189653568, frag ratio 0.033 , heap full: 1
--
I20251216 17:50:59.159667 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190718784, allocated 3189653568, frag ratio 0.033 , heap full: 1
I20251216 17:50:59.159759 339369 local_cc_shards.h:486] Table range memory report: allocated 1376784, committed 1575120, full: 0
I20251216 17:50:59.492799 339336 cc_shard.cpp:423] Abort cc request due to memory full. txn: 0, cc request type: N9txservice16FillStoreSliceCcE
E20251216 17:50:59.492874 339336 tx_execution.cpp:2966] ScanNextOperation failed for cc error: OUT_OF_MEMORY, txn: 9105, table: benchmark_db_11.benchmark_EloqDoc_10000_858993
I20251216 17:50:59.493212 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190718784, allocated 3189637008, frag ratio 0.034 , heap full: 1
I20251216 17:50:59.493242 339369 local_cc_shards.h:486] Table range memory report: allocated 1376800, committed 1575120, full: 0
I20251216 17:50:59.494503 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190718784, allocated 3189653568, frag ratio 0.033 , heap full: 1
--
I20251216 17:51:03.159874 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190743360, allocated 3189743872, frag ratio 0.031 , heap full: 1
I20251216 17:51:03.159920 339369 local_cc_shards.h:486] Table range memory report: allocated 1376848, committed 1575120, full: 0
I20251216 17:51:03.496354 339336 cc_shard.cpp:423] Abort cc request due to memory full. txn: 0, cc request type: N9txservice16FillStoreSliceCcE
E20251216 17:51:03.496397 339336 tx_execution.cpp:2966] ScanNextOperation failed for cc error: OUT_OF_MEMORY, txn: 9112, table: benchmark_db_11.benchmark_EloqDoc_10000_858993
I20251216 17:51:03.496645 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190743360, allocated 3189727312, frag ratio 0.032 , heap full: 1
I20251216 17:51:03.496672 339369 local_cc_shards.h:486] Table range memory report: allocated 1376864, committed 1575120, full: 0
I20251216 17:51:03.498168 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190743360, allocated 3189653568, frag ratio 0.034 , heap full: 1
--
I20251216 17:51:07.160240 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190743360, allocated 3189653568, frag ratio 0.034 , heap full: 1
I20251216 17:51:07.160246 339369 local_cc_shards.h:486] Table range memory report: allocated 1376912, committed 1575120, full: 0
I20251216 17:51:07.499845 339336 cc_shard.cpp:423] Abort cc request due to memory full. txn: 0, cc request type: N9txservice16FillStoreSliceCcE
E20251216 17:51:07.499886 339336 tx_execution.cpp:2966] ScanNextOperation failed for cc error: OUT_OF_MEMORY, txn: 9121, table: benchmark_db_11.benchmark_EloqDoc_10000_858993
I20251216 17:51:07.500118 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190718784, allocated 3189637008, frag ratio 0.034 , heap full: 1
I20251216 17:51:07.500140 339369 local_cc_shards.h:486] Table range memory report: allocated 1376928, committed 1575120, full: 0
I20251216 17:51:07.501459 339369 cc_request.h:3155] ccs 0 memory usage report, committed 3190718784, allocated 3189653568, frag ratio 0.033 , heap full: 1

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

_{You can configure auto-planning by selecting labels in the issue_enrichment configuration.}

🔗 Similar Issues

Possible Duplicates

• [Bug]: OOM when run mongo tpcc load #23

Related Issues

• [Bug]: Failed to insert batch: operation failed after 4 retries #278
• Concurrent DDL in a single core eloqdoc server eloqdoc#323
• https://github.com/eloqdata/tx_service/issues/235
• [Bug]: Some transactions hold write lock on the range table for more than 1 minute --- TPCC-load #178
• https://github.com/eloqdata/eloqdoc/issues/272

🔗 Related PRs

eloqdata/eloqdoc#243 - Refactor repo structure [merged]

👤 Suggested Assignees

• xiexiaoy
• zhangh43
• holiday-404
• lzxddz

---

🧪 Issue enrichment is currently in early access.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

[xiexiaoy]

Set bench_doc

	benchmarkConfigs = []BenchmarkConfig{
		// {DataVolume: 10000, DocumentSizes: []int{429497, 858993, 1503239, 1932735}},
		{DataVolume: 10000, DocumentSizes: []int{858993}},
	}

PR eloqdata/eloqdoc#369 has retried out-of-memory error.

$ cat ../etc/data_substrate.conf 
[local]
core_number=1
checkpoint_interval=10
node_memory_limit_mb=4096
enable_io_uring=true
raft_log_async_fsync=true
enable_data_store=true
enable_wal=true
eloq_data_path=/home/xiexy/edisk/service/eloqdata-mongodb/data/data
event_dispatcher_num=1
#auto_redirect=true
enable_mvcc=true
# txlog_rocksdb_cloud_bucket_prefix=eloqdoc-
# txlog_rocksdb_cloud_bucket_name=test
# txlog_rocksdb_cloud_object_path=txlog
# txlog_rocksdb_cloud_s3_endpoint_url=http://127.0.0.1:9900

[cluster]
ip_port_list=127.0.0.1:16379
#txlog_service_list=127.0.0.1:9000
#txlog_group_replica_num=1

[store]
# rocksdb_cloud_bucket_prefix=eloqdoc-
# rocksdb_cloud_bucket_name=test
# rocksdb_cloud_object_path=datastore
# rocksdb_cloud_s3_endpoint_url=http://127.0.0.1:9900
# aws_access_key_id=minioadmin
# aws_secret_key=minioadmin

[xiexiaoy]

Revert by #295.