Test the mongo tool mongodump/mongorestore

[lzxddz]

Task Description

Area Impacted

• CI/CD
• Documentation
• Dependencies
• Infrastructure
• Code Cleanup

[lzxddz]

Used the codes (https://github.com/mongodb/mongo-tools/, commit 61405b8ff290567a4e5d76300825ce4946b27b12) to build the tools of mongodump/mongrestore.

go build -o bin/mongodump mongodump/main/mongodump.go
go build -o bin/mongorestore mongorestore/main/mongorestore.go

Test Step:

• Use tpcc to load 1 warehouse and run tpcc-test for 5 minutes.

• Dump the data use follow commands:

    ./mongodump -h 127.0.0.1 --port 27017 -d tpcc -o tpcc-dump
  

• Stop eloqdoc and clear data

• Restore data use follow commands:

    ./mongorestore -h 127.0.0.1 --port 27017   tpcc-dump
  

[lzxddz]

crashed on restore

logs:

    finished restoring tpcc.STOCK (100000 documents, 0 failures)
2025-09-26T17:52:04.800+0800    restoring indexes for collection tpcc.ITEM from metadata
2025-09-26T17:52:04.801+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"I_W_ID_1_I_ID_1", "ns":"tpcc.ITEM", "unique":true, "v":2}, Key:primitive.D{primitive.E{Key:"I_W_ID", Value:1}, primitive.E{Key:"I_ID", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2025-09-26T17:52:04.801+0800    no indexes to restore for collection tpcc.HISTORY
2025-09-26T17:52:04.801+0800    restoring indexes for collection tpcc.NEW_ORDER from metadata
2025-09-26T17:52:04.801+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"NO_W_ID_1_NO_D_ID_1_NO_O_ID_1", "ns":"tpcc.NEW_ORDER", "unique":true, "v":2}, Key:primitive.D{primitive.E{Key:"NO_W_ID", Value:1}, primitive.E{Key:"NO_D_ID", Value:1}, primitive.E{Key:"NO_O_ID", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2025-09-26T17:52:04.801+0800    restoring indexes for collection tpcc.ORDERS from metadata
2025-09-26T17:52:04.801+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"O_W_ID_1_O_D_ID_1_O_ID_1_O_C_ID_1", "ns":"tpcc.ORDERS", "unique":true, "v":2}, Key:primitive.D{primitive.E{Key:"O_W_ID", Value:1}, primitive.E{Key:"O_D_ID", Value:1}, primitive.E{Key:"O_ID", Value:1}, primitive.E{Key:"O_C_ID", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2025-09-26T17:52:04.801+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"O_C_ID_1_O_D_ID_1_O_W_ID_1_O_ID_-1_O_CARRIER_ID_1_O_ENTRY_ID_1", "ns":"tpcc.ORDERS", "v":2}, Key:primitive.D{primitive.E{Key:"O_C_ID", Value:1}, primitive.E{Key:"O_D_ID", Value:1}, primitive.E{Key:"O_W_ID", Value:1}, primitive.E{Key:"O_ID", Value:-1}, primitive.E{Key:"O_CARRIER_ID", Value:1}, primitive.E{Key:"O_ENTRY_ID", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2025-09-26T17:52:04.801+0800    restoring indexes for collection tpcc.STOCK from metadata
2025-09-26T17:52:04.801+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"S_W_ID_1_S_I_ID_1_S_QUANTITY_1", "ns":"tpcc.STOCK", "unique":true, "v":2}, Key:primitive.D{primitive.E{Key:"S_W_ID", Value:1}, primitive.E{Key:"S_I_ID", Value:1}, primitive.E{Key:"S_QUANTITY", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2025-09-26T17:52:04.801+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"S_I_ID_1", "ns":"tpcc.STOCK", "v":2}, Key:primitive.D{primitive.E{Key:"S_I_ID", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2025-09-26T17:52:05.926+0800    Failed: tpcc.ORDERS: error creating indexes for tpcc.ORDERS: createIndex error: connection(127.0.0.1:27017[-4]) socket was unexpectedly closed: EOF
2025-09-26T17:52:05.926+0800    299764 document(s) restored successfully. 0 document(s) failed to restore.

core dump stack:

(gdb)
(gdb) bt
#0  0x00007146ddc969fc in pthread_kill () from /jepsen-test/eloqdoc-cloud/bin/../lib/libc.so.6
#1  0x00007146ddc42476 in raise () from /jepsen-test/eloqdoc-cloud/bin/../lib/libc.so.6
#2  0x00007146ddc287f3 in abort () from /jepsen-test/eloqdoc-cloud/bin/../lib/libc.so.6
#3  0x00007146de911906 in mongo::fassertFailedWithStatusWithLocation (msgid=msgid@entry=40113, status=...,
    file=file@entry=0x7146dd75c8b8 "src/mongo/db/storage/kv/kv_catalog.cpp", line=line@entry=326)
    at src/mongo/util/assert_util.cpp:165
#4  0x00007146dd74ee37 in mongo::fassertWithLocation (line=326, file=0x7146dd75c8b8 "src/mongo/db/storage/kv/kv_catalog.cpp",
    status=..., msgid=40113) at src/mongo/util/assert_util.h:287
#5  mongo::KVCatalog::FeatureTracker::putInfo (this=0x552a7c26f80, opCtx=0x55342080a00, versionInfo=...)
    at src/mongo/db/storage/kv/kv_catalog.cpp:326
#6  0x00007146dd7525b9 in mongo::KVCatalog::FeatureTracker::markRepairableFeatureAsInUse (this=0x552a7c26f80,
    opCtx=opCtx@entry=0x55342080a00,
    feature=feature@entry=mongo::KVCatalog::FeatureTracker::RepairableFeature::kPathLevelMultikeyTracking)
    at src/mongo/db/storage/kv/kv_catalog.cpp:257
#7  0x00007146dd75a786 in mongo::KVCollectionCatalogEntry::prepareForIndexBuild (this=0x553420917a0, opCtx=0x55342080a00,
    spec=0x553423d6e40, isBackgroundSecondaryBuild=<optimized out>) at src/mongo/db/storage/kv/kv_catalog.h:109
#8  0x00007146dc93cf41 in mongo::IndexCatalogImpl::IndexBuildBlock::init (this=this@entry=0x55342220280)
    at src/mongo/db/catalog/index_catalog_impl.cpp:417
#9  0x00007146dc94bce5 in mongo::MultiIndexBlockImpl::init (this=0x5534206a7c0, indexSpecs=...)
    at /usr/include/c++/11/bits/unique_ptr.h:173
#10 0x00007146d8393edb in mongo::MultiIndexBlock::init (specs=..., this=<optimized out>) at src/mongo/db/catalog/index_create.h:179
#11 mongo::CmdCreateIndex::errmsgRun(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, mongo::BSONObjBuilder&)::{lambda()#7}::operator()() const (__closure=0x5533d8eee00, __closure=0x5533d8eee00)
    at src/mongo/db/commands/create_indexes.cpp:397
#12 mongo::writeConflictRetry<mongo::CmdCreateIndex::errmsgRun(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, mongo::BSONObjBuilder&)::{lambda()#7}>(mongo::OperationContext*, mongo::StringData, mongo::StringData, mongo::CmdCreateIndex::errmsgRun(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, mongo::BSONObjBuilder&)::{lambda()#7}&&) (opCtx=0x55342080a00, opStr=..., ns=..., f=...) at src/mongo/db/concurrency/write_conflict_exception.h:91
#13 0x00007146d8396bc4 in mongo::CmdCreateIndex::errmsgRun (this=<optimized out>, opCtx=<optimized out>, dbname=..., cmdObj=...,
    errmsg=..., result=...) at src/mongo/base/string_data.h:62
#14 0x00007146db04561c in mongo::ErrmsgCommandDeprecated::run (this=<optimized out>, opCtx=<optimized out>, db=..., cmdObj=...,
--Type <RET> for more, q to quit, c to continue without paging--
    result=...) at src/mongo/db/commands.cpp:546
#15 0x00007146db04953d in mongo::BasicCommand::Invocation::run (this=0x55342066bc0, opCtx=0x55342080a00, result=<optimized out>)
    at src/mongo/db/commands.cpp:449
#16 0x00007146da48872c in mongo::(anonymous namespace)::runCommandImpl (opCtx=0x55342080a00, invocation=0x55342066bc0, request=...,

    replyBuilder=0x55342071c80, startOperationTime=..., behaviors=..., extraFieldsBuilder=0x5533d8ef560, sessionOptions=...,
    atomicCmd=false) at src/mongo/db/service_entry_point_common.cpp:568
#17 0x00007146da4897b0 in mongo::(anonymous namespace)::execCommandDatabase (opCtx=0x55342080a00,
    command=0x7146d83fd140 <mongo::cmdCreateIndex>, request=..., replyBuilder=0x55342071c80, behaviors=...)
    at /usr/include/c++/11/bits/unique_ptr.h:173
#18 0x00007146da48afb2 in operator() (__closure=0x5533d8efb90) at src/mongo/db/service_entry_point_common.cpp:1069

#19 0x00007146da48b921 in mongo::(anonymous namespace)::receivedCommands (behaviors=..., message=..., opCtx=0x55342080a00)
    at src/mongo/db/service_entry_point_common.cpp:1017
#20 mongo::ServiceEntryPointCommon::handleRequest (opCtx=0x55342080a00, m=..., behaviors=...)
    at src/mongo/db/service_entry_point_common.cpp:1390
#21 0x00007146ddebe3c4 in mongo::ServiceEntryPointMongod::handleRequest (this=<optimized out>, opCtx=<optimized out>, m=...)
    at src/mongo/db/service_entry_point_mongod.cpp:123
#22 0x00007146dba54bf6 in mongo::ServiceStateMachine::_processMessage (this=0x5533d5d0000, guard=...)
    at /usr/include/c++/11/bits/unique_ptr.h:173
#23 0x00007146dba52688 in operator() (sink=..., __closure=<optimized out>) at src/mongo/transport/service_state_machine.cpp:135
#24 std::__invoke_impl<boost::context::continuation, mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)::<lambda(boost::context::continuation&&)>&, boost::context::continuation> (__f=...) at /usr/include/c++/11/bits/invoke.h:61
#25 std::__invoke<mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)::<lambda(boost::context::continuation&&)>&, boost::context::continuation> (__fn=...) at /usr/include/c++/11/bits/invoke.h:97
#26 std::invoke<mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)::<lambda(boost::context::continuation&&)>&, boost::context::continuation> (__fn=...) at /usr/include/c++/11/functional:98
#27 boost::context::detail::record<boost::context::continuation, mongo::ServiceStateMachine::NoopAllocator, mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)::<lambda(boost::context::continuation&&)> >::run (fctx=<optimized out>,
    this=<optimized out>) at /usr/include/boost/context/continuation_fcontext.hpp:146
#28 boost::context::detail::context_entry<boost::context::detail::record<boost::context::continuation, mongo::ServiceStateMachine::NoopAllocator, mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)::<lambda(boost::context::continuation&&)> > >(boost::context::detail::transfer_t) (t=...) at /usr/include/boost/context/continuation_fcontext.hpp:80
#29 0x00007146da52524f in make_fcontext () from /jepsen-test/eloqdoc-cloud/bin/../lib/libboost_context.so.1.74.0
#30 0x0000000000000000 in ?? ()

[lzxddz]

Test mongodump/mongorestore between "Mongodb-v8.2.1" and "Eloqdoc"

Step:

1. Loading data into Mongodb-v8.2.1 ( tpcc load 1 warehouses)

2. dump data from Mongodb-v8.2.1
   ./mongodump -h 127.0.0.1 --port 27017 -d tpcc -o tpcc-dump

3. export (json) data from Mongodb-v8.2.1
   ./mongoexport -h 127.0.0.1 --port 27017 -d tpcc -c ORDERS -o ORDERS-1.json

4. restore data to Eloqdoc
   ./mongorestore -h 127.0.0.1 --port 27017 tpcc-dump

5. export (json) data from Eloqdoc
   ./mongoexport -h 127.0.0.1 --port 27017 -d tpcc -c ORDERS -o ORDERS-2.json

6. compare the file "ORDERS-1.json" and "ORDERS-2.json"