backport: merge bitcoin#21727, #22371, #21526, #23174, #23785, #23581, #23974, #22932, #24050, #24515 (blockstorage backports)

[kwvg]

Additional Information

• Dependent on refactor: drop usage of chainstate globals in Dash-specific code, merge bitcoin#21866 (goodbye to a global chainstate) #6078

• Dependent on backport: merge bitcoin#18344, #20867, #20286, #21359, #21910, #19651, #21934, #22722, #20295, #23702, partial bitcoin#23706 (rpc backports) #6074

• Dependent on refactor: move Get*Index to rpc/index_util.cpp, const-ify functions and arguments, add lock annotations and some minor housekeeping #6083

• Dependent on fix: move g_txindex initialization out of erroneous location and into constructor #6119

• Dependency for backport: merge bitcoin#22840, #22937, #23446, #23522, #24026, #24104, #24167, #20744, partial bitcoin#23469, #24169 (replace boost::filesystem with std::filesystem) #6138

• In bitcoin#24050, BlockMap is given ownership of the CBlockIndex instance contained within the unordered_map. The same has not been done for PrevBlockMap as PrevBlockMap is populated with pprev pointers and doing so seems to break validation logic.

• Dash has a specific linter for all Dash-specific code present in Core. The introduction of util/translation.h into validation.h has caused the linter to trigger shadowing warnings due to a conflict between the common use of _ as a placeholder/throwaway name (source) and upstream's usage of it to process translatable strings (source).

  Neither C++17 nor C++20 have an official placeholder/throwaway term or annotation for structured bindings (which cannot use [[maybe_unused] or std::ignore) but P2169 is a proposal put forth to make it the official placeholder, in that light, the linter will silence shadowing warnings involving an underscore.

Breaking Changes

None expected

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas (note: N/A)
• I have added or updated relevant unit/integration/functional/e2e tests
• I have made corresponding changes to the documentation (note: N/A)
• I have assigned this pull request to a milestone (for repository code-owners and collaborators only)

[knst]

Sanitizer tsan crashes like that in CI:

SUMMARY: ThreadSanitizer: heap-use-after-free /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:154:42 in _M_ptr

I guess, this crash happens due to re-creation of g_txindex inside TestChainSetup::mineBlocks

    g_txindex = std::make_unique<TxIndex>(1 << 20, true);                                  
    assert(g_txindex->Start(m_node.chainman->ActiveChainstate()));                         
    
    // Allow tx index to catch up with the block index.                                    
    IndexWaitSynced(*g_txindex);

so, g_txindex can be re-created several times, but there are async calls of g_txindex and they can be using g_txindex, when it called to re-create it.

I guess some guard code is needed for it.

[UdjinM6]

utACK 1bf0bf4 (with one nit)

[UdjinM6]

21727: nit: should drop this line I guess

[knst]

utACK 1bf0bf4

see nits if you will rebase it
+consider take these backports as useful follow-ups: bitcoin#23154 , bitcoin#24002, maybe new PR?

also maybe 24197

[knst]

nit: should not remove this empty line

[knst]

nit: (24515) have missing changes due to no namespace node::

[PastaPastaPasta]

utACK 1bf0bf4