Restore full functionality to trim_headers (untrim)

[psgreco]

Currently, trim_headers has certain limitations wrt the way it behaves in P2P protocol and in certain RPC calls. This PR brings back the full functionality and fixes a corruption issue when combined with pruning.

Fixes: #1241

[jamesdorfman]

This is a circular dependency.

"chain.h -> validation.h -> chain.h" 

I'm not quite sure how to fix it -- we can't simply forward declare ChainManager here since the line below requires knowledge of the fields of ChainManager:

m_pcontext->chainman->m_blockman.m_dirty_blockindex.insert(this);

[jamesdorfman]

Why was || node::fTrimHeaders removed?

[delta1]

This conditional is unsetting NODE_NETWORK, without this flag enabled it means this peer can't serve historical block data (ie. it's a pruned node). This was previously necessary because we weren't able to "untrim" the headers, which this PR fixes.

[delta1]

ACK eed85dc

I added the trim headers test from #1249 and confirmed that it passed here but fails on master (fixing #1334)

No noticeable effects during IBD from the additional locking.

[whitslack]

I have been running a pruned Elements node with this patch continuously for months and have seen no issues at all.

That said, memory usage is still unacceptably high at 2.3 GiB. Keeping the entire chain of block header hashes, going back to the genesis block, in RAM at all times is unsustainable, especially since a new block is added to the Liquid chain every minute. It means elementsd is almost entirely swapped out on my system — I see 20 MiB RSS and 2368 MiB in swap — and unfortunately elementsd accesses all of those pages during its shutdown, which means that my system takes many minutes to shut down, not great when the shutdown is triggered by UPS batteries nearing exhaustion. I know Elements is just copying what Bitcoin Core does, but they have one-tenth the block generation rate, so the issue is not as bad there, although it is increasingly becoming a problem over time. There has got to be a better way.

[psgreco]

UTACK eed85dc on all the additions.

[psgreco]

I have been running a pruned Elements node with this patch continuously for months and have seen no issues at all.

That said, memory usage is still unacceptably high at 2.3 GiB. Keeping the entire chain of block header hashes, going back to the genesis block, in RAM at all times is unsustainable, especially since a new block is added to the Liquid chain every minute. It means elementsd is almost entirely swapped out on my system — I see 20 MiB RSS and 2368 MiB in swap — and unfortunately elementsd accesses all of those pages during its shutdown, which means that my system takes many minutes to shut down, not great when the shutdown is triggered by UPS batteries nearing exhaustion. I know Elements is just copying what Bitcoin Core does, but they have one-tenth the block generation rate, so the issue is not as bad there, although it is increasingly becoming a problem over time. There has got to be a better way.

Hey, yeap, I agree that we need to continue reducing the usage, but this is the most we can do without diverging from what bitcoind does, any change after that would have to be much more invasive