Reduce lock contention in adding messages to mempool

[LesnyRumcajs]

Summary

There is a suspiciously long lock here

forest/src/message_pool/msgpool/msg_pool.rs

Lines 246 to 253 in 92ee2fb

	pub fn add(&self, msg: SignedMessage) -> Result<(), Error> {
	self.check_message(&msg)?;
	let tip = self.cur_tipset.lock().clone();
	self.add_tipset(msg, &tip, false)?;
	Ok(())
	}

In Lotus, there are some optimisations around it.

https://github.com/LesnyRumcajs/lotus/blob/896d3c33330a75a232b8332ee9632454c1765d6d/chain/messagepool/messagepool.go#L767-L796

This would explain long running mpool tests where ~10k messages are added (which shouldn't be that much, especially on a decent hardware; but it still takes well over 30s).

Completion Criteria

• Identify locks in the mentioned implementation and reduce the contention time.

Additional Links & Resources

[hanabi1224]

long running mpool tests

@LesnyRumcajs does the test refer to forest-cli chain head --format json -n 1000 | scripts/mpool_select_killer.rb? If so, I tested it locally against both forest and lotus, it takes

• ~50s against a calibnet Forest node (consistent across multiple runs)
• ~2m55s against a calibnet Lotus node on the first run
• ~1m15s against a calibnet Lotus node since the second run

[LesnyRumcajs]

No, I meant these.

forest/.config/nextest.toml

Lines 16 to 24 in 644df8e

	# This test checks the limitations of the message pool, which can take a while.
	[[profile.default.overrides]]
	filter = 'test(message_pool::msgpool::selection::test_selection::message_selection_trimming_msgs_two_senders)'
	slow-timeout = { period = "120s", terminate-after = 3 }
	# This test checks the limitations of the message pool, which can take a while.
	[[profile.default.overrides]]
	filter = 'test(message_pool::msgpool::selection::test_selection::message_selection_trimming_msgs_two_senders_complex)'
	slow-timeout = { period = "120s", terminate-after = 3 }

[hanabi1224]

@LesnyRumcajs I did time cost breakdown in message_selection_trimming_msgs_two_senders and it turns out message signature creation and verification are the slowest parts. In this case should we close this issue?

successes:

---- message_pool::msgpool::selection::test_selection::message_selection_trimming_msgs_two_senders stdout ----
msg_create_time: 13s 81ms 547us 881ns, msg_check_time: 30s 433ms 47us 885ns, msg_add_time: 2s 22ms 346us 554ns


successes:
    message_pool::msgpool::selection::test_selection::message_selection_trimming_msgs_two_senders

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 556 filtered out; finished in 46.03s