servers, util: fix deadlock caused by conflicting lock order

[BurtonQin]

• Description:
  This PR fixes two deadlock bugs caused by conflicting lock order in stratumserver and log.
  In servers/src/mining/stratumserver.rs
  add_worker() calls stratum_stats.write() before workers_list.write()
  

  grin/servers/src/mining/stratumserver.rs

  Lines 700 to 704 in c7c9a32

  	pub fn add_worker(&self, tx: Tx) -> usize {
  	let mut stratum_stats = self.stratum_stats.write();
  	let worker_id = stratum_stats.worker_stats.len();
  	let worker = Worker::new(worker_id, tx);
  	let mut workers_list = self.workers_list.write();

  
  remove_worker() calls workers_list.read() before stratum_stats.write()
  

  grin/servers/src/mining/stratumserver.rs

  Line 721 in c7c9a32

  self.stratum_stats.write().num_workers = self.workers_list.read().len();

  
  When add_worker() and remove_worker() are called simultaneously, the following deadlock may happen:

Thread-A	Thread-B
A.lock()
	B.lock()
B.lock()//deadlock!
	A.lock()//deadlock!

Similarly,
init_logger() calls LOGGING_CONFIG.lock() before WAS_INIT.lock()

grin/util/src/logger.rs

Line 154 in c7c9a32

let mut config_ref = LOGGING_CONFIG.lock();

grin/util/src/logger.rs

Line 253 in c7c9a32

let mut was_init_ref = WAS_INIT.lock();

init_test_logger() calls WAS_INIT.lock() before LOGGING_CONFIG.lock()

grin/util/src/logger.rs

Lines 261 to 262 in c7c9a32

	pub fn init_test_logger() {
	let mut was_init_ref = WAS_INIT.lock();

grin/util/src/logger.rs

Line 271 in c7c9a32

let mut config_ref = LOGGING_CONFIG.lock();

• How I fix:
  The fix is to enforce the order of the locks.

In stratumserver.rs, I use only one write lock of workers_list and remove the following read lock. This is to avoid possible atomicity violation when workers_list is written by another thread before being read.
In logger.rs, WAS_INIT is lifted before LOGGING_CONFIG in remove_worker to prevent the interleaving of init_logger() and init_test_logger().
Note that dropping the lockguard of Lock-A before calling Lock-B is also a possible solution for conflicting lock order. But I did not use it here to prevent possible atomicity violation.

• Minor suggestion:
  When the critical sections of different locks may overlap, I suggest always locking in the order that they are declared to prevent such deadlocks.

There is only one thing that concerns me:
fn remove_worker() calls update_stats() where stratum_stats.write() is called. Then stratum_stats.write() is called again. Is it okay to have the two critical sections interleaved with remove_worker() or add_worker() from another thread?

[lehnberg]

Ping @hashmap - you might find this up your alley

[quentinlesceller]

Thank you for the PR @BurtonQin !

1. This makes sense. 👍
2. Is likely to never happen since init_logger and init_test_logger should never been called together. But overall 👍 for safety sake.

There is only one thing that concerns me:
fn remove_worker() calls update_stats() where stratum_stats.write() is called. Then stratum_stats.write() is called again. Is it okay to have the two critical sections interleaved with remove_worker() or add_worker() from another thread?

Could you details what you think is the problem here?

Overall 👍 LGTM

[BurtonQin]

Well, after a second thought, I did not find any problem.

[quentinlesceller]

@BurtonQin thank you so much for the PR 🎆. This is merged!