[BUG] netlab status fails if the server was rebooted

[sdargoeuves]

Describe the bug

If the server is rebooted (which often happens outside of my control), when I do netlab status I see this error:

╰─ dnetlab status 
Traceback (most recent call last):
  File "/home/seb/code/netlab/netlab/netlab", line 15, in <module>
    netsim.cli.lab_commands(__file__)
  File "/home/seb/code/netlab/netlab/netsim/cli/__init__.py", line 524, in lab_commands
    mod.run(sys.argv[arg_start:])   # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb/code/netlab/netlab/netsim/cli/status.py", line 374, in run
    show_lab(args,lab_states)
  File "/home/seb/code/netlab/netlab/netsim/cli/status.py", line 253, in show_lab
    fetch_node_status(lab_state,topology)
  File "/home/seb/code/netlab/netlab/netsim/cli/status.py", line 180, in fetch_node_status
    node_stat.status = wk_state.get('status','Unknown')
                       ^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'

To Reproduce

My topology file uses both provider libvirt and clab, and the multipass plugin.

Expected behavior

It should show me the state of the lab (which would be down as they don't survive a reboot)

Lab topology

---
message: "Extreme Lab!"

plugin: [ multilab ]
defaults.multilab.id: 3 # subnet will be 10.194.58.0/24
defaults.addressing.mgmt.start: 10

provider: clab

nodes:
  exos1:
    device: exos
    image: netlab/exos:32.7.2.19
    provider: libvirt

  exos2:
    device: exos
    image: vrnetlab/extreme_exos:32.7.2.19
    mtu: 1800
    provider: clab

links:
  - exos1:
    exos2:
  - exos1:
    exos2:
    name: "interEXOS - backup"

Version

• dev

[DanPartelly]

A reboot of the machine does not clean up netlab artefacts and netlab locks, so the returned status shouldn't be "down". In all fairness the resulted state is "corrupted" imo.

[ipspace]

In all fairness the resulted state is "corrupted" imo.

But we should survive that gracefully. @sdargoeuves -- it's probably Vagrant not returning node information and me blindly trusting it does. I fixed a similar issue in 94b1976

[sdargoeuves]

I remember trying a while ago to bring back up the lab without doing a netlab restart, maybe using --snapshot, or something like that...
Now I much prefer restarting the lab. So indeed, if we want to say the lab is corrupted I'm fine with that.

Sorry i haven't looked further into it, I saw that a little while back and didn't do anything with it. This time when it happened I went to create the bug to ensure I won't forget!
Happy to dig deeper if you don't want to restart your server.

[sdargoeuves]

This PR (#3247) should help by at least not failing.
To test this, I did run netlab up and fairly quickly, pressed Ctrl+C. Then the lab is in a weird state, which will create this AttributeError when doing netlab status.

[ipspace]

FWIW

it's probably Vagrant not returning node information and me blindly trusting it does.

It turns out to be containers. We do docker ps to get the real state of the containers, and no containers are running after a reboot, resulting in empty status dictionary, which subsequently caused the crash in not-careful-enough "netlab status" function.

The fix by @sdargoeuves in #3247 was just what we needed. Thanks again!