recovery control of workers

[samuelcajahuaringa]

Dear Hyperqueue team,

My script for task farming is:

#!/bin/bash
rm -rf hq-server-dir
mkdir hq-server-dir
rm -rf job-*

export HQ_SERVER_DIR="$(pwd)/hq-server-dir"

start the login node as server
./hq server start --server-dir "$HQ_SERVER_DIR" &
sleep 5

script to reserve 2 compute nodes for specif time in this case 2 workers
./request_hq_workers.sh 2

task farming
./hq submit --server-dir "$HQ_SERVER_DIR" --array 0-39 --cpus=1 --resource gpus/nvidia=1 ./run_reaxff_lammps.sh

python script that stop the server and workers if all tasks were finish
python3 monitor_hq.py 1 40

That works fine without any problem for short times (for testing) but when I pretend uses for long time 10-24 hours, the server in the login could be fall and I lost the control of workers (cpu nodes) to assign new tasks and the nodes become idle until the rest of requested time finished. If there any way to recovery the control of workers if I have the information of fall server.

Best regards,
Samuel

[Kobzol]

Hi, I'm not sure if I completely understand the question, but my understanding is that you need to have resiliency against worker and/or server failures?

In terms of worker faillures, that is relatively easy, you can simply restart the worker if it crashes or its node disappears, and it will pick up the uncompleted tasks. If you want to avoid doing that manually, you can use HQ's automatic allocation system, which will automatically submit Slurm allocations with workers on your behalf, based on the current computational load.

In terms of server failures, these should be very rare. You can let your server run in the background on a login node and it should be able to keep running for days or weeks (you just need to make sure to run it in the background). If you really want to make sure that the server will survive crashes or login node failures, you can start the server with a journal (https://it4innovations.github.io/hyperqueue/stable/deployment/server/#resuming-stoppedcrashed-server), which provides resilience against server failures.

Btw, you don't need to do this:

python script that stop the server and workers if all tasks were finish
python3 monitor_hq.py 1 40

The hq submit command has a parameter --wait (https://it4innovations.github.io/hyperqueue/stable/jobs/jobs/#waiting-for-jobs), which will wait until all the tasks of the submitted job have been completed.