Skip to content

Fix nested directories being created when restarting#10

Merged
PatrickRMiles merged 4 commits into
LBANN:mainfrom
PatrickRMiles:fix/miles30/nested_dirs_in_restarts
Feb 17, 2026
Merged

Fix nested directories being created when restarting#10
PatrickRMiles merged 4 commits into
LBANN:mainfrom
PatrickRMiles:fix/miles30/nested_dirs_in_restarts

Conversation

@PatrickRMiles

Copy link
Copy Markdown
Collaborator

torchrun-hpc by default creates a new run directory, which is not what we want when restarting an existing run. This PR solves this by using the -l flag with torchrun-hpc and giving it the absolute path to the existing dir.
Also cleans up the restart scripts to more closely match the scaffold-tuolumne.job script we ship with the repo.

Patrick Miles added 4 commits January 29, 2026 13:20

@michaelmckinsey1 michaelmckinsey1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this tested?

Comment thread ScaFFold/utils/create_restart_script.py
Comment thread ScaFFold/utils/create_restart_script.py
@PatrickRMiles

Copy link
Copy Markdown
Collaborator Author

Is this tested?

Yup!

@michaelmckinsey1
michaelmckinsey1 self-requested a review February 6, 2026 02:44
@PatrickRMiles
PatrickRMiles merged commit c420941 into LBANN:main Feb 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants