Skip to content

[FSDP2] Shard eagerly initialized optimizer state - #4142

Draft
sdjasj wants to merge 1 commit into
huggingface:mainfrom
sdjasj:agent/fix-fsdp2-adagrad-state
Draft

[FSDP2] Shard eagerly initialized optimizer state#4142
sdjasj wants to merge 1 commit into
huggingface:mainfrom
sdjasj:agent/fix-fsdp2-adagrad-state

Conversation

@sdjasj

@sdjasj sdjasj commented Jul 29, 2026

Copy link
Copy Markdown

What does this PR do?

Preserves eagerly initialized optimizer state when FSDP2 replaces full model parameters with sharded DTensors.

Adagrad initializes step and sum state in its constructor. FSDP2 preparation previously replaced only optimizer param_groups, so the state remained keyed by the original parameters. Moving the key alone is insufficient because parameter-shaped state must also use the new parameter's DTensor mesh, placements, and local device.

This change moves state keys through the temporary parameter placeholders, rekeys them to the final FSDP2 parameters, and distributes parameter-shaped state tensors with the same layout and device. Scalar state remains unchanged. A multi-process regression test exercises an Adagrad forward/backward/step and optimizer serialization.

Fixes #3859

Validation

RUN_SLOW=1 python -m pytest -q \
  tests/fsdp/test_fsdp.py::FSDP2IntegrationTest::test_adagrad_with_eager_optimizer_state
# 1 passed

# Four-GPU bf16 FSDP2 smoke test
accelerate launch --num_processes=4 --num_machines=1 --machine_rank=0 \
  --use_fsdp --fsdp_version=2 --mixed_precision=bf16 \
  --fsdp_reshard_after_forward=true --fsdp_auto_wrap_policy=SIZE_BASED_WRAP \
  --fsdp_min_num_params=1 tests/fsdp/fsdp2_adagrad.py

ruff check src/accelerate/accelerator.py src/accelerate/utils/fsdp_utils.py \
  tests/fsdp/test_fsdp.py tests/fsdp/fsdp2_adagrad.py
ruff format --check src/accelerate/accelerator.py src/accelerate/utils/fsdp_utils.py \
  tests/fsdp/test_fsdp.py tests/fsdp/fsdp2_adagrad.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimizer Adagrad not working with Accelerate

1 participant