Fix double-shifted training loss in GitForCausalLM

[Yigtwxx]

What does this PR do?

GitForCausalLM.forward shifts the labels manually and then passes them positionally to self.loss_function, so they bind to labels and shift_labels stays None. ForCausalLMLoss (loss_utils.py:61-64) then shifts a second time, and position t is trained to predict token t + 2.

Because the manual shift flattens the labels to 1-D before the call, the internal pad-and-slice keeps the shape consistent (N → N+1 → N), so nothing raises — and each batch row's final target is silently pulled in from the next row.

This regressed in #35875, which swapped CrossEntropyLoss (no shift) for self.loss_function (shifts) while leaving the manual shift in place. First released in v4.49.0. Same mechanism as the Moonshine fix in #46784.

Unlike Moonshine, GIT cannot just drop the manual shift: logits carries num_image_tokens leading image positions that must be sliced off regardless. So this PR passes shift_labels explicitly to suppress the internal shift, following the convention already used by csm (modeling_csm.py:619-621), and keeps the tensors 2-D since ForCausalLMLoss flattens them itself — which also removes the cross-row leak.

Verified with the repro from the issue (CPU, no pretrained weights). Before:

model loss            : 4.584834575653076
aligned CE (expected) : 4.646134853363037
matches aligned       : False
matches double-shift  : True
row 0 correct targets : [28, 71, 8, 58, 61, 15]
row 0 actual targets  : [71, 8, 58, 61, 15, 19]
row 1 labels          : [19, 42, 30, 58, 94, 67]

After:

model loss            : 4.646134853363037
aligned CE (expected) : 4.646134853363037
matches aligned       : True
matches double-shift  : False

Also checked that the num_items_in_batch path used by Trainer under gradient accumulation still matches the mean reduction when all targets are valid.

Per review, no model-level regression test is added here — the common test coming from #46903 is expected to cover this. The change is source-only.

$ pytest tests/models/git/
189 passed, 297 skipped, 2208 subtests passed

ruff check and ruff format --check (pinned 0.14.10) are clean. No # Copied from blocks reference this code and GIT has no modular file, so nothing needs regenerating.

Fixes #47394

Disclosure: this fix was implemented with the assistance of a code agent, per my proposal in the linked issue, and reviewed and validated by me.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline and the
  Pull Request checks?
• Was this discussed/approved via a Github issue? GitForCausalLM double-shifts the training loss and leaks targets across batch rows (regression from #35875) #47394
• Did you make sure to update the documentation with your changes according to the guidelines?
• Did you write any new necessary tests?

Who can review?

@zucchini-nlp

[zucchini-nlp]

I'd prefer to keep a sinlge PR to fix all models and add a proper tests. A contrib was working on it in #46903 (will nudge them again) so let's close this PR

Fixing and reviewing the same issue in all models one-by-one takes too much time for maintainers and for contribs

[Yigtwxx]

Thanks for the review, and understood on wanting a single consolidated PR — that makes sense from a maintenance standpoint.

One thing worth flagging before this gets folded into #46903: the two cases have different root paths, and I don't think #46903 as currently written would cover GIT.

#46903 targets encoder-decoder models, where decoder_input_ids is built via shift_tokens_right so the logits are already aligned with labels. Its regression test (test_encoder_decoder_loss_no_double_shift) is gated on config.is_encoder_decoder, so it skips GIT entirely.

GIT is decoder-only. The double shift here comes from GitForCausalLM.forward shifting manually — which it has to do anyway, because logits carries num_image_tokens leading image positions that must be sliced off — and then passing the result positionally, so it binds to labels and ForCausalLMLoss shifts again. Dropping the manual shift isn't an option the way it was for Moonshine (#46784); the fix is passing shift_labels explicitly.

So a genuinely model-agnostic PR would need to cover both the encoder-decoder path and the decoder-only-with-prefix-tokens path, with a test that isn't gated on is_encoder_decoder.

Happy to go either way:

• close this and port the GIT case + an ungated test into Fix loss alignment and Trainer token counting for encoder decoder models #46903 (coordinating with @OmkumarSolanki), or
• leave this open if you'd rather keep the decoder-only variant separate.

Just let me know which you prefer and I'll act on it.

[zucchini-nlp]

Oh sorry, indeed the issue here is that we passed manually shifted labels as raw labels. Seems to be rather a miss when converging into unified self.loss since GIT is special of itself

Can you delete the test, I dont think it is needed at model level. The common test for encoder-decoder models from another PR should be fine imo, and let's merge

[Yigtwxx]

Done — the model-level test is removed, so tests/models/git/test_modeling_git.py is now identical to main and the PR is source-only (4 insertions, 3 deletions in modeling_git.py). Updated the description accordingly.

Locally: pytest tests/models/git/ → 189 passed, 297 skipped, 2208 subtests passed; ruff check and ruff format --check clean.

Ready to merge from my side. Note that the common test in #46903 is currently gated on config.is_encoder_decoder, so it won't exercise GIT as written — worth ungating it there so this path stays covered.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: git

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 29637186287:2
Result: success | Jobs: 5 | Tests: 485 | Failures: 0 | Duration: 2m 33s

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.