LocateAnything 3B

[SangbumChoi]

What does this PR do?

Adds native Transformers support for NVIDIA LocateAnything 3B, including configuration, image processor, processor, PyTorch modeling code, AutoClass registration, documentation, and focused tests.

This is currently draft/WIP because the required maintainer or issue coordination has not been linked yet, and several source files still carry restrictive NVIDIA license headers that need a clean licensing path before review.

Coordination and duplicate-work status

• Coordination/approval issue or forum link: missing. This must be added before requesting review.
• Duplicate PR search run: gh pr list --repo huggingface/transformers --state open --search "LocateAnything OR Locate Anything OR nvidia/LocateAnything" --limit 20. The only direct LocateAnything PR found was this PR (LocateAnything 3B #46300).
• Issue search run: gh issue list --repo huggingface/transformers --state open --search "LocateAnything OR Locate Anything OR nvidia/LocateAnything" --limit 20. No matching open issue was found.

Tests and checks run

• make style
• make typing
• PYTHONPATH=src python utils/check_repo.py
• make check-repo
• PYTHONPATH=src pytest tests/models/locateanything/test_modeling_locateanything.py tests/models/locateanything/test_image_processing_locateanything.py -q
• PYTHONPATH=src python - <<'PY' from transformers import AutoProcessor processor = AutoProcessor.from_pretrained("nvidia/LocateAnything-3B", trust_remote_code=False) print(type(processor).__name__) print(type(processor.image_processor).__name__) PY

AI assistance disclosure

AI assistance was used to draft and iterate on this patch. The human submitter is responsible for reviewing every changed line and validating the change end-to-end before moving this PR out of draft.

Before submitting

• This is not a pure code-agent PR.
• I have read the contributor guidelines.
• This was discussed/approved via a GitHub issue or forum thread, linked above.
• Documentation was added/updated.
• Tests were added/updated.

[SangbumChoi]

@molbap @merveenoyan soft ping if you have time to review

[github-actions]

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

• The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
• A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
• A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

[IMvision12]

@SangbumChoi it will be good to have a colab gist with inference

[molbap]

@IMvision12 Or a snippet is already good, plain copy-able snippets are great for verifiability!
@SangbumChoi see also my comment on the other PR. Not much bandwidth on either right now

[SangbumChoi]

Thanks, here is a plain Colab-copyable native inference snippet. It is pinned to the current PR head and model revision, and does not use trust_remote_code.

%pip install -q "git+https://github.com/SangbumChoi/transformers.git@0aff35412bdea44d9c140a6b2d4728f3857d1fb6"

import requests
import torch
from PIL import Image
from transformers import AutoProcessor, LocateAnythingForConditionalGeneration

assert torch.cuda.is_available(), "Select a GPU runtime first."

model_id = "nvidia/LocateAnything-3B"
model_revision = "c32291ca5e996f5a7a485845b4f57a233936bba0"
dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16

processor = AutoProcessor.from_pretrained(model_id, revision=model_revision)
model = LocateAnythingForConditionalGeneration.from_pretrained(
    model_id,
    revision=model_revision,
    dtype=dtype,
    attn_implementation="sdpa",
).to("cuda").eval()

response = requests.get(
    "https://images.cocodataset.org/val2017/000000039769.jpg",
    stream=True,
    timeout=30,
)
response.raise_for_status()
image = Image.open(response.raw).convert("RGB")

question = "Locate all the instances that matches the following description: cat."
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": question},
        ],
    }
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(images=image, text=[prompt], return_tensors="pt").to("cuda")

with torch.inference_mode():
    generated = model.generate(**inputs, do_sample=False, max_new_tokens=64)

answer = processor.decode(
    generated[0, inputs["input_ids"].shape[-1] :],
    skip_special_tokens=False,
)
print(answer)

Expected decoded output:

<ref>cat</ref><box><0><115><494><988></box><|im_end|>

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, locateanything

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 30056000146:2
Result: success | Jobs: 16 | Tests: 171,810 | Failures: 0 | Duration: 15h 44m

[SangbumChoi]

I also uploaded a native Transformers training example without changing this PR branch:

• LoRA training script
• README with Colab/GPU commands

%pip install -q \
  "git+https://github.com/SangbumChoi/transformers.git@38b9bd8d87cd52ffca787490de353004a71de1c1" \
  "peft>=0.17.0" requests pillow

!wget -q \
  https://raw.githubusercontent.com/SangbumChoi/transformers/codex/locateanything-training-demo/examples/pytorch/locateanything/locateanything_lora_train.py

# Processor and assistant-only label validation (no model weights):
!python locateanything_lora_train.py --dry-run

# One LoRA optimizer step and adapter save:
!python locateanything_lora_train.py --steps 1 --output-dir locateanything-warehouse-lora

The default is the harder warehouse/industrial integration case (forklift, pallets, stacked boxes, and aisle). It limits preprocessing to 8,192 vision patches, yielding 2,226 total tokens and 62 supervised assistant tokens. I validated exact prompt-prefix masking and target decoding, plus a tiny LocateAnything LoRA forward/backward pass with finite loss and gradients on all trainable adapter tensors; make style also passes.

Scope: this exercises standard autoregressive NTP SFT in native Transformers. It does not reproduce NVIDIA's custom PBD/MTP continual-SFT stack; the full pipeline remains in the official Eagle training guide. The full 3B optimizer step has not been run from this environment because Hugging Face Jobs is not authenticated here, so an A10G/L4/A100/H100 GPU run is still required for end-to-end verification.

[SangbumChoi]

@IMvision12 Would you willing to review the code?