Add deepseek ocr

[molbap]

What does this PR do?

As per title. Architecturally: Llava-next used as skeleton with a modified SamModel and a modified ClipVisionModel, keeping the deepseekV2 decoder untouched (using AutoModel) and changing using config only.

• Working config + random weights init
• Modular draft with subconfigs (two vision configs)
• Conversion from original checkpoint done
• Modular model finished
• Integration tests/OCR tests working as in original codebase
• Make modular slimmer
• Make processor faster
• Complete test suite for transformers
• Remap weights to avoid conversion / on-the-fly conversion? (cc @ArthurZucker )

Current branch is functional. You don't need to convert the weights, just run the following on your image and you'll get a nice OCR output.

import torch
from PIL import Image

from transformers import AutoProcessor, DeepseekOcrForConditionalGeneration

processor = AutoProcessor.from_pretrained("deepseek-ai/DeepSeek-OCR")
model = DeepseekOcrForConditionalGeneration.from_pretrained("deepseek-ai/DeepSeek-OCR", dtype=torch.bfloat16)

image = Image.open("handwritten_letter_small.png").convert("RGB")

conversation = [
    {
        "role": "<|User|>",
        "content": [
            {"type": "image", "path": "./handwritten_letter_small.png"},
            {"type": "text", "text": "<|grounding|>Convert the document to markdown."},
        ],
    }
]

inputs = processor.apply_chat_template(
    conversation,
    return_dict=True,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
    )

with torch.no_grad():
    generated = model.generate(**inputs, max_new_tokens=50)

text = processor.batch_decode(generated, skip_special_tokens=False)[0]
print(text.strip())

[molbap]

Implementation works. Processor remains to be optimized but getting similar results as in original repository.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, deepseek_ocr

[molbap]

What's missing here is a solid mapping between checkpoint state dict and canonical 😅 once it's solid, it'll be a good step to use that model in vLLM/SGLang etc with the transformers modelling backend.

[github-actions]

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41797&sha=6b3375

[ducviet00]

Hi @molbap, thanks for the work on this!! I'd like to ask about the status of this PR