llama: R-SWA reference sliding window attention for Unlimited-OCR

[sfallah]

Overview

Vibe-coded DRAFT to open the discussion about R-SWA support in llama.cpp.

• Adds baidu/Unlimited-OCR to mtmd: DeepSeek-OCR v1 with the original R-SWA decoder.
• R-SWA: each token attends the full prefix (image + prompt) + a window over the last n_swa generated tokens; unlike plain SWA, the image stays visible.
• New llama_swa_type REFERENCE; L_m latched implicitly in the KV cache at the prefill->decode boundary.
• Single full KV cache (no eviction); n_ref maintained across the cache lifecycle ops.
• n_ref is persisted in session/seq state -> LLAMA_SESSION_VERSION / LLAMA_STATE_SEQ_VERSION bumped.
• llama_model_n_swa returns 0 for deepseek2-ocr: the cache never evicts, so the server's SWA invalidation/checkpointing doesn't apply.
• F16 V cache garbles dense-table output; the default V type is promoted to F32 for R-SWA (logged; quantized -ctv still honored).

Scope: single-page parity.

Validation

The correctness is validated by reusing/extending a regression test that was introduced for DeepSeek-OCR (v1+2) against the HF reference implementation.

Single-page, bf16, against the HF reference in its release configuration (original bf16 weights, R-SWA active, CUDA):

	CER
llama.cpp (Metal and CUDA)	0.1574
HF reference	0.1591

Design note

R-SWA lives in core: LLAMA_SWA_TYPE_REFERENCE + the mask rule + per-seq n_ref in llama_kv_cache, activated only for deepseek2-ocr. Alternative: implement in deepseek2-ocr model code. Open to moving it into the model code if it is preferred.
Anyhow my immediate aim is that Unlimited-OCR is supported in llama.cpp with its original R-SWA.

Known limitations of the implicit latch

The prefill->decode boundary is inferred in the KV cache (first single-token append to a populated seq that requests output), not signaled by the caller. Where the heuristic breaks:

• multi-token decode (speculative, lookahead) never latches -> silently falls back to full causal.
• a prompt whose last ubatch is exactly 1 token latches one position early (the last prompt token leaves the always-visible prefix).
• context shift unlatches and re-latches at the current position -> near-full attention over the already-generated text afterwards.
• seq_cp copies the boundary unconditionally; a partial-range copy can transplant or clear it.

An explicit boundary signal from the caller (mtmd knows the media+prompt length) would remove the heuristic entirely. Open to that too if it's the preferred direction.

How to run

GGUF models: sabafallah/Unlimited-OCR-GGUF

build/bin/llama-mtmd-cli -hf sabafallah/Unlimited-OCR-GGUF:bf16 \
  --image tools/mtmd/test-1.jpeg -p "document parsing." \
  --chat-template deepseek-ocr \
  --temp 0 --flash-attn off --no-warmup \
  -n 4096 -c 16384 \
  --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 35 \
  --dry-penalty-last-n 128 --dry-sequence-breaker none

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - this is largely an AI-written first draft, opened to start the discussion. I designed the R-SWA approach (from the model's paper), directed and reviewed the implementation; the code itself is mostly AI-generated. I take responsibility for the contents.

[jason-ni]

It seems the impl in this branch has numeric precision issue:
https://huggingface.co/sabafallah/Unlimited-OCR-GGUF/discussions/1

@o7si could you please have a test on image in that discussion? Thanks.

[sfallah]

It seems the impl in this branch has numeric precision issue: https://huggingface.co/sabafallah/Unlimited-OCR-GGUF/discussions/1

@o7si could you please have a test on image in that discussion? Thanks.

@jason-ni
yes, I will.
Thanks for reporting this.

[sfallah]

@jason-ni
the root cause was found and fixed.
It was the DRY flags in my run example, not the model.
See details in the HF discussion: https://huggingface.co/sabafallah/Unlimited-OCR-GGUF/discussions/1