Skip to content

Refresh docs for the base-model migration #512

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
FuJacob merged 1 commit into from
Jun 1, 2026
Merged

Refresh docs for the base-model migration #512

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .claude/CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,13 @@ change as shipping to end users, not as an exercise.
- Focus comes from `FocusTracker`, `FocusSnapshotResolver`, `AXTextGeometryResolver`,
and `AXHelper`. Treat AX data as eventually consistent and app-specific.
- Visual context flows through `VisualContextCoordinator`,
`ScreenshotContextGenerator`, `ScreenTextExtractor`, `WindowScreenshotService`,
and `LlamaVisualContextSummarizer`.
`ScreenshotContextGenerator`, `ScreenTextExtractor`, and `WindowScreenshotService`.
OCR text is cleaned by the pure `OCRTextHygiene`; there is no model summarization step.
- Runtime generation flows through `SuggestionEngineRouter`,
`FoundationModelSuggestionEngine`, `LlamaSuggestionEngine`,
`LlamaRuntimeManager`, and the serialized `LlamaRuntimeCore` actor.
`LlamaRuntimeManager`, and the serialized `LlamaRuntimeCore` actor. The OSS
(llama.cpp) path drives base models via `BaseCompletionPromptRenderer`; Apple
Foundation Models stays instruct via `FoundationModelPromptRenderer`.

## Comments

Expand Down
21 changes: 15 additions & 6 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,8 @@ When adding a `struct`, `class`, `enum`, actor, or protocol, explain:
reconciliation, geometry helpers, and low-level bridging utilities.
- `CotabbyTests/`: unit and microbench tests. Prefer testing pure `Support/` and `Models/` logic
when possible.
- `LlamaRuntime/`: local llama.swift / llama.cpp integration artifacts.
- `CotabbyInference`: the llama.cpp wrapper, consumed as a SwiftPM package
(`github.com/FuJacob/cotabbyinference`, pinned to `main`) rather than vendored in-tree.

## App Ownership

Expand Down Expand Up @@ -112,10 +113,11 @@ native AppKit fields, and secure fields expose different AX shapes. Preserve sta
Visual context currently flows through:

- `VisualContextCoordinator`: field-scoped visual-context session lifecycle.
- `ScreenshotContextGenerator`: screenshot -> OCR -> optional summary -> bounded excerpt.
- `ScreenshotContextGenerator`: screenshot -> OCR -> `OCRTextHygiene` cleanup -> bounded excerpt.
- `WindowScreenshotService`: captures the relevant window or region.
- `ScreenTextExtractor`: Vision OCR extraction.
- `LlamaVisualContextSummarizer`: optional local summary using the selected llama runtime.
- `ScreenTextExtractor`: Vision OCR extraction, carrying per-line recognition confidence.
- `OCRTextHygiene`: pure cleanup of raw OCR (drops low-confidence lines and chrome noise). There is
no model summarization step; a base model conditions fine on cleaned raw context.
- `VisualContextModels`: configuration, status, and excerpt values.

Do not put raw screenshots, unbounded OCR dumps, or noisy AX tree text directly into prompts.
Expand All @@ -131,8 +133,15 @@ Runtime generation is split by responsibility:
- `LlamaSuggestionEngine`: request-to-prompt, llama result handling, and cache reset handoff.
- `LlamaRuntimeManager`: UI-facing runtime state, model selection, warmup, and lifecycle control.
- `LlamaRuntimeCore`: serialized actor around mutable llama.cpp pointers, prompt tokenization,
KV-cache reuse, sampling, and shutdown.
- `LlamaPromptRenderer`: prompt construction.
KV-cache reuse, sampling, an optional deterministic constrained decoder
(`runConstrainedDecode`, gated behind the default-off `cotabbyConstrainedDecoderEnabled`), and
shutdown.
- `BaseCompletionPromptRenderer`: prompt construction for the Open Source path. The llama models are
now *base* (non-instruct) GGUFs, so this renders a pure text continuation: no instruction preamble,
custom rules and context fold into a short conditioning preface (a base model conditions on
description, it does not obey commands), sections are character-budgeted via `PromptSectionBudget`,
and the caret prefix comes last. `FoundationModelPromptRenderer` stays instruct-shaped because
Apple's Foundation Models path gives us a first-class instructions channel.

Keep llama.cpp pointer work serialized inside `LlamaRuntimeCore`. The manager should publish state;
the core should own native correctness.
Expand Down
2 changes: 1 addition & 1 deletion ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ The coordinator should not own pure decision rules or low-level OS logic. Those
- `Cotabby/Support/SuggestionRequestFactory.swift`: pure request building
- `Cotabby/Support/SuggestionSessionReconciler.swift`: pure session and acceptance rules
- `Cotabby/Support/SuggestionAvailabilityEvaluator.swift`: pure gating logic
- `Cotabby/Services/Visual/VisualContextCoordinator.swift`: legacy screenshot/OCR lifecycle (deprecated during context rebuild)
- `Cotabby/Services/Visual/VisualContextCoordinator.swift`: screenshot/OCR lifecycle; OCR text is cleaned by the pure `OCRTextHygiene` filters (no model-summarization step)
- `Cotabby/Services/Runtime/LlamaSuggestionEngine.swift`: prompt/result normalization over the runtime

## Focus And Accessibility
Expand Down
19 changes: 11 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,14 +85,17 @@ Everything runs on-device. No hosted API, no cloud round-trip.

**Apple Intelligence**: uses Apple's on-device `FoundationModels` runtime on macOS 26 or later, no download required.

**Open Source**: runs local GGUF models in-process through llama.cpp via `CotabbyInference`. Cotabby ships with four built-in downloadable models:

| Model | File | Size | Source |
| -------------- | --------------------------------- | ------- | ------------------------------------------------------------------------------------------------------- |
| `tabby-1-nano` | `SmolLM2-135M-Instruct-q8_0.gguf` | ~0.1 GB | [Mungert/SmolLM2-135M-Instruct-GGUF](https://huggingface.co/Mungert/SmolLM2-135M-Instruct-GGUF) |
| `tabby-1-mini` | `Qwen3-0.6B-Q4_K_M.gguf` | ~0.4 GB | [unsloth/Qwen3-0.6B-GGUF](https://huggingface.co/unsloth/Qwen3-0.6B-GGUF) |
| `tabby-1-base` | `gemma-4-E2B-it-Q4_K_M.gguf` | ~3.1 GB | [unsloth/gemma-4-E2B-it-GGUF](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF) |
| `tabby-1-pro` | `gemma-4-E4B-it-Q4_K_M.gguf` | ~5.0 GB | [unsloth/gemma-4-E4B-it-GGUF](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) |
**Open Source**: runs local GGUF *base* models in-process through llama.cpp via `CotabbyInference`. Rather than instructing an instruction-tuned model, Cotabby treats the model as a pure text continuer and conditions it on your persona, style, language, and on-screen context. Cotabby ships with five built-in downloadable models:

| Model | File | Size | Source |
| -------------------- | ------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------- |
| `tabby-2-mini` | `Qwen3.5-0.8B-Base.i1-Q6_K.gguf` | ~0.8 GB | [mradermacher/Qwen3.5-0.8B-Base-i1-GGUF](https://huggingface.co/mradermacher/Qwen3.5-0.8B-Base-i1-GGUF) |
| `tabby-2-base` | `Qwen3.5-2B-Base.i1-Q4_K_M.gguf` | ~1.4 GB | [mradermacher/Qwen3.5-2B-Base-i1-GGUF](https://huggingface.co/mradermacher/Qwen3.5-2B-Base-i1-GGUF) |
| `tabby-2-pro` | `Qwen3.5-4B-Base.i1-Q4_K_M.gguf` | ~2.6 GB | [mradermacher/Qwen3.5-4B-Base-i1-GGUF](https://huggingface.co/mradermacher/Qwen3.5-4B-Base-i1-GGUF) |
| `tabby-2-gemma-mini` | `gemma-4-E2B.i1-Q6_K.gguf` | ~4.5 GB | [mradermacher/gemma-4-E2B-i1-GGUF](https://huggingface.co/mradermacher/gemma-4-E2B-i1-GGUF) |
| `tabby-2-gemma-pro` | `gemma-4-E4B.i1-Q4_K_M.gguf` | ~5.0 GB | [mradermacher/gemma-4-E4B-i1-GGUF](https://huggingface.co/mradermacher/gemma-4-E4B-i1-GGUF) |

`tabby-2-base` is the default. Apple Intelligence remains instruction-tuned and is unaffected by the base-model path.

### Bring your own model

Expand Down
1 change: 0 additions & 1 deletion docs/POLLING_AND_DELAYS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@ Update this file whenever you add, remove, or change a timing constant.
| Location | Value | Purpose |
|----------|-------|---------|
| `Cotabby/Services/Visual/VisualContextCoordinator.swift:29` | 250 ms | Session-start settle delay. Debounces visual context capture on focus change so a flapping Chromium focus doesn't retrigger screenshots and OCR. |
| `Cotabby/Services/Visual/LlamaVisualContextSummarizer.swift:20` | 3 s | Llama visual context summarization soft timeout. Cancels generation after 3 s and returns whatever partial text was produced. |

## Permissions

Expand Down