diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 02ba04db..06aa95e2 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -46,11 +46,13 @@ change as shipping to end users, not as an exercise. - Focus comes from `FocusTracker`, `FocusSnapshotResolver`, `AXTextGeometryResolver`, and `AXHelper`. Treat AX data as eventually consistent and app-specific. - Visual context flows through `VisualContextCoordinator`, - `ScreenshotContextGenerator`, `ScreenTextExtractor`, `WindowScreenshotService`, - and `LlamaVisualContextSummarizer`. + `ScreenshotContextGenerator`, `ScreenTextExtractor`, and `WindowScreenshotService`. + OCR text is cleaned by the pure `OCRTextHygiene`; there is no model summarization step. - Runtime generation flows through `SuggestionEngineRouter`, `FoundationModelSuggestionEngine`, `LlamaSuggestionEngine`, - `LlamaRuntimeManager`, and the serialized `LlamaRuntimeCore` actor. + `LlamaRuntimeManager`, and the serialized `LlamaRuntimeCore` actor. The OSS + (llama.cpp) path drives base models via `BaseCompletionPromptRenderer`; Apple + Foundation Models stays instruct via `FoundationModelPromptRenderer`. ## Comments diff --git a/AGENTS.md b/AGENTS.md index c542efa7..781d76f6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -54,7 +54,8 @@ When adding a `struct`, `class`, `enum`, actor, or protocol, explain: reconciliation, geometry helpers, and low-level bridging utilities. - `CotabbyTests/`: unit and microbench tests. Prefer testing pure `Support/` and `Models/` logic when possible. -- `LlamaRuntime/`: local llama.swift / llama.cpp integration artifacts. +- `CotabbyInference`: the llama.cpp wrapper, consumed as a SwiftPM package + (`github.com/FuJacob/cotabbyinference`, pinned to `main`) rather than vendored in-tree. ## App Ownership @@ -112,10 +113,11 @@ native AppKit fields, and secure fields expose different AX shapes. Preserve sta Visual context currently flows through: - `VisualContextCoordinator`: field-scoped visual-context session lifecycle. -- `ScreenshotContextGenerator`: screenshot -> OCR -> optional summary -> bounded excerpt. +- `ScreenshotContextGenerator`: screenshot -> OCR -> `OCRTextHygiene` cleanup -> bounded excerpt. - `WindowScreenshotService`: captures the relevant window or region. -- `ScreenTextExtractor`: Vision OCR extraction. -- `LlamaVisualContextSummarizer`: optional local summary using the selected llama runtime. +- `ScreenTextExtractor`: Vision OCR extraction, carrying per-line recognition confidence. +- `OCRTextHygiene`: pure cleanup of raw OCR (drops low-confidence lines and chrome noise). There is + no model summarization step; a base model conditions fine on cleaned raw context. - `VisualContextModels`: configuration, status, and excerpt values. Do not put raw screenshots, unbounded OCR dumps, or noisy AX tree text directly into prompts. @@ -131,8 +133,15 @@ Runtime generation is split by responsibility: - `LlamaSuggestionEngine`: request-to-prompt, llama result handling, and cache reset handoff. - `LlamaRuntimeManager`: UI-facing runtime state, model selection, warmup, and lifecycle control. - `LlamaRuntimeCore`: serialized actor around mutable llama.cpp pointers, prompt tokenization, - KV-cache reuse, sampling, and shutdown. -- `LlamaPromptRenderer`: prompt construction. + KV-cache reuse, sampling, an optional deterministic constrained decoder + (`runConstrainedDecode`, gated behind the default-off `cotabbyConstrainedDecoderEnabled`), and + shutdown. +- `BaseCompletionPromptRenderer`: prompt construction for the Open Source path. The llama models are + now *base* (non-instruct) GGUFs, so this renders a pure text continuation: no instruction preamble, + custom rules and context fold into a short conditioning preface (a base model conditions on + description, it does not obey commands), sections are character-budgeted via `PromptSectionBudget`, + and the caret prefix comes last. `FoundationModelPromptRenderer` stays instruct-shaped because + Apple's Foundation Models path gives us a first-class instructions channel. Keep llama.cpp pointer work serialized inside `LlamaRuntimeCore`. The manager should publish state; the core should own native correctness. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 6caf3eed..360783f9 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -63,7 +63,7 @@ The coordinator should not own pure decision rules or low-level OS logic. Those - `Cotabby/Support/SuggestionRequestFactory.swift`: pure request building - `Cotabby/Support/SuggestionSessionReconciler.swift`: pure session and acceptance rules - `Cotabby/Support/SuggestionAvailabilityEvaluator.swift`: pure gating logic -- `Cotabby/Services/Visual/VisualContextCoordinator.swift`: legacy screenshot/OCR lifecycle (deprecated during context rebuild) +- `Cotabby/Services/Visual/VisualContextCoordinator.swift`: screenshot/OCR lifecycle; OCR text is cleaned by the pure `OCRTextHygiene` filters (no model-summarization step) - `Cotabby/Services/Runtime/LlamaSuggestionEngine.swift`: prompt/result normalization over the runtime ## Focus And Accessibility diff --git a/README.md b/README.md index 35b30c4b..2cb8ab3f 100644 --- a/README.md +++ b/README.md @@ -85,14 +85,17 @@ Everything runs on-device. No hosted API, no cloud round-trip. **Apple Intelligence**: uses Apple's on-device `FoundationModels` runtime on macOS 26 or later, no download required. -**Open Source**: runs local GGUF models in-process through llama.cpp via `CotabbyInference`. Cotabby ships with four built-in downloadable models: - -| Model | File | Size | Source | -| -------------- | --------------------------------- | ------- | ------------------------------------------------------------------------------------------------------- | -| `tabby-1-nano` | `SmolLM2-135M-Instruct-q8_0.gguf` | ~0.1 GB | [Mungert/SmolLM2-135M-Instruct-GGUF](https://huggingface.co/Mungert/SmolLM2-135M-Instruct-GGUF) | -| `tabby-1-mini` | `Qwen3-0.6B-Q4_K_M.gguf` | ~0.4 GB | [unsloth/Qwen3-0.6B-GGUF](https://huggingface.co/unsloth/Qwen3-0.6B-GGUF) | -| `tabby-1-base` | `gemma-4-E2B-it-Q4_K_M.gguf` | ~3.1 GB | [unsloth/gemma-4-E2B-it-GGUF](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF) | -| `tabby-1-pro` | `gemma-4-E4B-it-Q4_K_M.gguf` | ~5.0 GB | [unsloth/gemma-4-E4B-it-GGUF](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) | +**Open Source**: runs local GGUF *base* models in-process through llama.cpp via `CotabbyInference`. Rather than instructing an instruction-tuned model, Cotabby treats the model as a pure text continuer and conditions it on your persona, style, language, and on-screen context. Cotabby ships with five built-in downloadable models: + +| Model | File | Size | Source | +| -------------------- | ------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------- | +| `tabby-2-mini` | `Qwen3.5-0.8B-Base.i1-Q6_K.gguf` | ~0.8 GB | [mradermacher/Qwen3.5-0.8B-Base-i1-GGUF](https://huggingface.co/mradermacher/Qwen3.5-0.8B-Base-i1-GGUF) | +| `tabby-2-base` | `Qwen3.5-2B-Base.i1-Q4_K_M.gguf` | ~1.4 GB | [mradermacher/Qwen3.5-2B-Base-i1-GGUF](https://huggingface.co/mradermacher/Qwen3.5-2B-Base-i1-GGUF) | +| `tabby-2-pro` | `Qwen3.5-4B-Base.i1-Q4_K_M.gguf` | ~2.6 GB | [mradermacher/Qwen3.5-4B-Base-i1-GGUF](https://huggingface.co/mradermacher/Qwen3.5-4B-Base-i1-GGUF) | +| `tabby-2-gemma-mini` | `gemma-4-E2B.i1-Q6_K.gguf` | ~4.5 GB | [mradermacher/gemma-4-E2B-i1-GGUF](https://huggingface.co/mradermacher/gemma-4-E2B-i1-GGUF) | +| `tabby-2-gemma-pro` | `gemma-4-E4B.i1-Q4_K_M.gguf` | ~5.0 GB | [mradermacher/gemma-4-E4B-i1-GGUF](https://huggingface.co/mradermacher/gemma-4-E4B-i1-GGUF) | + +`tabby-2-base` is the default. Apple Intelligence remains instruction-tuned and is unaffected by the base-model path. ### Bring your own model diff --git a/docs/POLLING_AND_DELAYS.md b/docs/POLLING_AND_DELAYS.md index dbfdb348..1b666166 100644 --- a/docs/POLLING_AND_DELAYS.md +++ b/docs/POLLING_AND_DELAYS.md @@ -37,7 +37,6 @@ Update this file whenever you add, remove, or change a timing constant. | Location | Value | Purpose | |----------|-------|---------| | `Cotabby/Services/Visual/VisualContextCoordinator.swift:29` | 250 ms | Session-start settle delay. Debounces visual context capture on focus change so a flapping Chromium focus doesn't retrigger screenshots and OCR. | -| `Cotabby/Services/Visual/LlamaVisualContextSummarizer.swift:20` | 3 s | Llama visual context summarization soft timeout. Cancels generation after 3 s and returns whatever partial text was produced. | ## Permissions