v0.3: n-gram suggestion data + query layer - #3
Conversation
Add tools/build_ngrams.py (builds a pruned unigram + bigram table from ainu-corpora: 4000 unigrams, 12k contexts, 460 KB) and src/suggest.rs which embeds and parses it (pure-std, bounds-checked) exposing next_words()/default_words(). Validated end-to-end: 'ruwe'->'ne', 'a=kor'->kinship terms. The candidate UI consumer lands next. Unit tests included. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Extend the generator and table to format v2: a (prev2 prev1) -> next trigram model (26k contexts) alongside the bigram model. suggest.rs gains predict(prev2, prev1) with trigram -> bigram backoff. Table grows to 1.3 MB. Verified on the real corpus: 'ruwe ne' -> na/gusu, 'somo ki' -> no/p/wa. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
complete(prefix, n) returns known words starting with the prefix, frequency-ranked (e.g. 'kam' -> kamuy/kamui, 'ay' -> aynu/aynumosir). Rounds out the query API: default words + next-word prediction + completion. Verified on the real corpus. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The n-gram table is derived from the rights-restricted ainu-corpora; per the ainu-llm/ainu-tts posture, corpus-derived artifacts are private-by-default and need a rights/ethics review before release. So: gitignore data/ngrams.bin (now untracked) and have build.rs generate the OUT_DIR table — the real one if a locally generated data/ngrams.bin is present, else an EMPTY table so public builds compile with suggestions disabled. suggest.rs includes from OUT_DIR; the embedded-table test tolerates the empty case. The publish decision for the real table is deferred to a rights review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Superseded — this branch is fully merged into |
Draft — the data + query layer for the suggestion feature. Stacked on #1 (retarget to
masterafter #1 merges). The candidate-window UI (the consumer) is a separate, Windows-side follow-up.What
tools/build_ngrams.py— builds a pruned n-gram table fromainu-corpora(~196k sentences): a 4000-word unigram list + a bigram model (prev word → top-8 next), serialized todata/ngrams.bin(460 KB). Stores aggregate counts only.src/suggest.rs— embeds the table (include_bytes!) and parses it (pure-std, bounds-checked, can't panic on a bad table). API:next_words(prev),default_words(n).Verified (offline, here)
cargo check --target x86_64-pc-windows-msvcgreen.ruwe→ne(17.6k; the evidential copula)a=kor(my-) →yupo / sapo / hapo / ona(kinship terms after a possessive)kamuy→ne / utar / menokoNext (not in this PR)
Candidate window (
ITfCandidateListUIElement) that shows these as katakana viaainconv, wired into the composition flow — Windows UI, needs the v0.2 foundation.