PocketTTS: feasibility of int8 weight quantization for memory-constrained iOS deployments

[Gaozhongpai]

Hi — opening this to check whether int8 quantization of PocketTTS is something that's been considered or evaluated for the FluidAudio CoreML port. I'm running into a memory budget issue on iPhone 13 Pro Max that the current fp32 release makes tight, and want to understand whether the path forward is selective int8, full int8, or something else entirely.

Context

Building a voice-first iOS app that runs Gemma 4 E2B (~1.5–2 GB resident via LiteRT-LM) alongside on-device TTS. iPhone 13 Pro Max has ~3 GB usable for third-party apps after iOS overhead. Current PocketTTS at ~650 MB on disk pushes us close to jetsam pressure when both engines are warm.

Two upstream signals suggest int8 might be viable:

1. Add int8 dynamic quantization support kyutai-labs/pocket-tts#147 (merged Apr 2026) — added quantize=True via torch.ao.quantization.quantize_dynamic on the FlowLM transformer's attention + FFN linear layers only, leaving flow_net and the Mimi VAE decoder at fp32. WER delta within noise (−0.022 ±0.024 for default config), 48% runtime memory reduction. Validated across 7 configs × 10 runs each.

2. KevinAHM/pocket-tts-onnx ships int8 ONNX exports of all four required models including mimi_decoder_int8.onnx and flow_lm_flow_int8.onnx, totaling ~120 MB. Existence proof that some int8 path produces audibly viable output, though I haven't verified ONNX RT vs CoreML quantization schemes produce equivalent quality.

I'm aware of the comment in PocketTtsModelStore.swift:39-43 explaining why ANE fp16 was rejected — Mimi decoder feedback-loop precision artifacts. The Kyutai PR's recipe explicitly avoids quantizing Mimi for the same reason, so a "selective int8" variant matching that recipe should sidestep the documented concern.

Questions

1. Has int8 been evaluated and rejected for reasons beyond the Mimi feedback loop? I.e. were there issues with cond_step or flow_decoder at int8 that aren't captured in the current ANE-fp16 comment?

2. Is there a PocketTTS variant lineage already planned that would address size? Noticed the repo has cond_step_v2.mlmodelc / cond_step_v3.mlmodelc alongside the fp32 default — wondering if any of those are precision experiments, or strictly graph-shape variants.

3. Would FluidAudio be receptive to a selective-int8 variant contribution following the Kyutai recipe (int8 weights on flowlm_step attention + FFN linears only, everything else fp32)? Estimated ~46% on-disk reduction on flowlm_step.mlpackage — total package ~650 MB → ~430 MB. Conservative target, no new precision risk beyond what Kyutai already validated.

4. Or is the right path full PyTorch re-conversion to match ONNX-export size more aggressively (~120 MB total)? My read is that's a much bigger lift — re-tracing autoregressive models with KV cache state, numerical equivalence validation, ABI-matching the existing Swift glue. Probably weeks of work where selective int8 is days. Wanted to check before assuming.

Thanks for the work on FluidAudio, the PocketTTS sessions API in particular is a nice fit for streaming voice assistants.

[Alex-Wengg]

https://github.com/FluidInference/FluidAudio/pull/540/changes @Gaozhongpai included in this pr

[Gaozhongpai]

Thanks for the pointer — I read through the branch, and want to make sure I'm not misreading it. From what I can see:

• The PR adds PocketTtsLanguage (10 cases including English + 9 new packs) plus an HF layout reorg with v2 subpaths under v2//, language-specific transformer layer counts (6 vs 24), and a v2 voice prebake pipeline.
• All variants on the branch appear to remain fp32 — I didn't find a precision enum, an int8-targeted resource path, or any
  coremltools.optimize / quantization config in the changed files.

If I missed where the int8 angle lands in this PR, I'd appreciate a pointer to the specific commit or file — it's possible the language pack expansion was meant to imply a refresh of model assets that includes int8 variants, but I couldn't see that on the branch.

[Alex-Wengg]

the int8 should be here
https://huggingface.co/FluidInference/pocket-tts-coreml/tree/main/v2/english/flowlm_stepv2.mlmodelc

Swift continues to load fp16 by default; clients should be able to swap in the int8 variant. if not please lmk so i may correct it

[Gaozhongpai]

Opened #558 — PocketTtsPrecision { .fp16, .int8 } threaded through manager/store/downloader, `+ post-download cleanup so int8 callers don't carry the unused fp16 FlowLM (~217 MB savings on English).

Verified the default is fp16 on disk by inspecting model.mlmodel for both variants (int8 has cast_fp16_to_fp32 scaffolding, default has none) and renamed the enum case accordingly before opening. 2 commits, no conflicts on #549. Happy to adjust API shape if you'd prefer a different approach.