audio.cpp

C++ inference for neural audio models — GGUF-native, no Python, no PyTorch.

Targets the models that existing C++ audio frameworks don't cover:

Model	Task	audio.cpp	audio.cpp (0xShug0)
HiFiGAN	Vocoder (mel→wav)	✅	❌
BigVGAN	Vocoder (mel→wav)	✅	❌
EnCodec	Codec (wav↔tokens)	✅	❌ (MioCodec only)
DAC	Codec (wav↔tokens)	✅	❌
AASIST	Anti-spoofing	✅	❌
AudioSeal	Watermarking	✅	❌

Built on ggml: same backend that powers llama.cpp and whisper.cpp. Weights stored in GGUF.

---

Quick start

git clone https://github.com/liodon-ai/audio.cpp
cd audio.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

With CUDA:

cmake -B build -DAC_CUDA=ON && cmake --build build -j

---

Models

HiFiGAN / BigVGAN — Vocoder

Synthesize waveforms from mel spectrograms (the missing link in any TTS stack).

# Convert from HuggingFace
python convert/convert_hifigan.py \
    --model jik876/hifi-gan --config universal_v1 \
    --output hifigan.gguf

# BigVGAN
python convert/convert_hifigan.py \
    --model nvidia/bigvgan-v2-22khz-80band-256x \
    --output bigvgan.gguf --bigvgan

# Run
./build/ac-vocoder -m hifigan.gguf -i input.wav -o output.wav

Architecture: Conv1d pre-net → 4× ConvTranspose1d upsampling → MRF ResBlocks → Conv1d post-net → Tanh. BigVGAN adds Snake activation (x + sin²(αx)/α) instead of LeakyReLU.

---

EnCodec — Neural Audio Codec

24 kHz and 48 kHz neural codec from Meta AI. Encode audio to discrete tokens (1.5–24 kbps), reconstruct from tokens.

python convert/convert_encodec.py \
    --model facebook/encodec_24khz \
    --output encodec-24khz.gguf

# Roundtrip
./build/ac-codec -m encodec-24khz.gguf -i input.wav -o reconstructed.wav

# Encode only (save codes)
./build/ac-codec -m encodec-24khz.gguf -i input.wav -o codes --encode-only

# Decode from codes
./build/ac-codec -m encodec-24khz.gguf --decode codes.bin -o output.wav

Architecture: SEANet encoder (Conv1d × 4, stride [8,5,4,2]) → 128-dim latent → RVQ (8 codebooks × 1024 entries) → SEANet decoder.

---

AASIST — Anti-Spoofing

Detect AI-generated or voice-cloned audio. Returns spoof probability in [0, 1].

python convert/convert_aasist.py \
    --checkpoint aasist_model.pth \
    --config aasist.conf \
    --output aasist.gguf

# Single file
./build/ac-antispoof -m aasist.gguf input.wav

# Batch
./build/ac-antispoof -m aasist.gguf --json audio1.wav audio2.wav audio3.wav

# Output
# audio1.wav    prob=0.0231  [BONAFIDE]
# audio2.wav    prob=0.9872  [SPOOF]

Architecture: LFCC features → Conv blocks → Spectro-Temporal Graph Attention → attention pooling → binary classifier.

---

AudioSeal — Audio Watermarking

Imperceptibly embed a 16-bit message into audio. Detect watermark presence and decode message from any segment of the audio.

python convert/convert_audioseal.py \
    --model audioseal/audioseal \
    --output audioseal.gguf

# Embed watermark
./build/ac-watermark -m audioseal.gguf embed \
    -i input.wav -o watermarked.wav -msg ABCD

# Detect + decode
./build/ac-watermark -m audioseal.gguf detect -i watermarked.wav
# Confidence: 0.9841  Message: 0xABCD  [WATERMARKED]

---

C API

#include <audio.h>

// Vocoder
struct ac_vocoder_context * ctx = ac_vocoder_init("hifigan.gguf",
                                                   ac_vocoder_default_params());
int n_out;
float * audio = ac_vocoder_generate(ctx, mel, n_frames, 80, &n_out);
// ... use audio ...
free(audio);
ac_vocoder_free(ctx);

// Codec
struct ac_codec_context * cc = ac_codec_init("encodec-24khz.gguf",
                                              ac_codec_default_params());
int n_cb, n_fr;
int32_t * codes = ac_codec_encode(cc, wav, n_samples, 24000, &n_cb, &n_fr);
int n_reconstructed;
float * rec = ac_codec_decode(cc, codes, n_cb, n_fr, &n_reconstructed);
free(codes); free(rec);
ac_codec_free(cc);

// Anti-spoof
struct ac_antispoof_context * as = ac_antispoof_init("aasist.gguf",
                                                      ac_antispoof_default_params());
float spoof_prob = ac_antispoof_score(as, wav, n_samples, sample_rate);
bool  is_fake    = ac_antispoof_is_spoof(as, wav, n_samples, sample_rate, 0.5f);
ac_antispoof_free(as);

// Watermark
struct ac_watermark_context * wm = ac_watermark_init("audioseal.gguf",
                                                      ac_watermark_default_params());
uint8_t msg[2] = {0xAB, 0xCD};
float * wm_audio = ac_watermark_embed(wm, wav, n_samples, 16000, msg, 2);
uint8_t decoded[2];
float confidence = ac_watermark_detect(wm, wm_audio, n_samples, 16000, decoded, 2);
ac_watermark_free(wm);

---

GGUF metadata keys

All audio.cpp models use the ac.* namespace:

Key	Type	Description
ac.arch	str	Model family name
ac.model_type	i32	Enum (1=HiFiGAN, 2=BigVGAN, ...)
ac.n_mels	i32	Mel channels (vocoders)
ac.use_snake	bool	BigVGAN snake activation
ac.sample_rate	i32	Native sample rate
ac.n_codebooks	i32	RVQ codebook count (codecs)
ac.codebook_size	i32	RVQ codebook size
ac.codebook_dim	i32	RVQ latent dimension
ac.n_message_bits	i32	Watermark payload bits
ac.has_generator	bool	AudioSeal includes generator

---

Roadmap

• DAC (Descript Audio Codec) — EnCodec alternative
• SNAC (multi-scale RVQ codec used by Orpheus TTS)
• RawNet3 (speaker verification)
• DeepFilterNet (noise suppression / bandwidth extension)
• CUDA kernels for conv upsampling
• Python bindings

---

License

MIT. Built on ggml (MIT).