Recipes

LiquidAI/LFM2.5-Audio-1.5B (Hugging Face). End-to-end audio–text model: FastConformer audio encoder → LFM2.5 1.2B backbone → depth transformer emitting Mimi codec tokens (8 codebooks, 24 kHz) → lightweight detokenizer. No separate ASR/TTS. GGUF builds exist for llama.cpp CPU inference; ONNX/WebGPU demos exist. A Japanese variant exists; broader multilingual output is announced; check current releases.

small enough for one 24 GB GPU (or two 4090s), upstream Python trainer (liquid_audio package), community full fine-tunes already demonstrate tool-aware voice flows trained in ~20 minutes on ~3k examples (reported), and it deploys on CPU/edge.

workflow adherence, short tool acknowledgements, narrating injected tool results, refusals when a tool is missing, tone/style, accent and code-switch understanding (e.g. Indian English, Hinglish input).

a new output language on the English base checkpoint, factual knowledge, full-duplex behaviour.

en-* native; hi-Latn-IN understand_only; others unsupported until multilingual checkpoints ship.

each user turn → one training session: system text (persona + tool list + "Known facts you must use" block when the previous turn was a tool result) → user audio clip → assistant target (text + audio). Tool calls are represented as the text convention chosen for this recipe and parsed back at serve time. Keep a "no-tool chitchat" class in the mix so the model does not over-trigger tools.

full fine-tune in bf16 via upstream trainer (LoRA via PEFT as an option if upstream supports it; verify). Defaults to start from: AdamW, lr 5e-5, cosine, ~50-step warmup, effective batch 8, 3–4 epochs, context ≈ 320 tokens (reported starting point, tune on our eval).

1× 24 GB (batch 2–4) or 2× 24 GB DDP. Colab L4/A100 works.

tool-call accuracy by class, entity accuracy (names, IDs, amounts, dates per locale), ASR round-trip WER of generated speech, TTFA.

native streaming via liquid_audio for GPU; GGUF path via llama.cpp for CPU (verify audio-out support in the llama.cpp build you pin).

Liquid AI's model licence; read it before commercial use; document in recipes/lfm25_audio/LICENSE_NOTES.md.

upstream API churn (the liquid_audio package is young); non-English text targets may degrade the English-centric speech decoder; measure.

• kyutai/moshiko-pytorch-bf16 / moshika (Kyutai Moshi, 7B, CC-BY-4.0, English).
• nvidia/personaplex-7b-v1 (Moshi architecture + weights, fine-tuned by NVIDIA for persona/voice control via text prompt + voice sample; code MIT, weights under NVIDIA Open Model License, gated on HF).

the only open path to real interruptions, backchannels, and natural timing. PersonaPlex is the better starting point for agents because persona prompting already works; its documented gap is that the text prompt shapes persona only and does not carry content, which is exactly what domain fine-tuning addresses.

en-* native; hi-Latn-IN understand_only (measure); others unsupported. Kyutai has said multilingual variants are in development; revisit.

stereo dual-stream conversations (channel 0 user, channel 1 agent) with transcripts. Mono call recordings are not enough; synth can produce stereo, and prepare can reconstruct pseudo-stereo from diarized mono (lower quality; flagged).

kyutai-labs/moshi-finetune (official, LoRA, includes dataset annotation tooling and a Colab). Pin a commit. Official example reports ~40 GB peak memory on one H100 for its recommended config (reported). nu-dialogue/moshi-finetune (unofficial, Accelerate/DeepSpeed, trains the user stream too) is an alternative for multi-GPU; J-Moshi shows language transfer is possible but needed 69k hours and 128 GPUs, so do not promise new-language support from this recipe.

canonical stereo → moshi-finetune layout (stereo WAV + JSON transcript with timestamps). Verify exact JSON keys against the pinned commit.

everything in Recipe A plus duplex metrics: interruption response time, barge-in stop time, overlap rate vs human reference, false-interruption rate.

Moshi's own server (PyTorch/Rust) or vLLM-omni's PersonaPlex duplex serving; LiveKit has an experimental PersonaPlex plugin. Wrap one of these behind our Realtime-compatible WebSocket; mark which path is verified.

40–80 GB GPU for training; ~19 GB bf16 for inference (reported), quantized ONNX variants exist.

CC-BY-4.0 (Moshi) vs NVIDIA Open Model License (PersonaPlex): different obligations; surface at vakforge train time.

Qwen/Qwen3-Omni-30B-A3B-Instruct (MoE, ~3B active) or the current Qwen3.5-Omni release. Thinker–Talker architecture: the Thinker is a text/multimodal LLM (long context, tool calling via <tool_call> tags), the Talker turns its hidden states into streaming speech. Multilingual speech in and out including Mandarin; the only open native S2S family with function calling today. Weights are also on ModelScope (matters for users without Hugging Face access).

unlocks non-English locale packs with native speech output, plus tool calling: the closest open analogue to GPT Realtime for global use.

zh-CN native; en-* native; hi-Latn-IN, es, de, fr, ja, ar; verify per language before marking native.

Thinker LoRA with Talker frozen, via ms-swift (supports Qwen3-Omni). This teaches workflow, tool use, and understanding without touching speech generation. Talker adaptation (speaker fine-tune stage) is upstream's multi-stage pipeline and is out of scope until packaged. Known pitfall: Thinker-only checkpoints saved by some frameworks use a key layout the full model cannot load without re-keying and merging Talker/code2wav from the vanilla checkpoint; the adapter must handle this and test it.

canonical → ms-swift multimodal chat JSONL (system, user audio, assistant text + tool calls). Audio-only training targets for the Thinker; speech output comes from the frozen Talker at inference.

large. bf16 weights for the 30B-A3B are tens of GB; LoRA training realistically needs an 80 GB GPU or multi-GPU; inference via vLLM-omni. Measure and record before publishing numbers.

tool-call accuracy, entity accuracy per locale, intelligibility of Talker output in the target language (ASR round-trip with a locale STT), latency (near-duplex, expect higher than Moshi).

vLLM-omni (supports Qwen3-Omni); wrap behind the Realtime WS. Note that streaming audio support in local serving may lag the hosted API; verify and document what actually streams.

Apache-2.0 for Qwen3-Omni at time of writing; confirm for the exact checkpoint.

framework version sensitivity (ms-swift/transformers/vLLM-omni move fast); hardware cost puts it out of reach for hobbyists, so position it as the "team with a server" recipe.

for any locale without a good native speech-to-speech model, a cascade of fine-tunable open parts works today, and each part is independently trainable and swappable. The locale pack picks the parts.

• STT: Whisper fine-tune (transformers / Unsloth) as the default; pack overrides: AI4Bharat IndicConformer (NeMo) for Indic, Paraformer/SenseVoice for zh, NVIDIA Canary/Parakeet for en throughput.
• LLM: any small instruct model (3–8B, Qwen3/Gemma/Llama-class) with LoRA via Unsloth or LLaMA-Factory on chat records from the canonical manifest; tool calling in OpenAI shape.
• TTS: pack default: Indic Parler-TTS / IndicF5 for Indic, CosyVoice / Qwen3-TTS for zh, Orpheus / Sesame CSM / Kokoro-class for en; speaker fine-tune where the licence allows.
• Orchestration: Pipecat pipeline with sentence-boundary aggregation; VAD for turn-taking.

public corpora chosen by the pack (Common Voice, AI4Bharat Vistaar/IndicVoices, WenetSpeech, etc.) plus your domain data.

STT WER/CER by language and entity type; LLM tool-call accuracy; end-to-end task completion; total latency (the cascade's weakness).

1× 24 GB for each component's fine-tune; inference fits on one GPU.