vakforgedocs

Architecture

Principles

  1. One canonical format in the middle. Raw data → canonical manifest → recipe adapters. Nothing crosses that line sideways. 1b. Locale packs own everything language- or market-specific. Core asks the pack; it never branches on a language string.
  2. Core has no ML dependencies. schema, validate, inspect, recommend, report import only the standard library, pydantic, numpy, soundfile, rich, typer.
  3. Recipes are plugins. Each lives in its own package with its own optional extra, lazy imports, and its own integration tests. Adding a recipe never touches core.
  4. Wrap upstream, don't fork. moshi-finetune, liquid-audio, Unsloth, NeMo are pinned dependencies called through thin adapters. If an upstream needs a patch, keep it as a patches/*.patch applied at install time, and open an upstream PR.
  5. Everything is reproducible from a manifest hash + config file + pinned versions.

System view

What vakforge owns, what the coding agent generates per project, and what comes from upstream open source.

vakforge system view Three layers. Your project holds a coding agent that runs the vakforge CLI and writes the per-project glue for prepare, synth, train, eval and serve. The vakforge core is a CLI over schema and validate, inspect and recommend, each of which asks the locale packs. The generated glue reads the core's output and calls the open-source models, trainers and serving stacks, which are pinned per recipe. YOUR PROJECT Coding agent + vakforge skill follows SKILL.md, step by step Generated per project prepare · synth · train · eval · serve writes VAKFORGE CORE pip install vakforge · no ML dependencies CLI typer schema + validate inspect recommend locale packs everything local lives here OPEN SOURCE pinned per recipe, wrapped never forked Voice models LFM2.5 · Moshi · Qwen-Omni Trainers liquid-audio · ms-swift Serving Pipecat · LiveKit · vLLM runs reads inspect.json and vakforge.jsonl calls

Locale pack inheritance

Children merge language tags and PII patterns from their parent and override scalar settings (formats, consent rule, privacy notes).

Locale pack inheritance A tree. LocalePack is the base, holding formats, PII patterns, consent, privacy notes and recipe support. The en pack inherits from it and adds email, card and IBAN patterns plus an English word-error-rate normalizer. en-US, en-GB and en-IN inherit from en, each adding its own national identifiers, phone formats and currency. hi-Latn-IN inherits from en-IN and adds Roman-Hindi detection, lang_mix and a Devanagari-safe normalizer. LocalePack formats · PII patterns · consent · privacy notes · recipe support en email · card (Luhn) · IBAN · WER en-US SSN · NANP phones dollars · MDY dates en-GB NI number · UK phones pounds · DMY dates en-IN Aadhaar (Verhoeff) · PAN +91 mobiles · lakh/crore hi-Latn-IN Roman-Hindi detection · lang_mix Devanagari-safe normalizer A child merges its parent's language tags and PII patterns, and overrides the scalar settings it names.

An agent run, step by step

An agent run, step by step A sequence between four participants: the developer, the coding agent running the vakforge skill, the vakforge CLI, and the open models and trainers. In the decide phase the developer asks for a Hinglish assistant and the agent runs init, inspect and recommend, with the CLI returning the data report and the decision. In the agree phase the agent brings a plan, cost, consent basis and hardware question back to the developer, who approves. In the build phase the agent verifies every upstream API against installed source, writes the glue, and validates the dataset. In the prove phase it trains only what recommend asked for, evaluates base against tuned, and reports back with an endpoint only if the tuned model wins. DECIDE AGREE BUILD PROVE Developer Coding agent + skill vakforge CLI Models + trainers Build a Hinglish assistant from data/raw vakforge init -l hi-Latn-IN vakforge inspect data/raw sources, languages, PII, tool candidates vakforge recommend retrieval and tools first; fine-tune only if it misses Plan, cost, consent basis, hardware Approve Verify every API against installed source, then write the glue vakforge validate vakforge.jsonl Train only what recommend asked for, then eval base vs tuned Report, and an endpoint only if it beats the base

Package layout

Everything below exists today, runs on CPU and imports no ML dependencies.

Module What it holds
cli.py typer app, one subcommand per stage; thin, delegates everything
config.py vakforge.yaml loading, pydantic settings, CLI override merge
schema.py the canonical models: Conversation, Turn, ToolCall, Meta, and the JSON Schema export
validate.py the file-level rules from DATA_FORMAT.md, returned as structured errors
locales/ base.py pack protocol and registry, checksums.py, then one module per pack
inspect/ sources.py classifies, profile.py gets per-kind facts, report.py summarises the folder
recommend/ rules.py: the decision guide as data, plus the bars and their evidence

Outside the package: skill/vakforge/ (the agent skill), site/ (landing page and these docs), examples/ (a synthetic project with its reports), tests/unit/ (CPU, no downloads).

The stages the skill generates — prepare, synth, train, eval, serve — are written into your project, not shipped here. docs/ROADMAP.md tracks which of them vakforge may ship itself later; the sections below are the design they would follow.

Key interfaces

python
class Adapter(Protocol):
    name: str

    def build(self, manifest: Path, out_dir: Path, cfg: AdapterConfig) -> AdapterOutput: ...

    # AdapterOutput: out_dir, manifest_hash, stats (rows kept/dropped and why)


class Recipe(Protocol):
    name: str
    extra: str  # uv extra that provides deps

    def check_env(self) -> list[EnvIssue]: ...  # missing deps, GPU, gated weights
    def train(self, dataset: AdapterOutput, cfg: TrainConfig) -> TrainResult: ...
    def load_for_eval(self, checkpoint: Path | None) -> Inferencer: ...  # None = base model
    def serve(self, checkpoint: Path | None, cfg: ServeConfig) -> StreamingBackend: ...


class Inferencer(Protocol):
    def respond(self, session: EvalSession) -> EvalTurnResult: ...

    # returns text, audio (24 kHz), tool_calls, timings (ttft, ttfa, total)


class StreamingBackend(Protocol):
    async def append_audio(self, pcm16: bytes) -> None: ...
    async def commit(self) -> None: ...
    async def responses(self) -> AsyncIterator[RealtimeEvent]: ...
    async def tool_result(self, call_id: str, content: dict) -> None: ...

None of this is built yet. The point of writing it down now is the constraint it puts on the rest: eval and serve are to depend on these protocols and nothing else, so that a recipe implementing them gets the full report and every protocol front end without touching either.

Data flow

Data flow, from raw files to a served endpoint A vertical pipeline. Documents, tables, chats and audio go into inspect, which writes inspect.json. recommend reads it and returns a recipe and the data gaps. prepare takes any source, and synth fills the gaps; both write the one canonical manifest, data/vakforge.jsonl. From there an adapter builds the recipe's dataset, the recipe trains and writes a checkpoint, the eval runner compares base against tuned and writes a report, and serve exposes a realtime WebSocket endpoint. inspect and recommend ship in the CLI today; the rest is generated per project. documents · tables chats · audio any source inspect classify and profile every file recommend what to change, and the evidence prepare synth data/vakforge.jsonl one canonical format in the middle adapters/<recipe>.build() recipes/<recipe>.train() eval.runner base vs tuned, same held-out split serve.realtime_ws inspect.json recipe + data gaps runs/<ts>/checkpoint + train.json runs/<ts>/ report.{md,json} ws://…/v1/realtime in the CLI today generated per project canonical format

Serving: protocol front ends

The model backend (StreamingBackend) knows nothing about the wire. Each protocol is a thin front end in serve/protocols/ that translates its messages into append_audio / commit / responses / tool_result. Every model served is open and local; "OpenAI Realtime compatible" names a message format, not a dependency.

Front end Why Status
OpenAI Realtime WebSocket de facto shape for voice agents; GPT Realtime users change one URL; Pipecat, LiveKit, Twilio clients work unchanged first
WebRTC (LiveKit / Pipecat transports) browsers and mobile, lowest latency, echo cancellation for free next
SIP telephony, the call-centre use case next
HTTP one turn per request for batch and simple integrations planned
Gemini Live format teams on Google's stack on request

A new front end ships with its own event list in serve/README.md and a contract test against a fake backend.

OpenAI Realtime WebSocket subset

We implement a documented subset, enough for common clients:

  • Client → server: session.update, input_audio_buffer.append, input_audio_buffer.commit, input_audio_buffer.clear, response.create, response.cancel, conversation.item.create (for function_call_output).
  • Server → client: session.created, input_audio_buffer.speech_started/stopped (VAD), response.created, response.audio.delta, response.audio_transcript.delta, response.function_call_arguments.done, response.done, error.
  • Audio: PCM16 24 kHz base64, matching the common default.

Unsupported events return a structured error naming the event. The exact list lives in serve/README.md and is tested.

Run directory

runs/2026-09-16T10-12-00_lfm25-audio/
  config.yaml            # fully resolved
  manifest_hash.txt
  versions.txt           # pip freeze of the recipe env
  train.log
  checkpoint/            # adapter or full weights
  report.md / report.json