Skip to content
View as Markdown

Audio

Inkling natively supports audio input, which you can combine with other modalities such as text and tool calls in the same conversation. Common uses include:

  • Speech transcription (ASR) — “Transcribe this speech exactly.”
  • Audio question answering — “What topic are the speakers discussing?”
  • Speaking-style classification — “Is the speaker whispering, laughing, or speaking normally?”
  • Information extraction — “List the names and dates mentioned in this recording.”

The Cookbook audio recipes demonstrate complete ASR and speaking-style workflows, including sampling, supervised fine-tuning, reinforcement learning, and evaluation.

Use either a native AudioPointer or an OpenAI-compatible input_audio part. tml-renderers decodes and DMel-encodes the audio on your client before the model request.

Native chat messages

Use AudioPointer to refer to a local file:

from tml_renderers import chat

user = chat.Author(chat.AuthorKind.User)
audio = chat.AudioPointer(
    location="speech.wav",
    format=chat.AudioFormat.Wav,
    num_frames=48_000,
    sample_rate=16_000,
)
messages = [
    chat.Message(content=chat.Text("Transcribe this."), author=user),
    chat.Message(content=audio, author=user),
]

num_frames is the number of source audio frames—not the number of DMel tokens. sample_rate describes the source file.

OpenAI-compatible Cookbook messages

For OpenAI-compatible messages, put base64-encoded audio in an input_audio content part:

audio_base64 = "..."  # Base64-encoded WAV bytes
audio_part = {
    "type": "input_audio",
    "input_audio": {"data": audio_base64, "format": "wav"},
}
text_part = {"type": "text", "text": "Transcribe this."}
messages = [{"role": "user", "content": [text_part, audio_part]}]

For WAV input, tml-renderers reads the frame count and sample rate from the WAV header. MP3 and FLAC inputs must include num_frames and sample_rate:

{
    "type": "input_audio",
    "input_audio": {
        "data": audio_base64,
        "format": "mp3",
        "num_frames": 48_000,
        "sample_rate": 24_000,
    },
}

The Cookbook Inkling renderer accepts these dictionaries directly.

Accepted audio input

  • Formats: WAV, MP3, and FLAC.
  • Locations: Local file paths for native AudioPointer; base64 audio for OpenAI-compatible input_audio.
  • WAV metadata: Frame count and sample rate can be read from a valid WAV header.
  • MP3 and FLAC metadata: Provide num_frames and sample_rate.
  • Remote media: HTTP and cloud-storage URLs are not fetched. Download the file in your application first.

The encoding is eager and client-side. By the time build_generation_prompt(...) returns, the audio is ready to send to Inkling.

Client-side audio encoding

AudioPointer holds the source location and metadata. When you build the generation prompt, tml-renderers:

  1. Reads the local or inline audio.
  2. Decodes and resamples it.
  3. Encodes it as quantized mel features (DMels).
  4. Places those features into the model-ready prompt.

You do not need to compute DMels yourself. Your application supplies audio bytes plus the correct format and metadata.

Preparing speech input

Mono 16 kHz WAV is a simple interchange format for speech datasets. Other accepted source rates are decoded and resampled by the client encoder.

Run the example

sample_audio.py demonstrates the complete sampling flow:

uv run python -m tinker_cookbook.scripts.inkling.sample_audio
uv run python -m tinker_cookbook.scripts.inkling.sample_audio message_format=chat

Pass audio_path=/path/to/audio.wav to use your own file.

Learn more