Audio playback and FFT

6 min read

Import system.audio to play WAV, MP3, and WebM/Opus on native desktops and in browsers. Format detection inspects the bytes; the file extension is not required to match. WebM decoding selects the first Opus track and ignores video tracks. The initial implementation supports mono and stereo.

Play a file

import system;
import system.audio;

async fun main(): void {
    let audio = Audio("music.mp3");
    audio.volume = 0.5f;
    let started = audio.play().await;
    if started.is_err() {
        System.println(started.unwrap_err().message());
        audio.close();
        return;
    }

    while !audio.ended && audio.error.is_none() {
        for (let event in audio.poll_events()) {
            switch (event) {
                AudioEvent.Error(_, error, operation, _) => {
                    System.println(operation + ": " + error.message());
                }
                _ => {}
            }
        }
        Time.delay(16).await;
    }
    audio.close();
}

Audio() creates an empty player. Assign src before playing. Constructors do no file I/O or device initialization. Audio(bytes) copies encoded bytes; Audio(buffer) uses an already decoded AudioBuffer. Call load().await to prepare a source without playing it. play().await prepares it if necessary and returns when output starts. Node and standalone WASI support buffer decoding and FFT analysis; speaker output is unavailable there.

pause() preserves position. play().await resumes. Set current_time in seconds to seek; streamed sources seek as described under URLs and limits. duration is absent until loading completes. volume ranges from zero to one; muted preserves its value. loop repeats a loaded source. close() is terminal and idempotent. Keeping a player alive keeps its playback resources alive; dropping it cancels loading and releases output.

URLs and limits

String sources also accept HTTP and HTTPS URLs. Native and Node requests use Dream's HTTP client; non-success HTTP responses produce EHTTP. Browser URLs use the browser's media element for loading, decoding, buffering, and seeking. Browser requests follow media permissions and CORS rules; cross-origin servers must allow the application origin so rendered samples can be visualized. Browser media load failures report EIO. Replacing src, calling load again, or closing cancels the prior load; its completion cannot overwrite the replacement source.

Native file and HTTP sources stream: playback starts once the first 256 KiB (or the whole source, if smaller) has arrived and a quarter second of audio has been decoded, and memory use does not grow with track length. load() reads only enough to learn the format, channel count, and (when the container declares it) the duration. Streamed MP3 files, and WebM files without a declared duration, keep duration absent; playback and Ended still work. Looping requires a seekable source.

If data arrives more slowly than it plays, output pauses with silence, Waiting is reported, and Playing follows once data resumes. Local file seeking reopens and decodes up to the target. Native and Node HTTP seeking currently supports PCM and IEEE-float WAV through validated frame-to-byte positions from dr_wav. The server must advertise byte ranges, supply a strong ETag or Last-Modified validator and Content-Length, and return the requested range of the same resource. Unsupported formats or servers report ENOTSUP; ignored or inconsistent range responses report ESEEK. Failure preserves playback at its previous position. No full download is performed to make a remote source seekable. Browser URLs use the media element's seekable ranges and the browser's container indexes and HTTP range implementation; supported formats depend on the browser's installed codecs. A seek on a stream is asynchronous: seeking stays true until Seeked arrives, and assigning current_time again before then reports EBUSY. A target beyond a known duration reports EINVAL.

The browser controls URL buffering independently of AudioOptions. Encoded bytes, local file sources, and offline AudioBuffer.decode still use the portable decoders.

Decoding never runs on the program's own thread. Native programs decode on a background thread; browser and Node programs decode in a dedicated worker that needs no cross-origin isolation headers. A streamed source decodes the next chunk ahead while the current one plays, so a busy program does not starve the speaker as long as it keeps awaiting.

import system.audio;

let options = AudioOptions();
options.max_compressed_bytes = 16 * 1024 * 1024;
options.max_decoded_bytes = 64 * 1024 * 1024;
let audio = Audio("https://example.com/song.webm", options);

For streamed sources, max_compressed_bytes (default 8 MiB) bounds the encoded data held while decoding, not the track length. A file whose headers or a single WebM element cannot be decoded within it fails with ELIMIT. buffer_seconds (default one second, at most 30) sets how much decoded audio is queued ahead of the speaker. max_decoded_bytes (default 256 MiB) applies to byte sources, which are decoded completely; a larger result returns ELIMIT. Limits are snapshotted by the constructor.

Receive events

poll_events() drains a typed queue. Events contain monotonic timestamps in nanoseconds. Play reports an accepted start and Playing reports running output. Seeking and Seeked surround an accepted seek. Pause, Ended, VolumeChange, SinkChange, and Error report the corresponding transitions. State properties remain queryable if events are not being polled.

Invalid property assignments preserve the previous value and enqueue Error. Fallible methods return Result; they may also enqueue an operation error. Events remain readable after close. Repeated pause/close calls do not duplicate those transitions. The queue holds 1,024 events; overflow discards the oldest and adds an EventsLost record on the next drain. Position events are limited to four per second and are generated while state is observed. Poll once per UI frame.

Waiting reports a streaming underrun; byte and buffer sources never produce it. Current device-loss detection observes a stopped native device or closed browser context. Browser permission changes and all forms of hardware hotplug have not been qualified on every platform.

Choose an output

Each player has its own sink_id; an empty ID uses the system default. Players using the same device share a mixer internally. Switching preserves the source position, volume, mute, and looping. A failed switch leaves the old output active.

import system.audio;

async fun select_output(audio: Audio): Result<Unit, AudioError> {
    let devices = Audio.devices().await?;
    if devices.length == 0 {
        return Result.Err(AudioError("ENODEVICE", "no output device is visible"));
    }
    return audio.set_sink_id(devices[0].id).await;
}

IDs are opaque and host-specific. Enumerate again rather than assuming an old ID still identifies the same device. Browsers may hide labels until permission has been granted. Audio.request_device().await opens the browser's output picker and must originate from a suitable user action in a secure context. Native programs use enumeration; request_device returns ENOTSUP. Unsupported browser output selection and denied permission return errors. Starting audio also requires browser user activation.

Inspect PCM and FFT data

Offline analysis does not open an audio device or include playback or decoders. The following produces a single-sided spectrum for a generated tone:

import system;
import system.audio;

fun main(): void {
    let samples = Array.alloc<float>(2048);
    for (let i = 0; i < samples.length; i = i + 1) {
        samples[i] = (float)Math.sin(6.283185307179586d * 440.0d * (double)i / 48000.0d);
    }
    let analyzer = AudioAnalyzer();
    let result = analyzer.analyze(samples, 48000, 1);
    if result.is_err() {
        System.println(result.unwrap_err().message());
        return;
    }
    let analysis = result.unwrap();
    System.println("Peak: " + analysis.peak + ", RMS: " + analysis.rms);
    System.println("Bin 19 frequency: " + analysis.frequency_hz(19).unwrap());
}

Samples are interleaved: stereo frames are left, right, left, right. The analyzer averages channels to mono, uses the latest FFT-sized block, and pads short input with silence. Sample rate must be positive, frames complete, and samples finite. It defaults to 2,048 samples and a Hann window. set_fft_size accepts powers of two from 32 through 32,768; AudioWindow.Rectangular disables tapering.

For FFT size N, results contain N / 2 + 1 bins at bin * sample_rate / N Hz. real and imaginary are unnormalized complex components after windowing. magnitudes divides complex magnitude by the sum of window weights and doubles all bins except DC and Nyquist. A unit, bin-centered sine therefore has amplitude one. magnitudes_db uses 20 * log10(amplitude), floored at -120 dBFS. Peak and RMS use the unweighted, padded waveform.

AudioBuffer.from_pcm copies supplied PCM. read_pcm(start_frame, frame_count) returns a checked copy.

AudioBuffer.decode(bytes).await detects WAV, MP3, or WebM/Opus content without opening a device. It copies the input and decodes on a background thread (a worker on the web), so timers and input keep running during a long decode. max_bytes (default 256 MiB) limits the decoded PCM; pass a CancellationToken to stop waiting:

import system;
import system.audio;
import system.io;

async fun main(): void {
    let bytes = File.read_bytes("clip.wav").await;
    if bytes.is_err() {
        System.println("could not read clip.wav");
        return;
    }

    let decoded = AudioBuffer.decode(bytes.unwrap()).await;
    if decoded.is_err() {
        System.println(decoded.unwrap_err().code() + ": " + decoded.unwrap_err().message());
        return;
    }

    let buffer = decoded.unwrap();
    System.println("Decoded " + buffer.duration + " seconds");
}

Unsupported or invalid input returns EFORMAT, audio over the limit or with more than two channels returns ELIMIT, an exhausted decoder returns ENOMEM, an unavailable decoder thread returns EBUSY, and a cancelled wait returns ECANCELLED. A cancelled decode finishes in the background and its result is discarded.

audio.capture(frame_count) lazily enables a bounded tap after this player's gain and mute and before mixing with other players. It returns stereo PCM at the output sample rate. The first calls can return an empty snapshot while the tap starts. Capture accepts at most 32,768 frames; frame_position counts rendered frames since capture began. overrun reports dropped frames or a concurrent update that prevented a consistent snapshot. Browser capture uses a bounded pool of transferable buffers and requires no shared memory.

Deployment

In optimized native deployments and browser WASM, an unused system.audio import adds no audio code, hosts, or assets. Native debug binaries still retain unused type metadata; byte-identical trimming there is not implemented yet. PCM playback excludes decoders and FFT; analysis-only code excludes output hosts and decoders. Automatic encoded-audio decoding includes all supported codecs because the source format is discovered at runtime. String-source playback includes file/HTTP routing.

The Audio API reference lists declarations.