EXPERIMENT_045 // ON-DEVICE.PRESENCE.SENSING
A local, open-weight `Silero VAD` model runs entirely in your browser via `onnxruntime-web`, reading your voice's presence and rhythm frame by frame with no cloud round-trip. This demonstrates the same physical-digital feedback loop Diana builds for experiential installations, driving a `p5.js` generative canvas in her brand palette on genuine local inference rather than an API call.
LOADING EXPERIMENT...
EXP_011 (Spatial Brand Mirror) proved a webcam-to-canvas feedback loop but leaned on a cloud vision API for every frame. This experiment asks the sharper question: can the same physical-digital installation feel come from a model that never leaves the browser tab? `Silero VAD` (MIT-licensed, ~2MB, via `@ricky0123/vad-web`) is a genuine open-weight neural network — not an amplitude meter — that returns a continuous speech-probability score for every ~32ms frame of audio, running on `onnxruntime-web`'s WASM backend with zero network calls after the one-time asset load. The visitor's voice presence and rhythm — not what they say, not any classification of ambient sound — becomes the entire input to a generative canvas tuned to Diana's brand palette. This is the local-open-weight-model half of Diana's dual positioning made concrete, sitting next to EXP_011's cloud-vision approach rather than replacing it.
HOW IT WORKS
`@ricky0123/vad-web` loads Silero VAD's "v5" ONNX weights (~2.2MB) and `onnxruntime-web`'s WASM runtime (~13MB) as same-origin static assets on first load — vendored into `public/vad/`, not fetched from a CDN, so the experiment's CSP never needs a third-party `connect-src` exception. `MicVAD.new()` performs this load without requesting microphone permission; permission is requested only on an explicit "Enable microphone" click, matching this Playground's existing camera-permission pattern from EXP_011. Once running, an `AudioWorklet` delivers 512-sample frames (32ms at 16kHz) to the model and every frame's `isSpeech` probability reaches the app via `onFrameProcessed` — the entire signal this experiment is built on.
Every frame updates a `PresenceDescriptor`: `intensity` (a fast exponential moving average of the raw probability), `presenceLevel` (a slower, ~8-second-window average, bucketed absent/intermittent/present/sustained), `cadence` (how many confirmed speech onsets have landed in the last 15 seconds — silent/sparse/steady/rapid), and `pattern` (whether the current or most recently ended speech segment reads as a short burst or a sustained run). All four are rolling summaries of two VAD-native signals — the continuous probability stream and its hysteresis-gated start/end events — never a typed event. Silero VAD does not identify what kind of sound is occurring; it only knows whether speech is present, and with what confidence, continuously. The descriptor is honest about that ceiling rather than dressing it up as more than it is.
`PresenceCanvas.tsx` is the same lerp-between-states `p5.js` particle system as EXP_011's `BrandCanvas.tsx` — 1.5-second continuous interpolation between descriptor states, colours resolved from CSS custom properties at runtime, the identical `prefers-reduced-motion` static-wash fallback. What differs is the mapping: `intensity` drives particle velocity, `presenceLevel` drives particle density, `cadence` drives flow-field turbulence, and `pattern` switches the rendering mode itself — ambient drift during silence, a radial burst on a fresh speech onset, continuous flowing trails through a sustained run. There is no directional vector, because audio presence has no direction to render; that's an intentional omission, not a missing feature.
WHAT THIS PROVES
The local-model half of a physical-digital installation claim can be made good on inside a browser tab, not just asserted in a pitch: a real, MIT-licensed neural network runs client-side via WASM, with the browser's own Network panel showing zero requests per analysis cycle after the one-time asset load — the same verifiable-not-claimed standard this repo holds its cost and latency claims to elsewhere.
A coarse, continuous signal is enough to drive a convincing generative system, the same architectural lesson EXP_011 established for vision: `Silero VAD`'s single `isSpeech` probability, rolled up into four derived fields (intensity, cadence, pattern, presenceLevel), produces a canvas with genuine range — calm ambient drift, sharp onset pulses, flowing sustained motion — without ever needing to know what kind of sound it is hearing, only that a voice is present and how it is moving through time.