EXPERIMENT_001 // VOICE.PARTICLE.INSTRUMENT
Microphone input drives GPU-accelerated particles and terrain mesh deformation in real-time. Speak, sing, or make noise to shape the visual landscape.
LOADING EXPERIMENT...
150,000 GPU-instanced particles respond to your microphone in real time. Frequency bands from the audio signal drive particle groups in different directions and colours — bass down, treble up, mids left and right — turning your voice into a visible, physical force. The experiment maps the Web Audio API directly to per-particle physics on the GPU.
HOW IT WORKS
When you grant microphone access, the browser captures your audio stream and feeds it through a Web Audio AnalyserNode running a Fast Fourier Transform at 1024 samples. That FFT splits your audio into 512 frequency bins — a real-time snapshot of how much energy exists at each pitch, updated every frame. The raw frequency data is normalised and passed to the GPU as a uniform array, which means the shader reads fresh audio data every frame without recompilation.
The 512 bins are grouped into four bands: bass (0-375 Hz), low-mid (375 Hz-1.9 kHz), high-mid (1.9-6 kHz), and treble (6 kHz and above). Each band drives a directional force on a different subset of the 150K particles. Bass energy pushes particles downward and turns them red. Low-mid pushes left, colouring them chartreuse. High-mid pushes right in cyan. Treble drives particles upward and fades them to white. The result is that a spoken sentence, a clap, and a whistle each produce visually distinct formations — you can see the spectral shape of the sound.
Without audio input, the particles hold a calm breathing sphere — a slow sine-wave oscillation that keeps the scene alive without being distracting. The moment the microphone detects signal above a threshold, the system transitions to active mode: per-band forces override the idle oscillation and the sphere explodes into a directional storm. When audio drops back below the threshold, particles gradually return to the sphere formation.
Below the particle field, a subdivided plane mesh acts as a terrain surface. Its vertex positions are displaced in real time by the same FFT data — each vertex samples a frequency bin based on its grid position, so the mesh ripples and deforms in sync with the audio. The terrain provides a grounded visual anchor while the particles move freely above it.
WHAT THIS PROVES
This is a GPU instancing stress test running in the browser. 150,000 individually animated particles — each with its own position, velocity, colour, and physics response — render at 60fps on commodity hardware using WebGL2 instanced buffer geometry. No compute shaders, no WebGPU, no native code. The bottleneck in browser-based particle systems has historically been the CPU-to-GPU data bridge; this experiment keeps all per-particle state on the GPU and only sends the audio uniform data (2KB per frame) across the bridge each tick.
It also demonstrates that the Web Audio API's AnalyserNode is fast enough to drive real-time visual feedback at animation-loop speeds — a capability that matters for any application mapping audio to visuals, from music tools to accessibility interfaces. The experiment runs entirely client-side with no server dependencies.