Audio Programming References — Where Web Audio API Meets Music Theory

#Introduction

sakimytocom currently has around 200 interactive components. Of those, 11 deal with sound.

Piano, Guitar, Kalimba, Theremin, Drum Machine, Wind Chimes, Music Box, Metronome, Color Organ, Music Visualizer, Sound Waves — these aren't the kind of components where you just build a UI and you're done. They generate sound in real time, calculate frequencies, control timbre with envelopes, and battle the browser's Autoplay Policy. This is territory that HTML and CSS can't reach.

This article covers what I learned while implementing these 11 music components and the books and resources that informed that learning. It's less a book review or technical explainer and more a practitioner's log: "What I read, what I built, and where I got stuck."

Audio programming looks intimidating at first. But in retrospect, the fundamental concepts are surprisingly few: AudioContext, OscillatorNode, GainNode, and envelopes — understand these four and you can build instruments in the browser. The problem is that it took quite a few detours to reach these four.

#Core Concepts of Web Audio API

The design of the Web Audio API can be summed up in one phrase: a "connect the nodes" model. Audio signals flow through a graph.

AudioContext → OscillatorNode → GainNode → destination (speaker)

This three-stage setup is the foundation of everything. OscillatorNode generates sound, GainNode controls volume, and destination outputs to the speaker. Simple, but this concept of a "graph-based signal flow" underpins all audio effect processing.

The most important implementation principle is "create one AudioContext and reuse it."

let audioCtx: AudioContext | null = null

function getAudioContext(): AudioContext {
  if (!audioCtx) {
    audioCtx = new AudioContext()
  }
  if (audioCtx.state === 'suspended') {
    audioCtx.resume()
  }
  return audioCtx
}

Creating an AudioContext is expensive. If multiple components each create their own AudioContext, it strains browser resources. Furthermore, some browsers limit the number of simultaneous AudioContexts.

Another ironclad rule is to always defer initialization. Don't create the AudioContext on component mount — initialize it on the user's first interaction (click or tap). This stems from an iOS Safari limitation — AudioContext remains fixed in a suspended state unless created within a user gesture's call stack.

This lesson was expensive. I once had to bulk-fix AudioContext-related bugs across 18 components. The most common pattern was "initializing AudioContext in a component's useEffect, resulting in no sound on iOS." The fix was simple: move initialization into an event handler and wrap it in try-catch. But the 18-file fix would have been unnecessary if I'd known the correct pattern from the start.

#Implementation Approaches by Instrument

The 11 music components fall into three implementation patterns.

1. Pitched instruments: Piano, Guitar, Kalimba, Theremin, Music Box, Wind Chimes
2. Rhythm instruments: Drum Machine, Metronome
3. Sound visualization: Sound Waves, Music Visualizer, Color Organ

Each uses the Web Audio API in completely different ways.

#Piano: Equal Temperament and Envelopes

The first thing Piano's implementation requires is the mapping between keys and frequencies. This is determined by a mathematically elegant rule called equal temperament.

function noteToFrequency(note: number): number {
  return 440 * Math.pow(2, (note - 69) / 12)
}

A4 (MIDI note number 69) is 440Hz. Each semitone up multiplies the frequency by 2^(1/12). One octave (12 semitones) up is exactly double. The formula itself is understandable with basic math.

But this formula alone won't produce a "piano-like sound." The raw square wave from OscillatorNode is pure electronic tone. What creates the "piano feel" is the ADSR envelope — the volume change curve from key press to release.

Attack (0.01s)  → Rapid rise to maximum volume
Decay (0.1s)    → Slight volume decrease
Sustain (0.3)   → Volume level while key is held
Release (0.3s)  → Decay after key release

Whether Attack is 0.01 seconds or 0.1 seconds creates a completely different instrument impression. Struck instruments like piano have extremely short Attack. Bowed instruments like strings have long Attack. The beauty of ADSR is that just four parameters can express the "feel" of an instrument.

The implementation key is to use GainNode's linearRampToValueAtTime() or exponentialRampToValueAtTime(). Rather than manually changing gain with setInterval, delegate to Web Audio API's scheduling system. Audio processing runs on a separate audio thread from the main thread, so relying on JavaScript timers causes timing drift.

#Theremin: Continuous Frequency Control

Theremin is the component where Web Audio API's "real-time" capabilities truly shine. Mouse movement becomes sound directly.

X-axis maps to pitch (frequency), Y-axis to volume. When mapping pointer position to frequency, use an exponential scale rather than linear.

const minFreq = 65   // C2
const maxFreq = 2093 // C7
const frequency = minFreq * Math.pow(maxFreq / minFreq, normalizedX)

The human ear perceives frequency logarithmically. The difference between 200Hz and 400Hz (one octave) feels smaller as a "pitch difference" than between 200Hz and 600Hz (one octave plus a perfect fifth), even though the Hz difference is the same 200Hz. Exponential mapping ensures that equal screen distance translates to equal perceived pitch change.

Another important technique I learned from Theremin is OscillatorNode's frequency.setTargetAtTime(). Switching frequencies instantly every time the mouse moves produces unpleasant clicking sounds (pop noise). Applying slight smoothing with setTargetAtTime achieves smooth frequency transitions. This is also closer to how a real Theremin behaves.

#Kalimba: Decay and Harmonics

Reproducing Kalimba's metallic timbre requires mimicking its harmonic structure.

function playKalimbaNote(frequency: number) {
  const ctx = getAudioContext()
  const harmonics = [
    { ratio: 1, gain: 1.0 },    // Fundamental
    { ratio: 2, gain: 0.6 },    // 2nd harmonic
    { ratio: 3, gain: 0.2 },    // 3rd harmonic
    { ratio: 5.4, gain: 0.08 }, // Non-integer harmonic (metallic quality)
  ]
  // Generate each harmonic as an OscillatorNode with individual GainNode volume control
}

Perfectly reproducing a real instrument's vibration modes is difficult, but by layering integer and non-integer frequency components relative to the fundamental, you can create a "convincing timbre." Adding a small amount of non-integer harmonics in particular creates a metallic shimmer. This is also what explains the timbral difference between a xylophone and a kalimba.

Another characteristic of kalimba is its relatively fast decay. Maximum volume at the moment of plucking, fading in 1-2 seconds. In envelope terms: extremely short Attack, zero Sustain, dominant Release. A different curve from piano.

#Reference Book: Web Audio API (Boris Smus)

The first book to read in this field is Boris Smus's "Web Audio API" (O'Reilly).

It's thin — only about 120 pages. Yet it's ideal for grasping Web Audio API's design philosophy and overall picture. The author is a Google engineer who was involved in the API specification. Throughout the book, there are explanations that make the "why it's designed this way" click.

The following chapters were particularly useful.

Chapter 2: Perfect Timing and Latency — Explains Web Audio API's scheduling model. Why currentTime-based absolute time specification is more accurate than setTimeout. Without this chapter's knowledge, rhythm drift occurs in Drum Machine and Metronome implementations.

Chapter 3: Volume and Loudness — Covers not just GainNode usage but volume control methods based on human auditory characteristics (Weber-Fechner law). Explains why you should think in decibel scale rather than linear gain values.

Chapter 5: Analysis and Visualization — Methods for frequency analysis using AnalyserNode. Music Visualizer and Color Organ were directly based on this chapter. The pattern of rendering spectrum data obtained from getByteFrequencyData() to Canvas was learned here.

The graph-based signal flow design philosophy originates from hardware modular synthesizers. The "connect the nodes" model feels intuitive because it's a metaphor for physically plugging and unplugging cables. After learning this concept, I could think about "where to insert a node" when adding new effects.

#Intersections with Music Theory

Building 11 components repeatedly required basic music theory knowledge. Particularly important were "equal temperament," "harmonics," and "consonance vs. dissonance."

#Equal Temperament vs. Just Intonation

The equal temperament used in the Piano implementation is a tuning system where all semitone intervals are equal. It allows free modulation and is used by most modern keyboard instruments.

In contrast, just intonation defines intervals as integer ratios. A perfect fifth is 3:2, a major third is 5:4. Mathematically clean, and chord harmony sounds more beautiful than equal temperament. But it breaks down on modulation.

I felt this difference when playing chords on Piano. An equal temperament C major chord (C-E-G) theoretically "produces slight beating." Just intonation would be perfectly harmonious. Yet when I actually tried it with OscillatorNode, the difference was almost imperceptible to an untrained ear. This "gap between theory and perception" is interesting. For implementation purposes, equal temperament works fine.

#Harmonic Content Determines Timbre

The same 440Hz sounds completely different on piano, flute, and clarinet. The difference is explained by harmonic content — which frequency components at what intensity are present relative to the fundamental.

• Flute: Nearly a pure sine wave. Almost no harmonics
• Clarinet: Strong odd harmonics (3x, 5x, 7x...)
• Violin: Many harmonics in good balance

This knowledge directly maps to OscillatorNode waveform selection. sine is close to flute, square to clarinet, sawtooth to strings. Of course, real instruments aren't this simple, but it's a sufficient starting point.

When I implemented the FourierSeries component, I visualized this relationship. Decomposing a square wave into its Fourier series reveals only odd harmonics. Writing the formula sin(x) + sin(3x)/3 + sin(5x)/5 + ... isn't intuitive, but watching the superposition process animated makes "a square wave is the sum of infinite sine waves" click. It was the best possible visualization of harmonics as the essence of timbre.

#The Nature of Code + Math Girls

Two books that aren't directly about audio programming but had a significant indirect influence.

#The Nature of Code (Daniel Shiffman)

Chapter 3 of this book, "Oscillation," was helpful for understanding the physical basis of sound. The three parameters of a sine wave — amplitude, frequency, and phase — correspond directly to OscillatorNode's properties.

The Sound Waves component is an extension of this chapter. It superimposes multiple sine waves and visualizes the waveform. It can also play as sound. The experience of "the waveform you see is exactly what you hear as sound" is a powerful intersection of programming and physics.

Music Visualizer was also heavily influenced by this book. Visualizing audio signals in both time domain and frequency domain — AnalyserNode's getByteTimeDomainData() for waveform, getByteFrequencyData() for spectrum. The intuition from Shiffman's book that "oscillation is a function of time and space" paid off here.

#Math Girls (Yuki Hiroshi)

The Fourier transform chapter of Math Girls provided intuition for "why frequency analysis works."

A time-domain signal and a frequency-domain signal are different representations of the same information — this duality concept is at the heart of Music Visualizer and Color Organ's design philosophy. When you play a song, you see the waveform and the spectrum simultaneously. The same music displayed in two different forms. The understanding that information isn't being "lost" but "transformed" came from Math Girls.

Yuki Hiroshi's writing lets you experience abstract mathematical concepts as a "story." The passage deriving the Fourier transform from the relationship between differential equations and inner products was far more understandable than reading a textbook. For practitioners, math is a tool, but knowing the philosophy behind the tool expands how you can use it.

#AudioContext Pitfalls

From the experience of bulk-fixing AudioContext-related bugs across 18 components, here are the common patterns.

iOS Safari: User Gesture Required — The most common bug. Initializing AudioContext in useEffect leaves it fixed in suspended state on iOS. The fix is to move initialization into click or touch handlers.

Chrome: Autoplay Policy — Chrome has blocked audio playback without user interaction since 2018. Checking AudioContext.state === 'suspended' and calling resume() is mandatory in every component.

OscillatorNode is Disposable — An OscillatorNode that has had stop() called cannot be reused. You need to create a new OscillatorNode for each new sound. Initially, I didn't know this and got an error calling start() twice.

Direct Assignment to GainNode.gain.value Causes Pop Noise — Setting gain.value = 0 directly causes clicking from the abrupt volume change. Use gain.linearRampToValueAtTime(0, ctx.currentTime + 0.01) to add a short fade instead. 0.01 seconds is sufficient.

Memory Leak: Forgetting to disconnect — Not calling disconnect() on nodes when a component unmounts prevents garbage collection. This is especially critical for components like Music Visualizer that continuously read AnalyserNode with requestAnimationFrame — cleanup is mandatory.

The common thread among these bugs is that they stem from "cross-browser behavior differences" and "Web Audio API design constraints." Some could be known from reading API documentation, but many don't sink in until you actually encounter them as bugs.

#Choosing Between use-sound (Howler.js) and Web Audio API

sakimytocom uses two different approaches for sound playback.

use-sound (Howler.js) — Plays pre-recorded WAV files. Used for short sound effects like BubbleWrap's pop sound and button clicks. File sizes are small (a few KB) and playback latency is negligible. Implementation is a single line: useSound('/sounds/pop.wav').

Web Audio API (OscillatorNode) — Synthesizes sound in real time. Used for instruments that need to dynamically generate sound based on user input, like Piano and Theremin. WAV files can't handle this — preparing WAV files for all 88 piano keys is impractical, and continuous frequency changes like Theremin are impossible with file playback.

The boundary between these approaches is clear: "Is the sound predetermined, or does it need to be generated in real time?" Recorded playback and real-time synthesis are fundamentally different programming models.

Howler.js's advantage is that the library absorbs cross-browser compatibility differences. Web Audio API's low-level control is essential for instruments, but creating an OscillatorNode just to play a click sound is overkill. Use the right tool for the job.

#Summary

Web Audio API is the "Canvas of sound." Just as Canvas API gives you the ability to "freely manipulate pixels," Web Audio API gives you the ability to "freely generate and process sound."

And in the world of sound, math determines timbre. Frequency ratios define pitch, harmonic composition defines timbre, and envelope curves define an instrument's "feel." Behind the single line 440 * Math.pow(2, (note - 69) / 12) lies a history of music theory stretching back to Pythagoras.

After 11 music components and AudioContext bug fixes across 18 components, I feel I've grasped the fundamentals of audio programming. The main resources I used were these three:

• Web Audio API (Boris Smus): API design philosophy and overall picture
• The Nature of Code (Daniel Shiffman): Physical intuition for oscillation and waves
• Math Girls (Yuki Hiroshi): Fourier transforms and the duality of frequency

One book, two references, and 11 implementations. As an entry point to audio programming, that was enough. All that's left is to pick an instrument you want to build and start connecting OscillatorNodes. At first, you'll only get electronic sounds, but as you adjust envelopes and layer harmonics, there comes a moment when a "convincing sound" emerges. That moment is the greatest reward of audio programming.