ICMC 2015 – Sept. 25 - Oct.

1, 2015 – CEMI, University of North Texas

Spatial Modulation Synthesis

Ryan McGee
Media Arts and Technology
University of California, Santa Barbara
[email protected]

ABSTRACT Sound Source

In 1962 Karlheinz Stockhausen’s “Concept of Unity in Elec- Bounded Periodic

tronic Music” introduced a connection between the param- Motion
eters of intensity, duration, pitch, and timbre using an ac-
celerating pulse train. In 1973 John Chowning discovered Source
that complex audio spectra could be synthesized by increas- Frequency

ing vibrato rates past 20Hz. In both cases the notion of in-
creased speed to produce timbre was critical to discovery.
Although both composers also utilized sound spatialization Source Source
in their works, spatial parameters were not unified with their Speed Position
synthesis techniques. Spatial Modulation Synthesis is intro-
duced as a novel, physically-based control paradigm for audio-
Spatialization Doppler Gain Amplitude
visual synthesis, providing unified control of spatialization, Algorithm Shift Attenuation Panning
timbre, and visual form using high-speed sound trajectories.
High Speed
1. INTRODUCTION
Spatial
Spatial modulation synthesis (SM) is a paradigm through which Modulation AM and AM and
Doppler FM
modulation synthesis and granulation effects can be produced Synthesis Granulation Spread

via high-speed sound spatialization within periodic trajectory

orbits. Hence the name, spatial parameters are modulated
to result in frequency and amplitude modulation of an input Figure 1. Spatial Modulation Synthesis
signal rather than controlling FM and AM directly. Spatial-
ization algorithms include a means for distributing the out-
put to each loudspeaker, such as VBAP [1] or Ambisonics over distance. Using existing spatialization, Doppler, and at-
[2], and are typically paired with multiple distance cues in- tenuation techniques, spatial modulation synthesis provides a
cluding Doppler shift and a distance-based gain attenuation. new control and visualization paradigm through which space
Additional distance cues may include air absorption filtering, and timbre are unified.
presence filtering, and reverberation. For now, the concept
of spatial modulation synthesis focuses only on Doppler shift 1.1 Background
and gain attenuation, leaving other possible cues as future re-
search. Beginning with Karlheinz Stockhausen’s monumental Gesang
While the spatialization of sounds along calculated trajec- der Junglinge in 1956, several 20th century works of elec-
tories and FM have been used together in many monumental tronic and computer music have utilized sound spatialization
compositions, spatialization has not been used to synthesize as an independent parameter of composition choreographed
tones through precise control of sound source trajectories. alongside timbre, pitch, intensity and duration. While early
Similarly, while the spatialization of sound grains has been choreography of spatial trajectories relied simply on ampli-
implemented by various composers, spatialization itself has tude panning between speakers, John Chowning’s 1971 pa-
not been used to create granular streams through rapid motion per, The Simulation of Moving Sound Sources [3], detailed
various techniques to implement localization cues. There-
after, many spatial compositions have utilized more realistic,
Copyright: © 2015 Ryan McGee. This is an open-access article distributed physically based distance simulation including Doppler shift
under the terms of the Creative Commons Attribution License 3.0 Unported, and gain attenuation with distance.
which permits unrestricted use, distribution, and reproduction in any Both FM and granular synthesis techniques have allowed
medium, provided the original author and source are credited. composers to fuse perceptibly discrete sound events into new

– 246 –
 ICMC 2015 – Sept. 25 - Oct. 1, 2015 – CEMI, University of North Texas

timbres. Chowing accomplished this by increasing vibrato Most digital audio Doppler simulations use an approxima-
rates into the audio domain (above 20 Hz) to discover FM tion for Doppler shift that is relatively accurate for vs <
synthesis [4], and Stockhausen morphed a series of impulses 100m/s and gives an equal shift of frequency both towards
into tone by altering their duty cycle. Stockhausen’s Concept and away from the listener [6]. This symmetric depth Doppler
of Unity in Electronic Music [5] states the composition and approximation is given by (Equation 2) and the resulting shift
decomposition of timbres as a fundamental principle of elec- is shown in figure 3.
tronic music and expresses the idea of unity between timbre,
pitch, intensity, and duration. To achieve this unity, Stock-
 vs 
f = 1± fs (2)
hausen had to take pulse trains to extremely short durations c
while Chowning took vibrato rates to extreme frequencies.
Likewise, spatial modulation synthesis is based on extremes
of sound source velocity to produce new timbres.

2. SPATIAL MODULATION SYNTHESIS

2.1 Doppler FM
Figure 3. Symmetric Doppler Appoximation, vs  c
Doppler shift is the physical phenomena of increased pitch for
sound sources moving toward a listener and decreased pitch Although Equation 2 is not physically accurate for high val-
for sound sources moving away from a listener. Through the ues of vs , the approximation conveniently provides a sym-
simulation of precise trajectories and velocities of moving metrical modulator than can used used to replicate exact FM
sound sources it is possible to sculpt the resulting Doppler timbres from SM. However, an additional implementation of
curve for use as a frequency modulator. The term Doppler the true Doppler shift (Equation 1) has also been implemented
FM is used to describe audio rate frequency modulation re- since its use is necessary for SM to achieve higher modula-
sulting from bounded, high-velocity sound source movement tion indices in addition to keeping the paradigm as physically
oscillating around a listener. First, the simplest case involving accurate as possible. Though not symmetrical in depth, us-
the motion of a sound source with a constant magnitude of ve- ing true Doppler modulation still produces more partials in
locity moving back and forth along one dimension is demon- the resulting spectra for greater timbral variation. For sim-
strated. Then, the acceleration required to produce classic si- plicity, the following equations will consider the symmetrical
nusoidal FM is described. Finally, the technique is extended Doppler approximation that shows similar relationships be-
to 3-dimensional motion resulting in parallel multiple modu- tween velocity, bounds, modulation depth, and frequency.
lator FM.
The Doppler shift for a stationary listener and moving sound 2.1.1 SM Square Modulation
source is given by Equation (1). Consider the scenario of a single sound source with frequency
  fs moving on the x-axis at a constant velocity, vs , past a lis-
c
f= fs (1) tener at x = 0. If the motion of the sound source oscillates
c ± vs such that the direction of its velocity is inverted at a bounds at
f is the observed frequency, fs is the frequency of the source, distance B in either direction from the listener then the result-
c is the speed of sound in air (≈ 340m/s) and vs is the ing Doppler shift becomes periodic. For this back and forth
speed of the moving sound source, positive when moving motion at constant velocity, the resulting Doppler curve will
away from the listener and negative when moving towards resemble a square wave.
the listener. For vs  c the magnitude of the shift is roughly
equal as the source moves towards and away from the listener
at a constant vs . However, for higher velocities, the change in
frequency, ∆f = |f − fs |, will be uneven and greater when
the source is approaching as shown in figure 2. For vs = c the
observed frequency goes to ∞ and, in reality, a sonic boom
occurs.
Figure 4. Oscillating Doppler Shift for Constant Velocity
.

Now, considering the Doppler curve as a frequency modu-

lator for our source with a rate of modulation determined by
the time it takes the source to move a distance of 2B and a
modulation depth directly related to the sound source’s ve-
Figure 2. Physically Accurate, Asymmetrical Depth Doppler locity, the modulation depth (∆f ), frequency (fm ), and index
. (I) are given by

– 247 –
 ICMC 2015 – Sept. 25 - Oct. 1, 2015 – CEMI, University of North Texas

vs and sawtooth modulators. Also, asymmetrical bounds may

∆f = fs (3) be used to further shape the modulation.
c
vs 2.2 Multi-dimensional Motion
fm = (4)
2B
Expanding source velocity to a 3-dimensional vector,
∆f 2Bfs hvsx , vsy , vsz i, produces complex modulation similar to par-
I= = (5) allel multiple modulator frequency modulation (MMFM) [8].
fm c
Not only are the timbres more complex, but the resulting 3D
The speed of the moving sound source,vs , is directly pro- orbits can produce beautiful Lissajous trajectories that modu-
portional to both the frequency and depth of modulation. The late in correlation with the changing FM parameters.
size of the bounds, B is inversely proportional to the fre-
quency of modulation, but directly proportional to the index
of modulation. The frequency of the moving sound, or car-
rier, is directly related to the depth and index of modulation.
This is a departure from classic FM in which the carrier fre-
quency is a separate control from the depth and index. Sim-
ilarly, this unique relationship between carrier frequency and
index of modulation was observed by Lazzarini et. al [7] in
their delay-line phase modulation implementation.
A fundamental concept of FM synthesis is that integer ratios
of the carrier to modulator frequencies will produce harmonic
spectra. In the case of SM square wave modulation the ratio
appears as Figure 6. 3D Asymmetrical SM Orbit

fs 2Bfs
= (6)
fm vs 2.3 Morphing Tones to Spatial Grains
Thus, proportions of bounding space and velocity become Sounds resulting from SM can replicate classic FM timbres,
critical to shape the harmonic content of a sound. but have the ability to morph into completely different modu-
lating or granular timbres by changing only a two spatial con-
2.1.2 SM Sinusoidal Modulation
trols, namely velocity and bounds. The novelty of SM lies in
The equation for classic FM synthesis involves a sinusoidal its simple, visual control paradigm which creates a continuum
modulator and the instantaneous frequency is given as between spatial choreography, tone, and granulation.
Within bounds of ±1 meter, Doppler FM can occur with
f (t) = fs + ∆f sin(2πfm t) (7) minimal amplitude modulation due to distance based gain at-
tenuation and panning. Using sinusoidal velocity along a sin-
By applying time-varying velocity (acceleration) to Equa-
gle axis to produce a sinusoidal modulator, Doppler produced
tion 2 we can derive the acceleration required of our sound
vibrato begins to morph into FM-like timbres starting at about
source to produce a sinusoidal Doppler curve.
125 m/s (≈ 280mph). Currently, sources can be simulated up
∆f sin(2πfm t)c to the speed of sound, 340 m/s, within a minimum bounds
vs (t) = (8) of 0.1 m (depending on sampling rate). By simply extend-
fs
ing the boundary of a Doppler FM tone, it can morph into
This sinusoidal velocity implies acceleration from rest at the a series of spatial sound particles. With larger boundaries
position of both boundaries and at the listener. The depth of imposing greater gain attenuation with distance, AM effects
modulation will remain the same as for square modulation, increase in depth, eventually becoming short duration ampli-
but the frequency of modulation is divided by a factor of π2 . tude envelopes. When the velocity of source movement is
sufficiently high, the duration of the resulting amplitude en-
velopes can enter the granular domain (≤ 100 ms). Oscil-
lating movement within a bounds produces a train of these
envelopes that varies in duration, shape, and spatial location
as the source moves in multiple dimensions.

Figure 5. Sinusoidal Doppler FM from Accelerating Motion 3. IMPLEMENTATION

While a plethora of tools exist for simulating moving sound
In addition to the square and sinusoidal frequency mod- sources such as GRM Tools Doppler 1 , the novelty of an SM
ulators produced by linear velocity and sinusoidal acceler-
ation respectively, linear accelerations can produce triangle 1 http://www.inagrm.com/doppler

– 248 –
 ICMC 2015 – Sept. 25 - Oct. 1, 2015 – CEMI, University of North Texas

implementation lies in the ability to have a simple control Acknowledgments

continuum between a tool for spatialization, tonal synthesis,
The author is grateful to the AlloSphere Research Group and
and granulation. Unlike other spatialization toolsets, the goal
Robert W. Deutsch Foundation for supporting this work.
of an SM implementation is to create a complete instrument
unifying space and timbre.
Spatial modulation synthesis is currently realized as a VST 5. REFERENCES
plug-in using the open-source AlloSystem C++ suite 2 . Im- [1] V. Pulkki, “Virtual Sound Source Positioning Using Vec-
plementation as a VST allows for timeline automation of all tor Base Amplitude Panning,” Journal of the Audio Engi-
parameters via the host DAW. The current parameters are neering Society, vol. 45, no. 6, pp. 456–466, 1997.
source type (sine/square/triangle/saw waveform or audio in-
put from DAW track), motion type (square, sine, saw, trian- [2] M. Gerzon, “With-height Sound Reproduction,” Journal
gle), Doppler type (symmetrical, physical), gain-attenuation of the Audio Engineering Society, vol. 21, no. 1, pp. 2–10,
curve (inverse distance and distance squared), spatialization 1973.
type (stereo, DBAP, VBAP, Ambisonics), x/y/z upper and
[3] J. Chowning, “The Simulation of Moving Sound
lower bounds, x/y/z velocity, x/y/z carrier to modulation ra-
Sources,” Journal of the Audio Engineering Society,
tio, visual trail length, and an adjustable ADSR envelope.
vol. 19, pp. 2–6, 1971.
OpenGL orbits are displayed in the plug-in interface, and an
accompanying synchronized graphics application allows the [4] ——, “The Synthesis of Complex Audio Spectra by
rendering of orbits over a cluster of machines in fully immer- Means of Frequency Modulation,” Journal of the Audio
sive stereo 3D graphics within the AlloSphere 3 laboratory. Engineering Society, vol. 21, no. 7, pp. 526–534, 1973.
The plug-in is capable of outputting to an arbitrary number of
output channels and implements a unique rendering paradigm [5] E. B. Karlheinz Stockhausen, “The Concept of Unity in
to compute sound trajectories and their corresponding graph- Electronic Music,” Perspectives of New Music, vol. 1,
ics at audio-sample rate. A dynamic anti-aliasing filter adjusts no. 1, pp. 39–48, 1962.
its cutoff to match the predicted bandwidth of the signal us-
ing Carson’s rule and prevent modulating delay line artifacts. [6] J. A. Julius Smith, Stefania Serafin, “Doppler Simulation
In addition to using arbitrary source input, the plug-in is ca- and the Leslie,” in Proc. Int. Conf. on Digital Audio Ef-
pable of scanning (audifying) its spatialization orbit as the fects, Hamburg, 2002.
source used for spatialization, creating feedback SM. When [7] T. L. Victor Lazzarini, Joseph Timoney, “The Generation
connected to a MIDI keyboard the plug-in features a carrier- of Natural Synthetic Spectra by Means of Adaptive Fre-
to-modulator ratio lock that results in dynamically changing quency Modulation,” Computer Music Journal, vol. 32,
size and velocity of the orbits with different notes, creating an no. 2, pp. 9–22, 2008.
intrinsic visual music accompaniment. Video examples and a
free stereo-only demo VST are available online 4 . [8] C. Roads, “Modulation Synthesis,” in The Computer Mu-
sic Tutorial. MIT Press, 1996, ch. 6, pp. 239–240.

4. CONCLUSION

Spatial modulation synthesis is a novel, spatial way to control

complex frequency modulation, amplitude modulation, and
granular synthesis. Looking back to Chowning’s work with
increasing vibrato rates to realize FM synthesis, SM takes
sound spatialization into the realm of timbre by increasing
speeds. Unlike other techniques that spatialize a series of
given sound grains, SM produces grains though the spatial-
ization of a non-granular source at high-speeds within a large
boundary. Using parameters of space and motion to fuse and
decompose sound timbre expands Stockhausen’s Concept of
Unity to include spatialization alongside timbre, pitch, inten-
sity, and duration. Visualization of SM trajectories also pro-
vides a physical, intrinsic correlation between visual space
and timbre.

2 https://github.com/AlloSphere-Research-Group/AlloSystem
3 http://www.allosphere.ucsb.edu
4 http://www.spatialmodulation.com

– 249 –