Brian Evans

University of Alabama
Foundations of a Visual
Department of Art and Art History
103 Garland Hall, Box 870270
Music
Tuscaloosa, Alabama 35487 USA
[email protected]

In the moving image we have a synthesis—the design, a tentative but useable grammar for structur-
spatial counterpoint of graphic art, and the tem- ing time-based visual art can be developed and even-
poral counterpoint of music. tually applied to the integration of sound and image.
—Sergei Eisenstein For this article, I created some simple video ex-
amples as illustrations. Where useful, I will also re-
There is a form of film that is trying to evolve
fer to the works of some visual music pioneers and
that area of thinking which I call “moving vi-
other filmmakers. (Pieces were selected based on
sual thinking.” And it is intrinsically a visual
relevance and availability, and the list is by no
music . . .
means exhaustive.)
—Stan Brakhage
Abstract animation, often called visual music, can
have a structural base similar to that of absolute Visual Music and its History
music. In this article, I propose a groundwork for a
practical theory of visual music composition. The Visual music can be defined as time-based visual
approach builds from the simple premise that the imagery that establishes a temporal architecture in
resolution of tension moves us through time. The a way similar to absolute music. It is typically non-
article will review some fundamentals from art and narrative and non-representational (although it need
design and apply them to temporal structure. To es- not be either). Visual music can be accompanied by
tablish the idea of visual consonance, the design sound but can also be silent.
principle of proportion will be presented. From this, One can imagine the origins of visual music going
I will develop and codify the idea that it is possible back to the discovery of fire and dancing shadows
to resolve visual dissonance to consonance, and so on cave walls, or perhaps reflections of clouds when
move a viewer through time in a way similar to first seen on the surface of rippling water. In more
tonal harmony in music. We will then briefly con- recent history, color organs (instruments that pro-
sider problems of color. The article will review jected colored light) were seen by the public as early
some traditional approaches to color harmony and as the 18th century with Castel’s Ocular Harpsi-
suggest a simple hierarchical approach to working chord. In the 1920s Thomas Wilfred toured the
with color in time. United States and Europe performing on his Clav-
Expanding to a more comprehensive system of ilux, an early electrical instrument that created
time-based design, the writings of Russian film- clouds and streams of continuous color. This his-
maker Sergei Eisenstein will be mined for raw mate- tory intensified over the last century, parallel with
rial (Eisenstein 1942, 1949). Eisenstein believed that the development of cinema.
“art is always conflict” (Eisenstein 1949, p. 46). For While the Hollywood-style narrative dominated
this article, conflict is defined as the opposition of (and dominates) cinema, pioneering filmmakers
forces that motivates and shapes action. The op- were working to develop a non-narrative language
posed forces are dissonance/consonance or tension/ of light. Typically non-academics, these pioneers
release. Eisenstein’s writings contain many ideas worked in an experimental tradition similar to that
that are useful in the development of a theory of vi- of music composers such as Eric Satie, John Cage,
sual music—especially his definition of cinematic and Karlheinz Stockhausen. Among the early film-
montage. From the concepts of montage and time- makers were Germans Walter Ruttman, Viking
Eggeling, and Hans Richter. Oskar Fischinger, also
Computer Music Journal, 29:4, pp. 11–24, Winter 2005 from Germany and moving later to the United
© 2005 Massachusetts Institute of Technology. States, worked over thirty years creating abstract

Evans 11
animations (Moritz 2004). The career of New A simple definition of absolute music is the
Zealander Len Lye, working mainly in London, structuring of time with the materials of sound pat-
spanned decades. American pioneers included John terns (excluding again program music and literary
and James Whitney, Mary Ellen Butte, Stan Brakhage, forms such as song, opera, and theater). Stravinsky’s
and Jordan Belson. Norman McLaren, Evelyn Lam- comment refers to music composition as the mak-
bart, and their colleagues at the National Film ing of “art for art’s sake.” A musical work seeks
Board of Canada also created significant works. from the listener an aesthetic response to the per-
There were many others (Russet and Starr 1976). ception of sonic pattern—the appreciation of “sig-
They all struggled against financial, institutional, nificant form.” This was a primary focus of
and technical barriers, yet they left a body of work modernism and its formalist leanings (Bell 1914).
that provides a base from which to construct a the- When visual artists discuss composition, they are
ory of visual music. This work is finding renewed in- generally referring to static design, the formal distri-
terest as evidenced by recent exhibitions. The 2003 bution of objects in the picture plane or in three-
Sonic Light Festival in Amsterdam featured many dimensional space. (This article will deal
of the pioneering animations, as did Sons & Lu- exclusively with two-dimensional space.) In the
mières, Une histoire du son dans l’art du 20e siècle 20th century, artists sought to express design in the
at the Centre Pompidou in Paris, closing in January abstract—significant form in visual space. They be-
2005 (Cruse 2004; see also www.sonicacts.com/03). gan to consider seriously their work in a musical
Also in 2005, the Los Angeles Museum of Contem- way, and the abstract form of the work was the con-
porary Art opened the exhibit Visual Music: Synaes- tent. Kandinsky, considered by many to be the fa-
thesia in Art and Music Since 1900, which then ther of abstract painting, said:
moved to the Hirshorn Museum in Washington, D.C.
A painter . . . in his longing to express his inner
(Brougher et al. 2005). Much of this work is becom-
life cannot but envy the ease with which mu-
ing easier to find on video and DVD at online sources
sic, the most non-material of the arts today,
such as the Iota Center (www.iotacenter.org), the
achieves this end. He naturally seeks to apply
Center for Visual Music (centerforvisualmusic.org),
the methods of music to his own art. And from
and even Amazon.com.
these results that modern desire for rhythm in
painting, mathematical, abstract construction,
for repeated notes of color, for setting color in
Visual Music/Absolute Music motion. (Kandinsky 1914)
The concept of absolute music can be constructed Using music as a basis for visual work was a con-
from Igor Stravinsky’s statement, “Music means cept explored by many artists in the early 20th cen-
nothing outside itself” (Stravinsky 1956). This illus- tury. Notable is the work by Sergei Eisenstein, a
trates a common mindset held by many composers Russian avant-garde filmmaker who put these ideas
for centuries. Form and content were often one in into practice and, fortunately for us, wrote down
instrumental music. (This of course does not include many of his theories of cinema. These ideas still
programmatic music and especially the occasional have potency and are especially relevant in discus-
attempts to imitate real-world sounds, such as bird sions of visual music, time design, and the combin-
calls, thunderstorms, etc.) Today, acoustic and elec- ing of sound and image.
tronic instruments can be used as generators of ab-
stract sounds without a referent in the real world
beyond the instruments themselves. These non- Visual Consonance
referential sounds are used to create abstract temporal
structures, from traditional musical forms to those For the visual artist, composition is “the arrange-
of a more experimental nature. (Many of these new ment of elements and characteristics within a de-
forms also find correspondences in visual music.) fined area . . . a grouping of related components that

12 Computer Music Journal

 Figure 1. My harmonic
analysis of Hokusai’s The
Great Wave off Kanagawa
(ca. 1831).

make sense together . . . balanced by an overall ap-

pearance of continuity” (Bowers 1999). Good vi-
sual composition achieves harmony. Art theorist
Rudolph Arnheim said, “One of the basic visual ex-
periences is that of right and wrong” (1966). This
idea forms the basis of design foundations as taught
in most art schools. First-year art students study
two-dimensional design and learn the principle of
visual “rightness” and how to achieve it with the
conceptual visual elements of point, line, plane, and
volume. These elements manifest themselves as ob-
jects in a defined space, expressed as size, shape,
color, and texture, establishing dynamic relation-
ships as these elements interact.
Visual “rightness” is visual consonance. This be-
comes an axiom from which we can build a gram- zontally and vertically illustrates its utility as a
mar of visual music. From this premise, composing compositional device. Many important points of at-
visual music is a simple process. If rightness is codi- tention lie on these formal lines: the highest wave
fied and understood, wrongness is easily defined by is placed on a vertical, the ocean level on a horizon-
not being right. We might call this wrongness visual tal, and Mount Fuji at an intersection of a horizon-
dissonance, that is, visually active moments of ten- tal and vertical.
sion in a temporal design. Progressing from visually The visual composition has a dynamic balance or
wrong to visually right moves us from dissonance to symmetry in the picture plane. We see elements of
consonance or tension to release, just as in music a repetition. (Note the wave at left-middle matching
dominant harmony resolves to a tonic (e.g., V–I). As the shape of Mount Fuji, and the foreground shape
in music, much of the interest and energy is in the of the big arching wave mirrored with the back-
moments of tension. (In tonal music theory texts, ground sky.) We are not concerned here with the
how many more pages are devoted to the dominant subject matter, but rather the underlying principle
chord and all its variations versus the tonic chord?) of composition or design.
Using the tension/release construction, we can Figure 2 shows my analysis of a drawing of an elk
move dynamically—some might say musically— from the walls of the Caves of Lascaux in France.
through time with visual materials. We can estab- (Dating from approximately 13,000 BCE, it is doubt-
lish visual cadence points. With cadences, we can ful that the golden section was applied consciously,
articulate units of time and so develop larger tem- although the drawing of the rectangle on the wall is
poral units such as motifs, periods, and phrases. curious as it predates Euclid by over 10,000 years.
Even more unusual is the fact that it is a golden rec-
tangle!) The animal seems to balance on the lower
Proportion left intersection of phi as it divides the bounding
rectangle on both the horizontal and vertical.
Consider a well-known design principle, the golden Whether by instinct or design, use of universal
section (usually symbolized as the Greek letter f or proportions is a common and effective formal de-
phi). A proportion of roughly 1:1.618, it is prevalent vice in visual and music composition. Proceeding
in the visual arts worldwide and throughout history from simple symmetry to a balanced but active
(Huntley 1970). Figure 1 shows my brief analysis composition is a useful skill in creating visual mu-
(what is sometimes called an harmonic analysis) of sic. Phi is just one of many proportions that has
Hokusai’s famous The Great Wave off Kanagawa been explored over the centuries. Jay Hambidge’s
(ca. 1831). The cut of the golden section both hori- studies in dynamic symmetry showed the applica-

Evans 13
Figure 2. My harmonic Figure 3. From a harmonic
analysis of a drawing of an analysis of Botticelli’s
elk from the walls of the Birth of Venus by Charles
Caves of Lascaux in France Bouleau (1963).
(ca. 13,000 BCE).

tion of proportions such as phi in temporal con-

structions as well. Bartók’s and Debussy’s uses of
phi are well documented (e.g., Lendvai 1971; Howat
1983). Stockhausen often used the Fibonacci series
(a number series from which phi can be derived) in
several of his early works (Griffiths 1981). Evidence
tion of many mathematical constants, including √2, of phi can even be found in the music of Mozart,
√3, and various other geometric constructions, illus- which should not be surprising as he was a Mason,
trating their application in everything from Grecian and the divine proportion was part of the Masonic
urns to furniture to typography (Hambidge 1919, “secret knowledge” (Evans 1993).
1932). The “secret” of the golden section was re- It should be stressed here that these proportions
vealed during the Renaissance in Luca Pacioli’s are not a panacea for the creation of good composi-
book Da Divina Proportione, which was widely tion or visual consonance. They are, however, un-
read by artists of the time. doubtedly useful and often used. Good visual
Also at this time, Leon Battista Alberti published composition requires sensitivity to all elements in
his writings on painting and architecture. He de- the picture plane. Anyone interested in composing
scribed the system of one-point perspective and the visual music should study two-dimensional design
application of the Pythagorean proportions of music in all its various aspects, much as any first-year art
to visual construction. Figure 3 is taken from a har- foundations course will provide.
monic analysis of Botticelli’s Birth of Venus by From an understanding of proportion and compo-
Charles Bouleau (1963). Here, we see an application sition in the picture plane, ideas can expand into
of phi on the cut of the horizon line along with a variations, and larger structures can build on rela-
composition based on the simple harmonic propor- tions of design elements. Figure 4 is from a design
tions of 9:12:16. Botticelli here used Alberti’s musi- lesson in a book by Arthur Wesley Dow written over
cal consonances. For Alberti, proportion was a visual a century ago. The figure illustrates variations of a
music. In Botticelli’s application of these propor- single design, which he calls the motif. Dow said,
tions, we see the interval of the perfect fourth (9:12, “Little can be expressed until lines are arranged in a
or 3:4) and what Alberti called the duple diatessaron, Space [which] I shall call PRINCIPLES OF COMPO-
or double fourth (9:16, or 32:42) (Alberti 1955). SITION . . . all being dependent upon a great general
Simple-number proportions are a foundation of principle, PROPORTION or GOOD SPACING”
Western music intervals, but we can find applica- (Dow 1899).

14 Computer Music Journal

Figure 4. From a design les-
son in a book by Arthur
Wesley Dow (1899).

and color. We see repetition. The two examples are

of equal duration of 8 seconds. They zoom in at
roughly the same rate, with a central focus of two
flowers moving to a single flower, focused left of
center. The examples feel similar, but there is varia-
tion. They contrast in their vertical motion, angles
of rotation, and colors; there is repetition, contrast,
and variation. We have the beginnings of a syntax
from which to construct larger forms and compose
visual music.
The construction of a pattern of tension/release
moves a viewer through time, and so we seek to
build tension. The desire to resolve the tension nat-
urally follows. Time passes dynamically. Video Ex-
ample 3 is an exercise illustrating a single shot with
Temporal Design elements entering and exiting the picture plane in
an overall rounded binary form (AABA). The ex-
In temporal design, we move the viewer from visual ample has four phrases that land on well-composed,
dissonance to cadences of visually balanced, well- visually “right” moments. Phrase 1 introduces
composed moments. Time passes musically through Mondrian-like graphic materials with red and blue
patterns of tension/release. Filmmaker Ken Burns’s squares, shifting in color and moving into a com-
use of slow pan and zoom of a camera over old still posed cadence, seen as a small yellow square spin-
photographs is a simple technique that allows ex- ning to combine with two thick black lines. Phrase
ploration of this basic approach to time design. Us- 2 follows with a repetition of blue and red squares
ing only camera motion, we can see this phrasing in spinning in from the opposite direction, cadencing
films such as The Civil War—A Film by Ken Burns by coming to rest on a balanced design. A new, con-
(Burns 1990). Now a common technique in histori- trasting phrase follows with thin lines rotating
cal documentaries, camera motion over still images around an axis and a return of the small yellow
can also tell stories. See for example French film- square and thicker black lines. This overlaps into a
maker Chris Marker’s 1962 short film La Jetée, an final phrase that again repeats the spinning red and
early and effective application of this device along blue squares while echoing the rotating lines that fi-
with montage and visual composition (Marker 1962). nally articulate closure, coming to rest on another
I created simple illustrations of camera move- complete and balanced design. The example illus-
ment with still images, included here as Video Ex- trates how visual time can be musical, using motifs
amples 1 and 2. [Editor’s note: video figures for this and phrases with the “camera” locked in place and
article can be found on the DVD accompanying the whole work unfolding in an single “shot.”
this issue.] A virtual camera moves over digitally Developing a visual music piece in a single shot
processed photos of flowers. Key frames are seen in with little or no editing is a common practice in vi-
Figures 5 and 6. Each example starts in balance with sual music composition; many of John Whitney’s
a dynamic or simple symmetry. Once motion starts, films do this. His multi-movement work Arabesque
a tension is established that seeks resolution in is built on the development of 360 points of light
stillness and a return to a balanced composition. moving in the picture plane (Whitney 1975). The
Each example can be considered as a phrase. We points move independently through various numeric
can eventually combine phrases, seeking points of processes, in and out of visual resonances where the
commonality and difference in the visual dimen- points resolve into circles or symmetries in simple
sions of motion (direction and speed), composition, numeric proportions. Whitney called this differential

Evans 15
Figure 5. Key frames from Figure 6. Key frames from
Video Example 1. Dots Video Example 2, illustrat-
show points of focus, and ing points of focus and ba-
lines indicate the basic tra- sic camera trajectory. The
jectory of camera motion. camera is moving down
The camera moves up and and rotates clockwise.
rotates counterclockwise.

Figure 5

Figure 6

dynamics or harmonic procession (Whitney 1980). movements of the dancers resolve into single fig-
These moments of resonance provide points of vi- ures, reminiscent of a musical canon at the cadence
sual consonance. Moving away from and back to the when all voices come together.
consonance creates tension and resolution. Oskar Fischinger’s Study No. 10 (1932), in black
Norman McLaren’s Pas De Deux (1968) shows and white, is a wonderful example of this single-
mostly single-shot action using time-lapse filming shot approach with movement through moments of
of two dancers. The piece presents another approach visual consonance. In this piece, Fischinger creates
to Whitney’s idea of differential dynamics. Definite abstract materials that directly follow and express
points of resolution occur when all the time-lapsed the phrases and feeling of the music.

16 Computer Music Journal

Figure 7. Two color circles:
(a) Itten’s (“painter’s”)
RYB color circle; (b) addi-
tive (“computer”) RGB
color circle.

(a) (b)

Considering Color tion of color, especially hue, can vary widely based
on genetics and experience (Livingstone 2002).
Color can also be used to create a tension that seeks At the very least, color in art is confusing. Take
resolution, although it is not an exact science. for example the two color circles seen in Figure 7.
Artists have developed many theories of color over Both are used to determine complementary colors.
the past centuries. A common idea in these theories A basic tenet of color theory is that complementary
is harmony. In discussing harmony, theorist Luigina colors (those directly opposite on the circle) create
De Grandis said, “All theories have in common an color harmony. The circle on the left shows the
assumption that the sensation of harmony, or con- standard color primaries as understood by artists
cord of parts of a whole . . . results exclusively from over the centuries: red, yellow and blue (“RYB”). Jo-
the relationships and proportions of its chromatic hannes Itten formalized this color circle in develop-
components” (De Grandis 1984). ing the color theories he taught at the Bauhaus.
The problem lies in how those chromatic compo- This approach to color is still taught in many art
nents should be organized, as there is no single uni- schools today (Itten 1973).
versal categorization of color. Margaret Livingstone, The circle on the right is the one used in com-
Professor of Neurobiology at Harvard University, puter graphics displays as they project light. It uses
studies how the eye and brain process color. She the additive primaries of red, green, and blue (RGB).
notes that “much of what has been written about When added together, these primaries will make
color in art is nonsense.” She points out that the white light. The complementary colors in this circle
brain processes hue and luminance (also called are cyan, magenta, and yellow (CMY), also called
value) separately. Some aspects of vision, such as the subtractive primaries, which theoretically pro-
depth perception and the separation of figure and duce black (the absence of light) when combined. As
ground, are colorblind, meaning that we get our vi- one mixes them, more and more light is absorbed or
sual cues from value alone. And the human percep- subtracted. Commercial and digital printing pro-

Evans 17
cesses use the subtractive primaries with black ink an RGB space designed to create grayscale when the
added (denoted K), as real-world pigments cannot RGB component values are equal.
create a true black, but rather a somewhat muddy With digital images, it is possible to calculate
brown. sums of the amounts of red, green, and blue in an
In Itten’s system, red and green are complemen- image. The closer the RGB sums are to being equal,
tary and concordant in combination. In additive the more “balanced” the image is with respect to
color, red and green are both primary and so are dis- color, as equal means gray. Color can move us
cordant. The two systems will create contradictory through time by resolving an emphasis of a single
harmonies. What is the measure we use to decide hue to a coloring that balances or equalizes RGB
which is right? Color continues to be problematic in summation values (Evans 1990). Tension/release
seeking visual harmony. Furthermore, the percep- with color can be established in this way, but it re-
tion of color is inexact, culturally influenced, and quires a detailed measurement of digital color that
personal. I might use a variation of green that re- may not be practical. What is more practical is a
minds me of the zest of a key lime. The same color general application of the perceptual hierarchies of
might remind someone else of rotting meat, trigger- brightness resolving to dark, and high saturation re-
ing a very different response. One must be careful solving to low saturation or gray.
with color. A familiar example is mentioned in Bordwell and
There are some generalizations we can draw, Thompson’s analysis of the 1939 film version of The
however. First, color space is a three-dimensional Wizard of Oz (Baum 1939; Bordwell and Thompson
construct. Familiar color spaces, all hue-based, in- 2001). The larger structure of the film is an ABA
clude RGB, RYB and CMY. Some color spaces in- form based on the settings Kansas-Oz-Kansas.
clude other visual dimensions. One artistically Dorothy is home in Kansas in the beginning, filmed
useful space maps a hue-only approach to one where in grayscale. Through her adventures in Oz, where
color is defined as hue, saturation, and value (HSV) conflict builds, the film is in full color. Conflict re-
(Smith 1978). There is no perceptual hierarchy of solves back to grayscale when she returns home.
hue, but there are of value (bright to dark) and satu- Shifting from color to grayscale is now a common
ration (full chroma to grayscale). Movement through device on television, used to articulate or simply
these hierarchies provides areas of exploration. end the action. A transition to grayscale or simple
Chemist and colorist Maurice Chevreul in the desaturation often takes us to a commercial.
19th century suggested that the eyes seek the bal- Working with color in HSV space, it is useful to
ance of grayscale, as evidenced through the visual think of brightness as visual amplitude and satura-
phenomena of simultaneous and successive con- tion as spectral intensity. Bright, full, saturated col-
trast (Chevreul 1854). A sensation of harmony or ors can be tiring to a viewer when used in excess,
“chromatic equilibrium” should occur with com- just as loud, full-spectrum sounds can become tiring
plementary colors, as they would mix to create to a listener.
grayscale, a sort of visual tonic. Gray is balanced, Oskar Fischinger’s Motion Painting 1 (1947) is a
whereas emphasis on one hue or another creates good example of a general approach to using satura-
tension. Chevreul also points out the difficulty in tion and brightness structurally. In the first section
practice using pigments or dyes as “we know of no especially, the movement of color is important as it
substance which represents a primary color.” builds large phrases. In the film, we see a single
Newton showed us that we see white when all painting develop layer upon layer over eleven min-
colors of the spectrum of visible light are combined; utes. We feel tensions resolve in short phrases as we
lower the value or brightness and we see gray. Digi- see shapes developing as motives, begun and then
tal technology simulates this with the additive RGB brought to completion. Spirals coil and unfold; mo-
color space. We can measure the equilibrium of saics of diamonds appear and cover what is under-
color combinations without concern for color circles neath. Larger structures are perceived as repetition
or complementarity. Computer monitors work in and contrast of hue choices. Tension resolves as

18 Computer Music Journal

high saturation, and bright colors give way to darker metric, rhythmic, tonal, overtonal, and intellectual.
colors and lower saturations. In discussing the five types, we will focus here on
the first three.

Montage and the Musical Phrase

Metric Montage
Cinematographer Joseph Mascelli lists the “Five C’s”
of cinematography as camera angles, close-ups, cut- In defining metric montage, Eisenstein writes that
ting, continuity, and composition (Mascelli 1968). “pieces are joined together according to their
All but the cutting require some degree of pre- lengths in a formula scheme corresponding to a
production planning. Through the practice of story- measure of music” (1949, p. 72). We are only con-
boarding, we see this planning in action (Begleiter cerned with the duration of each shot, cutting each
2001). With storyboards, filmmakers illustrate and shot to a beat without regard for the content of the
articulate many of the decisions to be made when shot. We establish meter by making each cell the
shooting a scene. The principles of composition same length—the same number of frames. Each cell
used in support of the story are key to this process. is like a phrase of music. With shots of short dura-
A director plans a scene as a series of moments, tion, cells are like bars of music—sometimes even
composed as a series of drawings. The drawings beats within a bar.
guide the eventual motion, framing, and design of A time signature can be loosely established
the picture plane over time. Continuity, camera through accent of some visual dimension. Occa-
angle, and composition in a scene delineate phrases sionally, the filmmaker will actually put an all-
that cadence with moments of visual rightness. black or all-white frame at the end of each cell,
Time passes musically. making the visual meter obvious. Sometimes cells
With visual time passing under compositional are edited to match musical phrasing—a common
control, a larger temporal structure can be built. Vi- device in music videos. Because the frame rate of
sual phrases are combined in film in a process called film or video is constant, we can accelerate or slow
montage, the cutting or editing of camera shots into down the meter by changing the length of the cells
a sequence. Eisenstein’s theory of montage can be as they sequence.
valuable to our developing grammar of visual music. Metric montage follows the “law of simple num-
He believed that it was montage that placed film at bers,” and it is applied for clarity when based on a
the “pinnacle of the arts.” The unit of montage, musical time passage (3/4, 2/4, 4/4, etc.), where it
what Eisenstein called the cell, combines with other can “bring into unison the ‘pulsing’ of the film with
units and creates a gestalt, expressing more than just the ‘pulsing’ of the audience” (Eisenstein 1949,
the sequence of images. The power for Eisenstein p. 73). Using an irregular measure, “16/17, 22/57,
lay in the heights of expression possible. “To deter- etc. . . . brings a degeneration of the method” (Eisen-
mine the nature of montage is to solve the specific stein 1949, p. 72). From the extremes of simple to
problem of cinema” (1949, p. 48). More simply, he irregular is a middle ground where lengths are var-
states “montage is conflict” (1949, p. 38). Montage ied, alternating between two different contexts, jux-
is a multi-tiered construct of tension and release. taposed temporally back and forth. What he refers
We can think of each cell as a musical phrase. to here are irregular-to-metered durations, such as
From good visual composition, we can build phrases rubato to a tempo, which is itself a basic process of
and expand into a larger musical form. Temporal de- tension/release.
sign at the level of the shot moves us forward to ca- Norman McLaren and Evelyn Lambart’s film Be-
dence points. Through montage, we construct gone Dull Care is an enjoyable example of metric
relationships from shot to shot, building a larger montage. Strips of clear celluloid were painted,
form and expressing our ideas, formally or other- scratched, textured, and processed in a variety of
wise. Eisenstein lists five basic types of montage: ways. They cut the strips into cell lengths to match

Evans 19
with music by the Oscar Peterson Trio (Lambart Camera motion and zoom come to rest at the end
and McLaren 1949). of each cell. Cells ease-in and ease-out at endpoints
in the larger phrases. Cells enter with cross-fades,
the cross-fade itself a visual equivalent to musical
Rhythmic Montage elision.

In rhythmic montage, the length of the cell is deter-

mined by what we see. “Abstract determination of Tonal Montage
the piece-lengths gives way to a flexible relation-
ship of the actual lengths” (Eisenstein 1949, p. 74). Eisenstein calls his third montage type tonal mon-
Durations of phrases support the content of the tage. Continuing his borrowing of musical function,
cells, and most specifically “movement within the he describes tonal montage as “melodic-emotive.”
frame impels movement from frame to frame . . . , We must be careful here to avoid misunderstanding
objects in motion, or of the spectator’s eye directed his use of the terms tonal and dominant, as they do
along the lines of some immobile object” (Eisen- not relate very well to the musical definitions of the
stein 1949, p. 75). This is apparent in basic continu- terms. The tone of a work is its expressive tone.
ity editing in which a long shot, usually containing The dominant is the primary emotional feel of the
lots of image detail, lasts longer than a close-up. shots. He here begins to bring the thematic content
The logic is that it takes longer to see all the details of the film and the meaning of the images into the
in a long shot. The passage of time becomes more discussion. He attempts to maintain an objective
psychological than strictly measured; experiential measure by focusing on the graphic or plastic as-
time is a consideration. Events and images within pects of the moving image: color and shape. He
the picture frame determine duration. calls it “graphic tonality”—the design elements of
Video Example 4 takes the simple camera motion color, line, and shape, and what they can represent.
with the flowers seen earlier and extends it into The time design begins to consider the subject
four phrases. The camera motion zooms in with matter and its signification. As Eisenstein explains
cells 1, 2, and 8, slowly panning and rotating from his ideas, he presages developments in semiotics and
and to a balanced composition. Consider these cells structuralist thought. This goes beyond our discus-
as phrases, cadencing with a similar design struc- sion, as we are focusing on the plastic elements alone
ture and a focus left-of-center. in composing visual music. The aim is to develop a
Cells 3–7 contrast this and together make up a simple design syntax built from a basic design idea.
contrasting phrase. They are much shorter in dura- We are sidestepping issues of content that go beyond
tion, with centered images and the camera zooming formal concerns. Bordwell and Thompson (2001)
in slightly. We see a basic rounded binary form: A write, “Since a camera movement consumes time
(cell 1), A (cell 2), B (cells 3–7), A (cell 8). The dura- on screen, it can create an arc of expectation and
tion of each cell reflects the amount of camera fulfillment.” This restates the premise: as camera
movement within the cell. movement alone fulfills an expectation, the viewer
Metric montage is illustrated with cells 1 and 2 is moved through time to a resolution of tension.
(each 8 sec long) and cells 3 through 7 (each 2.4 sec Thus, while in tonal montage, color and shape do
long). As the camera moves over more space in cell begin a move into connotation, we can reduce an
8, it is consequently slightly longer in duration than image-in-motion to its design elements. From this
cells 1 and 2, yet similar in camera motion and final distilled level of montage, we articulate phrases
composition. This keeps the perception of speed with movement in color (saturate to neutral, bright
more or less constant in these cells. Rhythmic mon- to dark) and movement from and to a harmonious
tage is developed so the durational feel of cells 1, 2, distribution of materials on the picture plane.
and 8 is similar, and the total duration of the B sec- In the flower example, the graphic tonality is seen
tion (cells 3–7), feels in balance. in the repetition of points of visual focus and in rep-

20 Computer Music Journal

etition or contrast of direction, rotation, and speed Returning to our simple, syntactical point of
of camera zoom and pan. Centered images of the B view, we can think about montage working at the
section, with little motion, are shorter, and they set three basic levels when building phrases of moving
up expectations for a return to rotation and a longer images. First is metric montage, which deals with
phrase. This is another iteration of the basic the simple duration of shots. This is expanded to
rounded binary structure, this time at the level of rhythmic montage, in which the duration of cells
tonal montage. supports the plastic elements in motion within the
Many of Stan Brakhage’s works, usually with no shot. Lastly, the graphic elements such as color,
sound score, are good examples of tonal montage. line, and shape and the dominant expression of
Mothlight (1963), for example, is an interesting the shot build phrases in tonal montage. Each shot
work that uses dead moths and other organic debris or phrase articulates cadences through a construct
he collected from light fixtures. He used no camera of tension/release. Phrases are combined and grow
but instead collaged twigs, blades of grass, dust, and to a larger form through repetition, contrast, and
moth parts onto sticky tape, and then he printed it variation.
to celluloid for viewing through a film projector.
Cells were then cut to create a fast-paced and
haunting montage. Visual Counterpoint
Chris Marker’s La Jetée and Ken Burns’s A Civil
War are good studies of tonal montage with mini- An artwork can be defined as a complex relation-
mal sources. For example, battle scenes in A Civil ship of elements; it is multi-dimensional. In devel-
War may have shots of short duration, in contrast to oping the elements particular to an image in motion,
longer duration pans over battlefields in the after- it is possible to build the unfolding of these elements
math. Both films show how much can be done in in a contrapuntal structure (Evans 1992). Various
developing large temporal structures with simple graphic elements can work independently in sup-
composition and effective editing. port of a larger structure. Eisenstein calls this visual
counterpoint.
Montage is conflict. Cinematic conflicts unfold in
Overtonal and Intellectual Montage the picture plane. These conflicts work within the
cell or shot and in “colliding” cells or montage.
The last two types of montage are not really mon- Eisenstein lists the conflicts (which deal mainly
tage in the strict sense. Eisenstein calls his fourth with the compositional elements) as follows (the
type of montage overtonal montage. He begins to italics are his):
develop the relationship between the previous three
conflict of graphic directions (lines either static
types and the cinematic work as a whole. He sees
or dynamic); conflict of scales; conflict of vol-
the elements of montage metaphorically as har-
umes; conflict of masses (volumes filled with
monic partials of a musical timbre. Conflict devel-
various intensities of light); and conflict of
ops and resolves as these elements wrestle and then
depth. With a bit more “intensification,” there
agree. Eisenstein himself admits that overtonal
can also be conflicts of close shots and long
montage is “not an altogether exact term” (1942,
shots; pieces of graphically varied directions
p. 86), but he is concerned with the larger structures
(pieces resolved in volume with pieces resolved
of the film and its impact on the audience. From
in area); pieces of darkness and pieces of light-
this, he takes us to his fifth level, intellectual mon-
ness; and conflicts between an object and its
tage, which will eventually build a “completely
dimension (and conflicts between an event and
new form of cinematography” (1949, p. 83). In later
its duration).
writing, he calls this intellectual cinema and is re-
ally considering the movement of thought rather [In this final conflict, he refers to optical distortion,
than moving light. stop-motion, or slow-motion.]

Evans 21
 What we have here is a comprehensive list of the movement” (1942, p. 81). This inner connection is
design materials. With an understanding of two- among the images, the sounds, and the meaning of
dimensional design and tension/release, we can de- the work. For the audience it is the gestalt of the
velop all of these materials independently within a work as a whole, with all elements contributing to
shot and integrate them through many shots into a the overall meaning. In moving beyond the visual
“unified system” of montage, the basis of a “tenta- music foundations, much can be learned from the
tive film syntax” (Eisenstein 1949, p. 55) techniques of opera, musical theater, film scoring,
Again, consider the first two video examples. and sound design.
They repeat the horizontal direction of focus from When we use digital tools to create time-based
left to right. They contrast in vertical direction (one work, layers of material are usually visualized on a
with the camera going up, the other with the cam- timeline. Events are placed graphically like notes on
era going down). They repeat the relative scale of staves in a musical score. Events unfold horizon-
zoom, and they repeat the duration. They contrast tally, but vertical relationships are also important—
in rotation angle (clockwise vs. counter-clockwise). multiple lines moving independently yet in
Different elements with different temporal struc- harmony. Eisenstein’s image of the orchestral score
tures—some in agreement and some opposed—cre- is realized in the user interface of current graphics
ate a counterpoint. software designed for time-based work. (Good ex-
amples include Adobe After Effects, Digidesign Pro
Tools, Macromedia Flash, and even Apple’s iMovie.)
Vertical Montage (And the Sound Image) Horizontal layers represent video, audio, still im-
ages, camera motion, interpolation paths, and more.
Eisenstein expanded his counterpoint to combine These lines can interact and mix both horizontally
the “visual image with the sound image, in the and vertically.
process of creating a single, unifying sound-picture
image” (1942, p. 73). He considered this a new
kind of montage, naming it vertical montage. Toward a Visual Music Theory
Again, he uses the term loosely, referring to the in-
teractions of various elements as they are synchro- Eisenstein believed that every artwork is conflict
nized in time. according to its social mission, according to its na-
He saw vertical montage as an orchestral score ture, and/or according to its methodology. The
with several staves of activity unfolding on the hori- methodology of cinema is montage. At the graphi-
zontal yet with all elements interrelating on the cal level, montage can move us in time through the
vertical, all moving forward with “intricate har- visual “rightness” one learns in two-dimensional
monic musical movement” (1942, p. 74). The mon- design. With experience and sensitivity to design
tage structure of the visuals is just one more line on foundations, the artist, filmmaker, or visual music
the musical score. The varied techniques of music composer can explore the higher levels of artistic
composition now can be applied to build a sound conflict, its nature and social mission. Even with
score that fits the overall contrapuntal texture of studies in composition and movement using static
the work. images, a design sense can be exercised. Sensitivity
For Eisenstein, it was important that this be a to musical time can be developed and practiced
true, polyphonic texture. He was not interested in with visual materials. A sound score can be added
direct synchronization of sound and image, but and musical sensibilities applied to a unified sound-
rather the “inner synchronization between the tan- image expression.
gible picture and the differently perceived sounds . . . Literacy these days means speaking the language
the ‘hidden’ inner synchronization where the plas- of new media, which is in many ways the language
tic and tonal elements will find complete fusion . . . of cinema. As it is a temporal medium, we can find
[where] we find a natural language common to both— at the foundations of cinema the concept of music.

22 Computer Music Journal

The tenets of musical expression give us a method Burns, K. 1990. A Civil War—A Film by Ken Burns.
for moving the viewer/listener through time. With Alexandria, Virginia: PBS Home Video.
visual composition, we create consonance and dis- Cheuvreul, M. E. 1854. The Principles of Harmony and
sonance. The ability to resolve dissonance to conso- Contrast of Colors and their Applications to the Artist.
nance provides a syntactical basis for temporal West Chester, Pennsylvania: Schiffer Publishing.
Cruse, V. 2004. Sons & Lumières, Une histoire du son
design and the composition of visual music. Our
dans l’art du 20e siècle. Paris: Centre Pompidou.
compositions can be a single shot moving through De Grandis, L. 1984. Theory and Use of Color. New York:
phrases. We build phrases through repetition, con- Harry N. Abrams.
trast, and variation of the many visual dimensions Dow, A. W. 1899. Composition. Garden City, New York:
including direction, speed, shape, size, and color. Doubleday.
Eisenstein’s montage provides a conceptual frame- Eisenstein, S. 1942. Film Sense. New York: Harcourt.
work for putting the phrases together into larger Eisenstein, S. 1949. Film Form. New York: Harcourt.
forms. New media may require a new literacy, yet Evans, B. 1990. “Temporal Coherence with Digital
traditional ideas still have much to offer. Color.” Digital Image—Digital Cinema, SIGGRAPH
’90, Art Show Catalog, published in Leonardo supple-
mental issue. Oxford: Pergamon Press, pp. 43–49.
Evans, B. 1992. “Elemental Counterpoint with Digital
Acknowledgments Imagery.” Leonardo Music Journal 25(4):13–18.
Evans, B. 1993. “Number as Form and Content: A Com-
This work was done with the support of a Research posers Path of Inquiry.” In M. Emmer, ed. The Visual
Advisory Council Grant, University of Alabama. Mind. Cambridge, Massachusetts: MIT Press.
Fischinger, O. 1947. Motion Painting No. 1. Recorded on
Oskar Fischinger: Pioneer of Abstract and Advertising
References Animation. LaserDisc. Carson, California: LaserDisc
Corporation (1986) Pioneer Special Interests, PSI-88-012.
Arnheim, R. 1966. “A Review of Proportion.” In G. Fischinger, O. 1932. Study No. 10. Recorded on Oskar
Kepes, ed. Module, Proportion, Symmetry, Rhythm. Fischinger: Pioneer of Abstract and Advertising Ani-
New York: George Braziller, pp 218–230. mation. LaserDisc. Carson, California: LaserDisc Cor-
Alberti, L. B. 1955. Ten Books on Architecture. London: poration (1986) Pioneer Special Interests, PSI-88-012.
Tiranti. Griffiths, Paul. 1981. Modern Music, the Avante Garde
Baum, L. F., director. 1939. The Wizard of Oz. 2004. (1939) Since 1945. New York: Braziller. p. 85.
Motion picture. Hollywood, California: Warner Stu- Hambidge, J. 1919. The Elements of Dynamic Symmetry.
dios, ASIN: B00000JS62. New York: Dover.
Bell, C. 1914. Art. London: Chatto and Windus. Hambidge, J. 1932. Practical Applications of Dynamic
Begleiter, M. 2001. From Word to Image: Storyboarding Symmetry. New Haven, Connecticut: Yale University
and the Filmmaking Process. Studio City, California: Press.
Michael Wiese Productions. Howat, R. 1983. Debussy in Proportion: A Musical Anal-
Bordwell, D., and K. Thompson. 2001. Film Art: An Intro- ysis. Cambridge: Cambridge University Press.
duction, 6th ed. New York: McGraw-Hill. Huntley, H. E. 1970. The Divine Proportion. New York:
Bouleau, C. 1963. The Painter’s Secret Geometry. New Dover.
York: Harcourt, Brace, and World. Itten, J. 1973. The Art of Color. New York: Van Nostrand
Bowers, J. 1999. Introduction to Two-Dimensional De- Reinhold.
sign: Understanding Form and Function. New York: Kandinsky, W. 1914. Concerning the Spiritual in Art.
Wiley. New York: Dover.
Brakhage, S. 1963. Mothlight. Recorded on By Brakhage— Lambart, E., and N. McLaren. 1949. Begone Dull Care.
Anthology—The Criterion Edition. DVD. New York: Recorded on Norman McLaren: The Collector’s Edi-
Criterion DVD CC1590D (2003). tion. DVD. Harrington Park, NJ: Milestone Film and
Brougher, K., et al. 2005. Visual Music: Synaesthesia in Video (2003) ASIN: B0000AOV4F.
Art and Music Since 1900. New York: Thames and Lendvai, E. 1971. Béla Bartók: An Analysis of His Music.
Hudson. London: Kahn and Averill.

Evans 23
Livingstone, M. 2002. Vision and Art: The Biology of See- Russet R., and C. Starr. 1976. Experimental Animation:
ing. New York: Harry Abrams. an Illustrated Anthology. New York: Van Nostrand
Marker, C. 1962. La Jetée. Recorded on Short 2-Dreams. Reinhold.
DVD. Hollywood: Warner Studios (1997) ASIN: Smith, A. R. 1978. “Color Gamut Transform Pairs.”
B000031VPS. Computer Graphics 12(3):12–18.
Mascelli, J. 1968. The Five C’s of Cinematography. Holly- Stravinsky, I. 1956. The Poetics of Music in the Form of
wood, California: Cine/Grafic Publications. Six Lessons. New York: Vintage Books.
McLaren, N. 1968. Pas De Deux. Recorded on Norman Whitney, J. 1975. Arabesque. Recorded on The World of
McLaren: The Collector’s Edition. DVD. Harrington John Whitney. LaserDisc. Carson, California: LaserDisc
Park, New Jersey: Milestone Film & Video (2003) Corporation (1984) Pioneer Special Interests, PSI-88-003.
ASIN: B0000AOV4F. Whitney, J. 1980. Digital Harmony: On the Complementar-
Moritz, W. 2004. Optical Poetry: The Life and Work of ity of Music and Visual Art. New York: McGraw-Hill.
Oskar Fischinger. Eastleigh, UK: John Libbey Publishing.

24 Computer Music Journal