VISUALIZING MUSIC IN ITS ENTIRETY USING ACOUSTIC FEATURES:

MUSIC FLOWGRAM

Dasaem Jeong and Juhan Nam

Graduate School of Culture Technology, KAIST
{jdasam, juhannam}@kaist.ac.kr

ABSTRACT one needs to read through the score for a while, having a
certain level of musical knowledge.
In this paper, we present an automatic method for visualiz-
As a way of making up the shortcomings of music no-
ing a music audio file from its beginning to end, especially
tation, audio-synchronized music scores have been devel-
for classical music. Our goal is developing an easy-to-use
oped [1]. Synchronized scores automatically follow music
visualization method that is helpful for listeners and can be
on the score so that listeners can easily track where the
used for various kinds of classical music, even for complex
currently playing measure is or select the measure on the
orchestral music. To represent musical characteristics, the
score to play the music from the position. However, such
method uses audio features such as volume, onset density,
systems require a synchronization process between audio
and auditory roughness, which describe loudness, tempo,
and score. Manual synchronization is too laborious to pro-
and dissonance, respectively. These features are visually
cess a large set of pieces, whereas automatic one, an active
mapped into static two-dimensional graph, so that users
research topic in the area of music information retrieval
can see how the music changes by time at a look. We im-
(MIR), is not accurate enough particularly for large orches-
plemented the method with Web Audio API so that users
tral music. Above all, these solutions still cannot show the
can access to the visualization system on their web browser
entire structure of a piece.
and make visualizations from their own music audio files.
Two types of user tests were conducted to verify the effects In the case of classical music, particularly for long in-
and usefulness of the visualization for classical music lis- strumental pieces, visualizing information about the entire
teners. The result shows that it helps listeners to memorize structure can be helpful to music listeners in that they do
and understand a structure of music, and to easily find a not contain lyrics or clear storytelling to follow. So addi-
specific part of the music. tional information about the music is required to help lis-
teners to understand the music. A traditional way of pro-
viding the information is giving a lecture or writing a pro-
1. INTRODUCTION gram note. But these requires professionals who can ex-
Music visualization is widely used in various music ac- plain the music. Many researchers instead have suggested
tivities for many purposes. Because music is an auditory a content-based approach to visualize the entire structure
art, its visual representations can contain information that of music from audio. Most of automatic music structure
cannot be transferred or preserved accurately with sound. visualization methods are based on self-similarity between
Music notation is a typical example of the visualized music each part of a piece [2, 3, 4]. These methods show a repeti-
representations. It is designed for communication between tive structure of the music based on the self-similarity. Fur-
composers and performers. The notation systems thus have ther information about the research is introduced in Sec-
been evolved to represent and deliver a composer’s inten- tion 2. In general, it is not easy to interpret the meaning of
tion as precise as possible. the visualizations. Finding the repetitive structure can help
For listeners, however, music notation has some limita- music structure analysis, but its usefulness on listeners has
tions. It contains too much information for listeners to in- been not verified yet.
terpret and so only a small part can be understood while To address this problem, we present an automatic music
following the music. Especially, in the case of orchestral visualization method named music flowgram, which aims
music, the score following task is quite difficult unless the to visualize an entire piece as an easy-to-understand im-
listeners are musically trained. Another problem is that the age. It extracts audio features from audio files and visual-
notation does not show the entire structure of a piece of izes them on a static two-dimensional graph. In our pre-
music. The time scope of a music score that can be read vious research, we found that a simple static graph show-
in a sight is limited to a few measures. The notation is ing the change of volume of a music piece can help lis-
focused on delivering information about what is happen- teners to concentrate more on classical music, compared
ing in a specific time. To understand the global structure, to spectrum-based real-time visualization [5, 6]. We have
improved this concept by adding additional features that
Copyright: c 2016 Dasaem Jeong et al. This is an open-access article distributed can represent other important characteristics of music, and
under the terms of the Creative Commons Attribution 3.0 Unported License, which conducted user test to verify its effect on listening to clas-
permits unrestricted use, distribution, and reproduction in any medium, provided sical music.
the original author and source are credited. The later part of this paper is organized as follows. First,
related work on visualizing music structure is briefly re- used an abstract representation and 21 of them were in con-
viewed. Then, we present our visualization method in two tinuous mode. This result indicates that a two-dimensional
sections: concept of the visualization and audio features in graph is natural in human sense for representing whole mu-
Section 3, and its implementation in Section 4. The de- sic piece.
tailed information about user tests are described in Section
5, and results with discussion in Section 6. The last sec-
3. MUSIC FLOWGRAM
tion concludes the paper with a summary and our plan for
future work. The idea of music flowgram for music visualization is based
on dramatic structure of storytelling. Freytag explained
2. RELATED WORK the structure of each story with two-dimensional graph vi-
sualization of tension progress [14]. Our idea is apply-
There has been some research on visualizing the structure ing a similar concept to music: drawing continuous two-
of a music piece, both in data visualization and MIR ar- dimensional graph that shows the change of music by time.
eas. The majority of them exploited the repetitive struc- If listeners can see a dramatic structure of music, they could
ture of music using self-similarity within a piece. Wat- feel more comfortable to concentrate on the music because
tenberg visualized it using an arc diagram that connects they can clearly see when the tension will increase or de-
each repetitive part with an edge being drawn as a semi- crease. This is similar to watching an opera, for which
circle [2]. Foote visualized the self-similarity as a two- people are encouraged to know dramatic structure before
dimensional matrix where each element is calculated from watching. The visualization will also help the listeners to
similarity between two audio frames [3]. Müller and Jiang recall the sequence of the music, as people remember the
extended it to a scape plot representation that visualizes order of opera story based on the order of important events.
the repetitive structure with varying segment size. Other A similar type of visualization is waveform visualization
researchers combined this self-similarity information with or volume graph. It shows the volume progress of the mu-
volume transitions over time [7]. There has been also re- sic so that users can see which part is loud or quiet. This
search that applies this structural information to music lis- type of visualization is used in SoundCloud 1 . Though vol-
tening interfaces [8, 9]. ume is a highly important factor in deciding characteristic
Other than those based on repetitions in music, some work of music, there are other quantitative parameters to explain
visualized the structure using tonality such as key change the music. Spectrogram is another way to show the vari-
over multiscale segments [10]. Malt and Jourdan presented ance of music as a two-dimensional image. However, it
a visualization method using statistical characteristics of contains too much details to deliver meaningful musical
spectral information, including spectral centroid and stan- information. Thus, more compact representations, which
dard deviation of the audio spectrum [11]. They illus- effectively extract musical elements, is needed.
trated the change of those information over time on a two- Considering that emotion is the most influential high-level
dimensional graph, adding amplitude information as a color concept on listeners, we focus on musical elements that are
of the graph. However, the most of the mentioned research associated with the emotional aspects of music. Among
have not released an end-user application so that general many suggested elements in this regard [15], we choose
users can render their own visualization. Furthermore, this loudness, tempo and harmony. For visualization, we repre-
research lacks user test or human side experiments that ver- sent them with volume, onset density and auditory rough-
ify its effect and usefulness for listeners. ness, respectively, as below.
Besides the automatic visualization methods using au-
dio files or MIDI files, visualization of semantic structure 3.1 Volume
of music is also proposed [12]. This method contains a
lot more information than repetitive structure, for exam- Unlike other genres of music, classical music consists with
ple, traditional structure analysis of sonata form, motif de- many different sub-parts, each of which has a different
velopment, and how the role of each instrument changes loudness characteristic. Therefore, temporal differences of
through the piece. But all of the information used in vi- loudness can explain the structural information of music
sualization is manually extracted from written explanation effectively. We represent the loudness with volume which
of the music, and cannot be automatically computed from is simply calculated as frame-level energy. Though more
audio files. complex measures of loudness could be adopted, we as-
There is also music psychological research about visual- sume that the volume is sufficiently effective in complex
izing whole music [13]. This research tested how people musical sound.
describe short music with graphical representations. Par-
ticipants are asked to “make any marks” to describe five 3.2 Onset Density
short orchestral works after listening to the music. The re- Emotion of music is highly dependent on the tempo char-
sult showed that musically trained participants more tended acteristic of music, i.e, whether the music is fast or slow.
to describe music with abstract representations such as sym- Beats per minute (BPM) is a typical way of representing
bols and lines. Most frequently used mapping was X-axis it. However, the single speed measure is not sufficient to
as time and Y-axis as pitch. The other type was pictorial describe the tempo characteristic of music because note
representations, which were mostly drawn by untrained
participants. Among 30 musically trained participants, 24 1 www.soundcloud.com
passages can vary dramatically in the same tempo. For
example, a long note and multiple short notes can be lo-
cated in a single beat but they produce a different nuance.
For this reason, we represent the tempo characteristic with
the number of notes per second. Since we need to have
overall trend of local note population rather than the exact
number of notes for visualization, we use a simple onset
detection algorithm which counts note onsets in a selected
frame based on amplitude information.

3.3 Auditory Roughness Figure 1. Change of roughness term Y by amplitude dif-

Quantifying the harmonic feature of music from audio is ference between two sinusoidal wave, where one ampli-
typically carried out by chord recognition. However, recent tude is fixed to 1 and the other varies
work pointed out the limitation of automatic chord recog-
nition [16]. Identifying chords can be arguable even for
musicologists, especially for complex classical music. The
research shows that the maximum agreement ratio between
two chord annotations among four annotations on “I Saw
Her Standing There” by The Beatles was only 65%. Also,
classical music includes atonal music or late-romantic mu-
sic like Wagner’s “Tristan und Isolde”. This makes hard to
employ automatic chord recognition for classical music.
Instead, we use auditory roughness which can represent
the tonality with a single value. It is a term used in the
acoustics and psychoacoustics literature to describe buzzing
sound quality that is produced by two sounds with differ- Figure 2. Change of roughness term Z by frequency dif-
ent pitch that is distinguishable but close to each other like ference between two sinusoidal wave. The frequency dif-
minor seconds interval. This feature is strongly associated ference is represented as fmax /fmin
with harmonic dissonance. For example, major seconds or
minor thirds in a low register, which are usually avoided as 3.4 Visualization Scheme
dissonant intervals in the western musical tradition, makes
high roughness. There are various models to calculate the As we mentioned above, representing music using a 2-D
auditory roughness quantitatively. Among others, we em- graph with x-axis in time is widely used mapping. Among
ploy a model presented by Vassilakis [17] that uses two three features, volume is the most accurate feature to cal-
sinusoidal components with frequency f1 and f2 and am- culate from audio data. Also it is the most dynamic fea-
plitude A1 and A2 : ture. Therefore, we use the volume as a Y value in our
visualization. For the other two features, a graph color
and a background color are used as mapping targets. Since
R = X 0.1 × 0.5Y 3.11 × Z the auditory roughness is often correlated with the volume,
X = Amin × Amax mapping it to the graph color can be somewhat redundant.
2Amin We thus map the onset density to a graph color and the
Y = auditory roughness to a background color.
Amin + Amax (1)
Z = e−b1s(fmax −fmin ) − e−b2s(fmax −fmin )
4. IMPLEMENTATION
0.24
s=
(s1 fmin + s2 ) Our purpose is building an automatic music visualization
system that can be easily utilized by general users. Pre-
where Amin = min(A1 , A2 ), Amax = max(A1 , A2 ), fmin vious visualization systems use symbolic data such as a
= min(f1 , f2 ), fmax = max(f1 , f2 ), b1 = 3.5, b2 = 5.7, MIDI file or pre-analyzed text data, which are not readily
, s1 = 0.0207, s2 = 18.96. The term X represents the obtainable by listeners. Also, many of them ask users to
dependence of roughness on intensity. For better under- install a stand-alone application, which might have some
standing of this equation, we illustrate how the term Y compatibility issue on user side or require some extra ef-
and Z change over the A1 and the frequency difference forts. Considering these problems, we design our system
fmax − fmin , and fmin , respectively, in Figure 1 and Fig- such that, if users can access to audio content on a web
ure 2. They show that roughness is higher when the am- browser, the visualization is immediately rendered from
plitude of two sin wave is similar, and the minimum fre- the audio file. We used web audio API which was devel-
quency is lower. The roughness of complex sound can be oped for various audio applications under HTML5 specifi-
calculated by summing the roughness of each combination cation. Therefore, our visualization system can be run on
of two sinusoidal components in the sound. many web browsers such as Google Chrome and Mozilla
Firefox 2 .

4.1 Feature Extraction

When a user loads an audio file on the system, the file
is decoded to linear PCM data and saved as a buffer on
the browser memory. Then, samples are segmented by a
Hann window. The volume is calculated using root mean Figure 3. Music flowgram of Salonen’s Violin Concerto
square of audio samples for each window. The algorithm
for counting onsets uses local maxima of the calculated
volume sequence. Specifically, it compares each volume
value in the array with the next value. If it increases, the
increased amount is saved. This is accumulated if the vol-
ume keeps increasing. If the volume decreases, the algo-
rithm compares the accumulated amount and a threshold.
If the accumulated amount is larger than the threshold, it
Figure 4. Music flowgram of Sibelius Finlandia
is counted as an onset and the accumulated value is reset.
Otherwise, only the accumulated amount is reset. Though
it is not very sensitive for detecting note onsets in legato 5. EVALUATION
passages, it is sufficient for detecting overall onsets in the
music. To evaluate the effectiveness of music flowgram, we set up
To calculate the auditory roughness, we use a DSP library a user test for classical music listeners. The test consists
for fast offline FFT processing 3 and detect 50 peaks from of two scenarios that imitate situations where they listen
the local maxima of the magnitude response. We then cal- to music on YouTube. The first case is listening to music
culate the auditory roughness from every pair of the peaks without any video, which contain a static image or slide
and add them all. show of images such as an album cover or a picture of com-
The result of these three features are all normalized and poser on YouTube. The second case is searching a specific
scaled to the size of HTML canvas. Since the auditory part of a video when a short extracted audio clip is given.
roughness tends to be somewhat correlated with the vol- The participants were 15 undergraduate students from Ko-
ume, we make it up by dividing the auditory roughness by rea Advanced Institute of Science and Technology. All of
the volume with a constant value. This compensation can them were a member of amateur orchestra, with a regu-
emphasize the dissonance in quiet passages. lar experience of listening to orchestral music. We divided
them into two groups.
4.2 Visual Mapping
5.1 User Test A: Listening and Recall
We visualize the features using a 2-D graph with a single
continuous curve on an HTML canvas. The x-axis repre- In our previous research [5], we found that those who listen
sents time and its width is fixed regardless of the length of to the music with a volume graph can identify an extracted
input files. The y-axis represents the volume curve along audio clip better than those who listened to the same music
with two color mappings. We downsample the features without it. In this test, we extend it to a more active recall
such that a set of values are mapped to a pixel by averag- task. That is, we evaluate how much music flowgram will
ing. Onset density is mapped to the color of vertical lines help a listener to concentrate on and memorize the music.
below the volume curve. Lines with high onset density are After listening to a movement from a symphony, partici-
colored with high saturated red on HSV scale. Auditory pants are asked to describe how music changed over time
roughness is mapped to the color of vertical lines above in an objective expression. This task requires much more
the volume curve. Lines with high auditory roughness are accurate musical memory compared to previous research
colored with bright clear blue on RGB scale and so those experiment, because participants need to recall the music
with low auditory roughness is with dark dim blue. without any audio cue.
To design the test environment to be more realistic, we
4.3 Output used YouTube as a listening interface. We uploaded two
videos for each music, one with a music flowgram and the
Once the visualization is generated and shown on the screen, other with an image of album cover 4 .
users can freely navigate the music through clicking on the Participants listened to three music pieces and described
visualization. It takes a mouse input and changes the play- them in three different situations: with an album cover, a
ing offset of the music immediately. A progress bar shows music flowgram only while listening, and a music flow-
the current playing position. Users can make a music flow- gram while listening and writing. Writing a note during
gram of a very complicate contemporary orchestral work, the listening was not allowed. That is, participants had to
for example, Salonens Violin Concerto, which is about 29 describe music only with their memory or the music flow-
minutes long as shown in Figure 3. gram. The music for the tests were selected from rarely
2 http://jdasam.github.io/visualization/main.html 4 https://www.youtube.com/playlist?list=
3 https://github.com/corbanbrook/dsp.js/ PLq7cRTjnYEi6fvwS3fSY1PsEDpNQdVRXA
 Song RK1-1 RK1-4 B1-4
(length) (9:04) (6:46) (7:18)
MF, maintained
Group A Album cover Music flowgram (MF)
after the listening
MF, maintained
Group B Music flowgram Album cover
after the listening

Table 1. The setting of the user test A

performed repertoire so that none of the participants had

possibly listened to the piece before. In addition, we se-
lected music with similar style to reduce the effect of dif-
ference between the music materials. Selected materials
are Rimsky-Korsakovs Symphony No. 1, first movement
(RK1-1) and fourth movement (RK1-4), and Borodins Sym-
phony No. 1, fourth movement (B1-4). The detailed set- Figure 5. An example of the proposed system
ting is shown in the Table 1.
To properly guide participants response, we provided an Song M5-3 R2-2
example paragraph like below: (length) (18:05) (10:11)
The piece starts with fanfare of trumpets. Then, cello Group A YouTube with MF YouTube only
plays quiet and smooth theme. Some variations of the theme Group B YouTube only YouTube with MF
are followed, and the dynamics get stronger. Then orches-
tra tutti play the main theme in faster tempo. Flute plays Table 2. The setting of user test B
fast and virtuosic solo passages. The same orchestra tutti
is followed. The key is transposed from major to minor,
and woodwinds play march-like melody. This melody de- of the web page included a YouTube player with a music
veloped further by brass and violin in fortissimo. The fan- flowgram of the audio of the selected video, and the other
fare from the beginning reappears in a more splendid way. page only included a YouTube player. We arranged a dif-
Main theme is played again by cello. The orchestra tutti ferent setting for each group, as described in the Table 2.
in the middle appears again with additional coda, which
finishes the music. (Dvořák Symphony No. 8, fourth move-
6. RESULT AND DISCUSSION
ment, translated into English by the first author)
After the listening and recall test, participants are asked 6.1 A. Listening and Recall
to score each visually represented feature, based on how
well it represents the musical characteristics. Also, partic- We checked the descriptions written by participants and
ipants were asked about how the listening experience with ma-tched the description to the corresponding part in the
the music flowgram was different from that without it. music. Any phrase or sub-phrase that can specify a re-
hearsal letter from music was counted as a correct answer.
Considering the participants are all non-professional musi-
5.2 User Test B: Searching an Excerpt
cian and the description was written after only one listen-
The second test was searching an excerpt of the music in ing attempt, we allowed a certain level of wrong descrip-
the YouTube video for the purpose of validating that a mu- tions, for example, incorrect instrument and melody iden-
sic flowgram can help a listener to find a specific part more tification. Many participants had confusions whether the
quickly. melody was reoccurred or newly introduced, and whether
We rendered a music flowgram from downloaded You- the solo instrument was flute or oboe. But the confusion
Tube videos, and attached it below the YouTube player, as between string and wind instrument was not allowed. The
shown in Figure 5. The music flowgram image is linked to paragraph below is an example of participants’ answer,
YouTube video so that users can select the playing position which is translated into English by the first author. Be-
by clicking a specific position on the graph. We compared cause Korean does not usually use a definite or indefinite
this setting to a YouTube player only page that contains the article, melody is translated without an article. A letter in
same video. Since the difficulty of searching task is largely a parenthesis is annotation made by authors, which means
influenced by a characteristic of the excerpt, we chose ex- a rehearsal letter of corresponding part in the score.
cerpts from movie scenes, rather than choosing arbitrarily. Music starts with brass and bass( A). Main theme starts
The selected movie is Lorenzos Oil and Birdman, which with bass and cellos, violas, violins takes over the theme
used a short clip from the third movement of Mahlers sym- and the pitch register goes higher(B). All the instruments
phony No. 5 (M5-3), and the second movement of Rach- play main theme in fortissimo(C). Clarinet plays melody
maninovs symphony No. 2 (R2-2), respectively. and string plays melody(F). Then, the brass section is added
Each participant watched the movie clips and searched and play majestic chord(G). The same pattern is repeated
the excerpted music parts on the YouTube video. There (Repetition). The flute plays melody and similar pattern is
were two movies and a corresponding web page including played(I). At the last part, flute and oboe appear(T). After
YouTube videos of the music used in the movie clips. One majestic brass, timpani finish music (U).
 Song RK1-1 RK1-4 B1-4 It was helpful for...
understanding an entire structure of music.
Group A 6.75 6.88 8.38 anticipating when will music get excited or relaxed.
(n=8) (SD: 1.49) (SD: 2.23) (SD:2.45) Positive knowing length of each section.
Group B 9.14 6.86 6.71 comments recognizing which part is repeating.
(n=7) (SD: 1.07) (SD: 1.68) SD: 1.98) memorizing change of dynamic.
concentrating on music by comparing my own
anticipation and the actual music
Table 3. Average score and standard deviation of correct It was disturbing to concentrate on listening to the music.
description in user test A It made listening less interesting because
Negative
it was easy to anticipate music.
comments
It made me keep thinking about how much the graph
By this criteria, we scored how many parts of the music is accurate.
is described in each description. The rehearsal letters are
Table 4. Examples of comments on listening with music
referenced from an edition of Muzgiz, Soviet State music
flowgram
publishing house, for both symphonies. The data analysis
was in blinded name to avoid bias. Parameter Average score (1 to 5)
The result on the Table 3 shows that the group with the Volume 4.6 (SD: 0.51)
music flowgram recalled more parts of the music than group Onset Density 3.4 (SD: 0.74)
with the album cover image, regardless of whether the mu-
Auditory roughness 1.9 (SD: 0.74)
sic flowgram was provided until the end of the writing step.
This result shows that the music flowgram can help a lis- Table 5. Evaluation for individual features by participants
tener to memorize music more precisely as we expected.
There was almost no difference between two groups in
the case of fourth movement of Rimsky-Korsakovs sym- standing the entire structure of the music.
phony, for which the only difference was the presence of Participants evaluation on each visual parameter is shown
the music flowgram during the writing step. From this re- in the Table 5. Most of the participants gave the highest
sult, we infer that the music flowgram was easy to remem- rate to loudness. The evaluation of the auditory roughness
ber and recall, so that there was almost no disadvantage of was not positive. This is partially because of its visual
not watching it again while writing the description. mapping to background color. Some of the participants
During the analysis, we found that the ratio of partici- pointed out that the change of background color was not
pants who mentioned the repetition of the piece was higher easily recognizable.
in the group with the music flowgram. Both the first move-
ment of Rimsky-Korsakovs Symphony No. 1 and the last 6.2 B. Searching an Excerpt
movement of Borodins Symphony No. 1 are in sonata- The result of user test B is on Table 6. Because the task
allegro form that includes a repetition of the exposition. is largely dependent on participants pre-knowledge about
Four out of seven participants of group B mentioned the the music material, we put answers from participants who
repetition of the first movement of Rimsky-Korsakovs sym- know the material well separately in Table 7.
phony, while only one out of eight participants mentioned
For both pieces, it took less time for the group with music
in the other group. In the case of Borodins symphony, six
flowgram to find the target part as we expected. In the case
out eight participants mentioned it in the group A, while
of Mahlers symphony, the excerpt was in the very last part
two out of seven in the group B mentioned it.
of the music, which contains a loud brass section. So it was
Recognizing the repetition is important for understand- a tough task to find it by navigating from the beginning of
ing a structure of music. Most of musical forms in clas- the music. But when the music flowgram was provided, the
sical music include repetition of main part. This is one of participants could only concentrate on the loud part. This
the reasons why the former research about music structure advantage greatly reduced the searching time, especially
focused on repetitive structure. The repetition of exposi- for the participants who do not know the piece very well.
tion is an important characteristic of a sonata-allegro form. On the other hand, the excerpt form Rachmaninovs sym-
This result also shows that our music flowgram is helpful phony is in an early part of the music. So many partici-
for listening and understanding classical music. pants can easily find the part by navigating without music
Comments from participants, which is shown in the Ta- flowgram. This is the reason why the difference between
ble 4, also support the previous results. Many participants the two group is slight. But it is worth mentioning that
answered that the music flowgram was helpful for under- the shortest record was only 6 seconds, which included the
standing the entire structure of music. There was only one
participant who wrote negative comments only. Some of
the comments contradicted each other; some participants Song M5-3 R2-2
answered that the listening was more interesting because Group A 170 seconds 59 seconds
he can anticipate, but the other answered that it was less (n=6) (SD: 138) (SD: 32.9)
interesting because it was easy to anticipate. Three out of Group B 377 seconds 40 seconds
fifteen participants answered that they preferred audio only (n=6) (SD: 153) (SD: 29.6)
listening environment. But they also commented that just
taking a look at the visualization was helpful for under- Table 6. Results of user test B (average of consumed time)
 Song M5-3 R2-2 [4] M. Müller and N. Jiang, “A scape plot representation
Group A 14 seconds 15 seconds for visualizing repetitive structures of music record-
(n=2) 43 seconds 21 seconds ings.” in Proceedings of the 13th International Society
Group B for Music Information Retrieval Conference (ISMIR).
85 seconds 6 seconds
(n=1) Citeseer, 2012, pp. 97–102.

Table 7. Results of user test B with participants who know [5] D. Jeong and Y. Noh, “Music visualization using vol-
well the material ume graph and its effect on classical music lstening,”
2014.
loading time for the YouTube player. One of the partici- [6] D. Jeong, “Music visualization using flow graph and its
pants found the excerpted part with a single click on the effect on listening to classical music,” Master’s thesis,
music flowgram. Since the excerpt contains legato passage Korea Advanced Institute of Science and Technology,
of strings, the participant could easily find it by searching 2015.
a part with low onset density.
[7] N. Kosugi, “Misual: music visualization based on
acoustic data,” in Proceedings of the 12th International
7. CONCLUSION Conference on Information Integration and Web-based
In this paper, we have presented an automatic visualiza- Applications & Services. ACM, 2010, pp. 609–616.
tion method for representing music in its entirety. The
[8] M. Goto, “Smartmusickiosk: Music listening station
goal of our visualization is showing how the music changes
with chorus-search function,” in Proceedings of the
from beginning to end. The method visualizes music with
16th annual ACM symposium on User interface soft-
three audio features like volume, onset density, and audi-
ware and technology. ACM, 2003, pp. 31–40.
tory roughness, which are highly associated with loudness,
tempo, and dissonance, respectively, in musical character- [9] M. Goto, K. Yoshii, H. Fujihara, M. Mauch, and
istics. These features are visualized as a two-dimensional T. Nakano, “Songle: A web service for active music
graph. We implemented the method on a web page us- listening improved by user contributions.” in Proceed-
ing Web Audio API and conducted user test for verifying ings of the 12th International Society for Music Infor-
the usefulness of our method in the listening and searching mation Retrieval Conference (ISMIR). Citeseer, 2011,
task. The results showed that listening to music with a mu- pp. 311–316.
sic flowgram helps listeners to memorize the music more
precisely. A music flowgram was also helpful for search- [10] C. S. Sapp, “Harmonic visualizations of tonal music,”
ing a specific excerpt from music. in Proceedings of the International Computer Music
Despite of the overall positive results, there is still a large Conference, vol. 1001. Citeseer, 2001, pp. 423–430.
margin for improvement. Auditory roughness, which is in-
[11] M. Malt and E. Jourdan, “Le bstd–une representa-
tended for representing the harmonic characteristic of mu-
tion graphique de la brillance et de lecart type spectral,
sic, was not satisfactory for many participants. For the fu-
comme possible representation de levolution du tim-
ture work, we are planning to improve our algorithm for
bre sonore.” in Proceedings of L’analyse musicale au-
detecting onset and calculating audio roughness. We are
jourd’hui. Crise ou (r)/’evolution ?, 2009.
also considering other audio features that can replace au-
ditory roughness such as tonal complexity [18]. Another [12] W.-Y. Chan, H. Qu, and W.-H. Mak, “Visualizing the
important challenge will be finding more intuitive and vi- semantic structure in classical music works,” Visualiza-
sually pleasing mappings for each parameter. tion and Computer Graphics, IEEE Transactions on,
vol. 16, no. 1, pp. 161–173, 2010.
8. REFERENCES
[13] S.-L. Tan and M. E. Kelly, “Graphic representations
[1] M. Müller, F. Kurth, and T. Röder, “Towards an effi- of short musical compositions,” Psychology of Music,
cient algorithm for automatic score-to-audio synchro- vol. 32, no. 2, pp. 191–212, 2004.
nization.” in Proceedings of the 5th International Con-
ference on Music Information Retrieval (ISMIR), 2004. [14] G. Freytag, Die Technik des Dramas. Hirzel, 1872.

[2] M. Wattenberg, “Arc diagrams: Visualizing structure [15] A. Gabrielsson and E. Lindström, “The role of struc-
in strings,” in Information Visualization, 2002. INFO- ture in the musical expression of emotions,” Handbook
VIS 2002. IEEE Symposium on. IEEE, 2002, pp. 110– of music and emotion: Theory, research, applications,
116. pp. 367–400, 2010.

[3] J. Foote, “Visualizing music and audio using self- [16] E. J. Humphrey and J. P. Bello, “Four timely insights
similarity,” in Proceedings of the seventh ACM inter- on automatic chord estimation,” in Proceedings of the
national conference on Multimedia (Part 1). ACM, 16th International Society for Music Information Re-
1999, pp. 77–80. trieval Conference (ISMIR), 2015.
[17] P. N. Vassilakis and K. Fitz, “Sra: A web-based re- [18] C. Weiss and M. Muller, “Tonal complexity features
search tool for spectral and roughness analysis of for style classification of classical music,” in Acoustics,
sound signals,” in Proceedings of the 4th Sound and Speech and Signal Processing (ICASSP), 2015 IEEE
Music Computing (SMC) Conference, 2007, pp. 319– International Conference on. IEEE, 2015, pp. 688–
325. 692.