Generative Art: Futuring Past

Audio-visuals in shared space as a metaphor for

mindscapes: generative creation in a network performance
Manzolli, J., Moroni, A., Partesotti, E., Hebling, E., Paiva, C.,
Dezotti, C., Spagnol, G., Antunes, M., Rossetti, D., Figueiredo, A.,
Cielavin, S., Pereira, C., Botechia, T.

Interdisciplinary Nucleus for Sound Studies (NICS),

University of Campinas (UNICAMP), Campinas, Brazil
www.nics.unicamp.br
e-mail: [email protected]

Abstract
This article presents an ongoing research focusing on the development of a platform
for generative networked creation. That is, a process of creation in which several
collaborators act on different parts during the artistic creation while a computer
network integrates communication between these agents and processes generative
transformations. Therefore, we established a working methodology with a group of
thirteen researchers of the Interdisciplinary Nucleus for Sound Studies (NICS), Ph.D.
students, postdoctoral researchers and senior researchers. Our proposal was to
distribute the stages of the collaborative process in separate generative modules and
develop a network architecture to enable interaction between the elements and,
finally, perform a multimodal work as a result of the information flow circulating in the
network.

Key words: generative art, audio-visual composition, networked performance,

collaborative creation.

1. Introduction
Studies on the possibility of using a network architecture and connectionism as a
paradigm for musical creation are already described in [3]. The authors argued that
"neural networks cooperate to produce a heuristic value that represents the worth of
each of musical fragments". Others developed the NEUROGEN with the idea of
using genetic algorithm and cooperating neural networks as architecture for a
generative musical composition [6].

Brown [2] explores computers as a vehicle for collaborative music making through
improvisation using computers on a network. They also point out that: "Networked
improvisation suggests the 'contemporary musicianship' which embraces the
computer as an instrument and the network as an ensemble and cyberspace as a
venue”. The authors comment on the last two aspects as follows: 1) The network as
an ensemble: "This allows ensemble activity to occur to separate sites for musicians
who share the network and the software, this means they can play together in a
collaboration where each can see and hear the result of their gestures have,
facilitating real musical communication between them” (idem, p. 3); 2) The
 Generative Art: Futuring Past

Cyberspace as a venue: "With algorithmic musical instruments that do not depend

upon continuous gestural input, the user can be both listener and performer, both
producer and consumer” (ibidem, p. 3).

The main concerning in our article is the conceptual viewpoint and interactive
perspectives that motivate our study. Next section presents a brief retrospect of
previous participation on the Generative Art Conferences which follows the
conceptual perspective that anchors our approach concerning collaborative practice
in contemporary art. After we shortly introduce the implementation of the network,
visualization and sonification, and finally, we discuss our understanding of generative
performing art as an open platform for collaboration and networking.

2. Retrospective: Generative Art

The study presented here dialogues with other presentations at the Generative Art
Conference. We presented a generative networked performance at the 17th
Generative Art Conference in Rome, 2014. The work MinDSounDS was a
performance involving a dancer, a flutist, BCI performer, and electronic music [10].
Similar to the work presented in this article, MinDSounDS was a performance where
a group of musicians and computers in a network produced a continuous exchange
of information between them1. The emerging audio-visuals were shaped by physical
actions, movements, music, and improvisation with interfaces and music instruments.
Therefore, the performance created meaningful relationships between agents while
they explore interactions with the visualization and sonification [10].

We have also studied how generative installations and interactive narratives can be
understood as an interdisciplinary research methodology to investigate new
paradigms on human cognition mediated by technology in the 18th Conference [11];
we presented how a multimodal opera was created and performed2 in the 19th
Conference [12]. In the 20th Conference we presented the Selfhood Installation3 [13].
Finally, a multimodal performance entitled as an Ode to Salvador Dalí's Christus
Hypercubus4 was discussed in the 21st Conference [14].

3. Conceptual Perspective
A tendency toward collaborative, participatory practice is undeniably one of the main
characteristics of contemporary art. Admittedly, these attempts are not new; one may
contend that this genealogy dates as far back as modern art itself. In the early
Romantic era, at the end of the eighteenth and beginning of the nineteenth centuries,
poets and artists started to form groups that bemoaned the separation of art from its
audience.

1
minDSounDS – 17th GenArt, Rome, https://youtu.be/cHJ1fRza9Ig
2
Descobertas – 19th GenArt, Florence, https://youtu.be/zCRq9zVPLew
3
Selfhood – 20th GenArt, Ravenna, https://youtu.be/B2Ryo6Y9Rz0
4
Ode to Christus Hypercubus – 21st GenArt, Verona, https://youtu.be/Ks3X80TZkMs
 Generative Art: Futuring Past

By the middle of the nineteenth century, the strategy Richard Wagner set forth in his
seminal essay “The Art-work of the Future” [3] is still central to any discourse in
Participatory Art. In order to reach the artwork of the future, the artists should
overcome the distinctions between various artistic genres or, as we call today,
different artistic media. The synthesis of artistic genres is for Wagner more a means
to an end: the unity of artists among themselves and the unity of artists and the
people. In this way participatory art can be understood not only as a reduction, since
the author forgoes his subjective authorial power by reducing his own creative role,
but also as an extension, of authorial power, whereby the viewer forfeits his secure
external position, his aesthetics distance from the artwork, and thus becomes not just
a participant but also an integral part of the artwork.

During the sixties, artists collectives, as well as happenings, performances, and

similar events, were famously reborn on a worldwide scale. Among them, to name
but a few examples, were Fluxus, Guy Debord´s Situationist International, and Andy
Warhol´s Factory. In all of these cases, the twofold aim was both the collaboration of
different artists and the synthesis of all artistic media. However, central to these
activities was the readiness of artists to forego their isolated, elevated, privileged
position in relation to the audience [7].

By the same time, in another context, an analysis methodology based on generative

theory emerged, focusing mainly on the field of language theory. Noam Chomsky
proposed "generative and transformative" models that analysed the different
sentence structures. Those were deductive models whose objective was to describe
the process of producing abstract chains and to determine a set of "generative" rules
that included syntax, interpretive semantics, and phonology.

The works of computer art, in its early beginnings, were based on rules or basic
parameters, from which it was established repetitions and variations. The roots of this
practice lied in the Neo-constructivist current in the plastic arts. Nevertheless, while
Constructivism focused on the application of mathematical and/or geometric models
to art, neo-constructivism, as well as Generative or Processual Art, worked with the
visualization of algorithms that enlarged their formal field by introducing new
processes. The difference is that the "manual" work of most neo-constructivists
compelled them, for practical reasons, to restrict themselves to structures of relative
simplicity, while computer-generated works allowed the creation of complex
structures.

The purpose of several computer art artists to generate the work from the
development of a certain process of repertoire selection and statistical distribution
certainly retained the proposal of the generative grammar of considering generative
mechanisms of construction, related to the specific characteristics of art based on
procedural creation. For Nake [15], the essentially new one in Computational
Aesthetics is the concept of algorithm. The works based on this generative aesthetic
allow the creation of aesthetic situations specified by several different but limited
steps. For Georg Nees [16], the computer is a generator of creative processes,
whose products are models of work of art. The essence of computer-aided work is,
 Generative Art: Futuring Past

according to him, the selection and distribution of signals in a given field, which he
calls Composition.

Still in the sixties, while the generative approaches emphasize the procedural
resolution of the work, other computational approaches look at the role of the
observer. Artists of the time analyse both in their work and in their discourse the
possibility of reaching a work of "dialog" art, where the point of reference is not the
mere circulation of information, but a true "aesthetic communication." Subsequently,
comes the telematics, or the art of telecommunication and network art. The art that
transits through the telecommunication network is based on a type of open
interpersonal communication and, therefore, is an art without a certain public. The
fundamental element is communication, art as traffic. Together with the
communication process is the idea of participation, which happens to the extent that
the artist offers the public a field of action, not a definite and finished work. In this
chain, all artistic activity is related to information. In the creative act, the information is
generated, when received by the public it is processed, and through the medium is
the transmission of information, which is known as communication.

In order to make possible communication between humans and machines, Turing

[20] had emphasized the need to generate an intermediary element that would
function as a translator of information transmitted in symbolic language to the
computer language (binary code). The idea of Turing predates the conception of the
interface between people and computational systems since, unlike direct
communication between machines, the direct coupling between minds and machines
until today is still not completely possible. Man-machine interfaces play a role similar
to the means humans need to communicate and enable coupling between different
systems.

At first, the information processed by the computer was not visible to the user. This
obstacle began to be overcome from 49 with the development of a first monitor with
an interactive, dynamic and visual display under the direction of Jay Forrester at MIT,
which allowed manipulating the information directly in the window with the optical
pens of the time. And from there we arrive in today's interactive systems, with the
immersive technologies, intelligent devices and the increasingly complete discourses
from the fields of cognitive sciences and neuroscience on the notions of reality and
observer that are placed at the core of the area of interactivity. What, then, of the
virtual spaces and interactions that progressively determine and enable cultural
practice, particularly in our time?

Perhaps as a remake of the happenings in the sixties, but with the possibilities of the
environments available now, this project was conceived and presented to an
interdisciplinary group of thirteen researchers, with musicians, artists, developers,
physiotherapists, gymnasts among them. The proposal was: Do it! The available
environments were presented: Processing, Pure Data, Max MSP and the
communication protocol, OSC. Like in Fluxus, the twofold aim was both the
collaboration of different artists and the synthesis of all artistic media. However,
central to these activities was the readiness of artists to forego their isolated position
 Generative Art: Futuring Past

in relation to the creative process towards an engaged, collaborative, participatory

approach. Simple as that.

4. Development: Network, Visualization and Sonification

We introduce in this section the three main elements of the study presented here.
Firstly, we describe the initial network architecture which is based on a data flow that
starts in the gesture captured by two interfaces (Kinect and Leap Motion) and then is
sent via OSC protocol to other computers in the network that control the visualization
and sonification. The subsection introduces the visualization that was developed in
two aspects: 1) geometric transformations and 2) video clips of silent movies. Finally,
we describe how the sound generation was developed from granular sounds and
sound textures.

4.1 Network

The first network architecture developed for the study is based on the principle that
the information flow has its origin in the gesture of a conductor that represents the
interpreting action of the whole system (see figure 1). The first tested performance
metaphor is that of a networked audio-visual generative instrument. Therefore, each
of the elements of the network is represented by computers that perform different
tasks and are connected to each other by the OSC protocol, all in the same using an
ad hoc network. In addition to the sound and image processing units, each entity of
the network has an editing and display controls. Finally, the data flow is stored so
that one can make future analyses of the interaction between the different
mechanisms of interaction between each agent after performances.

Fig
ure 1: network architecture

When in operation the network architecture obliges a circular causality, which is

generated with data of cinematic quality, derived from a movement interpretation that
is tracking from the Kinect or Leap Motion interfaces. Cognitively, acceleration carries
 Generative Art: Futuring Past

information of intention. An abrupt change of acceleration creates an expectation

breach and, in this system, corresponds to the start point. These two aspects of the
movement of the conductor are further explored in the sonification.

4.2 Sonification

The sonification developed for this study was based on principles already
implemented to compose the multimodal work “Ode to Christus Hypercubus”, which
was discussed during the 21th Generative Art Conference [14]. The main sound
devices explored in the sonification are: 1) long duration low-frequency sound with a
dense texture that remains during the whole work as a mechanism of connection
between the two distinct visual elements the geometric transformations and the video
clips; 2) granular sounds with a variety of grain density and attack mechanisms that
depend on the recognition of the gesture captured by the two control interfaces
(Kinect and Leap Motion); 3) a set of incidental sounds to signal the insertion of new
visual elements, parameter changes, and video clip changes.

As a compositional process aided by the computer that connects sounds and visual
the main task was to fulfil the space with many sound alliterations. We already
discussed that a multimodal performance can be seen as a way to create a unified
experience where sound, image, and audience are merged in space and evolve
coherently in time [11]. Therefore, a constant musical drone accentuates resonances
in which listeners’ memory is expanded. Our approach was to study the relation
between granular sounds and sound diffusion by controlling digitally the generated
sounds and correlating granular synthesis spatialization with the Ambisonics
technique.

4.3 Visualization

Using the Processing environment, the visualization process was based on the
construction of two distinct elements: 1) geometric images generated by parametric
transformations in real time, 2) silent movies clips in black and white. These two
visual contents with their different natures were woven into the work. The objective
was to create contact surfaces between two different representations and seek to
explore their similarities and differences.

4.3.1 Geometric Transformations

The images, generated by a rather spartan program implemented in Processing

(p5.js), reproduce frames of a spiral in perpetual motion (figure 2 and figure 3). More
than the adoption of a certain mathematical objectivism, as it could mislead the
ellipsis in rotation regulated by trigonometric functions that the code implements, the
spiral refers in particular to a visual analogy of obsession, of "idée fixée", as used in
cinema. The "Fraser spiral illusion" that the video, confined to dark shades
exclusively, aims to accomplish, refers to the spirals that Fritz Lang uses as
metaphor for his protagonist's psychopathy in M – Eine Stadt sucht einen Mörder
 Generative Art: Futuring Past

(1931) or those that Alfred Hitchcock and Saul Bass use in Vertigo (1958), one of
the most renowned cinematographic studies on human obsession.

Figure 2: spiral perpetual Figure 3: converging frames

4.3.2 Video Clips

Originally, these video clips were cut to fit their related musical themes. These
relationships follow a logic of cataloguing the moods used in the silent movies and
distributed in dedicated publications called photoplay music, mood music, motion
picture music and so on. The most comprehensive of these publications is Erdmann
& Becce's huge manual, called Allgemeines Handbuch der Film-Musik of 1927. The
macro-scheme is based on a taxonomy indexed by the following 2 parameters:
‘Incidenz’, or themes for general, incidental music, and ‘Inhalt’, or themes specified
by dramatic and narrative contents. There are more than three thousand small
thematic fragments organized like this on the Handbuch. The music suggested by the
publications, in particular by Erdmann & Becce [1], were used here as a starting
point, but the resultant sonification will be generated by the network interaction.
However, the cataloguing of emotions and meanings was a way of producing insights
into the generative process.

Video Clip 1: The meeting between Eva (Hedy Kiesler, then Hedy Lamarr), and
Adam (Aribert Mog). While Eva was swimming her horse runs away, and is captured
- and returned - by young Adam. The mismatch of the escape of the horse generates
the encounter between the couple, in causality that triggers the passion and ecstasy
that will follow its irrepressible path.
 Generative Art: Futuring Past

Videoclip 1: Ecstasy, 1933, dir. Gustav Machatý

Video Clip 2: This meeting creates a communication between the two sides of the
city Metropolis: the apparent side of wealth, prosperity and unconcern, and the
underground side of poverty, the enjoyment of the proletariat, and the insalubrity.

Videoclip 2: Metropolis, 1927, dir. Fritz Lang

5. Discussion
As we understand in the article presented here Generative performing art is an open
platform, where the agents can be at the same time the subject, the main character
of the poetic lyrics, and the public who enjoys the performance [8][16]. As usually
happens, the construction of a creative process is not built by linear actions and
interactions, either by the artist who creates a message or by the observer who
receives it. This process, in fact, of any art, remains an open work, an open art form
 Generative Art: Futuring Past

that offers different interpretations, always according to the subjective experience of

the viewer.

In the digital age, technological devices could be seen as extensions of the body [9],
it is the experience and interaction between the work of art and the human body that
determines the subjective understanding of the artistic message, as mentioned by
Kozel [7]. In this context, in fact, when we talk about the artistic process, therefore,
we are not referring only to the 'at source' process anymore, that of the artist himself.
We are, in fact, referring to the same process carried out by the observer, who
through his exploration and his Sensorimotor Contingencies can understand, interact
and therefore directly modify the entire creative process of the work of art.

For this reason, the work presented by our research group 'NICS' is an open work, an
ongoing process to remote and in-person interactions, to demonstrate how the
artistic process can be modified non-linearly and can determine a cyclic co-
determination (concrete) and a constant (abstract) perturbation, thus increasing the
creative potential that emerges from the interaction between the parts.

Therefore, a non-linear communication takes place with the other agents, which
transform the information they receive and then return this stream of data so
changed to the network. This is possible through the design of various associations,
both concrete and abstract, through the technological piece that we propose, and
bearing in mind what the paradigm of Embodied Cognition underlines; in particular
the Sensorimotor Contingency Theory in which the sensorimotor contingencies
governing perceptual exploration in the different modalities [17].

6. Conclusion
The study discussed here focuses on network architecture as a means of providing a
framework of collaboration between different researchers so that everyone can
contribute to the accomplishment of creative work. In this sense, project members
shared the same physical space during the performance or may be in remote
locations. During the creation, the network architecture was mainly used to enable
prototyping of structures and development of ideas. In a second moment, the actions
were integrated and the performance itself is the process of adjustment between the
parties of the work.

In a previous article, we have argued that a theory of mind, including one of creativity
and aesthetics, will be critically dependent on its realization as a real-world artefact
because only in this way can such a theory of an open and interactive system as the
mind can be fully validated [21]. In this way we understand that, while we develop a
platform that allows creative interaction among researchers, this same methodology
is a way of studying creativity using computational models that by extension will
reflect models of the mind.

Considering the broad Embodied Cognition paradigm, perception is connected to

embodied experience, and within an interactive landscape “this perception is
augmented and can be directly tied to our emotional experience” [19]. That means
 Generative Art: Futuring Past

that through this artistic and experiential process, we can represent the subjective
self-perception that the user experience.

Acknowledgments
The research described here, developed at the Interdisciplinary Nucleus for Sound
Studies (NICS), UNICAMP, is supported by the Brazilian agencies São Paulo
Research Foundation (FAPESP) and the National Council for Scientific and
Technological Development (CNPq).

References
[1] BECCE, Giuseppe; ERDMANN. (1927) Allgemeines Handbuch der Film-Musik.
Hans BRAV, Ludwig (ed). 2 tomos; Tomo 2: Thematisches Skalenregister.
Berlin-Lichterfelde: Schlesinger'sche Buch- u. Musikhandlung.
[2] Brown, A. R. and Dillon, S. C. (2007). "Networked Improvisational Musical
Environments: Learning through online collaborative music making." In Finney, J.
and Burnard, P. (Eds.) Music Education with Digital Technology. London:
Continuum, pp. 96-106.
[3] Wagner, Richard (1895) The Art-Work of the Future Richard Wagner's Prose
Works Volume 1 Pages 69-213. Original title: Das Kunstwerk der Zukunft
Published in 1849 Sämtliche Schriften und Dichtungen: Volume I Pages 194-206.
[4] Bown O., Lexer S. (2006) Continuous-Time Recurrent Neural Networks for
Generative and Interactive Musical Performance. In: Rothlauf F. et al. (eds)
Applications of Evolutionary Computing. EvoWorkshops 2006. Lecture Notes in
Computer Science, vol 3907. Springer, Berlin, Heidelberg.
[5] Giannetti, C, Estética Digital: Sintopía del arte, la ciência y la tecnologia. ACC
L´Angelot, 2002.
[6] Gibson, P.M., & Byrne, J.A. (1991) NEUROGEN, musical composition using
genetic algorithms and cooperating neural networks. In Proceedings of the IEE
Second International Conference on Artificial Neural Networks (pp. 309-313).
London: IEEE.
[7] Groys, B. A Geneology of Participatory Art. In: Frieling, R., Groy, B., Atkins, R.,
Manovich, L. The Art of Participation, SFMOMA, 2009.
[8] Kozel, S. (2007). Closer: performance, technologies, phenomenology. London,
England: MIT Press.
[9] Ihde, D. (2002). Bodies in technology. Minnesota, United States: University of
Minnesota Press.
[10] Manzolli, J. (2014) mindDSounDs: live performance of a Generative Networked
Music. In: 18th Generative Arts Conference, 2014, Rome, Italy. pg 59.
[11] Manzolli, J. (2015) Multimodal Generative Installations and the Creation of New
Art form based on Interactive Narratives. In: 18th Generative Arts Conference,
2015, Venece, Italy. pg 32-44.
[12] Manzolli, J. (2016) DESCOBERTAS: Creativity as Libretto of a Multimodal
Opera. In: 19th Generative Arts Conference, 2016, Florence, Italy. pg 252-266.
[13] Moroni & Manzolli J. SELFHOOD, an experience around the self In: Generative
Art International Conference, 2017, Ravenna, Italy.
 Generative Art: Futuring Past

[14] Manzolli, J. (2018) Multimodal Architecture of the Ode to Christus Hypercubus.

In: 21th Generative Arts Conference, 2016, Verona, Italy. pg 172-182.
[15] Nake, F. On generative aesthetics – two picture generating programs. Proc. of
the International Symposium Computer Graphics, Uxbridge: Brunel University
1970.
[16] Nees, G. ″Design und Programm″. In: Gesamttextil 1971, pg. 38–41.
[17] Noë, A., & O’Regan, J. K. (2002). On the brain-basis of visual consciousness: a
sensorimotor account. Vision and mind: Selected readings in the philosophy of
perception, 567-598.
[18] Partesotti, E. (2016). L'interazione nelle tecnologie musicali di realtá mista: Il
prototipo e-mocumu como esempio multimodale con propositi terapeutici.
[19] Partesotti, E., Peñalba, A., & Manzolli, J. (2018). Digital instruments and their
uses in music therapy. Nordic Journal of Music Therapy, 27(5), 399-418.
[20] Turing, A. Computing Machinery and Intelligence. IN; Mind, vol. LIX n. 2236.
Oxford, Oxford University Press, 1950.
[21] Verschure & Manzolli, "Computational Modeling of Mind and Music", In
Language, Music, and The Brain, ESF Reports, MIT Press, 2013.