Luisa Feiersinger

Kathrin Friedrich
Moritz Queisner
(Eds.)
 

2


Image – Action – Space

IMAGE – ACTION – SPACE
SITUATING THE SCREEN IN VISUAL PRACTICE

Luisa Feiersinger, Kathrin Friedrich, Moritz Queisner (Eds.)

This publication was made possible by the Image Knowledge Gestaltung. An Interdisciplinary L­ aboratory
Cluster of Excellence at the Humboldt-Universität zu Berlin (EXC 1027/1) with financial support from the
German Research Foundation as part of the Excellence Initiative.

The editors like to thank Sarah Scheidmantel, Paul Schulmeister, Lisa Weber as well as Jacob ­Watson,
Roisin Cronin and Stefan Ernsting (Translabor GbR) for their help in editing and proofreading the texts.

This work is licensed under a Creative Commons Attribution-NonCommercial-No-Derivatives 4.0 License.

For details go to https://creativecommons.org/licenses/by-nc-nd/4.0/.

Copyrights for figures have been acknowledged according to best knowledge and ability.
In case of legal claims please contact the editors.

ISBN 978-3-11-046366-8
e-ISBN (PDF) 978-3-11-046497-9
e-ISBN (EPUB) 978-3-11-046377-4

Library of Congress Control Number: 2018956404

Bibliographic information published by the Deutsche Nationalbibliothek

The Deutsche National­bibliothek lists this publication in the Deutsche Nationalbibliographie;
detailed bibliographic data are available on the internet at http://dnb.dnb.de.

© 2018 Luisa Feiersinger, Kathrin Friedrich, Moritz Queisner,

published by Walter de Gruyter GmbH, Berlin/Boston

The book is published with open access at www.degruyter.com, https://www.doabooks.org and

https://www.oapen.org.

Cover illustration: Malte Euler

Typesetting and design: Andreas Eberlein, aromaBerlin
Printing and binding: Beltz Bad Langensalza GmbH, Bad Langensalza
Printed in Germany

www.degruyter.com
 Inhalt

Editorial
7 115 Nina Franz and Moritz Queisner
Image – Action – Space. Situating the Screen in The Actors Are Leaving the Control Station.
Visual Practice The Crisis of Cooperation in Image-guided Drone
Warfare
11 Aud Sissel Hoel
Operative Images. Inroads to a New Paradigm 133 Shannon Benna
of Media Theory Systems and Practices to Produce
Stereoscopic
Space on Screen
29 Erkki Huhtamo
The Spell of the Catoptric Television. Media 147 Jens Schröter
Archaeology, Topos Study, and the Traces of Viewing Zone. The Volumetric Image, Spatial
Attention Knowledge and Collaborative Practice

41 Tristan Thielmann 159 Carolin Höfler

Early Digital Images. A Praxeology of the Display Sense of Being Here. Feedback Spaces Between
Vision and Haptics
55 Jan Distelmeyer
Carrying Computerization. Interfaces, 177 Timo Kaerlein and Christian Köhler
Operations, Depresentations Around a Table, around the World. Facebook
Spaces, Hybrid Image Space and Virtual
69 Luisa Feiersinger Surrealism
Spatial Narration. Film Scenography Using
Stereoscopic Technology 191 Sarah Atkinson
Synchronic Simulacinematics. The Live
79 Lasse Scherffig Performance of Film Production
From Action Capture to Interaction
 Gestalt
203 Ina Neddermeyer
93 Kathrin Friedrich I Want to See How You See. Curatorial Practices
Screening Bodies. Radiological Screens and of Exhibiting Virtual Reality
Diagnostic Operations

103 Luci Eldridge

Working on Mars. An Immersive Encounter
through the Screen
Editorial

Image – Action – Space

Situating the Screen in Visual Practice

With the improved capability of imaging, sensor and dis- tices that require the linking between screen-based visu-
play technology, screens have become mobile or touchable alization and physical space. A smartphone mapping app
and, most recently, transparent. While a visualization that indicates one’s current location and orientation, for
on-screen is not necessarily related to the spatial context instance, requires that users situate themselves in space
beyond the screen, transparent displays allow users to see based on a two dimensional map. During a surgical inter-
simultaneously the physical space behind the display and a vention, to give another example, surgeons must cope with
visualization on-screen. The two observers on the cover of the limiting architecture of image display in the operating
this volume look at a future construction site through the room, in which information on screen may not align with
transparent display of a head-mounted device that super- their perspective on the patient’s body or with the scale
imposes an architectural model on their view (fig. 1). Its and orientation of relevant anatomy. Accordingly, surgeons
rendering coincides with the scale of the actual building, must ascribe an image of the patient, for instance a comput-
and the visualization adapts to the observer’s point-of-view er tomography, to the patient’s body cognitively in order to
in real time. While a juxtaposition of building and model act appropriately in a particular situation. Even a regular
on a separate screen would require continuous comparison television screen on a living room shelf creates a specific
between image and object, the head-mounted display com- viewing situation. But this situation is not adaptive to the
bines them in a joint perceptual space. The missing offset images on-screen – it does not correlate image and space.
between image and object puts forward a new practice of An example for the way in which screen-based visual
interacting with spatially related information: users can practices dissolve the distinction between image and space,
navigate through space by superimposing a transparent creating a hybrid and adaptive form of visuality, is the
mobile interface onto their field of view. location-based augmented reality game Pokémon Go. The
This example shows how screen-based media trans- application encourages users to explore physical space in
form the way we see and act: transparent displays constitute order to catch virtual figures displayed within the camera
a form of images that only work when they are situated. They stream of a mobile phone (fig. 2). By aligning camera image
shift focus onto the situation rather than to the result of an and physical space, Pokémon Go players perform operations
imaging process. Of course, screens are always embedded both within and beyond the boundaries of the screen. What
in the context of a situation, particularly those visual prac- seems to be a simple moment in a game is actually an intri-
 Editorial

cately structured visual practice: The in-game view layers convenient catch-all used to describe the research and study
photographic and animated elements depending on the of what we access through screens, perpetuating the idea
player’s location and within the camera’s field of view. The of the screen as passive conduit”, is only until recently that
mechanism of merging image, action and space in Pokémon screen studies have started to investigate the impact and
Go transforms viewing into using and emphasizes an active application of screens in particular situations and with
role of the image in guiding a user’s action and perception. regard to its actionability and material affordances. 3 Con-
The visual practice of Pokémon Go points to the meth- cepts from media theory and visual studies, such as disposi-
odological issue of how to analyze and theoretically frame tif, spatial images, operative images or mobile screens help
the situation and situatedness of screens. By shifting the to scrutinize screenness.4
focus towards their “screenness”, we intend to examine The authors analyze how screens are situated in visu-
visual practices by asking what a screen does rather than al practices by scrutinizing the dynamic, transformational
asking what a screen is.1 From this perspective, the question and performative characteristics of screen-based media
is less about what becomes visible, or what can be seen, but with regard to image, action and space. By focusing on their
rather about how the interaction with and through screens dynamic yet deterministic relations the volume presents
structures action and perception. The terms image, action an approach to screenness that focuses on the actionability
and space serve as analytical reference points for investigat- of screen-based media in all their different hardware and
ing how screens engender a situated and dynamic relation software configurations. Speaking of actionability empha-
between them. sizes the way in which multiple forms and configurations
This volume draws on the evolving debate about the
screen as “a concept in progress”, which has started to IT University Copenhagen, 2010; Workshop “Screen Operations. Condi-
tions of Screen-based Interaction”, Humboldt University Berlin, 2016;
inquire its defining status.2 While the screen “has become Conference “Situation Space. How Spatial Images Define the User’s Dis-
position”, Humboldt University Berlin, 2017.
Lucy Suchman proposes looking at the socio-material practices surround-
1 Monteiro 2017 (as in fn. 2), p. 3.
3
ing the application of screens by using the term “screenness”. Brit Ross Giuliana Bruno, Atlas of Emotion. Journeys in Art, Architecture and Film,
4
Winthereik, Peter A. Lutz, Lucy Suchman, Helen Verran, Attending to New York: Vers, 2002; Frank Kessler, Dominique Chateau, José Moure, The
Screens and Screenness. Guest Editorial for special issue of Encounters, Screen and the Concept of Dispositif – A Dialogue, in: Dominique Chateau,
in: STS Encounters 4.2 (2011), pp. 1–6. In a different vein, Lucas Introna and José Moure (eds.), Screens. From Materiality to Spectatorship – A Historical
Fernando Ilharco develop the notion of screenness for their endeavour of and Theoretical Reassessment, Amsterdam: Amsterdam University Press,
introducing a “Heideggerian phenomenological analysis of screens”. Lucas 2016, pp. 264–271; Miriam Ross, Stereoscopic visuality. Where is the screen,
D. Introna, Fernando M. Ilharco, On the Meaning of Screens. Towards a where is the film?, in: Convergence. The International Journal of Research
Phenomenological Account of Screenness, in: Human Studies 29 (January into New Media Technologies 19.4 (2013), pp. 406–414; Jens Schröter, 3D.
2006), pp. 57–76, DOI: 10.1007/s10746-005-9009-y. History, Theory and Aesthetics of the Transplane Image, London: Blooms-
Dominique Chateau, José Moure, Introduction. Screen, a Concept in Prog-
2 bury, 2014; Aud Sissel Hoel in this volume, pp. 11–27; Erkki Huhtamo, Ele-
ress, in: Dominique Chateau, José Moure (eds.), Screens. From Materiality ments of Screenology. Toward an Archaeology of the Screen, in: ICON-
to Spectatorship – A Historical and Theoretical Reassessment, Amsterdam: ICS. International Studies of the Modern Image 7 (2004), pp. 31–82; Nanna
Amsterdam University Press, 2016, pp. 13–22. See further Stephen Mon- Verhoeff, Mobile Screens. The Visual Regime of Navigation, Amsterdam:
teiro (ed.), The Screen Media Reader. Culture, Theory, Practice, London: Amsterdam University Press, 2012; Ramón Reichert, Annika Richterich,
Bloomsbury, 2017; Workshop “Touching the Screen”, University of Oslo, Pablo Abend, Mathias Fuchs, Karin Wenz (eds.), Mobile Digital Practices,
8 2015; PhD course “Framing Screens: Knowledge, Interaction and Practice”, Digital Culture & Society (DCS) 3.2 (2017).
Image – Action – Space

1  Mixed reality in architectural planning superimposes building and model. 2  Augmented reality game merging image and place according to user sight-line.

of screens challenge users to integrate imaging techniques screens and screenness stem from the idea of a pragmatic
and visual information into their action routines, viewing and theoretical triangulation of image, action and space,
habits and working processes. which stresses the adaptive and situated alignment of visu-
Foci of investigation are: first, the analysis of codes, alization, operationality and spatiality.
data and software that form the technological basis for both
acquiring and displaying visual information that already
define and structure action as well as decision-making. Sec- Figures
ond, the exploration of interface design that constitutes both 1  Malte Euler, Creative Commons, Attribution-NonCommercial-
NoDerivatives 4.0 International (CC BY-NC-ND 4.0), 2018.
conceptual and epistemic considerations that render inter-
2  Picture alliance, REUTERS, Chris Helgren, 2016.
faces and their affordances as a screen-based space of its
own. And third, the perceptual level and the investigation of
how the screen intertwines human senses, cognitive capac-
ities and physical actions. These analytical perspectives on 9
Aud Sissel Hoel

Operative Images
Inroads to a New Paradigm of Media Theory

There is much talk these days about images being somehow While these thinkers continue to be influential, the central
operative. This notion, which foregrounds the active doing argument of this article is that there is something new about
of images, is often invoked to make sense of the disruption how today’s scholars of operative images approach the topic
in the image economy brought about by computerization. of mediation. What is new is that there seems to be a shift in
In today’s digital media environments, human dealings underlying assumptions about the nature and roles of media.
with the world increasingly take place via various kinds of The thinkers considered in this text, therefore, are treated
images and screens that do more than just display visual as transitional figures standing on the verge of a new para-
information. As cameras become ubiquitous, images net- digm of media theory. The emerging, operational paradigm
worked, image data geotagged and databases navigable in of media theory is characterized by its deeper recognition
real time, the status of images seems to be rapidly changing. of the active dimension of images and media.
Among scholars of the image, there is a growing realization The new line of research into the agency and efficacy of
of the shortcomings of existing theories and concepts when images is highly promising, breaking new ground by putting
it comes to explicating key features of today’s digital image image theory on an altogether new track. More work needs
applications. The current focus on the operational aspects to be done, however, when it comes to articulating what is
of images, therefore, is frequently accompanied by a call for meant by the term operation in this context. Addressing
conceptual revisions. this need, the article probes the literature on operative
This article contributes to the ongoing attempts to images, discussing and comparing different approaches
develop an operational basis for understanding images. To to operative images along four lines: from the perspective
this end, it considers a selection of contemporary approach- of art (­section 1), from the perspective of new media pro-
es that, each in their own way, grant centrality to the oper- duction and use (section 2), from the perspective of media
ational aspects of images. In the literature under consider- archaeology aspiring to become exact science (section 3),
ation, there is a great deal of focus on machine vision and and from the perspective of visual studies (section 4).1 In
automation, as well as on the roles of new media in warfare
and political conflict. These topics, of course, are not at all
These four approaches are certainly not exhaustive of how operative images
1
new. They were also at the forefront of 1980s media theory, are conceived in the current literature, but they suffice to unearth system-
with Jean Baudrillard and Paul Virilio as notable figures. atic differences in how the notion of operation is currently understood.
 Aud Sissel Hoel

all these ­sections, I seek to lay bare how the emphasis on the taneously of a new stage in the history of machine vision
operational aspects of images puts pressure on established where the machines have started to see for themselves. The
notions of images. catalyzing event for the Eye/Machine trilogy was the out-
The rough overview of the literature undertaken in rage and sensation of the 1990–1991 Gulf War, where point-
sections 1–4 shows that there is a tension regarding the of-view footage from laser-guided bombs (popularly known
boundaries and scope of operativity. While some approach- as smart bombs) was widely broadcasted to TV audiences.
es conceive operative images as a new kind of images that The military deployment of eye machines prepared the way
supplements the larger category of traditional images, other for a new type of warfare – a “war at a distance” 2 facilitated
approaches aim for a deeper revision that challenges the by a new kind of images that Farocki terms operative images
very idea of what an image is. The overview also shows that (operative Bilder).
the notion of operation is under-theorized as a media-the- Farocki’s work addresses the changing status of imag-
oretical concept, since in many cases it is simply imported es in the context of intelligent machines. Commenting on
from other research fields, such as computer science. Farocki’s work, Trevor Paglen notes: “Instead of simply
One note before I proceed: The reader may have noticed representing things in the world, the machines and their
that the question relating to the active doing of images is images were starting to ‘do’ things in the world”. 3 Volker
addressed here in the wider context of media theory. While Pantenburg adds that Farocki was “one of the first to exam-
this may cause some initial confusion, it is certainly no coin- ine in depth the various uses of images as instruments”.4
cidence. As we shall see, the slippage into media theory hap- Both aspects, the interventional and the instrumental, are
pens continually in the literature on operative images, and reflected in Farocki’s much-cited definition, which holds
it happens for a reason – indeed, as a consequence of the that operative images “do not represent an object, but rather
operational approach: If we are to follow through with the are part of an operation”. 5 Thus conceived, operative imag-
ideas suggested by the thinkers considered in this article – es are utility images – working images that typically serve
that images are instruments, interfaces, measuring media, practical purposes tied to specialized tasks, such as, in this
manipulable diagrams – the boundaries between image and case, guiding remote-controlled missiles. Similarly referring
medium start to become porous, leaving both terms trans- to Farocki’s work, Thomas Elsaesser goes further by charac-
formed. terizing operative images as “instructions for action” – and
not only that, in the digital media environment, Elsaesser

Representation versus Operation

Which is also the title of the English single-track film based on the Eye/
2
Harun Farocki’s three-part installation Eye/Machine Machine installation. Harun Farocki, War at a Distance, Germany 2003.
Trevor Paglen, Operational Images, in: e-flux 59 (2014), http://e-flux.com/
3
(2001–2003) is a key reference point in the literature on journal/59/61130/operational-images/ (accessed May 27, 2018).
operative images. The installation, which explores the Volker Pantenburg, Working Images. Harun Farocki and the Operational
4
Image, in: J. Eder, C. Klonk (eds.), Image Operations. Visual Media and Polit-
relation between humans, machines and modern warfare, ical Conflict, Manchester: Manchester University Press, 2017, p. 49.
12 announces the advent of a new visual regime, and simul- Harun Farocki, Phantom Images, in: Public 29 (2004), p. 17.
5
 Operative Images

maintains, the instructive function seems to have become by modernists like Dziga Vertov, does not extend the
“the new default value of all image-making”.6 human prosthetically so much as it replaces the human
The example of the smart bomb accentuates another robotically.9
feature of operative images. In the words of Farocki, they
provide phantom perspectives on things. The term phantom Farocki himself also alludes to the idea of replacement,
here alludes to the use of phantom shots in early cinema, characterizing today’s picture-processing apparatuses as
that is, of recordings taken from positions not normally “sensory automatons” destined to replace and outperform
occupied by humans (Farocki gives the example of a camera the work of the human eye.10 The main novelty of operative
hung under a train).7 The phantom perspective relates, in images, then, seems to be that they, in the words of Martin
other words, to the capacity of machine-made images to leap Blumenthal-Barby, “require neither human creators nor
beyond the human scale, reporting on events outside the human spectators”.11 What sets operative images apart from
scope of human sensibility. In this respect, Farocki’s Eye/ other images is that they are “not originally intended to be
Machine series resonates with the exploration of machine seen by humans”; instead they are “supposed to function
vision in art works such as Man with a Movie Camera (1929) as an interface in the context of algorithmically controlled
by the modernist, avant-garde filmmaker Dziga Vertov.8 guidance processes”.12
Nevertheless, in the current context of intelligent machines, The last remark, that operative images function as
the leap beyond the human scale seems to be of a more rad- interfaces, is a key observation to which I will return. For
ical nature. Hal Foster puts it thus: now, I will focus on the “posthuman” 13 aspects of operative
images, which have lead scholars to question their very sta-
[The images treated by Farocki] are not authored, and, tus as images. Pantenburg, for example, comments:
as they mostly survey the predetermined, they appear
to be more automatically monitored than humanly [T]he operational image emulates the look and feel of
viewed. In this way Farocki intimates that a new ‘robo traditional images, but on closer inspection, this turns
eye’ is in place, one that, unlike the ‘kino eye’ c­ elebrated out to be a secondary function, almost a gesture of
courtesy extended by the machines: The computer does
Thomas Elsaesser, Alexander Alberro, Farocki: A Frame for the No ­Longer
6 not need the image.14
Visible. Thomas Elsaesser in Conversation with Alexander Alberro, in:
e-flux 59 (2014), http://e-flux.com/journal/59/61111/­farocki-a-frame-for-
the-no-longer-visible-thomas-elsaesser-in-conversation-with-alexander- 9 Hal Foster, The Cinema of Harun Farocki, in: Artforum (November 2004),
alberro/ (accessed May 27, 2018). p. 160.
Farocki 2004 (as fn. 5), p. 13, p. 20.
7 10 Farocki 2004 (as fn. 5), p. 17.
The continuity between these works has been explored in the literature. As
8 11 Martin Blumenthal-Barby, ‘Cinematography of Devices’. Harun Farocki’s
pointed out by Volker Pantenburg, the connection is made explicit by Faroc- Eye/Machine Trilogy, in: German Studies Review 38.2 (May 2015), p. 329.
ki in his installation Counter-Music (2004). Pantenburg 2017 (as fn. 4), p. 59, 12 Ibid.
fn. 3; For a detailed exploration of the connection, see David Tomas, Vertov, 13 The term posthuman is sometimes invoked in the discussion of machine
Snow, Farocki. Machine Vision and the Posthuman, New York: Bloomsbury vision. See for example Tomas 2013 (as fn. 8).
Academic, 2013. 14 Pantenburg 2017 (as fn. 4), p. 49. 13
 Aud Sissel Hoel

In the strictest sense, therefore, operative images “would relationship characteristic of the modern era”.19 A similar
have to be characterized as visualisations of data that could idea is exposed by Ingrid Hoelzl and Rémi Marie, who con-
also take on other, different guises”.15 Fortunately, Panten- ceptualize the digital transformation of the image in terms
burg does not leave it at that. He goes on to call attention to of a shift from geometry to algorithm, and from projection
how Farocki’s work is deeply influenced by the philosopher to processing. Because of this shift, the image is “no longer
Vilém Flusser and his ideas about technical images. Accord- a passive and fixed representational form, but is active and
ing to Flusser, “technical images” (such as photographs and multiplatform, endowed with a signaletic temporality that
television images) differ from “traditional images” (Flusser is not only the result of digital screening (or compression),
gives the example of cave painting) in that they “owe their but also a transfer across digital networks”.20 This implies
existence to technical apparatuses”.16 Consequently, tech- that the image is “no longer a stable representation of the
nical images and traditional images mean in completely world, but a programmable view of a database that is updat-
different ways: While technical images are “computations ed in real-time”, and hence, that it “no longer functions as
of concepts” that arise “through a peculiar hallucinatory a (political and iconic) representation, but plays a vital role
power that has lost its faith in rules”; traditional images are in synchronic data-to-data relationships”.21
“observations of objects” that arise through “depiction”.17 The answer to Pantenburg’s question, whether opera-
While Flusser’s category of technical images comprises tive images are images at all, depends, of course, on how one
pre-digital images such as photographs and television imag- chooses to define the term image. Thanks to the frequent use
es, contemporary scholars typically draw the line in a differ- of a contrasting rhetoric by scholars of the image, we get a
ent place. William Uricchio, for example, in his attempt to rough sense of what the default notion of images might be:
conceptualize the distinguishing features of digital images, depictions based on an observation of objects, passive and
emphasizes the “algorithmic construction of the image”, fixed representations based on stable subject-object relation-
which is understood to disrupt “the long regime of three- ships. Farocki, too, provides clues to such a default notion of
point perspective”.18 In applications such as Microsoft Pho- images through his numerous negative definitions of opera-
tosynth and augmented reality systems, the interventions tive images. In the intertitles of the Eye/Machine series, for
of algorithms between the viewing subject and the object example, we learn that operative images are devoid of social
viewed introduce “cracks in the façade of the subject-object intent, that they are not meant for edification, and nor for
contemplation. We learn further that operative images are
not really intended for human eyes, and that they exceed the
15 Ibid., pp. 49–50. human scale. Elsewhere, he adds that operative images are
16 Vilém Flusser, Into the Universe of Technical Images, translated by Nancy made “neither to entertain nor to inform”, building toward
Ann Roth, introduction by Mark Poster, Minneapolis/London: University
of Minnesota Press, 2011, p. 7.
17 Ibid., p. 10. 19 Ibid., pp. 25–26.
18 William Uricchio, The Algorithmic Turn. Photosynth, Augmented Reality 20 Ingrid Hoelzl, Rémi Marie, Softimage. Towards a New Theory of the Digital
and the Changing Implications of the Image, in: Visual Studies 26.1 (March Image, Bristol, UK: Intellect, 2015, p. 3.
14 2011), p. 26. 21 Ibid., p. 4.
Operative Images

what seems to be the main negative characteristic given in Cultural Operations

his most-cited definition: operative images “do not represent
an object”.22 A traditional image, then, seems to be an image The next approach to be considered, that of Lev Manovich,
that represent an object in a way that, somehow, conforms differs from Farocki’s in that it no longer revolves around
to the human scale. the idea of automation. The relevance of Manovich’s work
The characteristics of operative images that have is confirmed by Werner Kogge, who, in an article discuss-
been unearthed so far are already starting to show signs ing Manovich’s The Language of New Media (2001), pro-
of inconsistency. Operative images are utility images, and poses operative images as a “paradigm of new media”. 24
as such they belong to a wider family of instruments and While Manovich himself does not use the term operative
tools, which are constructed by humans to serve practical image, Kogge’s suggestion is not unwarranted. According to
human purposes. Yet, in the literature under consideration, Manovich, new media “calls for a new stage in media the-
image-instruments are typically identified with intelligent ory”: If we want to understand the logic of new media, we
machines and automated systems, and as such they are con- need to turn to computer science, borrowing terms such as
ceived as images that no longer cater to human eyes, that “interface”, “database” and “operation”.25
disrupt the human scale and that roam about freely as if in As Kogge points out, Manovich’s approach is refresh-
defiance of petty, human intentions. Humans constructed ing in that it avoids sweeping generalizations about media,
them, but we no longer know whose purposes they serve. which seemed to be the trademark of 1980s media theory,
Even if he sometimes alludes to the imminent replacement exemplified, say, by Baudrillard’s ruminations about total
of humans by machines, Farocki himself, however, is reluc- simulation. 26 Manovich also avoids overemphasizing the
tant to take the human completely out of the loop. The ambi- newness of new media, sketching “archaeologies” that con-
guity is marked already in the title of Farocki’s installation, nect computer screens with classical screens, or comput-
namely by the slash separating Eye and Machine. As noted er-based techniques of media creation with previous tech-
by Foster, the slash raises the question of relation: “Does the niques of representation and simulation.27 Still, as Manovich
slash signify a split between eye and machine […] or a new makes clear, there are aspects of new media that lack histor-
elision of the two, or somehow both – a split that has pro- ical precedents. The newness of new media relates to their
duced an elision?” 23 As we shall see, the ambiguity prevails “programmability”, 28 which results from the merging into
in the literature on operative images. one of two separate historical trajectories, that of modern

24 Werner Kogge, Lev Manovich. Society of the Screen, in: A. Lagaay, D. Lauer
(eds.), Medientheorien. Eine philosophische Einführung, Frankfurt/New
York: Campus Verlag, 2004, pp. 297–315, p. 302.
25 Lev Manovich, The Language of New Media. Cambridge, MA: The MIT
Press, 2001, pp. 11–12, p. 48.
26 Kogge 2004 (as fn. 24), p. 303.
22 Farocki 2004 (as fn. 5), p. 17. 27 Manovich 2001 (as fn. 25), pp. 95–103, pp. 145–160.
23 Foster 2004 (as fn. 9), p. 160. 28 Ibid., p. 47. 15
 Aud Sissel Hoel

media and that of the computer. This meeting changes the and approach media through various cultural filters and
identity of both, giving rise to a “universal media machine”.29 representational schemes. 35 Moreover, like all media, the
The programmability of new media relates to how all computer works by “remediating” 36 older media. 37 Beyond
new media objects are numerical representations. 30 Com- that, Manovich reminds us that the computer level is not
posed of digital code, all new media objects can be described fixed and finished once and for all but continues to evolve
formally in mathematical terms, making them susceptible to as the computer is set to perform new tasks. The influ-
algorithmic manipulations. This means that if the contents ence between the levels, therefore, goes both ways, which
of old media are to be stored, displayed or distributed via means that the “new computer culture” is best conceived as
computers, they must be converted into numerical repre- a “blend of human and computer meanings, of traditional
sentations through a process of digitization. This require- ways in which human culture modeled the world and the
ment relates to what, in Manovich’s view, is the most con- computer’s own means of representing it”. 38
sequential effect of computerization: the transformation of Manovich’s idea about the two layers may seem rem-
media into computer data. 31 As a result, new media objects iniscent of the split between the human and the machinic
in general can be said to consist of two distinct layers: a as discussed in the previous section. Still, it is interesting
“cultural layer” whose structural organization “makes sense to note that, in Manovich’s case, the computer is not really
to its human users” and a “computer layer” whose structural outside the human as such. When he talks about the cultural
organization instead “follows the established conventions of layer, the term culture is taken in a narrow sense, reflecting
the computer’s organization of data”. 32 He gives the example his focus on “cultural software” – software that supports
of a digital image, which on one level is a “representation” 33 cultural actions such as “creating cultural artifacts and
that “belongs on the side of human culture”, and which on interactive services which contains representations, ideas,
another level, is a “computer file” that belongs, rather, to the beliefs and aesthetic values”. 39 Furthermore, when he says
“computer’s own cosmogony”. 34 Since today, media are for that a digital image on the level of representation “belongs
the most part created and accessed via computers, we can on the side of human culture”, he means that it belongs to
expect the computer to influence the traditional cultural the historical trajectory of visual representation with its
logic of media by imposing its own distinct computer logic. characteristic cultural forms, languages and conventions.40
Importantly, however, as Manovich sees it, this influence Thus, when the cultural layer is contrasted with the comput-
is not a one-way street. Just like traditional artists before er layer, the implication is not that the latter exists beyond
them, new media designers and users perceive the world
35 Ibid., pp. 117–118.
29 Ibid., p. 4, pp. 25–26, p. 69. 36 Jay D. Bolter, Richard Grusin, Remediation. Understanding New Media,
30 Ibid., p. 27. Cambridge, MA: The MIT Press, 1999.
31 Ibid., p. 45. 37 Manovich 2001 (as fn. 25), p. 89.
32 Ibid., p. 45. 38 Ibid., p. 46.
33 It is a representation in the sense of featuring recognizable objects. Ibid., p. 39 For a longer list, see Lev Manovich, Software Takes Command, New York:
45. Bloomsbury, 2013, p. 23.
16 34 Ibid., pp. 45–46. 40 Manovich 2001 (as fn. 25), p. 45.
Operative Images

human culture in a wider sense. The point is, rather, that the fitting together of heterogeneous elements into a single,
the computer layer belongs to a separate historical tra- seamless object.46 The operations of selection and compos-
jectory with its own distinct conventions – which is why iting both center on media production and use. It is only
the “language of cultural interfaces”, as we encounter it on when he turns to teleaction that Manovich addresses the
most of today’s computer screens, is often an “awkward mix kind of operations that are topical in the literature on oper-
between the conventions of traditional cultural forms and ative images. Manovich admits that teleaction is “qualita-
the conventions of HCI – between an immersive environ- tively different” from selection and compositing in that it
ment and a set of controls”.41 no longer concerns the “traditional cultural domain of rep-
Manovich’s refusal to draw a sharp line between the resentation”.47 Teleaction results from another meeting of
two layers – the human and the machinic – also informs historical trajectories, this time between media, computers
his approach to the notion of operation. While he acknow­ and telecommunication.48 In Manovich’s view, “teleaction”
ledges that operations behind computer programs can be is a more precise term of what is commonly referred to as
automated, and hence that “human intentionality can be “telepresence”, which he defines “as one example of repre-
removed from the creative process, at least in part”,42 he sentational technologies used to enable action, that is, to allow
refrains from identifying the notion of operation with the the viewer to manipulate reality through representations”.49
machinic. Instead, “operations” are defined more widely True to his habit of questioning the newness of new
as “typical techniques of working with computer media”.43 media, Manovich emphasizes that today’s action-enabling
As Manovich sees it, in the computer age, typical oper- images also have a prehistory. The common focus on “the
ations such as copy, cut, paste, search and filter are also history of visual representation in the West in terms of
used outside the computer, as “general cognitive strategies” illusion”, makes us prone to overlook the separate history
employed in the culture at large.44 Operations, in other of image-instruments. 50 To support his case, Manovich
words, are conceived as “technologically-based cultural draws on the work of Bruno Latour, 51 who, interestingly,
practices” that, despite being embedded in software, are uses perspectival images as well as photographs as exam-
not tied to it.45 Thus, when he sets out to analyze opera- ples of image-instruments. Paraphrasing Latour, Manovich
tions, Manovich focuses on general techniques (or “com- maintains that image-instruments are characterized by
mands”) that are common to many different software pro- their “precise and reciprocal relationship between objects
grams, such as selection, which relates to how in computer
culture authentic creation tends to be replaced by selection
from predefined menus, and compositing, which relates to 46 Ibid., pp. 123–35, pp. 136–60.
47 Ibid., p. 161.
48 Ibid., p. 162.
41 Manovich 2001 (as fn. 25), p. 91. 49 Ibid., p. 165 [original emphasis].
42 Ibid., p. 32. 50 Ibid., p. 167.
43 Ibid., p. 118. 51 More precisely on Bruno Latour, Visualization and Cognition. Thinking
44 Ibid., p. 118. with Eyes and Hands, in: Knowledge and Society. Studies in the Sociology of
45 Ibid., p. 118, p. 121. Culture Past and Present 6 (1986), pp. 1–40. 17
 Aud Sissel Hoel

and their signs”. 52 By systematically capturing features of ment of a systematic and reciprocal relation between objects
reality, a perspectival image, for example, is “more than just and signs, which is what enables humans to use images to
a sign system that reflects reality – it makes possible the manipulate reality. Yet there are tensions in Manovich’s
manipulation of reality through the manipulation of signs”.53 approach to image-instruments. While he continues to
Yet, in the history of image-instruments, the convergence refer to them as “representations” and “signs”, his explo-
with the trajectory of telecommunication makes a differ- rations of image-instruments lead to the realization that
ence, since the electronic transmission of video images and an image-instrument is “more than just a sign system that
the instantaneous construction of representations enable reflects reality”. 57 Manovich, however, stops there and does
real-time remote control – something that provides a unique not take the further step of considering why the traditional
kind of power: “I can drive a toy vehicle, repair a space sta- notions of representation and sign seem unable to properly
tion, do an underwater excavation, operate on a patient, or account for the reciprocity between instrument and real-
kill – all from a distance”.54 This is why, seen from the his- ity – not to speak of their shortcomings when it comes to
tory of action-enabling images, teleaction is a more radical elucidating the interventional and instructional aspects of
technology than, say, virtual reality, because it “allows the instrumental mediation.
subject to control not just the simulation but reality itself”. 55 There is also a second way that Manovich’s consider-
It is worth noting that, the way Manovich defines ations about image-instruments puts pressure on the tra-
image-instruments (as representations that systematical- ditional notion of images. While telepresence is typically
ly capture features of reality), it is not a requirement that associated with live video images, Manovich shows that
the representations in question be produced mechanically. teleaction does not depend on video. Instead, he observes
Leaning on Latour, Manovich seems rather to assume a con- that “different kinds of teleaction require different tempo-
tinuity between perspectival images and photographs (char- ral and spatial resolutions”. 58 In the case of radar-images,
acterizing the latter as perspectival images par excellence). 56 for example, “the image is so minimal that it hardly can be
By emphasizing such a continuity, Manovich differs from called an image at all”. 59 Lacking information about shape,
thinkers like Flusser as well as from thinkers like Friedrich texture and color, radar-images record nothing but the posi-
Kittler and Wolfgang Ernst (to be considered in the next tion of an object – which, however, suffices to destroy it.60 It
section), for whom the introduction of technical images seems, then, that for image-instruments to perform their
involves a momentous, cultural rupture. Manovich, on his operational roles, the visual aspect is not really needed. If
side, instead of identifying the operational and the instru- they do not reflect reality, and if they are no longer visual, in
mental with the machinic, concentrates on the establish- what sense are image-instruments still images? Manovich
does not answer this question. Overall, his investigation of
52 Manovich 2001 (as fn. 25), p. 167.
53 Ibid., p. 168. 57 Ibid., p. 168.
54 Ibid., p. 169. 58 Ibid., p. 170.
55 Ibid., p. 166. 59 Ibid.
18 56 Ibid., p. 167. 60 Ibid.
Operative Images

image-instruments remains an excursion, the bulk of his human-computer interface, seeking instead to explicate the
work being geared towards new media production. Thus, operational processes that play out below the “surface” of
while his investigation of image-instruments certainly software.64
puts pressure on received notions of images, Manovich Ernst’s approach belongs to a line of research that is
himself never explicitly questions their status as represen- commonly referred to as “German media theory” 65 and that
tations. Consequently, in Manovich’s account of teleaction, was opened by Friedrich Kittler, another notable figure of
the action is conceived as human action: Teleaction is the 1980s media theory. Kittler’s work stands out due to its
manipulation of reality by humans (the viewer, the subject, strong focus on the materiality and technicality of media.
the teleoperator) through representations. Images them- Taking inspiration from Michel Foucault’s 1969 treatise
selves are not considered actors.61 Archaeology of Knowledge, Kittler concerns himself with
epistemic ruptures in systems of knowledge, which in
Kittler’s view are related to media shifts. Hence, in Kit-
Technical Operations
tler’s work, the “historical apriori” of Foucault turns into
While in The Language of New Media Manovich experi- a “technical apriori”.66 As Kittler sees it, “media determine
ments with terms borrowed from computer science (includ- our situation” 67 by providing the material conditions under
ing interface and operation), he later comes to criticize this which something may become knowledge. Focusing less on
work for its tendency to regard computer science “as a kind discourses and more on the material substrates of media, he
of absolute truth”.62 Emphasizing even more strongly than conceives media as inscription systems. According to Kit-
before that “computer science is itself part of culture”, he tler, the introduction of the first technological media (“pho-
now aligns himself with the emerging field of software nographs and cinematographs”) marks a major epistemic
studies that approaches software as something more than a rupture in that they, in contrast to previous media (“texts
matter of engineering: “computers and software are not just and scores”), were able to store time.68 The introduction of
‘technology’ but rather the new medium in which we can technological media marks a rupture, more precisely, in that
think and imagine differently”.63 A very different approach
to the notion of operation is found in the work of Wolfgang 64 Wolfgang Ernst, Digital Memory and the Archive, edited and with an intro-
duction by Jussi Parikka, Minneapolis/London: University of Minnesota
Ernst, who seems to go in the opposite direction: Empha- Press, 2013, p. 71.
sizing the technical and engineering aspects of images and 65 Anthony Enns, Foreword. Media History versus Media Archaeology, in:
Wolfgang Ernst, Chronopoetics. The Temporal Being and Operativity of Tech-
media, Ernst dissociates the notion of operation from the
nological Media, translated with a foreword by Anthony Enns, London/New
York: Rowman & Littlefield, 2016, p. xiv.
66 Bernhard Siegert, Cultural Techniques. Or the End of the Intellectual Post-
61 Which they might have been, say, if Manovich had engaged more closely war Era in German Media Theory, in: Theory, Culture & Society 30.6 (2013),
with Latour’s work and adopted the broader notion of agency advanced by p. 50.
actor-network theory. Bruno Latour, Reassembling the Social. An Introduc- 67 Friedrich A. Kittler, Gramophone, Film, Typewriter, translated with an
tion to Actor-Network-Theory, Oxford: Oxford University Press, 2005. introduction by Geoffrey Winthrop-Young and Michael Wutz, Stanford,
62 Manovich 2013 (as fn. 39), p. 10. CA: Stanford University Press, 1999, p. xxxix.
63 Ibid., p. 13 [original emphasis]. 68 Ibid., p. 3. 19
 Aud Sissel Hoel

they involve a shift from inscription systems whose time is Since the heyday of Kittler’s anti-humanist theory,
“(in Lacan’s term) symbolic” to systems whose time runs “on many of his followers have gradually moved away from the
a physical or (again in Lacan’s terms) real level”.69 Further- exclusive focus on the material properties of media tech-
more, the shift from the symbolic to the real implies that nologies, centering instead on the notion of “cultural tech-
humans are no longer the ones doing the inscribing. The niques”.75 Other followers seem intent, rather, to “out-Kittler
introduction of technological media, in other words, occa- Kittler” 76 by affirming even more strongly the anti-human-
sions a displacement of “so-called man” (as Kittler tends to ist tendencies in Kittler’s work. This is the case with Ernst,
put it): machines, he maintains, and especially the intelli- whose resolute focus on machine agency is what makes his
gent machines introduced by digital technology, “are not approach particularly relevant.
there for us humans”.70 This idea, that the machines are not Ernst’s approach to media has been characterized as
there for us, ties into Kittler’s anti-humanist take on history, an “operative media archaeology”.77 In his own efforts to
whose ultimate subject is not humans but technology. With explicate his approach, Ernst positions himself against
the advent of intelligent machines, the self-processing of media archaeology as cultural history on the one hand, and
nature71 no longer needs human intermediaries: “Instead against media phenomenology on the other. As Ernst sees it,
of wiring people and technologies, absolute knowledge historical discourse and human perception are both prone
will run as an endless loop.72 Again according to Kittler, the to interpretation and ridded with subjectivity, which is why
introduction of digital technology also has the effect of eras- he seeks instead a “technoascetic” approach that “takes the
ing the differences among individual media, due to the way point of view of the machine itself”.78
that it reduces sound and image, voice and text to “surface This implies that, in the work of Ernst, archaeology
effects, known to consumers as interface”.73 This, clearly, does not mean genealogy. Drawing on Foucault’s notions
undermines the role of the human senses just as much as of archive and archaeology,79 media archaeology is defined,
it undermines meaning: “Sense and the senses turn into rather, as “a kind of epistemological reverse engineering,
eyewash”.74 and an awareness of moments when media themselves, not
exclusively humans anymore, become active ‘archaeologists’

69 Ibid., p. 4.
70 Kittler cited in Enns 2016 (as fn. 65), pp. xiv-xv.
71 Kittler’s idiosyncratic take on history (including the role of technology) is 75 Enns 2016 (as fn. 65), p. xvi; For an overview of approaches centering on the
succinctly summarized by Geoffrey Winthrop-Young as follows: “the ulti- notion of cultural techniques, see Bernhard Siegert, Cultural Techniques. Or
mate subject of history is technology, understood in a very broad sense as the End of the Intellectual Postwar Era in German Media Theory, in: Theo-
the processing of nature that for an extended period of time was dependent ry, Culture & Society 30.6 (2013), pp. 48–65; and Geoffrey Winthrop-Young,
on human intermediaries, but that now, with the arrival of digital tech- Cultural Techniques. Preliminary Remarks, in: Theory, Culture & Society
nology, is closer to a self-processing of nature that leaves humans behind”. 30.6 (2013), pp. 3–19.
Geoffrey Winthrop-Young, Kittler and the Media, Cambridge, UK: Polity, 76 Winthrop-Young 2013 (as fn. 75), p. 15.
2011, p. 80. 77 Jussi Parikka, Operative Media Archaeology. Wolfgang Ernst’s Materialist
72 Kittler 1999 (as fn. 67), pp. 1–2. Media Diagrammatics, in: Theory, Culture & Society 28.5 (2011), pp. 52–74.
73 Ibid., p. 1. 78 Ernst 2013 (as fn. 64), p. 24, p. 72.
20 74 Ibid. 79 For definitions, see ibid., p. 211, note 4.
Operative Images

of knowledge”.80 In contrast with Manovich, who, according This implies that the operational lifespan of technical
to Ernst, remains on the surface by investigating “monitors media objects is not identical to their cultural lifespan. He
and interfaces” and what they “offer to the human user”, gives the example of an old radio found in a museum, whose
Ernst is concerned with “technoepistemological configura- outer world has vanished. If such a radio, a historical muse-
tions underlying the discursive surface”.81 Thus conceived, um object, is reactivated so as to broadcast today’s radio pro-
the archaeology of media “is not simply an alternative form grams, it undergoes a change in status from “historical to
of reconstructing beginnings of media on the macrohistor- processual hardware”.87 Operationally speaking, therefore,
ical scale”, it describes, rather, “technological ‘beginnings’ the radio is still present, since “[t]here is no ‘historical’ dif-
(archai) of operativity on the microtechnological level”.82 ference in the functioning of the apparatus now compared to
These technological beginnings relate to the very essence then”.88 Thus, when the radio is reactivated, it truly becomes
of technical media, which Ernst conceives in operation- a medium again, which means that “there is a media-archae-
al terms: “It belongs to the specificity of technical media ological short circuit between otherwise historically clearly
that they reveal their essence only in their operation”.83 separated times”.89 This then is why, for Ernst, traditional
The essence of technical media relates to “microtemporal historical approaches will not do: By subjecting media pro-
processes” that are critical for the operations of technical cesses to a literary narrative, they misread and misrepresent
media, that is, for their performance as “processual hard- the Eigenzeit of technical media.
ware”.84 This means that, with a view to their operational According to Ernst, time-critical media provide a dif-
essence, technical media are not arbitrary or subject to dis- ferent (and better) kind of evidence of the past than the evi-
cursive cultural relativization; they have an “epistemolog- dence provided by historical-discursive accounts. As Ernst
ical existence” of their own, due to the way they produce sees it, machines have the power to “temporarily liberate”
their own machine-specific time – what Ernst refers to as us from the limitations of literary narrative and human per-
their “Eigenzeit”.85 Thus, the primary focus of Ernst’s kind ception.90 The unique evidential power of technical media is
of media archaeology is “time-criticality”, the time-giving directly connected with their time-giving agencies, which,
and time-differentiating aspects of technical media – the according to Ernst, induce “disruptions in human tempo-
way technical media “do not simply exist in time but result ral perception” due to their “asynchronous being in what
in timing agencies”.86 is known as ‘historical’ time”.91 Technical media (including
computers) differ from the “traditional symbolic tools of cul-
tural engineering (like writing the alphabet)” in that they
80 Ibid., p. 55.
81 Ibid. register and process “not just semiotic signs but physically
82 Ibid., p. 57.
83 Ibid.
84 Ibid. p. 50, p. 177. 87 Ernst 2013 (as fn. 64), p. 177 [original emphasis].
85 Ibid. p. 57. 88 Ibid., p. 57.
86 Wolfgang Ernst, Chronopoetics. The Temporal Being and Operativity of Tech- 89 Ibid.
nological Media, translated with a foreword by Anthony Enns, London/New 90 Ibid., p. 56.
York: Rowman & Littlefield, 2016, p. vii [original emphasis]. 91 Ernst 2016 (as fn. 86), p. vii. 21
 Aud Sissel Hoel

real signals”.92 Like Kittler before him, he articulates this in literature and musical notation”.100 Hence, as Ernst sees
opposition in terms of the symbolic versus the real: Techni- it, media archaeology is “media studies as exact science”: an
cal media “emancipate” the object from “an exclusive sub- approach that investigates “media-induced phenomena on
jection to textual analysis”, and in so doing, they remind us the level of their actual appearance”, that is, as “physically
about “the insistence and resistance of material worlds”.93 real (in the sense of indexical) traces of past articulation”.101
Ernst further develops the idea of the unique evidential Ernst’s kind of media-archaeology, then, as pointed out
power of technical media by invoking the Peircean notion by Jussi Parikka, is conceived as a “a way of stepping outside
of “index”: Media archaeology is “on the side of the index- a human perspective to the media-epistemologically objec-
ical”,94 which is seen as opposed to the side of the iconic tive mode of registering the world outside human-centered
and the symbolic.95 Hence, when it comes to photography, sensory perception”.102 Technical media (including comput-
he agrees with Roland Barthes, who “emphasizes photog- ers) are conceived by Ernst as “measuring media” – media
raphy as a decisive mutation in informational economies”.96 that, in contrast to mass media, are “able to decipher phys-
According to Ernst, photography is an example of a “true ically real signals technoanalogically”.103 In Ernst’s view,
media-archaeological tool” due to its “automatic registra- measuring media are closer to reality because they “behave
tion and self-inscription of light”.97 A similar rupture is ‘analogously’ to physics itself”.104 More precisely, they are
found in gramophonic recording, “which can record as well assumed to be closer to reality because they operate on the
the accompanying noise (i. e., the index) of the physically level of numbers and not on the “phenomenological multi-
real within and outside the recorded voice”.98 Technical media level” of text, image and sound.105 Media archaeology
media such as these provide a unique kind of evidence due to as conceived by Ernst is “close to mathematics”, which in
the way that they “immediately couples human perception turn is seen as close to nature.106 Hence, when human sens-
with the signal flow […], with or without their translation es are coupled with technological settings, “man is taken
into the iconological regime of cognition”.99 The immediate out of the man-made cultural world”.107 In this way, Ernst
coupling occasioned by technical media is then contrasted aspires toward a “cool” media-archaeological gaze, which
to the “indirect, arbitrary evidence symbolically expressed can be performed by algorithmic machines better than by

92 Ernst 2013 (as fn. 64), p. 58. 100 Ibid., p. 173.

93 Ibid., p. 43. 101 Ibid.
94 Ibid., p. 45. 102 Jussi Parikka, Archival Media Theory. An Introduction to Wolfgang Ernst’s
95 In Ernst’s treatment, the iconic and symbolic tend to be lumped together, Media Archaeology, in: W. Ernst, Digital Memory and the Archive, edited
since they are both associated with culturally variant human perception and with an introduction by Jussi Parikka, Minneapolis/London: Univer-
and history. sity of Minnesota Press, 2013, p. 9.
96 Ernst 2013 (as fn. 64), p. 38; see also Barthes Roland, Rhetoric of the Image, 103 Ernst 2013 (as fn. 64), p. 178.
in: Roland Barthes, Image Music Text, essays selected and translated by 104 Ibid., p. 62.
Stephen Heath, London: Fontana Press, 1977, p. 45. 105 Ibid., p. 71.
97 Ernst 2013 (as fn. 64), p. 47. 106 See ibid., pp. 71–73 for more details about how this (problematic) argument
98 Ibid., p. 64. goes.
22 99 Ibid., p. 67. 107 Ibid., p. 177.
Operative Images

human perception, since it is no longer dominated by “semi- revolves around the efficacy of images, which, as we shall
otically iconic, musically semantic, literally hermeneutic see, need not necessarily be identified with new media or
ways of seeing, hearing, and reading”.108 Thus, in contrast the machinic.
to Manovich, who emphasizes how the cultural layer and When it comes to visual studies approaches to opera-
the computer layer mutually influence each other, resulting tive images, an interesting case in point is the international
in a “blend of human and computer meanings”, Ernst pur- conference Image Operations,113 which, together with other
sues a firm anti-humanist approach that seeks instead to rid recent academic events,114 have contributed to the estab-
the analytical gaze of everything human.109 What is gained lishment of the field of “image operations studies”.115 In an
by this approach is that the non-human and time-critical edited volume following the conference, image operations
agencies of technical media come into view. However, again are discussed with a special emphasis on their roles in war-
as noted by Parikka, by pursuing a “happy positivism”, Ernst fare, insurgency/counterinsurgency and political activism.
comes close to “mythologizing the machine as completely In the introduction, Jens Eder and Charlotte Klonk consider
outside other temporalities, including the human”.110 More- three cases where imagery has been directly involved in
over, by defining the operational in a strictly technical sense, highly charged political situations: Kevin Carter’s Pulitzer
he seems to bracket out the very mediating aspects of media: Prize winning photograph showing a starving and collapsed
their roles as interfaces to the world and other people, their Sudanese child with a vulture in the background; a classified
status as meaningful forms of expression (images, texts, US military video released by WikiLeaks showing gunsight
sounds). footage from an attacking helicopter that opens fire against
a group of men including two Reuters news staff; and a You-
Tube video showing the beheading of the American journal-
Efficacious Images
ist James Foley by a member of the militant jihadist group
While Harun Farocki’s artistic explorations of operative ISIS. In what sense are these cases to be considered as image
images opens a complex array of questions relating to operations? Eder and Klonk provide some indications: they
pressing social, political and ethical issues, the approaches are image operations, first, in that they all provoked “a whole
of Lev Manovich and Wolfgang Ernst both stay “close to the series of largely uncontrollable events” that went “beyond
machine” 111 – focusing on software and hardware, respec-
tively. In this section, I consider operative images from 113 The conference took place at the Institute for Cultural Inquiry in Berlin on
April 10–12, 2014.
the perspective of visual studies,112 where the discussion
114 These events include the conference Media Acts (Trondheim 2011), the con-
ference What Images Do (Copenhagen 2014), a series of three conferences
108 Ibid., p. 27. Dynamis of the Image: An Archaeology of Potentialities (Düsseldorf 2014,
109 Manovich 2001 (as fn. 25), 46. Basel 2014 and Paris 2015), the workshop Screen operations: Conditions of
110 Parikka 2013 (as fn. 102), p. 7, p. 10. Screen-based Interaction (Berlin 2016) and the PhD course Operative Images
111 Interestingly, both thinkers use this exact phrase, see Manovich 2001 (as (Berlin 2017).
fn. 25), p. 117 and Ernst 2013 (as fn. 64), p. 59. 115 Zoya Brumberg, Book Review. Jens Eder and Charlotte Klonk (eds), Image
112 In the German-speaking parts of the world, more frequently referred to as Operations: Visual Media and Political Conflict, in: Journal of Visual Cul-
Bildwissenschaft. ture 16.3 (2017), p. 391. 23
 Aud Sissel Hoel

the original intentions of their producers”.116 In these cases, what is operating in image operations?”.122 They answer
the series of events lead to, among other things, Carter’s by pointing to a “complex network of agencies” in terms of
suicide, the imprisonment of the soldier who was charged actor-network theory.123 Certainly, people and organizations
for disclosing the military video, and a rigorous ban on the use images as tools, but there is an important sense that
footage showing Foley’s beheading. Furthermore, they are “images themselves also act”.124 This idea, that images have
image operations in that the production and circulation a “dynamic of their own”, is key to a highly influential line
of images “led directly or indirectly to the physical death of research in contemporary visual studies.125 It is somewhat
of real people”.117 Even if the images “operated within the surprising, therefore, that, when they go on to clarify the
seemingly disembodied digital sphere of the Internet”, they notion of images, they choose to focus their book on the
all had “serious consequences”, affecting bodies in “vital rather traditional idea of “visual pictures” understood as
ways”.118 Finally, they are image operations in that, in all “anything that visually represents or expresses something
three cases, the images were “crucial factors in the dynam- else without being written language”.126 In fairness, Eder
ics” of the conflicts in question, and as such, “the agens et and Klonk present a range of very different conceptions of
movens in the unfolding of events”.119 Thus, as conceived by images, including “image games” (invoking Wittgenstein’s
Eder and Klonk, image operations are primarily defined in notion of language games) and “image acts” (invoking Sear-
terms of their consequences, which in turn seem to be based le’s notion of speech acts) – the overall impression being that
primarily on the representational function of the imagery. the introduction wavers between established approaches to
Due to their disturbing contents, the images incite a series images in terms of representation and revisionist approach-
of uncontrollable events that have serious, real-world effects. es centering on the idea of image agency. The implication
At the same time, Eder and Klonk repeatedly emphasize of this all-embracing approach is that the operational
that, in all these cases, images do more than “just reflect comes across as a mere supplement to the more established
or represent conflicts”; rather, they “play performative and approaches. This becomes clear, for example, when Eder
constitutive roles within them”.120 They also call attention to and Klonk set out to clarify the specific powers of images,
how, in the digital media environment, the performative and listing the operational – which is now, rather unexpectedly,
constitutive roles of images grow stronger, amplifying “the defined in terms of the interactive use of images in digi-
volume, speed, reach and level of conflictual involvement”.121 tal media – as a fourth potential of images following their
After having proposed these characteristics, Eder and
122 Ibid., p. 6.
Klonk proceed to ask the pertinent question: “So who or
123 Latour 2005 (as fn. 61).
124 Eder, Klonk 2017 (as fn. 116), p. 6.
116 Jens Eder, Charlotte Klonk, Image Operations. Visual Media and Political 125 W. J. T. Mitchell, What Do Pictures Want? The Lives and Loves of Images,
Conflict, Manchester: Manchester University Press, 2017, p. 1, p. 4. Chicago/London: The University of Chicago Press, 2005; Gottfried Boehm,
117 Ibid., p. 3. Ikonische Differenz, in: Rheinsprung 11. Zeitschrift für Bildkritik 1 (2011),
118 Ibid., pp. 3–4. pp. 170–176, http://rheinsprung11.unibas.ch/archiv/ausgabe-01/glossar/
119 Ibid., p. 3 [original emphasis]. ikonische-differenz.html (accessed May 27, 2018); Horst Bredekamp, Der
120 Ibid., p. 4. Bildakt, Berlin: Klaus Wagenbach, 2015.
24 121 Ibid., p. 4. 126 Eder, Klonk 2017 (as fn. 116), p. 9.
Operative Images

mimetic, symbolic and aesthetic (including sensual and Bildlichkeit).130 Krämer’s take on operative images empha-
affective) potentials.127 sizes two interrelated points: the necessity of going beyond
Like Farocki and Ernst, Eder and Klonk accentuate the text-image dichotomy, and the promise of the diagram-
that images have an agency of their own, which implies matic approach.131 The background here is that traditional
that images cannot be fully understood by reconstructing notions of images, including classical ways of distinguishing
the intentions of their producers.128 In Eder and Klonk’s between semiotic modalities, are intimately bound up with
view, this is because, when images start to circulate, they more fundamental divisions – a highly influential exam-
have unforeseen effects that may even go against the orig- ple being Immanuel Kant’s opposition between the two
inal intentions of their producers. Hence, in contrast with stems of human knowledge: sensibility and understanding.
Farocki and Ernst, the operational is identified with the While the classical ways of conceptualizing the boundar-
real-world performative effects of images as they circulate ies between images, texts and numbers typically conform
in society, and not so much with their machinic element. to such long-established, fundamental oppositions, recent
The advantage of this approach is that it brings prominence attempts to rethink images are deep revisions in that they no
to the ethical dilemmas that arise on the level of images, longer assume the dualist worldview at the basis of the old
pointing to the need for a renewed focus on image ethics. distinctions – challenging received notions of images, there-
The notion of image operation, however, is rather vaguely fore, at their very root. Krämer contributes to the ongoing
defined and remains, as noted by Zoya Brumberg, a “nebu- revisionist endeavors, showing how the old philosophers
lous concept”.129 themselves provide resources to overcome unproductive
While Eder and Klonk for the most part approach the dualisms, such as Kant with his notion of schema 132 and
operational as a supplement to more established approach- Charles S. Peirce with his notion of diagram.133
es to images, they also at times seem to push in the direc- So why, then, this renewed interest in Peirce and dia-
tion of a deeper revision of the image category. If followed grammatics? Late in his career, Peirce developed a broad-
through, the idea that images have an agency of their own ened notion of diagrams that is highly relevant to the current
profoundly challenges received notions of images in terms attempts to conceptualize operative images, for two reasons:
of representation. Some thinkers, therefore, such as Sybille First, because it provides a dynamic and operational notion
Krämer, regard the current focus on the operational as an
occasion for a much-needed rethinking of the very idea of 130 Sybille Krämer, Operative Bildlichkeit. Von der ‘Grammatologie‘ zu einer ‘Dia-
grammatologie’? Reflexionen über erkennendes ‘Sehen,’ 2009, http://userp-
images. While in line with contemporary research advocat-
age.fu-berlin.de/~sybkram/media/downloads/Operative_Bildlichkeit.pdf
ing the agential powers of images, Krämer’s approach stands (accessed May 27, 2018).
out in its explicit focus on “operational iconicity” (operative 131 Ibid., pp. 1–3, pp. 10–12.
132 Ibid., pp. 12–15.
133 Ibid., pp. 10–12; In fact, Peirce’s notion of diagram takes its inspiration from
Kant’s notion of schema. Charles S. Peirce, (PAP) [Prolegomena for an Apol-
127 Ibid., pp. 9–10. ogy to Pragmatism], in: Charles S. Peirce, The New Elements of Mathematics,
128 Ibid., p. 1. Vol. IV: Mathematical Philosophy, The Hague: Mouton, 1976, p. 318; Ibid.,
129 Brumberg 2017 (as fn. 115), p. 389. pp. 10–12. 25
 Aud Sissel Hoel

of iconicity134 that pushes beyond static ideas of images in images to the wider family of instruments. For these rea-
terms of similarity (including Peirce’s own previous defi- sons, Peirce’s dynamic and operational notion of iconicity
nitions of iconicity); and second, because it provides a new promises to throw new light on the nature and workings of
notion of evidence that overcomes mechanistic accounts image-instruments, whether we are interested in how and
(including those based on indexicality). The diagrammatic why perspectival images or photographs are “more than just
approach, in other words, provides a fresh take on imag- sign systems that reflect reality” (to paraphrase Manovich),
es that scrambles the icon-index-symbol trichotomy as we or more concerned with the evidential and instructional
know it from textbooks in semiotics. Beyond that, the true powers of digital image applications.138
merit of a Peircean diagram is that it has the unique power
to generate new and surprising information when manipu-
Concluding Remarks
lated in systematic ways.135 Thus conceived, a diagram is not
necessarily visual. It is not a “visual picture” in the terms By maintaining that operative images are not representa-
of Eder and Klonk because the iconic element of diagrams tions but rather instruments that form part of operations,
has more to do with their demonstrative powers.136 The dia- Harun Farocki sets the stage for the ensuing discussions
grammatic structure is not exclusive to images and does not presented above. Identifying the notion of operation with
serve to distinguish them from texts or numbers, since, as automation, he frames the human-machine relationship
Peirce sees it, there is iconicity at the heart of linguistic as antagonistic. Farocki’s Eye/Machine installation and
propositions and mathematical formulas, just as there are its commenters also introduce the idea of the imminent
rules at the heart of images.137 The diagrammatic approach, replacement of humans by machines: Disrupting the human
in other words, redraws the boundaries between images, scale, sensory automations outperform the human eye. Hav-
texts and numbers as we have come to know them, empha- ing no need for human spectators, operative images serve,
sizing interconnections rather than oppositions. In the rather, as interfaces in algorithmically controlled processes.
same vein, it is the diagrammatic structure that connects Issues relating to the human-machine antagonism contin-
ue to resonate in the subsequent two sections. While Lev
Manovich seeks to resolve the antagonism by domesticating
134 A key source for Peirce’s operational notion of iconicity is an unpublished
manuscript that is referred to as “PAP”. See ibid. For a more detailed dis- the machine, Wolfgang Ernst instead chooses the opposite
cussion, see Frederik Stjernfelt, Diagrammatology. An Investigation on the strategy of bracketing everything human to secure the puri-
Borderlines of Phenomenology, Ontology, and Semiotics, Dordrecht: Spring-
ty of machinic operations. Jens Eder and Charlotte Klonk,
er, 2007, pp. 89–116; Aud S. Hoel, Lines of Sight. Peirce on Diagrammatic
Abstraction, in: F. Engel, M. Queisner and T. Viola (eds.), Das bildnerische on their side, identify the notion of operation with the
Denken. Charles S. Peirce, Berlin: Akademie Verlag, 2012, pp. 253–271; Aud performative effects of images as they circulate in ­society,
S. Hoel, Measuring the Heavens. Charles S. Peirce and Astronomical Pho-
tography, in: History of Photography 40.1 (2016), pp. 49–66. articulating the active dimension in terms of distributed
135 Stjernfelt 2007 (as fn. 134), p. 90. networks of agencies.
136 Eder, Klonk 2017 (as fn. 116), p. 9.
137 These rules are generative rules, and not the arbitrary rules of semiological
26 structuralism. 138 Manovich 2001 (as fn. 25), p. 168.
Operative Images

While all four approaches, each in their own way, The guiding idea of this article is that there is some-
strongly confirm the idea of images having an active dimen- thing new in the way that the scholars of operative imag-
sion, none of them provides a developed account of the oper- es approach the topic of mediation, which has to do with
ational as a media-theoretical concept. In this respect, the a deeper recognition of the active dimension of images
accounts considered in sections 1–4 remain too ambiguous, and media. Moreover, as already hinted in the final para-
too cultural, too technical and too wide, respectively. An graphs of the previous section, if the idea that images have
indication that the notion of operation remains under-theo- a dynamic of their own is followed through, we may come
rized as a media-theoretical concept can be seen in that none to question the classical ideas of images at their very root –
of the approaches in question gives a satisfactory account of including their underlying assumptions. This, then, is why,
the new role of operative images as interfaces – as interfaces, to the extent that we are currently standing on the verge
that is, not only in the HCI sense discussed by Manovich, of an emerging, operational paradigm of media theory, this
but in the epistemological and ontological sense as interme- paradigm will have to be a comprehensive one, not restrict-
diaries to the world and other people. In their operational ed to technical images, digital images or new media.
role as intermediaries, images cannot be reduced to Kittleri-
an “surface effects”.139 Moreover, a developed account of the
notion of operation as a media-theoretical concept would
also have to include a more satisfactory take on the relation
between technology and the human senses, not relegating
the latter to the “phenomenological multimedia level” (as
Ernst does).140 The tendency in the literature to distinguish
between images and media that supposedly conform to
the human senses and those that induce a disruption in
the familiar patterns of perception is yet another instan-
tiation of an unproductive opposition between human and
machines. Observing that the boundary between the two
can be drawn in several ways, Manovich raises a pertinent
question: “But what is human nature, and what is technol-
ogy?” 141

139 Kittler 1999 (as fn. 67), p. 1.

140 Ernst 2013 (as fn. 64), p. 71.
141 Manovich 2001 (as fn. 25), p. 171.
Thanks to Kathrin Friedrich and Moritz Queisner for their thoughtful
comments on a previous version of this article. Thanks also to Annamaria
Carusi for her helpful input. 27
Erkki Huhtamo

The Spell of the Catoptric Television

Media Archaeology, Topos Study, and the Traces of Attention

We are always turning in the same circle, Topoi can be compared to empty vessels or molds that are
always r­ olling the same stone. filled with new content as they reappear in changing cul-
Gustave Flaubert in a letter to Louise Colet, tural contexts. They represent continuities, but can also
April 22, 1854 point to ruptures and transitions. The interpretations and
meanings of topoi change in the course of their migrations,
as I explained in condensed form in my article Dismantling
The Power of Topoi
the Fairy Engine. Media Archaeology as Topos Study and will
How can we research media attention as a historical demonstrate in greater detail in a forthcoming book. 3
phenomenon? This essay suggests one way from a media Unlike Curtius, I do not think topoi only exist within
archaeological perspective. Media archaeology is a traveling literary traditions. Visual imagery functions in similar ways,
discipline, a bunch of approaches loosely tied together by as Aby Warburg demonstrated with his notion Pathosfor-
some common threads, but also with significant differences mel (pathos formula) that likely influenced Curtius’ topos.
among its practitioners.1 Because of this, it is necessary to Warburg’s unfinished Mnemosyne Atlas also anticipated
define in what sense the concept media archaeology is being media archaeology.4 It explored visual culture by means of
used here. I call the variant I have been developing for the thematic tableaux, collages that challenged the linear style-
past twenty-five years media archaeology as topos study or based accounts typical for art history. Warburg pointed out
simply topos archaeology. The idea of topos, which refers to migrations and transformations of visual motifs in terms of
recurring elements that travel within and across cultural fields instead of surface vectors of cause and effect. He broke
traditions for hundreds and even thousands of years came down artificial disciplinary barriers by linking academic art
from the German literary scholar Ernst Robert Curtius.2 with non-canonical forms like magazine illustrations and
advertisements, questioning the orthodoxy of academic art
Erkki Huhtamo, Jussi Parikka, An Archaeology of Media Archaeology, in:
1 history. The lives of topoi do not respect institutional and
E. Huhtamo, J. Parikka (eds.), Media Archaeology. Approaches, Applications,
and Implications, Berkeley: University of California Press, 2011, pp. 1–21, p. 3.
Ernst Robert Curtius, Europäische Literatur und lateinisches Mittelalter,
2 See Erkki Huhtamo, Dismantling the Fairy Engine. Media Archaeology as
3
Bern, 1948, translated into English in 1953 as European Literature and the Topos Study, in: Huhtamo, Parikka 2011 (as fn. 1), pp. 27–47.
Latin Middle Ages, new Version trans. by Willard R. Trask, London: Rout- The best edition is Aby Warburg, L’Atlas Mnémosyne, translated by Sacha
4
ledge & Kegan Paul, 1979. Zilberfarb, Paris: L’écarquillé – INHA, 2012.
 Erkki Huhtamo

cultural boundaries, least of all those separating high from one as well, as we can tell from the shadow cast on the sand.
popular culture. Neither do those who research them. The Similar scenes have become a token of everyday life. As
nature of topos archaeology is interdisciplinary. merely sitting in a crowded restaurant can often demon-
Cultural contextualization is more important for me strate, direct face-to-face contacts have been challenged by
than it was for Curtius. He operated within literary tradi- gazes into handheld screens.8 Ulriksen deliberate exagger-
tions, demonstrating how topoi such as the world upside ates, but only a little. Normally people still glance into the
down migrated from one text to another. 5 That is not suffi- lens when they are being photographed (or taking selfies
cient for media archaeology. Topoi must be treated as both themselves). The example seems to confirm Guy Debord’s
transhistorical and as symptomatic manifestations of the insight from The Society of the Spectacle: “[T]he real world
times and places where they appear. As I understand it, the becomes real images, mere images are transformed into real
task of media archaeology is therefore Janus-faced: it traces beings – tangible figments which are the efficient motor of
topoi between contexts and analyzes them within contexts. trancelike behavior.” 9
In media culture topoi serve at least three roles: as connec- Ulriksen’s cover illustration is worth comparing with
tors to older and broader cultural traditions; as discursive a television commercial created to promote Microsoft’s
commentaries on media cultural forms, themes and fanta- already forgotten Windows Phone 7 (2010). It too depicts
sies; and as motifs exploited by the culture industry. These a society mesmerized by mobiles. The commercial, code-
days they are modified and disseminated by potentially any- named Really, recalls, perhaps unintentionally, René Clair’s
one on the Internet.6 Topos transmissions can be detected in 1927 silent film The Crazy Ray (Paris qui dort), where Paris
all areas of media culture. They are not limited to the distant is collectively mesmerized by a mad scientist operating a
past; the Internet is both a topos disseminator and a gener- diabolic machine.10 All Parisians, except those who had been
ator, recycling age-old topoi and giving birth to new ones. up high enough (in the Eiffel Tower or in an airplane), have
Let us look at Mark Ulriksen’s satirical painting Cap- been turned into zombie-like sleepwalkers without a will of
turing the Memories, which was used in the cover of The their own. In Really there are also characters who are still
New Yorker in 2012.7 A family of four has been lined up for a awake, but for a different reason. They are free from the spell
vacation snapshot in a tropical paradise, their backs turned of the mobile phone, craving direct contact: a woman frus-
to the stereotypical lagoon with palm trees (a topos). Each trated with a phoning massage therapist; a man expressing
family member is fingering a mobile phone, utterly lost in its
8 As an indication of how fads change, the 2015 version might show all five
spell. The invisible person who is taking the photo is using
people posing together for the smartphone, attached to the end of a tele-
scoping selfie stick. Taking a selfie may still command a momentary eye
For an exploration of this topos, see Frédérick Tristan, Le monde à l’envers,
5 contact, albeit mediated.
Paris: Atelier Hachette & Massin, 1980. 9 Guy Debord, Society of the Spectacle, New York: Zone Books, 2004, para-
Erkki Huhtamo, Obscured by the Cloud. Media Archaeology, Topos Study,
6 grap. 18.
and the Internet, in: Thorsten Lomker (ed.), ISEA 2014 Dubai. Location. 10 Rene Clair, The Crazy Ray, France 1924. The Crazy Ray is also a meta-cinema,
Proceedings of the 20th International Symposium of Electronic Art, Dubai: an investigation of a medium to manipulate space and time. Clair was the
Zayed University Books, 2015, pp. 22–35. director of the dadaistic film classic Entr’Acte, France 1924, where similar
30 The New Yorker, July 23, 2012.
7 tricks were already used.
The Spell of the Catoptric Television

condolences over a mobile dropped in a toilet; a sexy wife addicted.14 The recurrence of the topos demonstrates that
losing her temper because of her husband’s disinterest; and the current discourse on sleepwalkers staring into mobile
a boy hitting his absent-minded father with a baseball.11 The screens, oblivious of their surroundings, is unprecedented
commercial ends with an ambiguous slogan: “It is time for only to a degree, more because of the scale of the phenome-
a phone to save us from our phones [...]. Designed to get you non than because of the motif itself. The excavations could
in and out and back to life.” 12 It is hard to say if the irony is probably be extended even further back in time to include
deliberate or not: the only way to solve the current ills of collective manias not related to technology.15
social life caused by a gadget is another – essentially simi- Media archaeology should not limit itself to textual
lar – gadget. or visual analysis, which would run the risk of ignoring or
On surface level both the magazine cover and the tele- misrepresenting the tangible – the material and performa-
vision commercial are comments on the same timely prob- tive – aspects of media culture. That is why I analyzed the
lem. Topos archaeology can demonstrate that they are also moving panorama in my book Illusions in Motion. Media
re-activated versions of a pre-existing topos, which has Archaeology of the Moving Panorama and Related Spectacles
been encountered numerous times in the formative devel- on three levels, which I called the painted, the performed,
opment of media culture, commenting on the consequences and the discursive panorama.16 I tried to demonstrate why it
of excessive attention to new gadgets. Before rushing to con- is important to relate media archaeological discourse anal-
clusions about the unique impact of smartphones, it is worth ysis to investigations of material and technological factors.
studying the media manias that are claimed to have broken I also pointed out the necessity of discussing media dispos-
out ever since the Kaleidoscope became a fad in the 1810s.13 itives and their actualizations within media practice as part
Media maniacs lose their loved ones to invisible suitors or of the topos archaeological endeavor. My research has led
end up in all kinds of accidents, over and over again. Some me to the conviction that topoi can manifest themselves in
people manage to stay unaffected, ridiculing or pitying the hardware design, technological features as documented in
patent documents, feedback mechanisms, and user inter-
11 Created by the ad agency Crispin, Porter and Bogusky. See W ­ indows7Phone, faces. However, they cannot be independent of contexts of
Really: New Windows Phone Ad, https://youtube.com/watch?v= ­55kOph conception and use.
D64r8 (accessed January 8, 2018). In the campaign’s pilot commercial, Sea-
son of the Witch, time has slowed down or stopped completely in a crowded
street – a bicyclist has fallen, a car has hit a pole, and a load of fruit have
dropped to the ground. A few people are moving like sleepwalkers, bumping
into each other, but most are entirely frozen; everyone is staring at a por-
table device. See windowsphonepro, Season of the Witch: New Windows 14 For examples, see Erkki Huhtamo, Illusions in Motion. Media Archeology
Phone 7 Ad, https://youtube.com/watch?v=Lg1gbVGk19k (accessed January of the Moving Panorama and Related Spectacles, Cambridge, MA/London:
8, 2018). MIT Press, 2013, pp. 77–79 and passim.
12 Except for this voiceover message and a few short remarks, the commercials 15 Charles Mackay pointed a possible way to proceed in his Victorian classic
are only accompanied by music, Donovan’s Season of the Witch and Edvard Memoirs of Extraordinary Popular Delusions and The Madness of the Crowds,
Grieg’s I Dovregubbens hall (In the Hall of the Mountain King) from the Peer Ware, Hertfordshire: Wordsworth Editions Ltd, 1995 [1841, 1852]. Remark-
Gynt Suite. ably, the book is still in print.
13 I will analyze this issues in detail in a forthcoming book. 16 Huhtamo 2013 (as fn. 14). 31
 Erkki Huhtamo

The Magic Mirror as a Magickal Transmitter practice.19 The web page “How to use a Scrying Mirror /
and Receiver Magick Mirror” describes it as a “powerful psychic tool”
for “seeing into darkness”, stating that it “allows commu-
Applying this media archaeological approach to the screen nication with higher realms and the subconscious”.20 The
in visual practice can link today’s ubiquitous personal scrying mirror is claimed to have numerous uses: “To con-
screens and their usages, dispositives, and discourses to an tact spirit guides, To access knowledge, For healing and self
example known by many names: magic(k) mirror, enchanted improvement, As a magickal [sic] transmitter and receiver,
mirror, sorcerer’s mirror, scrying mirror, etc. It is interesting For divining the past, present and future, As a Portal to the
not only because of its ambiguous relationship with media Astral Plane, For shamanic journeying, For ritual invoca-
history, but also because of its intertwined material and dis- tion and evocation, To improve visualization skills.” The
cursive identities. The magic mirror does not only belong comments left on the website discuss the making, purchase,
to the past, as an Internet search can easily demonstrate. and consecration rituals of the scrying mirror. Someone has
A website promoting New Age energy therapy defines it as recorded concerns: “1/isn’t it risky to go almost [sic] into
a “divination tool for looking into past lives, and past, pres- another world? 2/what if we wont be able to return?” etc.21
ent, and future events”. The writer provides the following For the commentators the scrying mirror seems a perfectly
operating instructions: “Sit before your mirror and begin to practical gadget, and in that sense little different from the
imagine objects on its surface, one after another. You should mobile phone. One of the comments could almost have been
try to see these images clearly in the mirror with your eyes left by a media archaeologist: “Basically a scrying mirror
open, just as if they were there in reality.” Most people, the is the first television and phone thing that ancient people
text continues, “do not see the images appear with the phys- used back then?” 22
ical eyes on the mirror’s surface but see within the mirror Except for their paranormal concerns, these descrip-
and in the mind’s eye. The mirror acts as a focal point, a tions evoke issues that are often associated with the inter-
gateway within.” 17 The website sells round and oval scrying face logic of media machines, that are subsumed by the term
mirrors made of glass or acrylic plastic, but the Internet also screen. Concentration is required to turn the magic mirror
offers how to instructions for making one’s own by covering into a gateway – to make it, as a Wiccan puts it, to “appear as
a clear glass plate with black matte spray paint.18
There are people who consider the magic mirror as an 19 The name “magic mirror” is also used for Asian metal mirrors with illusion-
istic properties. Janet Leigh Foster and Stephen Herbert have authored the
operational device that actually works when used in ­scrying
definitive book on them, Magic Mirrors. A 2000-Year History of Asia’s Most
Sensational Secret, forthcoming. Magic mirrors could also be related to the
Claude Mirror, black glass mirrors used by landscape artists as sketching
aids.
17 Scrying Mirrors – Natures Energies, https://naturesenergieshealth.com/ 20 Somenthingwiccan, How to use a Scrying Mirror / Magick Mirror,
metaphysics/divining/scrying-mirros/ (accessed June 22, 2017). https://ebay.com/gds/How-to-use-a-Scrying-Mirror-Magick-Mirror-
18 See in particular Donald Tyson, How to Make and Use a Magic Mirror. /10000000009366585/g.html (accessed June 22, 2017).
­Psychic Windows Into New Worlds, Custer, Washington: Phoenix Publish- 21 Ibid., comment by Adarsh Barman, posted March 5, 2017.
32 ing, 1995. 22 Ibid., comment by Rylan Sasaki, posted June 6, 2017.
The Spell of the Catoptric Television

a dark tunnel or window”.23 The surface is dynamic; it “will immemorial, reflecting surfaces have been used for observ-
begin to change and fade; a dark mist will appear”. 24 The ing one’s own likeness, but also “for looking into past lives,
Wiccan emphasizes that the magic mirror is a personal medi- and past, present, and future events”.28 In Le miroir (1978),
um, warning: “Do not let others look into its surface, except a media archaeological magnum opus avant la lettre, Jurgis
in ritual context.” 25 Anyone using a mobile phone in a public Baltrušaitis excavated a wealth of material and discursive
environment certainly shares this idea, although for differ- magic mirrors, covering huge stretches of space-time. 29
ent reasons. For another scrying practitioner, “the images Although he did not use that particular word, as a window or
are interactive”.26 Such comments resonate with issues that gateway to an elsewhere the magic mirror can be interpret-
occupy scholars interested in the media user’s sharing hab- ed as a topos, a recurring figure or formula. Together with
its. Is mental absorption into the screen more effective, safer countless other topoi, it contributed to the slow formation of
(especially when the user is in motion) or more dangerous media culture. Baltrušaitis used the anachronistic concept
than distracted multitasking? What is at stake in interac- “téléviseur catoptrique” (“catoptric television”) about the
tive media experiences? Does physically manipulating the visions said to have appeared on magic mirrors, implying
user interface keep the user from getting totally immersed that magic lore and technological screen-based media can
into the audiovisual realm or contrariwise lead to a tighter be related with each other. 30
bond with what is represented on the screen? How are social The best known of Baltrušaitis’s examples is a prophe-
and individual media experiences related with each other? sy the court sorcerer Michel de Nostredame (Nostradamus,
Investigating the bonds with seemingly superfluous 1503–1566) is said to have presented to Catherine de’ Medici
devices like the magic mirror may have lessons in store (1519–1589), the queen consort of France, by means of a magic
for researchers of media reception and interaction. Media mirror. 31 According to the story, the highly superstitious
archaeology does so by searching the past for excavation Catherine was anxious to know the future of her children,
sites where magickal transmitters and receivers may lie bur- and especially to find out who would succeed her on the
ied. Most often they are found as fragments of discourse throne. Nostradamus made each of her sons appear in turn
rather than as archives or material remains. 27 From time in a magic mirror, passing in front of Catherine’s credulous

23 Ibid. ­ ooking Glass.” For further information search the British Museum, Col-
L
24 Ibid. lections Online, for “Dr Dee’s Magical Mirror / Dr Dee’s Magical Speculum”
25 Ibid. http://britishmuseum.org (accessed June 22, 2017).
26 Honor Seed, comment at “Black Mirror Scrying – Has anyone tried it?”, 28 Sabine Melchior-Bonnet, The Mirror. A History, trans. Katharine H. Jewett,
See http://abovetopsecret.com/forum/thread190611/pg1 (accessed June New York, London: Routledge, 2002, pp. 108–110, pp. 195–196, pp. 262–264.
22, 2017). The same writer used the expression: “These interactive visions 29 Jurgis Baltrušaitis, Le miroir. Révélations, science-fiction et fallacies, Paris:
I saw.” Both messages posted January 12, 2006. Elmayan/Le Seuil, 1978. It is regrettable that this major work was never
27 An Aztec magic mirror, made of obsidian and thought to have been used by translated into English.
the Elizabethan alchemist John Dee in his scrying practice, is kept at The 30 Ibid., p. 208. Chapter VIII gives many examples about the tradition of magic
British Museum. It is almost circular but with a protruding part serving mirrors.
as a handle. In Hudinbras (1663), Samuel Butler is thought to have referred 31 Ibid., pp. 184–187, pp. 206–208; Melchior-Bonnet 2002 (as fn. 28), pp. 108–
to this mirror, used by Dee’s associate Edward Kell(e)y, as “The Devil’s 110, p. 195. 33
 Erkki Huhtamo

eyes as many times as he would have years to reign. The story but it may have been oral folklore. 35 Variants were produced
is still often evoked in paranormal circles and treated as fact. over the following centuries. In Honoré de Balzac’s version,
An extreme case is a book of conversations with Nostrada- a woman (or a man, say others), gifted with second-sight and
mus by Dolores Cannon, a “past-life regressionist and hyp- brought to the Castle of Chaumont by Nostradamus, placed
notherapist, who specializes in the recovery and cataloging the Queen “in front of a magical mirror in which a spinning
of ‘Lost Knowledge’”. 32 The conversations were conducted wheel was reflected, each child’s face appearing at the end
via a spirit medium. 33 At the beginning of each session, after of a spoke, the soothsayer made the wheel turn, and the
the medium had fallen into a state of trance, she established Queen counted the number of turns. Each turn was a year
a contact with the famous magus through a magic mirror she of a reign”. 36 Balzac’s imaginary device brings to mind a lot-
saw in Nostradamus’s study. Nostradamus agreed to meet tery wheel spinning on a television screen. In the tradition
the medium “in some other dimension” which was “gray of magia naturalis, writers from the phantasmagoric Éti-
and formless with no more substance than drifting clouds” enne-Gaspard Robertson (Mémoires, 1831) to Fulgence Mar-
(obviously this is what the medium saw in the mirror). Nos- ion (L’Optique, 1867, trans. The Wonders of Optics) suggested
tradamus confirmed that the magic mirror was the very that Nostradamus had performed a trick of technological
same one he had used to concoct the vision for Catherine. sleight-of-hand, which needed a rational explanation. 37 In
Was Catherine surprised to see the figures move in other variants there is no magic mirror at all, just spirits
the mirror, the medium asked? Not really, Nostradamus moving around within a magic circle drawn on the floor. 38
answered, because she was accustomed to court magicians,
but “felt uncomfortable, because she saw that all of her sons
wouldn’t survive.” Had Cannon studied Baltrušaitis’s work, 35 Simon Goulart, Le troisiesme et quatriesme volume du Thresor des Histoires
Admirables et Memorables de nostre temps [...], Cologny: Samuel Crespin, 1614,
she would have found out the apocryphal nature of the sto- pp. 438–439. Goulart gives the name of the source as “Marechalle de Raiz.”
ry. 34 In Trésor des histoires admirables (1614), Simon Goulart His version mentions that the scene in the mirror took place in a “hall”. The
trick would therefore have featured some kind of a stage set. If it ever took
recounted a version said to have been told by Albert de Gondi
place, it would most likely have happened in the summer of 1556 in Paris.
(1522–1602), duke of Retz, who was Catherine’s grandchild, 36 Honoré de Balzac, About Catherine de’ Medici and Other Stories, trans. Clara
Bell, Philadelphia: The Gebbie Publishing Co., Ltd, 1900, p. 254.
37 Robertson, Mémoirs récréatifs scientifiques et anecdotiques, I, Paris: Chez
32 From Cannon’s Amazon.com author’s page, https://amazon.com/ l’Auteur et a la Librairie de Wurtz, 1831, p. 344; Fulgence Marion, L’optique,
Between-Death-Life-Conversations-Spirit/dp/1940265002/ref=la _ Paris: Hachette, 1867; The Wonders of Optics, trans. and ed. Charles W. Quin,
B001K818HK_1_4?s=books&ie=UTF8&qid=1525703563&sr=1–4 (accessed New York: Charles Schribner & Co., 1871, pp. 199–200. For an illustration
June 22, 2017). of one possible arrangement of the dispositive, see fig. 56 (between pages
33 Nostradamus, Dolores Cannon, Conversations with Nostradamus, Vol. 2, 200 and 201). In his letter IV David Brewster gave a scientific explanation
Revised Edition, Huntsville, AR: Ozark Mountain Publishing Inc., 1992. of such devices (without referring to Nostradamus) in Letters on Natural
34 It is ironical that the magic mirror became used in the title of collections of Nos- Magic, London: John Murray, 1833, pp. 59–62.
tradamus’s writings. See Michel de Nostredame, The Complete Fortune-teller: 38 Nicolas Pasquier’s version, see Baltrušaitis 1978 (as fn. 29), p. 187. In an
being the Magic Mirror of Michael Nostradamus; Also, the Infallible Divination engraving published by Baltrušaitis both versions have been combined.
by means of Figures, or Arithmancy of Count Cagliostro, London: Lawrence Nostradamus is shown doing his tricks within a magic circle, while the
& Bullen, 1899; Nostradamus, The Magic Mirror; being the complete for- vision appears in a large horizontal mirror placed above a fireplace (ibid.).
34 tune-teller of Michael Nostradamus, New York: Zend Avesta Pub. Co., 1931. The source of the illustration is unknown. It can be traced back to Émile-
The Spell of the Catoptric Television

The appearances of views into a distance via an optical The device is kept “in a darker room without angles or
device, or catoptric televisions, to adopt Baltrušaitis’s term, resting-places for the eyes, – all its lines, as in a perspective,
form an extensive topos tradition. Beside the many forgot- tending to one point, in which was placed the speculum”.43
ten texts, magic mirrors can be found in classics like Shake- Father Wilhelm felt how “[t]he floating nebulous light that
speare’s Macbeth (Act IV, Scene I) and Sir Walter Scott’s My hung over its surface struck him with apprehension, for
Aunt Margaret’s Mirror (1828), where two noble ladies con- no lamp or other means of illumination was visible”.44 The
sult the advice of a mysterious savant from Padua, who has brightness was so intense that “after looking at the won-
a “very tall and broad mirror”. It happens that “as if it had derful light in the speculum, all other objects around were
self-contained scenery of its own, objects began to appear invisible in the gloom”.45 The mirror could only be consult-
within it”. 39 The scene the ladies saw was “as if represent- ed at night as “with the coming of dawn its brightness is
ed in a picture, save that the figures were movable instead dimmed, and at sunrise its power of reflection is gone.” 46
of being stationary”.40 Francis H. Underwood’s story The Whether this disk of polished steel, which Albrecht had
Exile of von Adelstein’s Soul (1872), set in fifteenth-century “obtained, while travelling in the East, from an Arabian phi-
Vienna, also featured a magic mirror.41 Baron von Adelstein losopher”, was a product of black or natural magic, “always
drives his coach over a young man, who happens to be the within the limits set by the great First Cause”, is left open.47
son of a witch, Frau Eldzeit. The young man dies, and the Father Wilhelm’s experience of looking at “the present occu-
witch casts a spell, forcing the soul of the baron to leave his pation of some of the Baron’s friends” 48 is worth quoting:
body at nights and to settle into the embalmed body of the
dead man. The baron’s chaplain and confessor, Father Wil- The priest fixed his eyes steadfastly upon the mirror,
helm, begins to investigate why the baron becomes lifeless and thought of the Baron. Slowly the mirror seemed to
in the night and only recovers in the morning. He consults become a window, expanding every moment like the
Albrecht Werner or Albertus Nyktalops, a mystical “philos- opening of an iris, and growing more transparent, until
opher” who has “a wonderful speculum or mirror of steel at last there was before his vision the family group in
[...] by means of which distant objects and even spirits of the the palace; the Baron rising from his chair, his mother
dead are brought within view at pleasure”.42 shedding tears as she was about to accompany him
to his chamber, and his sister hanging pensively upon
Jules Grillot de Givry, Le Musée des sorciers, mages et alchimistes, Com- his arm.49
piègne: Impr. de Compiègne; Paris: Libr. de France, 1929, but not further.
39 Sir Walter Scott, My Aunt Margaret’s Mirror, in: Waverley Novels, Vol. 40,
Chronicles of Canongate. First Series. The Surgeon’s Daughter. In Two Vol-
umes, II, Boston: Samuel H. Parker, 1833, pp. 198–235, pp. 224–225. 43 Ibid., pp. 36–37.
40 Ibid. 44 Ibid.
41 Francis Henry Underwood, The Exile of von Adelstein’s Soul, in: Cloud-Pic- 45 Ibid., p. 38.
tures, Boston: Lee and Shepard; New York: Lee, Shepard & Dillingham, 1872, 46 Ibid., pp. 41–42.
pp. 1–78. The Exile of von Adelstein’s Soul was written in 1858, but published 47 Ibid.
for the first time in this volume. 48 Ibid., p. 36.
42 Ibid., p. 35. 49 Ibid., p. 38. 35
 Erkki Huhtamo

It is tempting to interpret Underwood’s imaginary device inspired by earlier sources. Leprince de Beaumont’s ver-
as a media dispositive. The positioning of the mirror as the sion became the inspiration for Marmontel and Grétry’s
luminous focal point for observation in a darkened space successful opéra comique Zémire et Azor (1771). Jean Coc-
evokes cinema rather than television; the latter is often teau’s classic feature film La Belle et la Bête (1946) inspired
watched in lighted or semi-illuminated spaces. But there is a Philip Glass’s opera (1994), while Disney’s animation film
difference: unlike cinema, the system is interactive. Instead (1991) is only one of the many re-tellings of the topos within
of buttons or switches, it uses – anachronistically speaking – popular culture. 52 The magic mirror as a way of observing
a mind-machine interface (MMI): channels are switched by people from a distance was already featured in Villeneuve’s
thinking about the person one wishes to see. Further, as tale. Its role in Cocteau’s La Belle et la Bête is particularly
Father Wilhelm discovered when he wanted to consult the interesting because the film was released at the moment
scene of the accident where the young man had been killed when television was just beginning its triumphal march.53
a month earlier, the magic mirror was capable of presenting Although its adoption in France was slow, television was
reruns of past events. He saw “microscopic figures passing already an established idea. It is likely that some of the spec-
through a distant square” as the incident unfolded. 50 The tators, who saw Cocteau’s film, associated its magic mirrors
scenes were silent, which may well have been influenced with the television screen.
by a contemporary media form: the room camera obscura, The television pioneer Alan A. Campbell Swinton stated
a popular attraction at seaside resorts and elsewhere. 51 The in 1912: “[I]f there could be added to each telephone instru-
luminous real time image of events unfolding outside was ment what would indeed be a magic mirror, in which we
projected on a circular table in the center of the darkened could see even only in monochrome the faces of those with
chamber. These comparisons make huge leaps between whom we were communicating, the material advantages
times and places far apart from each other, but they are not would be great. In addition, there would be much senti-
necessarily arbitrary. Material forms of media culture have
52 The magic mirror tradition lives on, beside fairytales, in Halloween post-
been anticipated by discursive ones, technological solutions
cards. The magic mirror is depicted as a device for seeing one’s future hus-
by imaginary or even magic ones. band or wife. This idea was associated with Nostradamus and Catherine de’
A perfect example of the ways in which a topos tradi- Medici almost entirely erroneously in a masonic journal: “The famous Nos-
tradamus conjured up in a magic mirror the phantasmal form of her future
tion can migrate between cultural and mediatic contexts husband for Marie [sic] de Medicis.” Mysticus, A Corner of the Library: The
is the story La Belle et La Bête (1751), which became a car- Magic Mirror, in: The New Age, XXIX.6 (1921), p. 274.
53 Since the 1920s, Cocteau’s work often referred to contemporary media
rier of topoi. Jeanne-Marie Leprince de Beaumont (1711–
machines, often in relation to myths from classical antiquity. Magic mirrors
1780) adapted it in 1756 from a fairy tale published in 1740 play important roles in his poetic films Le Sang d’un Poête (1930) and Orphée
by Suzanne Barbot de Villeneuve, which in turn had been (1950) as concretized metaphors and entry points into alternate realities.
Lewis Carroll’s Through the Looking Glass (1871) can be related to the same
historical trajectory. It was also evoked in television promotion. See the
double-page ad “... and through this looking glass. the Wonderland of NBC
50 Ibid., p. 48. Big Color TV!” (c. 1956–1958), in: Steve Kosareff, Window to the Future. The
51 Underwood acknowledged he was “aware that the device of employing the Golden Age of Television Marketing and Advertising, San Francisco: Chroni-
36 magical speculum [...] is not a new one.” Ibid., p. vi. cle Books, 2005, pp. 116–117.
The Spell of the Catoptric Television

mental and other value.” 54 Such associations became com- Conclusion: the Topos Reified
monplace. Advertisers began comparing television sets with
crystal balls and magic mirrors early on. They were a form Returning to the issue of attention, it should be clear by now
of modern wizardry, “man’s strangest dream come true in that normally media archaeology can say little about actual
your home”. 55 In 1939 a British publication characterized experiences. On the contrary, it casts doubt on the truth-
television as “the magic mirror of the living room”.56 In 1944, fulness and accessibility of direct observations recorded in
on the eve of the television era, a children’s book looked back discourses. What seems authentic and personal often turns
into the past while anticipating the future: “If you have a out to be mere topoi in disguise. As members of societies,
magic mirror you can see a play at a theater without leav- humans inhabit vast topos spaces brimming with received
ing home, you can be with friends who live in another city. ideas and motifs that are used as molds for expressing con-
The magic mirror makes everything near. What is the magic temporary issues and concerns. The topos transmissions
mirror? It is a television machine.” 57 The Admiral Corpora- happening in these spaces may not be fully acknowledged
tion grasped the connection, branding its line of TV-sets as such by those living within their reach and even taking
Magic Mirror Television. 58 In 1940, the Archery News asked part in their dissemination. However, there are those who
the fundamental question: “May not the magic mirror be are fully aware of the power of the topoi and use them to fos-
nothing but an early conception of the possibilities of Tele- ter ideological and commercial ends. This applies to adver-
vision?” 59 The media archaeologist’s answer is affirmative, tisers, image and identity makers and other professionals
but excludes the words nothing but, for the magic mirror has of the culture industry. Their businesses are centered on
been other things as well. discovering recognizable formulas and revising them just
enough to maintain the customers’ interest. More often than
not, the formulas are found from the past. Topoi are effec-
54 Alan A. Campbell Swinton, Presidential Address. November 7th, 1911, in: tive as tools because their stereotypical components appeal
The Journal of the Röntgen Society VIII.30 (1912), p. 7. Swinton’s ideas about
to mainstream taste while they can also be dressed up and
electronic scanning contributed to the development of television.
55 Wulf Herzogenrath, Thomas W. Gaehtgens, Sven Thomas und Peter Hoe- presented as the Coolest Thing on the Planet. Camouflag-
nisch (eds.), TV Kultur. Fernsehen in der Bildenden Kunst seit 1879, Amster- ing the Old as the New suggests both a moment of sooth-
dam, Dresden: Verlag der Kunst, 1997, p. 146, p. 147, p. 157.
56 Reference to “magic mirror of the living room” is from The Nation’s Business ing recognition and an awe-inspiring encounter with the
27 (1939), p. 97. unprecedented.
57 Michail Il’in, Elena Segal, A Ring and a Riddle, trans. Beatrice Kinkead, Phil-
The magic mirror is a perfect example because its pres-
adelphia: Lippincott, 1944, p. 72 [original emphasis].
58 Questions and Answers about Admiral Magic Mirror Television, booklet, II ence is so widespread in today’s popular culture. It is less
printing, Chicago: Admiral Corporation, March 1948. Author’s collection. due to Nostradamus than to Walt Disney, whose classic ani-
An advertisement in Life explained that the “magic mirror” was an alumi-
num sheet inside the tube to make the picture “twice as bright as ordinary mation feature Snow White and the Seven Dwarfs (1937) has
TV.” Life 37.11, (1954), p. 25. The expression “the magic mirror of television” been seen by millions and has inspired countless ­products,
remained in generic use Television Magazine 13 (1956), p. 62.
59 Archery News 19 (1940), p. 36. Only a snippet view can be seen on Google
Books. 37
 Erkki Huhtamo

both industrial and artisanal.60 The iconic still image of fans of new technology. Applications vary, but the basic sys-
the Evil Queen staring into her magic mirror has turned tem configuration is relatively uniform: the magic mirror of
into an Internet meme, provided with captions and photo- 2017 is a screen overlaid with a two-way mirror (often a thin
shopped into other personalities, including (predictably) sheet of plastic) and connected to a computer. It allows the
Donald Trump.61 Compared with the examples discussed user to see one’s own likeness while consulting information
so far, something else is at stake: instead of displaying far- from the computer, the web or social media. The informa-
away scenes, Disney’s mirror provides the Queen a facetime tion floating around the user’s mirror image is called forth
session with an incarcerated spirit via both sound and image. by touches, gestures or voice commands. The screen is nor-
The talk is all about her. The theme is obsessive narcissism mally positioned vertically in the portrait mode, but taking
and the compulsion to constantly verify one’s own beauty. the mirror comparison literally, Samsung introduced in
Disney’s magic mirror is therefore closer to the mundane 2015 a perfectly round wall-mounted interactive display.62
mirror uses we all are familiar with. The popularity of this Commercial Magic Mirror Photo Booths and Magic Selfie
variant today must be associated with the selfie obsession, Mirrors enhance the posers’ likenesses with digital features
which has turned into a global phenomenon thanks to the (much like Sega’s Purikura arcade machines did years ago).
ubiquity of the smartphone. Millions of people use their Companies have also begun promoting magic mirrors to
phone screens as interactive mirrors, capturing, posting and fashion stores, persuading customers to try design clothes
transmitting their likenesses to others, even naked. The cap- and make-up virtually on their screen doubles.63 The magic
tive spirit questioned by the Evil Queen has been replaced by of the digital mirror is that of commodity fetishism; the awe
peers posting replies online. Anything but a Like can cause it produces pure appearance.
lurid or even tragic reactions. The magic mirror has also turned into a do-it-yourself
The idea, presented in the introduction, that a topos can phenomenon. Anything from old PC monitors and tablet
materialize as a technological device has been confirmed computers to flat panel LED screens have been converted
by recent developments. As I am writing this, magic mirror for the purpose. Encouragement and instructions can be
has become a popular buzzword among both developers and

62 A circular (and curved?) mirror display that recalls Hugo Gernsback’s TV

60 There are online discussions of whether Disney’s Evil Queen used the fantasies of the 1920s was introduced by Samsung in 2015 at the Interna-
expression “Mirror, Mirror on the Wall” or “Magic Mirror on the Wall” tionale Funkausstellung Berlin (IFA 2015) as part of its Smart Signage
when invoking the spirit. The latter is correct, although most people seem Portfolio presentation. It used the company’s OLED transparent display
to think it is the former. Such mistaken collective memories are sometimes technology, unveiled for the first time. The round mirror display was a con-
called The Mandela Effect. Fiona Broome, posted on December 15, 2015, at cept model for the store of the future. The design also resonated with the
http://mandelaeffect.com/mirror-mirror-research/ (accessed July 13, 2017). round displays of smartwatches, including Samsung’s own Gear S2, which
The question about the Mandela Effect is interesting, but controversial. On was released in 2015.
her website Fiona Broome defines herself as “author, researcher, and para- 63 Rina Raphael, Interactive ‘Magic Mirrors’ Are Changing How We See Our-
normal consultant”. Her forte is books about ghosts. http://fionabroome. selves – And Shop, in: Fast Company, April 6, 2017, article 3066781, http://
com/about-the-author (accessed July 13, 2017). fastcompany.com (accessed July 13, 2017). The writer evokes Disney’s Evil
61 Trump’s figure has been – equally predictably – also inserted into memes Queen as the point of comparison. As an indication of the current attention
38 inspired by The Beauty and the Beast. deficit disorders, a note states the article is a 9 minute read.
The Spell of the Catoptric Television

found from online resources like the Magicmirrorcentral.

com, which defines itself as a “website dedicated to DIY
magic mirror making (some people refer to them as Smart
Mirrors)”.64 The most common technical solution to power
the magic mirror is the inexpensive Raspberry Pi mini com-
puter with custom programming in Linux. Personal enthu-
siasm notwithstanding, there is something puerile in the
concept. In earlier manifestations of the magic mirror topos
one’s own reflection was normally effaced and replaced by
the data emanating from a distance. In an era of uncertainty,
superficiality and rampant narcissism, it now seems neces-
sary to constantly monitor one’s external appearance (and
body functions) and to be reminded that one still exists.
The obsessive selfie snapping with any place one visits as
background and anyone one meets and greets as pals serves
the same goal. The magic mirror tradition further looms
behind augmented reality applications. Whatever everyday
life used to mean is no longer enough; it has to be enhanced
and boosted, modified and converted.

64 On the home page, http://Magicmirrorcentral.com (accessed July 13, 2017). 39

Tristan Thielmann

Early Digital Images

A Praxeology of the Display

The digital image is non-existent. If anything is respon- attempts to re-introduce the separation between image and
sible for missing the point here, then it is inappropriate image carrier, in order to make a statement on the specific
essentialism. What there are, are innumerable analog properties of digitally generated pictorial worlds. By shed-
images that illustrate the data that are present in a dig- ding light on the historical and, simultaneously, the prac-
ital form: on monitors, televisions or paper, on movie tice-theoretical contribution of the display in the discourse
screens, displays and so on.1 on digital imagery, we can show that the scientific reflec-
tion on images in general has been unfoundedly loaded with
meaning in the truest sense of the word.
Shedding Light The “material turn” 4 that is currently being diagnosed
Media studies have postulated consistently that digital in the analysis of digital media practices as well as the dis-
images do not exist. 2 This has not prevented the visual course on “soft images” 5 reveals that we are increasingly
studies and arts from continuing with the attempt at pro- dealing with dynamic, transparent and malleable displays
claiming a phenomenology of images that refuses to negate adapting themselves to the individual user, context and sit-
their digital transformation or even origin. 3 The fact that uation. This raises the question as to what contribution a
we perceive images on digital displays has led to an analyt- practice theory can make toward conclusive media esthetics,
ical imprecision in the development of theory in the visual media history and a media theory of the display.
studies, in that the technical conditions underlying displays Such a practice theory must be assessed on the basis of
are referred to the materiality of images. This contribution “the practical procedures being given precedence over all
other explanatory parameters”.6 Under reference to Harold
Claus Pias, Das digitale Bild gibt es nicht. Über das (Nicht-)Wissen der
1
Bilder und die informatische Illusion, in: Zeitenblicke 2.1 (2003), http:// Bill Brown, Materiality, in: W. J. T. Mitchell, Mark B. N. Hansen (eds.), Criti-
4
zeitenblicke.de/2003/01/pias/ (accessed November 1, 2017). cal Terms for Media Studies, Chicago: The University of Chicago Press, 2010,
See ibid.; Wolfgang Hagen, Es gibt kein digitales Bild. Eine medienepis-
2 pp. 49–63; Sarah Pink et al. (eds.), Digital Materialities. Design and Anthro-
temologische Anmerkung, in: Lorenz Engell, Bernhard Siegert, Joseph pology, London/New York: Bloomsbury, 2016.
Vogl (eds.), Archiv für Mediengeschichte. Licht und Leitung, Weimar: Bau- Ingrid Hoelzel, Rémi Marie, Softimage. Towards a New Theory of the Digital
5
haus-Universität Weimar, 2002, pp. 103–110. Image, Bristol/Chicago: The University of Chicago Press, 2015.
See Gundolf S. Freyermuth, Lisa Gotto (eds.), Bildwerte. Visualität in der
3 Erhard Schüttpelz, Skill, Deixis, Medien, in: Christiane Voss, Lorenz Engell
6
digitalen Medienkultur, Bielefeld: transcript Verlag, 2013. (eds.), Mediale Anthropologie, Paderborn: Wilhelm Fink, 2015, pp. 153–182.
 Tristan Thielmann

Garfinkel, we could also formulate this as follows: “Praxeol- three-dimensional object that reveals an image on its open
ogy seeks to formulate statements of method, and to extend side to the thin and fluid two-dimensional digital surface
their generality, seeking as wide a domain of applicability that appears to be one with what it depicts.
as possible.” 7 While the term monitor (Latin: an overseer, instructor
The aim of a media practice theory of the display must or guide) still expresses specifications of an observing sub-
be to unveil the methods of the medium.8 Given the diversi- ject, at first sight, the display shifts what is being present-
ty and multiplicity of displays, this essay therefore pursues ed and exhibited into the center. The term “screen”, which
the question of what sociotechnical properties are exhibited primarily emphasizes protection from electronic radiation
phenomenologically by digital displays. What constitutes (electronic images), already points to this fundamental dif-
their specific media characteristics that distinguish them ference between it and the display.11 The type of visibility
from all other forms of electronic monitors and screens? We appears to be the key to understanding the display culture.12
initially need to take a step back to help us to better estimate The materiality of the display will therefore be at the focus
the scope of the current development. How long have we of a historical genealogy in the following, which reveals a
actually been in a position of referring to the display as an series of media practice-theoretical determinants.
independent medium? When contemplated from a technical and historical
perspective, the term display does not originate from the
medium of the television or computer but rather from the
The Top View of the Display
military medium of the radar: the radar display refers to the
In 2003 for the first time, more LC displays were sold in radar screen. The first field experiment using radar was con-
Germany than conventional monitors with cathode ray ducted in Great Britain as early as February 26, 1935, during
tubes (CRT).9 Since then, a fundamental change in screens which a test airplane produced an additional illuminated
can be diagnosed: from the stable, fixed CRT monitor to the dot on the screen of a cathode ray oscilloscope through the
flexible, mobile LC display;10 from the heavy, furniture-like, radio waves emitted by the BBC transmitter in Daventry
being reflected off the body of the plane.13
7 Harold Garfinkel, Some Sociological Concepts and Methods for Psychia-
trists, in: Psychiatric Research Reports 6 (1956), pp. 181–198, p. 191.
8 For visual media methodologies see: Gillian Rose, Visual Methodologies, becoming more and more prevalent, also in the form of its organic variant
London: Sage Publications, 2001. (OLED) that possesses a lower luminance density and therefore no longer
9 See Andreas Wilkens, Erstmals mehr LC-Displays verkauft als Röhren­ relies on the use of monocrystalline materials.
ge­r äte, in: heise.de, 16.03.2004, http://heise.de/newsticker/meldung/ 11 The French écran also originally means visual protection. The term screen
45600 (accessed November 1, 2017). that is generally used in English refers to the projection surface in movies
10 Liquid crystal display (LCD) is an umbrella term for liquid crystal screens. and to television, video and computers. Gunther Kress, “Screen”: Meta­
The thin film transistor (TFT) refers to specific LCD technology that is phors of Display, Partition, Concealment and Defence, in: Visual Commu-
used to created large-scale electronic circuits. TFT technology is currently nication 5.2 (2006), pp. 199–204.
the dominant flat screen technology, which is why LCD and TFT are used 12 See Sean Cubitt, The Practice of Light. A Genealogy of Visual Technologies
almost synonymously. More modern LCDs are also called LEDs as they from Prints to Pixels, Cambridge, MA, 2014.
use light-emitting diodes for background lighting. This allows a more com- 13 An oscilloscope detects changes in voltage in an electric circuit using a
42 pact construction and thus thinner displays. LED technology is currently light trace. This becomes visible on an analog computer based on the same
Early Digital Images

During this experiment, the radar display was already airplanes were represented as dots in a two-dimensional
calibrated so that distances were on a linear scale and it representation of the actual airspace. Here, for the first time
was thus easy to take a reading.14 A ruler was depicted on perhaps, a bridge is constructed between the representation
the oscilloscope, from which the distance of a flying object of technoscientific data, such as offered by the oscilloscope,
could be read off, based on where the pulse of the echo signal and the mimetic representation made possible by the tele-
amplitude was produced. However, in spite of their scaling, vision.” 19
the distances that were measured were not yet georefer- In Germany, the viewing tube of the all-round sensing
enced. Cartographic projections were not yet possible with system was called a “Sternschreiber”.20 The first decades of
the oscilloscope. The first radar displays simulated the prac- displays are not characterized by a (4:3 or 16:9) rectangular
tice of reading a ruler (fig. 1).15 section of reality, but by circular screens. Using radar sys-
This changed with the plan position indicator (PPI), tems (“Rundsuchanlagen”, lit. “circular search apparatus”) 1  Radar oscilloscope before 1940.
proposed in 1935 and used for the first time in 1940, that required panoramic viewing instruments.
allowed a top-down view of events.16 This is the classical All-round sensing with a panoramic display only
form of the radarscope with a panoramic display (fig. 2).17 became available in Germany in 1944. A distance indicator,
“Such a system with a rotating, or sweeping, line is what EAG 62 “Emil”, was used to locate enemy bombers and to
most people continue to associate with a radar display.” 18 guide the German fighter jets.21 However, joint depiction of
The PPI display indicates the distance and direction for all the fighter jets and enemy bombers was required to guide
altitudes through a sweep (a scan line that corresponds to the fighter jets, leading to the development of the “Himmel-
the position of the radar antenna in a given moment) that bett” method in 1941, led by Josef Kammhuber, to project
rotates around the center of the cathode ray tube and depicts the measurements for distance, azimuth and flight altitude
the echo blips as bright dots: “With this form of display, the as spots of light onto the glass disc of the “Seeburg plotting
table”.22
2  Schematic illustration of a
principle as the Braun tube, as is also still the case for the image signal in
In this case, the display is a window onto the turnta- classical displays on a plan position
current televisions. Robert Watson-Watt was already using an oscilloscope
as a display for locating storms in 1923. bles behind it that combine the different representations indicator.
14 See Robert Watson-Watt, Three Steps of Victory, London, 1957, p. 471. produced by the pixel projector (“Bildpunktwerfer”) into an
15 The History of Flight Radar, http://planefinder.net/about/the-history-of-
flight-radar/ (accessed November 1, 2017). indexed picture (fig. 3). The foundations for the display as an
16 Robert Watson-Watt, The Evolution of Radiolocation, in: Journal of the independent playback medium were thus laid, and the term
Institution of Electrical Engineers 93 (1946), pp. 374–382, p. 379.
display was transformed from the indicated to the indicator,
17 Manfred von Ardenne had already developed a precursor to the panoramic
display tube in the mid-1930s. See Manfred von Ardenne, Ein neuer Polar-
koordinaten-Elektronenstrahl-Oszillograph mit linearem Zeitmaßstab, in: 19 Charlie Gere, Genealogy of the Computer Screen, in: Visual Communication
Zeitschrift für technische Physik 17 (1936), pp. 660–666. However, Göring 5.2 (2006), pp. 141–152, p. 146.
rejected further development after looking at the drafts and photos of “fig- 20 Fritz Trenkle, Die deutschen Funkführungsverfahren bis 1945, Heidelberg:
ures on a fluorescent screen”. Manfred von Ardenne, Ein glückliches Leben Hüthig, 1987, p. 200.
für Technik und Forschung, Munich, 1976, p. 131. 21 David Pritchard, Durch Raum und Zeit. Radarentwicklung und -einsatz
18 Wikipedia, Radar in World War II, http://en.wikipedia.org/wiki/Radar_in_ 1904–1945, Stuttgart, 1992, p. 63.
World_War_II (accessed November 1, 2017). 22 Trenkle 1987 (as fn. 20), pp. 193–194. 43
 Tristan Thielmann

There were also enormous map tables in the English

command centers of the pre-display era, on which, however,
bits of cardboard cut into the shape of airplanes were still
being used to indicate their current position (fig. 4). While
the male officers studied the map, female soldiers continu-
ally changed the position of the miniature airplanes based
on the incoming radar information, by moving them with
long sticks.25 The human actors were also moved about in
the same way as the model airplanes. The display of the See-
burg plotting table has this in common with map tables and
paper maps (fig. 4).
The methods that are used to process and depict the
information collected from the radar have not changed
the puppet strings guiding the supposed beneficiaries. The
movement of distant objects determines the motion of the
person in front of the display. The display renders the loca-
3  Sketch of the Seeburg plotting table.
tion of a distant object into externally guided movement and
location of the person, as the position of the person in front
from the (re)presented to the (re)presenter, from the image of the display is determined by the position of the objects
to the medium. detected by radar. The display creates a uniform interac-
Instead of the now common sight of vertical screens, tion space in which distant objects that are out of view are
the dispositif of one of the first displays was determined by aligned with a subject making the observation. Coordina-
people moving around a table in the Himmelbett method. tion thus occurs without (visual) contact.
The compact all-round viewing instrument (“Rundsicht- Radar technology creates semipermeable spaces in
gerät”) in the “Jagdschloß” system also required observa- which object coordinates from an external space penetrate
tion from above.23 Even the frontally attached panoramic into an internal space, and resultant actions in the inter-
display in the 1944 mobile “Rundsuchanlage”, “Jagdwagen” nal space have effects on the external space. A sociospatial
or “Panotwiel”, still makes reference to this: it was called duplicity of vehicular coordinates is produced by the simul-
“Drauf ” (from above).24 It is this special dispositive structure, taneity of movement in geographical space and in the map
in particular, that characterizes the point of departure for room.
the display and its history (fig. 3).

23 Ibid., pp. 106–107. 25 Lev Manovich, The Language of New Media, Cambridge, MA/London: MIT
44 24 Ibid., p. 112. Press, 2001, p. 100.
Early Digital Images

What is depicted, how it is depicted and the dispositive

position of the subject are all still contingent here. The view
from above results in a unification that documents both the
dispositive structure of human and display, the relationship
between significats in the external space and signifiers in
the internal space, as well as all intermediaries, 26 bringing
about the translation from things to signs. The how of the
unification remains hidden, even though it takes place nei-
ther electronically, nor algorithmically, but optically and
electromechanically.
If we understand technology to be a distributed action,27
then this opens up a space that separates the space in front
and below the display from each other; but the causal
relationship between significat and signifier still remains
preserved. The external and internal worlds of the display
behave in a homomorphic fashion in relation to each other.
4  Plotting Room, Uxbridge, GB (1939).
In the case of the Seeburg plotting table, we are still
primarily dealing with interface agents who are standing
The Himmelbett method renders the display dispositif around a display. In contrast, in the case of the first digital
evident. It opens up three spaces to us that can be subjected displays, we start to talk about the particular importance
to a comparative investigation within the meaning of media of coordination agents, the active, no longer passive, move-
praxeology. (1.) Firstly, we are dealing with objects that are ment in space that was central to the functionality of the
out of sight, that are also moving rapidly and over great dis- first computer and its display.28
tances, and thus are scattered over a correspondingly large
external space. (2.) The internal space of the display is also
26 The actor-network theory defines an intermediate link or binding agent
not visible to the observer. In this respect, it represents a in a sociotechnical translation chain as an intermediary, which simply
miniaturized model of the external space, in that different transports meaning, while a mediator, or go-between, simultaneously also
transforms a meaning that is to be transported. See Bruno Latour, Reassem-
actors (fighter jets and bombers) are being represented 1:1
bling the Social. An Introduction to Actor-Network-Theory, Oxford: Oxford
here by their own medium (a ray of light). (3.) Finally, there University Press, 2005; Ann-Sophie Lehmann, Das Medium als Mediator.
is also the space in front of the display. This is where the Eine Materialtheorie für (Öl-)Bilder, in: Zeitschrift für Ästhetik und Allge-
meine Kunstwissenschaft 57.1 (2012), pp. 69–88. A media practice theory that
actors behind and outside the display are transformed into places digital displays in its center no longer needs to differentiate between
objects that are represented together on one area. In this intermediaries and mediators.
27 See Werner Rammert, Technik – Handeln – Wissen. Zu einer pragmatis-
interaction space, alliances are formed between different tischen Technik- und Sozialtheorie, Wiesbaden: Springer, 2007, pp. 79–81.
observable objects and the different observers. 28 See Ibid., p. 83. 45
 Tristan Thielmann

The Discrete Nature of the Display widely used as RAM up until the 1950s, for example, in
the mainframe computers ERA 1103 (with a Williams tube
For the first time in the history of computing and optical memory of 1024 words) and IBM 701 and 70231 (fig. 5–6).
media, the radar screen allows for directly addressing an What is decisive in relation to the Williams tube is that,
individual pixel on an illuminated area without having to with it, a new form of visibility or image processing appears
pass through “precursors” and “successors”.29 This is of rel- in the true sense of the word. However, the data that are to
evance to the optical media, as follows: In contrast to the be processed are not simply intended for the visualization
dispositif of the television, it is not only the rows but also of the invisible or the absent during completion of their pro-
the columns of an image that are decomposed into single cessing, but the dot images themselves are the data in the
elements. In terms of the computing media, this means: a RAM. They do not represent, but instead index. They are
number is broken down into its decimals and is no longer images that are not intended for the eyes of the users, but
depicted in its semantic unity, but instead distributed across are observed by the computer itself. 32
the area of a decimal digit(al) display. The Whirlwind computer was one computer that
5  The Manchester Mark 1 Williams This discrete nature of the geometric location and chro- worked with Williams tubes, the first computer that pro-
tube with the pickup plate. matic value distinguishes the radar display from the movie/ cessed data in real time and was simultaneously capable of
television picture and, long before the computer was to con- depicting anumeric data on a display.
quer media, already pointed toward our current media era in An early computer advertisement for the Whirlwind
which displays have become the signature of (full) digitality. computer shows how a cathode beam hits the phosphor
While calculating with decimal digits was still associated layer in a cathode ray tube and produces an illuminated
with the analog world, this now changes for the comput- dot (fig. 6). An illuminated point (digit) represents a 1 and
er-illiterate observer, through the use of binary digits. a non-illuminated point a 0 in this rectangular grid in the
The 1948 Manchester Mark 1 is regarded as the first Williams tube. In practice, however, as a pickup plate was
binary digital computer. Computer CRTs in this computer fixed opposite the data screen to protect the data storage
also acted as an accumulator, as well as control and arith- tubes from electromagnetic radiation, the data storage con-
metic registers. The computer pioneers Tom Kilburn and tents could not be read off directly during normal operation.
Frederick Williams used modified cathode ray tubes, called People in media studies therefore reached the conclusion
“Williams tubes”, as random access memory (RAM) for the that the digital per se is withdrawn from perception and
1948 Manchester Mark 1 (fig. 5). 30 Such vacuum tubes were that the hidden nature forms the condition of possibility for
6  Cover of the Whirlwind I brochure
29 Friedrich Kittler, Computergrafik. Eine halbtechnische Einführung, in:
with a schematic illustration of a
Herta Wolf (ed.), Paradigma Fotografie. Fotokritik am Ende des fotograf-
Williams tube.
ischen Zeitalters, Vol. 1, Frankfurt/M.: Suhrkamp Verlag, 2002, pp. 178–194,
p. 179. 31 The Airspace Company Convair later developed a 7 inch tube monitor for
30 Simon H. Lavington, Computer Development at Manchester University, in: the ERA 1103 which could display 6 × 6 characters. See Paul E. Ceruzzi, Eine
Nicholas Metropolis et al. (eds.), A History of Computing in the Twentieth kleine Geschichte der EDV, Bonn: mitp-Verlag, 2003, pp. 63–66.
46 Century, New York/London: Acdemic Press, 1980, pp. 433–443, p. 433. 32 Claus Pias, Computer-Spiel-Welten, Munich: Sequenzia Verlag, 2002, p. 75.
Early Digital Images

the function of digital data storage. 33 Full separation of data play screen”, 39 this is due to a perspective that attributes
and display, which can be regarded as the central adage of far greater importance to the media specifics of moving
digitality, was only achieved with Whirlwind I. characters than the (admittedly abstract) illustration of a
In 1945, project leader Jay Forrester started with the trajectory. Otherwise, the ENIAC demo program would
development of the Whirlwind computer at the Massachu- have to have been ranked first in a media history of the dis-
setts Institute of Technology (MIT) with the words: “We are play – especially if we consider the fact that the mediality of
no longer building an analog computer; we are building a digi- the ENIAC targets the contingent visualization and calcula-
tal computer.” 34 “One of the things that I think we did first was tion of trajectories, just like the Whirlwind. These displays
to connect a visual display to a computer”, 35 reports Robert R. reveal an immediate continuity in computing practices – at
Everett, the engineer on the Whirlwind project at that time. least when Whirlwind I is considered in the context of the
In this case, displays also initially served the purpose development of the Semi-Automatic Ground Environment
of checking for errors and carrying out tests, not for com- (SAGE).
plex data output or input. 36 For example, in 1949, only one
ball (dot) jumps across the 5” Tektronix oscilloscope of the The obvious updating of SAGE related to [...] the status
Whirlwind computers to demonstrate the speed and graph- of the picture. The radar systems that were connected
ics capabilities of the computer (fig. 5–6). When the Bounc- supplied the position of an object with the aid of angu-
ing Ball Program is referred to as the first demo program lar coordinates that were converted into Cartesian
in the history of software, 37 “the first display program ever coordinates based on the location of the radar and were
written” 38 and “the first significant use of the computer dis- indicated on the screen. The separation of data and dis-
play creates an arbitrariness of the depiction, such that
33 See Wolfgang Ernst, Den A/D-Umbruch aktiv denken – medienarchäolo-
gisch, kulturtechnisch, in: Jens Schröter, Alexander Böhnke (eds.), Ana-
it is no longer the screens doing the work (as is the case
log/Digital – Opposition oder Kontinuum? Zur Theorie und Geschichte einer for the Williams tube) but users working on them.40
Unterscheidung, Bielefeld: transcript-Verlag, 2004, pp. 49–65; Pias 2002 (as
fn. 32), p. 75.
34 Robert R. Everett, Whirlwind, in: N. Metropolis et al. (eds.), A History of During the Bedford Tests in 1950, the Whirlwind computer
Computing in the Twentieth Century, New York/London: Acdemic Press, was used as the central control station for the Cape Cod
1980, pp. 365–384, p. 365.
35 Ibid., p. 375. Experimental Air Defense System, a prototype for the ­aerial
36 “All we used the displays for was testing the various parts of the system defense and early warning system, SAGE. An additional
so displays were ancillary completely to the main event.” Norman Taylor,
“computer-generated visual display” 41 not only depicted
as cited in Jan Hurst et al., Retrospectives I. The Early Years in Computer
Graphics at MIT, Lincoln Lab and Harvard (Panel Proceedings of SIG-
GRAPH ‘89), in: Computer Graphics 23.5 (1989), pp. 19–38, p. 22. 39 Jan Hurst et al., Retrospectives II. The Early Years in Computer Graphics
37 Claus Pias, Die Pflichten des Spielers. Der User als Gestalt der Anschlüsse, at MIT, Lincoln Lab and Harvard (Panel Proceedings of SIGGRAPH ’89),
in: Martin Warnke et al. (eds.), Hyperkult II, Bielefeld, 2004, pp. 313–341, in: Computer Graphics 23.5 (1989), pp. 39–74, p. 40.
p. 321. 40 Pias 2002 (as fn. 32), p. 77.
38 Jan Hurst et al., Retrospectives I. The Early Years in Computer Graphics at 41 IBM Corporation – Military Products Division, On Guard! The Story of
MIT, Lincoln Lab and Harvard(Panel Proceedings of SIGGRAPH ‘89), in: SAGE, https://archive.org/details/OnGuard1956 (accessed November 1,
Computer Graphics 23.5 (1989), pp. 19–38, p. 21. 2017). 47
 Tristan Thielmann

7  Trajectory of a bouncing ball and a rocket on the Whirlwind I 5” Tektronix 8  Trajectory of a rocket (without afterglow). The x-axis of the coordinate
oscilloscope. system is slightly curved on the right display. The dot of light (the rocket)
is located at the highest point, the apex of the trajectory. The right bar
indicates the velocity of the rocket.

dots but also depicted and georeferenced lines and text as Even so, practices relating to the display still had a long way
Cartesian target coordinates in the visualization of airplane to go before it could be used like a map. The initial difficul-
movements (fig. 7–8): ties in depicting flying objects on electronic maps included
the fact that the radar system at Bedford airport that was
Such screens, referred to as ‘vector screens’ […] had no used to carry out the first Cape Cod tests allowed two dif-
filled areas, but potentially only lines, such as those ferent modi of data representation.43 The radar antennae
indicating borders and travel routes. The alphanumeric rotated four times a minute without suppression of inter-
text of coordinates also had the status of a map. Letters ference, while the radar only achieved two rotations when
and numbers were composed of dots in the blackness of suppression of clutter was switched on – the deletion of
the monitor that were connected by lines like an astro- undesirable data was required for improved legibility of
nomical constellation.42 the display (fig. 7–8).

43 Kent Redmond, Thomas M. Smith, From Whirlwind to MITRE. The R&D

42 Claus Pias, Punkt und Linie zum Raster, in: Markus Brüderlin (eds.), Orna- Story of the SAGE Air Defense Computer, Cambridge, MA: MIT Press, 2000,
48 ment und Abstraktion, Cologne: Dumont, 2001, pp. 64–69, p. 66. pp. 81–82.
Early Digital Images

In practice, the airplanes thus easily disappeared from belong to which aircraft, and 4) using the data to bring
the perceptual horizon as their blips only lit up twice a min- each track up-to-date with a new smoothed velocity
ute. Continuous tracking was therefore almost impossible.44 and position, and 5) predicting track positions in the
The determination of the location of mobile airborne future for the next correlation or for dead reckoning
objects and the depiction of their location on the display if data is missed. Once smoothed tracks have been
were strictly coupled to each other. Even though the digital calculated, the computer then solves the equations of
PPI screen therefore primarily visualizes the function of the ­collision-course interception and generates and dis-
rotating radar antenna itself and thus renders apparent the plays the proper vectoring instructions to guide an
media methodology of the display, all practical steps that are interceptor to a target.46
taken are tied to it as if the basis of the display image was
formed by a continuous “track-while-scan ability” 45. Even though the entire process was predetermined, such
On March 13, 1953, Robert Wieser explained in a lecture that the operators are denied independent coordinating
to visitors to the Cape Cod Experimental Air Defense Sys- practices on the displays in the literature, the interpreta-
tem what the different data processing steps were in order tion that is required in relation to the displays must not be
for the Whirlwind computer processes to make a special forgotten in a sociotechnical consideration of the issue.47
display available that monitors and controls the paths, path The radar screen shows radial snapshots in time, at reg-
guidance and path guidance aids: ular intervals, of the continuous movement of an object in
real space. The conditions underlying the visualization are
The radar data is fed into the Whirlwind I computer clear and apparent to the participating actors; they require
at the Barta Building in Cambridge, which processes no justification. Nobody would assume that the airplane that
the data to provide 1) vectoring instructions for mid- is being tracked in this way was moving discontinuously, as
course guidance of manned interceptors and 2) special depicted on the display. A significant praxeology of the dis-
­displays for people who monitor and direct the opera- play has developed based on the presentation of incomplete
tion of the system. path information and its completion by the actors, as prac-
In processing data, the computer automatically per- tices relating to interpretation and actions have arisen due
forms the track-while-scan function, which consists of to this medium that require no further justification, given
l) taking in radar data in polar coordinates, 2) convert- the strict adherence to instructions and incorporation into
ing it to rectangular coordinates referred to a common the rules of the system.
origin, 3) correlating or associating each piece of data
with existing tracks to find out which pieces of data 46 C. Robert Wieser, Cape Cod System and Demonstration. Memorandum VI –
L-86, MIT Lincoln Laboratory, Division 6, Cambridge, MA, March 13, 1953,
p. 2, http://dome.mit.edu/bitstream/handle/1721.3/41510/MC665_r28_L-
44 See Servomechanism Laboratory, Air Traffic Control Summary Report 7, 86.pdf (accessed November 1, 2017).
July 25, 1950 – October 25, 1950, Massachusetts Institute of Technology, 47 See Kjeld Schmidt, Cooperative Work and Coordinative Practices. Contri-
Cambridge, MA, 1950. butions to the Conceptual Foundations of Computer-Supported Cooperative
45 Redmond, Smith 2000 (as fn. 43), p. 77. Work (CSCW), London et al.: Springer, 2011, p. 318. 49
 Tristan Thielmann

Precisely because the computer users sitting in front of close together, and c) how data can be smoothed – something
the displays were not in a position to change the situation that was required due to the inaccuracy of the positional
or the program,48 practices relating to displays require a not data obtained by radar. 53 The unified planimetric depiction
entirely insignificant ability to normalize, as is also demon- on the display required “rationalization of sight” 54 – a mod-
strated in studies on current GPS navigation practices.49 ification to adapt the technically perceived reality to the
Lev Manovich’s reference to “visual nominalism”, the conditions of the display.
automated “use of vision to capture the identity of individ- Robert Everett, who had already made a significant
ual objects and spaces by recording distances and shapes”, 50 contribution with Jay Forrester toward driving forward
must therefore be contrasted with an equivalent visual the construction of the Whirlwind computer, developed a
norminalism. Manovich reduces the radar to “seeing with- light gun as a computer interface – “a photoelectric device
out eyes”, 51 whereby the praxeological changes that arise which is placed over the desired spot on a display scope” – as
from the radar display are overlooked: a modified seeing the joystick had proven too slow and cumbersome to oper-
with eyes under the conditions of remote sensing. ate. 55 The joystick was then discarded at the end of 1950.
In addition, the central change that is associated with The medium of the light gun corresponded to the planned
the digital radar display occurs at the interactional level: media practice of the Air Defense computer “to zero in on
in mid-1950, a joystick was initially used to pursue a target a selected target”, 56 not only on a screen but also in the air
object, with which a mobile dot of light was moved until it with the help of fighter jet interceptors.
covered the radar blip of the target object. The target thus The programing foundations for the interception tests
selected could then be pursued automatically. 52 were laid in the second half of 1950. The criteria for a suc-
In addition to this manual targeting, simultaneous work cessful computer test were a) the simultaneous tracking of
was being carried out on a) how targeting can be carried the target of the attack and the interceptor on the display, b)
out automatically within a selected corridor, b) how target the calculation of the target guidance data, and c) the trans-
objects can be separated from each other, even if they are mission of the control command to the fighter jet interceptor
or rocket interceptor. 57 However, one problem in relation to
48 Judy E. O’Neill, The Evolution of Interactive Computing Through Time-shar- tracking flying objects was that stationary targets were not
ing and Networking, University of Minnesota, 1992, p. 21.
49 See Barry Brown, Eric Laurier, The Normal Natural Troubles of Driving yet suppressed on radar displays until mid-1950.58 This only
with GPS, in: Proceedings of the ACM CHI 2012. Conference on Human Fac- changed in October 1950, with the patenting of the “Moving
tors in Computing Systems, Austin (2012), pp. 1621–1630.
50 Lev Manovich, The Mapping of Space: Perspective, Radar, and 3-D Comput-
er Graphics, in: Thomas Linehan (ed.) Computer Graphics Visual Proceedings, 53 See Redmond, Smith 2000 (as fn. 43), p. 80.
Annual Conference Series, ACM SIGGRAPH ’93, New York, 1993, pp. 143–147, 54 William M. Ivins, On the Rationalization of Sight. With an Examination of
http://manovich.net/content/04-projects/003-article-1993/01-article-1993. Three Renaissance Texts on Perspective, New York, 1973 [Original published
pdf (accessed November 1, 2017), p. 2. as Metropolitan Museum of Art Papers 8 (1938)].
51 Ibid., p. 4. 55 Servomechanism Laboratory 1950 (as fn. 52), p. 1.
52 See Servomechanism Laboratory, Air Traffic Control Summary Report 6, 56 Redmond, Smith 2000 (as fn. 43), p. 81.
April 25, 1950 – July 25, 1950, Massachusetts Institute of Technology, Cam- 57 See Ibid., pp. 83–84.
50 bridge, MA, 1950, p. 4, p. 8. 58 See Ibid., p. 79.
Early Digital Images

Target Indicator”, a module for the suppression of stationary The scope operator’s task was thus reduced to the selection
targets. 59 and passing on of information. The operator fulfilled a relay
Before radar targeting of mobile objects turned into the function62 that also could have been automated.
suppression of stationary targets, two 5” oscilloscopes were In this sense, selection was part of a “distributed cog-
used to separate mobile from stationary objects. A 12” PPI nition” process,63 because the power behind the action of
screen only became available to the MIT Digital Computer selecting the target with the light gun is limited to the sep-
Laboratory toward the end of 1950.60 One display was used aration of a mobile object from the stationary objects, to
to select objects with a light gun and a second display for extracting it and transferring it to a second display using
the presentation of the marked objects. Input and output copy/paste. This second display is based on a different con-
screens were still separated from each other. struction of reality: on a world that only knows vehicles.
The production of coordinates that had been conduct-
Tracking would be initiated manually by applying the ed previously outside the display was shifted to translation
light gun to a selected target on the main scope. So steps between two displays. With reference to the narrative
far, use of the test patterns indicated that the light- and visual complexity of the overall context that is being
gun technique should work. Holding the light gun on depicted, this therefore threw the development of displays
the location of the target long enough to detect the back to before the Seeburg plotting table, which was orient-
computer’s next scanned display would transfer the ed to an even greater extent based on mimetic procedures.
target’s display to the second scope. Doing the same The Himmelbett method required a consensual interpre-
to the interceptor’s spot on the first scope would select tation by the actors standing around the display. From an
it too for display on the second scope. From that time action-theoretical perspective, the development of the radar
on, the two selected blips would be tracked in isolation display that occurred in 1950 simultaneously constituted a
on the second scope, without further need for the light step in sociotechnical innovation: Through the separation
gun. Their courses would be predicted on the basis of into a data input and a data output display, “cooperation
the history of preceding sightings. A collision course
would be computed, proper heading instructions for the
interceptor would be displayed, and the scope operator
would pass on the information by voice to the pilot in
62 See Antoine Hennion, Cécile Méadel, In the Laboratories of Desire. Adver-
the interceptor. 61
tising as an Intermediary between Products and Consumer, in: Reseaux. The
French Journal of Communication 1.2 (1993), pp. 169–192. Within the scope
of the actor-network theory, a relay can also be understood as a team that
takes over and triggers a relay race of further actions. See Tristan Thiel-
mann, Digitale Rechenschaft. Die Netzwerkbedingungen der Akteur-Me-
59 Charles T. Baker Jr., Moving Target Indicator Radar, Patent-No. US 2811715 dien-Theorie seit Amtieren des Computers, in: Tristan Thielmann, Erhard
A, October 2, 1950. Schüttpelz (eds.), Akteur-Medien-Theorie, Bielefeld: transcript-Verlag, 2013,
60 See Redmond, Smith 2000 (as fn. 43), p. 83. pp. 377–424, p. 382.
61 Ibid., p. 84. 63 Edwin Hutchins, Cognition in the Wild, Cambridge, MA: MIT Press, 1995. 51
 Tristan Thielmann

without consensus” 64 became possible for the first time The display reveals its media methodology in the first
in graphic human-computer interaction, in that targeting computer applications, as demonstrated by these elabora-
(enemy flying objects) and target guidance (of the fighter tions. This shows that a technical component (an oscillo-
jet interceptors) were separated media-praxeologically and scope) is used in a different way from what was originally
became an action distributed across displays. intended – as an optical snapshot in time of the location of
At this point, it is interesting to note the use of the term dots, instead of one or more courses of waveforms. Put in
scope, used colloquially to refer to a) an oscilloscope and b) graphical terms, a loophole is left that must be closed by the
a viewfinder or a telescopic sight in military terms, and c) user through the determination of a mobile final destination.
the word more generally refers to a frame for movement Both at the technical and at the practical levels, this display
and latitude. is all about producing an endpoint coordinate. Furthermore,
These three different levels of meaning also actually a scope is not only etymologically linked to the media prac-
emerge in the SAGE Air Defense Computer and its proto- tice of searching, from a genealogical perspective, a scope is
types. On the one hand, this display is a converted measur- also focused on applications that try to fix movement.
ing instrument – an oscillograph that reduces waveform The different Whirlwind computer displays thus have
graphs to the depiction of dots. The scope is one such mea- a media-technical, practical, normalizing and nominalist
suring instrument and indicates that it once simply served dimension, all of which are also of importance for our cur-
to control the computer, as was the case for the ENIAC. On rent understanding of displays. This section has demonstrat-
the other hand, it also served as an instrument to search for ed that the separability and addressability of the individual
enemy objects. At the same time, the only demand that could pixel is a variable that takes precedence over the question of
be made was the displaying of a section of reality: a reality whether data are used for display or storage. The procedure
that is solely determined by the movement of the objects of discretization must therefore be considered as an essen-
that have been reduced to a dot. tial component of a praxeology of the display.65 The reason
Equally, the scope no longer serves the sole purpose of this is so important is that the discretization comprises
monitoring the internal and external world of the computer. both the fabrication of image and location dots. This step
With the advent of digitality, the internal world increasingly in the development in display media praxeology is there-
closes up; additional translation steps are incorporated by fore characterized by a sociotechnical duplicity of endpoint
new interfaces; the external world is simultaneously rep- coordinates.
resented in a more media-differentiated way – limited to a
circular excerpt, to mobile objects and discrete characters.

65 On understanding media praxeology, see special issue of Digital Culture &

Society 2 (2017) on Mobile Digital Practices; Erhard Schüttpelz, Epilogue.
64 Susan Leigh Star, Cooperation Without Consensus in Scientific Problem Media Theory Before and after the Practice Turn, in: Ulrike Bergermann
Solving. Dynamics of Closure in Open Systems, in: Steve M. Easterbrook et al. (eds.), Connect and Divide: The Practice Turn in Media Studies. The 3rd
52 (ed.), CSCW. Cooperation or Conflict?, London: Springer, 1993, pp. 93–106. DFG conference of Media Studies, Zürich/Berlin: diaphanes (forthcoming).
Early Digital Images

Flood Lights this respect, a far more comprehensive heuristic continuum

can be described based on the “duplicity of code”,67 through
The current analysis demonstrates that representations and which image and spatial production are constituted inter-
their arbitrary dimension can only form part of a media-es- actively and are reified in displays.
thetic investigation. The decisive media-methodological In addition, the praxeology of the display shows how
occurrence does not take place on the display, but behind it historical media methods reach into the present and are
from a technical perspective and in front of the display from still having a formative effect on current manifestations of
a practical perspective. A combined technical and practice displays. The display not only harbors within it the intrin-
theory is therefore required to understand the specific sic persistence of guidance and bearings, it cannot deny its
mediality of the display. genealogy, which stems from the medium of the radar. The
This kind of approach shows how the practice of coor- media methodology of the display aims at a mediated seeing
dination and the materiality of coordinates each determine over distance and the depiction of discrete and addressable
the media-specific nature of the display through their pictorial symbols in the form of co-existing light and loca-
duplicity. This socio-technology of the duplicity of image- tion dots. Even when considered in light, the dispositif of the
space coordinates has the capacity to more closely define the display remains seeing in the dark, inherent in surround-
dispositive structure of digital displays and to present them ings and in proximity. Displays show the immediate mediate.
in their different variants. In this process, the innovative That is its media practice-theoretical dimension.
steps that emerge in the media history of the display are Displays can therefore by all means be called visible
characterized by the elimination and inclusion of media- objects as per the meaning of Thomas Elsaesser, giving light
tors, through which the perceptibility of co-coordination is a spatial form and materiality that goes beyond the artist’s
newly materialized in each case. flat and framed canvas.68 They represent an a-modern return
Up to now, the duplication of spatial and image-related in the cloak of the digital modern, “that returns the fixed
coordinates was simply declared to be a cultural-geographic spectator facing the fixed rectangular screen to being a his-
characteristic of augmented reality applications.66 This essay torically contingent actor” 69 and thereby also allocates spatio-
shows that this scope of analysis that is driven by interven- temporally limited valence to the dispositif of the movie.70
tion is not far-reaching enough. Spatial co-coding not only
characterizes the layering technologies through which the
location-relevant internet information is merged with the
67 Ibid.
live camera image on mobile consumer devices. Co-coding 68 Thomas Elsaesser, The ‘Return’ of 3-D. On Some of the Logics and Gene-
of online and offline spaces already occurred with radars. In alogies of the Image in the Twenty-First Century, in: Critical Inquiry 39.2
(2013), pp. 217–246.
69 Ibid., p. 244.
70 See Hermann Kappelhoff, Der Bildraum des Kinos. Modulationen einer
66 See Mark Graham et al., Augmented Reality in Urban Places. Contested ästhetischen Erfahrungsform, in: Gertrud Koch (ed.), Umwidmungen.
Content and the Duplicity of Code, in: Transactions of the Institute of British Architektonische und kinematographische Räume, Berlin Vorwerk 8, 2005,
Geographers 38.3 (2013), pp. 464–479. pp. 138–149. 53
 Tristan Thielmann

The rectangular viewing window has been a “medi- Both are increasingly becoming one from a phenomeno-
um of visibility” 71 since Roman antiquity, which opens up logical perspective. This increases the lack of conceptual
elementary practical functions, such as illumination and clarity in relation to what we understand an image to be and
views. The display is the first medium that opposes this simultaneously moves into the foreground implicit knowl-
image-generating dispositif in its mediality, materiality and edge and media methods that are solely based on displays.
media practice. Based on the variability and adaptability of In point of fact, it will indeed not be much longer, as it
its shape and its co-coordinating function, it represents a has always taken place and is always occurring anew: Dis-
disillusioning feast for the eyes. Unlike the rectangular win- plays allow us to view images analytically.
dow or screen that is not subdivided, it no longer provides
a view in or out – is thus no longer constrained by archi-
tecture – but represents a socially canonized practice72 of Figures
telemetry and remote sensing. 1–2  Royal Air Force, Standard Technical Training Notes, Part 3: Elementary
When we take a look at the media history of the display, Radar, Great Britain, Air Ministry, London, 1965, p. 37, p. 87.

it becomes clear that the discourse in the pictorial sciences 3  Gyges Publishing Company, http://gyges.dk/Seeburg%20(Small).jpg
(accessed November 1, 2017).
on the discrete nature, operationality and spatial control of
4  Danielstirland, https://en.wikipedia.org/wiki/Battle_of_Britain_Bunker
images precedes the methodology of displays outlined here. (accessed November 1, 2017).
Stiegler has recognized this: 5 http://virtualtravelog.net/images/2003-08-Williams-Kilburn-tube.jpg
(accessed November 1, 2017).
[I]t will not be much longer before we can view i­ mages 6  Curtesy MIT Museum, image curtesy Computer History Museum.
analytically: screens [l’écran] and what is written 7  Claus Pias, Die Pflichten des Spielers. Der User als Gestalt der Anschlüsse,
[l’écrit] are not simply opposed to each other.73 in: Martin Warnke et al. (eds.), Hyperkult II, Bielefeld: transcript-Verlag, 2004,
pp. 313-341, p. 321.
8  Diego Hernan Sanhueza Martínez, See It Now: Jay W. Forrester and the
WHIRLWIND Computer - (1951), https://youtube.com/watch?v=5ZQP4G3Qwb4
(accessed November 1, 2017).

71 Gerd Blum, Epikureische Aufmerksamkeit und euklidische Abstraktion.

Alberti, Lukrez und das Fenster als Bild gebendes Dispositiv, in: Horst
Bredekamp, Christiane Kruse, Pablo Schneider (eds.), Imagination und
Repräsentation. Zwei Bildsphären der frühen Neuzeit, Paderborn: Wilhelm
Fink Verlag, 2010, pp. 79–118, p. 80.
72 See Hans-Jürgen Horn, Fenster (kulturgeschichtlich), in: Reallexikon für
Antike und Christentum, Vol. 7. Stuttgart: Anton Hiersemann, 1970, pp. 732–
737; Achatz von Müller, Der Politiker am Fenster. Zur historischen Iko-
nographie eines „lebenden Bildes“, in: Gottfried Boehm (ed.), Homo Pictor,
München/Leipzig: De Gruyter, 2001, pp. 323–338.
73 Bernard Stiegler, Das diskrete Bild, in: Jacques Derrida, Bernard Stiegler
(eds.), Echographien. Fernsehgespräche, Vienna: Passagen, 2006, pp. 162–
54 188, p. 180.
Jan Distelmeyer

Carrying Computerization
Interfaces, Operations, Depresentations

Indispensable and Invisible1 We can see these changes, but the most dramatic
change affecting human-computer interaction was
Interaction is dismissed. In the end of 2016 the cover of the invisible: what the computer does when we are not
Interactions magazine, published by the Association for Com- interacting with it. 6
puting Machinery since 1994, crossed out the last word in
human computer interaction and replaced it with integration. This correlation between integration, (autonomous) activ-
The “era of human-computer interaction”, the cover story stat- ity and hiddenness was accompanied by another market-
ed, “is giving way to the era of human-computer integration – ing-related connection between computers and invisibility,
integration in the broad sense of a partnership or symbiotic published at the same time. The international video cam-
relationship in which humans and software act with autono- paign “Feel connected all over Europe” of the German tele-
my, giving rise to patterns of behavior that must be considered communications company Deutsche Telekom presented the
holistically” (fig. 1).2 After a summary how “the nature of our singer Andrea Bocelli praising what is called laconically the
interaction has continuously evolved” from “switches, cards, network: “It gives me freedom. Reliably wherever I am. It
and tape to typing, mice, and styluses, adding speech and ges- transcends boundaries. It’s indispensable and invisible.” 7
ture” 3 and a forecast on “brainwave interaction” 4 (recalling Bocelli’s statements are decorated and elaborated by images
Vannevar Bush’s thoughts on “a couple of electrodes on the of him walking, riding, and boating in several iconic places
skull” 5), the most interesting question was raised: in Europe (fig. 2). The fact that Andrea Bocelli is blind and
that this promotional video about network technology and
Some parts of this paper have been published in the journal Cinéma & Cie
1 digitization shows no form of computer technology or infra-
(Vol. XVII). I would like to thank the participants and organizers of the
structure at all, is important for the message with which
workshop “Screen Operations. Conditions of Screen-based Interaction” at
Humboldt University Berlin, 2016, for discussions and comments.
Umer Farooq, Jonathan Grudin, Human-Computer Integration, in: Inter-
2
actions 23.6 (2016), pp. 27–32, p. 27.
Ibid., p. 28.
3
Ibid.
4 Farooq, Grudin 2016 (as fn. 2), p. 28.
6
Vannevar Bush, As We May Think, in: The Atlantic Monthly 176 (1945,
5 Serviceplan Group, Telekom Connecting Europe – TV commercial 2016,
7
https://theatlantic.com/magazine/archive/1945/07/as-we-may-think/ https://youtube.com/watch?v=6No-bDXIdEE (accessed September 20,
303881 (accessed September 20, 2017). 2017).
 Jan Distelmeyer

1  Cover of Interactions, 23/6 (2016). 2  Still from Telekom commercial Feel connected all over Europe
Association for Computing Machinery.

the video concludes: “The network – it’s the present and the ical match – (omni-)present and hidden at the same time.
future. You can’t see it. But you can feel it.” 8 From early plans of Ubiquitous Computing up to current
Both contributions to the (inexisting) presence of com- concepts of “calm” and imbedded technologies the linkage
puters are expressions of an important consilience: On the between ubiquity, efficiency, and invisibility is important,
one hand, the current discussions and developments of especially in concepts and presentations of developers.10
concepts like Ambient Intelligence, Internet of Things, and One of the most famous examples of the living contra-
Smart Environments promote the powerful and consequen- diction of invisibility and readiness-to-hand was given by
tial omnipresence of computers, which is, on the other hand, Steve Jobs’ farewell performance as CEO of Apple in San
understood and promoted as indiscernible, unobservable,
embedded, and (nonetheless or therefore) effective.9 A mag- potential paths for technocultural interrelations and interactivities.” Ulrik
Ekman, Individuations, in: Ulrik Ekman, Jay David Bolter, Lily Diaz, Maria
Engberg, Morten Søndergaard (eds.), Ubiquitous Computing, Complexity,
Ibid.
8 and Culture, New York: Routledge, 2016, pp. 77–90, p. 83.
“It is still a matter of some debate whether and how interaction designs for
9 10 Natascha Adamowsky, Vom Internet zum Internet der Dinge. Die neue Epis-
or ubicomp culture are to tend towards the transparent and calm invisibil- teme und wir, in: Florian Sprenger, Christoph Engemann (eds.), Internet
ity of an infrastructure for interactivity or towards a more personalized, der Dinge. Über smarte Objekte, intelligente Umgebungen und die technische
56 attention-getting, even exciting unfolding, and mediatory laying bare of Durchdringung der Welt, Bielefeld: transcript, 2015, pp. 231–265, p. 245.
 Carrying Computerization

Francisco on 6 June 2011. Introducing the service iCloud, environment that we experience simply by being and act-
Jobs illustrated his idea of an autonomous service, for which ing in space and time”),15 should not be misunderstood as
“I don’t even have to take the devices out of my pocket” or “to a disappearance of human computer interfaces or even of
be near my Mac or PC”.11 Since interfaces at all.

all these new devices have communications built into

Interfaces Carry
them; they can all talk to the cloud whenever they want.
[…] And now everything’s in sync with me not even Why do interfaces matter? Firstly, because the term refers
having to think about it. […] [S]o everything happens to different modes and processes of connectivity that ensure
automatically and there’s nothing new to learn. It just the functionality of (all forms of ) digital computers in the
all works. It just works.12 first place. It is important to remember that the term inter-
face, introduced by the physicists James and William Thom-
Obviously today’s interface culture is shaped very much by son in late 19th century, was originally used to describe the
various forms of interfacing with computers that cannot transmission of energy.16 Their usage of the term “would
be reduced to user interfaces, we address by touching or define and separate areas of unequal energy distribution
clicking, gestures or voices. N. Katherine Hayles remark, within a fluid in motion, whether this difference is given
“[m]obile phones, GPS technology, and RFID (radio frequen- in terms of velocity, viscosity, directionality of flow, kinetic
cy identification) tags, along with embedded sensors and form, pressure, density, temperature, or any combination
actuators, have created environments in which physical and of these”.17 With this in mind, the question of the pursued
virtual realms merge in fluid and seamless ways”,13 sums up ubiquity and networked embeddedness of computing, rely-
some forms of interfaces building and organizing seamless ing basically on transportation of signals and the carrying
processes of connectivity. But this development, mirrored of electricity, is inevitably a question of interfaces. The term
by the term “Post-Interface” 14 and Mark B. N. Hansen’s interface helps to describe the “interior telegraphy” of the
perspective on “twenty-first-century media” (“no longer computer as well as all forms of its networks, its relations to
a delimited temporal object that we engage with focally us and its incorporations.18 Hence, the ongoing development
through an interface such as a screen, media become an
15 Mark B. N. Hansen, Ubiquitous Sensation. Towards an Atmospheric, Imper-
sonal and Mircotemporal Media, in: Ulrik Ekman (ed.), Throughout. Art and
11 EverySteveJobsVideo, Steve Jobs introduces iCloud & iOS 5 – WWDC (2011), Culture Emerging With Ubiquitous Computing, Cambridge: MIT Press, 2013,
https://youtube.com/watch?v=gfj7UgCMsqs (accessed September 20, 2017). pp. 63–88, p. 73.
12 Ibid. 16 See Peter Schaefer, Interface. History of a Concept, 1868–1888, in: David W.
13 N. Katherine Hayles, Cybernetics, in: W. J. T. Mitchell, Mark B. N. Hansen Park, Nicholas W. Jankowski, Steve Jones (eds.), The Long History of New
(eds.), Critical Terms for Media Studies, Chicago: University of Chicago Media. Technology, Historiography, and Contextualizing Newness, New York:
Press, 2010, pp. 145–156, p. 148. Peter Lang, 2011, pp. 163–175.
14 Michael Andreas, Dawid Kasprowicz, Stefan Rieger, Technik | Intimität. 17 Branden Hookway, Interfaces, Cambridge: MIT Press, 2014, p. 59.
Einleitung in den Schwerpunkt, in: Zeitschrift für Medienwissenschaft 15 18 Hartmut Winkler, Prozessieren. Die dritte, vernachlässigte Medienfunktion,
(2016), pp. 10–17, p. 12. Munich: Fink, 2015, p. 294. 57
 Jan Distelmeyer

of an increasingly hidden dissemination, interconnection, towers and satellite networks; and preparatory to our
and implementation of computers cannot be understood using our digital devices or our laptops to communicate
without asking about interface processes.19 or to acquire information, the latter engage in complex
Graphical user interfaces are but one of the multilayered connections with wireless routers and network hosts.22
aspects characterizing interfaces in terms of digital com-
puting. These “symbolic handles”, as Florian Cramer and While these devices are constantly and so-called calmly
Matthew Fuller have called them, “which […] make software interfacing with networks and servers, we also do use our
accessible to users” depend on and are connected to other smart phones in active ways, which is why we pay for and
interface aspects and processes, such as hardware connect- update them. Even today, graphical user interfaces are so
ing humans/bodies to hardware, hardware connecting hard- obviously omnipresent that this manifestation of software
ware to hardware, software connecting software to hardware, is still “often mistaken in media studies for ‘interface’ as
and software providing software to software connections.20 a whole”.23 Screen operations belong to the chief activities
Secondly, our encounter with computers in all its forms in large parts the world; work and leisure activities are
by use of programmed and designed user interfaces is not increasingly involving screen activities, just as three most
superseded but accompanied by “pervasive” and “ubiqui- popular websites worldwide – Google, YouTube and Face-
tous” computing that Ulrik Ekman has described as “a socio- book – bank on our interactions with their offerings on dif-
cultural and technical thrust to integrate and/or embed ferent kind of screens.24 Despite this, media studies analyses
computing pervasively, to have information processing thor- of common user interfaces are still not common.25 This must
oughly integrated with or embedded into everyday objects change if we are to better understand of our relationship
and activities, including those pertaining to human bodies with (previous, current, and upcoming) forms of computing.
and their parts”.21 Mark B. N. Hansen’s description of the
“experiential shift” by “twenty-first-century media” depicts 22 Mark B. N. Hansen, Feed Forward. On the Future of Twenty-First-Centu-
ry-Media, Chicago: University of Chicago Press, 2015, p. 62.
the diversity of interconnected interface politics:
23 Cramer, Fuller 2008 (as fn. 20), p. 149.
24 Alexa web analytics, http://alexa.com/topsites (accessed September 20,
Thus, well before we even begin to use our smart 2017).
25 For exceptions, see Matthew Fuller, It looks like you’re writing a letter. Mic-
phones in active and passive ways, the physical devices rosoft Word, in: Matthew Fuller (ed.), Behind the Blip. Essays on the Cul-
we carry with us interface in complex ways with cell ture of Software, New York: Autonomedia, 2003, pp. 11–37; Christian Ulrik
Andersen, Søren Pold (eds.), Interface Criticism. Aesthetics Beyond Buttons,
Aarhus: Aarhus University Press, 2011; Margarete Pratschke, Interacting
19 “But alongside and interwoven with computational and networked digital with Images. Toward a History of the Digital Image: The Case of Graph-
media, more than one ‘environmental’ system of calculation, slipping in and ical User Interfaces, in: Horst Bredekamp, Vera Dünkel, Birgit Schneider
out of direct perception, and the multiple interfaces between them are to (eds.), The Technical Image. A History of Styles in Scientific Imagery, Chica-
be reckoned with.” Matthew Fuller, Foreword, in: Ekman 2013 (as fn. 15), go: University of Chicago Press, 2015, pp. 48–57; Teresa Martínez Figuerola,
pp. xi–xxvi, p. xx. Jorge Luis Marzo (eds.), Interface Politics, Barcelona: BAU, 2016; Florian
20 Florian Cramer, Matthew Fuller, Interface, in: Matthew Fuller (ed.), Soft- Hadler, Joachim Haupt (eds.), Interface Critique, Berlin: Kadmos, 2016; Jan
ware Studies. A Lexicon, Cambridge: MIT Press, 2008, pp. 149–152, p. 149. Distelmeyer, Machtzeichen. Anordnungen des Computers, Berlin: Bertz +
58 21 Ulrik Ekman, Introduction, in: Ekman 2013 (as fn. 15), pp. 1–59, p. 22. Fischer, 2017.
Carrying Computerization

These analyses are necessary because interfaces define Operative Images and Depresentation
today’s reality in manifold ways. Understood as the com-
plex of various processes of connectivity and conduction, The interdependence of aesthetics and dispositifs demands
interfaces do carry – on all levels of its acceptation – the attention be paid to the special status of these images and
worldwide computerization, in which graphical user inter- signs that – to quote a Windows 10 commercial from 2015 –
faces still build the real but underestimated blockbusters of “help you do your thing”. 27 Of course, these so-called com-
today’s visual politics. puter icons could likewise be symbolic, depending on the
Graphical user interfaces inform us (to some extent) specific interface design. Regardless of the potentially iconic
of the real and the imaginary, the well-prepared and con- or symbolic character of these images and signs, all clickable
sequential relations between humans and computers as or touchable appearances correspond to Peirce’s idea of indi-
applied in computers. They mediate interrelations between ces.28 These images and signs must have a physical relation
humans and computers. Studying its interface politics to the somehow presented processes of computing, to the
allows for the computer to be realized as a particular “power interior telegraphy of the computer. They “show something
machine”,26 which enables us to examine a key component about things, on account of their being physically connected
of computers and computerized media/things/beings: pro- with them” 29; otherwise they simply would not work. 30
grammability. Hence, images and videos on YouTube could combine
The fact that graphical user interfaces work so differ- different indexical qualities. Dealing with the YouTube
ently from, for instance, cinematic or televisual appearances interface by clicking/touching on a video thumbnail offered
and do inevitably rely on other interface processes between on the screen or by clicking/touching on a running video to
all sorts of hard- and software makes the task of interface stop or to enlarge it, is an expression of a certain semiotic
analysis and critique so urgent. The example I would like to shift. Because indexicality of these images and videos is no
comment on here is the YouTube interface – those immense- longer only generated by a potential trace to a pre-filmic
ly popular conditions with which we upload, search, identify, reality but also by the trace to the interior telegraphy of the
organize, tag, encounter, and negotiate the occurrence of networked computer that makes its existence and usage pos-
video material on this second most popular website world- sible. Since we are invited and enabled to interact with these
wide. To turn towards these special screen operations, it is
important to consider their operative images as depresen- 27 Windows, Windows 10 Highlights Reel, https://youtube.com/watch?v=j-
3ZLphVaxkg (accessed September 20, 2017).
tations.
28 See Marianne van den Boomen, Transcoding the Digital. How Metaphors
Matter in New Media, Amsterdam: Institute of Network Cultures, 2014,
pp. 37–41.
29 Charles S. Peirce, What is a sign, in: The Peirce Edition Project (ed.), The
Essential Peirce. Selected Philosophical Writings Volume 2 (1893–1913),
Bloomington: Indiana University Press, 1998, pp. 4–10, p. 5.
30 To specify this indexicality, it is helpful to remember the difference between
what Peirce called a genuine index and a degenerated index, because graph-
26 Distelmeyer 2017 (as fn. 25), pp. 82–92. ical user interfaces combine both forms of Peirce’s indexicality. 59
 Jan Distelmeyer

of causation, to machine processes to be executed in order

to yield a specific result.” 33
Graphical user interfaces visualize what the computer
offers to do in a particular way without, showing what is
actually happening inside the machines. “Software, or per-
haps more precisely OS,” as Wendy Chun has stated, “offer
us an imaginary relationship to our hardware: they do not
represent the motherboard or other electronic devices but
rather desktops, files, and recycling bins.” 34 Unsurprisingly
the YouTube interface does not represent any of the hard-
ware of the servers or network processes necessary to real-
ize my search request.
Nevertheless at the same time this hardware based rela-
tionship offered by software – depresented by symbolic or
iconic signs – offers more than just an imaginary relationship
3  Screenshot of the ABC video Trump Presidential Inauguration on YouTube.
to the working hardware of the computer, for example, in
the form of the motherboard. These clickable or touchable
images, they refer to another realm – to the processuality of signs are simultaneously linked electronically to the inner
my networked computer (fig. 3). processes of the machine, to its interior telegraphy, whose
The video Trump Presidential Inauguration 2017 and its flow of electronic signals connects, among others, the moth-
thumbnail image is photographically indexical because of erboard to the indexical signs of the graphical user interface.
its “physical relation between the object photographed and In terms of YouTube the click/touch on the offered thumb-
the image finally created”. 31 The additional indexical qual- nail of the video FULL VIDEO – The Inauguration of Donald
ities of the video – which we have learned to stop and con- J. Trump instructs the interior telegraphy of my computer
tinue by clicking/touching on it, as well as of the thumbnail, to use the exterior and protocol-driven telegraphy of the
which we click/touch to select and start the video in the first World Wide Web to stream the requested data. An imag-
place – are relying on the programmability of computers. inary and at the same time real and physical relationship
This programmatic indexicality is reliable, because these enables us to click/touch these images, to start the prom-
images and signs “materially refer to an act of executing
machine code” 32: “They refer to existential, physical chains

31 Tom Gunning, What’s the Point of an Index? Or, Faking Photographs, in: 33 Ibid.
Nordicom Review 25.1–2 (2004), pp. 39–49, p. 40. 34 Wendy Hui Kyong Chun, Control and Freedom. Power and Paranoia in the
60 32 van den Boomen 2014 (as fn. 28), p. 40. Age of Fiber Optics, Cambridge: MIT Press, 2006, p. 20.
 Carrying Computerization

ised and hidden algorithmic processes, which is why Frieder Volker Pantenburg has emphasized that operative imag-
Nake calls them “algorithmic images”. 35 es “aren’t intended to be released separately, and strictly
The contradictory character of these images and signs speaking don’t need to appear as images at all but emerge
has led Marianne van den Boomen to introduce the very as the intermediate product of a wider technical process”.41
fruitful term of depresentation. They show what we can Farocki described them as differentiated by purpose: “In my
do without showing the “procedural complexity” and the first work on this subject, Eye/Machine (2001), I called such
multitude of requirements and consequences attached: pictures, made neither to entertain nor to inform, ‘operative
“[T]he icons on our desktops do their work by representing images’.” 42 This last point is crucial and marks a productive
an ontologized entity, while depresenting the processual difference between Farocki’s concept and my application of
and material complexity involved. This is the way icons it.43 Whereas the operative images of a graphical computer
manage computer complexity, this is the task we as users (in interface may not be made for edification, information or
tacit conjunction with designers) have delegated to them.” 36 instruction in the classical sense (“Erbauung oder Beleh-
To address the special quality of these “symbolic rung” 44), they do (and must) instruct users on what can be
handles”, 37 I have discussed them as “operative images”, done. What they instruct and are part of through depresen-
adopting a concept coined by Harun Farocki to describe tation is a kind of knowledge about computers, about their
the production of images by machines for machines. 38 This usage, and about us – it forms an “implicit memory”.45
term  – translating “operative Bilder” Farocki has called The interdependency with technical execution (“tech-
them “operative images” as well as “operational pictures” nischen Vollzug” 46) differentiates this form of operativity
and “operational images” – is driven by the interest in pro- from others, as for instance the operative imagery, operative
cesses not represented by these operative images but rather writing, and diagrammatic operations of Sybille Krämer’s
of which operative images are part of themselves. 39 “These approach to diagrammatology.47 Operative images as depre-
are images”, Farocki explained, “that do not represent an
41 Volker Pantenburg, Farocki/Godard. Film as Theory, Amsterdam: Amster-
object, but rather are part of an operation.” 40
dam University Press, 2015, p. 210.
42 Farocki 2004b (as fn. 39), p. 17.
43 For other approaches to the term, see Werner Kogge, Lev Manovich. Society
of the Screen, in: Alice Lagaay, David Lauer (eds.), Medientheorien. Eine phil-
35 Frieder Nake, The Semiotics Engine. Notes on the History of Algorithmic osophische Einführung, Frankfurt/M.: Campus, 2004, pp. 297–315; Ingrid
Images in Europe, in: Art Journal 68.1 (2009), pp. 76–89. Hoelzl, The Operative Image. An Approximation, http://mediacommons.
36 van den Boomen 2014 (as fn. 28), p. 36. futureofthebook.org/tne/pieces/operative-image-approximation (accessed
37 Cramer, Fuller 2008 (as fn. 20), p. 149. September 20, 2017).
38 See Distelmeyer 2017 (as fn. 25), pp. 92–98. 44 Farocki 2004a (as fn. 39), p. 61.
39 Harun Farocki, Quereinfluss/Weiche Montage, in: Christine Rüffert et. 45 Jan Distelmeyer, An/Leiten. Implikationen und Zwecke der Computer­
al. (eds.), Zeitsprünge. Wie Filme Geschichte(n) erzählen, Berlin: Bertz isierung, in: Navigationen. Zeitschrift für Medien und Kulturwissenschaften
+ Fischer, 2004a, pp. 57–61, p. 61; Harun Farocki, Phantom Images, in:­ 17.2 (2017), pp. 37–53.
Public. Art, Culture, Ideas 29 (2004b), pp. 12–22; http://harunfarocki.de/ 46 Farocki 2004a (as fn. 39), p. 61.
installations/2000s/2003/eye-machine-iii.html (accessed September 20, 47 See Sybille Krämer, Operative Bildlichkeit. Von der Grammatologie zu einer
2017). ‘Diagrammatologie’? Reflexionen über erkennendes Sehen, in: Martina
40 Farocki 2004b (as fn. 39), p. 17. Heßler, Dieter Mersch (eds.), Logik des Bildlichen. Zur Kritik der ikonischen 61
 Jan Distelmeyer

sentations of computer performance are parts and thresh- I would like to highlight just two aspects of the last type:
olds of (at least) four types of mutually connected opera- The first aspect is related to the special indexicality of these
tions – that is, interface operations within the meaning of operative images, which leads back to the question of how
the multilayered interface facets: analyzing graphical user interfaces could help address
the dicey character of computerization. Addressing this
1. Operations as the various interrelations between indexicality inevitably confronts us with consequences of
hardware and software ensuring that these gen- programmability, which I understand as perhaps the most
eral-purpose machines and universal symbolic thought-provoking characteristic of computers and com-
machines fulfill their tasks. puterized media, things and beings. Graphical user inter-
2. Operations as the interrelations of several computers, faces always propose ideas and depresentations of more
leading to further co-action of hardware and soft- than just the computer; instead, “[i]nterfaces and operating
ware by protocol-driven networks. systems produce ‘users’ – one and all.” 51 And since all of
3. Operations as the connections and communications our computer use has to be envisaged and enabled by pro-
between computers and non-computer forms of gramming, computer interfaces always empower users to
interconnected materiality – such as human bodies regulate while at the same time forcing them to be regu-
or technical artifacts – that lead to the issues of sur- lated. 52 Hence, the interface mise-en-scène – the available
veillance and cybernetization of beings and (an inter- structure of operative images and depresentations – shapes
net of ) things under programmed control. the aesthetic appearance of the computer as an aesthetics of
4. Operations as us dealing with them – operations as regulation (Ästhetik der Verfügung). 53
the handling of and dealing with computers, hence: This aesthetics is marked by a particular power struc-
operations understood as technical, physical, and ture – a logic of regulation: Actively regulating users are
cognitive processes, including questions of the rela- being regulated in a system, in which they have to play under
tionship between software and ideology raised by the default rules with the provided tools and prerequisites.
Wendy Chun,48 Alexander Galloway,49 and Cynthia
and Richard Selfe. 50
51 Chun 2013 (as fn. 48), pp. 67–68.
52 I would like to stress the point that the common distinction between
users and programmers is highly problematic – especially when it comes
to interfaces. As Wendy Chun has pointed out, “programmers are users”
Vernunft, Bielefeld: transcript, 2009, pp. 94–123; Sybille Krämer, Christi- since “they create programs using editors, which are themselves software
na Ljungberg (eds.), Thinking with Diagrams. The Semiotic Basis of Human programs”: “The distinction between programmers and users is gradually
Cognition, Boston/Berlin: Mouton de Gruyter, 2016. eroding, not only because users are becoming programmers (in a real sense
48 Wendy Hui Kyong Chun, Programmed Visions. Software and Memory, Cam- programmers no longer program a computer; they code), but also because,
bridge: MIT Press, 2013. with high-level languages, programmers are becoming more like simple
49 Alexander Galloway, The Interface Effect, Cambridge: MIT Press, 2012. users. The difference between users and programmers is an effect of soft-
50 Cynthia L. Selfe, Richard J. Selfe, The Politics of the Interface. Power and ware.” Wendy Hui Kyong Chun, On Software. Or the Persistence of Visual
Its Exercise in Electronic Contact Zones, in: National Council of Teachers Knowledge, in: Grey Room 18 (2004), pp. 26–51, p. 38.
62 of English 45.4 (1994), pp. 480–504. 53 See Distelmeyer 2017 (as fn. 25), pp. 65–126.
Carrying Computerization

But this is not a one-way street: Precisely because every [M]ore than anything, my optimism comes from the
computer operation relies on programs, all programmed empowering nature of being digital. The access, the
functions, regulations, barriers, and presets are principally mobility, and the ability to effect change are what will
alterable and expandable by users or hackers. This processu- make the future so different from the present. 57
ality identifies dealing with computers as a power struggle
with which its political issues may begin. It confronts us Understanding digitalicity as one important discursive
with controllability resulting from programmability. aspect of computerization and – not least – the hopes of
The second aspect of operations in terms of human and investments in the “fourth industrial revolution” as
handling of computers is related to knowledge, informing for instance shown in the European Commission’s “path to
our actions. Criticized by various media scholars, the myth- digitise European industry” 58 – the question arises, how a
ical term digital has been an extremely powerful buzzword given interface mise-en-scène corresponds to the promises
and sales argument at least since the early 1990s. 54 To mark and fears that have shaped digitalicity. With this question I
the digital as a myth and to keep in mind the problems of would like to turn to YouTube as an example.
coping with mythical terms as shown by Roland Barthes, I
arranged myself some years ago with another not yet myth-
YouTube Operations
ical term: the neologism digitalicity. 55
In Western-European and US-American discourse If you enter the URL www.youtube.com or follow a cor-
since the early 1990s digitalicity is shaped to a special responding link, bookmark, or presetting, the front page
degree by promises (and fears) of interactivity, flexibility, of YouTube presents a deployment of selectable operative
control, freedom, and empowerment. Celebrated as a victory images, depresenting potentially upcoming video events
of digital media’s acclaimed elasticity as opposed to rigid, (fig. 4). 59 Even if you have no personal account to log in, the
inflexible, passive, and hierarchy-based predecessors, the personalizing you of YouTube is taken seriously right from
same programmatic linkage between flexibility and control the start: Thanks to recorded, evaluated, and conjugated
is now – at the latest since the Snowden disclosures and the former visits and dealings with YouTube, every front page
debates about dominating corporations and algorithmic reg- should be a customized performance. This personalization
ulation – also an object of criticism. 56 As just one example I is the outcome or yield of my work within the YouTube
would like to quote maybe the most influential protagonist
of digitalicity from the 1990s, Nicolas Negroponte:

57 Nicolas Negroponte, Being Digital, New York: Alfred A. Knopf, 1995, p. 230.
54 See Hartmut Winkler, Docuverse. Zur Medientheorie der Computer, Munich: 58 European Commission Directorate General for Communications Networks,
Fink, 1997; Lev Manovich, The Language of New Media, Cambridge: MIT Content & Technology, The Fourth Industrial Revolution, https://ec.europa.
Press, 2001; Chun 2006 (as fn. 34). eu/digital-single-market/en/fourth-industrial-revolution (accessed Sep-
55 See Jan Distelmeyer, Das flexible Kino. Ästhetik und Dispositiv der DVD & tember 20, 2017).
Blu-ray, Berlin: Bertz + Fischer, 2012. 59 I am describing the YouTube interface performed by a browser; the inter-
56 See Distelmeyer 2017 (as fn. 25), pp. 98–126. face designed for the YouTube app is a different formation. 63
 Jan Distelmeyer

power of control related to a variety of depresented audiovi-

sual material classified by taglines, genres, categories, and
other visualized metadata. It keeps up the empowerment
gesture and the ability to effect change: Even though I have
already chosen a video, this choice is accompanied by a
selection of another to-be-selected material.
This choice-empowerment relies heavily on a mode of
presentation that dominated and still is dominating more
than a few interface formations. This tradition presents the
aesthetics of regulation as an “order of selectivity” 61 – offer-
ing options and reassuring usability as a freedom of choice
in the form of menus, buttons, lists, and the like. This “free-
dom as control” is a question of strictly defined and prepared
choices.62
We encounter this traditional and surprisingly
4  Lists on the front page of YouTube.
long-lasting WIMP (abbreviation for windows, icons, menus,
and pointer that denotes an interface design paradigm in
interface that Till A. Heilmann has described as data labor human-computer interaction) cosmos by, for instance, using
in current capture-capitalism.60 popular online shops like iTunes or Amazon, the grid-ap-
The moment I make my selection, the former depresent- position of apps on multi-touch devices like Google Nexus
ed video starts in a frame, in which the video is a working and Samsung Galaxy, iPhone and iPad, the “active apps” and
as an operative (moving) image in its own right. If I click “ideal apps” arrangement on the Fairphone 2, the Launchpad
on the running video, it pauses until another click on the of MAC OS Yosemite, Windows 10, and the Linux-Interface
now freezed operative image starts the movement and GNOME 3 with its “activities overview” described by the
sound again. A double-click leads to the full screen mode, GNOME Project as “an easy way to access all your basic tasks.
another double-click brings back the YouTube website A press of a button is all it takes to view your open windows,
interface. Here the expandable video frame is escorted by launch applications or check if you have new messages.” 63
another arrangement of selectable operative images to the
right of the frame. This arrangement of thumbnails could be
described as a remaining gesture of wealth and richness – a
61 Jan Distelmeyer, Freiheit als Auswahl. Zur Dialektik der Verfügung compu-
terbasierter Medien, in: Jan-Henrik Möller, Jörg Sternagel, Leonore Hipper
(eds.), Zur Paradoxalität des Medialen, Munich: Fink, 2013, pp. 69–90.
60 Till A. Heilmann, Datenarbeit im ‘Capture’-Kapitalismus. Zur Auswei- 62 Chun 2006 (as fn. 34).
tung der Verwertungszone im Zeitalter informatischer Überwachung, in: 63 GNOME Project, https://gnome.org/gnome-3/ (accessed September 20,
64 Zeitschrift für Medienwissenschaft 13 (2015), pp. 35–47. 2017).
Carrying Computerization

Considering the familiarity with this widespread free-

dom as prepared choice-control, other widespread aes-
thetics of regulation could easily be overlooked. Especially
computer games challenge and play with this dominant
overview order. Examples can be found in different sorts of
games, most obvious maybe and long-lasting in first-person
shooters like the popular Tom Clancy’s Rainbow Six Siege
(Ubisoft, 2015), in which the crucial point is not to know
but to explore what actually is offered and waiting around
the corner. Nevertheless this exploring mode of aesthetics
is quite often supplemented by another order of selectivity,
showing available weapons, equipment, maps, and the like.
Hence, an order of selectivity, invoking our wealth of
choice by menus and similar arrangements, is not in the
least determined by technology. Instead this order of selec-
5  YouTube, navigation with progress bar.
tivity is a cultural construction and just one, yet dominant
mode of aesthetics of regulation. It presents the computer
as an empowering decision devise and shapes YouTube to a obvious and radical programming that changed the look of
special degree (fig. 5). YouTube already shortly after it has been sold to Google at
The aforementioned flexibility of the video appearance the end of 2006. In the early days of YouTube, right after a
in the YouTube frame is increased by the offer to transform video had been played, it still filled the whole video frame
the running video appearance in terms of language, subti- with one somehow representative image, ready to start
tles and resolution, which can be adjusted using the opera- anew. Since 2007 a finished video is replaced by a thumbnail
tive image of a gearwheel on the bottom right of the video collection of selectable videos: a new grid order of choice in
frame. Furthermore since 2012 each YouTube video is pre- exactly the frame supposedly reserved for moving images
sented in a paradigmatic way: When the cursor moves the (fig. 6). This programmatic displacement becomes peculiar
progress bar, the video blurs and a collection of somehow picturesque when the video is watched in full screen mode.
representative single frames pop up as a preview, offering Regarding this familiar mise-en-scéne – this grid of selec-
a navigation aid through the whole video by means of this tivity – Geert Lovink’s résumé about YouTube from 2008
frame collection. could be loaded with a new intention: “We no longer watch
Hereby the video is not playing but displayed as an area, films or TV; we watch databases.” 64
as a visible set of not yet operative images. This YouTube
approach to the order of selectivity touches upon fundamen- 64 Geert Lovink, The Art of Watching Databases. Introduction to the Video
tal questions of moving images elucidated by an even more Vortex Reader, in: Geert Lovink, Sabine Niederer (eds.), Video Vortex Reader. 65
 Jan Distelmeyer

6  YouTube, grid collection of recommended videos. 7  YouTube, autoplay feature.

Instead of the video appearance – that is, the chosen is, the idea and promise of this reference – is referring to the
succession and process of moving images and sounds as a recorded viewing and search history. Because these opera-
syntagmatic gesture – now the exact opposite is taking over: tive images are therefore both depresenting and acting, i. e.
the invitation to select among replaceable images as a para- part of an agency and an agenda, these aesthetical questions
digmatic gesture, which consists of operative images. Thus are also and unavoidably political.
YouTube’s magic – the additional transformation of moving With this in mind, a displacement and respectively a
images into operative images – is demonstrated once more diversification of film/video aesthetics by aesthetics of
insistently. The programmatic indexicality of these images regulation could be witnessed here. The logic of the film-
moves to the front. ic syntagm gets involved in the paradigmatic logic of dig-
Bearing in mind the second type of interface operations, italicity and its performed freedom as choice-control. To
this programmatic indexicality of the collected videos is this, I would like to add, another potential relationship: the
based not only on the fact that they “all refer causally and connection of this exhibited flexibility, a crucial promise of
physically to a set of software instructions to be execut- digitalicity, with the sociocultural ideal and pressure of flex-
ed” but also because of the operative trace to the process- ibility in today’s formations of flexible and communicative
ing of recorded and algorithmically evaluated data labor, capitalism. Jodi Dean and Franco Berardi describe “a key
with which these appearances are causally and physically contradiction of communicative capitalism” – if you “want
linked.65The grid collection of recommended videos – that to survive you have to be competitive and if you want to

Responses to YouTube, Amsterdam: Institute of Network Cultures, 2008,

pp. 9–13, p. 9.
65 Marianne van den Boomen, Interfacing by Material Metaphors. How Your Material. Tracing New Media in Everyday Life and Technology, Amsterdam:
66 Mailbox May Fool You, in: Marianne van den Boomen et al. (eds.), Digital Amsterdam University Press, 2009, pp. 253–264, p. 257.
Carrying Computerization

be competitive you must be connected, receive and process as a cultural form of presenting data in web-based social
continuously an immense and growing mass of data”.66 network services, heightening “the experience of the ‘data
The preliminarity and replaceability of the selected present’”.71 Another form of flexibility is performed here –
video can be interpreted as visualization and maybe famil- an ongoing flow of change that seems to be no longer under
iarization of what Dean calls “the competitive intensity of our (prepared and advised) control but that is controlled
neoliberal capitalism”.67 This aesthetic fate of chosen videos by information processing like a showcase for “algorithmic
may be understood as a reminder of the competitive pres- governmentality”.72
sure, analyzed by Boltanksi and Chiapello,68 and as an echo
of Gilles Deleuze’s Societies of Control69: Even these or them,
Monitoring
which may have been chosen once among the many, always
have to face a new competition right after the very selection. This deserves a closer look and further steps. My remarks
Ongoing flexibility and changeability is to learn and to rely here are intended as starting points for an analysis that
on (fig. 7). takes account of the complex procedures enabling and pur-
I would like to conclude with the observation that even suing the options of uploading, searching, watching/hear-
this well-established paradigmatic logic of YouTube is sub- ing, classifying, valuing, and exposing data in the form of
jected to changes. The installation of the “autoplay” mode, videos, requests, comments, clicks, and all sorts of metadata.
switched on by default since 2015, forms a counterpart to In the end all the options depend on processes that ask for a
the order of selectivity: “The Autoplay feature on YouTube new attention for intertwined interface operations.
makes it easier to decide what to watch next. After you How I operate with the YouTube interface is wedded
watch a YouTube video, we’ll automatically play another to other interface operations by which a request for a video
related video based on your viewing history.” 70 Thereby You- finds its way from, for instance, my processing smartphone
Tube creates a new emphasis of flow that can be discussed to the responding server located in one of the European
from various perspectives: for instance, both in terms of Google server farms in Dublin, Eemshaven, Hamina, or St
YouTube’s acclaimed reputation as the new television and Ghislain, relying on what Florian Sprenger has discussed
in terms of the “data stream”, estimated by Lev Manovich as “politics of micro-decisions” 73 and producing traffic that
is recorded both to customize my next visit on YouTube
66 Jodi Dean, The Limits of Communication, in: Guernica, http://­g uernicamag. (and other informed websites) as well as to profit from my
com/features/the-limits-of-communication/ (accessed September 20,
2017).
67 Ibid. 71 Lev Manovich, Data Stream, Database, Timeline, in: Software Studies Initia-
68 See Luc Boltanski, Ève Chiapello, The New Spirit of Capitalism, New York: tive, http://lab.softwarestudies.com/2012/10/data-stream-database-time-
Verso, 2007. line-new.html (accessed September 20, 2017).
69 See Gilles Deleuze, Postscript on the Societies of Control, in: October 59 72 Antoinette Rouvroy, Bernard Stiegler, The Digital Regime of Truth. From
(1992), pp. 3–7. the Algorithmic Governmentality to a New Rule of Law, in: La Deleuziana.
70 YouTube, Autoplay videos, https://support.google.com/youtube/ Online Journal of Philosophy 3 (2016), pp. 6–27.
answer/6327615?co=GENIE.Platform%3DAndroid&hl=en (accessed Sep- 73 Florian Sprenger, Politik der Mikroentscheidungen. Edward Snowden, Netz-
tember 20, 2017). neutralität und die Architekturen des Internets, Lüneburg, 2015. 67
 Jan Distelmeyer

ascending browsing record by customized advertising.74 Figures

These constructed, programmed, instructed, and performed 1  Cover of Interactions 23/6 (2016).
interface operations built the processual character of You- 2  Deutsche Telekom, Telekom Europe TV commercial with Andrea Bocelli,
Tube videos that Yuk Hui has described as “digital objects”, https://youtube.com/watch?v=eZu6oWfrieo (accessed November 20,
2017).
focusing “on data and metadata, which embody the objects
3  ABC News, Trump Presidential Inauguration 2017 (FULL EVENT) | ABC
with which we are interacting, and with which machines News, https://youtu.be/Nieiu8tmLIM?t=2h13m31s (accessed November
are simultaneously operating”.75 Interface operations char- 20, 2017).
acterize them as “new”, that is “dynamic and energetic”, 4  YouTube, https://youtube.com (accessed November 20, 2017).
forms of “industrial objects”.76 5  ABC News, Trump Presidential Inauguration 2017 (FULL EVENT) | ABC
News, https://youtu.be/Nieiu8tmLIM?t=2h24m30s (accessed November
Precisely because the question raised by Interactions –
20, 2017).
what computers are doing when we are not consciously
6–7  ABC News, Trump Presidential Inauguration 2017 (FULL EVENT) | ABC
interacting with them – is of prime importance, the com- News, https://youtu.be/Nieiu8tmLIM?t=8h33m4s (accessed November 20,
plex of human and automatized interface operations needs 2017).

to be explored. Interface operations include humans in front

of monitors as well as, for instance, sensor-based Ambient
Intelligence monitoring human or any other activities. Inter-
face analyses should therefore be interested in aesthetics
and a specific interface mise-en-scène as well as in the het-
erogeneous mesh of conditions enabling and determining it
together with the practices of use and understanding. Their
relevance is increasing the more interfaces and operations
between beings, things, and computers are built and relied
on – and the more these operations are planned and mediat-
ed as indispensable and invisible, heading towards new forms
of depresentation.

74 In the case of YouTube interface analyses overlap with “platform studies”

insofar as “platform” is understood as “a broad enough category to capture
a number of distinct phenomena, such as social networking, the shift from
desktop to tablet computing, smart phone and ‘app’-based interfaces as
well as the increasing dominance of centralised cloud-based computing”.
Joss Hands, Introduction. Politics, Power and ’Platformativity’, in: Culture
Machine 14 (2013), pp. 1–9, p. 1.
75 Yuk Hui, On the Existence of Digital Objects, Minneapolis, 2016, p. 48.
68 76 Ibid., pp. 49–57.
Luisa Feiersinger

Spatial Narration
F  ilm Scenography Using Stereoscopic Technology

Situating the Screen in Stereoscopic Practice am Ross, in her recent discussion on stereoscopic visuality,
actually begins her analysis by asking whether the screen
If the screen is a plane on which images are displayed in is even still present. 3 She reaches the conclusion that the
order to view them, then what precisely happens to the screen dissolves into a “field screen,” thereby facilitating
screen in those films that are presented under the catch- a “fundamentally different viewing experience.” 4 That a
phrase 3-D?1 The visual impression created by such stereo- change in the filmophanic space5 occurs is indisputable,
scopic films is based on a technical imitation of binocular but if one wishes to situate the screen in visual practice, as
vision, which allows for the perception of three-dimension- this volume suggests, then the bigger picture – so to speak –
ality. Through an intricate arrangement of projectors, filter must be considered. As such, the stereoscopic image space,
foils, specially coated screens, and distinctive eyeglasses, having dispensed with planar limitations, can only be gen-
two film tracks – each taken from a slightly different per- erated through particular filming techniques and mise-en-
spective – are simultaneously delivered to the eyes of the scène strategies. Fully in the tradition of narrative cinema,
spectator. The viewer is able to perceive a visual space
that is not merely restricted to the plane of the screen, but Bazin, Was ist Film?, Berlin: Alexander Verlag [1946] 2009. Although their
basic approaches are almost diametrically opposed, both film theoreticians
expands in front and behind it. This negation of the screen,
ascribe an “immediacy of appearance” and thus a direct influence on the
in which the abstraction of two-dimensionality appears viewer to the three-dimensional film. Arnheim 2002, p. 266; Bazin 2009,
to dissolve, has always been been understood within film p. 47.
Miriam Ross, Stereoscopic Visuality. Where is the Screen, where is the
3
theory as a means of achieving greater immediacy.2 Miri- Film?, in: Convergence. The International Journal of Research into New
Media Technologies, 19.4 (2013), pp. 406–414, p. 406.
This essay was first published in 2015, and has been translated and revised
1 Ibid., p. 413. It must be mentioned, that she also discusses the aesthetic
4
for this volume. I thank Deborah J. Curtis and Julia Sittmann. For the orig- changes in stereoscopic film based on its technical conditions.
inal text see Luisa Feiersinger, Räumliches Erzählen. Filmszenographie This text draws on the Vocabulary of Filmology used by Etienne Souriau
5
in stereoskopischer Technik, in: Annette Dorgerloh, Marcus Becker (eds.), to make a distinction between the reality that is independent of the film
Alles nur Kulisse?! Filmräume aus der Traumfabrik Babelsberg, Weimar: (“afilmic reality”), the reality that pertains to the film (“profilmic reality”),
VDG, 2015, pp. 140–145. and the narrated world (“diegesis”), as well as to separate the processes and
A comprehensive introduction to the historical background is beyond the
2 characteristics of film projection (“filmophanic reality”) from those of the
scope of this essay and, therefore, reference is only made here pars pro toto film material (“filmographic reality”). Etienne Souriau, Die Struktur des
to Arnheim, Film als Kunst, Frankfurt/M.: Suhrkamp, [1932] 2002, and to filmischen Universums und das Vokabular der Filmologie, in: montage/av,
André Bazin, Der Mythos vom totalen Film, in: Robert Fischer (ed.), André 6.2 (1997), pp. 140–157.
 Luisa Feiersinger

the mediation itself (the complex technological and cine- in stereoscopic film spaces, three vital questions must be
matographic arrangement that produces the image) must addressed: firstly, how does this diegetic space interact with
be rendered invisible to the viewer.6 the boundaries of its images, both on the plane of the screen,
The ostensibly negated screen is made expressly mani- and within the projection that extends beyond it? Secondly,
fest in the practices and techniques involved in producing how is the profilmic space prepared and translated for the
the immediacy of this perception, and stereoscopic films – shot, using cinematographic techniques? Finally, how has
just as any other screen-based images – are impossible to the understanding of these cinematographic practices and
conceive of outside their production and reception possibil- the existing technical requirements favoured the formula-
ities. An investigation of the film-image in films produced tion of specific narrative structures and motifs? In the fol-
since 2009 must be based on its interconnection with its lowing, Alfred Hitchcock’s film, Dial M for Murder (1954),9
production techniques – not because the visual space creat- produced using analog technology, and The Three Muske-
ed through stereoscopic technology is new,7 but because the teers, an exemplar of digital stereoscopic films, directed by
combination of stereoscopic alignments with digital record- Paul W. S. Anderson (2011),10 will be compared in order to
ing, processing and playback techniques is. Although the illuminate these issues.
shift from analog to digital techniques may not have been
apparent to the untrained eye, this transition was essen-
Translating Diegesis into Stereoscopic Film
tial to the development of the aesthetic qualities inherent
Space
to contemporary stereoscopic visual imagery.8 In order to
investigate how narration can take place through and with- In The Three Musketeers, Alexandre Dumas’ well-known
story is re-packaged as a action movie spectacle, meant to
This structure of technical images is known as the “principle of disjunction”
6
in the discipline of Bildgeschichte, Horst Bredekamp, Angela Fischel, Bir-
satisfy modern sensibilities: With the help of inordinate
git Schneider, Gabriele Werner, Bildwelten des Wissens, in: Bildwelten des amounts of weaponry, the three musketeers and the young
Wissens. Kunsthistorisches Jahrbuch für Bildkritik, 1.1: Bilder in Prozessen
D’Artagnan foil a conspiracy by Cardinal Richelieu to rob
(2003), pp. 9–20. Not by name but in principle, it was established for the
two-dimensional film by Bordwell, Thompson and Staiger in their seminal the inexperienced King Louis XIII of the throne. Central
study on the classical Hollywood cinema. David Bordwell, Janet Staiger, to the plot is a necklace belonging to the Queen, which –
Kristin Thompson, The Classical Hollywood Cinema. Film Style and Mode
of Production to 1960, London: Routledge, 1994. in the wrong hands – could trigger war between England
For in-depth information on the history of stereoscopic films and their
7 and France. This diegetic 17th Century France is located
occurrence in waves, see Ray Zone, Stereoscopic Cinema & the Origins of
in the profilmic spaces of Bavarian castles and Babelsberg
3-D Film, 1838 –1952, Lexington: University Press of Kentucky, 2007; Ray
Zone, 3-D Revolution. the History of Modern Stereoscopic Cinema, Lexing- green-screens, where a new world was created, which ful-
ton: University Press of Kentucky, 2012. filled the visual requirements of stereoscopic films, while
Thomas Elsaesser highlighted this link in an essay on the re-establishment
8
of stereoscopic films since 2009, emphasising the benefits of the visibly
different stereoscopic film image for the dissemination of digital techno­ 9 Alfred Hitchcock, Dial M for Murder, USA 1954, in: 3-D Blu-ray, Warner
logy. Thomas Elsaesser, The ‘Return’ of 3-D. On Some of the Logics and Bros. Entertainment Inc. 2012, 105 Min.
Genea­logies of the Image in the Twenty-First Century, in: Critical Inquiry 10 Paul W. S. Anderson, The three Musketeers, Germany/France/UK/USA 2011,
70 39 (Winter 2013), pp. 217–246, pp. 221–225. in: 3-D Blu-ray, Constantin Film 2011, 111 Min.
Spatial Narration

simultaneously serving as a visual subtext for the plot and

its characters. Richelieu’s room and its furnishings, for
example, convey his tactical cleverness: Our first encoun-
ter with the Cardinal is cinematically staged over a chess-
board, as the camera pans upwards from a close-up of the
chess pieces towards a medium close-up shot of his face.11 In
this sequence, the game of chess and the Cardinal’s face are
arranged not only on a vertical axis, but also positioned sep-
arately in the depth of space produced by the stereo­scopic
film. The elements of the image are distributed visually in
this stereoscopically-created space (depending on where
the optical axes of the two image tracks intersect) either in
front or behind the plane of the screen, the so-called zero
parallax.12 While the chess pieces are in front of this plane –
1  Semantics of space in The Three Musketeers (2011), screenshot, TC: 00.22.21.
referred to as negative parallax – the Cardinal’s face is in
positive parallax, namely behind it. The game of chess is
thus spatially accentuated, through its position directly in shot, while the room spreads out in positive parallax in the
front of the eyes of the audience. That the game serves as background around them.
an allegory for the Cardinal’s political manoeuvres – which For attentive film audiences, this specific mise-en-scène
he plans like chess moves – becomes abundantly clear as of objects in the foreground, actors in the middle ground,
the scene continues, and Richelieu reveals to his interloc- and a room in the background will already be familiar from
utrice, Mylady, that he only ever plays against himself – no shots in numerous stereoscopic films, including Hitchcock’s
other suitable challenger exists.13 Standing behind the table Dial M for Murder.14 In this 1954 film, a husband attempts
with the chessboard, the protagonists are shown in a two to have his unfaithful wife murdered in his absence. Even
shot – wherein the frame encompasses a view of two peo- though – or perhaps precisely because – the murder attempt
ple (fig. 1). Once again positioned in slight negative parallax, fails, the husband is found out by dint of a key, crucial to
the chess game continues to occupy the front of the image
14 The film was produced using stereoscopic technology, but has generally
space, framed symmetrically between two candlesticks and
been listed as a 2-D film due to the rapid decline of the 3-D boom in the
two small ornate cases. The two individuals dominate the 1950’s. For a history on the screening of the film, see R. M. Hayes, 3-D
­Movies. A History and Filmography of Stereoscopic Cinema, Jefferson/Lon-
don: McFarland & Co 1989, pp. 171–173 and Zone 2012 (as fn. 7), pp. 35, p. 42.
11 Ibid., TC: 00.21.52–00.22.02. David Bordwell discusses this particular mise-en-scène in the entry Dial M
12 For an in-depth description of the stereoscopic production of space and on for Murder: Hitchcock frets not as his narrow room on his blog David Bord-
the associated terminology, see the contribution by Shannon Benna and its well’s website on cinama, David Bordwell, Kristen Thompson, Observations
glossary in this volume, pp. 133-145. on film art, http://davidbordwell.net/blog/2012/09/07/­dial-m-for-murder-
13 Anderson 2011 (as fn. 10), TC: 00.22.02–00.23.04. hitchcock-frets-not-at-his-narrow-room/ (accessed May 12, 2015). 71
 Luisa Feiersinger

his ingenious plan, that ultimately betrays him. Almost and can be implemented more rapidly by digital means. As
all of the scenes in Dial M for Murder are filmed using the such, continuous minimal adjustments become feasible, per-
image composition mentioned above, with only a few crucial mitting – for instance – for the space to be flattened to spare
elements jutting out into the movie theatre in strong neg- the eyes of the viewer in a scene with rapid cutting. In The
ative parallax: first the wife’s hand, which she desperately Three Musketeers, these adjustments, the exaggeration and
stretches towards the viewers at the moment of her attempt- the flattening of the visual space, can be observed in the
ed murder, and then the key, which the police inspector sequences where the three musketeers encounter Rochefort,
displays in an equally dramatic fashion.15 These narrative chief of the Cardinal’s guardsmen, on airships.16
moments are foregrounded – quite literally – as exceptions Elements in the film that move towards the audience
to Hitchcock’s stereoscopic formula. Similarly, in The Three must be handled with the same care, as they entail an intrin-
Musketeers, the Queen’s necklace, in addition to the chess- sic contradiction: They are both expected to appear in a
board, also often appears in the visual foreground, thereby stereoscopic film, but when they do, are often condemned
marking its narrative importance in the film. as both cheap gimmickry and hard on the eyes.17 In addi-
However, spatiality is deployed at other levels as well. tion, they harbour the danger of destroying the illusion of
While the depth of space is relatively flat in dialogue scenes physicality produced in stereoscopic films, and thus laying
(such as the one previously mentioned between Richelieu bare the technical sleight of hand that brought them into
and Mylady), it is extended in more dramatic moments, as existence. The visual elements in negative parallax prac-
the so-called depth budget is enlarged. As the inter-axial dis- tically force themselves onto the viewer. But were they to
tance (the space between the cameras recording the imag- follow their natural impulse to test the image’s physicality,
es) is increased, the physical expansion of the image ele- the viewer would reach into nothingness, reinforcing the
ments is heightened. The stereoscopically produced space
is not dependent on the expansion of the actual space being
16 Anderson 2011 (as fn. 10), TC: 01.25.41–01.29.19. While the space is flattened
recorded by the cameras, but on specific cinematographic
in the battle sequences it is exaggerated in the sequences opening up the
strategies and conditions. This fact applies equally to analog view into the landscape. Glen MacPherson, who worked on this film as
and digital cinematography, although greater control can be a camera man, as well as on numerous other projects by Anderson, con-
firms these techniques for another joint 3-D project in the interview with
exerted over digital shots, since they can both be assessed R. Emmet Sweeney. R. Emmet Sweeney, Interview: Glen MacPherson, 3D
on the spot during filming, and corrected later in the pro- DP, http://filmcomment.com/entry/interview-glenn-macpherson-3d-dp-
resident-evil (accessed January 23, 2015).
duction process. In addition, the necessary manipulations –
17 The critics’ response to the film was mixed, mostly highlighting the
equally possible in analog films – appear easier to achieve excessive use of visual effects in a flat literary adaptation. For one exam-
ple, see Mark Feeny, The Three Musketeers Movie Review, in: The Bos-
ton Globe October 22, 2011, http://archive.boston.com/ae/movies/arti-
15 Hitchcock 1954 (as fn. 9), TC: 00.44.04 and TC: 01.39.29. The film director cles/2011/10/22/three_musketeers_when_swords_meet_cgi/ (accessed
confirms the positioning of these image elements in his interview with March 23, 2018). Elsaesser highlighted this type of criticism as a general
François Truffaut, although he has little praise for his only 3D project. trend in the discussion of 3-D in his essay on the genealogy of stereoscopic
François Truffaut, Mr. Hitchcock, wie haben sie das gemacht?, München: films and pointed out the contradictory demands placed on them. Elsaesser
72 Heyne, 2003, pp. 207–210, p. 208. 2013 (as fn. 8), p. 237.
Spatial Narration

images’ lack of corporality.18 Furthermore, even if the audi- space at these points does not extend forwards or backwards,
ence accepts the optical illusion as is, it is precisely these as is characteristic for stereoscopic technology, but moves
forward-moving elements that can produce perceptual to and over the sides. While the expansion of the diegetic
conflicts, through their positioning in the visual space rela- space over the side boundaries of the visual space is unprob-
tive to the screen’s boundaries. If, due to negative parallax, lematic in two-dimensional films,20 the frame appears more
an object appears to be placed in front of the screen, but is fundamentally to be recognized as a border in stereoscopic
simultaneously intersected by the framing of the film, then films: In Dial M for Murder, table lamps, which are placed
this results in competing and contrasting depth referenc- at the front of the image space often produce such a con-
es, since such an overlap normally indicates, by convention, flict. Specifically in the longer takes, the viewer perceives
that the object is positioned in the background.19 The visual the intersection between objects in negative parallax and
the frame as breaking the illusion of corporality that ste-
18 In the essay on stereoscopic visuality by Miriam Ross, already mentioned reoscopic films try to convey.21 Even if these lamps are only
above, the author focuses, in particular, on image elements presented in slightly in front of zero parallax, they are visually irritating,
negative parallax. The potential of the stereoscopic film to dissolve its illu-
sion of reality would be concentrated in these elements. She therefore refers since they exceed the full height of the image. Even when
to these elements as destabilising the screen and its illusion, Ross 2013 (as the objects do not produce any perceptual conflicts, their
fn. 3), p. 409. They simultaneously expand the sensory potential of the ste-
positioning in the foreground often distracts from the main
reoscopic film in its own fashion, as she demonstrates in reference to the
discourse on the haptic film. Jennifer Barker, The Tactile Eye. Touch and the action, which is in part covered up by them.22 Image com-
Cinematic Experience, Berkeley, CA: University of California Press, 2009; position and framing must therefore be re-conceived and
Guiliana Bruno, Atlas of Emotion. Journeys in Art, Architecture and Film,
New York: Vers, 2002; Laura Marks, The Skin of the Film, Durham: Duke re-learned for stereoscopic filming. The placement of these
University Press, 2000; Anne Rutherford, Cinema and Embodied Affect,
in: Senses of Cinema 25 (March 2003), http://sensesofcinema.com/2003/ Zone 2012 (as fn. 7), pp. 268–269. This frame, which is incorporated, but
feature-articles/embodied_affect/ (accessed January 23, 2018); Steven not perceived as such, just as is demanded by the tradition of narrative
Shaviro, The Cinematic Body, Minneapolis: University of Minnesota Press, cinema, is used much more frequently in digital cinema, predominantly
1993; Vivian Sobchack, The Address of the Eye. A Phenomenology of Film in individual shots, mainly thanks to the simplicity of the production of
Experience, Princeton: Princeton University Press, 1992; Vivian Sobchack, these stereo windows with digital techniques. Once again, see Benna 2018
Carnal Thoughts. Embodiment and Moving Image Culture, Berkeley: Uni- (as fn. 12), pp. 135-136.
versity of California Press, 2004; Christiane Voss, Film experience and 20 Instead of referring to the numerous publications that discuss the onscreen-
the formation of illusion. The spectator as ‘surrogate body’ for the cinema, offscreen relationship from specific points of view, it should be emphasised
in: Cinema Journal 50.4 (2011), pp. 136–150.; Ross 2013 (as fn. 3), p. 412. In here that David Bordwell and Kristin Thompson name framing, i. e. the
parallel with this argument on the disruptive potential of image elements relationship between what is depicted and its frame, as a central category
in negative parallax, I have interpreted these elsewhere as revenants of for analysis in their seminal book on the analysis of films. David Bordwell,
philosophical toys. Luisa Feiersinger, Berührung im stereoskopischen Kirstin Thomson, Narrative as a Formal System, in: David Bordwell, Kirs-
Film. Über das Ergreifen und Ergriffenwerden von optischen Illusionen, tin Thomson, Film-Art. An Introduction, New York: MacGraw-Hill, 2010,
in: Steffen Haug, Thomas Helbig, Tina Zürn (eds.), „Don’t touch! Touch pp. 186–212.
screen!“ Das Bild, der Blick und allerhand Formen taktiler Wahrnehmung und 21 Barbara Flückiger provides a clear discussion on this dissolution of the
Erkenntnis. Eine Tagung für Michael Diers, Munich: Fink, in preparation. physicality of objects when they are intersected by the margin. Barbara
19 Raymond and Nigel Spottiswoode were already working on this problem in Flückiger, Aesthetics of Stereoscopic Cinema, in: Projections 6.1 (2012),
the 1950’s. They therefore propose a stereo window that, printed around the pp. 101–122, pp. 116–117.
film image in the form of a black frame, also floats in space visually as an 22 This happens very often troughout the entire film, for one examplary
image element and thus eliminates the irritations produced by the overlap, instance, see Hitchcock 1954 (as fn. 9), TC: 00.15.03. 73
 Luisa Feiersinger

visual elements in Dial M for Murder raises the question as The specific constraints that shots with effects in nega-
to their function, whereby the likelihood is high that their tive parallax must adhere to, in order to avoid irritating the
purpose was simply to stagger the depth of space, but that viewer, are also liable to influence the narratives of stereo-
the chosen lamps were simply too large for the task.23 scopic films. The historically inconsistent re-imagination
In the later film, The Three Musketeers, greater atten- of Dumas’ The Three Musketeers to include airships is likely
tion was paid to the relationship between the larger objects a consequence of those conventions, insofar as flying ele-
shown in negative parallax and the frame. They are never ments are particularly well suited to the medium. Setting
truncated by more than one visual edge, and especially not the action at height, with the protagonists hovering in the
by the upper one, and then, only briefly. In addition, the air, facilitates not only the emergence of image elements in
viewer can observe a greater focus on the main action negative parallax, without the danger of encroaching on
within the general composition of the scene. There is also a the image frame, but also the subliminal introduction of
clear attempt to better integrate elements already in strong the motif of falling. A common theme in stereoscopic films,
negative parallax into the image as a whole, while simul- falling, with its ability to depict spectacular views into the
taneously maintaining the invisibility of the techniques depths below, produces a potent vacuum-effect which pulls
used in the medial transmission. In concrete terms, this at the viewer, and is, as such, a favoured cause of death in
means that an effort was made to ensure that elements The Three Musketeers, despite the plethora of actual weap-
entering into the viewer’s space respected the frame of ons available. The final battle between the adversaries
the screen. In one sequence, which follows the flight of a Rochefort and D’Artagnan takes advantage of precisely this
cannonball, the projectile is staged in a complex manner danger, impressively displayed through stereoscopic tech-
within the depth of space:24 the warhead initially hurtles niques.25 The duel on the gables of Notre Dame Cathedral
straight towards the viewer, crossing through the entire in Paris, with its steeply pitched roof, opens up numerous
stereoscopically created space, from positive into nega- opportunities for shots from above, looking down into the
tive parallax. But, before the cannonball reaches the view- depths below. Rochefort ultimately falls into the abyss –
er, the camera rotates around it, subsequently following effectively staged in positive parallax, emphasizing the
it in slow motion from the side, as it now, all of a sudden, dramatic nature of the location and his death.
floats in negative parallax in front of the audience, tan-
talizingly within reach. Safely out of range of any over-
Means of Constructing Stereoscopic Space
laps, its physicality appears beyond question. The camera
then pans around again, this time behind the cannon ball, It goes without saying that the risk to the actors in this scene
and follows its flight, back at normal speed, until impact. was minimal, since the gables were located no more than
half a metre above the ground on soft mats in a film-studio
23 On the production of space in Dial M for Murder, see also Jesco Jockenhövel, in Babelsberg (fig. 2a,b). The musketeers’ airship also flew in
Der digitale 3D-Film. Narration, Stereoskopie, Filmstil, Wiesbaden: Springer,
2014, pp. 60–64.
74 24 Anderson 2011 (as fn. 10), TC: 01.26.44–01.26.48. 25 Anderson 2011 (as fn. 10), TC: 01.32.31–01.36.00.
Spatial Narration

front of the green screen there, negating the need to recon- ability to control the outcome. With analog techniques, the
struct Notre Dame in Babelsberg, which thus only existed success or failure of the artificial creation of space can only
in virtual space (fig. 3a,b), 26 similarly to all elements lend- be assessed once the celluloid has been developed. Digital-
ing structure to the aerial space. What Hitchcock attempt- ly manipulated space can be checked on the control screen
ed with the help of table lamps, can now be accomplished during production and sometimes even instantaneously
in digital films with much smaller elements. Water vapour on-set. If the visual spaces are entirely digitally generat-
and clouds, for instance, demarcate the spatial expansion in ed, 30 control over the image is extended even further: Every
numerous scenes in The Three Musketeers. This “stereoscop- aspect of the various components of the simulated image
ic debris” is so easy to produce with CGI that it has become can be controlled and arranged. 31 In a stereoscopic set-up,
seemingly ubiquitous in recent films.27 Independent of the the ‘cameras’ are essentially viewpoints onto intricately cal-
fact that digital techniques have resulted in a simplification, culated generated worlds: their alignment, as well as their
and thus in an increase, of these types of cinematographic various stereo-parameters, can be perfectly synchronized.
manipulations,28 film space has nonetheless always been a Light reflections, for example, that present themselves dif-
synthetic space. Right from the beginning, film space was ferently to analog cameras taking the shot from different
untethered from physical reality, as shots of small-scale positions, can result in contradictory images, which dissolve
models (standing in for larger cityscapes) or even black and the spatial effect. 32 In CGI, they are introduced individual-
white, and deep focus shots, manipulated our visual per- ly, and as such become easily manageable. This element of
ception.29 control in digital film space facilitates its use in both two-di-
Within stereoscopic film techniques, the crucial differ- mensional and stereoscopic films. Whereas the creation of
ence between analog and digital manipulations remains the space was possible with analog techniques, implementing it
with two image tracks was far more difficult. The construc-
26 With reference to work on the virtual spaces, see the interview with Eric tion of artificial spaces was more noticeable in shots taken
Robinson, the head of the VFX team in The Three Musketeers. Vincent Frei,
with two instead of one camera. Due to their planar nature
The three Musketeers: Eric Robinson – Digital Effects Supervisor – Mr. X.,
http://artofvfx.com/?p=1713 (accessed May 12, 2015). matte paintings, used to introduce foreign environments
27 Ross calls these elements “stereoscopic debris”. It is precisely this debris into the backdrop, just like rear projections, provided the
that is capable of producing the “thick, tactile field screen” that is typical
for stereoscopic films in the 21st Century and she attributes a prominent cameras filming them from different perspectives with no
role to it in the construction of a “field screen”. Nonetheless she notes, that
these elements are not limited to the current productions techniques, but 30 Nowadays, the default construction of digital worlds is that of 3D anima-
the simplicity with which they can be controlled, made it easier to integrate tions. These are characterized by their volumetric figures in spatial settings.
them. Ross 2013 (as in fn. 3), pp. 409–410. They do not, however have an intrinsic connection to 3-D projection.
28 On the construction of these worlds and on their persuasive powers through 31 Prince 1996 (as fn. 28), as well as the discussions of the possibilities of the
the simulation of photographic appearance, see Stephen Prince, True Lies. camera in digital film. Jessica Aldred analyzed these considering their
Perceptual Realism, Digital Images and Film Theory, in: Film Quarterly effects on the viewer and their immersion into the film. Jessica Aldred, All
49.3 (1996), pp. 27–37. Aboard The Polar Express. A ‘Playful’ Change of Address in the Comput-
29 An overview on the advanced production of artificial worlds using analogue er-Generated Blockbuster, in: animation: an interdisciplinary journal 1.2
techniques is provided by Thomas G. Smith, Industrial Light & Magic. The (2006), pp. 153–172.
Art of Special Effects, New York: Ballantine Books, 1986. 32 Flückiger 2012 (as fn. 21), pp. 106–107. 75
 Luisa Feiersinger

2a,b  Duel on the roofs of Notre Dame in The Three Musketeers (2011), working photography before and after insertion of the digital background.

differentiating information. As can be observed, for example, to film a close-up, 35 today, digital camera can film such a
in Creature from the Black Lagoon (1954), they thus appear scene normally, as cameras have shrunk, permitting a short-
strangely flat in stereoscopic set-ups. 33 It is possible that for er inter-axial distance.
this very reason, Hitchcock decided to stick to a chamber
play in his stereoscopic film, thereby avoiding the tech-
Opening the Window
niques he otherwise favoured for the incorporation of any
external environment into his films. 34 The production pos- Overall, digital stereoscopic film is subject to fewer tech-
sibilities thus effect the options available for the setting. In nical limitations than its analog predecessor, permitting a
addition to the construction of virtual worlds and the possi- greater measure of control over profilmic objects and cin-
bilities inherent to post-production, digital filming devices ematographic staging. The wide scope for manipulation,
produce a liberty within their scenographic circumstanc- inherent to worlds that are stereoscopically reproduced,
es that stereoscopic films did not previously have. While allows spatial constructions to communicate the narrative.
Hitchcock still had to build a gigantic model of a telephone Despite this ostensible freedom for film producers, certain
conventions have established themselves, such as reducing
33 Jack Arnold, Creature from the Black Lagoon, USA 1954. the effects of negative parallax and the flattening of space
34 The entire film, apart from two short scenes outside, takes place in two during rapid cut-sequences, for instance. Some of these stan-
rooms in the couple’s apartment. Hitchcock states that the play the film was
based on was pivotal to this decision, see Truffaut 2003 (as fn. 15), pp. 208–
209. Jockenhövel also highlights that it is precisely the selection of a piece
that requires no outside space, which Hitchcock preferred to incorporate 35 David Bordwell shows this in his already mentioned in-depth analysis of
through matte paintings and rear projections, can be viewed as associated the stereoscopic variant of the film that he published on his blog Bordwell,
76 with the stereoscopic techniques. Jockenhövel 2014 (as fn. 23), p. 64. Thompson 2012 (as fn. 14).
Spatial Narration

3a,b  Airship above Notre Dame in The Three Musketeers (2011), CAD working photography in the raw and fully rendered version.

dards are also directly incorporated into film plots, which to a single point, in order to convey an illusionistic space. In
are adjusted to create stereoscopically suitable scenes and its two-dimensional alignment, film – even though it sets its
motifs. Above all, the mise-en-scène conventions outlined images in motion – continues in this tradition, to a certain
here serve the purpose of imitating natural perception: The degree.
technical mediation (between cinematographic manipula- The title sequence of The Three Musketeers, invokes this
tions and the viewer) remains as invisible as possible36 – a concept only to leave it behind. 38 The opening credits start
principle in line with a tradition that reaches as far back as with a view of a map that initially appears flat, positioned
Renaissance painting, with its emphasis on a central picto- slightly behind the plane of the screen (fig. 4). The map
rial perspective. 37 This classic art historical concept of the is framed by a dagger, a revolver and some coins, all ele-
picture as an open window binds the viewer and the image ments reminiscent of trompe l’œil paintings. Such paintings,
­usually depicting flatware, present themselves as illusion-
36 Benna points out the natural depth method, developed by stereoscopic istic expansions of the viewer’s space. Precisely this space
filmmakers Alan & Josephine Derobe, that mimics human binocular vision. is then burst open when the camera moves first towards
Benna 2018 (as fn. 12), p. 142.
and then into the map. Its flatness dissolves into different
37 In the 15th Century, the image in central perspective was described as an
open window by Leon Battista Alberti in his treatise on the art of painting. planes within the stereoscopic depth of space: the grid that
Leon Battista Alberti, On Painting, De Pictura, New Haven: Yale University delineates the map is revealed, floating in front of the map,
Press, 1966. This idea of immediacy has been discussed so comprehensively
as a metaphor in the discourses on the history of art and imagery, at least not dissimilar to the gridlines used in image composition.
since Panofsky’s Perspective as Symbolic Form (1927), that even an illustra- Moving through the transparent grid, and thus, quite lit-
tion of the central positions alone cannot be given here. In its place, refer-
ence is therefore made to the discussion of these metaphors specifically in
relation to the film. Sobchack 1992 (as. fn 18), pp. 14–25. 38 Anderson 2011 (as fn. 10), TC: 00.00.35–00.01.37. 77
 Luisa Feiersinger

as well as the artificial world with its population of figurines

have to be rendered on numerous screens. The impression of
human-sized three-dimensionality is then possible through
a careful arrangement of those viewpoints in hypostereo.
Furthermore, elements – such as the clouds – are chosen
for their ability to be displayed stereoscopically in arresting
ways. The dissolution of the screen and the abstraction of
two-dimensionality can thus only be left behind through
intricate technical and narrative alignments. The screen-
based image remains fixed in the mechanics and techniques
of its production and reception, even when the screen is
negated in a narrative sense.

4  Trompe-l’œil in the opening credits for The Three Musketeers (2011), screenshot, TC: 00.00.37.
Figures
1, 4  Paul W. S. Anderson, The Three Musketeers, Germany/France/UK/
erally, leaving it behind, the camera then opens up a visu- USA 2011, in: 3-D Blu-ray, Constantin Film Verleih GmbH 2011, 111 Min.,
TC: 00.22.21, 00.00.37.
al space that is only possible in digital stereoscopic films.
2–3  Digital Effects Supervisor, http://artofvfx.com/?p=1713 (accessed
Flying through an artificial space that is populated by fig- January 23, 2015).
urines (familiar from re-enactments of historical battles),
the camera’s movement is reminiscent of a physical camera,
moving freely within the space. But the zoom through the
sky and the clouds was created digitally. By making use of
stereoscopy, the film aims to produce sensation rather than
realism. The figurines and their stereoscopic viewpoints
are rendered in such a way that their three-dimensional-
ity equates to human size, with the camera’s flight up and
through the space causing a kind of a roller coaster sensa-
tion for the viewer.
The being there in an artificial world, which succumbs
to the screen as mediator, is nonetheless a worthy successor
in this longer tradition, if one considers both the invisible
screen-plane, and the technical and practical set-up that
78 produces the screen-based image. The various viewpoints
Lasse Scherffig

From Action Capture to Interaction

Gestalt
How can moving a small physical object on a table, observ- discussed in the humanities.2 Here, the focus often is on the
ing apparent motion on a computer screen and pressing a epistemological implications of this discipline, which, from
finger onto one part of the object be experienced as one inte- its very beginning, proclaimed it would erase the bound-
grated action: clicking on something? This text proposes a aries between animal and machine, living and non-living
novel answer to this question that goes back to the begin- systems. 3 One of its core tenets is the application of negative
ning of interactive computing and before, re-activating ideas feedback to the description and control of any process that
from cybernetics to arrive at a new understanding of screen- can be described as goal-directed behaviour.4
based interaction.1 Negative feedback implies that the output of a system
In this paper, I will briefly introduce cybernetics and its is fed back to its input as a negative quantity, resulting in a
role in interactive computing and show how at the transition system that operates on the difference between its output
from analogue to digital computing screen-based interac- and a desired goal. Systems using negative feedback hence
tion was introduced. I will then explain how screen-based use their own deviation from a given goal as a means of cor-
interaction is subject to questions regarding the percep- recting this error.
tion of motion that were first raised by gestalt psycholo- Cybernetics to some extend can be understood as a
gy, explain how these questions relate to the idea of direct science undertaking a “totalization” of feedback control. 5
manipulation and how we might have to rethink the gestalt Its importance for answering the question about clicking
of an interface as an effect of interaction.
Kathryn Hayles, How we Became Post-human. Virtual Bodies in Cybernetics,
2
Literature, and Informatics, Chicago, IL: Chicago University Press, 1999;
Cybernetics and Interaction Claus Pias, Zeit der Kybernetik. Eine Einstimmung, in: Claus Pias (ed.),
Cybernetics/Kybernetik – The Macy Conferences. Volume II, Zürich/Berlin:
While cybernetics played an important role in the formation Diaphanes, 2004, pp. 9–41.
of early computer science, in recent times it has mainly been This is already apparent in the title of Norbert Wiener’s Cybernetics. Or
3
Control and Communication in the Animal and the Machine, Cambridge, MA:
MIT Press, 1948.
Arturo Rosenblueth, Norbert Wiener, Julian Bigelow, Behavior, Purpose
4
For a much broader and more detailed development of this argument see
1 and Teleology, in: Philosophy of Science 10 (1943), pp. 18–24.
Lasse Scherffig, Feedbackmaschinen. Kybernetik und Interaktion, Köln: Peter Galison, The Ontology of the Enemy. Norbert Wiener and the Cyber-
5
Kunsthochschule für Medien Köln, 2017. netic Vision, in: Critical Inquiry 21.1 (1994), pp. 228–266, p. 233.
 Lasse Scherffig

applied to a variety of problems,6 paving the way for the

idea of building a flight simulator for arbitrary (existing and
future) airplanes as “a cockpit or control cabin connected,
somehow, to an analog computer” 7 (fig. 1).
Analogue computing, in this context, denotes a form of
computation where a physical system is built in analogy to
a phenomenon under study.8 At the Servomechanisms Lab-
oratory, during the early twentieth century, this practice
led to the construction of a series of feedback-based elec-
tro-mechanical devices to study the dynamics of the electri-
cal power grid and other complex systems (fig. 1). Likewise,
the ASCA was conceived as an electro-mechanical system
whose kinetic and electrical dynamics would resemble the
dynamics of flying. Crucially, this meant that the cockpit
1  The Product Integraph, an electro-mechanical analogue computer built
at the Servomechanisms Laboratory. would be an integral part of the computer – as the motion
of its instruments and controls would be inseparable from
the motion of computation.
on something is two-fold: on one hand, the idea of negative However, halfway through its construction, the com-
feedback can help us to understand how in clicking on some- puting part of the machine was turned into a digital comput-
thing, hands on the table and motion on the screen come er, because the project leads had realized the future poten-
together in one integrated sensorimotor act that, although tial of this emerging technology.9 This change meant that a
involving a variety of distinct processes and locations, is digital computer was to take over the role of an electro-me-
perceived as one. On the other, historically, the MIT Ser- chanical device intrinsically connected to an environment
vomechanisms Laboratory was behind much of the rise of (a cockpit, in this case). It hence had to be a special kind of
negative feedback and thus the emergence of cybernetics as digital computer: a computer that operates in real-time and
a field, as well as the construction of the first digital com- allows for the exchange of data with its environment while
puter that was interactive in today’s sense. operating.
This computer started as an analogue computer for
flight simulation, the Aircraft Stability and Control Analyz- David A. Mindell, Between Human and Machine. Feedback, Control, and
6
er (ASCA). The machine originally was planned as a con- Computing before Cybernetics, Baltimore: Johns Hopkins University Press,
2004, pp. 157–158.
tinuation of the laboratory’s successful work in analogue Kent C. Redmond, Thomas M. Smith, Project Whirlwind. The History of a
7
computing. Especially Vannevar Bush’s famous Differential Pioneer Computer, Bedford, MA: Digital Press, 1980, p. 32.
Charles West Churchman, Operations research. Eine Einführung in die
8
Analyzer had made clear that analogue computing can be Unternehmensforschung, München: Oldenbourg Verlag, 1971, pp. 151–152.
80 Redmond 1980 (as fn. 7), pp. 27–44.
9
From Action Capture to Interaction Gestalt

2  ASCA 1947 (left) and Whirlwind 1950 (right), a cockpit whose moving parts are part of a computational process versus the shape of computation to come.

During the construction of this novel machine, how- Reciprocal Visibility

ever, the task of building a versatile and fast digital com-
puter became so dominant, that the engineers involved in What was too new to make a digital ASCA possible comes
the project increasingly neglected the cockpit portion of the down to two questions: how to make digital data and pro-
system. As this cockpit still constituted a system of analogue cesses visible to human viewers, and how to make the view-
instruments and moving parts, it later became clear that er’s actions and reactions, in turn, visible to the computer?
connecting these instruments to a digital computer posed Both problems are rooted in the nature of digital com-
fundamental problems that had never been dealt with putation: the visibility of a representation in analogue com-
before: “These problems were not impossible, but neither puting is determined by the relationship between a phys-
did established solutions exist. The digital computer was ical system and the system it was made to model. Bush’s
too new.” 10 Differential Analyzer, like the other analogue computers of
In consequence, the simulator’s cockpit was scrapped the Servomechanisms Laboratory, was not so much a com-
in late 1948 and the result was named Whirlwind the first puter that solved differential equations as it was “an elegant,
interactive computer ever built and no longer a flight sim- dynamical, mechanical model of the differential equation”
ulator (fig. 2).11 that did “kinetically act out the mathematical equation”.12

10 Ibid., p. 49. 12 Larry Owens, Vannevar Bush and the Differential Analyzer. The Text and
11 Ibid., p. 60, pp. 43–44. Context of an Early Computer, in: Technology and Culture. The International 81
 Lasse Scherffig

Likewise, the ASCA would have been an electro-mechani- output that may affect future inputs – has only relatively
cal model of the aerodynamics of flight. This model would recently been acknowledged by theoretical computer sci-
seamlessly integrate the instruments and controls of the ence.18
cockpit, as well as any human action bearing on them, sim- During the transition from an analogue ASCA to a digi-
ply because the motion of instruments and controls would tal Whirlwind, both problems were addressed pragmatically.
be part of acting out the computation. In contrast, digital The problem of the visibility of digital data was approached
computation has no a priori relationship to the systems it by establishing the mode of representation that is still dom-
models.13 It is marked by discrete states, represented by the inant today: the computer drew arbitrary symbolic repre-
symbols of a formalism, and “carefully chosen rules that sentations on the screen. To that end, Whirlwind’s data
dictate how one symbol succeeds another”.14 In order to be registers were linked to the x/y-position of the electrode
visible, digital computation must be translated into repre- beam of a cathode-ray tube (CRT, fig. 3).19 By so doing, the
sentations that “stand in an arbitrary relationship to the discrete states of machine computation were translated into
objects they represent.” 15 representations that are readable by human observers, and
In addition, digital computation from the very begin- the computer screen was introduced.
ning was conceptualized as a time and context free process. Within the project, the establishment of this new form
The idea of the Turing Machine (and equivalent definitions of connecting people and computation was not seen as a
of computation) assumes that computation starts with a great leap. Robert Everett, one of Whirlwind’s engineers,
fixed input, operates on this input according to a fixed set simply noted later: “One of the things that I think we did
of rules and terminates after a finite number of steps (or first was to connect a visual display to a computer.” 20 It was
goes into an infinite loop of repetition).16 Hence “[t]uring understood as something I think we did first because the
machines cannot handle the passage of time”.17 engineering practice of the Second World War had already
Originally conceptualized as a machine in constant dia- established the possibility of thinking (and building) this
logue with a crew of flight operators in training, Whirlwind connection. With the Williams Tube a combination of dig-
had to deviate from this assumption. The fact that indeed ital computation and CRT was already in use. As the Wil-
almost every computer we use today does so – by constantly liams Tube was a form of digital memory that drew zeros
waiting for new input from its environment while producing and ones onto a CRT screen in order to store them for a
few milliseconds, it was not intended to be looked at by a
Quarterly of the Society for the History of Technology (1986), pp. 63–95, p. 75.
human observer.21 But in analogue radar technology, CRTs
13 Gerard O’Brien, Jon Opie, The Role of Representation in Computation, in:
Cognitive Processing 10.1 (2008), pp. 53–62.
14 Ibid., p. 56. 18 Ibid.
15 Ibid., p. 58. 19 Robert Everett, Whirlwind, in: J. Howlett, Gian Carlo Rota, Nicholas
16 Georg Trogemann, Jochen Viehoff, Code@Art. Eine elementare Einführung Metropolis (eds.), A History of Computing in the Twentieth Century, Orlan-
in die Programmierung als künstlerische Praktik, Wien/New York: Springer, do: Academic Press, 1980, pp. 365–384, p. 365.
2005, p. 85. 20 Ibid, p. 375.
17 Peter Wegner, Why Interaction is More Powerful Than Algorithms, in: 21 Claus Pias, Computer Spiel Welten, Dissertation, Weimar: Bauhaus-Univer-
82 Communications of the ACM 40.5 (1997), pp. 80–91, p. 83. sität, 2000, pp. 55–56.
From Action Capture to Interaction Gestalt

3  Light-gun and symbolic representations on an early screen of Whirlwind.

had already been employed as visual displays.22 Finished of interruption would be emitted from the very object the
after the war, even radar CRTs leftover from the war could gun was pointed at. It could thus be interpreted as a selec-
be used in the construction of Whirlwind.23 The project thus tion to be taken into account for further computation.24
simply had to connect both pre-existing practices (CRT- With this setup, Whirlwind was ready to become the
based digital memory and analogue radar displays) to create origin of SAGE, the Semi-Automatic Ground Environment
the arrangement of computation and representation we now air defence system – the largest computer built to date, man-
refer to as the computer screen. aging American air defence until 1983.25 More importantly,
The problem of the visibility of human action to the pro- however, it established a feedback loop between screen-
cess of computation was addressed by interrupting this pro- based representation and action. In consequence, the visi-
cess. A light-gun allowed for a literal handling of computa- bility of what was represented on screen became subject to
tion, as it made it possible to touch symbolic representations interactions between motor activity and visual perception.
by pointing at them (fig. 3). This was achieved by placing a
light sensor at the tip of the gun that would interrupt the
Direct Manipulation
computer’s drawing process. As Whirlwind did not draw a
rasterized image (organized in rows and columns of pixels), The closing of this loop, in which action would be taken on
but drew one representation after the other, interrupting a screen-based representation that in turn would react to
this process entailed that the light picked up at the moment that action, preconfigured how we interact with computers

22 Axel Roch, Die Maus. Von der elektrischen zur taktischen Feuerleitung, in:
Lab. Jahrbuch 1995/96 für Künste und Apparate, Köln: Verlag der Buchhan- 24 C. R. Wieser, Cape Cod System and Demonstration, Cambridge, MA: MIT
dlung Walther König, 1996, pp. 166–173, p. 170. Lincoln Laboratory, 1953, p. 2.
23 Everett 1980 (as fn. 19), p. 379. 25 Redmond 1980 (as fn. 7), p. 206. 83
 Lasse Scherffig

until today. It established a remarkably stable dispositive of Later, Edwin Hutchins, James Holland and Donald
interaction, sustainably structuring large parts of the field Norman expanded on Shneiderman’s work, providing a sem-
of human-computer interaction (HCI), which would later inal discussion of direct manipulation from a cognitive sci-
refer to the combination of (mostly screen-based) represen- ence perspective.28 Starting from the assertion that “[w]e see
tation with the capability to act on these representations as promise in the notion of direct manipulation, but as of yet
an interface. Nevertheless, it took the field until the 1980s to we see no explanation of it”, 29 they develop an explanation
conceptualize the closed loop between representation and that follows Shneiderman’s path by distinguishing between
action as direct manipulation. the physical reality of an interface and the non-physical
This discussion initially was framed by cognitive sci- “model-world” of what it represents. 30 Direct manipula-
ence and computational theories of the mind that treat tion, in this view, implies acting with the metaphors of that
interaction as a process of rule-based problem solving. For model-world, while well-chosen metaphors align this model
Ben Shneiderman, who introduced the term “direct manip- with a user’s problems. It is thus a function of the cognitive
ulation”, the phenomenon can accordingly be explained by or information processing “distance” between the mod-
assuming a difference between non-physical “semantics” of el-world and intention. 31
human problem solving and the physical “syntax” of repre- Surprisingly, however, this does not seem to account for
sentation and action at an interface.26 While, according to the whole phenomenon. Direct manipulation for the authors
this view, any form of HCI has to mediate between these seems to possess a qualitative or experiential component
two domains, direct manipulation reduces the difference that is hard to grasp in the terms of cognition and problem
between them by having users act in the world of seman- solving. In addition to cognitive distance, direct manipula-
tics as opposed to syntax: direct manipulation, the argu- tion relies on emotional engagement, resulting from the feel-
ment goes, allows a writer to, for instance, directly interact ing of being causally effective in that world – a phenomenon
with a paragraph of text (by marking it with the mouse) that cannot be understood in terms of goal-directed problem
as opposed to decomposing high-level semantic intentions solving. The authors thus admit that direct manipulation
into low-level abstract commands whose syntax is largely seems like an “atavistic […] return to concrete thinking”. 32 It
unrelated to the paragraph itself and the act of manipulat- may, however, be precisely the messy concrete thinking of
ing it.27 our hands engaged in syntactic activities (or sensorimotor
loops) that can help us to understand direct manipulation,
as will become apparent later in this paper.
26 Ben Shneiderman, Direct Manipulation. A Step Beyond Programming Lan-
guages, in: Noah Wardrip-Fruin, Nick Montfort (eds.), The New Media Read-
er, New York, NY/London: W. W. Norton & Company, 2001, pp. 486–498. 28 Edwin L. Hutchins, James D. Hollan, Donald A. Norman, Direct Manipu-
27 Ibid. This argument alone is enough to cast doubt on the supposed direct- lation Interfaces, in: Human-Computer Interaction 1 (1985), pp. 311–338.
ness of direct manipulation, as manipulating a paragraph with the mouse 29 Ibid., p. 316.
still presupposes a decomposition into low-level hand movements and but- 30 Ibid., p. 317.
ton presses – only that this low-level syntax is different from, say, a com- 31 Ibid., p. 311.
84 mand line interface. 32 Ibid., p. 337.
From Action Capture to Interaction Gestalt

Gestalt and Apparent Motion

The motion we see on computer screens is what the psychol-

ogy of perception calls stroboscopic or apparent motion, an
illusionary impression of motion created by the succession
of static frames. Historically, the systematic investigation of
apparent motion is closely connected to gestalt psychology, 33
as one of the texts defining the field is Max Wertheimer’s
(still untranslated) Experimental Studies about the Percep-
tion of Motion. 34
Wertheimer’s seminal study tries to understand how
stroboscopic stimuli that are objectively not moving cre-
ate the subjective percept of motion. For the study, Wert-
heimer employs the Schumann Tachistoscope as a strobo-
scope (fig. 4). This device uses rotation to quickly cover and
uncover stimuli. A setup using two stimuli, a and b, and a
prism allows Wertheimer to use the apparatus in a way that,
to a viewer, presents both stimuli in quick alternating suc-
cession.
Focusing on those cases of apparent motion that do
4  The Schumann Tachistoscope.
not yield a perfect illusion of seeing moving objects but, for
instance, fractured and partial motion percepts35, Wert-
heimer arrives at a remarkable conclusion that ultimate- ascribed by perception. Wertheimer, instead, sees motion,
ly reverses the relation of movement and object as it was named “pure φ” or “pure motion”, as a primary object of per-
understood by his contemporaries (fig. 5). These, he argues, ception, even reconstructing the identity of moving objects
assume that the perception of motion presupposes the as a limiting case of motion. 36 In this view, perception of
perception of a moving object, understanding the moving motion happens directly and immediately, preceding and
object as a primary and its motion as a secondary feature enabling the perception of the gestalt of an object. The latter
is hence conceived as a “short-circuit” of motion perception
33 Robert M. Steinman, Zygmunt Pizlo, Filip J. Pizlo, Phi is not beta, and why as a “duo-in-uno” when, for instance, two lines, a and b, in
Wertheimer’s discovery launched the Gestalt revolution, in: Vision Research
40 (2000), pp. 2257–2264.
34 Max Wertheimer, Experimentelle Studien über das Sehen von Bewegung,
in: Zeitschrift für Psychologie und Physiologie der Sinnesorgane 61 (1912),
pp. 161–265.
35 Ibid., p. 191. 36 Ibid., p. 221. 85
 Lasse Scherffig

a a a

b b b

a b

5  Apparent motion of a line from a to b (left) and partial apparent motion 6  Two lines in rapid succession 7  Ambiguous motion as described by Linke.
if the time-interval between frames a and b becomes too long (right), as forming an angle composed of two
described by Wertheimer. sides, as described by Wertheimer.

rapid succession lead to the perception of one object: an observation: His subjects were able to control the perceived
angle composed of two sides (fig. 6). 37 direction of motion most effectively by actively moving their
In special cases, the stroboscopic stimuli causing the hands and heads. This, he writes in a footnote, constitutes a
perception of apparent motion may be ambiguous. When- case of motor activity having a gestalt influence on visual per-
ever, for example, two or more concurring interpretations ception.40 Because the experimental systems of experimen-
of one stimulus are possible, their perception becomes tal psychology of that time, such as the tachistoscope, only
multi-stable: Subjects perceive one possible percept at a allowed for the precise control of the presentation of stimuli
time, while perception alternates between possibilities without connecting it to human action, this effect seemed
over time. This was first demonstrated by Paul Linke using too hard to control for him to warrant further ­investigation.41
a cross that is rotated by 45° from stimulus to stimulus and
that, as a bistable stimulus, can be perceived as clockwise
Action Capture
or counterclockwise rotation (fig. 7). 38 Termed “ambiguous
motion”, this effect was later studied by Paul von Schiller, During the past decades, the methods of experimental
who tried to isolate the factors that determine which possi- psychology have changed significantly in favour of quan-
ble percept is perceived at a time, trying to establish the laws titative research that relies on a universal experimental
of how ambiguous motion is disambiguated to distinct per- system, enabling not only the precise control of the expo-
cepts. 39 During this study, von Schiller made a remarkable sure of stimuli but also the measurement of human action.
This system is fundamentally structured by the interactive
37 Ibid., p. 251.
38 Paul von Schiller, Stroboskopische Alternativversuche, in: Psychologische
Forschung 17 (1933), pp. 179–214, p. 180. 40 Ibid., p. 196.
86 39 Ibid. 41 Ibid., p. 195.
From Action Capture to Interaction Gestalt

computer and was, for instance, pioneered in the famous that the perception of ambiguous auditory48 and tactile49
experimental studies by Douglas Engelbart42, which led to stimuli can be captured by the movement of the hands, eyes50
the decision to replace the light-gun for computer input with and the walking body51. For the interface this implies that
the mouse.43 Today, “the computer […] has taken over prac- we may have to understand interfaces as perceived in action.
tically all the experimental procedures used to examine the Their visual (and even tactile and auditory) qualities are influ-
perception of space and time”.44 enced by our actions, and interface design may have to take
It nevertheless took until 1994 for the first publication to into account that it is not only the functioning of an interface
present quantitative evidence of the influence described by that depends on its use, but also its perceptual qualities.
von Schiller.45 Since then, a series of studies have shown that Research examining action capture has shown that it
if ambiguous motion is coupled to physical motion, then the is facilitated by a close physical and temporal distance of
8  A typical ambiguous motion
bodily movement captures its perception by influencing it in the action and stimulus52, as well as a correspondence of stimulus as used in experiments.
the direction of motion. This capture effect is strongest for the axes and orientation of motion between both. 53 More Stroboscopic motion of the circles is
presented on a computer screen at
voluntary self-motion and has therefore been named “action importantly, the correspondence of stimulus and action that the same time as subjects perform
capture” 46 (or “priming by actions” 47), to do justice to the drives the effect is not an a priori. It is context dependent, as physical motion on an input device,
such as a keypad, knob, or mouse.
fact that the influence is caused by whole actions, comprised it can be influenced by expectations: when, for instance, a
of intentions, motor planning and execution. In most stud- button with a right arrow is pressed, perception of apparent
ies analysing the effect, computers are used to couple the rotation is captured in the clockwise direction, because we
movement of the hands with ambiguous motion stimuli pre- have learned that pushing a round object to the right will
sented on a screen (fig. 8). Action capture, one could hence most likely cause it to rotate in the clockwise direction. 54
argue, has been mostly studied as inter-action capture. The effect, in addition, can be modified and even reversed
The concept of action capture not only holds for visual by training. 55 And finally, it is already present when actions
stimuli and the motion of our hands; it also has been shown are merely planned and not yet carried out. 56

42 As a dispositive it structures the presentation of stimuli, the measurement 48 Bruno H. Repp, Günther Knoblich, Action Can Affect Auditory Perception,
of responses, the design and statistical analysis of experiments, and by that in: Psychological Science 18.1 (2007), pp. 6–7.
“the nature of the questions that can be addressed”. Nicholas J. Wade, Dieter 49 Olivia Carter, Talia Konkle, Qi Wang, Vincent Hayward, Christopher Moore,
Heller, Scopes of Perception. The Experimental Manipulation of Space and Tactile Rivalry Demonstrated with an Ambiguous Apparent-Motion Quar-
Time, in: Psychological Research 60.4 (1997), pp. 227–237, p. 235. tet, in: Current Biology 18 (2008), pp. 1050–1054.
43 William K. English, Douglas C. Engelbart, Melvyn L. Berman, Display-Se- 50 Ibid.
lection Techniques for Text Manipulation, in: IEEE Transactions on Human 51 Yoshiko Yabe, Gentaro Taga, Treadmill Locomotion Captures Visual Per-
Factors in Electronics 8.1 (1967), pp. 5–15. ception of Apparent Motion, in: Experimental Brain Research 191.4 (2008),
44 Wade, Heller (as fn. 42), p. 235. pp. 487–494.
45 G. Ishimura, S. Shimojo, Voluntary Action Captures Visual Motion, in: 52 G. Ishimura, Visuomotor for Action Capture, in: Investigative Ophthalmol-
Investigative Ophthalmology and Visual Science (Supplement) 35 (1994), ogy and Visual Science (Supplement) 36 (1995), p. 357.
p. 1275. 53 Wohlschläger 2000 (as fn. 47), pp. 927–929.
46 Ibid. 54 Ibid., p. 928.
47 Andreas Wohlschläger, Visual Motion Priming by Invisible Actions, in: 55 Ishimura 1995 (as fn. 52), p. 357.
Vision Research 40 (2000), pp. 925–930, p. 929. 56 Wohlschläger 2000 (as fn. 47), p. 929. 87
 Lasse Scherffig

Accordingly, action capture seems to not only depend 1.00

on how much an action corresponds with what we perceive

(in terms of spatio-temporal distance and orientation), but
rather seems to depend on how much a possible percept
corresponds to the result we expect an action to have. If I 0.75

expect (or plan) my hands to be involved in causing clock-

wise rotation, I am more likely to perceive an ambiguous

Action Capture
rotation as clockwise.
This corresponds to early findings from human fac- 0.50

tors indicating that the speed and error rate of actions at

an interface depend on the “compatibility” of stimulus and
response. 57 This compatibility has from the beginning been
0.25
understood as an acquired relationship, for which it holds
that “stimulus and response sets are optimally matched
when the resulting ensemble agrees closely with the basic
habits or expectancies of individuals”. 58
0.00
In order to further analyse the significance of action
0.0 0.4 0.8
capture for screen-based interaction, I have conducted a Similarity (cross-correlation)
study linking earlier research on action capture to the com-
9  Logistic regression of the relationship between a) the similarity of
patibility of mouse action and computer response. 59 Assess- mouse motion and a perfect rotation on screen, in the direction subjects
ing compatibility, however, is a messy task, since basic habits were asked to move the mouse, and b) the coincidence of perceived direction
of ambiguous motion with the direction of mouse motion (action capture).
or expectancies do not translate well into experimental pro- The more similar the motion of the hand is to a perfect rotation in the correct
tocols. But the computational tools of cybernetics – the field direction, the more likely is action capture. For details see Scherffig 2017
(as fn. 1), pp. 259–261.
whose heritage still defines the way interactivity works – at
least provide ways of measuring a non-semantic similarity
of stimuli and response, understood as the cross-correlation the mouse as they were asked to perform circular motion
of a time series of measurements. while looking at an ambiguous rotation on screen. It thus
Coupling an ambiguous motion stimulus to mouse relied on a paradigmatic case of interaction, linking the
movements, the experiment measured how subjects moved motion of the mouse with apparent motion on screen, while
making their interrelation measurable by using ambiguous
57 Paul M. Fitts, Charles M. Seeger, S-R Compatibility. Spatial Characteristics motion that can be captured by the body’s activity. In a
of Stimulus and Response Codes, in: Journal of Experimental Psychology series of trials, mouse motion was recorded together with
46.3 (1953), pp. 199–210, p. 199.
58 Ibid., p. 208.
the perceived direction of rotation of each trial, ascertained
88 59 Scherffig 2017 (as fn. 1), pp. 257–262. through questions (fig. 9).
From Action Capture to Interaction Gestalt

The result is as simple as it is statistically significant: mouse and observe its cursor and the on-screen reactions to
the more similar the mouse motion is to the motion sub- pressing a finger onto it, the motion of our hands on a phys-
jects were asked to perform, the higher the likelihood of ical object on a table and the perception of apparent motion
action capture (the significance level of their correlation is on a computer screen is fused internally into a sensorimotor
p<0.0001, see also fig. 5). In other words, the more similar unity that goes beyond the mere correlation of both.
the action of the hand on the mouse and the reaction on the This is exactly what was observed by computer scientist
screen seem to be, the more the former captures the latter. Dag Svanæs.61 Conducting experiments in which subjects
Action capture, therefore, seems to quantitatively incorpo- interacted with abstract interactive systems consisting of
rate our actions into what we perceive. Our perception seems black and white squares62 he analysed their behaviour in
to calculate with our actions and their expected results. correlation with their verbal descriptions of it, paying atten-
tion to the way the abstract black and white squares slowly
became perceived as objects:
Interaction Gestalt
We have seen the idea of calculating with one’s actions The objects described by the subjects in the experi-
before. It is the idea of comparing actions with changes in ments existed for them only through interaction.
perception, or – in the technical terms of cybernetics – the The objects emerged as a result of the interplay
idea of using the difference between a system’s output and between the intentions of the users, the users’ actions,
some goal to determine its future output. and the feedback given by the system. 63
Past and contemporary discourse in physiology, the
psychology of perception and cognitive science has often Observing the interaction with a simple system that would
identified this goal with expectations or predictions60: We eventually be perceived as a switch, he notes:
constantly compare the change in perception induced by
our activities with expected change. This is what becomes When the subjects said ‘It is a switch’, they did not
apparent (and even measurable) as action capture: our pre- come to this conclusion from a formal analysis of the
dictions regarding a physical action capture the way we State Transition Diagram of the example. Nor did they
perceive the results of that action. conclude it from the visual appearance of the square, as
Action capture thus suggests that at the interface, too, it the squares all looked the same. The switch ­behavior
is our actions that determine what we perceive. If we move a

60 See, for instance, Karl J. Friston, Christopher Thornton, Andy Clark,

Free-Energy Minimization and the Dark-Room Problem, in: Frontiers in
Psychology 3 (2012), pp. 1–7; Jack M. Loomis, Distal Attribution and Pres- 61 Dag Svanæs, Understanding Interactivity. Steps to a Phenomenology of
ence, in: Presence 1.1 (1992), pp. 113–119; or Sarah-Jayne Blakemore, Chris Human-Computer Interaction, Dissertation, Trondheim: Norges Tekni-
D. Frith, Daniel M.Wolpert, Spatio-Temporal Prediction Modulates the sk-Naturvitenskapelige Universitet, 2000.
Perception of Self-Produced Stimuli, in: Journal of Cognitive Neuroscience 62 Ibid., pp. 128–132, pp. 108–110.
11.5 (1999), pp. 551–559. 63 Ibid., p. 230. 89
 Lasse Scherffig

slowly emerged from the interaction as the square react to it. According to this model, we can indeed under-
repeated its response to the subject’s actions. 64 stand direct manipulation in terms of distance end engage-
ment. But distance would be reduced to simple spatio-tem-
This implies that the hands on the mouse in dialogue with poral distance of stimulus and response and the perceptual
the computer’s response yield the emergence of perceptual similarity between both. Or more generally, it would be
units having “gestalt properties” as their perception, once redefined as the distance of predicted and actual reaction,
emerged, is “direct and immediate” and not a cognitive which is the negative feedback at the heart of cybernetics.
interpretation of action and perception.65 Engagement, in turn, would become being engaged in sen-
Svanæs therefore suggests treating the objects that sorimotor loops that are continuously learned and exercised,
compose an interface as “interaction gestalts”, entities that forming the objects they deal with within this cyclical pro-
are “similar to visual gestalts in that they are wholes, and cess. If today’s touch-based interaction on mobile phone
not compositions of analytical elements”.66 The form or screens seems to constitute a return to Whirlwind’s com-
gestalt of an interface, understood as interaction gestalt, can bination of screen-based representation and the possibility
be seen as a perceptual or experiential whole that is based of touching it, this may be understood in light of HCI’s con-
on action and perception as duo-in-uno – as a limiting case stant effort to minimize distance and maximize engagement
of the loop of human action and machine reaction. in these literal terms.
From a sensorimotor perspective, the elements that The simple need to establish reciprocal visibility
make up an interface are hence not so much the discrete between computation and human environment thus intro-
entities that they have been designed and programmed to duced a dispositive of interaction in which bodily move-
be; instead they are the results of being used. Buttons, in this ment at the computer screen, its predicted and its observed
sense, look like buttons because they are used as such – and results together are integrated into coherent perceptions of
the other way round. Their form does not imply or com- interaction, that form the gestalt of the interface. This is the
municate their function. Instead, their (subjectively expe- integration of hand and screen-based representation that
rienced) form and function are interdependent and are the allows us to speak of clicking on something while we steer a
result of their use and its context. physical mouse on a table and watch apparent motion on a
This suggests a cybernetic model of the interface and screen. What, according to this view, creates interfaces, is
interaction, implying that what we see is enacted by how we interaction.67

64 State Transition Diagrams are formal graphical representations of how a

system of discrete states (such as combinations of black and white squares
that can switch their color) can transition from one state to another. In
Svanæs’ experiments these diagrams describe the actual behavior of the
systems used, as opposed to the perceived behavior described by his sub-
jects. Ibid., p. 206.
65 Ibid., p. 244. 67 As in: not only enables and shapes their functioning as interface but also
90 66 Ibid. their appearance.
From Action Capture to Interaction Gestalt

Figures
1  MIT Museum. Reprinted with permission from MIT Museum.
2  Copyright Computer History Museum. All Rights Reserved (left), The
MITRE Corporation. All Rights Reserved (right). Reprinted with permission
from The MITRE Corporation.
3  The MITRE Corporation. All Rights Reserved. Reprinted with permission
from The MITRE Corporation.
4  Collection Jirí Hoskovec. Reprinted with permission by Simona
Hoskovcová.
5–9  Author’s figure.

91
Kathrin Friedrich

Screening Bodies
Radiological Screens and Diagnostic Operations

Screens provide the basis for actions and diagnostics in and probably even instructive practice, the screen-bound
clinical radiology. From a media-historical perspective, the dispositive of radiological diagnostics in pre-digital form
transformation from light boxes to today’s digital screen (both regarding the imaging technique of radiography and
configurations provides the opportunity to explore the the diagnostic practice at the light box) involved several
epistemic and operational conditions of screens. In par- main objects and actors (fig. 1). These constitute the scenery
ticular, screen architectures and screen-bound tools are that might have been staged for the photograph to be taken.
central to the question of how screen operations prefigure Nevertheless, it becomes obvious that the hanging of x-ray
the diagnostic screening of patients’ bodies. By focusing films on the wall-sized light box is neatly ordered. A series
on the exploration of the operational and epistemic rela- of thorax and abdomen radiograms in different perspectives
tions between screens and screening, I will use the order of supposedly the same patient are provided. While two
of hanging or arranging film print-outs on light boxes as an eagerly interested physicians, probably radiological novic-
example of a site-specific practice which was significantly es, are sitting and staring at the light box in front of them, a
altered by the introduction of digital infrastructures into seemingly skilled radiologist instructs their sight by using
clinical radiology departments. The screen-based radiolog- a pointing stick.
ical hanging protocol, which specifies the arrangement of Even if this diagnostic dispositive is primarily direct-
visualizations on a light box or in a graphical user interface ed towards the screen of the light box, several other media
(GUI), reveals the extent to which media transformation technologies and infrastructures intersect in the process of
and epistemic practices are mutually contingent as well as crafting a diagnosis.1 Next to the telephone in the lower cor-
how profound and abundant the apparently flat and limited ner on the right there is also a Dictaphone to record diagno-
screen is. ses that are later on typewritten by busy clerks. Diagnostic
viewing is presented as a collective and distributed practice
rather than just an almost contemplative posture in front of
Protocols and Practices of Light Box Hanging
Diagnostic image viewing based on analog radiograms, On the design and conception of light box dispositive in early radiology
1
see Christian Vogel, Epistemischer Sinn und ästhetische Wirkung. Das
which are hung up on electrically illuminated light boxes, Betrachten von Röntgenbildern im Schaukasten, 1896–1930, in: Fotoges-
dates back to the 1910s. Later on, as an established collective chichte 138 (2015), pp. 19–28.
 Kathrin Friedrich

1  Photograph of x-ray diagnostics at Hermann Hospital in 1953 using a 2  Example of hanging CT scans on a light box.
light box.

a bright screen. As suggested by this photograph, viewing visualization for diagnostic purposes was not yet delegated
images on the light box is associated with perceiving from to digital representation modes, such as computer screens,
a distance static images that are strictly ordered in a stat- until the mid-1990s. One quite practical reason for this
ic frame. The x-rays could not be changed once they were technological difference between modes of data acquisition
printed out on film. Only their order at the light box was and modalities of visualization was the need for advanced
reconfigurable, and to a certain degree instruments such as and in particular networked software and also technically
the pointing stick or magnifying glasses could help to guide advanced computer screens, such as high-resolution dis-
the diagnostic gaze and bridge the operational and probably plays. Another, more epistemological reason was the estab-
epistemic gaps between distant users and static images. lished diagnostic routine and its persistence. The radiolo-
From the mid-1970s, digital imaging technologies such gists’ expertise slowly adapted to the technical features and
as computer tomography (CT) and later magnetic resonance diagnostic possibilities of new digital imaging techniques,
imaging (MRI) were introduced into clinical practice. These but it responded even more slowly to changing dispositives
genuinely digital imaging processes complemented analog of diagnostically screening images and bodies. For another
x-ray visualization, but the diagnostic viewing of images 15–20 years after digital imaging such as CT and MRI were
on the light box continued. Even though CT and MRI data introduced to clinical practice, the light box remained the
94 were produced and processed digitally, their tomographic primary place of radiological diagnostics.
Screening Bodies

The light box hanging of CT visualizations consists of leave a white space between the sequence of cross-sectional
individual cross-sectional images from a digital scan of a images, a cognitive bridging and spatial reformation must
specific body region (fig. 2). These cross-sectional image be performed in the viewer’s mind based on radiological
series are printed on films in a matrix. Typically, these films expertise. Finally, this imaginative reformation enables con-
consist of 4 × 5 matrices (4 cross-sectional images alongside clusions to be drawn as to the size, position and development
each other, arranged in 5 rows). Within each film sheet, the of a problematic structure in the patient’s body. The flatness
images are then read from left to right and from top to bot- of the view, the layout of the film hanging and the linear
tom. This schema is applied to the overall arrangement of arrangement of images all play a role in the mental summing
films on the light box. Anthropologist and radiologist Barry up of the slices on the light box into a body volume. At the
Saunders notes in his ethnographic observation of diagnos- same time, this kind of hanging films creates a form of clar-
tic CT reading on light boxes: ity that invites an elliptical and comparative way of seeing. 3
Simply by changing position, the viewer in front of the light
CT images are typically displayed in an order of mag- box can navigate between different cross-sectional planes or
nitude or so smaller than the specimen they reference, hang old and new images directly next to each other.
with many images on one sheet of film. Film size is Besides the arrangement of images on films as well as
standard, but the ‘matrix’ of slices on each sheet [...] is their hanging on the light box, cross-sectional image view-
variable, subject to differing conventions, even to ad ing is structured in another media technical respect. The
hoc specification by readers.2 size of the box determines the number of films that can be
examined at one time. To an extent, the available area lim-
The hanging sequence and hence the viewing sequence its the number of cross-sectional images per film sheet, as
applied to the image rows is based on the linear writing the slices as such would otherwise become too small to be
and reading direction of Latin scripts, whereas the actual examined in a detailed and diagnostically significant way.
diagnostic routine, i. e. reading the images, varies among Moreover, the viewer’s capacity would be challenged by
physicians depending on their operational routines, expe- an increased number of films and cross-sectional images.
rience and the diagnostic request as well as on body region. Fading out or integrating the gaps between individual tomo-
Formally, the hanging protocol and the screen install and graphical slices may be merely a question of focus for experi-
propose a certain order of viewing through their very own enced radiologists. But for the less experienced radiologists,
architecture. Hence, the gaps between individual cross-sec- it may cause a loss of orientation in the body volume.
tional images may be bridged in the direction of vision, but To a degree, the exterior form of the box specifies an
cognitively and epistemically, this bridging is performed epistemic and aesthetic framework in which diagnostic
by the radiologists themselves. Where the printouts simply
Eva Cancik-Kirschbaum, Bernd Mahr, Anordnung und ästhetisches Profil.
3
Die Herausbildung einer universellen Kulturtechnik in der Frühgeschichte
Barry F. Saunders, CT Suite. The Work of Diagnosis in the Age of Noninvasive
2 der Schrift, in: Bildwelten des Wissens. Kunsthistorisches Jahrbuch für Bild-
Cutting, Durham/London: Duke University Press, 2008, p. 18. kritik 3.1 (2005), pp. 97–114. 95
 Kathrin Friedrich

operations can take place. The hanging of the films and the image viewing and image operation is established through
arrangement of cross-sectional images within a film sheet and before the screen. If the term screen is used in the verb
lead the gaze through the image sequences, thereby ori- form to screen, the epistemic sphere of possibility becomes
enting the mental reconstruction of the body volume. The clear: “to test or examine someone or something to discover
format of the film and the selection of the image matrix in if there is anything wrong with the person or thing”.6 The
conjunction with the frame of the lightbox establish a phys- light box opens up, and calls for, both a position and dis-
ical and epistemic order for radiological diagnostics. position of the viewer that not only screens the visualized
In addition, radiologists sometimes use magnifying body but also draws closer to images or image sequences in
glasses or blinds integrated within the box in order to order to examine and test them.7 Here the instrumentation
emphasize certain aspects or limit the illuminated area. of screening plays an epistemically and aesthetically signifi-
In his ethnographic study Barry Saunders describes how cant role: image films are weighed up against each other, for
radiologists use a few tools such as magnifying glasses or example, by comparing them or using instruments such as
pointers for didactic purposes in particular: glasses; the light distribution is limited, and proportions are
scaled. In the transition from diagnosis using the lightbox to
Once seated, radiological vision uses few prostheses. diagnosis on the computer workstation, which is integrated
Occasionally one sees a reader of mammograms hold- in a software-based picture archiving and communication
ing a magnifying glass. [...] But diagnostic film viewing, system (PACS), interaction and interface designers are faced
including CT reading, is mostly macroscopic: it employs with the fundamental challenge of establishing new conven-
a ‘native’ vision, a repertoire of squinting and scanning tions of image viewing by also integrating these hanging and
and gazing, a few feet from the image surface.4 screening routines.8

On closer examination, diagnostic practice using light boxes

as diagnostic screens for both x-ray and CT visualizations is
revealed as a highly orchestrated and instrumented process
that requires whole-body involvement – not necessarily the
patient’s, but rather the radiologist’s. Hence, the notion of 6 Ibid.
screen undergoes an almost performative turn to become 7 Lisa Cartwright, Screening the Body. Tracing Medicine’s Visual Culture, Min-
neapolis: University of Minnesota Press, 1995.
screening. As a “flat surface [...] on which pictures or words
8 A more phenomenological analysis could also be helpful at this point as,
are shown”, the radiological light box is a screen in the more generally, proposed by Introna and Ilharco (2006) to account for the
media-technical sense of the term. 5 What is more, a site of “screenness” of screen dispositives and operations: “The screen is phenom-
enologically analyzed as the grounding intentional orientation that con-
ditions our engagement with certain surfaces in as much as we comport
ourselves towards them as screens [...]. This might be formally indicated as
Saunders 2008 (as fn. 2), p. 18.
4 the screenness of screen.” Lucas D. Introna, Fernando M. Ilharco, On the
Cambridge Dictionaries Online, http://dictionary.cambridge.org/us/dic-
5 Meaning of Screens. Towards a Phenomenological Account of Screenness,
96 tionary/american-english/screen (accessed February 8, 2018). in: Human Studies 29.1 (2006), pp. 57–76, p. 58 [original emphasis].
Screening Bodies

All Digital – Hanging Protocols and GUIs

The idea of not only digitally generating but also processing

and visualizing radiological image data dates back to the end
of the 1970s. One of the first subject-specific publications
that envisions an “all-digital department” that “includes,
besides all-digital diagnostic devices, a complete new digi-
tal communication structure and standard” was published
by German information science scholar Heinz Lemke and
colleagues in 1979.9 In their paper Application of Picture
Processing, Image Analysis and Computer Graphics Tech-
niques to Cranial CT Scans they stress the fact that such an
integrative system that serves all functions mentioned in
the paper’s title would need to include digital screen-based
workstations to provide “possible working modes in such
a system [distributed computing network, KF]”.10 In the
3  Radiological diagnostics at a workstation.
filmless era envisioned, the site of radiological diagnosis
would shift from the light box to the computer workstation.
With the broader realization of digital infrastructures in CT exams with a thousand images are becoming com-
radiology departments in the 2000s, and more specifical- mon and simply cannot be managed effectively on film.
ly with the implementation of software applications in a PACS viewing software can be used to dissect, analyze,
PACS, the computer screen and the workstation’s Graph- magnify, or reformat image data in an infinite number
ical User Interface (GUI) became the primary place of of ways.12
medical image data visualization and examination (fig. 3).11
The radiologist David Hirschorn notes: Hirschorn suggests that digital technologies not only make
the dissection, analysis, magnification or reconstruction of
9 Adrian M. K. Thomas, Arpan K. Banerjee, Uwe Busch (eds.), Classical Papers image series in real-time possible; all these processes can
in Modern Diagnostic Radiology, Berlin/Heidelberg: Springer, 2005, p. 332.
now take place in an unlimited number of forms and ways.
10 Heinz U. Lemke, Siegfried Stiehl, Horst Scharnweber, Daniel Jackél,
Applications of Picture Processing, Image Analysis and Computer Graph-
ics Techniques to Cranial CT Scans. Proceedings of the Sixth Conference
on Computer Applications in Radiology and Computer Aided Analysis of ing, informing, entertaining, affecting life, simply being there providing
Radiological Images, in: IEEE Computer Society Press, 1979, pp. 341–354, ground.” Nigel Thrift, Knowing Capitalism, London: SAGE Publications,
p. 341. 2005, p. 234.
11 In a broader perspective, geographer Nigel Thrift identifies screens as a 12 David S. Hirschorn, Introduction, in: Keith J. Dreyer, David Hirschorn,
constant place and locus of attention in times of digital processing and visu- James H. Thrall, Amit Mehta (eds.), PACS. A Guide to the Digital Revolution,
alization: “Screens are one of the constants of everyday life, communicat- New York: Springer, 2006, pp. 3–6, pp. 3–4. 97
 Kathrin Friedrich

What is proclaimed, in a technological euphoria, as a new, thereby establishes its own specific site that concretizes
infinite sphere of possibility, made possible by software data streams and renders them human amendable. Within
and GUIs, comes up against radio­logists’ existing skills and the digital infrastructure of radiological imaging software,
media competences. In the tension between technical possi- computer workstations establish a site where radiologists
bilities, established ways of image viewing and convention- consult their material and gain stationary access to data
alized methods of hanging and handling images, the screen streams. As in pre-digital times, it is a screen or a battery
and in particular the GUI become enabling yet authoritative of screens that frame where and how visualizations are to
interfaces in the act of communication and access. As media be viewed. However, the functionality of a workstation is
theorist Wendy Chun notes: bound a priori to the interaction with a number of “interfa-
cial devices”, such as the screen, keyboard and mouse.15 At
GUIs have been celebrated as enabling user freedom a workstation, the very external architecture of input and
through (perceived) visible and personal control on the output devices and the involvement of the user reveal that
screen. This freedom, however, depends on a profound the screen as the image surface is “just a specific sub-in-
screening: an erasure of the computer’s machinations terface within a broader human-computer interface”.16
and of the history of interactive operating systems as This relativization appears important in order to make
supplementing – that is, supplanting – human intelli- clear that, despite the similarity between the light box as
gence.13 a hardware dispositive and the workstation, both the diag-
nostic and theoretical focus undergoes a fundamental shift.
And in the realm of radiological diagnostics and medical “The screen just reassembles various interfacial processes,
screen operations in general the question of how software translating and returning them as visual representations
structures and screen-based disposition supplement or sup- on a flat visual plane.” 17 Therefore, the processes of digi-
plant human intelligence and action is even more pressing tal visualization and instrumentation need to be critically
as it touches upon responsibilities for making the choices untangled to examine which further interfacial processes,
regarding patients’ further treatments. Further, which new such as communication with colleagues or internet searches,
possibilities for diagnosis does the GUI create as an interac- are reassembled within the screen to shape the finding of
tively usable but nevertheless screen-bound interface, com- a diagnosis.
pared to film-based image viewing on the light box? Nevertheless, the issue of the ordered hanging and dis-
Viewed pragmatically, the GUI forms “a place where playing of cross-sectional image series is also relevant in
individuals and ‘communities’ meet infrastructures”.14 It the context of GUIs. Whether the diagnosis is performed
in tile mode (similar to a light box hanging) or stack mode
13 Wendy Chun, Programmed Visions. Software and Memory, Cambridge, MA:
MIT Press, 2011, p. 59. 15 Marianne Van den Boomen, Transcoding the Digital. How Metaphors Matter
14 Adrian Mackenzie, These Things Called Systems. Collective Imaginings in New Media, Amsterdam: Institute of Network Cultures, 2014, p. 33.
and Infrastructural Software, in: Social Studies of Science 33.3 (2003), 16 Ibid.
98 pp. 365–387, p. 366. 17 Ibid.
Screening Bodies

determines the possibilities for the diagnostic screening of

patients’ bodies. In tile mode, individual cross-sectional
images in a series can be displayed simultaneously in a num-
ber of viewports or tiles, thereby creating a synchronicity
similar to a light box hanging that foregrounds comparisons
between individual images and, in particular, between dif-
ferent computer graphic representation options (fig. 4).
In order to avoid the need to set up the arrangement lay-
out manually for each study, the hanging protocol function
has been integrated in recent software applications, such as
Agfa HealthCare’s IMPAX software.

[T]he purpose of a Hanging Protocol is to present spe-

cific types of studies and images in a consistent man-
ner. This can drastically reduce the amount of manual
image arrangement and display adjustment required
from the radiologist or clinician, thus improving overall
operational efficiency.18
4  Screenshot of Agfa HealthCare IMPAX EE GUI. Display of CT scan in tile mode.

The operational efficiency of the hanging protocol is intend-

ed to ensure that the user does not have to start by virtu- immediately delegated to sight with only minimal physical
ally hanging images in order to divide the GUI; instead the action involved by the co-thinking body as opposed to the
software automatically assigns specific types of studies (e. g. hanging practices of the light box.
thorax scans to determine the spread of lesions) to a par- In addition to tile mode, images can also be automatical-
ticular image layout. The software-based shortcut between ly arranged in stack mode. Whereas tile mode emphasizes
diagnostic query, visualization modality and operation in an order based on synchronous juxtaposition, the very name
the GUI may reduce the amount of time invested, and it may stack mode suggests a dimension of depth in this image lay-
be a response to both collective and individual diagnostic out on the screen.
conventions, but it omits the step of getting to grips with
the available image material. The comparison of images is Consequently, tile mode is used for cross-sectional
imaging only to get the ‘gestalt’ of one particular series
or of the entire examination […]. In stack mode, images
18 Agfa Healthcare, White Paper. Enhanced Hanging Protocols, 2012, p. 2,
http://agfahealthcare.com/global/en/main/resources/white_papers/index.
are conceptually placed one on top of each other, like
jsp (accessed January 9, 2015). cards in a deck. Only the image at the top of the stack is 99
 Kathrin Friedrich

visible. This display mode allows clinicians to create a Screen and Screening Operations
mental 3D model of the anatomical structure in which
they are interested.19 Screens – light boxes and computer displays – are not only
technical a priori of diagnostic radiology but also constitute
Stack mode creates the idea that a reconstruction of the possibilities for aesthetic and hence epistemic operations.
cross-sectional images has been piled up on top of each The interplay between screen and screening establishes a col-
other in a stack, and that the user can work through this lectively embedded and individually embodied practice that
stack, thereby reverting back at a conceptual level to the is guided by modes and orders of hanging images. By order-
fundamental idea of tomographic imaging: being able to ing images within the frame of a screen and applying differ-
slice a multidimensional space into flat sections to then ent kinds of instruments and by intersecting these with fur-
re-spatialize it virtually through aesthetic and epistemic ther media-based infrastructures, the screening of patient’s
operations. Stack mode creates the impression of taking a bodies appears to be a routine procedure guided by images.
virtual walkthrough of the represented body volume by the Yet, with the introduction of digital data processing and visu-
operation of scrolling back and forth with the mouse. The alization software the status of screens and images becomes
key characteristic of the symmetrical and synchronous com- contested. While with film-based screening, images and their
parison between different slices, as well as the horizontal order remained relatively static, even the application of digi-
comparisons within a series, is not a strictly linear working tal imaging techniques such as CT, software infrastructures
through of the image series, but rather involves repetitions and their GUIs introduced dynamic and instantaneous tools
and loops that constitute a “differential analysis” of images of hanging and handling images. The computer screen also
and bodies simultaneously.20 Even if scrolling is based on a remains a hardware frame that displays radiological visual-
particular individual routine, it is the dynamics of the imag- izations, but now radiologists themselves need to get to know
es that enables a visually guided questioning and searching. a different layer of screen and screening operations, i. e. the
The radiological finding crystallizes with each forward and GUI and tools of diagnostic software. While light box view-
backwards in the stack, with each software-based repetition ing established a hierarchy between archive clerks who hung
of a cross-sectional plane. In this respect, the interactive the films according to diagnostic requests and radiologist,
simultaneous interplay of the radiological gaze, visualiza- with digital technologies radiologists become users. With
tions and hand opens up the possibility of an epistemic iter- the available software applications, radiological experts
ation of diagnostic findings. are made responsible for structuring images on screen and
having the necessary tools at hand by knowing the possible
19 Adrian Moise, Designing Better User Interfaces for Radiology Interpre- operations that a certain software application offers. Screen-
tation, Dissertation, School of Computing Science, Simon Fraser Uni-
versity, 2003, p.  34, http://citeseerx.ist.psu.edu/viewdoc/download?- based actions are now streamlined within a GUI that is part
doi=10.1.1.71.8788&rep=rep1&type=pdf (accessed January 9, 2015). of a software which requires an operational knowledge of
20 Amit Prasad, Making Images/Making Bodies. Visibilizing and Disciplining
through Magnetic Resonance Imaging (MRI), in: Science, Technology and
its own, even if established analog routines were meant to
100 Human Values 30 (2005), pp. 291–316, p. 292. be predetermined in digital code.
Screening Bodies

Figures
1  McGovern Historical Center, Gift of Dr. Luther Vaughn, Photo Files,
Medical Equipment and Apparatus, Hermann Hospital X-ray equipment,
1953, https://mcgovernhrc.wordpress.com/2014/11/26/hermann-hospital-
radiology-department/ (accessed March 29, 2018).
2  Medicshots, Alamy stock photo, Alamy.com (accessed in March 20,
2018). 
3  Choia, iStockphoto.com, Photo-ID: 533334108.
4  Reprinted with permission of Agfa HealthCare.

101
Luci Eldridge

Working on Mars
An Immersive Encounter through the Screen

Lost in the kaleidoscopic colours captured by the Hubble NASA’s rovers provide viewpoints through which we regard
Space Telescope, lured in by the deathly blackness of Comet this alien terrain – windows upon unknown worlds, these
67P as represented by Rosetta, and straining to make out images bridge a gap between what is known and unknown,
Pluto through the first images taken by New Horizons, between what is visible and invisible.
we are awed by such vastness and intangibility captured This is a planet being explored remotely; data is sent
within the confines of our screens. Society seems to have back to Earth, examined and reconstructed into different
a fascination with things beyond the realm of perceptual visualisations, allowing for new commands to be sequenced
understanding; presenting us with scenes that we empiri- and uploaded, transforming image into action. Through
cally know nothing about, the image of a faraway planet on images, scientists and engineers make daily decisions on
the NASA website is a mysterious one, far more fascinating how to operate the rovers remotely; data gathered from
than our immediate setting. As viewers of such images we images is key to constructing visualisations of the terrain.
are reliant on space agencies like NASA and ESA to provide Scientific experiments and rover drive paths are simulated
us with pictures and information so that we may explore before being acted out by the real rovers on Mars. This is
these other worlds from the safety and security of our com- done in a number of ways, and the focus of this essay is one
puter screens, smartphones or tablets. Satisfying the public of the most current means scientists are using to explore
need for images they reflect humanity’s ancient impulse to three-dimensional visualisations of Mars: NASA and Mic-
explore, to discover places with their own eyes, and if not rosoft’s OnSight project. Using an augmented reality headset
our own, then those of our machines. to enable a more immersive encounter with an alien land-
This essay is drawn to one particular planet: Mars. scape, OnSight is an example of how screens are transmut-
More spacecraft, landers and rovers have been sent to Mars ing from the stationary to the mobile, from the two-dimen-
than any other planet and as such it is the most imaged (and sional to the spatial and temporal.
arguably imagined) otherworldly landscape. The landscape
is ostensibly familiar; comparable to the deserts of the
A Window onto Mars
American West, the plains of Chile, and the rugged land-
scapes of Iceland, we have an intuitive understanding as NASA’s Curiosity is the most recent rover sent to explore
to how it might feel to walk across its surface. The eyes of Mars (landed in 2012), and there are a whole host of other
 Luci Eldridge

spacecraft orbiting the planet, with a number of missions colour mast cameras) are located roughly 1.97m from the
planned for the near future. For many of the scientists work- ground, just above human eye height. The cameras then are
ing on Mars exploration, rovers are the next best thing to the eyes of the rover, and in turn an extension of our vision.
being there ourselves; stereo-vision, arms carrying instru- Clancey argues that it is through the image that scientists
ments and wheels that enable movement over long distanc- can experience Mars in the first person. When looking at a
es, the rovers facilitate what computer scientist William photograph of Spirit’s tracks in the sand, scientists liken the
J. Clancey has termed “virtual presence”.1 Whilst satellite marks to the scuffing of their boots: “we have been there and
imagery captures a sense of Mars as a whole, depicting we did this. These are our marks – our boots on the ground
the planet as distant and remote, the rovers, being on the of another planet.” 4 As “surrogate explorers” these scien-
ground, offer visions analogous with perception: windows tists become the rovers, referring to aspects of the landscape
onto a world. As Anne Friedberg points out: “We imagine as if they had stood there themselves. 5 By looking upon an
perception to be a kind of photographic view of things, taken image that places the human at the centre of the mediated
from a fixed point by that special apparatus which is called experience, the image makes way for the possibility of a vir-
an organ of perception.” 2 Although of course we do not see tual experience.
as the camera sees (from a fixed point, through one lens, But we are not there behind the viewfinder to compose
with a particular focal length, etc.) the notion that a pic- the image before releasing the shutter, nor are we able to
ture is a kind of window dates back to Alberti’s first modern compare the image with reality once we’ve tapped the cap-
treatise on painting, Della Pittura (1435). The camera, like ture button on our smartphone. We are not, therefore, able
Alberti’s gridded veil, is a mechanical means of translating to verify first-hand the referent these images signify. Jim
subject into image. Discussing the camera, art historian Bell, the Panoramic Camera’s lead investigator for NASA’s
Martin Kemp argues that imaging machines and the way Spirit and Opportunity rovers, states that “the relation to
data is presented is always linked to the eye because “it is reality is a particularly strange one” for Mars exploration;
the human visual system that initiates any kind of photo- the images captured by the rovers are not “abstract” but
graphic activity, […] the end product is rigged to work within they do not represent “a reality that any human has quite
the parameters of our sight, and […] images are irredeemably witnessed yet, either”.6 For Roland Barthes, writing about
subject to our ways and habits of seeing in all their variabil- analogue photography in 1980, the image always contains
ity.” 3 The cameras on Curiosity’s mast (the right and left the referent; the reflected light physically alters the surface
black and white navigation cameras and the right and left
Clancey 2012 (as fn. 1), p. 103.
4
William J. Clancey, Working on Mars. Voyages of Scientific Discovery
1 William J. Clancey, Becoming a Rover, in: Sherry Turkle (ed.), Simulation
5
with the Mars Exploration Rovers, Cambridge, MA: The MIT Press, 2012, and its Discontents, Cambridge, MA: The MIT Press, 2009, pp. 107–127,
pp. 59–60. p. 114. Clancey also explores the use of such phrases as “where are we going
Anne Friedberg, The Virtual Window. From Alberti to Microsoft, Cambridge,
2 to go” or “are we going to stay here?” to explore how each scientist projects
MA: The MIT Press, 2006, p. 142 [original emphasis]. themselves onto and into the body of the rover. Ibid., p. 115.
Martin Kemp, Seen Unseen. Art, Science and Intuition from Leonardo to the
3 Jim Bell, Postcards from Mars. The First Photographer on the Red Planet, New
6
104 Hubble Telescope, Oxford: Oxford University Press, 2006, pp. 268–269. York: Penguin Publishing Group, 2006, p. 3.
Working on Mars

of the negative and the image becomes indexically linked blue anaglyphs are used to get a sense of the topography,
to the object it represents, signifying its presence in some and 3D terrain models are constructed from stereoscopic
past moment.7 Mars is just over 225 million kilometres away data to simulate drives and build command sequences. The
and data – captured digitally – can take between 4 and 24 colour of images is manipulated by scientists to draw out
minutes to reach Earth.8 Barthes’ referent then is completely geological features and chemical compositions in a practice
invisible to the naked eye. The distance between Earth and of working with images to actively reveal what would oth-
Mars reduces our experience of the planet to two dimen- erwise remain unseen.9 Each virtual manifestation of Mars
sions, and the screen becomes an impenetrable boundary – becomes a screen through which a form of invisible vision is
both physically and metaphorically – that more immersive enabled. But this is a landscape humans have yet to witness
visualisation technologies are trying to break. We expe- first-hand. Distance, the unknown and the impenetrable
rience Mars through multiple forms of images; black and lie at the very heart of these images. These predominantly
white and colour composites standing in for an impossible two-dimensional windows often fail to offer their viewers
experience. These images are viewed principally on com- a more intuitive grasp of scale, distances and the rover’s
puter screens, but the image also becomes another kind overall context on the surface. And this is where recent
of screen, a mediator that is imbued with calibration pro- developments in the field of virtual reality are triumphing.
cedures, technological limitations and the science-driven Although scientists at NASA Ames have been working
demands of NASA. And yet images are a vital part of the pro- on head-mounted displays (HMDs) since the mid-1980s, the
cess when it comes to gaining a more immersive understand- tech world has recently witnessed a surge in developments
ing of the rover’s surroundings, enabling scientists and engi- in this field, particularly in the consumer market. Based
neers to make discoveries or decisions on where the rover on knowledge of stereoscopic vision, the left and right eye
should drive next. Panoramic visualisations for instance views of a virtual environment are projected into either eye
give a 360° view of the terrain, whereby the rover (and thus to give the illusion that the user is experiencing the image
the human viewer) is placed at the centre of the image. Red/ space in real life. Virtual environments may be fully immer-

Barthes was referring to un-manipulated analogue photography: “It is often

7
said that it was painters who invented Photography (by bequeathing it their These immersive forms of images were explored in my 2017 PhD thesis
9
framing, the Albertian perspective, and the optic of the camera obscura). I titled Mars, Invisible Vision and the Virtual Landscape. Immersive Encoun-
say; no, it was the chemists. For the noeme ‘That-has-been’ was possible only ters with Contemporary Rover Images. The thesis offered a new understand-
on the day when a scientific circumstance (the discovery that silver halo- ing of human interaction with a landscape only visible through a screen, and
gens were sensitive to light) made it possible to recover and print directly explored how contemporary scientific imaging devices aim to collapse the
the luminous rays emitted by a variously lighted object. The photograph is frame and increase a sense of immersion in the image. Arguing that these
literally an emanation of the referent. From a real body, which was there, representations produce inherently virtual experiences, their transportive
proceed radiations which ultimately touch me, who am here.” Roland Bar- power was questioned, highlighting the image as reconstructed – through
thes, Camera Lucida. Reflections on Photography, London: Vintage, 2000, the presence of a glitch, illusion is broken, revealing the image-as-image.
pp. 80–81. The research re-examined scientific forms of images against examples
The time it takes signals to reach Mars depends largely on the position of
8 from the history of visual culture (be it art or popular culture) to draw
Mars in relation to Earth with the minimum time delay being 4 minutes, parallels between different ways of seeing, representing and discovering
the maximum being 24 minutes. the unknown. 105
 Luci Eldridge

state: “images, even 3D stereo views, lack a natural sense

of depth that human vision employs to understand spatial
relationships […]. [OnSight] provides access to scientists
and engineers looking to interact with Mars in a more nat-
ural, human way.” 11 This technology is being heralded as an
immersive means to explore Mars from a scientific perspec-
tive, allowing scientists to plan which areas of the landscape
they would like to investigate, image and drill, but OnSight
is also being used to explore data from previous Martian
days (sols) in more detail. In addition, scientists from all over
the world can explore the data together; each scientist has
their own avatar within the virtual environment that the
other users can see. The avatar’s gaze ray (a line of coloured
light emanating from the avatar’s eyes) enables other users
to see where they are looking and each user is able to lay
flags to pinpoint areas of interest or possible spots for fur-
1  Screen view from OnSight, January 21, 2015.
ther exploration by the real Curiosity on Mars (fig. 1).

sive (virtual reality) or virtual objects may be overlaid onto

Navigating Mars From the Centre of the Image
real-life surroundings (augmented reality). Head-tracking
technologies enable the user to move about in real space and The OnSight software constructs a three-dimensional envi-
experience environments as if they were really there, plac- ronment from MastCam and NavCam images, together with
ing the viewer at the very centre of the visual experience. satellite imagery taken by the HiRISE camera on the Mars
In 2015 NASA and Microsoft launched OnSight, which Reconnaissance Orbiter.12 This technology is being used
uses Microsoft’s HoloLens (an augmented reality headset) by both NASA scientists and those working with the data
to display a virtual environment constructed from data cap- at planetary imaging facilities and academic institutions
tured by Curiosity. As a current screen-based medium that
synchronises image, action, and space on the spot, OnSight
11 Ibid.
is being used to explore Curiosity’s images in more detail, 12 Project Manager for OnSight Jeff Norris explains: “The 3D reconstruction
enabling scientists to “work on Mars”.10 As NASA/JPL is created via a terrain processing pipeline developed by my team that takes
as input the stereo images acquired by the rover’s cameras. The pipeline
extracts range information by using a process called stereo correlation, and
then uses that range data to build a 3D model of the shape of the terrain
10 NASA/Jet Propulsion Laboratory, NASA, Microsoft Collaboration Will called a ‘mesh’. The mesh is then coloured using texture maps that are also
Allow Scientists to ‘Work on Mars,’ http://jpl.nasa.gov/news/news.php?- derived from the images.” Jeffrey S. Norris (Founder and Director of JPL
106 feature=4451 (accessed December 14, 2016). Ops Lab, NASA, California), email message to author, March 26, 2015.
Working on Mars

around the world. The OnSight units access the data via
the internet by connecting to servers at the Jet Propulsion
Laboratory (JPL) in Pasadena, California. When the rover
moves to a new location, the scientists are emailed notifi-
cations about a new scene, which automatically downloads
when the headset is switched on.13
The kinds of promotional images that accompanied
the announcement of OnSight in January 2015 portrayed
an immersive experience whereby the user was able to
walk across the surface of Mars (fig. 2). However, the actu-
al experience of OnSight is fairly different; as the software
combines NavCam and MastCam images, the virtual envi-
ronment is a patchwork of colour and black and white. Upon
donning the headset and clicking through to the Mars data-
set what you see is essentially only a kind of window onto
Mars; the screen has an aspect ratio of 16:9 and it takes up
2  Erisa Hines, a driver for the Mars Curiosity rover, based at JPL, talks to participants in Destination: Mars, March 30, 2016.
the centre of the user’s vision. Unlike virtual reality where-
by you are totally immersed in a simulation, augmented real-
ity overlays the virtual and the real: peripheral vision (any- is revealed behind a window. As the user walks about in
thing outside of that screen to the right, left, bottom or top) real space, so too does the perspective through the window
is taken up by real-life surroundings. Although this might at change: through the image-as-screen Mars can be seen from
first be seen as a limitation, Research Associate in the Earth different viewpoints.
Science and Engineering Department at Imperial College Although this tool is not being used by engineers respon-
London, Dr. Steven Banham, states that an awareness of the sible for driving the rovers on Mars, there are similarities
user’s real surroundings prevents them from tripping up between how OnSight constructs a three-dimensional envi-
and counteracts the feeling of nausea so often experienced ronment and how tools used by JPL engineers model the
with fully immersive HMDs.14 As such, the technology can terrain through stereoscopic data captured by the rover’s
be used for prolonged periods of engagement. As the user cameras. Due to the time delay, driving rovers around the
moves their head or rotates on the spot, a three-dimensional surface of Mars in real time is not an option. Instead com-
rendering using photographic data of the Martian landscape mands are uploaded on a day-to-day basis. The data from the
previous sol is analysed by science and engineering teams
13 Dr. Steven Banham (Postdoctoral Research Associate, Earth Science and and decisions are made on where the rover should drive next
Engineering Department, Imperial College London), interview by author,
London, December 13, 2016.
and what experiments it should undertake. Commands are
14 Ibid. written and uploaded at the end of each Earth day and the 107
 Luci Eldridge

rover carries out these instructions, beaming back its data

for the following day, when the whole process starts again.
Although scientists return to images taken days, months or
even years before, engineering decisions on where to drive
the rover are made daily using the most recent data and a
different set of tools aid this process.
3D visualisation software is used by rover drivers at
JPL to assess the traversability of Curiosity’s surroundings
from stereoscopic data captured by the navigation cameras,
the hazard avoidance cameras and the mast cameras. The
multiple programmes used to drive the rover are part of a
suite of applications called the Rover Sequencing Visuali-
sation Programme, or RSVP. RSVP takes the stereoscopic
image data and automatically produces polygon meshes of
the terrain. The programme allows drivers to see the r­ over’s
3  Terrain model of Curiosity’s surroundings shown in RSVP.
positioning in relation to hazards and holes in image data
which are seen as potential obstacles (fig. 3–4). The rover
drivers study images and analyse the terrain models, simu- with images and 3D models allows teams at JPL to convert
late drives over them, and when they feel comfortable with image (simulations of the drive paths) into action (in reality
the planned traverse, upload the command sequence to the on Mars). In the models, the rover’s seeable surroundings
rover along with instructions from the science team. are represented by the black and white image data – these
These models are used alongside the raw images to images are draped over the underlying polygon mesh, filling
provide the rover’s up-to-date location (fig. 5); they are in only what can be seen from the rover’s central viewpoint.
interactable terrain models that display the rover in the Any holes in image data created by a ridgeline, rocks, or the
context of its seeable surroundings. Such modes of visuali- rover itself are represented in Martian brown – here the
sation generate what sociologist of science and technology polygon mesh is approximated. Rovers are never driven into
Janet Vertesi terms as an “immersive view of the Rover’s these spaces because they are unknowns, and for rover driv-
environment”, “draw[ing] Mars as [a] tangible, interactable ers these blind spots are as important as the terrain that can
terrain, and allowing engineers to conjure up the sense of be seen. The images then help to define and determine the
‘being there’ virtually”.15 In this case, the practice of working possibilities and the range of operations. Writing on vision
and perception in 1945, Maurice Merleau-Ponty asks how
15 Janet Vertesi, ‘Seeing Like a Rover’. Images in Interaction on the Mars we should experience the existence of absent objects, and
Exploration Rover Mission, Dissertation, Cornell University, 2009, http://
hdl.handle.net/1813/13524 (accessed March 15, 2014), p. 260, p. 262 [original
how we should experience the nonvisible parts of present
108 emphasis]. objects. “Should we say,” he asks “that I represent to myself
Working on Mars

4  Elevated terrain model of Curiosity’s surroundings shown in RSVP. 5  Photograph of Curiosity image mosaic shown on-screen in RSVP.

the sides of this lamp which are not seen?” The unseen sides your eyeball, also means effectively shifting your blindness,
of the lamp are anticipated, according to Merleau-Ponty, “as your own relative blindness.” 18 In the RSVP terrain-mapping
perceptions which would be produced” upon movement.16 tool we see according to blindness, distinctly aware of what
The terrain model in RSVP highlights the gaps in image is missing. Blindness pushes up from underneath the terrain
data, what cannot be seen by the camera’s lens. Unlike model, giving material form to that which remains invisible.
Merleau-Ponty’s lamp, the Martian landscape is not a space We cannot fully perceive the surrounding landscape, not
we can reach into or around to touch: we cannot physical- through Curiosity’s eyes or our own. Rover drivers can sim-
ly move through the landscape to reveal the invisible. The ulate Curiosity’s traverses through the virtual model, but by
visible landscape is framed by the flat colour of the polygon doing so they cannot reveal any more than what is already
model, showing the “limit of visibility”.17 there. Curiosity is at the centre of the image. RSVP allows
In a similar fashion, Paul Virilio writes on shifting one’s engineers to see what is on the periphery of vision, to rotate,
gaze, one’s vision and one’s blindness: “Shifting your gaze, zoom in and out, and generally gain a more encompassing
whether thanks to the mobility of your head or the mobility of understanding of the terrain they have to navigate, but this
vision is dictated by the rover’s capabilities, the limits of
16 Maurice Merleau-Ponty, The Primacy of Perception, Illinois: Northwestern
University Press, 1964, pp. 13–14.
17 Paul Virilio, A Landscape of Events, Cambridge, MA: The MIT Press, 2000,
p. 38. 18 Ibid., p. 39. 109
 Luci Eldridge

3  Mars Stereo View from John Klein to Mount Sharp, Raw. This 360° anaglyph combines dozens of images taken by Curiosity’s right and left Navigation Cameras on January 23, 25 and 26, 2013 (sols 166,
168 and 169). Photojournal image addition date: April 23, 2013.

its vision – in a visualisation that is, to quote one of JPL’s point which has been given artificial depth. Such images are
rover drivers John R. Wright, only “two and a half D”.19 shown on computer screens, and OnSight attempts to bridge
3D may be achieved through 3D-enabled screens or red/ the gap between an onscreen image and physical experience,
blue anaglyphs which are used to get a more kinaesthetic allowing a more intuitive exploration of landscape. Scien-
sense for the terrain. Yet images like Mars Stereo View from tists spend prolonged periods of time looking at image data
John Klein to Mount Sharp appear as windows (fig. 6); a view- and editing it to produce certain results. Before OnSight,
what was lacking was the ability to physically move about
19 John R. Wright (Data Visualisation Developer IV at Jet Propulsion Labora-
within the landscape, to inspect a particular rock, or to walk
110 tory, California), interview by author, Pasadena, CA, November 3, 2015. about and get a feel for the terrain surrounding the rover.
Working on Mars

But there are still limitations, primarily relating to the fact track the position of the scientist’s head as they move
that the environment can only be constructed through data around in their office, we can show them the views of
captured by the rover. Mars that they would have if they were moving in the
Donning the OnSight headset and airtapping through same way on Mars. This is what creates [a] ‘first-per-
the drop down menus to load the scene, the virtual environ- son perspective’.20
ment appears through a window. This window is overlaid
onto real-life surroundings and shifts with the motion of The importance of experiencing the world from within,
the user’s head, always directly in front of their eyes. From with the body as the locus of perception was set out by Mer-
its vantage point Curiosity can image its near surroundings leau-Ponty: “I do not see [a space] according to its exterior
in high resolution, but as with RSVP, seeing behind objects envelope; I live in it from the inside; I am immersed in it.
is not a simple case of walking into the landscape and look- After all, the world is all around me, not in front of me.” 21
ing from a different perspective; the virtual environment is The image is a space in which the objects do not surround
dictated by the rover and its cameras’ stereoscopic reach. us and immersive technologies attempt to deceive us other-
Unlike RSVP, which represents the unseen sides of the wise by enveloping us in the image. With OnSight the body
landscape in a different colour, OnSight’s objective is to is quite literally placed at the centre of the experience; the
increase levels of immersion, so unseen sides of rocks and user must move his/her body to reveal more of the land-
terrain features are estimated, rather than being left blank. scape. Artist and critic Brian O’Doherty describes a similar
The further away from the rover the user gets, the more experience: “[T]he Eye urges the body around to provide
the software has to fill in. As a result, these features appear it with information – the body becomes a data-gatherer.” 22
slightly distorted as the photographic data is stretched over With OnSight, however, our vision and body become
the underlying polygon mesh. As long as the user remains oddly detached; reaching our hand out in front of us – as if
close to the rover, the distortion does not impede a great to point towards something through the window – it disap-
deal on the level of immersion. pears, existing behind the screen within physical and not
Nonetheless, OnSight enables its user to gain greater virtual space. Despite our body being integral to how the
situational understanding for the terrain around the rover, illusion is revealed, the eye is isolated, being the only entity
just as they might if they were there on Mars. Project man- present in the image of Mars. To this end, Merleau-Ponty
ager for OnSight Jeff Norris elaborates on this: discusses the idea of the seer and the visible:

OnSight tries to engage many of the same senses […] without even entering into the implications proper
that a geologist would have when exploring a loca- to the seer and the visible, we know that, since vision is
tion on Earth. A very important sense that OnSight
­engages but a traditional computer monitor does not 20 Norris 2015 (as fn. 12).
21 Merleau-Ponty 1964 (as fn. 16), p. 178.
is ­proprioception, the body’s sense of its own position 22 Brian O’Doherty, Inside the White Cube. The Ideology of the Gallery Space,
and movement. Because we rapidly and accurately Berkeley: The Lapis Press, 1976, p. 52. 111
 Luci Eldridge

a palpation with the look, it must also be inscribed in holds the image of Mars just beyond our grasp. The level of
the order of being that it discloses to us; he who looks immersion for OnSight then is not the technology’s ability to
must not himself be foreign to the world that he looks give a full 360° view of an environment (for Mars to invade
at. As soon as I see, it is necessary that the vision (as all areas of vision) but is in the act of movement to reveal
is so well indicated by the double meaning of the word) the depth of the virtual image.
be doubled with a complementary vision or with anoth- With OnSight there is the definite wow-factor and
er vision: myself seen from without, such as another seductive novelty of new illusions; like the Victorian ste-
would see me, installed in the midst of the visible, occu- reoscope or 3D TV. With new technologies appearing all the
pied in considering it from a certain point […] he who time, perhaps there is something in the ephemeral nature of
sees cannot possess the visible unless he is possessed technologies in re-presenting images of Mars that reflects
by it, unless he is of it.23 our human desire to see ever more clearly and in a more
immersive manner, to get closer and closer to a feeling of
To appropriate another O’Doherty quote, with OnSight touching and being in the landscape, if only – for now at
“the eye is abstracted” from a mobile body “and projected least – on the level of vision. In a sense then, OnSight reveals
as a miniature proxy into the picture to inhabit and test the a deeper, more insatiable desire that lies at the heart of all
articulations of its space”.24 types of imaging; to re-live, re-construct or imagine some-
It is important to note that although OnSight is per- thing that is unseen because of its distance from us in time
haps the most immersive means of experiencing images of and space.
Mars in a screen-based technology that synchronises image,
action and movement, the tool is not without its limitations.
An Immersive Encounter
Being an augmented reality headset, the window onto Mars
appears against our real-life surroundings and as such it This essay concludes with a subjective encounter of OnSight
is not fully immersive. OnSight’s window onto Mars only which took place on December 13, 2016 at Imperial College
allows the user to glimpse a virtual image of Mars. Floating London, and was kindly facilitated by Dr. Steven Banham.
about, occupying a strange space between the user and their The virtual environment encompassed datasets from sols
real surroundings, the screen becomes the frame that with- 1526 – 1547 (November 22 –December 13, 2016). During this
time the rover had been parked for a few days whilst engi-
23 Maurice Merleau-Ponty, The Visible and the Invisible, followed by Working
neers ran diagnostics on the drilling mechanism; as such
Notes, Evanston: Northwestern University Press, 1968, p. 135 [original
emphasis]. the rover was able to image its immediate surrounds in
24 O’Doherty 1976 (as fn. 22), p. 18. Original quote: “One ‘steps’ firmly into high-resolution detail.
such a picture or glides effortlessly, depending on its tonality and colour.
The greater the illusion, the greater the invitation to the spectator’s eye;
the eye is abstracted from an anchored body and projected as a miniature
proxy into the picture to inhabit and test the articulations of its space. For
this process, the stability of the frame is as necessary as an oxygen tank is
112 to a diver. Its limiting security completely defines the experience within.”
Working on Mars

Lowering the headset over my eyes and adjusting the head- within a pixel-thin layer, a three-dimensional image held some-
band I looked through tinted glasses at the office surround- how within a two-dimensional display. This was a virtual open-
ings of the Royal School of Mines at Imperial College London. ing that did not require a click or swipe of the finger to reveal
A window slotted down into view. With an almost opaque but what lay beyond the borders. Here I was present virtually in
luminous translucency this window was hard edged and glow- the image, a presence that relied on my own physicality; the
ing against the dull grey of the real office carpet and surround- position of my head in relation to my body. A three-dimensional
ing white walls. But unlike Alberti’s fixed veil this window was image of Mars that I was in control of revealing.
mobile, almost fragile. Floating and glimmering the window Revealing. The act of revealing coincided with the act of con-
followed the motion of my head, persistently present within cealing. Movement enabled me to penetrate the environment
my direct field of vision, in front of and against, yet within the contained within the image, but did not allow me to bypass
office interior. A screen which was simultaneously a window, the screen. Movement revealed depth but concealed width.
appearing only for me. A personal portal out onto Mars. The window could not be enlarged, the frame could not be col-
The screen flickered and the laying out of a polygon mesh lapsed, the image-as-screen could not be stepped into.
announced the forthcoming emergence of landscape. The ter- I looked upon markers in the landscape, upright poles that
rain began to materialise, expanding outwards rapidly from my marked where Curiosity had been and for how long. My gaze
immediate surroundings and into the distance, a patchwork of lingered and the rover’s path became illuminated, snaking
greys and Martian browns in high and low resolution. Revolving through the landscape from one point to the next. A glowing
on the spot I looked out towards the mountainous rim of Gale path into the past of a landscape it would not see again. Upon
Crater; a dusty grey in the distance, offset against a shimmer- walking towards this point in the landscape I revealed the
ing soft pink sky. depth of image, a depth of space, a depth of time.
As I knelt down to examine a portion of the ground, the window And yet I was not limited to walking alone, nor to the four walls
shrank in size. Zooming in physically and virtually I saw cracks of the office. I could reach the outer edges of Curiosity’s vision
and crevices in the rocks, the strata in the bedrock, granules of through teleportation. Speeding back through time, through
sand and tiny pebbles. As I reached out to touch and feel the space, and into the reconstruction of Mars-as-image.
surface under my gaze my hand evaporated, my body belong-
ing to a space exterior to my vision.
As I stepped back Curiosity flickered into view. The large immo- Figures
bile body of the rover was coated in a thin film of dust, trapped 1  NASA/JPL-Caltech, http://jpl.nasa.gov/news/news.php?feature=4451
here, in the virtual image of Mars. As I advanced forth in an (accessed December 14, 2016). 

attempt to inspect its wheels, Curiosity vanished. In an instant 2  NASA/JPL-Caltech/Microsoft, http://jpl.nasa.gov/news/news.

php?feature=6220 (accessed December 14, 2016).
I became the rover, seeing the surrounding terrain from its van-
3–5  JPL-Caltech, Photo: Luci Eldridge.
tage point, its body merged with mine.
6  NASA/JPL-Caltech, http://photojournal.jpl.nasa.gov/catalog/PIA16847
As I walked backwards once more I looked out towards the (accessed March 15, 2014).
distant horizon. The environment appeared perversely trapped 113
Nina Franz and Moritz Queisner

The Actors Are Leaving the Control Station

The Crisis of Cooperation in Image-guided Drone Warfare

The Ground Control Station in Remote Warfare1 political operation  – the remote execution of a military
command to kill.
Under the condition of extreme remoteness, as is facilitat- Due to their remote setting, drone operations struc-
ed by computer-aided and robotic weapon technologies, turally rely on visibility and controllability via a complex
military interventions, both geographically and in relation system of sensors, control instruments, software inter­
to the laws of war, have become increasingly “limitless”.2 faces, and transmission technologies. In such cases, know­
Paradoxically, remote military interventions are, by their ledge of a situation in an area of operation, the situational
very nature, restricted to the closed system of computerised, awareness, is based mainly on operative images in the form
heavily mediated environments of control. The potential for of visualised sensor data, but even where images form the
crisis in the cooperation of human and non-human actors primary, often sole, basis for action, the analysis must not
is nowhere more apparent than in the ground control sta- be narrowed down to the level of depiction. The situation in
tions (GCS) in remotely-controlled drone warfare. A GCS which the images are being applied must also be considered.
provides image-guided control over the deployment of Against the background of an expanded understanding of
so-called unmanned weapon systems. It forms the central what constitutes a military operation, we will discuss the
operative unit for decision-making and action in remote ground control station as a site of image-operations in con-
warfare by linking human perception to the sensor techno­ temporary warfare.
logy of the drone. It is, thus, a crucial component of a control
setting, where questions of agency culminate in a d ­ istinctly
Contact Zones of Sensing
The terminology of unmanned aerial vehicles is misleading
This text first appeared in German in: Johannes Bennke, Johanna Seifert,
1
Martin Siegler, Christina Terberl (eds.), Prekäre Koexistenz, Paderborn: in the context of remote-controlled military interventions,
Fink, 2018. The authors would like to thank Timothy Cullen for the oppor- in that both the infrastructures for guiding and controlling
tunity for critical discussion, Deborah Curtis for the translation, and Habib
William Kherbek for editing. drones at a distance and the decisions underlying the actions
Derek Gregory, The Everywhere War, in: The Geographical Journal, 177, 3
2 are the result of intricate cooperation between a large
(2011), pp. 238–250; Caroline Holmqvist-Jonsäter, War as Perpetual Polic-
ing, in: Caroline Holmqvist-Jonsäter, Christopher Coker (eds.), The Char-
acter of War in the 21st Century, London/New York, 2010, pp. 103–118.
 Nina Franz and Moritz Queisner

­ umber of people in a situation of shared r­ esponsibility, 3 as

n damental problem of military intervention and, arguably, as
well as the complex technological processes that could be one of the most extreme means of political agency.
said to facilitate those actions. While traditional aviation, Analysing the use of the GCS, as illustrated in the
even if cockpits have become computerised and automa- available literature, and based on conversations with US
tised, still demands a comparatively high degree of inde- Air Force pilots, we want to highlight the ways in which
pendence on the part of pilots, the operational structure for the conditions for human-machine cooperation refer to the
the deployment of drones is essentially based on the com- circumstances of their design, and, thereby, to the tightly
munication between very different actors, such as cameras, interwoven political, scientific and economic relations that
relay stations, pilots, ground troops, military lawyers, data extend far beyond the dedicated technological processes of
analysts and imaging specialists.4 control. In doing this, we seek to reach beyond a description
Marie-Luise Angerer has highlighted the fact that of “distributed agency” and “chains of operations” to put the
the “the zones of contact between the interior and exterior emphasis on the question of how the distribution of control
domains of sensing can be understood to be simultaneous is implemented and negotiated, i. e., on which parameters it
processes of connection, disruption, and translation”, which is based, and where specifically human agency is distribut-
generally proceed with a high degree of friction and conflict ed in these human-machine configurations.6 In focusing on
and are ultimately shaped by the “radical in-translatability the moments of crisis within this cooperative process, we
(of ‘sensing’ and ‘sense-ability’)”. 5 Accordingly, the contact moreover seek to shed light on the apparent methodological
zones between the human sensorium and machine-based reduction that comes with the terminological symmetrisa-
sensing can be investigated as spaces in which some of these tion of human-technological co-existence which pervades
sources of friction and conflicts come to the surface while not only the discourse in current cognitive science and
others disappear entirely behind the surfaces of the inter- media theory, but, also, military policy and action. Thus, we
face design and are rendered imperceptible to the operators. hope to return the focus to the politically pertinent – and
As a particular contact zone of sensing, the CGS points to explicitly human – interests invested in these processes.
the question of human-technological co-existence as a fun-

As Lucy Suchman and Jutta Weber have argued in their theorisation of

6
Up to 200 people are required to run an aerial patrol with Reaper or Pred-
3 the use of drones in a military setting the position that in a situation of
ator drones. In addition to the crew and technical personnel, this includes shared “human-machine autonomy”, agency should not be thought of as
data and image analysts. See also Derek Gregory, From a View to a Kill. something pertaining specifically to humans or machines, but should be
Drones and Late Modern War, in: Theory, Culture & Society 28.7–8 (2012), treated as a conglomeration of “effects of specific human-machine config-
pp. 188–215, p. 195. See also M. C. Elish, Remote Split. A History of US Drone urations”, the analysis of which requires the careful setting of a framework,
Operations and the Distributed Labor of War, in: Science, Technology & a “cut in the network”, that allows a certain unit in the expansive system
Human Values 42.6 (2017), pp. 1100–1131. of agency to be rendered comprehensible, and its specific entanglement
More detail in ibid., pp. 195–197.
4 to be identified. Lucy Suchman, Jutta Weber, Human-Machine Autono-
Marie-Luise Angerer, Ecology of Affect. Intensive Milieus and Contingent
5 mies, in: Nehal Bhuta, Susanne Beck, Robin Geiß, Hin-Yan Liu, Claus Kreß
Encounters, translated from German by Gerrit Jackson, Lüneburg, 2017, (eds.), ­Autonomous Weapons Systems. Law, Ethics, Policy, Cambridge, 2016,
116 p. 46. pp. 75–102, p. 91.
The Actors Are Leaving the Control Station

1  Ground Control Station, external view, Holloman Air Force Base. 2  Pilot and sensor operator in a Ground Control Station, Holloman AFB.

The Ubiquity of Operations information, as well as software applications for chat and
email, in addition to other mission data, such as maps, com-
The GCS is composed of multiple workstations that are mand and control options, and the warning system.
usually installed in a mobile shipping container (fig. 1). The The pilots guide the aircraft and control the weapons
configuration of hardware and software in the central con- system while the sensor operators control the sensor sys-
trol unit defines the options of visual access to the combat tem and are responsible for targeting. The mission intelli-
zone. The members of the crew are referred to generally as gence coordinator prepares the deployment of the aircraft
operators, and are then subdivided into pilots and sensor and provides support in the analysis and interpretation of
operators on the one hand, and a mission intelligence coor- the incoming sensor data, their comparison with external
dinator overseeing the procedure on the other. Pilots and sources, and coordination with ground forces, lawyers,
sensor operators sit in front of multiple vertically and hori- and superior officers. The information exchange between
zontally arranged monitors (fig. 2). These provide a range of individual actors happens through chat clients and audio
options for visualising the sensor data from the multi-spec- link, which connect the crew to the other participants in
tral targeting system7, an array of different sensors that is the so-called remote split operations.8 Military research
attached to the body of the drone, also called the sensor ball. on the control and navigation of unmanned aircraft con-
Furthermore, the screens show air traffic and cartographic

Mark C. Elish, Remote Split. A History of US Drone Operations and the

8
The Multi-Spectral Targeting System is composed of an infrared sensor, an
7 Distributed Labor of War, in: Science, Technology & Human Values 42 (2017),
image amplifier, a daylight camera, a laser marker, and a laser illuminator. pp. 1100–1131. 117
 Nina Franz and Moritz Queisner

ceives the relationship between human and machine as an between humans and machines is taken as a given.13 The
immersive synthesis, in which processes relating to action, way in which these approaches contribute to clarifying
decision-making, and perception are realized in coopera- the precarious constellation of the actions of humans and
tion with humans. Correspondingly, military operations are machines in the context that concerns us here can only be
increasingly understood as a convergence between human elucidated based on the concrete observation of such oper-
users and technical processes at the interface between ations through relevant practices with attention paid to the
senses, sensors and computational processes. In cultural terminology of operation in a military context as well as in
and media theory this form of entanglement of the human current media theory.
and non-human is commonly framed by terms such as cul- The US military regards any military action that serves
tural technique 9, operational chain 10, a priori11 and hybrid 12. the purpose of achieving a planned objective or military
The debate about human machine cooperation has recently mission as an operation.14 Military operations are increas-
been expanded by the inclusion of older approaches from ingly understood as cooperation between automated and
various disciplines, in which the symmetrical relation (partially) autonomous technologies and the human oper-
ating crew, whose agency and decision-making abilities
are placed in a precarious relationship with the efficien-
cy of technical systems. Somewhat oblivious to current
9 See Bernhard Siegert, Cultural Techniques. Or the End of the Intellectual
Postwar Era in German Media Theory, in: Theory, Culture & Society 30 media-theoretical debates on the recognition of non-human
(2013), pp. 48–65; Thomas Macho, Christian Kassung (eds.), Kulturtechni- actors, in the military and technological discourse, agency
ken der Synchronisation, Munich, 2013, pp. 16–18; Erhard Schüttpelz, Die
medienanthropologische Kehre der Kulturtechniken, in: Lorenz Engell, is readily understood as co-agency, and the human actor is
Bernhard Siegert, Joseph Vogl (eds.), Archiv für Mediengeschichte 6. Kultur- regarded as an “element” 15 or “component” 16 of an operative
geschichte als Mediengeschichte (oder vice versa?), Weimar, 2006, pp. 87–110.
10 See André Leroi-Gourhan, Gesture and Speech, translated from French by
system. Against this background of the military understand-
Anna Bostock Berger, Cambridge, MA, 1993, pp. 219–235; as well as the ing of operation, we will discuss the ground control station
texts referred to in fn. 3. For a critique of the concept of operativity in
as a site of image-operations in contemporary warfare. To
current German media theory, see Dieter Mersch, Kritik der Operativi-
tät. Bemerkungen zu einem technologischen Imperativ, in: Dieter Mersch,
Michael Mayer (eds.), Techne/Mechane. Internationales Jahrbuch für Medi-
enphilosophie 2 (2016), pp. 31–52. 13 Among them Human Factor Studies, Work Place Studies or Science and
11 See Friedrich Kittler, Manfred Schneider, Editorial, in: Friedrich Kittler, Technology Studies. See: Erhard Schüttpelz, Sebastian Gießmann, Medien
Manfred Schneider (eds.), Diskursanalysen 2. Institution Universität, Oplad- der Kooperation. Überlegungen zum Forschungsstand, in: AG Medien der
en, 1987, pp. 7–11; Lorenz Engell, Joseph Vogl (eds.), Archiv für Medienges- Kooperation (eds.), Navigationen. Zeitschrift für Medien- und Kulturwissen-
chichte 1: Mediale Historiographien, Weimar, 2001; Erich Hörl (ed.), Die schaften 15.1 (2015), pp. 7–54.
technologische Bedingung. Beiträge zur Beschreibung der technischen Welt, 14 See Charles Messenger, Dictionary of Military Terms, published by the US
Frankfurt/M., 2011. Department of Defense, Greenhill, London, Stackpole, 1995, p. 274; see
12 See Bruno Latour, We Have Never Been Modern, translated from French also Jimena Canales, Operational Art, in: Niels van Tomme (ed.), Visibility
by Catherine Porter, Cambridge, MA, 1993; see also Gustav Roßler, Kleine Machines. Harun Farocki and Trevor Paglen, Baltimore, 2015, pp. 37–54, p. 37.
Galerie neuer Dingbegriffe. Hybriden, Quasi-Objekte, Grenzobjekte, epis- 15 Assistant Secretary of Defense for Research and Engineering, Human Per-
temische Dinge, in: Georg Kneer, Markus Schroer, Erhard Schüttpelz formance Training and Biosystems Directorate: Human Systems, http://acq.
(eds.), Bruno Latours Kollektive. Kontroversen zur Entgrenzung des Sozialen, osd.mil/rd/hptb/programs/human_systems (accessed January 8, 2017).
118 Frankfurt/M., 2008, pp. 76–107, pp. 79–82. 16 Ibid.
The Actors Are Leaving the Control Station

date, there is a notable lack of detailed descriptions of the within military strategy, has undergone a further expan-
interactions between drone crews and the systems of con- sion in recent times through the development of the related
trol, communication, and the sensors on which they rely,17 term of operational art. In the Field Manual of US Military
despite armed drones having been deployed by the US Air Doctrine, published in 1999, the term operational art, pre-
Force since the Yugoslav Wars in the mid-1990s, and by now viously used to refer exclusively to military interventions,
have become a cornerstone of US-military strategy.18 was used for actions of war, as well as for all “operations
outside warfare”, including “diplomatic activity, economics
and information”, as well as “political and other non-mili-
Cooperation and the Eroding Boundaries of
tary factors”.20 In our view, this expansion of the doctrinal
Agency
meaning of military operations must be taken seriously, as
The English term operation is ubiquitous in contemporary it refers to a change in the understanding of military actions
military discourse. The US Department for Defense Dictio- and of the actors in the military itself.
nary of Military Terms defines an operation as “[a] military A tendency to group together different military person-
action or the carrying out of a strategic, tactical, service, nel (e. g. pilots, operators of weapon and sensor systems)
training, or administrative military mission; the process under the term operator, previously only used for certain
of carrying on combat, including movement, supply, attack, participants in secret service missions becomes apparent.
defense and maneuvers needed to gain the objectives of any In the light of the numerous other meanings of the word
battle or campaign.” 19 In other words, the term operation operator, such as machine operator, user, or supervisor, mil-
describes all actions required to achieve a military objec- itary actions are correlated to a greater extent with tech-
tive, independent of their hierarchical level or the means nological equipment, without which modern warfare is
to carry them out. inconceivable. The interplay between the human operator
Jimena Canales, a historian of science, emphasises that and the increasingly complex and networked technological
the term operation, already very broad in its scope and usage infrastructure is described as a state of “cooperation” and
“collaboration” 21 in which human and non-human actors par-
ticipate on an equal level, leading to an apparent symme-
17 Notable exceptions are Peter M. Asaro, The Labor of Surveillance and
Bureaucratized Killing. New Subjectivities of Military Drone Operators, trisation between human and non-human actors. However,
in: Social Semiotics 23.2 (2013), pp. 1–29 and David J. Blair, Nick Helms, The under the condition of automated technological systems, the
Swarm, the Cloud, and the Importance of Getting There First. What’s at
military discourse characterises this cooperation primarily
Stake in the Remote Aviation Culture Debate, in: Air & Space Power Journal
(2014), pp. 33–52.
18 By now, the number of pilots for remotely piloted aircraft (RPA) exceeds
the number of those operating piloted aircrafts within the US Air Force. 20 Ibid., p. 37.
See Oriana Pawlyk, Drone Milestone: More RPA Jobs Than Any Other Pilot 21 James A. Winnefeld, Frank Kendall, Unmanned Systems Integrated Road-
Position, March 8, 2017, http://military.com/daily-news/2017/03/08/drone- map FY2013-2038, 2013; Robotics Collaborative Technology Alliance /
milestone-more-rpa-jobs-any-other-pilot-position.html (accessed August Army Research Laboratory, FY 2012 Annual Program Plan, 2011, https://
17, 2017). arl.army.mil/www/pages/392/rcta.fy11.ann.prog.plan.pdf (accessed
19 Messenger 1995 (as fn. 14), p. 274. See also Canales 2015 (as fn. 14), p. 37. ­December 7, 2017). 119
 Nina Franz and Moritz Queisner

as a technological process of action and decision-making, not only facilitate human actions within chains of opera-
which the human operator approaches in an increasingly tions via the screen, as in the case of camera-aided remote
passive manner, 22 namely, as a supervisor who monitors control, or in the context of graphical user interfaces, but
autonomous actors. In a different disciplinary context, but they also are attributed to act with a degree of autonomy.
in a somewhat parallel way, the term “operative image” has
recently attracted much attention in contemporary media
Distributed Cognition
studies and cultural theory. In this context the term links
the use of an image to an “object-constituting”, “generative” Thanks to the study The MQ-9 Reaper Remotely Piloted
function of “manageability and explorability” that is under- Aircraft: Humans and Machines in Action 25, carried out by
stood as “operativity”.23 Particularly in the context of auto- Timothy Cullen, a Lieutenant Colonel in the US Air Force,
mated image processing those practices constitute a new between 2009 and 2011 at the Engineering Systems Division
type of image that is defined by the exclusion of the human of the Massachusetts Institute of Technology (MIT), we can
actor.24 From this point of view, images become actors that base our considerations on a relatively detailed – albeit also
heavily redacted and undoubtedly partisan – description of
22 Grundel et al. provide the following definition for “cooperative systems”, the theatre of operations of the MQ-9 Reaper drone. We can
which also include weapons systems: “They have some common elements: 1)
add, for consideration, the perspective of the drone crews
more than one entity, 2) the entities have behaviors that influence the deci-
sion space, 3) entities share at least one common objective, and 4) entities and trainers working for the US Air Force, with whom we
share information whether actively or passively.” The objective of “cooper- held discussions in the jointly organised workshop on the
ative technical systems” in these contexts is no longer solely the revaluation
of non-human actors, but the minimisation of human participation: coop- topic Technology and Expertise in Remote Warfare at Max-
erative systems “capitalize on the availability of various interconnected well Air Force Base in Montgomery, Alabama, in February
resources and on the sharing of key information among the networked enti-
ties with minimal involvement of the operating crew”. Don Grundel, Robert
2017.
Murphey, Panos M. Pardalos, Oleg A. Prokopyev (eds.), Cooperative Systems. Cullen’s study, carried out using methods from Science
Control and Optimization, Berlin/Heidelberg, 2007, preface, without page
and Technology Studies, is a significant document, in that
numbering.
23 Sybille Krämer, Operative Bildlichkeit. Von der ‚Grammatologie‘ zu einer a member of the military who is familiar with the internal
‚Diagrammatologie‘? Reflexionen über erkennendes ‚Sehen’, in: Martina culture of knowledge within the U. S. Air Force provides
Hessler, Dieter Mersch (eds.), Logik des Bildlichen. Zur Kritik der ikonischen
Vernunft, Bielefeld, 2009, pp. 94–123, p. 98. information on the setting of remote warfare. The study not
24 Harun Farocki, on whose investigation of automated image-based naviga- only reveals some of the less known operative processes of
tion techniques the term is based, for example in remote-controlled rock-
modern warfare, but it also, inevitably, provides informa-
ets (Auge/Maschine I–III, 2001–2003 and Erkennen und Verfolgen, 2003),
defined “operative images” as those images that “do not represent an object, tion on the way in which members of the “RPA community”
but are rather part of an operation.” Harun Farocki, Phantom Images, in:
Public 29 (2004), pp. 12–24, p. 17. Volker Pantenburg explains further that
this new type of image is “in no way any longer a ‘separate entity’ and locat- machines”. Trevor Paglen, Operational Images, in: e-flux Journal 59 (2014),
ed opposite a potential observer, but becomes fully integrated into an elec- http://e-flux.com/journal/59/61130/operational-images (accessed May 7,
tronic technical operation”. Volker Pantenburg, Film als Theorie. Bildfor- 2017).
schung bei Harun Farocki und Jean-Luc Godard, Bielefeld, 2006, pp. 189–234. 25 Timothy Cullen, The MQ-9 Reaper Remotely Piloted Aircraft. Humans and
120 Trevor Paglen calls operational images “images made by machines for other Machines in Action, Cambridge, MA, 2011.
The Actors Are Leaving the Control Station

wish to be perceived, how they reflect on their own position apparent, each of which, in their own way, can be traced
within the scope of military interventions, and what role back to their origin in the systems thinking of cybernetics. 31
they claim for themselves in this process. Similar to Hutchins, Cullen follows the navigational
Cullen describes his work in Human and Machines in control sequences as examples of “socially distributed cog-
Action as an illustration of “how social, technical, and cog- nition” 32 through describing and mapping the actions of a
nitive factors mutually constitute remote air operations in crew in the cockpit. He finds the concept of computation
war”. 26 From a methodological perspective, the study is useful, without which a horizontal description (in contrast
guided both by Bruno Latour’s Science in Action 27 and by to the hierarchical description of a human as the sole actor)
Edwin Hutchins’ 1995 study, Cognition in the Wild 28, with its would not be possible. According to Hutchins, navigation
concept of situated and socially distributed cognition. The takes place based on a sequence of activities, “in which rep-
latter study develops this concept based on the example of resentations of the spatial relationship of the ship to known
pre-modern navigation practices in Micronesia, Hutchins’ landmarks are created, transformed, and combined in such
personal observations gleaned from his time serving on a way that the solution to the problem of position fixing is
the bridge of a US Navy ship, as well as on Lucy Suchman’s transparent”. 33 In his view, this results in a generalised defi-
descriptions from the 1980s of the “use, combination and nition of computation, a very broadly defined concept in cog-
re-representation” of information in so-called “intelligent nitive science, as “the propagation of representational states
machines”, for which she introduced the term “situated across a series of representational media”. 34 This definition
cognition”.29 is striking in its rejection of any clear distinction between
Hutchins’ approach lends itself to the description of the media of the representation, whether these are internal
the military control environment, as humans and things images produced by human imagination, a diagrammatic
are described as participants in one and the same “system” sketch, a map, or a computer-aided model that is shown on a
in both contexts, namely, as participants in “a distributed screen. Defining the process of computation as such an act of
process composed of emergent interactions among people constituted translation allows the nominal reduction of the
and tools”. 30 At this point, the common epistemic roots of friction between senses and sensors, algorithms and human
both the US military discourse on technology and the dis- cognition, decision-making, and programming.
course of cognitive science and media anthropology become
31 See Paul N. Edwards, The Closed World. Computers and the Politics of Dis-
26 Ibid., p. 37 course in Cold War America, Cambridge, MA, 1996. On the cybernetic ori-
27 Bruno Latour, Science in Action. How to Follow Scientists and Engineers gins of the cognitive sciences, see Jean-Pierre Dupuy, On the Origins of
through Society, Cambridge, MA, 1987. Cognitive Science. The Mechanization of the Mind, Cambridge, MA, 2000. On
28 Edwin Hutchins, Cognition in the Wild, Cambridge, MA, 1995; Edwin the history of cybernetics in US military science, see for example Antoine
Hutchins, Understanding Micronesian Navigation, in: Dedre Genter, Albert Bousquet, Cyberneticizing the American War Machine. Science and Com-
L. Stevens (eds.), Mental Models, Hillsdale, NJ, 1983, pp. 191–225. puters in the Cold War, in: Cold War History 8.1 (2008), pp. 77–102.
29 Lucy A. Suchman, Plans and Situated Actions. The Problem of Human-Ma- 32 Hutchins 1995 (as fn. 28), p. xii, xiii and 129, see also Cullen 2011 (as fn. 25),
chine Communication, New York, 1987, as cited in Cullen 2011 (as fn. 25), p. 30.
p. 29. 33 Hutchins 1995 (as fn. 28), p. 117.
30 Cullen 2011 (as fn. 25), p. 29. 34 Ibid., p. 117. 121
 Nina Franz and Moritz Queisner

Building on this idea, Cullen also recognises the drone visualisation technologies, through which operations at a
cockpit as a system in which both humans and automated distance are moved into the sphere of real time.
tools are participating: “[P]ilot, sensor operator, automated Imaging technologies, such as thermographic or elec-
tools, and other elements of Reaper were part of a larger tromagnetic measuring techniques, as well as light- or
computational system that performed in ways specific to the sound-based methods, allow visual and operative access to
environment and circumstances of operation.” 35 However, a situation in the conduct of war. While the pilots of manned
his characterisation of the operating crew can be seen as an aircraft are generally neither in the position to observe a tar-
attempt to separate human action from the automated and get over the longer term, nor of making it visible at a small
technologically defined sequences of action. distance or in high resolution, the visual practices of drone
The processes that are very broadly defined as compu- crews are based on a continuous video stream, which con-
tation by Cullen and Hutchins can be clarified based on the stitutes an important criterion for distinction according to
operations that are essential to remote-controlled warfare. Peter Asaro:
Central to operating the GCS are voice, images, and compu-
tation, these serve to visualise and synthesise complex rela- In most manned combat missions, the target is simply a
tionships, and render them legible on screen – a procedure set of geographic coordinates that were obtained from
that Cullen refers to as “building a picture”. 36 another source, such as soldiers on the ground, an air-
craft or satellite up above, or the outcome of the anal-
ysis of multiple intelligence sources. They also rarely
Vision at a Distance
remain close to a target to observe the consequences of
The deployment of drones for targeted killing, missions in their attack, a task called ‘battle damage assessment’
warfare, and surveillance presents a paradigm for a type that is often given to unarmed surveillance aircraft or
of military intervention that is defined and organised by soldiers on the ground. 37
imaging, sensor and network technologies. It is based on
a configuration of humans and machines that not only per- In contrast, drone operations present a visual practice in
mits seeing without being seen, but also killing in real time which vision becomes a cooperative process. Based on the
without being physically present. On the one hand, this is example of the merging of the visualisation process and the
made possible by the spatial mobility of sensor technologies ability of human vision, Cullen shows how the awareness of
that are ever more independent from human presence, and, the boundary between human and machine is strategical-
thus, become the preconditions for human decision mak- ly eliminated in military training and practice. According
ing. On the other hand, it is based on the almost immediate to Cullen, the operation of the sensor system by the sensor
temporal availability of data provided by transmission and operator will only function successfully when operators
dissociate the ability to see from the presence of their own
35 Cullen 2011 (as fn. 25), p. 32.
122 36 Ibid., p. 117. 37 Asaro 2013 (as fn. 17), p. 14.
The Actors Are Leaving the Control Station

bodies: “Instructor sensor operators taught their students to the presence of human actors.41 Further, they note that this
visualize themselves being on the Reaper aircraft, floating distance implies, above all, the option of the “projection of
above the ground and looking down at their quarry from agency without vulnerability”.42 Soldiers’ lives are not at risk
the belly of the aircraft” 38. The eye takes the place of the during an operation. Additionally, from a military techno-
sensor, negating machine action, but also acting as a sensor logical perspective, a positive cost-benefit ratio is attributed
itself: “A sensor operator’s close relationship with the sensor to surveillance at a distance in comparison to the options for
ball helped them to do their jobs well. Experienced sensor observation by ground troops or manned aircraft.43 To this
operators who ‘flew’ the sensor ball from an 18-inch moni- end, it is predominantly techniques for visualisation that
tor became the machine. They became the eye in the sky”. 39 are listed: the methods for obtaining ISR (intelligence, sur-
This demonstrates the extent to which the work of the veillance, reconnaissance), the argument proposes, produce
crews is dependent upon the production of visibility. Wher- a continuous and ubiquitous visibility that forms the basis
ever imagery intervenes between soldiers and the battle- for a clean, almost surgical conduct of a war.44
field, the interplay of structures and processes, behind and In contrast, critics regard drone technology as no less
in front of the screen, are crucial in order to understand than “the technical – and technological – solution par excel-
how operators act via sensor and imaging technologies. Pub- lence for the political problem of imperial overreach”.45 They
lished video feeds that are based on the sensor data have argue that “death of distance enables death from a distance,”
contributed to making the practice of drone warfare more
visible.40 However, the existence of such material hardly 41 See Peter W. Singer, Wired for War. The Robotics Revolution and Conflict in
the Twenty-First Century, New York, 2009.
reveals what the crews themselves see, and, above all, how 42 David Deptula, The Use of Drones in Afghanistan, CNN Amanpour, Novem-
they saw a given situation. The actors’ remote interaction, ber 24, 2009, as cited in Grégoire Chamayou, A Theory of the Drone, New
York, 2015, p. 12.
their diverging perspectives, and the underlying workflows 43 See Department of Defense, Unmanned Systems Integrated Road-
can only be vaguely surmised, if at all. Even so, the essen- map, FY2011-2036, 2011, http://acq.osd.mil/sts/docs/Unmanned%20
­Systems%20Integrated%20Roadmap%20FY2011-2036.pdf (accessed Jan-
tial arguments that are linked to vision at a distance can be
uary 8, 2017).
revealed based on the debate about these documents. 44 See Conor Friedersdorf, Calling U. S. Drone Strikes ‘Surgical’ is Orwellian
Advocates of drone use emphasise the aspects relat- propaganda, in: The Atlantic 27 (2012), http://theatlantic.com/politics/
archive/2012/09/calling-us-drone-strikes-surgical-is- orwellian-propa-
ed to safety and the minimisation of risk, in particular, as ganda/26292 (accessed January 8, 2017); John O. Brennan, The Ethics
vision and action at a distance do not necessarily require and Efficacy of the President’s Counterterrorism Strategy, Lecture at the
Woodrow Wilson International Center for Scholars, Washington, DC, April
30, 2012, http://cfr.org/counterterrorism/brennans-speech-counterterror-
ism-april 2012/p28100 (accessed January 8, 2017); Harold Koh, The Obama
Administration and International Law, in: Annual Meeting of the American
38 Cullen 2011 (as fn. 25), p. 166. Society of International Law, Washington DC, March 25, 2010, https://gene-
39 Ibid. va.usmission.gov/2010/04/01/obama-administration-international-law/
40 See, for example, projects such as Forensic Architecture (http://­forensic- (accessed July 27, 2017).
architecture.org), Dronestagram (http://dronestagram.tumblr.com) or 45 Laleh Khalili, Fighting over Drones, in: Middle East Report, No. 264; Pivot,
Airwars (https://airwars.org), which visualize the locations and conse- Rebalance, Retrench. The US Posture in the Middle East (2012), pp. 18–21,
quences of drone attacks (accessed March 1, 2018). p. 21. 123
 Nina Franz and Moritz Queisner

and that this “replaces one tyranny of geography with Peter Asaro suggests that most of these arguments miss the
another”.46 One of the ramifications of this that is often crit- point in relation to the actual complexity of drone technol-
icised is that the visual presentation and simulation of the ogies, as they only view the agency of the technology in the
events implies an emotional distance, which leads partici- sense of a simple reaction without considering how the use
pants to dissociate from the consequences of their action.47 of the technology also changes human agency. 50 Derek Greg-
For example, the aesthetics, as well as the software interfac- ory has pointed out that the visually-conditioned spheres
es and control instruments of the GCS, are often compared of action of drone operations are not technical, but tech-
to those of video games; the term, “push-button-war” is no-cultural phenomena, the problems of which can only
frequently used to characterise this phenomenon, in which be revealed through their use. 51 In order to understand the
it is no longer possible to distinguish between simulation actions and decisions relating to the deployment of drones,
and reality.48 Conversely, arguments are promulgated stat- it is, therefore, not only their workflows that must be docu-
ing that the spatial proximity and chronological continuity mented, and their consequences that must be made visible,
created by the real time video feeds moves the participants but, also, the pragmatic conditions of the intervention. In
far closer to events, and, also, renders the consequences of this sense, it appears necessary to question the interplay
actors’ actions visible: between human and technical actors in relation to how this
dynamic engenders or prevents specific forms of vision and
You’re 8,000 miles away. […] But it’s not really 8,000 visibility; this is the key practice of remote-controlled war-
miles away, it’s 18 inches away. […] We’re closer in a fare.
majority of ways than we’ve ever been as a service.
[…] There’s no detachment. […] Those employing the
Crafting a View
system are very involved at a personal level in combat.
You hear the AK-47 going off, the intensity of the voice Based on the example of the live video feed, Cullen describes
on the radio calling for help. You’re looking at him, 18 how the cooperative production of the image in real time
inches away from him, trying everything in your capa- becomes a source of identity for the different actors distrib-
bility to get that person out of trouble.49 uted in space: “The feed is distributed and networked. It is
the product of the aircrew. The crew ties their identity and
their worth to this feed.” 52 He emphasises the importance of
46 Gregory 2012 (as fn. 3), p. 192.
47 James Der Derian, Virtuous War. Mapping the Military-Industrial-Me-
dia-Entertainment Network, Boulder, 2001, pp. 9–10. home-takes-a-toll-on-drone-operators-1.95949#.WYSF0dPyjmE (accessed
48 Rachel Plotink, Predicting Push-Button Warfare. US Print Media and Con- December 8, 2017).
flict from a Distance, 1945–2010, in: Media, Culture & Society 34.6 (2012), 50 See Asaro 2013 (as fn. 17), p. 5.
pp. 655–672. 51 Gregory 2012 (as fn. 3), p. 190.
49 Col. Pete Gersten, commander of the 432nd Air Expeditionary Wing 52 Timothy Cullen, “MQ-9 Reaper Operations and the Evolution of Remote
at Creech Air Force Base, in Megan McCloskey, The War Room. Daily Warfare”, Presentation during the workshop Technology and Expertise in
Transition between Battle, Home takes a Toll on Drone Operators, 2009, Remote Warfare February 1, 2017, Air University, Maxwell Air Force Base,
124 https://stripes.com/news/the-war-room-daily-transition-between-battle- transcript by Nina Franz.
The Actors Are Leaving the Control Station

the chat room, in which members of the crew communicate as predominantly a “product of the crew”. 56 Contrary to
on the content of the feed, in which events are described and the strategic elimination of human-machine differences
interpreted. Vision becomes a cooperative process through that he observes in the training of Reaper crews, human
this interplay between language and image. The produc- actors remain superior to technology for Cullen: they not
tion of the video feed by the crew, as highlighted by Cullen, only recognise the errors and weaknesses of the technology,
implies the involvement of non-human actors, because in but also anticipate these by using their knowledge-based
order to recognise anything on the screen at all, the human experience, and they develop work-arounds to exploit the
observers require context. Context is not provided primarily technology in individual situations.
by the written communication in the chat rooms alone, but This reveals Cullen’s own perspective as an erstwhile
“is facilitated with the help of tactical displays, moving maps Air Force pilot, a tradition into which he attempts to enlist
[...] and chat rooms”. 53 the role of drone pilot. Cullen argues in favour of under-
The cooperative production of the feed as an actual standing drone crews as autonomous, and, thus, decidedly
task of the crew represents a stark contrast to the role of human decision-makers: “They struggled to be human”, he
fighter jet pilots, from whom the US Air Force drone pilots states, and, thereby, locates the problem in the transfor-
distinguish themselves, sometimes in a polemic form, in mation “from automatons and technicians into military
their self-image as “desk workers in flying uniform”.54 While professionals who viewed the interpretation and manipu-
aircraft pilots profess to act on an individual, and accord- lation of the virtual world they created as matters of life and
ingly autonomous basis, the operators of the GCS are co-au- death”. 57 According to Cullen, this battle for the identity of
thors55 in an interactive software environment. However, an acting human subject, who is not simply an element in
Cullen simultaneously positions this creation of visibility the technical process, takes place at the level of interfaces
that are designed by engineers without adequate input from
the users. The design of the control surfaces is revealed to
53 Ibid.
be a point of contention, based on statements by members
54 Workshop participants at Maxwell Air Force Base pointed out that the
requirement to wear a flying suit in the GCS appeared to them to be slightly of the crew:
absurd. On the modification of the use of the classification of “pilots”, and
the term Air Force Operator, Cullen 2011 (as fn. 25), p. 20.
55 In this, the situation in the GCS as opposed to the cockpit shares similarities General Atomics engineers initially designed the air-
with the set-up that Timothy Lenoir and Sha Xin Wei have described as the craft to fly autonomously for the bulk of a mission, but
“operative theatre” of computer-mediated surgery that is marked by the
pilots modified the ground control station and their
“necessary cooperation between human and machine”. The surgical, like the
remote military intervention, is today mediated by a technological infra- procedures to share aircraft control with the autopilot
structure, in which individual operators are “replaced by software-mediat-
ed, machine-human collectivities”. Accordingly, the unified authorship of
the operator/agent is transformed into “co-authorship” within “interactive 56 Cullen, MQ-9 Reaper Operations and the Evolution of Remote Warfare, 2017,
3-D simulations”. Timothy Lenoir, Sha Xin Wei, Authorship and Surgery. handout on the lecture with the same title given in the workshop Tech-
The Shifting Ontology of the Virtual Surgeon, in: Bruce Clark, Linda Hen- nology and Expertise in Remote Warfare February 1, 2017, Air University,
derson (eds.), From Energy to Information, Stanford, 2002, pp. 283–308, Maxwell Air Force Base, p. 13.
pp. 284–285. 57 Cullen 2011 (as fn. 25), p. 119. 125
 Nina Franz and Moritz Queisner

in order to maneuver more quickly and destroy a t­ arget ly possible in the Reaper cockpit described by Cullen, it
at a specific time. […] To make the system work for is improbable that this still applies in current and future
them, Reaper operators determined the time and place control stations in the same way. This is because modern
to use automated tools; avoided modes of operation computer systems can collect and analyse ever greater quan-
known to trigger failure; adjusted to the erroneous tities of data practically in real time and visualise these data
behavior of subsystems and other operators; and trans- on the screen at an increasingly uncircumventable level of
lated data into formats other humans and machines sensory perception; users can only accept or reject in the
could receive, interpret, and evaluate […]. 58 specific application contexts, but are now hardly capable of
comprehending the weight of such data.60
Here, Cullen emphasises the self-reliance of the prospective
drone pilots in the 29th Attack Squadron of Holloman Air
Interpretation and Decision
Force Base and highlights their options for intervention and
sovereignty of action in comparison to weapons systems that The high resolution, real time video feed that forms the basis
are designed for “autonomy”. According to Cullen, their own for action and decisions taken by drone crews amounts to a
expertise in the handling of the control panel allows the use of images that is fundamentally different from previous
members of the crew to intervene in the automated process- military practices in planning, surveillance, reconnaissance,
es and “share” control of the aircraft with the autopilot, such and intervention. The circumstance that both control and
that targets that are being aimed at can be destroyed “more navigation, as well as the visual access to the area of opera-
effectively”. What is notable here is the fact that it is not those tions, are guided by images, implies a structurally different
qualities that enable a secure and reliable assessment that image practice to that suggested by the visual methods and
are emphasised (for example, to identify a legitimate military competencies of traditional military reconnaissance.61
target under the laws of war), but those that have the objec- When visualization practices mediate between drone
tive of ensuring speed of decision. Likewise, “failure” and crews and the area of operations, this not only demands an
“erroneous behaviour” of subsystems and other operators examination of what can be seen and recognised in and on
that are designed to prevent intervention do not primarily
refer to causing collateral damage or civilian victims. 60 Mark Hansen, Feed-Forward, in: Robin Mackay (ed.), Simulation, Exercise,
The drone crew’s practice of “translation” of the data Operations, Falmouth, 2015, pp. 57–61, p. 57.
61 On this point, see Antoine Bousquet, The Martial Gaze. The Logistics of Mil-
produced by computer systems into “formats others could
itary Perception in the Age of Global Targeting, Minneapolis [publication in
receive, interpret and evaluate” 59 that Cullen observes, fol- 2018], therein Chapter 3 Imaging [We would like to thank Antoine Bousquet
lows the definition of computation introduced by Hutchins. for sharing an unpublished version of the manuscript]. On the civilian use of
image-guided navigation techniques, see Manovich, The Language of New
While these forms of human translation were apparent- Media, Cambridge, MA, 2001, therein The Poetics of Navigation, pp. 259–
268; Tristan Thielmann, The ETAK Navigator. Tour de Latour durch die
Mediengeschichte der Autonavigationssysteme, in: Georg Kneer, Markus
58 Ibid., p. 119. Schroer, Erhard Schüttpelz (eds.), Bruno Latours Kollektive. Kontroversen
126 59 Ibid., p. 43. zur Entgrenzung des Sozialen, Frankfurt/M., 2008, pp. 180–218.
The Actors Are Leaving the Control Station

these images. How images facilitate, complicate, or even in future operative scenarios where drones are deployed.
prevent vision and action in an operative process is also Given the newer sensor systems, the commonly cited “view
a subject for negotiation. In this case, the visualisation of through a soda straw” 66, which continues to be opposed
sensor data not only creates static situations for a temporal- to the popular concept of total visibility through omnipo-
ly and spatially subordinate reception, but also essentially tent seeing sensor systems, now hardly seems applicable to
guides and controls the actions and decisions of the crews. the visual practices of drone crews. Based on more recent
Drone operations, therefore, correspond to a new type of studies, human-based visual scanning of the surfaces of the
intervention in which military action is directly guided, or combat zone will hardly form part of the core competencies
misguided, by what and how images depict or obscure. of drone crews in the future.67 Target acquisition and obser-
The interpretation of sensor data in real time now rep- vation is, indeed, still carried out based on restricted fields
resents a significant number of decisions taken during mil- of view, zooms, and camera perspectives. However, sensor
itary interventions. While the techniques and methods of systems, such as the so-called Argus-IS (Autonomous real
obtaining information, surveillance, and reconnaissance time ground ubiquitous surveillance Imaging System), now
that are required for this were separated from the use of record moving images of areas on the scale of entire cities.68
weapons in conventional warfare, the selection and obser- There has been a corresponding rise in the requirement
vation of the target and the decision to kill are now part for personnel tasked with data analysis and interpretation.
of the remit of the crew. This confronts soldiers with com- In distributed common ground systems, infrastructures for
plex cognitive demands: decisions taken by drone crews the processing and analysis of the data from different sen-
are based on a practice that Asaro defines as a “fast-paced sor systems that are firmly established as weapons systems;
multimedia and social media environment of intelligence screeners are employed to monitor incoming data signals
gathering and killing”.62 The practices of seeing and visual- and video streams – an activity that is also increasingly
ising for the Reaper crews are based on a view according to being delegated to private companies (fig. 3).69 In this case,
which humans and machines must enter into a cooperative
“partnership”; they must become a “functional system” 63, as 66 Ibid., p. 130: “The ability of Reaper pilots, sensor operators, and mission
outlined by Hutchins. coordinators to communicate, develop, and execute a plan to ‘look through
a soda straw’ was a critical skill necessary to defend ground forces and to
Even if the active design, manipulation and interven- obtain the best video possible.”
tion in the feed, that Cullen describes as “building a pic- 67 See Valerie J. Gawron, Keven Gambold, Scott Scheff, Jay Shively, Ground
Control Systems, in: Nancy J. Coke, Leah J. Rowe, Winston Bennett Jr,
ture” 64 and as “growing a video track” 65, still forms part
DeForest Q. Joralmon (eds.), Remotely Piloted Aircraft Systems. A Human
of the core competencies and learning objectives of drone Systems Integration Perspective, Chichester, 2017, pp. 63–109.
crews, it may become progressively less of a requirement 68 Defense Science Board, Office of the Secretary of Defense, Autonomy, 2016,
p. 50.
69 Abigail Fielding-Smith, Crofton Black, Revealed. The Private Firms Track-
62 Asaro 2013 (as fn. 17), p. 13. ing Terror Targets at Heart of US Drone Wars, in: The Bureau of Inves-
63 Hutchins 1995 (as fn. 28), p. 170, see Cullen 2011 (as fn. 25), p. 198. tigative Journalism, July 30, 2015, https://thebureauinvestigates.com/
64 Cullen 2011 (as fn. 25), p. 117. stories/2015-07-30/revealed-the-private-firms-tracking-terror-targets-
65 Cullen 2017 (as fn. 56). at-heart-of-us-drone-wars (accessed December 8, 2017). 127
 Nina Franz and Moritz Queisner

The use of automated software systems for image analy-

sis, for example, to filter out abnormal activities, or to pursue
moving targets, demonstrates how seeing and visualising in
the deployment of drones cannot be understood as solely the
distributed activity of human actors, for example, as a result
of the collaborative observation of the screen by a crew. This
not only changes “the ideas about who the agent of image
production is in situations of war” 73, but the situations in
which data is depicted visually at all. Through the establish-
ment of forms of automated recognition and selection, data
visualisation may only be required if human actors need to
verify or falsify pre-filtered results of calculated decisions,
3  Distributed common ground system, US Air Force.
i. e. when they need to comply to juridical routines or mili-
tary workflows. In other words: the exclusion of the human
the forensic monitoring is still based on culturally acquired observer goes so far that action and decision-making do not
knowledge, as noted by Gregory.70 However, due to the enor- require images at all.
mous quantity of video material, the observation of images However, it is not only the visualisation, but also the
within the scope of anthropological vision is increasingly operationalisation of the combat zone that is increasingly
impossible as “no human eye is capable of analysing such system-controlled. The drone cockpit of the future intends
images with a volume of several terabytes per minute, to fuse the division of labour between the sensor operator
which is why movement profiles of humans and vehicles and pilot – which has existed to date – into labour carried
are pre-sorted through automated pattern recognition” 71, as out by only one person.74 Given the increasing automation of
Michael Andreas points out. Using a “Global Information the controlling of an ever greater number of aircraft by ever
Grid, i. e. a communication network or raster that is inter- fewer operators75, and the division and partition of combat
connected with the military databases […], the interconnec- zones into geometric decision spaces, so-called kill boxes,
tion of surveillance drones and precision weapons thereby this appears to herald the start of a new, worrying para-
renews the fiction of military real time […], the phantasms
73 Carolin Höfler, Eyes in the Sky. Körper, Raum und Sicht im bildgeführten
of which have penetrated through to the acronyms used in
Krieg, in: Martin Scholz, Friedrich Weltzien (eds.), Design und Krieg, Berlin,
military terminology” 72, Andreas argues. 2015, pp. 13–34, p. 31 [translated by the authors].
74 Maia B. Cook, Harvey S. Smallman, Human-Centered Command and Con-
trol of Future Autonomous System, Power Point presentation at the 18th
70 Gregory 2012 (as fn. 3), p. 195. International Command and Control Research & Technology Symposium
71 Michael Andreas, Flächen/Rastern. Zur Bildlichkeit der Drohne, in: Behe- Track C2 in Underdeveloped, Degraded, and Denied Operational Envi-
moth. A Journal on Civilization 8.2 (2015), pp. 108–127 [translated by the ronments, June 21, 2013, Alexandria, https://dodccrp.org/events/18th_
authors]. iccrts_2013/post_conference/presentations/090.pdf (accessed May 7, 2017).
128 72 Ibid., p. 114. 75 Gawron et al. 2017 (as fn. 67), p. 99, p. 102.
The Actors Are Leaving the Control Station

digm of control and operation in which the the human factor a single aircraft. In such cases, operators function as super-
becomes a precarious element which is only loosely attached visory decision-makers,78 instead of as observers.
to increasingly autonomous processes of computation. In conversation with the US Air Force drone crews on
the Maxwell Air Force Base, participants openly discussed
their frustration with the fact that more and more respon-
Crisis of Cooperation
sibility and power of decision-making is being transferred
Weapons manufacturers, with the Californian company from the operators of the weapons systems to the engineers,
General Atomics leading the way, are still designing GCS i. e. from members of the military to actors who are pur-
control and communication interfaces based on the classic suing fundamentally different, primarily market-driven,
cockpit architecture of manned aircraft, in which a crew interests and have no military responsibility. Criticism was
acts and observes within a specific aerial space. However, aimed, in particular, at the armaments group General Atom-
recent research in Human Factor Studies is resulting in the ics, which not only manufactures the most frequently used
design of the GCS more like platforms, where operators armed drone systems, Predator and Reaper, but which is
monitor automated processes. Gawron et al. view opera- also responsible for the design of the control stations that
tors as no longer being capable of translating the increasing are currently used. Complaints focused mainly on the grow-
quantities of data into actions: ing rigidity of the interfaces and the lack of scope for influ-
ence on the system by military personnel, especially when
[M]onitoring a systems status is burdensome and defining the requirements of the GCS.
requires continuous effortful filtering of relevant ver- David Blair and Nick Helms, both US Air Force drone
sus irrelevant information, but emerging technologies pilots, contrast what they refer to as the capability-oriented
can make this a supervisory task by presenting opera- view of the military users with a cybernetics-oriented view
tors only with those alerts that require operator atten- taken by developers and manufacturers:
tion, in turn freeing up operator resources for other
tasks or even making some monitoring tasks obsolete.76 From a capabilities view, crew members—in partner-
ship with a fleet of maintainers and support personnel –
Cook and Smallman, on the other hand, regard the demand take ‘their’ aircraft into the fight to hunt down threats.
for a new operative paradigm in the design of GCS as justi- Conversely, a cybernetics view uses a crew to supply a
fied by the fact that future crews will coordinate numerous set of inputs that in turn produces x number of hours of
activities carried out by different “autonomous platforms intelligence, surveillance, and reconnaissance.79
and agents” 77 in parallel, instead of tasks being allocated to

76 Ibid., p. 93. 78 See Gawron et. al. 2017 (as fn. 67), p. 93.
77 Cook, Smallman 2013 (as fn. 74), p. 2. 79 Blair, Helms 2014 (as fn. 17), pp. 40–41. 129
 Nina Franz and Moritz Queisner

Their “capability perspective” can be assigned to the hero­ alisations, and it is hardly recognisable as such, or even
ising tradition of classic pilots, who are first and foremost comprehensible. This circumstance is further complicated
self-reliant and use technology as “amplifiers of human by applications based on artificial intelligence or machine
will”. 80 Similar to Cullen, Blair and Helms according- learning; this is especially true in the case of automation of
ly view the problems of a cybernetics perspective mainly the data analysis84 in surveillance missions and for target
within the context of a diffusion of agency, through which recognition, wherein the identification of a legitimate target
crews become “subsystems within larger sociomechanical is precisely the critical function, a function that then sets
constructs”, which locks them into closed “control loops” 81 a precedent.
that regulate the systemic variables of specified parameters. A new Defence Advanced Research Projects Agency
In contrast, a “capabilities perspective”, such as is also (DARPA) programme called Explainable AI (XAI) demon-
advocated by Cullen, highlights the “technical” aspect strates that the US Air Force is aware of the problems posed
of the skills required for control. In order to “tease out” by increasing automation. The head of the programme,
details from an image, parameters such as “gain, level and David Gunning, was also responsible for the DARPA pro-
focus” must be manipulated manually, or different imaging gramme CALO (Cognitive Assistant that Learns and Organ-
modalities must be organised in various ways and over- ises), whose most prominent spin-off is Siri, Apple’s lan-
laid.82 According to this description, that is in contrast to guage recognition software. In a public statement on XAI,
more critical perspectives on what constitutes these image Gunning explains the objectives of the new programme, the
operations,83 what becomes visible is based primarily on the focus of which, as cited in the mission statement, “is the
competence of the operator. development of a model that will enable human users to
Even so, the crew members are convinced that the understand, appropriately trust, and effectively manage the
sphere of influence they are granted is significantly affect- emerging generation of artificially intelligent partners”.85
ed by software and hardware engineers and developers, and The anthropomorphism that pervades military ref-
that these actors are simultaneously determining, to an ever erences to cooperation, partners and human-machine
greater extent, what is visualised and how, i. e. what can teams, expresses a new turn in the rhetoric of Explainable
actually become the focus of attention. The selection pro- AI, where not only agency, but also the ability to think is
cess that precedes the workflow in the cockpit is not trans- attributed to the machines:
parent to the operators, who, as the human actors, bear the
84 An example of this is provided by the identification of military targets
responsibility for the decisions taken based on data visu-
through the analysis of behavioural patterns, see Patrick Tucker, A New
AI Learns Through Observation Alone. What That Means for Drone Sur-
80 Ibid., p. 40. veillance, in: Defense One, September 6, 2016, http://defenseone.com/tech-
81 Ibid. nology/2016/09/new-ai-learns-through-observation-alone-what-means-
82 Cullen 2011 (as fn. 25), pp. 165–167. drone-surveillance/131322/?oref=d-channeltop (accessed September 30,
83 See Kathrin Friedrich, Moritz Queisner, Anna Roethe (eds.), Image Guid- 2017); see also Nina Franz, Targeted Killing and Pattern-of-life Analysis.
ance. Bedingungen bildgeführter Operation. Berlin, 2016; Jens Eder, Char- Weaponised Media, in: Media, Culture & Society 30.1 (2017), pp. 111–121.
lotte Klonk (eds.), Image Operations. Visual Media and Political Conf lict, 85 Explainable AI, https://darpa.mil/program/explainable-artificial-intelli-
130 Manchester, 2016. gence (accessed September 30, 2017).
The Actors Are Leaving the Control Station

Continued advances promise to produce autonomous and the provision of trustworthy technological partners that
systems that will perceive, learn, decide, and act on also supply the comprehensibility of their own decisions
their own. However, the effectiveness of these systems remains a desideratum. It does not seem unlikely that this
will be limited by the machine’s inability to explain its initiative is more of a symptom anticipating an apparent cri-
thoughts and actions to human users. Explainable AI sis of the human operator rather than a realistic perspective
will be essential, if users are to understand, trust, and for opening the black box of highly complex neuronal net-
effectively manage this emerging generation of artifi- works, as Explainable AI seems to do little more than adding
cially intelligent partners. 86 further, deeper levels to the operating interfaces providing
instructions, and increase the epistemic distance between an
The step from the controlling human to “manager”, intelli- automated decision and the human executing the decision.
gent “partner” or decision-maker, therefore, appears com- While the symmetrisation of agency appears to be a
pleted, at least rhetorically. It is perhaps no coincidence progressive notion for thinking the complex relationships
that Timothy Cullen was asked to act as a consultant on between humans and machines in media theoretical reflec-
the programme, given his intensive investigation of the tion, especially in the wake of Latour, the tendency to put
practitioner’s perspective with reference to the user inter- human and machines on an equal plain in the context of mil-
face in the increasingly automated ground control stations. itary operative discourse is recognisable as a rhetoric that
He was invited to participate in a working group involv- nominally reduces the confrontational nature of this co-op-
ing behavioural psychologists from the field of naturalistic eration and obscures the influence of powerful actors that
decision making, the objective of which is the development are not necessarily part of the command chain. A report by
of an explanatory model for decision-making processes. In the Bureau of Investigative Journalism from the year 2015
turn, this model is to be used by computer scientists for the warned that service providers of powerful defense suppliers
development of a “system” to explain the performance of like General Atomics are increasingly taking over responsi-
“other systems”, such as so-called “deep neural networks”.87 bilities that are defined as “inherently governmental func-
In this case, cooperation remains primarily a matter of tions”.88 This is true for instance, when surveillance missions
technical requirements at the level of model development are outsourced to contractors, as one commentator notes:

86 David Gunning, Explainable Artificial Intelligence (XAI), DARPA/I20, The Pentagon may not have plans to allow contractors
Distribution Statement A, https://cc.gatech.edu/~alanwags/DLAI2016/
to fire missiles off drones. But allowing them to feed
(Gunning)%20IJCAI-16%20DLAI%20WS.pdf (accessed July 30, 2017)
[emphasis by authors]; Explainable Artificial Intelligence (XAI), [official targeting data to the uniformed trigger-puller takes the
website], https://darpa.mil/program/explainable-artificial-intelligence world one step closer in that direction. 89
(accessed July 30, 2017). A later version of the statement on the website for
the programme replaces “thoughts and actions” with “decisions”. Explain-
able Artificial Intelligence (XAI), [official website], https://darpa.mil/pro- 88 Fielding-Smith, Black 2015 (as fn. 69).
gram/explainable-­a rtificial-intelligence (accessed September 30, 2017). 89 Laura Dickinson, Drones and Contract Mission Creep, in: Just Security,
87 Timothy Cullen, email correspondence with the authors on July 22, 2017 August 5, 2015, https://justsecurity.org/25223/drones-contractors-mis-
and September 10, 2017. sion-creep (accessed September 15, 2015). 131
 Nina Franz and Moritz Queisner

Adding to that, over the course of the development of Figures

increasingly automated control interfaces that form the eye 1  US Air Force, Timothy Cullen, 2010.
of the needle for a technological authority, and through the 2  US Air Force, Christopher Flahive, https://media.defense.gov/2009/
introduction of the newest generation of AI technologies, Apr/20/2000593687/-1/-1/0/090409-F-0502F-001.jpg (accessed
January 2, 2018).
which diverge from the path of strictly rule-based cybernet-
3  US Air Force, 2014, http://af.mil/About-Us/Fact-Sheets/Display/
ic models, the Command and Control functions are increas- Article/104525/air-force-distributed-common-ground-system (accessed
ingly disappearing behind a rhetoric of cooperation that no January 3, 2018).
longer only places objects, humans, and algorithms on the
same cognitive plane, but essentially removes the agency
from the human element within the control environment.
This discourse, which attributes the ability to think and act
to things and which obliges humans to primarily believe in
the explanations provided by technology, pays no attention
to the fact that the real decisions have been taken by engi-
neers during the design process: The actors are leaving the
control station.

132
Shannon Benna

Systems and Practices to Produce

Stereoscopic Space on Screen
When we as filmmakers, photographers and artists try to to the viewer. I think this is more often the way stereoscopic
create an image, we are bound by the technical constraints 3D has been used, however, I don’t think it is always because
of the machines or mediums we use. Through increasing- it was the intent of the creator, but due to the way the con-
ly complex mechanisms or processes, we have been able to tent is delivered to the viewer. For example, if you create a
reproduce the images we perceive with our eyes more and 3D image that is true-to-life in depth and scale, accurately
more realistically. We have also been able to manipulate recreating the way the average person would view another
images to express an imagined aesthetic, intended to con- person, or an orthostereo image, you must also present it to
vey an artistic vision, an emotion, or a forced perception the viewer in such a way that they are viewing the image
designed to inspire sympathy/empathy for an unusual per- from the same position that the camera was capturing it,
spective. Whether we are discussing black and white, color, and without their being aware the image is being projected.
still, moving, realistic, fantastical, 2D, or stereoscopic 3D Then the viewer would be unaware they were experienc-
images, we must not only consider how the image is tech- ing an optical illusion. Whereas, if an orthoscopic image
nically created, but also how it is delivered to the viewer. is projected onto a screen in a theater, the viewer experi-
An image viewed on a small, personal screen (like a tablet ences the image as part of an audience, as opposed to as an
or cell phone) will create a different relationship with the one-on-one experience, and the image is now larger than
viewer than the same image viewed on a theatrical screen. life in scale, rendering it no longer realistic, but fantastical.
Likewise, a shared viewing experience in public will cre- With the advent of modern immersive media we are given
ate a different response than one experienced via a virtual the opportunity to place viewers inside of the world of our
reality (VR) headset. choosing in a whole new way. Although VR can give us a
As a filmmaker who has worked in both 2D and 3D complete sphere to visit, only with stereoscopic imaging can
media, I find that although 3D imaging provides an avenue the environment become truly immersive in the sense of
to recreating the most realistic version of the world that we granting the individual a real-life perception opportunity.
see every day, it is not always presented in a way that gives Via this presentation method we can engage a viewer in an
the viewer a real-to-life experience. Additionally, stereo- orthoscopic version of an alternate reality, whether it is real-
scopic 3D can provide a way for an artist to create a fantas- ism or fantastical, allowing for a whole new way for artists
tical world-view and present an alternate version of reality to connect with their audience.
 Shannon Benna

Therefore, to utilize stereoscopic imaging to its fullest Rules for Creating and Manipulating Stereo-
capability, we must understand its technical constraints and scopic Space
requirements and we must make a conscious decision about
the intended purpose of the images from the beginning. We To begin, one should understand the basic premise behind
must not only go through a procedure that ensures the imag- stereoscopic imaging. It requires two images, a left and
es we create will be viewable as 3D images, and not cause right, representative of and ultimately delivered to each of
pain or nausea to our viewer, but also choose the right set of the viewer’s eyes, discretely. As we all see in three dimen-
tools to capture and deliver our images, accurately express- sions everyday (unless we are part of the 6–12% of the pop-
1  Parallax, R/C (all images marked
R/C can be viewed stereoscopically ing our intentions towards reality or fantasy and immersion ulation that suffers stereo blindness, or an inability to see in
with red/cyan anaglyph glasses). vs. presentation. Then, once we have chosen our toolset, the depth) we are pre-equipped with the firmware in our brains
subtle adjustments we make to the settings of our tools will to take in these two separate right and left eye images and
convey the emotional and physical sensations we intend, as fuse them into an all-encompassing perception of space,
well as the perspective we present, including the immersive, complete with volumetric objects living at some distance
intimate, spectator and/or voyeuristic quality of the images. from us physically. This process is called stereopsis, and
As both a practitioner and an educator of stereoscop- those who can perceive depth possess binocular vision.
ic 3D imaging and media production, I believe the key to The amount of separation between the eyes is known
this and any discipline is communication. Herein lies a bit as inter-ocular (IO) separation, while the distance between
of a problem, as our terminology is not truly standardized. the imaging sensors is known as the inter-axial (IA) distance.
There are multiple ways to refer to the same effect, error, These terms are often used interchangeably and are mea-
process, or action, which are not only diversified between sured in a variety of different units: inches, centimeters,
production teams or academic institutions, but also between percentage of screen size, and pixels. Although it may seem
stages of production. The way one speaks of 3D in pre-visu- extreme, we are concerned with pixel accurate measure-
alization, development and appreciation is often different ments, as this can mean the difference between an image
than the terminology used in production, or in post-produc- that is comfortable to view, and too deep for the average per-
tion. There isn’t even consistency between editorial, finish- son to fuse. Through stereoscopic imaging we are able to
ing, and conversion, which are all post-production practices. trompe l’œil or fool the eye into believing these two images,
To aid us in this publication, any terms will be typeset in a captured from two slightly different perspectives are actual-
visually different manner for ease of recognition. There is a ly a three dimensional object, landscape or world. The wider
glossary of terms and accompanying illustrations to facili- the separation between the same object in each of the imag-
tate comprehension as well. es when overlaid, or parallax (fig. 1), determines the amount
of depth in the image. Too much parallax can render the
image unresolvable, as the eyes would be forced to diverge
or strain towards the periphery in order to attempt to fuse
134 the stereo pair into a single image.
Systems and Practices to Produce Stereoscopic Space on Screen

2 Z-Space. 3  Edge violation above, correct 4  Floating window 5  Convergence in stereo-space.

framing below, R/C. correction.

Along with being able to control the amount of depth We can angle the screen plane to accommodate this cre-
in a stereoscopic image, we are also able to choose where ative choice by employing a floating window (fig. 4), or a black
6  Parallax seperation in 3D image,
objects are placed in relation to the screen plane, or where mask that is applied to the side of one eye to bevel that side R/C.
they are placed along the Z axis (fig. 2). Objects behind the of the screen plane backwards in space. This technique is
screen plane are in positive Z space, while objects that are used exclusively in a traditional presentation method, in VR
in front of the screen, or float into the audience’s space are there is no off-screen, and thus no stereo window to consider.
in negative Z space. It is often difficult to perceive the actual Objects that appear at the screen plane are at zero par-
screen plane, and in some cases, especially VR presentation, allax and are considered to be the point of convergence. The
the screen plane does not truly exist. We do use the con- point of convergence is the point at which our eyes cross
cept of the screen plane to describe zero parallax , for ease when we naturally observe the world (fig. 5), or where the
of discussion, especially while creating images and using two imaging devices’ lines of sight cross when capturing
traditional on-set monitors. images. If we choose to set convergence during the initial
If an object is intended to be positioned in negative Z imaging stage, we are said to be shooting converged and
space, it must not be clipped off in any unnatural fashion but we set a degree of rotation that the cameras will turn in
be imaged as a whole. If part of the object exists off screen towards each other for each shot to decide where objects/
it must also be in positive Z space. If the object/subject is subjects will be positioned along the Z axis (fig. 6).
both partially off screen and in negative Z space, the effect It is not necessary to converge the lines of sight when
is called an edge violation (fig. 3) and causes confusion in the initially imaging. If we do not, it will give the effect that all
brain. A person or thing cannot be simultaneously outside of of the image appears in negative Z space and is referred to as
a window and inside at the same time without being angled. shooting parallel (fig. 7). Using this method, all of the image 7  Shooting Parallel. 135
 Shannon Benna

9  Stereo Pair, R/C. 10  Monoscopic / 2D Image, R/C.

can be pushed out into the audience’s space, all the time, like and can choose to fuse smaller portions of the image selec-
in an IMAX 3D film. When shooting parallel, the point of tively. If the content is also going to be available on home
8  Horizontal Image Translation
correction for edge violation, R/C. convergence can also be chosen during the post-production theater or TV screens, the images will all have to be adjusted
process. However, there will be a loss of resolution and an to push some of the image back into positive z space, so there
alteration of framing to compensate for the change in over- is not too much parallax separation present in all parts of
lap of the images when zooming in and cropping the image the screen to be fused at one time.
to eliminate superfluous edges. An adjustment to where the
images overlap and where the subsequent point of conver-
Single Camera Methods
gence exists, is called a Horizontal Image Translation, or H. I. T.
(fig. 8). 2D–3D conversion: Images captured with a single camera
then post-production processed to create second eye view.
This process can be very useful when native capture (ste-
Stereoscopic Imaging Toolsets
reoscopic imaging as produced using a naturally occurring
Below are brief descriptions of five ways to capture stereo- ­s tereo pair (fig. 9) or left & right eye images) is not prac-
scopic images, using either one or two cameras. The choice tical, i. e., when physical size, scale diversity, or camera/
of toolset will ultimately be decided by physical restrictions, rig mounting and/or operating space is very restricted.
as well as by the desired effect of the images to be captured. Although conversion can be expensive and time consuming,
Any of these toolsets can be used in a parallel or con- it can also be very helpful when creating fantastical scenar-
verged configuration, dependent upon the intended immer- ios, as the creation of a second eye image from the existing
sive or invasive quality of the content, and the chosen deliv- monoscopic (2D), or single eye image (fig. 10) can utilize
ery format(s). Projects shot and delivered in parallel will more than one inter-axial setting at a time, creating more
only be viewable on a Giant Screen, where the viewer is complex geometry than is achievable when using physical
136 relieved from looking at all of the screen at the same time cameras in a physical environment.
Systems and Practices to Produce Stereoscopic Space on Screen

Stop motion/stepper motor: Two images captured by get a close-up shot and will leave the audience feeling an
a single camera/lens, in succession, but after shifting per- emotional separation from the topic. Also, it could create too
spective between the first and second images. The amount large of a parallax separation in the background elements,
of shift between images is equivalent to the inter-axial dis- causing a divergence among viewers. This effect becomes
tance. The subject must remain static during the acquisi- exaggerated in an immersive delivery method, such as the
tion of both images intended for a single stereo pair, and the cinema or a VR set-up.
camera must only shift along the X axis. The stereo pairs Light levels that will be standard for the subject matter
11  Side-by-side rig.
are then strung together in an image sequence and played must also be considered as some camera systems handle
back at speed, each stereo pair equal to one frame in frames low light better than others. If a lack of light is innate in
per second. the subject, one should be aware that a greater than usual
amount of light is recommended for stereoscopic image
viewing, as there is light loss when projecting through
Dual Lens/Camera Methods
polarizing or color filters, and the viewer is wearing 3D
Single-body/dual-lens camera systems: There are a variety of glasses. Dim images can cause eye-strain, and low lumen
stereoscopic 3D video/digital cinema camera options raging projection in theaters has been the cause of much 3D media
in size, weight, image compression method and quality, but criticism and distaste. Additionally, lighting techniques for
with fixed inter-axial distance and permanent lens choices. 3D production are more demanding, as accenting individ-
These devices allow for less setup and no alignment neces- ual areas along the Z axis create depth cues, or planes of
sary or possible (some aberrance in hardware may require light for the viewer to distinguish, creating a richer depth 12 Hyperstereo.
post-production geometry and/or color correction). Depen- environment.
dent upon the inter-axial distance, the cameras innately Side-by-side rig (fig. 11): Dual camera configuration
restrict the amount of depth an image can have, as the dis- utilizing a simple bar with both cameras oriented next to
tance from subject and/or the furthest background point is one another. This is convenient for shooting larger, spec-
limited to avoid excessive parallax. They do grant extreme tacle style subjects with an increased potential inter-axial
freedom and mobility for the content creator, although they distance but limits cameras from getting any closer to one
limit creative flexibility. These solutions are ideal for ENG another than the physical body and or lenses will allow.
(Electronic News Gathering), Reality and documentary sit- Shooting with an exaggerated inter-axial setting can
uations, where time and crew/equipment space is limited. create an effect called hyperstereo (fig. 12), which allows us
One must camera test to assure the fixed inter-axial does not to perceive depth in environments or objects that are very
prohibit the necessary distance from the subject to create large or further away than we are able to observe depth
both ideal framing and intimacy/immersive effect, as well in naturally. Due to the average inter-ocular separation
as viewable 3D images. of human eyes (about 6.3–7.6 cm or 2.5–3 inches) we are
If the fixed inter-axial distance is too wide, the camera restricted from observing depth in anything more distant
will not be able to approach near enough to the subject to than 18.3–30.5 m or 60–100 feet away. 137
 Shannon Benna

While this process is ideal to allow the viewer to see which one can perceive the world as if one were very small.
depth in grand vistas, large objects, architectural structures This effect can be used creatively to immerse the viewer
or natural wonders, when used to perceive depth in normal into the world on a microscale, like documenting insects,
sized objects that are a great distance from the camera, an surgical procedures, electronics construction or granting
effect called miniaturization can occur. This effect makes the viewer a first-person POV as the inhabitant of a fanta-
objects and people seem very small, like toys, or as if the sy world of tiny people like in Jonathan Swift’s Gulliver’s
viewer were a giant. This can be used creatively to simulate ­T ravels (Benjamin Motte, 1726).
a fantastical scenario, rather than to observe sporting events Once we are able to communicate verbally about our
etc. from a distance, as miniaturization will pull the viewer intentions and desires concerning the stereoscopic images we
out of the experience and remind them they are watching wish to produce, we can begin to choose the toolset, pipeline
something artificially produced. and workflow that will help us to achieve these aims.
Beamsplitter rig (fig. 13): Dual camera configuration uti-
13  Beamsplitter rig.
lizing a mounting bracket with mirror to reflect the desired
Stereoscopic Imaging Systems Set-Up and
image into one camera lens (reflected camera), while the
Variables
second camera looks through the mirror to capture ( pass-
through camera), allowing for a far narrower inter-axial sep- No matter what camera or rig is chosen for the job, there
aration than the physical cameras would be able to achieve are a consistent set of parameters that must be adhered to
if mounted next to each other. There is significant light loss in order to ensure the two images that are produced are
due to the mirror in the rig, about 1.5 to 2 stops decrease, useable. The two images must be exactly matched in every
so lighting conditions must be of significant consideration way, except for their slight difference in vantage point along
when choosing to shoot with a beamsplitter. the X axis. Any discrepancy between the images is called
Shooting with a narrower inter-axial separation than retinal rivalry (fig. 15), as it can cause eye strain, because the
our average inter-ocular separation of 2.5 inches will allow viewer will be seeing something in one eye but not seeing it
for use of extreme/close-up shots, which are a standard part in the other. Each of the following items listed must be pre-
of cinematic storytelling when we want to convey focus or cisely matched and aligned to achieve a perfect stereo pair.
attention to a specific object or to intensify the emotional It is useful to label each piece from the cameras, lenses, and
14 Hypostereo.
connection with the subject. It is not a realistic situation, media cards to cables card to cables etc. as Left and Right.
as we rarely put our faces very close to an object or person Further, one should maintain use of each item in the same
when we are paying attention to a particular thing but is position consistently as it alleviates confusion and allows for
useful as a cue to the audience that something important easier troubleshooting if a problem arises. One basic guide-
is happening. line is to eliminate potential variables at every opportunity.
The narrowness of the inter-axial setting must be used Cameras should be the same make, brand, and have the
with care, because although it can help us to get closer to same firmware version. They should ultimately be able to
138 a subject, an effect called hypostereo (fig. 14) can result, in genlock. The frame rate and shutter speed/shutter angle set-
Systems and Practices to Produce Stereoscopic Space on Screen

tings should be the same on both cameras. Exposure and

color temperature should also match as closely as possible,
but note that the mirror will cause a color and light shift in
one camera. Final exposure and color balancing will need
to be done in post-production.
Lenses should be a matched pair (as close as possible),
lens mapping helps with and aberrations in the physical ele-
ments, and using wider lenses is preferred to long lenses as
zooming compresses space, which manifests in 3D images
15  Retinal rivalry, R/C.
as cardboarding, or the flattening of objects so they appear
like cardboard cutouts. Prime lenses (lenses with only 1
focal length) are most preferable, although for speed in to match the images in a stereo pair. A third FIZ control can
production, wider zoom lenses (lenses with multiple focal be used on some beamsplitter rigs to motorize the adjust-
lengths, but not to exceed 50 mm ideally) can be used as ments for inter-axial and convergence.
vari(able)-prime lenses, allowing one to have multiple focal Filters may be used but must be matched in density and
length choices without having to change the lenses on the orientation. The most commonly used filter is a quarter wave
rig and re-align every time. In order to perform the initial retarder, placed in front of the mirror in the beamsplitter.
alignment, zoom lenses must be set to the same actual focal This protects against polarization issues, which result from
length. Although the lens marking may denote a particular shooting reflective surfaces from slightly different angles.
focal length, the optical reality of 50 mm being exactly the As the reflection will look different in the left and right eye,
same on different lenses is very improbable. A visual confir- when viewed as a stereo image, strobing will occur and take
mation of zoom scale must be checked. the viewer out of the moment or cause eye-strain. This effect
The back of the lenses should be the same distance from can be used creatively but should be intentional.
the sensor when attaching them to their respective cameras. Timecode is important for both imaging and post-pro-
Not all lenses are adjustable, but most professional digital cine­ duction practices. Some cameras generate their own time-
ma cameras do have the option to adjust back focal distance. code, while others must be fed timecode. This provides not
Also be sure to focus both cameras to properly align them. only a point-of-reference for post-production to find specific
FIZ motor calibration is essential if we desire to have events within footage, but also allows for multiple camera
dynamic framing or to zoom in or out during a scene in 3D shoots to have a common standard. It also provides a regu-
imaging. These motor systems are used in 2D production lated cadence by which the cameras may be synced.
to control Focus, Iris, and Zoom of the lenses remotely. The Genlock is a process by which two cameras may be
same motor systems can be used during stereoscopic pro- linked together. Regardless of the shutter speed chosen,
duction, but two sets will be needed (one for each camera). the cameras must open and close shutter at the exact same
They will need to be calibrated to work in unison if they are time. Genlock guarantees the two cameras will maintain 139
 Shannon Benna

their sync with each other, so that the left and right camer- When an inversion of image occurs, whether it is
as capture the exact same frame without any discrepancies because one of the cameras on a beamsplitter is recording
being introduced by the shutter angle. If not there will be a reflected image and it appears backwards, or because
temporal differences and the images will be unusable. Also, the camera is mounted upside down, that image must be
if a camera system is not genlocked but both cameras are flipped and/or flopped to allow for alignment and monitor-
started at the same time, the shuttering tends to drift out ing during production. Flip/flop can be done in camera (if
of sync over time. the option is available on the camera) or at the monitoring
16  Anaglyph 3D Image, R/C.
There are a few configurations one might apply to the level. It does not matter which stage it is done at, but it must
cameras being linked, but in stereoscopic imaging, we tend be done at some point, either during initial image capture,
to use the master/slave configuration. This allows chang- or during the post-production process.
es on one camera (master) to automatically be applied to Some specialty monitors exist for onset 3D monitoring
the other camera (slave), so we are certain the settings are and make the process much easier. They usually have mul-
always the same on both cameras and eliminate user error. tiple modes that are helpful during alignment, and when
Keeping cameras in sync is not just concerned with setting inter-axial & convergence. They range in size from
shuttering, but also with scanning. Film cameras capture 5 inches (which would be used for onboard camera monitor-
images by the opening and closing the physical aperture, ing for the operator) to large TV size, which would be used
allowing light to pass through the iris and strike the film, by the stereographer when setting the depth in each scene
leaving an impression. Digital cameras have two types of or by the director to view and compose shots and on set
shutters available: A global shutter, which behaves like a action or playback and review. They can be fully polarized
film camera and takes one massive impression each time stereoscopic monitors (that use polarized lens glasses) or an
the camera shutters, or a rolling shutter, which basically acts anaglyph (fig. 16) display (which utilize the red/cyan glasses).
like a scanner or copy machine, scans the lines of resolu- Polarizing filters orient the left and right eye images’ light
tion in succession starting at the upper left hand corner and waves to only pass through their correlating glasses’ eye.
ending with the lower right corner. When using cameras This same technology is used in theaters and on 3D TVs.
with a rolling shutter, the scan progression must be in sync There are also some pass-through devices that will out-
as well. This is what we are referring to when we describe put a 3D image in a variety of formats, although anaglyph
the cameras as being in phase. There are some additional images are able to be viewed on any standard monitor or
issues when using rolling shutter cameras including dealing computer screen. Some form of 3D monitoring is essential
with fast motion. One arises when the first part of the image for at least the alignment stage, and is highly recommended
is scanned before the last part of the image, thus causing during production, although with the use of complex cal-
motion artifacts. Additionally, one or both cameras on the culations, inter-axial and convergence can be set without
rig are sometimes mounted in inverted positions (because of it. This will definitely increase on-set production time and
mounting necessity), so the cameras may actually be scan- will often increase the required amount of post-production
140 ning in the opposite direction. corrections.
Systems and Practices to Produce Stereoscopic Space on Screen

Alignment Settings on Beamsplitter Rigs and 3. Scale: The images must be at the same scale in order
in Post-Production Correction to match geometrically. Therefore, when mount-
ing the cameras on the beamsplitter rig, the lenses
Alignment is one of the most important aspects of 3D imag- should be equidistant from the mirror and zoomed
ing, as it can be the difference between communicating an to the same focal length (optically). Also, the images
emotional message via an image on screen and having a output from the cameras should be in focus, as it is
meaningful experience lost in translation due to a techni- nearly impossible to ascertain pixel accurate align-
cal aberration that prevents it being viewed comfortably. As ment in an out of focus image.
there are so many variables in these imaging systems, and 4. Vertical alignment: This setting is sometimes con-
the adjustments that effect the outcome are so minute, the trolled by adjusting the height of the camera. If this is
following procedure is very helpful in establishing sound the case, the potential for pitch and roll issues arises.
geometry for image capture. Because we are working in To combat this, many rig designers place the vertical
three-dimensional space, any defects will create alignment alignment adjustment at the mirror.
issues, and render the images un-viewable. Although some 5. Pitch: This is the measurement of the height of the
corrections can be made during the post process, some front or back of the physical camera body as mounted
aspects are permanent and thus must be matched as closely to the rig. As some internal sensors are not always
as possible before imaging. physically squarely seated in the camera, adjustment
The following are the basic points of alignment and/or of pitch can help to correct for this geometrical issue.
geometry correction: However, it is very easy to use this to compensate
for misalignments created via another variable and
1. Mounting: When mounting the cameras onto the rig, thus make the geometry incorrect in either the back-
they must be square and level, meaning the camera ground or foreground. Ideally one should be able to
must be exactly perpendicular to the mirror at a 90 zoom from one end of the lens to the other without
degree angle and when a bubble level is placed on top loosing alignment, but as lenses have physical weight,
of the camera rig it should be level to the ground. they can shift center of gravity when on one end or
2. Horizontal: During the alignment, set the horizontal the other, so one must align for the object of interest
separation between the cameras to zero (zero rig); on a shot by shot basis.
this will produce the exact same image in both eyes, 6. Roll: This is a measure of the camera body’s flush
essentially creating a 2D image. Once the images are mounting to the rig as well but is concerned with the
exactly the same, as you proceed through the pro- tilt left or right.
cess of alignment, your images will be geometrically 7. Rotation: As one camera either hangs down or points
matched, and the rig will be properly aligned for ste- upward into the mirror, the camera can have a ten-
reoscopic imaging. dency to pivot on the mounting plate.
141
 Shannon Benna

Process of Formulating Depth in Images on sured in percentage of screen size, as dictated by

Screen c­ reative design or distribution channel):
9. Check for distortion within the subject/object of
After properly aligning the beamsplitter rig, we are ready interest.
to begin capturing stereoscopic images. We need to make a 10. If there is too much IA + convergence, objects will
series of creative choices to decide how to set the inter-axial distort, elongating along Z axis, creating a Pinocchio
and convergence (if we are shooting converged). There are effect (fig. 17).
many methods that involve complex processes of measuring 11. If there is too little IA + convergence, object will distort
17  Distortion effects.
and mathematical computation. However, a simple method elongating along X axis, either flattening the image,
for creating natural and comfortable 3D is called the Nat- or rendering a lack of volume or roundness (fig. 17).
ural Depth Method, developed by stereoscopic filmmakers
Alan & Josephine Derobe. This process creates stereoscop-
Controlling Parallax for various Delivery For-
ic images by choosing a natural convergence point for the
mats
viewer to focus on, then balancing the amount of parallax
separation between the negative and positive Z spaces. The A larger size of screen proportionally increases the amount
procedure is as follows: of depth a given parallax separation will display. There are
often several versions of a 3D product that have been re-con-
1. Set IA to roughly 2.5 inches. verged for each distribution channel. For example, media
2. Set the convergence on the object/subject you want intended to be viewed on a cell phone, personal tablet or
to be at the screen plane. small monitor will have greater parallax separation than the
3. Check the parallax of the nearest object/subject, at version distributed for movie theater projection. Conversely,
the closest point it will pass in front of the rig, to projects designed for Giant Screen Cinema (IMAX) will be
make sure it is not too diverged to be fused. unviewable on a smaller screen. Most feature movies/doc-
4. Check the parallax of the furthest object or back- umentaries are routinely reconverged before being released
ground for excessive disparity. (If the background is for home theater consumption.
out of focus, there can be more forgiveness, as the
audience will not be attempting to fuse a blurry
Creative Design in Producing Stereoscopic
image.)
5. Decrease the IA to adjust for acceptable amount of
Images
parallax in both. When designing a piece of stereoscopic media one must take
6. Re-converge. into consideration the demands of production, as per the
7. Repeat until entire image is comfortable to view. physical elements (scale of subject, size of environment, size
8. Finally, when parallax is within acceptable depth of budget, desired pace of production, un/predictive nature
142 budget (allowed amount of separation, usually mea- of subject matter), but also the way the eye receives infor-
Systems and Practices to Produce Stereoscopic Space on Screen

mation and the time it takes the brain to process the stereo- throw something into the viewer’s face or to create a sense
scopic images into a three-dimensional environment. Often of intrusion into the audience’s space, while placement of
to ease this process, we will divorce from traditional cine- an objects/subject in positive Z space can create a sense of
matic practice and embrace a cinematic theatre approach. voyeurism. Increased depth in a scene, with the subject in
This means longer takes of action in which we move within the distance, can make a viewer feel as though the space
space as if the camera is a character/anthropomorphized, is vast and provide a sense of occhiolism, while increased
rather than executing the traditional set-ups (establish- depth with the subject and environment filling up the scene
ing shot, two-shot, over the shoulder, coverage, etc.). This can create a sense of claustrophobia. A diminished depth
18  Stereo window in display
allows for a more natural viewing experience, as if we were when observing a subject or their environment can provide environment.
there, as opposed to being reminded that we are viewing a sense that something is lacking in that person’s life, while
a captured image. This serves us well in both orthoscopic realistic volume and depth can create a sense of well being
imaging as well as fantastical settings and helps the viewer and richness in the subject’s life. The freedom of the cam-
to stay immersed in the scene. era to move within the three-dimensional space can cre-
If we do adhere to traditional set ups, it is advisable to ate a sense of weightlessness or buoyancy, while a static or
use longer shots and cut less often when editing, as it takes locked-off shot can provide the physical sensation of restric-
the brain several seconds to fuse a new shot. One should also tion or captivity. The only true way to learn these tricks is to
attempt to transition from one shot to the next while having watch as much stereoscopic content as possible, both with
the point of interest at the same position along the Z axis, glasses on and off. When an emotional reaction is noted, look
at least at the end of one shot and the beginning of the next. at the image without glasses to ascertain the parallax and
The convergence point, or placement along the Z axis can be convergence setting. Also, there is nothing like practicing
dynamic, softly easing the viewer from one shot to the next. and experimenting to help one grasp the subtle art that is
Once the images have been fused, the subject can visually stereography or designing depth for expression on screen.
move freely within the stereo space, and in and/or out of
the stereo window (fig. 18), into positive or negative Z space.
The use of more or less depth as well as placement of Figures
objects/subjects along the Z axis can render many diverse 1–18: Shannon Benna.
psychological effects on the viewer. There is no set of
defined rules by which one can construct specific reactions.
On the production side we discover them as the stereoscop-
ic medium is explored. Dynamic inter-axial and/or conver-
gence can be used to draw a viewer into the image or to
push objects away from the viewer, creating a sense of dra-
matic/emotional tension, or loss. Objects/subjects that are
pushed into negative Z space can be used as a gimmick, to 143
 Shannon Benna

Glossary of Stereoscopic 3D Terms

2D–3D conversion Postproduction Binocular vision  The ability to per- Edge violation  An object/subject
process by which a second eye ceive depth while viewing with that is both partially off screen
view is generated from a single two eyes. and in negative Z space.
eye image. Cardboarding  An effect in stereo- FIZ  Motorization & controls for the
3D image  A volumetric image con- scopic imaging that comes from focus, iris, and zoom of a lens.
sisting of a left and right eye shooting an object/subject at Flip/flop  The vertical/horizon-
view, delivered discretely to each great distance while zoomed in tal re-orientation of an image.
of the viewer’s eyes, and per- using telephoto lenses. (There is not a standardized
ceived to have depth. Also called Converged  a method of stereoscopic direction = term.)
stereoscopic 3D. imaging where the cameras will Floating window  A black mask that
Alignment  The matching of geo- turn in towards each other a is applied to the side of one eye
metric elements/configuration specified degree of rotation for to bevel that side of the screen
when assembling a stereoscopic each shot. This decides where plane backwards in space.
imaging system, or the postpro- objects/subjects live along the Fuse  The physical process of com-
duction process of correcting Z axis. bination of separate images
geometric mismatches in image Convergence  The point at which in the brain, creating a single
pairs. our eyes cross when we naturally image, complete with volumetric
Anaglyph  A method of displaying observe the world, or where two objects living in space at some
stereo images that utilizes con- imaging devices’ lines of sight distance from us physically.
trasting color images & filtering cross. Global shutter  Digital imaging shut-
glasses to discretely deliver left Depth cues  Points of interest along ter that captures all of the image
and right images to their respec- the Z axis, accenting layers of at once, like a traditional film
tive eyes. depth in space. aperture.
Beamsplitter rig  Dual camera con- Depth budget  The sanctioned Horizontal Image Translation (H. I. T.)
figuration utilizing a mounting amount of overall parallax sepa- Post-production process where
bracket with mirror to reflect the ration in a 3D image, as dictated the convergence point is changed
desired image into one camera by creative design or the distri- in a shot, by shifting one or both
lens (reflected camera), while bution channel. of the images left or right in rela-
the second camera looks through Diverge  The effort to fuse two tion to one another.
the mirror to capture (pass- images by straining towards the Hyperstereo  A method of stereo-
through camera). periphery. scopic imaging using exaggerat-
144 ed inter-axial settings, intended
Systems and Practices to Produce Stereoscopic Space on Screen

to give depth to objects at a great right images of a stereo pair Rolling shutter  A digital imaging
distance. when the images are overlaid. shutter that scans lines of res-
Hypostereo  A method of ste- Parallel  A method of stereoscopic olution starting with the upper
reoscopic imaging using very imaging where the cameras do right corner and finishing in the
narrow inter-axial settings, not tow in towards one anoth- lower left.
allowing for extreme close-up er, placing the screen plane at Side-by-side rig  Dual camera con-
or macro images of very small infinity. figuration utilizing a simple bar
objects/subjects. Pass-through camera  The camera with both cameras oriented next
Inter-Axial (IA)  The distance that looks through the mirror to one another.
between the nodal centers of of a beamsplitter to capture an Stereo blindness  An inability to
acquisition point on digital imag- image. Also called the Direct perceive depth, when viewing
ing sensors. Camera. with two eyes.
Inter-Ocular (IO)  The amount of Polarized  A method of displaying Stereo pair  A left and right eye
separation between the pupils of stereo images that utilizes spe- image that when viewed create a
the eyes. cial filters to orient the left and 3D image.
Miniaturization  A side effect of right eye images’ light waves. Stereo window  The screen plane
shooting hyperstereo, where Electrons from the left eye light boundary between the negative
objects appear to be miniatur- waves are spun to the left, while and positive portions of the Z
ized. electrons from the right eye light axis.
Monoscopic  A single eye or 2D/flat waves are spun to the right. Left Stereography  The creative art and
image. and right polarizing filters are technical science of creating ste-
Native capture  Stereoscopic imag- worn in front of the viewer’s eyes, reoscopic 3D images.
ing using 2 cameras or lenses to which only allow the correlating Stereopsis  The perception of depth
create a left and right eye image. light waves to pass through. by fusing two images into one
Negative Z space  In front of the Positive Z space  Behind the screen volumetric image.
screen plane/stereo window. plane/stereo window. Z axis  Towards or away from the
Orthostereo  A realistic depth Reflected camera  The camera that audience.
image, reproducing the natural captures the image reflected in Zero parallax  At the screen plane.
scale and volume of an object/ the mirror of a beamsplitter.
subject. Retinal rivalry  When all or part of
Parallax  The separation between an object appears in one eye of a
the same object in the left and stereo pair, but not in the other. 145
Jens Schröter

Viewing Zone
The Volumetric Image, Spatial Knowledge and Collaborative
Practice

Volumetric Images Information from the outside is collected and processed to

generate strategies for action – in the case of Avatar: the
James Cameron’s Avatar (2009) is undoubtedly not the most evil capitalists trying to convince the native species on the
original contribution to science fiction cinema – think of the planet to leave their village, which is located in a very big
stereotypical the-good-indians-against-the-evil-capital- tree, because of the deposit of a demanded material with
ist-plot. Nevertheless it was hailed as a major contribution the funny name Unobtainium in the ground. The scene in fig.
to cinematic techniques because of its use of stereoscopy. 1a–c shows a visualization of a landscape based on fictional
While the use of stereoscopy might be quite conventional (or nearly fictional, I will come back to that) display technol-
in Avatar, there is a sequence, which is quite interesting and ogies, which render an image space even more convincingly
convincing (fig. 1a–c). by the stereoscopic presentation of the film. However, the
The sequence shows a kind of control room, a “center of shown fictional images are not stereoscopic: they are volu-
calculation” 1, as Bruno Latour would have pointed out, or a metric, no glasses are needed to see a spatial image, which
“center of coordination” as Lucy Suchman notes: can be seen from all sides, is colorful and half-transparent.
It has several properties:
Centers of coordination are characterizable in terms 1. It represents space not only like an image based on
of participants’ ongoing orientation to problems of linear perspective, it is itself spatial. That means it is a
space and time, involving the deployment of people post-perspectival image: it does not need to represent the
and equipment across distances, according to a canon- scenery with the aid of perspective because it does not
ical timetable or the emergent requirements of rapid render the scene on a plane. It is a transplane image (this is
response to a time-critical situation.2 similar to holography, but volumetric displays are normally
not holographic). 3 Thereby it avoids the spatial problems of
Bruno Latour, Science in Action. How to Follow Scientists and Engineers
1 perspectival representation: By foreshortening, perspective
through Society, Cambridge, MA: Harvard University Press, 1987, Chap. 6.
Lucy Suchman, Centers of Coordination. A Case and Some Themes,
2
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.627.4590& See Jens Schröter, 3D. History, Theory and Aesthetics of the Transplane
3
rep=rep1&type=pdf (accessed September 19, 2017). Image, New York, NY: Bloomsbury Academic, 2014.
 Jens Schröter

1a–c  Screenshots of Avatar, TC: 00:47:49, 00:48:03, 00:48:10.

changes the relative length of all lines to each other and it ing requires representations, which allow immediate under-
changes all angles – and in this sense a perspectival image standing of spatial situations and structures.
is not a very reliable representation of space (except under 2. This shows another important aspect: Such a trans-
very controlled conditions). That’s why technical and archi- plane image is ideally suited for collaborative work in which
tectural drawings made for the construction of objects (and decisions often have to be made. People stand around the
this is still true for the images in the instruction manuals representation, see the scene from different angles, point to
which accompany IKEA furniture) are not in linear per- specific aspects and discuss what to do. Of course this can
spective but in different forms of parallel perspective.4 be done (and is done) with flat images as well, and normal-
Another way to avoid the distortions and misrepresenta- ly these are preferred for the simple reason that they are
tions of perspective is to make the image itself spatial, which more available, but the technical arrangement (in Avatar) is
of course already begins with sculpture or scaled models especially helpful in situations where the structure of space
made from wood or clay used by engineers and architects plays a central role. In this sense the display shown is in
for example. The fictional volumetric computer display in the tradition of, what in German is so beautifully called, a
Avatar, designed by the special effects company Prime Focus, Lagebesprechungstisch or a table for discussing the situation.
stands in a long tradition of images which avoid perspective There is for instance a long tradition of planning and educa-
to represent a situation undistorted and therefore in a more tion in the military done with sand tables (fig. 2–3).6
efficient and operational way. 5 It is not surprising that we One important point that connects with the aforemen-
see military personnel in the scene. Military decision-mak- tioned aspect is that a spatial representation of this kind
does not align viewers in the same way a screen (or a lin-
See Jeffrey Z. Booker, A History of Engineering Drawing, London: Chatto
4 ear-perspectival representation on a screen) does. Linear-
and Windus, 1963.
Prime Focus World, http://primefocusworld.com/projects/ (accessed Sep-
5
148 tember 30, 2017). See Hans Hemmler, Die Arbeit am Sandkasten, Aarau: Sauerländer, 1942.
6
Viewing Zone

2  Sand table (conventional). 3  Sand table (virtual). 4–5  Jesuit church in Vienna.

and even more central perspectives, with one vantage point, no specific place at all).7 The representation in Avatar can be
position the viewer at the eye point – at least in principle: scaled up and down, it is enriched with further information,
Extreme cases are, for example, the remarkable trompe l’œil and it can be scrolled and rotated and therefore allows dif-
dome in the Jesuitenkirche in Vienna, executed by Andrea ferentiated analysis. As it was put in one review of Avatar:
Pozzo in 1703, which looks correct only from one standpoint.
And that point is explicitly marked on the floor (fig. 4–5). The Holotable was a lovely way of displaying an inter-
Screens usually direct the attention in the direction active map. As opposed to being displayed on a screen,
where the screen is. Spatial displays like a sand table or the 3D hologram allows people to view the map from all
the fictional strategic volumetric display in Avatar do not angles and have a better sense of scale. It’s also much
prescribe any specific positions for the viewer (except of better than a physical model, as it allows users to see
course that you have to look towards the display). The rep- the internal structure of the terrain and to point inside
resentation can be seen from different perspectives – in the model, it’s also more movable, updatable, can be
the literal, and that’s the point, also in the metaphorical endless and takes up less room. There’s countless ways
sense. It does not privilege any position but rather opens this can be used to display information. 8
up a viewing zone (the direct positioning of the viewer is
also avoided in parallel perspective, which is used in, e. g., On different forms of parallel perspective see Benjamin Beil, Jens Schröter,
7
simulation games, like Sim City, and is often called god’s eye Die Parallelperspektive im digitalen Bild, in: Zeitschrift für Medienwissen-
schaft 4 (2011), pp. 127–137.
view, because it prescribes no position. The gaze comes from HUDS+GUIS, The Design of Avatar UI, https://hudsandguis.com/
8
home/2011/01/16/the-design-of-avatar (accessed September 20, 2017). 149
 Jens Schröter

The display in Avatar is purely fictional, but that makes it no with normative assessments of what such futures could
less interesting: The fictional representation in Avatar is a and should mean for present-days societies.” 12 Often mov-
projection of a futuristic technological practice intertwined ies show futuristic technologies that become central and
with real developments. Often there is an immediate con- driving metaphors for a certain line of technological devel-
nection, like when technology developers become film con- opment – as became the fictional “Holodeck” from Star Trek
sultants. Kirby uses the notion of the “diegetic prototype” to for the discourses in the development of virtual reality in
designate fictional technologies that operate “normally” in the nineties.13 This is an example for how an imaginary tech-
the diegetic world of a film.9 Some people explicitly connect nology directly becomes part of a development process.
the displays in Avatar to real developments: As a technician Fictional three-dimensional displays exist in popular
discussing the possibilities of realizing such a “holo table” cinema as well as in technical papers. But in both of these
puts it in a blog post, beautifully titled “holo-tables-avatar- discursive fields they have different functions. In popular
style-are-cool”: “The solution, both hardware, software cinema the representation of future technology can work –
and computer power, seems pretty expensive. An elliptic as I analyzed in my PhD and as was discussed in the already
3D display used horizontally as a ‘holo table’, but price aside, mentioned text by David Kirby – as a means to produce the
it seems doable with today’s technology unless I’m missing desire for potential new technologies in mass audiences,
something.” 10 And then he adds: “Zebra Imaging, a long- normalize them as parts of a potential future and in this
time producer of 3D holographic prints has been awarded way help developers to receive funding, which historically
a contract by DARPA back in 2005 to develop a real-time has happened.14 In that sense Kittler once described popular
interactive holographic display map. The Urban Photoinc cinema as an instruction manual for new media.15
Sandtable Display (UPSD) is the result of that. It supports In technological papers such fictional entities help to
up to 20 participants, 360 degree view points, 12 inch depth orient researchers towards a common goal and help to get
and displays that scale up to 6 feet in length, enabling full funding too, think of the role of projected technological
Parallax without requiring special glasses or goggles.” 11 artifacts in patents. In this sense the fictional presentation
Sheila Jasanoff argues: “Science Fiction […] is a repos- of future technological practices is by no means external or
itory of sociotechnical imaginaries, visions that integrate secondary but rather an integral part of the development of
futures of growing knowledge and technological mastery
12 Sheila Jasanoff, Imagined and Invented Worlds, in: Sheila Jasanoff, Sang-
Hyun Kim (eds.), Dreamscapes of Modernity. Sociotechnical Imaginaries and
9 David Kirby, The Future Is Now: Diegetic Prototypes and The Role of Pop- The Fabrication of Power, Chicago/London: The University of Chicago Press,
ular Films in Generating Real-World Technological Development, in: Social 2015, pp. 321–341, p. 337.
Studies of Science 40 (2010), pp. 41–70. See Jens Schröter, Das Holodeck als 13 See Jens Schröter, Das Netz und die virtuelle Realität. Zur Selbstprogram-
Leitbild, in: Bildwelten des Wissens 14 (2018), pp. 90–99. mierung der Gesellschaft durch die universelle Maschine, Bielefeld: transcript,
10 Arie Tal, Augmented Reality Science-Fiction vs. Science-Fact: Are We 2004.
There Yet?, http://augmentech.blogspot.de/2012/12/holo-tables-avatar- 14 Ibid.
style-are-cool-and.html (accessed September 20, 2017). The author alludes 15 See Friedrich Kittler, Synergie von Mensch und Maschine, in: Florian
to https://zebraimaging.com (accessed September 20, 2017). Rötzer, Sarah Rogenhofer (eds.), Kunst Machen. Gespräche über die Pro-
150 11 Ibid. duktion von Bildern, Leipzig: Reclam, 1993, pp. 83–102, p. 101.
Viewing Zone

technology. In the next part I will analyze the steps in the on space. This can also be seen in a paper published in 1963,
actual development process of volumetric display technol- regarded as an important early text: “A real need exists for
ogies and how fictional representations of potential future a three-dimensional display in almost any spatial naviga-
practices are to be found in developer’s accounts and other tion problem, whether it is through water, air, or outer space.
forms of diegetic prototypes. Faster and faster vehicle velocities have outmoded visual
navigation, even when direct visual observations are possi-
ble. […] The navigator’s ability to react should not be limited
History of Volumetric Displays
by his position display.” 18
In 1948 a paper called Three-Dimensional Cathode-Ray Tube The solution to the problem of the ineffectiveness of
Displays by Parker and Wallis appeared, where they state: the human operator could be to develop a real three-dimen-
sional display: “A truly three-dimensional display is one in
Since the screen of a c. r. tube [= cathode ray tube] is which the echoes appear as bright spots in an actual volume
only two-dimensional, only two coordinates of the of light, at points representing the spatial positions of the
object’s position can be thus directly displayed. This corresponding objects.” 19 This is the decisive point in volu-
has until relatively recently been adequate, the radar metric displays. The image is not being created on a plane,
set being called upon to scan in only a single angular nor on two, as in stereoscopy; it is created in a volume. As
coordinate, usually with a ‘fan beam’, but the modern a result the image is perceived as spatial. How can this be
set may scan in two angular co-ordinates with a ‘pen- done? According to the authors,
cil beam’. It is with these volume-scanning radar sets,
where the object’s position in three coordinates is deriv- [t]he echoes are displayed in the volume of light as
able, that we are concerned here.16 bright spots, by an intensity modulation of the c. r. t.
spot. The deflections must be suitably synchronized
Obviously, the concern here is to represent spatial informa- with the scan of the aerial beam, in order that the
tion in a three-dimensional way and – since we are dealing echoes may appear consistently at points representing
with radar – to achieve this as fast as possible in critical the objects’ spatial positions. The deflection produced
situations where decisions have to be made quickly. “When mechanically can be either ‘real’ or ‘apparent’. An
a human operator is involved in the loop, however, all the example of the former would be obtained if the c. r.
n channels have to pass simultaneously through the bottle- tube itself were moved axially. This is, for mechanical
neck of his senses, consciousness and movements.” 17 The reasons, undesirable. A similar effect can be obtained,
slow human operator thus has to get optimal information however, by projecting the c. r. t. picture on to a moving

16 E. Parker, P. R. Wallis, Three-Dimensional Cathode-Ray Tube Displays, in: 18 R. D. Ketchpel, Direct-View Three-Dimensional Display Tube, in: IEEE
Journal of the IEEE 95 (1948), pp. 371–390, p. 371. Transactions on Electron Devices 10 (1963), pp. 324–328, p. 324.
17 Ibid., p. 379. 19 Parker, Wallis 1948 (as fn. 16), p. 372. 151
 Jens Schröter

screen. An ‘apparent’ deflection can be obtained, for The motivation for this work is the dream of realizing
example, by observing the c. r. t. picture in a mirror real stereovision images in space. Most of us remember
which is moved in a suitable manner.20 ( fig. 6) the scene in the 1977 movie ‘STAR WARS’ in which
the robot R2-D2 projects a three-dimensional image
Here, Parker and Wallis describe two fundamental types of Princess Leia, who begs Obi-Wan Kenobi for help.
of the class of volumetric displays that create the volume of Besides ‘STAR WARS,’ there have been many movies
the image with movable parts (“swept volume”). In the first that contain scenes in which holograms appear […].
case the screen is rotating and the light-points are projected These films indicate a desire or a premonition in many
onto it. In the second case the plane is multiplied into a vol- of us to see this kind of technology brought to life.21
ume through a translational moving mirror. This means that
similar to film, volumetric displays function on the basis of Even though the “desire for 3D” does not have to be hyped
the series of physiological optics with the addition of the into an anthropological constant, it seems that it is an actor
third dimension. Human perception visualizes a three-di- in this matter.22 Here we can already glimpse that fictional
mensional image produced by the fast succession of projec- representations do play a role in orienting research towards
tions onto the rapidly moving planes. It can (in principle) be certain goals to be achieved. Fig. 7 shows a (fictional) rep-
viewed from all sides without additional glasses. Contrasted resentation of a volumetric display from a text published in
to geometrical optics, this plane is being moved, thereby 1989. This sort of centralized traffic control room is remark-
becoming transplane. The image then appears in the volume, ably similar to representations of futuristic displays in Star
described by many authors as image-space or image-volume. Wars Episode III – Revenge of the Sith (USA 2005, George
6  Early Diagrams for a Moving
Screen and a Moving Mirror Display, Obviously, these are very primitive concepts to realize a Lucas) (fig.8).
Two Fundamental Forms Of volumetric image – at least compared with the presumably Specific institutions subject a distant place to analysis
Volumetric Display Of The Swept
Volume-Type. computer-generated smooth image in Avatar. Avatar shows and bring it under control with the help of volumetric dis-
that technological progress is, of course, unavoidable – and plays. Bruno Latour has argued that the “simple drift from
so we are in the midst of the discursive level of these imag- watching confusing three-dimensional objects, to inspect-
ing technologies. ing two-dimensional images which have been made less
Even though (or maybe because) volumetric imaging confusing” is a central technique of producing knowledge.23
technologies require extensive funding, fictional, phantas-
21 Rieko Otsuka, Takeshi Hoshino, Youichi Horry, Transpost. All-Around Dis-
magorical ideas are surrounding this topic even in research
play System for 3D Solid Image, in: Proceedings of the ACM Symposium on Virtual
projects. In a text on volumetric displays from 2004 Rieko Reality Software and Technology, Hong Kong, 2004, pp. 187–194, p. 187.
Otsuka and others state: 22 Meaning the thesis, that there is a naturally given “desire for illusionistic
images”, which sometimes seems implied in positions like André Bazin’s
The Myth of Total Cinema, in: idem., What is Cinema?, Berkeley: University
of California Press, 1967, pp. 23–27.
23 Bruno Latour, Visualisation and Cognition. Drawing Things Together, in:
Knowledge and Society. Studies in the Sociology of Culture and Present 6
152 20 Ibid. (1986), pp. 1–40, p. 15.
Viewing Zone

“No matter what they [the scientists, but one could also say:
the military, J. S.] talk about, they start talking with some
degree of confidence and being believed by colleagues, only
once they point at simple geometrized two-dimensional
shapes.” 24 However, as the above quoted discussions on vol-
umetric displays have shown, it is at least problematic if, for
locating something within an image, collaborative viewing
as well as the discussions and the decision-making connect-
ed with it are perhaps more successful with three-dimen-
sional representations. Through these display technol-
ogies space or spatial constellations themselves become
“immutable mobiles”, in the sense of Latour. 25 In this way
a spatial situation is opened up to discussions and control.
The literature on volumetric displays goes beyond our
current discussion, but in these texts one can repeatedly
find commentaries on the viability and necessity of volu-
7  Fictional volumetric display used in a sort of traffic control.
metric display technologies:

With vendors lowering the barrier to adoption by

providing compatibility with new and legacy appli-
cations, volumetric displays are poised to assume a
commanding role in fields as diverse as medical imag-
ing, mechanical computer-aided design, and military
visualization.26

Military and medical visualizations are the most mentioned

fields of applications. Most often the goal is to control spac-
es filled with people or to control the human body itself.
According to Blundell and Schwarz the means of control is
usually a god’s eye view in which the user either observes
8  Fictional volumetric display from Star Wars Episode III – Revenge of the Sith.
24 Ibid., p. 15–16.
25 See Ibid.
26 Gregg E. Favalora, Volumetric 3D Displays and Application Infrastructure,
in: Computer 38 (2005), pp. 37–44, p. 37. 153
 Jens Schröter

Volumetric systems therefore provide a ‘God’s-eye’ view of

any image scene.” 27
To conclude I will discuss a paper on a military develop-
ment of a volumetric display, which is rich in fictional projec-
tion. Fig. 9 shows a volumetric display in which a transplane
image is projected onto a rotating plane in form of a helix.
In the paper the authors discuss possible usages of this
display: “A logical application for the 3-D volumetric display
is for control and management of air traffic in a volume of
aerospace for the FAA, Air Force, or Navy.” 28 It should be
noted that in this military setting that only men are watch-
ing the display and thereby direct their controlling gaze on
the targets (be they hostile or friendly), although women
are not excluded from the military in the USA (fig. 10). The
corresponding text explains:

The Department of Defense Science and Technology

Initiative identifies seven thrust areas. One of these is
Global Surveillance and Communications, a capabil-
ity that can focus on a trouble spot and be responsive
to the needs of the commander. A three-dimensional
display of the battle area – such as the LaserBased 3-D
Volumetric Display System – will greatly facilitate this
capability. Tactical data collected for command review
9  Volumetric Display in which a transplane image is projected onto a
rotating plane in form of a helix. can be translated and displayed as 3-D images. The
perspective gained will contribute to quicker and more
accurate decision-making regarding deployment and
a space from outside or is able to effortlessly penetrate the management of battle resources.29
body: “In the case of all volumetric display systems known
to the authors, the generation of images occurs within a 27 Barry Blundell, Adam Schwarz, Volumetric Three Dimensional Display Sys-
tems, New York, NY: John Wiley & Sons Inc, 2000, p. 4.
containing vessel from which the [observer] is excluded. 28 Parviz Soltan et al., Laser-Based 3D Volumetric Display System, in: Rich-
ard M. Satava et al. (eds.), Interactive Technology and the New Paradigm for
Healthcare, Amsterdam et al.: IOS Press, 1995, pp. 349–358, p. 356.
29 Parviz Soltan et al., Laser-Based 3-D Volumetric Display System (The
154 Improved Second Generation), 1996, http://handle.dtic.mil/100.2/ADA
Viewing Zone

10  Usage of a (Fictious) Volumetric Display 11  Usage of a (Fictious) Volumetric Display for the Control of a Female
for a Command-and-Control Situation. Body during Birth.

This is quite explicit. laborative work. 30 And the discourses on these images are
In fig. 11 it is a woman who is at the center – but in this full of fictionalized projections about their possible future
case she is the object of the medical gaze via the volumet- uses especially in military and medical practices, centered
ric display. It seems as if there is a gendered bias of the on control and surveillance. These potential image technol-
space-controlling gaze. ogies are understood as assembling and situating people in a
This is only a very small fraction of the rich literature situation room or a center of calculation to control a situation.
on volumetric display technologies, but some elements are Such projections of futuristic medial practices are of
clearly visible: These displays aim to produce truly three-di- course more revealing about the present than about any
mensional and therefore post-perspectival images. These future to come. Obviously, all the different extrapolations
images render scenes and situations, in which the spatial on volumetric display technologies converge in that they
structure is of paramount importance, without the distor- are mainly used for strategic planning, control, surveillance,
tions of perspective. Because the images do not prescribe
and situate viewers in any strict sense, they are ideally suit-
30 Of course: In difference to flat images, where everyone sees the same image
ed to be seen, discussed and used by a group of people in col- (even if not standing on the position prescribed by the perspectival con-
struction), while watching volumetric images everyone in a group of observ-
ers sees a different image – as would be the case with real three-dimension-
al objects. That the image does not prescribe a certain viewpoint opens up
306215 (accessed August 30, 2007), p. 17. the space for several viewpoints, which can be brought into discussion. 155
 Jens Schröter

analysis – in Avatar to plan the exploitation of Pandora by First there is the obvious problem that it might be too
the military-industrial complex. Avatar is explicitly about big an argument to correlate a certain form of representa-
anthropocracy, about humans trying to establish their con- tion with a certain anthropocentric episteme. Might not
trol over the resources of a foreign world by, amongst other the anthropocentrism of perspective be more gradual and
things, three-dimensional displays. The following part will depend on different practices with perspectival images?
analyze the cultural imaginary of anthropocratic power as These were arguments already made to criticize so-called
sedimented in the discursive history of volumetric displays – apparatus theory. 33
be they real or fictional. Closely connected with this is, secondly, a more funda-
mental problem with Panofsky’s argument. Perspective was
invented in the renaissance. But it was only decades later,
Volumetric Display as Symbolic Form
namely in the 19th century, that industrial modernity had
Erwin Panofsky’s famous paper Perspective as Symbolic its big takeoff, an anthropocracy if you will, resulting now-
Form ends with the sentence: “It is thus no accident if this adays in ecological disaster. And here is the central point:
perspectival view of space has already succeeded twice in Very important forms of representations to be used in this
the course of the evolution of art: the first time as the sign upheaval for the constructing of technologies, for increas-
of an ending, when antique theocracy crumbled; the second ing the effectiveness and speed of individuals in decision
time as the sign of a beginning, when modern ‘anthropocra- making etc. were not at all structured by linear or central
cy’ first reared itself.” 31 Here anthropocracy is directly con- perspective, but were, as I said, parallel-perspectival or
nected with (central-)perspectival representation. Panof- material 3D models or maps – e. g. in engineering drawing,
sky’s argument is, put simply, that perspective shows the meaning the transfer of technological knowledge or archi-
world as seen or at least: approximately as seen by a human tecture. Although parallel perspective and maps on the one
observer, man becomes the center of the shown world: “This hand and real volumetric representations on the other are of
view of space […] is the same view that will later be rational- course different in that the former two are still forms of pro-
ized by Cartesianism and formalized by Kantianism” 32 – and, jection, where the latter is not (it is more a scaling and filter-
as we all know, in Descartes the cogito is the only secure ing), they are similar in that they do not imply a positioned
knowledge and in Kant the world appears only according viewing subject. 34 Does that mean, when we follow Panof-
to the transcendental structures of consciousness. Man is sky’s argument, that they are not anthropocentric because
in the center. Panofsky argued that perspectival represen- they do not imply a viewing (although one-eyed) body?
tation is an expression and/or one performative realization Yes, perhaps that is what it means – but in a very special
of an anthropocentric worldview. sense: The forms of power relevant during the renaissance

31 Erwin Panofsky, Perspective as Symbolic Form, trans. by Christopher S. 33 See Hartmut Winkler, Der filmische Raum und die Zuschauer. ‘Apparatus’ –
Wood, New York, NY: Zone Books, 1991, p. 72. Semantik – ‘Ideology’, Heidelberg: Carl Winter, 1992.
156 32 Ibid., p. 66. 34 An exception may be the implied position of potential observers in sculpture.
Viewing Zone

were mostly feudal forms, implying personalized forms of In orthographic projection the projectors do not all
power, e. g. peasants in serfdom to their landlords or power converge to a point, but remain parallel. Because this is
concentrated in the two bodies of the king, according to not the way we see things, orthographic drawing seems
Kantorowicz. 35 Panofsky’s perspectival anthropocracy is a less easy to place. It does not correspond to any aspect
personalized form of power, the world made to conform to of our perception of the real world. It is a more abstract
the gaze of an idealized person. One shouldn’t forget that and more axiomatic system. [...] The advantage of
in some types of baroque theater architecture the emperor orthographic projection is that it preserves more of
or another type of king had the only place from which the the shape and size of what is drawn than perspective
perspectival scenery on the stage was completely coherent – does. It is easier to make things from than to see things
here the body of the emperor and the eye point of central with. 37
perspective were literally matched. 36
But in modernity, as has often been noted, personalized These abstract, measurable representations – think of how
power disappears and new objectified, anonymous forms of in Avatar the three-dimensional representation is enhanced
power took its place; a power we describe as Sachzwänge, by information – are the expression of modern power, which
factual constraints, the subject of deep analysis by Marx, has always already transcended human standards and
Weber, Schelsky and others. scale. 38 The combination of spatiality, the enhancement of
A question becomes unavoidable: Can we formulate the collaborative work and the saturation of images with infor-
speculative thesis that the view from nowhere – in parallel mation is characteristic for a wide field of display technolo-
perspective, material models and volumetric display tech- gies. Such display technologies are less the expression than
nology – is the view of objectified power, which no one in the performative realization of modern power. The view is
particular possesses? Which is only to be found in abstract not a god’s eye view – but it is the view of the successor of
structures – as Marx, Luhmann and Foucault, for example, god, what was already precisely described by Benjamin in
have shown in very different ways? his beautiful fragment Kapitalismus als Religion – it is the
In parallel perspective it is more important that the rel- view of Capital (or of capital and the military as one of its
ative length of the lines and the angles are preserved, that executive forces – as is shown in Avatar). 39 It is a view or a
you can measure it: It is a representation of the object in
itself and not as someone sees it, or as architectural theorist
37 Robin Evans, Architectural Projection, in: Eve Blau, Edward Kaufman
Robin Evans put it:
(eds.), Architecture and Its Image. Four Centuries of Architectural Repre-
sentation. Works from the Collection of the Canadian Centre for Architecture,
Montreal, Cambridge, MA: The MIT Press, 1989, pp. 18–35, p. 20.
38 See Jens Schröter, Tristan Thielmann (eds.), Display I: Analog, in: Naviga-
tionen 6.2 (2006) and Tristan Thielmann, Jens Schröter (eds.), Display II:
35 See Ernst H. Kantorowicz, The King’s Two Bodies. A Study in Medieval Polit- Digital, in: Navigationen 7.2 (2007).
ical Theology, Princeton, NJ: Princeton University Press, 1957. 39 See Walter Benjamin, Capitalism as Religion [Fragment 74], in: Eduardo
36 See John Peacock, Inigo Jones’s Stage Architecture and Its Sources, in: The Mendieta (ed.), The Frankfurt School on Religion. The Key Writings by the
Art Bulletin 64 (1982), pp. 195–216. Major Thinkers, New York/London: Routledge, 2005, pp. 259–262. 157
 Jens Schröter

gaze that makes something measurable and quantifiable as Figures

well as controllable and reproducible. 1a–c  20th Century Fox, James Cameron, Avatar, USA/UK 2009, TC:
But isn’t that too big a thesis too? Are all god’s eye rep- 00:47:49, 00:48:03, 00:48:10.

resentations the gaze of capital? I guess that would indeed 2  Cheryl Rodewig, US Army.

be too strong. But in certain practices these types of rep- 3  John Fischer, US Navy.

resentations become operational for domination – and it’s 4-5  Jens Schröter.

precisely their non-linear-perspectival character and their 6  E. Parker, P. R. Wallis, Three-Dimensional Cathode-Ray Tube Displays, in:
Journal of the IEEE, 95 (1948), pp. 371-390, p. 373.
non-positioning of viewers that is their strength. And the
7  Rodney Don Williams, Felix Garcia, Volume Visualization Displays, in:
least one could say is that the potentially subjectivist and Information Display 5 (1989), pp. 8-10, p. 9.
anthropocentric linear perspective is not the characteristic
8  Star Wars Episode III – Revenge of the Sith (USA 2005, George Lucas).
expression and technology for modern power.
9  Reprinted from Parviz Soltan et al., Laser-Based 3D Volumetric Display
System, in: Richard M. Satava et al. (eds.), Interactive Technology and the New
Paradigm for Healthcare, Amsterdam et al.: IOS Press, 1995, pp. 349-358, p.
352, with permission from IOS Press.
10–11  Parviz Soltan et al., Laser-Based 3-D Volumetric Display System
(The Improved Second Generation), http://handle.dtic.mil/100.2/
ADA306215 (accessed August 30, 2007), pp. 16-20.

158
Carolin Höfler

Sense of Being Here

Feedback Spaces Between Vision and Haptics

Vertiginous Bodies according to which it would be possible to generate virtual

experien­ces able to persuade and convince our senses:
Facebook, Samsung, Google, and HTC are making headway
on the mass consumer market with their latest virtual real- The ultimate display would, of course, be a room within
ity glasses, promising users a total immersion into picto- which the computer can control the existence of matter.
rial worlds. By means and with the help of head-mounted A chair displayed in such a room would be good enough
displays – as the advertising slogans unanimously stress – to sit in. Handcuffs displayed in such a room would be
virtual reality (VR) will eventually become authentic and confining, and a bullet displayed in such a room would
real. It is in particular the combination of two technological be fatal. 1
methods that causes hope for increasing the realism of an
observer’s visual experience: Stereoscopic 3D and precise, In the realm of the design and development areas of
low-latency head tracking are supposed to boost the impres- human-computer interaction, virtual and augmented real-
sion of being not just in front of an image but being part of ity, and game and interaction design today, the discourse
an image space and interacting with things and process- centers more than ever around a fusion of the virtual and
es happening there. If the movements watched via head-­ physical space into a mixed reality that is prevalent in
mounted displays do genuinely match the perception of body research and discussion. In that respect, mixed reality is
movements, this will not only alleviate the user’s feeling of understood to consist of environments and systems aiming
motion sickness but also strengthen the user’s impression at coupling a real-physical action space to a synthetic, com-
of realness of the visually perceived environment. In this puter-generated image space – with those two spaces hardly
perceptual situation, the user might stress that the techno- distinguishable from each other.2 In such a mixed reality, it
logical system can be controlled, thus providing the virtual
system with credibility and enhancing its acceptance. Ivan E. Sutherland, The Ultimate Display, in: Information Processing 1965.
1
Proceedings of IFIP Congress 65.2 (New York, May 24–29, 1965), pp. 506–
The promise of a nullification of the discrepancies 508, p. 508.
between virtual and physical reality immediately evokes Paul Milgram, Haruo Takemura, Akira Utsumi and Fumio Kishino, Aug-
2
mented Reality. A class of displays on the reality-virtuality continuum, in:
associations related to the prophetic description of the Proceedings of SPIE. Telemanipulator and Telepresence Technologies 2351
­Ultimate Display by Ivan Sutherland in the year 1965, (Boston, October 31–November 1, 1994), pp. 282–292.
 Carolin Höfler

is either the physical surroundings that are enriched with this holistic view of space is to be supported by ­f urther
virtual information, like in augmented reality scenarios, or sensory perceptions. What the new VR installations and
it is the virtual surrounding that is extended by physical experiments have in common is the relation to the recipi-
information, like in virtual reality applications using tactile ent’s body: The recipient no longer serves as a pure watcher
feedback. Both variations are considered to represent inter- alone, instead their entire body should be capable of expe-
im stages in the so-called “reality-virtuality-continuum”, as riencing the virtual surroundings. 5 On the one hand, the
the researchers Paul Migram and Fumio Kishino stressed impression of realness of the virtual world is supposed to
in their writings from the early 1990s. 3 be reinforced through real-world elements in virtual ­spaces.
The discourse on a seamless fusion of the physical On the other hand, it is intended to let the virtual world
and virtual worlds grows in importance in the current appear even more intensely than its physical counterpart. In
discussion on “phygital” objects and surroundings which order to find stimuli to actually achieve the state of intensi-
are touted as new forms for a computerized world access.4 fied perception, facets and spots of virtual 3D environments
Having its origin in the USA, the marketing term phygital are brought into the game, physically materialized, and
consists of the words physical and digital and refers to the internalized into the VR setting.
linkage of physical objects and surroundings with their One such work trying to intensify and enhance the user’s
virtual representation, or vice versa the coupling of virtu- perception is delivered by the Australian game developing
al things to a physical equivalent in a digital network. The collective Toast with its project Plank Experience.6 The recip-
network enables these things and surroundings to connect ient, with his VR glasses on his head, finds himself situated
and exchange data. The development of phygital objects in a big city, heading up the elevator of a skyscraper. The door
1 Toast, Plank Experience, 2016.
Exhibition view, art association and spaces is in particular spurred by mobile display tech- opens high above the skyline. A plank juts out over a deep
Frankfurt (Frankfurter Kunstverein), niques, through which the interaction between display and abyss. According to the real physical presentation space, it
2017.
user is enhanced. In contrast to augmented reality scenarios, is just a wooden board on which the VR user balances (fig. 1).
in which digital elements can be blended into the physical But for the recipient, perception changes since the pictures
environment, phygital applications instead use physical of the VR glasses make him believe that he is stepping onto a
elements to be projected into a VR setting. In both cases, loose and life-threatening balancing plank, while a ventilat-
however, the key idea is a fusion of both worlds. ing fan blows into his face simulating wind (fig. 2). His glass-
In the discourse on the melting of the boundaries es let him look into dizzying depths with the wind growing
between physical, virtual and imaginary worlds, much stronger and his body reacting more and more energetically.
attention is paid to the idea of the holistic nature of the space
as a whole being primarily seen. In current developments, For example, see the exhibition Perception is Reality. On the Construction of
5
Reality and Virtual Worlds, exhibition, curated by Franziska Nori (Frank-
furt/M., Frankfurter Kunstverein, October 7, 2017–January 7, 2018), https://
Ibid., p. 283.
3 fkv.de/en/content/perception-reality-construction-reality-and-­v irtual-
For example Alica Rosenthal, Phygital Marketing. Die analoge und digi-
4 worlds (accessed January 1, 2018).
tale Welt verschmilzt, https://webmatch.de/blog/phygital-­marketing-die- Toast, Richies’s Plank Experience, https://toast.gg (accessed January 1,
6
160 analoge-und-digitale-welt-verschmilzt (accessed January 1, 2018). 2018).
Sense of Being Here

2 Toast, Plank Experience, 2016. Rendering.

With the display on his head, the user fumbles around with swings, the higher one gets. Thus, the swing as an interface
his feet. The physical ground is hard and stable, but the pic- intensifies the physical perception in such a way so that not
tures give the impression that the user is about to fall. only the sense of sight and touch are stimulated, so is the
There are many examples of similar projects and pieces sense of balance. Such a sensual state of involvement of the
of work that are willing not only to stimulate the eye but are viewer leads to a special kind of strengthening and mobi-
trying to stimulate all senses, for instance the project Swing lizing one’s physical senses and powers: the picture direct-
by Christin Marczinzik and Thi Binh Minh Nguyen. Here, ly located in front of the viewer, diminishing the distance
wearing one’s VR glasses, one can place oneself on a swing between subject and object, between the person watching
and gaze out at a wonderful landscape.7 The stronger one and the things being watched, makes the (re)acting body
become itself a medium for its individual imagination and
self-deceit.
Christin Marczinzik, Thi Binh Minh Nguyen, Swing VR. An Immersive VR
7
Experience, http://christin-marczinzik.de/portfolio/swing-vr (accessed
January 1, 2018). 161
 Carolin Höfler

which the user may have the intuitive experience of being a

bird. In contrast to ordinary flight simulators, the user takes
the flight position of a bird rather than simply piloting the
run-of-the mill airplane or spaceship; and, in doing so, he
interacts with the installation via whole body movements
that mimic a bird in flight. To more intensely experience
the scene presented in VR, the viewer is lying face down
on a rack with two wings which he can grasp ahold of and
move up and down through vigorous arm movements (fig. 3).
With the palm of the hand spinning upward, one gains alti-
tude, the palm of the hand spinning down, one loses height.
During gliding flight, the viewer will have a soft wind blow-
ing in one’s face, generated, similar to Plank Experience, by
a ventilating fan. The airflow increases with increased
flapping. In order to achieve this state of acting, and being
a genuine bird, Birdly uses photorealistic 3D graphics pro-
vided via head-mounted displays of the newest generation
and sensorimotor couplings. The display shows pictures of
a flight over Manhattan, Dubai, or Singapore (fig. 4). The
3 Somniacs, Birdly, 2015–2018. Photo.
bird’s-eye perspective emphasizes panoramic views and
monumentalizes the surrounding spaces. The recipient can-
Trompe-corps not see all of the scene before him at once but rather has to
look back and forth, side to side. This gives the impression
A corporeal experience even further out there is the inten- of a closer proximity and a stronger involvement for the
tion behind the installation Birdly by designer Max Rheiner, recipient in flight.
which provides the viewer with fictional pictorial spaces But it isn’t the immersive pictures alone that involve
to explore along with a fictitious body as well.8 Birdly is the user in this fictional space and special pictorial scene;
intended to make movements and situations instrumentally rather it’s a physical-material apparatus through which the
perceptible – movements that are not normally tangible for special flight movements in these pictorial spaces can be
the human body. It is about modeling an artificial nature in steered, managed, and controlled that really solidifies the
illusion. The body’s physical presence experiences much
See Max Rheiner, Birdly, Zurich University of the Arts (ZHdK). Interaction
8 greater feedback, reinforcing the user’s experience of the
Design, 2013–2014, http://iad.zhdk.ch/de/projekte/birdly (accessed Janu-
ary 1, 2018). In 2015, Rheiner founded out of ZHdK the company Somniacs
physical world while – paradoxically – the actual physical
162 which established Birdly as a commercial product. space in which the apparatus and the display and the user
Sense of Being Here

4  Somniacs, The Birdly aerial view of Manhattan, New York, 2015. VR Rendering (Oculus Rift). Generated by an earlier version of the simulator (current version
Birdly Serial Edition, 2016).

are s­ ituated is left aside and replaced by totally different Maak recently pointed out: “The classical ‘Trompe-l’œil’ is
virtual surroundings, similar to typical flight simulators and followed by the ‘Trompe-corps’.” 10 But do these VR scenar-
VR games.9 Thus, the rather small laboratory-like indoor ios amount to nothing more than deceiving the recipient’s
space is turned into an urban, large-scale outdoor space body, for example by provoking dizzying heights? Is not
being viewed from dizzying heights. another main idea here to allow the body to (re-)identify
These and similar other media installations represent with the physical world? Plank Experience not only aims
an attempt at intensifying bodily perception and playing at the body’s deception through imagery, it also aims at
down common knowledge about real-physical surroundings the image’s materialization through the body. The image
and situations as well, as German architectural critic Niklas
10 Niklas Maak, Kunst und virtuelle Realität. Der schwindelnde Körper, in:
Frankfurter Allgemeine Zeitung, January 5, 2018, http://plus.faz.net/feuille-
Principally Michael Friedman, Head-Mounted Display Screens. A (De)
9 ton/2018-01-05/der-schwindelnde-koerper/99295.html (accessed January
construction of Sense-Certainty, in: MediaTropes VI.1 (2016), pp. 114–136. 5, 2018). Translation from German by the author. 163
 Carolin Höfler

becomes real if the beholder is standing on physical ground, virtual space are physically reconstructed and are thus
for instance, even if that ground is portrayed spatially and perceptible for the body. VR glasses seem to appear trans-
visually differently. The sensory and motor perceptions, the parent since they let the viewer look into a photorealistic,
experiences of balance and imbalance, cause the body to stereoscopic digital model of his direct environment. Thus,
reconnect to the physical space. However, the vision stays only a comparison between the contrasts and similarities of
unbound by physical and practical limits. The visual per- the virtual pictorial space and the physical model space can
spective can be reinforced by other sensory impressions, for serve as a tool to gain reliable and informative insights on
instance caused by a ventilating fan, serving as boosting ele- the construction of reality and virtual worlds, as the com-
ments for heavy bodily reactions to digital imagery. puter scientist and founder of the Media Research Lab at
The recollection of things, actions, and experiences New York University, Ken Perlin, recently stressed.12 It is
in virtual space is often more intense than recollecting rather through the analysis of these alternative experien­
an experience in physical space. In the virtual space, it is ces of space that the new conditions of human perception
possible to experience situations that can’t be felt in the in relation to technically constructed realities can be made
physical space, situations that are maybe too dangerous to tangible. In order to illustrate the analytic potential of vir-
expose oneself to in the real world. Involved in interactive tual-physical spaces, a case study from the game and leisure
moving images, and in line with the theory of enactivism, sector will serve as an example.
the body directly and actively frames and generates an
experience that is stored not as a virtual imagination but
Phygital 3D Spaces
as a real experience.11 As a consequence, associations and
meanings, which the recipient assigns to the physical and In 2016, a US start-up in the city of Pleasant Grove, Utah,
media-generated impressions, do blur and shift. What this opened an amusement park called The Void which features
shift can ultimately evoke and entail for our perception, con- something quite different from the commonly known fun-
sciousness, and memory is not yet clear and remains a little fair rides and flying constructions. Visitors to this park are
researched area. equipped, rather, with advanced VR hardware in order to
Closely related to this is the question of how and how interact with virtual and physical 3D settings and accom-
much our perception changes if the physical room where the plish feats they could not otherwise experience due to
user is located is not replaced by fictitious pictorial worlds spatial and physical limits of the real world and since the
but instead digitally constructed and can be operated and laws of physics cannot be repealed, like fighting dragons or
steered via VR glasses. By the same token, the question even being hit by enemy bullets. Visitors wear data glasses
­a rises as to how our perception changes if facets of the called Rapture HMD with two curved, extremely high-res-
olution screens, integrated headphones, and a microphone.
11 Thiemo Breyer, Philosophie der Verkörperung. Grundlagen und Konzepte,
in: Gregor Etzelmüller, Annette Weissenrieder (eds.), Verkörperung als
Para­digma der theologischen Anthropologie, Berlin/Boston: De Gruyter, 12 Munich ACM SIGGRAPH Chapter, Ken Perlin – Prototyping the Future,
164 2016, pp. 29–50, p. 43. https://vimeo.com/145127565 (accessed January 1, 2018).
Sense of Being Here

5  A player at The Void; Player’s view, Pleasant Grove, Utah, 2015/16. Screen shots.

Additionally, they wear a waistcoat and gloves comprising intervenes in the game process, thereby exercising partial
numerous sensors for haptic feedback and body tracking. control and managing the moving images projected onto the
With this virtual reality hardware, they pass through an digital 3D surfaces. The intended loss of control, primarily
ensemble of physically built rooms with numerous motion caused by the unfamiliar linkage between complex virtu-
and interaction sensors. The Void’s key component is a pitch al-physical image space data, is thus an essential constituent
called the Gaming Pod, an area of almost 330 square meters of the game; it adds to the thrill.
with a collection of moveable walls creating a maze of bend- “The goal is to attain total immersion”, claims James
ing corridors. When players walk on virtual trails, through Jensen, co-founder and Chief Visionary Officer of The Void.13
these corridors, they can move freely, without being pushed In order to incorporate players more (and even fully) into
to an obstacle. The physical boundary and the objects of the the physical-digital settings of reality, special mechanisms,
pitch are integral components of the digitally constructed tools, and devices are provided to let players feel sensations
3D image spaces. The physical game architecture is digitally like heat, cold, humidity, vibrations, height differences, and
remodeled and visualized and inserted into the VR glasses, to allow them to touch objects or perceive bad or pleasant
where it overlaps with interactive moving images and 3D scents. Gaming sickness, as a common characteristic of many
figures (fig. 5). These projections appear exclusively in the photorealistic PC or console games, is eliminated, the pro-
virtual space, whereas the player’s actions are carried out viders of The Void stress, citing the doubling of two differ-
simultaneously both in the physical and virtual space. Thus, ent spaces as a main reason: “Any movement made in the
head-mounted displays and tactile interfaces are ideally
called upon to give consistent information in order to trig- 13 James Jensen, as cited in Angela Gruber, Virtual Reality Theme Park: The
Void. Der erste virtuelle Freizeitpark, in: Die Zeit, June 8, 2015, http://zeit.
ger parallel sensations. In this setting, the visitor, equipped de/digital/games/2015-06/virtual-reality-the-void-freizeitpark/ ­komplett
with a display and wearable devices, physically and digitally ansicht (accessed January 1, 2018). Translation from German by the author. 165
 Carolin Höfler

v­ irtual world is to the visitor also a movement made in the gloved hands on the display, but does not wear gloves at all
real world. The visitor doesn’t feel any discrepancy, so he (fig. 5)? Is it possible to replace sensation with imagination?
will not feel ill. The game is real.” 14
The Void’s promise thus consists of a fully-immersive
Hierarchy of the Senses
imaginary world, of an extension of images brought into
the depths of space for viewers, with their visual perception Phygital experiences like The Void are based on a holistical-
being boosted through given and complementary, mutually ly oriented model of the structure of the senses according
supportive, sensory impressions in the physical space. But to which the senses, being part of different fields of percep-
what kind of idea and conception of the senses and their tion, render synthetic performances in the process of the
peculiarities is this promise motivated and influenced by? constitution of space. Until today, this idea of an entity of
What does it mean for our perception, orientation, and nav- the senses, and also of its hierarchical structure, has char-
igation if digital real-time images reproduce the surround- acterized the interpretation and the application of imaging
ings but deprive us of a direct view of the physical space – if techniques. Although sensory perceptions are closely inter-
the visual space (Sehraum) and the tactile space (Tast­raum) connected, physical-digital VR environments, in particu-
are disconnected, only to be reconnected through digital, lar in the gaming and entertainment area, are based on a
moving images of the surrounding space generated in real model of perception in which vision plays the key role and
time?15 What dependencies and reciprocal relationships ranks first, followed by the senses of hearing, touch, taste
exist between the physical here and the pictorial there? What and smell. For example, this is expressed by the fact that the
categories are being developed if the depicted and discon- geometric forms of physical objects and surfaces are radical-
nected space complies with its physically real spatial dimen- ly reduced and simplified, whereas the interactive moving
sions, but differs in its qualities? What perception shifts and images gain in detail and complexity. The method of screen-
scaling effects come into play or originate if – as it is the based, stereoscopic vision, which aims at heightening and
case with regard to The Void – a stage set architecture of intensifying the visual perception, can thus be regarded as
simplified form and materiality can be recognized by touch, a possible starting point for the reconceptualization and
while a space of high density, fluidity, light, and informa- radical expansion of the traditional hierarchy of the senses
tion is experienced visually? How does the visual perception with vision at its top. The classificatory scheme, in which
influence our tactile perception if the VR player sees his priority is given to the sense of sight, is now motivated and
influenced by the use and interpretation of mobile display
14 Ibid. techniques.
15 The matter of the reciprocal relationships between the visual and tactile This revived dispute on the interaction between phys-
space in VR settings was a subject at the interdisciplinary conference Mit
weit geschlossenen Augen. Virtuelle Realitäten entwerfen (Eyes Wide Shut. ical and cognitive performances in the experience of space
Designing Virtual Realities), May 31–June 1, 2017, at KISD – Köln Inter- can be regarded as the continuation of a tradition stem-
national School of Design of TH Köln, organized by Carolin Höfler and
Philipp Reinfeld in cooperation with the Institute of Media and Design of
ming from the ideas of sensualistic aesthetics in the 18th
166 TU Braunschweig. century. Following that tradition, theorists and architects
Sense of Being Here

developed a perspective towards the idea of built space, the “tactile space” to the “face space”, as Schmarsow points
stressing that it only exists in dependence upon the recipi- out – space comes into being as a concatenation of mental
ent and the entirety of his perceptions and emotions.16 The imagery.19 It was philosopher Edmund Husserl who recapit-
decisive idea behind it was the approach of an emotional ulated – skipping the idea of psychologism – the approach of
merger of subject and object in aesthetic perception, which a sensomotoric linkage of all senses and the sensuous-bodily
later, in 1872, would be coined by the German philosopher state of perception more systematically. Modern phenome-
Robert Vischer as the term of “Einfühlung” or “empathy”, nology of space, as Husserl has established it in his lectures
as it is often translated.17 Instead of following traditional at the beginning of the 20th century, assumed the impres-
proceedings of representation, ornamentation, and iconog- sion of spatiality to be in connection with the awareness of
raphy, aesthetic efforts at that time were rather aimed at one’s own body movement and thus the result of a lasting
reconceptualizing architecture in the context of a synaes- sequence of perception.20
thetic, optical-tactual perception. Insight into that shift of Referring to these phenomenological and psychological
perspective provide, for example, the essays Prolegomena approaches of the first half of the 20th century, Hungar-
zu einer Psychologie der Architektur (Prolegomena to a Psy- ian philosopher Alexander Gosztonyi tried to define the
chology of Architecture) by Heinrich Wölfflin, Ueber den peculiarities of the senses in his Grundlagen der Erkenntnis
Werth der Dimensionen im menschlichen Raumgebilde (On (Fundamentals of Knowledge) from 1972.21 In his analysis,
the Importance of Dimensions in Human Spatial Creation) he not only took into account the classical senses like sight,
by August Schmarsow, and Das räumliche Sehen (The Spatial hearing, taste, smell, and touch but also the “senses of bodily
Vision) by Paul Klopfer.18 Instead of having an idea of space feeling” categorizing the senses of vibration, temperature,
as an immutable entity, the named authors advocate for a balance, gravity, and proprioception as the most important
more dynamic principle according to which space is gener- ones.22 His interest was mainly aimed at analyzing how the
ated in the very moment of perception. The idea of a mov- different senses either compete or diffuse.
ing, active recipient is thus a prerequisite for space and its A constitutive element for the dominant physiologi-
creation. From the bodily movement on – in transition from cal-rational understanding of human-computer interac-
tion, virtual and augmented reality, as well as game and
16 An overview on the aesthetics of empathy is given by Jörg H. Gleiter, interaction design, is primarily Gosztonyi’s emphasis on
Architekturtheorie heute, Bielefeld: transcript, 2008, pp. 113–126. the “quality of reality” according to which every sense is
17 Robert Vischer, Über das optische Formgefühl. Ein Beitrag zur Ästhetik,
supposed to have a “quality of realness” (Wirklichkeits-
Leipzig: Hermann Credner, 1873.
18 Heinrich Wölfflin, Prolegomena zu einer Psychologie der Architektur
(1886), in: Idem, Kleine Schriften (1886–1933), Basel: Benno Schwabe & Co.,
1946, pp. 13–47; August Schmarsow, Ueber den Werth der Dimensionen 19 Schmarsow 1896 (as fn. 18), p. 50, pp. 54–55.
im menschlichen Raumgebilde, in: Berichte über die Verhandlungen der 20 Edmund Husserl, Ding und Raum. Vorlesungen 1907, Den Haag: Martinus
Königlich Sächsischen Gesellschaft der Wissenschaften zu Leipzig. Philolo- Nijhoff, 1973.
gisch-Historische Classe 48 (1896), pp. 44–61; Paul Klopfer, Das räumliche 21 Alexander Gosztonyi, Grundlagen der Erkenntnis, Munich: C. H. Beck, 1972,
Sehen, in: Zeitschrift für Ästhetik und Allgemeine Kunstwissenschaft XIII pp. 67–97.
(1919), pp. 135–149. 22 Ibid., pp. 67–68. 167
 Carolin Höfler

wert), sub­dividing it into a “quality of reality” (Realitäts- Feedback Design

wert) and a “quality of evidence” (Evidenzwert).23 According
to Gosztonyi, the sense of touch serving as sense of near- The new entanglements between the physical “form of being”
ness (Nahsinn) therefore has a high quality of reality since (Daseinsform) of the tactile space and the digital “form of
it allows for a feeling of material resistance, whereas the effect” (Wirkungsform) of the visual space in phygital VR
sense of sight has a lower quality of reality. The sense of environments fundamentally alter the idea, concept, and
sight serving as sense of farness (Fernsinn), however, is sup- design of architectural spaces.25 With regard to the realm
posed to have a high quality of evidence as it allows having of designing, the construction of specific spaces of action
oversight of and insight into complex formal contexts. But, characterized by the interplay between human bodies, tech-
as Gosztonyi posits, only the interplay of quality of reality nical things, and physical surroundings is currently coming
and quality of evidence determines the degree of realness more and more to the focus of attention. On the one hand,
of the environment being perceived. the built spaces are being cross-linked through chips, tags,
Even if Gosztonyi identified the reciprocal effects of and sensors; on the other hand, they are designed to evoke
the senses as prerequisites for the construction of reality, specific sensory experiences. This means that the material
he assumed a hierarchical structure to exist: “The sense of surfaces and objects of the physical space are modeled in
touch is not dominant. […] The one who sees subordinates such a way that certain (expected or desirable) sensory per-
the things touched, ranks qualities and forms of touch, and ceptions, environmental experiences, and behavior patterns
arranges it in order according to his field of view.” 24 Such come into being in the virtual space. Relevant to the design
traditional approaches of the peculiarity and the hierarchy and the construction of the physical space is the question
of visual and tactile-haptic sensory perceptions had a lasting of how to develop its parameters so that the VR glasses user
effect on the debate on physical-virtual realities. It is the will accept the virtual space as a real space. How can one
idea of the tactile sense as a simple pressure sense with a design, create, and arrange a physical space and its form so
high quality of reality that currently dominates the design that the impression of a sensory and emotional immersion,
of mobile devices and interactive surroundings whose inter- of control and intervention in virtual environments can be
faces give haptic feedback. In contrast to the forms of sight, strengthened and best achieved?
the forms of touch are rather poorly developed. This question is based on the assumption that the dis-
play user will accept the virtual surroundings as realistic
and authentic as possible if he can move through them as
naturally as possible. However, body perception can some-
times be deceptive, especially if the recipient wearing the
23 Ibid., p. 68. head-mounted display has no visual access to the space
24 Ibid., p. 81. Translation from German by the author. Original quote: “Der being identified by touch. The intense visual perception
Tastsinn ist nicht dominant […]. Der Sehende ordnet das Ertastete den
Sehformen unter und ordnet die Tastqualitäten und die Tastformen in das
168 Sehfeld ein.” 25 Schmarsow 1896 (as fn. 18), p. 50.
Sense of Being Here

can alleviate other physical experiences: Visually perceived

paths are therefore usually different from physical paths. It
is this discrepancy between physical and virtual movement
that the approach of redirected walking takes into account.
Redirected walking assumes that the display user, while
passing through virtual worlds, is relatively insensitive to
rotations and curves and underestimates egocentric dis-
tances.26 No matter how hard a test person tries to walk in
a straight line while blindfolded or with VR glasses on his
head, he often ends up going in circles without realizing
it. These observations are utilized for the construction of
physical-digital VR environments, in particular in those
cases where the physical space is limited compared with
the potentially infinite virtual space. As a result, the display
user is physically guided around a curve while he thinks
he is moving straight. According to current knowledge, it
only requires a 22-meter radius in order to make the user
think that he is walking a straight line, while he is actually
walking in circles.27
6  Keigo Matsumoto & Team, Unlimited Corridor, University of Tokyo, 2016. Photo.
A typical space configuration that permanently redi-
rects the walking direction is the Unlimited Corridor devel-
oped in 2016 by engineers and computer scientists of the Virtual crossroads and turnoffs are physically reproduced
University of Tokyo in cooperation with the US company through an additional corridor in the center. The Void is
Unity Technologies (fig. 6). 28 In this spatial installation, an advocate of the redirected walking principle, too. Its
the display user touches the corridor wall with one hand Executive Illusionist Curtis Hickmann developed a similar
in order to enhance the virtual environment’s credibility. endless corridor for the VR game hall.29 At the very same
time, the system can be used by several users simultane-
26 Frank Steinicke, Being Really Virtual. Immersive Natives and the Future of
ously, movement sensors and images actively guide users
Virtual Reality, Cham: Springer, 2016, pp. 59–86.
27 Ibid., p. 77. not to bump and crash into each other, with virtual doors
28 Keigo Matsumoto, Yuki Ban, Takuji Narumi et al., Unlimited Corridor. serving as barriers.
Redirected Walking Techniques Using Visuo-Haptic Interaction, in:
Proceedings of ACM SIGGRAPH 2016 Emerging Technologies, Article
No. 20 (Anaheim, CA, July 24–28, 2016), https://dl.acm.org/citation.cfm?-
doid=2929464.2929482 (accessed January 1, 2018); see also the video of VR 29 See the model of The Void’s playing field  https://theverge.com/2016/
experiments: Keigo Matsumoto, Unlimited Corridor, https://youtube.com/ 7/1/12058614/vr-theme-parks-disney-six-flags-the-void-ghostbusters-­
watch?v=THk92rev1VA (accessed January 1, 2018). virtual-reality (accessed January 1, 2018). 169
 Carolin Höfler

respect, the space of the ball is rather a gigantic joystick

managed, operated, and steered with the users’ feet. Three
of these balls are located in a Las Vegas casino, one ball can
be found at the University of Bremen in the department for
cognitive neuroinformatics, for the purpose of research on
human orientation in virtual and physical spaces. 30 This
experimental system also supports the idea that the VR user
has the best orientation in unknown virtual worlds if he
is provided with as many sensory impressions as possible.
Besides the feeling of moving and watching 3D pictures of
the virtual world, the setting can be complemented with
sounds and odors. Such a maneuverable ball is character-
ized by an ideology insinuating that space perception and
space movement follow sensory impressions. That idea has
a strong impact on the development of today’s phygital VR
settings in the realm of game and interaction design. Yet,
it is quite astonishing though that the material and atmo-
spheric characteristics of real space receive so little atten-
tion in those settings.

Hallucinatory Interface
It even seems that the contradiction between physical spa-
7  VirtuSphere, Mounted Warfare TestBed at Fort Knox, Kentucky, 2007.
tiality and virtual imagery will sharpen: The visual-mate-
rial, tactile-haptic, and olfactory features of space are being
A multidirectional version of the Unlimited Corridor is decoupled from their physical shape and materiality and
the so-called VirtuSphere originally used by the US military transferred into visual information. The physical diffuses
for training purposes (fig. 7). It is a ball to walk-in with a into the virtual for the benefit of a performance that per-
diameter of 3 meters supported on rollers. After entry, users manently reconfigures the potential relationship between
can walk in any direction without changing their position physical and virtual reality.
in the physical space. With the help of a head-mounted dis-
play, test persons are transferred to virtual worlds in which 30 Kerstin Schill, Räumliche Exploration (VirtuSphere), University of
­Bremen, working group Cognitive Neuroinformatics, http://cognitive-­
they can move about freely. Sensors beneath the ball record neuroinformatics.com/en/research/projects/raeumliche-exploration
170 any step and transmit the information to the display. In that (accessed January 1, 2018).
Sense of Being Here

8  Keiichi Matsuda, Hyper-Reality, 2016. AR rendering.

Number and complexity of this projected imagery will niques and procedures in computer graphics and computer
in the near future certainly increase. Given the ubiquity vision, it is possible to generate, from a variety of images,
of cameras and the availability of a billion photos online, photo­realistic and editable digital models of those scenes,
recent years have witnessed new options for image-based which can then later be integrated into virtual reality sur-
3D data capture and 3D reconstruction of physical spac- roundings. In the ideal case, the display user can record
es and objects. 31 With the help of photogrammetric tech- his physical surrounding space via a digital video camera
attached to his glasses, can process that information into
31 Simon Fuhrmann, Fabian Langguth, Michael Goesele, MVE – A Multi-View 3D spaces and 3D objects through image-based geometry
Reconstruction Environment, in: Proceedings of the Eurographics Work- ­reconstruction software, and can eventually embed them
shop on Graphics and Cultural Heritage (Darmstadt, October 6–8, 2014),
https://gcc.tu-darmstadt.de/media/gcc/papers/Fuhrmann-2014-MVE.
into the VR scene – and all this in real time.
pdf (accessed January 1, 2018). 171
 Carolin Höfler

The possible consequences of this continuous transfor- they deciphered the aesthetic functionalism of a commer-
mation of physical spaces into virtual imagery is outlined cial entertainment industry whose symbolism and imagery
by Japanese architect Keiichi Matsuda in his short movie were particularly oriented towards the visual perception of
Hyper-Reality (fig. 8). 32 Just like in a computer game, the motorists and pedestrians. 33 From the buildings along the
viewer is watching the action from first-person perspective Las Vegas Strip, they derived the building type of the dec-
of a protagonist moving through a consumer’s day-to-day orated shed which is simple in configuration and form, but
world, totally saturated by media and promotions. He takes whose message is intricately designed and offensively put to
the bus, gambles online, chats with a job manager about the front of the façade – as ultra-large illuminated panels
current vacancies, and enters the question “Who am I?” and signs. Hyper-Reality then appears to be an even more
into a search engine. In the supermarket, there are new ads exaggerated version of the Strip. The digital symbols cover-
and apps perpetually popping up, and a small dog on the ing the city like a virtual skin are individually customized to
shopping cart serves as a shopping companion, reporting those who pass through, can be dynamically modified, and
special offers via GIF animation. While in this visual super- are, in the truest sense of the term, transparent and easily
saturation, innumerable background sounds rumble, roar, comprehensible.
and ring. In Hyper-Reality, Matsuda coped with a specific Hyper-Reality emphasizes the leveling of deep struc-
kind of the interplay of virtual and physical realities – with tures as a constitutive feature of physical-digital worlds,
augmented reality (AR). In contrast to VR displays, visual even though the pioneers of VR and AR systems originally
access to the physical space is still possible, but it is mod- came forward with a contrary promise. Given the revocation
ified through interactive 3D projections. While doing so, of deep structures, one is tended to consider the Hyper-Real-
the user is looking through transparent screens on which ity environments as an epitome of the late capitalist culture
he is watching the projections. They constitute the front. of spectacles and simulacra, as Fredric Jameson in his 1984
Behind it, a real-physical space opens up. Thus, this proce- book Postmodernism, or The Cultural Logic of Late Capital-
dure interlinks a flat with a deep space. ism had stressed. 34 With regard to a postmodern space that
The technique of combining detailed virtual image appeals to all senses, Jameson coined the term of “hyper-
information up front, with less detailed physical space con- space”. 35 He understood it as a spatiality characterized by a
figurations in the back, is closely associated with postmod- plenty of indissoluble and interwoven surfaces. He exempli-
ern strategies of space-formation. The Hyper-Reality archi- fied this idea of “depthlessness” in particular via the lobby of
tectures, for instance, can be regarded as contemporary the Westin Bonaventura hotel in Los Angeles: “I am tempted
versions of the decorated shed which Robert Venturi and
Denise Scott Brown considered to be a postmodern build-
33 Robert Venturi, Denise Scott Brown, Steven Izenour, Learning from Las
ing type par excellence. In Learning from Las Vegas (1972), Vegas. The Forgotten Symbolism of Architectural Form, revised edition of the
1972 publication, Cambridge, MA/London: The MIT Press, 1977, pp. 87–103.
34 Fredric Jameson, Postmodernism. Or, The Cultural Logic of Late Capitalism,
32 Keiichi Matsuda, Hyper-Reality. A New Vision of the Future, https://vimeo. Durham: Duke University Press, 1991.
172 com/166807261 (accessed January 1, 2018). 35 Ibid., p. 44.
 Sense of Being Here

to say that such space makes it impossible for us to use the modernism.40 In the age of VR and AR and the Internet of
language of volume or volumes any longer, since these are things, Hyper-Reality can be understood as an amplified
impossible to seize.” 36 He stressed that such a space was sequel and fearsome intensification of those experiences
filled with diaphanous materials, illuminating phenome- and emotions.
na, and ongoing, continuous movements to such an extent
that its material-constructional spatial boundary is forced
Other Images, Other Spaces
back to vagueness: “Hanging streamers indeed suffuse this
empty space in such a way as to distract systematically and With the growing penetration of new VR and AR glasses
deliberately from whatever form it might be supposed to onto the mass market, relations between bodies and outer
have […]”. 37 It is as if the effectual spatial boundary seems and inner environments become increasingly intercon-
to dissolve, dilute, and overlay all spatial regions. As if it nected via screen-based, interactive moving images. The
merges into diversified layers, into limiting structures being interactions with and through screen images lead to a spe-
at once open and enclosed floating through space as aerial cific alignment of the body in both the virtual and physical
objects. From these observations, Jameson concluded the space. Being a player in physical-digital theme parks like
existence of a modified spatiality – one that substitutes The Void or supposedly flying via the full-body installation
“depth” with “surface” and allows for an intensified form of of Birdly are aimed at generating an intense perception and
what in German would be called Sehenlassen or letting itself a powerful body mobilization. The visualizations of objects
be seen. 38 The numerous activities, moods, and ambiances and spaces installed through the glasses actively affect the
in the hotel lobby make the observer feel as if he had totally user’s position and movement in the physical space and also
immersed into the space: “[…] a constant busyness gives the have an impact on the user’s spatial disposition towards the
feeling that emptiness is here absolutely packed, that it is an display, the apparatus, and the architecture. If the physical
element within which you yourself are immersed, without space with its haptic surfaces and things to be perceived and
any of that distance that formerly enabled the perception of identified is digitally reproduced and visually brought into
perspective or volume. You are in this hyperspace up to your the displays, the physical space will transform into phygital
eyes and your body.” 39 The feelings of disorientation and surroundings, inducing the beholder to go through specific
dizziness which Jameson identified to be the very effects of experiences and actions. It is thus not only important how
the hyperspace are, by implication, fundamental experien­ the phygital 3D space is visually presented and perceived;
ces also detected by various observers – from Rem Koolhaas critical is how the given data of both environment and body
to Jean-François Lyotard – when faced with the forces of interact and communicate – also with regard to an inter-
capitalism unleashed and the aesthetic experiences of post- action pattern between display imagery and the beholder’s
perceptions and actions.
36 Ibid., p. 43.
37 Ibid.
38 Ibid., p. 49, p. 12. 40 Jean-Francois Lyotard, Leçons sur l’Analytique du sublime, Paris: Galilée,
39 Ibid., p. 43. 1991. 173
 Carolin Höfler

Considering the increasing integration of VR and AR 2006.41 This case mingled 3D image space analysis with
systems into economic and design-related processes and criminology, political enlightenment, and legal proceedings.
given the growing impact of display images on a comput- Knowing that virtual reality tools provoke perception shifts
erized world access, many questions arise, in particular and scaling effects, it is quite astonishing that the actors
with regard to the issues of authorship, final authority, and involved have such a high confidence in imaging techniques
decision-making autonomy of the actors involved: What being applied in those cases for the purpose of fact- and
industries and sectors will further develop and advance truth-finding.
the potentials of these physical-virtual spaces of experience Criticism and skepticism regarding how the recipient
and action, and for what purposes? How will the observer can be captured, deceived, and manipulated in VR environ-
be influenced? Who designs and is in charge of the content ments is more effective through designing critical spaces.
and type of physical-virtual experiences? Which interactive This raises the question of how alternative forms to physi-
screen images will be provided? What emotional and mental cal immersion into 3D spaces can be devised and developed.
experiences shall be triggered? Even though the action tak- Is it possible to maybe reduce the intensity of virtual 3D
ing place in the huge spectacle of games and artistic instal- surroundings? In coping with VR, which medial strate-
lations has, in the first place, no far-reaching effect for the gies exist that circumvent the almost total immersion into
real-physical world, the perceptions and experiences made virtual spaces and which provide transparency about the
in those physical-virtual settings do create an intensity of techniques that users are succumbing to? How can more
experience which must be perceived to be real. As a result, heterogeneous, fragmented, and conflicting formal systems
virtual reality experiences will certainly be establishing be applied in immersive environments?
themselves as a new category of spatial experience and will Those strategies of undermining and disturbance are
thus give a strong impetus to sensory perceptions, actions, based on a conceptional approach which tries to overcome
and decisions in non-digital spheres of life. the traditional antagonisms between body and mind and
Further, the scientific utilization of virtual reality tools which interprets sensory perceptions neither individual-
will have far-reaching implications for knowledge produc- istic-hierarchically nor collectivist-holistically. From this
tion and evidence acquisition, for instance in investigative perspective, seeing, hearing, and feeling are not being
analysis and 3D crime scene reconstruction. The fact that understood as naturally given skills, but as specific effects
architecture and the arts also harness forensic methods of socio-technical assemblages and as medial design prac-
based on virtual reality is illustrated by the interdisciplin- tices.42 This involves not only experiences of visual-spatial
ary research group Forensic Architecture and its project relations, but experiences of social relations as well. How
77sqm_9:26min, which through imaging techniques tried to can display users get in touch and interact with each other?
resolve the case around German intelligence officer Andreas
Temme with regard to the NSU-murder of Halit Yozgat in 41 Forensic Architecture, 77sqm_9:26min, investigation, 2016–2017, http://
forensic-architecture.org/case/77sqm_926min (accessed January 1, 2018).
42 Beate Ochsner, Robert Stock (eds.), senseAbility. Mediale Praktiken des
174 ­Sehens und Hörens, Bielefeld: transcript, 2016.
Sense of Being Here

And how can they get in touch and interact with those not Figures
equipped with a display? 1 Toast, Plank Experience, Frankfurter Kunstverein, Photo: N.
In such discontinuous settings, what is and will be the Miguletz. https://fkv.de/sites/default/files/styles/680x540/public/
FKV_Perception%20is%20Reality_ Ausstellungsansicht_Toast_2.
function of architecture? In hyper-realistic phygital VR jpg?itok=YHOUdOBg (accessed January 1, 2018). Reprinted with permission.
environments, the materially bound spatial forms primarily 2 Toast, Plank Experience, https://toast.gg/release/ (accessed January 1,
serve as image carriers and step behind the digital inter- 2018). Reprinted with permission.

face. The question arises as to how to envision, design, and 3  Somniacs, http://somniacs.co/media.php (accessed January 1, 2018).
Reprinted with permission.
develop an architecture that does not constantly validate
4  Adam Savage, Tested, Flying the Birdly Virtual Reality Simulator, https://
the perception of being in a virtual space (sense of being
youtube.com/watch?v=gWLHIusLWOc (accessed January 1, 2018).
there), but rather challenges it through creating a sense of Reprinted with permission.
presence in the physical space (sense of being here). What 5  THE VOID, First look at THE VOID, https://youtube.com/watch?v=
other perceptions could be triggered by physical spatial cML814JD09g, 0:58, 0:59.

forms if only they provided meanings and messages beyond 6  Unlimited Corridor Project Team (Keigo Matsumoto, Representative),
Japan, http://cyber.t.u-tokyo.ac.jp/~matsumoto/image/uc/uc_web.jpg
the moving images of the displays – to be decoded and inter- (accessed January 1, 2018). Reprinted with permission.
preted by the recipient in relation to the visually perceived 7  Paul Monday, https://en.wikipedia.org/wiki/VirtuSphere#/media/
3D ­spaces? What if these perceptions, meanings, messages, File:Virtusphere.jpg (accessed January 1, 2018).
codes, i­ mages did not rely on affirmation and affective adap- 8  Keiichi Matsuda, Still from Hyper-Reality, 2016, http://hyper-reality.
tion, but on disturbance and doubt instead? co/assets/HQ_images/hyper-reality_03.jpg (accessed January 1, 2018).
Reprinted with permission.

175
Timo Kaerlein and Christian Köhler

Around a Table, around the World

Facebook Spaces, Hybrid Image Space and Virtual Surrealism

On 6 October 2016, at the developer conference Oculus Con- including their palette of facial expressions and hand ges-
nect 3 in San José, California, Facebook CEO Mark Zuck- tures, a series of immersive photo and video environments –
erberg presented the first working prototype of Facebook a deep-sea scenario, the surface of the planet Mars, and
Spaces, the social media company’s ambitious foray into the Zuckerberg’s own office –, and interactive affordances like
emerging virtual reality (VR) sector.1 Unsurprisingly, Face- playing games at a virtual table and creating 3D objects in
book’s vision of VR is a social one, i. e. an attempt is made at space, the audience experienced a dizzying moment of ref-
translating the company’s core business model of capitaliz- erential disorientation. Zuckerberg transported the group of
ing social relations into a VR setting. three into a different setting again, this time (supposedly) a
During the product demonstration on stage, which live video-feed of his actual living room. Then he suddenly
included Lucy Bradshaw and Michael Booth, two senior received a video call from his wife via the Facebook Messen-
Facebook employees working in the VR team, Zuckerberg ger application on his virtual wristwatch: While her moving
discussed all major features of Facebook’s VR proposal that image appeared in VR on an oversized virtual phone display
is available in an open beta version at the time of this writ- for all participants to see, she in turn saw on the display of
ing. Only Zuckerberg himself was physically present wear- her phone her husband’s VR avatar standing in their living
ing a head-mounted display, while Bradshaw and Booth room with two other comic characters, the unsuspecting
were somewhere off-stage using a similar set-up to meet family dog Beast reclining on the couch in the background.
with Zuckerberg within Facebook Spaces, Facebook’s social The climax of the product demonstration induces at the
VR application. While the audience could only see Zucker- same time a media theoretical reeling: Zuckerberg turned
berg on stage talking into the air, they could turn to giant around for a “modern family selfie” 2, using a virtual self-
screens on which Zuckerberg’s perspective of the virtual ie stick handed to him by Michael Booth’s avatar, aligning
space was projected: It displayed what the headset let him the gigantic phone display with his wife’s image next to his
see, Bradshaw and Booth virtually present via their ava- virtual avatar and the moving image of the dog, in the same
tars. After having showcased the comic-figure-like avatars, act suturing diverse layers of referentiality into a series of
photo-like static representations that appeared on the virtu-
Road to VR, Facebook Social VR Demo – Oculus Connect 2016, https://youtube.
1
com/watch?v=YuIgyKLPt3s (accessed November 3, 2017). Ibid.
2
 Timo Kaerlein and Christian Köhler

promises to change this situation. 3 What people will even-

tually do in a social VR scenario, how they will interact
with each other and with diverse media content, is first and
foremost an as yet unsettled question of interface design.
This is evident to the designers of Facebook Spaces,4 but we
will address this question not from a design perspective, but
from a media-theoretical point of view that follows ­Johanna
Drucker’s definition of “interface” understood not as an
object, but as “a set of conditions, structured relations, that
allow certain behaviors, actions, readings, events to occur”. 5
This allows for an analysis that considers the interface mise-
en-scène of Facebook Spaces as constituting a proper media
dispositif or apparatus.6 This apparatus assigns subject posi-
tions, orders relations between participants and – crucially –
sets the stage for a radically hybrid image space, in which
1  Mark Zuckerberg taking a “modern family selfie” in Facebook Spaces.
See Thilo Hagendorff, Virtual-Reality-Datenbrillen im Spannungsfeld
3
zwischen Empathie- und Isolationsmaschinen, in: Institut für immersive
Medien (ed.), Jahrbuch immersive Medien 2016, Marburg: Schüren Verlag,
al table in front of him (fig. 1). Afterwards, the couple chose 2017, pp. 71–79.
their favorite picture that was then effortlessly delivered to Christophe Tauziet, leading designer in Facebook’s social VR team, makes
4
this point explicit in an extensive and insightful article on medium.com:
Zuckerberg’s Facebook feed via the push of a virtual button “One of the biggest challenges for our design team was to design the
on his other wrist. In Facebook’s vision of social VR, the old user interface of Spaces. Unlike with traditional web, desktop or mobile
design where we can rely on existing UI elements and interaction pat-
dichotomies of actual and virtual, real and imagined, per-
terns that people have learned over the years, most of those patterns
ception and action that structured major debates in the VR have yet to be invented for VR.” Christophe Tauziet, Designing Facebook
discourse of the 1990s seem to have irretrievably collapsed. Spaces, https://medium.com/@christauziet/designing-facebook-spac-
es-part-4-creating-a-vr-interface-821861159495 (accessed November 3,
The above-described scene taking place in Facebook 2017). We will refer to his development report repeatedly throughout our
Spaces raises a series of questions concerning the character contribution.
Johanna Drucker, Performative Materiality and Theoretical Approaches to
5
and shape of visual practices that are intended to constitute
Interface, in: digital humanities quarterly 7.1 (2013), http://­digitalhumanities.
sociality in VR. Whereas screen-based practices in VR are org/dhq/vol/7/1/000143/000143.html (accessed November 3, 2017).
often associated with anti-social behavior, Facebook S ­ paces, See Jan Distelmeyer, Machtzeichen. Anordnungen des Computers, Berlin:
6
Bertz + Fischer Verlag, 2017, pp. 81–82. Distelmeyer introduces the concept
advertised with the slogan “VR is better with friends”, of interface mise-en-scène in explicit reference to film studies to signify the
need to pay closer attention to the heterogeneous aesthetic arrangements
organizing the use of computers. Analyzing the complex staging of inter-
face processes can offer different approaches to digital cultures than just
178 assuming the computer to be a functional tool.
Around a Table, around the World

the line between physical and virtual entities increasingly We will explore the strategies employed by the designers of
loses significance. Applying the designers’ own language, Facebook Spaces to achieve this end, primarily the virtual
this image space can best be characterized as giving rise table that acts as the central interface element and center
to a virtual surrealism in which long contested dichotomies of control. One major finding of our analysis amounts to the
concerning the status of images, perception and action dis- observation that the apparatus of Facebook Spaces, contrary
solve into a scenario of reality-agnosticism that is equally to popular rhetorics of presence and immersion associated
frightening and exhilarating.7 Our main interest concerns with VR, creates a strong impression of unreality by decon-
the ways that the ensemble of interface techniques and pro- textualizing images and severing referential links. Finally,
cesses of Facebook Spaces attempts – and ultimately fails – we discuss over-arching theoretical and normative concerns
to keep together this heterogeneous action space, especially raised by our analysis: Facebook Spaces is understood as a
by rather surprisingly falling back to an almost-forgotten step towards an emerging virtual surrealism – a scenario in
“old medium”, namely the virtual table around which the which the affordances of digital media are taken very seri-
users gather.8 ously to the extent that the referential status of images and
The essay is divided into three parts. First, we give an actions in VR altogether ceases to be a relevant parameter
introductory account of Facebook Spaces. The focus of our for design and use. This also has implications for the subject
description lies in understanding the brandscape of Face- positions assigned by the apparatus: In a scenario of mixed
book Spaces as a dispositif as described in the works of unreality, a tendency towards moral indifference can be
Jean-Louis Baudry, i. e. as a spatial arrangement that reg- observed and criticized.
ulates the behavior of participants and favors specific psy-
chic dispositions.9 The second part of our contribution then
Setting the Table – The Dispositif of Facebook
delves deeper into the theoretical ramifications of this set-
Spaces
up: The hybrid image space constituted by Facebook Spaces
is reminiscent of a heterotopia in that it involves a confronta- Due to the commercial availability of affordable VR hard-
tion of widely disparate image spaces that have to be sutured ware for the consumer electronics market, several compa-
together to constitute a reliable and secure action space. nies have developed applications and platforms for social
interaction in VR.10 The discussions around these offerings
See Gabriel Valdivia, Identity Transfer and the Rise of Virtual Surrealism,
7 are reminiscent of the ones that accompanied the early
https://artplusmarketing.com/identity-transfer-and-the-rise-of-virtu-
text-based MUDs (multi-user dungeons/dimensions) and
al-surrealism-bac751e6342c (accessed November 3, 2017).
See Walter Seitter, Möbel als Medien. Prothesen, Paßformen, Menschen-
8 MOOs (MUD, object-oriented) common in the 1980s and
bildner. Zur theoretischen Relevanz Alter Medien, in: Annette Keck, Nico-
las Pethes (ed.), Mediale Anatomien. Menschenbilder als Medienprojektionen,
Bielefeld: transcript, 2001, pp. 177–192, pp. 184–187. 10 For an overview discussing different services like Bigscreen, vTime,
See Jean-Louis Baudry, The Apparatus. Metapsychological Approaches
9 AltspaceVR and Rec Room, see Adario Strange, Social Networking in VR
to the Impression of Reality in Cinema, in: Philip Rosen (ed.), Narrative, is Here, and it Feels Like the Future, http://mashable.com/2017/01/12/
Apparatus, Ideology. A Film Theory Reader, New York: Columbia University virtual-reality-social-networks-vr/#iaf1.9tSSOqq (accessed November 3,
Press, 1986, pp. 299–318. 2017). 179
 Timo Kaerlein and Christian Köhler

early 1990s, which was also the period when the cultural an abstract geometrical shape like a triangle proposed by
imaginary concerning virtual reality peaked in the all-en- Jaron Lanier –, the premise and imperative of Facebook
compassing term cyberspace.11 Similar rhetorics concern- Spaces is simply: “Be Yourself in VR!” 14 Rachel Franklin,
ing the general idea of online communities are applied in head of Social VR at Facebook and former general manager
the Spaces context, including an understanding of virtual for the Sims series at Electronic Arts, further qualifies this
tribes defined “not by proximity but personal choice”.12 statement: “It’s easy to create an identity that represents the
These notions are accompanied by a set of body and identi- real you in Facebook Spaces. This helps people recognize
ty politics that were already common in the VR discourse you and makes VR feel more like hanging out in person. […]
of the 1990s, like the idea of experimenting with different You can change your eye color, hairstyle, facial features and
types of embodiment, a fragmentation of the sense of self more until your look fits your identity. It’s all about being
experienced in the relationship to one’s virtual avatar, and a yourself.” 15 On the one hand, this idea differs greatly from
prevalent logic of mentally being somewhere else while the the (supposedly) wildly experimental character of virtual
body is left behind in the physical world. identities in the 1990s, while on the other, it hints at a notion
But there are also marked differences in how social VR of idealization and purification of the self that is tightly con-
is imagined in the present. In fact, one could go as far as nected to the necessities of social media self-curation.16 It is
to claim that VR companies apply metaphors and mental noteworthy in this context that avatars in Facebook Spaces
images concerning the affordances of the new medium par- can neither look unhappy nor have a body that deviates too
tially dressed in the language of the 1990s, while something far from the norm set by its designers.17
entirely different is happening.13 We will focus our analysis Topologically speaking, Facebook Spaces can further
on Facebook Spaces because the multinational social media be understood as a curious kind of virtual brandscaping:
enterprise already has access to a base of two million month- Whereas Lev Manovich discusses examples of companies
ly active users, a fact that makes it especially well-positioned giving their brand a material shape via architecture – e. g.
in the emerging social VR market. In contrast to older VR the design of OMA/Rem Koolhaas’ Prada store in New York
discourses that stressed possibilities of identity play and
experimentation with different body types and shapes –
14 Facebook, https://facebook.com/spaces (accessed November 3, 2017).
including the wish to experience a virtual embodiment as See Jaron Lanier, Technology, http://globetrotter.berkeley.edu/people5/­
Lanier/lanier-con2.html (accessed November 3, 2017).
15 Rachel Franklin, Facebook Spaces: A New Way to Connect with Friends in
11 See Sherry Turkle, Life on the Screen. Identity in the Age of the Internet, New VR, https://newsroom.fb.com/news/2017/04/facebook-spaces/ (accessed
York: Simon & Schuster, pp. 9–19, pp. 180–186. November 3, 2017).
12 Yaser Sheikh from the Oculus research team, as cited in Matt ­Weinberger, 16 See Mark Zuckerberg: “You have one identity. […] Having two identities
Facebook’s Vision of the Year 2026 is Scary and Awesome, http://­ for yourself is an example of a lack of integrity.” As cited in Karl Wolfgang
businessinsider.de/facebooks-world-of-virtual-reality-in-2026-2016- Flender, #nofilter? Self-Narration, Identity Construction and Meta Story­
4?r=US&IR=T (accessed November 3, 2017). telling in Snapchat, in: Florian Hadler, Joachim Haupt (ed.), Interface Cri-
13 Concerning the role of such legitimizing ideologies in the history of the tique, Berlin: Kulturverlag Kadmos, 2016, pp. 163–182, p. 172.
internet, see Patrice Flichy, The Internet Imaginaire, Cambridge, MA: MIT 17 See Kyle Riesenbeck, Facebook Won’t Let Me Be Fat in VR, http://­revvrstudios.
180 Press, 2007, pp. 10–12. com/facebook-fat-in-vr (accessed November 3, 2017).
Around a Table, around the World

that makes heavy use of electronic displays to create an “aug- Baudry’s account, the apparatus of cinema served first and
mented space” –, the challenge for Facebook lies in trans- foremost to create an “impression of reality […] dependent on
lating their product into a VR environment that serves as a subject effect”, i. e. the apparatus tends to make itself invis-
the condition and backdrop of user interactions.18 Where the ible in order to constitute a simulation of the real.20 This only
brandscaping described by Manovich worked by integrating works because the cinematographic subject enters a “state of
screens into physical architecture, within the apparatus of artificial regression” which leads to “a lack of differentiation
VR screens create an image space which must be provided between the subject and its environment” and thus a “par-
with virtual architecture. Although this environment is a tial elimination of the reality test”.21 Without reconstructing
thoroughly virtual one, the actions that are possible in it the intricacies of apparatus theory at this point, it suffices to
are fundamentally physical, i. e. they comprise a set of ges- say that the main thrust of the argument is to claim that the
tures and interface operations involving the whole body. In technical apparatus of cinema produces ideological effects
the following, we will first give an introductory description independently of what is projected.
of the environment of Facebook Spaces by understanding As others have shown, apparatus theory can deliver an
it as a media dispositif or apparatus in the sense of Baudry. adaptable conceptual framework to describe the ideological
This serves to prepare a more detailed analysis of the hybrid effects not of media content but of media themselves.22 But,
image space constituted by Facebook Spaces in the following of course, cinema and VR are two fundamentally different
chapter. Baudry distinguished the media. Not only is the production process of VR applica-
tions, at best, only partly comparable to the production of
basic cinematographic apparatus [l’appareil de base], movies, but the act and context of reception differs widely
which concerns the ensemble of the equipment and from cinema. The apparatus of cinema consisted of viewers
operations necessary to the production of a film and its who were physically restricted in a darkened room to watch
projection, from the apparatus [le dispositif ] […], which unreachable images projected from behind their back.23 In
solely concerns projection and which includes the sub- contrast, users of VR look onto light-emitting screens direct-
ject to whom the projection is addressed.19 ly in front of their eyes locking out non-screen reality to see
images which they can interact with. In fact, the positioning
It is the latter apparatus in the sense of a spatial arrange- of screens is one of the core differences between the appa-
ment of objects and bodies that interests us most; although ratus of cinema and the apparatus of VR/Facebook Spaces:
it is impossible to separate this completely from the data Whereas spectators in the cinema are always in principal
infrastructures and economies constituting the position
of Facebook in the contemporary social media business. In
20 Ibid., p. 312.
21 Ibid., p. 313.
18 Lev Manovich, The Poetics of Augmented Space, in: visual communica- 22 See Knut Hickethier, Dispositiv Fernsehen. Skizze eines Modells, in: mon-
tion 5/2 (2006), pp. 219–240, pp. 234–235. The term brandscaping is here tage a/v 4/1 (1995), pp. 63–83.
attributed to Otto Riewoldt. 23 See Jean-Louis Baudry, Ideological Effects of the Basic Cinematographic
19 Baudry 1986 (as fn. 8), p. 317. Apparatus, in: Film Quarterly 28.2 (1974), pp. 39–47, p. 44f. 181
 Timo Kaerlein and Christian Köhler

2  Apparatus_1: Users with VR headsets. 3  Apparatus_2: The dispositif of Facebook Spaces.

able to see the edges of the screen, screens in VR occupy the dances and limitations that assigns subject positions and
user’s whole field of vision which even makes it necessary to regulates participants’ social behavior. We are interested in
simulate screens virtually to allow for specific operations. the set-up of this second apparatus and will further specify
Therefore, one defining trait of VR one must consider if its its various components as the interface mise-en-scène of
ideological effects are to be analyzed is that, unlike cinema, it Facebook Spaces (fig. 3).
not only consists of the technological apparatus and content In the most recent public beta version, the environ-
of the images but also of another mediating virtual layer. ment of Facebook Spaces is imagined not so much as the
In the case of Facebook Spaces, one would thus have to “infinite wonderland” of cyberspace so prevalent in the
assume a double apparatus in the sense that the user first cyberpunk imaginary of the 1980s and 90s but rather more
has to set up a space for the hardware, don a VR headset, closely resembles the familiar surroundings of a “dinner
and get proficient with a set of physical controllers. These party” with family and friends.24 That is, if one is willing
taken together comprise a material interface arrangement to ignore the highly technical composition of this virtual
or apparatus_1 that has become an iconic visual reference
to VR technology in press reporting and advertising alike 24 The juxtaposition of wonderland and dinner party is taken from Rachel
(fig. 2). The hardware in turn allows entry into a virtual Rubin Franklin. See Peter Rubin, Facebook’s Bizarre VR App Is Exactly
Why Zuck Bought Oculus, https://wired.com/2017/04/facebook-­spaces-
action space that will be addressed as apparatus_2 in the vr-for-your-friends/ (accessed November 3, 2017). Both, of course, can be
182 following: It constitutes a visual setting with specific affor- traced back to Carroll’s literary blueprint.
Around a Table, around the World

4  Birthday party at the virtual table of Facebook Spaces. 5  VR Dock 2.0, main navigation tool of Facebook Spaces. 6  VR Watch 2.0, notification center in Facebook Spaces.

dinner party: The participants’ cartoon-like avatars – that or – in the case of 360° video content – onto the (imagined)
can be modelled after the users’ likings using a photograph walls of the virtual enclosure to constitute a shared envi-
in the initial setup phase – gather around a virtual table that ronment. Sociality in Facebook Spaces is fundamentally and
acts as the center for a diverse range of practices (fig. 4). 25 purely a screen-based practice, with the added twist that
Each avatar is assigned a fixed position at the table of which screens only ever appear as virtual constructs inside the
four are available in total. Directly in front of each avatar user’s perceivable action space. The scenario could also be
is a small projector – dubbed VR Dock 2.0 by the develop- described as an echo chamber of projections cohabited by
ers – that acts as an individual access point to the users’ up to four subjects that share traces of their memories in a
personal Facebook content like images and videos, as well consensually constructed dream world (fig. 4–6).
as third party content and a palette of interface tools like While a lot of the elements of the interface mise-en-
a pencil, a selfie-stick with a camera and a mirror (fig. 5). scène of Facebook Spaces are quite innovative, the design-
Apart from the dock, each user has at his or her disposal a ers of Facebook Spaces draw on several interrelated HCI
virtual watch fitted to the avatar’s left wrist that delivers (human-computer interaction) conventions and estab-
notifications, e. g. about incoming calls via the Messenger lished action patterns to define the operational modalities
app, and a user interface placed on the right wrist which of the virtual environment. First among these is the idea
gives contextual options equivalent to a right mouse-click of “direct manipulation” that has been popularized by the
in a desktop graphical user interface (fig. 6). The middle of desktop metaphor of graphical user interfaces (GUIs) since
the table houses the so-called Display Center: Media content the 1960s: Users can handle graphical representations of
can be placed in this circular area via hand gesture and is data like personal files and folders with intuitive gestures
then either projected onto a large display visible to all par- ­w ithout the need to attain expert programming skills. 26
ticipants in the case of traditional photographs or videos,

26 See Florian Hadler, Daniel Irrgang, Instant Sensemaking, Immersion and

25 The following description of the various interface elements of Facebook Spaces Invisibility. Notes on the Genealogy of Interface Paradigms, in: Punctum
is mainly adopted from Tauziet’s informative article mentioned above. 1.1 (2015), pp. 7–25. 183
 Timo Kaerlein and Christian Köhler

Many action routines situated in this paradigm have been Other user-created objects like sketches, drawings and pho-
further simplified and extended with the popularization tographs made with the selfie-stick, or drawn from users’
of touchscreen interfaces, especially in mobile devices like accounts constantly and increasingly litter the shared space
smartphones.27 Thus, many of the surfaces inside the appara- of the virtual table or float freely around the avatars. All in
tus_2 of Facebook Spaces are “touch-sensitive” and react to all, this quickly leads to a dizzying array of visual elements
gestural inputs.28 The designers also resort to general ideas that can get overwhelming and messy, which is document-
from the tangible interaction paradigm first introduced by ed by user experience videos uploaded to YouTube. 31 In the
Hiroshi Ishii and Brygg Ullmer from the MIT Media Lab: next section, we will generalize from these usability issues
Whereas the original vision of “tangible bits” aimed at aug- and understand them as indicating a representational crisis
menting physical objects to bridge “the gap between the of the hybrid image space constituted by Facebook Spaces.
worlds of bits and atoms”, in Facebook Spaces users act in a The virtual table, employed as an element of the interface
completely virtual environment inside which abstract data mise-en-scène to constitute a “space of affordances and pos-
processes are translated into physical activities with a spa- sibilities structured into organization for use”, inadvertently
tial dimension.29 For example, when using an in-built feature produces this crisis in the first place. 32
to live-broadcast from Facebook Spaces, a stream of friend’s
comments is visualized in the virtual environment and users
Plights of the Round Table – How to Control a
can pull single comments out of this stream and interact
Hybrid Image Space
with them spatially as if they were large sheets of paper. 30
While Facebook has been discussed as a heterotopia before,
27 See Timo Kaerlein, Aporias of the Touchscreen. On the Promises and Perils we propose that this holds true even more for Facebook
of a Ubiquitous Computing, in: NECSUS. European Journal of Media Studies
1/2 (2012), https://necsus-ejms.org/aporias-of-the-touchscreen-on-the-
Spaces. 33 Its heterotopic character can be described on sev-
promises-and-perils-of-a-ubiquitous-technology/ (accessed November 3, eral levels: We would like to argue that VR in general consti-
2017).
tutes a heterotopia on the level of apparatus_1, whereas the
28 The question of whether interactions inside a virtual environment can and
should still be addressed as screen operations or whether it makes more social VR scenario on the level of apparatus_2 intensifies this
sense to treat them as a new category in HCI has been debated as early hetero­topic character by drawing together and juxtaposing
as 1991 in Meredith Bricken, Virtual Worlds. No Interface to Design, in:
Michael Benedikt (ed.), Cyberspace. First Steps, Cambridge, MA: MIT diverse types of images and screens. The hybrid image space
Press, 1992, pp. 363–382. Bricken wholeheartedly affirms a paradigm shift thus constructed is the subject of interface design efforts to
“between traditional interface design and designing virtual worlds” that
make it cohere and counter its diverging tendencies.
is compared to the difference between watching the ocean from a boat and
diving into it with a scuba gear set. Ibid., p. 364.
29 Hiroshi Ishii, Brygg Ullmer, Tangible Bits. Towards Seamless Interfaces
Between People, Bits and Atoms, in: Proceedings of the ACM SIGCHI Con- 31 See TWit Netcast Network, Facebook Spaces VR Test Drive, https://­youtube.
ference on Human Factors in Computing Systems (CHI ’97), New York: ACM, com/watch?v=_kGRpSd4vnc (accessed November 3, 2017).
1997, pp. 234–241, p. 240. 32 Drucker 2013 (as fn. 5), p. 31.
30 See Mike Booth, Live from Facebook Spaces: A New Way to Share VR with 33 See Robin Rymarczuk, Maarten Derksen, Different Spaces. Exploring Face-
Friends, https://newsroom.fb.com/news/2017/07/live-from-facebook-­ book as Heterotopia, in: First Monday 19.6 (2014), http://firstmonday.org/
184 spaces/ (accessed November 3, 2017). ojs/index.php/fm/article/view/5006/4091 (accessed November 3, 2017).
Around a Table, around the World

First and foremost, Facebook Spaces is built on a system virtual environment mimic highly conventionalized actu-
of openings and closures that is fundamental for its opera- al practices with the help of virtual objects characterized
tions. As Foucault describes in his well-known lecture Of by certain affordances, e. g. taking a selfie with a virtual
Other Spaces, entering a heterotopia is often regulated by selfie-stick. 37 Glitches aside, the interface arrangement of
rites or acts of purification, which play an import part in Facebook Spaces creates a “regime of control” which, at
setting these places apart by isolating them from normal first glance, contradicts the rhetorics of limitless freedom
places. 34 In the case of Facebook Spaces, as with any other generally applied to VR. 38 But, as became apparent during
virtual environment, this division is implemented by the the design process, another system of opening and closing,
apparatus_1, i. e. the hardware and software necessary to not unlike the one granting access to the greater heteroto-
enter VR. In particular, the head-mounted display serves pia of VR itself, had to be established to connect users to
to exclude the user’s perception of the actual surrounding their friends via Facebook Spaces. The designers soon dis-
space while enclosing him or her in a virtual image space. 35 covered that one particularity of VR is that many problems
At the same time, the user’s body moves in a space measured of actual space repeat themselves within the virtual space
by several sensors, which capture head and hand movements they created. The reproduction of more traditional social
to map them onto the avatar’s body in virtual space, thereby settings chosen in older virtual worlds and other contem-
inducing a strong reality effect. 36 porary social VR applications (like living rooms or bars)
But once they enter Facebook Spaces, users find that did not bring their users together effectively enough to let
they are not allowed to wander around in virtual space at them engage in social interactions. On the contrary, “when
their leisure, rather they are embodied as leg-less avatars able to freely move around, people tended to get lost and
gathered around a virtual table. These avatars are fixed weren’t really interacting with each other”, according to
in a position from which every user could at any time see Facebook Spaces lead designer Christophe Tauziet. 39 There-
any of the up to three other users in their instantiation of fore, the seemingly rigidly controlled virtual action space we
Facebook Spaces and with whom they could now engage. addressed earlier as apparatus_2 was implemented to more
The possible interactions themselves are highly preformat- closely define the range of possible social connections and
ted by the interface, making it difficult to talk of actions in interactions granted by the user interface. The key design
an emphatic sense. Especially screen practices within the element of this solution is the virtual table. Media theorist
Walter Seitter acknowledged the mediality of tables early
on, describing their ability to keep things – and people,
34 See Michel Foucault, Of Other Spaces. Utopias and Heterotopias, in: Neil
Leach (Hg.), Rethinking Architecture. A Reader in Cultural Architecture, New
York: Routledge, 1997, pp. 330–336.
35 See Michael Friedmann, Kathrin Friedrich, Moritz Queisner, Christian 37 The status of the resulting pictures remains unclear: Should one consider
Stein, Conceptualizing Screen Practices. How Head-Mounted Displays them as photographic images or rather as screenshots?
Transform Action and Perception, in: Media Tropes VI.1 (2016), pp. i–v. 38 Sabine Wirth, Between Interactivity, Control, and ‘Everydayness’. Towards
36 See Hartmut Winkler, Reality Engines. Filmischer Realismus und Virtu- a Theory of User Interfaces, in: Florian Hadler, Joachim Haupt (ed.) Inter-
elle Realität, http://homepages.uni-paderborn.de/winkler/reality.html face Critique, Berlin: Kulturverlag Kadmos, 2016, pp. 17–35, p. 18.
(accessed November 3, 2017). 39 Tauziet (as fn. 4). 185
 Timo Kaerlein and Christian Köhler

space, it also brings together different spaces.41 Tellingly,

among the examples he chose to illustrate the concept of
heterotopia are the theater, which brings several places onto
the stage, and the cinema, “a very odd rectangular room,
at the end of which, on a two-dimensional screen, one sees
the projection of a three-dimensional space”.42 Foucault’s
examples seem rather tame compared to Facebook Spaces.
In fact, if you have stood around a virtual table floating in a
neon-colored shifting psychedelic space age scenario while
video chatting with a friend wearing a digital cat on his head
and then attempt to document the action using a virtual sel-
fie stick, you might wish yourself back in the manageable
space of a cinema (fig. 7).
All these heterogeneous spaces, like the users them-
selves, are centered around the virtual table to create a
disturbing onslaught of many different screens and images
with varying degrees of realism that seem to collapse onto
7  Hybrid image space – Video chatting within Facebook Spaces.
the user. The designers foresaw this possibility which led to
the decision to support users with the ability to pause their
one might add – together and present them to make them experience should it become too overwhelming.43 What the
available for communication (“Verkehr”).40 The designers of designers did not anticipate, or at least not explicitly provide
Facebook Spaces tried to employ exactly this ability to keep for, is the effect that the interface design has on the referen-
things together in VR. tial status of images churning through the apparatus.
However, the fundamental logistical capabilities of the We would like to argue that the effects of apparatus_1,
table to draw things and people together in social VR are the HMD and sensors in your living room, and appara-
challenged by the same acts of communication it makes pos- tus_2, the virtual table and its plethora of gateways into
sible in the first place. One of the ways users can interact is
41 See Foucault 1997 (as fn. 34), p. 334.
by taking and sharing pictures and videos, thereby perfo-
42 Ibid.
rating the virtual space and linking it with other media and 43 See Tauziet (as fn. 4): “Whenever people want to take a break from their
actual spaces. Foucault described the ability of heterotopias experience, whether that’s because the pizza delivery guy is knocking at the
door, there’s a destabilizing shaky 360 video around them, or they’re feel-
to juxtapose several spaces in one space that are in them- ing uncomfortable, they can pause their experience by pressing the pause
selves incompatible – a heterotopia is not just a different button located on the inside of their wrist, or by taking their headset off.
This teleports them out of the space momentarily and into a ‘paused space’,
giving them a chance to catch their breath and take action if needed (reset-
186 40 Seitter 2001 (as fn. 8), pp. 178–179. ting the space, muting/kicking people out, reporting content…).”
Around a Table, around the World

other image spaces, interfere in a specific way. Not unlike perceivable space.45 HMDs make it specifically their point
the apparatus described by Baudry, the apparatus_1 of VR to place the user within a symbolic space which claims to
evokes an impression of reality that is, if anything, stronger be real while shutting out non-symbolic space – screens
than in cinema, because the subject in VR actually occupies are no longer an object within the users’ field of perception
the perspective focal point from which space is construct- but their only means of visual perception.46 The distinc-
ed. As perception and action are tightly coupled in VR, the tion between symbolic and non-symbolic space is further
ensuing sensorimotor coupling between the image and the undermined by Facebook’s advertising language, which
user’s body can create highly convincing illusions of embod- reproduces well-known topoi of presence and immersion,
iment.44 But at the same time apparatus_2 leaves a strong by promising to bring people together in one room and to
impression of unreality as it cuts any referential links images “transport you to new places” with the help of 360° videos.47
might have held in the past. This effect is increased by the In effect, one may say that Facebook Spaces, due to the inter-
CGI-based user avatars whose positioning and appearance ference of apparatus_1 and _2, generates a real symbolic,
is completely arbitrary with reference to the photographic within which the distinction of real and symbolic collapses,
image backgrounds. Images in Facebook Spaces may change thereby evoking an impression of surrealism, i. e. seemingly
places, be replaced or be subjected to post-production effects realistic representations of an unlikely and often bizarre
at a whim, whether or not they themselves were calculated character.
or taken. Even though it is well-known that digital or dig-
itized images can be manipulated in this way, traditional
screen practices constituted a symbolic space explicitly dis-
tinct from actual space to enable these kinds of operations.
The same does not hold true for VR which does not place its
screen before a user as a manipulable object but wraps itself
around the user’s head.
Whereas cinema, as described by Baudry, depended on
an interplay of psychological and architectural mechanisms
45 In turn, the general invisibility of apparatus_1 often makes it necessary to
to render its apparatus invisible, apparatus_1 of VR is phys- simulate visual representations of physical controllers, keyboards and other
ically invisible because it is situated outside of the user’s input devices inside apparatus_2. This leads to the effect that users interact
with images of devices they are actually holding in their hands because their
perceivable space while simultaneously constituting this
field of vision is blocked by the head-mounted display.
46 This observation is supported by an article on wired.de reporting from
Facebook’s developer conference F8. Visitors who wanted to test Facebook
Spaces were presented the headset by Facebook employees with the words:
44 To some extent, this has already been the case for videogames. See “Here are your eyes.” Elisabeth Oberndorfer, F8. Die neue Social-VR-App
­Serjoscha Wiemer, Körpergrenzen. Zum Verhältnis von Spieler und Bild von Facebook im Test, https://wired.de/collection/tech/facebook-­spaces-
in Videospielen, in: Britta Neitzel, Rolf F. Nohr (eds.), Das Spiel mit dem vr-virtual-reality-oculus-rift-app (accessed November 3, 2017).
Medium. Partizipation – Immersion – Interaktion, Marburg: Schüren, 2006, 47 Facebook, https://oculus.com/experiences/rift/1036793313023466/
pp. 240–260. (accessed November 3, 2017). 187
 Timo Kaerlein and Christian Köhler

offices – “one of the things that’s really magical about VR is

you can get the feeling you’re really in a place” (fig. 8).49 It
is here, in this “bizarre” 50 and “awkward” 51 video that was
almost instantly perceived as a huge PR disaster, that the
apparatus of Facebook Spaces instantiates what can most
accurately be described as a disturbing kind of virtual sur-
realism. Despite all assurances of experiencing a sense of
presence, the effect of the apparatus amounts not so much to
an illusion of transparency, but an illusion of homogeneity of
the images acting as the background for the virtual sociality
of Facebook Spaces. (fig. 8)
This illusion – the sense that there are no substantial
differences between the referents of the images processed
by the apparatus – is ultimately grounded in the digital char-
acter of these images: Digital photography has finally lost
all traces of indexicality in the world of Facebook Spaces.
The smartphone camera has indeed advanced to “the first
8  Mark Zuckerberg and Rachel Franklin visiting Puerto Rico in VR.
[mass-distributed, TK/CK] augmented reality platform”
that includes possibilities of enhancing one’s images with
“A new era of reality has arrived.” 48 Virtual various special effects and of manipulating photos with
Surrealism and the Loss of Referentiality the help of advanced object recognition capabilities, e. g.
to remove elements from them that disturb the staging of
A different scene, uploaded to Facebook on 9 October 2017, a perfect vacation shot. 52 Accordingly, images on Facebook
shows Mark Zuckerberg and Rachel Franklin in a promo-
tional video, advertising the features of Facebook Spaces 49 Mark Zuckerberg, as quoted in: Olivia Solon, Mark Zuckerberg “Tours”
Flooded Puerto Rico in Bizarre Virtual Reality Promo, https://­t heguardian.
while teleporting to different locations around the world, com/technology/2017/oct/09/mark-zuckerberg-facebook-puerto-rico-­
most remarkably a 360° video of Puerto Rico devastated by virtual-reality (accessed November 3, 2017).
50 Ibid.
hurricane Maria. Their good-humored avatars frolicking
51 Alison Maine, Mark Zuckerberg Took His VR Avatar to Puerto Rico, and
around and high-fiving in front of images of the catastro- It Was Just So Awkward, http://mashable.com/2017/10/09/mark-zucker-
phe, Zuckerberg and Rubin indulge in the experience of berg-virtual-reality-fail-puerto-rico/#Zgoe6d4m_qqC (accessed November
3, 2017).
being there without leaving the comfort of their respective 52 Mark Zuckerberg in Engadget, Facebook Spaces Announcement | F8 in
Under 10 Minutes, https://youtube.com/watch?v=JXYmpqdhHzg (accessed
November 3, 2017). In the same video, Zuckerberg explains another feature
48 HTC Vive, Vive Pre CES 2016, https://youtube.com/watch?v=CB9ecPgZlq0 with the words: “You can add a second coffee mug so it looks like you’re not
188 (accessed November 3, 2017). having breakfast alone.”
Around a Table, around the World

are becoming more and more self-referential in that they Figures

cease to point to any external referents whatsoever and 1  Facebook, Mark Zuckerberg.
instead exhibit the near limitless possibilities of digital 2  BagoGames, https://flickr.com/photos/bagogames/26104037025,
image manipulation. Creative Commons Attribution 2.0 Generic (CC BY 2.0).

Currently, influential voices like VR filmmaker Chris 3–6  Facebook, permission to use granted by Christophe Tauziet.

Milk talk of VR as an “empathy machine”, and some atten- 7  Jan Kaerlein, Timo Kaerlein, Christian Köhler.

tion has been attracted by the application of VR technolo- 8 Facebook.

gy to treat US soldiers’ PTSD by confronting veterans with

vivid simulations of traumatic events. 53 Our analysis of Face-
book Spaces runs counter to these expectations and concep-
tualizations by grounding the unsettling moral indifference
documented by the Zuckerberg/Rubin video media-theoret-
ically in the fundamental disinterest towards the referential
status of the images that are made part of the apparatus of
Facebook Spaces. Virtual surrealism in the sense described
here entails a form of reality-agnosticism: The question if
and how something is grounded in any materiality outside
the apparatus seems to have lost all relevance in Facebook
Spaces. By providing the sort of interface criticism we have
undertaken in our article, we can shed light on some of the
ways that the operationality of VR interfaces itself contrib-
utes, once again, to an agony of the real.

53 Chris Milk, How Virtual Reality Can Create the Ultimate Empathy
Machine, https://ted.com/talks/chris_milk_how_virtual_reality_can_
create_the_ultimate_empathy_machine, (accessed November 3, 2017).
See also Kathrin Friedrich, Therapeutic Media: Treating PTSD with Virtual
Reality Exposure Therapy, in: MediaTropes eJournal VI.1 (2016), pp. 86–113. 189
Sarah Atkinson

Synchronic Simulacinematics
The Live Performance of Film Production

When I think back, the notion of a simulacinematic space The screenwriter, Gregory Widen, was a former fire-
was first invoked in my own mind by an unpleasant memory man and he made it very clear that in our movie the
I had whilst visiting Universal Studios Hollywood in 2000 firefighters had to be right there in the middle of those
where I experienced the Backdraft theme park attraction. blazing infernos. But I knew that getting these shots
Backdraft, a 1991 film directed by Ron Howard, starring Wil- would be very hazardous for the actors and the entire
liam Baldwin and Kurt Russell, focused on fire fighters and film crew. Everyday, they found themselves in the mid-
their dangerous encounters with extreme conflagrations. dle of flame, smoke, flying ash and toppling scenery.2
The theme park attraction simulated the most d ­ ramatic
conditions of the film replete with flames, smoke and dan- This quote is indicative of the frequency, during interviews,
ger. The Backdraft attraction follows a lineage of theme where film industry practitioners seek to call our attention
park attractions that involve inferno-like conditions – for to the drama of the film’s making, and thus, by extension,
example “Fighting the Flames” was one such attraction at how materials and attractions, such as Backdraft not only
Dreamland in Coney Island in the early 20th century.1 The seek to position audience members in the fictional world of
main Backdraft experience took place upon a 500,000 cubic the film, but into the subjectivity of one of the film’s makers –
foot soundstage, dressed as the empty warehouse simulat- to vicariously experience the drama of the film’s production.
ing the climactic moment of the film. As the director yells I stood there in abject terror throughout, shuffling
Action a fire is seen to start in an office at the other side of myself toward the exit door, as most of the other thrill-seek-
the warehouse. When searching for documentation relat- ers shouted and cheered in appreciation, I experienced a vis-
ing to the Backdraft attraction, I found this quote from the ceral sense of panic, what if this is real? – no one will know or
director Ron Howard: realize. Unbeknown to me at the time of that experience, but
on recent research, I discovered that on 24 September 1992,
a fire had broken out in the air conditioning ducts above

Ron Howard, Scene by Scene Walkthrough. Scene 1, http://­t hestudiotour.

2
Andrea Stulman Dennett, Nina Warnke, Disaster Spectacles at the Turn of
1 com/wp/studios/universal-studios-hollywood/theme-park/past-­attrac
the Century, in: Film History 4.2 (1990), pp. 101–111. tions/backdraft/ (accessed January 23, 2018).
 Sarah Atkinson

the attraction. Around 500 people were in the attraction at folded together into simultaneous experiences in which the
the time, and to quote a news article – “No members of the two temporalities converge in an experiential modality.4
public panicked as they thought the black smoke was part Simulacinema is becoming an increasingly frequent
of the presentation.” 3 So my unease was entirely founded! phenomena as a result of digital technologies and their
This particular experience is not just of interest to me, use and application in cinema spectatorship, as well as a
because of its conflation of the filmic text and the condi- result of the evolution of cinematic commodification – the
tions of its making (the presence of a director’s voice, and expansion of the cinematic text across different forms and
the acknowledgement by the director of the same sense felt platforms – coupled with the commodification of cinematic
on set), but because of the experiential affective space that experience. Where Thomas Elsaesser has previously made
is occupied by the audience. It is a space that is routinely a distinction “between ‘cinema’ (event and experience) and
inhabited by performing artists, actors, film production per- ‘film’ (text and work)”, 5 I would introduce a third axis – that
sonnel (as the example of Backdraft illuminates) – it is the of filmmaking (process). In simulacinema, I would con-
assimilation by the audience into a simulacinematic space tend that the dividing line has become increasingly blurred
which is the central concern of this essay. between these three – text, experience and process.
Simulacinema is a portmanteau term – a combination Simulacinemic phenomena, which are characterized by
of the words simulation and cinema – which I am using to both aesthetic and affective qualities, have manifested in a
account for a phenomenon in which an audience simulta- number of different forms and contexts whereby the ontolo-
neously experiences both the space of the filmic diegesis gies of cinematic production and reception are experienced
and/or the cinematic spectacle, and the attendant, but cru- by the audience. Instances of simulacinema have occurred
cially, simulated space of its production. Simulacinematic on the set of film locations in major cities, within immer-
spaces are characterized by the uncanny sense of inhabit- sive theatrical cinema experiences (such as Secret Cinema)
ing two conflicting ontological spaces (fiction and reality) and on-stage theatrical performances that blend stage and
whilst also embodying two diametrically opposed subjec- screen techniques and aesthetics.
tivities (observer and participant). Simulacinematics refers
to the aesthetic and affective qualities of these spaces that
I here build on Guy Debord’s concept of the spectacle, Jean Baudrillard’s
4
merge film style and visual cinematic codes with production notion of simulacra and the work of Tom Gunning, in examining the history
­aesthetics, as well as the live and the mediated elements of of fairground and cinematic trajectories of showcasing of technological
apparatus from the birth of cinema. See Guy Debord, The Society of the Spec-
their experience. The making and reception of a film tend
tacle, Detroit: Black & Red, 1970; Jean Baudrillard, Simulacra and simulation,
to be chronologically displaced moments – but within sim- Ann Arbor: University of Michigan Press, 1994; Tom Gunning, The Cinema
ulacinematic phenomena, by contrast, the two moments are of Attraction. Early Film, Its Spectator and the Avant-Garde, in: Thomas
Elsaesser, Adam Barker (eds.), Early Cinema. Space, Frame, Narrative, Lon-
don: British Film Institute, 1990, pp. 56–62.
Thomas Elsaesser, Digital Cinema. Convergence or Contradiction?, in:
5
Backdraft On Fire, http://thestudiotour.com/wp/studios/universal-­
3 C. Vernallis, A. Herzog, J. Richardson (eds.), The Oxford Handbook of
studios-hollywood/theme-park/past-attractions/backdraft/ (accessed Sound and Image in Digital Media, Oxford: Oxford University Press, 2013,
192 January 23, 2018). pp. 13–44, p. 25.
Synchronic Simulacinematics

Within simulacinematic phenomena, I have identified Prochronistic simulacinematic moments are created
three different types of subjectivities or experiential modal- and experienced during the production of the film. I point
ities relating to temporalities – future, retrospective and to examples of the manifestation of prochronistic simu-
present, I refer to these as: Prochronistic, Parachronistic lacinema, Transformers 4: Age of Extinction (2014) and Sui-
and Synchronic. The first two types relate specifically to the cide Squad (2016).7 These are films that were both filmed in
marketization of cinematic experience, and are symptom- various cities across the world. In the case of Transform-
atic of the confluence between audience and fan practices ers 4, multiple locations were used in the USA, Hong Kong
and their exploitation by film marketers and the film indus- and in mainland China. Suicide Squad was filmed at var-
try. The simulacinematic in these two cases emerges as a ious locations across Canada (in particular in downtown
symptom and as effect of these two phenomena, as opposed Toronto) and Chicago. In both cases, spectacular car-chase
to a deliberate intervention or strategy on the part of the sequences, crashes, explosions and destruction were filmed
filmmakers or distributors. I position this in the wider trend in the inner-city locations. Audience members were given
towards the exploitation of the economy of film production a glimpse of the stunts and effects that were to come in the
(what John Caldwell has referred to as the Para-Industry) final films, and in close enough proximity to capture these
where making-of content becomes the marketing material.6 moments on their portable devices and then to distribute
The synchronic manifestations, which are the central con- them across various social media channels.8 Given the con-
cern of this essay, are at the very creative edges and man- text of the co-production between China and America, par-
ifest as a deliberate experimentation in the mediation and ticularly in relation to Transformers 4, these highly public
manufacture of screen-based texts. In all three instances we choices of location appear deliberate. In the case of Suicide
are able to examine what happens in the conflation between Squad, the sets were left in-situ on the streets of Toron-
the live and the mediated, what happens between the screen to, after filming had completed, as on-street installations
and the physical space – and in each instance there is a dif- which audience members could visit and be photographed
ferent relation. As I will go on to examine, in prochronistic against.9 Here, there is a complete collapse between pro-
moments, the peripheral screen practices and engagements duction, promotion and reception. Audience members are
of the audience characterize these moments, in parachro- present at the time of the spectacle whilst are also witness
nistic, the cinema screen is embedded in the experience, the to the point of its capture. They are immersed in the milieu
screen is centralized in synchronic simulacinema. of the fictional space at the same time at the point of its
I will firstly outline the key principles and character- making. And this is the key principle of simulacinematic
istics of both prochronistic and parachronistic phenomena
before examining synchronic simulacinema in more detail.
Michael Bay, Transformers 4. Age of Extinction, USA/China 2014; David
7
Ayer, Suicide Squad, USA 2016.
Kevin B. Lee, Transformers: The Premake, 2014, https://vimeo.com/
8
94101046 (accessed February 20, 2018).
John Thornton Caldwell, Para-Industry, Shadow Academy, in: Cultural
6 Aynne Kokas, Hollywood made in China, Oakland: University of California
9
Studies 28.4, 2014, pp. 720–740. Press, 2017. 193
 Sarah Atkinson

phenomena – the simultaneous presence of the audience in Parachronistic moments are created long after produc-
both the manufacture and the experience of the cinema­ tion has taken place – during the formal reception phase of
tic spectacle. As S
­ tephen Heath stated in 1980: “Resting on a film. Such instances emerge under a different commercial
an industrialisable technological base, cinema, different to (pre-promotional) imperative to the former category. They
theatre, offers the possibility of an industry of spectacle.” 10 have predominantly emerged in contexts of the generation
This notion of the industry being a spectacle in and of itself of (3rd party) retrospective revenue, through the screening
is bound up within the aesthetic and affective pleasures of of old, cult films. These instances can be aligned to the film
the film theme park, which I alluded to in the introduction theme park modality, where films, and their making, are
to this chapter. James Moran also alludes to this sensation repackaged and re-experienced for audience’s years after
at Universal Studios written in 1994: their release – i. e. the Jaws exhibit at Universal Studios.
UK-based organization Secret Cinema deliver immer-
Universal’s blockbusters spill over the screens as inter- sive experiences around a film screening through a recre-
active spectacles, which in turn sprawl onto the studio ation and reinterpretation of the fictional world of the film.
lots where they were originally spawned in a cycle that On the surface, these productions encourage and engender a
increasingly blurs production and exhibition, ‘reality’ variety of fan practices such as singing, dancing, cos-playing
and representation, ‘art’ and entertainment.11 and quoting-along to the film being screened. Furthermore,
and as a by-product, these productions, in their elaborate
It is through the presence of digital technologies that these restaging of a cinematic fictional universe, invariably mobi-
lines are not only being blurred, but they collapse entirely. lizes the mechanics of the film production industry machine,
In prochronistic moments this occurs through the unifica- through the hiring of film production personnel (set build-
tion of the screen and the space of production through the ers, sound designers, actors and stunt performers), and
presence of audience members’ screen capture equipment. through working with film distributors to secure licenses
The use of mobile phones and recording devices pre-medi- for screenings and in some cases new releases. In an arti-
ate the onscreen action in these moments. In parachronistic cle on Secret Cinema’s instantiation of Back to the Future,
moments the cinema screen acts as the linking screen inter- we drew out the significance of the emulation of the filmic
face between the action taking place around it. world and how an aesthetics of production was embedded
within the experience:

As with the Back to the Future event, it became appar-

ent very early on in the experience that as an audience
10 Stephen Heath, The Cinematic Apparatus. Technology as Historical and
Cultural Form, in: Teresa De Lauretis, Stephen Heath (eds.), The Cinema­tic member you are not actually immersed in the world
Apparatus, New York: St. Martin’s Press, 1980, pp. 1–13, p. 7 [emphasis added of Hill Valley, you are immersed in the world of its
by the author].
11 James Moran, Reading and Riding the Cinema of Attractions at Universal
­making – such was the presence of the physical ­
194 Studios, in: Spectator 14.1 (1994), pp. 78–91, p. 79.
Synchronic Simulacinematics

evidence of its construction (scaffolding, light rigs and output of the finished text. Synchronic simulacinematics are
scenery), populated by stunt vehicles, production and highly reflexive and afford a critical edge to understanding
security personnel.12 this phenomena of dual audience pleasures. In this manifes-
tation of simulacinema we witness a celebration of the cine-
Parachronistic simualcinematic moments therefore become matic apparatus and the visual spectacle that this creates. It
characterized by these unintentional, accidental instances is a complex space where theatre and cinema coalesce and
of staging filmmaking aesthetics. The presence of the screen the cinematic production process is itself conceptualized as
calls to attention the audience’s awareness of the construc- a form of live theatre. In synchronic simulacinematic case
tion and the artifice of film – as it arguably happens in all studies, the screen is absolutely central to the concurrent
simulacinematic moments. In these cases, the screen is creation of an image for the screen. The creative actions of
embedded into the experience itself (the screen is literally film production and practice come into focus and transform
framed within the experiential space – in the screening of themselves into theatre-show.
Back to the Future, the screen is centralized in the Hill Valley I am putting forward two case studies where the act of
town hall façade; in Moulin Rouge, the screen is framed by production becomes the act of performance and there is a
the stage of the famous Parisian night club). simultaneous collapse of production, performance, capture,
The reception of a filmic text is bound up in the appre- transmission and reception. The first is a strand of work by
ciation of its making, so much so, that the two conflate. As theatre director Katie Mitchell – and its evolution over three
with the former category – this is not necessarily a con- productions – Waves (2006), … some trace of her (2008), and
scious decision undertaken on the part of the creators but Forbidden Zone (2014).13 The second is Kid Koala’s Nufonia
rather a symptom of the always-intertwined nature of film Must Fall Live (2014).14 The analysis of these two different
production and film reception, as well as dual audience and examples has been undertaken through the study of their
fan pleasures of meta-filmic awareness. associated documentary videos in order to examine both the
The most sophisticated and complex form of simulacin- form and techniques of the pieces, as well as the discourse of
ema and the most conscious form through artistic intention- their description. The documentation and framing of these
ality (I have argued that the other forms are unconscious/ two examples are key to conveying their simulacinematic
unintentional on the part of the creators) is synchronic qualities, and themselves become part of the economy of
simu­lacinema. These are moments created during produc- film and theatre production. It is the simultaneity of screen
tion – designed to be appreciated in synchronicity with the
13 Katie Mitchell, Waves, London: National Theatre 2006; Katie Mitchell, …
12 Sarah Atkinson, H. W. Kennedy, From Conflict to Revolution. The Secret some trace of her, London: National Theatre 2008; Katie Mitchell, Forbidden
Aesthetic and Narrative Npatialisation in Immersive Cinema Experience Zone, London: 59 Productions 2014.
Design, in: Participations. Journal of Audience & Reception Studies 13.1 14 K. K. Barrett, Kid Koala (Erik San), Nufonia Must Fall, Montreal: ­Envision
(2016), pp. 252–279, p. 274. See Sarah Atkinson, H. W. Kennedy, ‘Tell no one’: Management & Production 2016, Live Performances, Ann Arbor Center,
Cinema as Game-space. Audience Participation, Performance and Play, in: March 11–12, 2016; K. K. Barrett, KID KOALA, NUFONIA MUST FALL
G|A|M|E. The Italian Journal of Game Studies 4 (2015), https://gamejournal. LIVE! [Official Trailer], 2016, https://youtube.com/watch?v=s_DhuuHt76M
it/atkinson_kennedy/ (accessed August 20, 2017). (accessed August 20, 2017). 195
 Sarah Atkinson

space and physical space showing the same events that is production which is simultaneously being performed, filmed,
the principle difference to the previous instances of simu- projected and observed live on a screen above the stage,
lacinema that I have described. underneath which audience members are able to see the inner
The two case studies are drawn from a number of workings of the film set in which the on-screen action is being
notable examples of projects, which have sometimes been shot. Production crew, i. e. camera operators and sound
referred to as “Live Cinema”.15 These include works by Film recordists, negotiate the film set in full view of the audience as
Live in Italy, a group of artists whose practice is to make and they frame the action and capture the performance. This
broadcast films live – “a movie that is filmed at the same viewing mode, in which the audience can constantly switch
time that it is screened”.16 Francis Ford Coppola has also between the registers of fictionality and its construction
worked in a similar mode with his project Distant Vision, invokes a metafictional experience and awareness, which
which he also refers to as “live cinema” where a film was can on the one hand create as Patricia Waugh states a “­fiction
shot and broadcast live to screenings rooms on 22 July 2016 that both creates an illusion and lays bare that i­ llusion” 19
after 26 days of rehearsal.17 Coppola positions live cinema in or as Thomas Elsaesser has noted: “the production process
contradistinction to live multi-camera broadcast, associated can take on a textual form”.20 There is a key distinction to
with the televisual: be made here between live cinema and simulacinema. Syn-
chronic simulacinema occurs when audiences have access
I felt the need to experiment in order to learn the actual and are witness to both the on and off screen spaces, and not
methodology of live cinema, which is a hybrid of theater, just the on screen-output as is the case in the Coppola exam-
film and television. The shot is the basic element, as ple. In synchronic simulacinema, the audience can take an
in film; the live performance is from theater; and the active role in what they choose to focus their attention upon;
advanced television technology to enable it is borrowed they take the vantage point of a director who watches both
from TV sports.18 the monitor of the camera output on set and the produc-
tion itself. The distinction is that the director has the active
59 Productions and Katie Mitchell refer to the third and final power to make choices about where the camera directs its
piece Forbidden Zone production as live cinema – a t­ heatre lens, whereas the audience are passive in this regard.
The format of Forbidden Zone is based on Mitchell’s
15 Sarah Atkinson, H. W. Kennedy (eds.), Live Cinema. Cultures, Economies, earlier productions Waves and … some trace of her, which
Aesthetics, New York: Bloomsbury, 2017.
at that point were referred to as a multi media productions.
16 http://film-live.org (accessed January 18, 2018).
17 Bill Desowitz, Francis Ford Coppola Completes ‘Distant Vision’ Live Cinema
Workshop at UCLA, http://indiewire.com/2016/07/­f rancis-ford-coppola-
completes-distant-vision-live-cinema-workshop-at-ucla-1201709229/ 19 Patricia Waugh, Metafiction. The Theory and Practice of Self-Conscious Fic-
(accessed August 15, 2017). tion, London: Routledge, 1984, p. 6.
18 Dave McNary, Variety Francis Ford Coppola Starts Experimental ‘Live 20 Thomas Elsaesser, Fantasy Island. Dream Logic as Production Logic, in:
Cinema’ Project at UCLA, http://variety.com/2016/film/news/fran- Thomas Elsaesser, Kay Hoffman (eds.), Cinema Futures. Cain, Abel or Cable?
cis-ford-coppola-experimental-live-cinema-ucla-1201820998/ (accessed The Screen Arts in the Digital Age, Amsterdam: Amsterdam University Press,
196 August 15, 2017). 1998, pp. 143–157, p. 143.
 Synchronic Simulacinematics

Waves was a work devised from the fragmented text of Vir- speaks his thoughts and how he “just provides his face”. 23
ginia Woolf’s novel, The Waves (1931). …some trace of her is This proves consistent with traditional filmmaking conven-
inspired by Fyodor Dostoevsky’s The Idiot (1868–1869). Both tions in what could be referred to as the “cinefication” of the
later productions include the visual production of live sound theatre.24 In these examples, we see how theatre explores
effects and real-time video captured and projected on-stage.21 and reveals, whilst cinema and film continue to conceal.
For the purposes of this essay I am focusing upon the princi- These techniques raise questions around whether what is
ples, techniques and aesthetics that are deployed in Mitch- being produced is for screen or stage consumption, or in
ell’s productions in relation to the staging of production aes- this case, the in-between – the simulacinematic space. In
thetics and mechanics. I posit that these are not transparent the most advanced and technically sophisticated iteration
filmic productions-techniques but artistic techniques. of the simulacinemtic aesthetic in Forbidden Zone, it is the
Within all three Mitchell productions, the performative digital which is foregrounded in both form and content. Dig-
aspects of film production are staged through the use of real- ital technologies make possible the live and simultaneous
time production aesthetics. These provide a simulational capture and broadcast of image and sound, whilst digital
tendency themselves as on the set of actual film productions aesthetics are laid bare in its presentation. In Mitchell’s
action is fragmented across scenes and takes, action is cut earlier works, it is the analogue, the craft-based, the hand-
for cameras, lights and production personnel to re-set. Clas- made (foley production techniques) that are experimented
sical narrative film production is a mode that has persisted with. Mitchell describes the approach to the live creation
since its establishment in the 1890s – very often dictated of sound effects through foley as the theatricalization of
by the economics of production and the availability of loca- sound “where the image you see on screen is matched with
tions and performers. Mitchell describes this approach as something entirely different, the realization in the perfor-
“fragmenting the stage picture, combining video output with mance and then the audiences as they watched this cre-
the live construction of it”, whilst Ben Whishaw who plays ation live”.25 There is an example where a performer taps a
the character of Myschkin in …some trace of her explains piece of chalk on a blackboard in order to create the sound
it as “the juxtaposition of image and the artificialness of of fingers tapping keys on a typewriter. Here we witness
the way that image is made”.22 Whishaw goes onto describe the sound being created for the film soundtrack, the film
how another performer plays his hands, how another audience. Performing it in this way creates an audio/visual
disjuncture, whilst also invoking the loss of the real, the
absence of the authentic.
21 For in-depth analyses of Waves see Louise LePage, Posthuman Perspectives
and Postdramatic Theatre. The Theory and Practice of Hybrid Ontology in
Katie Mitchell’s The Waves, in: Cultura, lenguaje y representación. revista 23 National Theatre Discover, Ben Whishaw on Acting in a multimedia produc-
de estudios culturales de la Universitat Jaume I 6 (2008), pp. 137–149; Janis tion, 2011, https://youtube.com/watch?v=5hK0y8tN29w (accessed August
Jefferies, ‘… some trace of her’. Katie Mitchell’s Waves in Multimedia Per- 20, 2017).
formance, in: Women. A Cultural Review 22.4 (2011), pp. 400–410. 24 Vsevolod Meyerhold, The reconstruction of the theatre, in: Edward Braun
22 National Theatre Discover, Katie Mitchell on directing multimedia produc- (ed.), Meyerhold on Theatre, New York: Bloomsbury, 2014, pp. 253–273.
tions, 2011, https://youtube.com/watch?v=rAij9r9RvF0&t=2s (accessed 25 National Theatre Discover, Sound design for ‘...some trace of her’, 2011,
August 20, 2017). https://youtube.com/watch?v=THpcmuKNumY (accessed August 20, 2017). 197
 Sarah Atkinson

A principle characteristic of synchronic simulacinema

is that the productions are experienced in real-time – the
seduction of this aesthetic prevails in one-take cinema. 26
Time on stage unfolds at the same time as time on screen,
and it is the complexities and complications of this audiovi-
sual achievement that are laid bare in the production. The
fluid on-screen action and the flowing movement of the
characters is juxtaposed and contradicted by the on-stage
presence of the fragmented set and the urgent choreography
of the production personnel as they negotiate the presence
of the apparatus – the cabling, lights, set etc. This invokes
the labour in the viewer who also has to visually negoti-
ate and cohere the audio/visual complexity – this affective
labour is a key trait of simulacinema: while time is parallel
in the theatre-space and screen-space, dimensionality or
space is not.
The accompanying documentary by 59 Productions
presents the drama of the production and the high stakes
that at any moment – anything can go wrong, all at once
deliberately inhabiting, celebrating and performing “the
commercial drama of a movie’s source”. 27 This sense of
1  Forbidden Zone. The camera operator captures the performer looking in the mirror in the set in the bottom left.
liveness is intrinsic to the synchronic simulacinematic. The
presence of the spectator at the moment of capture is cen-
As Figure 1 illustrates, the overall formal quality of tral to the experience, whilst the live is also a marketing
Mitchell’s stage production creates a visual multi-screen and promotional tool, serving to historicize these instanc-
picture-in-picture effect, invoking a digital aesthetic (a trope es as on-off, unique and ground-breaking. Similarly, in the
in various multi-screened films that proliferated in the early next case study under consideration, Nufonia Must Fall
2000s with the advent of digital editing, and an aesthet- Live, the director K. K. Barrett emphasizes the importance
ic of digital postproduction edit interfaces). The viewer is of ­liveness:
positioned as voyeur and is witness to the various screened
vignettes, reminiscent of the aesthetic of Alfred Hitchcock’s 26 See Sarah Atkinson, “You sure that’s a film, man?” Audience anticipation,
Rear Window (1954). expectation and engagement in Lost in London LIVE, in: Participations.
Journal of Audience and Reception Studies 14.2 (2017), pp. 697–713.
27 Timothy Corrigan, A Cinema without Walls. Movies and Culture after Viet-
198 nam, New Brunswick: Rutgers University Press, 1991, p. 118.
Synchronic Simulacinematics

[I]n this world of being able to get anything on-demand,

you have to come to the theatre currently to see this
show, that’s what makes it special […] it’s going to evap-
orate after this until its next performance.28

Nufonia Must Fall Live was a simultaneous stage and screen

performance described as a live silent film conceived by Eric
San, better known as Kid Koala adapted from his silent comic
book of the same name. In the 60-minute performance piece,
the on-screen animation was performed live through pup-
petry which the audience could watch on the screen above
the stage and below the screen (fig. 2). The stage consisted
of 12 different model sets lit with LED, which were animated
by puppets and puppeteers, all visible to the audience, along
with a camera operator, sound engineer and video editor.
The performance was accompanied by a string quartet and
Koala’s own scratch DJ music. San describes it as

the most complicated show that I’ve ever been a part

of, it’s like this eight-ring circus, amazing energy and
chaos on stage; but what we hope about this eight-ring
2  Nufonia Must Fall Live. The three puppeteers to the right of the stage are visibly manipulating the puppets within the
circus is this fluid, unified feeling that we create on set that we see on the screen above the stage. The four musicians on the left play the accompanying soundtrack.
screen and for the people in the audience just to drift
away with that story; but then they can have that pic-
ture-in-picture vibe where they can look down and see Here San alludes to a similar effect created in Mitchell’s
how it’s all happening in real time.29 work using the language of new digital media – real-time
and picture-in-picture. The unification which San seeks to
achieve between the performance and its making on the
surface appears to be highly challenging since animation
production is the antithesis of live viewing – it requires
28 Banff Centre for Arts and Creativity, Nufonia Must Fall: A Making of the
Stage Production, 2014, https://youtube.com/watch?v=K01BWCWk6ek extensive and timely production work. The meticulousness
(accessed February 4, 2018). and effort of stop-frame animation processes is very often
29 The Creators Project, Puppets, Turntables, And A String Quartet | K. K.
Barrett & Kid Koala’s “Nufonia Must Fall Live”, 2014, https://youtube.com/
celebrated and foregrounded in industry discourses (i. e.
watch?v=HFOImWFUL7k (accessed August 20, 2017). Aardman animation feature films reportedly take eighteen 199
 Sarah Atkinson

months to shoot with 25 to 30 animators working across 25 Secondly, the revealing of the secrets of animation
sets at once). 30 production has its historical antecedents at the turn of the
In Nufonia Must Fall Live, it is a theatrical form of 20th century where craft-based techniques and the human
animation – live puppetry – that is produced and staged. labour behind the production were revealed in early pro-
As such, the performance follows two different cultural to-making-of documentaries. 33
trajectories, the first being that of the fairground and the Through this study, the emergent functions, leitmotifs
show­casing of film apparatus and the second that of the and aesthetics of simulacinema phenomena have emerged
making-of and behind the scenes, the fascination with the through a fusion of forms that merge the live with the
process and magic of filmmaking which is as old as the his- mediated, the screen with the stage, and  – through the
tory of cinema itself. mobilization of film production machinery – its production
Firstly, San alludes to the spectacle of the moving mechanics, processes and techniques. The creation of the
image through his reference to the circus. The simultane- screen image is central: in parachronistic simulacinema, the
ous viewing of the illusion and its source has its trajectory screen image is cohered by the spectators, in prochronsit-
in cine-fairground attractions as Gunning states “the earli- ic the screen image is recreated by performers and sceno­
est years of exhibition the cinema itself was an attraction”, 31 graphy, and in simulacinema the screen image is subject to
using the term “attraction” to emphasize “that of exhibition- live creation.
ist confrontation rather than diegetic absorption”. 32 Film In Mitchell’s work, attention is deliberately placed on
has always showcased its techniques and artistry since its the artifice of film-based production; the inauthentic arti-
early history and has always delighted through this. Today’s fice of the construction of the screen image is highlighted.
viewer knows how film is made and thus takes pleasure in In Nufonia Must Fall Live, it is the spectacle of screen-image
film-production being made into its own form of spectacle, creation that is foregrounded – the production is a perfor-
thus becoming an image for a screen without being present- mance and the artform and craft of animation production
ed on a screen. With the zoetrope for example, the audience is celebrated.
first see the mechanics and the machinery kick into action Simulacinema is a concept which most usefully helps
before looking deeper and immersing themselves in the to extend understandings of the complexities of audience
illusion within. It is here that we see a return to fairground viewing pleasures; the evolution of creative practice in per-
practices of cinema exhibition, where the technology is formance and the complexities of the commodified ecosys-
revealed and showcased before the illusion itself. tem of film and cinema including its intrinsic and endemic

33 Examples include Wallace Carlson, How Animated Cartoons Are Made,

USA 1919; How Walt Disney Cartoons Are Made, USA 1939; Dave Fleischer,
30 Kate Abbott, Nick Park, Peter Lord, How we made Wallace and Gromit, in: How the Fleischer Studios, Miami, Florida, made ‘Aladdin and His Wonder-
The Guardian, March 3, 2014, https://theguardian.com/tv-and-­radio/2014/ ful Lamp’, USA 1939; Alfred L. Werker, The Reluctant Dragon, USA 1941.
mar/03/how-we-made-wallace-and-gromit (accessed January 18, 2018). See a detailed study in Sarah Atkinson, From Film Practice to Data Process.
31 Gunning 1990 (as. fn. 3), p. 58. Production Aesthetics and Representational Practices of a Film Industry in
200 32 Ibid., p. 59. Transition, Edinburgh: Edinburgh University Press, 2018.
Synchronic Simulacinematics

politics. The synchronic simulacinematic examples are sug-

gestive of a politicization of production through the per-
formance of labour and the simultaneous “pseudo visible,
hyper invisible” economy of film production. 34
There is significant labour required of the audience to
comprehend and to process these complex experiences. Sim-
ulacinema is affectively taxing, and laborious, the viewer
has to always shift in focus between two realities to grasp
the overall production.
As cinema and theatre continue to evolve, merge
and converge, the continued dissimilation of simulations
becomes an increasingly complex, yet important task, par-
ticularly in relation to the decoding of mediations of film
production so that we may evolve critical understandings
of the emergent and seductive economy of live within the
contemporary film and cinema ecology.

Figures
1  Katie Mitchell, The Forbidden Zone, Photo: Stephen Cummiskey.
2  Kid Koala, Nufonia Must Fall Live, Festival Noorderzon, Photo: Pierre
Borasci.

34 Sarah Atkinson, From Film Practise to Data Process: Production Aesthetics

and Representational Practices of a Film Industry in Transition, Edinburgh:
Edinburgh University Press, 2018, pp. 161–168. 201
Ina Neddermeyer

I Want to See How You See

Curatorial Practices of Exhibiting Virtual Reality

Displaying time-based media art poses major challenges for the process of seeing. Despite their diversity all works in
the presentation and reception in exhibitions. Sound over- the following essay can be characterized as VR-HMD thus
lays, distraction effects or the missing reset button, which virtual realities consisting of CG Imagery generated in real
would make it possible to start an artwork as soon as visitors time and displayed on HMDs.
enter the room are everyday problems in curatorial practice.
Virtual reality (VR) installations can be understood in this
VR Headsets as Unbounded Displays
tradition of time-based media art, while at the same time
a fundamental change is taking place with the increasing In VR installations with HMDs, the displays are no lon-
availability of head-mounted displays (HMD) in exhibi- ger presented in a statically installed form, e. g. a monitor
tions. While VR installations can be displayed on tradition- or a video projection, with which the visitors are usually
al screens (on VR monitors or as cave installations), most confronted in time-based media exhibitions. Instead, the
artists aim to let their virtual worlds be accessible through mobile display is flexibly available. The application starts
HMDs (fig. 1–2). So even if the VR artwork cannot be clearly whenever the visitors put the headset on and stops as soon
defined, its usual combination with such a screen-set up has as the HMD is taken off.
to be taken into account discussing the changes in the use In this moment the screen as a physical object fades into
and function of VR artworks. the background. While video projections and monitors as
Approaching the topic through curatorial practice displays are clearly delineated and always visible, the head
gained in the exhibition Beautiful new worlds. Virtual real- mounted displays seem to disappear. In the immersive
ities in contemporary art in the Zeppelin Museum Fried- installations the visitors perceive the virtual images pri-
richshafen, this essay aims to take stock of the possibilities marily as unframed image spaces. The boundary between
and challenges of VR installations exhibited with HMDs, image and non-image blurs as the virtual image offers a 360°
reflecting the presuppositions and specific modalities of view in which the screen is not visible anymore. As soon as
individual artworks. The spectrum ranges from interac- the VR-HMD is put on, there is no possibility to look away;
tive, experience-orientated works to linear-narrative works, viewers are completely in the virtual world and with the
from VR displayed on headsets to room-filling installation only possibility to leave being to take the HMD off. Instead
settings, which also critically deal with the technology and of merely viewing an art work from an outside view, virtual
 Ina Neddermeyer

1  Beautiful new worlds. Virtual realities in con­ 2  Banz & Bowinkel: Palo Alto, 2017, anodized and powder coated aluminium tubes, 55¨ LCD monitor,
tem­porary art, Zeppelin Museum Friedrichs­h afen, HTC Vive VR, VR PC, custom printed carpet, installation view.
November 11, 2017–April 8, 2018, exhibition view.

reality means experiencing works of art from an inside view installations of the artist duo Banz & Bowinkel the viewer
and thus in a supposedly immersive way. Visitors become a navigates through virtual worlds by means of controllers
part of the virtual world. operated by touching a button that leads to action sequences,
Further, viewers have an active part. They are directly such as movements through the space (fig. 2). The narra-
addressed and involved by wearing VR headsets or experi- tive is therefore developed non-linearly and always differ-
encing interactive elements within the VR installation. The- ent depending on the choices that were made. These three
oretically, it is even possible to move with the HMD relative- elements, the inner and outer freedom of action as well as
ly freely through the exhibition space. This mobility relates the disappearance of the technical equipment, lead to the
not only to the visitors’ external freedom of movement but impression that VR-HMDs are unbounded displays. But
also to a large area of ​​actions within the VR installation – this is more a felt impression, as there are a lot of technical
when, for example, there is no longer an auctorial narrative, limitations that such installations, as almost all VR instal-
but the viewers become protagonists who interact with the lations, impose.
204 virtual environment that is created by the technology. In the
I Want to See How You See

First of all, the so-called invisible technology is revealed

by cabling which curbs the movement space of the viewers.
Secondly, there are the headsets themselves, which remain
visible with their edges and noticeable in their weight. To
create a perfect illusion, heavy displays are necessary that
counteract with the immersive experience. Thirdly, the size
of the exhibition space that can be tracked by infrared cam-
eras is limited due to the bandwidth, and so the scope of
action is limited as well. Standalone VR headsets might be
launched within near future but quite possibly to the disad-
vantage of quality. Furthermore, elaborated VR installations
that include, besides the VR environment, live simulation,
tracking systems and interactive experiences are not as
reliable and stable enough for everyday use. For instance,
synchronization problems occur (moving one’s head does
not necessarily result in a simultaneous virtual movement),
PC and HMD loose connection, the tracking system might
not work or it is necessary to reset the PC due to technical
3  Harun Farocki: Parallel II, 2014, single channel video installation, color, 9 min., still.
problems.
With his series Parallel (2012–2014) Harun Farocki
already showed very early that the seemingly unbounded
virtual image spaces have perceivable boundaries (fig. 3). aries of virtual grid cages are very quickly perceivable, since
In these 2D videos the artist deals with computer gener- viewers might step out of the programmed environments
ated images, especially with video games. The boundaries and cause the VR installation to fail.
are, on the one hand, spatial ones that can be understood
as consciously or unconsciously programmed endpoints.
Immersion and Corporeality
They limit the scope of action. Users are asked to return or
come back if they are leaving the official part of the game, or Most of the VR installations tend to aim to sensorily over-
insurmountable objects, barriers or parts of the landscapes whelm the user with immersive effects. Although, probably
appear. On the other hand, there are limits on social norms in the not too distant future, VR-HMDs will become an inte-
and interactions (i. e. which behaviors cause a reaction gral part of everyday life; presently, they represent a new
and how other players, authorities and leaders in the game technological setup for a large part of the population, pro-
impose sanctions). These experiences from video games can viding an entirely different visual experience. These lead to
be transferred to virtual environments in which the bound- a strong physical involvement, which is a challenge for the 205
 Ina Neddermeyer

unique experience: When they look down at their bodies,

often there is no virtual representation; their bodies are
invisible. Some artists therefore use avatars to overcome
this difference of the real and the virtual world, and thus
giving the invisible body a virtual stand in-representation.
It is interesting to note that processes of identification take
place very quickly, and the virtual body of an avatar is per-
ceived as one’s own physical body.
At the same time, the physical body is strongly empha-
sized by interactive elements that enable new physical expe-
riences in a lot of VR installation. Head movements lead to a
direct change in perspective and angle of view. With the aid
of controllers, viewers can move through space. Or the other
way around, in Florian Meisenberg’s VR installation hand
movements in the exhibition room are transferred directly
into the virtual space by means of infrared cameras (fig. 4).
Meisenberg created a VR installation on site for the
Zeppelin Museum in which he works with the Leap Motion
4  Florian Meisenberg: Pre-Alpha Courtyard Games (raindrops on my cheek), 2017, VR-Installation, custom made
VR Software build with Unity, VIVE VR Station, Leap Monitor Controller, custom made carpet, scaffold, plexiglass, technology as a tracking tool. Instead of a “traditional” VR
projectors, installation view. installation in which visitors have to use a controller to
move around, the Leap Motion device makes it possible to
sense of balance, and reveals a different spatial experience track the movements of the hands without using control-
of images as users move and change their perspective in the lers or gloves. The hand movements are directly tracked
virtual worlds. At the same time this corporeal involvement and translated to actions in the virtual space, where visitors
is juxtaposed with a distance to the virtual images, because can shape the appearance of wireframe objects intuitively.
it is not possible to touch and feel objects as physical objects Here the hands are represented as avatars, as it would be
(The Swayze Effect).1 too confusing for the visitors to have no corporal represen-
VR-HMDs change the relationship between body, space tation in the virtual world. Furthermore, it would be diffi-
and display. It is not only the exhibition space that is omit- cult to interact with the technology if the user is unable to
ted by the virtual reality simulations, but the body seems see what their hands are doing. In a second step the visitor
to disappear too. For the visitors this might offer a very can place a texture on the wireframe to create a sculpture
that can be uploaded on sketchfab.com, an online p ­ latform
https://oculus.com/story-studio/blog/the-swayze-effect/ (accessed June
1
for publishing 3D, AR or VR content. Both the wireframe
206 12, 2018). objects and textures are part of pre-programmed data-
I Want to See How You See

base. Consequently, the viewers who are engaged actively VR works place new demands on art education especial-
are co-producers of these artworks. It has to be mentioned ly. To be confronted with VR installations for the first time
as well that Florian Meisenberg works closely togeth- raises simple questions such as: How do I put on a VR-HMD?
er with a programmer to create the application, thus What about my belongings, if I cannot see (and hear) what
invigorating the interdisciplinary interface between art and is happening right next to me? Do I have to sit down while
technology. watching VR videos? The supervisory staff is now no longer
The VR simulation is embedded in a space-filling instal- solely responsible for guarding the exhibits but must also
lation of a giant carpet hanging from the ceiling that gives provide support as a VR sitter. They have to pay attention
the impression of a photographic studio. So Meisenberg that the visitors do not get tangled up or stumble over a cable
is weaving together the virtual and physical layers of the while they are travelling through virtual worlds.
works. A live VR simulation is projected on this carpet. But This support is also necessary as the new spatial expe-
viewers from outside can only see the isolated gestures of riences make viewers feel their bodies in a very strong and
the hand avatars; they are not able to see the artworks. direct sensation. In some cases they suffer from motion sick-
Through movement and without further tools, one’s ness. Nausea is caused by the lack of proprioceptive signals
own physicality becomes even more strongly represented corresponding to the impression that you should be moving
in the virtual environment. This simultaneity of disembod- physically as your point of view changes. For this reason, the
iment on the one hand and at the enhancement of corpo- first-time or extensive use of VR-HMD is associated with a
reality on the other is to be seen as a defining trait of VR great deal of effort, especially for untrained visitors, which
installations, which is reflected on by numerous artists. is revealed in quite physical reactions.
With only a few exceptions, it has generally been pos-
sible to watch time-based media art together with others.
Challenges for Mediation
Viewing a VR installation is a thoroughly solitary experi-
VR installations are a major challenge to master in the ence. It is only possible that one person wears the VR-HMD
exhibition context. Just the simple fact that, the VR-HMDs at a time, so that another person is not able to see the video
used in exhibitions are regularly worn by many different installation. The other visitors are thus excluded to a cer-
people naturally raises questions about hygiene. Due to the tain extend, and therefore the process of being observed
close eye contact there is the danger of transmitting viruses while observing is reinforced. Furthermore, talking about
and bacteria, e. g. a conjunctivitis. For proper care the use the artworks thus becomes a challenge. The process of
of special VR cleaning sets is necessary, also to maintain simultaneous art viewing and verbal exchange becomes a
the function of the lenses. Despite the fact of producing an downstream process, and a very practical question for guid-
incredible amount of waste, we decided that every visitor ed tour arises: How is it possible to speak about VR works
should receive a personal hygiene cover that can be used at that not everyone has seen or cannot even get a glimpse at
every VR station in the exhibition. Before putting the HMD together?
on, the disposable hygiene cover is placed on the face. 207
 Ina Neddermeyer

This evokes the question of how many VR headsets are

installed for each artwork in the exhibition space. If only
one headset is available, the mediation is difficult at well-at-
tended exhibitions or when groups are present. Projections
or monitors have generally been available for (almost) all
visitors; here a curatorial decision must be made in the case
of works that are not bound to a certain setting. Installing
more than one VR-HMD means also financial challenges. A
perfect solution would be to enable as many individual art
experiences as possible.
A provisional solution, which is already practiced in
some exhibitions, is to show, in addition to the VR-HMD, a
projection of what the visitor sees in the VR installation. The
artists Salome Asega, Reese Donohue and Tongkwai Lulin
decided to show their VR installation ASM(V)R combined
with a welcome screen/image, before the visitor jumps into
the Oculus Rift VR-HMD (fig. 5). This way the visitors can
get an impression of the work by looking at a 2D video snip-
pet of the VR simulation. It is, of course, important to note
that a transfer from VR-HMD set-up to a conventional 2D
screen is a profound intervention in the work, whereby the
core characteristics of VR, the individual experience of the
viewer and the specific reaction of the program to the view-
er, are completely eliminated.
Discussing these challenges, it should also be mentioned
that some common problems do not occur: Sometimes time-
based media art raises questions about the protection of
minors when watching videos with pornographic or vio-
lent content. This problem is not as virulent when using
VR-HMDs, since actively donning the headset is necessary
to watch the simulations, and additional warning signs will
5  Salome Asega & Reese Donohue & Tongkwai Lulin: ASM(V)R, 2017, Oculus
Rift VR, PC, headphones, projector, bean bag, 6.22 min., installation view. make sure that no one steps into the artwork by accident.

208
I Want to See How You See

Artistic Potentials of VR Technologies

VR technologies open up many new possibilities for artists

to reflect on their aesthetic potential while at the same time
critically examining their technical conditions. Most VR
artists give precise instructions how the artworks should
be installed, which VR system they prefer, how many head-
sets have to be installed in an exhibition, if headphones are
necessary, the sitting (or standing) accommodations (e. g.
color and material of the bean bags) as these aspects are a
fundamental part of the artworks. Of course these instruc-
tions depend on the specific spatial (and financial) situation
of the museum and are adapted for every exhibition.
Sidsel Meineche Hansen’s VR installation Dickgirl
3D(X) deals with VR pornography as the porn industry is
one of the biggest engines of innovation for VR technologies.
It is a critical pornographic VR production that examines
6  The Nest Collective: Let This Be A Warning – A VR Short Film, 2017, VR video, 10.43 min., Oculus Rift VR, still.
not only post-human sexuality but also the voyeuristic view.
By using VR-HMD and because of the isolation from the out-
side world, the installation of immersive pornography gives tage, which makes the visual process appear in a new light
the impression of a private setting. The setting is specifically that underlines the performative aspect of the installation.
defined by the artist: The VR-HMD has to be positioned on a The artist duo Banz & Bowinkel deals with the question
black vegan leather beanbag that allows viewers to sit com- of how virtual image worlds and physical spaces can be con-
fortably. This setting of the installation creates the illusion nected to one another and how virtual reality can be made
of privacy, of watching porn on your own sofa or bed. At the visible to outsiders. Their VR headsets are embedded in
same time, however, you find yourself in the public setting of installations with sculptural elements and printed images.
an exhibition space, which allows other visitors to watch you The virtual environment is transmitted to a monitor, so the
watch porn. Thereby the act of waiting for a set of VR-HMD actions are also visible to outsiders. Banz & Bowinkel thus
to be freed up while observing other visitors becomes an discuss the visual process as a central element of VR works,
important part. But only by watching the videos themselves which raises questions of participation: Who can see which
can the viewers understand what they just saw the person video and who is excluded? Within their VR installations,
before them do. A special situation arises as the knowl- there is no linear timeline. There is no particular starting or
edge about the VR video generates an information advan- end point, so the visitors will become protagonists who act
209
 Ina Neddermeyer

7–8  micha cárdenas: Becoming Dragon, 2008, VR-HMD, photographs, video, 3 min., still and exhibition view.

in the virtual and physical world simultaneously. Thus the Curatorial Potentials (and Challenges) of VR
boundaries of physical and simulated spaces blur. Technologies
VR technologies open up new forms of narration as well.
As viewers are addressed and involved directly, they can From a curatorial perspective, the aforementioned challeng-
become part of the narration. The Nest Collective, a Kenyan es underlie the pressing question of how VR works should be
art collective, created a science fiction film as a VR experi- installed in an exhibition. Should the technical infrastruc-
ence. The storyline follows a group of African people who ture be hidden according to the claims of invisibility or is it
have left earth to colonize a distant planet. Putting on the necessary to counteract at this point, to show the physical
VR headsets, visitors are instantly immersed in the installa- presence of technology? The latter choice enables viewers
tion as they arrive on the planet and have to ask themselves to adopt a distance to the artwork they are confronted with
if they would be welcomed in a black world (fig. 6). and hopefully to open up a critical point of view. Therefore,
we decided to make the whole technical infrastructure,
with all the cables, computers and displays, visible in the
exhibition. Next to VR-HMD that can be used as displays,
210 we show a headset that micha cardenás wore in her perfor-
I Want to See How You See

9  Beautiful new worlds, exhibition view.

mance (fig. 7–8). It is the only exhibited object that cannot be um, a purely virtual exhibition space was developed (pro-
used. It is placed in a showcase, turning into a museum piece. grammed), which can be entered through a virtual reality
Another focal aspect of the exhibition is showing the lounge in the physical museum. Once the visitors have put
long tradition of immersion in illusionary worlds that is on the VR-HMD, they can move through the simulated
by no means without its history. Panoramas, dioramas or exhibition rooms with the help of a controller, and zoom
stereo­scopy illustrate the centuries-old history of mankind’s into the artworks in the different virtual rooms. Such dis-
interest in immersive media. We have therefore decided that play formats are, of course, only possible in the case of VR
the starting point of the exhibition should be stereoscopic works, in which the installation in the surrounding physical
images, which accompany the history of the Zeppelin from exhibition room is irrelevant to the work. Artistic positions
1900 to the 1930s (fig. 9). in which the physical exhibition space is part of a specific
VR technologies open up new exhibition formats. In installation, e. g. with Halil Altindere, who installed his VR
a very radical understanding, the Kunstsammlung NRW work Journey to Mars in an oval room and with an airspace
established a completely new exhibition format for VR wallpaper, are excluded. In Journey to Mars the HMD as a
technology with the exhibition Unreal in 2017. In contrast display becomes a fundamental part of the artwork as visi-
to the exhibition Beautiful new worlds at the Zeppelin Muse- tors look like astronauts who are flying to Mars. Therefore, 211
 Ina Neddermeyer

10–11  Halil Altindere: Journey to Mars, 2016, 360 VR Video, 4K, 5.10 min., wallpaper, installation views.

the display is transferring the storyline of the VR work (the exhibitions and collection accessible for people far away.
artist ironically proposes that the refugees be relocated to That the visit to the physical museum itself will be replaced
mars, if no country will receive them: outer space as refuge completely seems rather doubtful nonetheless.
for the refugees) in the physical exhibition room (fig. 10–11). At the moment, it seems more sensible to open an exhi-
Thus works which operate at the interface between ana- bition for the visitor that focuses not only on the experience
logue and digital to query traditional perceptions and defi- effect, overwhelming and immersion. Rather, a parcours is
nitions of virtual and real worlds cannot be integrated into necessary, one that places the critical reflection in the fore-
these solely virtual exhibition formats. ground. To show the visitors the displays and technologies
Nevertheless, the exhibition Unreal has made it very themselves, as the VR-HMD must be consciously raised
clear that curators and artists have just begun to explore and lowered, is a central component of the exhibition. For
the potentials the new technology creates for the conception the exhibition Beautiful new worlds. Virtual realities in
of exhibitions with VR works. With its Arts & Culture App, contemporary art in the Zeppelin Museum a circuit was
Google is offering virtual tours through traditional exhibi- developed together with the exhibition architects Kooper-
tions of the MoMA in New York, the Naturkundemuseum ative für Darstellungspolitik that makes the entanglement
212 in Berlin and the Louvre in Paris to make museums, their of virtual and physical spaces perceptible for the audience
I Want to See How You See

12–13  Beautiful new worlds, exhibition views.

through bodily experience. One focal point is to open the many of these available applications do not offer multiuser
window shutters of the exhibition space, which are normally experiences, i. e. Tilt Brush gives the chance to paint in 3D
closed to create black boxes for the presentation of time- space with virtual reality but not as a multiplayer activity.
based media works. Thus it is possible to make a connection Critical voices already complain that the physical muse-
between the exhibition space and the world outside and to um abolishes itself, but they are only partly right. It might
question what is real, what is virtual and what is just an be true that in the future it might become unnecessary to
exhibition space. In the exhibition a system of yellow hand- physically visit exhibitions to view virtual art, but these
rails known from public transport is installed – a metaphor challenges are not uniquely linked to the development of
of orientation that counteracts the feeling of getting lost in VR technology. They have accompanied museums for many
virtual reality (fig. 12–13). years, when one looks at the discussions about technical
It is very likely that within the next years new exhibi- reproducibility and digitization. Nevertheless, physical
tion formats and new ways of art mediation will be devel- museums have not lost their fascination. The museums,
oped, even to the extent of purely virtual exhibition formats however, are now more than ever challenged and encour-
that allow for meeting other visitors in virtual environments aged to rethink themselves as social places that offer and
in order to exchange ideas or to interact. At the moment create discourse about art. 213
 Ina Neddermeyer

Figures
1, 9, 12–13  Beautiful new worlds. Virtual realities in contemporary art,
Zeppelin Museum Friedrichshafen, November 11, 2017 - April 8, 2018,
exhibition views. © Zeppelin Museum Friedrichshafen, photos: Markus
Tretter.
2  Banz & Bowinkel: Palo Alto, 2017. © Zeppelin Museum Friedrichshafen,
photo: Markus Tretter.
3  Harun Farocki: Parallel II, 2014. © Harun Farocki GbR.
4  Florian Meisenberg: Pre-Alpha Courtyard Games (raindrops on my cheek),
2017. © Zeppelin Museum Friedrichshafen, photo: Markus Tretter.
5  Salome Asega & Reese Donohue & Tongkwai Lulin: ASM(V)R, 2017.
© Zeppelin Museum Friedrichshafen, photo: Markus Tretter.
6  The Nest Collective: Let This Be A Warning – A VR Short Film, 2017.
© Electric South.
7  micha cárdenas: Becoming Dragon, 2008. © the artist, photo: Elle
Mehrmand.
8  micha cárdenas: Becoming Dragon, 2008. © Zeppelin Museum
Friedrichshafen, photo: Markus Tretter.
10–11  Halil Altindere: Journey to Mars, 2016. © Zeppelin Museum
Friedrichshafen, photo: Markus Tretter.

214
 ation
n- ba se d me dia , su ch as touch screens, navig
­Scr ee image, ac tion
st em s an d vir tu al re ali ty applications , merge
sy es
e pa rti cu lar co nf igu ra tion of the screen shap
and space. Th g what
wi ng ha bit s an d wo rk ing pr ocesses. By askin
user s’ vie e focuses
en s do ra th er th an wh at screens ar e, the volum
scre author s
th e ac tio na bil ity of sc reen-based media . The
on tices and
e situated in visual pr ac
analyze how screens ar ative
th eir dy na mi c, tra ns formational and perform
scrutinize space.
ac te ris tic s wi th re ga rd to image, ac tion and
char

www.degruyter.com
ISBN 978-3-11-046366-8