EDITORS: Andre Stork, [email protected].

de
Rahul C. Basole, [email protected]

DEPARTMENT: GRAPHICALLY SPEAKING

Why Visualize? Arguments for Visual

Support in Decision Making
€ t Konstanz, Konstanz, 78464, Germany
Dirk Streeb, Mennatallah El-Assady, and Daniel A. Keim, Universita
Min Chen, University of Oxford, Oxford, OX1 2JD, U.K.

Just like the numerous applications of visualization, there are plenty of

theoretical arguments for why visualization can aid in knowledge generation
and communication. Meanwhile, to date, these arguments have been presented
independently, which hinders the exploitation of underlying properties of
visualization in practice. We present a network of arguments that explicate
not only how different arguments build on each other, but also which
arguments demand for tradeoffs in the design of visualizations. Furthermore,
we call for the extension of our network to the application of visualization in
decision making, as well as visual analytics. We highlight the practical
applicability of our theoretical approach with examples, including one from the
text analysis.

D
ecision making is one of the most prominent in measuring the quality of visualizations and propos-
areas of application for visualizations (see the ing suitable views.
work by Padilla et al.,1 for a review), next to To give one example, let us consider the prominent
knowledge generation and communication. Mean- argument that viewers can identify visual patterns in
while, from a theoretic point of view, there is no com- visualizations, which resemble patterns in the data.
prehensive and convincing explanation for the This argument in favor of visualization is connected to
successes and failures of visualizations, yet.2 We several other arguments. In particular, we explicate
started to construct a network of more than 100 argu- that the argument on visual patterns builds on argu-
ments on the merits of visualization.3 ments from at least three groups, as depicted in
Our network of arguments3 provides a structural Figure 1. First, perception plays a role not only by driv-
foundation for the interdisciplinary research on visual- ing attention toward some visual structures, but also
ization. It combines different argumentative stand- by performing visual inferences. Second, the visual
pattern can only be interpreted in case it provides pre-
points and relates the properties of visualization to
cise and detailed access to the underlying data.
the visualization process, as well as to their dependen-
Finally, viewers learn how to interpret visual patterns
cies on tasks, users, and data. It also includes some
by interacting with the visualization and/or similar
limitations of visualization and highlights needs for
visualizations.
tradeoffs between opposing properties, for instance,
Explicating the relationships between the argu-
between the flexible usage of visualizations and their
ments, and, thus, the properties of visualization that
specific tailoring to particular tasks. The plethora of
the arguments cover, provides a better theoretical
properties and task demands describes the difficulties
understanding of the benefits and limitations of visual-
ization. As a next step, it is crucial to take our findings
from the abstract why visualization works to the more
practical how does visualization work, and conse-
0272-1716 ß 2021 IEEE
quently how to make visualizations work. The transfer
Digital Object Identifier 10.1109/MCG.2021.3055971 of theoretical insights will benefit the use of visualiza-
Date of current version 15 March 2021. tion in decision making and visual analytics.

March/April 2021 Published by the IEEE Computer Society IEEE Computer Graphics and Applications 17
GRAPHICALLY SPEAKING

FIGURE 1. Small section of our network of arguments show-

ing the foundations of potential benefits of visual patterns.
We identified three groups of arguments that support the
commonly expressed argument on visual patterns. Adapted
from Fig. 5 in the work by Streeb et al.3

FIGURE 2. Decision-making strategies differ in the propor-

DECISION-MAKING STRATEGIES tions to which the three basic components are involved.
Decision making always takes place in practical envi- Human-centered machine learning relaxes the data fixation
ronments,4 which typically contrast the assumptions of machine learning.
made by many mathematical models.5 Thus, any deci-
sion-making process starts from the real world and
ends by acting within the same real world, including all learning focuses on measured datasets, in contrast
the costs and consequences. It does not merely con- to traditional statistics that defines formal relation-
nect data to the outputs of an algorithm (e.g., class ships between attributes in more detail based on
labels in classification). For instance, decision making domain-specific theories. Which (combinations of)
involves different kinds of costs for measuring attri- strategies lead to appropriate and intelligent deci-
bute values (e.g., time, money, and side-effects) and sions depends on the application environment. For
depending on how (much) decisions are off the mark example, physicists achieved great successes in
(e.g., false positives versus false negatives). applying formal theories such as Maxwell’s equa-
In Figure 2, we roughly contrast a number of deci- tions of electromagnetism or Einstein’s theory of
sion-making strategies. Strategies differ with respect relativity.
to the proportions of three basic components. First, The current trend to automatize machine learning
measurement can be used to quantify attributes and processes explores the bottom (right) of the triangle
compile datasets, which may, for example, be used to and promises to relax demands on the other two com-
train machine learning models. Second, formal models ponents. On the one hand, it aims at compensating a
of the application domain can be used to describe the potential lack of domain expertise and avoiding costs
relationships between attributes mathematically, introduced by incorporating the mental component
such as by laws of physics. These two components and, thus, humans. On the other hand, learning rela-
can be seen as different kinds of data. By contrast, the tionships from measurements instead of formalizing
third component covers the not quantified aspects. them reduces the demand for domain-specific theo-
This mental component includes implicit knowledge ries, as well as the ability to express these theories in
about the task and domain expertise that, for mathematical and statistical terms.
instance, lead to the detection and interpretation of However, the focus on measured data increases
visual patterns. Here, we do not want go into more the need for high-quality datasets. Although the struc-
details than having one component, which covers ture of the decision environment, structural breaks,
everything that is neither formal nor measured. and costly or time-consuming measurement proce-
Importantly, practical decision-making strategies dures can prohibit the timely access to sufficient data-
combine components. For instance, intuition com- sets (see also the work by Taleb.5) High-quality
bines cognitive processes with perceived observa- measurements typically also require domain expertise
tions and, thereby, balances the limitations of the for coming up with reliable measurement procedures
mental and measurement components.4,5 Machine in the first place.

18 IEEE Computer Graphics and Applications March/April 2021

 GRAPHICALLY SPEAKING

Further uncertainty is introduced by the quantiza-

tion/formalization of tasks, as it often does not faith-
fully cover all constraints. Usually a single (formal)
objective function needs to be known a priori, which is
the exact opposite of the data-driven appeal of
machine learning, and demands for the formalization
of domain constraints.
Needless to say, machine learning is applied many
times successfully. But even then, accessibility and
auditability, which may be required by contractors or
law, suffer from the weak incorporation of the mental
and formal components. For example, complex
machine learning models may facilitate the training
process, but hide the effective structures in large
networks.6

FIGURE 3. Visualization can bring the distinct components

VISUAL ANALYTICS
together. Task constraints often cannot be exactly formalized
Visual analytics promotes to include humans and the
mental component explicitly.7 Visualization provides (e.g., the price constraint 9 $1m ¼)  $1m) and measure-
the powerful interfaces to models (e.g., from machine ments are rarely exact (e.g., negotiable house prices).
learning), datasets, and the general analysis process Demands on visualizations diverge depending on the propor-
(e.g., via provenance). With our theoretical work on tions of basic components involved.
visualization,3 we lay a foundation for the field and
guide future research. For example, we integrate theo-
retic work from the visualization community with argu- learning (e.g., in the form of a decision tree) can help in
ments from psychology and other disciplines. Any finding a suitable house, once the potential buyer pro-
human-centric approach needs to consider not only vided labels for some offers. Depicting (III) the trained
the interplay between visualizations and analysts, but model can enable the potential buyer to grasp how
also the work that analysts contribute,8 such as exter- well it aligns with her expectations.
nalizing task constraints and embedding the analysis On their own, the three components can be valu-
within the application environment. Visualization can able, but only their combination fully exploits the
aid analysts, for instance, by providing the external power of visualization. Visual interactive machine
memory and a shared structure for integrating the learning (IV) makes models more accessible by
knowledge of experts across involved domains. showing their formal and measurement components
One particular strength of visualization is the together. Visual analytics tries to exploit the ana-
potential to combine the three distinct components lyst’s domain expertise to improve models and
involved in decision making, as given in Figure 3. counter limitations of data quality. For instance,
Within a single visualization, the diverse entities from (II) joining formal and mental components aids in
the three components (e.g., formal functions, data explicating and refining task constraints, and
points, and mental task constraints) are mapped to a (VI) labeling and annotation enrich datasets by
more homogeneous set of visual entities (e.g., lines, externalizing domain knowledge.
circles, and polygons). These visual entities often can We envision that the combination of all three com-
be related more easily, if relevant aspects are faithfully ponents can be fostered. From the perspective of
mapped to visual features. visual analytics, there is a need to provide closer inter-
Let us consider the simple example of buying a actions with machine learning models. In complement,
house. Clearly, (I, in Figure 3) the potential buyer has a machine learning may benefit from the more concise
quite good mental picture of the task and the con- alignment of intended tasks and formalized optimiza-
straints on price, size, and whether the house should tion criteria through visual means for evaluating mod-
have a garden or a fireplace. For example, offers likely els. This also demands for the exploration of the limits
include some ridiculously expensive houses that are of visualization and resulting needs for tradeoffs in
not feasible with the given budget. Visualizing practical human–machine environments, which goes
(V) available offers can provide an overview. Machine beyond our investigations to date.

March/April 2021 IEEE Computer Graphics and Applications 19

GRAPHICALLY SPEAKING

FIGURE 4. Interactive topic model optimization using visual analytics. This approach9 is based on IHTM to refine a topic tree,
employing speculative execution of different optimization strategies (based on multiobjective optimization of quality metrics).

APPLICATION OF VISUAL given a document selection and asked to compare the

ANALYTICS suitability of two topic modeling results. Using a rein-
While there already are numerous application exam- forcement learning loop, the system progressively
ples in the field of visual analytics, and more recently adjusts the topic modeling parameters to mirror
visual interactive machine learning,10 only lately Sacha human expectations.
et al. contributed an ontology for sharing a general Finally, the third approach9 emphasizes the
workflow and terminology.11 Making the iterative pro- training of the model by combining the formal and
cess human centric enables the interactive externali- the measurement components. Here, analysts can
zation and evaluation of constraints that are, at the follow along while an incremental topic model is
start, only available mentally, and often cannot be being constructed and optimized. This approach
externalized faithfully in all their subtleties. Consider integrates the human using different dynamics. In
the following example from the text analysis. the following, we illustrate this last technique and
Topic modeling algorithms have become a promi- how it implements our first attempt to integrate all
nent machine learning approach for automatically three components.
subdividing document corpora into thematically As depicted in Figure 4(a), analysts start the tai-
related groups of documents, labeled by keywords lored refinement process by externalizing their domain
(i.e., topic descriptors). While these models are widely understanding in a topic backbone (i.e., a priori for
applicable, they are notoriously difficult to optimize priming the topic modeling algorithm). Different tasks
and adapt to analysts’ expectations. To tackle this demand for distinct backbones to faithfully reflect the
challenge for different user groups, the lingvis.io diverse set of understandings of domain experts.
framework12 offers three visual analytics techniques Based on this given structure, the incremental hierar-
for refining topic models. Each of the three techniques chical topic modeling (IHTM) starts constructing a
prioritizes different components of our model. topic tree that is updated on the visual interface for
For instance, focusing on the mental component every measured document. Our example in Figure 4(b)
analysts externalize their domain knowledge using shows a portion of a topic tree from a presidential
Semantic Concept Spaces.13 Here, they define high- debate between Romney and Obama. Since, for analyz-
level relations between concepts to adapt the underly- ing such data, the semantic understanding of concept
ing language model used for measuring word relations relationships is central, we depict the incremental
that form the basis of the subsequent topic modeling. development of the topic tree. Thus, analysts are able
The second approach14 focuses on the formal compo- to interrupt the processing to include their domain
nent. Based on layered topic matching,15 analysts are knowledge at any time.

20 IEEE Computer Graphics and Applications March/April 2021

 GRAPHICALLY SPEAKING

Besides the refinement, the model itself utilizes a inappropriate in others. We see a need to investigate
multiobjective optimization to monitor its internal the big picture of visualization within the decision-
quality. These objectives are various quality metrics making environments it is applied in. One of the
that each captures one aspect of an optimal topic strengths of visualization is its encompassing nature
model, such as thematic coherence within a topic and regarding the combination of the different compo-
separation across topics, as shown in Figure 4(c). nents as depicted in Figure 3. While we see this
Using different trigger strategies, the visual analytics potential, we are aware of many open questions on
system halts the topic modeling process when the how to combine the distinct components, for
quality of the model deteriorates and activates optimi- instance, in the form of sketching, mathematical dia-
zations, such as splitting large topics or merging grams, and data visualization. At the same time, the
related ones. application of visualization may boost creativity not
As potentially multiple optimizations are beneficial only with regard to the (machine learning) models
(depending on the analyst’s mental model) and the being inspected, but also with respect to how to
optimization criteria are task-dependent (not known a improve the visual interfaces and visual analytics sys-
priori), several approaches are performed in parallel tems themselves. Our network of arguments3 con-
using Speculative Execution (SpecEx).16 These specu- nects the beneficial properties of visualization, such
lations create sandboxes to evaluate the impact of a that designers can navigate and integrate the theo-
selected optimization on the model (projected to the retical aspects to consider more effectively.
future). Using such steerable preview mechanisms, Machine learning is a technological strategy for deci-
the system offers options, which enable choices and sion making that needs to be investigated in application
allow for making actual decisions and tradeoffs. environments and not only as a set of purely mathemati-
Hence, this topic model optimization approach uses cal or computational problems. Combining the fields of
visual analytics to connect the formal (i.e., the hierar- visualization/visual analytics and machine learning in
chical topic model), the measurement (i.e., the docu- practical contexts as visual human-centered machine
ments and their individual effects on the topic tree), learning is a major challenge for research. Extending our
and the mental (i.e., the prior backbone and the steer- network on visualization toward visual analytics and
ing of the incremental process) components. visual interactive machine learning will contribute to
Making decisions and involved tradeoffs demands underpin and advance the design of visual interfaces.
for knowing alternatives as, for instance, generated by While machine learning facilitates the exploitation of
Speculative Execution. Machine learning should be large and high-dimensional datasets, it promotes con-
used to come up with these options in the form of dif- servative one-dimensional decision making based on
ferent models, instead of prescribing one model as past data and single optimization criteria postulated a
optimal based on a single criterion that has been priori. Visual analytics and visual human-centered
defined a priori. Visualization and visual analytics can machine learning enable decision makers to make pro-
aid in the multiobjective optimizations that are rele- active,as well asadaptivelybalanced decisions based on
vant to decision making, for example, by facilitating multiple objectives evolving over time and competitive
the evaluation and comparison of models. Figure 4(c) modeling options provided by a number of machine
shows several quality measures and their changes learningmodels.
throughout the modeling process. In combination
with difference views for comparing topic model
trees,9 the system integrates relevant information ACKNOWLEDGMENTS
visually and enables direct interaction. As a result, our €fer. This
The authors would like to thank Hanna Scha
visual analytics system, while highlighting the training work was supported by the Deutsche Forschungsge-
process and, thus, the formal and measurement com- meinschaft (DFG, German Research Foundation)
ponents, also promotes the externalization of domain within the Project VALIDA/SPP-1999 RATIO (Project-
knowledge. This human-centric exchange of informa- ID 376714276).
tion between the machine and the human incorpo-
rates the mental component (cf. Tam et al.8).
REFERENCES
1. L. M. Padilla, S. H. Creem-Regehr, M. Hegarty, and
CONCLUSIONS J. K. Stefanucci, “Decision making with visualizations: A
The decision-making strategies in Figure 2 are cognitive framework across disciplines,” Cogn. Res.:
options that are appropriate in some situations and Princ. Implications, vol. 3, 2018, Art. no. 29.

March/April 2021 IEEE Computer Graphics and Applications 21

GRAPHICALLY SPEAKING

2. M. Chen, G. G. Grinstein, C. R. Johnson, J. Kennedy, and 14. M. El-Assady, R. Sevastjanova, F. Sperrle, D. Keim, and
M. Tory, “Pathways for theoretical advances in C. Collins, “Progressive learning of topic modeling
visualization,” IEEE Comput. Graph. Appl., vol. 37, no. 4, parameters: A visual analytics framework,” IEEE Trans.
pp. 103–112, Jul./Aug. 2017. Vis. Comput. Graph., vol. 24, no. 1, pp. 382–391,
3. D. Streeb, M. El-Assady, D. A. Keim, and M. Chen, Jan. 2018.
“Why visualize? Untangling a large network of 15. M. El-Assady, F. Sperrle, R. Sevastjanova, M. Sedlmair,
arguments,” IEEE Trans. Vis. Comput. Graph., vol. 27, and D. Keim, “LTMA: Layered topic matching for the
no. 3, pp. 2220–2236, Mar. 2021. comparative exploration, evaluation, and refinement
4. G. Gigerenzer, P. M. Todd, and ABC Research Group, of topic modeling results,” in Proc. Int. Symp. Big
Eds., Ecological Rationality: Intelligence in the World. Data Vis. Immersive Analytics, Oct. 2018, pp. 1–10.
London, U.K.: Oxford Univ. Press, 1999. 16. F. Sperrle, J. Bernard, M. Sedlmair, D. Keim, and
5. N. N. Taleb, Statistical Consequences of Fat Tails: Real M. El-Assady, “Speculative execution for guided visual
World Preasymptotics, Epistemology, and Applications. analytics,” Workshop Mach. Learn. User Interaction
RESEARCHERS.ONE, 2020. [Online]. Available: https:// Vis. Anal. IEEE Vis., 2018. [Online]. Available: http://nbn-
www.researchers.one/article/2020-01-21 resolving.de/urn:nbn:de:bsz:352-2-1vfpbv3lvivg2
6. J. Frankle and M. Carbin, “The lottery ticket hypothesis:
Finding sparse, trainable neural networks,” in Proc. Int.
DIRK STREEB is currently a member of the Data Analysis
Conf. Learn. Representations, 2019. [Online]. Available:
€t Konstanz, Konstanz,
and Visualization Group, Universita
https://openreview.net/forum?id=rJl-b3RcF7
7. D. A. Keim, J. Kohlhammer, G. P. Ellis, and F. Mansmann, Germany. His research interests include theoretical aspects
Mastering the Information Age Solving Problems With of information visualization and visual analytics as aids in
Visual Analytics. Goslar, Germany: Eurographics Assoc., human decision-making processes. He received the M.Sc.
2010. degree in social and economic data analysis in 2016 from Uni-
8. G. K. L. Tam, V. Kothari, and M. Chen, “An analysis of €t Konstanz, where he is currently working toward the
versita
machine- and human-analytics in classification,” IEEE
Ph.D. degree with the Graduate School of Decision Sciences.
Trans. Vis. Comput. Graph., vol. 23, no. 1, pp. 71–80,
He is the corresponding author of this article. Contact him at
Jan. 2017.
9. M. El-Assady, F. Sperrle, O. Deussen, D. Keim, and [email protected].
C. Collins, “Visual analytics for topic model optimization
based on user-steerable speculative execution,” IEEE MENNATALLAH EL-ASSADY is currently with the Data Analysis
Trans. Vis. Comput. Graph., vol. 25, no. 1, pp. 374–384,
€t Konstanz, Konstanz, Germany.
and Visualization Group, Universita
Jan. 2019.
Contact her at [email protected].
10. F. M. Hohman, M. Kahng, R. Pienta, and D. H. Chau,
“Visual analytics in deep learning: An interrogative
survey for the next frontiers,” IEEE Trans. Vis. Comput. DANIEL A. KEIM is currently a Professor and the Head of the
Graph., vol. 25, no. 8, pp. 2674–2693, Aug. 2019. €t Konstanz,
Data Analysis and Visualization Group, Universita
11. D. Sacha, M. Kraus, D. A. Keim, and M. Chen, “VIS4ML: Konstanz, Germany. Contact him at [email protected].
An ontology for visual analytics assisted machine
learning,” IEEE Trans. Vis. Comput. Graph., vol. 25, no. 1,
MIN CHEN is currently a Professor of Scientific Visualization
pp. 385–395, Jan. 2019.
12. M. El-Assady et al.“lingvis.io—A linguistic visual analytics with Oxford University, Oxford, U.K., and a Fellow of Pem-
framework,” in Proc. Annu. Meeting Assoc. Comput. broke College, Cambridge, U.K. He is a Member of IEEE. Con-
Linguistics, Syst. Demonstrations, 2019, pp. 13–18. tact him at [email protected].
13. M. El-Assady, R. Kehlbeck, C. Collins, D. Keim, and
O. Deussen, “Semantic concept spaces: Guided topic Contact department editor Andre Stork at andre.stork@igd.
model refinement using word-embedding projections,” fraunhofer.de and department editor Rahul C. Basole at
IEEE Trans. Vis. Comput. Graph., vol. 26, no. 1, [email protected].
pp. 1001–1011, Jan. 2020.

22 IEEE Computer Graphics and Applications March/April 2021