Chapter 5

Analysis: Four Level for Validation

Vis/Visual Analytics, Chap 5 Validation 1 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Contents
• Why Validate? • Validation Approaches
• Why is Validation – Domain
difficult? – Abstraction
• Four Levels of Design – Idiom
– Algorithm
– Domain Situation
– Mismatches
– Task & Data Abstraction
– Visual Encoding & • Examples
Interaction – Genealogical Graphs
– Algorithm – MatrixExplorer
• Angles of Attack – … and 4 more
• Threats to Validity
– Different for what, why,
how

Vis/Visual Analytics, Chap 5 Validation 2 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Why Validate?
• Why?
– The vis design space is huge, and most
designs are ineffective

– Think about how you might validate your

choices from the very beginning of the
design space, rather than leaving it at the
end
Vis/Visual Analytics, Chap 5 Validation 3 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design
• Why four nested
levels?
– Splitting the
complex vis design
into four cascading
levels provides an
analysis framework
that lets you to
address different
concerns separately

Vis/Visual Analytics, Chap 5 Validation 4 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design
• Four nested levels
– Consider the details
of a particular
application domain
– The what-why
abstraction
• Map domain-
specification
problems and data
into forms that are
independent of the
domain

Vis/Visual Analytics, Chap 5 Validation 5 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design
• Four nested levels
– How level: design of
idioms that specify
the visual encoding
and interaction
– Design of algorithms
to instantiate those
idioms
computationally

Vis/Visual Analytics, Chap 5 Validation 6 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design
• Four nested levels
– The output from an
upstream level is
input to the
downstream below
– Choosing a poor
choice at an
upstream level
inevitably cascades
to all downstream
levels

Vis/Visual Analytics, Chap 5 Validation 7 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design
• Domain situation
– A domain situation
includes
• a group of target users
– e.g. computational biologists
• domain of interest
– comparative genomics
• Questions
– e.g. genetic source of
adaptivity in a species
• Data
– e.g. genomic sequence data

– Methods to identify
domain
• Interviews
• observations
• careful research

User introspection is
insufficient!
Vis/Visual Analytics, Chap 5 Validation 8 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design
• Domain situation
– Output
• A detailed set of
questions asked or
actions carried out by
the target uses, about
a possibly
heterogeneous
collection of data

Two of the questions asked by the computational biologist working

on the comparative genomics:
1. What’s the difference between individual nucleotides (核甘酸) of feature pairs
2. Where are the gaps across a chromosome (染色體)?
Other:
What is the genetic basis of a disease? (X, not specific enough)
Vis/Visual Analytics, Chap 5 Validation 9 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design
• Data/Task Abstraction
– Domain-specific into
generic representation
• Identify abstract tasks
– Ex. Browsing, comparing,
summarizing
• Design abstract data
forms
– Are designed, a creative
design step
» Often choose to
transform the original
data to something
quite different

Vis/Visual Analytics, Chap 5 Validation 10 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design
• Data/Task Abstraction
– Explicitly considering the
choices made in
abstracting tasks and
data can be very useful in
design process.
– Bad alternative: to do
this implicitly and w/o
justification
» Solving the “lost in
hyperspace” problem
done by showing the
searcher a website
hyperlink connectivity
graph?
» Wrong, too much
cognitive load.
Vis/Visual Analytics, Chap 5 Validation 11 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design
• Visual encoding and
interaction idiom
– Decide on the specific
way to create and
manipulate the visual
rep. of the abstract
data block, guided by
the abstract tasks
– Idiom: each distinct
possible approach
• Visual encoding
– How to represent data
visually (what users see)
Vis/Visual Analytics, Chap 5 Validation 12 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design
– Idiom
• Visual encoding
– How to represent data
visually (what users see)
• Interaction
– How to manipulate that
representation
dynamically (how users
change what they see)
• Possible to analyze
encoding and interaction
as separate decision. In
some cases, need to be
considered as a combined
idiom
Vis/Visual Analytics, Chap 5 Validation 13 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design
– Idiom blocks are
designed
• A big design space…
• Abstracting data and task
can be used to rule out
many bad options
• Should make decisions
about good and bad
matches based on
understanding human
abilities, especially
– Visual perception
– memory

Vis/Visual Analytics, Chap 5 Validation 14 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design

Word Tree combines the visual encoding idiom of a hierarchical tree of

keywords laid out horizontally and the interaction idiom of navigation
based on keyword selection.

Vis/Visual Analytics, Chap 5 Validation 15 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Nested Levels of Design
• Algorithm
– A detailed procedure
that allows a computer
to automatically carry
out the desired goal
• To efficiently handle visual
encoding and interaction
– Are designed
• Could have many
algorithms for the same
idiom. Ex. Many algorithms
for direct volume rendering Visual/interaction:
– Primary concerns: primary concerns are human
computational issues perceptual issues
Vis/Visual Analytics, Chap 5 Validation 16 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Four Levels of Design
• Dependency:
– Wrong block upstream
cascades downstream
choices
• poor task, perfect
idiom => X
• Iterative process:
– Consider each level
separately
– A better
understanding of one
block will refine other
levels

Vis/Visual Analytics, Chap 5 Validation 17 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Angles of Attack for Vis Design
• Two angles of attack for vis design
– Top down (problem-driven) or bottom up
(technique driven)

Vis/Visual Analytics, Chap 5 Validation 18 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Angles of Attack for Vis Design
• Problem-driven (Top-down)
– Start with the problems of real-world user and
attempt to design solution that that helps them
work more effectively
– Often the problem can be solved using existing
visual coding and interaction idioms
• Much of the challenge lies at the abstraction level
– Sometimes the problem motivates the design of
new idioms, if no existing ones will adequately
solve the abstracted design problem

Vis/Visual Analytics, Chap 5 Validation 19 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Angles of Attack for Vis Design
• Problem-driven (Top-down)
– Considering the four levels of nested model
explicitly can help you avoid the pitfall of
skipping important steps in problem-driven
approach
• Some designers skip over the domain situation level
completely, short-circuit the abstraction level by
assuming that the first abstraction is right and jump
immediately into the third level
• THE ABSTRACTION LEVEL IS OFTEN THE HARDEST TO
GET RIGHT!!
• The design process for problem-driven work involves
iterative refinement at all levels.

Vis/Visual Analytics, Chap 5 Validation 20 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Angles of Attack for Vis Design
• Technique driven (Bottom-Up)
– Start with idiom or algorithm design
• Goal is to invent new idioms that better support
existing abstractions, or new algorithms that better
support existing idioms
– Considering the four nested model can help you
articulate your assumptions at the level just
above your focus
• Articulate the abstraction requirement for new idiom,
or articulate the idiom requirement for new algorithm

Vis/Visual Analytics, Chap 5 Validation 21 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Treats to Validity
• Validating the effectiveness of a vis design
is difficult because there are so many
possible questions
– Considering the validity questions at each level
separately helps
– Each level has a different set of treats to validity
• Different fundamental reasons why you might have
made wrong choices

Vis/Visual Analytics, Chap 5 Validation 22 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Threats to Validity

Vis/Visual Analytics, Chap 5 Validation 23 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Approaches
• Different treats require very different
approaches
• Difference between immediate and
downstream validation
– Most kinds of validation for the outer levels are
not immediate because they require results
from the level nested within them
– A poor showing of a test may misdirect
attention upstream, when in fact the problem
results from a poor choice at the current level
– Downstream validation is necessary!!
– The immediate validation only offer partial
evidence of success
Vis/Visual Analytics, Chap 5 Validation 24 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Approaches

Vis/Visual Analytics, Chap 5 Validation 25 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Approaches
• Domain Validation
– The primary treat is that the problem is
mischaracterized
• The target users do not in fact have these problems
– An immediate validation is to interview and
observe the target users to verify the
characterization
• Field study
– Observe how people act in real-world scenarios
– Ask questions when clarification is needed
– Interview people about their needs
– Downstream validation: Report adoption rates
– What the target users do of their own accord

Vis/Visual Analytics, Chap 5 Validation 26 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Approaches
• Abstraction Validation (downstream)
– The treat: the identified task abstraction and
the designed data abstraction do not solve the
characterized problems
– Key validation: the system must be tested by
target users doing their own work, rather than
doing an abstracted task specified by the
designer
• Collect anecdotal evidence that the tool is in fact useful
– insights found or hypothesis confirmed
– A more rigorous validation: to conduct a field
study to observe and document how the target
user uses the deployed system
Vis/Visual Analytics, Chap 5 Validation 27 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Approaches
• Idiom Validation
– Treat: the chosen idioms are not effective at
communicating the desired abstraction to the
users
– One immediate validation:
carefully justify the idiom w.r.t. known
perceptual and cognitive principles
• Heuristic evaluation
• Expert review

Vis/Visual Analytics, Chap 5 Validation 28 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Approaches
• Idiom Validation
– A downstream validation
• Lab study
– A controlled experiment in a lab setting
– Tease out the impact of specific idiom choice by measuring
human performance on abstract tasks
– Quantitative measurement
» Time spent, errors made
» Performance
» Logging actions such as mouse moves and clicks
» Tracking eye movements
– Qualitative measurement
» Reflect about the strategies through questionnaires
» The number of participants needs to be sufficient for
statistical significance
Vis/Visual Analytics, Chap 5 Validation 29 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Approaches
• Idiom Validation
– Another downstream validation
• Presentation of and qualitative discussion of results in
the form of still images or video
• Sometimes occur as usage scenarios, supporting that
the tool is useful for a particular task-data abstraction
– Third downstream validation
• The quantitative measures of result images by using
quality metrics
– e.g. Measure # of edge crossings and edge bends for
graphs
– Informal usability study
• Lead to better and more usable systems, but neither
offer validation nor provide evidence of the superiority
of an approach for a particular context
Vis/Visual Analytics, Chap 5 Validation 30 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Approaches
• Algorithm Validation
– Threat: the algorithm is suboptimal
• Either to a theoretical minimum or in comparison with
previous methods
• Is the algorithm correct?
Does it have good performance?
– Immediate validation
• Analyze complexity and memory
– Downstream validation
• Measure wall-clock time & memory usage
• Primary consideration: scalability, how data size affect
the speed
• One trickier question: what data used?
– Use standard benchmarks
Vis/Visual Analytics, Chap 5 Validation 31 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Approaches
• Algorithm Validation
– Another threat: incorrectness at the algorithm
• Poor algorithm design, or
• The implementation

Vis/Visual Analytics, Chap 5 Validation 32 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Approaches
• Mismatches
– Mismatch between the level at which the
benefit is claimed and the validation
methodology
– Example
• The benefit of a visual encoding cannot be validated by
wall-clock timings of the algorithm In practice, not all
validation tests are adopted
– The nested model explicitly separates the
design problem into levels in order to guide
validation according to the unique threats at
each level

Vis/Visual Analytics, Chap 5 Validation 33 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples
• Genealogical (宗譜) Graphs
– Proposed
• Multiple new visual encoding
– Classical dual-tree, classicial left-to-right, new indented
graph
• Interaction idioms
– Automatic camera framing, animated transitions, and a
new widget for ballistically dragging out subtrees to
arbitrary depth
– Explicitly cover all four levels
• Domain situation
– Domain is genealogy, discuss the need and current tools

Vis/Visual Analytics, Chap 5 Validation 34 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples
• Abstraction level
– Point out the term fam ily tree is highly misleading because
the data type is a more general graph with specialized
constraints on its structure
– Discuss conditions for which the data type is a tree, a
multi-tree, or a directed acyclic graph
– Map the domain problem of recognizing nuclear family
structure into an abstract task of determining subgraph
structure
• Visual encoding and interactions
– Discuss the strengths and weakness of several visual
encoding design choices
» Connection, containment, adjacency and alignment,
indentation
– Address interaction idiom design
• Algorithm design
– Dual-tree layout

Vis/Visual Analytics, Chap 5 Validation 35 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples
Genealogical Graphs

Genealogical graphs. (a) Three layouts for the dual-tree:

classical node–link top-to-bottom at the top, classical left-to-right on the left,
and the new indented outline algorithm on the right.
(b) Widget for subtree collapsing and expanding with ballistic drags.
Vis/Visual Analytics, Chap 5 Validation 36 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples
– Validation
• Abstract level
– Downstream informal testing of a system prototype with a
target user to collect anecdotal evidence
• Visual encoding and interaction design
– Immediate justification of Established principles
– Downstream method of a qualitative discussion of result
images and video

Vis/Visual Analytics, Chap 5 Validation 37 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples
• MatrixExplorer for social network analysis
– Proposed
• Matrix representation to minimize clutter for large and
dense graph
• More intuitive node-link graph for smaller network
– Explicitly cover all four levels
• Domain situation
– Explicit characterization of social network analysis domain
– Validated with qualitative interviews and exploratory
study with social scientists
• Abstraction level
– Include a detail list of the requirements in abstract form

Vis/Visual Analytics, Chap 5 Validation 38 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples
• MatrixExplorer for social network analysis
• Visual encoding and interaction
– A thorough discussion of the primary encoding idiom
– Discussion of basic and more complex interaction
» Interactive reordering and clustering
– Validation
» Use the immediate validation
» An extensive downstream validation using qualitative
discussion of result images
• Algorithm design
– Focus on reordering algorithm
– Validation
» Downstream benchmark timing

Vis/Visual Analytics, Chap 5 Validation 39 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples
(MatrixExplorer)

MatrixExplorer features both node–link and matrix representations in an

interface designed for sociologists and historians to explore social
Networks (Munzner 84)

Vis/Visual Analytics, Chap 5 Validation 40 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples
(MatrixExplorer)

MatrixExplorer validation methods

Vis/Visual Analytics, Chap 5 Validation 41 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples (Flow Maps)

Flow maps showing migration patterns from 1995 to 2000 US Census

data. (a) Migration from California. (b) The top ten states that sent
migrants to California shown in green, and to New York in blue
(Munzner 86)

Vis/Visual Analytics, Chap 5 Validation 42 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples (Flow Maps)

Vis/Visual Analytics, Chap 5 Validation 43 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples (LiveRAC)

LiveRAC supports exploration of system management time-series data with

a reorderable matrix and semantic zooming. (a) The first several dozen rows
have been stretched out to show sparklines for the devices. (b) The top three
rows have been enlarged more, so the charts appear in full detail.
(Munzner 87)
Vis/Visual Analytics, Chap 5 Validation 44 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples (LiveRAC)

Vis/Visual Analytics, Chap 5 Validation 45 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples (LinLog)

Vis/Visual Analytics, Chap 5 Validation 46 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
 Validation Examples (Sizing the
Horizon)

Vis/Visual Analytics, Chap 5 Validation 47 CGGM Lab., CS Dept., NCTU Jung Hong Chuang