AAAS

Health Data Science

Volume 2022, Article ID 9840519, 19 pages
https://doi.org/10.34133/2022/9840519

Review Article
A Review of Three-Dimensional Medical Image Visualization

Liang Zhou,1 Mengjie Fan,1 Charles Hansen,2 Chris R. Johnson,2 and Daniel Weiskopf3
1
National Institute of Health Data Science, Peking University, Beijing, China
2
Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, USA
3
Visualization Research Center (VISUS), University of Stuttgart, Stuttgart, Germany

Correspondence should be addressed to Liang Zhou; [email protected]

Received 25 December 2021; Accepted 17 March 2022; Published 5 April 2022

Downloaded from https://spj.science.org on September 26, 2024

Copyright © 2022 Liang Zhou et al. Exclusive Licensee Peking University Health Science Center. Distributed under a Creative
Commons Attribution License (CC BY 4.0).

Importance. Medical images are essential for modern medicine and an important research subject in visualization. However,
medical experts are often not aware of the many advanced three-dimensional (3D) medical image visualization techniques that
could increase their capabilities in data analysis and assist the decision-making process for specific medical problems. Our
paper provides a review of 3D visualization techniques for medical images, intending to bridge the gap between medical
experts and visualization researchers. Highlights. Fundamental visualization techniques are revisited for various medical
imaging modalities, from computational tomography to diffusion tensor imaging, featuring techniques that enhance spatial
perception, which is critical for medical practices. The state-of-the-art of medical visualization is reviewed based on a
procedure-oriented classification of medical problems for studies of individuals and populations. This paper summarizes free
software tools for different modalities of medical images designed for various purposes, including visualization, analysis, and
segmentation, and it provides respective Internet links. Conclusions. Visualization techniques are a useful tool for medical
experts to tackle specific medical problems in their daily work. Our review provides a quick reference to such techniques given
the medical problem and modalities of associated medical images. We summarize fundamental techniques and readily available
visualization tools to help medical experts to better understand and utilize medical imaging data. This paper could contribute
to the joint effort of the medical and visualization communities to advance precision medicine.

1. Introduction of Health (NIH), USA, released the NIH Strategic Plan for
Data Science, which proposed that technological innova-
In recent years, with advances in computing power and tions such as machine learning, deep learning, AI, and vir-
algorithms, the impact of technology in medicine is greater tual reality (VR) could revolutionize biomedical research
than ever and keeps increasing. Strategic plans focusing on over the next 10 years (https://datascience.nih.gov/nih-
promoting the application and development of medical strategic-plan-data-science).
technologies have been made worldwide. In 2018, the Medical images, such as computerized tomography
General Office of the State Council of China issued “Opin- (CT), magnetic resonance imaging (MRI), and diffusion ten-
ions on Promoting the Development of ‘Internet Plus sor imaging (DTI), are the backbone of modern medical
Healthcare,’” calling for strengthening the integration, practices and research. Furthermore, medical images are a
sharing, and application of clinical and research data and core data source and target for analysis in AI and VR in
supporting the research and development of health- aforementioned strategic plans. Therefore, the analysis and
related artificial intelligence (AI) technology, medical understanding of medical images are of utmost importance
robots, etc. (http://www.gov.cn/zhengce/content/2018-04/ in medical technologies. In practice, most if not everyone
28/content_5286645.htm). A three-year action plan was in health science areas is familiar with medical images, and
released to develop AI, which puts a priority on expanding many use medical images in daily work. Medical images
the clinical application such as medical image-assisted are massive and complex and hard to explore and gain
diagnosis systems (http://www.cac.gov.cn/2017-12/15/c_ insights with traditional statistical methods that do not
1122114520.htm). In the same year, the National Institutes involve the expertise of humans. Therefore, technologies
2 Health Data Science

that have humans in the loop are needed for medical image (Section 3) and visualization papers from mainstream visu-
exploration and analysis, and medical experts could make alization venues, e.g., IEEE Transactions on Visualization
the best of the potential values of medical images to enable and Graphics (TVCG), Computer Graphics Forum (CGF),
high-quality healthcare solutions. Vision is known to be the IEEE VIS conference, the EuroVis conference, the IEEE
the major and most efficient perception mechanism for PacificVis conference, and Computers and Graphics. Specif-
humans. Visualization transforms data into interactive ically, with a few exceptions, papers covered in Section 4
visual representations to facilitate data understanding and were selected by first searching for the term “medical visual-
exploration through visual perception and human- ization” on the two major venues—IEEE TVCG (including
computer interaction, and it can present medical images in the VIS conference) and CGF (including the EuroVis con-
3D with high accuracy. Visualization integrates the cognitive ference)—with the publication time within the last 15 years
advantage of humans and the computational advantage of (2006–2021), and then, we carefully examined each paper
computers for data mining [1] and decision-making [2] found in the search and included papers with direct rele-
and is an effective data analysis technology for medical vance to medicine and 3D visualization of medical images.
images. Throughout this paper, we understand the term Unlike a previous review on biomedical visualization
medical image visualization as 3D visualization of medical techniques [9], our overview covers advanced visual analysis
images. methods for medical images that are arranged based on a
We first report on the scope and sources of papers taxonomy of medical problems from individuals to popula-
involved (Section 2). We then summarize fundamental tech- tions (Section 4) and also includes advances in fundamental
niques that enable the visualization of various types of med- visualization methods (Section 3). This paper also features
ical images (Section 3). Next, in Section 4, specialized an overview of techniques that could improve spatial percep-

Downloaded from https://spj.science.org on September 26, 2024

medical imaging visualization methods are reviewed based tion, which is vital for the understanding of medical images
on our medical procedure-oriented and scale-based taxon- in 3D. Overall, rather than a comprehensive review of the lit-
omy. In Section 5, we summarize visualization techniques erature on the topic of medical image visualization, this
that may have potential medical applications and discuss review provides a big picture of the subject of study and
limitations of medical image visualization. Finally, we list introduces state-of-the-art techniques driven by specific
free software tools that are readily available online for med- medical problems.
ical image visualization (Section 6) and conclude the over-
view in Section 7. 3. Fundamental Medical Image
Visualization Techniques
2. Scope
We understand medical images as data over 3D spatial
Medical visualization, in general, is systematically covered in domains that have values defined for each point in space
the textbook by Preim and Bartz [3]. In this paper, we focus and possibly in time (e.g., time-varying data), namely, fields.
on techniques of 3D medical image visualization and specific Medical imaging data can be abstracted as fields where data
techniques for various medical problems based on imaging values are defined everywhere within the spatial domain, for
categorized by the medical procedure and the scale of example, fluid characteristics of tissues in an MR image.
studies. In practice, the field data is sampled and stored as dis-
There are several reviews of visualization techniques for crete data points in images of various formats, e.g., DICOM
medical images [4–8]. However, the target audience of these (Digital Imaging and Communications in Medicine), NII
reviews is visualization researchers rather than medical (NIfTI-1 data format), TIFF (Tag Image File Format), and
experts. These technical-oriented reviews offer in-depth dis- RAW (Raw image). We denote a data point v of a medical
cussions on visualization techniques for specific medical image data as
data types with fine-grained taxonomy, including perceptu-
ally motivated 3D medical image data visualization [4], mul- v : ðx, y, z, t, s1 , s2 , ⋯, sm Þ, ð1Þ
timodal medical data [6], medical flow data [8], cardiac 4D
data [5], and flattening-based medical visualization [7]. where x, y, z are the spatial coordinates, t is the time, and
Our paper, instead, is to provide medical experts with a s1 , s2 , ⋯, sm are physical attributes of this data point, for
general overview of this highly relevant field. We believe that example, the Hounsfield scale (for CT) or T1- (longitudinal
our review achieves a balance between the coverage of tech- relaxation time) weighted, T2- (transverse relaxation time)
niques and the relevance for medical experts, and our goal is weighted, and FLAIR-fluid-attenuated inversion recovery,
to convey existing visualization techniques that are poten- and m denotes the number of attributes.
tially useful for health care experts in their research and clin- Medical image data can be classified into three types: sca-
ical practices. lar, vector, and tensor, based on the mathematical properties
We focus on techniques that generate 3D visualizations of physical attributes s. When m > 1, the medical image data
of medical images. While other representations of 3D medi- are multifield data that can be a mixture of various data
cal images exist, e.g., 2D representations with flattening visu- types, e.g., scalars and vectors or scalars and tensors, that
alization techniques, we do not include them in our review are typical in real medical practices. Advanced techniques
and readers are referred to a survey elsewhere [7]. Our for multifield visualization are covered in Section 4.
review includes classic papers for fundamental techniques Figure 1 shows the classification of visualization techniques
Health Data Science 3

Type of
Si

Scalar, e.g., Vector, e.g., Tensor, e.g.,

Si = 0.5 si = [1, 0, 0]T 1 2 3
si = 2 1 2
3 2 1

Downloaded from https://spj.science.org on September 26, 2024

(a) (b) (c)

Figure 1: Classification of fundamental visualization techniques for different types of medical image data. Scalar, vector, and tensor image
data visualization is discussed in this order in the remainder of this section. Example visualizations include a direct volume rendering of a
CT scan of a torso [10] for scalar data, a streamline visualization of a bioelectric field simulation of a torso for vector data, and a tube-based
tractography visualization of a brain [11] for tensor data. Reprinted from Computers & Graphics, Vol. 36, No. 6, Zhou and Hansen [10],
Transfer Function Combinations, 596-606, Copyright 2012, with permission from Elsevier (a). Reprinted, with permission, from the SCI
Institute (b). ©2007 SPIE. Reprinted, with permission, from Weldeselassie et al. [11] (c).

for medical images with examples (Figures 1(a)–1(c)) of a angle mesh is then visualized with a color palette specified
scalar CT scan of a torso, a flow simulation of a torso, and by the user. Isosurfaces can be generalized to “thick” interval
a tensor field visualization of a brain. In the remainder of volumes and even be combined with direct volume render-
this section, we focus on the case of single-typed data, i.e., ing in a unified framework [13]. However, indirect volume
each attribute of s has the same type, and discuss fundamen- rendering shows only a limited number of features as sur-
tal visualization techniques for each type of data. faces, and it does not provide the volumetric look within
medical images.
3.1. Scalar Image Visualization. If each attribute of s of equa-
tion (1) is a scalar, i.e., a quantity without direction, the data 3.1.1. Direct Volume Rendering. For a systematic introduc-
is scalar volumetric data in the context of 3D medical imag- tion to direct volume rendering, we refer the reader to the
ing. In general, volume visualization techniques can be cate- course notes [14] or the book on real-time volume graphics
gorized into two classes: indirect volume visualization and [15]. Volume rendering is based on optics, and all tech-
direct volume visualization. Indirect volume visualization niques are approximations to the solution of the light prop-
extracts surfaces with certain data values to indirectly visual- agation equation [16] in volumetric materials. Later, volume
ize a volumetric data with surface meshes. Since the rendering was extended to include medical images—initially
extracted surfaces there have the same data value, for exam- focusing on CT human head scans—along with surface
ple, parts of the skin that have a specific Hounsfield scale in a shading [17, 18].
CT scan, such methods are also called isosurface rendering. An optical model determines how particles in the media
In contrast, direct volume visualization or volume rendering interact with light [19]. We illustrate the direct volume ren-
directly visualizes the volume data without extracting any dering problem with typical optical models in Figure 2: the
surfaces and allows the “see-through” of internal structures eye symbol represents the viewer, the cloud indicates the
of a medical image, for example, the brain within a head scalar volumetric data, a ray (the straight line) is shot from
MR scan. the viewer through the volume, the current volume sample
Isosurface rendering is usually realized with the march- is indicated by a gray dot, and the light source is drawn as
ing cubes method [12]. The marching cubes algorithm tra- the sun. The final color of the ray “seen” by the viewer is
verses data cells, i.e., cubes, through the data volume and determined by solving the volume rendering integral with
uses a lookup table to efficiently determine the topology of a given optical model. Particles emit and absorb light in
the isosurface and computes the intersection of the isosur- the emission and absorption model (Figure 2(a)), and the
face and the cell to construct triangles accordingly. The tri- volume sample only receives lights from samples to its back,
4 Health Data Science

(a)

(b) (d)

Downloaded from https://spj.science.org on September 26, 2024

(c) (e)
Figure 2: Optical models of direct volume rendering: (a) emission and absorption, (b) local illumination, and (c) global illumination. A CT
human head scan is visualized in (d) using local illumination with (e) a 1D transfer function.

e.g., the white dot, along the ray. On top of the emission and 3.1.2. Visual Data Exploration with Transfer Functions. The
absorption model, a local illumination model adds local medical image data and its rendering are linked by a transfer
reflections of light from the light source. Here, light from function—a mapping from the data value to optical proper-
the source along the straight line to the volume sample ties, i.e., color and opacity of the volume. The idea is already
may be attenuated as illustrated in Figure 2(b). Alternatively, realized in early volume rendering papers [17, 18], but they
attenuation between the light source and the point in the did not use transfer functions as a data exploration tool.
volume might be neglected. Finally, global illumination However, for medical visualization, transfer functions are
(Figure 2(c)) considers a full model of scattering that leads the main means of interactive visual mining/exploration of
to complex light paths and interactions, as indicated by the scalar medical images. A survey of transfer functions for vol-
irregular polylines. Currently, most applications rely on local ume rendering can be found elsewhere [25].
illumination (often, without attenuation from the light The basic and most frequently used transfer functions
source to the point of local illumination) as a compromise are one-dimensional (1D) transfer functions that typically
between rendering quality and computational cost. In con- map data values of the scalar volume, i.e., the grayscale value
trast, the global illumination model can achieve photorealis- of images, to color and opacity. An example of volume ren-
tic visual effects, such as shadows and translucency, that dering of a CT head scan is shown in Figure 2 with a screen-
improve spatial perception, which is important for medicine shot of a 1D transfer function widget (Figure 2(e)). The
as discussed in Section 3.1.3. distribution of data values is shown in blue in the back-
Volume rendering, even with a simple optical model, has ground, and the transfer function is shown in the fore-
high computational cost. Due to the limitation of computer ground: the line in black indicates the opacity, and the
hardware, interactivity was not achieved even with hardware color is set by linear interpolation of the colored dots. Here,
acceleration [20] in early work. Therefore, practical use of direct the skin and muscles are set to yellow and brown of low
volume rendering of medical images was not feasible back then. opacity, blood vessels are assigned red with high opacity,
With the advent of graphics processing units (GPUs), and bones are set to white of high opacity. Although rela-
interactive volume rendering techniques were devised tively easy to use, 1D transfer functions have limited feature
[21–23]. Among them, the GPU-based ray casting [23] classification capability; for example, they cannot clearly
method is currently the standard volume rendering tech- separate the bone and the blood vessels as shown in
nique in most applications. The main advantages of GPU- Figure 2(d), due to the partial-volume effect of medical
based ray casting lie in that no geometry has to be generated, imaging. A preintegrated technique can be employed to
and the implementation is straightforward. Interactive vol- improve the quality of volume rendering with 1D transfer
ume rendering is also available on mobile devices [24] that functions [26].
can facilitate ubiquitous visualization and analysis of medi- Artifacts caused by the partial-volume effect cannot be
cal images. resolved by using only the single scalar image data value.
Health Data Science 5

(a) (b) (c)

Figure 3: Direct volume rendering of a brain scan using (a) the traditional local illumination model and with (b) the directional occlusion
shading [35] that approximates global illumination to enhance depth perception. ©2009 John Wiley & Sons, Inc. Reprinted, with
permission, from Schott et al. [35]. A more advanced global illumination technique [36] generates the photorealistic rendering of a CT
scan in (c). ©2014 IEEE. Reprinted, with permission, from Ament et al. [36].

Therefore, more attributes are measured as in the multi- premature. Shadows and translucency are enabled by half-
modal medical imaging, e.g., T1, T2, and FLAIR channels angle slicing [22], and directional occlusion is also available

Downloaded from https://spj.science.org on September 26, 2024

in MRI, or derived from the original image data. Accord- [35]. Figure 3(b) shows the volume rendering of an MRI
ingly, multidimensional transfer functions use the attributes brain scan with directional occlusion. Compared to
jointly for improved classifications. Two-dimensional trans- Figure 3(a) with the traditional Phong illumination—a
fer functions that enhance boundaries of volumes are the frequently used local illumination method—directional
typical choice on most occasions as the second attribute is occlusion (Figure 3(b)) creates important depth cues that
easily derived from the original data [27, 28]. Interaction improve the perception of the complex creases of the brain.
widgets for 2D transfer functions [28] have become standard Scattering and shadows are also made possible with
in most of the current visualization tools described in GPU-based ray casting that achieves high image quality
Section 6. [39]. Advanced global illumination models [36, 40] are
Designing multidimensional transfer functions for available to include scattering and soft shadowing in ray-
more than two attributes is challenging. One common casting volume rendering. Figure 3(c) shows the volume
approach is to explore the value domain of multimodal rendering of a CT scan with a low-pass shadowing model
medical images. Several techniques [29–31] based on mul- with scattering that provides translucency. These methods
tidimensional diagrams facilitate selections of specific value could aid medical experts in their clinical work to quickly
ranges of each data attribute. However, value domain and accurately locate features of interest in the visualiza-
approaches are unfamiliar to medical experts. In contrast, tion, which is not possible with traditional local illumina-
the spatial domain, e.g., on the slices of images, is pre- tion models.
ferred. Accordingly, slice-based multidimensional transfer
function methods are available. For example, a machine 3.2. Vector Image Visualization. When attributes s (equation
learning-based method allows the user to draw directly (1)) are vectors, the medical image to be visualized is a vec-
on slices for training [32]. Another example is slice- tor field. Vector field visualization is extensively used and
based sketching combined with parallel coordinates and studied in computational fluid dynamics for science and
scatterplots for multimodal medical images [33]. A semi- engineering, and a survey of general vector field visualization
automatic method sets transfer functions through drawing techniques is available [43]. In medicine, vector field data in
features of interest on slices with probabilistic boundary terms of fluid flow from phase-contrast MR (PC-MR) scans
lassos and approximate optimization [34]. are typically used for understanding pathological cerebral
aneurysm haemodynamic, nasal aerodynamics, and aortic
3.1.3. Techniques for Improved Spatial Perception. Although haemodynamics; bioelectric simulations based on electrocar-
local illumination is widely used in volume rendering, more diography (ECG) in cardiology and electroencephalography
realistic rendering is often necessary for clinical purposes as (EEG) or magnetoancephalography (MEG) in neurology are
it provides important depth cues allowing accurate spatial another source of vector image data [44].
perception. Therefore, global illumination has been a core For more details on medical vector field data visualiza-
research topic in volume rendering. A general and compre- tion, we refer the reader to several surveys [5, 8, 45]. In this
hensive global illumination model that creates various visual section, we briefly review medical vector field visualization
phenomenon is available in computer graphics [37]. techniques in the following order: direct methods, geometry
In the case of direct volume rendering, Monte Carlo methods, feature methods, and techniques for accurate spa-
sampling enables global illumination [38]; however, the tial perception. Segmentation and mesh generation are also
gradual rendering from a coarse to fine visualization and important processes in vector medical image visualization
the noisy look makes its integration to the clinical pipeline but are beyond of the scope of our review.
6 Health Data Science

(a) (b) (c)

Figure 4: Vector field visualization of medical images. The LIC method is used to visualize (a) a simulation of the bioelectric field of the
heart [41]. ©2008 IEEE. Reprinted, with permission, from Li et al. [41]. Streamlines are visualized as colored lines in bioelectric field
simulations in (b) a torso (reprinted, with permission, from the SCI Institute). The visualization of a brain in (c) applies LIC with
isosurfacing for an MRI scan along with volume rendering of a functional MRI (fMRI) [42]. ©2007 SPIE. Reprinted, with permission,
from Schafhitzel et al. [42].

3.2.1. Direct Methods. Direct methods do not explicitly tex regions is a vital process in a number of vector medical

Downloaded from https://spj.science.org on September 26, 2024

extract any geometry or features in a vector field. One strat- image visualization techniques [52, 53]. Vortex cores in
egy visualizes vector information with glyphs on 2D slices blood vessels are related to blood transportation mecha-
[46]. Another strategy converts vector information into mul- nisms in the aorta and the left ventricle [53]. Vortex regions
tiple scalars and visualizes them with color coding in slices are extracted for analysis of potential malfunctions [52].
or volumes using direct volume rendering. However, direct Line predicates [54] are functions used for querying inte-
methods do not visualize trajectories of particles in the vec- gral lines (streamlines or pathlines) that meet certain prop-
tor field, i.e., global path features, which is one of the focal erties of interest. These functions are flexible and especially
points of vector field analysis. Therefore, direct methods suitable for vector medical images as flow features such as
see limited use in vector medical image visualization. velocity, shapes of lines, and distances to the vessel wall
can be encoded. Blood with high residence times [55] and
3.2.2. Geometry Methods. Extracting and visualizing repre- impingement zones of cerebral aneurysms [56] can be
sentative geometry in the vector field, e.g., streamlines, path- extracted with line predicates.
lines, or streaklines, are effective and well-received methods.
Geometry methods extract global path features from seeded 3.2.4. Techniques for Improved Spatial Perception. Spatial
points in the image [47]. Streamlines are visualized as 3D perception of path geometry, lines or tubes, is difficult in
lines or tubes colored by the desired property, for example, 3D due to many factors, e.g., cluttering, inaccurate depth
the electrical potentials of electric fields of a torso in perception, occlusion, and inaccurate perception of orienta-
Figure 4(v) or a brain with the volume rendering as context tion. Methods that improve spatial perception are available
in Figure 4(d). The analysis of flow patterns is aided with to complement aforementioned visualization techniques
image slices [48] similar to the case of volume rendering, (Sections 3.2.1–3.2.3). Particles in the flow field can be
where a cutting plane is often used. The extraction of global drawn as different shaped glyphs [52] or cartoon-styled
path features is sensitive to seed points; a probing tool facil- stretched ellipsoids or short pathlines (pathlets) [49]. Draw-
itates seeding in images [49]. However, automatic seeding ing whole extracted lines as illuminated streamlines [57] or
remains as a challenging and open question. with halos [58, 59] is another strategy for visualization with
Alternative to visualizing 3D geometry as lines, tubes, or enhanced spatial perception.
surfaces, global path features can be shown with textures. Global illumination is superior to local illumination in
Line integral convolution (LIC) [50] visualizes the vector user performance for locating and comparing features with
field by convolving a noise image with a low-pass filter along tube/line renderings as shown in a comparative study [60].
streamlines extracted from the data image. The LIC method Shadows are especially important for depth perception of
can be extended to curved surfaces [41, 51], for example, on vector field visualization as they provide depth cues. Direc-
the surface of a heart [41] as shown in Figure 4(a) or a torso tional ambient occlusion [61] or ray tracing for tubes [62]
as shown in Figure 4(b). is able to create shadow effects for vector field data or field
lines extracted from tensor fields (Figures 5(c) and 5(d)).
3.2.3. Feature Methods. Feature methods extract features of
interests, e.g., sinks, sources, saddle points, vortices, and 3.3. Tensor Image Visualization. Tensor image visualization
regions of abnormal flow velocity, in vector fields. We cover techniques concern medical images with the attribute(s) s
specialized methods for medical fluid flows; for other vector of equation (1) as tensors—here, we consider only second-
field data, we refer the readers to a general overview of order tensors [65], i.e., matrices, and most applications in
texture-based vector field visualization [43]. Extracting vor- medicine assume symmetric matrices. DTI captures the
Health Data Science 7

(a)

(b) (c) (d)

Figure 5: Visualizations of diffusion tensor medical images with (a) glyphs [63], (b) hybrid voxel-based and tractography method [64], and

Downloaded from https://spj.science.org on September 26, 2024

(c, d) tractography methods [61, 62]. Different uncertainty-aware glyph encodings are shown in (a), from left to right: ellipsoids,
superquadrics, and fourth-order homogeneous polynomial [63]. A combined rendering of voxel-based visualization of characteristic
curves along with extracted tractography is shown in (b) [64]. For tractography visualizations, extracted fiber tracts of the brain are
visualized as tubes with shadows with (c) ray tracing [62] and are combined into the context of volume rendering of scalar MRI with
unified volume and surface occlusion shading [61] (d). ©2012 IEEE. Reprinted, with permission, from Jiao et al. [63] (a). ©2011 IEEE.
Reprinted, with permission from, Hlawatsch et al. [64] (b). ©2019 John Wiley & Son, Inc. Reprinted, with permission, from Han et al.
[62] (c). ©2013 IEEE. Reprinted, with permission, from Schott et al. [61] (d).

diffusion information of water molecules in tissues as symmet- ellipsoids [69, 70] (spherical glyphs) or cuboids that change
ric matrices at each point. In medicine, clinical experts often shape and orientation based on diffusion information—the
use the term diffusion-weighted imaging (DWI) to refer to anisotropy and orientation of principal components. A com-
the isotropic diffusion map—a scalar measurement of DTI. posite glyph can directly encode linear, planar, and spherical
To avoid confusion, we use the term DTI to refer to the imag- information [71]. Superquadric glyphs [72] with general
ing that measures tensor fields and do not use DWI. We clas- principles of usage [73] avoid the shortcomings of spherical
sify techniques into three categories for tensor image or cubical glyphs. Uncertainty-aware visualization is avail-
visualization: voxel-based, glyph-based, and tractography. As able for augmenting spherical, superquadric glyphs for DTI
in previous sections, we also discuss available techniques for and fourth-order homogeneous polynomial glyphs for high
improved spatial perception for tensor medical images. angular resolution diffusion imaging (HARDI) [63]. Afore-
mentioned glyph shapes can be found in Figure 5(a). The
3.3.1. Voxel-Based Methods. By computing scalar descrip- spatial arrangement of glyphs is another crucial issue. A
tions of tensors and using 2D slices and/or volume render- glyph packing technique leads to efficient use of space and
ing, voxel-based methods convert tensor data visualization correct perception of the visualization [74].
to scalar data visualization. One such description is frac-
tional anisotropy [66], and other anisotropy measurements, 3.3.3. Tractography. The main goal of DTI is to reconstruct
including linear anisotropy, planar anisotropy, and isotropy the white matter fiber tracks of the brain. Tractography extracts
are also helpful for describing the shape and size of tensors. fibers or tracts from DTIs and visualizes extracted fibers with
Moreover, measurements based on characteristic curves, for vector field visualization techniques. Therefore, tractography
example, finite separation ratio, are calculated to describe is probably the most well-known and popular tensor visualiza-
coherent fiber structures [64]. Tract-based spatial statistics tion technique among medical experts. A review on tractogra-
uses nonlinear registration and fractional skeleton to phy can be found elsewhere [75]. Fiber tracts are typically
improve the analysis of multisubject DT images [67]. Strate- visualized as lines [57, 76] or tubes [77]. GPU acceleration
gies of voxel-based visualization using direct volume render- enables interactive visualization of fiber tracks as tubes with
ing are available, including designing transfer functions tuboids and level-of-detail techniques [78] or hybrid rendering
based on tensor information and using diffusion volume tex- of triangle stripes and point sprites [79].
tures [68]. Visualizing fiber tracts in the context of the scalar med-
ical image [80] helps medical experts better analyze the data.
3.3.2. Glyph-Based Methods. Glyphs allow encoding the Grouping and clustering fiber tracts reduce clutter and facil-
complex tensor information using multiple visual channels, itate understanding the anatomy of these tracts. Grouped or
for example, shape, size, and color. Classic glyph design uses clustered tracts are typically color labeled by anatomical
8 Health Data Science

bundles [81]. Similar to graph visualization, edge bundling A summary of papers reviewed in this section can be
of fiber tracts can also enhance the readability of the visual- found in Tables 1–3 for diagnosis, treatment, and prognosis,
ization [82, 83]. respectively. Each paper is documented with the following
properties for readers to quickly locate techniques of their
3.3.4. Techniques for Improved Spatial Perception. Percep- interests: supported data types (data type), scale of studies
tion of glyphs is difficult in 3D due to loss of information (scale), body locations of the problem (location), and modal-
from projection to the 2D image plane and the ambiguity ities of medical images (modalities). We classify data type
in the 3D shape representation. There are superquadric into scalar, vector, and tensor; scale is classified into individ-
glyphs that avoid the ambiguity and improve over spherical ual and population; location is defined by the target body
and cubical glyphs [72, 73]. The glyphs are typically ren- part(s) of medical image(s); and modalities show the modal-
dered with illuminated surfaces and shadow effects in 3D ity of scanners of medical images.
to further enhance depth perception.
Similarly, perception is an important issue in tractogra- 4.1. Diagnosis. Methods designed for diagnosis are summa-
phy. Illustrative visualization that provides depth cues helps rized in Table 1. While the majority of these methods handle
reduce visual clutter and enhance spatial perception. A sur- studies on the individual level, a few recent works support
vey on illustrative visualization is available elsewhere [84]. studies of populations.
Depth-dependent halos [59] enhance depth perception with
line-based visualization and depth-based contrast enhance- 4.1.1. Individual. Diagnosis for individuals often requires
ment. Line-based ambient occlusion [85] is devised to multimodal medical images to provide sufficient informa-
enhance depth cues of the tractography visualization in both tion. In oncology, positron emission tomography (PET)

Downloaded from https://spj.science.org on September 26, 2024

grayscale and color. Ambient occlusion also enhances depth images that show physiological functions and CT images
perception for combined direct volume rendering and fiber that represent anatomical structures are used jointly for the
tract visualization [61] as shown in Figure 5(b). Shadows diagnosis of tumors. Multimodal visualization techniques
could also be generated with ray tracing. In Figure 5(a), a are, therefore, required to analyze the combination of the
whole brain tractography is visualized by 3D tubes with two types of scans. Focusing only on potential PET anomaly
ray-traced shadows [62]. These techniques complement regions with the CT anatomy as the context, i.e., focus-and-
visualization in context [86] and fiber clustering and bun- context visualization, is an effective visualization strategy for
dling [83] for more effective analysis of fiber tracts. PET+CT images. An illustrative technique allows us to visu-
alize the CT data as the context in cartoon style and the PET
4. Medical Visualization Methods for data as focus with a see-through lens that quickly draws the
Health Applications attention of medical experts [87]. There, the focus can be
interactively manipulated and contents within focal regions
Medical studies on individuals are aimed at providing pre- are controlled by interactive transfer functions. Alterna-
cise and tailored solutions for the specific anatomy and tively, a visibility-based transfer function for PET+CT data
pathology for individual patients. With the power of medical allows users to select regions of interest for further analysis
image visualization, medical experts could deepen the [88]. Shape-encoded rendering combines shape analysis
understanding of the data and, therefore, potentially with volume rendering to highlight tubular and nodular
improve the quality of personalized medicine. structures [89]. The method aids the diagnosis of anomalies
Of equal importance is to understand health problems in in CT scans of lungs or PET/CT scans for oncological
populations. More specifically, medical image visualization practices.
for studies of populations involves ensemble data, i.e., a col- In the field of cardiology diagnosis, perfusion data from
lection of data members that are individual 3D medical MR or single photon emission computed tomography
images. Understanding such datasets requires specialized (SPECT) are used in conjunction with the regular MR or
ensemble visualization and visual analysis techniques built CT images that are of higher resolutions to indicate the
on top of basic methods covered in Section 3. Ensemble underlying anatomy. The analysis of coronary artery disease
visualization is aimed at achieving one or more of the fol- is achieved by integrating the perfusion MR with the mor-
lowing goals: visualizing the main trend of the ensemble, phologic data from CT angiography (CTA) using visual
visualizing outliers of the ensemble, and comparing specific analysis with multiple-linked views [91]. A well-accepted
members to the main trend or other members. Therefore, analysis tool in cardiac diagnosis, the bull’s eye plot is
we believe that it is necessary and helpful to distinct visual- extended for rest and stress comparison and is used interac-
ization methods that support medical studies for individuals tively to drive the 3D exploration with colored height fields,
from those for populations. icons, and synchronized lenses. The bull’s eye plot is further
In this section, we focus on medical image visualization extended to be continuous and as a volume to assess trans-
methods specifically designed to address clinical and public murality in a 3D anatomical context [90]. Diagnosis is
health problems. We cover a broad range of medical visual- achieved with visual analysis supported by a comprehensive
ization methods focusing on medical problems to be tackled visualization and interactive exploration with multiple-
(diagnosis, treatment, or prognosis) and with a classification linked views, and several segmented volumes and enhanced
of the scale of the medical study—from individuals to MR images are used for the joint rendering. A multivariate
populations. glyph-based method enables the structured analysis of
Health Data Science 9

Table 1: Summary of specialized medical image visualization techniques for diagnosis.

Reference Data type Scale Location Modalities

Lawonn et al. 2016 [87] Scalar Individual Whole body PET+CT
Jung et al. 2013 [88] Scalar Individual Whole body PET+CT
Wiemker et al. 2013 [89] Scalar Individual Lymph nodes, lungs, breast, whole body PET+CT+MR
Termeer et al. 2007 [90] Scalar Individual Heart Perfusion-MR+MR
Oeltze et al. 2006 [91] Scalar Individual Heart Perfusion-MR+CTA
Hennemuth et al. 2008 [92] Scalar Individual Heart MR
Kirisli et al. 2014 [93] Scalar Individual Heart CTA+SPECT
Meyer-Spradow et al. 2008 [94] Scalar Individual Heart SPECT
Williams et al. 2008 [95] Scalar Individual Colon CT
Mirhosseini et al. 2019 [96] Scalar Individual Colon CT
Song et al. 2017 [97] Scalar Individual Chest, abdomen CT
Viola et al. 2008 [98] Scalar Individual Liver US+CT
Zhou and Hansen 2014 [34] Scalar Individual Brain MR
Jösson et al. 2020 [99] Scalar Population Brain fMR+MR
Elbaz et al. 2014 [53] Vector Individual Heart PC-MR

Downloaded from https://spj.science.org on September 26, 2024

Meuschke et al. 2016 [100] Vector Individual Heart PC-MR
Köhler et al. 2013 [52] Vector Individual Heart PC-MR
Born et al. 2013 [55] Vector Individual Heart PC-MR
van Pelt et al. 2010 [101] Vector Individual Heart PC-MR
van Pelt et al. 2011 [49] Vector Individual Heart PC-MR
Zhang et al. 2016 [102] Tensor Population Brain DT
Zhang et al. 2017 [103] Tensor Population Brain DT

Table 2: Summary of specialized medical image visualization techniques for treatment.

Reference Data type Scale Location Modalities

Rieder et al. 2008 [104] Scalar Individual Brain MR
Weiler et al. 2011 [105] Scalar Individual Brain MR
Khlebnikov et al. 2011 [106] Scalar Individual Abdomen CT
Beyer et al. 2007 [107] Scalar Individual Brain MR
Dick et al. 2011 [108] Scalar Individual Bone CT
Lundstrom et al. 2011 [109] Scalar Individual Bone CT
Smit et al. 2007 [110] Scalar Individual Pelvic MR
Butson et al. 2013 [111] Scalar Individual Brain MR
Vorwerk et al. 2020 [112] Scalar Individual Brain MR
Bock et al. 2013 [113] Scalar Individual Brain MR
Athwale et al. 2019 [114] Scalar Individual Brain MR
Blaas et al. 2007 [115] Tensor Individual Brain fMR+MR+DT
Born et al. 2009 [116] Tensor Individual Brain fMR+MR+DT
Diepenbrock et al. 2011 [117] Tensor Individual Brain fMR+MR+DT
Joshi et al. 2008 [118] Tensor Individual Brain fMR+MR+DT
Rieder et al. 2008 [119] Tensor Individual Brain fMR+MR+DT
Dick et al. 2009 [120] Tensor Individual Bone CT+sim
Note: sim: simulation.

myocardial perfusion using 3D glyphs encoding parameters and perfusion SPECT images are combined and jointly ana-
of the left ventricular myocardium [94]. By linking the 3D lyzed to diagnose coronary artery disease [93]. A study com-
view with 2D slices, the method supports the analysis of nor- paring the method to the traditional practice shows that the
mal case, various types of ischemia, and heart failure. CTA visualization method is advantageous in terms of diagnostic
10 Health Data Science

Table 3: Summary of specialized medical image visualization techniques for prognosis.

Reference Data type Scale Location Modalities

Raidou et al. 2016 [127] Scalar Population Prostate MR
Karall et al. 2018 [128] Scalar Population Breast MR
Raidou et al. 2018 [129] Scalar Population Bladder CT
Furmanová et al. 2021 [130] Scalar Population Prostate, bladder, rectum CT

performance. Contrast-enhanced cardiac images, including applied to healthy and pathological hearts and shows aspects
perfusion images, whole-heart coronary angiography, and of flow that cannot be seen with traditional methods.
late enhanced images, are analyzed by aligning different
datasets together and visualized as multiple isosurfaces [92]. 4.1.2. Population. Diagnosis can benefit from studying
Another important medical image visualization-based health problems in a population, for example, with a cohort
diagnosis approach is virtual endoscope visualization, e.g., study, and by comparing different individuals. Traditionally,
virtual colonoscopy and gastroscopy, where the visualization cohort studies with medical images rely on hypothesis for-
of inner surfaces of tubular structures is the main focus. For mation and statistical analysis, but the visualization and
example, an immersive virtual colonoscopy method sup- exploration of the imaging data are ignored. A visual analy-
ports the exploration of the colon within volume visualiza- sis method combines hypothesis formation and reasoning
tion in a virtual reality environment [96]. A hybrid with interactive volume rendering of multivariate brain

Downloaded from https://spj.science.org on September 26, 2024

technique that combines the inner surface rendering and MRI and fMRI cohort study data [99]. With multiple-
volume rendering of colons is available [95]. For a full review linked views, the method supports the exploration of the
of flattening visualization techniques, we refer readers to a bidirectional correlations between the volume rendering
survey elsewhere [7]. and clinical parameters and the comparison of different
Extensive training is required for diagnosis with medical patient groups.
images. A visual analysis method enables comparative visu- Diagnosis can be potentially further improved by includ-
alization of gaze data of several radiologists reading slices ing tensor information of DTIs. However, visualization of
and volume rendering of medical images [97]. By setting in DTIs in a population is challenging because, on top of the
a real diagnostic environment, the method is useful for train- occlusion issue of spatially overlapping images, each voxel
ing radiologists. Ultrasound (US) images are frequently used there encodes complex information. As a first step, effective
in clinical practice, but effective diagnosis with such images comparative visualization of two DTI images is required. A
requires extensive training. A joint 2D US and 3D CT image glyph-based technique visualizes three aspects of tensors,
visualization method registers the 2D plane of the US image namely, the scale, the anisotropy type, and the orientation
to segmented 3D structures of CT to assist the learning of [102]. By showing the glyphs on 2D slices, the method is able
liver examinations [98]. to compare two DTI images. As an example, the brain DTI
The 4D phase-contrast MR (PC-MR) is a recent of a healthy subject is compared to an HIV-infected subject.
advancement in medical imaging that is designed for mea- An overview+detail visualization is devised for DTI image
suring time-varying flow fields in the body, which is specifi- ensembles: aggregate tensor glyphs show an overview of
cally used for hemodynamics analysis. Among other the ensemble in the spatial layout, and visualizations of ten-
features, the vortex is especially useful in the analysis and sor properties (scale, shape, and orientation) are used for
diagnosis of cardiac flow data. Vortex rings in the left ventri- detailed analysis [103]. A case study demonstrates that the
cle are extracted and visualized in 3D to analyze inflow dur- method is able to visualize and analyze a cohort DTI study
ing early and late diastolic filling of normal subjects [53]. of 46 subjects.
Quantitative parameters characterizing vortex flow for these
phases are formulated for normal subjects. Aortic vortex 4.2. Treatment. Treatments aided by medical visualization
flow is classified based on the orientation, shape, and tempo- are mainly surgical planning and therapeutical intervention
ral occurrence of the vortex for PC-MR data of healthy sub- planning. Therefore, treatment-related visualization
jects and ones with cardiovascular diseases [100]. The methods are individual-based as shown in Table 2. A thor-
classification results are visualized with 2D vortex plots ough introduction of various applications of visualization
and 3D glyph visualization. in surgical planning can be found elsewhere [121].
The flow field in the heart and aorta is analyzed by semi- Neurosurgery preoperative planning is a major task for
automatic segmentation with line predicates that extract visualization techniques for brain imaging. Due to the com-
vortices and visualized with arrows [52]. The most suitable plexity and importance of the brain, multimodal 3D medical
cardiac blood flow vortex extraction criterion is found images are used in neurosurgery to locate different anatom-
through comparison, investigating pathologies like coarcta- ical structures. Heterogeneous pathological tissues are visu-
tions, tetralogy of Fallot, and aneurysms. A visual analysis alized with volume rendering by registering multimodal
method provides flexible interactive exploration of cardiac volumes, e.g., T1, T2, and FLAIR MRI, and automatically
blood flow using line predicates that generate bundles with segmented mask volumes [104]. A slice-based interface that
similar flow characteristics [55]. The technique can be is familiar to medical experts is used to drive the visual
Health Data Science 11

exploration of multimodal brain images with direct manipu- by registering the MRIs of a patient to visualize the context
lation on the 2D images with lassos [34]. Transfer functions (organs around the tumor), the target (the tumor), and risks
are semiautomatically designed based on user-selected 2D (autonomic nerves) of the surgery. Distances between nerves
regions, and then, brain tumors and edema can be visually to the mesorectum and the tumor to the mesorectum, which
segmented in 3D and visualized with volume rendering. are critical to the surgery, are calculated as a distance field. A
Vascular structures in the brain are extracted and visualized linked-view tool comprising a 3D model view, an MRI view,
with volume rendering to aid neurosurgery planning for and a distance field-based unfolded view is implemented.
arteriovenous malformations [105]. Here, feeding arteries, Five medical experts evaluated the method and considered
draining veins, and arteries “en passage” are segmented that it has potential in surgical planning and surgical train-
and visualized together with the brain rendered as the con- ing for oncologic surgeons.
text. A high-quality multimodal scalar volume visualization Precise preoperative planning is also critical in ortho-
method facilitates the actual planning of neurosurgeries pedics. A medical visualization table is available to visual-
[107]. The method supports specific operation planning, ize CT scans using volume rendering with user interface
for example, the optimal skin incision and skull opening and interactions designed for low learning effort and sim-
for the pathology and customized surgery of deep-seated ilarity to the real working scenarios for surgeons [109]. A
lesions for a given patient; specialized visualization of super- user study shows that the table system is liked by surgeons
ficial brain anatomy, function, and metabolism facilitates the and potentially beneficial for planning. Hip joint replace-
planning. ment is an important surgery in orthopedics. The optimal
Using DT images jointly with fMRI and regular MR implant positioning can be aided by an interactive distance
scans could further improve the quality of surgical planning field visualization technique that uses glyphs and slices in

Downloaded from https://spj.science.org on September 26, 2024

as fiber tracts and functional regions of the brain around the a 3D isosurface context to show distances between the
tumor could be analyzed. DTI, fMRI, and regular MRIs are implant and the bone boundaries [108]. Stress simulation
combined and visualized with volume rendering and tube- is an effective method for the design and the planning of
based rendering for brain tumor resection planning [115]. the implant; however, the resulting stress tensor fields
Fiber bundles can be interactively selected so that those need special visualization methods as most methods are
around the tumor could be avoided in the planning. The designed for diffusion tensors. With volume rendering
fMRI activation areas, functional areas of the brain, and fiber and line rendering, a focus-and-context method is pro-
tracts connecting these areas are jointly visualized with illus- posed to visualize time-varying stress tensor fields gener-
trative rendering [116]. Interactive probing of fMRI, DTI, ated by such simulations [120]. The method supports the
and MRIs within the brain visualization is proposed for neu- interactive exploration of the simulation and reacts to
rosurgical planning [117]. Uncertainty of these images is changes in the simulation and therefore could compare
also visualized in the method to provide additional informa- the physiological stress distribution before and after the
tion to the user. Interactions, especially, cropping or cutting simulated replacement surgery.
operations from the surface of the brain to the inside, are Deep brain stimulation (DBS) is an accepted neuromod-
critical for neurosurgery planning. Volume clipping with ulation therapy for treating the motor symptoms of Parkin-
complex geometries [122] is the foundation for such tailored son’s disease. The DBS device that generates electrical
cropping operations. Cropping views with different shapes, stimulation as an alternation of neural activity is comprised
e.g., sphere, cube, and cylinder, are combined with an of a multielectrode lead implanted in the brain and a con-
image-guided navigation system that visualizes MRI, fMRI, nected subcutaneous implantable pulse generator. The accu-
DTI, and SPECT for epilepsy neurosurgery [118]. Distance racy of the multielectrode lead placement is the key to the
information is critical in preoperative planning, and DTI effectiveness of the therapy. A mobile device-based visualiza-
and fMRI provide such data for fiber tracts and functional tion tool supports volume rendering and isosurface render-
regions. In a comprehensive method designed for neurosur- ing to compare different settings of DBS and help
gical planning, the tumor and neighboring fiber tracts are healthcare providers to choose the optimal configuration
rendered as the focus with distance-based enhancements for a patient [111]. A further improvement uses a client-
while the volume-rendered brain provides the context server approach to achieve efficient interactive visualization
[119]. The planned path can be interactively manipulated and simulation of DBS [112]. The usefulness of the method
and is visualized as a line and as a cylindrical cropping win- is demonstrated by a postoperative example and another
dow on the brain. example of DBS surgery pre- and interoperative planning.
In an oncology surgery, multiple possible paths to a The precision of DBS electrode positioning is related to the
tumor may exist. However, the safety of paths is not equal uncertainty introduced by the resolution of brain imaging.
and has to be considered during the planning. A ray-based The positional uncertainty of each electrode is quantified
method estimates the safety of all straight access paths to and visualized with uncertainty-aware volume and isosur-
the tumor in volume rendering and provides the area and face visualization techniques [114]. Multimodal volumes
path safety information [106]. Clear evidence shows that and their associated uncertainty are quantified and fused in
the method is liked by medical experts and can be used in a multiview visual analysis method to assist the planning of
clinical practice with little overhead. Pelvic oncology surgery DBS electrodes [113]. This comprehensive method covers
planning is aided with a visual analysis method based on the planning, recording, and placement phases of the treat-
preoperative MR scans [110]. The method is built on an atlas ment and uses volume and geometry rendering,
12 Health Data Science

Table 4: Techniques with potential health science applications.

Reference Scale Location Modalities Features

Zachow et al. 2009 [131] Individual Nose CT+sim Fluid dynamics: speed, pressure, humidity, temperature
Gasteiger et al. 2012 [56] Individual Brain CTA+sim Fluid dynamics: inflow jet and impingement zone
Meuschke et al. 2019 [132] Individual Brain CTA+sim Fluid dynamics: vortex, blood flows
Rosen et al. 2016 [133] Individual Heart MR+DT+sim Bioelectric fields
Meuschke et al. 2017 [134] Individual Brain Sim Rupture risk of aneurysms, stress tensor
Zhou et al. 2021 [135] Population Brain, heart MR+sim Quantitative comparison of scalar medical images
Note: sim: simulation.

spatiotemporal visualization, and uncertainty visualization [130], and its usefulness has been demonstrated through
for corresponding elements in the procedure. the exploration of cohort studies by health experts.
A relevant topic that requires high-quality visualization
is the segmentation of 3D anatomy and lesions from med- 5. Future Directions and Limitations of Medical
ical images. Segmentation is often necessary for the diag- Image Visualization
nosis and treatment of individuals and, therefore, is a
prerequisite for many of the aforementioned visualization Some visualization methods support exploring and analyz-
techniques. Accurate and efficient segmentation of specific ing medical image data from basic medical research that

Downloaded from https://spj.science.org on September 26, 2024

anatomical structures, for example, the heart [123, 124] can potentially address unsolved health challenges in the
and the prostate [125], has been an active and long- future. In this section, we discuss some of these important
standing research area in medical imaging. A discussion future directions and also limitations of medical image visu-
of state-of-the-art segmentation techniques is beyond the alization. In Table 4, we summarize medical image visualiza-
scope of our paper and can be found elsewhere [126]. In tion techniques with potential health science applications.
Section 6, we list free visualization software dedicated to Here, we list the features of these methods as a reminder
3D segmentation. to the readers of what are available in the visualization tool
box in the future.
4.3. Prognosis. In contrast to the case of treatment, the value Simulations are an important approach in medical
of visualization lies in its capability of comparison in a pop- research to understand complex, invisible, and/or perpetual
ulation to aid the prognosis evaluation. Therefore, all tech- activities in human bodies. Nasal flow simulation is an
niques for prognosis summarized in Table 3 are important means for understanding the physiological nasal
population-based. A number of techniques focus on radio- breathing that improves the overall traditional statistics-
therapy treatment evaluation. A visualization method is based summary of flow behaviors. A visual analysis method
available for exploring and understanding tumor control aids in the exploration of a computational fluid dynamics
probability models for cohorts [127]. By combining visuali- simulation on an anatomically correct model of the upper
zations of medical images and statistical models, the method respiratory tract [131]. With multiple-linked views, the
supports the exploration of uncertainty, parameter sensitiv- method allows users to analyze multiple attributes, e.g.,
ity analysis, interpatient response variability identification, speed, pressure, humidity, and temperature, of the complex
and finding treatment strategies that result in the desired flow simulation to derive intervention plans. Hemodynamic
outcome. characteristics in cerebral aneurysms are studied with com-
A cohort study of patients who underwent breast cancer putational fluid dynamic simulations and segmented data
chemotherapy treatment is analyzed with a visualization from CTA scans [56]. The visual analysis tries to understand
method that shows different aspects of the study using the inflow jet and impingement zone that are correlated with
multiple-linked views [128]. The method combines medical the risk of rupture. A vortex classification method automat-
images and nonimage information in an interactive visuali- ically classifies blood flows in cerebral aneurysms and visual-
zation tool that allows for analyzing individual patients, izes the clusters as streamlines in an aneurysm-based and a
comparing different chemotherapy treatment strategies and hemisphere-based visualization [132].
comparing different patients. Cardiac diseases are typically related to malfunctions in
Radiotherapy-induced bladder toxicity is analyzed with a bioelectric fields in the body that are difficult to measure
visualization technique tailored to investigating individual in vivo. Simulation of such bioelectric fields then becomes
patients and cohorts in the whole treatment process of a a feasible alternative, and, therefore, in-depth analysis of
cohort study [129]. This method focuses on the analysis of simulation results is important. Early work focuses on the
the impact of shape variations on the accuracy of dose deliv- visualization of torso electric field simulations [136]. Special-
ery by integrating the spatial visualization of bladders, ized for the 3D myocardial ischemia simulation, a multiple-
dimensionality reduction and clustering, and dose distribu- linked view approach is devised to perform the visual analy-
tion visualizations. The idea is further extended to visualize sis of the multiple simulation runs of bioelectric fields on the
and analyze more organs that may impact the accuracy of heart [133], which could also potentially be used for diagno-
dose delivery in radiotherapy treatment for prostate cancer sis. A systematic discussion of computational and numerical
Health Data Science 13

Table 5: Medical image visualization tools.

Name Data types Features

ParaView Scalar, vector, tensor Analysis, large datasets, parallel/super computing
Voreen Scalar, vector Rapid prototype
Inviwo Scalar, vector Rapid prototype
MegaMol Scalar, vector Particles, rapid prototype
SCIRun Scalar, vector, tensor Modeling, simulation, analysis
3D Slicer Scalar AI, segmentation
Seg3D Scalar Segmentation
ImageVis3D Scalar Large datasets
FluoRenderer Scalar Confocal microscopy data

methods for bioelectric fields problems can be found in a 6. Software Tools for Medical
review, where related visualization techniques are also dis- Image Visualization
cussed [44].
Comparative visualization is important for understand- Both commercial tools and free software are available for

Downloaded from https://spj.science.org on September 26, 2024

ing an ensemble of simulation runs and comparing between medical image visualization. In this paper, we list representa-
patients or different measures of medical images. Scalar tive free software that can be found on the Internet and read-
image ensembles, for example, brain MR atlas data ily used for 3D medical images of various types and file
(https://www.oasis-brains.org/), are a common form of formats. As shown in Table 5, we briefly summarize these
ensemble medical image data. Direct visualization of image tools with data types that can be handled (data types) as well
members in 2D and 3D is not effective due to occlusion, as their featuring characteristics (features).
and quantitative comparison is not feasible in this way Visualization and analysis tools for general scientific
either. One possible solution is to reduce the dimensionality problems and data provide flexible rapid prototyping frame-
of image data to one (1D) with space-filing curves [135, 137, works. ParaView (https://www.paraview.org/) is a cross-
138]. Data-driven space-filling curves [135] better preserve platform open-source visualization tool designed for interac-
spatial coherency in the resulting 1D representation than tive visualization and data analysis for a wide range of
static curves [137, 138], which potentially break spatially research and engineering areas, and various types of medical
coherent features into distant 1D fragments. The method is images (scalar, vector, and tensor) are supported [139].
applied for visualizing brain MRI atlas and an ensemble of Voreen (voreen.uni-muenster.de) is an open-source rapid
3D myocardial ischemia simulations and could potentially application development framework for the interactive visu-
be used for diagnosis or research by finding anomalies of alization and analysis of multimodal volumetric datasets
subjects through comparison to the main trends. Compara- [140]. Inviwo (https://inviwo.org/) is a framework for rapid
tive visualization of stress tensors is available for analyzing prototyping visualizations and provides a rich visual inter-
rupture risks of cerebral aneurysms based on computational face for creating customized visualization [141]. MegaMol
fluid dynamic simulations [134]. With several glyph designs, (https://megamol.org/) is a comprehensive cross-platform
the method supports the comparison of local stress tensors visualization prototyping framework evolved from particle
on the inner and outer vessel walls. Medical experts consider rendering for molecular datasets [142, 143]. SCIRun
that this method introduces the often overlooked wall struc- (https://www.sci.utah.edu/software/scirun.html) is a soft-
ture information for rupture assessment and could contrib- ware environment for scientific problem simulation, model-
ute to the development of a comprehensive risk factor of ing, and visualization, and it supports various types of
aneurysms in the future. medical images.
Medical image visualization has its limitations. First, cus- A number of tools are available for scalar medical image
tomized techniques and tools are required for specific medical visualization and analysis. 3D Slicer (https://download.slicer
problems, which demand close collaborations between the .org/) is a tool for visualization and analysis of medical
visualization and medical experts. Typically, an iterative pro- images and features interface for medical devices, for exam-
cess with several prototypes is required for a method and its ple, surgical navigation system and robotic devices [144].
associated software tool to become usable, which is often Seg3D (https://www.sci.utah.edu/software/seg3d.html) is a
time-consuming. Second, a learning process is required for medical volume segmentation and processing tool that
medical experts to familiarize themselves with new concepts allows for flexible manual segmentation and a number of
or interactions, for example, transfer function design in vol- automatic segmentation algorithms. ImageVis3D (https://
ume rendering. Nevertheless, medical image visualization www.sci.utah.edu/software/imagevis3d.html) is a scalable
has the advantage of combining the expertise of humans and and multiplatform volume visualization tool that supports
the computational power of machines, which is vital for health large datasets and works on mobile devices [145]. FluoRen-
science, making it a promising research direction. derer (https://www.sci.utah.edu/software/fluorender.html)
14 Health Data Science

features the visualization of confocal microscopy data and [2] D. Streeb, M. El-Assady, D. A. Keim, and M. Chen, “Why
multichannel scalar volume datasets. visualize? Arguments for visual support in decision making,”
These software tools contain example datasets, tutorials, IEEE Computer Graphics and Applications, vol. 41, no. 2,
detailed user guides, and supporting communities. Readers pp. 17–22, 2021.
are encouraged to try out these tools with included datasets [3] B. Preim and D. Bartz, Visualization in Medicine, Morgan
and their own medical images to have first-hand experience Kaufmann, Burlington, 2007.
of some key techniques discussed in this paper. [4] B. Preim, A. Baer, D. Cunningham, T. Isenberg, and
T. Ropinski, “A survey of perceptually motivated 3D visuali-
7. Conclusion zation of medical image data,” Computer Graphics Forum,
vol. 35, no. 3, pp. 501–525, 2016.
In this paper, we have provided an overview of 3D medical [5] B. Kohler, S. Born, R. F. P. van Pelt, A. Hennemuth, U. Preim,
image visualization techniques. Starting from a classification and B. Preim, “A survey of cardiac 4D PC-MRI data process-
of medical images in terms of data characteristic, i.e., the ing,” Computer Graphics Forum, vol. 36, no. 6, pp. 5–35, 2017.
mathematical properties of field data, we review fundamen- [6] K. Lawonn, N. Smit, K. Buhler, and B. Preim, “A survey on
tal visualization techniques for scalar, vector, and tensor multimodal medical data visualization,” Computer Graphics
medical images. The discussion covers seminal work that Forum, vol. 37, no. 1, pp. 413–438, 2018.
lays foundations for each type of visualization and features [7] J. Kreiser, M. Meuschke, G. Mistelbauer, B. Preim, and
techniques that allows for accurate spatial perception that T. Ropinski, “A survey of flattening-based medical visualiza-
is important in medical practices. Next, specialized medical tion techniques,” Computer Graphics Forum, vol. 37, no. 3,
visualization techniques are categorized based on the sup- pp. 597–624, 2018.

Downloaded from https://spj.science.org on September 26, 2024

ported data type, medical procedures, the scale of the con- [8] S. Oeltze-Jafra, M. Meuschke, M. Neugebauer et al., “Genera-
cerned medical problems, locations of the studied tion and visual exploration of medical flow data: survey,
problems, and modalities of images. We describe the medi- research trends and future challenges,” Computer Graphics
cal aspects as well as technical aspects of each technique to Forum, vol. 38, no. 1, pp. 87–125, 2019.
facilitate medical experts to choose proper techniques for [9] C. R. Johnson and X. Tricoche, “Biomedical visualization,” in
their specific problems. Then, we discuss works that have Advances in Biomedical Engineering, P. Verdonck, Ed.,
potential health science applications and the limitations of pp. 211–273, Elsevier, Amsterdam, Netherlands, 2009.
medical image visualization. Finally, state-of-the-art free [10] L. Zhou, M. Schott, and C. Hansen, “Transfer function com-
visualization software is listed so that medical experts can binations,” Computers Graphics, vol. 36, no. 6, pp. 596–606,
have a first-hand experience of some of the aforementioned 2012.
techniques and experiment with their own data to gain [11] Y. T. Weldeselassie, G. Hamarneh, and D. Weiskopf, “Ten-
insights into visualization. sor dissimilarity based adaptive seeding algorithm for DT-
MRI visualization with streamtubes,” in Medical Imaging
2007: Visualization and Image-Guided Procedures, K. R.
Conflicts of Interest Cleary and M. I. Miga, Eds., vol. 6509 of International
Society for Optics and Photonics, , pp. 900–908, SPIE,
The authors declare no conflicting interests.
2007.
[12] W. E. Lorensen and H. E. Cline, “Marching cubes: a high res-
Authors’ Contributions olution 3D surface construction algorithm,” SIGGRAPH
L. Zhou provided the overall supervision of the work, col- Computer Graphics, vol. 21, no. 4, pp. 163–169, 1987.
lected papers for review, and drafted and edited the manu- [13] M. Ament, D. Weiskopf, and H. Carr, “Direct interval volume
script. MJ. Fan provided clinical support as the medical visualization,” IEEE Transactions on Visualization and Com-
puter Graphics, vol. 16, no. 6, pp. 1505–1514, 2010.
expert and contributed to the structuring of the manuscript
and the collection and summarizing of papers. C. Hansen [14] K. Engel, M. Hadwiger, J. M. Kniss, A. E. Lefohn, C. R. Sal-
and C. R. Johnson contributed to the editing of the manu- ama, and D. Weiskopf, “Real-time volume graphics,” in
ACM SIGGRAPH 2004 Course Notes, ser. SIGGRAPH '04,
script. D. Weiskopf contributed to the structuring of the
p. 29, Association for Computing Machinery, Los Angeles,
manuscript and critically edited the manuscript. Liang Zhou CA, USA, 2004.
and Mengjie Fan contributed equally to this work.
[15] M. Hadwiger, J. M. Kniss, C. Rezk-salama, D. Weiskopf, and
K. Engel, Real-Time Volume Graphics, A. K. Peters, Ltd.,
Acknowledgments 2006.
This research was supported by the Data for Better Health [16] J. T. Kajiya and B. P. Von Herzen, “Ray tracing volume den-
Project of Peking University-Master Kong and by NIH sities,” SIGGRAPH Computer Graphics, vol. 18, no. 3,
pp. 165–174, 1984.
(R01 EB031872).
[17] M. Levoy, “Display of surfaces from volume data,” IEEE
Computer Graphics and Applications, vol. 8, no. 3, pp. 29–
References 37, 1988.
[1] D. Keim, “Information visualization and visual data mining,” [18] R. A. Drebin, L. Carpenter, and P. Hanrahan, “Volume ren-
IEEE Transactions on Visualization and Computer Graphics, dering,” SIGGRAPH Computer Graphics, vol. 22, no. 4,
vol. 8, no. 1, pp. 1–8, 2002. pp. 65–74, 1988.
Health Data Science 15

[19] N. Max, “Optical models for direct volume rendering,” IEEE ration,” in 2013 IEEE Pacific Visualization Symposium,
Transactions on Visualization and Computer Graphics, vol. 1, pp. 73–80, Sydney, NSW, Australia, 2013.
no. 2, pp. 99–108, 1995. [34] L. Zhou and C. Hansen, “GuideME: slice-guided semiauto-
[20] B. Cabral, N. Cam, and J. Foran, “Accelerated volume render- matic multivariate exploration of volumes,” Computer
ing and tomographic reconstruction using texture mapping Graphics Forum, vol. 33, no. 3, pp. 151–160, 2014.
hardware,” in Proceedings of the 1994 Symposium on Volume [35] M. Schott, V. Pegoraro, C. Hansen, K. Boulanger, and
Visualization, pp. 91–98, Tysons Corner, VA, USA, 1994. K. Bouatouch, “A directional occlusion shading model for
[21] S. Roettger, S. Guthe, D. Weiskopf, T. Ertl, and W. Strasser, interactive direct volume rendering,” Computer Graphics
“Smart hardware-accelerated volume rendering,” in Euro- Forum, vol. 28, no. 3, pp. 855–862, 2009.
graphics/IEEE VGTC Symposium on Visualization, G.-P. [36] M. Ament, F. Sadlo, C. Dachsbacher, and D. Weiskopf, “Low-
Bonneau, S. Hahmann, and C. D. Hansen, Eds., pp. 231– pass filtered volumetric shadows,” IEEE Transactions on
238, The Eurographics Association, 2003. Visualization and Computer Graphics, vol. 20, no. 12,
[22] J. Kniss, S. Premoze, C. Hansen, P. Shirley, and pp. 2437–2446, 2014.
A. McPherson, “A model for volume lighting and modeling,” [37] M. Ament, C. Bergmann, and D. Weiskopf, “Refractive radi-
IEEE Transactions on Visualization and Computer Graphics, ative transfer equation,” ACM Transactions on Graphics,
vol. 9, no. 2, pp. 150–162, 2003. vol. 33, no. 2, 2014.
[23] J. Kruger and R. Westermann, “Acceleration techniques for [38] T. Kroes, F. H. Post, and C. P. Botha, “Exposure render: an
GPU-based volume rendering,” in IEEE Visualization Con- interactive photo-realistic volume rendering framework,”
ference 2003, pp. 287–292, Seattle, USA, 2003. PLoS One, vol. 7, no. 7, 2012.
[24] M. Moser and D. Weiskopf, “Interactive volume rendering on [39] T. Ropinski, C. Döring, and C. Rezk-Salama, “Interactive vol-
mobile devices,” in In Vision, Modeling, and Visualization

Downloaded from https://spj.science.org on September 26, 2024

umetric lighting simulating scattering and shadowing,” in
VMV '08 Conference Proceedings, pp. 217–226, Konstanz, 2010 IEEE Pacific Visualization Symposium (PacificVis),
Germany, 2008. pp. 169–176, Taipei, Taiwan, 2010.
[25] P. Ljung, J. Krüger, E. Groller, M. Hadwiger, C. D. Hansen, [40] M. Ament, F. Sadlo, and D. Weiskopf, “Ambient volume scat-
and A. Ynnerman, “State of the art in transfer functions for tering,” IEEE Transactions on Visualization and Computer
direct volume rendering,” Computer Graphics Forum, Graphics, vol. 19, no. 12, pp. 2936–2945, 2013.
vol. 35, no. 3, pp. 669–691, 2016. [41] G.-S. Li, X. Tricoche, D. Weiskopf, and C. D. Hansen, “Flow
[26] K. Engel, M. Kraus, and T. Ertl, “High-quality pre-integrated charts: visualization of vector fields on arbitrary surfaces,”
volume rendering using hardware-accelerated pixel shading,” IEEE Transactions on Visualization and Computer Graphics,
in Proceedings of the ACM SIGGRAPH/EUROGRAPHICS vol. 14, no. 5, pp. 1067–1080, 2008.
Workshop on Graphics Hardware, pp. 9–16, Los Angeles, [42] T. Schafhitzel, F. Roler, D. Weiskopf, and T. Ertl, “Simulta-
CA, USA, 2001. neous visualization of anatomical and functional 3D data by
[27] G. Kindlmann and J. Durkin, “Semi-automatic generation of combining volume rendering and ow visualization,” in Med-
transfer functions for direct volume rendering,” in IEEE Sym- ical Imaging 2007: Visualization and Image-Guided Proce-
posium on Volume Visualization, pp. 79–86, Research Trian- dures, K. R. Cleary and M. I. Miga, Eds., vol. 6509 of
gle Park, NC, USA, 1998. International Society for Optics and Photonics, , pp. 21–29,
[28] J. Kniss, G. Kindlmann, and C. Hansen, “Multidimensional SPIE, 2007.
transfer functions for interactive volume rendering,” IEEE [43] R. S. Laramee, H. Hauser, H. Doleisch, B. Vrolijk, F. H. Post,
Transactions on Visualization and Computer Graphics, and D. Weiskopf, “The state of the art in flow visualization:
vol. 8, no. 3, pp. 270–285, 2002. dense and texture-based techniques,” Computer Graphics
[29] H. Akiba and K.-L. Ma, “A tri-space visualization interface Forum, vol. 23, no. 2, pp. 203–221, 2004.
for analyzing time-varying multivariate volume data,” in [44] C. R. Johnson, “Computational and numerical methods for
Eurographics/IEEE-VGTC Symposium on Visualization, K. bioelectric field problems,” Biomedical Engineering, vol. 25,
Museth, T. Moeller, and A. Ynnerman, Eds., pp. 115–122, no. 1, pp. 1–81, 1997.
The Eurographics Association, 2007. [45] A. Vilanova, B. Preim, R. V. Pelt, R. Gasteiger, M. Neugebauer,
[30] X. Zhao and A. Kaufman, “Multi-dimensional reduction and and T. Wischgoll, “Visual exploration of simulated and mea-
transfer function design using parallel coordinates,” in IEEE/ sured blood flow,” in Scientific Visualization: Uncertainty,
EG Symposium on Volume Graphics, R. Westermann and G. Multifield, Biomedical, and Scalable Visualization, C. D.
Kindlmann, Eds., pp. 69–76, The Eurographics Association, Hansen, M. Chen, C. R. Johnson, A. E. Kaufman, and H.
2010. Hagen, Eds., pp. 305–324, Springer, London, UK, 2014.
[31] H. Guo, H. Xiao, and X. Yuan, “Scalable multivariate volume [46] D. Laidlaw, R. Kirby, C. Jackson et al., “Comparing 2D vector
visualization and analysis based on dimension projection and field visualization methods: a user study,” IEEE Transactions
parallel coordinates,” IEEE Transactions on Visualization on Visualization and Computer Graphics, vol. 11, no. 1,
and Computer Graphics, vol. 18, no. 9, pp. 1397–1410, pp. 59–70, 2005.
2012. [47] M. H. Buonocore, “Visualizing blood flow patterns using
[32] F.-Y. Tzeng, E. Lum, and K.-L. Ma, “An intelligent system streamlines, arrows, and particle paths,” Magnetic Resonance
approach to higher-dimensional classiffication of volume in Medicine, vol. 40, no. 2, pp. 210–226, 1998.
data,” IEEE Transactions on Visualization and Computer [48] L. Wigström, T. Ebbers, A. Fyrenius et al., “Particle trace visu-
Graphics, vol. 11, no. 3, pp. 273–284, 2005. alization of intracardiac flow using time-resolved 3D phase
[33] L. Zhou and C. Hansen, “Transfer function design based on contrast MRI,” Magnetic Resonance in Medicine, vol. 41,
user selected samples for intuitive multivariate volume explo- no. 4, pp. 793–799, 1999.
16 Health Data Science

[49] R. van Pelt, J. Olivan Bescos, M. Breeuwer et al., “Interactive 2012 IEEE Pacific Visualization Symposium, pp. 193–200,
virtual probing of 4D MRI blood-flow,” IEEE Transactions Songdo, Republic of Korea, 2012.
on Visualization and Computer Graphics, vol. 17, no. 12, [64] M. Hlawatsch, J. E. Vollrath, F. Sadlo, and D. Weiskopf,
pp. 2153–2162, 2011. “Coherent structures of characteristic curves in symmetric
[50] B. Cabral and L. C. Leedom, “Imaging vector fields using line second order tensor fields,” IEEE Transactions on Visualiza-
integral convolution,” in Proceedings of the 20th Annual Con- tion and Computer Graphics, vol. 17, no. 6, pp. 781–794,
ference on Computer Graphics and Interactive Techniques, 2011.
SIGGRAPH'93, pp. 263–270, Anaheim, CA, USA, 1993. [65] P. Basser, J. Mattiello, and D. Lebihan, “Estimation of the
[51] D. Weiskopf and T. Ertl, “A hybrid physical/device-space effective self-diffusion tensor from the NMR spin echo,” Jour-
approach for spatio-temporally coherent interactive texture nal of Magnetic Resonance, Series B, vol. 103, no. 3, pp. 247–
advection on curved surfaces,” in Proceedings of Graphics 254, 1994.
Interface 2004, pp. 263–270, London, Ontario, Canada, 2004. [66] P. J. Basser and C. Pierpaoli, “Microstructural and physiolog-
[52] B. Köhler, R. Gasteiger, U. Preim, H. Theisel, M. Gutberlet, ical features of tissues elucidated by quantitative-diffusion-
and B. Preim, “Semi-automatic vortex extraction in 4D PC- tensor MRI,” Journal of Magnetic Resonance, Series B,
MRI cardiac blood flow data using line predicates,” IEEE vol. 111, no. 3, pp. 209–219, 1996.
Transactions on Visualization and Computer Graphics, [67] S. M. Smith, M. Jenkinson, H. Johansen-Berg et al., “Tract-
vol. 19, no. 12, pp. 2773–2782, 2013. based spatial statistics: voxelwise analysis of multi-subject
[53] M. S. M. Elbaz, E. E. Calkoen, J. J. M. Westenberg, B. P. F. diffusion data,” NeuroImage, vol. 31, no. 4, pp. 1487–
Lelieveldt, A. A. W. Roest, and R. J. van der Geest, “Vortex 1505, 2006.
flow during early and late left ventricular filling in normal [68] G. Kindlmann, D. Weinstein, and D. Hart, “Strategies for
subjects: quantitative characterization using retrospectively- direct volume rendering of diffusion tensor fields,” IEEE

Downloaded from https://spj.science.org on September 26, 2024

gated 4D flow cardiovascular magnetic resonance and Transactions on Visualization and Computer Graphics,
three-dimensional vortex core analysis,” Journal of Cardio- vol. 6, no. 2, pp. 124–138, 2000.
vascular Magnetic Resonance, vol. 16, no. 1, p. 78, 2014. [69] C. Pierpaoli and P. J. Basser, “Toward a quantitative assess-
[54] T. Salzbrunn, C. Garth, G. Scheuermann, and J. Meyer, ment of diffusion anisotropy,” Magnetic Resonance in Medi-
“Pathline predicates and unsteady flow structures,” The cine, vol. 36, no. 6, pp. 893–906, 1996.
Visual Computer, vol. 24, no. 12, pp. 1039–1051, 2008. [70] D. Laidlaw, E. Ahrens, D. Kremers, M. Avalos, R. Jacobs, and
[55] S. Born, M. Pfeie, M. Markl, M. Gutberlet, and C. Readhead, “Visualizing diffusion tensor images of the
G. Scheuermann, “Visual analysis of cardiac 4D MRI blood mouse spinal cord,” in Proceedings of IEEE Visualization
flow using line predicates,” IEEE Transactions on Visualization Conference, pp. 127–134, Research Triangle Park, NC, USA,
and Computer Graphics, vol. 19, no. 6, pp. 900–912, 2013. 1998.
[56] R. Gasteiger, D. J. Lehmann, R. van Pelt et al., “Automatic [71] C. F. Westin, S. E. Maier, B. Khidhir, P. Everett, F. A. Jolesz,
detection and visualization of qualitative hemodynamic char- and R. Kikinis, “Image processing for diffusion tensor mag-
acteristics in cerebral aneurysms,” IEEE Transactions on netic resonance imaging,” in Medical Image Computing and
Visualization and Computer Graphics, vol. 18, no. 12, Computer-Assisted Intervention– MICCAI'99, C. Taylor and
pp. 2178–2187, 2012. A. Colchester, Eds., pp. 441–452, Springer, Berlin, Heidel-
[57] M. Zockler, D. Stalling, and H.-C. Hege, “Interactive visuali- berg, 1999.
zation of 3D-vector fields using illuminated stream lines,” in [72] G. Kindlmann, “Superquadric tensor glyphs,” in Euro-
Proceedings of IEEE Visualization Conference, pp. 107–113, graphics/IEEE VGTC Symposium on Visualization, O. Deus-
San Francisco, CA, USA, 1996. sen, C. Hansen, D. Keim, and D. Saupe, Eds., pp. 147–154,
[58] V. Interrante and C. Grosch, “Strategies for effectively visual- The Eurographics Association, 2004.
izing 3D flow with volume LIC,” in Proceedings of IEEE Visu- [73] T. Schultz and G. L. Kindlmann, “Superquadric glyphs for
alization Conference, pp. 421–424, Phoenix, AZ, USA, 1997. symmetric second-order tensors,” IEEE Transactions on
[59] M. H. Everts, H. Bekker, J. B. Roerdink, and T. Isenberg, Visualization and Computer Graphics, vol. 16, no. 6,
“Depth-dependent halos: illustrative rendering of dense line pp. 1595–1604, 2010.
data,” IEEE Transactions on Visualization and Computer [74] G. Kindlmann and C.-f. Westin, “Diffusion tensor visualiza-
Graphics, vol. 15, no. 6, pp. 1299–1306, 2009. tion with glyph packing,” IEEE Transactions on Visualization
[60] C. Weigle and D. Banks, “A comparison of the perceptual and Computer Graphics, vol. 12, no. 5, pp. 1329–1336, 2006.
benefits of linear perspective and physically-based illumina- [75] S. Mori and P. C. M. van Zijl, “Fiber tracking: principles and
tion for display of dense 3D streamtubes,” IEEE Transactions strategies – a technical review,” NMR in Biomedicine, vol. 15,
on Visualization and Computer Graphics, vol. 14, no. 6, no. 7-8, pp. 468–480, 2002.
pp. 1723–1730, 2008. [76] L. Zhukov and A. Barr, “Oriented tensor reconstruction: trac-
[61] M. Schott, T. Martin, A. P. Grosset, S. T. Smith, and C. D. ing neural pathways from diffusion tensor MRI,” in Proceed-
Hansen, “Ambient occlusion effects for combined volumes ings of IEEE Visualization Conference, pp. 387–394, Boston,
and tubular geometry,” IEEE Transactions on Visualization MA, USA, 2002.
and Computer Graphics, vol. 19, no. 6, pp. 913–926, 2013. [77] S. Zhang, C. Demiralp, and D. Laidlaw, “Visualizing diffusion
[62] M. Han, I. Wald, W. Usher et al., “Ray tracing generalized tensor MR images using streamtubes and streamsurfaces,”
tube primitives: method and applications,” Computer IEEE Transactions on Visualization and Computer Graphics,
Graphics Forum, vol. 38, no. 3, pp. 467–478, 2019. vol. 9, no. 4, pp. 454–462, 2003.
[63] F. Jiao, J. M. Phillips, Y. Gur, and C. R. Johnson, “Uncertainty [78] V. Petrovic, J. Fallon, and F. Kuester, “Visualizing whole-
visualization in HARDI based on ensembles of ODFs,” in brain DTI tractography with GPU-based tuboids and LoD
Health Data Science 17

management,” IEEE Transactions on Visualization and Com- [92] A. Hennemuth, A. Seeger, O. Friman et al., “A comprehen-
puter Graphics, vol. 13, no. 6, pp. 1488–1495, 2007. sive approach to the analysis of contrast enhanced cardiac
[79] D. Merhof, M. Sonntag, F. Enders, C. Nimsky, P. Hastreiter, MR images,” IEEE Transactions on Medical Imaging,
and G. Greiner, “Hybrid visualization for white matter tracts vol. 27, no. 11, pp. 1592–1610, 2008.
using triangle strips and point sprites,” IEEE Transactions on [93] H. A. Kirişli, V. Gupta, R. Shahzad et al., “Additional diag-
Visualization and Computer Graphics, vol. 12, no. 5, nostic value of integrated analysis of cardiac CTA and SPECT
pp. 1181–1188, 2006. MPI using the SMARTVis system in patients with suspected
[80] T. Schultz, N. Sauber, A. Anwander, H. Theisel, and H.- coronary artery disease,” Journal of Nuclear Medicine, vol. 55,
P. Seidel, “Virtual Klingler dissection: putting fibers into con- no. 1, pp. 50–57, 2014.
text,” Computer Graphics Forum, vol. 27, no. 3, pp. 1063– [94] J. Meyer-Spradow, L. Stegger, C. Doring, T. Ropinski, and
1070, 2008. K. Hinrichs, “Glyph-based SPECT visualization for the diag-
[81] S. Zhang, S. Correia, and D. H. Laidlaw, “Identifying white- nosis of coronary artery disease,” IEEE Transactions on Visu-
matter fiber bundles in DTI data using an automated alization and Computer Graphics, vol. 14, no. 6, pp. 1499–
proximity-based fiber-clustering method,” IEEE Transactions 1506, 2008.
on Visualization and Computer Graphics, vol. 14, no. 5, [95] D. Williams, S. Grimm, E. Coto, A. Roudsari, and
pp. 1044–1053, 2008. H. Hatzakis, “Volumetric curved planar reformation for vir-
[82] G. Kindlmann, X. Tricoche, and C.-F. Westin, “Delineating tual endoscopy,” IEEE Transactions on Visualization and
white matter structure in diffusion tensor MRI with anisot- Computer Graphics, vol. 14, no. 1, pp. 109–119, 2008.
ropy creases,” Medical Image Analysis, vol. 11, no. 5, [96] S. Mirhosseini, I. Gutenko, S. Ojal, J. Marino, and
pp. 492–502, 2007. A. Kaufman, “Immersive virtual colonoscopy,” IEEE Trans-
[83] M. H. Everts, E. Begue, H. Bekker, J. B. T. M. Roerdink, and actions on Visualization and Computer Graphics, vol. 25,

Downloaded from https://spj.science.org on September 26, 2024

T. Isenberg, “Exploration of the brain's white matter structure no. 5, pp. 2011–2021, 2019.
through visual abstraction and multi-scale local fiber tract [97] H. Song, J. Lee, T. J. Kim, K. H. Lee, B. Kim, and J. Seo, “Gaze
contraction,” IEEE Transactions on Visualization and Com- Dx: interactive visual analytics framework for comparative
puter Graphics, vol. 21, no. 7, pp. 808–821, 2015. gaze analysis with volumetric medical imagesIEEE Transac-
[84] T. Isenberg, “A survey of illustrative visualization techniques tions on Visualization and Computer Graphics,” vol. 23,
for diffusion-weighted MRI tractography,” in Visualization no. 1, pp. 311–320, 2017.
and Processing of Higher Order Descriptors for Multi-Valued [98] I. Viola, K. Nylund, O. K. Øye, D. M. Ulvang, O. H. Gilja, and
Data, I. Hotz and T. Schultz, Eds., pp. 235–256, Springer H. Hauser, “Illustrated ultrasound for multimodal data inter-
International Publishing, Cham, 2015. pretation of liver examinations,” in Eurographics Workshop
[85] S. Eichelbaum, M. Hlawitschka, and G. Scheuermann, “Line- on Visual Computing for Biomedicine, C. Botha, G. Kindl-
AO—improved three-dimensional line rendering,” IEEE mann, W. Niessen, and B. Preim, Eds., The Eurographics
Transactions on Visualization and Computer Graphics, Association, 2008.
vol. 19, no. 3, pp. 433–445, 2013. [99] D. Jönsson, A. Bergström, C. Forsell et al., “VisualNeuro: a
[86] P. Svetachov, M. H. Everts, and T. Isenberg, “DTI in context: hypothesis formation and reasoning application for multi-
illustrating brain fiber tracts in situ,” Computer Graphics variate brain cohort study data,” Computer Graphics Forum,
Forum, vol. 29, no. 3, pp. 1023–1032, 2010. vol. 39, no. 6, pp. 392–407, 2020.
[87] K. Lawonn, S. Glaer, A. Vilanova, B. Preim, and T. Isenberg, [100] M. Meuschke, B. Köhler, U. Preim, B. Preim, and K. Lawonn,
“Occlusion-free blood flow animation with wall thickness “Semi-automatic vortex flow classification in 4D PC-MRI
visualization,” IEEE Transactions on Visualization and Com- data of the aorta,” Computer Graphics Forum, vol. 35, no. 3,
puter Graphics, vol. 22, no. 1, pp. 728–737, 2016. pp. 351–360, 2016.
[88] Y. Jung, J. Kim, S. Eberl, M. Fulham, and D. D. Feng, “Visibil- [101] R. Van Pelt, J. O. Bescós, M. Breeuwer et al., “Exploration of
ity-driven PET-CT visualisation with region of interest (ROI) 4D MRI blood flow using stylistic visualization,” IEEE Trans-
segmentation,” The Visual Computer, vol. 29, no. 6, pp. 805– actions on Visualization and Computer Graphics, vol. 16,
815, 2013. no. 6, pp. 1339–1347, 2010.
[89] R. Wiemker, T. Klinder, M. Bergtholdt, K. Meetz, I. C. [102] C. Zhang, T. Schultz, K. Lawonn, E. Eisemann, and
Carlsen, and T. Bulow, “A radial structure tensor and its A. Vilanova, “Glyph-based comparative visualization for dif-
use for shape-encoding medical visualization of tubular fusion tensor fields,” IEEE Transactions on Visualization and
and nodular structures,” IEEE Transactions on Visualiza- Computer Graphics, vol. 22, no. 1, pp. 797–806, 2016.
tion and Computer Graphics, vol. 19, no. 3, pp. 353–366, [103] C. Zhang, M. Caan, T. Höllt, E. Eisemann, and A. Vilanova,
2013. “Overview + detail visualization for ensembles of diffusion
[90] M. Termeer, J. Olivan Bescos, M. Breeuwer, A. Vilanova, tensors,” Computer Graphics Forum, vol. 36, no. 3, pp. 121–
and F. Gerritsen, “CoViCAD: comprehensive visualization 132, 2017.
of coronary artery disease,” IEEE Transactions on Visuali- [104] C. Rieder, M. Schwier, H. K. Hahn, and H.-O. Peitgen, “High-
zation and Computer Graphics, vol. 13, no. 6, pp. 1632– quality multimodal volume visualization of intracerebral
1639, 2007. pathological tissue,” in Proceedings of the First Eurographics
[91] S. Oeltze, A. Ku, F. Grothues, A. Hennemuth, and B. Preim, Conference on Visual Computing for Biomedicine, pp. 167–
“Integrated visualization of morphologic and perfusion data 176, Delft, The Netherlands, 2008.
for the analysis of coronary artery disease,” in Proceedings [105] F. Weiler, C. Rieder, C. A. David, C. Wald, and H. K. Hahn,
of the Eighth Joint Eurographics/IEEE VGTC Conference on “AVM-explorer: multi-volume visualization of vascular
Visualization, pp. 131–138, Lisbon, Portugal, 2006. structures for planning of cerebral AVM surgery,” in
18 Health Data Science

Eurographics 2011 - Dirk Bartz Prize, K. Buehler and A. [118] A. Joshi, D. Scheinost, K. Vives, D. Spencer, L. Staib, and
Vilanova, Eds., pp. 9–12, The Eurographics Association, X. Papademetris, “Novel interaction techniques for neurosur-
2011. gical planning and stereotactic navigation,” IEEE Transac-
[106] R. Khlebnikov, B. Kainz, J. Muehl, and D. Schmalstieg, “Cre- tions on Visualization and Computer Graphics, vol. 14,
puscular rays for tumor accessibility planning,” IEEE Trans- no. 6, pp. 1587–1594, 2008.
actions on Visualization and Computer Graphics, vol. 17, [119] C. Rieder, F. Ritter, M. Raspe, and H.-O. Peitgen, “Interactive
no. 12, pp. 2163–2172, 2011. visualization of multimodal volume data for neurosurgical
[107] J. Beyer, M. Hadwiger, S. Wolfsberger, and K. Bühler, “High- tumor treatment,” Computer Graphics Forum, vol. 27, no. 3,
quality multimodal volume rendering for preoperative plan- pp. 1055–1062, 2008.
ning of neurosurgical interventions,” IEEE Transactions on [120] C. Dick, J. Georgii, R. Burgkart, and R. Westermann, “Stress
Visualization and Computer Graphics, vol. 13, no. 6, tensor field visualization for implant planning in orthope-
pp. 1696–1703, 2007. dics,” IEEE Transactions on Visualization and Computer
[108] C. Dick, R. Burgkart, and R. Westermann, “Distance visuali- Graphics, vol. 15, no. 6, pp. 1399–1406, 2009.
zation for interactive 3D implant planning,” IEEE Transac- [121] B. Preim and C. Botha, Visual Computing for Medicine: The-
tions on Visualization and Computer Graphics, vol. 17, ory, Algorithms, and Applications, Morgan Kaufmann, 2014.
no. 12, pp. 2173–2182, 2011. [122] D. Weiskopf, K. Engel, and T. Ertl, “Interactive clipping tech-
[109] C. Lundstrom, T. Rydell, C. Forsell, A. Persson, and niques for texture-based volume visualization and volume
A. Ynnerman, “Multi-touch table system for medical visuali- shading,” IEEE Transactions on Visualization and Computer
zation: application to orthopedic surgery planning,” IEEE Graphics, vol. 9, no. 3, pp. 298–312, 2003.
Transactions on Visualization and Computer Graphics, [123] Z. Shi, G. Zeng, L. Zhang et al., “Bayesian VoxDRN: a proba-
vol. 17, no. 12, pp. 1775–1784, 2011. bilistic deep voxelwise dilated residual network for whole

Downloaded from https://spj.science.org on September 26, 2024

[110] N. Smit, K. Lawonn, A. Kraima et al., “PelVis: atlas-based sur- heart segmentation from 3D MR images,” in Medical Image
gical planning for oncological pelvic surgery,” IEEE Transac- Computing and Computer Assisted Intervention – MICCAI
tions on Visualization and Computer Graphics, vol. 23, no. 1, 2018, A. F. Frangi, J. A. Schnabel, C. Davatzikos, C. Alber-
pp. 741–750, 2017. ola-López, and G. Fichtinger, Eds., pp. 569–577, Springer
[111] C. R. Butson, G. Tamm, S. Jain, T. Fogal, and J. Krüger, “Eval- International Publishing, Cham, 2018.
uation of interactive visualization on mobile computing plat- [124] X. Zhuang, L. Li, C. Payer et al., “Evaluation of algorithms for
forms for selection of deep brain stimulation parameters,” multi-modality whole heart segmentation: an open-access
IEEE Transactions on Visualization and Computer Graphics, grand challenge,” Medical Image Analysis, vol. 58, pp. 101–
vol. 19, no. 1, pp. 108–117, 2013. 537, 2019.
[112] J. Vorwerk, D. McCann, J. Krüger, and C. R. Butson, “Inter- [125] Y. Jin, G. Yang, Y. Fang et al., “3d pbv-net: an automated
active computation and visualization of deep brain stimula- prostate MRI data segmentation method,” Computers in Biol-
tion effects using duality,” Computer Methods in ogy and Medicine, vol. 128, pp. 104–160, 2021.
Biomechanics and Biomedical Engineering: Imaging & Visual-
[126] G. Litjens, T. Kooi, B. E. Bejnordi et al., “A survey on deep
ization, vol. 8, no. 1, pp. 3–14, 2020.
learning in medical image analysis,” Medical Image Analysis,
[113] A. Bock, N. Lang, G. Evangelista, R. Lehrke, and T. Ropinski, vol. 42, pp. 60–88, 2017.
“Guiding deep brain stimulation interventions by fusing mul-
timodal uncertainty regions,” in 2013 IEEE Pacific Visualiza- [127] R. Raidou, O. Casares-Magaz, L. Muren et al., “Visual analy-
tion Symposium (PacificVis), pp. 97–104, Sydney, NSW, sis of tumor control models for prediction of radiotherapy
Australia, 2013. response,” Computer Graphics Forum, vol. 35, no. 3,
pp. 231–240, 2016.
[114] T. M. Athawale, K. A. Johnson, C. R. Butson, and C. R. John-
son, “A statistical framework for quantification and visualisa- [128] N. Karall, M. E. Gröller, and R. G. Raidou, “ChemoExplorer:
tion of positional uncertainty in deep brain stimulation a dashboard for the visual analysis of chemotherapy response
electrodes,” Computer Methods in Biomechanics and Biomed- in breast cancer patients,” in EuroVis 2018 – Short Papers, J.
ical Engineering: Imaging & Visualization, vol. 7, no. 4, Johansson, F. Sadlo, and T. Schreck, Eds., pp. 49–53, The
pp. 438–449, 2019. Eurographics Association, 2018.
[115] J. Blaas, C. P. Botha, C. Majoie, A. Nederveen, F. M. Vos, [129] R. Raidou, O. Casares-Magaz, A. Amirkhanov et al., “Bladder
and F. H. Post, “Interactive visualization of fused fMRI runner: visual analytics for the exploration of RT-induced
and DTI for planning brain tumor resections,” in Medical bladder toxicity in a cohort study,” Computer Graphics
Imaging 2007: Visualization and Image-Guided Procedures, Forum, vol. 37, no. 3, pp. 205–216, 2018.
K. R. Cleary and M. I. Miga, Eds., vol. 6509 of Interna- [130] K. Furmanová, L. P. Muren, O. Casares-Magaz et al., “PRE-
tional Society for Optics and Photonics, , pp. 599–610, VIS: predictive visual analytics of anatomical variability for
SPIE, 2007. radiotherapy decision support,” Computers Graphics,
[116] S. Born, W. Jainek, M. Hlawitschka et al., “Multimodal visu- vol. 97, pp. 126–138, 2021.
alization of DTI and fMRI data using illustrative methods,” [131] S. Zachow, P. Muigg, T. Hildebrandt, H. Doleisch, and H.-
in Bildverarbeitung für die Medizin, pp. 6–10, Springer, Berlin C. Hege, “Visual exploration of nasal airflow,” IEEE Transac-
Heidelberg, 2009. tions on Visualization and Computer Graphics, vol. 15, no. 6,
[117] S. Diepenbrock, J.-S. Prassni, F. Lindemann, H.-W. Bothe, pp. 1407–1414, 2009.
and T. Ropinski, “2010 IEEE visualization contest winner: [132] M. Meuschke, S. Oeltze-Jafra, O. Beuing, B. Preim, and
interactive planning for brain tumor resections,” IEEE Com- K. Lawonn, “Classification of blood flow patterns in cerebral
puter Graphics and Applications, vol. 31, no. 5, pp. 6–13, aneurysms,” IEEE Transactions on Visualization and Com-
2011. puter Graphics, vol. 25, no. 7, pp. 2404–2418, 2019.
Health Data Science 19

[133] P. Rosen, B. Burton, K. Potter, and C. R. Johnson, “muView: a

visual analysis system for exploring uncertainty in myocar-
dial ischemia simulations,” in Visualization in Medicine and
Life Sciences III, L. Linsen, B. Hamann, and H.-C. Hege,
Eds., pp. 49–69, Springer International Publishing, Cham,
2016.
[134] M. Meuschke, S. Voß, O. Beuing, B. Preim, and K. Lawonn,
“Glyph‐based comparative stress tensor visualization in cere-
bral aneurysms,” Computer Graphics Forum, vol. 36, no. 3,
pp. 99–108, 2017.
[135] L. Zhou, C. R. Johnson, and D. Weiskopf, “Data-driven
space-filling curves,” IEEE Transactions on Visualization
and Computer Graphics, vol. 27, no. 2, pp. 1591–1600, 2021.
[136] R. MacLeod, C. Johnson, and M. Matheson, “Visualization of
cardiac bioelectricity-a case study,” in Proceedings of IEEE
Visualization Conference, pp. 411–418, Los Alamitos, CA,
USA, 1992.
[137] I. Demir, C. Dick, and R. Westermann, “Multi-charts for
comparative 3D ensemble visualization,” IEEE Transactions
on Visualization and Computer Graphics, vol. 20, no. 12,
pp. 2694–2703, 2014.

Downloaded from https://spj.science.org on September 26, 2024

[138] J. Weissenböck, B. Fröhler, E. Gröller, J. Kastner, and
C. Heinzl, “Dynamic volume lines: visual comparison of 3D
volumes through space-filling curves,” IEEE Transactions
on Visualization and Computer Graphics, vol. 25, no. 1,
pp. 1040–1049, 2019.
[139] J. Ahrens, B. Geveci, and C. Law, “36-ParaView: an end-user
tool for large-data visualization,” in Visualization Handbook,
C. D. Hansen and C. R. Johnson, Eds., pp. 717–731, Butter-
worth-Heinemann, Burlington, 2005.
[140] J. Meyer-Spradow, T. Ropinski, J. Mensmann, and
K. Hinrichs, “Voreen: a rapid-prototyping environment for
ray-casting-based volume visualizations,” IEEE Computer
Graphics and Applications, vol. 29, no. 6, pp. 6–13, 2009.
[141] D. Jönsson, P. Steneteg, E. Sundén et al., “Inviwo - a visuali-
zation system with usage abstraction levels,” IEEE Transac-
tions on Visualization and Computer Graphics, vol. 26,
no. 11, pp. 3241–3254, 2020.
[142] S. Grottel, M. Krone, C. Müller, G. Reina, and T. Ertl, “Mega-
Mol—a prototyping framework for particle-based visualiza-
tion,” IEEE Transactions on Visualization and Computer
Graphics, vol. 21, no. 2, pp. 201–214, 2015.
[143] P. Gralka, M. Becher, M. Braun et al., “MegaMol – a compre-
hensive prototyping framework for visualizations,” The Euro-
pean Physical Journal Special Topics, vol. 227, no. 14,
pp. 1817–1829, 2019.
[144] A. Fedorov, R. Beichel, J. Kalpathy-Cramer et al., “3D Slicer
as an image computing platform for the Quantitative Imag-
ing Network,” Magnetic Resonance Imaging, vol. 30, no. 9,
pp. 1323–1341, 2012.
[145] T. Fogal and J. Krüger, “Tuvok, an architecture for large scale
volume rendering,” in Proceedings of the 15th International
Workshop on Vision, Modeling, and Visualization, pp. 139–
146, Siegen, Germany, 2010.