EUROGRAPHICS 2011 / Short Paper

Interactive Visualization Techniques

for Neurosurgery Planning

Stefan Diepenbrock†1 , Jörg-Stefan Praßni1 , Florian Lindemann1 , Hans-Werner Bothe2 , Timo Ropinski1

1 University of Münster, 2 University Hospital Münster

Abstract
We present concepts for pre-operative planning of brain tumor resections. The proposed system uses a combination
of traditional and novel visualization techniques rendered in real-time on modern GPUs in order to support neu-
rosurgeons during intervention planning. A set of multimodal 2D and 3D renderings conveys the relation between
the lesion and the various structures at risk and also depicts data uncertainty. To facilitate efficient interactions
while providing a comprehensible visualization, all employed views are linked. Furthermore, the system allows
the surgeon to interactively define the access path by clicking in the 3D views as well as to perform distance
measurements in 2D and 3D.
Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Picture/Image
Generation—Three-Dimensional Graphics and Realism

1. Introduction
We propose an application prototype for neurosurgical plan-
ning that supports a large variety of modalities (fMRI, DTI,
T1 pre, T1 post, T2, FLAIR, SWI, CT). Our application
integrates proven solutions as 2D slice rendering with state
of the art techniques and some novel approaches, into a
workflow leading towards finding an optimal access path for
neurosurgery. Our focus are computer graphics techniques
utilized to enable these visualizations, including multimodal
volume raycasting with enhanced shading techniques for
brain rendering, two novel projection techniques, enhanced Figure 1: Exploring the data and planning an initial access
lift charts and uncertainty visualization techniques for DTI path can be done by exploiting multimodal 3D and 2D views
and fMRI. We will also discuss the feedback received from (a). Furthermore, the tumor view allows a close-up inspec-
domain experts, who evaluated the clinical value of the tion of the vicinity of the resection region (b). (Workflow Step
proposed application. 1)

2. Workflow
First, the initial investigation of the data in combination with
In cooperation with our medical partner, we have identi- an interactive access path specification. Second, the deeper
fied two main steps for the pre-operative planning workflow. analysis of the chosen access path and the actual preparation
for the surgery. Figures 1 and 2 show the views exploited in
workflow step 1 and workflow step 2. During the planning,
† {diepenbrock,j-s.prassni,f_lind03,hwbothe,timo.ropinski}@uni- the surgeon can always go back from step 2 to step 1 in order
muenster.de to choose an entirely new access path.

c The Eurographics Association 200x.

 S. Diepenbrock et al. / Interactive Visualization Techniquesfor Neurosurgery Planning

Figure 3: Enhanced lift charts indicate the current position

in the slice stack and display the amount of malignant tissue
(red) as well as fMRI signal (yellow).

Figure 2: In-detail inspection as well as modification of the

access path and operation preparation by exploiting a probe
3. Uncertainty Extraction
view, a cylindrical access path projection, a surgeon micro-
scope slice view and an access path distance plot. (Workflow To deal with the uncertainty introduced through DTI, we in-
Step 2) corporate the fiber context as well as the fiber anisotropy.
Since DTI is less certain in regions near bone or air, we have
applied a volume analysis that first applies a masking in or-
der to extract bone and air structures. Based on this mask-
ing volume we perform a distance transformation that com-
putes the distance d to these structures for each fiber seg-
ment. Thus, a user-defined security margin can be defined
As can be seen in Figure 1 (a), in the first step we com- around air and bone structures. We normalize the computed
bine classical 2D slice views with a 3D context view that in- distance to obtain a structure uncertainty US . To get the fi-
tegrates relevant information from the available modalities. nal uncertainty for a fiber segment, we combine US with the
The 2D slice views provide insight into structures and allow anisotropy uncertainty UA , to obtain the overall uncertainty
the neurosurgeon to identify structures inside the tumor and U = max(US ,UA ).
diagnose its type. The location of the tumor as well as its
relation to risk structures is depicted in the 3D view, which
4. Visualization
is further overlaid with a tumor map showing a projection of
relevant structures as seen from the tumor. As shown in Fig- 2D and 3D Views. The 2D slice views integrated into our
ure 1 (b), an additional close-up view of the tumor and the system are standard slice views, which have been extended
relevant adjacent structures is available to identify vessels by using enhanced lift charts [TMS∗ 06]. To quickly identify
and fibers that may be infiltrated or displaced by the tumor. the most important structures, we overlay one chosen modal-
Using these views the neurosurgeon can place one or more ity with the tumor segmentation mask, the fMRI signal and
access paths before analyzing and comparing them in step 2. the vessels. Within the lift charts (see Figure 3), we depict
for each slice the extent of malignant tissue (red curve) as
In the second step (see Figure 2) we exploit a classical well as the fMRI signal (yellow curve). The current slice in
probe view (top left), which shows all structures inside the the stack is also indicated to help the user to navigate through
access path and provides a preview of how the access path the slices.
would look like during operation. It also enables the sur-
To generate high-quality 3D views, we exploit GPU-
geon to fine tune the previously chosen access path. To lo-
based volume ray-casting [KW03], which has been shown
cate structures close to the access path the top right view
[SHC∗ 09] to generate images superior to other volume ren-
displays a projection of the structures surrounding the ac-
dering techniques. Our multi volume raycaster is inspired
cess path onto the surface of the access path cylinder. A plot
by the approach presented by Lindholm et al. [LLHY09]. In
showing the minimum distances of relevant structures along
order to integrate the fiber geometry into the 3D views, we
the access path is located at the bottom right to allow an easy
have modified the exit points used by our GPU-based ray-
comparison of several possible paths. Finally, in the bottom
caster, as proposed by Scharsach [Sch05] (see Figure 4).
left the system offers a slice view that is centered around the
access path and oriented perpendicular to it, thereby provid- Because gradients in MRI scans are unreliable due to
ing a view corresponding to the operation microscope focus- noise, we use a distance based darkening (dark means deep)
ing at a certain depth. In the following we will discuss the and depth darkening [LCD06] to render the brain and simu-
preprocessing, visualization and interaction techniques we late the effects of a global illumination model with a mini-
employed and give a short evaluation based on the feedback mal performance impact. We render the brain without shad-
we received from neurosurgeons. ing (see Figure 5 (b)) and then apply the depth darkening to

c The Eurographics Association 200x.

 S. Diepenbrock et al. / Interactive Visualization Techniquesfor Neurosurgery Planning

Figure 4: Integration of geometry and single volume ray-

casting results into the EEPs of our multi volume raycaster. (a) (b) (c)
Figure 5: Comparing different techniques to shade the brain
(using the same transfer-function): Gradient based shading
(a), no shading (b), depth darkening and dark means deep
the image. The resulting image depicts the structures of the (c).
brain more comprehensible manner (compare Figure 5 (a)
and (c)). We then integrate the rendering of the brain into our
multi volume raycasting using the EEP modification tech-
tions can calculated by simply passing spherical/cylindrical
nique we employed for geometry integration (see Figure 4).
Entry-Exit-Points (EEPs) to the raycaster (see Figure 6). Be-
There are three different 3D views provided by our sys- cause OpenGL does not allow for easy spherical/cylindrical
tem. The 3D context view, shown during the first step of projections we use an OpenCL kernel to raytrace the Proxy-
the workflow, integrates all relevant modalities into a com- Geometry of one or multiple volumes and generate these
prehensible rendering. It can be used to identify the various EEPs.
structures at risk and understand how they relate and interact Tumor Map. The tumor map, which is overlaid on the 3D
with each other. To prevent cluttering, all modalities can be context view, is inspired by the projection type presented by
easily deactivated through on-screen buttons (see Figure 1 Rieder et al. [RWS∗ 10]. The tumor map provides all neces-
(top right)). Additionally, the user can specify a region of in- sary information that is required to identify the target loca-
terest based on the distance to the tumor surface for the ves- tion of the access path, which can be done by clicking on the
sel volume. Moreover, the 3D context view allows the sur- tumor map. Distances to structures at risk, such as vital func-
geon to efficiently define the access path by using the mouse. tional areas or the pyramidal tract, can easily be measured by
After the access path has been defined, a 3D probe view is moving the mouse over the tumor map. The measured dis-
exploited during the second step of the workflow in order tances are also displayed in the context view (see Figure 2
to identify all related risk-structures lying along the chosen (top left)). We generate the map by performing two spheri-
path. This probe view allows the physician to see the access cal raycastings from the center of the tumor. First, we ren-
path in a similar way as during the resection. Furthermore, der the tumor mask using an inverted transfer function (i.e.,
the surgeon may use the ring widget (see Figure 2 (top left)) the inside of the tumor is transparent, the rest is opaque).
to rotate the head around the fixed access path axis in order to We then use the first-hit points as entry points for a second
simulate the possible orientations during the resection. The raycasting of all risk-structures. By calculating the distance
bigger ring marker can be used as rotation widget, while the between entry- and first-hit points for this raycasting we get
smaller one indicates the direction pointing to the patient’s a distance map to which we apply a red-blue color mapping.
nose.
Access Path Projection. Besides their distance to the tu-
The third 3D view is the tumor view, which focusses on mor, the distance of structures at risk to the access path is
the spatial relation between the tumor and nearby structures also important. Therefore, the access path projection shows
at risk. As the context view, it also integrates all relevant the distance to all relevant structures as seen from the access
modalities, but displays a close-up showing only the tumor path (see Figure 7 (right)). The result of this cylindrical pro-
and structures in the proximity of the tumor. Thus, the tumor jection is mapped to a disc that is rendered in a view, which
view can be used to support the surgeon when analyzing how is linked to the probe view. The center of the projection rep-
close the tumor is to vital functional areas. The surgeon can resents the deeper end of the access path. As with the tumor
especially analyze which fibers infiltrate the tumor to what map, moving the mouse cursor over a red region of the map
extent, or which fibers are displaced. will automatically measure the distance to the structure at
risk and display it in the probe view (see Figure 7 (left)). For
4.1. Projection Techniques the access path projection we also display the distance along
the access path, which is an important information for the
To provide informative views of the structures at risk, we ex- surgeon.
ploit two projection techniques. One, showing the structures
close to the tumor and one showing the structures close to
4.2. Uncertainty Visualization
the access path. For both projection techniques we employ
a red-blue color mapping, where structures at risk nearby DTI Uncertainty. When visualizing the DTI fiber tracts,
are displayed in red. Using a volume raycaster these projec- we incorporate the derived uncertainty information intro-

c The Eurographics Association 200x.

 S. Diepenbrock et al. / Interactive Visualization Techniquesfor Neurosurgery Planning

6. Evaluation
Our application prototype was reviewed as an entry in the
IEEE Visualization Contest 2010. Two of the five review-
ers were neurosurgeons and the clinical value was rated as
high (7 and 9 out of 9). We have indepedantly shown our
application to several domain experts and got very positive
Figure 6: Cylindrical entry- and exit points (a), result of feedback about the integration of a wide range of modalities
the cylindrical raycasting (first hit points) (b), resulting dis- into an intuitive 3D visualization. The enhanced lift charts
tances color coded and mapped to a disc (c). were also appreciated for providing a simple indication of
the current position in the slice stack.

References
[KW03] K RÜGER J., W ESTERMANN R.: Acceleration tech-
niques for GPU-based volume rendering. In IEEE Vis (2003),
pp. 287–292. 2
[LCD06] L UFT T., C OLDITZ C., D EUSSEN O.: Image enhance-
ment by unsharp masking the depth buffer. ACM Transactions on
Graphics 25, 3 (jul 2006), 1206–1213. 2
[LLHY09] L INDHOLM S., L JUNG P., H ADWIGER M., Y NNER -
MAN A.: Fused Multi-Volume DVR using Binary Space Par-
Figure 7: Access Path Projection: The measured distances
titioning. In Computer Graphics Forum (2009), vol. 28, John
are displayed in the probe view using linking. Wiley & Sons, pp. 847–854. 2
[NYE∗ 10] N GUYEN T. K., Y NNERMAN A., E KLUND A.,
L JUNG P., H ERNELL F., O HLSSON H., A NDERSSON M.,
K NUTSSON H., F ORSELL C.: Concurrent volume visualization
duced in Subsection 3. We encode the uncertainty in the sat-
of real-time fmri. In IEEE/EG Volume Graphics (2010). 4
uration and value of the displayed fiber color in the HSV
[RWS∗ 10] R IEDER C., W EIHUSEN A., S CHUMANN C., Z ID -
color space. While we in general use the standardized direc- OWITZ S., P EITGEN H.-O.: Visual support for interactive post-
tional fiber color-mapping MDs are used to, we lower the interventional assessment of radiofrequency ablation therapy. In
saturation and value in regions of high uncertainty (see Fig- EuroVis (2010). 3
ure 8). Thus, uncertain fibers become less emphasized and [Sch05] S CHARSACH H.: Advanced GPU raycasting. Proceed-
their orientation, which can also be considered as less cer- ings of CESCG 5 (2005), 67–76. 2
tain, is less prominent. [SHC∗ 09] S MELYANSKIY M., H OLMES D., C HHUGANI J.,
L ARSON A., C ARMEAN D., H ANSON D., D UBEY P., AUGUS -
fMRI Uncertainty. To depict the uncertainty of the fMRI TINE K., K IM D., K YKER A., ET AL .: Mapping high-fidelity
signal, we have applied an approach inspired by the work of volume rendering for medical imaging to cpu, gpu and many-
Nguyen et al. [NYE∗ 10]. We display the core of each fMRI core architectures. IEEE Vis (2009), 1563–1570. 2
region by exploiting a diffusely emitting light signal. Addi- [TMS∗ 06] T IETJEN C., M EYER B., S CHLECHTWEG S., P REIM
tionally, in order to express the uncertainty regarding the size B., H ERTEL I., S TRAUSS G.: Enhancing slice-based visuaiza-
of theses regions, we add an uncertainty margin, depicted by tions of medical volume data. In EuroVis (2006), pp. 123–130.
2
orange borders (see Figure 8). To generate this visualization
we render the fMRI signal twice: In the first pass we render
the core regions using a higher threshold, while in the second
pass we use a lower threshold and apply an edge detection
filter. We than composit this border-image with the result of
the first pass.

5. Interaction Techniques
A detailed exploration of the different modalities would not
be possible without exploiting sophisticated interaction tech-
niques. Therefore, we have integrated several interaction
techniques which support the mental linking of the different Figure 8: Uncertainty visualization. fMRI (green inset):
views as well as a deeper understanding of the data. Thus, Core regions are rendered using a diffusely emitting light
the surgeon can measure distances in the image, specify and signal, uncertainty borders are rendered in orange. DTI (red
alter the access path and interactively navigate through all inset): Fibers close to bone and air are rendered with less
3D views. The whole set of interaction techniques is demon- saturation and brightness to mark them as uncertain.
strated in the accompanying video.

c The Eurographics Association 200x.