Automated Multi-Modality Image Registration Based On Information Theory
Automated Multi-Modality Image Registration Based On Information Theory
1. Introduction
Rigid body registration is a fundamental task in three-dimensional (3-D)
medical image processing that serves to align two or more 3-D scans that
were taken at a di erent time, or generated on di erent medical imaging
modalities. There are numerous applications. Medical diagnosis bene ts
from the complementarity of the information in images of di erent modal-
ities. During radiation therapy planning dose calculation needs to be done
based on the CT data while tumor outlining is often much better performed
in the corresponding MR scan. The analysis of brain function requires the
correlation of physiological data from PET data with anatomical data from
the corresponding MR data, etc.
A general overview of image registration techniques has been compiled by
Brown [1]. Two other articles survey image registration techniques and their
264 A. COLLIGNON ET AL.
applications in medical imaging [9], [13]. From these surveys we learn that
the bulk of registration algorithms are either frame based, point landmark
based (using external and/or internal landmarks), surface based, or voxel
based. Point landmark and surface based registration algorithms are typ-
ically used for the alignment of anatomical scans, while voxel similarity
based registration algorithms seem to be more robust for alignment prob-
lems involving functional scans ([7] and [15]), although many people are
using surface based registration algorithms in the latter case also.
Voxel based (VB) registration algorithms optimize a function measuring
the similarity of all possible pairs of voxel grey-values (or derived features,
e.g. gradients). The main advantage of VB methods is that feature calcula-
tion is straightforward or even absent when only grey-values are used and
thus, in theory, accuracy is not limited by segmentation errors as in surface
based algorithms. Simple correlation of grey-values works only for di erent
modalities if there exists a linear mapping between grey-values [1], which is
unfortunately not always the case. Nonetheless, interest in VB registration
algorithms has recently revived by the success of Woods' AIR package in
clinical practice [16], which has been made more exible by Hill et al [7].
Their basic assumption is that the grey-value mapping should be a func-
tion (e.g. to each MR grey-value correspondends at most one PET-value).
They accomplish this heuristically by demanding that a global measure of
the variance of the grey-value ratios, with the function's domain variable
as the ratios' denominator, be minimal. We recently compared these and
some other VB matching criteria [2]. All of them are heuristic in nature.
In the same work we rst suggested an information theoretic approach to
the design of a non-heuristic multi-modality matching criterion and we pro-
posed to use the entropy of the joint probability distribution p(g1 ; g2 ) of
the combined grey-values of all common voxel pairs in the two images to
be matched:
X
E (g1 ; g2 ) = , p(g1 ; g2 ) log (p(g1 ; g2 )) (1)
g1 ;g2
TABLE 1. This table gives the registration error for our algorithm when
applied to an MR and its transformed version MRT . Transformation pa-
rameters were (10.39, -3.27, 2.03) degrees and (7.06, 1.21,18.26) mm.
Voxels are (1.33 x 1.33 x 4.0) mm3 in both images. Subsample \3.0/1.0"
means that only 1 in 3 voxels are sampled in X and Y direction of the
image planes, and that all voxels are used in the perpendicular Z di-
rection. The \error"-column gives us an arithmetic average registration
error for all voxels in the central quarter of all slicesT of the MR volume.
Binning parameter values were: ni = 16. For MR : gmax = 2550 and
gmin = 0. For MR: gmax = 3359 and gmin = 2.
alistic than the one in the previous section, because: 1) the noise will be
di erent now, 2) because of the Gd-enhancement, and 3) due to the visi-
bility of pathology (a tumor) in the MRE volume. A true multi-modality
test is of course the registration of CT with MR. Table 2 gives the results.
TABLE 2. Same as table 1, but for MRT combined with MRE. For MRE:
gmax = 3582 and gmin = 2. For CT: gmax = 4094 and gmin = 2.
Figure 1. Left: MRI/CT combination registered using mutual information with partial
volume distribution, ni = 16, sampling factors equal to 1. Right: PET/MRI combination
of an MS patient. The exact registration solution is unknown. This result was found to
be clinically acceptable.
robust enough to handle this type of deviations from the one-to-one map-
ping assumption. Therefore, his algorithm requires prior segmentation of
the brain from the MR image before application of his matching criterion to
that part of the image. After careful inspection of the dispersion behaviour
of the grey-value scatter plot Hill et al propose to select only speci c bins
of the scatter plot to obtain meaningful dispersion measurements. Their
approach can be automated but in a data dependent manner only. Our
rst attempt in [2] to increase robustness in a data independent way was
based on an intuitive information theoretic reasoning only: the scatter-plot
entropy as a measure of multi-modality information content or signal com-
plexity has all the properties of a data independent measure of scatter-plot
dispersion. The main problem with the entropy matching criterion, how-
ever, is its sensitivity to the problem of partial overlap, and, even worse,
also to the problem that the information content of the separate overlap-
ping parts of the images is not constant, causing it to have multiple local
272 A. COLLIGNON ET AL.
minima all over -space. In this paper we have solved the latter problem
also using relative entropy.
The results from sections 3.1 and 3.2 are all subvoxel accurate if the largest
voxel size (mostly equals the slice distance) is taken as a reference, and if
the stereotactic registration solutions are taken to be correct. The latter
assumption was clearly not true for the pair of CT/MR volumes in section
3.2. In this case visual inspection showed that our mutual information based
registration solution is better than the stereotactic one, which was particu-
larly noticable in the caudal-cranial direction. This can be explained by the
fact that the stereotactic localiser was xed to the head by means of a mask
and not by screws. Taking into consideration the nature of the data in these
experiments we may also conclude that the mutual information is inherently
robust against noise and local di erences (e.g. pathological). Comparison
of the results for the di erent interpolation methods tells us that nearest
neighbour interpolation o ers a very rapid solution with an acceptable,
near subvoxel accuracy, even when it is used with a subsample of the oat-
ing volume, f2 . All experiments with nearest neighbour interpolation and
sub-sampling were performed in less than 7 minutes. The trilinear partial
volume distribution method was especially designed to overcome some edge
e ects, and e ects of image grid interference, that are both inevitable when
calculating the mutual information of a nite sample. Therefore, it does not
go well with sub-sampling, which explains why no sub-sampling was used in
tables 1 and 2. When the same tests were performed using the sampling fac-
tors that were used with nearest neighbour and trilinear interpolation the
search algorithm got stuck in a local minimum far from the registration so-
lution. Our implementation of the trilinear partial volume distribution has
approximately the same complexity as that of trilinear interpolation which
was about twice as slow as nearest neighbour interpolation. This factor two
together with the di erences in the in-plane sampling factor explains why
the partial volume distribution based experiments in tables 1 and 2 took 90
minutes. The results from section 3.3 show us that it is possible to register
images of di erent modalities without using any prior knowledge about the
grey-value mapping between parts of the image, and yet obtain essentially
the same registration results. E.g. both methods proposed by van den Elsen
require the application of feature extraction algorithms to make simple cor-
relation work. This approach makes her algorithms data dependent. The
results obtained in section 3.4 have not yet been compared to that of other
algorithms. However, our algorithm is more user-friendly than Woods' AIR
software [16] because it does not require the prior segmentation of the brain
surface from the MR images. Our algorithm does not require the selection
of a part of the histogram, as does Hill's more robust version of Woods'
AUTOMATED MULTI-MODALITY IMAGE REGISTRATION 273
algorithm.
In all our experiments so far, using the search procedure described in
section 2.2.4 we have observed translations up to 2:5 cm and rotations up
to 11o that were all recovered fully automatically. Thus, for our practical
purposes the mutual information criterion o ers a well behaved optimi-
sation function in a reasonably large region around the initial estimate
of the registration parameter values. In theory we need to prove that it is
strictly quasi-convex to guarantee its unimodality. Further experiments will
be needed in the future to obtain an estimate of the range of quasi-convexity
around the global optimum.
5. Conclusion
The new voxel similarity based (VB) registration algorithm has been ap-
plied successfully to real world images of the human head (CT, MR, and
PET volumes) which satisfy the rigid body assumption well. Comparison
of the registration outcome with that of stereotactic and 2 other VB regis-
tration solutions indicates that mutual information is a subvoxel accurate
matching criterion and that it is robust in multi-modality situations.
The experimental results indicate that the mutual information matching
criterion is a well-behaved multi-modality 3-D registration criterion, and
that we have found satisfactory solutions for all implementation problems.
Most importantly, we have found that due to the niteness of the sample
of grey-value pairs involved in the calculation of the mutual information
matching criterion the use of partial volume distribution of grey-values is
required to obtain the most accurate results.
6. Acknowledgments
This work is part of COVIRA (Computer Vision in Radiology), project
A2003 of the AIM (Advanced Informatics in Medicine) programme of the
European Commission. Special thanks go to prof. C. Schiepers, and dr. P.
Dupont from the PET Center of the Department of Nuclear Medicine, at
the University Hospital Gasthuisberg of Leuven for supplying us with the
PET/MRI data and for visually inspecting our results. Special thanks also
go to Petra van den Elsen for our informal cooperation. Her volumetric
MRI was acquired in the Hospital \Medisch Spectrum" in Enschede, with
special thanks to dr. G. Wilts. The high resolution CT was acquired in
Utrecht University Hospital by prof. dr. F.W. Zonneveld, Drs. D.N. Velis,
and P.A. van den Elsen coordinated the multimodal acquisition.
274 A. COLLIGNON ET AL.
References
1. L.G. Brown, "A Survey of Image Registration Techniques", ACM Computing Sur-
veys, Vol. 24, No. 4, pp. 325-376, December 1992.
2. A. Collignon, D. Vandermeulen, P. Suetens, G. Marchal, "3D Multi-Modality Med-
ical Image Registration Using Feature Space Clustering", accepted for presentation
at the CVRMed'95 conference on april 3-5, 1995, and publication in the Lecture
Notes in Computer Science series, Springer-Verlag.
3. T.M. Cover, J.A. Thomas, Elements of Information Theory, John Wiley & Sons,
Inc., New York, 1991.
4. AIM Programme, Project A2003, COVIRA Computer Vision in Radiology, Deliver-
able (2) D1/1.1 Set of raw images and clinical description, 26 June 1992, distributed
by the Commission of the European Communities DG XIII, Brussels.
5. R.O. Duda, P.E. Hart, Pattern Classi cation and Scene Analysis, Stanford Research
Institute, Menlo Park, CA, USA, A Wiley-Interscience Publication, 1973.
6. P. Dupont, G.A. Orban, R. Vogels, G. Bormans, J. Nuyts, C. Schiepers, M. De
Roo, L. Mortelmans, \Di erent Perceptual Tasks Performed with the Same Visual
Stimulus Attribute Activate Di erent Regions of the Human Brain: A Positron
Emission Tomography Study", Proc. Natl. Acad. Sci., USA, Vol. 90, pp. 10927-
10931, December 1993, Neurobiology.
7. D.L.G. Hill, D.J. Hawkes, N.A. Harrison, C.F. Ru , \A Strategy for Automated
Multimodality Image Registration Incorporating Anatomical Knowledge and Im-
ager Characteristics", 13th Int. Conf., Information Processing in Medical Imaging,
IPMI'93, Flagsta , Arizona, USA, June 1993, Lecture Notes in Computer Science,
Vol. 687,pp. 182-196, Springer-Verlag, Berlin Heidelberg.
8. D.L.G. Hill, C. Studholme, D.J. Hawkes, "Voxel Similarity Measures for Automated
Image Registration", Proc. SPIE Vol. 2359, Visualization in Biomedical Computing,
SPIE Press, Bellingham, WA, Rochester, MN, October 4-7, 1994.
9. C.R. Maurer, J.M. Fitzpatrick, "A Review of Medical Image Registration", Inter-
active Image-Guided Neurosurgery, R.J. Maciunas (Ed), Park Ridge, IL, American
Association of Neurological Surgeons, 1993, pp. 17-44.
10. W.H. Press, B.P. Flannery, S.A. Teukolsky, W.T. Vetterling, Numerical Recipes in
C, The Art of Scienti c Computing, Cambridge University Press, 1988.
11. I. Vajda, Theory of Statistical Inference and Information, Kluwer Academic Pub-
lisher, Dordrecht, 1989.
12. P.A. Van den Elsen, Multimodality matching of brain images, Ph.D. Thesis, Utrecht
University Thesis, June 1993.
13. P.A. van den Elsen, E-J.D. Pol, M.A. Viergever, "Medical Image Matching - A
Review with Classi cation", IEEE Eng. in Medicine and Biol., pp. 26-38, March
1993.
14. P.A. Van den Elsen, E.J.D. Pol, T.S. Sumanaweera, P.F. Hemler, S. Napel, J. Adler,
\Grey value correlation techniques used for automatic matching of CT and MR brain
and spine images", Proc. SPIE Vol. 2359, Visualization in Biomedical Computing,
pp. 227-237, SPIE Press, Bellingham, WA, Rochester, MN, October 4-7, 1994.
15. R.P. Woods, J.C. Mazziotta, S.R. Cherry, "MRI-PET Registration with Automated
Algorithm", JCAT, 17(4):536-546, July/August, 1993.
16. R.P. Woods, Automated Image Registration (AIR): User's Manual and Program
Documentation, Division of Brain Mapping Neuropsychiatric Institute, UCLA
School of Medicine, 8/7/1993.