Computational Vision and Medical Image Processing.

VipIMAGE 2015 contains invited

lectures and full papers presented at VIPIMAGE 2015 - V ECCOMAS Thematic Conference on
Computational Vision and Medical Image Processing (Tenerife, Canary Islands, Spain, 19-21
Computational Vision and
Medical Image Processing
October, 2015). International contributions from 19 countries provide a comprehensive coverage
of the current state-of-the-art in the fields of:

3D Vision;
Computational Bio- Imaging and Visualization;

Computational Vision and

Medical Image Processing
Computational Vision;
Computer Aided Diagnosis, Surgery, Therapy and Treatment;
Data Interpolation, Registration, Acquisition and Compression;
Industrial Inspection;
Image Enhancement;
Image Processing and Analysis;
VIPIMAGE 2015
Image Segmentation;
Medical Imaging;
Medical Rehabilitation;
Physics of Medical Imaging;
Shape Reconstruction;
Signal Processing;
Simulation and Modelling;
Software Development for Image Processing and Analysis;
Telemedicine Systems and their Applications;
Tracking and Analysis of Movement and Deformation;
Virtual Reality.

Computational Vision and Medical Image Processing. VipIMAGE 2015 will be useful
to academics, researchers and professionals in Biomechanics, Biomedical Engineering,
Computational Vision (image processing and analysis), Computer Sciences, Computational
Mechanics, Signal Processing, Medicine and Rehabilitation.

Sponsored by

Joao Manuel R.S.Tavares

R.M. Natal Jorge
EDITORS
an informa business
 COMPUTATIONAL VISION AND MEDICAL IMAGE PROCESSING V

VIPIMAGE_FM.indd i 9/9/2015 7:30:13 PM

This page intentionally left blank
 PROCEEDINGS OF VIPIMAGE 2015 – V ECCOMAS THEMATIC CONFERENCE ON
COMPUTATIONAL VISION AND MEDICAL IMAGE PROCESSING, TENERIFE, SPAIN, 19–21
OCTOBER 2015

Computational Vision and Medical

Image Processing V

Editors
João Manuel R.S. Tavares & R.M. Natal Jorge
Faculdade de Engenharia, Universidade do Porto, Porto, Portugal

VIPIMAGE_FM.indd iii 9/9/2015 7:30:13 PM

 CRC Press/Balkema is an imprint of the Taylor & Francis Group, an informa business

© 2016 Taylor & Francis Group, London, UK

Typeset by V Publishing Solutions Pvt Ltd., Chennai, India

Printed and bound in Great Britain by CPI Group (UK) Ltd, Croydon, CR0 4YY

All rights reserved. No part of this publication or the information contained herein may be reproduced,
stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical,
by photocopying, recording or otherwise, without written prior permission from the publisher.

Although all care is taken to ensure integrity and the quality of this publication and the information
herein, no responsibility is assumed by the publishers nor the author for any damage to the property or
persons as a result of operation or use of this publication and/or the information contained herein.

Published by: CRC Press/Balkema

P.O. Box 11320, 2301 EH Leiden, The Netherlands
e-mail: [email protected]
www.crcpress.com – www.taylorandfrancis.com

ISBN: 978-1-138-02926-2 (Hbk)

ISBN: 978-1-315-64279-6 (eBook PDF)

VIPIMAGE_FM.indd iv 9/9/2015 7:30:13 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Table of contents

Acknowledgements ix
Preface xi
Invited lectures xiii
Scientific committee xv

Invited lectures
An adaptive non-rigid image registration technique using hierarchical B-splines 3
A. Pawar, Y. Zhang, X. Wei, Y. Jia, T. Rabczuk, C.L. Chan & C. Anitescu
Medical image segmentation using Object Shape Models: A critical review on recent trends,
and alternative directions 9
A.X. Falcão, T.V. Spina, S.B. Martins & R. Phellan
Primal-dual method for continuous max-flow approaches 17
K. Wei, X.-C. Tai, T.F. Chan & S. Leung
Image restoration using variational approaches: Some recent advances 25
A. Lanza, S. Morigi & F. Sgallari
Virtual and augmented medical imaging environments: Applications to simulation, training,
surgical planning and interventional guidance 33
C.A. Linte

Contributed papers
The fast method of creating High Dynamic Range image for fluorescent microscopy
applications 41
A. Bal
Automatic cheek detection in digital images 49
M. Frackiewicz, H. Palus & K. Radlak
A variational model for image fusion with simultaneous cartoon and texture
decomposition 57
M. Dodangeh, I.N. Figueiredo & G. Gonçalves
Image contrast enhancement using split Bregman method 63
S.Gh. Bardeji, I.N. Figueiredo & E. Sousa
Extraction of spectral drowsy component from the resting electroencephalographic signal
for quick, objective and direct testing of sleepiness in absolute terms 69
A.A. Putilov, O.G. Donskaya & E.G. Verevkin
The illusion of a blackbody at the human ear and the human temperature measurement 75
A. Cardoso
Video-based Turkish Sign Language recognition systems 81
M. Aktaş & E.B. Sonmez

VIPIMAGE_FM.indd v 9/9/2015 7:30:13 PM

 A nonsmooth nonconvex sparsity-promoting variational approach for deblurring images
corrupted by impulse noise 87
A. Lanza, S. Morigi & F. Sgallari
Classification-based blood vessel segmentation in retinal images 95
J. Odstrcilik, R. Kolar, V. Harabis & R.P. Tornow
Line extraction via phase congruency with a novel adaptive scale selection for Poisson
noisy images 101
V.A. Krylov & J.D.B. Nelson
Diagnosis of human intestinal parasites by deep learning 107
A.Z. Peixinho, S.B. Martins, J.E. Vargas, A.X. Falcão, J.F. Gomes & C.T.N. Suzuki
Texture-energy features for microaneurysms detection 113
D. Veiga, N. Martins, C. Pereira, M. Ferreira & J. Monteiro
An iterative algorithm for Total Variation minimization in DBT imaging 119
A.M. Mota, N. Matela, N. Oliveira & P. Almeida
No-reference wavelet based Retinal Image Quality Assessment 123
L.S. Abdel Hamid, A. El-Rafei, S. El-Ramly, G. Michelson & J. Hornegger
Vessel segmentation of retinal images with fuzzy morphology 131
P. Bibiloni, M. González-Hidalgo & S. Massanet
Identification of subendocardial infarction—a feasibility study using synthetic ultrasonic
image data of a left ventricular model 137
J. Żmigrodzki, S. Cygan, B. Leśniak-Plewińska & K. Kałużyński
Microcalcification segmentation in full field digital mammography 143
N. Martins, D. Veiga, C. Pereira, M. Ferreira, N. Alves & M. Delgado
Unsupervised delineation of the vessel tree in retinal fundus images 149
N. Strisciuglio, M. Vento, G. Azzopardi & N. Petkov
A registration algorithm for microscopy images of the capillary bed 157
H. Rieiro, J.L. Alba Castro, S. Martinez-Conde & S.L. Macknik
Bilateral filtering based biomedical image colorization 163
A. Popowicz & B. Smolka
Comparison of the internal structures of bones by microtomography 171
J.C.L. Stutz, J.S. Domínguez & J.T. de Assis
Model adaptation for mesh generation of biological structures 175
A. Ramos-de-Miguel, R. Montenegro & J.M. Escobar
Medical volume rendering based on gradient information 181
T.F. de Moraes, P.H.J. Amorim, J.V.L. da Silva, H. Pedrini, M.I. Meurer
Chaos theory-based quantification of ROIs for mammogram classification 187
J. Kurek, B. Świderski, S. Dhahbi, M. Kruk, W. Barhoumi, G. Wieczorek &
E. Zagrouba
M5L: A web-based Computer Aided Detection system for automated search of lung
nodules in thoracic Computed Tomography scans 193
A. Traverso, M. Agnello, P. Cerello, M. Saletta, S. Bagnasco, C. Peroni, E. Fiorina,
M.E. Fantacci, A. Retico & E. Lopez Torres
Non-intrusive and calibration free visual exploration analysis in children with Autism
Spectrum Disorder 201
D. Cazzato, F. Adamo, G. Palestra, G. Crifaci, P. Pennisi, G. Pioggia, L. Ruta,
M. Leo & C. Distante
Semi-automatic tumor contouring method using PET and MRI medical images 209
S. Urbán, L. Ruskó & A. Nagy

vi

VIPIMAGE_FM.indd vi 9/9/2015 8:56:36 PM

 Automatic detection of bones based on the confidence map for Rheumatoid
Arthritis analysis 215
K. Radlak, N. Radlak & B. Smolka
Improved computer recognition of Fuhrman grading system in analysis
of Clear-Cell Renal Carcinoma 221
M. Kruk, J. Kurek, S. Osowski & R. Koktysz
A proof of concept of an augmented reality system for Nuss surgery 227
A. Ferreira, P. Morais, S. Queirós, F. Veloso, N.F. Rodrigues,
J. Correira-Pinto & J.L. Vilaça
Usage of mobile devices in a bone fracture reduction process 233
J.R. Jiménez, F. Paulano, J.M. Noguera & J.J. Jiménez
Automated peritumoral edema segmentation in preoperative brain MRI scans 239
E. Binaghi, P. Melchiorre, L. Romitelli, S. Balbi & D. Lattanzi
Evaluation of segmentation techniques for wound area identification 245
G. Zhang, P. Xiberta, A. Bardera, I. Boada & A. Romero
Automated image segmentation based on multiobjective optimization and machine
learning 251
S.M. Shontz, J.S. Tahara, D.O. McLaurin, D.J.L. Colbry & B. Parikh
Fractal descriptor on holographic images of cervical cells 255
M. Mihailescu, E.I. Scarlat, I.A. Paun, I. Grigorescu, R. Radu & O.T. Nedelcu
An integrated two Time-of-Flight camera system to determine knee flexion movement:
Comparison with standard motion analysis system 261
E. Veron-Tocquet, V. Burdin, J. Savéan, J. Leboucher & O. Rémy-Néris
Mechanics–based analysis of the left atrium via echocardiographic imaging 267
S. Gabriele, L. Teresi, V. Varano, P. Nardinocchi, P. Piras, G. Esposito,
P.E. Puddu, C. Torromeo & A. Evangelista
A new thick-walled conical model of the Left Ventricle 273
B. Leśniak-Plewińska, S. Cygan, J. Żmigrodzki & K. Kałużyński
Augmented Reality in radiofrequency ablation of the liver tumors 279
L.T. De Paolis & F. Ricciardi
Anthropomorphic robot forefinger virtually simulated 283
H.J. Rabiela, B.V. González & G.D.A. Miranda
CT based identification problem for the multicompartment model of blood
perfusion 289
E. Rohan, V. Lukeš & J. Brašnová
A study on discrimination of SIFT feature applied to binary images 295
I. Setitra & S. Larabi
Time-constrained detection of colored objects on raw Bayer data 301
A.J.R. Neves, A. Trifan & J.L. Azevedo
Image scanning techniques for speeded-up color object detection 307
A. Trifan, A.J.R. Neves & B. Cunha
Improving deep neural networks classification by preprocessing images 313
H. Erdmann, F.T. Ito, D. Takabayashi & D.N. dos Santos
Surface reconstruction of bone fragments: A comparative study 321
F. Paulano, J.J. Jiménez & J.R. Jiménez
Towards a robust patch based multi-view stereo technique for textureless and occluded
3D reconstruction 327
B. Haines & L. Bai

vii

VIPIMAGE_FM.indd vii 9/9/2015 8:12:21 PM

 Forming tool inspection using fiber-optic sensor heads 335
S. Matthias, M. Kästner, E. Reithmeier, P. Sieczkarek & A.E. Tekkaya
Navigation of robotics platform using advanced image processing navigation methods 341
L. Beran, P. Chmelar & L. Rejfek
SNIP: Smile—neutral facial display intensity predictor 347
K. Nurzynska & B. Smolka

Author index 355

viii

VIPIMAGE_FM.indd viii 9/9/2015 7:30:13 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Acknowledgements

The editors and the Conference co-chairs acknowledge the support towards the publication of the Book
of Proceedings and the organization of the V ECCOMAS Thematic Conference VipIMAGE to the
following organizations:
– Universidade do Porto (UP)
– Faculdade de Engenharia da Universidade do Porto (FEUP)
– Instituto de Ciência e Inovação em Engenharia Mecânica e Engenharia Industrial (INEGI)
– European Community on Computational Methods in Applied Sciences (ECCOMAS)
– International Association for Computational Mechanics (IACM)
– Fundação para a Ciência e a Tecnologia (FCT)
– Associação Portuguesa de Mecânica Teórica Aplicada e Computacional (APMTAC)

ix

VIPIMAGE_FM.indd ix 9/9/2015 7:30:13 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Preface

This book contains invited lectures and full papers presented at VipIMAGE 2015—V ECCOMAS
Thematic Conference on Computational Vision and Medical Image Processing, which was held in Ten-
erife, Canary Islands, Spain, during the period 19–21 October 2015. The event had 6 invited lectures,
and 50 contributed presentations originated from 19 countries: Algeria, Brazil, China, Czech Republic,
Egypt, France, Germany, Hungary, Italy, Mexico, Norway, Poland, Portugal, Romania, Russia, Spain,
The United Kingdom, The United States of America and Turkey.
Computational methodologies of signal processing and analyses have been commonly used in our
society. For instances, full automatic or semi-automatic Computational Vision systems have been increas-
ing used in surveillance tasks, traffic analysis, recognition process, inspection purposes, human-machine
interfaces, 3D vision, deformation analysis and aided medical procedures.
One of the notable aspects of the Computational Vision domain is the inter- and multi-disciplinarily.
Actually, principles and methodologies of other sciences, such as Informatics, Mathematics, Statistics,
Psychology, Mechanics and Physics, are regularly embraced into this domain. One of the key motives
that contributes for the continually effort done in this field of the human knowledge is the high number of
applications that can be easily found in Medicine. For instance, computational algorithms can be applied
on medical images for shape reconstruction, motion and deformation analysis, tissue characterization or
computer-assisted diagnosis and therapy.
The main objective of these ECCOMAS Thematic Conferences on Computational Vision and Medi-
cal Image Processing, initiated in 2007, is to promote a comprehensive forum for discussion on the recent
advances in the related fields in order to identify potential collaboration between researchers of differ-
ent sciences. Henceforth, VipIMAGE 2015 brought together researchers representing fields related to
Biomechanics, Biomedical Engineering, Computational Vision, Computer Graphics, Computer Sciences,
Computational Mechanics, Electrical Engineering, Mathematics, Statistics, Medical Imaging, Medicine
and Rehabilitation.
The expertises spanned a broad range of techniques for Signal Processing and Analysis, Image Acqui-
sition, Image Processing and Analysis, Data Interpolation, Registration, Acquisition and Compression,
Image Segmentation, Tracking and Analysis of Motion and Deformation, 3D Vision, Computer Simula-
tion, Medical Imaging, Computer Aided Diagnosis, Surgery, Therapy and Treatment, Computational
Bio-imaging and Visualization, Telemedicine, Virtual and Enhanced Reality, Satellite imagery, Software
Development and Applications.
The conference co-chairs would like to take this opportunity to express gratitude for the support given
by The International European Community on Computational Methods in Applied Sciences and The
Portuguese Association of Theoretical, Applied and Computational Mechanics, and thank to all spon-
sors, to all members of the Scientific Committee, to all Invited Lecturers, to all Session-Chairs and to all
Authors for submitting and sharing their knowledge.

João Manuel R.S. Tavares

Renato M. Natal Jorge
Conference Co-Chairs

xi

VIPIMAGE_FM.indd xi 9/9/2015 7:30:13 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Invited lectures

During VipIMAGE 2015, Invited lectures were delivered by 6 Expertises from 4 countries:
− Alexandre Xavier Falcão, Universidade de Campinas, Brazil
− Cristian A. Linte, Mayo Clinic, USA
− Fiorella Sgallari, University of Bologna, Italy
− Yongjie Zhang, Carnegie Mellon University, USA
− Xiaochuan Pan, The University of Chicago, USA
− Xue-cheng Tai, University of Bergen, Norway

xiii

VIPIMAGE_FM.indd xiii 9/9/2015 7:30:14 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Scientific committee

All works submitted to VipIMAGE 2015 were evaluated by an International Scientific Committee
composed by 116 expert researchers from recognized institutions of 26 countries:

Ahmed El-Rafei, Ain Shams University, Egypt

Alberto Gambaruto, Barcelona Supercomputing Centre, Spain
Alexandre Cunha, California Institute of Technology, USA
Amr Abdel-Dayem, Laurentian university, Canada
Ana Mafalda Reis, Universidade do Porto, Portugal
Ana Maria Mendonça, Universidade do Porto, Portugal
André Correia, Universidade do Porto, Portugal
André R.S. Marçal, Universidade do Porto, Portugal
Angel Alberich-Bayarri, Quiron Hospitals Group, Spain
António Luís Pereira do Amaral, Instituto Superior de Engenharia de Coimbra, Portugal
Antonios Gasteratos, Democritus University of Thrace, Greece
Armando Jorge Sousa, Universidade do Porto, Portugal
Audenaert Emmanuel, Ghent University, Belgium
Begoña Calvo, Universidad de Zaragoza, Spain
Bernard Gosselin, University of Mons, Belgium
Christos Grecos, University of the West of Scotland, UK
Constantine Kotropoulos, Aristotle University of Thessaloniki, Greece
Cristian A. Linte, Mayo Clinic, USA
Da-Chuan Cheng, China Medical University, Taiwan
Daniel Cremers, Technische Universität München, Germany
Daniela Iacoviello, Sapienza Università di Roma, Italy
Djemel Ziou, Université de Sherbrooke, Canada
Eduardo Borges Pires, Universidade de Lisboa, Portugal
Eduardo Soudah, International Center for Numerical Methods in Engineering, Spain
Elisabetta Binaghi, Università degli Studi dell’Insubria, Italy
Emmanuel Audenaert, Ghent University, Belgium
Emre Celebi, Louisiana State University in Shreveport, USA
Enrique Alegre, University of Leon, Spain
Fátima Nunes, Universidade de São Paulo, Brazil
Fernao Vistulo de Abreu, Universidade de Aveiro, Portugal
Filipa Sousa, Universidade do Porto, Portugal
Francesco Iacoviello, University of Cassino and South Latium, Italy
Francisco P.M. Oliveira, Universidade de Coimbra, Portugal
Gerald Schaefer, Loughborough University, UK
Gerhard Holzapfel, Graz University of Technology, Austria
Giuseppe Placidi, University of L’Aquila, Italy
Helcio R.B. Orlande, Universidade Federal do Rio de Janeiro, Brazil
Hemerson Pistori, Universidade Católica Dom Bosco, Brazil
Henryk Palus, Silesian University of Technology, Poland
Hongen Liao, Tsinghua University, China
Isabel N. Figueiredo, Universidade de Coimbra, Portugal
J. Paulo Vilas-Boas, Universidade do Porto, Portugal
Jaime S. Cardoso, Universidade do Porto, Portugal
Jan D’hooge, Catholic University of Leuven, Belgium

xv

VIPIMAGE_FM.indd xv 9/9/2015 7:30:14 PM

 Javier Melenchón, Universitat Oberta de Catalunya, Spain
Jeff Weiss, The University of Utah, USA
Jimmy T. Efird, East Carolina Heart Institute, USA
João L. Vilaça, Instituto Politécnico do Cávado e do Ave, Portugal
João Paulo Papa, Universidade Estadual de São Paulo, Brazil
Jordi Gonzàlez, Universitat Autònoma de Barcelona, Spain
Jorge A. Silva, Universidade do Porto, Portugal
Jorge Barbosa, Universidade do Porto, Portugal
Jorge Manuel Miranda Dias, Universidade de Coimbra, Portugal
Jorge S. Marques, Universidade de Lisboa, Portugal
Jos Vander Sloten, Katholieke Universiteit Leuven, Belgium
José Augusto Ferreira, Universidade de Coimbra, Portugal
José C. Reis Campos, Universidade do Porto, Portugal
Josef Slapal, Brno University of Technology, Czech Republic
Kristian Sandberg, Computational Solutions, Inc., USA
Krzysztof Rechowicz, Old Dominion University, USA
Lasse Riis Østergaard, Aalborg University, Denmark
Laurent Cohen, University Paris Dauphine, France
Luis Amaral, Universidade do Minho, Portugal
Luis Paulo Reis, Universidade do Minho, Portugal
Luiz Neves, Universidade Federal do Parana, Brazil
Lyuba Alboul, Sheffield Hallam University, UK
Mads Nielsen, University of Copenhagen, Denmark
Mahmoud El-Sakka, The University of Western Ontario, Canada
Manuel Filipe Costa, Universidade do Minho, Portugal
Manuel González-Hidalgo, University of the Balearic Islands, Spain
Manuel Ujaldón, University of Malaga, Spain
Manuele Bicego, Università degli Studi di Verona, Italy
Masud Rahman, Department of Infrastructure and Regional Development, Australia
Marc Thiriet, Université Pierre et Marie Curie, France
Mário Forjaz Secca, Universidade Nova de Lisboa, Portugal
Mengxing Tang, Imperial College London, UK
Michela Cigola, University of Cassino and South Latium, Italy
Miguel A. González Ballester, Universitat Pompeu Fabra, Spain
Miguel Tavares da Silva, Universidade de Lisboa, Portugal
Miguel Velhote Correia, Universidade do Porto, Portugal
Nguyen Dang Binh, Hue University, Vietnam
Nikos Chrisochoides, Old Dominion University, USA
Nuno Machado, Instituto Politécnico do Lisboa, Portugal
Paola Lecca, University of Trento, Italy
Paola Nardinocchi, Sapienza Università di Roma, Italy
Paolo Di Giamberardino, Sapienza Università di Roma, Italy
Paolo Emilio Puddu, Sapienza Università di Roma, Italy
Paulo Flores, Universidade do Minho, Portugal
Pedro Almeida, Universidade de Lisboa, Portugal
Pedro Martins, Universidade do Porto, Portugal
Petia Radeva, University of Barcelona, Spain
Philippe Young, University of Exeter, UK
Ragnar Nortvedt, Haukeland University Hospital, Norway
Reneta Barneva, SUNY Fredonia, USA
Rui Lima, Universidade do Minho, Portugal
Sabina Tangaro, Istituto Nazionale di Fisica Nucleare, Italy
Sanderson L. Gonzaga de Oliveira, Universidade Federal de Lavras, Brazil
Sandra Rua Ventura, Instituto Politécnico do Porto, Portugal
Sebastian Kurtek, The Ohio State University, USA
Sgallari Fiorella, University of Bologna, Italy
Slimane Larabi, USTHB University, Algeria
Susana Oliveira Branco, Instituto Politécnico de Lisboa, Portugal

xvi

VIPIMAGE_FM.indd xvi 9/9/2015 7:30:14 PM

 Suzanne M. Shontz, Mississippi State University, USA
Thierry Brouard, Université François Rabelais, France
Vaclav Skala, University of West Bohemia, Czech Republic
Valentin E. Brimkov, SUNY Buffalo State College, USA
Victor Hugo C. de Albuquerque, Universidade de Fortaleza, Brazil
Waldir Leite Roque, Universidade Federal da Paraíba, Brazil
Xiongbiao Luo, Nagoya University, Japan
Xue-Cheng Tai, University of Bergen, Norway
Yongjie (Jessica) Zhang, Carnegie Mellon University, USA
Young Lae Moon, Chosun University Hospital, South Korea
Yubo Fan, Beihang University, China
Zbisław Tabor, Cracow University of Technology, Poland
Zeyun Yu, University of Wisconsin at Milwaukee, USA
Zhen Ma, Universidade do Porto, Portugal

xvii

VIPIMAGE_FM.indd xvii 9/9/2015 7:30:14 PM

This page intentionally left blank
 Invited lectures

VIPIMAGE15_Book.indb 1 9/8/2015 7:40:39 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

An adaptive non-rigid image registration technique

using hierarchical B-splines

Aishwarya Pawar, Yongjie Zhang & Xiaodong Wei

Department of Mechanical Engineering, Carnegie Mellon University, Pennsylvania, USA

Yue Jia, Timon Rabczuk, Chiu Ling Chan & Cosmin Anitescu
Institute of Structural Mechanics, Bauhaus-Universität Weimar, Weimar, Germany

ABSTRACT: An efficient non-rigid image registration framework aims at accurately constructing a

spatial transformation, which enables the smooth deformation of an image in order to correctly align it
with the target image. We present a new approach to implement non-rigid image registration, in which
the spatial transformation is constructed by using hierarchical B-spline basis functions. Based on the
concept of minimization of an energy functional, which drives the registration process by minimizing
the difference between the images, the control points are dynamically updated resulting in a new spatial
transformation function at each stage of the registration process. Moreover the registration is carried
out on a set of successively refined grids, which makes the method more efficient and faster. Instead of
uniformly refining the entire control grid, this dynamic implementation is carried out by an adaptive grid
refinement procedure, wherein we detect large deformation areas and locally refine these regions. This
makes the computation of the spatial transformation more efficient and accurate with fewer number of
control points. A comparison between the adaptive grid refinement and uniform grid refinement methods
is explained using synthetic and medical images as examples.

1 INTRODUCTION implementation has been proposed in (Rueckert

et al. 1999) and (Tustison et al. 2007). According
Image registration is an image processing tech- to the type of transformation, image registration
nique, which has a broad range of applications, methods are classified into rigid and non-rigid reg-
but is most prominently applied in the field of istration models. Three types of transformations
computer vision and medical image analysis (Gos- carried out under rigid registration are translation,
htasby 2005), (Zhang et al. 2012). It involves the rotation and reflection. Rigid registration is gen-
alignment of two or more images so that there is erally used to register images that undergo small
proper integration of useful data from separate deformations. In non-rigid registration, we can deal
images. The important features from each of these with large deformation problems. However, certain
images can be combined together to achieve a bet- regularization constraints need to be imposed on
ter understanding of the complete picture of the the deformation to ensure the smoothness of the
image data that is being studied. There have been solution and proper convergence to get the desired
many methods, by which image registration is car- result.
ried out. A comprehensive survey of several such In spline based registration (Jia et al. 2015), (Leng
methods with their respective advantages and limi- et al. 2013) and (Szeliski and Coughlan 1994), a
tations has been covered in detail in (Maintz and spline-based control grid overlays the image, and a
Viergever 1998) and (Zitova and Flusser 2003). spatial transformation is constructed to deform this
The process of image registration involves grid. An efficient and stable algorithm was suggested
deforming one image, usually called the source by (Jia et al. 2015), in which the spatial transforma-
image, by computing a spatial transformation tion was based on B-spline basis functions, and the
function, which aligns it with the other image, control points were computed dynamically to enable
called the target image. The spatial transformation non-rigid registration. Moreover the method was
is defined in a parametric form, and the param- made more efficient by computing the spatial trans-
eters are computed through minimization of an formation on a set of successively refined grids. In
energy functional that is based on using a similar- our work, instead of uniformly refining the grid, we
ity metric to drive the registration process. Such an employ an adaptive mesh refinement scheme using

CH01-KN01.indd 3 9/11/2015 12:19:37 PM

 hierarchical B-splines. Hierarchical B-spline based f ( ), such that, I1 f ( x))
)) I 2 ( ) . The transforma-
image registration methods have been previously tion of the source image is driven by minimizing an
used in (Hansen et al. 2008) and (Xie and Farin energy functional, which uses the Sum of Squared
2004). In the current work, we formulate an adap- Differences (SSD) of the two images as the driving
tive grid refinement strategy by using hierarchical force to align the images. The detailed description
B-splines and interfacing it with the minimization of the formulation of the energy functional can be
of the energy functional as given in (Jia et al. 2015). found in (Jia et al. 2015) and (Leng et al. 2013). The
Thus we can solve large deformation problems energy functional is given as:
with adaptive grid refinement strategy through the
dynamic computation of fewer control points. This
method is more efficient and faster as compared to
E( ) ∫Ω ( I 2 ( x ) I1 f ( x )))2 d Ω
the uniform grid refinement method. + λ1 ∫ (|| f,ξ 2
f,η ( x ) ||22 ) d Ω
Ω
The paper is organized as follows. In section 2,
we first present the framework of image registra- + λ2 ∫ (|| f,ξ 2
f,η ( x ) ||22
Ω
tion and introduce our adaptive grid refinement − < f,ξ f,η ( ) >2 ), (3)
strategy using hierarchical B-splines. In section 3,
we apply our approach to solve two numerical where λ1 and λ2 are regularization parameters
examples, by comparing the results in terms of that are set according to the characterization of
computational time and accuracy with the results the problem. λ1 and λ2 are set as 0.0001 in our
obtained using uniform grid refinement strategy. method. f,ξ ( ) and f,η ( ) are the derivatives
The paper ends with a brief summary in section 4. of f ( ) in ξ and η directions respectively and
< f,ξ , f,η ( ) > is the inner product operator
which is equal to f,ξ T f,η ( ) . The first term
2 IMAGE REGISTRATION MODEL in Equation (3) drives the registration process by
minimizing the differences between the images.
2.1 Registration framework Unlike rigid deformations, non-rigid deforma-
The process of registration of two images, I1( ) tions can achieve image deformations of larger
(the source image) and I 2 ( ) (the target image), magnitude. But this sometimes results in unrealis-
is based on aligning I1( ) with I 2 ( ) through a tic changes within the image. Thus the second and
computed spatial transformation function. We use third terms in Equation (3), called the regulariza-
bi-cubic B-splines to obtain a C 2 mapping from tion constraints, are used to prevent these unreal-
R 2 → R 2 . In particular, the spatial transformation istic changes and also make the transformation as
function f ( ) is defined as: smooth as possible.
Since the energy functional is a function of
Nb
f ( ), we differentiate E( ) with respect to the con-
f( ) ∑ Pkφk ( ), (1) trol points P, and update their values according to
k =1
the equation:
where Pk are a set of control points and φk ( ) are
Pt 1
Pt − ∈ δ E ( x ), (4)
the bivariate basis functions. Nb represents the total
number of basis functions used. φk ( ) is the ten- t+1 t
where P and P are control points evaluated at
sor product of univariate B-spline basis functions the current and previous iterations respectively. ∈
Ni p (ξ ) and N j q ( ) , defined on the knot vectors, denotes the time step which is chosen based on the
Ξ = {ξ1, , ξn1 + p +1} and H = {η1, , ηn2 + qq++1} in problem. δ E ( ) is defined as given in (Jia et al.
ξ and η directions, respectively. n1 and n2 are the 2015).
number of univariate basis functions. p and q are Similarity metrics such as Sum of Squared Dif-
the degree of polynomials in ξ and η directions, ferences (SSD) and the Ratio of Similarity (RS)
respectively. This is written as (Jia et al. 2015), provide a way to quantitatively
evaluate the results of the registration process. RS
φi j i p ξ )N j q (η ), (2) is defined as

where i 1, 2, ..., n1 and j 1 2 n2 . The details ⎛ || I 2 ( ) − I k ( f ( )) || 2 ⎞

on B-spline functions can be found in (Piegl and RS ( ) = ⎜1 − L
⎟ × 100, (5)
Tiller 2012) and (Cottrell et al. 2009). The initial ⎝ || I 2 ( ) − I ( ) ||L2 ⎠
control points are defined to create an identity map,
f ( ) x , at the beginning of the registration. This where I1k f ( x )) is the image obtained at the kth
represents the initial source image. After representing time step. When RS = 100%, this corresponds to
I1( ), we proceed to find the optimal transformation the completely registered image.

VIPIMAGE15_Book.indb 4 9/8/2015 7:40:39 PM

 2.2 Hierarchical B-splines The refinement procedure is implemented in a
recursive manner till we reach the refinement level
This section begins with a brief review of hier-
lmax −1.
archical B-spline basis functions. Let us consider
univariate basis functions defined on the open
2.3 Adaptive local refinement
knot vectors Ξ = {ξ1, , ξn1 + p +1}, where n1 and p
are the number of univariate basis functions and According to the multilevel technique described
the degree of the polynomials in the ξ direction, in (Jia et al. 2015), the errors in the registration
respectively. The local support of the basis function will blow up after certain number of iterations
Ni p (ξ ) is in the knot interval [ξi ξi + p +1 ] , denoted when we use only one mesh to compute the spa-
by supp( Ni p (ξ ))) . We define the maximum level of tial transformation f ( ). Hence a new technique
refinement as lmax . The knot vectors of the current is developed, wherein the registration process is
level are obtained by bisecting g those of the previ- carried out on a set of grids which are refined at
ous level. A basis function Nil p (ξ ) at level l can be each level. The final image obtained at a particu-
represented as a linear combination of a subset of lar level l is used as the source image to compute
the basis functions from the next refinement level. the solution on the refined grid at level l + 1. This
These basis functions are called as the children is done to prevent large distortion of the coarse
basis functions of Nil p (ξ ) . We have level elements when the image undergoes large
deformation. This process continues till we reach
Nc −1 the grid at the maximum refinement level. Thus
Nil p (ξ ) ∑ Sk p Nkl +p1 (ξ ), (6) the solution is computed on a set of grids that are
k =0 refined uniformly in the entire image domain. It
can be seen however, that in the image domain,
where N kl +1p (ξ ) are the children basis functions and there are some regions where there will be large
Nc is the number of children basis functions. The deformation, such as areas where the differences
refinement coefficients Sk p , are calculated using in the intensities of the source and target image
the Oslo Algorithm (Cohen et al. 1980). In the are large, whereas the remaining regions are either
similar manner the children of the bivariate basis well-matched or contain low information, such as
functions φil j ( ) (Equation (2)), are the basis func- image background. Thus we can use this infor-
tions found in the next refinement level, which are mation and locally refine the regions where there
completely contained in the region of local sup- is large deformation keeping the rest of the grid
port, [ξi ξi + p ] × [ j j q +1 ] . elements coarse since they will not distort too
The hierarchical refinement procedure involves much to give unrealistic results. We endeavor to
replacing a selected set of basis functions at a par- use the dynamical implementation on a hierarchy
ticular refinement level with their children basis of locally refined grids so that we can avoid uni-
functions. This enables local refinement of the formly increasing the number of control points in
selected basis functions at different refinement lev- the whole domain.
els, (l , , , lmax ). Further details on the imple- The detailed outline of the algorithm is described
mentation of local refinement by using hierarchical in the following: Given a pair of images, I1( )
B-splines can be found in (Bornemann and Cirak and I 2 ( ). The process starts at level-0, which is a
0
2013). We explain the implementation of local uniform grid. The initial set of control points Phb
refinement of B-spline basis functions on two con- are defined such that f ( ) ∑ i =1 Phhb ,iφ hhb ,i = x . N is
N 0 0

secutive levels, l and l + 1, in the following steps: the total number of basis functions at level-0. For
l , , ..., lmax − 1,
1. At a particular refinement level l, a subset of
B-spline basis functions satisfying the refine- 1. Compute the absolute difference in the intensi-
ment criterion are identified, which form the set ties, I d , between I1l f ( x )) and I 2 ( ) , at the cell
φ l . The remaining B-spline basis functions are coordinates of the hierarchical B-spline con-
set as active (φal ). trol grid at level-l. Here I1l f ( x )) is the evolved
2. The children B-spline basis functions of all the image obtained from the previous level-(l −1). It
basis functions φ l are set as active (φal+1 ). is the initial source image if l = 0. Let Dmean be
3. The process ends by collecting all the active the average difference in the intensities between
B-spline basis functions at levels l and l + 1, the two images.
in order to get the hierarchical B-splines basis 2. Loop over the active B-spline basis functions
l +1.
functions, φhb φ lj , j 1 2 Nb, where Nb is the total number
of active B-spline basis functions at level-l. Loop
Thus we can obtain, over supp(φ lj ), and find the average of the I d val-
l +1
ues in supp(φ lj ), denoted as D j . If D j Dmean ,
φhb φal ∪ φal +1. (7) then we refine the particular B-spline according

VIPIMAGE15_Book.indb 5 9/8/2015 7:40:47 PM

 to the refinement procedure explained in
section 2.2. ρ is the threshold parameter set by
the user. A higher value of ρ results in less con-
trol grid refinement.
3. After carrying out the refinement, we collect
all the active B-spline basis functions, control
points and cells at level-l, to get the hierarchical
l +1
basis functions, control points and cells, φhb ,
l +1
Phb and Ωl +1 respectively.
4. Begin the loop for iterations, k , , ..., imax ,
where imax is the maximum number of iterations
for a particular level.
a. Compute the energy functional value,
δ E k ,l +1( ) .
b. We update the control points, according to
k l +1 k ,l +1
the relation Phb Phb − ∈δ E k ,l +1( x ).
c. Using the updated control points, the spa-
tial transformation is computed and we have
f ( ) ∑ Nj =hb1 Phhbk , j ,l +1φ hhbl +1, j , where N hb is the total
number of hierarchical B-splines.
d. Using the spatial transformation f ( ), we
obtain image I1 f ( x ) ).
5. The final image obtained at the end of the itera-
tions loop is used as the source image for the
next refinement level.
6. The process terminates upon reaching the
desired accuracy, that is, the best possible match
between the source and target images.

3 IMPLEMENTATION AND DISCUSSIONS

The efficiency of the proposed registration frame-

work is evaluated on a pair of synthetic images
and a pair of medical images. The resulting images Figure 1. Synthetic image results. The initial source
obtained at the end of each refinement level are image, the target image and initial differences are shown
recorded. The difference between the resulting in (a–c). Meshes generated at different levels based on
image and the target image at each refinement hierarchical B-spline basis refinement are shown in (d–f).
level is illustrated. The results were generated on The resulting images obtained after each stage of regis-
a computer with 2.5 GHz quad-core Intel Core i7 tration are shown in (g–i). The differences between the
processor and 16 GB RAM. Finally, a compari- target and resulting images obtained at each level are
shown in (j–l).
son of the convergence of the solution using the
similarity metrics and the time taken for each level
are shown in a tabulated manner along with the
results obtained with uniform grid refinement remaining iterations, the time step was reduced to
technique. 0.001 in order to achieve a smooth convergence.
The solution reached the desired accuracy at the
end of the third refinement level. For the first two
3.1 Synthetic image example (ring to sun)
levels, the total number of iterations for each level
The registration model is tested on a pair of syn- is 40, whereas for the third level, 20 iterations are
thetic images (see Figs. 1 (a–b)) to evaluate the used. The values of RS at the end of each level
efficiency of the algorithm. At the first level, we are 91.36%, 97.02% and 97.96%. The final images
use a uniform control grid consisting of 30 × 30 obtained after each stage of refinement are shown
elements and 33 × 33 control points. The threshold in Figures 1 (g–i). The difference between the
parameter ρ, is set as 1.5. The number of control resulting image and target image at the end of
points in the second and third refinement levels each level are shown in Figures 1 (j–l). It can be
are 2,328 and 6,254 respectively. The time step seen that the maximum deformation in the image
for the first 20 iterations is set as 0.002. For the is observed on the coarse level and the spatial

VIPIMAGE15_Book.indb 6 9/8/2015 7:40:54 PM

 Table 1. Synthetic image (Ring to Sun example, image size = 220 × 220): Comparison of the convergence of the solu-
tion and the time taken using our method (HBS) with uniform B-spline grid refinement (UBS).

SSD
4
RS (%) CPU Time (in seconds) ( initial = 2.2943 × 10 ) Control points used
Levels
(Iterations) HBS UBS HBS UBS HBS UBS HBS UBS

Level 1 (40) 91.36 91.00 74.06 70.84 1,982.28 2,064.87 1,089 1,089
Level 2 (40) 97.02 96.65 155.25 303.60 682.76 768.80 2,328 3,969
Level 3 (20) 97.96 97.80 333.67 393.27 (at 6th 468.62 505.55 6,254 15,129
iteration)
Total time 562.98 767.71
(seconds)

Table 2. Medical image (Brain MRI example, image size = 320 × 320): Comparison of the convergence of the solution
and the time taken using our method (HBS) with uniform B-spline grid refinement (UBS).

SSD
3
RS (%) CPU time (in seconds) i ii Control points used
Levels
(Iterations) HBS UBS HBS UBS HBS UBS HBS UBS

Level 1 (10) 54.05 54.05 81.81 37.40 1,170.30 1,170.30 1,849 1,849
Level 2 (20) 75.41 73.73 208.60 326.61 626.25 669.12 3,997 6,889
Level 3 (20) 79.07 79.51 770.10 2,247.60 532.95 521.93 11,089 26,569
Total time 1,060.51 2,611.61
(seconds)

transformation is then evaluated on the finer levels to capture the features of the images correctly.
to further improve the accuracy. Table 1 shows the Also, the time step is reduced as compared to syn-
comparison of the time taken, the value of RS and thetic images in order to gradually and smoothly
the number of control points used for each refine- deform the image. In this example, we use a uni-
ment level of the grid. From Table 1, it can be seen form control grid consisting of 40 × 40 elements
that the total time taken to achieve the desired and 43 × 43 control points at the first level. The
accuracy using the adaptive refinement scheme threshold parameter ρ, is set as 1.5. The number
is less than that of the uniform refinement strat- of control points in the second and third refine-
egy. Although the initialization time taken to set ment levels are 3,997 and 11,089 respectively. The
up the hierarchical data structure in adaptive grid time step for the entire registration process is set
refinement is more than that taken for the uni- to be 0.00025. The solution reached the desired
form grid refinement, during the computation accuracy at the end of the third refinement level.
of the deformed image, the time taken by the For the first level, the total number of iterations
adaptive refinement method is much less. This equals to 10, whereas for the second and third
is because the number of control points used to level they were increased to 20. The values of RS
compute the spatial transformation of the deform- at the end of each level are 54.05%, 75.41% and
ing image is much lower than the uniformly refined 79.07%. The final images obtained at each refine-
grid. This makes the adaptive registration process ment level are shown in Figures 2 (g–i). The dif-
much more efficient. ference between the resulting and target images
at the end of each level are shown in Figures 2
(j–l). Table 2 shows the comparison of the time
3.2 Medical image example (Brain MRI)
taken, the value of RS and the number of control
The registration model is tested on a pair of med- points used for each refinement level of the grid.
ical images as shown in Figures 2 (a–b). Unlike From Table 2, it can be seen that the total time
synthetic images, medical images have more taken to achieve the desired accuracy using adap-
complex feature information. Thus we use more tive refinement scheme was less than that of the
control points at the first refinement level so as uniform refinement strategy.

VIPIMAGE15_Book.indb 7 9/8/2015 7:40:56 PM

 such as medical image interpolation and further
extension to 3-D registration problems.

ACKNOWLEDGEMENTS

The medical images were provided from (http://

overcode.yak.net/15). The research at Carnegie
Mellon University was supported in part by NSF
CAREER Award OCI-1149591. The research at
Bauhaus University Weimar was supported in part
by the ITN-INSIST funded by the EU-FP7.

REFERENCES

Bornemann, P.B. & F. Cirak (2013). A subdivision-based

implementation of the hierarchical B-spline finite ele-
ment method. Computer Methods in Applied Mechan-
ics and Engineering 253, 584–598.
Cohen, E., T. Lyche, & R. Riesenfeld (1980). Discrete
B-splines and subdivision techniques in computer-
aided geometric design and computer graphics. Com-
puter graphics and image processing 14(2), 87–111.
Cottrell, J.A., T.J. Hughes, & Y. Bazilevs (2009). Isogeo-
metric analysis: toward integration of CAD and FEA.
JohnWiley & Sons
Goshtasby, A.A. (2005). 2-D and 3-D image registration:
for medical, remote sensing, and industrial applications.
John Wiley & Sons.
Hansen, M.S., R. Larsen, B. Glocker, & N. Navab (2008).
Adaptive parametrization of multivariate B-splines
for image registration. In IEEE Conference on Compu-
ter Vision and Pattern Recognition, pp. 1–8.
Jia, Y., Y. Zhang, & T. Rabczuk (2015). A novel dynamic
multilevel technique for image registration. Computers
Figure 2. Medical image results. The initial source image,
& Mathematics with Applications 69(9), 909–925.
the target image and initial differences are shown in (a–c).
Leng, J., G. Xu, & Y. Zhang (2013). Medical image inter-
Meshes generated at different levels based on hierarchical
polation based on multi-resolution registration. Com-
B-spline basis refinement are shown in (d–f). The resulting
puters & Mathematics with Applications 66(1), 1–18.
images obtained after each stage of registration are shown
Maintz, J.A. & M.A. Viergever (1998). A survey of medical
in (g–i). The differences between the target and resulting
image registration. Medical image analysis 2(1), 1–36.
images obtained at each level are shown in (j–l).
Piegl, L. & W. Tiller (2012). The NURBS book. Springer
Science & Business Media.
Rueckert, D., L.I. Sonoda, C. Hayes, D.L. Hill, M.O.
4 CONCLUSION Leach, & D.J. Hawkes (1999). Nonrigid registration
using free-form deformations: application to breast
MR images. IEEE Transactions on Medical Imaging
A non-rigid image registration method using adap-
18(8), 712–721.
tive grid refinement based on hierarchical B-spline Szeliski, R. & J. Coughlan (1994). Hierarchical spline-
basis functions has been presented. By detecting based image registration. IEEE Conference on Compu-
the large deformation areas in the image, the con- ter Vision and Pattern Recognition, 194–201.
trol grid is locally refined in these regions. Hence Tustison, N.J., B. Avants, G.J.C., et al. (2007). Improved
we use fewer number of control points resulting FFD B-spline image registration. In IEEE 11th Inter-
in an efficient and accurate computation of the national Conference on Computer Vision, pp. 1–8.
spatial transformation function. A comparison Xie, Z. & G. Farin (2004). Image registration using hier-
between our method and uniform grid refinement archical B-splines. IEEE Transactions on Visualization
and Computer Graphics 10(1), 85–94.
shows that our method performs better by achiev-
Zhang, Y., Y. Jing, X. Liang, G. Xu, & L. Dong (2012).
ing the same accuracy in a shorter amount of time. Dynamic lung modeling and tumor tracking using
We demonstrate the method on artificial and medi- deformable image registration and geometric smooth-
cal images. The algorithm has worked successfully ing. Molecular & Cellular Biomechanics 9(3), 213–226.
on medical images and hence provides a promising Zitova, B. & J. Flusser (2003). Image registration methods:
direction for future work in several applications a survey. Image and vision computing 21(11), 977–1000.

VIPIMAGE15_Book.indb 8 9/8/2015 7:40:56 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Medical image segmentation using Object Shape Models: A critical

review on recent trends, and alternative directions

Alexandre Xavier Falcão, Thiago Vallin Spina & Samuel Botter Martins*
Institute of Computing, University of Campinas, Campinas, Brazil

Renzo Phellan
Eldorado Research Institute, Campinas, Brazil

ABSTRACT: Segmentation is important to define the spatial extension of body anatomic structures
(objects) in medical images for quantitative analysis. In this context, it is desirable to eliminate (at least
minimize) user interaction. This aim is feasible by combining object delineation algorithms with Object
Shape Models (OSMs). While the former can better capture the actual shape of the object in the image,
the latter provides shape constraints to assist its location and delineation. We review two important
classes of OSMs for medical image segmentation: Statistical (SOSMs) and Fuzzy (FOSMs). SOSMs
rely on the image mapping onto a reference coordinate system, which indicates the probability of each
voxel to be in the object (a probabilistic atlas built from a set of training images and their segmentation
masks). Imperfect mappings due to shape and texture variations asks for object delineation algorithms,
but the methods usually assume that the atlas is at the best position for delineation. Multiple atlases per
object can mitigate the problem and a recent trend is to use each training mask as an individual atlas.
By mapping them onto the coordinate system of a new image, object delineation can be accomplished
by label fusion. However, the processing time for deformable registration is critical to make SOSMs suit-
able for large scale studies. FOSMs appear as a recent alternative to avoid reference systems (deformable
registration) by translating the training masks to a common reference point for model construction.
This relaxes the shape constraints, but asks for a more effective object delineation algorithm and some
efficient approach for object’s location. One of the solutions, named optimum object search, translates
the model inside an estimated search region in the image while a criterion function guides translation and
determines the best delineated object among candidates. This makes segmentation with FOSMs consid-
erably faster than with SOSMs, but SOSMs that adopt the optimum object search can be more effective
and with less atlases per object. We then discuss the pros and cons of the recent FOSM and SOSM
approaches by providing alternative directions, which also include the user to correct segmentation errors
and improve the models.

1 INTRODUCTION object delineation. This has motivated the use of

interactive image segmentation methods that exploit
Medical image segmentation is a crucial step in the a synergism between user and computer, by combin-
quantitative analysis of body anatomic structures ing user input for approximately locating the object
(objects) for the non-invasive diagnosis and with some image-based delineation method to extract
treatment of diseases. We may divide image segmen- it from the background (Falcão et al. 1998, Falcão et
tation into two tightly coupled tasks: object recogni- al. 2000, Falcão et al. 2004, Rother et al. 2004, Boykov
tion and delineation (Falcão et al. 1998). Recognition and Funka-Lea 2006, Grady 2006, Couprie et al.
involves locating the whereabouts of the object in 2011, Spina et al. 2014). The main drawback in inter-
an image and verifying errors in segmentation, while active segmentation is its limited capacity for large
delineation defines the object’s spatial extent. scale studies in medical image analysis. Moreover,
Humans usually outperform computers in object image segmentation is a tedious task prone to human
recognition, while computers are more precise for errors, specially when dealing with 3D MR and CT
images, since the results must be visually assessed in
multiple planes (sagittal, axial, and coronal).
*The authors would like to thank CNPq (grants In order to eliminate or at least minimize user
302970/2014-2 and 479070/2013-0), FAPESP (grant intervention in segmentation, the shape and tex-
2011/01434-9), and CAPES for the financial support. ture properties of the object are an important

VIPIMAGE15_Book.indb 9 9/8/2015 7:40:57 PM

 prior information. Active Shape Models (ASMs) image to the model’s reference image. This is not
(Cootes et al. 1995), for instance, were proposed valid in practice, given that a single reference image
to learn the possible manifestations of the object’s cannot represent all object instances available in
shape from a set of training images with their seg- nature. This problem has motivated the use of mul-
mentation masks, and encode them into a linearly tiple SOSMs per object, converging to the point of
deformable shape model. The shape is represented considering each training mask as an independent
as a set of boundary points (landmarks) whose cor- atlas in recent works (Wang et al. 2013, Akhondi-
respondence must be specified in each mask. The Asl and Warfield 2013). That is, for a new test
landmarks are mapped onto a common coordinate image, each training image must be registered with
system in order to estimate their mean position the test image, the deformation field is applied to
and possible variations, limiting the deformations the corresponding mask, and the registered seg-
of the model. Similarly, texture variations can be mentation masks go through a process referred to
incorporated into an Active Appearance Model as label fusion to output the final delineation of the
(AAM) (Cootes et al. 1998). target image. The argument is that the strategy of
ASM/AAM approaches may be interesting solu- Multiple Atlas Label Fusion (MALF) makes bet-
tions for object delineation, but they do not solve ter use of the training dataset, by increasing the
recognition. Usually, the user has to place the model chance of correctly segmenting the target image.
near the object for subsequent model deformation Although MALF strategies seem to improve
(with possible translation) to match the land- accuracy, the time consumed by the registration
marks with the strongest edges for object delinea- step increases prohibitively, thereby making them
tion (Rueda 2010, Ghose et al. 2010, Inamdar and unfeasible for large scale studies. Moreover, they
Ramdasi 2013). The specification of correspond- assume that the dataset used for training is fixed
ing landmarks in different masks is a challenging and will never increase during testing. Updating
task (Rueda 2010), whose the interactive solution OSMs is necessary in clinical settings to ensure
makes model construction a tedious and error- adaptability to new challenging images. Lastly, as
prone step. Such a problem can be avoided with pointed out in (Phellan Falcão & Udupa 2014),
deformable image registration (Pluim et al. 2003), registration does not solve object recognition, since
but this essentially suggests the use of another type the object’s boundary might not even be confined
of Object Shape Model (OSM)—a probabilistic in the uncertainty region of the model afterwards
atlas model (Mazziotta et al. 1995). (Phellan, Falcão, & Udupa 2014) (i.e., the set of
In this paper, we review two important classes of voxels with prior probability between (0, 1)).
OSMs for medical image segmentation: Statistical FOSMs have been recently proposed as an alter-
(SOSMs) and Fuzzy (FOSMs). SOSMs are popu- native to SOSMs, with the aforementioned issues
lar approaches, being traditionally called statistical in mind (Miranda et al 2008, Miranda et al. 2009).
atlas-based models (Cabezas et al. 2011, Harrigan Instead of registering the training images to obtain
et al. 2015, Rusu et al. 2011, Lötjönen et al 2011). a prior probability map, the segmentation masks
Similarly to ASMs, the model is built from a set of are aligned by translation to a common reference
training images with their corresponding segmen- point (e.g., the masks’ geometric centers) and com-
tation masks. The simplest approach for model bined to output a fuzzy image. As a consequence,
construction requires the selection of one reference the construction of a FOSM is much faster. The
image from the training set and the deformable values in the fuzzy image are proportional to the
mapping of the remaining ones onto the refer- membership that each voxel has to the object of
ence coordinate system. The resulting deformation interest, although they do not reflect prior prob-
field is applied to the corresponding masks and the abilities since the voxel values do not correspond to
object label frequency at each coordinate defines a same location in the actual anatomy. The lack of
a prior probability map (a probabilistic atlas). By registration makes the uncertainty region of a fuzzy
mapping a new image onto the atlas coordinate image thicker than that of a classic SOSM’s prior
system, the object can be delineated by threshold- probability map (Fig. 1). As a consequence, the
ing the prior probability map (Mazziotta et al. uncertainty region of a FOSM requires more effec-
1995), by estimating and thresholding a posterior tive delineation, although it better accommodates
probability map (Vos et al. 2013), or by applying the boundary of the real object when the model is
some image-based delineation algorithm (Grau placed at the object’s center in the test image.
et al. 2004). After construction, the automatic search for the
The main drawbacks of classical SOSMs are object of interest and segmentation using FOSM
related to their dependency on the deformable involves finding the optimal position for the model
image registration algorithm. They assume that in the test image. This is typically achieved by a
object recognition is solved by registration, which local search for the object inside a given region
grants a near-accurate mapping from any test of the test image. In one of the possible solutions,

10

VIPIMAGE15_Book.indb 10 9/8/2015 7:40:57 PM

 cannot assume that the dataset used to compute
the models captures all possible nuances observed
in nature, nor that the models are capable of han-
dling abnormalities. The corrected segmentation is
also paramount to update the model in order to
improve the delineation of future test images.
In the remainder of the paper, we highlight recent
Figure 1. Sagittal view of an SOSM’s prior probability improvements developed for SOSMs (Section 2)
map for the brain’s left hemisphere on the right, and of a and FOSMs (Section 3), with their pros and cons.
FOSM’s fuzzy image on the left. We then provide details on how the automatic seg-
mentation result may be interactively corrected by
using image-based delineation methods (Section 4),
before stating conclusions in Section 5.

2 STATISTICAL OBJECT SHAPE MODELS

MALF approaches typically compute a confidence

weight for each atlas (template), which dictates the
importance that each mask should have in the
fusion process. In (Wang et al. 2013), the local inten-
sity similarity between the test image and each pair
of template images is considered for this purpose.
To mitigate errors in registration of the atlases, the
test image is further divided into patches and, for
each test patch, the method searches locally for the
most similar patch in each atlas image (finding the
Figure 2. Overall pipeline for effective medical image corresponding patch in each mask). The labels in
segmentation using OSMs. the test patch are defined by weighted label fusion
from the labels of the corresponding mask patches.
The object is then defined by local shape imposi-
named optimum object search, the model trans- tion from the models. In contrast, Akhondi-Asl
lates inside an estimated search region in the test and Warfield (2013) argue that local intensity simi-
image while a criterion function guides translation larity is a poor measure for relating the true target
and determines the best delineated object among image segmentation with the dataset templates.
candidates. Delineation often considers the exter- They learn instead Gaussian Mixture Model clas-
nal and internal portions of the fuzzy image as sifiers using the intensity and label maps of the
seeds (i.e., voxels with values {0, 1}, respectively), registered atlases, which output probabilistic seg-
but other delineation methods can be used as well. mentations of the target image. These probabilistic
The key idea is that FOSMs assume that the segmentations are then fused together to determine
model is initially mispositioned and try to correct the final binary segmentation mask.
it. This assumption is in stark contrast to SOSMs’ To reduce the time consumed by the registra-
reliance on the registration algorithm. Hence, Phel- tion of multiple atlases, Sanroma et al. (2014)
lan, Falcão, & Udupa (2014) have proposed to use argue that not all atlases are necessarily useful for
the FOSM’s optimum object search procedure to segmentation of a given target image. In fact, the
improve object delineation in SOSMs, after ini- inclusion of misleading atlases may undermine
tially positioning the model through deformable labeling performance, thereby making atlas selec-
registration. Their promising results suggest a tion a necessity. They propose a selection technique
new interesting direction of research, which would based on supervised learning that is applied before
combine classic SOSMs and FOSMs instead of deformable registration, which ranks the atlases
using MALF. according to the expected segmentation accuracy
Both statistical and fuzzy OSMs have been that each one will have on the target image.
actively improved in recent years to address their The work of Sanroma et al. (2014) supports our
shortcomings. Notwithstanding, we envision that earlier claims when they indicate that smaller and
effective image segmentation systems must allow more selective may perform better in practice. We
user intervention when automatic segmentation believe that improving the classic framework is
fails. Figure 2 depicts such scheme. Giving the user more fruitful since it is supported by a solid sta-
the opportunity to correct mistakes is key, since we tistical background. During the optimum object

11

VIPIMAGE15_Book.indb 11 9/8/2015 7:40:57 PM

 The FOSM whose best candidate segmentation
mask achieves the highest score of the criterion
function, among all of the ones in the Cloud Bank,
is selected to output the final delineation.
The Cloud Bank Model was later extended to
search and segment multiple objects in the Cloud
System Model (Miranda et al. 2010). The interest-
ing advancement in this case was that the objects
may be searched together, taking their relative posi-
tions into account to improve object recognition.
This inspired recent works on body-wide hierar-
chical Automatic Anatomy Recognition (AAR)
of multiple organs, which separately explored the
construction, recognition, and delineation aspects
of FOSMs (Udupa et al. 2011, Udupa et al. 2012,
Udupa et al. 2014). investigated the modeling stage
to create hierarchical FOSMs. They included the
Figure 3. Example of segmentation using FOSM. scaling factor of each object of interest and align-
ment based on the orientation of each principal
axis in the fuzzy model’s construction. These extra
search with SOSM in (Phellan, Falcão, & Udupa data were used in (Udupa et al. 2012) to help in
2014), the assessment of whether the model is cor- the object recognition step. In this case, some cri-
rectly positioned is done synergistically by using terion functions were evaluated, including super-
the probability map to constrain the location vised learning by Fisher Linear Discriminant, the
where a delineation algorithm is applied (i.e., only minimization of a ball-scale energy, and a one-
on the voxels of the test image superimposed by the shot approach that positions every object directly,
uncertainty region of the model). The delineation from the parent positioning relationship observed
generates a candidate segmentation mask that is during training. In (Udupa et al. 2014), the work
evaluated according to a criterion function, whose on hierarchical AAR was completed by analyz-
score is expected to be maximum when the mask ing the delineation stage using the Iterative Rela-
corresponds to the real object. Such an approach tive Fuzzy Connectedness framework (Ciesielski
object search (Fig. 3) comes from the inception of et al. 2007). Lastly, (Rittner et al. 2014) included
FOSMs (Miranda et al. 2008). multiple FOSMs per object for hierarchical AAR,
similarly to the Cloud Bank Model (Miranda et al.
2009) and the Cloud System Model (Miranda et al.
3 FUZZY OBJECT SHAPE MODELS 2010), although only for object recognition.
A few aspects distinguish the works in FOSMs.
We may define a FOSM as a triple composed of In (Udupa et al. 2011, Udupa et al. 2012, Udupa
a fuzzy image, a delineation algorithm, and a cri- et al. 2014), the hierarchy of the objects guides seg-
terion function (Miranda et al. 2008). Different mentation, and the object search involves limited
approaches devised for each of these elements translation and rotation of the models, according
may lead to improvements of the model. For to parameters learned by training. Object recogni-
instance, an important issue to be considered when tion is done first and then delineation is applied
using FOSMs is that a thicker uncertainty region as a separated step. In contrast, the FOSMs in
may degenerate the model’s fuzzy image, thereby (Miranda et al. 2008, Miranda et al. 2009, Miranda
decreasing performance. In (Miranda et al. 2009), et al. 2010, Phellan 2014, Phellan et al. 2015)
we proposed a solution to circumvent this prob- exploit the synergism between model and deline-
lem by grouping the training images according to ation algorithm for recognition and delineation as
their shape similarities, before generating multiple a single step in the optimum object search guided
FOSMs to represent a single object of interest. The by a criterion function. This process requires only
proposed algorithm starts by computing the simi- translations (since there is no object alignment for
larity between all training shapes using the Dice model construction) in multiple scales (Miranda
coefficient. Then, maximal cliques are selected et al. 2008, Miranda et al. 2009, Miranda et al.
to form groups whose similarity is above a given 2010) or, as in (Phellan 2014, Phellan et al. 2015),
threshold. Each group produces a FOSM, previ- translations constrained within an estimated search
ously denoted as Object Cloud Model, that is used region and guided by an optimization algorithm,
to search for the object of interest using a multi- named Multiscale Parameter Search (Ruppert et al.
scale version of the procedure shown in Figure 3. 2015). The search region is estimated based on the

12

VIPIMAGE15_Book.indb 12 9/8/2015 7:40:58 PM

 observed displacements between training masks
and the center of a reference object (e.g., the body
region), which can be automatically segmented by
image-based techniques.
The optimized object search of FOSM has
indeed improved SOSM (Phellan 2014), making it
more effective than FOSM for single object deline-
ation. Its extension to multiple objects is straight-
forward, being the criterion function applied to all
delineated boundaries and taking into account the
relative position of the objects with respect to the
center of the multiobject cloud system (Miranda
et al. 2010). On the other hand, the speed gains
of FOSMs over SOSMs make the former more
attractive for large scale studies and interactive
applications.

4 INTERACTIVE CORRECTION
OF OSM SEGMENTATION

Regardless of the object model selected for auto-

matic segmentation, errors may still occur after
the model is applied to a new image. It is impor-
tant to have a generic image-based delineation
method that allows the user to make corrective
actions. To the best of our knowledge, there
is only one method capable of taking as input
any automatic segmentation result and allowing
the user to correct errors without starting over
the process. We have devised such a method in
(Miranda et al. 2010, Miranda et al. 2011). The
principle behind it is that our method automati- Figure 4. Interactive correction of a pre-existing seg-
cally transforms the input segmentation into mentation computed using our FOSM on a brain MR
an Optimum Path Forest (OPF) with minimum image of a patient who underwent surgery to treat epi-
number of roots—i.e., a data structure that cap- lepsy, in which part of his left brain hemisphere was
tures the optimum connectivity between voxels removed.
and seeds (root voxels), as used by seed compe-
tition methods for object delineation (IFT-SC)
minimum cardinality (Fig. 4b). The user may then
based on the Image Foresting Transform (Falcão
draw new scribbles (add trees) or delete scribbles
et al. 2004). This allows very efficient corrections
(remove trees) and the forest is updated differen-
by adding or removing trees of the forest (Falcão
tially (Falcão and Bergo 2004), taking time pro-
and Bergo 2004).
portional to the modified regions (Fig. 4c). Such a
IFT-based delineation treats an image as a
procedure provides interactive response time to the
graph by taking all of the voxels as the nodes
user and the corrected segmentation may be used
and an adjacency relation to define the arcs (e.g.,
to improve the OSM.
6-neighbors in 3D images). In interactive segmen-
tation, a seed set composed of user-drawn scribbles
on the foreground and background is fed to IFT 5 CONCLUSIONS
for delineation. These seeds compete among them-
selves to conquer every voxel in the image, parti- We have presented a critical review on trends in
tioning the graph into an OPF where each seed is SOSMs and provided alternative directions that
the root of an optimum path tree. The object is include the more efficient FOSMs and SOSMs with
defined as the set of trees rooted at internal seeds. a reduced number of models and optimum object
Given any input segmentation mask, such as search. In the latter, it is also interesting to create a
the one in Figure 4a the resuming method does multi-object system model in order to exploit their
the reverse process of IFT-based segmentation, relative position and simultaneous delineation for
by computing a set of seeds (forest roots) with more accurate segmentation.

13

VIPIMAGE15_Book.indb 13 9/8/2015 7:40:59 PM

 OSM-based segmentation methods have been Falcão, A.X., J.K. Udupa, & F.K. Miyazawa (2000). An
developed by assuming an initial training set ultrafast user-steered image segmentation paradigm:
with images and their corresponding segmenta- Live-wireon-the-fly. IEEE Trans. Med. Imag. 19(1),
tion masks. The training set is created by interac- 55–62.
Falcão, A.X., J.K. Udupa, S. Samarasekera, S. Sharma,
tive image segmentation methods. However, for a B.E. Hirsch, & R.A. Lotufo (1998). User-steered
large number of images, the user does not known image segmentation paradigms: Live-wire and live-
the minimum training set size and which images are lane. Graph. Model. Im. Proc. 60(4), 233–260.
more representative for training the OSMs, such Ghose, S., A. Oliver, R. Mart, X. Llad, J. Freixenet, J.C.
that the segmentation of the remaining images Vilanova, & F. Meriaudeau (2010). Prostate Segmen-
will be more accurate. These challenges actually tation with Texture Enhanced Active Appearance
require an active learning approach for model con- Model. In SITIS, Volume 1, Kuala Lumpur, Malay-
struction. In this context, the efficiency gain of is a sia, pp. 18–22.
considerable advantage over SOSMs. Grady, L. (2006). Random walks for image segmenta-
tion. IEEE Trans. Pattern. Anal. Mach. Intell. 28(11),
On the other hand, SOSMs with optimum object 1768–1783.
search can be more effective than FOSMs (Phellan Grau, V., A. U.J. Mewes, M. Alcaiz, R. Kikinis, & S.W.
2014). This suggests hybrid approaches that take Warfield (2004). ImprovedWatershed Transform for
advantage of the tighter shape imposition of to Medical Image Segmentation Using Prior Informa-
delineate object parts with lack of boundary con- tion. IEEE Trans. Med. Imaging 23(4), 447–458.
trast, while can better capture the object’s bound- Harrigan, R.L., A.J. Plassard, L.A. Mawn, R.L. Gal-
ary as it appears in other parts of the image. loway, S.A. Smith, & B.A. Landman 2015). Con-
Additionally, SOSMs- and FOSMs-based have structing a statistical atlas of the radii of the optic
not considered that automatic segmentation may nerve and cerebrospinal fluid sheath in young healthy
adults. In SPIE, Volume 9413, Orlando, USA, pp.
fail and solutions to correct them without starting 941303–941303–7.
over the process are required. We have presented Inamdar, R.S. & D.S. Ramdasi (2013). Active Appear-
an efficient solution for this problem based on the ance Models for Segmentation of Cardiac MRI Data.
Image Foresting Transform. In ICCSP, Volume 1, pp. 96–100.
Lötjönen, J., J. Koikkalainen, L. Thurfiell, R. Lundqvist,
G. Waldemar, H. Soininen, & D. Rueckert (2011,
REFERENCES March). Improved Generation of Probabilistic Atlases
for the Expectation Maximization Classification. In
Akhondi-Asl, A. & S. Warfield (2013, Oct). Simultane- ISBI, Chicago, USA, pp. 1839–1842.
ous truth and performance level estimation through Mazziotta, J.C., A.W. Toga, A. Evans, P. Fox, &
fusion of probabilistic segmentations. IEEE Trans. J. Lancaster (1995). A Probabilistic Atlas of the
Med. Imaging 32(10), 1840–1852. Human Brain: Theory and Rationale for Its Develop-
Boykov, Y. & G. Funka-Lea (2006, Nov). Graph cuts and ment: The International Consortium for Brain Map-
efficient N-D image segmentation. Int. J. Comput. Vis. ping (ICBM). NeuroImage 2(2), 89–101.
70(2), 109–131. Miranda, P., A. Falcão, & J. Udupa (2008). Clouds:
Cabezas, M., A. Oliver, X. Llad, J. Freixenet, & M. B. A model for synergistic image segmentation. In ISBI,
Cuadra (2011). A review of atlas-based segmentation pp. 209–212.
for magnetic resonance brain images. Comput. Meth- Miranda, P., A. Falcão, G. Ruppert, & F. Cappabianco
ods Programs Biomed. 104(3), e158–e177. (2011, March). How to fix any 3d segmentation inter-
Ciesielski, K.C., J.K. Udupa, P.K. Saha, & Y. Zhuge actively via image foresting transform and its use in
(2007). Iterative relative fuzzy connectedness for mul- mri brain segmentation. In ISBI, Chicago, USA, pp.
tiple objects with multiple seeds. Comput. Vis. Image 2031–2035.
Underst. 107(3), 160–182. Miranda, P.A.V., A.X. Falcão, & J.K. Udupa (2009).
Cootes, T.F., G.J. Edwards, & C.J. Taylor (1998, Jun). Cloud bank: a multiple clouds model and its use in
Active Appearance Models. In ECCV, Volume 2, MR brain image segmentation. In ISBI, Boston,
Freiburg, Germany, pp. 484–498. USA.
Cootes, T.F., C.J. Taylor, D.H. Cooper, & J. Graham Miranda, P.A.V., A.X. Falcão, & G.C.S. Ruppert (2010).
(1995). Active shape models – their training and appli- How to complete any segmentation process interac-
cation. Comput. Vis. Image Underst. 61(1), 38–59. tively via image foresting transform. In SIBGRAPI,
Couprie, C., L. Grady, L. Najman, & H. Talbot (2011). Gramado, Brazil, pp. 309–316.
Power watershed: A unifying graph-based optimiza- Miranda, P.A.V., A.X. Falcão, & J.K. Udupa (2010,
tion framework. IEEE Trans. Pattern. Anal. Mach. March). Cloud models: Their construction and
Intell. 33(7), 1384–1399. employment in automatic MRI segmentation of the
Falcão, A.X. & F.P.G. Bergo (2004, sept.). Interactive brain. Technical Report IC-10-08, IC, University of
volume segmentation with differential image foresting Campinas.
transforms. IEEE Trans. Med. Imag. 23(9), 1100–1108. Phellan, R. (2014). Medical image segmentation using
Falcão, A.X., J. Stolfi, & R.A. Lotufo (2004). The image statistical and fuzzy object shape models. Master’s the-
foresting transform: theory, algorithms, and applications. sis, Institute of Computing— University of Campinas
IEEE Trans. Pattern Anal. Mach. Intell. 26(1), 19–29. (UNICAMP).

14

VIPIMAGE15_Book.indb 14 9/8/2015 7:41:01 PM

 Phellan, R., A.X. Falcão, & J.K. Udupa (2014). Improv- Sanroma, G., G. Wu, Y. Gao, & D. Shen (2014, Oct).
ing atlas-based medical image segmentation with Learning to rank atlases for multiple-atlas segmenta-
a relaxed object search. In Y. Zhang and J. Tavares tion. IEEE Trans. Med. Imaging 33(10), 1939–1953.
(Eds.), CompIMAGE, Volume 8641, pp. 152–163. Spina, T.V., P.A.V. Miranda, & A.X. Falcão (2014, Dec).
Pittsburgh, USA. Hybrid approaches for interactive image segmenta-
Phellan, R., A.X. Falcão, & J.K. Udupa (2015). Medical tion using the Live Markers paradigm. IEEE Trans.
image segmentation using object atlas versus object Image Process. 23(12), 5756–5769.
cloud models. In SPIE, Volume 9415, Orlando, USA, Udupa, J.K., D. Odhner, A.X. Falcão, K.C. Ciesielski,
pp. 94151M–94151M–11. P.A.V. Miranda, M. Matsumoto, G.J. Grevera, B.
Pluim, J. P. W., J. Maintz, & M. Viergever (2003, Aug). Saboury, & D.A. Torigian (2012). Automatic anatomy
Mutualinformation-based registration of medical recognition via fuzzy object models. In SPIE, San
images: a survey. IEEE Trans. Med. Imaging 22(8), Diego, USA.
986–1004. Udupa, J.K., D. Odhner, A.X. Falcão, K.C. Ciesielski,
Rittner, L., J.K. Udupa, & D.A. Torigian (2014). Multi- P.A.V. Miranda, P. Vaideeswaran, S. Mishra,
ple fuzzy object modeling improves sensitivity in auto- G.J. Grevera, B. Saboury, & D.A. Torigian (2011). Fuzzy
matic anatomy recognition. In SPIE, Volume 9034, object modeling. In SPIE, Lake Buena Vista, USA.
San Diego, USA, pp. 90342U–90342U–7. Udupa,J.K.,D.Odhner,L.Zhao,Y.Tong,M.M.Matsumoto,
Rother, C., V. Kolmogorov, & A. Blake (2004). “Grab- K.C. Ciesielski, A.X. Falcao, P. Vaideeswaran,
Cut”: Interactive foreground extraction using iterated V. Ciesielski, B. Saboury, S. Mohammadianrasanani,
graph cuts. ACM Trans. Graph. 23(3), 309–314. S. Sin, R. Arens, D.A. Torigian (2014). Body-wide hier-
Rueda, S. (2010, june). 2D and 3D Digital Shape Model- archical fuzzy modeling, recognition, and delineation
ling Strategies. Ph. D. thesis, University of Notting- of anatomy in medical images. Med. Image Anal. 18(5),
ham, Nottingham. 752–771.
Ruppert, G.C.S., G. Chiachia, F.P.G. Bergo, F.O. Favretto, Vos, P.C., I. Išgum, J.M. Biesbroek, B.K. Velthuis, &
C.L. Yasuda, A. Rocha, & A. X. Falcão (2015). Medi- M.A. Viergever (2013). Combined Pixel Classifica-
cal image registration based on watershed transform tion and Atlasbased Segmentation of the Ventricular
from greyscale marker and multi-scale parameter System in Brain CT images. In SPIE, Volume 8669,
search. Comput. Methods Biomech. Biomed. Eng.: pp. 1–6.
Imaging & Visual., 1–19. Wang, H., J.W. Suh, S.R. Das, J.B. Pluta, C. Craige, &
Rusu, M., N. Bloch, C.C. Jaffe, N.M. Rofsky, E.M. Genega, P.A. Yushkevich (2013). Multi-atlas segmentation
E. Feleppa, R.E. Lenkinski, & A. Madabhushi (2013). with joint label fusion. IEEE Trans. Pattern Anal.
Statistical 3D Prostate Imaging Atlas Construction Mach. Intell. 35(3), 611–623.
via Anatomically Constrained Registration. In SPIE,
Volume 8669, Lake Buena Vista, USA, pp. 1–15.

15

VIPIMAGE15_Book.indb 15 9/8/2015 7:41:01 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Primal-dual method for continuous max-flow approaches

Ke Wei
Department of Mathematics, Hong Kong University of Science and Technology, Hong Kong

Xue-Cheng Tai
Department of Mathematics, University of Bergen, Bergen, Norway

Tony F. Chan
Office of the President, Hong Kong University of Science and Technology, Hong Kong

Shingyu Leung
Department of Mathematics, Hong Kong University of Science and Technology, Hong Kong

ABSTRACT: We review the continuous max-flow approaches for the variational image segmentation
models with piecewise constant representations. The review is conducted by exploring the primal-dual
relationships between the continuous min-cut and max-flow problems. In addition, we introduce the
parameter free primal-dual method for solving those max-flow problems. Empirical results show that the
primal-dual method is competitive to the augmented Lagrangian method.

1 INTRODUCTION In this paper, we focus on the variational

approach for the image segmentation problem. Let
Multi-phase image segmentation (or multi- Ω denote the domain of an input image. Then the
labelling) is an important problem in image process- task is to find a partition { i }in=1 of Ω which mini-
ing and has a wide range of applications in related mises the following energy functional
areas such as computer vision (Paragios, Chen, & n
Faugeras 2005), stereo (Kolmogorov & Zabih 2002,
Kolmogorov & Zabih 2004a) and 3D reconstruc-
minn
{ i }i =1
∑ ∫Ω i
fi ( x ) ⎛
{Ωi }in=1 ⎞⎠
⎝{Ω
i =1
tion (Vogiatzis, Esteban, Torr, & Cipolla 2007).The
basic task in image segmentation is to optimally s.t. ∪in=1 Ωi = Ω and
d k l ∅ for
f k ≠ l , (1)
assign different labels to different pixels of an
image with respect to some energy functional. where R( ) is a regularisation term. Concrete repre-
In the spatially discrete setting, the image is usu- sentations for { i }in=1 are needed so that we can design
ally modelled as a graph, and the solution of the numerical algorithms to solve (1). Over the last 30
multi-phase image labelling problem can be found years, many representations have been proposed, for
by computing the min-cut or max-flow solutions example the level set method (Osher & Sethian 1988,
of the graph, see (Kolmogorov & Boykov 2005, Chan & Vese 2001). In this work, we are interested
Kolmogorov & Zabih 2004b, Boykov, Veksler, & in the piecewise constant representations where each
Zabih 2001, Boykov & Kolmogorov 2003) and ref- region is represented by a unique binary or integer
erences therein. In the spatially continuous setting, value. With the piecewise constant representations,
the variational approach has been widely studied, convex relaxations can often be constructed for (1)
where the problem is formulated as the minimiza- and so efficient and robust numerical algorithms
tion problem of a continuous energy functional. can be developed based on those relaxations. So far
Compared with the graph-based method in the there are three typical piecewise constant representa-
discrete setting, the variational method has several tions for { i }in=1 and each representation will result
advantages: 1) it can avoid the metrication errors in a particular form of (1).
thanks to the crucial rotation invariance property; Binary value representation: For each partition
2) a wide range of reliable numerical schemes are Ωi , we can define an indicator function
available, and these schemes can be easily imple-
mented and accelerated; 3) it requires less mem-
⎧1 x ∈ Ω i
ory in computation; 4) it is easy to use GPU and ui ( x ) = ⎨ i 1, …, n.
parallel processors. ⎩0 x ∉ Ω i ,

17

VIPIMAGE15_Book.indb 17 9/8/2015 7:41:01 PM

 With these indicator functions, model (1) with n n −1
the Potts regulariser R ⎛⎝⎜{ i }in=1 ⎞⎠ ∑ in=1 | ∂Ωi | can be λi {
min
}, λ ∈B
∑ ∫Ω ( λi λi ) fi dx α ∑ ∫ | ∇λi | dx
d ,
Ω
rewritten as1 i =1 i =1
(4)
n
where
min
ui ∈{ },u ∈S
∑ ∫Ω ui fi dx + ∑ ∫Ω | ∇ui | dx
d (2)
B={ }.
i =1 i =1
, , n : = ≥ ≥ n =
where
After solving (4), the labelling function u can be
⎧ n ⎫ recovered by u ∑ in=1 ( λi λi ) i .
S ⎨(u1, …, un ) : ∑ ui = 1, ui ≥ 0 ⎬ . Product of binary values representation: This rep-
⎩ i =1 ⎭ resentation was was first given in (Lie, Lysaker, & Tai
2005, Lie, Lysaker, & Tai 2006a) extending the mul-
Such a representation was used in (Lellmann, tiphase level set framework of Vese-Chan (Vese &
Kappes, Yuan, Becker, & Schnörr 2009, Zach, Chan 2002). In the case of two-phase labelling, its
Gallup, Frahm, & Niethammer 2008). In (Bae, convex relaxation and global minimization was
Yuan, & Tai 2011), global minimization property proposed in (Chan, Esedoglu, & Nikolova 2006).
for this model was proven. In (Yuan, Bae, Tai, & For simplicity, assume n = 2 m . Define w0 ( s ) = 1 − s
Boykov 2010), this model was interpreted as a min- and w1( s ) = s and introduce m binary function
cut problem with a corresponding max-flow model. φ1 φm : Ω → {0,1} such that x ∈Ωi if and only if
Integer value representation: Let u : Ω → { , , n} φ1 φm is a binary representation of the integer i.
be a labelling function such that u( x ) = i if x ∈Ωi , With these notations, model (1) with the regulariser
R ⎛⎝{ i in=1 ⎞⎠ = ∑ i =1 ∇φi ddx can be formulated as
m
i 1, …, n . Let ψ i (u ) be the corresponding indi-
cator functions for Ωi which is related to u. Then n m m
model (1) with the regulariser R ⎛⎝⎜{ i }in=1 ⎞⎠ ∑ in=1 | ∂Ωi | wa (φk ) fi d
reduces to
min ∫ ∑∏
φ ∈{ , } Ω
k =1
i
k
+ α ∑ ∫ ∇φk dx,
Ω
(5)
i i =1 k =1

n n
where a1i … ami is a binary representation of i, denoted
u ∈{
min
}
∑ ∫Ω fi i (u ) ∑ ∫Ω | ∇ i ( u)) | dx
d , (3) by i [a1i ami . The interpretation of this model as
i =1 i =1
a min-cut problem and its corresponding max-flow
model was given in (Bae, Lellmann, & Tai 2013,
This model was first introduced in (Lie, Bae & Tai 2015, Liu, Tai, Leung, & Huang 2014).
Lysaker, & Tai 2005, Lie, Lysaker, & Tai 2006b).
Augmented Lagrangian was used in (Lie, Lysaker, &
Tai 2005, Lie, Lysaker, & Tai 2006b) to solve it. In 2 CONTINUOUS MAX-FLOW
(Bae & Tai 2009), discrete graph cut was used to APPROACHES
solve this model which is very much related to the
graph based method in (Ishikawa 2003, Darbon & Three piecewise constant representations and the cor-
Sigelle 2006). The interpretation of this model as a responding formulations for labelling problems were
min-cut problem and its corresponding max-flow introduced in the last section. All the three formula-
model was given in (Bae, Yuan, Tai, & Boykov tions are non-convex because either the constraints
2010, Bae, Yuan, Tai, & Boykov 2014a). This prob- are non-convex or both the objective functional
lem can also be transformed into a binary mini- and the constraints are non-convex. A number of
zation problem. If we replace the regulariser by approaches have appeared in the literature recently
R ⎛⎝{ i }in=1 ⎞⎠ = ∫ ∇u dx and introduce showing that these models can be interpreted as
Ω continuous min-cut problems. There exists a corre-
sponding continuous max-flow model for each of
⎧1 if u( x ) > i these three min-cut problems. Exploring the connec-
λi ( x ) = ⎨ , i 1, …, n − 1
⎩0 if u( x ) ≤ i tion between these continuous min-cut and max-flow
problems, convex global minimization method could
be derived for these non-convex problems.
and λ0 ( ) = 1, λn ( x ) = 0 . Then model (1) can also Since the pioneer work of Chan et al. (Chan,
be written as a binary minimization problem (Liu, Esedoglu, & Nikolova 2006), one of the important
Tai, Leung, & Huang 2014) research topics in image segmentation is to find
good convex relaxations for the non-convex mini-
mization problems arising from the variational
1
In the sequel, we will omit the notation x when there is methods and then design numerical algorithms for
no confusion from the context. the corresponding convex relaxations. In a series of

18

VIPIMAGE15_Book.indb 18 9/8/2015 7:41:02 PM

 research papers (Yuan, Bae, Tai, & Boykov 2014, where in the first equality, we use the fact (Giusti
Bae & Tai 2015, Bae, Yuan, Tai, & Boykov 2014b, 1977)
Bae, Lellmann, & Tai 2013, Yuan, Bae, Tai, &
Boykov 2010), the authors have developed a set of
computationally efficient algorithms for the mod-
∫Ω | |≤α
∫Ω u div qdx
els in the last section. The essential idea is to inter-
pret these models as continuous min-cut problems in the distributional sense; and the third equal-
and each of them has a corresponding continuous ity follows from the minmax theorem (Ekeland &
max-flow problem. This section reviews the con- Teman 1999, Chapter 6, Proposition 2.4). Instead
tinuous max-flow approaches from the primal-dual of regarding (7) as a convex relaxation, our view
perspective. However, we use a different derivation point is that (6) is a min-cut problem over a continu-
to get these continuous max-flow models. ous domain and the last maximization problem in
(8) is a max-flow problem. Problem (7) is the dual
2.1 Two-phase image segmentation problem of (8). The pair of min-max problems (7)
and (8) are continuous analogue of the min-cut and
We first consider the Chan-Vese model (Chan, max-flow problems in the graph based methods
Esedoglu, & Nikolova 2006) for the two phase seg- (Yuan, Bae, & Tai 2010, Yuan, Bae, Tai, & Boykov
mentation problem 2014). In the discrete case, it is well-known that
the min-cut problem is equivalent to the max-flow
} ∫Ω ∫Ω uCt dx + ∫Ω ∇u ddx.
min ( u )Cs dx
d (6) problem. In the continuous setting, the following
u ∈{ ,
three problem are equivalent, (Yuan, Bae, Tai, &
Boykov 2014, Prop. 3.1), and thus gives another
Model (6) can be viewed as a special case of way to explain the convex relaxation model (7):
either the formulation (2) or (5). Since (6) is a non-
convex optimization problem, it is not tractable to Min-cut problem ( ) ⇔ Max-flow problem
p oblem ( )
find its global solution. Thus the authors propose ⇔ Dual of max-flow problem ( )
to relax the binary valued function u to 0 1
and solve the following convex problem From the above derivation, we can see that the
so called cut for (7) is the Lagrangian multiplier for
the flow conservation constraint div q − ps + pt = 0
min ∫ ( u )Cs dx
d ∫Ω uCt dx + ∫Ω ∇u dx
d . (7)
u ∈[ ,
] Ω of the max-flow problem.

Moreover, it is proved in (Chan, Esedoglu, & 2.2 Multi-phase image segmentation

Nikolova 2006) that the solution to (6) can be We will derive the max-flow models for the
obtained by thresholding the solution to (7). How- multi-phase image formulations (2), (4) and (5)
ever, the numerical algorithms for (7) usually suf- one by one and show that the equivalence between
fer from the non-smoothness of the TV term. We the min-cut and max-flow problems can also be
can form a dual problem of (7) as follows: extended to multi-phase cases.

min ∫ ( u )Cs dx
d ∫Ω uCt dx + ∫Ω ∇u ddx 2.2.1 Binary value representation
u∈[ ,
] Ω Following the convex relaxation technique for the
= min max ∫ ∫Ω upt dx + ∫Ω u div qdx two-phase Chan-Vese model, a natural convex
u∈[ , ] ps Cs Ω
pt ≤Ct
relaxation for (2) is
| |≤α n
= min max ∫ min ∑ ∫ ui fi dx + ∑ ∫Ω ∇ui dx.
u∈R ps ≤Cs Ω
∫Ω upt dx + ∫Ω u div qdx u S
i =1
Ω
i =1
(9)
p ≤Ct
|q|≤α
We can form a dual problem of (9) by introducing
= max min ∫ ∫Ω upt dx + ∫Ω u div qdx another n variables pi i = 1, , n which corresponds
ps ≤Cs u∈R Ω
p ≤Ct to the data fidelity term and an auxiliary variable ps:
| |≤α
n n
= max min ∫
ps ≤Cs u∈R Ω
∫Ω u ( − + ) dx ( 1
min
n )∈S
∑ ∫Ω ui fi dx + ∑ ∫Ω ∇ui dx
i =11 i 1
p ≤Ct ⎛ n ⎞ n
| |≤α

⎧ p ≤ Cs , pt ≤ Ct , | | ≤ α
= min max ∫
ui ≥ 0 ps
⎜1 −
Ω⎜
⎝
∑ ⎟
ui ⎟ ps dx ∑ ∫Ω ui fi dx
i =1 ⎠
= max ∫ ps dx
d s. t. ⎨ s i =1

⎩div q − ps + pt = 0,
Ω n
+ ∑ ∫Ω ∇ui dx
(8) i =1

19

VIPIMAGE15_Book.indb 19 9/8/2015 7:41:06 PM

 ⎛ n ⎞ n follows from the minmax theorem. The maximisa-
= min max ∫ ⎜⎜1 − ∑ ⎟
⎟ + ∑ ∫ ui fi dx tion problem (12) also corresponds to a continuous
ui ∈R ps Ω Ω
⎝ i =1 ⎠ i =1 maxflow model (Bae, Yuan, Tai, & Boykov 2014b).
p ≤ fi
| |≤α
n 2.2.3 Product of binary values representation
+ ∑ ∫ ui div qi dx The energy functional in (5) is also non-convex in
Ω
i =1 addition to the non-convexity of the constraints.
⎛ n ⎞ n In order to find a convex relaxation of (5), we
= max min ∫ ⎜⎜1 − ∑ ui ⎟⎟ ps ddx + ∑ ∫ ui fi ddx can compute the pixel-wise convex envelop of
ps ui ∈R Ω ⎝ ⎠
Ω the data term, which can be obtained by com-
i =1 i =1
p fi
|qi |≤α puting the bi-conjugate of the functional twice
n
(Bae, Lellmann, & Tai 2013). The bi-conjugate of
+ ∑ ∫ ui div qi dx g (u ) := ∑ in 1 ∏ mk =1 wak (uk ) fi uk ∈{ } ( k = , , m ) is
Ω
i =1
⎧ m n m ⎫
g ∗ ( p ) = max ⎪
∑ pk uk − ∑ ∏ wa (uk ) fi ⎪⎬ .
n
= max min ∫ ps dx + ∑ ∫ ui ( qi ps pi ) dx
d ⎨
uk ∈{ , } ⎪
i
k ⎪
ps ui ∈R Ω Ω ⎩ k =1 i =1 k =1 ⎭
i =1
p ≤ fi
|qi |≤α
Consequently the bi-conjugate of g ∗ is
⎧ pi fi , | qi | ≤ α
= max ∫ ps dx s. t. ⎨
⎩div qi − ps + pi = 0,
Ω (10) ⎧m ⎫
g ∗∗ (φ ) maxm ⎨∑ φk pk − g ∗ ( p )⎬
p∈R ⎩k =1 ⎭
where the third equality follows from the minmax
⎧m ⎛ n m m ⎞⎫
theorem. The maximization problem (10) corre-
= max⎪⎨∑ φk pk + in ⎜
⎜ ∑∏ (uk ) fi − ∑ pk uk ⎟⎟ ⎪⎬
sponds to a type of max-flow model in the spatially p∈R m ⎪⎩k =1 uk ∈{ , } ⎝ ak
⎠ ⎪⎭
i =1 k =1 k =1
continuous domain (Yuan, Bae, Tai, & Boykov
2010). m
= am ∑ { }
p∈R
k =1
2.2.2 Integer value representation
Relaxing λi ∈{ , } to λi ∈[ 0,1] gives the convex m
relaxation of (11) s. t. p0 ≤ −∑ uki pk + fi , for any i = [ u1i … umi ],
k =1
n n −1
min ∑ ∫ ( λi − λi ) fi dx + α ∑ ∫ | ∇λi | d . (11)
λ ∈B Ω Ω where the constraint in the last equality follows
i =1 i =1
from the fact
By fixing λ0 = 1 and λn = 0, the dual problem of
n m
(11) can be computed as follows:
∑ ∏ wa i
k
⎛ i ⎞
⎝ uk ⎠ fi fi if i [ u1i umi ].
i =1 k =1
n n−1
min ∑ ∫ ( λi − λi ) fi dx + α ∑ ∫ | ∇λi | dx
λ ∈B
i =1
Ω
i =1
Ω Thus the problem (5) with the regularisation
n n−1 term can be relaxed to
= min max ∑ ∫ ( λi − λi ) pi dx + ∑ ∫ λi div qi dx
λi ∈R pi ≤ fi Ω Ω m m
i =1 i =1
|qi |≤α
n n−1
minmax ∫ p ddx
φ p Ω ∫Ω ∑ φk kd α∫
Ω
∑ | ∇φk | dx
k =1 k 1
k=
= max min ∑ ∫ ( λi − λi ) pi dx + ∑ ∫ λi div qi dx m m
pi fi λi ∈R Ω Ω
|qi |≤α
i =1 i =1 = min max ∫
φ p ,|qi |≤α Ω
∫Ω ∑ φk kd +∫
Ω
∑ φk div qk dx
d
n−1 k =1 k =1
d + ∑ λi (
= max min ∫ p1dx ) dx m
pi ≤ fi λi ∈R Ω
| |≤α
i =1
+
= max min ∫ p0dx + ∫
p ,|qi |≤α φ Ω Ω
∑ φk (di qk + pk dx
d
k =1
⎧ p fi ,| qi | ≤ α ⎧p ≤
∑ k=1 uki pk
m
= max ∫ p1dx
d s. t. ⎨ i fi
Ω
⎩div qi pi + pi +1 = 0, ⎪⎪ 0
(12) = max ∫ p0dx
d s. t. ⎨| qk | ≤ α
Ω
⎪div q + p = 0.
where in the first equality 1 i n 1 for λi and qi , ⎪⎩ k k

whereas 1 ≤ i ≤ n for pi ; again the second equality (13)

20

VIPIMAGE15_Book.indb 20 9/8/2015 7:41:08 PM

 Moreover, if the data term of (5) is sub-modular, 3.2 Primal-dual method
(13) is a tight convex relaxation for (5). In particu-
In this section, we will provide another method
lar, the data term is sub-modular when n = 4 (Bae
for solving the continuous max-flow problems in
& Tai 2015).
Sec. 2. As it can be observed from the derivation
process, each continuous max-flow approach is
associated with a min-max problem of the form
3 GENERAL ALGORITHMIC
FRAMEWORK min max F ( p γ ), (16)
γ ∈Cγ p ∈C p
3.1 Augmented lagrangian method q∈Cq

The maximisation problems (8), (10), (12) and (13) where F ( p q γ ) is (linear) functional of p, q
share the following general form and γ . Then following the general set-up of the
primal-dual method in (Chambolle & Pock 2011,
⎧ p Cp Esser, Zhang, & Chan 2010), we can solve (16) as
⎪ follows (Alg. 2):
max F ( p ) s. t. ⎨q Cq , (14)
p
⎪⎩Lq + Dp = 0,
∂F ( p ,q l ,γ l )
• pl pl + τ lp ,
∂pp
where ∂F ( pl ,q ,γ l )
• ql q l + τ lq ,
∂q
• F ( p ) is a (linear) functional of p,
∂F ( pl ,q l +1 ,γ )
• L and D are linear mappings, • γl γ l − βl ,
∂γ
• C p and Cq are convex sets.
where τ lp ,τ lq and β l are adaptive stepsizes. The
The augmented Lagrangian formulation of (14)
primal-dual method is parameter free compared
is
with the augmented Lagrangian method in Sec. 3.1
c 2 as there is no proximal regularizer in Alg. 2. More-
L( γ ) = F ( p ) + γ , Lq + Dp − Lq + Dp . over, we can choose the stepsizes automatically to
2 (15) accelerate the algorithm, for example see Table 1.
In general, τ li (i p q ) and β l should be inverse
Then with a triple of intial points ( 0 0 γ 0 ), to each other, which are proportional to l and 1/l
we can solve (14) by the augmented Lagrangian respectively.
method as follows (Alg. 1):

• p
l +1
arg max p C p L( p, q l ,γ l ), 4 NUMERICAL EXPERIMENTS
• q
l +1
q Cq
( p , q, γ ),
l l
This section compares the aforementioned two
• γ l
γ − c ( Lq Dp
l
D ). algorithms on the image segmentation problems.
The tests are conducted on a Mac Pro laptop with
When there are no explicit solutions to the first Intel Core i5 CPUs @ 2.4 GHz and executed from
two subproblems, they can be solved approximately Matlab. For the discretized version of the gradient,
by several projected ascent iterations (Yuan, Bae, divergence and TV operators, we refer the readers
Tai, & Boykov 2014, Bae & Tai 2015, Bae, Yuan, to (Zhu 2008, Ch. 1.3).
Tai, & Boykov 2014b, Bae, Lellmann, & Tai 2013,
Yuan, Bae, Tai, & Boykov 2010). The penalization
parameter c is chosen by trial-and-error in numeri-
cal tests.

Table 1. Stepsize selections for the primal-dual method

(Alg. 2).

τ lp τ lq βl

Cameraman 0.1 ∗ l 0.04 ∗ l − 0.02 1

0.08∗l + 0.2
Figure 1. Input images for image segmentation. Left:
3 color 0.005 ∗ l + 0.03 0.005 ∗ l 1
cameraman for two-phase image segmentation; Right: 3
0.02 ∗l
color for multi-phase image segmentation.

21

VIPIMAGE15_Book.indb 21 9/8/2015 7:41:11 PM

 Figure 2. The first row: rescaled labelling function returned by the primal-dual method (Alg. 2) before ((a) and (c))
and after ((b) and (d)) thresholding; The second row: rescaled labelling function returned by the augmented Lagrangian
method (Alg. 1) before ((e) and (g)) and after ((f) and (h)) thresholding. The thresholding value is 0:5.

Table 2. Computational results of the image segmentation for the two-phase (cameraman) and multi-phase (3 col-
our) image segmentations by the primal-dual and the augmented Lagrangian methods. The algorithm terminates if el
defined in (17) is less than ε.

Cameraman 3 color

ε =1 −3 ε =1 −5 ε =1 −3 ε =1 −5

Iter Time (s) Iter Time (s) Iter Time (s) Iter Time (s)

Primal-dual (Alg. 2) 32 0.15 118 0.51 25 0.11 65 0.25

Augmented Lagrangian 32 0.16 149 0.67 40 0.18 215 0.89
(Alg. 1)

In our tests, the data term has the form We test the primal-dual and the augmented
fi | u − ci |β , where u is the input image and ci Lagrangian methods on the standard cameraman
is the average intensity of the region i which is image (see the left of Fig. 1) for the two-phase
assumed to be fixed. Apparently, fi is convex image segmentation problem and on the standard
when β ≥ 1 and non-convex when β < 1. Here β noisy 3 color image (see the right of Fig. 1) for the
is set to 1. multi-phase image segmentation problem. Due to
In order to compare the convergence rate of the the limited space, we only conduct the tests for the
two tested algorithms, we first (numerically) com- variational model with the integer value represen-
pute the optimal primal energy, denoted by E ∗ , by tation in Sec. 2.2.2. In Alg. 1, c is selected to be 0.3,
running either of the algorithms 10000 iterations. and the adaptive stepsize selections in Alg. 2 are
Then we evaluate the progress of the algorithm by listed in Table 1.
computing the relative error of the primal energy The segmentation results are plotted in Figure 2.
defined as By comparing the plots before and after thresh-
olding, we can see that both the primal-dual and
| El − E∗ | the augmented Lagrangian methods can achieve
el = , (17)
E∗ very good performance since they all converge to
a result which takes the values 0 and 1 almost eve-
where E l is the primal energy in the lth iteration. rywhere. The computational results are presented

22

VIPIMAGE15_Book.indb 22 9/8/2015 7:41:16 PM

 in Table 2 which includes the number of iterations Esser, E., X. Zhang, & T.F. Chan (2010). A general frame-
and computational time needed for the algorithm work for a class of first order primal-dual algorithms
to converge to certain accuracy. Table 2 clearly for convex optimization in imaging science. SIAM J.
shows that the primal-dual method with the adap- Imaging Science 3(4), 1015–1046.
Giusti, E. (1977). Minimal surfaces and functions of
tive stepsize selections can be faster than the aug- bounded variation. Australian National University.
mented Lagrangian method. Ishikawa, H. (2003). Exact optimization for markov
random fields with convex priors. IEEE Transac-
tions on Pattern Analysis and Machine Intelligence 25,
REFERENCES 1333–1336.
Kolmogorov, V. & Y. Boykov (2005). What metrics can
Bae, E., J. Lellmann, & X.-C. Tai (2013). Convex relaxa- be approximated by geo-cuts, or global optimiza-
tions for a generalized chan-vese model. In A. Hey- tion of length/area and flux. In ICCV 2005, pp.
den et al. (Eds.), EMMCVPR 2013, LNCS 8081, pp. 564–571.
223–236. Springer-Verlag Berlin Heidelberg 2013. Kolmogorov, V. & R. Zabih (2002). Multi-camera scene
Bae, E. & X.-C. Tai (2009). Graph cut optimization reconstruction via graph cuts. In European Conference
for the piecewise constant level set method applied on Computer Vision, pp. 82–96.
to multiphase image segmentation. In Scale space Kolmogorov, V. & R. Zabih (2004a). What energy func-
and variational methods in computer vision, pp. 1–13. tions can be minimized via graph cuts. IEEE Transac-
Springer. tions on Pattern Analysis and Machine Intelligence 26,
Bae, E. & X.-C. Tai (2015). Efficient global minimization 65–81.
methods for image segmentation models with four Kolmogorov, V. & R. Zabih (2004b). What energy func-
regions. J Math Imaging Vis 51, 71–97. tions can be minimized via graph cuts. IEEE Transac-
Bae, E., J. Yuan, & X.-C. Tai (2011). Global minimiza- tions on Pattern Analysis and Machine Intelligence 26,
tion for continuous multiphase partitioning problems 65–91.
using a dual approach. International journal of compu- Lellmann, J., J. Kappes, J. Yuan, F. Becker, & C. Schnörr
ter vision 92(1), 112–129. (2009). Convex multi-class image labeling by simplex-
Bae, E., J. Yuan, X.-C. Tai, & Y. Boykov (2010). A fast constrained total variation. In X.-C. Tai et al. (Eds.),
continuous max-flow approach to non-convex mul- SSVM 2009. LNCS, Volume 5567, pp. 150–162.
tilabeling problems. UCLA, Applied Mathematics, Springer, Heidelberg 2009.
CAM-report-10–62. Lie, J., M. Lysaker, & X.-C. Tai (2005). A piecewise con-
Bae, E., J. Yuan, X.-C. Tai, & Y. Boykov (2014a). A fast stant level set framework. Int. J.Numer. Anal. Model
continuous max-flow approach to non-convex multi- 2(4), 422–438.
labeling problems. In Efficient Algorithms for Global Lie, J., M. Lysaker, & X.-C. Tai (2006a). A binary level set
Optimization Methods in Computer Vision, pp. 134– model and some applications to mumford-shah image
154. Springer. segmentation. Image Processing, IEEE Transactions
Bae, E., J. Yuan, X.-C. Tai, & Y. Boykov (2014b). A fast on 15(5), 1171–1181.
continuous max-flow approach to non-convex multi- Lie, J., M. Lysaker, & X.-C. Tai (2006b). A variant of the
labeling problems. In A. Bruh et al. (Eds.), Global level set method and applications to image segmenta-
Optimization Methods, LNCS 8293, pp. 134–154. tion. Math. Comp. 75(255), 1155–1174.
Springer-Verlag Berlin Heidelberg 2014. Liu, J., X.-c. Tai, S. Leung, & H. Huang (2014). A new
Boykov, Y. & V. Kolmogorov (2003). Computing geo- continuous max-flow algorithm for multiphase image
desics and minimal surfaces via graph cuts. In ICCV segmentation using super-level set functions. Journal
2003, pp. 26–33. of Visual Communication and Image Representation
Boykov, Y., O. Veksler, & R. Zabih (2001). Fast approxi- 25(6), 1472–1488.
mate energy minimization via graph cuts. IEEE Trans- Osher, S. & J. Sethian (1988). Fronts propagating with
actions on Pattern Analysis and Machine Intelligence curvature dependent speed: algorithms based on
23, 1222–1239. hamilton-jacobi formulations. J. Comput. Phys. 79(1),
Chambolle, A. & T. Pock (2011). A first-order primal- 12–49.
dual algorithm for convex problems with applications Paragios, N., Y. Chen, & O. Faugeras (2005). Handbook
to image. J Math Imaging Vis 40, 120–145. of Mathematical Models in Computer Vision. Springer
Chan, T.F., S. Esedoglu, & M. Nikolova (2006). Algo- New York.
rithms for finding global minimizers of image segmen- Vese, L. & T. Chan (2002). A new multiphase level set
tation and denoising models. SIAM J. Appl. Math. framework for image segmentation via the mumford
66(5), 1632–1648. and shahs model. International Journal of Computer
Chan, T.F. & L.A. Vese (2001). Active contours without Vision 50, 271–293.
edges. IEEE Transactions on Image Processing 10(2), Vogiatzis, G., C.H. Esteban, P.H. Torr, & R. Cipolla
266–277. (2007). Multi-view stereo via volumetric graph-cuts
Darbon, J. & M. Sigelle (2006). Image restoration with and occlusion robust photo-consistency. PAMI 29(12),
discrete constrained total variation part i: Fast and 2241–2246.
exact optimization. Journal of Mathematical Imaging Yuan, J., E. Bae, & X. Tai (2010). A study on continuous
and Vision 26(3), 261–276. maxflow and min-cut approaches. In Computer Vision
Ekeland, I. & R. Teman (1999). Convex analysis and vari- and Pattern Recognition (CVPR), 2010 IEEE Confer-
ational problems. SIAM. ence on, pp. 2217–2224. IEEE.

23

VIPIMAGE15_Book.indb 23 9/8/2015 7:41:18 PM

 Yuan, J., E. Bae, X.-C. Tai, & Y. Boykov (2010). A contin- Zach, C., D. Gallup, J.-M. Frahm, & M. Niethammer
uous max-flow approach to potts model. In K. Danii- (2008). Fast global labeling for realtime stereo using
lidis, P. Maragos, and N. Paragios (Eds.), ECCV 2010, multiple plane sweeps. In Vision, Modeling and Visu-
Part VI, LNCS 6316, pp. 379–392. Springer-Verlag alization Workshop (VMV).
Berlin Heidelberg 2010. Zhu, M. (2008). Fast numerical algorithms for total vari-
Yuan, J., E. Bae, X.-C. Tai, & Y. Boykov (2014). A study ation based image restoration. Thesis.
on continuous max-flow and min-cut approaches.
Numerische Mathemetik 126(3), 559–587.

24

VIPIMAGE15_Book.indb 24 9/8/2015 7:41:18 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Image restoration using variational approaches: Some recent advances

A. Lanza, S. Morigi & F. Sgallari

Department of Mathematics, University of Bologna, Bologna, Italy

ABSTRACT: In image formation, the observed images are usually blurred by optical instruments and/
or transfer medium and contaminated by noise, which makes image restoration a classical inverse prob-
lem. A general principle for dealing with the intrinsic numerical instability of most inverse problems is that
of regularization. Among the regularization approaches the most common is Tikhonov regularization.
Another very popular choice in the literature for regularization is based on the Total Variation (TV) semi-
norm. Since then, in order to provide more reliable and efficient models, several variational formulations
have been investigated which involve two terms, the data-fidelity term and the regularization term, with
the goal to capture characteristic and essential image features. In this review, we will focus on variational
methods for the restoration of images corrupted by additive noise which we assume to be sampled from a
known a-priori distribution. In particular, we will discuss recent advanced proposals for the regularization
and fidelity terms in the optimization problem. Numerical results illustrate the efficacy of the different
models.

1 INTRODUCTION functions, in such a way that important informa-

tion, encoded in image features like textures and
A real captured image is deteriorated during for- details, is often compromised in the restoration
mation, transmission, and recording processes. process. In this review, we will focus on variational
Classically, this degradation is the result of two methods for the restoration of images corrupted
phenomena. The first one is deterministic and is by additive noise which we assume to be sampled
related to the acquisition, to possible defects of from a known a-priori distribution.
the imaging system (blur created by an incorrect Without loss of generality, we consider grayscale2
lens adjustment, by motion, for example) or to images with a square d × d domain. Let u ∈R d
phenomena such as atmospheric turbulence. The be the unknown d × d clean 2 2
image concatenated
second phenomena is a random one and corre- into an d 2 -vector, K2 ∈R d d be a known blurring
sponds to the noise coming from any signal trans- operator and n ∈R d be an unknown realization of
mission. There are basically three standard noise the random noise process, which we assume white
models in imaging systems: additive, multiplicative with zero mean and standard deviation σ . The
and impulse noise. Moreover, typical noise models discrete imaging model of the degradation process2
are characterized by the shape of their probability which relates the observed degraded image g ∈R d
density function, which in the discrete setting is with u , can be expressed as follows:
represented by the noise histogram.
Image restoration refers to the recovery of a g Kuu + n .
K (1)
clean sharp image from a noisy, and potentially
blurred, observation and it consists of both image Given K and g, the goal of restoration is to solve
deblurring and denoising processes. There are the inverse problem of recovering an estimate u*
several advanced mathematical image restora- of u from g, which is known as deconvolution or
tion approaches that have been developed and are deblurring. When K is the identity operator, recov-
being used today: stochastic modeling, wavelets ering u from g is referred as denoising. It is well
(Daubechies & Teschke 2005), and variational/ known that deblurring by direct inversion of K,
Partial differential Equation (PDE) approaches. i.e. u∗ K −1g , can yield meaningless results due to
In general, the nonlinear-diffusion PDE-based noise and to the fact that the linear blur operator
methods and TV-based regularizations, succeeded K is typically ill-conditioned or singular (Hansen
in obtaining good quality, edge preserving resto- 1998). To obtain a meaningful restoration, some
rations, especially for noise removal. However, prior information is commonly included in the
they modify images towards piecewise constant formulation.

25

VIPIMAGE15_Book.indb 25 9/8/2015 7:41:18 PM

 Variational approach to restoration typically One aspect generally missing from state-of-the-
consists in solving the optimization problem art image restoration methods is a full exploitation
of all the available information about the noise.
u∗ ← g i 2
{ u + u g }, (2) Instead of the standard L2 fidelity term in (2), in
u ∈R d order to explicitly exploit the assumption that the
noise distribution is known, in the third approach
where the regularization term R( ) || A( ) ||qq we present (Sect. 4) we introduce a novel fidelity
(A is a regularization operator, not necessarily lin- term that enforces the similarity between the resid-
ear) enforces certain prior constraints on the clean ual distribution and a target noise distribution.
image u, the fidelity term F ( ) = || Ku − g || pp In Sect. 5 we will report some experimental
measures how u fits the observed image g, and μ is results which allow to point out the potentials of
a positive regularization parameter balancing the the three considered variational approaches.
two terms. In particular, the fidelity term is typi-
cally related to the type of noise corrupting the
image, while the regularization term commonly 2 ADAPTIVE FRACTIONAL
enforces smoothness of the solution. REGULARIZER
Many different variational restoration mod-
els have been proposed, ranging from the clas- In this section, we briefly summarize the work in
sical Tikhonov ( p = 2, q = 2) (Hansen 1998) (Chan, Lanza, Morigi, & Sgallari 2013), which
(Calvetti & Reichel 2003), to Total Variation presents a variational model for restoration of tex-
(Rudin, Osher, & Fatemi 1992), Mumford-Shah tured images corrupted by blur and impulse/Gaus-
(Jung, Bresson, Chan, & Vese 2011), and many sian additive noise. In particular, the model relies
other variants of these. on an adaptive fractional order regularization term
Total variation regularization was originally and on a L1 fidelity. The model is as follows:
introduced for noise reduction (Rudin, Osher, &
Fatemi 1992), and is obtained from (2) by setting u∗ ← g i { K
Ku g A u }, (3)
p = 2, q = 1, and A(u ) the gradient magnitude of u ∈R d
2

u. If we let ∇ i ::=
= (Gx ,i u,G
,G y,i u )T , with Gx,i, Gy,i rep-
resenting the ith rows of the x and y-directional where Λ = diag( λ1 λd ) is an d 2 d 2 diago-
finite difference operators Gx, Gy, respectively, then nal matrix with λi representing the regularization
the regularization term R( ) is defined by the parameter for the ith pixel, Aα (ui ) = || ∇α i ui ||2 ,
TV-semi-norm where α i represents the fractional order of differ-
entiation for the ith pixel, and
d2
|| u ||TV = || A( ) ||1: ∑ ( x ,i )2 ( yi )2 .
∇α i = (Gxα,ii ,,G
::= G yα,ii u )T (4)
i =1

The distinctive feature of TV regularization is is the fractional-order discrete gradient opera-

that image edges can be preserved, but the restora- tor, with components representing the x and
tion can present staircase effects. y-directional fractional finite difference operators.
For what concerns the fidelity term, the L2 norm Let us motivate our model by analyzing the
is used for images corrupted by additive Gaussian high-pass filtering character of the fractional
noise. This standard choice corresponds to setting order derivative operator. In Figure 1 we show
p = 2 in (2), which indirectly encodes the prior the restored images of the blurred and noisy
knowledge on the noise standard deviation σ . image in Figure 3(b) by applying model (3) with
When noise is of impulsive type, it is well known different α while keeping λ = 1.0. With α = 1 we
that better performance are obtained by using the get the 1-TV model which preserves edges but
L1 norm.
In this work, we will first present two variants
of the general variational restoration model in (2):
the first relies on an adaptive regularization opera-
tor which is constructed by using fractional order
derivatives (Sect. 2), the second proposes a fidelity
term which imposes the resemblance of the residue
image to a white noise realization by constraining
its autocorrelation function (Sect. 3). These two Figure 1. Restoration of image shown in
approaches are particularly suitable for the resto- Figure 3(b) using model (3) with different values of α
ration of partly-textured perturbed images. while λ = 1.0 is fixed.

26

VIPIMAGE15_Book.indb 26 9/8/2015 7:41:20 PM

 fails to preserve fine scale features such as tex- In this section we illustrate the variational
tures in the images, see Figure 1(a). The high-pass method proposed in (Lanza, Morigi, Sgallari, &
capability becomes stronger with larger α , so Yezzi 2013) for image denoising and extended in
more and more texture regions are well preserved (Lanza, Morigi, & Sgallari 2015) for image resto-
when α increases to 1.5, 1.8 and then 2.0, see Fig- ration, which relies on a TV regularization term
ures 1(b)–(d), respectively. However, for α = 2.0 (simply for its popularity, any other regularizer
the high pass capability of fractional order deriva- could be used as well) and on a fidelity that con-
tives is too strong so that most components of the strains the residual of the restoration to resemble a
image remain and only little high frequencies noise white noise process.
is removed. Thus a best filtering is obtained by The model proposed in (Lanza, Morigi, &
adaptively applying the fractional order regulari- Sgallari 2015), referred to as TV-W, extends the
zation in texture regions and TV regularization in classical constrained TV-L2 restoration model (5),
non-texture regions. by incorporating constraints on the whiteness of
The method consists of two phases. In the first the residue image. The model replaces the feasible
phase, we apply a texture detector (we refer to set defined in (6) as follows:
(Chan, Lanza, Morigi, & Sgallari 2013) for more
details) to obtain a texture map. The texture map u∗ ← g i || u ||TV , (7)
contains a texture measure (a real positive value) u W
for each pixel and is thresholded to obtain the tex- 2
ture and non-texture classes. The texture class is where the new feasible set W ⊂ R d contains solu-
further partitioned into C subclasses according tions u such that the corresponding residue image
to the texture measure. Following this classifica- Ku − g satisfies the discrepancy principle and
tion a value α i is associated to each pixel i, with is guaranteed to be white. We refer to (Lanza,
α i = 1 if the ith pixel belongs to the non-texture Morigi, & Sgallari 2015) for a detailed descrip-
class, and α i ∈{{ 1 C } if the ith pixel belongs
tion of the minimization algorithm based on the
to one of the C texture subclasses. The regulariza- Alternating Directions Method of Multipliers
tion parameters λi in the diagonal matrix Λ in (3) (ADMM) (Chan, Tao, & Yuan 2013).
are then chosen according to the α i ’s. In particular, In order too define W, we consider the periodog-
2
each texture class has an associated regulariza- ram pn ∈R d of the noise realization n defined as
tion value, while for the non-texture class we set pn =| T n |, where | ⋅ | denotes the component-wise
2 2
λ = 1.0. In the second phase, we apply the classi- modulus of the complex vector T n , T ∈Cd d is
cal TV regularization (1-TV) to the non-texture obtained using the unitary matrix representing 1D
regions while applying a fractional order TV regu- Discrete Fourier Transform and the Kronecher
larization (1-TVα) in the texture classes. The cor- product. The vector pn is suitably scaled by multi-
responding minimization problem is solved by the plication with a diagonal normalization matrix M,
half-quadratic method in (Nikolova & Ng 2005) i.e. pˆ n = Mpn .
(see (Chan, Lanza, Morigi, & Sgallari 2013) for Then, the elements of the normalized peri-
details) odogram are sorted in increasing order of spatial
frequency by multiplication with a suitable permu-
tation matrix Π. The Normalized Cumulative Peri-
3 WHITENESS CONSTRAINING odogram m2
(NCP) of n is thus defined as the vector
FIDELITY cn ∈R d given by

When the standard deviation σ of noise is avail- cn := S Π pˆ n , (8)

able, under the discrepancy principle, the uncon-
strained TV-L L2 model obtained by setting p = 2, where S is the d2 × d2 partial sums matrix, i.e. a
q = 1 in (2), can be equivalently reformulated as the lower triangular matrix with all nonzero entries
following constrained optimization problem: equal to one.
Hence, the whiteness set W used in model (7) is
u∗ ← g i || u ||TV , (5) as follows:
u U

with the feasible set defined as W {u ∈ d

: b ≤ S M | T Ku − g | b },
U {u ∈ d
: Ku
K −g d }, (6) where the inequalities must be intended component-
wise and b + , b − ∈ R d are vectors containing
2

where τ is a scalar parameter controlling the vari- the upper and lower limits of the whiteness set,
ance of the residue image Ku − g . respectively.

27

VIPIMAGE15_Book.indb 27 9/8/2015 7:41:24 PM

 4 HISTOGRAM CONSTRAINING where Hg−u denotes the normalized cumulative his-
FIDELITY togram of the residue image g − u, i is the index
of Q bins of the histogram, and H L and H U rep-
Similarly to the approach illustrated in the pre- resents the lower and upper limits, respectively, of
vious section, the work in (Lanza, Morigi, the band around the theoretical noise histograms.
Sgallari, & Yezzi 2014) presents a constrained For the computation of the H L and H U limits, a
variational model for restoration based on a Monte Carlo approach is used: given a target prob-
TV regularization and on an extension of the ability density function for the noise and a selected
discrepancy-based constraint (6). In particular, image dimension d, we generate a large number of
instead of considering only the variance of the different noise realizations by sampling from the
residue image, the approach in (Lanza, Morigi, target distribution. For each realization, the nor-
Sgallari, & Yezzi 2014) constrains the overall distri- malized and cumulative histograms are computed.
bution of the residual to resemble the distribution Finally, the minimum and maximum values within
of noise, which is assumed to be known. all the realizations for all the histogram bins are
This concept is better illustrated in Figure 2. obtained.
Three synthetic examples of additive image noise all To compute the minimizer of the proposed
with the same mean μ = 127.5 and standard devia- model (i.e., the restored image), an efficient mini-
tion σ = 27.5 are depicted in the first row, neverthe- mization algorithm based on the ADMM tech-
less the first noise example is completely different nique is presented (Chan, Tao, & Yuan 2013).
from the other two. In particular, the first is a simple
step function, the second and the third are specific
realizations of Gaussian and uniform noise distri- 5 EXPERIMENTAL RESULTS
butions, respectively. In the second row the associ-
ated histograms are shown, which reflect the visual In this section we will give some examples related to
difference we can notice in the illustrations. Even the reviewed restoration algorithms when applied
if the second and the third noisy images in the top to images synthetically corrupted by blur and
row appear visually similar, the associated histo- white Gaussian noise. We compare the proposed
grams are significantly different from each other. algorithms with the well-known Rudin-Osher-
Therefore, by exploiting the complete information Fatemi (ROF) model (Rudin, Osher, & Fatemi
on the noise distribution we expect to obtain a 1992), based on the minimization of the TV-L2 or
residual which better fits the probabilistic model of TV-L1 functionals.
the noise process and, as a consequence, we expect The regularization parameter of the TV-L2
an improvement in the restored image. model is adjusted so that the solution is guaran-
The variational model takes the form: teed to satisfy the discrepancy principle, that is the
variance of the residual must be equal to the vari-
u∗ ← g i || u ||TV , ance of the noise.
u H (9) We stopped the algorithms as soon as the relative
difference between two successive iterates satisfies
where the feasible set H is defined as follows: || u k u k −1 ||22 || u k −1 ||22 < 10 −4 or after a maximum
of 100 iterations.
H= { ∈ : ≤ ≤ ∀ = , , }, The accuracy of the methods is evaluated by the
Signal to Noise Ratio (SNR) defined by

∗ || u ||
SNR( ) g10 dB, (10)
|| u∗ u ||
2
where u∗ ∈ R d is the computed
p estimate of the
2
uncorrupted image u ∈R d . This quantity pro-
vides a quantitative measure of the quality of the
restored image: a high SNR value indicates that u∗
is an accurate approximation of u .
The corrupted images taken as input by the
algorithms have been obtained as follows. First,
the original d d image u is blurred by a Gaussian
kernel characterized by two parameters and
Figure 2. Top row: step image (left), realization of and generated through the MATLAB com-
Gaussian (center) and uniform (right) noise distribu- mand ( ).
tions; bottom row: associated histograms. The larger the , the stronger the blur will be.

28

VIPIMAGE15_Book.indb 28 9/8/2015 7:41:28 PM

 Then, the blurred image Ku
K is corrupted by addi- Table 1. Example 1. Results for restorations of test
tive zero-mean white Gaussian noise with standard images that have been corrupted by spatially-invariant
deviation σ . Gaussian blur, defined by = 3 and = 1.5,
and by noise corresponding to noise-level ν.
Example 1: Adaptive Fractional (AF) algorithm.
We consider the restoration of a blur- and noise- ν SNRi SNRAF SNRL1–TV SNRL2–TV
contaminated test image represented by 255 × 255
pixels. The desired blur- and noise-free image is 0.01 15.98 20.46 20.00 18.22
depicted in Figure 3(a). The resulting image is dis- 0.05 13.29 15.86 15.06 15.48
played in Figure 3(b). 0.10 9.44 13.06 11.56 12.44
In Table 1 the second column with header SNRi
reports SNR values for image that have been
corrupted by spatially-invariant Gaussian blur, the SNR values for restorations by AF algorithm,
characterized by = 3 and = 1.5, and by and columns four and five, labeled SNRL1 TV and
noise with different e2 levels ν . In the case of Gaussian SNRL2 TV , show the SNR values for restorations
noise, let g ∈R d be the associated vector with the by TV regularizer with L1 and L2 data fidelity terms,
unknown noise-free entries, i.e., g = g + e where the respectively.
vector e represents the noise. We define the noise- The adaptive regularization process applies
level ν =|| e ||/||  ||. The values of ν are shown in the different levels of denoising in different image
first column. Column three, labeled SNRAF, shows regions. This improves the result visually, in terms
of texture preservation and smoothing of the
homogeneous regions, and in terms of signal-to
noise-ratio. Figure 3(d) shows the restoration by
AF algorithm and Figure 3(f) depicts the texture-
map used for the restoration. The texture measures
are obtained on the corrupted image in Figure 3(b)
and the four texture classes are shown using gray-
level values. In particular, in black regions the AF
algorithm is applied with α 4 1 0 α 2 = 1.7 , and
α 3 = 1.8 in the gray regions, and α1 = 1.9 in white
regions. The restoration by L1-TV is illustrated in
Figure 3(c) using λ = 0.1 and it preserves textures
but it does not remove the noise. However, when
we apply L2 -TV restoration with λ = 1.0 it does the
opposite, that is, it preserves very well the edges but
it destroys the textures. The restoration by L2 -TV,
illustrated in Figure 3(e), shows less sharpness and
smoothing of the texture regions.
Example 2. Whiteness constraining fidelity
We consider the restoration of skyscraper ((d = 256)
image, which presents interesting mixtures of textures,
flat regions and shaded areas; the noise-free version
of the image is depicted in Figure (4)(a).
In this example we investigate the comparison
between the restoration by TV-L2 and by TV-W
algorithms for a fixed additive Gaussian noise
with standard deviation σ = 5 and variable Gaus-
sian blur degradations, characterized by band
and sigma , which significantly worsen the ill-
conditioning of the image restoration problems.
Table 2 reports in the first and second columns
the parameters band and sigma of the Gaussian
blur kernel, and the third column, labeled SNR0,
Figure 3. Example 1. images: (a) blur- and
noise-free image; (b) corrupted image by Gaussian blur,
reports the SNR-values of the available corrupted
= 3 and = 1.5, 10% of Gaussian noise; (c) image g, i.e., SNR( ) . The columns labeled
L1-TV restoration λ1 = 0.1; (d) restored image by AF TV-L2 and TV-W display SNR-values for the
algorithm; (c) L2-TV restoration λ1 = 0.1; (f) texture restored images obtained by the two considered
classes computed on (b). algorithms. For increasing values of the band and

29

VIPIMAGE15_Book.indb 29 9/8/2015 7:41:30 PM

 The regularization parameter of the TV-L2
model is adjusted so that the solution is guaran-
teed to satisfy the discrepancy principle, that is the
variance of the residual is equal to the variance of
the noise.
The accuracy of the methods is evaluated by the
Root Mean Squared Error (RMSE) defined as

RMSE ( ) := || u u ||22 d 2 , (11)

2
where u ∈R d is the computed approximation
p
2
of
the desired noise-free image u ∈R d . This quantity
provides a quantitative measure of the quality of
the restored image u. A small RMSE value indi-
cates that u is an accurate approximation of u ;
we recall, however, that the RMSE values are not
always in agreement with visual perception.
Figure 4. Example 2. Restoration obtained by TV-L2 We consider the restoration of test (d = 256)
(c) and TV-W (d) models on the image (a)
degraded (d) by Gaussian blur with = 5,
image, which present interesting mixtures of tex-
= 1.5 and additive zero-mean white Gaussian noise with tures, flat regions and shaded areas; the noise-free
standard deviation σ = 10. version of the image is depicted in Figure (5)(a).
The quantitative results in Table 3 demonstrate
that the proposed TV-CDF model outperforms the
Table 2. Example 2. SNR values from restoration of TV-L2 approach in terms of obtained RMSE val-
image corrupted by additive zero-mean white ues on the test image for all the considered noise
Gaussian noise (fixed σ = 5 and varying Gaussian blur). levels.
The visual results for test image corrupted by
SNR0 TV-L2 TV-W zero-mean AWGN with standard deviation σ = 10,
7 2.5 4.89 6.54 7.07
7 1.2 7.18 9.06 10.01
5 1.0 8.39 10.82 11.45
5 0.7 11.35 14.15 14.33
3 1.0 9.88 12.24 12.64
3 0.5 15.85 17.66 17.91

sigma parameters, the textured parts of the images

tend to disappear, consequently the benefits of the
whiteness constraint decrease until the worst case
with band = 7 and sigma = 2.5 , where the SNR-
values for the two algorithms are almost the same.
In Figure 4 we display the degraded images and
the results by applying the two different image res-
toration methods.
A visual inspection of the figures of this exam-
ple together with the results in Table 2 allow us
to conclude that by constraining the whiteness of
the residual, we can go beyond the use of just the
residual norm or the variance of the noise model,
thus providing better restorations.
Example 3: Histogram constraining fidelity
We compare the proposed algorithm, referred to
as TV-CDF, with the well-known Rudin-Osher-
Fatemi (ROF) model (Rudin, Osher, & Fatemi Figure 5. Example 3. Restoration results on
1992), based on the minimization of the TV-L2 image corrupted by zero-mean AWGN with standard
functional (5). deviation σ = 10.

30

VIPIMAGE15_Book.indb 30 9/8/2015 7:41:35 PM

 Table 3. RMSE values for REFERENCES
image corrupted by zero-mean
AWGN with standard deviation σ. Calvetti, D. & L. Reichel (2003). Tikhonov regularization
of large linear problems. BIT Numerical Mathematics
σ TV-L2 TV-CDF 43(2), 263–283.
Chan, R.H., A. Lanza, S. Morigi, & F. Sgallari (2013, 2).
10 7.41 6.82 An adaptive strategy for the restoration of textured
20 11.00 10.34 images using fractional order regularization. Numeri-
30 13.55 13.02 cal Mathematics: Theory, Methods and Applications 6,
276–296.
Chan, R.H., M. Tao, & X. Yuan (2013). Constrained
total variation deblurring models and fast algorithms
based on alternating direction method of multipliers.
are illustrated in Figure (5). In particular, the noise- SIAM J. Imaging Sciences 6(1), 680–697.
free image is depicted in Figure (5)(a), the noise- Daubechies, I. & G. Teschke (2005). Variational image
corrupted image is shown in Figure (5)(b), the restoration by means of wavelets: Simultaneous
images restored by TV-L2 and by TV-CDF are decomposition, deblurring, and denoising. Applied
given in Figure (5)(c) and Figure (5)(d), respec- and Computational Harmonic Analysis 19(1), 1–16.
tively. In the last rows of Figure (5) we show the Hansen, P.C. (1998). Rank-deficient and Discrete Ill-
normalized histograms of the residuals associated posed Problems: Numerical Aspects of Linear Inver-
with the restored images depicted in Figure (5)(c) sion. Philadelphia, PA, USA: Society for Industrial
and Applied Mathematics.
and Figure (5)(d).
Jung, M., X. Bresson, T. Chan, & L. Vese (2011, June).
The effect of the proposed constraint is evi- Nonlocal mumford-shah regularizers for color image
dent from the residual histograms. In fact, the restoration. Image Processing, IEEE Transactions on
residual histograms obtained by the proposed 20(6), 1583–1598.
TV-CDF method and displayed in Figure (5)(f) Lanza, A., S. Morigi, & F. Sgallari (2015). Variational
adhere more faithfully to the target Gaussian image restoration with constraints on noise whiteness.
probability density function (red curve) than Journal of Mathematical Imaging and Vision 53(1),
those reported in Figure (5)(e) and computed by 61–77.
the TV-L2 model. Lanza, A., S. Morigi, F. Sgallari, & A.J. Yezzi (2013).
Variational image denoising based on autocorrelation
whiteness. SIAM J. Imaging Sciences 6(4), 1931–1955.
Lanza, A., S. Morigi, F. Sgallari, & A.J. Yezzi (2014).
6 CONCLUSIONS Variational image denoising while constraining the
distribution of the residual. ETNA. Electronic Trans-
We reviewed three recent proposals of vari- actions on Numerical Analysis [electronic only] 42,
ational approaches for image restoration. We 64–84, electronic only.
Nikolova, M. & M.K. Ng (2005). Analysis of half-
have shown as the choice of suitable regulari-
quadratic minimization methods for signal and image
zation and data fidelity terms is crucial for the recovery. SIAM Journal on Scientific Computing 27(3),
success of the variational model. The aspect of 937–966.
computational efficiency in solving the optimi- Rudin, L.I., S. Osher, & E. Fatemi (1992, November).
zation problem has not been considered in the Nonlinear total variation based noise removal
present short review. algorithms. Phys. D 60(1–4), 259–268.

31

VIPIMAGE15_Book.indb 31 9/8/2015 7:41:37 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Virtual and augmented medical imaging environments: Applications

to simulation, training, surgical planning and interventional guidance

Cristian A. Linte
Department of Biomedical Engineering, Chester F. Carlson Center for Imaging Science,
Rochester Institute of Technology, Rochester, NY, USA

ABSTRACT: While effective therapy cannot be delivered without the appropriate equipment and
instrumentation that enables access to the internal organs and tissues under minimally invasive condi-
tions, real-time visualization and navigation information is critical to guide the instruments to the correct
targets and manipulate the diseased tissues in absence of direct vision. Over the past decades, thanks to
the advances in medical image acquisition, visualization and display, surgical tracking and image comput-
ing infrastructure, a wide variety of technology has emerged that facilitates diagnosis, procedure planning,
intra-operative guidance and treatment monitoring while providing safer and less invasive approaches for
therapy delivery. Nevertheless, successful exploitation, use and integration of the multi-modality images
and signals available from both pre- and intra-operative sources require the development and validation
of engineering tools that allow users to extract critical information for decision making during planning,
visualization, guidance or training. This article serves as a quick snapshot of various applications of
biomedical imaging and modeling, their integration into platforms for planning and guidance of inter-
ventions, and their potential to develop new paradigms for virtual anatomy visualization for simulation,
teaching and training applications.

1 INTRODUCTION (Soares & Murphy 2005). Despite the limited soft

tissue visualization provided by two-dimensional
Several decades ago many surgical interventions (2D) X-ray fluoroscopy imaging, this modality has
required the clinician to have full physical and visual become standard-of-care thanks to its real-time
access to the internal organ that required treatment: capabilities and wide clinical use.
abdominal procedures involved large incisions and The limited extent of direct visualization of the
exposure of inner organs, neurosurgical interven- surgical scene and organs being treated, and often
tions involved the opening of the skull, and cardiac lack thereof, require significant reliance on medi-
interventions were performed on the open-chest, cal imaging and computer-assisted navigation.
while the patient was placed on cardiac arrest and While the less invasive treatment approaches have
connected to a heart-lung machine. Despite the proved to be beneficial to patients, they have intro-
success of the therapy itself, often times patients duced new challenges for the physicians. One such
would experience severe infections, delayed recovery challenge, for example, has been voiced by cardiac
and other complications mainly induced by the large interventions being among the last disciplines to
incisions and tissue exposure needed to access the adopt the minimally invasive treatment techniques,
surgical targets rather than by surgically repairing mainly due to the challenges associated with access
the tissue. Hence, less invasive interventions have and visualization inside the beating heart.
been motivated by improved patient outcome, fewer In addition to its use in the planning and guidance
complications, reduced morbidity, shorter recovery of minimally invasive intervention, medical imaging
times, and improved overall quality of life. and image-guided navigation present a significant
One crucial step toward reducing invasiveness potential for paradigm-shifting approaches for simu-
was the substitution of direct vision by real-time lation, teaching and training. We live in the “digital”
camera views: the introduction of endoscopic age surrounded by technology that enables acqui-
imaging enabled the use of miniature cameras, sition of 3D, 4D and 5D medical images (Fig. 1)
dramatically reducing incision size (Litynski 1999). and stereoscopic and 3D visualization displays have
Another crucial step was the adoption of interven- become commodities. Yet, traditional pedagogical
tional radiology as a surgical sub-specialty, opening approaches still resort to using standard didactic
up the horizon for percutneous, catheter-based equipment (i.e., textbooks, slides) to present human
interventions conducted under X-ray guidance anatomy and describe the interconnection between

33

VIPIMAGE15_Book.indb 33 9/8/2015 7:41:37 PM

 trans-esophageal echocardiograpy has become
standard-of-care for real-time patient monitoring
during most cardiac interventions.
Lastly, MRI has enabled non-invasive, high-
quality visualization inside the human body with
exquisite soft tissue contrast, yet accompanied by its
inherent high cost and demanding infrastructure.
While modalities such as CT and MRI have been
predominately employed for diagnostic imaging
and pre-procedural planning, recent efforts have
been dedicated to adapting and developing the nec-
essary infrastructure to use CT and MRI for intra-
operative real-time guidance and visualization
in the interventional suite. These interventional
Figure 1. Visualizing the possibilities: examples of slice-, approaches require access to specialized technol-
surface- and volume-rendered representations of different ogy compatible with specific high-end imaging
anatomical organ models generated from medical imag- equipment, imposing a highly invasive technologi-
ing datasets. Image courtesy of the Biomedical Imaging cal “footprint” in the interventional suite requiring
Resource, Mayo Clinic, Rochester MN USA. significant modifications to procedure workflow
and re-training. Therefore, despite the envisioned
benefits, such highly-specialized diagnostic imaging
different organs and systems. Teaching and train- systems adapted for intra-operative guidance seem
ing experiences would be much improved by lever- unlikely to become standard-of-care, but rather a
aging the availability of open source, high-quality specialized process only available to those centers
medical imaging datasets to generate 2D slices, 3D that can afford it, thereby limiting access to afford-
surfaces and 4D volume renderings of the anatomy. able, efficient healthcare.
These methods enable the user to interact with the The primary challenge in image-guided
imaging data via commercially available track- interventions is to provide an optimal visualization
ing systems, visualize the interconnected data via environment that spares the clinician from having
widely available and many affordable virtual and/or to mentally fuse information from the pre- and
augmented reality head-mounted displays or other inta-operative images during the procedure. The
routine visualization and display paradigms. development of an ideal environment is not trivial:
pre-procedural images are typically 3D and feature
a higher quality and volume of data, while intra-
2 ENABLING TECHNOLOGIES FOR operative images feature lower resolution, lower
BIOMEDICAL IMAGE VISUALIZATION quality and are often 2D, trading the amount of
AND IMAGE-GUIDED NAVIGATION information displayed for real-time visualization.
While an optimal visualization paradigm is hard
Over the last ten years, all medical imaging modali- to accurately identify, an image guidance system
ties have evolved and their use has expanded much is deemed successful if it provides sufficient infor-
beyond their traditional diagnostic utilization. mation for accurate navigation of the therapeutic
X-ray imaging, along with endoscopy, have been instruments to the target tissue and successful on-site
the first to provide real-time visualization during target tissue manipulation while avoiding damage
procedure guidance in the Operating Room (OR), to critical anatomical structures. Most often, this
and have departed significantly from their initial task is achieved by integrating pre-operative images
use. X-ray images are used to generate three-di- and derived models into the intra-operative setting
mensional (3D) reconstructions of the anatomy (Cleary & Peters 2010, Yaniv & Cleary 2006).
and internal vasculature via Computer Tomog-
raphy (CT), cone-beam CT and CT angiograms.
3 IMAGE-GUIDANCE WORKFLOW
Single-camera endoscopy has been upgraded to
stereoscopic endoscopy, accompanied by several
3.1 Imaging and instrument tracking
attempts toward augmented reality visualization.
Shortly afterwards Ultrasound (US) imaging The typical computer-assisted intervention
found its way into the interventional suite, mainly workflow (Fig. 2) includes several steps, each serving
thanks to its real-time capabilities, relatively low a specific scope. The pre-operative images and ana-
cost (compared to CT or Magnetic Resonance tomical models provide the necessary contextual
Imaging (MRI), wide availability, portability information for the surgeon to identify the region of
and lack of ionizing radiation. Consequently, interest and navigate the surgical instrument to the

34

VIPIMAGE15_Book.indb 34 9/8/2015 7:41:38 PM

 equipment, other paradigms have been explored,
including the projection of virtual information on
the patient’s body surface either directly or using
video augmentation (i.e., in the form of an aug-
mented or virtual reality, depending on the extent
of “real” and “virtual” information presented), or
via the use of stereoscopic head-mounted displays
or commercially available 3D displays (Linte, Dav-
enport, Cleary, Peters, Vosburgh, Navab, Edwards,
Jannin, Peters, Holmes III, & Robb 2013).

3.3 Targeting accuracy: How close are we?

A critical, yet often not explicit step of the image guid-
ance workflow is “verification”: what is the uncer-
tainty associated with the navigation, targeting and
on-site therapy delivery? Typically, the true answer is
provided by analyzing the intra-operative images, as
these images depict the location of the surgical target
most accurately, along with the instrument position
and orientation. While the other components of the
surgical navigation environment and their integra-
Figure 2. Components of a computer-assisted interven-
tion aim to provide minimal error, each component
tion workflow: pre-operative imaging, surgical instrument
localization, data integration via registration, visualization of the platform has its inherent uncertainty, in the
and information display, surgical navigation, and intra- end contributing to the overall targeting accuracy of
operative verification and evaluation. the tracking system. Hence, the verification step is
critical in evaluating the achieved targeting accuracy
of the navigation platform, and while it is not explic-
target, serving as a road map. Provided the patient itly performed during the procedure workflow, users
anatomy is registered to the pre-operative road assess accuracy at each step—navigation and posi-
map (typically using a tracking system), the surgical tioning, by ensuring the information provided by
instrument representations can be visualized in the the guidance platform indeed matches the informa-
same coordinate system as the road map, much like tion provided by the intra-operative images.
a driver using a GPS navigation system to obtain
real time positioning information along the route.
4 APPLICATIONS
Lastly, since pre-operative images cannot faith-
fully depict the intra-operative anatomy given their
4.1 Cardiac intervention applications
acquisition prior to the procedure, real-time images
such as US or cone-bean CT are critical to visualize Limitations and Challenges: Catheter-based tissue
the true, intra-operative target location and confirm ablation interventions for treatment of arrhythmic
accurate and precise on-site therapy delivery. conditions still rely heavily on X-ray fluoroscopy
guidance despite the limited visualization it provides.
The intracardiac ablation path is typically planned
3.2 Integration, visualization and display
prior to the procedure, but cannot be faithfully
All of the above components (pre- and intra- reproduced during the intervention. This challenge
operative images and tracked instrument rep- arises due to the lack of sufficient visualization
resentations) are integrated into the same and navigation information for accurate targeting,
coordinate system that includes the patient via as well as the lack of information with regards to
their inherent registration transformations, a the geometry and pathophysiology of the ablation
mathematically complex step hidden from the lesions. In these cases, despite the complex planning
surgeon, but requiring heavy engineering involve- of the planned irreversible isolation of the electric
ment. However, the appropriate visualization and pathway, patients end up with temporary edema
display of the complex surgical environment is the and recurring arrhythmia requiring continued
critical stage that involves the surgeon and clini- treatment and further procedural interventions.
cal team, as optimal presentation of the available Tentative Solution: Left atrial CT or MRI
information is paramount for procedure guidance images enable the integration of the pre-procedural
and therapy delivery. While traditional 2D displays plan into the guidance data via registration of
have been a part of the routine interventional suite the pre- and intra-operative anatomical models

35

VIPIMAGE15_Book.indb 35 9/8/2015 7:41:39 PM

 (Fig. 3). This approach facilitates the visualization the catheter, their size and colour intensity is mean-
and interpretation of pre-operative information in ingless in terms of providing realistic information
the context of the intra-operative data, augmented with respect to the quality (temperature, duration,
by real-time US imaging (Linte, Lang, Rettmann, thermal dose etc.) of the generated lesion. However,
Cho, Holmes III, Robb, & Peters 2012). the proposed surrogate models can portray such
In addition, the pre-procedural images can information in real time, facilitating ablation lesion
be employed as substrate for modeling the tis- monitoring and providing cues with regards to the
sue response to ablation, along with the physical location at which subsequent ablation lesions should
and physiological tissue parameters, as well as the be delivered to avoid the presence of gaps in the abla-
parameters provided by the ablation system. As a tion line, which would, in turn, lead to incomplete
result, fast and accurate surrogate thermal models isolation of the arrhythmic pathway (Linte, Camp,
that predicts the lesion geometry and lesion quality Holmes III, Rettmann, & Robb 2013).
(extent of reversible and irreversible thermal damage
induced in the tissue) can be developed, providing a
4.2 Spine intervention applications
clear visualization of the superimposed on the ana-
tomical model (Fig. 4). While current approaches Limitations and Challenges: Spine instrumentation
enable users to “drop” spherical glyphs of various procedures require accurate and precise implanta-
size an colour intensity at the locations visited by tion of fixators (surgical screws) into the vertebral
pedicle to immobilize motion at specific vertebral
segments. These procedures entail detailed planning
to identify the optimal implant size and trajectory
to ensure firm fixation and also avoid their interfer-
ence with proximal vascular or neural structures.
Moreover, knowledge of the mechanical strength of
the pedicle body is critical to avoid the instrumenta-
tion of weak bone regions, which would compromise
fixation strength. Nevertheless, current planning
approaches rely on the use of 2D radiographs or
axial CT images to depict the 3D anatomy of the
spine and estimate implant trajectory, despite the
curvature of the spine. In addition, although 3D CT
images of the patient are typically acquired several
days before the procedure, the planning still occurs
in the interventional suite with the patient under
anesthesia, and often continues after opening up the
patient, leading to prolonged intervention times.
Figure 3. a) Example of a high-quality pre-operative Tentative Solution: The proposed application
left atrial model augmented with real-time US image; leverages the availability of the pre-operative
b) Electro-anatomical map of the left atrium generated images several days prior to the procedure, enabling
intra-operatively using a tracked catheter; c) Registration the planning to be conducted by the surgeon or
of pre- and intra-operative left atrial model using point-
cloud to surface registration.
surgical staff prior to the intervention and allow-
ing for for additional consultations if necessary.
The planning platform provides the user with full
control to browse through and select virtual screws
of appropriate dimension and specification from
a database of virtual implants that replicate the
actual implants manufactured by different spine
instrumentation suppliers. The selected pedicle
implants can then be virtually inserted in a 3D
volume-rendered representation of the patient’s
spine CT data and their position and orientation
can be altered as needed to avoid interference
with critical neural or vascular structures (Fig.
Figure 4. a) Example of modeled ablation lesion
showing the irreversible lesion core surrounded by
5). Furthermore, the screw trajectory is opti-
the reversible lesion penumbra; b) Volume-rendered mized according to a surrogate fixation strength
representation of a pre-operative anatomical image estimated based on the dimension of the selected
thresholded into tissue and blood showing the ablation implant and the strength (i.e., bone mineral density)
catheter tip and real-time lesion. of the pedicle bone traveled by the screw (Linte,

36

VIPIMAGE15_Book.indb 36 9/8/2015 7:41:39 PM

 2D displays of the anatomy from textbooks and
slides. Moreover, the training of medical profes-
sionals also entails the use of traditional, “old
school” approaches, further explaining the under-
utilization of available medical imaging data, as
well as visualization and display technology.
Tentative Solution: Here we describe two appli-
cations that leverage the power of virtual and aug-
mented reality to explore different paradigms for
presenting anatomical and functional information
to viewers while exploiting the multi-dimensional-
ity of the data.
The first application aims to provide a virtual
environment for learning the anatomy of the skull
decomposed into several key bone segments. The
dataset was generated using the BodyParts3D
libraries that provide virtual representations of
the anatomy of several organs and their inter-
connected components. The environment allows
users to browse through different components of
the skull anatomy while providing a brief audio
description of the selected component and its adja-
cent, neighbouring components, guiding the user
during the assembly process (Fig. 6). The virtual
environment was designed in Vizard, a well-known
software application for virtual and augmented
reality implementations. The user controls the
view and manipulates the components within the
Figure 5. Snapshop of the spine intervention planning virtual environment using either a 3D mouse or
platform comprising a volume-rendered display of the
(less intuitive) pre-defined key-board strokes. The
CT image with implanted virtual screw templates (a);
orthogonal views showing the screw templates within the environment can be displayed either on a 3D moni-
pedicles (b, c, d), as well as the resulting volume-rendered tor, requiring each user to wear passive polarizing
representation of the instrumented spine segment glasses to enable stereoscopic vision, or on virtual
featuring the virtual implants. reality head-mounted displays, such as the popular
and relatively inexpensive Oculus RIFT.
The second application demonstrates the use of
Camp, K., Huddleston, Robb, & Holmes III 2015). augmented reality displays to enable the visualiza-
As such, provided several trajectories are identi- tion of underlying information beyond the surface
fied, their associated fixation strength is estimated, view of an object seen using real-time video (Pot-
enabling the user to select the trajectory that maxi- ter, Besnch, Dawson-Elli, & Linte 2015). An exam-
mizes fixation strength while avoiding interference ple of such an application in medical practice
with critical anatomy. Therefore, the proposed would be the augmentation of laparoscopic video
platform provides optimal planning solutions showing the surface of the liver surface with inter-
based on objective metrics and yields automati- nal information extracted from pre-procedural CT
cally generated reports that prescribe the selected
implant dimension, trajectory and achieved fas-
tening strength deemed optimal at each vertebral
level, enabling the full extent of the planning (and
additional revisions) to be conducted prior to the
procedure.

4.3 Virtual and augmented visualization

for simulation, teaching and training
Figure 6. Example of a user-interactive virtual anatomy
Limitations and Challenges: Despite the availability building platform for understanding the anatomical com-
of high quality 2D, 3D 4D (3D + time) and 5D ponents and their assembly of the skull. Upon selection,
(space + time + function) medical imaging data, each component is accompanied by a brief audio
current learning techniques still employ abstract, description of itself and its neighbouring components.

37

VIPIMAGE15_Book.indb 37 9/8/2015 7:41:41 PM

 needed to confirm whether less information may,
in fact, be more valuable.
What does the ideal image guidance system look
like? An ideal system best positioned to make
an impact in current clinical practice seamlessly
blends diagnostic, planning, and guidance infor-
mation into a single visualization environment dis-
played at the interventional site. In addition, the
system provides accurate registration and tracking,
high-fidelity visualization and information display,
Figure 7. Example of an augmented display using the enables smooth data transfer, and fits within the
IGT Phantom. When filed with rice, the “IGT” letters traditional interventional suites without interfering
inside the phantom are not visible; however, this informa- with current workflows and typical equipment.
tion can be extracted from a CT image of the phantom
and overlaid onto the real video, using the camera itself
as a tracking device. REFERENCES

Cleary, K. & T.M. Peters (2010). Image-guided interven-

tions: technology review and clinical applications.
or MRI images, enabling the surgeon to “see” what Annu. Rev. Biomed. Eng. 12, 119–142.
lies beneath the surface in the attempt to optimize Hynynen, K. (2011). MRIgHIFU: A tool for image-
the path to resect the tumour. guided therapeutics. J Magn Reson Imaging. 34(3),
The key constraints for these applications is the 482–493.
proper alignment of the real and virtual representa- Linte, C.A., J. Camp, A.K., P.M. Huddleston, R.A.
tions of the objects or organs, which entails accurate Robb, & D.R. Holmes III (2015). Estimating pedi-
tracking of the camera and imaged scene. Several cle screw fastening strength via a virtual templating
options entail the use of external spatial tracking and platform for spine surgery planning: a retrospective
preliminary clinical demonstration. Comput method
localization systems to determine the position and Biomech Biomed Eng: Imaging & Visualization In
orientation of the image scene, camera and other Press, DOI:10.1080/21681163.2014.900725.
elements. Another solution is to employ the camera Linte, C.A., J.J. Camp, D.R. Holmes III, M.E. Rettmann, &
itself as a tracking system and determine in real time R.A. Robb (2013). Towards online modeling for
the position and orientation of the imaged scene rel- lesion visualization and monitoring in cardiac abla-
ative to the camera and use this information, along tion therapy. In Proc. Med Image Comput Comput
with the magnification information, to provide the Assist Interv., Volume 8149 of Lect Notes Comput
correct position, orientation and scaling of the vir- Sci., pp. 9–17.
tual object model when superimposed onto the real Linte, C.A., K.P. Davenport, K. Cleary, C. Peters, K.G.
Vosburgh, N. Navab, p. J. Edwards, P. Jannin, T.M.
video view used as background image (Fig. 7). Peters, D.R. Holmes III, & R.A. Robb (2013). On
mixed reality environments for minimally invasive
therapy guidance: Systems architecture, successes and
5 IMAGE-GUIDED THERAPY: RESEARCH challenges in their implementation from laboratory to
OR COMMODITY clinic. Comp. Med. Imag. And Graph. 37(2), 83–97.
Linte, C.A., P. Lang, M.E. Rettmann, D.S. Cho, D.R.
Computer-assisted intervention environments pro- Holmes III, R.A. Robb, & T.M. Peters (2012). Accu-
vide clinicians with less invasive approaches for racy considerations in image-guided cardiac interven-
accurate and intuitive visualization, replacing or tions: experience and lessons learned. Int J Comput
Assist Radiol Surg. 7, 13–25.
augmenting their direct view of the real world with Litynski, G.S. (1999). Endoscopic surgery: the history,
computer graphics information extracted from the the pioneers. World J Surg. 23(8), 745–753.
pre- and intra-operative medical images. While Potter, M.J., A. Besnch, A. Dawson-Elli, & C.A. Linte
some image guidance environments feature highly (2015). Augmenting real-time video with virtual mod-
complex infrastructures that often overwhelm the els for enhanced visualization for simulation, teach-
user, other systems empower the user, enabling ing, training and guidance. In Proc. SPIE Medical
him/her to carefully select and focus only on the Imaging: Image Perception, Observer performance and
relevant information. Moreover, although human- Technology Assessment, Volume 9416, pp. 941615–8.
computer interaction in image guidance has been Soares, G.M. & T.P. Murphy (2005). Clinical interven-
tional radiology: Parallels with the evolution of gen-
explored to a limited extent, for example when eral surgery. Semin Intervent Radiol 22(1), 10–14.
selecting the optimal visualization strategy for per- Yaniv, Z. & K. Cleary (2006, April). Image-guided
cutaneous procedures in interventional radiology procedures: A review. Technical Report CAIMR
(Hynynen 2011), such studies are the exception TR-2006-3, Image Science and Information Systems
rather than the norm and additional research is yet Center, Georgetown University.

38

VIPIMAGE15_Book.indb 38 9/8/2015 7:41:44 PM

 Contributed papers

VIPIMAGE15_Book.indb 39 9/8/2015 7:41:45 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

The fast method of creating High Dynamic Range image

for fluorescent microscopy applications

A. Bal
Institute of Automatic Control, Silesian University of Technology, Gliwice, Poland

ABSTRACT: In this work simple and effective method for creating of High Dynamic Range (HDR)
images for use in the fluorescence microscopy applications is presented. For creating of the HDR image
the Camera Response Function (CRF) is used, so the pixel values of the HDR image, in general, are pro-
portional to the signal which have been reached the camera sensor (i.e. irradiance of sensor). Therefore
the obtained HDR image can be used not only for qualitative analysis of the specimen, but also for its
quantitative analysis. The proposed method has been tested on real data.

1 INTRODUCTION An example of a case in which the camera

dynamic range is too low for proper acquisition
The fluorescence microscopy is one of the most of the specimen image is presented in Figure 1. In
versatile and widely used imaging method in biol- this Figure the sequence of images, acquired with
ogy and medicine researches (Waters 2009). This different exposure time t, of the same cell nuclei
method is mainly used for visualization of the loca- is presented. These images were obtained with the
tion of cells or tissue components that have been usage of fluorescence microscopy and DAPI fluo-
stained whit fluorescent molecules. It is also used for rescent dye, which is used for detection of DNA.
the quantitative assessment of fluorescent stained Hence, in this case, the signal intensity is related
components. The advantages of the fluorescence to the amount of DNA contained in different cell
microscopy are e.g. its: relative simplicity, versatility, sites. As it can be seen the dynamic range of object
high sensitivity and selectivity. signal is too high for the used camera. Therefore,
The usage of the fluorescence microscopy is there is no possibility of acquisition of a single,
connected with the problem of limited dynamic proper exposed, image on which, both, low and
range of cameras used for image acquisition. This high object signals are properly reproduced.
problem is particularly acute in applications in The Figure 2 presents profiles of images (a)–(c)
which e.g.: from Figure 1. The profiles are calculated for pixels
which are lying on the line segment AB (the red
• signal associated with a single object has greater
line in the image 7(a)). The profiles are presented
dynamic than can be acquired by a camera,
for image brightness (pixel values) normalized to
• group of objects such that signal associated sep-
the range [0; 1], which corresponds to the dynamic
arately, which each object has a dynamic range
range of a LDR image. It can be seen in the image
lower than camera dynamic range, but the dif-
7(b) that any of the images 1(a)–(c) is properly
ference between signals from different objects is
exposed—the image 1(a) is underexposed and the
greater then the dynamic range of the camera.
images 1(b) and 1(c) are overexposed.
In the second case objects can be located not In those applications where dynamic range
only in the near neighbourhood by also in different of LDR image is insufficient the idea of the
places of one or more microscopy specimens. High Dynamic Range (HDR) imaging can be
In such cases the limitation of camera dynamic used (McCann & Rizzi 2012). The usage of HDR
range leads to under or over exposition of the images in microscopy imaging is not a new idea.
obtained single images. For clarity, since now, It was presented, e.g. in Bell et al. (2009) and in
such images i.e. images obtained directly from Eastwood and Childs (2012), but there is lack of
the camera will be described as the Low Dynamic example of usage of the HDR imaging method in
Range (LDR) images. The usage of LDR images fluorescence microscopy for qualitative and quan-
leads to the loss of information contained in the titative specimen analysis. The aim of this work
analysed specimen, and therefore the analysis of was the investigation of the possibility of the usage
such images can give false results. of HDR imaging in such applications.

41

VIPIMAGE15_Book.indb 41 9/8/2015 7:41:45 PM

 Figure 1. Images of the same cell nuclei acquisited in DAPI channel with different exposure times t; the signal satura-
tion (τ > 0.975) is indicated on red.

All images I i ∈ I —except the last image I imax

i.e. image acquires with shortest exposure time—
contain areas of overexposure (signal saturation).
It was assumed that overexposure area ωi in image
Ii satisfy following condition
ωi { i ( , y) Ii ( y ) > τ }, (1)

where (x, y) are the coordinates of the pixel, Ii (x, y)

is the value of the pixel pi (x, y) in the image Ii , and
τ is a threshold parameter. For a normalised range
of image values, i.e. I i ( x y ) ∈ [ 0 1] , the value of τ
is close to 1.
The image I is created in an iterative process.
Let Ii is the HDR image after ith iteration, hence
I Iimax . The whole process starts form the image
obtained for the longest exposure time t1 i.e.
I1 = I1. (2)

The image Ii for i > 1 is calculated as follows

Ii ( i 1) α i Ii ( i ) τ + Δi , (3)

where

ti −1
Figure 2. Profiles A–B of images 1(a)–(c) i.e. for αi = (4)
DAPI channel: (a) cell image with marked line segment
ti
AB (red) for which the profiles have been calculated,
(b) image profiles in normalized range of pixel values. and

1
2 DESCRIPTION OF THE METHOD
Δi = ∑ Ii −1( x, y).
|| Ci −1 || pi −1∈Ci −1
(5)

In the proposed method the HDR image I is The C( j ) is an inner contour of region ωj and
assembled from two or more LDR images I i ∈ I , it is defined as follows
where I = { 1, 2 , , i , , imax } . The index i repre-
sents exposure time ti of the image Ii, and images Cj { j( y) p j ( ) j ∈ (ω j , m )}, (6)
I i ∈ I are sorted in the descending order of its
exposure time i.e. ti ti +1. The images Ii and Ij are where ∈( j , ) is a result of morphological erosion
differ only in the exposure times ti and tj used for of region ωj with structuring element m, which in
their acquisition. this case is defined as 3 × 3 array of 1.

42

VIPIMAGE15_Book.indb 42 9/8/2015 7:41:45 PM

 In the proposed method of HDR image creation
the LDR images Ii used in this process have to be
linear in the sense of values Ii (x, y), i.e. image bright-
ness, which are the camera response on the amount
of light reaching each pixel of the sensor (sensor
irradiance) Ei (x, y). However, in case of the real
camera, such response is non linear. This camera
response can be described by the Camera Response
Function (CRF) (Grossberg and Nayar 2003), and
this function can be used for linearisation of non
linear images obtained from a camera.
Let Ii ( x y ) represents image value for pixel
pi (x, y) obtained directly from the camera after
the acquisition process with exposure time ti. The Figure 4. Comparison of HDR and LDR image
profiles A–B (Fig. 7(a)) for DAPI channel: black line—
image I, which is linearised version of I , is calcu-
profile of HDR image, coloured lines—profiles of LDR
lated as follows images obtained for different exposure times; pixel values
are shown in normalized range, where 1 is the maximum
I ( x y ) CRF 1( I( x y))
y )), (7) pixel value in LDR image.

where CRF−1 is the inverse function of CRF, which is

calculated for the camera used to capture images I .
The images after linearisation, i.e. I, are the input
data to the process of creating HDR image.

3 EXPERIMENT AND ITS RESULTS

The proposed method was tested on imaging of

human cells specimen stained with two fluorescent
stains i.e. DAPI and FITC. For image acquisi-
tion the Olympus cellR excellence system based on
Olympus IX81 inverse fluorescent microscope with
a Hamamatsu Orca-R2 camera was used (Fig. 3).
The image acquisition was performed for each
imaged cell in two channels, i.e. independently for
both fluorescent dyes, and for each channel appro-
priate configuration (i.e. the dichroic mirror, exci-
tation and emission filters) of the microscope was
used. For each cell two series of LDR images, one
in each channel, were acquired. The differences

Figure 5. Maps of SNR for LDR and HDR image in

Figure 3. The Olympus cellR xcellence microscope DAPI channel: (a) LDR image acquired with exposure
system with Hamamatsu Orca-R2 camera which was used time t = 175 ms, (b) HDR image; (c) difference ΔSNR for
in the experiment. images (a) and (b).

43

VIPIMAGE15_Book.indb 43 9/8/2015 7:41:49 PM

 between images Ii in each series I are only results
of difference in their exposition time ti.
The exemplary series of LDR images obtained
for the same cell are presented: for DAPI channel in
the Figure 1 and for FITC channel in the Figure 6.
In the both figures the places in each image where
the signal is saturated is marked in red.
In Figures 4 and 7 the profiles of the obtained
HDR images are presented. For creation of HDR
images for both channels the τ = 0.975 was used.
Additionally, in Figures 5 and 8 comparison
between theoretical Signal-to-Noise Ratio (SNR)
of LDR and HDR images are presented. The SNR
for LDR images is calculated as follows

ψ Ii ( x y)
NRLDRi ( x y ) = 20 log10
SNR , (8)
ψ I i ( x y ) + ηr 2 + ηd 2

where ψ is a full well capacity of the used camera,

ηr is a readout noise and ηd is a dark noise. In case
of the used camera, according to the manufacturer
data (Hamamatsu Photonics):ψ 18000 ηr = 5.5 e

Figure 7. Comparison of image profiles for FITC

channel: (a) cell image in FITC channel with marked line
segment AB (red), (b) profiles A–B: black line—profile
of HDR image, coloured lines—profiles of LDR images
obtained for different exposure times; pixel values are
shown in normalized range (1 is the maximum pixel value
in LDR image).

and ηd = 5 ⋅ 10 −4 e// i l/s ⋅ ti . Because maxi-

mal exposition time (ti) used during g experiment
max(ti ) = 10s the value ηd  ψ I I ( x y ) and also
ηd  ηr therefore the value of ηd has not been
taken into account in the calculation of SNR.
For calculation of SNR for HDR image
(SNRHDR) crucial is that adding a constant
to the value Ii(x, y) and multiplying the value
Ii(x, y) by a constant does not change the value of
SNR
NRLDRi ( x y ) . Therefore

SNR
NRHDR ( x y ) SNRLDR
S y ) |Ii ( x y ) = f
Ri ( x y) I i ( x y )) , (9)

where Ii ( x y ) = f I i ( x y )) means that the value

of pixel p(x, y) of HDR image is obtained from
the value of LDR image II (x, y) according to the
equation (3).
Additionally, in Figures 5 and 8 the difference
ΔSNR between SNR for HDR image and SNR of
the best LDR image for appropriate channel is pre-
sented (Fig. 5(c) and Fig. 8(c)); ΔSNR is defined
as follows
Figure 6. Images of the same cell nuclei acquisited in
FITC channel with different exposure times t; the signal
saturation (τ > 0.975) is indicated on red. ΔS ( , y ) = SNRHDR ( x, y ) − SNR
N LDR ( x, y ). (10)

44

VIPIMAGE15_Book.indb 44 9/8/2015 7:41:54 PM

 contains the whole information about signals of
the analysed specimen what is very important for
obtaining proper results of analysis. There is no
overexposure in the HDR images, therefore the
information about strong signal is not loose, and
the weak signals are also maintained.
A good example of this is the Figure 4 where
small changes of the profile height at about 50
and 200 pixels of the A–B profile are maintained
together with maintaining of the very strong signal
at about 280 pixel of the profile. As can be seen
on the profiles of LDR images presented in this
figure, the LDR image can not maintain whole
information about those two kind of signals. What
is important, because of the linearisation of the
images Ii , the pixel values of HDR image are pro-
portional to the signal—in case of LDR images Ii
this relationship is not true.
The proposed method can be used not only
for fluorescence microscopy imaging, but also
for other imaging methods used in biology and
medicine. An a good example of such new appli-
cation area of the proposed method are e.g.
methods based on the gel electrophoresis (Corley
2005) like western blot technique (Mathews
et al. 2009) or 2D gel electrophoresis (Berth et al.
2007).
As it is presented in Figures 5 and 8 the
SNR values of the HDR images for both chan-
nels are much better than the SNR values of
the corresponding LDR images without signal
saturation. What is important SNR for HDR
images is much bigger, than SNR of LDR images,
for weak signals.
Of course with the creation of HDR images a
lot of problems are associated. The first of them
is the choice of the τ parameter value. Differences
between HDR images obtained for different values
of τ can be expressed as

Iτ ( , y ) − I ( , y )
Δ τ ( , y) = , (11)
I ( , y)

where
Figure 8. Maps of SNR for LDR and HDR image in
FITC channel: (a) LDR image acquired with exposure
1
time t = 2.5 s, (b) HDR image; (c) difference ΔSNR for
images (a) and (b).
I( y) ∑ Iτ (
|| T || τ ∈T
y ) ⋅ 100%, (12)

and T = { } is a set of tested τ values. The Figure 9

4 CONCLUSIONS presents in pseudocolour maximum absolute val-
ues of Δ τ ( , y ) , i.e.
The obtained result shows that proposed method
can be useful for image acquisition in applications  ( , y) =
Δ (| Δ τ ( , y ) |), (13)
in which not only qualitative but also quantita- τ ∈T
tive analysis have to be performed. The method
of creating HDR images presented in this work for DAPI and FITC channels obtained for
is simple and effective. The obtained HDR image T = { . , . , . , . }. As it can be seen, in

45

VIPIMAGE15_Book.indb 45 9/8/2015 7:41:57 PM

 images obtained for different exposure times
means better HDR image. However, because of
e.g. photobleaching of fluorescent stain the total
exposure time

imax
T ( x y) ∑ ti ( x y), (14)
i =1

which is independently calculated for each place of

the specimen, cannot be increased without limit.
This problem will be one of the aim of the future
research.
There are at least two important problems which
have to be investigated during future researches.
One of them is the calculation of CRF. This func-
tion can be treated as a camera fixed parameter.
Therefore, in practice, for each camera CRF can
be determined only once, and then it can be used
in each experiment. Thus, there is a need for effec-
tive and accurate methods for calculation of CRF.
The second issue is to develop an objective method
of assessing the accuracy of the images obtained
HDR. These both problems will be the aim of the
future researches.

ACKNOWLEDGEMENTS

This work was partially supported by Polish

National Science Centre (grant no. NN518 283640)
and from funds for statutory activities of the Insti-
tute of Automatic Control, Silesian University of
Technology.
The calculations were performed using the
infrastructure supported by POIG.02.03.01-24-
 values for T = {0.9, 0.95, 0.975, 0.99}.
Figure 9. The ΔI 099/13 grant: GeCONiI—Upper Silesian Centre
for Computational Science and Engineering
Presented experiment was performed in the
Laboratory of Digital Image Processing and Bio-
both channels, for most pixels Δ  ( , y ) ≤ 2.5% .
medical Visualization, which works in the Bio-
However, for a larger differences of between used technology Centre of the Silesian University of
τ values the ΔI  ( x, y ) value is bigger, e.g. for
Technology.
T = { . , . , . , . , . , . , . } for most
pixels Δ ( , y ) ≤ 5% . In both cases, the error
associated with the creation of HDR images is REFERENCES
acceptable from the viewpoint of the biological
purpose of the specimen analysis. However, the Bell, A.A., J. Brauers, J. Kaftan, D. Meyer-Ebrecht,
reason of such large relative change of Δ  ( , y) A. Bocking, & T. Aach (2009). High dynamic range
value requires an explanation during further microscopy for cytopathological cancer diagnosis.
IEEE Journal of Selected Topics in Signal Processing
researches.
(Secial Issue on: Digital Image Processing Techniques
Another interesting problem connected with for Oncology) 3(1), 170–184.
HDR image creation is the choice of expo- Berth, M., F.M. Moser, M. Kolbe, & J. Bernhardt (2007).
sure times t for acquisition of LDR images. The The state of the art in the analysis of two-dimensional
choice of ti influences the quality of the obtained gel electrophoresis images. Applied Microbiology and
HDR images. Usually the usage og more LDR Biotechnology 76(6), 1223–1243.

46

VIPIMAGE15_Book.indb 46 9/8/2015 7:42:01 PM

 Corley, R.B. (2005). A Guide to Methods in the Biomedical Mathews, Suresh, T., P. Plaisance, Eric, & T. Kim (2009).
Sciences. The Wiley—IS&T Series in Imaging Science Imaging systems for westerns: Chemiluminescence vs.
and Technology. Springer US. infrared detection. In B.T. Kurien and R.H. Scofield
Eastwood, B.S. & E.C. Childs (2012, January). Image (Eds.), Protein Blotting and Detection, Volume 536 of
alignment for multiple camera high dynamic range Methods in Molecular Biology, pp. 499–513. Humana
microscopy. In Proceedings IEEEWorkshop on Press.
Applications of Computer Vision, pp. 225–232. McCann, J.J. & A. Rizzi (2012). The Art and Science of
Grossberg, M.D. & S.K. Nayar (2003). Determining the HDR Imaging. The Wiley—IS&T Series in Imaging
camera response from images: What is knowable? Science and Technology. John Wiley & Sons, Ltd.
IEEE Transactions on Pattern Analysis and Machine Waters, J.C. (2009). Accuracy and precision in quantitative
Intelligence 25(11), 1455–1467. fluorescence microscopy. Journal of Cell Biology
Hamamatsu Photonics. ORCA-R2. Digital CCD camera 185(7), 1135–1148.
C10600-10B. Nechnical note. Hamamatsu Photonics.

47

VIPIMAGE15_Book.indb 47 9/8/2015 7:42:02 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Automatic cheek detection in digital images

M. Frackiewicz, H. Palus & K. Radlak

Institute of Automatic Control, Silesian University of Technology, Gliwice, Poland

ABSTRACT: This paper deals with some problems of cheek detection in colour images. Authors
reviewed the literature in this field and proposed a new effective method for cheek detection. A character-
istic feature of the proposed method is the use of 13 essential face landmarks generated by a known face
landmark detector. In the paper four methods from the literature and one’s own method are described
in detail. All these methods have been tested experimentally using a suitable face image database PUT.
Preliminary tests showed the advantages of our method. Additionally we investigated the impact of head
movements on the location detected cheeks. A small number of false cheek pixels lying outside the face
contour and a small colour error caused by movements of the head predetermines the proposed method
for further research and applications.

1 INTRODUCTION (Su et al. 2013) both cheeks are detected after the
previous using the Viola-Jones algorithm for eye
Several applications of face image processing may and mouth detection. Based on the colours of
require detecting facial regions such the mouth, cheek pixels the haemoglobin and melanin com-
eyes, eyebrows, nose, cheeks and chin. Automatic ponents are separated, making it easier to assess
cheek detection is an important operation in this the health of a person. Sometimes the colours of
field. The cheek colour well represents the face the face in selected locations (five blocks each of
colour, because the cheeks are not covered by hair 32 × 32 pixels) are determined and on the basis
as sometimes the forehead or the chin and not of these colours such liver diseases as hepatitis
generate highlights as the nose area. Each cheek are diagnosed (Liu and Guo 2007). Two of these
is a consistent patch of skin. Among the typical blocks are located on the surface of both cheeks
cheek detection applications are medical diagnos- through finding another facial features. Some
tics, human face beautification as well as coding of other application of this approach is described in
videophone sequences. the paper (Kim et al. 2008), that presents an idea to
Person’s emotional state generates colour use the colours of the forehead centre and the lips
changes in its facial skin (Ramirez et al. 2014). for cardiovascular disease detection.
In this work the colour changes were considered An original application of the cheek detection is
and measured in three regions of the face, i.e. the used to beautify the human face in photographic por-
forehead and both cheeks. These regions of inter- traits (Lai et al. 2013). Beautification is performed
est limited to small squares were determined on the by the total removal or the magnitude reduction of
basis of data from the facial feature tracker. Good unwanted skin details: scars, pores, blemishes, wrin-
results are obtained from the combination of the kles, etc. The developed skin changing system can,
forehead and the left cheek. Automatic cheek on the one of the early processing stages, automati-
detection may be particularly useful in medical cally extract the cheeks by using the active shape
diagnostics. Human health evaluation based on modelling and the watershed algorithm.
the colour of his/her face is used in traditional Automatic cheek detection was also applied in
oriental medicine for thousands of years (Yang the field of model-based coding of videophone
et al. 2012, Seo et al. 2014, Zhao et al. 2014). The sequences (Hess and Martinez 2004). The human
result of the latest research is a strong statistical face in such sequence can be segmented into few
relationship between perceptually uniform colour facial regions such as e.g. cheeks. The cheek borders
space CIELAB components in specific regions of were detected based on the well-known SUSAN
the face and pathological patterns used in orien- algorithm (Smith and Brady 1997) and additional
tal medicine. In the paper (Yang et al. 2012) both knowledge-based rules.
cheek skin boxes are detected after the previous The main goal of this study is to compare the
eye and mouth detection. Similarly, in the work automatic cheek detection methods existing in the

49

VIPIMAGE15_Book.indb 49 9/8/2015 7:42:02 PM

 literature with a new method proposed in this paper. between the eye centres, mouthx—mean value of x
This article is organized as follows. In Section 2 we coordinates of mouth corners, eyex—mean value
shortly present four known methods and one our of x coordinates of eye centres.
proposal for automatic cheek detection. The exper- The third known method (Su et al. 2013) gener-
imental results for colour face images from PUT ates three bounding boxes: two located on the eyes
face database are included in Section 3. Finally, the and one located on the mouth. Detected region
Section 4 concludes the paper. representing the left cheek is above the mouth box
and reaches the left eye box and has its width. The
right cheek region is also above the mouth box and
2 CHEEK DETECTION METHODS reaches the right eye box preserving its width.
The fourth method described in (Hess and
In the literature we searched the cheek detection Martinez 2004) uses for each cheek detection
methods which determine the cheeks by means the the landmarks located in the mouth corners,
primary and secondary facial landmarks. There on the lower edge of eyes and six landmarks on
are methods based on a small number of face land- the face contour. Cheek regions are defined by the
marks, e.g. 27 landmarks (Campadelli et al. 2007) segments that connect these facial landmarks. The
and a large number of landmarks e.g. 192 land- cheek region is limited from the face centre by a
marks (Kazemi and Sullivan 2014). Unfortunately, vertical line passing through the mouth corner,
for our problem was found in the literature only a from the top by a line passing through the lower
few solutions. edge of the eyes and from the bottom by a line
The first known method (Xian 2011) is passing through a contact place between the head
based on the coordinates of eye centres: and shoulders.
Leye ( xl , yl ), Reye ( xr , yr ) and coordinates of The new cheek detection method, proposed by
mouth centre: M ( xm , ym ). This method assumes the authors, is based on selected facial landmarks.
that the face is oriented vertically: xl xr = xlr. We decided to use open-source Davis King’s Dlib
The horizontal distance between the eye centres: library (King 2009, King 2015a), in which the faces
B = | yr − yl | and the vertical distance between the are detected using Histograms of Oriented Gradient
eye centres and mouth centre: A = | xm − xlr | are (HOG) descriptors (Dalal and Triggs 2005) trained
calculated. Detected cheek regions are rectangles with structural SVM based training algorithm (King
with following coordinates of the left upper cor- 2015b) and facial landmarks are detected using the
ners and following lengths of sides: method proposed in (Kazemi and Sullivan 2014).
This method generates 68 landmarks in the well-
Lcheek [( xlr + A 4 yl ); A/ ; B / ] (1) established landmark configuration of Multi-PIE
(Gross et al. 2010) presented in Figure 1.
Rcheek [( xlr + A 4 yr − B /2 ); A/ ; B / ] (2)

The second method described in (Yang et al. 2012)

determines the cheek centres: ( cl ycl ), ( cr ycr ),
where xcl xcr = xclrl. Unfortunately, the size of the
square areas around of these cheek centres has not
been defined. The authors of paper present three
ways of determining the cheek centres depending
on the accuracy with which are known the eye and
the mouth positions. In our implementation of this
method assumes that the cheek regions are squares
with defined centres and side lengths equal: B/2.
Generator of landmarks fairly accurately locates the
eyes and the mouth. Therefore, for the determination
of cheek centres we adopted following formulas:

xclrl xcr = xclr ( mouthhx y x ) /2

eeye (3)
ycl = mouthhyl mouthhD /3 (4)

ycr mouthhyr mouthhD /3 (5)

where: mouthhD m hyr | —distance

mouthyl mouth Figure 1. The 68 Multi-PIE landmarks scheme and the
between the mouth corners, B = | yr − yl | —distance landmarks selected for our method marked by the circles.

50

VIPIMAGE15_Book.indb 50 9/8/2015 7:42:02 PM

 In our work we additionally considered also
Zhu and Ramanan face detection and landmarks
localization method (Zhu and Ramanan 2012).
The performance of both methods were evaluated
using an error measure introduced in (Cristinacce
and Cootes 2006) on a subset of 2193 near-frontal
images from PUT face database for which a con-
tour model were manually annotated. This meas-
ure is the mean value of distance between the
landmarks detected by tested algorithm and manu-
ally marked points divided by the inter-ocular dis-
tance (measured as the Euclidean distance between
the outer corners of the eyes). The comparison of
these algorithms is presented in Figure 2. As can
be observed the facial landmarks localization pro-
posed in (Kazemi and Sullivan 2014) and imple-
mented in the Dlib library outperforms the Zhu
Ramanan method (Zhu and Ramanan 2012) and
was chosen for later tests. Exemplary results of
landmarks detected on PUT database and manu-
ally marked landmarks are presented in Figure 3.
Among so many landmarks our method uti-
lizes thirteen selected points (eight points on the
Figure 3. Exemplary results of landmarks detection
face contour, two mouth corners, two inner eye
and comparison with ground truth.
corners and one nose landmark). The border of
the right cheek region is created by a connection
of the following four landmarks (2, 3, 4 and 5)
from the right face contour and is located between cheek is detected in a similar manner. In this case
the ear and the chin. Last fourth landmark (pt.5) the landmarks from left face contour, left corner of
is connected to the right corner of the mouth the mouth and the inner corner of the left eye, are
(pt.49). Then, from the right corner of the mouth used. The same nose landmark (pt.29) is used for
a line is passing in the direction of inner corner detection of both cheeks.
of the right eye (pt.40). Another line is run from The described methods use different facial
the starting point on the face contour (pt.2) in landmarks. The first three methods apply the hori-
the direction of the landmark located on the nose zontal and vertical lines of the mouth and eyes only
(pt.29). Both lines are running until they intersect. and cheek regions are rectangles. It is known that
The result is a closed region of the cheek. The left rectangles inaccurately describe the cheek regions.
The fourth method additionally uses facial con-
tour points and the above mentioned lines, which
causes an unnatural shape of cheeks. The proposed
method is based on facial contour points and does
not use those lines, which creates an opportunity
for more detailed description of the shape of the
cheek.

3 EXPERIMENTAL TESTS

The four described above methods and one

new method proposed by the authors have been
implemented. The detected cheek regions, as
results of initial tests, are shown in the image of
hairdresser styling head (Fig. 4).
In our further tests the PUT face database
(Kasinski et al. 2008) was applied because of its
advantages: high resolution (2048 × 1536) images
Figure 2. Cumulative error curves on the PUT database of 100 people realized in partially controlled illu-
using landmarks chosen for cheek detection. mination on a uniform background.

51

VIPIMAGE15_Book.indb 51 9/8/2015 7:42:05 PM

 Figure 5. Cheeks detected by the first method.

Figure 4. Detected cheeks: (a) 68 face landmarks used

for cheek detection, (b) first method, (c) second method,
(d) third method, (e) fourth method, (f) our method.

The first two cheek detection methods, despite

their simplicity, designate the cheek areas lying
partly outside the face contour (Fig. 5c) and
(Fig. 6c). These methods are very sensitive to small
lateral turns of the head. As a result of their action
are found small, central areas of the cheeks.
The third method (Fig. 7) determines the greater
cheek region than two previous methods, but often
adds to the detected cheek area the pixels of nasal
region. This method is also sensitive to small lateral
turns of the head.
The fourth method (Fig. 8) allows detect larg-
est part of the cheek. Detected area does not go Figure 6. Cheeks detected by the second method.
beyond the face contour, but covers the lower
lashes and an area under the eyes.
The area found by the authors method (Fig. 9) of the figure is placed a face image of the head
does not go beyond the face contour and safely acquired in a central position. Each image con-
bypasses the nasal region and an area under tains both cheek regions marked with black line
the eyes. In this case the detected cheek area is borders and found with the help of the proposed
slightly smaller than the area found by the fourth method. All images show the correct detection of
method. cheeks. Detection results are not always as positive,
PUT image database, used in this study, includes especially for other methods and other persons.
also the face images acquired after different move- Such examples are shown in Figure 8, where on
ments of the head: to the left, right, up, down and the magnified faces the regions belonging to the
four diagonal movements (left-up, right-up, left- cheeks and at the same time extending beyond the
down, right-down). Figure 10 shows these eight face contour, have been manually marked. Pixels
images for the first tested person. At the centre in these regions can be specified as false detected

52

VIPIMAGE15_Book.indb 52 9/8/2015 7:42:07 PM

 Figure 7. Cheeks detected by the third method. Figure 9. Cheeks detected by our method.

Figure 10. Cheeks detected by our method at different

head poses.

The average results computed for all persons and

Figure 8. Cheeks detected by the fourth method. placed in the last row of the table, well describe
the quality of each method. Methods 1–2 carry the
largest errors, while the methods 4–5 character-
cheek pixels. These pixels were coloured with a ize by the smallest errors, especially the proposed
contrasting colour (e.g. cyan), which allows easily method. In this paper we did not consider a clas-
and automatically to count them. sification of pixels, belonging to the other parts of
Further studies were carried out for four per- the face such as the mouth, nose and the ears, as
sons and for each of them have been used the cen- cheek pixels.
tral image and eight images created as the results Another method of assessing impact of head
of head movements. Then, using five cheek detec- movements on the tested cheek detection methods
tion methods for each image, were determined the is a comparison of average cheek colour in each
cheek regions. For each person and each method of the eight images resulting from the movement
was calculated an average percentage of false clas- of the head with the average cheek colour in the
sified pixels in the areas designated as cheeks. The image acquired in the central position. For each
results of this experiment were recorded in Table 1. person and each method of cheek detection we

53

VIPIMAGE15_Book.indb 53 9/8/2015 7:42:11 PM

 one of eight test images acquired after movement
of the head.
The obtained results of the colour errors for the
left and right cheeks were averaged. Table 2 shows
the results of this experiment and, as in the case of
Table 1, shows also the advantages of the proposed
method over other methods.

4 CONCLUSION

We tested several methods found in literature for

automatic cheek detection using the PUT face
database. Each of these methods based on previ-
ously established face landmarks shown some dis-
advantages. We have proposed a new method for
cheek detection where the above mentioned dis-
advantages are minimized while almost all cheek
regions are correctly detected. Based on visual
and quantitative assessment, we find that our
method accurately models the shape of the cheek
and generally not extend it to other facial features.
Automatic cheek detection will allow to evaluate
the colour changes caused by different emotional
Figure 11. Examples of detected false cheek pixels. states.

Table 1. Average percentage of false cheek pixels. ACKNOWLEDGEMENTS

Meth. 1 Meth. 2 Meth. 3 Meth. 4 Meth. 5
This work was supported by the Polish Ministry
Person 1 11.91 12.52 0.34 0.08 0.00 for Science and Higher Education under internal
Person 2 17.54 17.51 3.68 0.00 0.00 grant BK-227/RAu1/2015 for Institute of Auto-
Person 3 17.56 15.95 3.66 0.79 0.75 matic Control, Silesian University of Technology,
Person 4 4.13 6.43 0.25 0.02 0.00 Gliwice, Poland.
AVG 12.79 13.10 1.98 0.22 0.19

REFERENCES
Table 2. Average colour errors of cheek regions.
Campadelli, P., G. Lipori, & R. Lanzarotti (2007).
Meth. 1 Meth. 2 Meth. 3 Meth. 4 Meth. 5 Automatic facial feature extraction for face recognition.
INTECH Open Access Publisher.
Person 1 6.23 6.23 3.52 4.12 4.12 Cristinacce, D. & T.F. Cootes (2006). Feature detection
Person 2 10.58 15.14 6.42 4.41 4.30 and tracking with constrained local models. In Proc.
Person 3 7.18 12.14 5.82 3.98 4.02 BMVC, pp. 95.1–95.10. doi:10.5244/C.20.95.
Person 4 5.75 8.89 4.01 3.29 2.95 Dalal, N. & B. Triggs (2005). Histograms of oriented
gradients for human detection. In IEEE Computer
AVG 7.44 10.60 4.94 3.95 3.85
Society Conference on Computer Vision and Pattern
Recognition, Volume 1, pp. 886–893.
Fairchild, M. (1998). Color appearance models. Addison-
calculated an average colour error ΔE E76 using a Wesley.
Gross, R., I. Matthews, J. Cohn, T. Kanade, & S. Baker
standard distance formula in the perceptually uni-
(2010). Multi-pie. Image and Vision Computing 28(5),
form CIELAB colour space (Fairchild 1998): 807–813.
Hess, M. & G. Martinez (2004). Facial feature extraction
Δ 76 = ( L0 − Li )2 + (a − ai )2 + (b0 − bi )2 (6) based on the smallest univalue segment assimilating
nucleus (susan) algorithm. In Proceedings of Picture
Coding Symposium, Volume 1, pp. 261–266.
where L0 , a0 , b0 are the components of average Kasinski, A., A. Florek, & A. Schmidt (2008). The PUT
cheek colour for central positioned image and face database. Image Processing and Communications
Li , ai , bi —components of average cheek colour for 13(3–4), 59–64.

54

VIPIMAGE15_Book.indb 54 9/8/2015 7:42:15 PM

 Kazemi, V. & J. Sullivan (2014). One millisecond face Seo, J.-H., Y.-B. Park, & Y.-J. Park (2014). Reliable facial
alignment with an ensemble of regression trees. In color analysis using a digital camera and its relation-
IEEE Conference on Computer Vision and Pattern ship with pathological patterns: A pilot study. European
Recognition (CVPR), pp. 1867–1874. Journal of Integrative Medicine 6(3), 322–327.
Kim, B.-H., S.-H. Lee, D.-U. Cho, & S.-Y. Oh (2008). Smith, S.M. & J.M. Brady (1997). SUSAN—a new
A proposal of heart diseases diagnosis method using approach to low level image processing. International
analysis of face color. In International Conference on Journal of Computer Vision 23(1), 45–78.
Advanced Language Processing and Web Information Su, R., T.M. Dockins, & M. Huber (2013). ICA analysis
Technology, pp. 220–225. of face color for health applications. In The Twenty-
King, D.E. (2009). Dlib-ml: A machine learning toolkit. Sixth International FLAIRS Conference.
Journal of Machine Learning Research 10, 1755–1758. Xian, W. (2011). Face region recognizer. B.E. Thesis,
King, D.E. (2015a). Dlib. http://dlib.net. Accessed on Malaysia University of Technology, Johor, Malaysia.
March, 2015. Yang, Y., J. Zhang, L. Zhuo, Y. Cai, & X. Zhang (2012).
King, D.E. (2015b). Max-margin object detection. CoRR Cheek region extraction method for face diagnosis
abs/1502.00046. of traditional chinese medicine. In IEEE 11th Inter-
Lai, Y.-R., C.-Y. Yao, H.-S. Hu, M.-T. Chi, & Y.-C. Lai national Conference on Signal Processing (ICSP),
(2013). A facial skin changing system. In Asia-Pacific Volume 3, pp. 1663–1667.
Signal and Information Processing Association Annual Zhao, C., G.-z. Li, F. Li, Z. Wang, & C. Liu (2014).
Summit and Conference (APSIPA), pp. 1–7. Qualitative and quantitative analysis for facial com-
Liu, M. & Z. Guo (2007). Hepatitis diagnosis using facial plexion in traditional Chinese medicine. BioMed
color image. In Medical Biometrics, pp. 160–167. Research International 2014.
Springer. Zhu, X. & D. Ramanan (2012). Face detection, pose
Ramirez, G.A., O. Fuentes, S.L. Crites, M. Jimenez, & estimation, and landmark localization in the wild.
J. Ordonez (2014). Color analysis of facial skin: In IEEE Conference on Computer Vision and Pattern
Detection of emotional state. In IEEE Conference on Recognition (CVPR), pp. 2879–2886.
Computer Vision and Pattern Recognition Workshops
(CVPRW), pp. 474–479.

55

VIPIMAGE15_Book.indb 55 9/8/2015 7:42:18 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

A variational model for image fusion with simultaneous

cartoon and texture decomposition*

Mahdi Dodangeh & Isabel N. Figueiredo

Department of Mathematics, CMUC, University of Coimbra, Coimbra, Portugal

Gil Gonçalves
INESC—Coimbra, Portugal

ABSTRACT: Image fusion is a technique that merges the information of multiple images, represent-
ing the same scene, to produce a single image that should gather the major and meaningful information
contained in the different images. On the other hand, cartoon+texture image decomposition is another
image processing technique, that decomposes the image into the sum of a cartoon image, containing the
major geometric information, i.e., piece-wise smooth regions, and a textural image, containing the small
details and oscillating patterns. We propose a model to perform the fusion of multiple images, relying on
gradient information, that provides as well a cartoon and texture decomposition of the fused image. The
model is formulated as a variational minimization problem and solved with the split Bregman method.
The suitability of the proposed model is illustrated with some earth observation images and also medical
images.

1 INTRODUCTION same time it also does the decomposition of the

fused image into cartoon and textural parts, adopt-
Image fusion is an image processing tool, that ing the generalized denoising model of (Bresson,
generates a single composite image from multi- Esedoglu, Van dergheynst, Thiran, & Osher 2007),
source images, of the same scene. It is particularly based on total variation regularization. Finally
important in remote sensing (it makes possible, we apply the split Bregman method (Goldstein &
for example, to combine the information of earth Osher 2009) to solve the model.
observation images acquired by different sensors After this introduction, the rest of the paper is
or satellites, for improving earth surveillance and organized in three sections: Section 2 describes the
monitoring) and in medicine (the fused image model, Section 3 explains the application of the
obtained from different medical imaging modali- split Bregman method and finally in Section 4 some
ties may improve diagnosis and prognosis). experimental results are shown. The paper ends
In this paper we propose a variational model for with some conclusions and future work.
image fusion (we refer to (Fang, Li, Zhang, & Shen
2013, Piella 2009) for two related variational models)
and that performs simultaneously a cartoon + 2 PROPOSED MODEL
texture decomposition of the fused image.
The proposed model fuses the gradients of the Let I i : Ω → [ 0,1], i = 1, , N , be (normalized) gray-
input images and combines this fusion with percep- scale and multisource images, where Ω ⊂ R 2 repre-
tual enhancement and intensity correction. At the sents the pixel domain, and for a pixel x Ω I i ( x )
is the intensity of Ii at x. We define g : ∑ iN wi ∇I i , a
weighted average of the gradients of Ii with weights
defined by wi | ∇I i | / ∑ Nj =1| ∇I j |, where | . | is the
*This work was partially supported by the research Euclidean nom and ∇ the gradient operator. We
project PTDC/MATNAN/0593/2012 of FCT (the Portu-
also define I 0 : ∑ iN=1 zi I i , a weighted average of the
guese national funding agency for science, research and
technology) and by CMUC (the Centre for Mathematics input images Ii (a possible choice for these weights
of the University of Coimbra—UID/MAT/00324/2013), is zi : wi or zi | I i | / ∑ Nj =1 | I j ).
funded by the Portuguese Government through FCT/ In (Fang, Li, Zhang, & Shen 2013) the fusion
MEC and co-funded by the European Regional Develop- model corresponds to the following minimization
ment Fund through the Partnership Agreement PT2020. problem

57

VIPIMAGE15_Book.indb 57 9/8/2015 7:42:18 PM

 ⎡ 2⎤ min ⎡|| ∇( + ) − g ||
2
η 1 μ TVh ( ) + α || v ||L1 ( )
min ∫ ⎢ ∇I − g + I − I − I 0 ⎥ ddx ( )⎢⎣ L( )
I Ω⎣ 2 2 2 ⎦
1 η 1 ⎤
⎡ η 1 2 + || ( ) − I 0 ||2L ( ) + u + v − ||2L2 ( ) . ⎥ .
= min
i || ∇ ||L ( ) || I − ||L2 ( ) 2θ 2 2 ⎦
I ⎣ 2 2
μ ⎤ where for simplicity h = 1.
+ || I − I 0 ||2L2 ( ) ⎥ ,
2 ⎦
3 NUMERICAL SOLUTION
where L1( ) and L2 ( ) are the spaces of, respectively,
absolutely and square integrable functions in Ω, with Since the objective functional in P is convex, the
norms denoted by || . ||L1 ( ) and || . ||L2 ( ) , respectively. solution of P can be computed by minimizing sep-
In this model, the fused image I is searched in such arately, with respect to u and v, that is by solving
a way, that its gradient matches the combined gra- the cartoon and texture subproblems (denoted by
dient g (the function g merges the gradient infor- C and T, respectively):
mation of all the different input images Ii). In Subproblem C—v being fixed, find u solution of
addition the linear combination of fitting terms
η μ
2
|| I 12 ||2L2 ( ) 2 || I − I 0 ||2L2 ( ) (as proposed in ⎡ 1
(Bertalmío, Caselles, Provenzi, & Rizzi 2007)) makes min || ∇(u ( + v) ||L ( ) h (u ) (u + v )
|| (u
u ⎣ 2θ
the fused image I to be perceptually more uniform η 1 ⎤
and close to the average I0 (the weighted sum of − I 0 ||2L2 ( ) + || ( ) − ||2L2 ( ) ⎥ .
the several input images); η and μ are two positive 2 2 ⎦
parameters that balance the influence of each term.
Subproblem T—u being fixed, find v solution of
On the other hand, in (Bresson, Esedoglu,
Vandergheynst, Thiran, & Osher 2007) the follow- ⎡ 1
ing model for image cartoon+texture decomposi- min || ∇((u + v ) ||L ( ) || ||L ( ) (u + v )
|| (u
tion is defined
v ⎣ 2θ
η 1 ⎤
− I 0 ||2L2 ( ) + || ( + ) − ||2L2 ( ) ⎥ .
2 2 ⎦
⎡ 1 ⎤
min ∫ ( x ) | ∇u | + | u + v − f |2 + | v | ddx
( ) Ω⎣ 2θ ⎦ Due to the structure of subproblems C and
⎡ 1 ⎤ T, involving particular L1 and L2 terms (that are
= min h (u ) + || u + v − f ||L2 ( ) +α || v ||L1 ( ) ⎥
2
convex functions) a suitable iterative method
( )⎣ 2θ ⎦
for approximating their solutions, and that we
adopt in this paper, is the split Bregman method
Here f R 2 → R is the input (grayscale)
(Goldstein & Osher 2009).
image and the goal is to find a decomposition u + v
Firstly, extra variables are introduced through
of f, that is f u + v , where the image u represents
the constraints d1 ∇ u + v ) g d2 = ∇u , for sub-
the geometric information of f (i.e., the cartoon
or piecewise smooth regions) and the image v cap- problem C, and e = ∇(u + v ) g , z = v for subprob-
tures the texture information. The function h is an lem T. The goal is to separate completely the L1
edge detector, that vanishes at object boundaries and L2 terms in each subproblem. Then we enforce
(such as h | ∇ff |) = 1 /(( β | ∇ff |2 ), with β an arbi- these constraints with the Bregman iteration proc-
ess (Bregman 1967). Therefore we have a sequence
trary positive constant), TV Vh (u ) is the total vari-
of cartoon and texture subproblems, respectively
ation norm of the function u weighted by h and
C k and Tk, for k = 1, 2, …, as follows:
θ > 0 is a very small parameter, so that we almost
Subproblem
p Ck—v k being fixed, find
have f u + v . The terms TV Vh (u ) and || v ||L1 ( ) act k +1
as regularizers for u and v, respectively, and α > 0 ( k 1
k
2 ) solution of
is a regularization parameter.
⎧ ⎡ 1
By combining the ideas of these two aforemen- ⎪( min || 1 |||L ( ) ||| ||L1 ( ) || (u + v k ) 0 ||2L2 ( Ω )
tioned models, we propose in this paper a variational 1 2) ⎣ 2θ
⎪
model for multisource image fusion that simultane- ⎪ η 1 λ
⎨ + || ( ) − ||2 2 ( Ω ) + 1 || d1 − ∇( k
ously performs a cartoon and texture decomposition )
⎪ 2 2 2
of the fused image. Given the normalized grayscale
⎪ λ2 ⎤
multi-source images { i }iN , the combined gradient ⎪ + g − b1 ||L2 ( Ω ) +
k 2
|| d 2 ∇u − b2k ||2L2 ( ) ⎥ ,
function g and the average image I0, already defined, ⎩ 2 ⎦
we search for a fused image in the form u + v , as the
solution of the following unconstrained optimiza- b1k 1
b1k − d1k 1
∇( u k 1 + v k ) g ,
tion problem (herein denoted by P ) b2k b2k − d 2k ∇u k +1 .

58

VIPIMAGE15_Book.indb 58 9/8/2015 7:42:19 PM

 Subproblem
p Tk—uk+1 being fixed, find where for z, γ in R
k k +1 k +1
( ) solution of
z
⎧ shrink( γ ) = ⋅ max((| z | −γ , 0 ).
⎡ 1 |z|
⎪(min) ⎣|| ||L ( ) || ||L ( ) || (u k +1 + v )
⎪ 2θ
⎪ η 1 − The formula for v k +1 is derived again
g from
⎪⎪ − I 0 ||L2 ( Ω ) + 2 || (
2 k 1
) − ||2 2 ( Ω ) the Euler-Lagrange equation, thus v k +1 is the
2
⎨ solution of the following PDE in Ω, hereafter
λ
⎪ + 3 || e − ∇(( k +1 ) + g − c k ||2 2 denoted by “texture PDE”
⎪ 2 L ( )
⎪
⎪ + λ4 || z v − w k ||2 2 ⎤ , ⎛ 1 ⎞ ⎛ 1 ⎞
η + + λ4 λ3Δ v k + = − η + − λ3Δ u k +1
L ( )⎥ ⎝ ⎠ ⎝ ⎠
⎩⎪ 2 ⎦ θ θ
1 1
∇((uu k +1 + v k +1 ) − g , − λ3 ( ) λ4 ( ) + η + I0 ,
k k k k
ck 1
ck − ek 1 g
2 θ
wk wk − zk + v k +1.
with the non-homogeneous Neumann boundary
The constants λi, for i = 1, 2, 3, 4 are the fixed pen- condition
alty parameters, used in the Bregman approach.
The minimization problem in subproblem Ck is
∂v k +1 k+
solved by iteratively minimizing with respect u, d1 = (e − ∇ + g − c k ) ⋅ n, ∂Ω.
and d2 , alternatively. Similarly, in subproblem T k ∂n
the minimization problem is also solved by itera-
tively minimizing with respect v, e and z, separately. − The formulas for e k +1 and z k +1 are also explicit
Consequently we have that: using the shrinkage operators (likewise for d1k +1
and d2k +1).
− The formula for u k +1 is derived from the Euler- Summarizing, the split Bregman method for P is:
Lagrange equation, i.e. it is the solution of the
following PDE (partial differential equation) in Algorithm
Ω, hereafter denoted by “cartoon PDE” Input—Multisource images I i , i 1, 2, , N .
Initialize—u 0 I 0 , fix θ, η η, α, λi (i = 1, 2, 3, 4 ),
⎛ 1 ⎞ + ⎛ 1 ⎞ tol, and fix v0, b10 , b20 , c0, d10 , d20 , e0 ω0, z0 equal
η + − ( λ1 λ2 )Δ u k +1 = − η + − λ1Δ v k
⎝ θ ⎠ ⎝ θ ⎠ to zero.
While max{| k − u k −1 ||,|| k − v k −1 |} > tol
( )
− λ1di d1k + g − b1k − λ2div d2k − b2k ( ) Cartoon part -
1 1
+ η + u0 u k +1 solution of the " cartoon PDE ",
2 θ d1k (∇(u k 1 + v k ) g + b1k , λ1 ),
1
k
where Δ and div are the Laplace and diver- d2 (∇u k 1 + b2k , λ1 ),
2
gence operators, respectively, with the non- b1k b1k − d1k ∇(u k 1 + v k ) g ,
homogeneous Neumann boundary condition
b2k b2k − d2k ∇ u k 1.
on the boundary ∂Ω of Ω
Texture part -
∂u k +1 λ1 v k +1 solution of the "texture PDE" ,
∂n
=
λ1 + λ2
(
d1 + g − b1k − ∇v
∇v k ⋅ n )
ek (∇(u k +1 + v k ) g + c k , λ1 ),
λ2
( )
3

+ d2 − b2k ⋅ n z k
s (v k +1 + w k , λα ),
λ1 + λ2 ck ck − ek
4
∇(u k +1 + v k +1 ) g,
where n is the unit outward normal to ∂Ω and w k
w −z
k k
v k +1.
“⋅” denotes the inner pproduct in R 2 . End
− The formulas for d1k +1 and d2k +1 are explicit using Output—Fused image u k v k , where u k is the
the shrinkage operators, so at each pixel (i, j) in Ω cartoon part and v k the texture part.

d1ki j
⎝
(
⎛ ( k
∇ u 1
+ v k ) g + b1k )i j , λ1 ⎞⎟⎠ , 4 EXPERIMENTS
1
⎛
d2ki+1j = shri
r nk ⎜ ∇
⎝
( k+
+ b2k )i , j , λ1 ⎞⎟⎠ , In this section we apply the proposed tech-
2 nique to three different types of images. In the

59

VIPIMAGE15_Book.indb 59 9/8/2015 7:42:25 PM

 experiments,
p iments, −w we use the following g p parameters
r tures on the polyp surface. Finally, Figure 3 shows
η θ = 10 5 , λ1 λ3 = 1, λ2 , λ4 = 10 , α = 1, the outcome of our method for a standard test
and tol = 10 −4. (RGB) image (with 256 × 256 pixels), downloaded
The first example, displayed in Figure 1, shows from the IPOL archive (http://www.ipol.im/).
an EO (earth observation) image, obtained with For assessing the quality of the results, we have
the remote sensing satellite WorldView-2. The EO used the following different evaluation measures:
image was downloaded freely from Digital Globe
1. Average Gradient (AG), which for a grayscale
Inc. (www.digitalglobe.com), and acquired on April
image u is defined by (Liu, Huang, & Zhao
3rd 2011 over Sydney (Australia). For this experi-
2006).
ment we consider the panchromatic band (with
a high spatial resolution of 0.46 meters at nadir)
2 2
and three visible multispectral bands (Red, Green, 1 ⎛ ∂u ⎞ ⎛ ∂u ⎞
| ∫Ω
Blue) (with a low spatial resolution of 1.84 meters AG = ⎜⎝ ∂x ⎟⎠ + ⎜⎝ ∂x ⎟⎠ dx ,
| 1 2
at nadir and a high spectral resolution). The goal is
to fuse the high spatial and high spectral images in
a single image and extract from it the cartoon and and generally, a big AG indicates a good fusion
texture information. We fuse each multispectral result.
band (64 × 64 pixels) with the panchromatic image 2. Entropy (E), which for a discrete grayscale image
(256 × 256 pixels), and to this end we first perform u is defined by (Liu, Huang, & Zhao 2006)
a pre-processing step, where each multispectral
L −1
image is up-sampled to the panchromatic image
E (u ) = − ∑ pi log pi
size, by using a bi-cubic interpolation. In Figure 1 i =0
we see that our model produces a fused image with
significant improvements. Also note that the car- where pi is the probability of the ith gray level in
toon is almost the same as the fused image, and the the image u, and L is the gray level. Similarly, the
texture information is meaningful. larger the entropy is, the better the fusion result.
Figure 2 shows the results of our method applied 3. Peak Signal to Noise Ratio (PSNR), that is evalu-
to a medical (RGB) image (with 536 × 536 pixels), ated in decibels and is inversely proportional the
acquired with the wireless capsule Pillcam Colon 2 mean squared error. For a grayscale restored
of Given Imaging. It displays a colonic polyp (the image u and rinput image u it is defined by the fol-
reddish region) exhibiting strong texture. Here we lowing formula (Huynh-Thu & Ghanbari 2008)
fuse each channel with the grayscale version of the
three color image. Again the results confirm that (2 n − ) | Ω |
the cartoon has almost all the details of the fused PSNR = 10log10
image, and the textural image captures the small fea- || u − u ||2L2 ( )

Figure 1. Top left: Multispectral (R, G, B) scene of Sydney, Australia. Top middle: Panchromatic image. Bottom left:
Cartoon. Bottom middle: Fused Image. Right: Texture.

60

VIPIMAGE15_Book.indb 60 9/8/2015 7:42:33 PM

 where P represents average power (Gonzalez &
Woods 2006). Generally a big SNR indicates
that the image reconstruction is of high quality.
The results obtained with the proposed method
are reported in Tables 1 and 2, where the columns
are organized as follows:
• The first column corresponds to the figure
numbers.
• The second column indicates the images: Pan-
chromatic (Pan.), multi-spectral image (RGB)
and grayscale version of the multi-spectral image
(Gray).
• The third column corresponds to input and out-
put images, where
− Input stands for the multi-spectral image.
− Average refers to the weighted average of the
multi-spectral image and the panchromatic
Figure 2. Top left: Polyp image obtained with Pillcam band (using the weights wi, based on the gra-
Colon 2, by courtesy of University Hospital of Coimbra, dients, previously defined).
Portugal. Top right: Fused Image. Bottom left: Cartoon. − Fused represents the fused image.
Bottom right: Texture. − Cartoon indicates the cartoon image.
• The fourth and fifth columns correspond to the
measures E, AG, PSNR and SNR.

Table 1. Entropy (E) and Average Gradient (AG) for

the three figures.

E AG × 103

Figure 1 Pan. – 7.08 65.88

RGB Input 6.91 43.18
Average 7.03 58.46
Fused 7.03 58.39
Cartoon 7.03 58.34
Gray Input 6.96 42.36
Average 7.03 56.83
Fused 7.03 56.77
Cartoon 7.03 56.72
Figure 2 Pan. – 6.35 6.14
RGB Input 6.13 6.19
Average 6.30 10.11
Figure 3. Top left: Garden standard test image. Top Fused 6.30 10.10
right: Fused Image. Bottom left: Cartoon. Bottom right: Cartoon 6.30 10.09
Texture. Gray Input 6.35 6.14
Average 6.35 7.20
Fused 6.35 7.19
Cartoon 6.35 7.19
Generally a big PSNR indicates that the Figure 3 Pan. – 7.60 65.28
image reconstruction is of high quality. RGB Input 7.56 64.94
4. Signal-to-Noise Ratio (SNR), which is defined Average 7.59 66.01
in decibels as the ratio of the power of a sig- Fused 7.59 65.94
nal (meaningful information) and the power of Cartoon 7.59 65.89
background noise (unwanted signal): Gray Input 7.60 65.28
Average 7.60 65.39
⎛ Pssignal ⎞ Fused 7.60 65.31
SNR = 10 log10 ⎜ ⎟,
⎝ Pnnoise ⎠ Cartoon 7.60 65.26

61

VIPIMAGE15_Book.indb 61 9/8/2015 7:42:38 PM

 Table 2. Peak-Signal-to-Noise-Ratio (PSNR) and same time was to have a simultaneous information
Signal-to-Noise-Ratio (SNR) of the three figures. The in real-time. In the future we are interested in com-
reference images to compute PSNRs and SNRs are the paring our (simultaneous) approach with a two-
input images. step approach.
PSNR SNR
The numerical experiments reveal the good
performance of the proposed model, in terms of
Figure 1 RGB Fused 20.22 9.21 appropriate quality metrics.
Cartoon 20.23 9.21 However, the proposed model depends on sev-
Gray Fused 21.90 10.98 eral parameters, that are fixed manually. In the
Cartoon 21.90 10.98 tests, we have noticed that the quality of the results
Figure 2 RGB Fused 24.00 13.00 depends on the size of these parameters. For
Cartoon 24.00 13.00 instance, experimentally we realize that the size of
Gray Fused 45.41 34.45 λ4 should be relatively large when compared to the
Cartoon 45.42 34.46 other parameters. Note that a large λ4 enforces the
Figure 3 RGB Fused 24.21 18.10 texture constraint “z v” to hold. In the future, we
Cartoon 24.21 18.10 intend to address these parameter related issues
Gray Fused 48.41 42.61 and in particular to perform a deep study to find
Cartoon 48.42 42.62 the optimal parameters, in an automatic way.

REFERENCES
Table 3. The computational time in seconds for the
three figures. Columns 2–4 correspond to the execution Bertalmío, M., V. Caselles, E. Provenzi, & A. Rizzi
time of the proposed method (fusion and simultaneous (2007). Perceptual color correction through vari-
cartoon+texture decomposition of the fused image) of ational techniques. Image Processing, IEEE Transac-
the red, green and blue bands, respectively. tions on 16(4), 1058–1072.
Bregman, L.M. (1967). The relaxation method of finding
Red Green Blue
the common point of convex sets and its application
to the solution of problems in convex programming.
Figure 1 0.3276 0.3120 0.3744
USSR Computational Mathematics and Mathematical
Figure 2 1.7940 1.6224 1.7472 Physics 7(3), 200–217.
Figure 3 0.2184 0.2652 0.1560 Bresson, X., S. Esedoglu, P. Vandergheynst, J.-P. Thiran, &
S. Osher (2007). Fast global minimization of the active
contour/snake model. Journal of Mathematical Imag-
ing and vision 28(2), 151–167.
All the numbers reported in Tables 1 and 2 are Fang, F., F. Li, G. Zhang, & C. Shen (2013). A varia-
averages of the three channels when it is applica- tional method for multisource remote-sensing image
ble. The displayed results confirm that the average fusion. International Journal of Remote Sensing 34(7),
gradient AG and entropy E for the fused images 2470–2486.
are improved when compared to AG and E of the Goldstein, T. & S. Osher (2009). The Split Bregman
method for L1-regularized problems. SIAM Journal
input (see Table 1). Also, from Table 2, one can
on Imaging Sciences 2(2), 323–343.
see that PSNR and SNR of the cartoon and fused Gonzalez, R.C. & R.E. Woods (2006). Digital Image
images are similar. Processing (3rd Edition). Upper Saddle River, NJ,
To evaluate the computational efficiency of USA: Prentice-Hall, Inc.
our model, we computed the execution time. Huynh-Thu, Q. & M. Ghanbari (2008, June). Scope of
Table 3 shows that the proposed method is quite validity of PSNR in image/video quality assessment.
fast. Electronics Letters 44(13), 800–801.
Liu, W., J. Huang, & Y. Zhao (2006). Image fusion based
on PCA and undecimated discrete wavelet transform.
In I. King, J. Wang, L.-W. Chan, and D. Wang (Eds.),
5 CONCLUSIONS
Neural Information Processing, Volume 4233 of Lec-
ture Notes in Computer Science, pp. 481–488. Springer
We have presented a variational model for image Berlin Heidelberg.
fusion with a simultaneous cartoon and texture Piella, G. (2009). Image fusion for enhanced visualiza-
decomposition of the fused image. The main moti- tion: a variational approach. International Journal of
vation for performing these two operations at the Computer Vision 83(1), 1–11.

62

VIPIMAGE15_Book.indb 62 9/8/2015 7:42:42 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Image contrast enhancement using split Bregman method*

Somayeh Gh. Bardeji, Isabel N. Figueiredo & Ercília Sousa

Department of Mathematics, CMUC, University of Coimbra, Coimbra, Portugal

ABSTRACT: In this paper we propose a variational method for image contrast enhancement, by keep-
ing the image details and correcting the non-uniform illumination. It is a minimization problem, where the
objective functional consists of two different fitting terms: a L1 term that matches the gradients of the input
and reconstructed images, for preserving the image details, and a L2 term that measures the misfit between
the reconstructed image and the mean value of the input image, for reducing the image variance and thus
correcting the illumination. For solving this minimization problem we apply the split Bregman method,
which is an efficient and fast iterative method suitable for this type of non-differentiable and convex mini-
mization problem, involving a L1 term. Some experimental results show the effectiveness of the method.

1 INTRODUCTION solved by a suitable computational method. Our

energy functional is a modification of that pro-
Image contrast enhancement is an image processing posed in (Morel, Petro, & Sbert 2014). It is com-
technique, whose purpose is to improve the image posed of two fitting terms. One fits the gradients
quality, for human interpretation of the image con- of the input and reconstructed images, it is meas-
tents or for supplying a good input in automated ured with the L1-norm, and aims at preserving
image processing systems. image details. The L1-norm is the chosen measure
In the literature there exists a plethora of con- because it has the property of better preserving
trast enhancement methods, which are based, discontinuities, when compared to the L2-norm.
for example, on histogram equalization, edge The second term fits the reconstructed image with
enhancement, edge sharpening, filtering and res- the mean value of the input image, in the L2-norm,
toration. For a detailed description of these type for correcting non-uniform illumination. Due to
of methods see for example the book (Gonzalez & its particular structure, we then choose the split
Woods 2008). In this paper we focus on a particu- Bregman method (Goldstein & Osher 2009) to
lar variational PDE (partial differential equation) solve this variational problem. This is a particular
approach for contrast enhancement. We refer for efficient iterative method applicable to a wide class
example to the book (Aubert & Kornprobst 2006) of L1-regularized optimization problems.
for an overview of the application of functional After this introduction the rest of the paper
analysis techniques and the theory of partial dif- includes the description of the variational problem
ferential equations to different image processing in Section 2, its numerical solution in Section 3,
problems, such as restoration of degraded images, some applications in Section 4 and finally the paper
denoising, segmentation, inpainting, decompo- ends with some conclusions and comments.
sition into cartoon and texture, optical flow and
image classification.
In the variational approach, used herein, the 2 DESCRIPTION OF THE MODEL
objective is to minimize an appropriate energy (or
functional), whose corresponding Euler-Lagrange In (Morel, Petro, & Sbert 2014) it is proposed
equation involves a PDE, that can be afterwards the following variational model for contrast
enhancement

{∫ }
*This work was partially supported by the research
project PTDC/MATNAN/0593/2012 of FCT (the Por- min
u Ω
u f ddx
x ∫Ω ( ) dx

{ },
tuguese national funding agency for science, research and
technology) and by CMUC (the Centre for Mathematics = min || ∇u − ∇f ||2L2 ( ) + λ || u − u ||2L2 ( ) (1)
of the University of Coimbra—UID/MAT/00324/2013), u
funded by the Portuguese Government through FCT/
MEC and co-funded by the European Regional Develop- where f R is the original (grayscale) input
ment Fund through the Partnership Agreement PT2020. image, Ω ⊂ R 2 represents the image pixel domain,

63

VIPIMAGE15_Book.indb 63 9/8/2015 7:42:42 PM

 x is a point (i.e. a pixel) in Ω, ∇ denotes the gradi- 3 NUMERICAL SOLUTION BASED
ent operator, | ⋅ | is the Euclidean norm in R2, u is the ON SPLIT BREGMAN METHOD
mean value of the reconstructed image u : Ω → R ,
L2(Ω) is the space of square integrable functions in A critical and first aspect of the split Bregman
Ω, and || . ||L2 ( ) denotes the L2-norm. This model method is the separation of the L1 and L2 terms,
contains two quadratic fitting terms. The first fits which is achieved by introducing an auxiliary
the gradients of u and f, and consequently aims at variable. Thus, we first replace (2) by the following
preserving image details. The second term intends constrained optimization problem
to reduce the effect of nonuniform illumination by
fitting u and its mean value u , thus by decreasing
the image variance. The parameter λ is a positive
constant that balances the influence of the two fit-
u { 2
|| u − f ||L2 (
n || d ||L ( ) ) }
subject to d = ∇u − ∇ff ,
ting terms. After replacing u by f , the mean value
of the input image f, it is shown that the problem and then reformulate it as an unconstrained prob-
does not depend
p on the value f , and the problem is lem, by introducing a quadratic penalty function,
solved, for f = 0 in the Fourier domain, using the that is
discrete Fourier transform.
In this paper we modify the model (1) by replac-
ing the L2-norm by the L1-norm for measuring the u {
min || d ||L ( )
λ
2
|| u − f ||2L2 ( )

}
misfit between ∇u and ∇f and we also replace the
α
mean value u by f . This yields the following vari- + || d (∇ ∇f ) ||2L2 ( )
. (3)
ational model 2

{∫ }
Then, the split Bregman method consists in
min
u Ω
u f dx
d ∫Ω ( f ) dx solving the following sequence of problems for
k = 0, 1, 2, …

u {
= min || ∇u − ∇f ||L1 ( )
λ
+ || u − f ||2L2 (
2 ) } (2)
⎧ k
⎪(
k +1
{
) = arg min || d ||L ( )
λ
|| u − f ||2L2 ( )
d ,u 2

}
⎪
1
where L ( ) is the space of absolutely integrable ⎨ α
+|| d − (∇u ∇ff ) − b k ||2L2 ( ,
functions in Ω and || . ||L1 ( ) denotes the L1-norm. The ⎪ 2 )
⎪b k
reason for replacing the L2-norm by the L1-norm is ⎩ b k + ∇uu k f − d k +1.
related to the fact that the L2-norm of the gradient
tends to smear image discontinuities, as opposed to (4)
the L1-norm that tends to preserve the discontinui- k
where the new vector b results from the Bregman
ties, which in image processing corresponds to sharp iteration (Bregman 1967), that is a strategy for
edges. In addition to this advantage, the presence of enforcing the constraint d = ∇u − ∇ff , using a fixed
the L1 fitting term in (2) permits the use of fast and penalty parameter α. This strategy is an alternative
effective algorithms for computing its solution. In to the conventional continuation technique to solve
effect, problem (2) belongs to the general class of (3) with an increasing sequence of penalty param-
L1-regularized problems of the form eters α1 < α 2 < … < α n tending to ∞, for accurately
enforcing the constraint. The minimization prob-
min{|| φ ( ) ||L1 ( )
H ( )} lem in (4) is solved by iteratively minimizing with
u
respect u and d, alternatively, which means the fol-
lowing two steps are performed.
where both || φ ( ) ||L1 ( ) and H(u) are convex func-
tions. This kind of models can be efficiently solved Step 1—Minimization with respect to u (with d
with the split Bregman method of (Goldstein & fixed)
Osher 2009). This is an appropriate algorithm for
solving non-differentiable convex minimization ⎧λ α
u k +1 arg min ⎨ || u f ||2L2 + || d k
problems, involving L1 or TV (total variation) u ⎩2 2
terms. We refer to (Fang, Li, Zhang, & Shen 2013,
Getreuer 2012b, Goldstein, Bresson, & Osher 2010,
−(∇u ∇f ) − b k ||2L2 ( ) , } (5)
Yin, Osher, Goldfarb,& Darbon 2008) for a few
examples of different applications of the method. for which the optimality condition (derived from
In our case φ ( ) = ∇u − ∇ff and H (u ) = the Euler-Lagrange equation) is, in Ω
λ || u f ||L2 ( ), and in the next section we apply
split Bregman method to solve (2). (λ )u k + = λ f α div( k
∇f k
),

64

VIPIMAGE15_Book.indb 64 9/8/2015 7:42:43 PM

 with Δ and div denoting the Laplace and diver- Summarizing, the split Bregman method for
gence operators, respectively, along with the non- model (2) is as follows:
homogeneous Neumann boundary condition on
Algorithm -
the boundary ∂Ω of Ω
Input—Original image g f.
∂u k +1 Initialize—u 0 f , d 0 b0 = 0, and fix λ ,α ,tol.
= (d k − ∇f − b k ) ⋅ n, While | k − u k −1 ||> tol
∂n
uk λ +4α
[αU + λ f v k ], in Ω,
∂u k +1
where “⋅” denotes the inner product in R2 and n is ∂n
= (d k − ∇f − b k ) ⋅ n, in ∂Ω,
the unit outward normal to ∂Ω.
This problem can be solved efficiently with the
d k
( )
= shrink ∇u k +1 − ∇ff b k , α1 ,
Gauss-Seidel method, since the system is diago- bk b k + ∇uu k f − d k +1.
nally dominant. The solution ik +1 End
j at each pixel (i,j) Output—Image uk.
in Ω (excepting in ∂Ω) is defined by

1 4 APPLICATIONS
uik +j 1 [αU i j + λ fi j − α vik j ]
λ + 4α
Some results obtained with our proposed model
U = (U i j ) := (uik +1j + uik+1, j + ik, +j 11 + uik, j +1 )
are shown in this section. All the experiments
vk = ( i , j ) ::=
k
( (d k + ∇f − b k ))i , j . were implemented with the software MATLAB®
R2014a (The Mathworks, Inc.)
Here we use finite differences for approximating Figure 1 shows the contrast enhancement with
the derivatives in the gradient ∇ and divergence div our method for a standard test image (a scalar
operators, respectively.
p In particular the discretiza- image with 512 × 343 pixels), downloaded from
tion used for vik j is obtained by applying
y backward the IPOL archive (http://www.ipol.im/). As this
finite differences for divd k and divb k and centered figure demonstrates, the details are kept and the
finite differences for Δf
Δ dark regions become more visible in the enhanced
image. In addition, and as expected, when λ
vik j dik j − dik − dik j −1 − 2bik j bik + bik j −1 increases (λ is the parameter associated with the
j j
fitting term intended to reduce the non-uniform
+ fi k 1 j + fi k j fikj −1 + fikj +1 fikj . illumination) the result tends to the mean value of
the input image.
The Neumann boundary condition is implicitly Figure 2 depicts the results of our method
imposed in ∂Ω, the boundary of the rectangular applied to a medical (RGB—red, green, blue)
pixel domain, by using backward finite differences, image (with 536 × 536 pixels), acquired with the
in the right and top sides, and forward finite differ- wireless capsule Pillcam Colon 2 of Given Imag-
ences in the left and bottom sides. ing. It displays a colonic polyp (the reddish region
Step 2—Minimization with respect to d (with u
fixed)

k +1
{
rg min || d ||L1 (
d )
α
2
∇u k +1 ∇f )
|| d − (∇u

− b k ||2L2 ( ) }.
This problem can be explicitly solved using
shrinkage operation (known as well as soft thresh-
olding) at each pixel (i, j)

⎛ 1⎞
dik j = shrink (∇u k +1 − ∇f
∇f b k )i j , ⎟
⎝ α⎠

where for z ,γ R

z Figure 1. Top left: Original image. Top right: λ = 0.005,

i (z γ ) =
shrink × max((| z | −γ , 0 ). α =1. Bottom left: λ = 0.01, α = 1. Bottom right: λ = 0.05,
|z| α = 1.

65

VIPIMAGE15_Book.indb 65 9/8/2015 7:42:47 PM

 Figure 2. Top left: Polyp image obtained with Pillcam
Colon 2, by courtesy of University Hospital of Coimbra,
Portugal. Top right: λ = 1, α = 150. Bottom left: λ = 1,
α = 200. Bottom right: λ = 2, α = 200.

in the top left subfigure) exhibiting strong texture. Figure 3. First column: Original (RGB) retinal fun-
We applied the algorithm independently to each dus image and results with (λ = 1, α = 5) in 2nd row and
channel. The original medical image (the top left (λ = 5, α = 5) in 3rd row. Second column: Grayscale ver-
subfigure) has a non-uniform illumination, with sion and results with (λ = 1, α = 5) in 2nd row and (λ = 5,
low contrast in some regions, that is corrected and α = 5) in 3rd row.
enhanced with the proposed method. The influ-
ence of the model parameters (λ and α) is also
illustrated in these results. Increasing λ results in
an averaged image, tending to the mean value of
the input image, and by increasing α the contrast
enhancement is enforced.
In Figure 3 we can see the results for another medi-
cal image (with size 536 × 536 pixels). It is an inhomo-
geneous illuminated retinal fundus image, provided
by the company Retmarker (http://www.retmarker.
com/), and obtained from a patient screened accord-
ing to the Diabetic Retinopathy Screening Program
of Portugal. We have processed with our method
the grayscale version (second column) as well as
each color channel separately (first column). Again
these results show the good contrast enhancement
improvement achieved with our method.
Figure 4 shows the result of the proposed meth-
odology for a MRI image, downloaded from the
IPOL archive (http://www.ipol.im/), and also a
comparison to the result obtained with the online
demo of (Morel, Petro, & Sbert 2014). We remark Figure 4. Top left: Original MRI and input for the pro-
posed method. Top right: Original MRI with 2% satu-
that our method operates directly on the input
ration and input for the model solved with the screened
image (exhibiting intensity inhomogeneities) Poisson equation (Morel, Petro, & Sbert 2014). Bottom
whereas the displayed result with the technique of left: Result for the proposed method with (λ = 1, α = 10).
(Morel, Petro, & Sbert 2014) involve pre and post Bottom right: Result applying a simplest color balance
processing steps that consist on the application of with 2% saturation to the solution obtained with the
a simplest color balance technique (Limare, Lisani, screened Poisson equation and with λ = 0.05.

66

VIPIMAGE15_Book.indb 66 9/8/2015 7:42:51 PM

 Morel, Petro, & Sbert 2011). We can notice that those obtained with the original model in (Morel,
our method performs slightly better in the dark Petro, & Sbert 2014) will be performed, as well as
down region on the left part of the image. We also a comparison with the ACE method (Getreuer
emphasize that our method doesn’t use any pre or 2012a), which is one of the most effective contrast
post processing methodology. enhancement methods.

5 CONCLUSIONS REFERENCES

In this paper we propose an inverse variational Aubert, G. & P. Kornprobst (2006). Mathematical prob-
model for image contrast enhancement. It is a modi- lems in image processing: partial differential equations
fication of the screened Poisson equation for image and the calculus of variations, Volume 147. Springer
contrast enhancement as proposed by (Morel, Science & Business Media.
Bregman, L.M. (1967). The relaxation method of finding
Petro, & Sbert 2014). The difference with respect the common point of convex sets and its application
to this latter model is in the term that matches the to the solution of problems in convex programming.
gradients of the input and reconstructed images, USSR Computational Mathematics and Mathematical
that is now measured in the L1-norm (instead of Physics 7(3), 200–217.
the L2-norm as in (Morel, Petro, & Sbert 2014)), Fang, F., F. Li, G. Zhang, & C. Shen (2013). A varia-
because the former has the advantage of better tional method for multisource remote-sensing image
preserving edges in image processing applications. fusion. International Journal of Remote Sensing 34(7),
The presence of this L1-norm term enables the 2470–2486.
proposed model to be solved by the split Bregman Getreuer, P. (2012a). Automatic color enhancement (ace)
and its fast implementation. Image Processing On Line
technique, which is an efficient procedure that has (2012).
the advantage of not requiring regularization, Getreuer, P. (2012b). Rudin-Osher-Fatemi total variation
continuation, or the enforcement of inequality denoising using split Bregman. Image Processing On
constraints. Line 10.
Applications of the proposed method to dif- Goldstein, T., X. Bresson, & S. Osher (2010). Geometric
ferent types of images show its good perform- applications of the split bregman method:
ance. We also remark that our methodology does segmentation and surface reconstruction. Journal of
not include pre and post processing steps as it Scientific Computing 45(1–3), 272–293.
happens in the methodology presented in (Morel, Goldstein, T. & S. Osher (2009). The Split Bregman
method for L1-regularized problems. SIAM Journal
Petro, & Sbert 2014), where the best reconstructed on Imaging Sciences 2(2), 323–343.
images have been obtained with pre and post Gonzalez, R.C. & R.E. Woods (2008). Digital Image
processing steps, involving the application of a Processing (3rd Edition). NJ, USA: Prentice-Hall, Inc.
simplest color balance technique (Limare, Lisani, Limare, N., J.-L. Lisani, J.-M. Morel, A.B. Petro, &
Morel, Petro, & Sbert 2011) applied before and C. Sbert (2011). Simplest color balance. Image Process-
after solving the model with the screened Poisson ing On Line 1.
equation. Morel, J.-M., A.-B. Petro, & C. Sbert (2014). Screened
The model we propose involves some parameters Poisson equation for image contrast enhancement.
that are tuned and fixed manually. In the future Image Processing On Line 4, 16–29.
Yin, W., S. Osher, D. Goldfarb, & J. Darbon (2008).
an automatic or self-adapting method for choos- Bregman iterative algorithms for L1-minimization
ing these parameters will be studied. Moreover an with applications to compressed sensing. SIAM
additional analysis concerning the comparison of Journal on Imaging Sciences 1(1), 143–168.
the results obtained with the present method with

67

VIPIMAGE15_Book.indb 67 9/8/2015 7:42:56 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Extraction of spectral drowsy component from the resting

electroencephalographic signal for quick, objective and direct
testing of sleepiness in absolute terms

A.A. Putilov, O.G. Donskaya & E.G. Verevkin

Research Institute for Molecular Biology and Biophysics, Novosibirsk, Russia

ABSTRACT: In spite of the extraordinary progress that has been made by sleep and behavioral sciences
in the last decades, the assessment of sleepiness is one of the remaining challenges. The lack of practical
tools for objective sleepiness testing has become one of the critical barriers to reducing the threats of
increased sleepiness to public health, productivity, and safety. The first aim of the present report was to
examine whether objective sleepiness testing can be facilitated by the quantitative analysis of the resting
Electroencephalographic (EEG) signal recorded at the first minute following eyes closing. The second aim
was to examine applicability of a new spectral EEG index for differentiation of 4 alertness-sleepiness sub-
states (alert, neither, dozy, and sleepy) in participants of two sleep deprivation experiments. In the first
(exploration) study of applicability of the new index, sleepiness of 15 young adults was self-scored every
second hour simultaneously with recording of the resting EEG in the course of 43–61-hour wakefulness.
In another (confirmation) study, one-minute eyes closed EEG recordings were obtained with a 3-hour
interval during 24-hour wakefulness of 130 adolescents and adults. The sleepiness testing was based on
calculation of Spectral Drowsy Component Score (SDCS). To compute SDCS, 16 (1–16 Hz) single-Hz
powers of the EEG spectra were obtained for the first one-minute interval of eyes closed EEG signal, log-
transformed, ipsatized, weighted, and summed. The 16 weights were the correlation coefficients between
the time course of self-scored sleepiness and the time courses of single-Hz powers. The results indicate that
SDCS provided reliable criteria for distinguishing between four sub-states of alertness-sleepiness. These
criteria performed even better than the traditional criteria based on latency to sleep onset. Moreover,
the occipital SDCS changed their sign exactly at the boundary between neither and dozy sub-states thus
providing yes-or-no criterion of onset of sleepiness state as opposed to the preceding alert state. We con-
cluded that SDCS might be recommended for quick (app. one-minute), direct and objective testing of
sleepiness in absolute terms.

1 INTRODUCTION Although it is believed that sleep deprived indi-

viduals are able to monitor their own sleepiness,
The Electroencephalogram (EEG) is becoming the vast majority of people seem poor in such a
increasingly important measurement of brain monitoring (Berka et al., 2004, van Dongen et al.,
activity. Developing and understanding signal 2004, Smith et al., 2005, Tremaine et al., 2010). For
processing methods for the analysis of EEG sig- instance, the experimental results indicate that sub-
nals seem to be crucial in the distinct areas of jective sleepiness fails to predict the falling asleep
fundamental and applied biomedical research event (Kaplan et al., 2007, Herrmann et al., 2010).
(Motamedi-Fakhra et al., 2014). It appears that, similar to the effects of alcohol,
The EEG signal processing seems to be of spe- the negative effects of drowsiness are often subjec-
cial interest for the distinguishing between different tively underestimated (Czeisler, 2011).
vigilance states and sub-states. In particular, the The EEG signs of sleepiness can be revealed
ability to monitor sleepiness is recognized as one objectively by means of the Multiple Sleep Latency
of important research areas where signal process- Test (Carskadon et al., 1986, Roehrs & Roth,
ing methods for the analysis of EEG signals can be 1992) or the Maintenance of Wakefulness Test
successfully applied (Subasi et al., 2005). Research (Doghramji, 1999). However, these tests are costly,
in this area is also of practical importance due time consuming, and usually administered in spe-
to obvious implication of the ability to monitor cial settings, such as a sleep research laboratory.
sleepiness for safety critical occupations and pro- Sleepiness can be also measured objectively by
cedures, such as driving (Horne & Burley, 2010). using other than EEG signals. For example, simple

69

VIPIMAGE15_Book.indb 69 9/8/2015 7:42:56 PM

 performance tests (e.g., a test on reaction time) sleepiness test would be a search for such criteria.
might be applied for this purpose, and such test- These criteria were recently proposed by one of us
ing does not require the application electrodes on (Putilov, in press). It was hypothesized that the rest-
the scalp. Moreover, the test can be administered ing EEG signal might be used for calculating spec-
in less restricted environmental conditions, and the tral alert and drowsy component scores. Weights for
duration of testing procedure might be reduced alert components score can be derived as a differ-
to 5 minutes (Roach et al., 2006, Horne & Burley, ence between spectra for the extreme (e.g., alert and
2010). sleepy) sub-states or between spectra for distant
However, similar shortening of the testing pro- (e.g., first and forth) deprivation phases. Weights
cedure was also suggested for such quicker and for drowsy components score can be provided by
simpler EEG tests of sleepiness as the Karolinska correlating the time courses of single-Hz powers
Drowsiness Test (Åkerstedt & Gillberg, 1990) and with the time course of either subjective or objec-
the Alpha Attenuation Test (Stampi et al., 1995). tive sleepiness measure (Putilov, in press).
These tests combine estimation of decline of alpha The major purpose of the present report was to
activity in eyes closed condition with measurement examine one of such recently suggested (Putilov, in
of “alpha bursts” reflecting a drowsiness-related press) spectral EEG indexes of sleepiness, Spectral
rise of alpha activity in eyes open condition. Drowsy Component Score (SDCS). We hypoth-
The general disadvantage of any of the pro- esized that, as compared to the EEG indexes of
posed tests is that the experimental and field stud- sleep onset, the extraction of SDCS from the EEG
ies of sleep-wake behavior in real life situations spectrum can provide more suitable criteria for
require even a much quicker and simpler testing demarcation of the boundaries between several
procedure. It was recognized that the area of prac- alertness-sleepiness sub-states of still waking indi-
tical applications of such quick test would be very viduals. For instance, it is expected that, in contrast
broad (Mullington et al., 2011, Quan et al., 2011). to the existing sleepiness and performance tests,
Alloway et al. (1997) noted that the increase such criteria can allow the description of the appar-
of alpha activity in eyes open condition did not ent level of sleepiness in absolute numbers related
contribute much to accuracy of distinguishing to understandable terms, and, consequently, the
between drowsy and alert individuals compared to drowsy component scoring procedure can be rec-
the contribution of the alpha rhythm decrease in ommended for simple and quick (e.g., one minute)
eyes closed condition. Moreover, our earlier stud- objective testing of sleepiness.
ies showed that attenuation of alpha-based EEG
indexes right after closing the eyes strongly corre-
lates with the feeling of sleepiness and shortened 2 METHOD
latency to the EEG signs of sleep onset (Donskaya
et al., 2012, Putilov et al., 2012, Putilov & Participants of the first study were practically
Donskaya, 2014b). Therefore, one can question healthy young males (N = 7) and females (N = 8).
the necessity of the EEG recordings in both condi- Their ages ranged from 22 to 26 and from 19 to
tions for accurate testing of sleepiness. If the EEG 24 years old (average ± standard deviation: 24.7 ± 1.9
sampling in eyes open condition can be excluded and 21.6 ± 2.2, respectively). The EEG signal was
from a testing procedure, this can open a new per- recorded with a 2-h interval in the eyes open and
spective of reducing duration of the procedure to a eyes closed conditions (2 and 5 min, respectively)
minute or so (Putilov & Donskaya, 2014b). starting from Friday evening (19:00 hours). The
We also showed in several our earlier publica- last session was scheduled on Sunday evening
tions that an epoch of sleep onset usually coincides (19:00). Each participant could terminate his/her
with zero-crossing decline of some of alpha-based participation in the experiment when felt irresist-
EEG indexes, such as alpha-theta power difference ible desire to sleep during one of the 2-hour inter-
in occipital derivation and score on the second prin- vals between the EEG recordings. All participants
cipal component of the EEG spectrum in any (e.g., completed, at least, 16 recording sessions, but only
frontal, occipital, etc.) derivation (Putilov, 2011, four of them completed all 25 sessions. Before
2014, 2015, Putilov & Donskaya, 2013a, b, 2014a). and after each EEG recording, the participants
However, prior to the transition from wake to sleep self-rated their alertness-sleepiness on the 9-step
state, these measures are always more or less higher Karolinska Sleepiness Scale or KSS (Åkerstedt &
than zero and, additionally, demonstrate remarka- Gillberg, 1990). Mean scores obtained by averag-
ble inter-individual variation (Putilov & Donskaya, ing scores self-reported before and after each EEG
2014b). Therefore, they cannot serve as yes-or-no recording session were used in the present analysis
criteria for distinguishing between sleepiness sub- as the indicators of subjective sleepiness.
states of still waking people. Consequently, a fur- Two (frontal and occipital) electrodes were used
ther step on the way of development of a quick for the EEG recordings. The active electrodes were

70

VIPIMAGE15_Book.indb 70 9/8/2015 7:42:56 PM

 placed in the middle from the top of the head to powers (5–8 Hz) changed from positive to negative.
forehead (Fz-A2) and midway between O1 and Four sub-states of alertness-sleepiness correspond-
O2 of the International 10–20 system of electrode ing to self-perceived minimal, mild, moderate, and
placement (Oz-A2). To fix the electrodes, Ten20 severe sleepiness were named Alert (SL = −6 min),
conductive paste was used (Nicolet Biomedical, Neither (SL from −5 to −3), Dozy (SL = −2 min),
Madison, WI, USA). The exact positions of active and Sleepy sub-states (SL = −1 min). Another
electrodes were preliminary inked by permanent (subjective) 4-sub-state classification was based on
marker, and the electrodes were removed after KSS score (1–4.5 for Alert, 5–6 for Neither, 6.5–7
each EEG measurement. The EEG signals were for Dozy, and 7.5–9 for Sleepy sub-state).
recorded via a 16-channel electroencephalograph The absolute power values in the frequency
(Neuron-Spectrum-2, Neurosoft, Ivanovo, Russia), range from 1 to 16 Hz were converted into a natu-
conditioned by the high-pass, low-pass and notch ral logarithmic scale. In the present analysis, such
filters (0.5 Hz, 35 Hz, and 50 Hz, respectively), sets of single-Hz spectral powers calculated for the
sampled and stored on a hard disc with a frequency first minute of eyes closed relaxation were used
of 200 Hz. The EEG records were inspected visu- for computation of SDCS. In order to compute
ally to remove 2-s epochs containing artifacts from SDCS for the participants of the first study, each
further analysis. Power spectra for artifact-free of 630 (Fz and Oz) sets of 16 ipsatized powers was
2-s epochs were computed using the FFTW (the weighted in accord with correlations between the
Fastest Fourier Transform in the West) package time courses of these powers and the time course
(Frigo & Johnson, 2005; see also www.fftw.org). of KSS score (25 time points) and then summed:
Absolute spectral powers (μV2) were calculated for
single-Hz frequency bandwidths (i.e., 0.50–1.49 Hz, SDCSi = Σw *f
j j,j (1)
1.50–2.49 Hz, etc.). The spectra of the artifact-free
2-s epochs were averaged within each one-min where SDCSi is the i-th SDCS (i refers to either
interval of the eyes closed EEG records. frontal or occipital score), wj is a weight of the j-th
Data from another study were used to provide evi- frequency bin (j from 1 to 16 Hz), and fi,j is an ipsa-
dence for reliability of the measures obtained in the tized power value for a j-th power density bin of
analysis of the first (internal) dataset. Participants an i-th score.
of this confirmation study were practically healthy The procedure of calculation of SDCS was also
adolescents and adults. The mean ages of 54 male applied for computing scores for the participants
and 76 female participants were 27.4 ± 10.1 (range from the second (external) sample. Additionally,
from 15 to 55 years) and 30.8 ± 13.4 (range from the weights provided by the analysis of one sample
16 to 66 years), respectively. In the experimental were used for calculating SDCS for participants of
morning (between 08:00 and 8:30 o’clock), they another study.
were admitted to a research unit of the institute Moreover, the Oz SDCS provided criteria for
and remained there until approximately 11:00 the the third classification of alertness-sleepiness sub-
next morning. Over the 24 hour period (from 9:00 states. Such (SDCS-based) sub-states were distin-
to 9:00) each study participant completed 9 3–4- guished for assigning KSS scores and SL to the
min EEG recording sessions divided by 3-hour following 4 intervals of SDCS: below −0.2 (Alert),
intervals. KSS was self-scored after each session. −0.2–0 (Neither), between 0 and 0.2 (Dozy), and
The EEG recordings were taken at frontal and above 0.2 (Sleepy sub-state).
occipital scalp sites corresponding to derivations Differences between four sub-states of alert-
Fz and O2 referenced to the right mastoid, A2. For ness-sleepiness were tested by applying two-way
one half of the participants, the recordings with ANOVAs with a fixed factor “Alertness-sleepiness
close eyes were made prior to the recordings with sub-state” and a random factor “Participant”
eyes open, and for another half of the participants (N = 15 or 130). Prior to applying these ANOVAs,
the eyes closed recordings were following the eyes normality of distribution of the EEG variables was
open recordings. In the present analysis, only, at tested by means of the exploratory analysis (SPSS)
least, one minute of eyes closed section of each that was performed for each time point of each
EEG record was used for comparison with similar index. The hypothesis of normality (Kolmogorov-
data obtained in the first study. Smirnov criterion) was always supported.
The EEG records from the first study were
sorted into 4 groups in accord with the duration of
wakefulness, i.e., from only wakefulness (−6 min) to 3 RESULTS
sleep initiated already during the first minute with
eyes closed (−1 min). Latency to Sleep onset (SL, Very strong correlations (Spearman’s coefficients
min) was determined as a minute during which of correlation from 0.93 to 0.96, p < 0.001) were
the difference between alpha (9–12 Hz) and theta attained between the 25-point time courses of

71

VIPIMAGE15_Book.indb 71 9/8/2015 7:42:56 PM

 sleepiness measures and the corresponding time Table 2. Differences between two sleepiness sub-states
courses of SDCS (Table 1). (dozy and sleepy) in occipital SDCS.
The occipital SDCS calculated for dozy and
sleepy sub-states were positive, whereas scores CI
Mean
calculated for alert and neither sub-states were Weighting Sub-state score −95% +95%
negative. The borderline between neither and dozy
sub-states was close to zero score. Thus, such scor- Dataset, derivation: Internal, Oz (N = 15)
ing provided yes-or-no criteria of sleepiness onset. Internal Fz Averaged 0.002 −0.030 0.035
Namely, positive SDCS suggested sleepiness state, Dozy 0.040 −0.034 0.115
whereas negative SDCS corresponded to the pre- Sleepy 0.249 0.172 0.326
ceding alert state. Difference 0.209** 0.064 0.353
The two-way ANOVAs detected highly signifi- Internal Oz Averaged 0.020 −0.013 0.053
cant main effect of the sub-state factor irrespective Dozy 0.061 −0.014 0.137
of the criteria applied for distinguishing between Sleepy 0.269 0.190 0.347
4 sub-states of alertness-sleepiness, KSS-, SL-, and Difference 0.207** 0.061 0.354
SDCS-based. Particularly, SL- and KSS-based
classifications gave F(N = 15, df = 3) = 60.6 and Dataset, derivation: External, O2 (N = 130)
F = 22.1, respectively, in analysis of data on occipi- Internal Fz Averaged 0.075 0.066 0.084
tal SDCS. SL- and SDSC-based classifications Dozy 0.081 0.058 0.104
gave F = 23.2 and F = 23.9 for KSS score. SDSC- Sleepy 0.182 0.155 0.209
and KSS-based classifications gave F = 59.6 and Difference 0.101*** 0.053 0.148
F = 23.0 for SL, min (p < 0.001 for any of these Internal Oz Averaged 0.091 0.082 0.100
F-ratios). Dozy 0.097 0.074 0.120
The results of post hoc pairwise comparisons Sleepy 0.203 0.176 0.229
suggested that any of 4 objectively defined sub- Difference 0.106*** 0.059 0.153
states demonstrated highly significantly differ- Notes: SDCS were derived from either the first (inter-
ence from any of 3 other sub-states. Moreover, nal) or second (external) datasets from subsets of data
on occipital derivations (Oz and O2, respectively) using
weights derived from the first (internal) dataset, either Fz
Table 1. Correlations between the time courses of or Oz subsets. Averaged: General mean score obtained
SDCS and traditional sleepiness measures. by averaging over 4 sub-states. Difference: Difference
between Sleepy and Dozy sub-states. ***p < 0.001,
Derivation Source of weights Correlation **p < 0.01: Level of significance for t-value in post-hoc
pairwise comparisons of the dozy and sleepy sub-states
KSS score (with Bonferroni adjustment for the number of compari-
Fz Internal 0.938***/0.938*** sons). CI: Confidence Interval.
Oz Internal 0.950***/0.945***
Oz External 0.944***/0.954***
KSS sub-state some of the results suggested that SDSC-based
Fz Internal 0.940***/0.940*** classification over-performed SL-based clas-
Oz Internal 0.935***/0.928*** sification in distinguishing between KSS scores
Oz External 0.928***/0.939*** assigned to two sleepiness (dozy and sleepy) sub-
SL sub-state states.
Fz Internal 0.953***/0.953*** Very similar results were obtained in the analy-
Oz Internal 0.961***/0.959*** sis of the second (external) dataset irrespective of
Oz External 0.919***/0.942*** which weights were used for calculating SDSC for
SL, min the participants of this experimental study, either
Fz Internal 0.944***/0.944***
own or those provided by the analysis of the first
Oz Internal 0.952***/0.950***
(internal) dataset (Table 2).
Oz External 0.913***/0.931***

Note: The time courses of SDCS were correlated with 4 DISCUSSION

the time courses of four major sleepiness measures based
on KSS and SL (the first dataset). SDCSs were calcu- In spite of the extraordinary progress that has
lated for either frontal or occipital subset (Fz or Oz)
using weights derived from either the same (internal)
been made by sleep and behavioral sciences in the
or another (external) dataset, and either from frontal or last decades, the assessment of sleepiness is one
occipital subset, Fz/Oz or Fz/O2. Asterisks (***) denote of the remaining challenges. The lack of practical
highly significant (p < 0.001) Spearman’s coefficients of tools for objective sleepiness testing has become
correlation (N = 25). one of the critical barriers to reducing the threats

72

VIPIMAGE15_Book.indb 72 9/8/2015 7:42:56 PM

 of increased sleepiness to public health, produc- a new perspective of development of objective,
tivity, and safety (e.g., Mullington et al., 2011). reliable, simple and quick (app. one-minute) test-
Unlike subjective (KSS) scoring, the existing ing of sleepiness.
objective sleepiness and performance tests do not It is necessary to stress that, in despite of a
allow quick assessment of the apparent level of clear tendency of sleep deprived participants to
sleepiness in absolute numbers related to under- underestimate the negative effects of sleep loss,
standable terms (Horne & Burley, 2010). The the comparison of the results on KSS based
duration of any of testing procedures that were SDCS with the results on SL that is a tradi-
earlier proposed for objective sleepiness meas- tional measure of sleepiness showed that, due
urement cannot be reduced to less than 5-minute to close correlation between the time courses
time interval (see Introduction). However, our of subjective and objective sleepiness, this score
earlier studies showed that attenuation of alpha- can be regarded as a reliable objective rather
based EEG indexes right after closing the eyes than subjective measure of sleepiness. In other
strongly correlates with the feeling of sleepiness words, the effect of underestimation of sleepi-
and shortened latency to the EEG signs of sleep ness was evident only for subjective (KSS) scor-
onset. Therefore, such quick EEG measurement ing, whereas data on SDCS and SL did not reveal
can allow the reduction of duration of the pro- this effect.
cedure to a minute or so (Putilov & Donskaya, However, it is also necessary to stress that the
2014b). Data provided by such measurement present results cannot be regarded as one of the
can be, in particularly, used for calculating such final steps on the way of development of a quick
objective sleepiness index as SDCS (Putilov, in and simple sleepiness test. First, the present
press). In the resent study the weights for calculat- analysis included such additional procedures as
ing SDCS were 16 coefficients of correlation of manual artifact detection, ipsatization of each par-
the time course of subjective sleepiness with the ticipant power values, averaging over several scores
time courses of single-Hz powers. The time course obtained from the same participant, etc. Second,
consisted of either 15 or 9 time points (the first the analysis was limited to only two derivations
and second studies, respectively). Irrespective of and did not test contribution of other derivations
the dataset, the occipital SDCS of −0.2, 0, and 0.2 and such additional indicators of sleepiness as
provided the borderline scores for subdivision of changes in coupling between activities of distant
the alertness-sleepiness continuum into four dif- cortical areas. Third, it remains to be explored in
ferent sub-states (alert, neither, dozy, and sleepy, the future research whether yes-or-no criterion of
respectively) in the vast majority of two study transition between alert and sleepy states can be
participants. applied without preliminary subject-specific cali-
Thus, the present results suggest that the find- bration in other studies including the field studies
ings on SDCS were irrelevant to the source of of shift workers, clinical studies of patients with
weights for calculation of scores (internal vs. sleep disorders, studies of individuals without
external). Such results supported reliability of the prominent alpha rhythms and people representing
SDCS-based testing of alertness-sleepiness. They, distinct ages.
in particular, indicated that the weights for scoring
of drowsy component can be obtained from stud-
ies with rather distinct sleep deprivation protocols. 5 CONCLUSION
Namely, these weights were found to lead to very
similar scores (Table 2) in despite of remarkably The spectral component scoring of the EEG sig-
different clock times chosen for subjective sleepi- nals recorded during the first minute of eyes closed
ness evaluation. Furthermore, the weights derived relaxation provides a possibility of quantifying
from one dataset were found to be applicable for sleepiness and distinguishing between alertness
another dataset (i.e., led to similar classifications and sleepiness states in absolute terms, i.e., by
of sleepiness sub-states) in despite of additional using zero-crossing criterion of sleepiness onset.
very remarkable difference between these two sleep
deprivation experiments in the finally reached lev-
els of sleepiness. ACKNOWLEDGEMENT
In general, the present results indicate that
i) objective sleepiness testing can be facilitated The experimental studies were supported by the
by the quantitative analysis of the resting EEG Russian Foundation for Basic Research (grants
signal recorded at the first minute following eyes 07-06-00263-a, 10-06-00114-a, and 13-06-00042-a),
closing, ii) this analysis can include calculation and the Russian Foundation for Humanities
of occipital SDCS as a zero-crossing criterion of (grants 06-06-00375-a, 12-06-18001-e, and 15-06-
sleepiness onset, and iii) the results on SDCS open 10403-a).

73

VIPIMAGE15_Book.indb 73 9/8/2015 7:42:56 PM

 REFERENCES Putilov, A.A. 2015. Rapid changes in scores on princi-
pal components of the EEG spectrum do not occur in
Åkerstedt, T. & Gillberg, M. 1990. Subjective and objec- the course of “drowsy” sleep of varying length. Clin
tive sleepiness in the active individual. Int J Neurosci EEG & Neurosci 46 147–152.
52: 29–37. Putilov, A.A. (in press) Can we evaluate sleepiness
Alloway, C.E.D., Ogilvie, R.D. & Shapiro, C.M. 1997. quickly, directly, objectively, and in absolute terms?
The alpha attenuation test: assessing excessive daytime Results of scoring of alert/drowsy components of the
sleepiness in narcolepsy-cataplexy. Sleep 20: 258–266. resting EEG spectrum. Somnologie 2015; 19(3).
Berka, C., Levendowski, D.J., Olmstead, R.E., Putilov, A.A. & Donskaya, O.G. 2013a. Rapid changes
Popovic, M.V., Cvetinovic, M., Petrovic, M.M., in scores on the two largest principal components of
Davis, G., Lumicao, M.N. & Westbrook, P. 2004. the electroencephalographic spectrum demarcate the
Real-time analysis of EEG indices of alertness, cogni- boundaries of drowsy sleep. Sleep Biol Rhythms 11:
tion and memory with a wireless EEG headset. Int J 154–164.
Hum Comput Interact 17: 151–170. Putilov, A.A. & Donskaya, O.G. 2013b. Construction
Carskadon, M.A., Dement, W.C., Mitler, M., Roth, T., and validation of the EEG analogues of the Karolin-
Westbrook, P.R. & Keenan, S. 1986. Guidelines for ska sleepiness scale based on the Karolinska drowsi-
the Multiple Sleep Latency Test (MSLT): a standard ness test. Clin Neurophysiol 124: 1346–1352.
measure of sleepiness. Sleep 9: 519–24. Putilov, A.A. & Donskaya, O.G. 2014a. Calibration of
Czeisler, C.A. 2011. Impact of sleepiness and sleep defi- an objective alertness scale. Int J Psychophysiol 94:
ciency on public health: utility of biomarkers. J Clin 69–75.
Sleep Med 7(5 Suppl): S6–S8. Putilov, A.A. & Donskaya, O.G. 2014b. Alpha attenua-
Donskaya, O.G., Verevkin, E.G., Putilov, A.A. 2012. The tion soon after closing the eyes as an objective indi-
first and second principal components of the EEG cator of sleepiness. Clin Exp Pharmacol Physiol 41:
spectrum as the indicators of sleepiness. Somnologie 956–964.
16: 69–79. Putilov, A.A., Donskaya, O.G. & Verevkin, E.G. 2012.
Doghramji, K. 1999. Maintenance of Wakefulness Test. Quantification of sleepiness through principal com-
In S. Chokroverty (ed). Sleep disorders Medicine: ponent analysis of the EEG spectrum. Chronobiol Int
231–236. Woburn MA: Butterworth-Heinemann. 29: 509–522.
Frigo, M. & Johnson, S.G. 2005. The design and imple- Quan, S.F., Shaw, P.J., Naidoo, N., Haeggström, E.,
mentation of FFTW3. Proc IEEE 93: 216–231. Krueger, J.M. & Church, G.M. 2011. Panel discus-
Herrmann, U.S., Hess, C.W., Guggisberg, A.G., Roth, C., sion: can there be a biomarker for sleepiness? J Clin
Gugger, M. & Mathis, J. 2010. Sleepiness is not Sleep Med 7(5 Suppl): S45–S48.
always perceived before falling asleep in healthy, sleep- Roach, G.D., Dawson, D. & Lamond, N. 2006. Can a
deprived subjects. Sleep Med 11: 747–751. shorter psychomotor vigilance task be used as a rea-
Horne, J.A. & Burley, C.V. 2010. We know when we are sonable substitute for the ten-minute psychomotor
sleepy: subjective versus objective measurements of vigilance task? Chronobiol Int 23: 1379–1387.
moderate sleepiness in healthy adults. Biol Psychol 83: Roehrs, T. & Roth, T. 1992. Multiple Sleep Latency
266–268. Test: technical aspects and normal values. J Clin
Kaplan, K.A., Itoi, A. & Dement, W.C. 2007. Awareness Neurophysiol 9: 63–67.
of sleepiness and ability to predict sleep onset: Can Smith, S., Carrington, M. & Trinder, J. 2005. Subjective
drivers avoid falling asleep at the wheel? Sleep Med and predicted sleepiness while driving in young adults.
9: 71–79. Accid Anal Prev 37: 1066–1073.
Motamedi-Fakhra, S., Moshrefi-Torbati, M., Hill, M., Stampi, C., Stone, P. & Michimori, A. 1995. A new quan-
Hill, C.M., Paul, R. & White, P.R, 2014. Signal titative method for assessing sleepiness: The alpha
processing techniques applied to human sleep EEG attenuation test. Work & Stress 9: 368–376.
signals—A review. Biomed Signal Process Control 10: Subasi, A., Kiymik, M.K., Akin, M., Erogul, O. 2005.
21–33. Automatic recognition of vigilance state by using
Mullington, J.M., Czeisler, C.A., Goel, N., Krueger, J.M., a wavelet-based artificial neural network. Neural
Balkin, T.J., Johns, M. & Shaw, P.J. 2011. Panel dis- Comput Appl 14: 45–55.
cussion: current status of measuring sleepiness. J Clin Tremaine, R., Dorrian, J., Lack, L., Lovato, N.,
Sleep Med 7(5 Suppl): S22–S25. Ferguson, S., Zhou, X. & Roach, G. 2010. The rela-
Putilov, A.A. 2011. Principal components of electro- tionship between subjective and objective sleepiness
encephalographic spectrum as markers of opponent and performance during a simulated night-shift with a
processes underlying ultradian sleep cycles. Chronobiol nap countermeasure. Appl Ergon 42: 52–61.
Int 28: 287–299. van Dongen, H.P., Baynard, M.D., Maislin, G. &
Putilov, A.A. 2014. When does this cortical region drop Dinges, D.F. 2004. Systematic inter-individual differ-
off ? Principal component structuring of the EEG ences in neurobehavioral impairment from sleep loss:
spectrum yields yes-or-no criteria of local sleep onset. evidence of trait-like differential vulnerability. Sleep
Physiology & Behavior 133: 115–121. 27: 423–433.

74

VIPIMAGE15_Book.indb 74 9/8/2015 7:42:57 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

The illusion of a blackbody at the human ear

and the human temperature measurement

António Cardoso
Materiais e Produtos, CATIM—Centro de Apoio Tecnológico à Indústria Metalomecânica, Porto, Portugal

ABSTRACT: The insertion of probes in the human ear or any other technique involving contact ther-
mometers, or the use of the inner canthus for temperature measurement, using infrared thermal imagers,
with the purpose of obtaining a reference to determine the core temperature in humans, can be compro-
mised by a wide number of factors that may lead to faulty results.
In this paper, an innovative approach based on the use of a thermographic camera and on the char-
acteristics of the external auditory meatus is proposed. This new approach enables to solve the problems
associated with the previously proposed methods, adding new possibilities namely alowing the precise
measurement of the core temperature in humans in a quick and simple way, without body contact, open-
ing the possibility of setting self-screening stations, working without operators in fully automatic mode.

1 INTRODUCTION another plastic probe for hygienic reasons, generat-

ing plastic waste. It also presents some emissivity
The majority of the methods used to determine related problems, derived from placement issues. To
the core temperature in humans are affected by a get an accurate measurement the ear thermometer
number of factors that apart other clinical consid- must be directed to the ear drum or otherwise, if
erations may yield to faulty measurements. the reading angle is less than 45°, it will be reading
One method to determine the core temperature in the radiation emanated from the walls of the exter-
humans is the measuring of the temperature in the nal auditory meatus. At the interior of the external
eye corner (inner canthus). The ophthalmic artery, auditory meatus, the emissivity has a value of 98%.
located beneath the inner cantus, is supplied directly When a probe at room temperature is inserted, its
by the internal carotid artery, reason why screening own reflected radiation will affect the reading,
thermography utilizes it to determine the core tem- causing the disappearance of the blackbody illu-
perature in humans. However, and by the same sion and introducing an error in the measurement,
reason, the superficial temperature measured at the even if the probe is properly aimed at the ear drum.
inner cantus by the thermographic camera may be Only after three or four readings it produces a cor-
affected by other factors like some cerebrovascular rect measurement value.
diseases that affect the internal carotid artery. These The insertion of a Pt100 or of a thermocouple
diseases may cause a diminishing in the blood flow, on the human ear is not a good solution either,
and consequently, in the amount of heat dissipated because the lack of air flow inside the external
at the inner canthus, leading to a drop in the super- auditory meatus, and therefore the associated
ficial temperature. Another factor that may affect good level of insulation provided by still air, con-
reading is the accumulation of gound, the mucous tributes to a faulty measurement. In fact, still air is
that is discharged by the eyes while sleeping, which an effective insulator because it eliminates convec-
accumulates in their corners, at the inner canthus, tion and has low conduction. Furthermore, in this
enhancing temperature dropping. In thermographic technique the metallic wires of the thermometer
terms, the inner canthus area has a very well deter- are another problem. Metals are good thermal con-
mined emissivity: 98%, which implies the need to ductors and thus the wires act as thermal drains,
determine the reflected apparent temperature to get compromising the final measurement value.
an accurate measurement. A range of other methods based on contact
The ear thermometer is another non-contact thermometers is used in clinical scenarios. Apart
method that, however, requires the use of a dispos- hygienic measures that makes mandatory the
able plastic probe cover, since mechanical support disinfection of the thermometers to avoid the
is needed which involves physical contact with the propagation of diseases like flu, hepatitis, her-
ear. Therefore, it poses an economic and ecological pes, tuberculosis and other bacterial and viral
problem, as each measurement requires the use of related diseases, the temperature given by contact

75

VIPIMAGE15_Book.indb 75 9/8/2015 7:42:57 PM

 thermometers is the internal sensor temperature tube with a scale, the “display”. The increase in
and not the patient’s temperature. temperature forces the expansion of a liquid placed
Even the clinical temporal artery thermometers inside the tube, thus indicating the temperature
involve contact with the patient, and thus they are of the patient’s body. In the past, the liquid inside
affected by the same problems. Furthermore, the clinical thermometers was mercury, a poison-
measurement accuracy is affected by the position- ous heavy metal. Due to these concerns, mercury
ing of the thermometer. If the thermometer is not has been phased out in favour of alcohol-filled or
properly positioned over the temporal artery, the galinstan-filled thermometers. Lately, these ther-
reading will be inaccurate. mometers are being substituted by digital ones with
Contact thermometers always measure their own thermistors as the sensing element. They can be used
temperature and not the temperature of the object in cavities of the human body covered with mucous
they are in contact with. Equation (1) represents membranes, like the rectum (rectal) or mouth (oral),
how temperature reading evolves over time. When or in the armpit (axillary), the last one being the
t → ∞ the thermometer temperature will eventually most usual and also the least accurate of the three.
reaches the temperature of the patient.

⎛ t ⎞ 2.2 Esophageal probes

−
Tsensor (T patient
i − Troom ) ⎜1 e
⎜⎝
τ ssensor
⎟ + Troom
⎟⎠
(1) Esophageal probes are commonly used when a
constant monitoring is mandatory, like in Inten-
sive Care Units (ICU) and Continuing Care Units
The biggest advantage of thermographic cameras (CCU). Usually, a Negative Temperature Coefficient
is that they don’t measure the temperature. They cal- (NTC) thermistor or a thermopile is used as sensing
culate it. Indeed, thermographic cameras are imag- element. Like the clinical thermometers, esophageal
ing devices where each pixel represents an amount probes are contact thermometers. Besides, its place-
of energy radiated towards the camera by a surface. ment in the patient’s esophagus causes discomfort.
This energy has two components: the emitted radia-
tion, and the reflected radiation. Since the majority
of the surfaces belong to opaque objects, the trans- 2.3 Ear thermometers
missivity component is usually considered to be zero As mentioned in the Introduction section, one of the
(this is indeed the case of the application described main problems with ear thermometers is the narcis-
herein). In this case, Equation (2) is used to express sism of the probe; the amount of radiation emitted
the relations among the relevant physical quantities. by the probe itself and reflected on the ear drum
surface whose radiation is intended to be measure.
(
= σ ⋅ ε ⋅T
Tsurfac
f e
4
Treflected 4 ) (2) Due to the scale factor and other constraints, the
probe has to be pointed towards the ear drum with
In (2) W represents the magnitude of the point- an angle higher than 85°, because the sensor has
ing vector in the surface, σ the Stefan-Boltzmann roughly the same size of the object measured and
constant, ε the emissivity component and Treflected is placed at a distance of less than 3 cm. To get
the reflected apparent temperature that has to be more accurate readings the measurement may not
determined (ISO 18434-1 2008). be done immediately after inserting the probe. The
In this paper, it is described a new thermo- probe is at room temperature and after insertion
graphic method to accurately measure human body it is necessary to wait a certain amount of time to
temperature. This method is based on the thermo- allow the establishment of the thermal equilibrium
graphic measuring of the external auditory meatus between the patient and the probe.
temperature. It is shown that this method provides
accurate results, overcoming the problems faced
2.4 Temporal artery thermometers
by previous contact and contact-less measurement
methods, while enabling quick temperature meas- As already mentioned, temporal artery thermom-
urements, broadening its application possibilities. eters are contact thermometers with an additional
accuracy problem related with positioning. If not
properly placed over temporal artery the reading is
2 TRADITIONAL HUMAN
inaccurate. Furthermore, like the ear thermometer
TEMPERATURE MEASUREMENT
each measurement consumes a disposable cap.
METHODS

2.1 Clinical thermometers 2.5 Inner canthus

The clinical thermometer is a contact thermometer During the epidemic of Severe Acute Respiratory
composed by a bulb, the sensor, and a transparent Syndrome (SARS) it was observed that infected

76

VIPIMAGE15_Book.indb 76 9/8/2015 7:42:57 PM

 patients experienced an increase of temperature 3 EXTERNAL AUDITORY MEATUS
on the eyes zone (Ring 2012). This fact, in com-
bination with the morphology of the ophthalmic The set formed by the external auditory meatus—
artery led to considerer the inner canthus a pos- the external ear canal—, and the eardrum, Figure 1,
sible spot to estimate core temperature in human is no more than a mechanical waveguide terminated
beings. Indeed, the ophthalmic artery, fed directly by an adapted mechanical load. This condition is
by the internal carotid artery, is located beneath essential to not have echo formed within our ears.
the inner cantus where it splits in two, the nasal This condition is complemented by another one:
artery and the frontal artery (Guyton & Hall the isothermal nature of the external auditory mea-
2000). Despite these characteristics, the inner tus cavity assures that the air inside it has an aver-
canthus has a series of snags that compromise, in age speed equal to 0. This second condition ensures
thermographic terms, the accuracy of the calcu- the absence of the Doppler Effect in the external
lations done to determine the core temperature auditory meatus, which can only be achieved when
(Wood 1964). there are no temperature differences, thus ensur-
Human skin is known to have an emissivity of ing that there are no static pressure differences. If
98% (Ring 2012), regardless of the color of the no pressure differences are observed, no flow will
skin. This feature imposes the need to estimate exist, no mass transport from one side to the other.
the reflected apparent temperature. If this is not On the other hand, it also ensures that the speed
done, a reference source with a very well-known of the sound propagation does not change along
emissivity and a very well defined temperature the external auditory meatus, from the moment the
has to be put in the field of vision of the thermo- wave enters it until it reaches and is absorbed by
graphic camera to allow for the computation of the eardrum.
the reflected apparent temperature. As a rule, this The canal and the eardrum have the same coat-
reference source has to be on the same plane of the ing characteristics as both are coated with the
inner canthus, in order to get a correct reference same material, skin. Therefore, there are no dif-
measurement. However, filling 75% of the field of ferences on the material in contact with the wave
vision of the camera (ISO/TR 13154 2009) with while it travels through the waveguide and until
the face of the patient to ensure a Measurement being absorbed by the charge, the eardrum.
Field of View (MFOV) smaller than the inner can- From an electromagnetic point of view, the
thus size, compromise in several ways the estima- geometry of the cavity creates in its opening the
tion of the reflected apparent temperature. First optical illusion of a blackbody surface. This phe-
because the rule mentioned before cannot be kept, nomenon prevents us from seeing into the cavity,
and second because the narcissism effect of the since the electromagnetic waves, after entering the
camera, and possibly of the operator over the canal, and after successive reflections, are com-
scene, is huge. pletely absorbed within the cavity. This means that
Since the outer layer of the skin has no irriga- the incoming electromagnetic radiation gets com-
tion, heat flow can only be achieved in a passive pletely trapped inside the cavity. Consequently, this
way through thermal conduction. This outer layer makes the entrance of the external auditory mea-
acts as a low-pass filter and the temperature calcu- tus a dark spot.
lated is always below or above the core temperature
depending on room temperature—if this is below
or above core temperature. Gound accumulation
at the inner canthus may amplify this effect.
Cerebrovascular diseases may also compro-
mise readings because they may cause a dimin-
ishing in the blood flow in the ophthalmic artery
and consequently in the amount of heat dissi-
pated at the inner canthus, leading to a drop in
the superficial temperature and promoting false
negatives as a result (Ring 2012). The opposite
effect takes place when a healthy person makes
physical efforts. The increase of the muscular
metabolism during exercise generates heat that
needs to be dissipated in order to maintain body
heat balance. To help dissipate this excess heat,
capillary blood vessels dilate, increasing facial
temperature and thus the possibility of produc- Figure 1. External auditory meatus (Gray &
ing false positives. Lewis 1918).

77

VIPIMAGE15_Book.indb 77 9/8/2015 7:42:57 PM

 However, and according to the law of Stefan- Unlike ear thermometers which are affected by
Boltzmann, all bodies with a temperature above crooked external auditory meatus or wax, because
absolute zero emit radiation. So there is radiation they have to “see” the eardrum, thermographic
coming out of the external auditory canal. Since cameras do not have their accuracy compromised
the external auditory meatus and the eardrum by these issues, provided that they are not visible
are at the same temperature, the set acts as a heat from the outside. If they are not seen from the
source with a single temperature. Taking into outside, crooked external auditory meatus or wax
account that all the radiation that enters the exter- accumulation are no longer a problem.
nal auditory meatus is absorbed inside, there are Therefore, none of the previously identified
no reflected radiations coming from the external sources of error interfere with the thermographic
auditory meatus. Therefore it is possible to cor- measurement, thus guaranteeing an accurate and
relate the radiation radiated through the external quick core temperature measurement. In all situa-
auditory meatus opening with the temperature of tions, the external auditory meatus, when observed
the heat source and thus to measure the tempera- from the outside, always have an emissivity of, or
ture of the eardrum located 3.5 cm away from the very close to, 100% (99,971% (Saunders 2009)),
entrance of the external auditory canal. relieving the measurement from the determination
In sum, due to its geometrical configuration and of the reflected apparent temperature. For that
characteristics, an illusion of blackbody is always reason,
present at the interface between the external audi-
tory meatus and the exterior. W σ T4 (3)
As in any isothermal cavity, the meatus is com-
pletely surrounded by a volume, the head. The cav- which may be derived from (2) when emissivity
ity has a truncated conical shape with a ratio of tends to 1.
3.5 between the length and the opening diameter
(1 cm × 3.5 cm) (Guyton 2000), with the eardrum
in the truncated end. 4 SOURCES OF UNCERTAINTY
Like a blackbody of room temperature, all radi-
ation entering the meatus is trapped inside. The The MFOV poses the greatest concern when using
radiation that exits the external auditory meatus is thermographic cameras to perform temperature
only the radiation that is radiated inside. The exter- measurements. MFOV corresponds to the smallest
nal auditory meatus, when observed from the out- object whose temperature the camera can measure.
side with a thermographic camera appears as an Typically its value varies between 3 (3-by-3 pixels =
isothermal surface. No details can be seen inside. 9 pixels) and 5 (5-by-5 pixels = 25 pixels) times the
Only a flat surface at the same temperature (same size of each pixel in the camera image sensor. If
level of radiation) can be observed, as seen in the MFOV is 3 then 9 pixels are need to measure
Figure 2, an “optical” illusion. This virtual surface, the object temperature. Therefore, the minimum
by definition, has an emissivity of 1. No calculus measurement size depends on the distance—
or estimation of reflected apparent temperature closer the object, smaller the measurement area.
and emissivity are needed. However, the distance to the object may not be
reduced to zero as there is the need to focus the
camera. Furthermore, changing the lenses changes
the MFOV and the minimum focus distance.
To be able to correctly read the temperature at
the external auditory meatus the thermographic
camera needs to have an MFOV smaller than
the diameter of the meatus—the diameter of the
external ear canal. In addition, the camera needs
to have the emissivity parameter selectable to 1 and
preferably the possibility to be programmed to find
and measure the maximum temperature within a
defined area.
The thermographic cameras available on the
market have an MFOV smaller enough to allow
in the limit an accurate temperature reading in
adult patients. However, in pediatric practice,
as seen in Figure 3, a camera with a detector of
320 × 240 pixels may be necessary. Cameras with
Figure 2. Thermogram of the ear. detectors of 120 × 120 pixels, a field of view of

78

VIPIMAGE15_Book.indb 78 9/8/2015 7:42:58 PM

 hand held thermographic cameras can be used to,
in less than 5 seconds, measure patient’s tempera-
ture without annoying or waking up her/him. In
cases where a tight control of the temperature is
required, like in operation theatres, measurements
can be done continuously during surgery without
having the patient wired.
However, other no traditional applications
where human control temperature may be impor-
tant may also be envisaged. That’s the case of
airports, train stations or other wide open public
spaces, like receptions, where fixed stations with
or without an operator can be installed to moni-
tor passers’-by temperature. One possibility is to
add thermographic cameras to ePassport gates,
Figure 3. Child being monitoring during sleep. automated self-service passport barriers recently
installed in airports worldwide. To pass the barrier,
a person needs to stand still in front of a camera
25° and minimum focus distance of 12 cm, giv- for just some seconds, before being authorize to
ing a minimum measurement size of 2.31 mm, are walk through the gate. At the same time, she/he
examples of ideal cameras for all types of patients may be warned to pull back her/his hair from in
(Cardoso 2012a, b). front of her/his ears to allow an accurate tempera-
Another source of uncertainty is the Noise ture measurement.
Equivalent Temperature Difference (NETD),
defined as the amount of infrared radiation
required to produce an output signal equal to the 6 CONCLUSIONS
systems own noise. In practice, it is the minimum
level of thermal sensitivity exhibit by the camera. The approach proposed in this paper to measure
Currently, typical values for NETD are below, or human temperature based on the use of a thermo-
equal to, 100 mK, allowing measurement resolu- graphic camera and on the characteristics of the
tions of, at least, 100 mK. external auditory meatus has several advantages
In stand still stations, depth of field, out of over other traditional methods. No disposable
focus and positioning of the camera in relation parts are needed and therefore it does not produce
to the spot where persons should stand may also waste. It is a completely non-intrusive method and
be sources of problems. The place where a person therefore it does not cause any discomfort to the
stands must be close enough to minimize the ratio patient. It is a predictable, more accurate method
MFOV/meatus diameter, and at the same time far than the other previously described methods, if a
apart to maximize depth of field. Measuring at a set of proper practices are implemented.
distance too close to the minimal focus distance Currently, the longest flight takes less than
may introduce uncertainty, because any focus error 24 hours long and this is the time needed for a
may not be compensated by the depth of field. H5N1, or other virus, to travel across the world.
Further studies are needed to establish the con- Long-haul flights are therefore a major con-
tributions of each one of these uncertainty sources cern for the role they can play in disease spread-
and to implement proper operation practices with ing. At airports, the current method implies the
the goal to reduce or compensate them. In the removal of glasses to observe the temperature in
limit, if a proper operation practice is achieved, the inner canthus. Since a 98% emissivity should
only the single and unavoidable NETD source of be expected, errors are to be consider. False posi-
uncertainty will remain. tives are annoying for those spotted, but the major
concern is with false negatives. In airports, where
the access from the plane to the terminal is done
5 APPLICATIONS exposed to the elements, the patient’s face can chill
in frosty weather and the alarm at the thermo-
The simplicity of the measurement procedure graphic camera may not sound, failing a potential
effectively broadens its number of applications. positive (Chan et al. 2004).
Traditional clinical uses like physicians’ offices, While further studies are needed to reduce
ICUs, CCUs, Cardiac/Coronary Care Units, Emer- uncertainty sources to a minimum and to imple-
gency Rooms or in the nursing practice in patients ment proper operation practices, this method
wards are just a small number of examples where seems to be the most accurate and promising and

79

VIPIMAGE15_Book.indb 79 9/8/2015 7:43:00 PM

 able to cover a broader spectrum of applications ISO 18434-1 2008. Condition monitoring and diag-
when compared to existing methods. nostics of machines. International Organization for
Standardization, Thermography—Part 1: General
procedures.
ISO/TR 13154 2009. Medical electrical equipment—
REFERENCES Deployment, implementation and operational
guidelines for identifying febrile humans using a
Cardoso, A. 2012a. Human Temperature Measurement screening thermograph. International Organization
with Infrared Cameras. 7th International Infrared for Standardization.
Forum infraR&D. Hannover, Germany. Ring, F. 2012. Fever Detection in Children with Infrared
Cardoso, A. 2012b. Screening fever, a new approach. XII Thermal Imaging using the new ISO Recommenda-
EAT Congress on Thermology. Porto, Portugal. tions. 7th International Infrared Forum infraR&D.
Chan L.S., Cheung G.T., Kumana C.R. & Lauder I.J. Hannover, Germany.
2004. Screening for fever by remote-sensing infra- Saunders, P. 2009. Calibration of Low-Temperature
red thermographic camera. J Travel Med. 2004 Infrared Thermometers. Measurement Standards
Sep–Oct;11(5):273–9. Laboratory of New Zealand. Techniqueal Guide 22.
Gray, H. & Lewis, W.H. 1918. Anatomy of the human Wood, E.H. 1964. Thermography in the Diagnosis
body. Philadelphia: Lea & Febiger. of Cerebrovascular Disease: Preliminary Report.
Guyton, A.C. & Hall, J.E. 2000. Textbook of medical Radiology. September 1964, 83:540–542.
physiology London: W.B. Saunders.

80

VIPIMAGE15_Book.indb 80 9/8/2015 7:43:01 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Video-based Turkish Sign Language recognition systems

M. Aktaş & E. Battini Sonmez

Computer Engineering Department, İstanbul Bilgi University, Istanbul, Turkey

ABSTRACT: Automatic recognition of sign language will be extremely beneficial for hearing impaired
people, by allowing them to easily communicate with the outside world. Sign language is a non-verbal and
visual language, which evolved detached from the spoken language and it is based on the simultaneous
contribution of both temporal and spatial configurations. Even if it is spread all over the world, sign lan-
guage is not a universal language and every country needs to build its own automatic system. This paper
challenges the issue of the development of automatic video-based Turkish Sign Language (TSL) recogni-
tion systems; it presents two computer vision-based automatic systems for TSL recognition and compares
their performances. Since we are using a newly released TSL database, this paper has also the side-effect
to divulge the database to the research community.

Keywords: video-based sign recognition; Turkish Sign Language; spatial-temporal features

1 INTRODUCTION processing techniques to extract informative fea-

tures and classify body movements, when a deaf
Sign Language (SL) is the most important com- person makes a sign. We believe that a system
munication channel among hearing impaired peo- designed to work in a public place must be as natu-
ple; however, since the majority of subjects outside ral as possible and, because of this, we decided to
the deaf community do not know this language, build different computer vision-based systems and
the communication between deaf and hearing to compare their performances.
societies is still too weak. Automatic systems capa- To summarize, this paper challenges the issue of
ble to translate SL into text and/or speech and vice the development of automatic video-based Turk-
versa could bridge the existing gap between deaf ish Sign Language (TSL) recognition systems; it
and hearing people. uses the Turkish Sign Language database recently
Sign language is characterized by manual and introduced by Memis and Albayrak (Memiş &
non-manual features; examples of non-manual Albayrak 2013).
parameters are gaze, facial expression, and mouth, The rest of the paper is organized as fol-
head and trunk movements. Since sign language low: Section 2 discusses the related work;
evolved and separated from the spoken language, Section 3 introduces the TSL database used in this
their grammars are fundamentally different from paper; Section 4 gives an overview of the proposed
each other. systems; Section 5 details experiments. Finally, con-
Furthermore, to make the problem more diffi- clusions and future work are drawn in Section 6.
cult, every country has its own sign language, and,
therefore, there is also the need to build different
systems, one for every country. 2 LITERATURE REVIEW
In 1995, Waldron and Kim (Waldron & Kim
1995) published one of the first studies on auto- Automatic sign language recognition systems have
matic American Sign Language; since then, differ- been investigated since few decades; the most com-
ent researchers tacked this issue, but the problem is mon classification techniques used in this field are
still open, because it is particularly challenging due Hidden Markov Models (HMM) and Neural Net-
to the complexity of translating manual and non- works (NN).
manual characteristics into a word or sentence. In 1997, (Vogler & Metaxas 1997) Vogler and
In order to easier the recognition task, some Metaxas used HMM for continuous American
signers wear special gloves with motion-sensors; Sign Language (ASL) recognition and they were
in the following, we will refer to those methods as able to recognize 53 signs out of a completely
‘data gloves-based systems’. In alternative ‘com- unconstrained sentence; in 1999, the same authors
puter vision-based systems’ rely on classical image were able to recognize ASL sentences with 22 signs

81

VIPIMAGE15_Book.indb 81 9/8/2015 7:43:01 PM

 based on ASL phonemes; and, in 2004, (Vogler & “time” (category 2, 61 words), “jobs” (category 3,
Metaxas 2004), they extended their study to par- 58 words), “associations and places” (category 4,
allel HMM to implement a multi-channel ASL 39 words), and “school” (category 5, 54 words).
system, with sensory gloves. In 2001, (Wang, Every word or sentence is played in a short video
Gao, & Xuan 2001), Wang et al. used HMM to by a professional deaf signer; 4 different people (or
design a data gloves based system for Chinese actors) mimic all words.
Sign Language (CSL) recognition; in 2003, (Allen, The database is divided into 2 sections, the first
Asselin, & Foulds 2003) Allen et al. developed an section stores words and sentences belonging to
Artificial Neural Network (ANN) to recognize category 1 (storing 75 videos of player 1, 72 videos
24 ASL letters; in 2003, Chen et al. used HMM of player 2, 72 videos of player 3 and 75 videos of
to implement a Chinese Sign Language recogni- player 4), category 2 (with 184 172 185–184 videos
tion system, with sensory gloves; in 2004, (Wang, respectively played by actor 1, 2, 3 and 4) and cate-
Sarawate, & Leu 2004), Wang et al. used ANN and gory 3 (storing 171 172 171–167 videos respectively
HMM to design an ASL gesture recognition sys- played by player 1, 2, 3 and 4); the second section
tems with sensory gloves; in 2005, (Mohandes & stores words and sentences belonging to category 1
Deriche 2005), Mohandes and Deriche used HMM (with 72 videos of player 2 and 72 videos of player
to design a system for Arabic Sign Language rec- 4), category 4 (storing 119 videos of player 2 and
ognition; in 2007, (Munib, Habeeb, Takruri, & Al- 125 videos of player 4) and category 5 (with 163
Malik 2007), Munib et al. used Hough transform videos of player 2 and 168 videos of player 4). To
and neural networks for ASL recognition; in 2009, summarize section 1 stores a total of 1700 videos
Lee and Tsai used 3D data and Neural Networks and section 2 contains 1450 videos, respectively.
to recognize Taiwan sign language. Table 1 shows the distribution of the 1700 vid-
The study of the Turkish Sign Language (TSL) eos belonging to section 1 among the 4 players and
is still at the very beginning: in 2005 Haberdar and the 3 categories.
Albayrak (Haberdar & Albayrak 2005) used local
features with a HMM based system for TSL rec-
ognition and reached 95.5% recognition rate over Table 1. Distribution of the 1700 videos in section 1.
50 sign words; in 2011, (Altun & Albayrak 2011),
Altun and Albayarak, used Hough transform to Section 1
create a Turkish finger-spelling recognition system.
Players Category 1 Category 2 Category 3
In 2013, (Memiş & Albayrak 2013), Abbas and
Albayrak introduced to the research community 1 75 184 171
a new TSL database together with a benchmark 2 72 172 172
algorithm which classifies 1002 videos representing 3 72 185 171
111 words and short sentences of the Turkish Sign 4 75 184 167
Language; they reached their best performance of
91.52% using both RGB and depth videos.
The aim of this paper is to contribute to the
study of automatic TSL recognition systems
by using the recently introduced TSL database
of Abbas and Albayrak; in this initial study, we
extend the benchmark experiment of Abbas et al
by working with all videos of section 1.

3 THE TURKISH SIGN LANGUAGE

DATABASE

The TSL database of Abbas et al. consists of a

total of 6300 videos divided into two sections, stor-
ing 1700 and 1450 videos, respectively; all videos
are in both RGB and depth image formats. The
label of every video records information related to
the section, the category, the words (class), and the
signers.
The entire database has a total of 236 words
and sentences, which are grouped into the 5 cat-
egories of “daily life” (category 1, 24 words), Figure 1. Selected frames of the sign ‘excuse me’.

82

VIPIMAGE15_Book.indb 82 9/8/2015 7:43:01 PM

 Like Abbas et al. we worked only with section 1,
which stores words and sentences of daily life’s
conversation (e.g. excuse me, I’ll see you then,
goodbye, good morning, where, when, what, etc.)
as well as time’s related terminology (e.g. August,
month, sometimes, this week, today, in the after-
noon, etc.), and job’s key words (e.g. cook, soldier,
lawyer, shoe repairer, hairdresser, etc.).
Figure 1 shows selected frames of the sentence
‘excuse me’.
The database was recorded using the Microsoft
Kinect camera, which gave the possibility to record
RGB images and depth images; key landmarks of
the body posture, were not recorded.

4 THE PROPOSED SYSTEMS

Computer vision-based systems for automatic rec-

ognition of TSL rely on classical image processing
techniques to extract informative features and clas-
sify body movements, when a deaf person makes
a sign.
In this paper we compare the performance of
two systems, both of them are an extension of the
benchmark algorithm proposed by Abbas et al.
together with the new TSL database, (Memiş &
Albayrak 2013).

4.1 Reading and preprocessing Figure 2. A sample gray frame before (a) and after
(b) moving the useless background information.
We used all the 1700 videos of section 1, the aver-
age number of frames per video is 43, their initial
size is 640 × 480. Due to out of memory problems, successive frames is to be able to determine the area
it has been necessary to extract and memorize only where the motion of hands is significant because,
the relevant information already during the read- if the values of some pixels change from one frame
ing step. to another, these form a lighter area within the dif-
First of all, RGB frames were converted into ference of images, IDi.
gray frames, after that, a naive cropping method Subsequently, the difference image is converted
was developed and applied on the frames of each into a binary image by a threshold function; the
video. The aim of this cropping is to eliminate value of the threshold is database depended and it
useless background information, and keep only has been fixed, manually, to 50. Figure 3 shows the
the player’s body figure; this results in frames of thresholded image of differences.
size 361 × 427. The input and output frames of Finally, all thresholded images of differences are
naive cropping function can be seen in Figure 2. summed up to calculate a spatial-temporal feature,
named by Abbas et al. the accumulated motion
4.2 Feature extraction image:
Every two consecutive frames in a video produce n −1
an ‘image of differences’, which is calculated using Avi ∑ i 1 Ith i

the following formula:

where Ithi stores the ith thresholded image of dif-
I Di I i − I i +1, ∀i = 1, 2, ..., n ference of the video. The accumulated image, Avi,
represents the whole video in one frame; it is the
where Ii represents the ith frame of the video, spatial-temporal feature of that video in the form
and IDi stores the difference of images; hence, of a matrix.
n − 1 ‘image of differences’ are obtained from a In an alternative system, we extracted features in
video with n frames. The logic behind subtracting the frequency domain. The only difference of this

83

VIPIMAGE15_Book.indb 83 9/8/2015 7:43:02 PM

 Table 2. Confusion matrix of first 6 classes within
fold 1 in trial 1.

c1 c2 c3 c4 c5 c6

c1 1 0 0 0 0 0
c2 0 4 0 0 0 0
c3 0 0 3 1 0 0
c4 0 0 1 3 0 0
c5 0 0 0 0 5 0
c6 0 0 0 0 0 3

Table 3. The corresponding words represented in

Table 2.

Corresponding sign in Turkish In English

Figure 3. A sample thresholded difference image. c1 Afedersiniz Excuse me
c2 Afiyet olsun Bon appetit
c3 Gorusmek uzere See you soon
system is the application of 2D Discrete Cosine c4 Gorusuruz See you
Transform (DCT) after obtaining the spatial- c5 Gule gule Goodbye
temporal feature, Avi. c6 Gunaydin Good morning

4.3 Classification
Table 4. Accuracy table using spatial-
We used the K-fold cross validation method to temporal features.
separate all samples into training and test sets;
we fixed K = 3, and we divided the data set into Fold 1 Fold 2 Fold 3
three nearly equal subsets. Since the algorithm uses
random selection to assign samples to folders, the Trial 1 84.30 82.54 84.63
whole process is repeated for 5 times and the given Trial 2 82.54 80.78 83.04
performance is the average of 3 × 5 = 15 trials. Trial 3 79.72 85.89 80.57
We used the Euclidean distance to built a Trial 4 83,95 84.10 81.48
Distancei,j matrix of dimensions M × N, where M Trial 5 82.54 80.92 78.48
is the number of test samples and N is the number
of training samples:
Table 5. Accuracy table using frequency
Distancei j T ri − TrainVectorj )2
(TestVector domain features.

Fold 1 Fold 2 Fold 3

Classification is performed by assigning every
test sample to the class of the nearby training sam- Trial 1 72.13 75.84 76.50
ple; that is, we used the K-Nearest Neighbor algo- Trial 2 78.09 75.49 79.01
rithm with K = 1. Trial 3 78.66 75.44 75.49
Trial 4 76.50 76.72 79.89
Trial 5 78.80 75.66 75.49
5 EXPERIMENTAL RESULTS

We worked with 1700 RGB videos of session 1

of the Turkish Sign Language database created of confusion matrix belonging to one sample fold
by Abbas et al., (Memiş & Albayrak 2013). We of one trial.
randomly assign an order to all samples and we Because the confusion matrix is of dimension
repeated the benchmark algorithm of Abbas et al.; C × C, where C = 143 is the number of classes, it is
that is, we applied the 3-fold cross validation tech- not possible to properly display the whole matrix.
nique to divide the database into train and test sets; Obviously, different accuracy rates were
all experiments are repeated 5 times and the given achieved for each fold within each trial; the follow-
performance is their average. Table 2 is a portion ing table shows the performances obtained with

84

VIPIMAGE15_Book.indb 84 9/8/2015 7:43:03 PM

 both system 1, based on temporal features, and Memiş, A. & S. Albayrak (2013). A kinect based sign
system 2, using frequency domain features: language recognition system using spatio-temporal
The overall performance is determined by calcu- features. In Sixth International Conference on Machine
lating the mean of all accuracies. Our best perform- Vision (ICMV 13), pp. 90670X–90670X. International
Society for Optics and Photonics.
ance is achieved using spatial-temporal features; it Mohandes, M. & M. Deriche (2005). Image based arabic
is equal to 82.37%. We claim that it is comparable sign language recognition. In Signal Processing and
with the benchmark result given by Abbas et al. Its Applications, 2005. Proceedings of the Eighth
since we worked with all 1700 videos of section 1. International Symposium on, Volume 1, pp. 86–89.
IEEE.
Munib, Q., M. Habeeb, B. Takruri, & H.A. Al-Malik
6 CONCLUSION AND FUTURE WORK (2007). American sign language (asl) recognition
based on hough transform and neural networks.
Automatic recognition of sign language helps to Expert systems with Applications 32(1), 24–37.
Vogler, C. & D. Metaxas (1997). Adapting hidden markov
strength the connection between deaf and hear- models for asl recognition by using three-dimensional
ing societies. It is a very challenging and important computer vision methods. In Systems, Man, and
research field which requires more attention. Cybernetics, 1997. Computational Cybernetics and
We worked on Turkish Sign Language recogni- Simulation., 1997 IEEE International Conference on,
tion and we challenged the given benchmark algo- Volume 1, pp. 156–161. IEEE.
rithm released together with the database. Current Vogler, C. & D. Metaxas (2004). Handshapes and move-
results are promising, but we are willing to do better ments: Multiple-channel american sign language
by using the full potentiality of depth images. recognition. In Gesture-Based Communication in
Human-Computer Interaction, pp. 247–258. Springer.
Waldron, M.B. & S. Kim (1995). Isolated asl sign
recognition system for deaf persons. Rehabilitation
REFERENCES Engineering, IEEE Transactions on 3(3), 261–271.
Wang, C., W. Gao, & Z. Xuan (2001). A real-time large
Allen, J.M., P.K. Asselin, & R. Foulds (2003). American vocabulary continuous recognition system for chinese
sign language finger spelling recognition system. In sign language. In Advances in Multimedia Information
Bioengineering Conference, 2003 IEEE 29th Annual, Processing-PCM 2001, pp. 150–157. Springer.
Proceedings of, pp. 285–286. IEEE. Wang, H., N. Sarawate, & M. Leu (2004). Recognition of
Altun, O. & S. Albayrak (2011). Turkish fingerspelling american sign language gestures with a sensory glove.
recognition system using generalized hough trans- In Japan USA Symposium on Flexible Automation,
form, interest regions, and local descriptors. In Pattern Denver, CO, pp. 102–109.
Recognition Letters, pp. 1626–1632. Springer-Verlag,
Berlin Heidelberg.
Haberdar, H. & S. Albayrak (2005). Real time isolated turk-
ish sign language recognition from video using hidden
markov models with global features. In Computer
and Information Sciences (ISCIS 2005), LNCS, pp.
677–688. Springer-Verlag, Berlin Heidelberg.

85

VIPIMAGE15_Book.indb 85 9/8/2015 7:43:03 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

A nonsmooth nonconvex sparsity-promoting variational approach

for deblurring images corrupted by impulse noise

A. Lanza, S. Morigi & F. Sgallari

Department of Mathematics, University of Bologna, Bologna, Italy

ABSTRACT: We propose a new variational approach for the restoration of images simultaneously cor-
rupted by blur and impulse noise. Promising results have been obtained by the 1-TV variational model
which contains a Total Variation (TV) regularization term and a nonsmooth convex 1-norm data-fidelity
term. We introduce a sparsity-promoting nonconvex data-fidelity term which performs better but makes
the problem more difficult to handle. The main contribution of this paper is to develop a numerical algo-
rithm based on the Alternating Direction Method of Multipliers (ADMM) strategy and on the use of
proximal operators. Preliminary experiments are presented which strongly indicate that using nonconvex
versus convex fidelities holds the potential for more accurate restorations at a modest computational
extra-cost.

1 INTRODUCTION several filters were proposed, e.g. Adaptive Median

Filter (AMF) is used to detect salt-and-pepper
In this paper we present a variational approach for impulse noise, Adaptive Center-Weighted Median
the restoration of blurred images in the presence Filter (ACWMF) and Rank-Ordered Logarithmic
of impulse noise, which can be due to transmission Difference (ROLD) have been utilized to detect
errors, malfunctioning pixel elements in camera random-valued impulse noise, see (Dong, Chan, &
sensors, faulty memory locations, etc.. An impor- Xu 2007). In (Chan, Dong, & Hintermuller 2010),
tant characteristic of impulse noise is that only a a two-phase image restoration method based on
subset of the pixels is corrupted by noise while
2
the TV regularization combined with an 1−data-
rest is noise-free. In particular, let ∈R d be the fitting term for impulse noise removal and deblur-
unknown clean image concatenated into a column ring has been proposed.
vector, and defined, without loss of generality, on Given K and g , our goal is to solve the inverse
a square d d domain Ω, with values in [ 0 1] . The problem of recovering an as accurate as possible
estimate u∗ of the clean image u. Since the blur-
2
observed corrupted image g ∈R d at pixel location
i ∈Ω is modelled as follows: ring operator is typically very ill-conditioned or
even singular, some a-priori preferences are usually
⎧( Ku )i for i ∈ Ω0 added in the form of a regularization term in order
gi = ⎨ (1) to get meaningful estimates.
⎩ ni fo i ∈ Ω1 := Ω \ Ω0 ,
We propose the following restoration model:
2 2
where K ∈R d 2d is a known linear blurring opera- ⎧⎪ d 2 ⎫⎪
tor and n ∈R d is an unknown realization of the min2 ⎨ ∑ φ (((( )i ; a ) μ (u ) ⎬ , (2)
impulsive noise process. For images corrupted by u Rd ⎩ ⎪ i =1 ⎭⎪
salt-and-pepper impulse noise, ni take either 0 or 2
1 values, while for images corrupted by random- where TV (u ) := ∑ id=1 || (∇u )i ||2 denotes the dis-
valued impulse noise, ni ∈[ 0,1] are uniformly dis- crete TV semi-norm of u, and φ is the so-called
tributed random numbers. In the discrete model penalty function φ : R → R with parameter a.
(1) of the image degradation process we assume The cost functional reflects the need to deblur
that Ω1 is unknown. the image, recover its discontinuities and at the
The most popular methods to solve this image same time remove the impulse noise. The regulari-
restoration problem are based on a two-stage zation parameter μ controls the trade-off between
approach which estimates the noise-corrupted the fidelity term and the regularization term. It
region Ω1 before estimating u. In these approaches, is well known that the TV regularizer success-
the second stage becomes a regular image deblur- fully allows for discontinuities while smoothing
ring problem, while for the accurate detection of Ω1 the other regions, and thus we are motivated to

87

VIPIMAGE15_Book.indb 87 9/8/2015 7:43:04 PM

 consider TV norm for our proposed model. In this A1) φ(.;a)∈C0 (R) (continuous)
work we draw a particular attention to the choice A2) φ(−x;a) = φ(x;a) ∀x∈ R*+(even)
of the data-fidelity term. A3) φ(.;a)∈C2 (R*+)
A popular choice for the penalty function φ in A4) φ′(x;a) > 0 ∀x∈ R*+(increasing)
presence of additive Gaussian noise is φ ( ) x 2. A5) φ″(x;a) ≤ 0 ∀x∈ R*+(concave)
However, this choice seems to be inadequate for the A6) infx∈ R* φ″(;a) = −a
+

degradation model (1) since the quadratic function

assigns too much weight to distant points which Due to the last assumption A6), we will refer
could be potential outliers. to as the concavity parameter. Three examples
Nonsmooth, convex 1-norm penalty terms, of penalty functions, referred to as φlog, φrat and
φ ( ) = | x |, were shown to be more useful towards φatan, which satisfy the above assumptions A1)–A6)
this aim, since they are robust to outliers, i.e., to and which have been considered, e.g., in (Chen &
impulse noise, see (Nikolova 2004) and (Chan, Selesnick 2014), (Lanza, Morigi, & Sgallari 2015),
Dong, & Hintermuller 2010). are reported in Table 1 together with some prop-
For the data fidelity term in (2) we propose in erties. We notice that for a  0, the three penalty
Sect. 2 nonconvex and nonsmooth penalty func- functions tend to the convex absolute value func-
tions φ aimed at strongly promoting sparsity of the tion | |, thus yielding the 1-norm fidelity. In
residual. However, this leads to a nonconvex mini- Figure 1 we show the plots of the three considered
mization problem (2) which cannot benefit from penalty functions for two different values a = 2 and
convex optimization theory and methodology to a = 3 of the concavity parameter: the higher is a,
obtain robust algorithms and ensure convergence. the more concave is φ .
In Sect. 3 the minimization of the proposed non-
smooth nonconvex functional (2) is carried out
by a suitable variant of the ADMM which allows 3 ADMM FOR THE PROPOSED MODEL
to split the original nonconvex problem into a
sequence of simple convex subproblems whose In this section we introduce a suitable variant of
(unique) solutions can be obtained very efficiently. the basic ADMM approach (Boyd, Parikh, Chu,
The numerical experiments in Sect. 4 demonstrate Peleato, & Eckstein 2011) to solve the proposed
that the proposed method is effective in deblurring nonconvex nonsmooth minimization problem in
images corrupted by impulse noise. (2). Toward this aim, first we introduce two o2 aux-
iliary variables
2
t Q and r V , with V := R d and
Q := R 2 d , which allow to reformulate (2) into the
2 NONCONVEX PENALTY FUNCTIONS following linearly constrained equivalent form:

d2
In the following, we denote the sets of non-negative
and positive real numbers as R + : { ∈ R : ≥ } { ∗ ∗ ∗
}← in ∑ { (r
g min (ri a)) || i || }
u ,t ,r
and R∗+ : { ∈ R : > }, respectively. To induce i =1 (3)
sparsity of the residual Ku − g according to the s.t.: t = Du, r = Ku
K − g,
proposed restoration model (2), we consider 2 2 2
parameterized nonconvex penalty functions where D : ( Dh ;DDv ) ∈ R 2 d d , Dh Dv ∈ R d are lin-
φ ( ) : R → R, with parameter a ∈ R∗+, satisfying ear operators representing the discretization of the
the following assumptions: first-order horizontal and vertical partial derivatives,

Table 1. Three examples of sparsity-inducing penalty functions.

φlog φrat φatan

φ( ) 1 | | ⎛ 1 + 2 | x |⎞ π
a log(1 + a | x |) 1 + | x |//2 atan −
⎝ 3 ⎠ 6
a 3 /2
φ( ) 0 0 0
φ ′( ±
) ±1 ±1 ±1
±
φ ″( ) −a −a −a
lim φ ( ) +∞ 2
x→±∞ a 2π 3
a 9

88

VIPIMAGE15_Book.indb 88 9/8/2015 7:43:07 PM

 In the following subsections we show in detail
how to solve the three sub-problems (6)–(8) for the
variables t, r and u, respectively, then we present
the overall iterative ADMM-based minimization
algorithm.

3.1 Solving the sub-problem for t

Figure 1. Plots of the penalty functions φlog ( x a ), Recalling the definition of the augmented Lagrang-
φrat ( x a ), φatan ( x a ) defined in Table 1 for two different ian functional L in (4), after some simple manipu-
values a = 2 (left) and a = 3 (right) of the concavity lations the sub-problem for t in (6) can be written
parameter. as follows:
d2 ⎧
β 2⎫
respectively, and i (( Dhu )i ( Dv u )i ) ∈ R repre- 2 t k +1 ←arg
ag i
t Q
∑ ⎪⎨⎪ ti 2 + 2 μt ti − qik 2 ⎪⎬⎪ , (10)
i =1 ⎩ ⎭
sents the discrete gradient of image u at pixel i .
To solve (3), we define the augmented Lagrang- with the constant (w.r.t. t) vectors qik ∈R 2 given
ian functional by
d2
L( λt λr ) = ∑ {φ ((rri ; a)) || i ||2 } qik ⎛ k⎞
⎝ Du ⎠ i
1 ⎛ k⎞
λt , i = 1, , d2 , (11)
i =1 βt ⎝ ⎠ i
βt
− 〈 λt , t − Du 〉 +
2
|| t − Du ||22 ( )
and where ( Du k )i , tk ∈ R 2 denote the dis-
i
crete gradient and the Lagrange multipliers at
β
− 〈 λr , r −( K
Ku − g )〉 + r ||r −( K
Ku g )||22 , (4) pixel i, respectively. The minimization in (10) is
2 thus equivalent to the following d 2 independent
2-dimensional problems:
where βt β r > 0 are scalar penalty parameters
and λt ∈Q , λr ∈V are the vectors of Lagrange
⎧ β 2⎫
multipliers. tik +1 ← arg
ag i ⎨ ti 2 + t ti − qik ⎬ , (12)
Solving (3) is thus equivalent to search for the ⎩
ti ∈R 2 2 μ 2
⎭
solutions of the following saddle-point problem:
which, according to (Wang, Yang, Yin, & Zhang
∗ ∗
Find ( ) X ×Λ 2008), admit closed-form solutions given by the
following soft-thresholding operators:
s.t. L( ∗ ) ≤ L( ∗ λ ∗ ) ≤ L( ∗
)
∀ ( λ) X ×Λ, (5) ⎧ k μ ⎫ qik
tik +1 ⎨|| qi || − , 0 ⎬ k , (13)
with L defined in (4) and where, for simplic- ⎩ βt ⎭ || qi ||
ity of notations, we set x (u,t, r ) , λ ( λt , λr ) ,
X V × Q V and Λ = Q × V . where 0⋅(0/0) = 0 is assumed. The computational
The kth iteration of the ADMM iterative scheme cost of this sub-problem is linear in the number of
applied to the solution of (3) reads as follows: pixels d 2 .

t k +1 ← g i L(u k , t, r k ; λtk , λrk ) (6) 3.2 Solving the sub-problem for r

t Q
Given the definition of L in (4), after some manip-
r k +1 ← g (u k , t k +1, r; λtk , λrk ) (7) ulations the sub-problem for r in (7) reduces to:
r V

u k +1 ← g (u, t k , r k +1; λtk , λrk ) (8) 2

d
⎧ β 2⎫
u V
r k +1 ← argg in ∑ ⎨φ (ri ; a ) + r ⎛⎝ ri − zik ⎞⎠ ⎬ , (14)
i =1 ⎩ 2 ⎭
r V
⎧ k +1
⎪ λt
⎧⎪λ k γ βt ( t k Du k +1 )
⎨ ← ⎨t
⎪ λ k +1 Ku k +1
⎩⎪λr γ β r (r
k k
⎩ r (K g )) (9) where zik ∈R denotes the 2
ith entry of the constant
(w.r.t. r) vector z k ∈R d defined as follows:
where γ is a relaxation parameter chosen in the
interval ( ( 5 + 1)) ) , as analyzed in (Chan, 1 k
zk Ku k
K g+ λr . (15)
Tao, & Yuan 2013). βr

89

VIPIMAGE15_Book.indb 89 9/8/2015 7:43:13 PM

 The minimization in (14) is thus equivalent to 1
the following d 2 independent scalar problems: f y ) := ( y b)2 + ηφ ((yy a ) (21)
2

⎧ β 2⎫ is strictly convex if and only if

rik +1 ← g i ⎨φ (ri ; a ) + r ⎛⎝ ri − zik ⎞⎠ ⎬ . (16)
⎩
ri ∈R 2 ⎭ η a < 1. (22)
To the purpose of imposing that (16) are convex Proposition 2. Let φ : R → R be a function sat-
problems, we report Proposition 1 below, whose isfying assumptions (A1)–A6), a,η ∈ R∗+ be given
proof can be found in (Chen & Selesnick 2014), constants satisfying condition (22) and f : R → R
Proposition 2. After noticing that the minimized be the function defined in (21). Then, the proxim-
functions in (16) have the same form of function ity map proxηϕ : R → R of function φ defined as:
f defined in (21), we can conclude that (16) are
strictly convex problems if and only if the follow-
proxηφ (b ) arg min
in f ( y ) b ∈R , (23)
ing condition holds: y ∈R

βr > a . (17) is given by

In case that (17) is satisfied, the unique solu- ⎧0 if | b | ≤ η
tions of the strictly convex problems in (16) can be proxηφ ( ) = ⎨ ∗
(24)
computed based on the proximity maps introduced ⎩sign(b ) y if | b | > η ,
in Proposition 2 below. In particular, according to
Proposition 2, for a generic penalty function φ the where y∗ is the unique solution of the following
solutions of (16) can be obtained as follows: constrained nonlinear equation:

⎧
if | zik | ≤ 1/β r y∗ + ηφ
φ ′ ( y∗ ; a ) | b | = 0 , 0 < y∗ |b|. (25)
k +1 ⎪0
ri = ⎨ , (18)
⎪sign( z k ) r∗ if | zik | > 1/β r
⎩ i Moreover, in case that | b | > η , the Newton-
Raphson iterative scheme defined by:
where r∗ denotes the (unique) solution of the fol-
lowing constrained nonlinear equation: y0 b
y kφ y k ;;aa φ ′( y k ;aa ) b/η
1∗ y k +1 = , (26)
r + φ ′(r∗; a ) − | zik | = 0 , 0 < r∗ < | zik | . (19) φ ″( y k ;aa ) + 1 / η
βr

According to (26), the solution r∗ of (19) can be converges with quadratic rate to the unique solu-
obtained as the limit point of the following (quad- tion y∗ of the constrained nonlinear equation
ratically convergent) Newton-Raphson iteration: in (25).
Proof First, we notice that for any penalty
function φ satisfying assumptions A1)–A6), the
r 0 zik
proximity map proxηφ in (23) has the following
r kφ r k ;;a − φ ′(r k ;a ) + zik β r properties:
r k +1 = . (20)
φ ″(r k ;a ) + β r
proxηφ ( ) 0 proxηφ ( ) = − proxηφ ( ) ∀ b ∈ R , (27)
For what concerns computational efficiency of
(18)–(20), first we notice that iterations (20) has to which follow immediately from the definitions in
be carried out only for the subset of indexes i for (21) and (23). Since according to (27) proxηφ is a
which the quantity | k | is above the threshold odd function, it follows that:
in (18). Then, the quantity zik β r in the numera-
tor of (20) is constant during iterations and can proxηφ ( ) sign( ) proxηφ (| |)) b ∈R . (28)
be precomputed. Last but not the least, very few
iterations (20) are sufficient to compute accurate
estimates of r∗ , as it will be demonstrated in the Therefore, in the rest of the proof we focus on
experimental section. the case b ∈ R∗+ and then apply (28) to derive the
Proposition 1. Let φ : R → R be a function sat- expression of proxηφ for b ∈R .
isfying assumptions A1)–A6) and a,η ∈ R∗+, b ∈R For any a,η ∈ R∗+ satisfying condition (22) and
be given constants. Then, the function f : R → R any b ∈ R∗+ , the function f in (21) is strictly convex
defined as and, in particular, has the following properties:

90

VIPIMAGE15_Book.indb 90 9/8/2015 7:43:18 PM

 ⎧
⎪ f ′ y) = y − b + η φ ′( y; a ) < 0 ∀ y ∈ R∗− In our case, condition (31) is satisfied. In fact, K
⎪ represents a blurring operator, which is a low-pass
⎪⎪ f ′ − = − b −η < 0 filter, whereas the regularization matrix D is a finite
⎨
⎪f′ + = − b +η ≥ 0 ∀ b ∈ ]0,η ] difference operator and, hence, is a high-pass filter.
⎪
⎪ In case that (31) is satisfied, the coefficient
⎪⎩ f ′′( y ) = 1 + η φ ″( y ) >0 ∀ y ∈ R∗+ matrix in (30) is symmetric positive definite and,
(29) typically, is highly sparse. Hence, the linear sys-
tem in (30) can be solved quite efficiently by the
Hence, in case that b ∈ ] 0, ] , the function f has iterative (preconditioned) Conjugate Gradient
its unique global minimizer at y = 0, so that (28) method. Moreover, under appropriate assump-
allows to conclude that the first part of statement tions about the solution u k +1 near the image
(24) holds. boundary, the linear system can be solved even
In case that b ∈ ] ,+∞ , ], all the properties in more efficiently. For example, under periodic,
(29) hold true but the third one which becomes reflective or anti-reflective boundary conditions
f ′( + ) < 0 . Hence, in this case the global minimizer for u k +1, the coefficient matrix in (30) can be diag-
of function f coincides with its unique (positive) onalized by the 2D discrete Fourier transform
stationary point, thus concluding the proof of (24). (FFT implementation), 2D discrete cosine trans-
form (FCT implementation) or 2D discrete sine
3.3 Solving the sub-problem for u transform (FST implementation), respectively.
Provided that the penalty parameters βt and β r
The sub-problem for u in (8) is a quadratic minimi- are kept fixed for all the ADMM iterations, the
zation problem, whose first-order optimality con- coefficient matrix in (30) does not change and it
ditions lead to the following linear system: can be diagonalized once for all at the beginning.
Therefore, at each ADMM iteration the linear
βr T
Au k = b k , with A = DT D + K K, system (30) can be solved by one forward FFT/
βt FCT/FST and one inverse FFT/FCT/FST, each
⎛ λtk ⎞ β r T ⎛ k +1 λrk ⎞ at a cost of O(d 2 log d ) .
bk DT ⎜ t k +1 K ⎜r − + g⎟ (30)
⎝ βt ⎠ βt ⎝ βr ⎠
3.4 ADMM-based minimization algorithm
2 2
The d d linear system in (30) is solvable if To summarize previous results, in Alg. 1 we report the
the coefficient matrix has full-rank, that is if the main computational steps of the proposed ADMM-
following condition holds: based iterative scheme used to solve the nonconvex
nonsmooth minimization problem in (2).
T
Ker( ) ∩ Ker(
Ker( T
) = { }, (31)

where Ker (M ) denotes the null space of matrix 4 EXPERIMENTAL RESULTS

M and 0 indicates the d 2 -dimensional null vector.
We present some preliminary experiments assess-
ing the performance of the proposed nonsmooth
nonconvex model (2), referred to as φ -TV, when
applied to the restoration of the and
test images depicted in Figure 2(a) and
Figure 2(e), respectively. The images are syntheti-
cally corrupted by Gaussian blur and impulse
noise of both salt & pepper ( ) and random-
valued ( ) type with different noise levels. We
recall that the level of an impulse noise corruption,
expressed in percent, represents the fraction of
image pixels which are affected by the noise.
To provide evidence of the advantages of using
nonconvex versus convex fidelity terms, we com-
pare the proposed φ -TV model with the popular
nonsmooth but convex 1-TV model defined by
(2) with φ ( ) =| x |, see (Yang, Zhang, & Yin 2009).
Accuracy of the obtained restored images u∗ is
evaluated by the Improved Signal-to-Noise Ratio
(ISNR) defined as

91

VIPIMAGE15_Book.indb 91 9/8/2015 7:43:23 PM

 Figure 2. Restoration results for brain (top row) and cameraman (bottom row) test images.

|| g u ||22 Table 2. ISNR values obtained by φ-TV on the

ISNR
S (u∗ g u ) = 10 log10 dB , image.
|| u∗ u ||22
50% 50%
where u denotes the uncorrupted image.
For both the compared models and for all its φlog φrat φatan φlog φrat φatan
the examples, we used the iterative minimiza-
tion ADMM scheme reported in Section 3 with 1 25.68 25.69 25.78 17.40 17.43 17.48
5 25.67 25.70 25.79 17.40 17.44 17.48
the same parameter values βt = 1/100, β r = 100,
γ = 1.6, and taking as restored image u∗ the iter-
ate u k obtained as soon as the following stopping
condition is satisfied:
r variable sub-problem. We notice that the penalty
k k −1 k −1 −5 function φatan provides slightly better results than
|| u u ||2 || u ||2 < 10 . φ log and φrat and that performing only one iteration
of the Newton-Raphson scheme in (20) does not
For all the experiments, the regularization affect the accuracy of the restorations.
parameter μ is tuned independently for the φ -TV In Table 3 and Table 4 we report the ISNR val-
and the 1-TV models so as to get the highest pos- ues obtained by 1-TV and φ -TV models on the
sible ISNR value of the restored image. and test images corrupted
In our φ -TV model, we used a value a = 10 of by Gaussian blur ( = 11, = 2.5) and
the concavity parameter, such that condition (17) impulse noise of increasing levels 30%, 50%, 70%.
is satisfied and the sub-problem for the variable The benefits of using the nonconvex fidelity term
r is convex. Moreover, in the regularization term in the proposed φ -TV model with respect to the
of our φ -TV model we used the penalty function convex fidelity term of the 1-TV model are well
φatan defined in Table 1 and we performed only represented by the results in Tables 3–4, where the
one iteration of the Newton-Raphson scheme in highest ISNR values are marked in boldface. This
(20) for the solution of the variable r sub-problem. is confirmed by the restored images reported in
To support these choices, in Table 2 we report the Figure 2. The nonconvex data-fitting allows us to
ISNR values obtained by running our φ -TV model weight more the deblurring term, which leads to
on the test image corrupted by Gaussian better quality solutions.
blur and impulse noise of level 50%, with differ- These preliminary experiments strongly indi-
ent penalty functions φ and different number of cate that using nonconvex versus convex fidelities
Newton-Raphson iterations for the solution of the allows for more accurate restorations at a modest

92

VIPIMAGE15_Book.indb 92 9/8/2015 7:43:27 PM

 Table 3. ISNR values obtained for the test image. particular, results of the experimental comparison
with the 1-TV model strongly indicate that using
30% 50% 70% nonconvex versus convex fidelity terms holds the
potential for more accurate restorations at a mod-
est computational extra-cost.
1-TV 23.71 18.80 22.36 12.90 17.93 9.04 In future work we will investigate the use of
φ-TV 27.68 23.52 25.78 17.48 18.67 10.43 sparsity-promoting penalty functions for both the
fidelity term and the regularization term.

Table 4. ISNR values obtained for the test REFERENCES

image.
Boyd, S., N. Parikh, E. Chu, B. Peleato, & J. Eckstein
30% 50% 70% (2011). Distributed optimization and statistical learn-
ing via the alternating direction method of multipli-
ers. Foundations and Trends in Machine Learning 3(1),
1–122.
1-TV 20.13 13.77 17.90 11.39 15.99 8.70 Chan, R., Y. Dong, & M. Hintermuller (2010, July). An
φ-TV 25.00 19.13 19.27 14.50 17.40 10.17 efficient two-phase l1-tv method for restoring blurred
images with impulse noise. Image Processing, IEEE
Transactions on 19(7), 1731–1739.
Chan, R., M. Tao, & X. Yuan (2013). Constrained total
computational extra-cost which can be approxi- variational deblurring models and fast algorithms
mately evaluated as a 30% overhead. based on alternating direction method of multipliers.
SIAM J. Imag. Sci. 6, 680–697.
Chen, P.-Y. & I. Selesnick (2014, July). Group-sparse
signal denoising: Non-convex regularization, convex
5 CONCLUSIONS optimization. Signal Processing, IEEE Transactions
on 62(13), 3464–3478.
In this paper, we presented a nonsmooth noncon- Dong, Y., R. Chan, & S. Xu (2007, April). A detection
vex variational model for non-blind deblurring statistic for random-valued impulse noise. Image
of images corrupted by impulse noise of both Processing, IEEE Transactions on 16(4), 1112–1120.
salt-and-pepper and random-valued type. In the Lanza, A., S. Morigi, & F. Sgallari (2015). Convex image
model, a classical TV regularization term is cou- denoising via non-convex regularization. In Scale
Space and Variational Methods in Computer Vision,
pled with a nonsmooth nonconvex fidelity term Volume 9087, pp. 666–677.
which allows to promote sparsity of the residual Nikolova, M. (2004). A variational approach to remove
image more strongly than the convex 1-norm outliers and impulse noise. Journal of Mathematical
fidelity commonly used for this kind of noise. Dif- Imaging and Vision 20(1–2), 99–120.
ficulties arising from non-smoothness and non- Wang, Y., J. Yang, W. Yin, & Y. Zhang (2008). A new
convexity of the proposed model are dealt with alternating minimization algorithm for total varia-
by an efficient minimization algorithm based on a tion image reconstruction. SIAM J. IMAGING SCI,
suitable variant of the popular ADMM procedure 248–272.
which allows to split the nonconvex problem into a Yang, J., Y. Zhang, & W. Yin (2009). An efficient tvl1
algorithm for deblurring multichannel images cor-
sequence of strictly convex sub-problems. rupted by impulsive noise. SIAM Journal on Scientific
Some preliminary experiments demonstrate Computing 31(4), 2842–2865.
the effectiveness of the proposed approach. In

93

VIPIMAGE15_Book.indb 93 9/8/2015 7:43:30 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Classification-based blood vessel segmentation in retinal images

J. Odstrcilik, R. Kolar & V. Harabis

St. Anne’s University Hospital—International Clinical Research Center (ICRC), Brno, Czech Republic
Department of Biomedical Engineering, Faculty of Electrical Engineering and Communication,
Brno University of Technology, Brno, Czech Republic

R.P. Tornow
University of Erlangen–Nuremberg, Department of Ophthalmology, Erlangen, Germany

ABSTRACT: Automatic and precise segmentation of retinal blood vessels can help in computer aided
assessment of various retinal diseases, especially diseases related to cardiovascular system or glaucoma.
Hence, an accurate detection of retinal vascular structures is one of the most addressed topic in the field
of retinal image processing today. Most of the methods are designed directly for utilization with a special
dataset only or have problems to segment blurry and noisy images. In this study, we showed that com-
bination of three standard approaches—matched filtering, Hessian-based approach, and morphological
processing, together with support vector machine classification technique, can be satisfactorily used as a
universal segmentation approach in retinal images acquired using entirely different devices.

1 INTRODUCTION different kinds of diagnostically important param-

eters, e.g. for glaucoma diagnosis (Tornow 2015 &
An accurate segmentation of retinal vasculature Odstrcilik 2014).
is a necessary step preceding higher-level analysis
of fundus images. It can be consequently used for
detection of various diseases, especially diseases 2 IMAGE DATASETS
related to cardiovascular system (Bhuiyan 2014)
or glaucoma (Wang 2007). Other goals to segment Two datasets of standard static color fundus
vasculatures in retinal images can be related to bio- images and one dataset of dynamic low-quality
metric applications as well (Lajevardi 2013) or as video-sequences with different resolution and dif-
a preprocessing step for image registration (Kolar ferent Field of Views (FOV) were used for devel-
2013, Kolar 2013). Therefore, segmentation of opment and testing of the proposed methodology:
retinal vessels is one of the most addressed topic in A. HRF (high-resolution fundus) database (color
retinal image processing today (Annunziata 2015, images, size of 3504 × 2336 px, 60-degree FOV), B.
Rathod 2015, Cheng 2014, Hannink 2014, Budai local patient fundus image dataset (color images,
2013, Yu 2012 & Fraz 2012). Most of the meth- size of 2048 × 1536 px, 45-degree FOV) acquired in
ods are designed directly for one dataset or have St. Anne’s Faculty Hospital, Brno, Czech Repub-
problems to accurately segment blurry and noisy lic, and C. dataset of experimental dynamic fun-
images. dus video-sequences (gray-level video format, size
In this study, we showed that combination of 640 × 480 px, 15-degree FOV, framerate: 11
of three standard and computationally simple to 39 fps) acquired at collaborating ophthalmol-
approaches—Matched Filtering (MF), Hessian- ogy clinic at University of Erlangen, Erlangen-
based Approach (HA), and Morphological Nuremberg, Germany. Figure 1 shows respective
Processing (MP), together with Support Vector examples of images from particular datasets.
Machine (SVM) classification technique, can
be used successfully as a universal segmentation
approach in different kinds of retinal images with 3 METHODS
various resolution and image quality. Especially, a
proper segmentation of vasculature in retinal vid- Three methods, namely Matched Filtering (MF),
eo-sequences, acquired non-mydriatically under Hessian-based Approach (HA), and Morphologi-
the low illumination conditions, can be highly cal Processing (MP) were used for generation of
useful as a preprocessing step for extraction of image features to segment blood vessels. Supervised

95

VIPIMAGE15_Book.indb 95 9/8/2015 7:43:31 PM

 respects to the appearance of typical blood vessel
segments of different width (diameter) and orien-
tation. These kernels were created according to
measurement of numerous perpendicular cross-
sectional profiles of retinal vessels in standard
images of the HRF database. The cross-sectional
profiles were then heuristically classified into five
classes (indexed as k = 0, 1, …, 4) of different blood
vessel thicknesses to achieve a reliable and precise
detection of all possible blood vessel segments with
an acceptable width resolution. Particular kernels
were rotated into the angular direction ϕ = 0°,
15°, …, 165° in order to cover sufficiently all pos-
sible orientations of the blood vessel segments.
A square shape of the obtained masks is assumed.
Thus, a number of 12 differently oriented ker-
nels hk,ϕ for each of the width classes is obtained.
Furthermore, each kernel is convolved with the
input image I:

MFR
FRk ,ϕ I ∗ hk ,ϕ . (1)

In this way, a number of 60 (5 × 12) paramet-

ric images (MFRk,ϕ—matched filter responses) is
obtained, related to the corresponding width class
k and the orientation of blood vessel segment ϕ.
A joint parametric image OMFR is then computed
by selection of a maximum response from the set
of resulted parametric images in pixel-wise manner
(Fig. 2a):

OMFFR
k ,ϕ
{MFR
M R }. k (2)

Figure 1. Examples of images from a) HRF database,

local patient fundus image dataset, and c) one frame of a
low-quality video-sequence.

Support Vector Machine (SVM) was further utilized

for pixel-wise classification to get a binary image.
Figure 2. Example of parametric images obtained via
3.1 Feature extraction (a) MF, (b) HA, (c) MP approaches, and (d) resulted
binary representation of blood vessel tree classified via
3.1.1 Matched filtering SVM classifier. Particular images represent an output of
According to our previous methodology (Odstrci- processing of one frame of a low-quality video-sequence,
lik 2013), 2D filtration kernels were designed with which is shown in Figure 1c.

96

VIPIMAGE15_Book.indb 96 9/8/2015 7:43:31 PM

 The five kernels designed can be used for vessel from local patient fundus image database (dataset
segmentation in retinal images with high resolution. B), only the elements with R = 7 and R = 15 were
In case of low-quality video-sequences, the size of utilized. In case of low-resolution fundus video-
large and medium blood vessels (∼5–10 pixels) in sequences (dataset C), utilization of SE with R = 7
central and peripheral part of the images (frames) is sufficient.
is appropriately comparable with the thinnest The final parametric image OMP is then obtained
blood vessels in the images of HRF database. This in the same way as in case of MF via selection of
means that only the kernel k = 0 can be used for maximum response from the set of individual par-
vessel segmentation in studied video-sequences. ametric images in pixel-wise manner (Fig. 2c).

3.1.2 Hessian-based approach

3.2 Classification
The HA is used to segment blood vessels (Odstrcilik
2014). First, the input image I is smoothed by Support Vector Machines (SVMs) have become
Gaussian kernel to suppress the noise: very popular machine learning technique that are
used in many areas for classification as well as
f I ∗ gσ (3) regression, and other learning tasks (Chang 2011).
Standard version of C-SVM classifier with RBF
The size of kernel gσ is set to 22 × 22 px for (radial basis function) kernel was used to classify
images in datasets A and B and 9 × 9 px for dataset blood vessels (Fig. 2d). The value of the regulariza-
C. Standard deviation is set σ = 8 (datasets A, B) tion parameter C = 100 has been identified experi-
and σ = 3 (dataset C). Both parameters were iden- mentally via cross-validation. All other parameters
tified in a heuristic manner with respect to the size of the SVM classifier have been left with default
of blood vessels in images of particular datasets. values as proposed in (Chang 2011). The classifier
Since the blood vessels appear mainly as the valleys was trained on image features extracted from images
in the image (darker than background), their posi- of HRF database. A number of 498 square-shaped
tion can be determined using Hessian matrix. This (5 × 5 pixels) image regions of interest (ROIs) were
matrix of the second-order partial derivatives is manually extracted from regions of blood vessels
defined as follows: of different diameters in 5 randomly chosen images
from HRF database, which contains corresponding
⎛ ∂ xx fij ∂ xy fij ⎞ ground truth segmentations. Next, a number of 468
Hij = ⎜ , (4)
∂ yy fij ⎟⎠
ROIs were extracted from the image background.
⎝ ∂ yx fij For each ROI, the mean value of proposed features
was computed. Thus, a dataset of instances of two
where fij represents a grey-scale image and i, j are groups (“vessels” and “background”) was estab-
spatial coordinates along the axes x and y. Since lished, which can be used for training the SVM
∂ yx fij = ∂ xy fij , Hessian Hij has two real eigenval- classifier. Figure 3 shows a depiction of instances
ues λ1,2. Hij and corresponding eigenvalues λ1,2 in three-dimensional parametric space. It can be
are computed in a pixel-wise manner using Sobel
difference kernel. Then, only the higher positive
eigenvalue representing image valleys is taken into
account. In this way, a parametric image OHA is
computed (Fig. 2b).

3.1.3 Morphological processing

The third feature to segment blood vessels is
obtained via morphological bottom-hat filtering
of the grayscale input image I with predefined
structuring element (SE):

Bhat ( I ) = (I
(I S
SE ) I , (5)

where the operation • represents morphological

closing (Soille 2010). Three disk-shaped structur-
ing elements with radiuses R = 7, 15, and 30 were
defined to segment blood vessels in images from
HRF database (dataset A). The elements’ size
was estimated to best fit the diameter of blood Figure 3. A depiction of instances (green—vessels, red—
vessels in these high-resolution images. For data background) in three-dimensional parametric space.

97

VIPIMAGE15_Book.indb 97 9/8/2015 7:43:36 PM

 noticed that both groups can be quite well sepa-
rated. Occasional outliers of background instances
are most probably caused by noise or artefacts in
parametric images.
Finally (after the classification), morphologi-
cal cleaning is applied by deleting non-connected
objects with size less than ∼ 500 and 1000 px in case
of datasets A and B, and dataset C, respectively.

4 RESULTS AND DISCUSSION

4.1 Classification performance

In the first step, the classification performance
was evaluated via the repeated random sub-sam-
pling cross-validation procedure using extracted
instances of vessels and background. Seventy and
thirty percent of randomly selected instances were
used for training and testing the classifier in each
run of cross-validation, respectively. The repetition
of random sub-sampling procedure was set to 100.
The results of cross-validation procedure (Table 1)
signalize rather high performance of the method
in term of sensitivity (SE), specificity (SP), and
classification accuracy (ACC), which corresponds
with visual findings in Figure 3. The improvement
was also achieved in comparison with our previous
approach for vessel segmentation in low-quality Figure 4. a) Example of one segmentation result of
video-sequences (Odstrcilik 2014). All quantitative image from HRF database and b) corresponding ground
results of cross-validation (SE, SP, and ACC) are truth segmentation. The image corresponds with the
a bit higher in case of the proposed method (by image in Figure 1a.
2–5%).

4.2 Segmentation of static fundus images ground truth segmentations are available, quanti-
tative evaluation of the proposed method has been
In the second step, the proposed method was tested
performed via computation of SE, SP, and ACC
on static fundus images in HRF database—dataset
parameters (Table 2). Values of other authors are
A (Fig. 4). The HRF database contains three sets
also presented to compare the proposed approach
of fundus images: one of healthy retinas, one of
with recent state-of-the-art methods, which were
glaucomatous, and one of Diabetic Retinopathy
evaluated on HRF dataset. It is shown the proposed
(DR) retinas. Each set is composed of 15 color
method is well comparable with other methods.
fundus images with corresponding gold standard
Next, the proposed approach was tested on
segmentation made by a human observer. Since
static patient fundus image dataset—dataset B.
(Fig. 5). The dataset contains 45 images of patients
Table 1. Classification performance. with various diseases—DR, glaucoma, and mac-
ular degeneration. In this case, the method was
Evaluation parameters Average results evaluated only qualitatively, since no gold standard
segmentation is available. The original method was
SE = TP/(TP + FN) 96.86 ± 1.66% developed especially for images of HRF database.
SP = TN/(TN + FP) 98.76 ± 0.89% However, the results revealed the utilization for
ACC = (TP + TN)/ 97.78 ± 0.81% other dataset is possible.
(TP + FN + TN + FP)

TP—number of ROIs correctly classified as the 4.3 Segmentation of dynamic video-sequences

blood vessels, FN—number of ROIs incorrectly
classified as the background, TN—number of ROIs In the third step, the proposed method was tested
correctly detected as the background, FP—number to segment blood vessels on dataset of dynamic
of ROIs incorrectly classified as the blood vessels. low-quality video-sequences acquired by an

98

VIPIMAGE15_Book.indb 98 9/8/2015 7:43:37 PM

 Table 2. Evaluation of the method on HRF database. Table 3. Evaluation of the method using dynamic low-
quality fundus video-sequences. A mean value along with
SE SP ACC standard deviation of Avessel is provided.

Proposed H 0.7887 0.9931 0.9581 Video

method, G 0.7982 0.9850 0.9554 Sequence frame Number of Acquisition
2015 DR 0.7588 0.9480 0.9554 no. rate [fps] frames [–] time [s] Avessel [%]
Budai et al., H 0.6620 0.9920 0.9610
2013 G 0.6870 0.9860 0.9650 1 12 189 16 2.96 ± 0.90
DR 0.6580 0.9770 0.9550 2 39 836 21 2.89 ± 0.25
Odstrcilik H 0.7860 0.9750 0.9530 3 12 191 16 6.49 ± 0.77
et al., 2013 G 0.7900 0.9640 0.9490 4 12 189 16 0.41 ± 0.37
DR 0.7460 0.9610 0.9440 5 24 376 16 1.51 ± 0.26
Annunziata H 0.6820 0.9935 0.9587 6 35 878 25 2.47 ± 0.33
et al., 2015 G 0.7566 0.9785 0.9603 7 11 189 17 2.50 ± 0.35
DR 0.6997 0.9787 0.9554 8 21 403 19 6.69 ± 0.88
Cheng et al., H 0.7889 0.9865 0.9647 9 12 189 16 1.14 ± 0.28
2014 G 0.6921 0.9875 0.9634 10 22 389 18 4.04 ± 0.69
DR 0.6313 0.9851 0.9561 11 20 838 42 0.76 ± 0.11
Yu et al., H 0.7938 0.9767 0.9566 12 12 189 16 1.38 ± 0.64
2012 G 0.7890 0.9662 0.9518 13 24 376 16 2.24 ± 0.47
DR 0.7604 0.9625 0.9460 14 19 964 51 2.37 ± 0.53
H—dataset of healthy images, G—dataset of glauco-
15 20 400 20 2.37 ± 0.53
matous images, DR—dataset of diabetic retinopathy
images.

Figure 6. Variations of the area covered by blood ves-

Figure 5. Example of segmentation result of one image sels expressed by statistical boxplot diagrams. On each
of patient with DR from local patient fundus image data- box, the central mark is the median, the edges of the box
set. The image corresponds with the image in Figure 1b. are the 25th and 75th percentiles, and the whiskers extend
to the most extreme data points not considered outliers.
Outliers are then plotted individually by ‘+’ symbols.
experimental video ophthalmoscope—dataset C.
The dataset contains 15 video-sequences of vari-
ous acquisition time ∼ 16–50 seconds. Since the The results in Table 3 and Figure 6 show rather
experimental dataset has no ground truth segmen- high stability of the segmentation across the
tation, evaluation of the proposed methodology sequences (σ < 1% in all cases). The area varia-
was performed in a heuristic manner: the area tion as well as outliers (Fig. 6) can be caused by
covered by segmented blood vessels (Avessel [%]) was inter-frame variability of the image quality across
computed for each frame in the sequence and its the sequence as well as by fast eye movements of
variation over the entire sequence have been inves- the subject during data acquisition, which finally
tigated (Table 3). Figure 6 shows variations of results in blurred frames. Comparing the results
Avessel for each sequence in a dataset via statistical with earlier approach (Odstrcilik 2014), when
boxplot diagrams. Avessel fluctuated more (σ ∼ 1–3%), these of the new

99

VIPIMAGE15_Book.indb 99 9/8/2015 7:43:37 PM

 proposed technique reveal the method is more sta- Budai A., R. Bock, A. Maier, et al., “Robust Vessel Seg-
ble and robust. As can been seen from an example mentation in Fundus Images,” International Journal
of resulted segmentation of one frame of a video- of Biomedical Imaging, vol. 2013, pp. 11 (2013).
sequence in Figure 2d, mainly the blood vessels Chang C.C., C.J. Lin, “LIBSVM: a library for support
vector machines,” ACM Trans Int Sys Tech, vol. 27,
inside the Optic Disc (OD) region are segmented. no. 2, pp. 31–39 (2011).
It is because these vessels are more contrasted in Cheng E., L. Du, Y. Wu, et al., “Discriminative vessel seg-
our low-quality video-sequences. However, this mentation in retinal images by fusing context-aware
can be further utilized as an advantage for further hybrid features,” Machine Vision and Applications,
processing towards evaluation of cardiac cycle vol. 25, no. 7, pp. 1779–1792 (2014).
induced changes in fundus reflection in the OD Fraz M.M., P. Remagnino, A. Hoppe, “Blood vessel
region (Tornow 2015). segmentation methodologies in retinal images—A
Survey,” Computer methods and Programs in Bio-
medicine, vol. 108, no. 1, pp. 407–433 (2012).
Hannink J., R. Duits, E. Bekkers, “Crossing-preserving
5 CONCLUSIONS multi-scale vesselness,” Medical Image Computing
and Computer-Assisted Intervention—MICCAI
The paper presents an ongoing work on the blood 2014, vol. 8674, pp. 603–610 (2014).
vessel segmentation, which can be used univer- Kolar R., B. Hoeher, J. Odstrcilik et al., “Registration of
sally for different types of fundus image data. Image Sequences from Experimental Low-Cost Fun-
The experiments showed that the fusion of three dus Camera,” Lecture Notes in Computer Science,
known approaches—MF, HA, and MP can be Proc. of Workshop on Biomedical Image Registra-
used successfully for accurate and precise detection tion, pp. 174–183 (2014).
of blood vessels in combination with supervised Kolar R., V. Harabis, J. Odstrcilik, “Hybrid retinal image
registration using phase correlation,” Imaging Science
classification technique given by SVM classifier. It Journal, vol. 61, no. 4, pp. 269–384 (2013).
has been also shown that the SVM model trained Lajevardi S.M., A. Arakala, S.A. Davis, et al., “Retina
on standard database can be used for segmentation Verification System Based on Biometric Graph Match-
of blood vessels in completely different data with ing Image Processing,” IEEE Transactions on Image
different image quality (with different resolution Processing, vol. 22, no. 9, pp. 3625–3635 (2013).
and FOV). Especially, the method reached good Odstrcilik J., R. Kolar, J. Jan, R.P. Tornow, et al., “Blood
results in case of experimental and noisy fundus vessel segmentation in video-sequences from the
video-sequences with low resolution. For further human retina,” IEEE International Conference on
development, testing on other datasets and possi- Imaging Systems and Techniques (IST 2014), pp.
129–133 (2014).
bly combination with other types of features can Odstrcilik J., R. Kolar, T. Kubena, et al., “Retinal vessel
be considered. In addition, also enhancement and segmentation by improved matched filtering: evalua-
restoration techniques for preprocessing of low- tion on a new high resolution fundus image database,”
intensity and noisy video-sequence data could be IET Image Process., vol. 7, no. 4, pp. 373–383 (2013).
utilized to improve the segmentation. Rathod D., R.R. Manza, Y.M. Rajput, “Retinal Blood
Vessels Extraction using Matched Filter on High
Resolution Fundus Image Database,” International
ACKNOWLEDGEMENT Journal of Computer Applications (2015).
Soille P., “Morphological image analysis: principles and
applications,” Springer, p. 392 (2010).
This work has been supported by European Tornow R.P., R. Kolar, J. Odstrcilik, “Non-mydriatic
Regional Development Fund—Project FNUSA- video ophthalmoscope to measure fast temporal
ICRC (No. CZ.1.05/1.1.00/02.0123). changes of the human retina,” Proc. SPIE 9540,
Novel Biophotonics Techniques and Applications III
(2015).
REFERENCES Wang S., L. Xu, Y. Wang, et al., “Retinal vessel diam-
eter in normal and glaucomatous eyes: the Beijing eye
Annunziata R., A. Garzelli, L. Ballerini, et al., “Lever- study,” Clin. Exp. Ophth., vol. 35, no. 9, pp. 800–807
aging Multiscale Hessian-Based Enhancement with a (2007).
Novel Exudate Inpainting Technique for Retinal Ves- Yu H., S. Barriga, C. Agurto, et al, “Fast vessel segmen-
sel Segmentation,” IEEE Journal of Biomedical and tation in retinal images using multiscale enhancement
Health Informatics, vol. 00, no. 00 (2015). and second-order local entropy,” Proc. SPIE 8315,
Bhuiyan A., R. Kawasaki, E. Lamoureux, et al., “Retinal Medical Imaging 2012: Computer-Aided Diagnosis,
Vascular Features for Cardio Vascular Disease Pre- 83151B (2012).
diction: Review,” Recent Patents Comput. Sci., vol. 3,
no. 3, pp. 164–175 (2014).

100

VIPIMAGE15_Book.indb 100 9/8/2015 7:43:38 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Line extraction via phase congruency with a novel adaptive

scale selection for Poisson noisy images

Vladimir A. Krylov
Department of Electrical, Electronic and Telecommunication Engineering and Naval Architecture,
University of Genoa, Genoa, Italy

James D.B. Nelson

Department of Statistical Science, University College London, London, UK

ABSTRACT: In this paper we address the problem of extracting curvilinear structures from images
in the presence of Poisson-distributed noise that is typically found in low-quality, low-contrast data, as
well as in numerous medical imaging modalities. A contrast-robust phase congruency estimation method
is proposed such that the optimal range of scales is selected in an adaptive manner for each location in
the image. The adaptive regime is driven by a statistically well-principled variance test statistic which is
based upon a local measure of the dispersion index. Experimental validation confirms that the adap-
tive scheme delivers superior line extraction results on mammographic imagery when compared to other
recent attempts including the non-adaptive phase congruency.

1 INTRODUCTION elements and propagating them in such a way as to

take into account the line-object interactions, e.g.
The extraction of lines and curvilinear structures in (Verdié and Lafarge 2014). At a steep compu-
(CLS) from noisy and cluttered imagery is one of tational complexity price, this allows one to take a
the fundamental problems in pattern recognition. far more adaptive and flexible approach which can
Myriad applications rely on accurate extraction of accommodate various CLS properties by suitable
lines or line-like features either as an end goal in design of prior energy terms. Other approaches
itself or as a means to support other image process- combine elements of various extraction techniques,
ing tasks. Curvilinear extraction can pose serious like isotropic non-linear filtering (Liu et al. 2007),
challenges when the analysed imagery is affected space-scale line profile analysis (Steger 1998),
by strong noise or when the image is cluttered by skeleton extraction (Jang and Hong 2002), local
objects that do not contain useful line features (Lee Radon maximization (Krylov and Nelson 2014).
and Kweon 1997). A particularly difficult line Specialized line detection techniques have been
detection problem arises when the imagery con- adopted in various application domains, such as
tains CLS at various scales and when these features remote sensing (Verdié and Lafarge 2014), medical
overlap. This can happen as the result of 3D-data imaging (Obara et al. 2012), biometrical applica-
projection, which is often the case of medical scans. tions (Huang et al. 2008). Often CLS detection is
These images typically contain multiple occlusion a necessary preprocessing stage to extract com-
and superposition artefacts, due to overlaps of fea- plex objects that are comprised of distinct com-
tures at different layers of tissue. binations of linear features, such as, e.g., spicule
Recent decades have seen much, and continued, patterns that often surround malignant masses
interest in the design of line and CLS detection in mammographic images (Sampat et al. 2008,
methods and various different line-models and Muralidhar et al. 2010).
assumptions have evolved. Notable multi-scale In this paper we focus on the scenario of images
techniques have been designed based on direction- affected by Poisson-distributed noise, whereas the
ally adaptive transforms such as steerable filters majority of existing state-of-the-art methods explic-
(Jacob and Unser 2004), contourlets (da Cunha itly or implicitly operate under the noise Gaus-
et al. 2006), feature-adapted beamlets (Berlemont sianity assumption. The Poisson distribution is
and Olivo-Marin 2010), and curvelets (Ma and characterized by one parameter, referred to as inten-
Plonka 2010). In contrast stochastic geometry sity, and both the distribution’s mean and variance
approaches proceed by randomly sampling line are equal to its value. The corresponding noising

101

VIPIMAGE15_Book.indb 101 9/8/2015 7:43:38 PM

 model is particularly appropriate for discretized, The rest of the paper is arranged as follows. In
positive-value data and characterizes several widely Section 2 we recall the employed concepts of phase
used image modalities, including low-light and low- congruency and Gabor filtering. In Section 3 we
quality sensor data, astronomy imagery, and many design a novel scale selection approach for the
kinds of medical imagery (Gravel et al. 2004), such Poisson-noisy imagery. In Section 4 we report exper-
as, mammographic, X-ray, PET, SPECT, fluores- iments, and in Section 5 we draw the conclusions.
cent confocal microscopy imaging, etc.
Along with the degradation of the analysed
imagery by the Poisson noise, we also assume the 2 PHASE CONGRUENCY
presence of CLS at different scales, with possible AND GABOR FILTERING
occlusions and superpositions. In order to address
the line extraction on such imagery, we will assume Phase congruency exploits the expected behaviour
that the CLS of interest manifest as image ridges, of the image line- and edge-like features in the fre-
i.e. smooth elongated curves, whose points are quency domain (Kovesi 2003). In particular, the fre-
local maxima in at least one direction. Typically, quency components of such features share similar
these maxima are identified by resorting to local phases—e.g., observe the behaviour of the square
gradient information (Canny 1986, Lindeberg wave frequency components at points { } cor-
1998). In this work we will, instead, employ the responding to the edges in Figure 1(a). The phase
phase congruency characteristic, which evaluates congruency concept disregards the amplitudes of
the arrangement of phases of the signal at dif- the frequency components and, similarly to coher-
ferent frequencies and has an advantage of being ence, analyses solely the phase information.
stable under translation, geometric deformations The phase congruency measurement procedure
and contrast variations (Kovesi 2003). This advan- at a signal point is exemplified geometrically in
tage is of significance for various types of imagery, Figure 1(b). The Fourier components in the ana-
especially in case when some of the relevant CLS lysed signal have amplitudes Cn and phase angles
demonstrate very weak contrast. In order to extract γn; the magnitude of the sum vector is the local
the phases, we will employ the directional Gabor energy E. The phase congruency at point z is clas-
filters, see (Kovesi 2003). sically defined as PC ( z ) = | E ( z ) | ∑ n Cn ( z ). In this
Since we assume that the imagery is inherently paper we will employ a slightly different definition
multiscale, the linear features can be superimposed of phase congruency that allows a better localiza-
one on top of the other. As a result, this can signifi- tion (Kovesi 2003):
cantly interfere with the phase congruency estima-
tion which requires non-cluttered line features. To
tackle this problem we adaptively identify the opti- PC =
∑ n Cn ( z ) max[0,α n ( z )] , (1)
mal range of scales. Unlike (Schenk and Brady 2003), ∑ n Cn ( z )
we propose an automatic adaptive scheme that is
statistically well-founded in that it is based upon the where α n ( ) (γ n ( z ) − γ ( )) | i (γ n ( z ) − γ ( x ))| .
dispersion index—an oft-utilised goodness-of-fit Instead of resorting to the Fourier transform, the
measure in statistical estimation of Poisson distri- local frequency information is obtained via banks
butions (Karlis and Xekalaki 2000). The dispersion
index allows to evaluate the degree of homogeneity
of the Poisson-noisy image, and, thus, helps to iden-
tify the scale at which the local Gabor filter encoun-
ters an edge. Note that the employed adaptive scale
selection approach is consistent with the scale-space
edge concept defined in (Lindeberg 1998), accord-
ing to which, the features represent connected sets of
points characterized by some measure that persists
over a consecutive range of scales.
The contribution of this work lies in the design
of dispersion index-based automatic algorithm for
adaptive optimal scales identification to extract
CLS using phase congruency on Poisson-noisy
images. The experimental validation is performed Figure 1. (a) First four components in the Fourier series
on mammographic images, where CLS extrac- of a square wave. (b) Fourier components of a signal
tion supports an array of important tasks such as plotted as a sum: Cn, γn are the components’ amplitudes
coregistration or inference regarding tissue malfor- and phase angles, γ–—weighted mean phase, and E—the
mation or pathology (Muralidhar et al. 2010). total magnitude.

102

VIPIMAGE15_Book.indb 102 9/8/2015 7:43:38 PM

 of Gabor filters tuned to different spatial frequen- 3 ADAPTIVE SCALE SELECTION
cies (Kovesi 2003).
Gabor filters have received considerable atten- Consider imagery affected by Poisson noise. Unlike
tion because they have an analytical expression in additive Gaussian noise (which is the limit case for
both time and frequency, are easy to implement, Poisson noise (Lehmann and Romano 2005)), it is
and are highly tunable. It has also been argued that signal-dependent. This can make separating sig-
the filters resemble the behaviour of certain cells in nal from noise a difficult task. Assume that some
the visual cortex of mammals (Serre et al. 2005). image area depicts a constant signal affected by
We use these filters here as a means to estimate the noise—this means that the data will be Poisson-
time-frequency coefficients necessary for the esti- distributed with intensity λ equal to the pure signal
mation of phase congruency. Other wavelet bases intensity (Gravel et al. 2004). The purpose of adap-
could be used in their place instead, however the tive scale selection is at every image point to iden-
choice of wavelet is beyond the scope of this work. tify the resolution at which this point is no longer
A Gabor filter is defined as providing an impulse contained inside a homogeneous area. In other
response equal to a harmonic function multiplied words, to identify the minimum scale s* where
by a Gaussian function. In the frequency domain the neighbourhood of the given pixel includes the
its filtering function can be written as Poisson-distributed values with distinct intensity
parameters, i.e., multiple sample populations are
1 ⎧ ( 0)
2
( 0)
2⎫ represented in this neighbourhood characterised
Ψ( )= exp ⎨− − ⎬, by distinct Poisson distributions.
2πσ uσ v ⎩ 2σ u
2
2σ v ⎭
2
To estimate s* we employ the dispersion index,
which is a widely used tool to construct goodness-
with σ u 1//(2ππσσ x ), σ u = 1//(2πσ x ), and σx, σ y—the of-fit tests and analyse the homogeneity property
standard deviations in the time domain. The Gabor of the Poisson distribution (Karlis and Xekalaki
filter can be thought of as a Gaussian function 2000). The dispersion index, also called variance-
shifted in frequency
q to position ( 0 0 ) at orienta- to-mean ratio, is defined as D = σ 2 /μ , with vari-
tion tan −1 ( ), see Figure 2. In this paper the Gabor
filters are convolved with the signal at S distinct
ance σ 2 and mean value μ. It is immediate, that
the Poisson distribution is characterised by D ≡ 1.
scales and O uniformly spread orientations. In order to employ the dispersion index to test the
The Gabor filters have been successfully used homogeneity of a sample of N random observa-
for various image processing applications (Kovesi tions { i }iN=1, we employ the Variance Test (VT)
2003, Serre et al. 2005, Teuner et al. 1995). In par- statistic (Karlis and Xekalaki 2000) defined as
ticular, in case of Poisson-noisy images in (Buciu
and Gacsadi 2011) the Gabor wavelets were
∑ i =1 (X i
N
X )2 N
reported as an efficient tool for directional feature VT = − , (2)
analysis for mammographic imagery. Nevertheless, X 2N 2
one of the conclusions in the study was that high
frequency image components are rather sensitive with the sample mean X ∑ iN=1 X i /N . This sta-
to the Poisson noise. tistic constitutes the locally most powerful unbi-
ased test for the Poisson homogeneity hypothesis
{ 0 λ1 λN } against the mixed Poisson alter-
native { A λ λm }. It is imme-
diate that this alternative hypothesis definition
corresponds exactly to the specified above region
homogeneity specification. The value of VT is a
slight modification of the dispersion index, which
is obtained by introducing a correction with respect
to the sample size N. The values of the VT statistic
are standard normal distributed (Karlis and Xeka-
laki 2000) under H0.
In this paper we employ the Gabor filters to
evaluate the phase congruency, and, therefore, we
will use the VT on the samples originating from
inside the ellipses constructed according to the
Figure 2. Gabor filters in the frequency domain. Each levels of filter responses in square magnitude—
of the ellipses represents the range of the corresponding from 0.5 to 1, see Figure 2. Thus, for every scale
filter response in the squared magnitude values from 0.5 s = 1, …, S and orientation o = 1, …, O we retrieve
to 1.0. a data sample of size Nso. For each of these samples

103

VIPIMAGE15_Book.indb 103 9/8/2015 7:43:39 PM

 we calculate the corresponding value of the sample enough (O > 8), the former is selected with respect
dispersion index VT Tso . Our target is to select the to the CLS present, but in the general case should
optimal scale s* such that the deviation of the VT Tso not be small. The scale range parameter Δs was set
from the standard normal distribution is meaning- equal to 3.
ful. At every scale we calculate O distinct values of The results on three typical mammographic
the variance test and then average over the differ- regions of interest are demonstrated in Figure 3.
ent orientations because (i) we do not aim to iden- Note that except for thresholding by the mean
tify the optimal orientation in this procedure, and PC value for every set of pixels extracted at the
(ii) the averaging improves robustness with respect same scale s*, we report the raw results, without
to the signal fluctuations. Summarizing the above, resorting to image morphology tools, that can
the optimal scale can be identified as improve the visual quality, nor do we discard any
scales, which is an option if the CLS are a priory
O expected at some particular scales. We observe that
s∗ =
s ,…,S
∑ | VTTso | . the scale maps are informative on their own and
o =1 contain the CLS, which are visible as regions with
the adaptive scale different from the surrounding
In the experimental study we do not assign sta- background’s scale. By construction the designed
tistical test meaning to the VT values, due to a high method is local and can be successfully applied
variability of confidence levels obtained with the to non-stationary imagery that contains regions
test. This observed variability can be explained characterized by distinct statistical properties. The
by the employed two simplifying assumptions: (i) scale maps (after appropriate smoothing) can be
pixel independence and (ii) constant-intensity sig- employed as an indicator to distinguish between
nal in the homogeneous zones. Nevertheless, the different layers of the low-fibroglandularity tissue
experiments suggest that at a certain scale a sub- on the mammograms.
stantial increase of VT is typically observed, and The designed method demonstrates an ability to
we will use this behaviour as an indicator to obtain automatically adapt the scale such that, on the one
the scale estimates. Specifically, we will set hand, the blob in the top image in Figure 3 is sepa-
rated from some of the CLS but, on the other hand,
O O
the higher scale (red) line on the right half of the
s∗ =
s ,…,S
∑ | VTTso | ∑ | VTTs 1,o | . (3)
image is clearly distinguished from the neighbour-
o =1 o 1
o=
ing low scale CLS. Likewise, in the second image
After having identified for every pixel the the finer line structures are extracted against the
optimal scale s*, we perform phase congruency- slightly coarser scale content of the fibroglandular
based edge extraction as in Eq. (1) using scales tissue more clearly than by the other methods. The
{ min(
mi ( )}. It is worth noting that in order third mammogram demonstrates a high variety of
to benefit from adaptive scale selection, Δs should CLS at different scales and various levels of occlu-
be set with a small value, we suggest using Δs  4. sion in the tissue. The designed method performs
Once the features are extracted, it is necessary to well with higher scale (finer) features but confuses
pick the bright ridges from all the line- and edge- some low scale features.
like features. To do so we propose to multiply the In Figure 3 we present results obtained with
extracted PC values by the factor max 2 [ 0, 2γ /π ], three benchmark methods. Comparison between
where γ is the pixel weighted mean phase (corre- the standard phase congruency extractor (based on
sponding to the magnitude E, see Figure 1(b)). This global manual setting of scales) and the proposed
manipulation removes all the dark lines and edges method demonstrates a substantially stronger
(with −π / 2 γ  0 ), and penalizes the less distinct capacity of the latter to extract multiscale features.
non-symmetric bright line features (0 γ π /22). Comparing with the Radon-based enhancement
Finally, a hard thresholding could be employed to method (Sampat et al. 2008) reveals the contrast
obtain a binary CLS map. dependence of the latter and its weaker perform-
ance in modeling different scale features. As com-
pared with a primitive-based monoscale method
4 EXPERIMENTAL VALIDATION (Krylov and Nelson 2014), the proposed approach
produces noisier but also more CLS-sensitive
In this section we report experiments on mam- maps. The designed method can be used to reduce
mographic images from the Digital Database for the parametric initialization in (Krylov and Nelson
Screening Mammography (Heath et al. 2001). We 2014) and address the multiscale line-features.
have selected the number of scales S = 8 and con- In Figure 4 we demonstrate the performance
sider O = 16 orientations. The choice of the lat- of the scale selection based on s* at several image
ter parameter is not significant as long as it is big points and visualize the process of the final map

104

VIPIMAGE15_Book.indb 104 9/8/2015 7:43:41 PM

 Figure 3. CLS extraction on mammograms containing architectural distortions (with their centers marked by red
dots) and surrounded by CLS. The proposed method’s results are coloured according to the optimal scale at which
the CLS are extracted. The comparisons are with Radon-based method (Sampat et al. 2008) (Benchmark 1) and CLS
extractor (Krylov and Nelson 2014) (Benchmark 2).

test statistic based on the dispersion index value.

We thus formulate an application-independent
Poisson-noise specific technique, which in this
respect is unlike the majority of existing state-of-
the-art methods that typically target the Gaussian
noise. The proposed line detection method has
been experimentally validated and compared with
benchmark methods on mammographic images.
The principal direction for future development
is in designing of primitive-based multiscale CLS
extraction techniques that could benefit from the
Figure 4. Example of scale selection by the proposed pixel-based result and scale selection procedure
method and construction of line map from different scale designed in this study, e.g., by integration with
components. local Radon transforms (Krylov and Nelson 2014),
active contours (Muralidhar et al. 2010), etc.

construction as a sum of phase congruency com-

ponents extracted at adaptively estimated scales. It REFERENCES
can be seen how CLS of different scales contribute
Berlemont, S. & J.-C. Olivo-Marin (2010). Combining
to the final line-detection map. local filtering and multiscale analysis for edge, ridge,
and curvilinear objects detection. IEEE Trans. Image
Process. 19(1), 74–84.
5 CONCLUSIONS Buciu, I. & A. Gacsadi (2011). Directional features for
automatic tumor classification of mammogram
We have proposed a novel automatic approach images. Biomed Signal Process Control 6(4), 370–378.
to address the problem of multiscale CLS extrac- Canny, J. (1986). A computational approach to edge
detection. IEEE Trans. Pattern Anal. Mach. Intell.
tion from Poisson-noisy images. This approach PAMI-8(6), 679–698.
employs the phase congruency principle and per- da Cunha, A.L., J. Zhou, & M. Do (2006, Oct). The non-
forms the adaptive scale selection for the phase subsampled contourlet transform: Theory, design,
extraction. The latter is performed in a statisti- and applications. IEEE Trans. Image Process. 15(0),
cally well-founded way by employing the variation 3089–3101.

105

VIPIMAGE15_Book.indb 105 9/8/2015 7:43:43 PM

 Gravel, P., G. Beaudoin, & J.A. De Guise (2004). A Ma, J. & G. Plonka (2010). The curvelet transform. IEEE
method for modeling noise in medical images. IEEE Signal Process. Magazine 27(2), 118–133.
Trans. Med. Imaging 23(10), 1221–1232. Muralidhar, G., A. Bovik, J. Giese, M. Sampat, G.
Heath, M., K. Bowyer, D. Kopans, R. Moore, & Whitman, T. Haygood, T. Stephens, & M. Markey
W. Kegelmeyer (2001). The digital database for screen- (2010). Snakules: A model-based active contour algo-
ing mammography. In M. Yaffe (Ed.), Proc. of the Fifth rithm for the annotation of spicules on mammogra-
Int. Workshop on Digital Mammography, pp. 212–218. phy. IEEE Trans. Med. Imaging 29(10), 1768–1780.
Medical Physics Publ. available online at http://mara- Obara, B., M. Fricker, D. Gavaghan, & V. Grau (2012,
thon.csee.usf.edu/Mammography/Database.html. May). Contrast-independent curvilinear structure
Huang, D.-S., W. Jia, & D. Zhang (2008). Palmprint detection in biomedical images. IEEE Trans. Image
verification based on principal lines. Pattern Recogn. Process. 21(5), 2572–2581.
41(4), 1316–1328. Sampat, M., G. Whitman, A. Bovik, & M. Markey
Jacob, M. & M. Unser (2004). Design of steerable filters (2008). Comparison of algorithms to enhance spi-
for feature detection using Canny-like criteria. IEEE cules of spiculated masses on mammography. J. Digit.
Trans. Pattern Anal. Mach. Intell. 26(8), 1007–1019. Imaging 21(1), 9–17.
Jang, J.-H. & K.-S. Hong (2002). Detection of curvilinear Schenk, V.U.B. & M. Brady (2003). Improving phase-
structures and reconstruction of their regions in gray- congruency based feature detection through automatic
scale images. Pattern Recogn. 35(4), 807–824. scale-selection. In Progress in Pattern Recognition,
Karlis, D. & E. Xekalaki (2000). A simulation com- Speech and Image Analysis, pp. 121–128.
parison of several procedures for testing the Poisson Serre, T., L. Wolf, & T. Poggio (2005). Object recogni-
assumption. The Statistician 49(3), 355–382. tion with features inspired by visual cortex. In Proc.
Kovesi, P. (2003). Phase congruency detects corners and Comp. Vision Pattern Recogn (CVPR), Volume 2, pp.
edges. In Proc. Australian Pattern Recogn. Soc. Conf. 994–1000.
(DICTA). Steger, C. (1998). An unbiased detector of curvilinear
Krylov, V.A. & J.D.B. Nelson (2014). Stochastic extraction structures. IEEE Trans. Pattern Anal. Mach. Intell.
of elongated curvilinear structures with applications. 20(2), 113–125.
IEEE Trans. Image Process. 23(12), 5360–5373. Teuner, A., O. Pichler, & B.J. Hosticka (1995). Unsu-
Lee, J.-W. & I.-S. Kweon (1997). Extraction of line fea- pervised texture segmentation of images using tuned
tures in a noisy image. Pattern Recognition 30 (10), matched Gabor filters. IEEE Trans. Image Process.
1651–1660. 4(6), 863–870.
Lehmann, E. & J. Romano (2005). Testing statistical Verdié, Y. & F. Lafarge (2014). Detecting parametric
hypotheses. London, UK: Springer. objects in large scenes by Monte Carlo sampling. Int.
Lindeberg, T. (1998). Feature detection with automatic J. Comput. Vision 106(1), 57–75.
scale selection. Int. J. Comput. Vision 30(2), 79–116.
Liu, L., D. Zhang, & J. You (2007). Detecting wide lines
using isotropic nonlinear filtering. IEEE Trans. Image
Process. 16(6), 1584–1595.

106

VIPIMAGE15_Book.indb 106 9/8/2015 7:43:48 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Diagnosis of human intestinal parasites by deep learning

A.Z. Peixinho, S.B. Martins, J.E. Vargas & A.X. Falcão

Institute of Computing, University of Campinas, Campinas, Brazil

J.F. Gomes
Institute of Biology and Institute of Computing, University of Campinas, Campinas, Brazil

C.T.N. Suzuki
Immunocamp Science and Technology LTDA, Brazil

ABSTRACT: Intestinal parasitic infections can cause serious health problems, especially in children
and immunodeficient adults. In order to make the diagnosis of intestinal parasites fast and effective, we
have developed an automated system based on optical microscopy image analysis. This work presents a
deep learning approach to discover more effective parasite image features from a small training set. We
also discuss how to prepare the training set in order to cope with object scale and pose variations. By
using random kernels, our approach considerably simplifies the learning task of a suitable convolutional
network architecture for feature extraction. The results demonstrate significant accuracy gains in classifi-
cation of the 15 most common species of human intestinal parasites in Brazil with respect to our previous
state-of-the-art solution.

1 INTRODUCTION architecture. We discuss how to prepare the train-

ing set in order to cope with class object scale and
Infections by human intestinal parasites can cause pose variations. The method is validated in com-
death, physical and mental disorders in children parison with our previous state-of-the-art solution
and immunodeficient adults (Popejoy 2012). in (Suzuki et al. 2013b), by using a dataset with
The detection of parasites in optical microscopy 16,437 images whose automatic annotation was
images obtained from fecal samples is determinant confirmed/corrected by experts in Parasitology.
for the diagnosis. However, this procedure is not Section 2 shortly describes the automated sys-
effective when executed by humans. In order to tem for the diagnosis of human intestinal parasites
circumvent the problem, automated image analy- and the previous feature extraction and clas-
sis methods have been investigated (Dogantekin sification methods, while Section 3 presents the
et al. 2008, Avci and Varol 2009, Al-Sameraai et al. proposed ConvNet-based approach for feature
2013, Flores-Quispe et al. 2014), but they are usu- extraction and classification. Experimental results
ally limited to a few species of parasites and do are presented in Section 4 and Section 5 draws the
not take into account automatic image acquisi- conclusive remarks.
tion problems, such as focus adjustment, object
motion, overlapping with impurity components,
etc. In (Suzuki et al. 2013a), we proposed a first 2 AUTOMATED SYSTEM FOR
solution to automate the diagnosis of the 15 most THE DIAGNOSIS OF INTESTINAL
common species of parasites (protozoa and PARASITES
helminths) in Brazil. Subsequently, the effective-
ness and efficiency of this solution was improved The automated system for the diagnosis of human
in (Suzuki et al. 2013b). intestinal parasites consists of an optical micro-
Inspired by the recent success of deep learning scope with motorized stage, focus driver, and
techniques (Krizhevsky et al. 2012, Chiachia et al. digital camera, all controlled by a computer for
014a, Menotti et al. 2015), we present a method automated image acquisition and analysis. The
for parasite feature learning from optical micro- acquired images are first segmented into objects
scopy images. Our approach uses convolutional (single connected components), which can be
networks (ConvNets) with random kernels to con- either impurities or parasites. These objects are
siderably simplify the learning task of the ConvNet located through the color difference between the

107

VIPIMAGE15_Book.indb 107 9/8/2015 7:43:48 PM

 image background, with blue color, and candidate within the object, the difference between the ellipse
objects, with brown color. Objects with area out- and the object, energy, entropy, variance, and
side the range of parasites are eliminated, and the homogeneity of the co-occurrence matrix. These
remaining ones are used as markers for a fine delin- features were used with different weights in a sin-
eation via the IFT-watershed algorithm (Falcão et gle feature vector. Some features are more relevant
al. 2004). For image analysis, the parasites images than others to differentiate the classes (e.g., major
are divided into three different groups: (a) proto- and minor axes of the best fit ellipse, perimeter,
zoan cysts; (b) helminth eggs; and (c) helminth lar- area and histograms of texture and flat regions),
vae. The three groups can be easily distinguished and the weight of each feature was found by opti-
by simple shape measures, being helminth eggs mization in order to maximize classification accu-
and larvae about 20 times bigger than protozoa. racy within the training set by cross-validation.
Hence, the pipeline extracts more complex features The dissimilarity between objects was measured
and classify the objects within each group into one by the Euclidean distance of their feature vec-
out of the 15 species of interest or as impurity. The tors, considering the weight of each feature. We
main challenge is to learn the most effective fea- also used optimum-path forest classification (Papa
tures to distinguish similar objects from different et al. 2009), since it outperformed Support Vector
classes (Fig. 1). Machines (SVMs) with this feature set.
In (Suzuki et al. 2013b), we defined features In this work, we learn the ConvNet architecture
based on our experience: color histograms of tex- that produces effective object feature vectors for
ture and flat regions, object area, perimeter, sym- SVM-based classification. In this case, the result-
metry, major and minor axes of the best fit ellipse ing high-dimensional feature space favored linear
SVMs.

3 CONVOLUTIONAL NETWORK

Our ConvNet-based feature learning approach

relies on random kernels to considerably simplify
the hyper-parameter optimization of the ConvNet
architecture (Pinto and Cox 2011).
ConvNet models are biologically inspired mod-
els, that includes feature detecting layers, each
of which alternated with sub-sampling layers to
obtain properties leading to some translation and
deformation invariance (Matsugu et al. 2003).
Basically, a ConvNet is composed of linear and
non-linear image processing operations. When
stacked, these operations essentially extract higher
level representations, named multiband images,
whose pixel attributes are concatenated into a
high-dimensional feature vector per input image
for subsequent classification.
We consider a layer as the combination of well-
known operations (i) convolution with a bank of
filters, (ii) activation, (iii) spatial pooling, and (iv)
local normalization. Figure 3 (a) shows how these
operations are stacked within a layer.
This section describes ConvNets from  the image
processing perspective. Let Iˆ = ( I ) be a multi-
band image, where DI Z 2 is the image domain
and I ( p ) = {I
{I1( pp)) I 2 ( p ) I m ( p )} is the attribute
vector of a pixel p ( x p , y p ) DI .

3.1 Filter bank convolution


Figure 1. (a) The 15 most common species of human Let Φ i =(A Wi ) be a multiband filter with weight
intestinal parasites in Brazil and (b) similar impurity vector W i (q ) {wi , (q ) wi ,2 (q ) wi ,m (q )}, where
objects. q ∈A( p ) is adjacent to the origin p of the filter.

108

VIPIMAGE15_Book.indb 108 9/8/2015 7:43:48 PM

 A multiband filter bank Φ {Φ1, Φ 2 , , Φ n } is
a set of filters Φ i= (A ,W i ) , i { , , , n}. The
weights in each W i (q ) are randomly generated
from a uniform distribution and normalized to
zero mean and unit norm.
The convolution between an input image Iˆ and
a filter Φi produces a band i of the filtered image
Jˆ = ( J ) , where DJ DI and J ( J1 , J2 , , J n )
as shown in Figure 2.

3.2 Activation
The activation
 operation simply creates an image
Kˆ = ( J ) (a sparse code since we use zero mean
and unit norm kernels) by K i ( p ) = max( Ji ( p ), 0 ),
where p DJ are pixels of the image, and
i { , , , n} are the image bands.

3.3 Spatial pooling Figure 3. Schematic diagram of the considered frame-

work that illustrates how the fundamental operations are
Spatial pooling aims at bringing small transla- combined in a layer (a) and how the ConvNet is instan-
tional invariance to the features by aggregating tiated and evaluated according to the hyper-parameter
activations from the same filter in a given region values (b). The tables show the adopted values for all
(LeCun et al. 1998). We adopted the max-pooling considered hyper-parameters of each layer (a) and glo-
operation. bal hyper-parameters (b). This figure was adapted from
Let DM DK /s be a regular subsampling of (Menotti et al. 2015).
every s ≥ 1 pixels in an adjacency B. We call s the
stride of the pooling operation which results into

a spatial resolution reduction for s > 1. The pool-
 where M ( p ) = {M{M1( pp)) M 2 ( p ) M n ( p )} and
ing operation creates the image M ˆ =( ),
M Mi ( p ) = max q ∈B ( p ){Mi (q )}, i 1, 2, , n.

3.4 Divisive normalization

The last operation is the divisive normalization,
which is based on gain control mechanisms found
in cortical neurons (Geisler and Albrecht
 1992). It
creates an output image Oˆ = ( O ) , DO DM ,
where O( p ) = {O
{O1( pp)) O2 ( p ) On ( p )} and

Mi ( p)
Oi ( p ) = (1)
∑ j = ∑ ∀q∈C ( p ) M j (q )M j (q )
n

for some adjacency C . It promotes a competition

among pooled filter bands, such that high responses
predominate over low ones. The output feature
 vec-
tor results from the concatenation of O( p ) for all
p DO when Ô is the output of the last layer.

3.5 Hyper-parameter optimization

As presented in Figure 3 (a), each layer has seven
Figure 2. The convolution of an m-band with the
m-band filter bank. For a better understanding of the hyper-parameters (a considerable reduction in
operation, we have assigned a different color for each fil- comparison with ConvNets that learn kernel
ter band and highlighted a specific region in the input weights).
image that follows the same sequence of colors. (Martins The seven hyper-parameters determine the local
and Falcão 2015). features to be highlighted in each layer.

109

VIPIMAGE15_Book.indb 109 9/8/2015 7:43:53 PM

 1. Convolution window size: neighborhood con- According to shape size and symmetry meas-
sidered for convolution with the filter bank. ures, candidate objects were automatically divided
It determines the scale of texture details to be into three groups, protozoa/impurities, helminth
extracted from the input. eggs/impurities, and larvae/impurities, with 5,112,
2. Number of filters in the bank: higher is the 9,568, and 1,757 images, respectively.
number of filters more information is captured Each group of images was randomly divided
from the input. into three subsets, Z1, Z2 , and Z3 , with 10%, 45%,
3. Pooling window size: neighborhood considered and 45% of the images from each class, respec-
in the pooling step. It determines how much tively, such that Z1 Z2 ∩ Z3 = ∅. The ConvNet
translational variation is discarded. architecture (hyper-parameters) was learned from
4. Pooling stride: it determines the reduction fac- set Z1, by using the software developed in our
tor applied to the input resolution. group Chiachia (2014b). Then the ConvNet-based
5. Pooling strength: it determines how suscepti- features were extracted from all images in Z2 and
ble the pooling is to the highest response in the Z3 , set Z2 was used to train a linear SVM classi-
input. fier with a precomputed kernel and a hard margin
6. Local normalization: it indicates whether or not (C = 105 ). The classification accuracy (Papa et al.
local normalization must be applied to its input. 2009) was obtained on set Z3 . Training from Z2
7. Local normalization window size: neighbor- and test on Z3 was repeated 10 times with random
hood considered for local normalization. The sets, Z2 and Z3 , for statistical validation.
highest response within this neighborhood pre- In order to evaluate the impact of input image
vails over the lower ones. resolution (object scale) for architecture learning,
we evaluated two types of input image resolutions:
There are also four network hyper-parameters.
200 × 200 and 400 × 400 pixels. The impact of class
1. Input image size: it must reveal enough texture imbalance in architecture learning was evaluated by
details about the objects of interest. cross-validation in Z1, when features were learned
2. Local normalization: it indicates whether or not from the actual Z1 and from a modified version of
local normalization must be applied to its input. Z1 with a same number of samples per class.
3. Local normalization window size: neighbor-
hood considered for local normalization. The
highest response within this neighborhood pre- 4.1 Feature learning
vails over the lower ones. Considering the discrete hyper-parameter search
4. Number of layers in the network: each layer space in Figure 3, the candidate ConvNets were
is defined by the aforementioned layer hyper- evaluated by training a SVM classifier on 50% of
parameters. the samples in Z1 and testing it on the remaining
For this application, it was enough to learn deep samples. This process was also repeated for 50%
representations up to three layers (Fig. 3 (b)). The −50% random splits of Z1 in order to select the
hyper-parameters above define the network archi- best architecture and evaluate the impact of input
tecture and their possible values define a search image resolution and class imbalance.
space for optimization. Table 1 shows the accuracy results of the best
We adopted the strategy in (Menotti et al. 2015), ConvNet on 50% of Z1 for the cases of balanced
which uses random search of the possible discrete val- and unbalanced versions of it.
ues for all hyper-parameters to find a ConvNet that Class imbalance is not a main issue for feature
produces the highest classification accuracy, using learning in the cases of protozoa and larvae, but it
multiple random subsampling in the training set. considerably affects the learning process in the case
Additionally, we evaluated the impact of the of helminth eggs. Therefore, the final ConvNet
input image resolution and class imbalance, since architecture for each group was learned by forcing
helmith eggs and larvae are considerably larger a same number of samples per class in a modified
than protozoa, the number of impurities is con- version of Z1.
siderably higher than the number of parasites, and
some species are less frequent than others.
Table 1. Accuracy results of the best ConvNet on 50%
4 EXPERIMENTS AND RESULTS of Z1, using the balanced and unbalanced versions of it.

Balanced Unbalanced
The experiments used a dataset with 16,437 objects
(segmented images with their corresponding class Protozoa 0.8288 ± 0.0150 0.8171 ± 0.0190
as confirmed by experts), including examples from Eggs 0.8994 ± 0.0212 0.8310 ± 0.0358
the 15 most common species of human intestinal Larvae 0.8735 ± 0.0778 0.8688 ± 0.0240
parasites in Brazil and impurities.

110

VIPIMAGE15_Book.indb 110 9/8/2015 7:43:57 PM

 Table 2 shows the accuracy results of the best Table 5. Average recall and precision of the proposed
ConvNet on 50% of the balanced version of Z1, for approach on each class of parasite and impurity.
two input image resolutions.
This experiment shows that ConvNets cannot Class Recall Precision
deal with large object scale variations in a same data- Entamoeba coli 0.9867 0.9753
set. Therefore, we fixed the input image resolution in Entamoeba histolytica/E. dispar 0.8868 0.8594
400 × 400 pixels for protozoa feature learning and in Endolimax nana 0.9433 0.9175
200 × 200 pixels for eggs and larvae feature learning. Giardia duodenalis 0.9391 0.9364
Iodamoeba butschlii 0.9521 0.9532
4.2 Feature vector evaluation Blastocystis hominis 0.8833 0.9164
Given the best ConvNet architecture for each Protozoa impurities 0.9740 0.9782
group, we extract the ConvNet features from all Hymenolepis nana 0.9946 0.9856
images in Z2 and Z3 , use Z2 to train a SVM clas- Hymenolepis diminuta 0.9258 0.9345
sifier, and test its accuracy on Z3 , for each 50% − Hookworms 0.8510 0.9581
50% random split of Z2 Z3 into Z2 and Z3 . Enterobius vermicularis 0.9313 0.9679
Ascaris lumbricoides 0.9043 0.9199
Simard et al. (2003) points out the benefits of
Trichuris trichiura 0.9950 0.9888
applying transformations in the training set to
Schistosoma mansoni 0.9044 0.9153
increase the ConvNet effectiveness. In order to
Taenia spp. 0.9904 0.9734
cope with possible object variations in pose, which
Helminth eggs impurities 0.9827 0.9765
often occurs for parasites and impurities in the real
Strongyloides stercoralis 0.9269 0.9116
scenario, we also evaluated the classifier when Z2 is Helminth larvae impurities 0.9871 0.9894
extended by sample rotations of 90°, 180° and 270°
for training. Table 3 shows the average accuracy
of the classifier over Z3 when features are learned
without and with additional rotated samples in Z2 . Table 6. Hyperparameters defining the best ConvNet
architecture for each parasite group.
A slight improvement can be observed when
adding rotated samples to Z2 , except for the case Parameters Larvae Proto Eggs
of helminth eggs.
Network
Number of layers 3 2 3
Table 2. Accuracy results of the best ConvNet on Input size 200 × 200 400 × 400 200 × 200
50% of the balanced version of Z1, for two input image
resolutions. Input normalization No Yes Yes
Input norm. window – 5 7
200 × 200 400 × 400 Layer #1
Number of filters 32 128 32
Protozoa 0.8288 ± 0.0150 0.8970 ± 0.0583 Convolution size 7 5 7
Eggs 0.8994 ± 0.0212 0.8983 ± 0.0207 Pooling size 7 5 9
Larvae 0.8735 ± 0.0778 0.8568 ± 0.0467 Pooling stride 4 2 1
Pooling strength 1 2 1
Normalization No No No
Table 3. Average accuracy over Z3, when features are Normalization size – – –
learned without and with additional sample rotation in Z2. Layer #2
Dataset Without rotation Using rotation Number of filters 128 256 256
Convolution size 3 9 9
Protozoa 0.9559 ± 0.0074 0.9649 ± 0.0043 Pooling size 5 7 5
Eggs 0.9646 ± 0.0048 0.9679 ± 0.0033 Pooling stride 1 4 4
Larvae 0.9511 ± 0.0090 0.9570 ± 0.0079 Pooling strength 2 1 10
Normalization Yes Yes Yes
Normalization size 9 7 3

Table 4. Average accuracy over Z3 of the ConvNet- Layer #3

based approach and the method in (Suzuki et al. 2013b). Number of filters 256 – 256
Convolution size 5 – 3
Dataset Suzuki et al. (2013b) ConvNet Pooling size 9 – 9
Pooling stride 8 – 4
Protozoa 0.9028 ± 0.0038 0.9649 ± 0.0043 Pooling strength 10 – 10
Eggs 0.9316 ± 0.0074 0.9679 ± 0.0033 Normalization No – No
Larvae 0.9536 ± 0.0062 0.9570 ± 0.0079 Normalization size – – –

111

VIPIMAGE15_Book.indb 111 9/8/2015 7:44:00 PM

 The same procedure involving Z2 and Z3 was Dogantekin, E., M. Yilmaz, A. Dogantekin, E. Avci, &
A. Sengur (2008). A robust technique based on invari-
executed with the original feature vector (which
ant moments—anfis for recognition of human para-
is not based on deep learning) and the optimum- site eggs in microscopic images. Expert Systems with
path forest classifier, as proposed in (Suzuki et al. Applications 35(3), 728–738.
2013b). Finally, we compare the results obtained Falcão, A., J. Stolfi, & R. Lotufo (2004). The image
by the ConvNet-based approach with those of the foresting transform theory, algorithms, and applica-
method in (Suzuki et al. 2013b) (see Table 4). tions. IEEE Transactions on Pattern Analysis and
Except for the case of larvae, the proposed Machine Intelligence 26(1), 19–29.
approach can considerably improve effectiveness of Flores-Quispe, R., R.E.P. Escarcina, Y. Velazco-Paredes, &
our automated system for the diagnosis of human C.A.B. Castanon (2014). Classification of human par-
asite eggs based on enhanced multitexton histogram.
intestinal parasites. This can be explained by the
In IEEE Colombian Conference on Communications
uniformly good performance in classification over and Computing (COLCOM), pp. 1–6.
all 15 species of parasites (see Table 5). Geisler, W.S. & D.G. Albrecht (1992). Cortical neurons:
The best architecture for each parasite group is Isolation of contrast gain control. Vision Research
shown in Table 6. 32(8), 1409–1410.
Krizhevsky, A., I. Sutskever, & G. Hinton (2012).
Imagenet classification with deep convolutional neural
5 CONCLUSION AND FUTURE WORK networks. In Advances in Neural Information Process-
ing Systems (NIPS), pp. 1097–1105.
LeCun, Y., L. Bottou, Y. Bengio, & P. Haffner (1998).
We proposed a ConvNet-based approach for the
Gradient based learning applied to document recogni-
diagnosis of human intestinal parasites and pre- tion. Proceedings of the IEEE 86(11), 2278–2324.
sented a srategy to deal with class imbalance and Martins, S.B. & A.X. Falc˜ao (2015). A fast and robust
considerable object scale variations in the data- negative mining approach for user enrollment in face
set. Our approach has demonstrated the ability recognition systems. Master’s thesis, Universidade
to learn features from a small dataset and obtain Estadual de Campinas, Campinas, São Paulo—Brazil.
significant gains in classification accuracy for pro- Matsugu, M., K. Mori, Y. Mitari, & Y. Kaneda (2003).
tozoa and helminth eggs, maintaining the same Subject independent facial expression recognition with
good performance for larvae, as compared to our robust face detection using a convolutional neural net-
work. Neural Networks 16(5–6), 555–559. Advances in
previous state-of-the-art solution (Suzuki et al.
Neural Networks Research: {IJCNN} ’03.
2013b). Menotti, D., G. Chiachia, A. Pinto, W. Schwartz, H.
The ConvNet-based feature extraction can be Pedrini, A.X. Falc˜ao, & A. Rocha (2015). Deep rep-
efficiently implemented in parallel and the lin- resentations for iris, face, and fingerprint spoofing
ear SVM classification is a simple inner product detection. IEEE Transactions on Information Forensics
between feature vectors and the normal vector of and Security (TIFs) PP(99), 1–1.
the hyper-plane of each class. Therefore, we may Papa, J.P., A.X. Falc˜ao, & C.T.N. Suzuki (2009). Super-
conclude that our ConvNet-based approach can be vised pattern classification based on optimum-path
an efficient and effective solution for the automated forest. International Journal of Imaging Systems and
Technology 19(2), 120–131.
diagnosis of human intestinal parasites. We intend
Pinto, N. & D. Cox (2011). Beyond simple features: A
to further investigate ConvNet architectures for the largescale feature search approach to unconstrained
specific least accurate cases in Table 5 and extend face recognition. In IEEE International Conference on
the work to the diagnosis of parasites in animals. Automatic Face Gesture Recognition and Workshops
(FG), pp. 8–15.
Popejoy, M.W. (2012). Working to overcome the global
REFERENCES impact of neglected tropical diseases. Perspectives in
Public Health 132(4), 192.
Al-Sameraai, R.S.H., G.K. Hawari, & M. Zeehaida Simard, P.Y., D. Steinkraus, & J.C. Platt (2003). Best
(2013). Automated system for diagnosis intestinal practices for convolutional neural networks applied to
parasites by computerized image analysis. Modern visual document analysis. In International Conference
Applied Science 7(5), 98–114. on Document Analysis and Recognition, pp. 958–963.
Avci, D. & A. Varol (2009). An expert diagnosis system Suzuki, C.T.N., J.F. Gomes, A.X. Falcão, J.P. Papa, &
for classification of human parasite eggs based on S.H. Shimizu (2013a). Automatic segmentation and
multi-class svm. Expert Systems with Applications classification of human intestinal parasites from
36(1), 43–48. microscopy images. IEEE Transactions on Biomedical
Chiachia, G. (2014b). Hyperparameter Optimization Engineering 60(3), 803–812.
made Simple. Suzuki, C.T.N., J.F. Gomes, A.X. Falcão, S.H. Shimizu, &
Chiachia, G., A.X. Falc˜ao, N. Pinto, A. Rocha, & D. J.P. Papa (2013b). Automated diagnosis of human
Cox (2014a, Dec). Learning person-specific repre- intestinal parasites using optical microscopy images.
sentations from faces in the wild. IEEE Transactions In IEEE International Symposium on Biomedical
on Information Forensics and Security (TIFs) 9(12), Imaging, pp. 460–463.
2089–2099.

112

VIPIMAGE15_Book.indb 112 9/8/2015 7:44:02 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Texture-energy features for microaneurysms detection

D. Veiga
Enermeter, Parque Industrial de Celeirós, Lugar de Gaião, Braga, Portugal
University of Minho, Centro Algoritmi, Azurém, Guimarães, Portugal

N. Martins
Centro de Computação Gráfica, Azurém, Guimarães, Portugal
Enermeter, Parque Industrial de Celeirós, Lugar de Gaião, Braga, Portugal

C. Pereira & M. Ferreira

Enermeter, Parque Industrial de Celeirós, Lugar de Gaião, Braga, Portugal

J. Monteiro
University of Minho, Centro Algoritmi, Azurém, Guimarães, Portugal

ABSTRACT: The appearance of microaneurysms is the first clinical sign in Diabetic retinopathy. These
lesions appear as small red dots in color fundus images. Their detection in a preliminary phase could
prevent disease progression which leads to blindness. This paper addresses the problem of automatic
microaneurysms detection in color fundus images. Laws texture-energy measures were selected as feature
extractor and descriptor. The suitability of these measures were investigated by testing several kernel sizes
followed by support vector machine classification.
Tests were made on the LaTIM (laboratoire de Traitement de l’Information Médicale) database and
results are evaluated in terms of the Free-Response Receiver-Operating Characteristic (FROC) curve. The
best performance was achieved with 31 × 31 kernel size.
To our knowledge, this is the first time that Laws local texture-energy transform is used to segment
microaneurysms. Results are promising and a next step to analyse each microaneurysm candidate is needed.

1 INTRODUCTION Cazuguel, Cochener, & Roux 2008, Zhang, Karray,

Li, & Zhang 2012) and supervised learning-based
Detection of microaneurysms (MAs) provides (García, Sánchez, Lopez, Diez, & Hornero 2008,
insight about Diabetic Retinopathy (DR) presence, Niemeijer, van Ginneken, Staal, Suttorp-Schulten, &
a frequent complication of diabetes mellitus. These Abrámoff 2005). The morphological top hat trans-
lesions are often an early manifestation of the disease form is the most common morphological opera-
and their premature detection can delay the pathol- tion used to detect MAs. Here, a morphological
ogy evolution by induction of a tighter metabolic opening of the green channel image is performed
control. In addition, if the disease is detected on with linear structuring elements at different orien-
time, ophthalmologists can act therapeutically using tations, short enough to fit inside curved vessels,
laser treatment before irreversible damage occurs but, at the same time long enough to not fit inside
in the retina. MAs appear as small red dots in the MAs. Larger objects like blood vessels are conse-
superficial retinal layers and can be observed in dig- quently detected and removed, leaving only MA
ital color fundus images (Neubauer & Ulbig 2007). candidates. This approach has been modified and
Published methods on automatic segmenta- used by other authors (Walter, Massin, Erginay,
tion of microaneurysms can be categorized in Ordonez, Jeulin, & Klein 2007, Niemeijer, van Gin-
three groups: mathematical morphology (Walter, neken, Staal, Suttorp-Schulten, & Abrámoff 2005).
Massin, Erginay, Ordonez, Jeulin, & Klein 2007, Filter-based algorithms aim to produce an appro-
Cree 2009, Niemeijer, van Ginneken, Staal, priate kernel to match MAs. Wavelet and Gaus-
Suttorp-Schulten, & Abrámoff 2005), filter- sian kernels are the most common. Quellec et al.
based (Niemeijer, Van Ginneken, Cree, Mizutani, (Quellec, Lamard, Josselin, Cazuguel, Cochener, &
Quellec, Sánchez, Zhang, Hornero, Lamard, Roux 2008) match lesions templates in subbands
Muramatsu, et al. 2010, Quellec, Lamard, Josselin, of wavelets transformed images. In learning-based

113

VIPIMAGE15_Book.indb 113 9/8/2015 7:44:02 PM

 methods, classifiers like artificial neural networks, Cochener, & Roux 2008). This database contains
support vector machines, k-NN and pixel clas- 36 manually annotated eye fundus color photo-
sification have been reported. Usually these algo- graphs from Brest University Hospital. Images
rithms start with a preprocessing step, candidate have 2240 × 1488 pixels and were stored in tiff file
extraction and feature selection for classification. format. Images were cropped in order to remove
An ensemble-based system to detect MAs has most of the black background, characteristic of
recently been proposed which consists in the selec- retinal images.
tion of the optimal combination of preprocessing The performance of the proposed method is
and candidate extractor methods (Antal & Hajdu evaluated in terms of the FROC curve, that plots
2012). This method achieved the best results on the sensitivity against the average number of False
ROC dataset. Despite the undoubted evolution of Positives (FP) per image.
the reported approaches, there is still the opportu- The proposed method has a training phase. A
nity to contribute with more advanced and effec- total of 93 window samples were manually selected
tive methods. from 10 normalized images of the LaTIM data-
In this paper, Laws texture-energy measurements base, with the following distribution: 63 windows
for MAs detection are studied. Texture consists of containing MAs, 22 windows containing vessels
spatially extended patterns, of more or less accurate and 8 windows containing only background. From
repetitions of some basic texture element, called these 93 images, train was performed with 5-fold
textels. These are continuous set of pixels with cross-validation.
some tonal and/or regional property. To be able to Tests to the developed method were made on the
find changes in the texture of an image, a simple remaining 26 images from LaTIM.
strategy is to perform texture measurements in a
sliding window. Then, window size and texture fea-
2.2 Preprocessing
tures need to be selected. To detect MAs we decided
to use a higher order statistics method such as Laws Preprocessing aims to attenuate retinal images
texture masks. First order statistics from intensity imperfections such as non-uniform illumination,
histogram like mean and variance, and second poor contrast and noise as well as retina’s variabil-
order statistics like gray-level co-occurrence matri- ity from different persons.
ces are popular texture description methods due to The background intensity variation can be
their simplicity. However, since the MA detection eliminated by the background estimation and
problem presents various challenges, a higher order subsequent subtraction from the original green
statistics method might be a good strategy for this plane image. The background image is deter-
purpose. Its advantages over first and second order mined by applying a median filter with a kernel
statistics are that they can describe geometry and size corresponding to 1/30 the image height as in
context. These features have been used in other (Giancardo, Meriaudeau, Karnowski, Li, Garg,
biomedical images studies such as (Elnemr 2013) Tobin, & Chaum 2012). The obtained shaded
and (Dheeba & Selvi 2011). Five onedimensional corrected image shows a characteristic gray level
kernels were chosen as in (Dheeba & Selvi 2011) distribution with dark structures belonging to the
to create the two-dimensional masks. Following, a histogram left tail with negative values. At this
SVM classifier performs pixel-based classification. point, and since all normalized images have similar
The results of several kernel’s size are analysed in histogram characteristics, it is possible to elimi-
36 images from LaTIM database and performance nate all bright structures such as the optic disc and
is evaluated by FROC curves for each kernel size. exudates by putting these elements equal to zero.
New insights are given related with detection of Then, only the absolute value of all pixels belong-
MAs in color fundus images. ing to the dark structures is kept.
This paper is organized as follows. Section 2
presents the database utilized and details of the pro-
2.3 Microaneurysms detection
posed approach. Section 3 discusses the obtained
results on LaTIM database. Finally, major conclu- The automatic detection of microaneurysms in
sions are presented in Section 4. digital fundus images remains a challenge in image
processing. These small red dots have typically
low contrast and they may be hard to distinguish
2 MATERIALS AND METHODS
from variations in the background. Texture is a
key component of the human visual perception,
2.1 Database
along with color and shape. Here, we will focus
In this study the LaTIM (Laboratoire de Traite- on texture, and the Laws masks are proposed to
ment de l’Information Médicale) database was calculate the description features of MA for their
used (QuelQuellec, Lamard, Josselin, Cazuguel, texture discrimination. A classification step is then

114

VIPIMAGE15_Book.indb 114 9/8/2015 7:44:02 PM

 performed to find the MA candidates. A schematic detects ripples and the W5 vector detects waves.
representation of the proposed approach is shown Consequently, the weighted masks will measure
in Figure 1. content related with each vector used to generate
Laws texture energy transform combines pre- the mask. For example, E5L5 measures horizon-
determined one-dimensional convolution kernels tal edge content, and L5E5 measures vertical edge
into various convolution masks with moving win- content. The filters are binomially weighted masks
dow updating. Figure 2 shows the vectors that that enhance image spots, edges, or high-frequency
are convolved with their own transposes to gener- components and measure the spatial frequencies or
ate twenty-five 5 × 5 weighted masks (Shapiro & texture structures (Laws 1985, Lee & Schenk 1993).
Stockman 2001, Dheeba & Selvi 2011). The names A particular texture-energy measure thus depends
of the vectors describe their purposes. The L5 vec- on the spatial filters used, the window size, and the
tor gives a center-weighted local average, the E5 method of measuring average variation within the
vector detects edges, the S5 detects spots, the R5 window.
The first step is to filter the preprocessed images
with the convolution masks. Several kernel sizes
were tested in order to find out which one achieved
better performance in MA identification. The fol-
lowing sizes were selected to test: 5 × 5, 11 × 11,
15 × 15, 21 × 21, 25 × 25, 31 × 31, 35 × 35, 41 × 41
and 45 × 45. The comparison method was based
on FROC curves. The different kernel sizes were
obtained using bilinear interpolation of the 5 × 5
kernels.
After convolving each preprocessed image with
Figure 1. Proposed approach schematic. the 25 convolution kernels, a 25-dimensional fea-
ture vector is obtained for each pixel. Assuming
that Fk [i j ] is the result of filtering with the kth
mask at pixel [ j ] . Then, after the convolution
process with Laws masks, the texture energy map
Ek for filter k L5L5 is defined by
c+w r +w
Ek [ r c ] ∑ ∑ j ] |,
| Fk [i j] (1)
j c −w i r −w

with,

KernelSize × 3
w= − 1, (2)
2

The texture energy map consists in a moving

window operation, where every pixel in the image
is replaced by the sum of pixels (magnitudes) in its
neighbourhood. In literature (Laws 1985), the win-
dow operation is applied to all filtered images, Fk.
However, this operation may smooth small lesions
or even be responsible for their disappearance and
thus we consider it to be more adequate for macro-
texture features detection. Consequently, and since
the main purpose is to discriminate MAs from noise
and background, we decided to apply the window
operation only in the FL5L5 which is then used in the
contrast normalization step. The unique texture
energy map is obtained by filtering the preprocessed
Figure 2. (a) 5 × 5 texture vectors used to create two- image with Level kernel (L5L5), FL5L5, followed by
dimensional texture masks. L, E, S, W and R stand for the window operation. Windows with three times
level, edge, spot, ripple and wave. (b) The convolution of the kernel size were used. The texture energy map
E5 and L5 vectors. has the same size of the original image.

115

VIPIMAGE15_Book.indb 115 9/8/2015 7:44:02 PM

 Filtered images from convolution with Laws detected by the algorithm, while false positives are
masks are then normalized for contrast by dividing the number of non-MAs identified as MAs. To
each resultant filtered image with the EL5L5 map. construct the FROC curve, the real SVM output
MAs segmentation is then performed by pixel- values were kept and then, a threshold with val-
classification using SVM. Each pixel from the final ues between [−1 1] was applied to that gray level
texture energy transform image consists in a 25- image. Figure 4 displays the binary output of
dimensional vector and corresponds to the input the SVM pixel classification for six kernel sizes.
of the classifier which then determines if that pixel Kernels larger than 31 × 31 are not shown since
is a candidate to MA or not. The OpenCV function they are not much different from 31 × 31 kernel.
train_auto was used to train the SVM model auto- It is possible to observe that MAs are detected,
matically by choosing the optimal parameters C, as well as many other objects. As expected, the
gamma, p, nu, coef0 and degree. Parameters were kernel dimension interferes in the identification
selected using a grid search and a cross-validation of MAs and, with larger kernels the number of
with 5 folds. Parameters with the best results were
saved and used in the final tests.

3 RESULTS AND DISCUSSION

The main objective of this paper was to detect

microaneurysms in color fundus images with tex-
ture analysis. For this purpose, Laws texture energy
values were computed in 36 retinal images. Nine ker-
nel sizes were tested and processed images classified
with a SVM classifier. The SVM was trained using
manually selected regions from images. Using the
entire image for training would be time consuming
since the number of MAs is very small compared to
the non-MAs regions. Due to this fact, and to reduce
class imbalance, 93 small representative regions of
10 images were selected with the different regions
present in retina (blood vessels and background).
In addition, this image selection for train aims to
reduce redundant information that could be intro-
duced if the entire image was used. The training
algorithm used applied 5-fold cross-validation and
data was split equally. By minimizing the samples
used, overtraining is avoided. Nine trainings were
computed, one for each tested size. Figure 3 shows
some examples of training images. For better visu-
alization, images are contrast enhanced.
Performance was evaluated in terms of the
FROC curve. This seemed to be the best choice
to quantitatively evaluate the MAs location iden-
tification. In the FROC plot, sensitivity values are
plotted against the average number of false posi-
tives per image. Sensitivity refers to MAs correctly

Figure 4. Images resulting from the proposed method:

(a) Original color fundus image; (b) ground truth; (c) MA
candidates with kernel 5 × 5; (d) MA candidates with ker-
Figure 3. Example of manually selected images to train nel 11 × 11; (e) MA candidates with kernel 15 × 15; (f) MA
SVM: (a) image containing vessels; (b) image containing candidates with kernel 21 × 21; (g) MA candidates with
MAs and vessels; (c) image containing only background. kernel 25 × 25; (h) MA candidates with kernel 31 × 31.

116

VIPIMAGE15_Book.indb 116 9/8/2015 7:44:03 PM

 outliers diminishes. Moreover, these texture fea- to include them, but at the same time, not enough
tures proved to be very good in vessel segmenta- small only to embrace MA pixels. Since the number
tion with small kernels. In Figure 4c, vessels are of false positives per image is still very high for a
clearly visible with 5 × 5 kernel. This result could real application, we concluded that further process-
be compared with other vessel segmentation ing of each MA candidate is needed.
approaches in the future. By looking at images
in Figure 4 is possible to conclude that micro-
texture filters detects patterns of texture energy 4 CONCLUSION
features, and with different kernels sizes different
retinal structures are discriminated. An adequate In this paper we present a MA detection method
size of filter must be selected. based on Laws texture-energy measurements. As
Figure 5 and 6 shows the FROC curves for each far as we know these features were never reported
kernel dimension. For less than 30 false positives to MAs detection in fundus images. MAs are
all kernels are below 60% sensitivity. Above that extracted using only these features in an SVM.
number of FP, sensitivity values start to increase Nine kernels sizes were tested to analyse the impact
and for 50 False Positives (FP), approximately 70% on MAs identification. Results showed that Laws
sensitivity is reached with kernels 31 × 31, 35 × 35 texture features can be used to find MA candidates,
and 41 × 41. In addition, sensitivities of 80% are and 31 × 31 kernel size achieved best results. How-
achieved for 100 FP, and 90% for 200 FP, with ker- ever, the number of false positives is still an issue.
nels 31 × 31 and 35 × 35. These two kernels show Consequently, this method may be considered to
very close performances but for more than 100 FP find MA candidates but further processing should
kernel size 31 × 31 appears to be above kernel 35 be performed. Furthermore, it was concluded that
× 35. The performance of the classifier does not small kernels like 5 × 5 are very good in retinal ves-
improve for kernels 41 × 41 and 45 × 45. The worst sels segmentation. Finally, an advantage of this
performance was achieved with kernel 15 × 15. We approach is its simplicity, since no vessel or optic
presume this size is probably very close to the MAs nerve segmentation was needed. As future work
dimension, and consequently not sufficiently high results using Laws texture features will be com-
pared with some other form of texture quantifica-
tion method, such as Haralick features.

ACKNOWLEDGMENTS

Work supported by FEDER funds through the

Programa Operacional Factores de Competitivi-
dade COMPETE and by national funds by FCT
Fundação para a Ciência e a Tecnologia. D. Veiga
thanks the FCT for the SFRH/BDE/51824/2012
and ENERMETER.

Figure 5. FROC curves of the proposed approach in

LaTIM images. REFERENCES

Antal, B. & A. Hajdu (2012, June). An ensemble-based

system for microaneurysm detection and diabetic
retinopathy grading. Biomedical Engineering, IEEE
Transactions on 59(6), 1720–1726.
Cree, M.J. (2009). Automated microaneurysm detection
for screening. Automated Image Detection of Retinal
Pathology, 155–184.
Dheeba, J. & S. Selvi (2011). Classification of malignant
and benign microcalcification using svm classifier. In
Emerging Trends in Electrical and Computer Technol-
ogy (ICETECT), 2011 International Conference on,
pp. 686–690. IEEE.
Elnemr, H.A. (2013). Statistical analysis of laws mask
texture features for cancer and water lung detection.
Figure 6. FROC curves in more detail between 0 and International Journal of Computer Science Issues
50 false positives. (IJCSI) 10(6).

117

VIPIMAGE15_Book.indb 117 9/8/2015 7:44:06 PM

 García, M., C.I. Sánchez, M.I. Lopez, A. Diez, & Niemeijer, M., B. van Ginneken, J. Staal, M.S. Suttorp-
R. Hornero (2008). Automatic detection of red lesions Schulten, & M.D. Abrámoff (2005). Automatic
in retinal images using a multilayer perceptron neu- detection of red lesions in digital color fundus photo-
ral network. In Engineering in Medicine and Biology graphs. Medical Imaging, IEEE Transactions on 24(5),
Society, 2008. EMBS 2008. 30th Annual International 584–592.
Conference of the IEEE, pp. 5425–5428. IEEE. Quellec, G., M. Lamard, P.M. Josselin, G. Cazuguel, B.
Giancardo, L., F. Meriaudeau, T.P. Karnowski, Y. Cochener, & C. Roux (2008). Optimal wavelet trans-
Li, S. Garg, K.W. Tobin, & E. Chaum (2012). form for the detection of microaneurysms in retina
Exudate-based diabetic macular edema detection photographs. Medical Imaging, IEEE Transactions on
in fundus images using publicly available datasets. 27(9), 1230–1241.
Medical Image Analysis 16(1), 216–226. Shapiro, L. & G. Stockman (2001). Computer Vision.
Laws, K.I. (1985). Goal-directed textured-image segmen- Paperback.
tation. In 1985 Technical Symposium East, pp. 19–26. Walter, T., P. Massin, A. Erginay, R. Ordonez, C. Jeulin, &
International Society for Optics and Photonics. J.-C. Klein (2007). Automatic detection of microaneu-
Lee, D.-C. & T. Schenk (1993). Image segmentation from rysms in color fundus images. Medical image analysis
texture measurement. International Archives of Photo- 11(6), 555–566.
grammetry and Remote Sensing 29, 195–195. Zhang, B., F. Karray, Q. Li, & L. Zhang (2012). Sparse
Neubauer, A.S. & M.W. Ulbig (2007). Laser treatment in representation classifier for microaneurysm detec-
diabetic retinopathy. Ophthalmologica 221(2), 95–102. tion and retinal blood vessel extraction. Information
Niemeijer, M., B. Van Ginneken, M.J. Cree, A. Mizutani, Sciences 200, 78–90.
G. Quellec, C.I. Sánchez, B. Zhang, R. Hornero, M.
Lamard, C. Muramatsu, et al. (2010). Retinopathy
online challenge: automatic detection of microaneu-
rysms in digital color fundus photographs. Medical
Imaging, IEEE Transactions on 29(1), 185–195.

118

VIPIMAGE15_Book.indb 118 9/8/2015 7:44:06 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

An iterative algorithm for Total Variation minimization in DBT imaging

A.M. Mota, N. Matela, N. Oliveira & P. Almeida

Instituto de Biofísica e Engenharia Biomédica, Faculdade de Ciências, Universidade de Lisboa, Lisboa, Portugal

ABSTRACT: Digital Breast Tomosynthesis (DBT) is a new technique that takes several projection
images with different angles and combines them to reconstruct 3D image using approximately the same
dose of two mammograms. Currently, handling noise is a major problem in DBT. Total Variation (TV)
based minimization algorithms have achieved great success due to its virtue of preserving edges while
reducing image noise. Because TV denoising depends on noise model, we tested an algorithm for Poisson
noise in real DBT images.
For unfiltered images we have a Contrast to Noise Ratio (CNR), a spatial resolution and TV of 1.110,
0.483 mm and 2.319E+08, respectively. After filter application, the CNR improves up to 5.3%, spatial
resolution value doesn’t suffer degradation (variations up to 0.4%) and TV decrease up to 40.4%.
We conclude that application of TV denoising to DBT images is a promising technique and allows a
wide field of research in the future.

1 AIM tested an iterative algorithm based on ROF model

but for Poisson noise (Jonsson et al. 1998, Le et al.
Digital Breast Tomosynthesis (DBT) is a new 2007, Sawatzky et al. 2009). We applied it to DBT
x-ray imaging technique for breast cancer detec- images and carried out visual and quantitative
tion (Jähne 2005, U.S. Food and Drug Administra- analysis of our method.
tion (FDA) 2011, Sechopoulos 2013) that has the
ability to overcome Digital Mammography (DM)
limitations on tissue superposition (Sechopoulos, 2 MATERIALS AND METHODS
2013). It acquires a set of 2D projections from dif-
ferent source positions, within a limited angular 2.1 Acquisition and reconstruction
range, and combines them to reconstruct 3D image
A Mammographic Accreditation Phantom Model
using approximately the same dose of two mam-
156 by Gammex was acquired to perform image
mograms (Timberg et al. 2008).
quality assessment. It was used a Siemens Mammo-
Currently, handling noise level is a major problem
mat Inspiration that makes 25 low dose acquisition
in DBT images mainly due to low dose projections
within a limited angular range from approxi-
and incomplete sampling of the projection fre-
mately −25º and +25º. The equipment that performed
quency space due to the limited angular range used.
the acquisition was installed in a clinical facility
Image denoising is a challenging issue in image
that makes DBT examinations in a daily basis.
processing field and Total Variation (TV) based
The acquired data was reconstructed with the
minimization algorithms have achieved great suc-
image reconstruction algorithm Algebraic Recon-
cess among several imaging techniques due to its
struction Technique (ART), with a constant relax-
virtue of preserving edges while reducing image
ation coefficient of 0.20 (this choice was made
noise (Ertas et al. 2014, Feng et al. 2009, Jonsson
based on previous studies—scientific paper under
et al. 1998, Joshi et al. 2009, Mota et al. 2015, Nett
approval in the international Journal of Digital
et al. 2008, Sidky et al. 2011, Zhiqiang et al. 2013).
Imaging) and finalized after convergence iteration
TV is a quantity that characterizes how smoothly
(7th), using IDL from Exelis with a pixel size of
the intensity of an image is changing and it increases
0.34 mm and a slice thickness of 1.0 mm.
significantly in the presence of noise. TV minimiza-
tion algorithm was originally developed by Rudin,
2.2 Image analysis
Osher and Fatemi (ROF model) (Rudin et al. 1992),
where the TV of an image is minimized subject to For image analysis, quantitative and qualitative
constraints involving Gaussian noise statistics. comparisons were performed between unfiltered
In the present work our aim was to implement and filtered reconstructed images. Two Figures of
a method to reduce DBT images noise. For this we Merit were used for quantitative analysis of image

119

VIPIMAGE15_Book.indb 119 9/8/2015 7:44:06 PM

 quality: Contrast to Noise Ratio (CNR) and Full state solution of gradient descent is the minimizer
Width at Half Maximum (FWHM) of one micro- of the problem. Considering Poisson noise, this
calcification. For CNR, a Region of Interest (ROI) means that we solve:
over the lesion (mass) and another over the back-
ground were drawn. Rhis latter was obtained as a uin+j 1 uin j
square centered in the lesion, excluding the lesion.
CNR was calculated with Equation 1, where μL Δt
fi
and μB stand for mean pixel values in ROI over =
j
− uin+1
j
the lesion and background, respectively; and σB ⎡ ⎛
1 1 Δ +x ui , j
stands for standard deviation in background ROI. 1 − ⎢ 2 ⎜ Δ −x
λ ⎢h ⎜ ε2 + ( + 2
+ ( Δ 0yui , j )2
Signal Difference was calculated as the difference
⎣ ⎝ x ui , j )
between mean pixel values in the lesion and in the
Δ +y ui , j ⎞⎤
background.
+ Δ −y ⎟⎥
μ − μB ε2 + ( 0
x ui , j )
2
( y ui j )2 ⎟⎠ ⎥
⎦
N = L
CNR (1)
σB (4)

FWHM of a Gaussian curve fitted to the Point Here Δx+, Δx−, Δy+ and Δy− denote forward and
Spread Function of one microcalcification was backward one-sided differences in x and y direc-
considered as an indicator of spatial resolution. tions, respectively; and Δx0 and Δy0 indicate central
CNR and FWHM values were calculated before difference in x and y directions, respectively. This
and after filter application for the two scenarios. formulation is valid only for interior points, and
Boundary conditions were defined as in Ref Rudin
et al. 1992. To solve the problem with these bound-
2.3 Formulation of TV minimization algorithm ary conditions for u, a routine was implemented
With this TV minimization algorithm, starting and integrated in the image reconstruction soft-
from a constrained minimization problem, it is ware implemented in IDL.
obtained an unconstrained problem that is com- Δt should be a small number that forces the
posed by a fidelity term corresponding to the algorithm to make small jumps. Its optimal value
assumed noise model and a regularization term depends on the function. If it is too large, the algo-
that is the TV function. rithm will diverge and if it is too small, it will take
Our unconstrained TV minimization problem, a long time to converge.
based on ROF model but for Poisson noise, can be λ is the Lagrange multiplier, also called as regu-
formulated as Equation 2 (Le et al. 2007, Sawatzky larization or fitting parameter since it controls
et al. 2009): image regularization, between removing noise and
preserving edges.
In a first phase, results for Δt ranging between
min { (u)) ( l u )}
f ln (2)
u 0.05 and 0.25 segs, with a fixed λ = 60 were ana-
lyzed (from previous studies we know that this is an
where f and u are the original and denoised image, acceptable value for this level of noise. See (Mota
respectively, and TV(u) is the TV of denoised et al., 2015)). It is considered a good Δt the one that
image. TV values were determined according to is effective in minimizing the TV and that can achieve
Equation 3, considering the selected 2D slice. convergence with a reasonable number of iterations.
In the next stage, taking into account Δt values cho-
m n sen, different λ values were compared. Finally, with
TV (u ) ∑∑ ( x ui , j )
2
+( y ui j )
2
(3) the most appropriate Δt and λ, filtered images were
i =1 j =1 quantitatively and quantitatively analyzed.

In Equation 3, ui,j is the intensity value of pixel

(i, j), with i = {1, …, m}, j = {1, …, n} and m × n 3 RESULTS
the image dimensions. Δx and Δy is the discretiza-
tion of the horizontal and vertical derivatives, In Figure 1 we can observe TV values for Δt ranging
respectively. between 0.05 and 0.50 segs, with a fixed λ = 60.
To solve the problem of Equation 2, for Gaus- As Δt = 0.05 segs segs is too small to converge in
sian noise, ROF proposed the gradient descent 20 iterations and values from 0.30 to 0.50 diverge,
method (Rudin et al. 1992), a procedure based we choose to continue the study with Δt = 0.25 segs.
on a parabolic equation with an artificial time This value is the one which reaches convergence
parameter. Since the problem is convex, the steady faster (iteration number 10), also allowing a high

120

VIPIMAGE15_Book.indb 120 9/8/2015 7:44:06 PM

 Figure 1. TV values for phantom unfiltered and filtered Figure 4. FWHM values for phantom unfiltered and
reconstructed images plotted as a function of iteration filtered reconstructed images plotted as a function of
number and Δt values for λ = 60. iteration number, for λ = 60 and Δt = 0.25 segs.

Figure 5. Images of a mass obtained for the unfiltered

(a) and filtered (b) images with Δ = 0.25 segs and λ = 60
(iteration 10).
Figure 2. TV values for phantom unfiltered and filtered
reconstructed images plotted as a function of iteration
number and λ values for Δt = 0.25 segs.

Figure 6. Images of a cluster of microcalcifications

obtained for the unfiltered (a) and filtered (b) images
with Δ = 0.25 segs and λ = 60 (iteration 10).

4 DISCUSSION
Figure 3. CNR values for phantom unfiltered and fil-
We presented here improved results that can be opti-
tered reconstructed images plotted as a function of itera-
tion number, for λ = 60 and Δt = 0.25 segs. mized in the sense of achieve an objective correla-
tion between the parameters and image noise level.
The combination Δ = 0.25 segs and λ = 60 works well
TV minimization. The results obtained for differ- and can achieve convergence in iteration number 10.
ent λ values, taking into account Δt = 0.25 are pre- λ = 10 doesn’t ensure the fidelity term has enough
sented in Figure 2. weight to keep the fundamental information of the
For the image quantitative and qualitative anal- initial image and λ between 30 and 50 diverge. In
ysis, λ = 60 was the value chosen and the results this way those values were automatically excluded.
obtained for CNR and FWHM are shown in From λ = 60 to 120 we choose λ = 60 because it
Figure 3 and Figure 4. For a visual inspection images allows the better CNR in 10 iterations.
of a mass and a cluster of microcalcifications of the We can see the a decrease in image TV of 40.4%
phantom are presented in Figure 5 and Figure 6. as well as the fact that this method improves CNR

121

VIPIMAGE15_Book.indb 121 9/8/2015 7:44:07 PM

 (up to 5.3%) without losing spatial resolution Jonsson, E., Huang, S.C. & Chan, T. 1998. Total varia-
(variation up to 0.40%) which means an important tion regularization in positron emission tomography.
progress in handling noise in these images. Further Joshi, S.H., Marquina, A., Osher, S.J., Dinov, I., Van
optimizations (about Δt, λ and number of itera- Horn, J.D. & Toga, A.W. 2009. Mri Resolution
Enhancement Using Total Variation Regularization.
tions needed to convergence) can greatly reduce the Proc IEEE Int Symp Biomed Imaging, 2009, 161–164.
computation time which, for this particular case Le, T., Chartrand, R. & Asaki, T. 2007. A Variational
and for 10 iterations, was approximately 7 minutes. Approach to Reconstructing Images Corrupted by
Also, as we know, λ depends on the image noise. Poisson Noise. Journal of Mathematical Imaging and
As the image noise is different at the end of each Vision, 27, 257–263.
iterations, we think that a λ variable over iterations Mota, A.M., Matela, N., Oliveira, N. & Almeida, P. 2015.
may produce even better results. Total variation minimization filter for DBT imaging.
Medical Physics, 42 (6), June.
Nett, B., Tang, J., Leng, S. & Chen, G.H. 2008. Tomo-
synthesis via Total Variation Minimization Recon-
5 CONCLUSIONS struction and Prior Image Constrained Compressed
Sensing (PICCS) on a C-arm System. Proc Soc Photo
Taking into account the results presented, we con- Opt Instrum Eng, 6913, nihpa92672.
clude that application of TV denoising to DBT Niklason, L.T., Christian, B.T., Niklason, L.E.,
images is a very promising technique and allows a Kopans, D.B., Castleberry, D.E., Opsahl-Ong, B.H.,
wide field of research in the future. Landberg, C.E., Slanetz, P.J., Giardino, A.A.,
Moore, R., Albagli, D., Dejule, M.C., Fitzgerald, P.F.,
Fobare, D.F., Giambattista, B.W., Kwasnick, R.F.,
Liu, J., Lubowski, S.J., Possin, G.E., Richotte, J.F.,
ACKNOWLEDGEMENTS Wei, C.Y. & Wirth, R.F. 1997. Digital tomosynthesis
in breast imaging. Radiology, 205, 399–406.
This work was supported in part by Fundação para Rudin, L.I., Osher, S. & Fatemi, E. 1992. Nonlinear total
a Ciência e a Tecnologia—Portugal (grant number variation based noise removal algorithms. Phys. D, 60,
SFRH/BD/43376/2008 and projects Pest-OE/ 259–268.
SAU/IU0645/2013, Pest-OE/SAU/IU0645/2014 Sawatzky, A., Brune, C., Müller, J. & Burger, M. 2009.
and PTDC/BBB-IMG/3310/2012). Total Variation Processing of Images with Poisson
Statistics. Computer Analysis of Images and Patterns,
5702, 533–540.
Sechopoulos, I. 2013. A review of breast tomosynthesis.
REFERENCES Part I. The image acquisition process. Med Phys, 40,
014301.
Ertas, M., Yildirim, I., Kamasak, M. & Akan, A. 2014. Sidky, E.Y., Duchin, Y., Pan, X. & Ullberg, C. 2011. A
An iterative tomosynthesis reconstruction using total constrained, total-variation minimization algorithm
variation combined with non-local means filtering. for low-intensity x-ray CT. Med Phys, 38 Suppl 1,
BioMedical Engineering OnLine, 13, 65. S117.
Feng, Y., Liu, P., Li, B., Yu, L. & Chen, W. 2009. Improved Timberg, P., Båth, M., Andersson, I., Svahn, T.,
reconstruction of non-Cartesian magnetic resonance Ruschin, M., Hemdal, B., Mattsson, S. & Tingberg,
imaging data through total variation minimization A. Impact of dose on observer performance in breast
and POCS optimization. Conf Proc IEEE Eng Med tomosynthesis using breast specimens. Proc. SPIE
Biol Soc, 2009, 2676–9. 6913, Medical Imaging: Physics of Medical Imaging
Food And Drug Administration. Available on: http:// (2008).
www.fda.gov/Radiation-EmittingProducts/Mammog Zhiqiang, C., Xin, J., Liang, L. & Ge, W. 2013. A limited-
raphyQualityStandardsActandProgram/FacilityCerti- angle CT reconstruction method based on anisotropic
ficationandInspection/ucm114148.htm. Last accessed TV minimization. Physics in Medicine and Biology, 58,
July 24, 2014. 2119.

122

VIPIMAGE15_Book.indb 122 9/8/2015 7:44:08 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

No-reference wavelet based Retinal Image Quality Assessment

L.S. Abdel Hamid

Electronics and Communications Department, Faculty of Engineering, Misr International University, Cairo, Egypt

A. El-Rafei
Engineering Physics and Mathematics Department, Faculty of Engineering, Ain Shams University, Cairo, Egypt

S. El-Ramly
Electronics and Communication Engineering Department, Faculty of Engineering, Ain Shams University,
Cairo, Egypt

G. Michelson
Department of Ophthalmology, Friedrich-Alexander University, Erlangen-Nuremberg, Germany

J. Hornegger
Pattern Recognition Laboratory, Department of Computer Science, Friedrich-Alexander University,
Erlangen-Nuremberg, Germany

ABSTRACT: Retinal Image Quality Assessment (RIQA) is an essential preliminarily step in Automatic
Retinal Screening Systems (ARSS) to avoid misdiagnosis of retinal disease. In this work, a no-reference
wavelet based RIQA algorithm suitable for real-time ARSS is introduced. Retinal images are classified
into good and bad quality images using two proposed metrics. These measures are extracted from the
wavelet decomposition subbands of the image to reflect sharpness and contrast. Wavelet level and regional
analyses are performed to determine the most relevant levels and retinal region to be used in the RIQA.
The algorithm is tested on two publicly available datasets with different resolutions. Area under the ROC
curve (AUC) of 1 and 0.985 are obtained for the low and high resolution datasets respectively with run-
time ranging between 0.1 and 2 seconds depending on image resolution. Comparison with other RIQA
algorithms shows superiority of the proposed algorithm in both performance and time.

Keywords: Retinal Image Quality Assessment; wavelet transform; real-time system

1 INTRODUCTION Several factors could affect the quality of retinal

images including patient movement, loss of focus
According to the World Health Organization and non-dilated pupil. Studies have shown that
(WHO), there is an estimate of 285 million people the percentage of inadequate images for screening
in 2014 suffering from blindness or low vision systems is about 10% and 20.8% for dilated and
worldwide due to different retinal diseases such as non-dilated pupils respectively (Yu et al. 2012).
glaucoma and diabetic retinopathy. Most of these Accordingly, RIQA is considered an essential
vision impairments can be avoided if the retinal preliminarily step in ARSS used to ensure that
disease is early on diagnosed and properly treated. the retinal image being processed, is of adequate
ARSS can facilitate early retinal disease detection quality to allow for correct medical diagnosis.
by being available in public places, like airports or
malls, and immediately advising patients to seek
further medical assistance if signs of a retinal dis- 2 RELATED WORK
ease are present.
Substantial research effort was put in the creation RIQA approaches can be broadly divided into
of reliable ARSS that are able to provide accurate generic and segmentation based approaches.
diagnosis in real-time settings. Results were found Generic methods were initially based on the
promising, yet strongly reliant on the quality of intuition that good retinal images had similar
the image being processed (Patton et al. 2006). histograms. Early works compared the histogram

123

VIPIMAGE15_Book.indb 123 9/8/2015 7:44:08 PM

 (Lee et al. 1999) or edge magnitude histogram in literature (Ferzli et al. 2005, Vu et al. 2012,
(Lalonde et al. 2001) of the observed image to Zhao et al. 2013), yet WT based algorithms have
that of a template histogram created from a set of scarcely been adopted for RIQA. In context of
good images. More recent generic approaches use full-reference RIQA, a wavelet based full refer-
textural features, such as Haralick features (Davis ence distortion measure, has been introduced
et al. 2009) or Run-Length-Matrix (RLM) features to quantify diagnostic information loss due to
(Fasih et al. 2014), along with histogram (statisti- image compression (Nirmala et al. 2010). In the
cal) features extracted from global or local regions measure introduced by Nirmala et al., different
in retinal images. weighs were given to different wavelet levels based
Segmentation methods are based on the on an innovative analysis performed on the rela-
intuition that good quality retinal images tend tion between retinal blood vessels and the differ-
to have more separable anatomical structures, ent wavelet levels.
specifically the blood vessels. Segmentation meth- In the context of no-reference RIQA, a wavelet
ods use either vessels segmented from the entire based focus measure defined as the mean value of
image (Usher et al. 2003, Giancardo et al. 2008, the sum of detail coefficients in Level (L1) was used
Thomas et al. 2013) or vessels segmented from along with moment and statistical measures (Veiga
the macula region (Fleming et al. 2006, Hunter et al. 2014). Veiga’s algorithm achieved high clas-
et al. 2011). The later argue that macular vessels sification results, however the system was rather
are more diagnostically important as well as being slow taking approximately 33 seconds to process
the first to become blurred with any slight image a new image making it unsuitable for real-time
degradation whereas thick blood vessels can still applications. Furthermore, the wavelet measure
be clearly visible, even when the retinal images are used by Veiga only used L1 subbands thus not fully
significantly blurred. exploiting the WT multiresolution potential.
Generic methods have the advantage of being A wavelet based RIQA algorithm that does
simple yet they completely ignore any retinal not require the presence of a reference image is
structures while segmentation methods are slow introduced in this paper. It is designed to distin-
and prone to error. In order to improve the per- guish between good and bad quality retinal images
formance, hybrid approaches were introduced that and to be suitable for real-time screening applica-
combined both generic and segmentation tech- tions. A couple of metrics computed from wavelet
niques by using features extracted from segmented features are proposed based on the intuition that
blood vessels along with histogram and textural good quality retinal images have visibly clear sharp
features (Paulus et al. 2010, Yu et al. 2012, Sevik blood vessels and adequate contrast. Two types of
et al. 2014). analyses are conducted using the proposed metrics.
Nevertheless, all previous methods dealt with First, level analysis is performed in order to deter-
images in the spatial domain. Recently, the trans- mine the most relevant wavelet level for feature
form domain has been gaining great popularity extraction and the effect of image resolution on the
in image quality assessment as it is more consist- level selection. Then, the significance of using glo-
ent with the Human Visual System (HVS), which bal features, extracted from the entire image, and
states that the eye decomposes images into low and local features, extracted from the diagnostically
high subbands and processes each differently (Gao significant center region containing the macular
et al. 2008). vessels, is analyzed. Results from these two analy-
The Wavelet Transform (WT) is a multiresolution ses are concatenated to identify the best set of fea-
technique having the advantage of being localized tures. Finally, results are compared to some of the
in both time and frequency. It is computed by existing RIQA algorithms from both performance
applying low (L) and high (H) pass filtering along and run-time perspectives.
the rows then the columns of an image giving four
subbands LL, LH, HL and HH referred to as the
3 MATERIALS AND METHODS
approximate, vertical, horizontal and diagonal
subbands respectively. The horizontal, vertical
Several retinal images datasets exist in literature,
and diagonal subbands together are known as
however only few have quality information. In this
the detail subbands of the image holding its high
work, two publicly available macula centered data-
frequency (edge) information as opposed to the
sets having different resolutions are used to test the
approximate subband being a smoothed version of
proposed algorithm:
the original image. In order to capture finer image
details, further decomposition of the approximate a. Diabetic Retinopathy Image Database
subbands can be performed. (DRIMDB) (Sevik et al. 2014): which includes
Several WT based general image quality 69 bad and 125 good quality retinal images. All
assessment algorithms have been introduced images were obtained with a Canon CF-60UVi

124

VIPIMAGE15_Book.indb 124 9/8/2015 7:44:08 PM

 Fundus Camera using 60° FOV and stored in entropy (EntrHoriz) and vertical entropy (Entrvert) as
JPEG files at 570 × 760 pixels. Most bad images given by the following equations:
in DRIMDB suffer from severe blur where thin
vessels are invisible and thick blood vessels are 1
either severely smeared or completely invisible. Entrrlevel = iz + Entr
( EntrrHori
Horiz rVVe t ) (1)
2N
b. High Resolution Fundus (HRF) Image Data-
Entrrsubband = − ∑ i =1 Ci
N 2 2
base (Thomas et al. 2013): which includes 18 bad Ci (2)
and 18 good quality retinal images taken using
a Canon CR-1 fundus camera with a 45° FOV
where N is the number of coefficients in the wavelet
having resolutions varying from 2592 × 3888
subband, Ci is the wavelet coefficient having index i
to 3456 × 5184 pixels. The bad images in HRF
and subband is the horizontal or vertical subband.
have varying degree of blur. Some images are
only slightly distorted having blurred thin ves-
3.2.2 Contrast measure
sels yet clear thick vessels, while other images
Contrast of an image can be defined as the ratio
suffering more severe blur have smeared thick
of high to low frequencies (Pu 2000). In this work,
vessels.
contrast is calculated for a specific level as the
ratio between the average of the summation of
3.1 Preprocessing the horizontal (SHoriz) and vertical (SVert) subband
coefficients to the summation of the approximate
Two preprocessing steps are performed prior to
(SApprox) subband coefficients as given by the fol-
feature extraction from the retinal images. Initially,
lowing equations:
the dark regions around the retina are cropped to
remove useless information and reduce processing (SHori SVert ) 2
time. Then, a 3 × 3 median filter is used to reduce Contrasttlevel = Horiz
iz
(3)
noise from the cropped images while preserving SApprox
edges.
∑ i = 1|
N
The green and red channels of the preprocessed i |
retinal images are used for feature extraction. Ssubband = (4)
N
However, the blue channel is excluded from the
analysis since it is under-illuminated and scarcely where N is the number of coefficients in the wavelet
contributing any relevant information. subband, Ci is the wavelet coefficient having index i
and subband is the horizontal or vertical subband.
3.2 WT features computation
3.3 Wavelet level and region analyses
Good retinal images are characterized by hav-
ing high sharpness and contrast making retinal In their work, Nirmala et al. have shown a rela-
structures and possible disease lesions clearly tion between different wavelet levels and retinal
visible and easily separable. In order to measure blood vessels. In order to find the most significant
retinal image sharpness and contrast, two wavelet wavelet level to be used for feature extraction the
based metrics are introduced. The green and red aforementioned analysis is re-implemented where
channels of the image are decomposed into five small sections from different retinal images, having
levels using daubechies4 ‘db4’ wavelet. Then, the either thick or thin vessels, are extracted and coef-
measures are calculated from the coefficients of ficients of subsequent detail subbands are zeroed.
the horizontal and vertical detail subbands. These Then, the Mean Structural Similarity Index Meas-
subbands are found to carry more significant ure (Wang et al. 2004) is calculated to measure
information related to the retinal vessels than the which level have more effect on the retinal vessels.
diagonal subband. Another experiment is performed to validate the
results from the previous analysis in which the pre-
3.2.1 Sharpness measure sented sharpness and contrast features from each
Sharp blood vessels in good quality images cor- wavelet level are used for classification and results
respond to more information present in the detail from classification using each of the different lev-
subbands compared to bad quality images. An els are compared.
information theoretic metric that reflects the infor- Moreover, global features are calculated from
mation content in the detail subbands is proposed the entire image while local features are computed
to be used with retinal images. This metric is the from the center region. The center region is cho-
wavelet Shannon entropy which relates to the sen to include the macular vessels which are diag-
retinal image sharpness. The entropy of each nostically important as well as being the first to
level is calculated as the average of the horizontal be affected by any slight blur (Hunter et al. 2011).

125

VIPIMAGE15_Book.indb 125 9/8/2015 7:44:08 PM

 This center region is defined by dividing the image the SVM classifier gave an AUC larger than 0.999
horizontally into four equal sections then regard- for the individual levels L2, L3, or L4 and for both
ing the two center sections together as the center global and local features. A perfect differentia-
region. The choice of a relatively wide center region tion between good and bad images (AUC = 1) was
allows taking into account any slight vertical shift achieved when using L3 (for both global and local
in the macula centered retinal image. Results from features). In (Sevik et al. 2014), the authors used
using global and local features in classification are 144 hybrid features for RIQA of the DRIMDB
compared. images resulting in F1-score of 0.996, 0.965 and
For all analyses in this subsection, the K-Nearest 0.842 for classifying the images into good, bad and
Neighbor (KNN) classifier is utilized as it can be outliers (images having no retinal information)
easily tuned and found to provide consistent results. respectively.
Conclusions from the feature analyses are used to
determine the most relevant wavelet level and type
4.2 HRF classification results
(global or local) of the features. Finally, the clas-
sification using the selected set of features is fine The most significant levels to thin and thick blood
tuned by the SVM classifier to improve the results. vessels were found to be L3 and L4 respectively
by applying the method in (Nirmala et al. 2010)
on several HRF images. Classification using the
4 RESULTS sharpness and contrast features from different
levels led to best results for L3 (AUC = 0.917) fol-
In this section, the results from features analyses lowed closely by L4 (AUC = 0.907). Unlike the
for both DRIMDB and HRF datasets are results from the DRIMDB images, classification
demonstrated. Performance and run-time of the using global and local features demonstrated better
proposed algorithm are evaluated and compared accuracy for using local features, for example L3
to other RIQA techniques in literature. local features gave an AUC of 0.944 while global
Feature extraction computations were performed features resulted in AUC of 0.917. Results from the
under the MATLAB (Mathworks, Inc., Natick, MA, HRF dataset are summarized in Table 2.
USA) development environment. Classification As evident from the feature analysis results, the
results were obtained using 10-fold cross validation feature choice was more critical for the HRF dataset
in the Weka platform (Hall et al. 2004). than the DRIMDB dataset. For final classification,
wavelet level L3 was selected as it had the largest
correlation to the thin retinal vessels. In addition,
4.1 DRIMDB classification results
features from local region were utilized to prevent
Performing analysis similar to that in (Nirmala the relatively clear thick vessels from overweighing
et al. 2010) on several DRIMDB images showed the noticeably blurred thin vessels in the slightly
that the levels most relevant to thin and thick vessels blurred images. The SVM classifier with local
were L2 and L3 respectively. Classification using features from L3 resulted in an AUC of 0.985.
proposed sharpness and contrast features for sepa- Results from the proposed algorithm on the HRF
rate levels gave best results for L3 (AUC = 0.998) dataset were compared to other RIQA techniques
followed closely by L2 (AUC = 0.987) and L4 from literature and results are shown in Table 3.
(AUC = 0.981). Results for classification using
global against local features were very close for
all levels with a maximum AUC difference of Table 2. Area under the receiver operating characteristic
0.02. Results from the DRIMDB dataset are curve (AUC) for the HRF database.
summarized in Table 1.
Feature analysis results using the DRIMDB L1 L2 L3 L4 L5
dataset, showed that high classification results
Global 0.799 0.778 0.917 0.907 0.62
were obtained regardless of the selected level and
Local 0.938 0.87 0.944 0.904 0.799
type of features (global or local). Final results from

Table 1. Area under the receiver operating characteristic Table 3. Comparison of the proposed algorithm with
curve (AUC) for the DRIMDB database. existing RIQA techniques.
L1 L2 L3 L4 L5 Proposed Fasih Thomas Davis
Algorithm 2014 2013 2009
Global 0.973 0.987 0.998 0.981 0.876
Local 0.963 0.995 0.998 0.998 0.895 AUC 0.985 0.796 0. 883 0.852

126

VIPIMAGE15_Book.indb 126 9/8/2015 7:44:08 PM

 Two generic RIQA algorithms relying on local higher order wavelet decomposition levels become
features were re-implemented by the authors for more significant comparison to lower resolution
the sake of comparison, namely those proposed images.
in (Davis et al. 2009) and (Fasih et al. 2014). Davis In order to further explore this conclusion,
et al. calculated statistical, textural and sharpness the HRF dataset was resized to 1024 × 1024 and
features from seven local regions covering the entire 512 × 512 pixels and level analysis was repeated
retinal image. Fasih et al. used textural and sharp- for both cases. Classification results, summarized
ness features extracted only from the optic disc and in Table 3 and Table 4, confirmed the previous
macula regions. The results reported from the seg- conclusion. For example, when using local features
mentation based approach in (Thomas et al. 2013) from L3, the AUC was 0.944. However, the AUC
applied to the HRF dataset were also included in the decreased to 0.923 and further to 0.764 for the
comparison. Thomas et al. computed a global qual- resized images at 1024 × 1024 and 512 × 512 pixels
ity score based on the segmented blood vessel tree. respectively.
Feature analysis also indicated that the rel-
evance of wavelet levels and type of feature (global
4.3 Run-time results
or local) was dependent on the type of blur in the
The proposed method is intended for real-time bad quality retinal images. For images with severe
ARSS, therefore immediate results reporting is blur, as in the DRIMDB dataset, high classifica-
essential. The time required to process a new image tion accuracy was achieved regardless of the wave-
using the proposed algorithm was recorded and let level and type of feature used. An AUC larger
found to be less than 0.1 second for small resolu- than 0.96 was obtained from levels L1 to L4. The
tion images (DRIMDB of 570 × 760 pixels), reach- difference in AUC between the usage of global and
ing a maximum of ∼2 seconds for high resolution local features was less than 0.02 for all mentioned
HRF images (3456 × 5184 pixels). levels. These values could be explained by the
The run-time of the re-implementation of the severe blur in this dataset. Both the thick and thin
algorithms of Davis et al. and Fasih et al. was vessels are affected by the high degree of blur in
found to be 7–10 seconds and 9–20 seconds per these images. This in turn is reflected on the differ-
image respectively for the HRF dataset depending ent wavelet levels carrying information related to
on the image size. All the previous time analysis vessels of varying thickness. Moreover, since both
was computed on the same PC using MATLAB. thin and thick vessels are significantly affected by
the severe blur, both global and local features were
equally significant.
5 DISCUSSION On the other hand, for slightly blurred images in
the HRF dataset, classification results were found to
A no-reference RIQA algorithm using wavelet be dependent on the wavelet level and feature type
based features is proposed. Sharpness and contrast (global or local). Best results were achieved when
measures extracted from the detail wavelet sub- using L3 local features (AUC = 0.985). The slight
bands were used for classification of retinal images blur in the bad images mainly affected the thin ves-
into good and bad quality images. Evaluation of sels present in the local macular region. Wavelet
the introduced method showed high efficiency in level L3 was found to have the highest sensitivity to
determining the quality of the images regardless of
their resolution and degree of blur.
In order to find the most relevant features to be Table 4. AUC for the HRF dataset resized to
used in classification, wavelet level and feature type 1024 × 1024 pixels.
(global or local) analyses were performed. Level
analysis showed that there is a correlation between L1 L2 L3 L4 L5
image resolution and level choice. Nirmala et al. Global 0.781 0.892 0.944 0.92 0.787
have shown that for 512 × 512 pixel retinal images, Local 0.96 0.954 0.923 0.79 0.778
L2 and L3 detail subband coefficients had more
influence on thin and thick blood vessel respec-
tively. Re-implementation of Nirmala et al.’s pro-
cedure for the DRIMDB dataset (570 × 760 pixels) Table 5. AUC for the HRF dataset resized to
gave similar results. On the other hand, for the 512 × 512 pixels.
high resolution HRF dataset L3 and L4 detail sub-
band coefficients were found to have more influ- L1 L2 L3 L4 L5
ence on thin and thick blood vessel respectively. Global 0.907 0.886 0.836 0.611 0.423
Accordingly, it can be concluded from the level Local 0.96 0.926 0.765 0.472 0.481
analysis results that for higher resolution images,

127

VIPIMAGE15_Book.indb 127 9/8/2015 7:44:09 PM

 slight blur as the analysis showed that it is the most relevant wavelet level for classification was depend-
related level to thin vessels. Additionally, the small ent on the image resolution. The resizing of high
percentage of the thin vessels to the total vessel net- resolution images changed the order of the sig-
work makes it easily overweighed by the visible thick nificant wavelet levels as found from the second
vessels. Thus, global features could lead to wrong analysis. Then, it was indicated in the third analy-
classification in lightly blurred images. This explains sis that local features extracted from regions with
the higher accuracy associated with the usage of thin vessels performed better than global features
local features for the high resolution dataset. for slightly blurred images.
RIQA of the HRF dataset is considered specifi- The conclusions from the previous analyses were
cally challenging due two reasons: 1) The slight blur incorporated in the algorithm to decide on the
affecting only the fine details (thin macular vessels) final set of features. The proposed algorithm was
of the retinal images are hard to capture using tradi- tested against other RIQA methods showing supe-
tional RIQA algorithms. 2) The high resolution of rior classification performance. Also, as a result
the HRF dataset images with some images having of the very limited number of computationally
a size of 3456 × 5184 pixels (which is almost 6 times inexpensive features that were carefully selected,
the size of images from DRIMDB) make feature cal- the proposed algorithm was distinguishably faster
culations computationally expensive. Nevertheless, than the other RIQA algorithms in comparison for
the proposed sharpness and contrast wavelet based both low and high resolution images.
measures extracted from L3 of the local central Wavelet domain’s consistency with the HVS
region were found to efficiently capture the slight method of image analysis makes it a valuable tool
blur affecting the thin macular vessels in the bad for image quality evaluation. Specifically, RIQA
images. Thus, achieving results that surpassed both wavelet based algorithms have the potential to
generic and segmentation based techniques imple- overcome the limitations of both generic and seg-
mented for the HRF dataset. The accuracy of the mentation based algorithms while achieving supe-
wavelet based algorithm exceeded the other RIQA rior results.
methods by at least 10%.
The reduced number of features and the
simplicity of their calculation from the wavelet REFERENCES
subbands had a significant effect on the algorithm’s
speed. The introduced RIQA algorithm was found Davis, H., S. Russell, et al. (2009). Vision-based, real-
to be 5 times and 10 times faster than the methods time retinal image quality assessment. Proc. of IEEE
by Davis et al. and Fasih et al. respectively for International Symposium on Computer-Based Medical
higher resolution images. This speed ratio largely Systems (CMBS), pp. 1–6.
increased in case of lower resolution images. This Fasih, M., J.M. Langlois, et al. (2014). Retinal image
can be attributed to the computationally extensive quality assessment using generic features. Proc. of
SPIE 9035, Medical Imaging, International Society for
textural feature used in these methods. Optics and Photonics, pp. 90352Z–90352Z−7.
Fleming, A. D., S. Philip, et al. (2006). Automated assess-
ment of diabetic retinal image quality based on clarity
6 CONCLUSIONS and field definition. Investigative Ophthalmology &
Visual Science, 47(3): 1120–1125.
Automatic screening systems for retina diseases Ferzli, R., L.J. Karam, et al. (2005). A robust image
should provide the users with an immediate accurate sharpness metric based on kurtosis measurement
report. RIQA associated with ARSS is crucial and of wavelet coefficients. Proc. of Int. Workshop on
should operate in real-time and perform efficiently Video Processing and Quality Metrics for Consumer
Electronics.
while taking into account the lack of a reference Gao, Z. & Y.F. Zheng, (2008). DWT-Based Image Quality
image. In this paper, a wavelet based algorithm for Metric. IEEE Transactions on Circuits and Systems for
retinal image quality grading is designed to satisfy Video Technology, 18(7): 910–922.
the aforementioned requirements imposed by the Giancardo, L., E. Chaum, et al. (2008). Elliptical Local
screening application. The algorithm exploited the Vessel Density: A Fast and Robust Quality Metric for
relation between the clarity of the retinal blood ves- Fundus Images. Proc. of International Conference of
sels and the information content within the image the IEEE Engineering in Medicine and Biology Society
wavelet levels. Metrics are proposed to address (EMBS), pp. 3534–3537.
the sharpness and contrast of the images. These Hall, M., E. Frank, et al. (2009). The WEKA data min-
ing software: an update. ACM SIGKDD explorations
measures are calculated directly from the multi- newsletter, 11(1): 10–18.
resolution decomposition and used as features for Hunter, A., J.A. Lowell, et al. (2011). An automated reti-
classifications. Therefore, no reference is required. nal image quality grading algorithm. Proc. of Interna-
Three comprehensive analyses were performed. tional Conference of the IEEE Engineering in Medicine
In the first analysis, it was shown that the most and Biology Society (EMBC), pp. 5955–5958.

128

VIPIMAGE15_Book.indb 128 9/8/2015 7:44:09 PM

 Lalonde, M., L. Gagnon, et al. (2001). Automatic visual Usher D.B., M. Himaga, et al., (2003). Automated assess-
quality assessment in optical fundus images. Proc. of ment of digital fundus image quality using detected
Vision Interface, Vol. 32, pp. 259–264. vessel area. Proc. of Medical Image Understand-
Lee, S.C., & Y. Wang (1999). Automatic retinal image ing & Analysis: British Machine Vision Association
quality assessment and enhancement, Medical Imag- (BMVA), pp. 81–84.
ing International Society for Optics & Photonics, pp. Veiga, D., C. Pereira., et al. (2014). Quality evaluation of
1581–1590. digital fundus images through combined measures.
Nirmala, S.R., S. Dandapat, et al. (2010). Wavelet Journal of Medical Imaging, 1(1):014001–014001–9.
weighted blood vessel distortion measure for retinal Vu, P.V. and D.M. Chandler (2012). A Fast Wavelet-
images. Biomedical Signal Processing and Control, Based Algorithm for Global and Local Image Sharp-
5(4): 282–291. ness Estimation. IEEE Signal Processing Letters,
Patton, N., T.M. Aslam, et al. (2006). Retinal image anal- 19(7): 423–426.
ysis: concepts, applications and potential. Progress in Wang, Z., A.C. Bovik, et al. (2004). Image Quality
Retinal and Eye Research, 25(1): 99–127. Assessment : From Error Visibility to Structural Simi-
Paulus, J., J. Meier, et al. (2010). Automated quality larity. IEEE Transactions on Image Processing, 13(4):
assessment of retinal fundus photos. International 600–612.
journal of computer assisted radiology and sur- Yu, H., C. Agurto, et al. (2012). Automated image qual-
gery, 5(6): 557–564. ity evaluation of retinal fundus photographs in dia-
Pu, T., & G. Ni (2000). Contrast-based image fusion betic retinopathy screening. Proc. of IEEE Southwest
using the discrete wavelet transform. Optical Engi- Symposium on Image Analysis and Interpretation, pp.
neering, 39(8): 2075–2082. 125–128.
Şevik, U., C. Köse, et al. (2014). Identification of suit- Zhao, H., B. Fang., et al. (2013). A no-reference image
able fundus images using automated quality assess- sharpness estimation based on expectation of wave-
ment methods. Journal of Biomedical Optics, 19(4): let transform coefficients. Proc. of IEEE International
046006–046006. Conference Image Processing (ICIP), pp. 374–378.
Thomas, K., A. Budai, et al. (2013). Automatic No-
Reference Quality Assessment for Retinal Fundus
Images Using Vessel Segmentation. Proc. of IEEE
International Symposium on Computer-Based Medical
Systems (CMBS), pp. 95–100.

129

VIPIMAGE15_Book.indb 129 9/8/2015 7:44:09 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Vessel segmentation of retinal images with fuzzy morphology

P. Bibiloni, M. González-Hidalgo & S. Massanet

Department of Mathematics and Computer Science, University of the Balearic Islands, Palma de Mallorca, Spain

ABSTRACT: Distinguishing vessels in a retinal image is a solid first step towards a computer-aided
diagnosis of several medical conditions. Mathematical morphology provides simple yet powerful trans-
formations to process images. Extended with fuzzy techniques, it achieves the ability to handle the uncer-
tainty present in medical imaging, both in the images and in the expert diagnosis. Within this paradigm,
we propose a method that leverages the geometric and photometric features that characterize vessels as
such. It is tested with two databases, DRIVE and STARE. The first promising results outperform some
of the state of the art methods with which it was compared, positioning it among the most competitive
techniques.

1 INTRODUCTION segmentation of the vessel tree, some features of the

veins and arteries can be estimated (e.g. their width
Vessel segmentation in retinal images consists on and tortuosity) and other manifestations can be
specifying which pixels of an image would be consid- segmented out more easily. The automatic segmen-
ered by an expert as part of a vessel. These images tation of blood vessels is, therefore, the first step to
include part of the retina (light-sensitive tissue), infer more information towards a computer-aided
the optic disc (termination of the optic nerve), diagnosis of a condition with noticeable effects on
the macula (area of the retina with high-density the retina. A more accurate and sistematic diag-
of cone cells) and the blood vessels in this region, nosis of diabetic retinopathy, glaucoma and other
besides a number of manifestations that may indi- conditions may be possible due to image process-
cate a medical condition. ing of eye-fundus images.
Given their nature, these images may present a Vessel segmentation in eye-fundus images is a
variety of particular artifacts and noise. Although problem that researchers have faced with a wide vari-
an expert can easily distinguish most artifacts from ety of approaches [5]. Thus, a number of techniques
actual manifestations, it is a computationally com- to segment vessels have been already published. In
plex task. The image as a whole should be proc- 1999, Martínez-Pérez et al. designed a method that
essed to convey as much information as possible. extracted features from the second-derivative of the
For instance, the position of the vessel tree may pixel’s intensity, obtained as the convolution of the
help to flag a dark region as the macula, which is image with derivatives of Gaussian functions. They
located somewhat in the middle of that structure, merged these features obtained at several scales to
as shown in Figure 1. then apply a local region-growing procedure. [10].
Some diseases may be detected by means of Later in 2000, Hoover et al. published a vessel seg-
examining an eye-fundus retinal image. Given the mentation method using the database they created,
STARE, which is publicly available [8]. They used a
matched filter (correlation of the image with a tem-
plate to detect its presence) and a local thresholding
process. Chanwimaluang and Guoliang presented
a method that also uses matched filtering, to later
apply a threshold based on local entropy [3]. In 2004,
the also publicly-available database DRIVE was
presented by Niemeijer et al., including a method in
which pixel-wise features were obtained as the out-
put of linear filters (Gaussian and its derivatives),
and further processed with a kNN classifier [11].
Figure 1. Image number 14 of the STARE database, Extending this work, Staal et al. presented another
with annotations. The macula is hardly visible, whereas method in which additional features are considered
the illumination is noticeably non-uniform. after detecting the main ridges of the image [13].

131

VIPIMAGE15_Book.indb 131 9/8/2015 7:44:09 PM

 Using fuzzy techniques, Ayala et al. published a OC ,I ( , ) DC (EI ( A, B ), B ),
method that uses averages of fuzzy sets to project
CC ,I ( , ) EI ( DC ( A, B ), B ),
grayscale images into binary estimations [1]. Like-
wise, Sun et al. presented a method based on multi- THOOC ,I ( A, B ) C , I ( A, ),
scale morphology, fuzzy filtering and the watershed THCC ,I ( , B ) A, ) − A.
, I ( A,
transform [14].
Although some methods use fuzzy techniques
As expected, this definition of fuzzy dilation
and others are based on morphological operators,
expands the foreground object, whereas the ero-
there are no studies involving eye-fundus vessel
sion diminishes it. The opening removes sharp or
segmentation with fuzzy mathematical morphology,
thin regions of the foreground object, whereas the
as far as we are aware. Such a paradigm perfectly
closing operation enlarges the foreground object
suits the need of handling uncertainty in medical
by filling sharp or thin areas of the background.
images, present in both the pixels’ value and the
The Top-Hat transformations enhance, therefore,
expert diagnosis annotations.
sharp or thin regions. These last operators are of
interest due to their characteristic behaviour—they
have already been used to process noisy biomedical
2 VESSEL SEGMENTATION METHOD
images [6]. Specifically, we use a Top-Hat by Clos-
ing, to extract the dark, thin objects that represent
In this section, the method is presented following
vessels in eye-fundus images.
the description of the vessels’ features. Firstly,
We will consider that B(0) = 1. Besides, our
we give an overview of fuzzy mathematical
conjunctor C will be a t-norm and I its residu-
morphology, which constitutes the core of our
ated implication, so the following inclusions will
algorithm.
hold [9]:

2.1 Fuzzy mathematical morphology EI ( ) OC ,I ( A, B ) ⊂ A

⊂ CC ,I ( , ) DC ( A, B ).
Mathematical morphology is a paradigm to proc-
ess images by means of basic yet powerful transfor-
mations. It is based on simple and local operations Depending on the choice of the conjunctor—
that, when properly combined, transform images which would also affect its residuated implicator—
to extract borders, skeletons or more complex different results are obtained [7]. One of the focus of
features. our study is pinpointing which are the best settings
The mathematical morphology paradigm that regarding the problem of vessel segmentation.
we use is the one introduced by De Baets [4], which
is based on fuzzy operators. A conjunctor is defined 2.2 Modelization of vessels
as a function C : [ 0,1] × [ 0,1] ⎯→ [ 0,1] increasing in
both variables such that C ( ) C ( ) = C ( ) = 0 In order to construct a method that segments ves-
and C( ) = 1, whereas an implicator sels out we should identify which features charac-
I : [ 0,1] × [ 0,1] ⎯→ [ 0,1] is monotonically decreas- terize a pixel as part of a vessel. We include both
ing in the first variable and increasing in the sec- geometric and photometric features—so we can
ond one, and fulfills I ( ) I ( ) = I ( ) = 1 and infer information regarding the shape of objects
I ( ) = 0 [2]. Let B be a two-dimensional struc- and their colour components. Likewise, it is impor-
turing element (i.e. a template), and A an image. tant to describe the noise found in these images so it
Then, the dilation D and the erosion E are defined can be properly faced.
as follows: Vessels in eye-fundus images, like the one shown
in Figure 1, are thin curvilinear structures. They
are darker regions than the background surround-
DC ( A B )(y
( y) C (B(x y ) A( x )) ,
x ing them. Besides, they form a tree-like structure:
EI ( A B )(y
( y) i f I (B(x y ) A( x )) , a connected region with several bifurcations. Their
x width is monotonic, and tends to grow towards
a central region, in which the optic disk is found.
where the variable x ranges over the values in which These features are considerably good indicators in
both A(x) and B(x − y) are defined. order to discriminate vessels across a great number
From these two basic operations and the of images.
reflected structuring element, B (x) = B(−x), we There are distinct types of elements that may
can construct the opening O , the closing C and the interfere with the detection of the vessels. Insuf-
top-hat transformations by opening THO and by ficient or non-uniform illumination always causes
closing THC : fluctuations in the image’s bright, and sometimes

132

VIPIMAGE15_Book.indb 132 9/8/2015 7:44:09 PM

 big rounded shapes may also appear. Big vessels, through f(x) = ( / 255)k . The equalization factor
specially arteries, present another illumination k is chosen so that all pixels are adequately distrib-
problem, central vessel reflex, consisting on a uted in [0, 1], providing a well contrasted image.
lighter area along the interior of the vessel. The 3. Top hat by closing. To take advantage of the fact
optic disk is a very bright area, roughly circular. that vessels are thin regions which are darker
Moreover, several pathologies may also appear: than their surroundings, a top hat by clos-
microaneurysms and other types of hemorrhages ing THC is the perfect choice to extract them
(darker redish areas), hard exudates (yellow points from both the background and the big artifacts
caused by lipid residues), cotton wool spots (yellow/ present on the image.
gray regions produced by damage to nerve fibers) 4. Border removal. The exterior of the original
and several others. image can not contain any vessel, but the top
hat transform may have produced some false
2.3 High-level description positives there. This background area is esti-
Our method leverages the features that character- mated and then removed.
ize a vessel as such to create an automatic segmen- 5. Filtering. Although big artifacts have been rejected,
tation. It is composed of several stages, each one small ones may still be present. To remove isolated
of them with a well-defined behaviour towards points, a filtering step is applied. Given a point x0,
taking advantage of one or more of those features. its 8-neighbourhood x1, …, x8 is considered. The
The steps of our algorithm, which can be visually filter then outputs: H(x) = min(x0, max(x1, …,
observed in Figure 2, work as follows: x8)). After applying this operation, the estimation
is normalized to the range [0, 1].
1. Green channel. From the original RGB image, 6. Dilation. The top-hat transform slightly reduces
only the third channel is considered, since it is the width of the vessels. A dilation step is thus
the one which conveys the most information needed to cancel this effect out.
about the vessels’ location. 7. Hysteresis. To further leverage the model of the
2. Fuzzification. Each pixel, originally within the set vessels, a hysteresis step is applied so that con-
{0, 1, …, 255} is transformed to the interval [0, 1] nected components are more likely detected.
Two thresholds are specified, 0 < tlow ≤ thigh ≤ 1. A
pixel will be classified as “vessel” as long as it is
placed in a connected region of pixels greater or
equal than tlow such that at least one of them is
also greater or equal than thigh.

3 EXPERIMENTATION RESULTS

3.1 Input images

The two eye-fundus databases used are DRIVE and
STARE, which have low levels of noise but a wide
range of manifestations. DRIVE [13] is a database
that contains 40 colour images of mostly healthy
patients, each of them with a resolution of 565 ×
584 pixels, along with one or two expert segmenta-
tions. The STARE database [8] contains 397 colour
images of 605 × 700 pixels, 20 of which have been
manually segmented twice. They are very appropri-
ate to test and optimize vessel segmentation meth-
ods: the low noise that they present reduces the
difference between distinct manual segmentations,
which is a recurrent problem to robustly verify the
correctness of a method.

3.2 Implementation details

The method has been fully implemented in
Figure 2. Step by step visualization of our method, with MATLAB® 2014a and in the C programming
the second image of the DRIVE database as input. language.

133

VIPIMAGE15_Book.indb 133 9/8/2015 7:44:11 PM

 There are several parameters that can be
adjusted to obtain the optimum result—including
the structuring element and the t-norm used to
perform the Top Hat transform or the hysteresis
thresholds tlow thigh . They have been optimized for
each of the databases independently, since each of
them presents different characteristics. Besides, a
batch optimization procedure has been performed
to avoid overfitting.
The best equalization factor k varies from 0.51
to 0.61, depending on the database. The Top-Hat
transform uses a 7 × 7 flat structuring element and
the Mayor-Torrens t-norm with λ = 0.7:

⎧ y λ ), if ( x y ) [ λ ]2 ,
λ
TMT ( x y ) = ⎨max( x
⎩max( x y ), otherwise .

The dilation, on the other hand, is instanti-

ated with a 3 × 3 gaussian-shaped structuring
element (with σ = 3/4) and the Product t-norm,
TP(x, y) = x ⋅ y. Lastly, the hysteresis thresholds
also vary: tlow = 0.12 and thigh ∈ [0.24, 0.26].

3.3 Comparison and objective metrics

Our method has been tested and compared with
other state-of-the-art algorithms. A visual com-
parison can be observed in Figure 3, in which the
results of several methods are shown for compari-
son. The image chosen is the 255-th of the STARE
database, which corresponds to a healthy patient Figure 3. Comparison of different methods with the
255-th sample of the STARE database. The detail cor-
[8] and presents uniformly distributed vessels.
responds to the positions (x, y) ∈ [300, 500]2.
The good contrast that it presents highlights thin
vessels, which are therefore tagged in the ground
truth—in contrast with high-noise images, in which
only the widest vessels can be identified. where TP(x) are the true positives of our estima-
Figure 3 remarks one of the strengths of our tion x, FP(x) the false positives and FN(x) the false
method: it is very sensitive to small and thin ves- negatives. It is sensitive to both types of errors,
sels, which are usually the most difficult ones to unlike other error metrics such as precision—which
detect. is not affected by false negatives—or recall—which
However, in order to properly compare methods, is independent of the number of false positives.
objective indicators of their correctness are essen- In Figure 4, it is depicted a comparison of the
tial. The ground truth is a classification of each distributions of the F1 Score by different meth-
pixel as either “vessel” (positive) or “non vessel” ods throughout all the test images of the DRIVE
(negative). An automatic vessel segmentation aims database.
to mimic this classification. We will consider two The second metric that we use is Accuracy due
indicators based on these two pixel-wise masks: the to its extensive use in the related literature, which
F1 Score and the Accuracy. helps comparison between different methods, and
The F1 Score is an appropriate metric due to its due to its simplicity, which provides an intuitive
good behaviour against different deviations from insight regarding the behaviour of different meth-
the ground truth. It is the harmonic mean of preci- ods. It is defined as the rate of pixels correctly
sion (rate of selected pixels that correspond to a classified:
vessel) and recall (rate of vessel pixels that are actu-
ally selected). It is computed as: TP ( x ) TN
T (x)
Acc( x ) = .
#{x}
2 ⋅ TP ( x )
F1( x ) = .
2 ⋅TP(x
TP ( x ) FP F ( x ) + FN ( x ) where #{x} is the total number of pixels.

134

VIPIMAGE15_Book.indb 134 9/8/2015 7:44:13 PM

 Figure 4. Boxplot showing the distributions of different methods with the 20 images of test of the DRIVE dataset.

We remark that only around 10%–15% are “ves- Table 1. Comparison of the average of the F1 Score and
sel” pixels, so the trivial estimation that considers Accuracy of all possible images in both databases, where
each pixel as “non-vessel” would obtain an Accu- both the results and the ground truth were available.
racy of 0.85–0.90.
DRIVE STARE
A statistical test is needed to compare two meth-
ods with each other. Given n images, each method F1 Acc F1 Acc
Mi provides a series of results ri (ri( ) , …, ri( n ) ) ,
which can be interpreted as samples of a probabil- 2nd expert 0.788 0.964 0.740 0.952
ity distribution. We use the Wilcoxon signed-rank Martínez [10] 0.697 0.950 – –
test to contrast whether the mean of two paired Niemeijer [11] 0.750 0.960 – –
distributions differ or not. It is robust against non Hoover [8] – – 0.651 0.946
normal data, which is our case. The significance Chanwim. [3] 0.670 0.942 0.638 0.935
level is set to α = 0.05. Staal [13] 0.764 0.962 – –
For each database, the method has been executed Soares [12] 0.748 0.960 0.698 0.957
and tested against the ground truth provided. In Our method 0.733 0.958 0.689 0.960
the DRIVE database, the 20 test images have been
considered for experimenting purposes, using the
other 20 images as training samples when needed.
The 20 manually segmented images of the STARE of our method. However, with the STARE data-
database are considered, using the strategy leave- base, which presents images with more noise and
one-out1 when training is required. The mean of lower contrast, our method obtains extremely good
the metrics of all samples is showed in Table 1. segmentations, with a F1 Score only improved—
Along the 20 test images of the DRIVE data- without statistical significance—by Soares et al. It
base, the method provides a mean F1 Score of also yields the best Accuracy result, which is signifi-
0.733 and an Accuracy of 0.958. The 20 STARE catively better than all the other methods except for
images yield an average F1 Score of 0.689 and an the results by Soares et al. Surprisingly, the Accu-
Accuracy of 0.960. racy result is even better than that of the second
As we can see in Table 1, the metrics of our expert segmentation—being the only method pro-
method are in all cases similar to the best ones. viding a better result than the second expert. This
With respect to the DRIVE database, our method is due to the fact that our method and the second
is statistically better than the one presented by ground truth differ from the first manual segmenta-
Chanwimaluang, although the method presented by tion in different ways: the F1 Score of the latter is
Staal et al. has significative better results than those clearly above the one of the former.
The methods by Chanwimaluangand Guoliang
[3] and Soares [12] were executed using the source
code published by their authors. Hoover et al.
1
Leave-one-out consists on computing each estimation [8] published the segmentations of their method
by training a new classifier with the remaining samples. with the STARE database, and Staal et al. did so

135

VIPIMAGE15_Book.indb 135 9/8/2015 7:44:14 PM

 with the rest of the methods compared, including [2] G. Beliakov, A. Pradera, and T. Calvo. Aggregation
theirs. functions: A guide for practitioners, volume 361.
In a computer with an Intel®i5 at 3.10 Ghz Springer, 2007.
with 16 GB of RAM, our method lasts an aver- [3] T. Chanwimaluang and G. Fan. An efficient blood
vessel detection algorithm for retinal images using
age of 0.48 seconds after having being optimized. local entropy thresholding. In Proceedings of the 2003
In contrast, the method by Chanwimaluang needs International Symposium on Circuits and Systems,
1.76 seconds in average, whereas the one by Soares ISCAS’03, volume 5, pages V–21. IEEE, 2003.
uses 22.59 seconds per sample after a learning [4] B. De Baets. A fuzzy morphology: a logical approach.
stage of 391.12 seconds. The rest of segmenta- In Uncertainty Analysis in Engineering and Sciences:
tions were not computed on the same machine, so Fuzzy Logic, Statistics, and Neural Network Approach,
their time measurements are not comparable. Just pages 53–67. Springer, 1998.
for the sake of completeness, and remarking that [5] M.M. Fraz, P. Remagnino, A. Hoppe, B. Uyyanon-
this measure is not comparable to the others, the vara, A.R. Rudnicka, C.G. Owen, and S.A. Barman.
Blood vessel segmentation methodologies in retinal
method by Staal et al. lasts around 15 minutes on images–a survey. Computer methods and programs in
an Intel® Pentium III PC at 3.10 Ghz with 1 GB biomedicine, 108(1):407–433, 2012.
of RAM [13]. [6] M. González-Hidalgo and S. Massanet. A fuzzy
mathematical morphology based on discrete tnorms:
fundamentals and applications to image processing.
4 CONCLUSIONS AND FUTURE WORK Soft Computing, 18(11):2297–2311, 2014.
[7] M. Gonzalez-Hidalgo, S. Massanet, A. Mir, and D.
Emerging paradigms and approaches can be Ruiz-Aguilera. On the choice of the pair conjunction-
applied to design finer-grained methods, so their implication into the fuzzy morphological edge detec-
tor. IEEE Transactions on Fuzzy Systems, 2013.
performance can potentially outstand current [8] A. Hoover, V. Kouznetsova, and M. Goldbaum.
state-of-the-art solutions. Although the perform- Locating blood vessels in retinal images by piece-
ance of our method is promising, some more wise threshold probing of a matched filter response.
work can be done to take full advantage of the IEEE Transactions on Medical Imaging, 19(3):
features of the vessel structure. A line-enhancement 203–210, 2000.
procedure could be implemented, and noise filters [9] E.E. Kerre and M. Nachtegael. Fuzzy techniques
may be used so that the method is even more sound in image processing, volume 52. Springer Science &
against highly-interfered images. Business Media, 2000.
Taking advantage of technological improve- [10] M.E. Martínez-Pérez, A.D. Hughes, A.V. Stanton,
S.A. Thom, A.A. Bharath, and K.H. Parker. Retinal
ments, like better-quality eye-fundus cameras or blood vessel segmentation by means of scale-space
faster processors, is of paramount importance. analysis and region growing. In Medical Image
Optimizing our method to process high-resolution Computing and Computer-Assisted Intervention–
images is the next step to verify that our methodol- MICCAI’99, pages 90–97. Springer, 1999.
ogy can face a variety of situations. [11] M. Niemeijer, J. Staal, B. van Ginneken, M. Loog,
and M.D. Abramoff. Comparative study of reti-
nal vessel segmentation methods on a new pub-
ACKNOWLEDGMENTS licly available database. In Medical Imaging 2004,
pages 648–656. International Society for Optics and
Photonics, 2004.
This work was partially supported by the project [12] J.V. Soares, J.J. Leandro, R.M. Cesar Jr, H.F.
TIN 2013-42795-P. P. Bibiloni also benefited from Jelinek, and M.J. Cree. Retinal vessel segmentation
the fellowship FPI/1645/2014 of the Conselleria using the 2-D Gabor wavelet and supervised clas-
d’Educació, Cultura i Universitats of the Govern de sification. Medical Imaging, IEEE Transactions on,
les Illes Balears under an operational program co- 25(9):1214–1222, 2006.
financed by the European Social Fund. [13] J. Staal, M.D. Abrámoff, M. Niemeijer, M.A.
Viergever, and B. van Ginneken. Ridge-based ves-
sel segmentation in color images of the retina. IEEE
REFERENCES Transactions on Medical Imaging, 23(4):501–509,
2004.
[1] G. Ayala, T. León, and V. Zapater. Different aver- [14] K. Sun, Z. Chen, S. Jiang, and Y. Wang. Morpho-
ages of a fuzzy set with an application to vessel seg- logical multiscale enhancement, fuzzy filter and
mentation. IEEE Transactions on Fuzzy Systems, watershed for vascular tree extraction in angiogram.
13(3):384–393, 2005. Journal of medical systems, 35(5):811–824, 2011.

136

VIPIMAGE15_Book.indb 136 9/8/2015 7:44:15 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Identification of subendocardial infarction—a feasibility study

using synthetic ultrasonic image data of a left ventricular model

J. Żmigrodzki, S. Cygan, B. Leśniak-Plewińska & K. Kałużyński

Institute of Metrology and Biomedical Engineering, Faculty of Mechatronics, Warsaw University of Technology,
Warszawa, Poland

ABSTRACT: The distinction between a transmural and subendocardial infarction is important for
both therapeutic and prognostic reasons. To investigate the possibility to differentiate these two cases,
a numerical model of a LV phantom with uniform wall and wall with inclusion simulating subendocar-
dial infarction was created. Using meshes of these phantoms synthetic ultrasonic data was generated
using the Field II software. Block matching based estimation of displacements was carried out using
the 2-dimensional cross correlation coefficient. Eulerian strain tensor and principal strain components
were obtained using displacement gradient tensors. The components of the principal strain, particularly
the first component, show strong differences in the affected wall as compared with the homogenous wall.
The methodology has the potential for identification of non-transmural changes in the left ventricle wall.

1 INTRODUCTION signals or baseband signals. Numerous similarity

measures can be used, eg. cross-correlation func-
Contrary to the transmural infarction, in the case tion, cross-correlation coefficient (Kaluzynski et al.,
of the subendocardial infarction the viability of 2001; de Korte et al., 1997; Langeland et al., 2003;
the cardiac muscle is preserved. The distinction Lee & Konofagou, 2008; Luo & Konofagou, 2010),
between these two types of infarction is important zero crossing of the phase of the cross-correlation
for both therapeutic and prognostic reasons. One function (Pesavento & Ermert, 1998), sum of abso-
can expect, that after revascularization the cardiac lute differences and sum of squared differences
contractility will improve. A method allowing for (Langeland et al., 2003). Speckle tracking has also
quantitative evaluation of the transmural distribu- been shown to enable selective assessment of epi-
tion of cardiac deformation may help in estimat- cardial, midwall, and endocardial function in both
ing the disease extent and is currently sought by animal (Sakurai et al., 2014; Tee et al., 2015) and
the cardiologists. Studies have been conducted to humans (Abate et al., 2014; Adamu et al., 2009;
measure the transmural distribution of myocardial Becker et al., 2009; Leitman et al., 2010). This
deformation in animal models (Hashimoto et al., layer-specific analysis of myocardial deformation
2003; Ishizu et al., 2010; Waldman et al., 1985). parameters has not been validated in an experi-
Most of them used invasive techniques, and their mental model.
conclusions may be biased because of potential This study investigates the possibility of distin-
alteration of the cardiac function, resulting from guishing a wall with nontransmural inclusion, sim-
opening of the pericardium (Belenkie et al., 2004) ulating subendocardial infarction, and a uniform
or the implantation of microdevices. Noninvasive wall in Left Ventricular (LV) model, using synthetic
studies in humans indicated the possibility to dis- ultrasonic data. The strain distribution within the
tinguish transmural and non-transmural infarction model wall is used to assess the effect of the inclusion.
(Chan et al., 2006; Zhang et al., 2005). This study is the part of an ongoing project which
The attention of cardiologists is concentrating will comprise also experiments in Polivinyl Alcohol
on noninvasive imaging of 2D strain on the basis of (PVA) cryogel ventricular phantoms. It is consid-
speckle tracking, which appears to be highly repro- ered as both source of reference data and introduc-
ducible and demonstrated good correlation with tory phase to these experiments.
results of tissue strain estimation using sonomi-
crometry or tagged MRI (Cho et al., 2006; Heyde
et al., 2012). The speckle tracking determines the 2 MATERIALS AND METHODS
local displacement of speckle patterns between
sequential images, usually using a block-matching The numerical model of the LV phantom was
algorithm applied to the RF (Radio Frequency) designed as a half of an ellipsoid with uniform

137

VIPIMAGE15_Book.indb 137 9/8/2015 7:44:15 PM

 15 mm wall thickness (Fig. 1). Proposed geom- pressure amplitude curve corresponded to the
etry resulted in 52 ml end-systolic volume. This physiological LV volume change (Fukuta and Lit-
geometry is based on previously studied ventricu- tle, 2008) with magnitude reaching 36 kPa. Sets
lar models (Cygan et al., 2014; Heyde et al., 2012; of deformed meshes of the numerical models
Lesniak-Plewinska et al., 2010). A tubular collar obtained from FEM have been created with the
was added for mounting physical phantoms in time-step corresponding to the desired frame rate
the laboratory setup. A homogenous phantom of the generation of synthetic ultrasonic data and
and a phantom with an inclusion approximately were used to define the shape of the phantom dur-
2.5 times stiffer than the rest of the phantom mate- ing this generation.
rial, with a diameter of 20 mm and 10 mm thick- Because the results obtained using the FEM
ness were considered. The inclusion was created as model will serve as a reference to studies of physi-
an intersection of a cylinder with the thick-walled cal phantoms, the FEM model should take into
ellipsoid of the phantom. The main axis of inclu- account the properties of the material. PVA gel
sion is normal to the surface of the phantom. is often assumed to be linearly elastic (Jia et al.,
This axis intersects the wall 40 mm from the base 2006), but it has been shown to be hyperelastic
(Fig. 1). or even visco-hyperelastic (Karimi et al., 2014).
Numerical model of the phantom geometry, If the behavior of the physical phantom is to be
created using the Autodesk Inventor 2012 soft- adequately modeled, the properties of the material
ware (Autodesk Inc., USA) was exported in CAD model should come from physical measurements.
format to the Finite Element Method (FEM) The PVA cryogel was modeled as a hyperelastic
software—Abaqus 6.13-3 (Simulia, USA) (Cygan material using strain-stress relation obtained from
et al., 2015). The phantom was meshed into linear measurements of the material samples. The hyper-
hexahedral elements (type C3D8H) with 1.5 mm elastic material was defined as isotropic, with 3-rd
seed. In the simulations both the internal and exter- order Yeoh strain energy potential definition and
nal surfaces of the mounting collar of the phan- Poisson’s ratio of 0.45 (Cygan et al., 2015).
tom were immobilized. Deformation was forced
by applying pressure load to the inner surface of
2.1 Synthetic data generation
the elliptical part of the phantom. Shape of the
The FIELD II package was used in the present
study (Jensen, 1996; Jensen & Svendsen, 1992). It
enables incorporating realistic transducer features
into the simulation (Touil et al., 2010).
The concentration of the scattering elements
within the resolution cell volume was set to 3, result-
ing in a speckle amplitude distribution close to the
Rayleigh distribution. Increasing this number did not
reduce significantly the discrepancy between these
distributions and incurred a heavy computational
cost. The synthetic ultrasonic data were generated
using parameters corresponding to the Ultrasonix
Sonix Touch RP scanner and phased array trans-
ducer SA4-2/24, featuring central frequency of 2.8
MHz and 65% bandwidth. A Long Axis View (LAX)
of the single wall of the phantom was simulated
(Fig. 1). Scan parameters were as follows: excitation
frequency 2.3 MHz, sector angle 35°, 126 lines, frame
rate 100, transmit focus at 8 cm, dynamic focusing on
receive. Resolution cell dimensions (−6 dB FWHM)
were x—3.12 mm, y—4.24 mm, z—0.39 mm, resolu-
tion cell volume—5.24 mm3, where 0 xz is the imag-
ing plane and the z axis is perpendicular to the probe
section (Fig. 1).
The RF lines were generated at a sampling fre-
quency of 80 MHz. Each RF line was then filtered
with bandpass filter. Lowpass section of the filter
Figure 1. Section of the phantom. * marks the LV and has a cutoff frequency of 9.69 MHz and an attenu-
** the fixing collar. LAX view sector shadowed, coordi- ation slope equal to −24 dB/octave. Highpass sec-
nate system in gray. tion of the filter has a cutoff frequency of 2.05 MHz

138

VIPIMAGE15_Book.indb 138 9/8/2015 7:44:15 PM

 and attenuation slope equal to −22 dB/octave. This of displacement estimation was increased by
mimics the properties of the frequency response interpolating the cross-correlation coefficient
of the analog front-end section of the Ultrasonix by a factor of 5 in the radial direction (wave
Sonix Touch RP scanner. Prior to the displace- propagation direction) and by 10 in the angular
ment estimation the RF lines were decimated by direction. The outliers of displacement estima-
a factor of 6. tion were detected by analyzing regional values of
the mean and standard deviation and corrected
(Garcia, 2010; Wang et al., 2012). Subsequently
2.2 Displacement and strain estimation displacement fields were smoothed using two-
The displacement fields were estimated using the dimensional median and averaging filters, of size
2D cross-correlation coefficient as the measure of 30 × 15 [samples × lines] and 10 × 5 [samples × lines]
similarity (Langeland et al., 2003). The size of the in the case of the radial displacement map. In
kernel and search region were 7 × 9 [samples × lines] the case of the angular displacement map, the
and 65 × 23 [samples × lines]. The kernel size corre- dimensions of the filters were respectively 10 × 10
sponded to the average speckle size. The resolution [samples × lines] and 30 × 30 [samples × lines].

Figure 2. Maps of the first component of the principal strain, LAX view, data without scan conversion. FE simula-
tion reference data—homogenous model (top-left), model with insertion (top-right). Estimation results—homogenous
model (bottom-left), model with insertion (bottom-right). The effect of the non-transmural insertion can be seen in
the (top-right) and (bottom-right) parts. Abscissa is scaled in scan line number, ordinate in RF sample number, strain
nondimensionalised [%].

139

VIPIMAGE15_Book.indb 139 9/8/2015 7:44:16 PM

 Displacement gradient tensors were obtained gradient tensor estimation as the displacements
in the polar coordinate system (the centre of the estimated from synthetic data, but using different
ultrasonic probe was in the centre of this system) derivative lengths: in the radial direction N equaled
as directional derivatives of the displacement fields 50 points and in the angular direction N equaled
(radial and angular). The values of the derivatives 5 points.
were estimated as a value of the slope of the line
fitted to N points around the point of interest.
For derivatives in the radial direction, N was equal 3 RESULTS AND DISCUSSION
to 150 points and for angular direction N was
equal to 10 points. The Eulerian strain tensor The principal strains were analyzed for two con-
and principal strain components were obtained secutive images from the part of the deformation
using displacement gradient tensors (Mase et al., cycle featuring the highest slope of the internal
2009), a methodology similar to that presented in pressure curve (corresponding to the early diasto-
(Fung-Kee-Fung et al., 2005). lic phase). The maps of the first component of the
The FEM data were used as the reference to the principal strain obtained for the normal wall model
results obtained from the synthetic ultrasonic data. and the affected wall model differ. Both maps cor-
The displacements exported from Abaqus were respond well to the reference maps obtained from
submitted to the same procedure of deformation the FEM data (Fig. 2).

Figure 3. Maps of the second component of the principal strain, LAX view, data without scan conversion.
Organization as in Figure 2.

140

VIPIMAGE15_Book.indb 140 9/8/2015 7:44:16 PM

 ACKNOWLEDGEMENTS

This study was supported by the Polish National

Science Centre by the decision DEC-2012/07/B/
ST7/01441.

REFERENCES

Abate, E., Hoogslag, G.E., Leong, D.P., Bertini, M.,

Antoni, M.L., Nucifora, G., Joyce, E., Holman, E.R.,
Siebelink, H.-M.J., Schalij, M.J., et al. (2014).
Association between Multilayer Left Ventricular
Rotational Mechanics and the Development of Left
Ventricular Remodeling after Acute Myocardial Inf-
Figure 4. Plots of the first component of principal arction. J. Am. Soc. Echocardiogr. 27: 239–248.
strain: estimated across the affected area (continuous Adamu, U., Schmitz, F., Becker, M., Kelm, M., and
circled line); reference across the affected area (continu- Hoffmann, R. (2009). Advanced speckle tracking
ous line); estimated across the corresponding area in the echocardiography allowing a three-myocardial layer-
normal model (broken circled line); reference across the specific analysis of deformation parameters. Eur. J.
corresponding area in the normal model (broken line). Echocardiogr. J. Work. Group Echocardiogr. Eur. Soc.
Cardiol. 10: 303–308.
Becker, M., Ocklenburg, C., Altiok, E., Füting, A.,
Balzer, J., Krombach, G., Lysyansky, M., Kühl, H.,
The maps of the second component of the prin- Krings, R., Kelm, M., et al. (2009). Impact of infarct
cipal strain also show differences between the nor- transmurality on layer-specific impairment of myo-
mal wall model and the affected wall model, but cardial function: a myocardial deformation imaging
study. Eur. Heart J. 30: 1467–1476.
due to more noise the differences are less visible.
Belenkie, I., Sas, R., Mitchell, J., Smith, E.R., and Tyberg, J.V.
In the major part of the corresponding maps, the (2004). Opening the pericardium during pulmonary
estimated values of the strain are close to the refer- artery constriction improves cardiac function. J. Appl.
ence (Fig. 3). The biggest errors are observed in Physiol. Bethesda Md 1985 96: 917–922.
lower third part of the model. The maps of direc- Chan, J., Hanekom, L., Wong, C., Leano, R.,
tion (angle) of both principal strain components Cho, G.-Y., and Marwick, T.H. (2006). Differentiation
very well correspond to reference maps, however, of Subendocardial and Transmural Infarction Using
the difference between the normal wall model and Two-Dimensional Strain Rate Imaging to Assess
the affected wall model is very subtle. Short-Axis and Long-Axis Myocardial Function.
J. Am. Coll. Cardiol. 48: 2026–2033.
The slope of the first component of the princi-
Cho, G.-Y., Chan, J., Leano, R., Strudwick, M.,
pal strain in the normal wall and the affected wall and Marwick, T.H. (2006). Comparison of two-
at the place of the inclusion show different sign dimensional speckle and tissue velocity based strain
(Fig. 4). This means that the presence of the inclu- and validation with harmonic phase magnetic reso-
sion strongly affects the strain distribution within nance imaging. Am. J. Cardiol. 97: 1661–1666.
the model wall and also the transmural gradient Cygan, S., Werys, K., Błaszczyk, Ł., Kubik, T., and
of the first component of the principal strain, Kałużyński, K. (2014). Left ventricle phantom and
changing its sign. This indicates that the strain gra- experimental setup for MRI and echocardiography—
dient may be considered as an indicator of presence Preliminary results of data acquisitions. Biocybern.
Biomed. Eng. 34: 19–24.
of a stiffer area in the vicinity of the inner wall of
Cygan, S., Żmigrodzki, J., Leśniak-Plewińska, B.,
the model. Karny, M., Pakieła, Z., and Kałużyński, K. (2015).
Influence of Polivinylalcohol Cryogel Material
Model in FEM Simulations on Deformation of
4 CONCLUSIONS LV Phantom. In H. van Assen, P. Bovendeerd, &
T. Delhaas, eds. Functional Imaging and Modeling
This preliminary study indicates, that the identifi- of the Heart, (Springer International Publishing),
cation of the nontransmural inclusion in a numeri- 313–320.
cal LV phantom, based on simulated synthetic De Korte, C.L., van der Steen, A.F., Dijkman, B.H.,
and Lancée, C.T. (1997). Performance of time delay
ultrasonic data, is possible by analyzing estimated
estimation methods for small time shifts in ultrasonic
principal strain distribution within the wall and signals. Ultrasonics 35: 263–274.
potentially also its gradient. Further work will Fukuta, H., and Little, W.C. (2008). The Cardiac Cycle
include diversification of the size and material and the Physiological Basis of Left Ventricular
properties of the inclusion and experiments with Contraction, Ejection, Relaxation, and Filling. Heart
physical LV phantoms. Fail. Clin. 4: 1–11.

141

VIPIMAGE15_Book.indb 141 9/8/2015 7:44:17 PM

 Fung-Kee-Fung, S.D., Lee, W.-N., Ingrassia, C.M., Leitman, M., Lysiansky, M., Lysyansky, P., Friedman, Z.,
Costa, K.D., and Konofagou, E.E. (2005). Angle- Tyomkin, V., Fuchs, T., Adam, D., Krakover, R.,
independent strain mapping in myocardial elas- and Vered, Z. (2010). Circumferential and Longitu-
tography 2D strain tensor characterization and dinal Strain in 3 Myocardial Layers in Normal Sub-
principal component imaging. In 2005 IEEE jects and in Patients with Regional Left Ventricular
Ultrasonics Symposium, 516–519. Dysfunction. J. Am. Soc. Echocardiogr. 23: 64–70.
Garcia, D. (2010). Robust smoothing of gridded data Lesniak-Plewinska, B., Cygan, S., Kaluzynski, K.,
in one and higher dimensions with missing values. D’hooge, J., Zmigrodzki, J., Kowalik, E.,
Comput. Stat. Amp Data Anal. 54: 1167–1178. Kordybach, M., and Kowalski, M. (2010). A dual-
Hashimoto, I., Li, X., Hejmadi Bhat, A., Jones, M., chamber, thick-walled cardiac phantom for use in car-
Zetts, A.D., and Sahn, D.J. (2003). Myocardial strain diac motion and deformation imaging by ultrasound.
rate is a superior method for evaluation of left ven- Ultrasound Med. Biol. 36: 1145–1156.
tricular subendocardial function compared with tissue Luo, J., and Konofagou, E.E. (2010). A fast normal-
Doppler imaging. J. Am. Coll. Cardiol. 42: 1574–1583. ized cross-correlation calculation method for motion
Heyde, B., Cygan, S., Choi, H.F., Lesniak-Plewinska, B., estimation. IEEE Trans. Ultrason. Ferroelectr. Freq.
Barbosa, D., Elen, A., Claus, P., Loeckx, D., Control 57, 1347–1357.
Kaluzynski, K., and D’hooge, J. (2012). Regional Mase, G.T., Smelser, R.E., and Mase, G.E. (2009).
cardiac motion and strain estimation in three- Continuum Mechanics for Engineers, Third Edition
dimensional echocardiography: a validation study in (Boca Raton: CRC Press).
thick-walled univentricular phantoms. IEEE Trans. Pesavento, A., and Ermert, H. (1998). Time-efficient and
Ultrason. Ferroelectr. Freq. Control 59: 668–682. exact algorithms for adaptive temporal stretching and
Ishizu, T., Seo, Y., Enomoto, Y., Sugimori, H., 2D-correlation for elastographic imaging using phase
Yamamoto, M., Machino, T., Kawamura, R., and information. In, 1998 IEEE Ultrasonics Symposium,
Aonuma, K. (2010). Experimental validation of left 1998. vol.2: 1765–1768.
ventricular transmural strain gradient with echocar- Sakurai, D., Asanuma, T., Masuda, K., Hioki, A., and
diographic two-dimensional speckle tracking imaging. Nakatani, S. (2014). Myocardial layer-specific analysis
Eur. J. Echocardiogr. J. Work. Group Echocardiogr. of ischemic memory using speckle tracking echocardi-
Eur. Soc. Cardiol. 11: 377–385. ography. Int. J. Cardiovasc. Imaging, 30: 739–748.
Jensen, J.A. (1996). FIELD: A Program for Simulating Tee, N., Gu, Y., Murni, and Shim, W. (2015). Compara-
Ultrasound Systems. In 10th Nordicbaltic Conference tive Myocardial Deformation in 3 Myocardial Lay-
on Biomedical Imaging, Vol. 4, Supplement 1, Part 1: ers in Mice by Speckle Tracking Echocardiography.
351–353. BioMed Res. Int. 2015, e148501.
Jensen, J.A., and Svendsen, N.B. (1992). Calculation of Touil, B., Basarab, A., Delachartre, P., Bernard, O., and
pressure fields from arbitrarily shaped, apodized, and Friboulet, D. (2010). Analysis of motion tracking in
excited ultrasound transducers. IEEE Trans. Ultrason. echocardiographic image sequences: influence of sys-
Ferroelectr. Freq. Control 39: 262–267. tem geometry and point-spread function. Ultrasonics
Jia, C., Kim, K., Weitzel, W.F., Jia, C., Rubin, J.M., and 50: 373–386.
Kolias, T.J. (2006). P1E-1 Left Ventricular Phantom Waldman, L.K., Fung, Y.C., and Covell, J.W. (1985).
with Pulsatile Circulation for Ultrasound Strain Rate Transmural myocardial deformation in the canine
Imaging. In IEEE Ultrasonics Symposium, 2006, left ventricle. Normal in vivo three-dimensional finite
1317–1320. strains. Circ. Res. 57: 152–163.
Kaluzynski, K., Chen, X., Emelianov, S.Y., Wang, G., Garcia, D., Liu, Y., de Jeu, R., and Johannes
Skovoroda, A.R., and O’Donnell, M. (2001). Strain Dolman, A. (2012). A three-dimensional gap filling
rate imaging using two-dimensional speckle tracking. method for large geophysical datasets: Application
IEEE Trans. Ultrason. Ferroelectr. Freq. Control, 48: to global satellite soil moisture observations. Environ.
1111–1123. Model. Softw. 30: 139–142.
Karimi, A., Navidbakhsh, M., and Beigzadeh, B. (2014). Zhang, Y., Chan, A.K.Y., Yu, C.-M., Yip, G.W.K.,
A visco-hyperelastic constitutive approach for mode- Fung, J.W.H., Lam, W.W.M., So, N.M.C., Wang, M.,
ling polyvinyl alcohol sponge. Tissue Cell 46: 97–102. Wu, E.B., Wong, J.T., et al. (2005). Strain Rate Imag-
Langeland, S., D’hooge, J., Torp, H., Bijnens, B., and ing Differentiates Transmural From Non-Transmural
Suetens, P. (2003). Comparison of time-domain dis- Myocardial Infarction: A Validation Study Using
placement estimators for two-dimensional RF track- Delayed-Enhancement Magnetic Resonance Imaging.
ing. Ultrasound Med. Biol. 29: 1177–1186. J. Am. Coll. Cardiol. 46: 864–871.
Lee, W.-N., and Konofagou, E.E. (2008). Angle-
independent and multi-dimensional myocardial
elastography—from theory to clinical validation.
Ultrasonics, 48, 563–567.

142

VIPIMAGE15_Book.indb 142 9/8/2015 7:44:18 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Microcalcification segmentation in full field digital mammography

N. Martins
Centro de Computação Gráfica, Azurém, Guimarães, Portugal
Enermeter, Parque Industrial de Celeirós, Lugar de Gaião, Braga, Portugal

D. Veiga, C. Pereira & M. Ferreira

Enermeter, Parque Industrial de Celeirós, Lugar de Gaião, Braga, Portugal
University of Minho, Centro Algoritmi, Azurém, Guimarães, Portugal

N. Alves
Centro de Computação Gráfica, Azurém, Guimarães, Portugal

M. Delgado
Dr Campos Costa, Imagiologia Clínica, Rua de Avis, Porto, Portugal

ABSTRACT: Breast Cancer is one of the most common types of cancer in Women. Mammography
is widely used as a screening method because it can detect pathologies in an early stage. There are some
Computer Aided Diagnosis (CAD) systems, but their performance can be improved with new and bet-
ter image processing algorithms. In this paper, we present a new microcalcification detection method for
Full Field Digital Mammograms (FFDM), based on the wavelet transform, Expectation Maximization
with Markov Random Fields bootstrap and a final SVM classifier. Tests were made in a public database
(InBreast) and results shown that our approach achieved better False Positive (FP) rates and sensitivities
compared to the literature. Features were also presented and the problems with the train addressed.

1 INTRODUCTION of cells. In digital images, they can be as small as

1–2 pixels, and with very low signal to noise ratio.
Breast cancer is the most common type of cancer This represents a challenge to both, physicians and
in woman. In the US, it was estimated 226,870 new image processing algorithms and thats why it still
cases of invasive cancer, and around 40.000 deaths, is an open problem (A. Bria 2014).
making it a serious public health problem (Jiemin
Ma 2013). The most used and globally accepted
1.1 Related word
method for breast cancer detection is the Mam-
mography. With this technique is possible to find The number of publications in this field is
breast cancer clues, like Microcalcification (MC) extremely large. In (Karthikeyan Ganesan 2013,
clusters in an early stage. Is known that an early Afsaneh Jalalian 2013) is presented a recent review
detection is crucial for an efficient treatment, for of the stat-of the-art. The problem, is that most
instance, when diagnosed in an early stage, the of them use screen films with less quality than
success rates are about 98.6% and with much less FFDM images. In fact, to our knowledge, there is
invasive treatments. When the diagnosis is in a late only one public FFDM database which is InBreast
stage, the success rate drops to about 23.3% and in (Domingues 2011).
most cases needs a surgical intervention to remove Some recent and important papers, regarding
part of the breast (mastectomy) (Jiemin Ma 2013). MC detection, are presented next. Supervised algo-
This kind of treatment is painful and is psycho- rithms have been used more often than the unsu-
logic traumatic due to the removal of an important pervised, due to the complexity of the problem. In
organ for the woman self esteem. Given the impor- (E. Zhang & Bai 2014) an SVM was used to reduce
tance of an early detection, several algorithms have the False Positives (FP), after a contrast enhance-
been proposed in the literature to help physicians ment technique based on morphologic filters and a
in the diagnosis. MCs are one of the hardest clues dual threshold technique. In (A. Bria 2014) a cas-
to find in a mammogram, they consist in very small cade of classifiers was used to classify unbalanced
white dots, resulting from the necrotic activities data, first they used haar-like features to detect

143

VIPIMAGE15_Book.indb 143 9/8/2015 7:44:18 PM

 MCs and then a new set of graph based features where, DWT is the discrete wavelet transform and
to remove non-malign clusters, they claim results IDWT is the inverse discrete wavelet transform.
comparable with existent commercial systems A and D represent the approximation and detail
(R2CAD), but the database used was not pub- coefficients, s is the scale, which varies between
lic, and so no comparisons are possible. In (S.-Y. 0 and S, where 0 is the original image and S the
Shin & Yun 2014), a deep learning technique was coarser scale. At last, h, v, d represent respectively
used. They started with a peak detector, followed the horizontal, vertical and diagonal orientations.
by a manual classification to distinguish between In (Martins, Pereira, & Ferreira 2014), contrast
calcification and non-calcification, to teach the is defined as the ratio between the details and
algorithm the characteristics of the calcifications. approximations coefficients in the wavelet domain.
After that, 33 unknown images are presented to After some transformations, the enhancement is
the classifiers and a ROC of 0.83 was achieved. achieved using equation (3)
Once again the database was private and the test-
ing set small, making the results hard to replicate. As ( m n ) k
In (Domingues & Cardoso 2014), they propose Dsk ( m n ) Gsk ( m n ) Ds ( m n ) (3)
As ( m n )
a Bayesian surprise method to detect MCs, they
compare their approach with three other com-
mon algorithms and they claim that they achieve a where, Dsk are the enhanced detail coefficients, Gsk
sensitivity of 60.3% with a FPR per image of 108. the gain matrix and As the enhanced approxima-
In this work they used their own public database tion coefficients.
(inBreast), so the present work will use the same In order to use equation (3) we must find As . This
database and metrics in order to make the com- step is achieved using the IDWT with the enhanced
parisons fair. detail coefficients of the scale above (s + 1). So, the
enhancement starts at the coarse scale, S, until the
scale 0 is reached. At each scale the information
from the scale above is used. At the beginning, we
2 PROPOSED WORK define AS as AS , because no scale above exists and
only the Gain matrix, GSk , is used.
Our work consists in three main steps. First a pre- The parameters selected in our approach are
processing is applied to enhance the MCs, then a based on the image resolution. So a maximum
segmentation step where the MCs are extracted scale, S, of 4 was chosen and the Gain matrix
from the background and finally a classification was simplified to a fixed value, 1.5. The wavelet
step to reduce false positives. In the next sections, filter used was the “biorthogonal 1.5” because it
all techniques will be explained in more detail. resemble the MC shape. The value of S = 4 was
chosen, because this is the last scale where MCs
2.1 Pre-processing are expected to shown up. In Figure 1 is shown the
result of the pre-processing.
In order to enhance the signal to noise ratio of
the MC, a wavelet-based enhancement technique
was applied. The technique presented in (Martins, 2.2 Segmentation
Pereira, & Ferreira 2014) is used, more precisely Two main segmentation techniques where used.
the Wavelet Contrast method. Next an explanation The first is the segmentation of the breast tissues in
of this method is done and the parameters chosen glandular and adipose. This is done with a Expecta-
for the given images. tion Maximization (EM) algorithm with an MRF
The Wavelet Transform is a well know signal bootstrap. The second is a wavelet based contrast
transform tool. It uses wavelet filters to decompose segmentation, for each tissue. The segmentation in
the image in details and approximations (high and only computed in the breast area and, to detect it,
low frequencies) in multiscale. These characteris- the OTSU algorithm was used, followed by an ero-
tics make it interesting to use in mammography, sion to refine the contours of the resulting mask.
because MCs can be seen as high frequencies in
these images. Moreover, their size is variable which
the multiscale property can naturally deal with.
A very simple way to express the wavelet transform
is shown in equation (1 and 2)

{A s+ }W ( As )
Dsh+ Dsv+ Dsd+ = DWT (1)

{As } ⎛ h v d⎞
⎝ As , Ds , Ds , Ds ⎠ (2) Figure 1. Microcalcifications enhancement result.

144

VIPIMAGE15_Book.indb 144 9/8/2015 7:44:18 PM

 2.2.1 Breast tissue segmentation
Looking at the histogram of a Mammogram,
is possible to find two overlapping Gaussians
Figure 2. So, knowing that the glandular tissue is
brighter than the adipose, we assume that the two
overlapping Gaussians represent the adipose and
glandular tissues.
In order to separate them, we use the EM algo-
rithm to calculate the parameters of both gaus-
sians and the MRF to give a spatial sensitivity to
the algorithm as shown in (Zhang, Brady, & Smith
2001). We use the k-means algorithm to initialize
the segmentation and then the EM-MRF to refine Figure 3. Histogram of the contrast image.
it. The idea of using the MRF is to enhance the
separation of the tissues in the overlapping area. If
a pixel is classified as glandular tissue, but is sur- Table 1. Features extracted from each candidate.
rounded by adipose tissue, odds are that the pixel is
also adipose tissue, and the other why around. This Features Description
technique results in a much smoother segmenta- Shape based Area, perimeter, circularity
tion and robust against noise. Intensity based Mean, std, min, max
Zernike moments Radial degree 0 to 4
2.2.2 Microcalcifications segmentation Hu moments Invariant to rotation and size
After the tissue segmentation, the algorithm looks Fractal based Box counting method
for MCs in each tissue. This approach follows the Gaussian similarity 8 directions std, skew, kurt
idea that the noise levels are different in the adi-
pose and glandular tissue. So they are processed
in separate to avoid over segmentation in one and
under segmentation in the other. This was expected noise in the image. The subtraction of 1, removes
in our approach due to the use of a statistical con- around 84% of the pixels with less contrast and the
trast approach. The segmentation step starts with remaining 16% are considered as candidates to the
the image decomposition using the DWT. After next phase.
that, the coarser approximation coefficients are set Next the images are merged and the resulting
to zero, and then the ISWT is used to obtain the image is once more normalized, dividing the inten-
contrast image. The idea is to remove the low pass sities by the maximum value between the maximum
band of the image, which correspond to the tissues. intensity of the image and 20. The value 20 was
Higher frequencies are considered to be possible obtain after observation of the images without MCs.
MCs. The contrast image is then separated, using In these cases the maximum value drops, because
the tissue segmentation, in adipose and glan- no high frequencies are present in the image, which
dular component. Both tissues are assumed to would force the algorithm to always find MCs that
have a gaussian like histogram (Fig. 3) and so a are not present. Consequently the number of false
Z-normalization of the intensities is performed fol- positives would drastically increases. After that a
lowed by a subtraction of 1 from all pixels. This simple linear threshold is used to differentiate the
step aims to reduce and uniform the quantum MCs from the background.

2.3 False positive reduction

To reduce the FP rate, an SVM was used. In this
step, a set of features are extracted from each MC
candidate. The selected features are presented in
Table 1.
First, the candidates are extracted with the seg-
mentation step. A small threshold was used in order
to get a high sensitivity. Next, each candidate is
compared with the GT and separated between cor-
rect and incorrect detections. The number of incor-
rect detections is much higher than the number of
the correct ones. Because of that, a training strat-
Figure 2. Histogram of a mammogram. egy similar to (A. Bria 2014) was used. 75% of the

145

VIPIMAGE15_Book.indb 145 9/8/2015 7:44:19 PM

 correct detected MCs were selected to train the In Table 2 is shown the published results of
SVM and the false group was randomly picked other methods.
from the incorrect detected MCs, so that the same Comparing with the Bayesian Surprise, our
number of correct and incorrect detected MCs is approach performed considerable better. In their
obtained. After that, a 5-fold cross-validation was operation point (FP = 108), our method, achieved
used to train the SVM and tests were made in the a sensitivity of 77.2% with the SVM and 69.97%
remaining MC candidates (not used in the train). before the SVM, in comparison with a sensitivity
This process is repeated until a desired SVM score of 60.3% obtain by the Bayesian Surprise. Regard-
is achieved. High sensitivities are desired, and so ing the other detection methods, our method out-
a minimum sensitivity of 85% was used and the performed all.
SVM with the best specificity was kept.
4 CONCLUSIONS
2.4 Metrics
The performance metric used was the same as in A fully automatic segmentation method was pro-
(Domingues & Cardoso 2014), not only because it posed for MC segmentation. Results shown that,
allows a direct comparison with their results, but for the InBreast database, our approach achieved
also, because it is based in a mathematical proven higher sensitivity and lower FP per image than the
formulation. The final results are presented in a existing methods in the literature. Results still have
FROC curve. The FROC curve test several different a long way to go, higher sensitivities and lower
thresholds to create a curve of False Positives per FP rates are necessary to make the system more
image vs sensitivity. In our case, 2 tests were made, trustful. Regarding the proposal itself, we highlight
one before the SVM and the other after the SVM. the modularity, each step is independent from the
other. For instance, the preprocessing could help
improve existing methods and the segmentation
3 RESULTS could be used as a starting point to other meth-
ods. The Segmentation also have the advantage of
In the Figure 4 is shown the FROC curve of our being simple and that no training is needed, so it
approach. can be used, in theory, in any kind of mammogra-
As expected, the SVM reduce the false positive phy image with only small parameters adjustments.
rates considerably. For a sensitivity of 70% the The SVM performed as expected reducing consid-
reduction was from 75 FP to less then 45, and for erably the number of FPs, proving that the features
a sensitivity of 80% the reduction was from 400 chosen correctly, making it a valuable contribution
to 170 FP. of this work.

ACKNOWLEDGMENTS

Work supported by European Union funds through

the FUNDO EUROPEU DE DESENVOLVI-
MENTO REGIONAL and by national funds
by Quadro de Referência Estratégico Nacional.
N. Martins thanks the Centro de Computação
Gráfica, Dr Campos Costa and ENERMETER.

REFERENCES

Figure 4. FROC of the proposed method. A. Bria, N.K. (2014). Learning from unbalanced data:
A cascade-based approach for detecting clustered micro-
calcifications. Medical Image Analysis 18, 241–252.
Table 2. Operation points of some published algorithms. Afsaneh Jalalian, S.B.M. (2013). Computer-aided detec-
tion/diagnosis of breast cancer in mammography
Method Sensitivity (%) FP and ultrasound: a review computer-aided detection/
diagnosis of breast cancer in mammography and
Fixed threshold 34.5 164 ultrasound: a review. ELSEVIER—Clinical Imaging.
Outlier detection 45.8 60 Domingues, I., C.J. (2011). Inbreast: Toward a full-field
Mathematical morphology 40.3 225 digital mammographic database inbreast: Toward a
Bayesian surprise 1 60.3 108 full-field digital mammographic database. Technical
Report: Acad Radiology.

146

VIPIMAGE15_Book.indb 146 9/8/2015 7:44:20 PM

 Domingues, I. & J. Cardoso. (2014, Aug). Using bayesian Shin, S.-Y., S.L. & I.D. Yun. (2014). Classification
surprise to detect calcifications in mammogram based microcalcification detection using discrimi-
images. In Engineering in Medicine and Biology Society native restricted boltzmann machine in digitized
(EMBC), 2014 36th Annual International Conference mammograms. SPIE Medical Imaging 115.
of the IEEE, pp. 1091–1094. Zhang, Y., M. Brady, & S. Smith (2001, Jan). Segmentation
Jiemin Ma, Ahmedin Jemal Jiemin Ma, A.J.J.M.A.J. of brain mr images through a hidden markov ran-
(2013). Breast Cancer Metastasis and Drug Resistance. dom field model and the expectation-maximization
Springer-Verlag New York Springer-Verlag algorithm. Medical Imaging, IEEE Transactions on
New York. 20(1), 45–57.
Karthikeyan Ganesan, U.R.A. (2013). Computer- Zhang, E., F. Wang, Y.L. & X. Bai. (2014). Automatic
aided breast cancer detection using mammograms: detection of microcalcifications using mathemati-
A review. IEEE REVIEWS IN BIOMEDICAL cal morphology and a support vector machine. Bio-
ENGINEERING 6. medical materials and engineering 24(1), 53–59.
Martins, N., C. Pereira, & M. Ferreira. (2014, June).
Digital mammograms contrast enhancement
using wavelets—a comparative study. In Medical
Measurements and Applications (MeMeA), 2014
IEEE International Symposium on, pp. 1–6.

147

VIPIMAGE15_Book.indb 147 9/8/2015 7:44:20 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Unsupervised delineation of the vessel tree in retinal fundus images

N. Strisciuglio
Department of Computer Engineering and Electrical Engineering and Applied Mathematics, University of Salerno,
Salerno, Italy
Johann Bernoulli Institute for Mathematics and Computer Science, University of Groningen, Groningen, The Netherlands

M. Vento
Department of Computer Engineering and Electrical Engineering and Applied Mathematics, University of Salerno,
Salerno, Italy

G. Azzopardi
Department of Intelligent Computer Systems, University of Malta, Msida, Malta
Johann Bernoulli Institute for Mathematics and Computer Science, University of Groningen, Groningen, The Netherlands

N. Petkov
Johann Bernoulli Institute for Mathematics and Computer Science, University of Groningen, Groningen, The Netherlands

ABSTRACT: Retinal imaging has gained particular popularity as it provides an opportunity to diag-
nose various medical pathologies in a non-invasive way. One of the basic and very important steps in
the analysis of such images is the delineation of the vessel tree from the background. Such segmentation
facilitates the investigation of the morphological characteristics of the vessel tree and the analysis of any
lesions in the background, which are both indicators for various pathologies. We propose a novel method
called B-COSFIRE for the delineation of the vessel tree. It is based on the classic COSFIRE approach,
which is a trainable nonlinear filtering method. RE filter to be configured by the automatic analysis of any
given vessel-like pattern. The responses of a B-COSFIRE filter is achieved by combining the responses of
difference-of-Gaussians filters whose areas of support are determined in an automatic configuration step.
We configure two types of B-COSFIRE filters, one that responds selectively along vessels and another that
is selective to vessel endings. The segmentation of the vessel tree is achieved by summing up the response
maps of both types of filters followed by thresholding. We demonstrate high effectiveness of the proposed
approach by performing experiments on four public data sets, namely DRIVE, STARE, CHASE_DB1 and
HRF. The delineation approach that we propose also has lower time complexity than existing methods.

Keywords: delineation; retinal image analysis; trainable filters; vessel segmentation

1 INTRODUCTION There are two categories of methods that have

been proposed so far, namely supervised (Staal et al.
Retinal fundus imaging is a non-invasive technique 2004, Soares et al. 2006, Ricci and Perfetti 2007,
that has gained popularity for the diagnosis of Marin et al. 2011, Fraz et al. 2012) and unsupervised
several pathologies, including diabetic retinopathy, (Hoover et al. 2000, Jiang and Mojon 2003, Men-
glaucoma, hypertension, arteriosclerosis and mul- donca and Campilho 2006, Al-Rawi et al. 2007)
tiple sclerosis. The automatic analysis of retinal approaches. Supervised approaches form pixel-wise
images may be used to assist medical specialists to feature vectors and learn a binary classifier that
diagnose such pathologies. labels pixels as vessel or non-vessel. On the other
The separation of the blood vessels from the hand, unsupervised approaches, typically rely on
rest of the image is an important segmentation tracking, filtering or morphological operations fol-
step, which facilitates the subsequent analysis. For lowed by thresholding. Supervised methods require
instance, the detection of vessel bifurcations, whose reasonably-sized data sets of training images and
progression may provide signs for cardiovascular corresponding ground truth, which is usually very
diseases, may be better detected in the segmented difficult and expensive to obtain.
image. Moreover, subtracting the delineated vessel We propose a novel unsupervised delineation
tree from the background provide better opportunity algorithm that is based on the Combination of
for the automatic detection of any retinal lesions. Receptive Fields (CORF) computational model of
149

CH23-MI9449.indd 149 9/9/2015 9:34:26 PM

 a simple cell in visual cortex (Azzopardi and Petkov use one DoG filter and thus the σ values of all
2012, Azzopardi et al. 2014) and its implementation tuples are the same. This design decision allows
called Combination Of Shifted Filter Responses the use of multi-scale DoG filtering to a prototype
(COSFIRE) (Azzopardi and Petkov 2013). It uses pattern of an irregular shape. As an example, the
two types of COSFIRE filters, one that gives a filter model automatically determined from the
response along the vessels and the other one that prototype pattern in Figure 1b is:
selectively responds to vessel endings. In general,
the two COSFIRE filters that we use respond to ⎧(σ 1 = .6, ρ1 = 0, φ1 = 0 ), ⎫
bar structures and thus we call them bar-selective ⎪(σ = .6, ρ = 2, φ = 0 ), ⎪
COSFIRE filters or B-COSFIRE for brevity. ⎪⎪ 2 2 2 ⎪⎪
The rest of the paper is organized as follows: S = ⎨(σ 3 = .6, ρ3 = 2, φ3 = 3.14
1 ),⎬ (1)
in Section 2 we present the B-COSFIRE filters; ⎪( 2.6 ρ4 = 4, φ4 0 ), ⎪
⎪ 4 ⎪
in Section 3, we present the experimental protocol ⎪⎩( 5 2.6 ρ5 = 4, φ5 3.14 ) ⎪⎭
and the results achieved on four public data sets
of retinal fundus images, which we compare with
We compute the response of a B-COSFIRE fil-
state-of-the-art methods in Section 4. Finally, we
ter as the geometric mean of the responses of the
draw conclusions in Section 5.
concerned DoG filters at the positions determined
in the configuration step:
2 PROPOSED METHOD
1/ |S |
def ⎛ |S | ⎞
2.1 B-COSFIRE filter rS ( x y ) ⎜∏
⎝ i =1
( sσ i )
ρi ,φi ( x y ) ⎟
⎠
(2)
A B-COSFIRE filter is trainable, in that its selec-
tivity is determined by an automatic configuration where sσ i ρi ,φi ( x, y ) is the blurred response of a
process that analyzes a given prototype pattern. DoG filter with σ i . Blurring is achieved by taking
Figure 1a shows a synthetic bar structure that we the maximum DoG response in a local neighour-
use as prototype. The configuration procedure hood weighted by a Gaussian function. The
involves the convolution of a center-on Difference- Gaussian function is centered around the polar
of-Gaussians (DoG) filter—with a given standard coordinates ( ρi φi ) with respect to the center of
deviation σ of the outer Gaussian function1—and the support of the B-COSFIRE filter (i.e. the pixel
the synthetic bar pattern followed by the determi- under consideration). It has a standard deviation
nation of local maximum DoG responses along a σ ′ i that is a linear function of the distance ρi :
number of concentric circles, Figure 1b. For each σ ′ i = σ ′ 0 + αρi. Further details about the param-
local maximum point i we extract the polar coordi- eters σ ′ 0 and α are provided in Section 3.3. The
nates ( ρi φi ) with respect to the center of the pro- blurring step is important as it allows for some tol-
totype pattern. This location is indicated by the ‘x’ erance with respect to the preferred positions of
marker in Figure 1b. the concerned DoG responses.
This configuration procedure results in a set In Azzopardi et al. (2014), we also demonstrate
S {( i i ,φi ) | i = 1, , n} of 3 − tuples. Here we that the AND-type function (geometric mean) that
we use is more robust to noise than an OR-type
function (e.g. arithmetic mean).

2.2 Tolerance to rotations

The B-COSFIRE filter configured above is selective
to horizontal vessels. By using a rotated prototype
pattern we can configure a filter that has prefer-
ence to a different orientation. Alternatively, we
create a new set Rψ (S ) = {(σ i i φi + ψ ) | i 1,…, n}
Figure 1. (a) A synthetic vessel-like structure used that represents a new B-COSFIRE filter with an
to configure a B-COSFIRE filter. (b) The output of a orientation preference that is offset by ψ radians
center-on DoG filter to the image in (a). The cross marker from that of the original filter S. We denote by
is the center of support of the B-COSFIRE filter while rˆS ( y ) the rotation-tolerant response at location
the enumerated spots represent the positions at which the ( y ) that is achieved by taking the maximum
strongest DoG responses are achieved.
value of various B-COSFIRE filters with different
orientation preferences:
1
The standard deviation of the inner Gaussian function def
is half the standard deviation of the outer Gaussian rˆS ( x, y ) = max{rRψ ( S ) ( x, y )} (3)
function. ψ ∈Ψ

150

CH23-MI9449.indd 150 9/9/2015 9:34:27 PM

 Figure 3. (b) Example of the configuration of an asym-
Figure 2. (a) Input retinal image and (b) the (inverted) metric B-COSFIRE filter by a prototype vessel-ending (a).
output of a rotation-tolerant B-COSFIRE operator The point of interest of the B-COSFIRE filter is indicated
using 12 orientation preferences. by the cross marker and lies on the end of the prototype.

Figure 4. Examples of retinal fundus images (first column) from the four concerned data sets (DRIVE, STARE,
CHASE DB1 and HRF) together with the manually segmented ground truth (second column). The output of the
delineation algorithm (third column) is thresholded in order to obtain the final segmented vessel tree (fourth column).

151

CH23-MI9449.indd 151 9/9/2015 9:34:29 PM

 π π
where Ψ = { , 12 , 6 , , 1112π }. Figure 2b shows the higher than images of the former data sets. It com-
rotation-tolerant output of a B-COSFIRE filter prises of three sets of 15 images each collected
when applied to the green channel of the image in from healthy people, patients with signs of glau-
Figure 2a. coma and patients with signs of diabetic retin-
opathy. Each image is coupled with one manually
2.3 Delineation algorithm segmented image used as ground truth.

The delineation algorithm that we propose uses

two types of B-COSFIRE filters that we refer to 3.2 Pre-processing
them as symmetric and asymmetric. A symmet- For our experiments we consider only the green
ric B-COSFIRE filter is of the type configured channel of the retinal fundus images as it pro-
above which considers the same number of DoG vides the best contrast between the vessels and
responses on both sides of the support center of the background. Furthermore, we apply the pre-
the filter. Such a filter gives strong responses along processing algorithm proposed by Soares et al.
vessels but responds weakly on the vessel endings. (2006) to smoothen the pixels that are close to the
An asymmetric B-COSFIRE filter is one which circumference of the FOV. This is achieved in an
achieves strong responses on vessel endings and iterative way by increasing the radius of the FOV
it is configured by the prototype pattern shown by one pixel in each iteration. The value of a new
in Figure 3a. Delineation is then achieved by pixel is the average of the neighbouring pixels
thresholding the sum of the response maps of the within the FOV that is determined in the previous
symmetric and asymmetric B-COSFIRE filters. iteration. The initial FOV masks for the images of
In Figure 4, we show examples of the output of the DRIVE and HRF data sets are provided with
the delineation algorithm (third column) and its the ground truth data. For the other data sets this
thresholded version (fourth column) for each information is, however, missing and we obtain the
of the four data sets, together with the manually initial FOV masks by thresholding2 the luminos-
segmented ground truth (second column). ity channel of the CIELab version of the original
RGB image. For the HRF data set, we downsam-
pled the original images by a factor of 2.
3 EXPERIMENTS AND RESULTS

3.1 Data sets 3.3 Selection of parameters

We use four data sets to evaluate the proposed The B-COSFIRE filters that we propose have four
approach: DRIVE (Staal et al. 2004), STARE parameters, namely the standard deviation of the
(Hoover et al. 2000), CHASE_DB1 (Owen et al. outer Gaussian function of the afferent DoG filter,
2009) and HRF (Odstrcilik et al. 2013). a set of ρ values that are the radii of the concentric
The DRIVE data set is composed of 40 images circles used in the configuration, together with σ ′ 0
(20 for training and 20 for testing) with resolution and α that are used by the blurring function.
768×584 pixels. For each image, a mask of the Field For each data set we consider a validation set of
Of View (FOV) of the retina is provided together images and use a grid search to first estimate the
with the binary manual segmentation of the blood- parameter values of the symmetric B-COSFIRE
vessel tree. The images in the training set have been filter and then to estimate the parameter values
manually segmented by a single observer, while the of the asymmetric B-COSFIRE filter that best
images in the test set have been segmented by two improves on the performance of the symmetrical
other observers. one. For DRIVE we use the provided 20 training
The STARE data set consists of 20 images images as the validation set, and for the other three
(700×605 pixels), 10 of which contain signs of data sets we form validation sets with the first half
pathologies. Each image is distributed together of the images.
with two manually segmented images provided by In Table 1 we report the estimated parameter
two different observers. values of the two B-COSFIRE filters that we use to
The CHASE_DB1 data set contains 28 colour process the images in the four considered data sets.
images (999×960 pixels) from 14 patients in the pro-
gram Child Heart And Health Study in England. 3.4 Evaluation method
The data set contains two groups of manually seg-
mented images provided by two observers. For the We binarize the summation of the response maps
three dataset the first set of segmented images is of the two B-COSFIRE filters by using a threshold
used as ground truth.
The HRF data set consists of images with a 2
The thresholds are 0.5 and 0.1 for the STARE and
resolution (3504×2336 pixels) that is substantially CHASE_DB1, respectively.

152

CH23-MI9449.indd 152 9/9/2015 9:34:31 PM

 Table 1. Estimated parameter values of the B-COS- published methods. The results of these methods
FIRE filters used in four public data sets. are achieved by thresholding the response of a fil-
ter or the score output of a classifier. However, the
Data set Filter σ ρ σ0 α evaluation of algorithms for vessels segmentation
DRIVE Symmetric 2.4 {0, 2, …, 8} 3 0.7
in retinal fundus images is complex. The accuracy
Asymmetric 1.8 {0, 2, …, 22} 2 0.1 is not a suitable metric because it is biased by the
STARE Symmetric 2.7 {0, 2, …, 12} 1 0.6 high number of non-vessel pixels (background).
Asymmetric 2.1 {0, 2, …, 24} 1 0.1 Thus, in order to compare the proposed method
CHASE_DB Symmetric 4.8 {0, 2, …, 18} 3 0.2 with other published methods we move along
Asymmetric 4.3 {0, 2, …, 34} 1 0.1 the ROC curves. For the same specificity values
HRF Symmetric 3.2 {0, 2, …, 14} 3 0.3 reported in state-of-the-art studies, we compare
Asymmetric 2.7 {0, 2, …, 24} 1 0.1 the value of the sensitivity. For the DRIVE data set
and for the same specificity (Sp = 0.9764) reported
by (Mendonca and Campilho 2006) we achieve a
sensitivity of 0.7376 , which is marginally better.
Table 2. Experimental results of the proposed delinea-
Similarly, for the STARE data set and for the same
tion algorithm on public data sets.
specificity reported by (Mendonca and Campilho
Data set Se Sp AUC Acc 2006) and (Al-Diri et al. 2009) (Sp = 0.9730 and
Sp = 0.9681) we achieve a sensitivity of 0.7554 and
DRIVE 0.7655 0.9704 0.9614 0.9442 0.7848 respectively, which is a significantly better
STARE 0.7763 0.9695 0.9555 0.9496 result. We also achieve the best AUC value for the
CHASE_DB1 0.7699 0.9476 0.9497 0.9305 DRIVE data set with respect to all other unsu-
HRF (Healthy) 0.7467 0.9850 0.9557 0.9665 pervised approaches (AUC = 0.9435, Al-Rawi et
HRF (DR) 0.7620 0.9604 0.9413 0.9663 al. 2007 – AUC = 0.9558, Ricci and Perfetti 2007
HRF (Glaucoma) 0.7446 0.9781 0.9561 0.9603 – AUC = 0.9407, Cinsdikici and Aydin 2009 –
AUC = 0.9614 , Lam et al. 2010).
For the HRF data set we achieve better results
value as a fraction of the maximum response. In than the other published method (Odstrcilik et al.
this way, the pixels whose sum of the two B-COS- 2013). Indeed, for the same specificity achieved by
FIRE filters is greater than the threshold are clas- Odstrcilik et al. 2013 for the healthy, diabetic retin-
sified as vessels and the remaining as non-vessels. opathy and glaucoma sets, we achieve sensitivity
We use a systematic set of threshold values values of 0.8063, 0.7528 and 0.8168, respectively.
between 0 and 1 in intervals of 0.01 and match Such values are higher than the ones reported by
every resulting binary image with the correspond- Odstrcilik et al. 2013 (0.7861, 0.7463 and 0.79). As
ing ground truth image. For each threshold value we for the CHASE_DB1 there are no other unsuper-
compute the number of true positives, false positives, vised methods to compare with. The only other
true negatives and false negatives, and subsequently state-of-the art method is a complex supervised
use them to compute the Matthews Correlation approach (Fraz et al. 2012) based on an ensemble
Coefficient (MCC) as a performance indicator. The of classifiers.
MCC is a suitable measurement for unbalanced As regards the time required to process an image
classes and thus it is appropriate for the problem at (from DRIVE and STARE data sets), the proposed
hand where the background pixels outnumber the method is the most efficient with respect to other
vessel pixels approximately by a factor of 7. methods from the state of the art (Table 3).
Finally, we choose the threshold that provides
the maximum average MCC value over a a valida-
tion data set, and report the Accuracy (Acc), Sen- Table 3. Comparison of the time required to process an
sitivity (Se) and Specificity (Sp) for that threshold. image from the DRIVE and STARE data sets.
For the sake of completeness we also compute
Method Processing time
the ROC curve and its underlying area (AUC). In
Table 2 we report the results that we achieve for the B-COSFIRE 10 seconds
concerned four data sets. (Jiang and Mojon 2003) 20 seconds
(Staal et al. 2004) 15 minutes
(Mendonca and Campilho 2006) 2.5 minutes
4 DISCUSSION
(Soares et al. 2006) 3 minutes
(Lam et al. 2010) 13 minutes
In Table 4, we report the results achieved by the (Marin et al. 2011) 1.5 minutes
proposed method on the four considered data sets, (Fraz et al. 2012) 2 minutes
in comparison with the ones achieved by other

153

CH23-MI9449.indd 153 9/9/2015 9:34:31 PM

 Table 4. Performance results of the proposed unsupervised B-COSFIRE filter approach on the concerned data sets
compared to other methods from the state of the art.

DRIVE STARE

Method Se Sp AUC Acc Se Sp AUC Acc

Unsupervised B-COSFIRE 0.7655 0.9704 0.9614 0.9442 0.7716 0.9701 0.9563 0.9497
(Mendonca and 0.7344 0.9764 – 0.9463 0.6996 0.9730 – 0.9479
Campilho 2006)
(Martinez-Pérez et al. 2007) 0.7246 0.9655 – 0.9344 0.7506 0.9569 – 0.9410
(Al-Rawi et al. 2007) – – 0.9435 0.9535 – – 0.9467 0.9090
(Ricci and Perfetti 2007) – – 0.9558 0.9563 – – 0.9602 0.9584
(Lam et al. 2010) – – 0.9614 0.9472 – – 0.9739 0.9567
Supervised (Niemeijer et al. 2004) – – 0.9294 0.9416 – – – –
(Staal et al. 2004) – – 0.9520 0.9441 – – 0.9614 0.9516
(Soares et al. 2006) 0.7332 0.9782 0.9614 0.9466 0.7207 0.9747 0.9671 0.9480
(Ricci and Perfetti 2007) – – 0.9633 0.9595 – – 0.9680 0.9646
(Marin et al. 2011) 0.7067 0.9801 0.9588 0.9452 0.6944 0.9819 0.9769 0.9526
(Fraz et al. 2012) 0.7406 0.9807 0.9747 0.9480 0.7548 0.9763 0.9768 0.9534

CHASE_DB1 HRF

Method Se Sp AUC Acc Se Sp AUC Acc

Unsup. B-COSFIRE 0.7585 0.9587 0.9487 0.9387 0.7511 0.9745 0.9510 0.9644
(Odstrcilik et al. 2013) – – – – 0.7740 0.9669 0.9493 0.9668
Sup. (Fraz et al. 2012) 0.7224 0.9711 0.9712 0.9469 – – – –

5 CONCLUSIONS Azzopardi, G. & N. Petkov (2012). A CORF computa-

tional model of a simple cell that relies on LGN input
The results that we achieve on DRIVE, STARE, outperforms the Gabor function model. Biol Cybern
CHASE_DB1 and HRF data sets demonstrate 106(3), 177–189.
Azzopardi, G. & N. Petkov (2013). Trainable COSFIRE
the high effectiveness of the proposed method. filters for keypoint detection and pattern recognition.
B-COSFIRE filters are versatile as they can be IEEE Trans Pattern Anal Mach Intell 35, 490–503.
automatically configured to be selective for any Azzopardi, G., A. Rodrguez-Snchez, J. Piater, & N.
given vessel-like pattern, including bifurcations Petkov (2014, 07). A push-pull corf model of a simple
and crossovers. We have made the Matlab imple- cell with antiphase inhibition improves snr and con-
mentation available online3. The processing of a tour detection. PLoSONE 9(7), e98424.
B-COSFIRE filter is also very efficient. In fact, Azzopardi, G., N. Strisciuglio, M. Vento, & N. Petkov
the proposed method is the most time-efficient (2014). Trainable cosfire filters for vessel delineation
algorithm for vessels delineation in retinal fundus with application to retinal images. Medical Image
Analysis 19(1), 4657.
images published so far. Cinsdikici, M.G. & D. Aydin (2009). Detection of
blood vessels in ophthalmoscope images using mf/
REFERENCES ant (matched filter/ant colony) algorithm. Computer
Methods and Programs in Biomedicine, 85–95.
Fraz, M., P. Remagnino, A. Hoppe, B. Uyyanonvara, A.
Al-Diri, B., A. Hunter, & D. Steel (2009, Sept). An active
Rudnicka, C. Owen, & S. Barman (2012). An ensem-
contour model for segmenting and measuring retinal
ble classification-based approach applied to retinal
vessels. Medical Imaging, IEEE Transactions on 28(9),
blood vessel segmentation. IEEE Trans Biomed Eng
1488–1497.
59(9), 2538–2548.
Al-Rawi, M., M. Qutaishat, & M. Arrar (2007). An
Hoover, A., V. Kouznetsova, & M. Goldbaum (2000).
improved matched filter for blood vessel detection of
Locating blood vessels in retinal images by piecewise
digital retinal images. Computer in biology and medi-
threshold probing of a matched filter response. IEEE
cine 37(2), 262–267.
Trans Med Imaging 19(3), 203–210.
Jiang, X. & D. Mojon (2003). Adaptive local threshold-
ing by verification-based multithreshold probing with
3
http://www.mathworks.com/matlabcentral/fileexchange/ application to vessel detection in retinal images. IEEE
49172. Trans. Pattern Anal. Mach. Intell. 25(1), 131–137.

154

CH23-MI9449.indd 154 9/9/2015 9:34:32 PM

 Lam, B., Y. Gao, & A.-C. Liew (2010). General retinal Owen, C.G., A.R. Rudnicka, R. Mullen, S.A. Barman,
vessel segmentation using regularization-based mul- D. Monekosso, P.H. Whincup, J. Ng, & C. Paterson
ticoncavity modeling. IEEE Trans on Med Imaging (2009). Measuring retinal vessel tortuosity in 10-year-
29(7), 1369–1381. old children: validation of the computer-assisted
Marin, D., A. Aquino, M. Emilio Gegundez-Arias, & image analysis of the retina (caiar) program. Invest
J. Manuel Bravo (2011). A New Supervised Method Ophthalmol Vis Sci 50(5), 2004–10.
for Blood Vessel Segmentation in Retinal Images by Ricci, E. & R. Perfetti (2007). Retinal blood vessel seg-
Using Gray-Level and Moment Invariants-Based mentation using line operators and support vec-
Features. IEEE Trans Med Imaging 30(1), 146–158. tor classification. IEEE Trans Med Imaging 26(10),
Martinez-Pérez, M.E., A.D. Hughes, S.A. Thom, A.A. 1357–1365.
Bharath, & K. H. Parker (2007). Segmentation of Soares, J.V.B., J.J.G. Leandro, R.M. Cesar, Jr.,
blood vessels from red-free and fluorescein retinal H.F. Jelinek, & M.J. Cree (2006). Retinal vessel seg-
images. Medical Image Analysis 11(1), 47–61. mentation using the 2-D Gabor wavelet and super-
Mendonca, A.M. & A. Campilho (2006). Segmentation vised classification. IEEE Trans Med Imaging 25(9),
of retinal blood vessels by combining the detection of 1214–1222.
centerlines and morphological reconstruction. IEEE Staal, J., M. Abramoff, M. Niemeijer, M. Viergever, &
Trans Med Imaging 25(9), 1200–1213. B. van Ginneken (2004). Ridge-based vessel segmen-
Niemeijer, M., J. Staal, B. van Ginneken, M. Loog, & M. tation in color images of the retina. IEEE Trans Med
Abramoff (2004). Comparative study of retinal ves- Imaging 23(4), 501–509.
sel segmentation methods on a new publicly available
database. In Proc. of the SPIE—The International
Society for Optical Engineering, pp. 648–56.
Odstrcilik, J., R. Kolar, A. Budai, J. Hornegger, J. Jan,
J. Gazarek, T. Kubena, P. Cernosek, O. Svoboda, &
E. Angelopoulou (2013, Jun). Retinal vessel segmen-
tation by improved matched filtering: evaluation on
a new high resolution fundus image database. IET
Image Processing 7(4), 373–383.

155

CH23-MI9449.indd 155 9/9/2015 9:34:32 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

A registration algorithm for microscopy images of the capillary bed

H. Rieiro
Universidade de Vigo, Vigo, Spain
Barrow Neurological Institute, Phoenix, AZ, USA

J.L. Alba Castro

Universidade de Vigo, Vigo, Spain

S. Martinez-Conde & S.L. Macknik

SUNY Downstate Medical Center, Brooklyn, NY, USA
Barrow Neurological Institute, Phoenix, AZ, USA

ABSTRACT: We propose a new feature-based registration algorithm to corregister microscopy images

of brain microvasculature. This algorithm is a three-dimensional adaptation of SIFT, modified here to
further filter keypoint selection to highly distinctive regions of blood vessels, and to add three-dimensional
rotational invariance. We show the algorithm is robust against rotations and additive noise in two-photon
microscopy vasculature images. Future work will explore corregistration across different imaging modali-
ties based on this method.

1 INTRODUCTION those requiring the acquisition of data at distant

points in time (Holtmaat et al. 2009). We propose
Neuroimaging techniques are powerful tools in a feature-based registration algorithm that will
neuroscientific research. They are not only used provide an automatic and precise alignment of
to study neural disease and its models, but also images of the capillary bed acquired through
as tools to research normal behavior of the brain, microscopy techniques.
both from a structural point of view, such as
studies of homeostatic regulation, and also from
a functional perspective, since blood flow increases 2 ALGORITHM DESCRIPTION
and decreases are directly correlated to neural
activity (Kleinfeld et al. 1998; Chaigneau et al. An image contrast registration algorithm functions
2007). The most well-known functional imaging by maximizing a similarity metric such as mutual
technique, functional Magnetic Resonance Imag- information (Pluim et al. 2003). A full affine trans-
ing (fMRI), uses blood oxygenation and volume as formation model in three dimensions would thus
a proxy to study functional activity over large scale require minimization in a twelve-dimensional
brain areas (Hyder et al. 2001). space (rotations, scale, translation and shear along
There are a number of different imaging the three XYZ axes), which is computationally
techniques that can be employed to study blood very expensive. The usual way to mitigate this
flow, each of them with a set of advantages and problem is to use a multiresolution algorithm that
disadvantages. For example, fMRI can be used carries out the maximization across increasing
to study full brain activation with limited spatial levels of detail, potentially using simplified models
and temporal resolution, on the order of 1 mm3 at low resolution stages (Jenkinson & Smith 2001).
and 1 second respectively (Logothetis 2003). When we tried to apply this approach to images
Microscopy techniques, such as two-photon of the capillary bed, we found that, due to the
microscopy and confocal microscopy, work at small size of the capillaries and the high density
much higher spatial and temporal resolutions of the microvascular network, it was not possible
(Denk et al. 1990). This makes them excellent tools to downsample the images by a high enough factor
to study blood flow regulation and microvascular to improve the method performance while keeping
structure at the microscopic level. the results satisfactory.
Several types of studies of microscopic blood Instead, we developed an feature-based algo-
flow require alignment of images, for example rithm for registration of capillary images, based

157

VIPIMAGE15_Book.indb 157 9/8/2015 7:44:26 PM

 on a previous implementation of SIFT (Scale 2.2 Keypoint localization
Invariant Feature Transform) (Lowe 2004). SIFT,
Keypoints over the preprocessed image were
described originally to work in two-dimensional
initially localized using a method like the one
images, defines points in the image (keypoints) that
described by (Cheung & Hamarneh 2007),
are invariant to image scaling, and feature descrip-
where maxima and minima of the multiscale
tors that allow to obtain rotational invariance.
image were found over 3 different Difference of
We first tested a previous adaptation of SIFT for
Gaussians (DoG) scales (with σ = 1.5), and three
three-dimensional images (Cheung & Hamarneh
different image magnifications (0.5, 1, and 2). The
2009), but this implementation showed poor per-
keypoints found were further filtered using three
formance for our specific problem, specifically
application-specific criteria. We first removed
related to its rotational invariance and its capacity
those keypoints with negative amplitude in the
to extract robust features, so we enhanced it with
DoG images, to remove keypoints that would fall
application-specific knowledge.
outside the edge of a vessel, which therefore kept
only keypoints falling within a vessel. Second, we
2.1 Image preprocessing set up a minimum intensity threshold to remove
Two-photon images of brain microvasculature spurious maxima (0.1). Finally, we used a ves-
present two sources of noise relevant to the selness measure (Frangi et al. 1998) to keep only
registration algorithm. One is electrical noise in the keypoints that were localized in areas of the
the sensors, with the main effect of causing back- vessel that had high curvature, or, ideally, where
ground noise, which can have amplitude similar to the capillaries branched. The vesselness criteria
that of very small vessels. The second noise source, used was:
of biological origin, is due to small changes in R
serum concentration over time, which can produce − 2αa
V 1 e (1)
slight spatial mislocalizations of the features. These
sources of noise, combined with the reduction of where Ra is the ratio between the two eigenvalues
vessel brightness (signal) when imaging on deeper with the higher modulus of the Hessian matrix of
layers of tissue, can make small vessels hard to the image at the keypoint, and α is a constant set
detect based on their contrast alone. to 0.3. Figure 2 shows the result of the keypoint
To diminish the effect of these noise sources localization process.
we first filtered the images using a 3 × 3 × 3 voxel
median filter, and then transformed the filtered
image using a multilocal creaseness measure to 2.3 Modified descriptor
detect ridges and valleys (López et al. 1999). After localizing the keypoints we built the feature
Since the objects of interest were blood ves- descriptors as follows. The first step was to deter-
sels, which presented as bright structures over mine the keypoint orientation by building a
dark background, we further removed negative histogram of orientations of the gradients (azimuth
creaseness values from the image, which corre-
sponded to image valleys. The result was an image
with reduced background noise and enhanced
blood vessels contrast, in exchange for a small
blurring (Fig. 1).

Figure 1. Input image and result of applying the crease- Figure 2. Example of keypoint localization in a two-
ness transformation. The creaseness transformation has photon microscopy section. Grey dots over imposed on
a two-fold effect, blurring the image and removing most blood vessels mark the detected keypoints, which tend to
of the background noise. fall in the points of vessel branching.

158

VIPIMAGE15_Book.indb 158 9/8/2015 7:44:26 PM

 and elevation) in a region of 16 × 16 × 16 voxels that the azimuth information was ignored in bins
surrounding the keypoint. The histograms were with maximal and minimal elevation (±π). This is
weighed by the magnitude of their gradients and because at the extreme values of elevation the azi-
by the solid angle of the bin, to avoid giving too muth component becomes highly susceptible to
much weight to the low elevation bins (Scovanner noise, and therefore unreliable to use for matching.
et al. 2007). As in the original description of SIFT,
we allow each keypoint to have several features if 2.4 Keypoint matching and model estimation
any orientation has an amplitude over 80% of the
maximum value. We compared the keypoints across two images
While the original description of SIFT included (I1 and I2) using a histogram intersection metric
2-D rotational invariance, the base implementation (Sinha & Shukla 2013). The search for the best
we used for our three dimensional images (Cheung & neighbor (the keypoint in the second image that
Hamarneh 2009) did not provide such a feature. maximizes the histogram intersection) was per-
Whereas rotational invariance within two dimen- formed using a k-d tree based approximate method
sional images is relatively easy to implement due (Muja & Lowe 2009). To remove poor matches, we
to the fact that there is only one possible rotation enforced two conditions:
axis, when working with three dimensional images 1. We required the metric value for the best neigh-
this rotation axis can be any three dimensional bor to be at least 25% higher than the value for
vector. Three-dimensional rotations lack other the second best (Lowe 2004).
desirable properties such as commutability or the 2. Due to the set of keypoints between the images
capability of being easily decomposed in orthog- being aligned is not identical, it is possible that a
onal components. We opted to add rotational keypoint from I1 is being matched to a keypoint
invariance to our features by adapting the original, from I2 that does not exist in I1. To reduce this
two-dimensional mechanisms defined for the origi- problem, we removed cases in which the match
nal SIFT descriptor to the 3-D case. was not reciprocal (i.e. the nearest neighbor for
To achieve this, we rotated the feature gradients feature fA is fB, but the nearest neighbor for fB is
surrounding the keypoint so as the assigned feature a different feature fA’).
orientation was always aligned with the positive
direction of the X axis. The first step for this was From the matches of keypoints, we can estimate
to rotate by the opposite angle of azimuth (rotation the transformation between images using a least-
around Z axis), and elevation (around Y axis) of squares method, where an affine transformation
the feature. Since this mechanism only corrects rota- A is estimated from the matched points P (from
tion around two angles, an uncompensated rotation I1) and Q (from I2), corresponding to the first and
around the X axis remains for all gradients that fail second images, as:
to precisely correspond to the main orientation,
−1
causing different rotated versions of the same fea- Aˆ = PQt ⎛⎝⎜ QQt ⎞⎠⎟ (2)
ture, such as those found in two different images of
the same object, to have different orientation histo- Since the results of the matching process are
grams, reducing the registration robustness to rota- likely to produce a significant number of outliers,
tions. To compensate this problem, we projected the we used a RANSAC (Yaniv & Joskowicz 2005;
realigned gradients on the YZ plane and calculated Fischler & Bolles 1981) model estimation technique
the principal components of the resulting cloud of to minimize their effect in the estimated matrix.
points, to estimate the orientation of the main com-
ponent with regards to the positive direction of the
Y axis. We then used an additional rotation around
X by the opposite angle to align the main compo- 3 EXPERIMENTAL RESULTS
nent to the Y axis. We compensated for mismatched
signs on the main component vector between dif- We tested the developed algorithm using two-photon
ferent versions of the same feature by creating two images acquired from the mouse olfactory bulb.
different descriptors, one where the calculated main
component is aligned with the positive direction of
3.1 Animal preparation
the Y axis, and another where it is aligned to the
negative direction. C57B6 mice (n = 7) were anesthesized using a mix-
Finally, for each feature the 16 × 16 × 16 region ture of ketamine and xylazine. Following surgical
surrounding the keypoint was divided into 64 procedures similar to those described in (Kleinfeld
4 × 4 × 4 subregions, and 6 × 6 gradient orienta- et al. 1998), a small window (5 × 5 mm) was drilled
tion histograms, weighed by gradient magnitude, over the left or right olfactory bulb. The blood
were built for each subregion, with the particularity serum was labeled through tail vein injection of

159

VIPIMAGE15_Book.indb 159 9/8/2015 7:44:27 PM

 2000 kDa Tetramethylrhodamine, and two-photon images to rotated versions of themselves. The rota-
imaging of the microvasculature was acquired tions were performed around an arbitrary rotation
using a laser wavelength of 857 nm, with resolu- axis, with rotations around other axes yielding sim-
tion of 2 × 2 × 2 μm. ilar results. The values of the metric for this figure
are normalized for each image such that a value of
3.2 Experimental procedure one (maximum radius in the figure)represents the
maximum mutual information possible for each
To test the performance of our corregistration (i.e. the image entropy), to make the average and
method when using biological images, and specifi- confidence intervals more meaningful when aver-
cally to its robustness to changes in orientation, we aging over images with disparate contrasts. The
compared each image to rotated versions of itself method shows a uniform behavior over the range
around an arbitrary axis. We used the normalized of three-dimensional rotations. In addition, regis-
mutual information as a measure of registration tration of unrelated images provides much lower
quality, since it is a measure commonly used in metric values, as represented by the darker line in
other registration problems (Pluim et al. 2003). Figure 3, which are used as the baseline for a bad
We set an intensity threshold of 0.1 for registration.
keypoints in the DoG space. Keypoints where the To test the algorithm robustness against noise,
vesselness value was below 0.5 were removed to we tested a 30 degree rotation while adding additive
avoid finding keypoints in straight vessel segments. gaussian noise of varying energy (standard devia-
The estimation of the model was performed using tion between 0.02 and 0.1) to both the original and
a RANSAC algorithm where the probability of rotated images. The images themselves, without
outliers in the final estimation was set to be less any added noise on top of that coming from the
than 0.05, and with matches classified as outliers acquisition itself, have a standard deviation in the
when the Euclidean distance between the original 0.04–0.2 range, so the noise levels used have sig-
and transformed keypoint was above 5 μm. nificant magnitude. Figure 4 shows the values of
the non-normalized mutual information metric
3.3 Results when performing the registration under this con-
ditions, with each line representing a single image
Figure 3 shows the average values of the mutual used. This representation allows us to examine the
information metric between images when registering robustness of the algorithm to noise as a function
of the quality of the non-noisy registration. We
can see that the algorithm is robust to the noise
levels tested, with values of the registration metric

Figure 4. Registration results with additive gaussian

Figure 3. Results of registering images to rotated versions noise over a 30 degree rotation. The value of the mutual
of themselves (gray) and unrelated images (black). Dashed information between the original and transformed image
lines show the 95% confidence interval for the registration is shown for different standard deviations for the noise.
metric values. Values of the mutual information metric are Each line represents the result for a different image, each
normalized to their maximum, which coincides with zero from a different animal. The algorithm is robust to noise
rotation (registration with itself). in images having good contrast.

160

VIPIMAGE15_Book.indb 160 9/8/2015 7:44:27 PM

 holding steady for noise in the lower and middle Frangi, A.F. et al., 1998. Multiscale vessel enhancement
range except for the lowest contrast image (bottom filtering. Medial Image Computing and Computer-
line, Fig. 4). Assisted Invervention—MICCAI’98. Lecture Notes in
Computer Science, vol 1496, 1496, pp.130–137.
Holtmaat, A. et al., 2009. Long-term, high-resolution
imaging in the mouse neocortex through a chronic cra-
4 CONCLUSIONS nial window,
Hyder, F. et al., 2001. Quantitative functional imaging of
We have developed an algorithm that provides the the brain: towards mapping neuronal activity by BOLD
capability of automatically align images of blood fMRI. NMR in biomedicine, 14(7–8), pp.413–31.
vessels acquired through microscopy techniques. Jenkinson, M. & Smith, S., 2001. A global optimisation
The algorithm is robust to changes in orientation method for robust affine registration of brain images.
over the whole 3-D range, and shows good behavior Medical image analysis, 5(2), pp.143–56.
against the addition of significant levels of gaus- Kleinfeld, D. et al., 1998. Fluctuations and stimulus-
induced changes in blood flow observed in individual
sian noise. capillaries in layers 2 through 4 of rat neocortex. Pro-
Future research lines will test the robustness of ceedings of the National Academy of Sciences of the
the method on images taken at disparate points United States of America, 95(26), pp.15741–15746.
in time, its behavior and changes in preprocessing Logothetis, N.K., 2003. The underpinnings of the BOLD
needed to apply it to images acquired through dif- functional magnetic resonance imaging signal. The
ferent modalities such as confocal microscopy, and Journal of neuroscience: the official journal of the
finally investigate the possibility of registering the Society for Neuroscience, 23(10), pp.3963–71.
images acquired across those different modalities. López, A.M. et al., 1999. Evaluation of methods for ridge
and valley detection. IEEE Transactions on Pattern
Analysis and Machine Intelligence, 21(4), pp.327–335.
Lowe, D.G., 2004. Distinctive Image Features from
REFERENCES Scale-Invariant Keypoints. International Journal of
Computer Vision, 60(2), pp.91–110.
Chaigneau, E. et al., 2007. The relationship between blood Muja, M. & Lowe, D.G., 2009. Flann, fast library for
flow and neuronal activity in the rodent olfactory bulb. approximate nearest neighbors. In International Con-
The Journal of neuroscience: the official journal of the ference on Computer Vision Theory and Applications
Society for Neuroscience, 27(24), pp.6452–60. (VISAPP’09).
Cheung, W. & Hamarneh, G., 2007. n-SIFT: n-dimensional Pluim, J.P.W., Maintz, J.B.A. & Viergever, M. a, 2003.
scale invariant feature transform for matching medi- Mutual-information-based registration of medical
cal images. In Proc. 4th IEEE Int. Symp. Biomedical images: a survey. IEEE transactions on medical
Imaging: Biomedical Imaging: From Nano to Macro. imaging, 22(8), pp.986–1004.
pp.720–723. Scovanner, P., Ali, S. & Shah, M., 2007. A 3-dimensional
Cheung, W. & Hamarneh, G., 2009. n-SIFT: n-dimensional sift descriptor and its application to action recogni-
scale invariant feature transform. IEEE transactions tion. 15th international conference on Multimedia, (c),
on image processing: a publication of the IEEE Signal pp.357–360.
Processing Society, 18(9), pp.2012–21. Sinha, A.K. & Shukla, K.K., 2013. A Study of Distance
Denk, W., Strickler, J.H. & Webb, W.W., 1990. Two- Metrics in Histogram Based Image Retrieval. Inter-
photon laser scanning fluorescence microscopy. national Journal of Computers & Technology, 4(3),
Science, 248(4951), pp.73–76. pp.821–830.
Fischler, M. & Bolles, R., 1981. Random sample Yaniv, Z. & Joskowicz, L., 2005. Precise robot-assisted
consensus: A paradigm for model fitting with appli- guide positioning for distal locking of intramedullary
cations to image analysis and automated cartography. nails. IEEE transactions on medical imaging, 24(5),
Communications of the ACM, 24(6), pp.381–395. pp.624–635.

161

VIPIMAGE15_Book.indb 161 9/8/2015 7:44:27 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Bilateral filtering based biomedical image colorization

A. Popowicz & B. Smolka

Institute of Automatic Control, Silesian University of Technology, Akademicka, Gliwice, Poland

ABSTRACT: Most of biomedical images are monochromatic and thus their assessment may be sig-
nificantly improved by means of the colorization techniques. Such colorized versions of images not only
benefit from more pleasing appearance, but also some fine details are easily noticeable, what may substan-
tially improve the diagnosis. In this article we present a novel method of colorization, which is based on
a combination of distance transformation and modified bilateral filtering approach. Tunable parameters
are introduced to allow for precise adjustment of final colorization outcomes. Our proposed algorithm
enables a simple and efficient region highlighting and interactive segmentation of biomedical images. It
may be also employed for the educational and presentation purposes. We show several examples of appli-
cations of our algorithm using various biomedical images obtained by ultrasonography, transmission
electron microscopy, computer tomography, magnetic resonance and infrared imaging.

1 INTRODUCTION color reference to the gray scale target image uti-

lizing the analysis of intensity statistics in distinct
Biomedical images obtained using the most wide- image patches. This approach was also adopted for
spread imaging techniques are usually presented in the colorization of biomedical images in (Zhao,
a gray scale form. As examples, we may enumerate Wang, Jin, & Shi 2007).
the techniques such as the Computer Tomography Another automatic colorization algorithms can
(CT), the Ultrasonography (USG), the Magnetic be found in (Zhen, Yan, & Lizhuang 2012) and in
Resonance (MRI), the Transmission Electron (Yan & Kankanhalli 2003). The authors of the first
Microscopy (TEM) or the Infrared Imaging (IR) paper suggest the pattern continuity approach in
(Bankman 2000, Lin, Olivo, & Kung 2009, Walus M. contrast to traditional luminance-based schemes.
2015). Some new techniques emerge, like the reti- The second publication presents the algorithm for
nal optical coherence tomography (Acharya U, infrared image colorization, wherein the reference
Yun, Ng, Yu, & Suri 2008) or elastography (Ophir, image is obtained from the flash-assisted frame.
Cspedes, Ponnekanti, Yazdi, & Li 1991), which One of the most prominent semiautomatic
also provide gray scale images (or video sequences) colorization method was presented in (Levin,
as the imaging output. Lischinski, & Weiss 2004a, Levin, Lischinski, &
It is worth to notice, that the images presented in Weiss 2004b). The authors utilize the minimiza-
the gray scales are mostly less perceptible, than the tion of color difference between adjacent pixels
ones with colors included. Therefore, the coloriza- and assume that the neighboring pixels with simi-
tion techniques are important tools utilized dur- lar intensities should also be given the same colors.
ing the image assessment phase. They may also Their proposed minimization formula may be
improve the accuracy of the diagnosis. easily solved using typical methods, like the Least
We can divide the colorization techniques into Square algorithm.
two categories: automatic and semiautomatic In several papers (Lagodzinski & Smolka 2014,
(interactive). Although the methods from the first Yatziv & Sapiro 2006, Popowicz & Smolka 2014)
category allow for colorization without any user the authors investigate the digital paths between
effort, they usually provide less appealing results. adjacent image pixels taking into account the
Moreover, the physicians prefer to supervise any encountered intensity gradients. They obtain
image processing procedures, thus the semiauto- the degree of the membership of pixels to each
matic methods are usually the solution of choice. of inserted scribbles and estimate the unknown
The pioneer colorization algorithm (Welsh, color using the weighted averaged of all scribbled
Ashikhmin, & Mueller 2002) was based on colors.
the color transfer idea presented in (Reinhard, In this article we present a novel method of
Ashikhmin, Gooch, & Shirley 2001). In this origi- semiautomatic image colorization based on the
nal method, the colors were transferred from the algorithms presented in (Lagodzinski & Smolka

163

VIPIMAGE15_Book.indb 163 9/8/2015 7:44:27 PM

 2014, Popowicz & Smolka 2014). We combine the
distance transformation (Rosenfeld & Pfaltz 1968)
and bilateral filtering (Tomasi & Manduchi 1998)
to achieve high quality colorization outcomes. In
the next Sections of this paper we show the basics
steps of our concept, present the role of tunable
parameters and exhibit exemplary results of appli-
cation of the algorithm using various biomedical
images. Figure 2. The masks utilized in double-scan algorithm.

2 ALGORITHM

The presented colorization method method belongs

to the family of semiautomatic approaches, in
which the user indicates the color hints of so-called
scribbles, which provide the information about
required color in a given region of the image. Our
aim is to assign an unknown chrominance value to
each image pixel using the colors from the nearest
scribbles. The weights utilized in such color blend- Figure 3. Direction of mask sliding in double-scan
ing are obtained by the magnitude of the intensity algorithm.
gradients encountered between the pixel and a
given scribble. Therefore, the weights correspond
with the membership measure of pixels to the where di denotes the distance for pixel Pi in corre-
inserted scribbles. The main steps of our algorithm sponding mask (see Fig. 2). Eventually, after both
are presented schematically in Figure 1. scans, the estimations of Euclidean distance are
In the first step, we calculate the topographic obtained (see the result in Fig. 4d). The procedure
distances on the image domain between scribbles is repeated for each of inserted scribbles.
of the same color and any other pixel within the After the distance calculations, we initialize
image. To do so, we employ the distance trans- the membership arrays for each scribbled color.
formations (Borgefors 1986) and the double-scan In such an array all the pixels are initially given 0
algorithm (Rosenfeld & Pfaltz 1968). In this algo- membership, except for the scribbled pixels, which
rithm we utilize two processing masks, which receive 1 (full membership). We also make use
are used in respectively first and second scan of the previously calculated metric distances to
(see Fig. 2). The scans are performed from the obtain the order of pixels, which will be processed
left to right, up to bottom (first scan) and in the in the bilateral filtering membership estimation,
opposite direction—right to left, bottom to top explained below. The initial membership array
(second scan). For details of scans directions see for the scribble from Figure 4 with the processing
Figure 3. order depicted by the filled gray levels, is presented
In the double scan algorithm, initially all the in Figure 5.
pixels are given infinite distance value except for In the next part of the algorithm we perform a
the pixels belonging to the scribble, which are given modified version of classic bilateral filtering. The
0 (see Fig. 4b). On such initialized distance matrix, bilateral filtering is nonlinear image smoothing
the double scan algorithm slides twice, performing algorithm developed for suppression of Gaus-
following distance replacement in each step: sian noise in both gray scale and color images
(Tomasi & Manduchi 1998). The idea is based
d0 d1, d2 d3 , d 4 } (1) on the non-linear, adaptive filtering, in which the
local window slides the image matrix, replacing the
intensity of its central pixel with the weighted aver-
age of intensities of other pixels in window:

N2
I0 ∑ I k wk (2)
k =1

where I0 is the filtered intensity of central pixel in

Figure 1. Three-step algorithm schema. N × N local window, Ik and wk denote respectively the

164

VIPIMAGE15_Book.indb 164 9/8/2015 7:44:28 PM

 linear filtering algorithms, is capable to preserve
the sharp edges in filtered outcomes.
We utilize the presented filtering idea for the cal-
culation of membership of a given pixel to scrib-
bled color. However, instead of using the weighted
average, we replace the membership by the maxi-
mum over the weighted membership values in a
local filtering window. Similarly to the bilateral fil-
tering approach, the weights are obtained from the
intensity difference between the pixels. In each step
of sliding window, the corresponding membership
value replacement is performed:

⎧ ⎛ ( I I )2 ⎞ ⎫
M0 a ⎨M k exp ⎜ − 0 2k ⎟ ⎬ , (4)
k , ,N 2
⎩ ⎝ 2σ ⎠⎭

where M0 is the new estimation of membership

of central pixel in a window, M k ( k ) is the
Figure 4. An exemplary application of distance membership of k-th pixel in a window, Ik stands
transformations. for the intensity of k-th pixel and σ is the param-
eter responsible for the strength of membership
decrease.
In contrast to classic image filtering methods, in
our algorithm the window slides through the pixels
in the order defined by previously calculated met-
ric distance (as shown in Fig. 4d and visualized by
shades of gray in Fig. 5). Therefore, our bilateral
filtering window replaces the membership values
starting from pixels neighboring with a scribble
and moves gradually toward the image boarders.
This bilateral filtering based procedure assigns
higher membership values to pixels, which are
connected with scribble by a digital, pixel-to-
pixel path, within which the intensity changes
smoothly. The image regions, which are separated
from a scribble by high intensity gradients (like the
boundaries of an image region), receive very low
membership coefficients. Some membership arrays
for four scribbled colors in an exemplary gray scale
Figure 5. Initial membership array with order of pixel microscopy image of blood cells are presented in
processing indicated by the gray scale levels. Figure 6.
The final step of the algorithm is the assignment
of chrominance to each pixel. We perform the fol-
intensity and the weight of corresponding k-th pixel lowing color blending using weights taken directly
in the window. In the bilateral filtering the weights from the membership arrays:
are inversely proportional to the intensity differences
and the metric distances to a central pixel:
CP =
∑ s S Cs M s , (5)
1 N2 ⎛ (I I ) 2⎞
⎛ d ⎞
∑ s S Ms
I0 = ∑ I k exp ⎜⎝ − 02σ 2k ⎟⎠ exp ⎜⎝ − 2σk2 ⎟⎠ ,
Z k =1
(3)
I d where CP is an assigned chrominance of pixel P,
Cs is the chrominance of scribbled color s from the
where Z is normalizing factor, dk is the metric set S of all scribbled colors, Ms is the membership
distance between k-th and central window pixel, of pixel P to scribbled color s. After such a color
σI and σd tune the influence of respectively, the blending procedure, each pixel receives the color in
intensity difference and the pixels distance on final accordance to its membership value to the corre-
smoothing. The bilateral filtering, in contrast to sponding color scribble.

165

VIPIMAGE15_Book.indb 165 9/8/2015 7:44:29 PM

 Figure 7. Examples of utilization of various mask sizes
N: (a) is the original USG image with indicated color
scribbles depicted also separately in (b); (c), (d) and (f),
(g) are the membership maps, respectively for 3 × 3 and
Figure 6. Exemplary membership arrays. The gray scale 7 × 7 masks; (e) and (h) are the corresponding coloriza-
in presented arrays was chosen so that white and black tion outcomes.
level correspond respectively to full and no membership
(M = 1 and M = 0).
The second parameter σ is responsible for the
degree of final color blending. The higher the σ,
3 TUNABLE PARAMETERS the more blurred is the final colorization outcome.
For very small σ values, the edges between the
Our method enables fine tuning of colorization regions of different colors become clearly visible.
outcomes by modifying the size of the window For higher σ the blending is stronger and the color-
(N) and the parameter σ in the bilateral filtering ization result becomes in general more pleasing for
based membership replacement (4). While the human perception. Two extreme values of σ were
first parameter influences the algorithm robust- used for the examples depicted in Figure 8.
ness against the noise, the second one modifies the We present an example of image colorization
strength of final color blending. quality as a function of both tunable parameters. To
The size of the window (N) allows to set the size evaluate the quality of the colorization, we utilized
of small details, which will be neglected during the known color image reference. Such test image,
the memberships assignment. For large windows, the depicted in Figure 9, was converted to gray scale (by
algorithm will be robust against the local noise and leaving only the luminance channel in YCbCr color
will be able to pass over thin lines that are associated space), scribbled and colorized by our algorithm. The
with noise rather than with the scene objects. On the outcomes were compared with the original image
other hand, reducing the window size to the mini- employing Peak-Signal-to-Noise (PSNR) measure:
mum (i.e. N = 3) may be desirable if one wishes to
colorize some tiny objects. The membership arrays 1 X Y
and colorization results of ultrasound image of MSE = ∑ ∑ ∑ [ I rc (i , j ) − I sc (i, j )]2 ,
3XY c∈{R ,G ,B } i =1 j =1
(6)
breast mass for different N parameters are depicted
in Figure 7. As it can be seen, the larger mask (7 × 7)
enabled the accurate colorization of whole region of ⎛ 2552 ⎞
PSNR = 10 log ⎜ ⎟, (7)
breast mass starting from very simple scribbles. ⎝ MSE ⎠

166

VIPIMAGE15_Book.indb 166 9/8/2015 7:44:31 PM

 biomedical images, the optimal parameters should
be found within the same range.

4 COMPARISON WITH OTHER

COLORIZATION TECHNIQUES

We compared the quality of colorization of our

method with the results of the state-of-the-art
algorithms. For this purpose we either imple-
Figure 8. Colorization results for various σ param- mented or utilized the available code for the
eter values. The scribbled original image can be seen in
following methods: Colorization Using Optimi-
Figure 6a.
zation (Levin, Lischinski, & Weiss 2004a, Levin,
Lischinski, & Weiss 2004b) (CUO), Chrominance
Blending Colorization (CBC) (Yatziv & Sapiro
2006), and Distance Transformation based Colori-
zation (DTC) (Lagodzinski & Smolka 2014).
In our experiment the microscopic gray scale
image was scribbled using 8 different colors (70 scrib-
bles, 680 × 490 pixels), as presented in Figure 11a.
Then, for each method, we modified its parameters,
so that the outcomes were visually most attractive.
The final results are depicted in Figure 11. The com-
Figure 9. Original and scribbled gray scale test image putational time for each method is given in Table 1.
samples utilized for colorization quality evaluation. As it can be noticed, the colorized versions of
gray scale image are of very similar quality. It con-
firms, that our method is capable to provide fine
and comparable colorization outcomes. Some

Figure 10. Colorization quality as a function of tunable

parameters when colorizing the image Figure 9b.

where X,Y are width and height of the image, c is

color channel (R, G or B), I rc (i j ) is the intensity
of pixel at position (i,j) in reference color image
(color channel c), I sc (i j ) is the same as above but
for the colorized version of the image.
In our tests, we used σ range from 2 to 20 and
window sizes respectively 3, 5, …, 23. In such
parameters space we managed to find an optimal
solution, which, as it can be observed in Figure 10,
was obtained for N = 5 and σ = 14. Extensive
experiments revealed that for most of typical, prop- Figure 11. The outcomes of colorization of biomedical
erly sampled (i.e. no over- and under-sampling) microscopy image provided by various algorithms.

167

VIPIMAGE15_Book.indb 167 9/8/2015 7:44:35 PM

 Table 1. Computational time comparison. imaging purposes. They include miscellaneous
images obtained by various imaging techniques:
Method CUO CBC DTC Proposed USG, TEM, CT, MRI and IR. We inserted sev-
eral scribbles for better discernibility of dis-
Evaluation time 21 s 105 s 8s 44 s
tinct regions, depicting tissues, cells, human
organs etc.

5 SUMMARY

In the paper we have proposed a novel approach

to colorization of biomedical images. Our method
employs simple operations within a digital image,
which involve the distance transformations and
the replacement procedures based on the bilateral
filtering scheme. The tunable parameters enable
adjusting the final colorization outcomes, so that
they meet the application requirements.
We presented some examples of applications of
our method to images obtained by different tech-
niques widely used in the medicine. The proposed
algorithm allowed for achieving fine and medi-
cally valuable colorization results comparable with
the results of other state-of-the-art algorithms.
We plan to extend the capabilities of the algo-
rithm after investigating the impact of non-square
bilateral processing windows and different kernel
functions.

ACKNOWLEDGMENT

The research leading to these results has received

funding from the Polish-Norwegian Research
Programme operated by the National Center for
Research and Development under the Norwegian
Financial Mechanism 2009–2014 in the frame of
Project Contract No. Pol-Nor/204256/16/2013.
This work was performed using the infra-
structure supported by POIG.02.03.01-24-099/13
grant: GeCONiI-Upper Silesian Center for
Computational Science and Engineering.
Figure 12. Examples of colorization of biomedical
Ultrasound image depicted in Figure 12,
imaging by our algorithm. Corresponding images include
(from up to bottom): USG image of human joint, the was created for MEDUSA project at Section
TEM image of a human leukocyte, a CT slice through for Rheumatology; Department for Neurology,
the upper human abdomen, MRI image of human brain, Rheumatology and Physical Medicine, Central
infrared finger vascular system. Hospital, Førde, Norway.

slight unwanted color leakage may be noticed REFERENCES

for each of employed method, however it may be
reduced by better scribbles placing. The evaluation Acharya U, R., W.L. Yun, E.Y. Ng, W. Yu, & J.S. Suri
(2008, August). Imaging systems of human eye:
time differs significantly and indicates, that our
A review. J. Med. Syst. 32(4), 301–315.
proposed algorithm (44 s) is much slower than the Bankman, I.N. (Ed.) (2000). Handbook of Medical Imag-
fastest method DTC (8 s), but still is more compu- ing. Orlando, FL, USA: Academic Press, Inc.
tationally efficient than CBC (105 s). Borgefors, G. (1986, June). Distance transformations in
In Figure 12 we present some examples of digital images. Comput. Vision Graph. Image Process.
applications of our method for the medical 34(3), 344–371.

168

VIPIMAGE15_Book.indb 168 9/8/2015 7:44:39 PM

 Lagodzinski, P. & B. Smolka (2014, March). Application Tomasi, C. & R. Manduchi (1998, Jan). Bilateral filtering
of the extended distance transformation in digital for gray and color images. In Sixth International
image colorization. Multimedia Tools Appl. 69(1), Conference on Computer Vision, 1998, pp. 839–846.
111–137. Walus M., Bernacki K., K.J. (2015, March). Nir finger
Levin, A., D. Lischinski, & Y. Weiss (2004a, August). vascular system imaging in angiology applications.
Colorization using optimization. ACM Trans. Graph. In Mixed Design of Integrated Circuits and Systems
23(3), 689–694. (MIXDES), 2015 22nd International Conference on.
Levin, A., D. Lischinski, & Y.Weiss (2004b). Colorization Welsh, T., M. Ashikhmin, & K. Mueller (2002, July).
using optimization. In ACM SIGGRAPH 2004 Transferring color to greyscale images. ACM Trans.
Papers, SIGGRAPH ’04, New York, NY, USA, pp. Graph. 21(3), 277–280.
689–694. ACM. Yan, W.-Q. & M. Kankanhalli (2003, July). Colorizing
Lin, F., M. Olivo, & S.-Y. Kung (2009). Biomedical Imag- infrared home videos. In Multimedia and Expo, 2003.
ing (1st ed.). Springer Publishing Company, Inc. ICME ’03. Proceedings. 2003 International Conference
Ophir, J., I. Cspedes, H. Ponnekanti, Y. Yazdi, & X. Li on, Volume 1, pp. I–97–100 vol.1.
(1991). Elastography: A quantitative method for Yatziv, L. & G. Sapiro (2006, May). Fast image and video
imaging the elasticity of biological tissues. Ultrasonic colorization using chrominance blending. Image
Imaging 13(2), 111–134. Processing, IEEE Transactions on 15(5), 1120–1129.
Popowicz, A. & B. Smolka (2014, March). Isoline Zhao, Y., L. Wang, W. Jin, & S. Shi (2007, May). Colorizing
based image colorization. In Computer Modelling biomedical images based on color transfer. In Complex
and Simulation (UKSim), 2014 UKSim-AMSS 16th Medical Engineering, 2007. CME 2007. IEEE/ICME
International Conference on, pp. 280–285. International Conference on, pp. 820–823.
Reinhard, E., M. Ashikhmin, B. Gooch, & P. Shirley Zhen, Z., G. Yan, & M. Lizhuang (2012, July). An
(2001, September). Color transfer between images. automatic image and video colorization algorithm
IEEE Comput. Graph. Appl. 21(5), 34–41. based on pattern continuity. In Audio, Language
Rosenfeld, A. & J. Pfaltz (1968). Distance functions on and Image Processing (ICALIP), 2012 International
digital pictures. Pattern Recognition 1(1), 33–61. Conference on, pp. 531–536.

169

VIPIMAGE15_Book.indb 169 9/8/2015 7:44:42 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Comparison of the internal structures of bones by microtomography

J.C.L. Stutz
Universidade do Estado do Rio de Janeiro—Instituto Politécnico, Nova Friburgo, Rio de Janeiro, Brasil
Universidade Estácio de Sá, Nova Friburgo, Rio de Janeiro, Brasil

J.S. Domínguez & J.T. de Assis

Universidade do Estado do Rio de Janeiro—Instituto Politécnico, Nova Friburgo, Rio de Janeiro, Brasil

ABSTRACT: The use of computed microtomography (μCT) has revolutionized many areas of research,
such as non-invasive, fast and high precision techniques, which allows immediate visualization of internal
structures without any risk to the object of study. Its usefulness is undeniable for medicinal purposes, is also
important for purposes of zoological research and paleontological. In this work, we used the μCT to inves-
tigate the internal structure of bones from mammalian and poultry. We studied the bones of rats and quail.
Through of microtomography images, we observed that the bones of the poultry have a bony structure in
the form of a trellis that is not present in bones of mammals. These trellises bony is an evolutionary adapta-
tion that allowed the bones of the birds they become longer and lighter maintaining its strength. It was also
observed that the percentage of the trabecular area in poultry is almost half of that observed in mammals.
The results obtained validate the use of μCT as a technique that allows the study of bone structures in small
samples. Enabling explore the morphological differences between the bones of those species.

Keywords: microtomography; bones; X-ray

1 INTRODUCTION poultry bones, using rat bones (RattusNorvegicu)

and quail bones (CoturnixCoturnix).
The computed microtomography (μCT) is a non- The main motivation of this work is to explore
invasive method, which allows the study of the the internal morphological structures of pneumatic
complex internal structures of an object. This bones of birds and compare them with mammals.
technique has been used in various areas of knowl- The pneumatization of bird bones is linked to its
edge. However, the biomedical applications are the ability to fly, to provide strength and lightness of
most common (Morton et al., 1990). The μCT is his bone structure.
being used in many medical studies as osteoporosis
(Lima et al., 2006), osteoarthritis (Amanai et al.,
2 MATERIALS AND METHODS
2008), pseudoarthrosis (Qu and Schroeder, 2008),
fragility and fracture risk (Hanson and Bagi, 2004)
We analyzed four samples, two bones of rats and
and the differences between healthy bones and
two bones of poultries. Figure 1 is a 3D rendering
unhealthy bones (Lima et al., 2011). These stud-
of the samples used.
ies use μCT to investigate the internal structures of
To explore the samples of rat bones we use the
osseous tissue.
scanner Skyscan model 1174, operated at 50 kV,
In microtomography samples are explored using
some kind of radiation to produce detailed images of
axial body. μCT can be analyzed as a two-step process:
exploitation of samples and image reconstruction.
During exploitation, the bodies of the samples under
study are radiographed at different angles generating
a set of radiographs. These radiographs are digit-
ally processed in tomographic image reconstruction
process to generate a three-dimensional image of the
interior of the object under study from a series of
two-dimensional images.
In this work, we use the μCT to compare the
internal structure of mammalian bones with the Figure 1. Scanner Skyscan model 1174.

171

VIPIMAGE15_Book.indb 171 9/8/2015 7:44:42 PM

 Table 1. Dimensions of the samples.

Number Pixel size Size in

Sample of slices (μm) mm

Poultry1 2020 22.08 44.61

Poultry 2 2033 22.08 44.90
Rat 1 1323 33.29 44.04
Rat 2 1149 33.29 38.25

Figure 3. Poultry bone slices examples.

Figure 2. 3D rendering of samples: quail bones a and b,

rat bones c and d.

with aluminum filter of 0.75 mm, and pixel size of

33.29 μm. A scanner Skyscan model 1173, oper-
ated at 50 kV, with aluminum filter of 1.00 mm and
pixel size of 22.08 μm, produced the radiographs
of poultry bones.
Table 1 summarizes the dimensions of the Figure 4. Examples of rat bone slices.
samples and the number of horizontal slices
reconstructed from radiographs obtained in the
exploration of samples. Table 2. Percentage of different types of internal bone
structures.

3 RESULTS AND DISCUSSION % trabecular % trellis % cortical

Sample area area area
Figure 3 shows the poultry bones and their slices. Poultry 1 27.48 31.63 40.89
In her is possible to observe the bone of the bird at Poultry 2 32.86 24.5 42.65
its extremity, which is formed by a trabecular bony Rat 1 51.17 0 48.83
structure, followed by a trellis section followed by Rat 2 56.22 0 43.86
a cortical section. Figure 4 shows the rat bones and
its slices, can see that the bone also begins with a
trabecular bone structure, but followed by cortical
section of bone, not presenting in its interior a trel- Based on this set of slices the 3D volume was
lis structure. rebuilt and is represented in Figure 7. In this image
As we can see from Table 2, there is a significant can be observed clearly the structural nature of the
decrease in the area of trabecular bone of the poul- bone trellis.
try, this area is replaced by bone trellises. Those Comparing the trellis region with the trabecular
trellises are displayed in 3D, from different angles region of the quail bone, a huge difference between
for better viewing in Figure 5. the two regions is observed. The trabecular region
For a better view of the trellis region was cho- of the bone, by its definition, has a spongy structure
sen and separate a section of quail 1, composed of without defining pattern filling the entire inside of the
40 slices, displayed in Figure 6. The selected area bone. The trellis region has a large empty area with
corresponds to a section of ∼0.883 mm or ∼2% of several narrow and straight structures in the form
the sample area and ∼6.41% of the trellis region in of pillars which support between themselves and
the sample. inner walls of the bone in a triangular arrangement.

172

VIPIMAGE15_Book.indb 172 9/8/2015 7:44:42 PM

 Table 3. Morphometric parameters of the samples.

Poultry 1 Poultry 2 Rat 1 Rat 2

3
TV (μm ) 4.01E+12 4.11E+12 4.74E+12 3.56E+12
BV (μm3) 1.68E+11 1.45E+11 2.36E+11 2.77E+11
BV/TV 4.1850 3.5274 4.9816 7.7837
BS (μm2) 1.84E+09 2.09E+09 1.97E+09 2.07E+09
BS/BV 0.01099 0.0144 0.00833 0.007443
(1/μm)

cubes of the model. The BV is the total volume of

binarised objects within the VOI. The BV/TV is
the proportion of the VOI occupied by binarised
solid objects. The BS is the surface area of all the
Figure 5. Rendering 3D bone trellis at different angles. solid objects within the VOI and the BS/BV is the
surface to volume ratio or “specific surface” is a
useful basic parameter for characterizing the thick-
ness and complexity of structures.

4 CONCLUSION

These results show that by using μCT images you

can view and separate the existing bone trellises
inside the long bones of poultries.
With the segmentation of these structures will
Figure 6. Set of slices of trellis area of Poultry1. be possible, using computational tools to study
the load capacity and resistance they provide to
the bone. This information may be used to com-
pare with the bones of other birds and so describe
the different adaptations in bone structures that
contribute to their flight.

REFERENCES

Amanai, T., Nakamura, Y., Aoki, S., Mataga, I., (2008).

Micro-CT analysis of experimental Candida osteoar-
thritis in rats. Mycopathologia 166, 133–141.
Hanson, N.A., Bagi, C.M., (2004). Alternative approach
to assessment of bone quality using micro-computed
tomography, Bone 35, 326–333.
Figure 7. 3D volume based on a set of slices of trellis Lima, I.C.B., Oliveira, L.F., Lopes, R.T., (2006). Bone
area. architecture analyses of rat femur with 3D microto-
mographics images. J. Radio anal. Nucl. Chem. 269,
639–642.
Lima, I., Taam, P., da Costa, V., Fleiuss, M.F.,
This kind of structure reduces the amount of bone Rosenthal, D., Lopes, R.T., (2011). Study of strontium
volume, making the bone bird lighter. ranelate bone issues by X-ray microtomography. Nucl.
In Table 3 the values of five morphometric Instrum. Methods Phys. Res. A 652, 779–782.
parameters used in bone assessment studies are Morphometric parameters measured by Skyscan™ CT-
presented: Total Volume (TV), Bone Volume (BV), analyser software. Disponível em: <www.skyscan.be/
Bone Volume to Tissue Volume (BV/TV), Bone next/CTAn03. pdf>. Acesso em: junho de 2015.
Surface (BS) and Bone Surface to Bone Volume Morton, E.J., Webb, S., Bateman, J.E., Clarke, L.J.,
Shelton, C.G., (1990). Threedimensional X-ray micro-
(BS/BV). These values were calculated by CTAn® tomography for medical and biological applications.
software that composes the software package of Phys. Med. Biol. 35, 805–820.
Skyscan scanner used at work. Qu, G., Schroeder, H.P.V., (2008). Trabecular microstruc-
The TV is the total volume 3D Volume-of- ture at the human scaphoid Nonunion. J. Hang Surg.
Interest (VOI) it is measured based on the marching 33, 650–655.

173

VIPIMAGE15_Book.indb 173 9/8/2015 7:44:44 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Model adaptation for mesh generation of biological structures

A. Ramos-de-Miguel, R. Montenegro & J.M. Escobar

Division Discretization and Applications, University Institute on Intelligent System and Numerical
Application in Engineering, Las Palmas of Gran Canaria, Spain

ABSTRACT: In this work we propose a technique for the modellisation of biological structures for its
applications in Finite Elements Methods (FEM). The key of this method consists on deforming the mesh
of a generic structure in order to obtain a mesh of a specific one. The input data are both the mesh of the
generic structure and the data collected by Computerised Tomography (CT) scan, Magnetic Resonance
Imaging (MRI), etc... This technique involves genetic algorithms for adjusting the nodes of the input
mesh to the collected points, Radial Basis Functions (RBF) to deform the input mesh, and optimisation
of the final mesh.

1 INTRODUCTION

The generation of FEM meshes of biological

structures for simulating pathologies, prothesis and
behaviour of different organs (1) (2) (3), presents a
great challenge when little information is available.
This problem can be found, for example, in CT
for medical image acquisition and later 3D recon-
struction. In this case, a compromise is required
between emitted radiation, exposure time and
desired image quality. Moreover if we use high
radiation and long exposure time, there can be no
guarantees that sufficient data will be collected.
What we propose in this work is a method to
generate an FEM mesh by doing small deforma-
tions on a model mesh. Our inputs are a tetrahe- Figure 1. Summary of the main stages of the method in
dral mesh that represents the organ of a generic a simplified example. a) Input data: a generic mesh of a
patient and the acquired data, AD, from a particu- structure M (black) and the acquired data of the desired
lar patient, i.e., a CT scan, MRI, etc. structure, AD (red); b) Affine transform of M after the
M represents the tetrahedral mesh of a generic point cloud superposition; c) Selection of pairs (p, h) to
patient’s organ, SM the surface mesh of M and NS guide the deformation, where h are the handle points; d)
its nodes. The mesh after deformation.
The first stage is to superpose, searching for the
best fit, NS to AD, by doing rigid transformations
of M in order to minimise the Hausdorff distance must be applied. This optimisation consists on the
between NS to AD. We define two sets of points, application of an untangling and smoothing pro-
PS NS , known as selected points, and H AD, cedure: one for the surface SM ′ and another for the
known as handle points. The pair (p, h) with p ∈ PS tetrahedral mesh M ′ .
and h ∈ H is selected among NS and AD if the dis-
tance between them is below a certain threshold,
2 METHODS
given by the user. The surface mesh SM is deformed
in order to interpolate the points of H.
2.1 Point cloud superposition
We use RBF to deform M due to the good
behaviour of these functions and their low compu- Our purpose is to minimise the Hausdorff distance
tational cost (4) (5) to get M ′ . The intrinsic prob- between the points NS to AD using rotations, trans-
lem of this method is that the resulting deformed lations and scaling. This distance indicates how far
mesh can be tangled, so an optimisation procedure apart the two sets are.

175

VIPIMAGE15_Book.indb 175 9/8/2015 7:44:45 PM

 A genetic algorithm was used for this task of the linear polynomial space. The chosen kernel
because there is not an analytic function that function to deform the model was ϕ ( ) r3 (7).
relates the affine transformation of a point cloud The weights wj and qj are determined by imposing
with the obtained Hausdorff distance. the interpolation conditions, d ( p j ) = δ h j p j − h j .

2.1.1 Genetic algorithm

2.3 Mesh untangling and smoothing
Each individual has 9 genes, 3 for translation, 3 for
rotation and 3 for scale in each axis x, y and z. As explained above, the deformation process on
The genetic algorithm comprises 5 stages. In the M can produce M ′ to be tangled, so mesh optimi-
first one, the initial population is created, where the sation is required.
genes of each individual were generated randomly. This procedure is divided in two stages: one for the
Then, an iterative process is initiated where, for each surface mesh SM ′ and another for M′ inner nodes.
iteration, each individual is compared with the fitness The optimisation process (untangling and
function. After evaluating each individual, the leader smoothing) is an iterative process in which each
is selected and a crossover and a mutation stage fol- node of the mesh is moved to a new position that
lows. The crossover and mutation are done gene by minimises a certain objective function. The objec-
gene by using the gene information from the parents. tive function is derived from some quality meas-
The last step is the generation of the new individuals ure of the local mesh, that is, the set of elements
to keep the size of the population constant. (triangles for 2D or tetrahedrals for 3D) connected
to the adjustable or free node. (6) (7) (8).
The optimisation process for SM ′ takes place in a
2.2 Deformation of the model mesh
local parametric space. We search ideal triangles in
Once the superposition of NS and AD have been this local parametric space that become optimal in
done, M is transformed into a deformed model M ′ the surface SM ′ . After finding the optimal position
by computing updated point locations x′ = x + d( x ) in the parametric space, the nodes are projected to
for each original point x ∈ M . the local approximation using RBF. Finally, M ′
inner nodes are optimised using (9).
2.2.1 Deformation function
RBF are used for computing g d ( x ), x = ( y ) . The 2.4 Skeleton deformation
deformation function d : R3 → R3 is given by:
When we work with geometries with very small dif-
m 4 ferences, the superposition of M ′ in AD matches.
d( x ) = ∑ w j j (x) + ∑ q kπ k ( x ) (1) But, some geometries are impossible to superpose
j =1 k =1 well because they have structural differences. For
example, a full body will be badly adapted to
Equation (1) is defined as a linear combina- another because the arms and legs have different
tion of radially symmetrical nucleus functions position on each mesh.
ϕ j ( ) ϕ( j ||), located in centres h j ∈ H and To solve this problem we propose to apply the
a linear polynomial to guarantee the affine transfor- deformation procedure to the skeletons of M and
mations, where { 1 2 3 4 } = { y } is a base AD, which we will name as SkM and SkAD. First

Figure 2. Skeleton deformation stages. a) Model input mesh; b) Patient input mesh; c) Superposed skeletons of model
(blue) and patient (red) meshes; d) Deformed model (blue) skeleton to fit the patient (red) skeleton; e) Deformed model
surface mesh.

176

VIPIMAGE15_Book.indb 176 9/8/2015 7:44:47 PM

 we voxelise M and AD and then we calculate the
skeleton 3D medial axis from a 3D binary volume,
to get SkkM and Sk kAD (as described in (10)). Then,
SkkM must be deformed to be adapted to Sk kAD ,
proposed in 2.1 and 2.2. Rigid transformation and
deformations done to Sk kM are also applied to M.
Figure 2 shows the model (a) and patient (b)
meshes and the corresponding skeletons (c). The
skeleton of the model is deformed using RBF in
order to fit with the skeleton of the patient (d).
Finally, (e) shows the deformed mesh surface.
With this optional step, we reduce the fitting
error between M ′ and AD.

3 RESULTS

3.1 Superposition test

The superposition is a critical stage of this method. Figure 3. Different simulated CT scans extracted from
If a good adjustment between the input point the original mesh. a) CT 1 with 10 Femur slices; b) CT 2
clouds and the mesh is not reached, the deforma- with 20 Femur slices; c) CT 3 with 40 Femur slices; d) CT
tion will select the wrong handle points and the 4 with 80 Femur slices; e) Original Femur mesh.
final mesh will not fit the input data.
In this superposition test one mesh is adjusted
with itself. In this case AD is a set of points extracted
from the input mesh that simulates M’s CT scan.
Applying the Hausdorff distance minimisa-
tion procedure 2.1 to { 1 2 3 4 },
results in the affine transformations { 1 2 3 4 }.
The Hausdorff distance is measured between
Ti ( N s ), i = 1, ... , 4 and NS to evaluate the goodness
of the fitting between patient data and input mesh.
The data are shown in Table 1.
A genetic algorithm was used to calculate aff-
ine transformations with 1000 generations and a
population size of 20 individuals.

3.2 Deformation test

AD will be taken as a set of points extracted from Figure 4. Left: 3D mesh model M; right: mesh used to
the mesh in Figure 4-b, that simulate a CT scan. extract the input data (patient) AD.
Also, it will be assumed that AD and M are already
superposed.
This test consists on deforming M to adapt it to In Figure 5 the different adjustments between
AD following the procedure 2.2. AD and M are illustrated in terms of the number
of handle points. It should be noted that the greater
the number of handle points the better the adjust-
Table 1. Evolution of the Hausdorff ment between M and AD, Table 2.
distance as the number of points of the The faces shown in Figure 5 have been optimised
acquired data increases. following the procedure 2.3. In Figure 6 a detail of
the rightmost head in Figure 5 is shown, before a)
Num. points Hausdorff and after applying the optimisation procedure b).
Mesh CT scan dist. Table 3 shows the number of tangled elements and
the mean quality before and after optimisation of
CT 1 101 0.1152
CT 2 200 0.1084
each head in Figure 5. Mean ratio quality metric
CT 3 413 0.0997
was used to measure the mesh quality (11). The 0
CT 4 825 0.0894 value corresponds to a degenerated element and 1
to an equilateral triangle.

177

VIPIMAGE15_Book.indb 177 9/8/2015 7:44:50 PM

 Figure 5. Deformation test after untangling stage, changing the number of handle points. From left to right, the
number of handle points is increased. The first line shows the handle points used and the second line the deformation
mesh.

Table 2. Evolution of the Hausdorff distance Table 3. Evolution of the final mesh quality after opti-
as the number of handle points increases. misation and untangling. All case finished with 0 tangled
elements.
Mesh n. handle P. Hausdorff dist.
n. tangled ele. Mean Q. Mean Q.
Case 1 67 0.3833
Case 2 1094 0.3725 Mesh Pre Pre Post
Case 3 1453 0.3179
Case 4 1557 0.3112 Case 1 571 0.568 0.756
Case 2 595 0.566 0.755
Case 3 637 0.557 0.755
Case 4 624 0.559 0.755

4 DISCUSSION

We propose a method to create meshes suitable to

be used in FEM and matched to the patient organ
geometry.
Figure 6. A detail of the rightmost head in Figure 5, The highest computational cost corresponds to
before a), and after the smoothing and untangling stage b). the computation of the Hausdorff distance in the

178

VIPIMAGE15_Book.indb 178 9/8/2015 7:44:52 PM

 present implementation. This computation can be [3] A. Idhammad, A. Abdali and N. Alaa; “Computa-
improved by arranging the data in an octree struc- tional simulation of the bone remodeling using the
ture and also through parallel computing. finite element method: an elastic-damage theory for
The handle points guide the deformation. small displacements”; Theoretical Biology and Medi-
cal Modelling, Volume 10, April 2013.
Hence, as illustrated in Table 2, the number of [4] D. Sieger, S. Menzel and M. Botsch; “On Shape
handle points is strongly related to the final mesh Deformation Techniques for Simulation-based
adjustment. Also, as shown on Figure 5, the dis- Design Optimization”; New Challenges in Grid Gener-
semination of these points makes it possible to ation and Adaptivity for Scientific Computing; SEMA
capture the singularities of the mesh. SIMAI Springer Series, 2014, Springer Milan.
In this method, the handle points are selected [5] Eloi Ruiz-Gironés, Xevi Rocab, Jose Sarratea;
using the nearest point strategy, but, probably, it “Mesh deformation based on radial basis function
would be better to work with some landmarks on interpolation”; Procedia Engineering; Volume 82
the mesh (13), to enforce a better adaptation of (2014) 101–113.
[6] J.M. Escobar, G. Montero, R. Montenegro and E.
some parts of the mesh. Rodríguez; “An algebraic method for smoothing
On the skeleton deformation stage, the proce- surface triangulations on a local parametric space”;
dure tends to flatten the surface mesh. This prob- Int. J. Numer. Meth. Engng; 2006; 66:740–760.
lem and a possible solution is described in (12). [7] A. de Boer, M.S. van der Schoot, H. Bijl; “Optimiz-
Finally, it should be noted that the mesh defor- ing mesh distortion by hierarchical iteration reloca-
mation induced by RBF can sometimes cause tan- tion of the nodes on the CAD entities”; Computers
gled meshes, so the application of an untangling and Structures; Volume 85, Issues 11–14, June–July
and smoothing procedure is mandatory in order to 2007, Pages 784–795.
obtain a valid mesh for the application of FEM. [8] José María Escobar, Eduardo Rodríguez, Rafael
Montenegro, Gustavo Montero and José María
González-Yuste; “Simultaneous untangling and
smoothing of tetrahedral meshes”; Computer Meth-
5 CONCLUSIONS ods in Applied Mechanics and Engineering; 2003;
Vol. 192 25:2775–2787.
We propose a technique for the modellisation of [9] José María Escobar, Eduardo Rodríguez, Rafael
biological structures for its application in FEM Montenegro, Gustavo Montero and José María
when a geometry is given with sparce data. This González-Yuste; SUS code: simultaneous mesh
technique involves genetic algorithms for adjusting untangling and smoothing code”; http://www.dca.
the nodes of the input mesh to the collected points, iusiani.ulpgc.es/SUScode; 2010.
[10] Ta-Chih Lee, Rangasami L. Kashyap and Chong-
Radial Basis Functions (RBF) for deforming the Nam Chu; “Building skeleton models via 3-D medial
input mesh, and mesh optimization. surface/axis thinning algorithms.”; Computer Vision,
A critical point is the selection of handle points. Graphics, and Image Processing; 56(6):462478, 1994.
Future research will involve the improvement of [11] Patrick M. Knupp; “Algebraic Mesh Quality Met-
the method adopted for this selection, e.g. using rics.”; SIAM Journal on Scientific Computing; 23(1),
landmarks (13). A proper handle point choice has 193–218, 2001.
positive effect in the fitting of the acquired data [12] Zohar Levi and David Levin; “Shape Deforma-
to mesh. tion via Interior RBF.”; 1062 IEEE Transctions on
Visualization and Computer Graphics; Vol. 20, No. 7,
2014.
[13] Lorenzo Grassi, Najah Hraiech, Enrico Schileo,
REFERENCES Mauro Ansaloni, Michel Rochette and Marco Vice-
conti; “Evaluation of the generality and accuracy
[1] I. Borazjani; “Fluid-structure interaction, immersed of a new mesh morphing procedure for the human
boundary-finite element method simulations of femur”; Medical Engineering and Physics, Volume
bio-prosthetic heart valves”; Computer Methods in 33, 2011, 112–120.
Applied Mechanics and Engineering, Volume 257,
15 April 2013, Pages 103–116.
[2] L. Hou-Guang, T. Na, R. Zhu-Shi; “Numerical mod-
eling of human middle ear biomechanics using finite
element method”; Journal of Donghua University,
Volume 28, April 2011, Pages 115–118.

179

VIPIMAGE15_Book.indb 179 9/8/2015 7:44:53 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Medical volume rendering based on gradient information

Thiago F. de Moraes, Paulo H.J. Amorim & Jorge V.L. da Silva

Tridimensional Technology Division, Center for Information Technology Renato Archer, Campinas-SP, Brazil

Helio Pedrini
Institute of Computing, University of Campinas, Campinas-SP, Brazil

Maria Inês Meurer

Department of Pathology, Federal University of Santa Catarina, Florianópolis-SC, Brazil

ABSTRACT: Several medical imaging techniques have allowed the acquisition of volumetric images
for non-invasive diagnosis purposes, such as Computerized Tomography, Magnetic Resonance Imaging,
X-Ray Microtomography, Positron Emission Tomography and Ultrasonography. A substantial increase
in both data size and image resolution has been possible due to recent technological advances. Interactive
exploration of volumetric data through visualization approaches can be used to enhance internal struc-
tures of interest. This work describes and evaluates a novel projection technique for volume rendering
based on gradient information and viewing direction to generate values of color and opacity from the
data values. Experiments are conducted on several data sets to demonstrate the effectiveness of the pro-
posed technique compared to other projection approaches.

1 INTRODUCTION rendering techniques are volume ray casting,

splatting, shear warp, and texture-based volume
Volume rendering can be defined as a set of tech- rendering. Due to the intrinsically parallel nature
niques for visualizing bidimensional (2D) projec- of direct volume rendering, Graphics Processing
tions of tridimensional (3D) data sets. Volume Unit (GPU) volume rendering approaches have
rendering has applications in several knowledge been proposed to speed up the visualization proc-
domains, such as medicine, biology, geology, ess (Rodríguez et al. 2014, Hadwiger et al. 2006,
among others. Typically, 3D medical data sets Okuyan and Güdükbay 2014).
are acquired through Computerized Tomography In contrast to the approaches that require every
(CT), Magnetic Resonance Imaging (MRI), X-Ray voxel value to be mapped to color and opacity
Microtomography (microCT), Positron Emission values, projection techniques do not use transfer
Tomography (PET) and Ultrasonography scanners functions. A review of some relevant projection
for diagnostic purposes. techniques is presented in Section 2.
The generation of 2D images from 3D data sets This paper proposes and evaluates a novel projec-
can be performed via rendering approaches, which tion technique for volume rendering based on gra-
are in general categorized into two groups: isosur- dient information and viewing direction, combined
face rendering and direct volume rendering. with Maximum Intensity Difference Accumulation
The isosurface rendering techniques involve the (MIDA) projection, to map data values to color
visualization of intermediate structures extracted and opacity values. Experiments conducted on sev-
from the data. The algorithm of Marching Cubes eral data sets demonstrate the effectiveness of the
(Lorensen and Cline 1987) is one of the most proposed technique compared to other projection
known techniques for extracting a polygonal mesh techniques for medical volume rendering. The eval-
of an isosurface from a 3D scalar data set. uated techniques were implemented in InVesalius
In direct volume rendering (Levoy 1990), (2015), an open-source imaging framework.
attributes of color and opacity are based on voxel
intensity, such that anatomical structures can be
reconstructed by selecting proper threshold lev- 2 BACKGROUND
els for each attribute. The mapping of each voxel
value to color and opacity is done through a The Maximum Intensity Projection (MIP) (Bruckner
transfer function. Some widely used direct volume and Gröller 2009) selects and projects only the

181

VIPIMAGE15_Book.indb 181 9/8/2015 7:44:53 PM

 voxels with maximum intensity along the viewing A comparative analysis with other projection tech-
rays. Since a single value is displayed along each ray, niques (described in Section 2) is conducted on a
visual ambiguities can be caused by the lack of depth number of medical data sets.
information and shape perception of the structures. The 3D volume data sets are represented by a
Some variants of the MIP technique have been finite number of 2D cross-sectional slices. A vol-
proposed to mitigate its main problems. The Local ume element (voxel) corresponds to a position in
Maximum Intensity Projection (LMIP) (Sato the data set and stores a data value. In a direct
et al. 1998) displays the first maximum value that volume rendering method (Levoy 1998), a ray is
is above a certain threshold, instead of the global shot through an image plane into the 3D volume,
maximum value, which adds spatial information. such that color and opacity values along the ray are
The Depth-Shaded Maximum Intensity Projection computed for each pixel.
(DMIP) (Heidrich et al. 1995) weights the data The accumulated color and opacity values can
value by a depth-dependent term, which reduces be computed following the emission-absorption
the chance of high data values to be projected in model (Fernando 2004) as
case they are located far way from the image plane.
n i −1
The Shape-Enhanced Maximum Intensity Projec-
tion (SEMIP) (Zhou et al. 2011) searches for a C ∑ Ci ∏ (1 − j)
i =1 j −1
valid gradient for the maximum intensity of each n
(1)
viewing ray and applies a shading based on gra- α = 1 − ∏ (1 − j)
dient to improve depth and shape perception of j −1
internal structures. Depth-Enhanced Maximum
Intensity Projection (DEMIP) (Díaz and Vázquez where C and α are the accumulated color and
2010) uses depth and simple color blending to dis- opacity values computed iteratively, whereas Ci
ambiguate the ordering of the structures. A related and α i are the color and opacity obtained from the
projection technique is the Minimum Intensity segment i along the viewing ray.
Projection (MinIP) (Mistelbauer et al. 2013), Equation 1 can be efficiently computed if sam-
which displays the minimum value along the view- ples are sorted in a certain order. In back-to-front
ing ray and can be used to provide information order, the accumulated color and opacity values
about blood flow deviations, such as soft plaques. are computed as
The Average Intensity Projection (AIP) (Wu et al.
2008) displays the mean of all values found in the Cˆ i = Ci + (1 − α i )Cˆ i +1 (2)
viewing ray, which can lose image sharpness. αˆ i = α i + (1 − α i )αˆ i +1
The Maximum Intensity Difference Accumula-
tion (MIDA) (Bruckner and Gröller 2009) provides where Cˆ i and αˆ i are the accumulated color and
a method for controling the volume compositing opacity, respectively, from the back of the volume.
effect by introducing a weighting coefficient that In front-to-back order, the accumulated color
takes into account new higher values along the and opacity values are computed as
viewing ray. Thus, when the maximum value changes
from a low to a high value, the corresponding Cˆ i = (1 − αˆ i −1)Ci + Cˆ i −1
sample contributes a higher intensity in comparison (3)
αˆ i = (1 − αˆ i −1)α i + αˆ i −1
to other samples. Occluded structures of interest
can be displayed through MIDA, where the accu-
mulated opacity is modulated according to the where Cˆ i and αˆ i are the accumulated color and
maximum intensity difference. opacity, respectively, from the front of the volume.
The Contour MIP (Csébfalvi et al. 2001) displays As described in Section 2, there are some vari-
the data values based on the magnitude of local ations of the composition strategies, such as MIP,
gradient information and on the angle between the MinP, LMIP, DEMIP, SEMIP, AIP, MIDA,
viewing direction and gradient vector. The tech- among others.
nique enhances the voxels with higher value of The accumulated color Cˆ i and opacity αˆ i for the
gradient information, emphasizing the surfaces of i -th sample along viewing ray in the MIDA tech-
different objects. nique are expressed as

Cˆ i = β i Cˆ i −1 + (1 − β i αˆ i −1)α iCi
(4)
3 METHODOLOGY αˆ i = βi αˆ i −1 + (1 − βi αˆ i −1)α i

This work proposes a new projection technique where β i δ i and δ i = max( I i I maxi , 0 ) , such
for volumetric rendering, which is based on gradi- that I maxi is the current maximum intensity value
ent information and viewing direction to produce along the viewing ray and I i is the intensity from
values of color and opacity from the data values. the segment i along the viewing ray.

182

VIPIMAGE15_Book.indb 182 9/8/2015 7:44:53 PM

 The gradient information can also be used to In our work, the gradient information is also
determine the optical properties (color and opac- employed to emphasize the surface of object.
ity) of voxels. The gradient vector ∇f
∇ points in the However, the MIDA technique is used to generate
direction of largest variation the color and opacity values from the data values
and gradient information.
⎛δ f δ f δ f ⎞ Initially, the method calculates the pixel inten-
∇f = ⎜ , , (5)
⎝ δ x δ y δ z ⎟⎠ sity based on the magnitude of the slices taking
the viewing direction into account, similarly as the
In the Contour MIP (Csébfalvi et al. 2001), the work proposed by Csébfalvi et al. (2001)
data values are mapped to color and opacity val-
ues based on both the magnitude of local gradient s( P V ) ( | ∇( P ) V |)n (6)
information and the angle between the gradient
vector and viewing direction. The MIP technique where ∇( ) is the gradient magnitude of pixel P, V
is used as the projection technique for rendering is the viewing direction and n is a sharpness coef-
the image. ficient for the contour. Viewing direction is (0,0,1)

Figure 1. Results for Manix data set for different projection techniques.

183

VIPIMAGE15_Book.indb 183 9/8/2015 7:44:55 PM

 for axial, (0,1,0) for coronal and (1,0,0) for sagittal MIP is useful for rendering high contrast regions,
plane. for instance, visualization of bone regions and nod-
Equation 6 is computed for all pixels of the ules. MinIP allows visualization of regions with
input slices. Then, the MIDA technique is applied low data values, for instance, detection of excessive
to the generated set of slices and produces a 2D fat in the liver. AIP offers an overall view of the
image as final result. One advantage of the pro- slices, since it corresponds to a mean of the data
posed method is to reduce effects caused by lack values. MIDA provides results with depth informa-
of depth perception. tion, which helps the comprehension of lesions or
other interest regions. Contour MIP and Contour
MIDA highlight regions with high contrast, allow-
4 EXPERIMENTAL RESULTS ing radiologists to focus on such regions.
In Figure 1, it is interesting to observe some
We conducted tests on several medical volumetric differences between MIP (Fig. 1b) and MIDA
data sets available in InVesalius (2015) and OsiriX (Fig. 1d). Both images show bone regions, how-
(2015). Due to space limitation, only three data ever, the depth information is not present in the
sets will be used to illustrate the results. All evalu- MIP image, such that it is very difficult to distin-
ated projection techniques were implemented in guish the bone structures from the aorta and other
the open source imaging framework InVesalius. structures in the neck region. On the other hand, it
Figures 1, 2 and 3 show the results for Manix, is possible to observe bone orientation and other
Chest and Colonix data sets, respectively, obtained volumetric information from the MIDA image. A
through previously described projection tech- similar comparison is possible between Contour
niques: MIP (Maximum Intensity Projection), MIP (Fig. 1e) and Contour MIDA (Fig. 1f), where
MinIP (Minimum Intensity Projection), AIP both show bone structures, soft tissues and the
(Average Intensity Projection), MIDA (Maximum pulmonary bronchioles (which were not visible in
Intensity Difference Accumulation), Contour MIP the MIP and MIDA images). However, in the Con-
and Contour MIDA (proposed method). tour MIDA image, it is possible to observe the soft

Figure 2. Results for Chest data set for different projection techniques.

184

VIPIMAGE15_Book.indb 184 9/8/2015 7:44:56 PM

 Figure 3. Results for Colonix data set for different projection techniques.

tissue covering the jaw bone, which is not visible in on gradient information and viewing direction to
the image generated by Contour MIP. generate a final image.
In Figure 2, it is possible to observe that only Experiments are conducted on several data sets
Contour MIP (Fig. 2e) and Contour MIDA and the results obtained through the proposed method
(Fig. 2f) are able to show the pulmonary bronchi- are compared with other projection approaches.
oles and the grooves on the base of the left lung.
The main difference between both is the depth and
ACKNOWLEDGEMENTS
volumetric information. The depth information
makes it possible to perceive that the bones of the
The authors are grateful to FAPESP, CNPq and
ribs are behind the lungs, not in front of them.
CAPES for the financial support.
In Figure 3, only Contour MIP (Fig. 2e) and
Contour MIDA (Fig. 2f) are able to show the colon
in the images. However, the Contour MIDA allowed REFERENCES
to differentiate the bends of the colon, as well as to
show the internal parts of the pelvis bone. Bruckner, S. & M.E. Gröller (2009, June). Instant Vol-
ume Visualization using Maximum Intensity Differ-
ence Accumulation. Computer Graphics Forum 28(3),
775–782.
5 CONCLUSIONS Csébfalvi, B., L. Mroz, H. Hauser, A. König, & M.E. Gröller
(2001). Fast Visualization of Object Contours by
This work proposed and evaluated a novel projec- Non-Photorealistic Volume Rendering. In Computer
tion technique for medical volume rendering based Graphics Forum, Volume 20, pp. 452–460.

185

VIPIMAGE15_Book.indb 185 9/8/2015 7:44:57 PM

 Díaz, J. & P. Vázquez (2010). Depth-Enhanced Maxi- Okuyan, E. & U. Güdükbay (2014). Direct Volume Ren-
mum Intensity Projection. In 8th IEEE/EG Interna- dering of Unstructured Tetrahedral Meshes using
tional Symposium on Volume Graphics, pp. 93–100. CUDA and OpenMP. The Journal of Supercomputing
Fernando, R. (2004). GPU Gems: Programming Tech- 67(2), 324–344.
niques, Tips and Tricks for Real-Time Graphics. Pear- OsiriX (2015, January). DICOM files. http://www.osir
son Higher Education. ixviewer.com/datasets/.
Hadwiger, M., A. Kratz, C. Sigg, & K. Bühler (2006). Rodríguez, M.B., E. Gobbetti, J.I. Guitián, M. Makhinya,
GPU-Accelerated Deep Shadow Maps for Direct Vol- F. Marton, R. Pajarola, & S. Suter (2014). State-of-
ume Rendering. In SIGGRAPH/EUROGRAPHICS the-art in Compressed GPU-Based Direct Volume
Conference On Graphics Hardware: 21st ACM SIG- Rendering. In Computer Graphics Forum, Volume 33,
GRAPH/Eurographics Symposium on Graphics Hard- pp. 77–100.
ware, Volume 3, Vienna, Austria, pp. 49–52. Sato, Y., N. Shiraga, S. Nakajima, S. Tamura, &
Heidrich, W., M. McCool, & J. Stevens (1995, October). R. Kikinis (1998). Local Maximum Intensity Projec-
Interactive Maximum Projection Volume Rendering. tion (LMIP): A New Rendering Method for Vascular
In IEEE Conference on Visualization, pp. 11–18. Visualization. Journal of Computer Assisted Tomogra-
InVesalius (2015). Open Source Software for Reconstruc- phy 22(6), 912–917.
tion of Computed Tomography and Magnetic Reso- Wu, Q., F. Merchant, & K.R. Castleman (2008, April).
nance Images. http://www.cti.gov.br/invesalius/. Microscope Image Processing (1st ed.). Academic
Levoy, M. (1988). Display of Surfaces From Volume Data. Press.
IEEE Computer Graphics and Applications 8(3), 29–37. Zhou, Z., Y. Tao, H. Lin, F. Dong, & G. Clapworthy
Levoy, M. (1990). Efficient Ray Tracing of Volume Data. (2011). Shape-Enhanced Maximum Intensity Projec-
ACM Transactions on Graphics 9(3), 245–261. tion. The Visual Computer 27(6–8), 677–686.
Lorensen, W.E. & H.E. Cline (1987, July). Marching
Cubes: A High Resolution 3D Surface Construction
Algorithm. Computer Graphics 21(4), 163–169.
Mistelbauer, G., A. Morar, A. Varchola, R. Schernthaner,
I. Baclija, A. Köchl, A. Kanitsar, S. Bruckner, &
E. Gröller (2013, June). Vessel Visualization using
Curvicircular Feature Aggregation. Computer Graph-
ics Forum 32(3), 231–240.

186

VIPIMAGE15_Book.indb 186 9/8/2015 7:44:58 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Chaos theory-based quantification of ROIs for mammogram

classification

Jaroslaw Kurek & Bartosz Świderski

The Faculty of Applied Informatics and Mathematics, Warsaw University of Life Sciences, Warsaw, Poland

Sami Dhahbi
Research Team on Intelligent Systems in Imaging and Artificial Vision (SIIVA)—RIADI Laboratory,
Ariana, Tunisia

Michal Kruk
The Faculty of Applied Informatics and Mathematics, Warsaw University of Life Sciences, Warsaw, Poland

Walid Barhoumi
Research Team on Intelligent Systems in Imaging and Artificial Vision (SIIVA)—RIADI Laboratory,
Ariana, Tunisia

Grzegorz Wieczorek
The Faculty of Applied Informatics and Mathematics, Warsaw University of Life Sciences, Warsaw, Poland

Ezzeddine Zagrouba
Research Team on Intelligent Systems in Imaging and Artificial Vision (SIIVA)—RIADI Laboratory,
Ariana, Tunisia

ABSTRACT: The paper presents improved method for breast cancer diagnosis. The previous method
has been presented in Dhahbi et al. 2015. The suggested improvements regards applying more diagnostic
features such as wavelet packet decomposition, Hilbert matrix, fractal texture features, etc. Moreover we
investigated several classifiers such as Random Forest, Support Vector Machine and Decision Tree. In this
paper larger database (number of images/trials) has been used and reached better accuracy.

Keywords: mammography; diagnostic feature; breast cancer diagnosis; wavelet transform

1 INTRODUCTION this [2]. It is the most common cancer among

people of highly civilized countries, such as USA,
Breast cancer is one of the common women- Canada, Australia and Western European coun-
cancer worldwide. It does not matter which region tries. Least of breast cancer recorded in southern
we take into consideration: the developed or less Asia and Africa. Breast cancer, which is the most
developed world, statistic states that over 508 000 common cancer in women, in men is fortunately
women died in 2011 due to breast cancer [1] and rare.
522 000 in 2012 [2]. To cope with this issue, screening program is
Breast cancer is arising from cells of the breast applied. Usually for women this screening pro-
that develops locally in the breast and metasta- gram made up of palpable breast examination by a
sizes to the lymph nodes and internal organs (e.g. doctor (like self-examination performed regularly
lungs, liver, bones and brain). Breast cancer is carried out by the woman herself) and mammog-
the most common malignancy in women [1]. This raphy (ionizing radiation to create mammography
represents approximately 23% of all cases of can- images). Such research is usually done in women
cers in women and approximately 14% of deaths over 50 years of age every three years, although
because of this [2]. It is estimated that each year many experts believe that it is worth it carried out
breast cancer is diagnosed in 1.5 million women every two years. In the USA, women in first-degree
worldwide, and about 522,000 die because of relatives of individuals who develop breast cancer

187

VIPIMAGE15_Book.indb 187 9/8/2015 7:44:58 PM

 are advised to use mammography every 2 years
starting at age test 10 years younger than that in
which the person became ill relative.
Screening mammograms can detect lumps or
other breast abnormality at an early stage when
they are not detectable by a woman or doctor,
which increases very significantly the chance of
cure [3]. A significant negative effect of the research
is a phenomenon observed that the negative result
of mammography (i.e. not confirming the presence
of the disease) in many women creates a false sense
of security and reluctance and negligence in sys-
tematic self-examination of the breast. This pro-
cedure is wrong, because in about 20–25% cases,
mammography cannot detect a cancer developing
(these are called false negative results).
The Mammography screening is currently the
best tool for nationwide mammogram screening
programs reducing the mortality rate of breast Figure 1. Example of Region of Interest (ROI) extraction.
cancer [4, 5]. Hence the improved methods of the
breast cancer are usually based on the analysis of
mammography images.

2 DATABASES

Total number of mammography images used in

this paper is 10168 and it is a part of Digital Data-
base for Screening Mammography (DDSM Uni-
versity of South Florida) which is currently the
largest public mammogram database [6]. The data-
set includes 2604 cases. Screening mammography
typically involves taking two views of the breast,
from above (Cranial-Caudal view, CC) and from
an oblique or angled view (Medio-Lateral-Oblique,
MLO). Hence when we take into consideration left
and right breast, we have 4 images for every case in
the used challenging dataset.
Figure 2. Sample of used ROIs from DDSM database.
2.1 Region of Interest (ROI) Cropping
This is an important step to extract and focus on
ROIs has been rejected due to small quality of
appropriate part of mammography image. Since the
image. The dataset consists of three group of
whole mammographic image comprises the pectoral
mammography result (Fig. 2):
muscle and the background with a lot of noise, fea-
tures were computed on a limited ROI that contains 1. Normal tissue—8254 ROIs
the prospective abnormality [2]. Thus, cropping 2. Benign lesion—862 ROIs
a region of interest removes the unwanted parts. 3. Malignant lesion—1052 ROIs
Image cropping was performed manually based on
Examples of ROIs for every of three results are
the physician annotation provided in the dataset.
depicted on Figure 2.
We can describe ROI as rectangular space where in
Due to the complexity of separating the three
its center we can observe the lesion (Fig. 1).
classes, we decided to merge ROI’s belong to benign
lesions and malignant ones. It means the issue is
2.2 Trial subsets used in experiments reduced to binary classification of normal tissue
and abnormal tissue trials. Finally, we obtained:
Based on the above-described database, 10168
ROIs have been used in this work among the 10416 1. Normal tissue—8254 ROIs
available ones [7]. Some number of all available 2. Abnormal tissue—1914 ROIs

188

VIPIMAGE15_Book.indb 188 9/8/2015 7:44:58 PM

 3 FEATURES GENERATION – Standard deviation:
n
Based on the all extracted ROI’s, we gener- 1
ated 122 potential diagnostic features for binary σ
n
∑( i μ )2
i =1
classification. Dataset of potential diagnostic fea-
tures consist of the following group: – Variance:
• features generated based on Hilbert’s image rep- 1 n
resentation (23 features). σ2 ∑(
n i =1
i μ )2
• features generated based on segmentation-based
fractal texture analysis (36 features).
– Skewness:
• Kolmogorov-Smirnov descriptors (38 features).
• statistics from the gray-level co-occurrence 1 n ⎛ xi − μ ⎞
3

matrix and other statistic (15 features). S= ∑⎜

n i =1 ⎝ δ ⎠
⎟
• Maximum sub-region descriptors (10 features)
using K-S statistic and Minkowsky approach. – Kurtosis:
4
3.1 Hilbert’s image representation 1 n ⎛ xi − μ ⎞
K= ∑⎜
n i =1 ⎝ δ ⎠
⎟ −3
Every image, which has regular representation in
form of matrix of size n × m, has been transformed – Root Mean Squared (RMS):
to one vector using our previously proposed
Hilbert curve and recursive modeling. Building
∑ i =1 xi
n
Hilbert curve of order 4 is depicted on Figure 3. M =
RMS
n
3.2 Trial subsets used in experiments – Crest factor (peak-to-rms ratio):
By means of Hilbert transformation of every X peak
image, we can obtain image vector representation C=
X RM
MS
for each image, which can be further treated as
“time-series”. Based on every image vector X rep- – Wavelet energy of 4-level wavelet packet decom-
resentation, (where n denoted the vector length), position of db10 wavelet family:
we can calculate the following potential features:
After wavelet packet decomposition, the portion
– Mean: of energy for every terminal node has been
1 n calculated based on the following equation:
μ= ∑ Xi
n i =1 N
2
E ∑ S jk
k =1

where, Sjk is appropriate coefficient discrete

wavelet transformation.

3.3 Segmentation-based fractal texture analysis

This approach let us generate 36 texture features
based on SFTA algorithm (Segmentation-based
Fractal Texture Analysis) and returns 1 × 6 vector
D extracted from the input grayscale image. When
we apply 6th fractal order, we can obtain 36 texture
features.
The SFTA algorithm decomposes images into
various thresholded images using several sets of
lower and upper threshold values. Implemen-
tation has been based on Costa approach [9].
Thresholded images are used to extract the frac-
tal dimension. Since images with more jagged
Figure 3. Example of Hilbert curve, which is a space edges and prominent color differences tend to
filling curve that visits every pixel in a matrix (image). have higher fractal dimension, images with more

189

VIPIMAGE15_Book.indb 189 9/8/2015 7:45:00 PM

 uniform texture properties will have closer fractal features generated based on K-S approach has
dimension. been described in [10].

3.4 Kolmogorov-Smirnov descriptors 3.5 Statistics from the gray-level co-occurrence

matrix and other statistic
Kolmogorov-Smirnov descriptors reflect the
change distribution of intensity of pixels placed Four features have been generated using Gray-
in the rings of the increasing geometrical dis- Level Co-occurrence Matrix (GLCM) from a
tances from the central point. The division of the given image. GLCM calculates how often a pixel
images into coaxial rings is illustrated in Figure 4. with gray-level (grayscale intensity) value i occurs
It represents 4 concentric rings of equal number horizontally adjacent to a pixel with the value j.
(56) pixels in each ring (equal number of pix- Based on GLCM approach, the following statistic
els leads to more stable distribution of KS sta- features were generated:
tistics) [10]. a. Contrast: measure of the intensity contrast
The central point is travelling along pixels between a pixel and its neighbour over the
uniformly distributed in the image. The results whole image.
of the statistical analyses of these coaxial rings b. Correlation: measure of how correlated a pixel
will be combined together by concatenating the is to its neighbour over the whole image.
pixel intensities corresponding to the same rings c. Energy: sum of squared elements in the
placed in equal distances, at different positions of GLCM.
the central pixel. Then cumulative Kolmogorov- d. Homogeneity: value that measures the closeness
Smirnov (KS) distances between the intensity of the distribution of elements in the GLCM to
of pixels xi and xj belonging to two different the GLCM diagonal.
rings using KS test is then estimated. Based on
above approach we can generate the following Further 11 features are generated from general
7 features: statistic measures of the treated image as one-
dimensional vector. Hence, the following features
a. dKS_12 (mean KS statistics between ring no 1 were generated:
and ring no 2).
b. dKS_13 (mean KS statistics between ring no 1 a. median
and ring no 3). b. mean
c. dKS_14 (mean KS statistics between ring no 1 c. standard deviation
and ring no 4). d. kurtosis
d. the ratio dKS_13/dKS_12. e. min
e. the ratio dKS_14/dKS_12. f. max
f. the coefficient α0 of the approximation line g. high order cumulates and moments
d KS α 0 α1l + ε .
g. the slope coefficient α1 of the approximation 3.6 Maximum sub-region descriptors using K-S
line d KS α 0 α1l + ε . statistic and Minkowsy approach
Generally, we obtained 14 K-S features: 7 for The box-counting fractal dimension is measure
mean and 7 for median approach. Rest of 24 similar characterizing the fractal complexity. It is the partic-
ular case of the Mandelbrot fractal dimension and
is based on the notion of self-similarity of the struc-
ture at different scales. It measures how the length
of the complex curve is changing when the measure-
ment is performed with the increased accuracy [11].
To characterize any curve, this curve is covered
with the set of regular squared areas of the size ε.
Thus, we have to calculate the number of squared
areas containing any part of the given curve.
Number of founded areas is denoted as N(ε).

log (N )
d = lim
ε →∞ log( )

In our approach, we have used 8 boxes. We sug-

Figure 4. Illustration of 4 coaxial rings around the cen- gested 122 various features based on which we can
tral pixel. choose the best subset for binary classification.

190

VIPIMAGE15_Book.indb 190 9/8/2015 7:45:02 PM

 Table 1. Result of binary classification (normal vs. because numerical tests are performed on 3828
abnormal tissues) of mammography images. total trials: 1914 trials of abnormal tissue and
1914 trials of normal tissue randomly chosen from
Standard 8254 available trials. In [2], accuracy is 86.46% but
Setting Accuracy [%] deviation [%]
numerical experiments have been performed based
SVM 80% 5.1% on only 200 trials (ROIs): 100 trials normal and
Decision tree 79% 5.5% 100 trials abnormal. So, current method presented
Random forest 81% 4.5% in this paper is more reliable based on the cho-
sen selected features. The best result obtained for
random forest classifier applying 200 trees in this
algorithm.
4 FEATURES SELECTION

To choose appropriate diagnostic features, from all REFERENCES

122 potentially available, the best separating two
class of tissue (normal and abnormal) sequential [1] “Breast cancer: prevention and control”, WHO
2015.
feature selection has been performed. This approach
[2] S. Dhahbi, W. Barhoumia, E. Zagroubaa “Breast
selects a subset of features from the data matrix cancer diagnosis in digitized mammograms using
X that best predicts the data in y by sequentially curvelet moments”, Comp. Biol. Med. 64 (2015)
selecting features until there is no improvement in 79–90.
prediction [8]. There is possibility to apply chosen [3] A. Jotwani, J. Gralow, Early detection of breast can-
classifier (e.g. no-linear) for this selection method cer, Mol. Diagnosis and Ther. 13(6) (2009)349–357.
e.g. SVM, random forest, decision tree. For every [4] H. Nelson, K. Tyne, A. Naik, C. Bougatsos, B.
candidate feature subset, sequential feature selec- Chan, P. Nygren, L. Humphrey, Screening for breast
tion performs 10-fold cross-validation by repeatedly cancer: Systematic evidence review update for the U.
S. preventive services task force, Ann. Intern. Med.
calling function with different training subsets of X
151(10) (2009) 727–W242.
and y, XTRAIN and ytrain, and test subsets of X [5] S. Hofvind, G. Ursin, S. Tretli, S. Sebuodeg°ard,
and y, XTEST and ytest. The result is logical vector B. Moller, Breast cancer mortality in participants
indicating which features are finally chosen. After of the norwegian breast cancer screening program,
applying sequential features selection, we obtained Cancer 119 (2013) 3106–3112.
48 diagnostic features from the 122 available ones. [6] M. Heath, K. Bowyer, D. Kopans, W. Kegelmeyer,
R. Moore, K. Chang, S. Munishkumaran, Current
status of the digital database for screening mam-
5 NUMERICAL EXPERIMENTS mography, in Digital Mammography, Springer
Netherlands (1998) 457–460.
[7] M. Jiang, S. Zhang, H. Li, N. Metaxas, Computer-
The authors have applied three type of classi- aided diagnosis of mammographic masses using
fier: SVM, decision tree and random forest. For scalable image retrieval, IEEE Transactions on Bio-
numerical experiments, the equal subset of normal medical Engineering 62(2) (2015) 783–792.
and abnormal tissues trials have been chosen in [8] Matlab user manual, MathWorks, Natick, 2014.
randomize approach. The 10-fold stratified cross [9] Costa, Alceu Ferraz, Humpire-Mamani, Gabriel,
validation has been performed to compute the mis- and Traina, Agma Juci Machado. An efficient
classification error for each classifier. Results are algorithmfor fractal analysis of textures. 2012 25th
given in Table 1. SIBGRAPI Conference on Graphics, Patterns and
Images 39–46, 2012.
[10] Swiderski, S. Osowski, M. Kruk, J. Kurek, Texture
characterization based on the Kolmogorov-Smirnov
6 CONCLUSION distance, Expert Systems with Applications 42(1)
(2015) 503–509.
The obtained accuracy result (= 81%) is more [11] Schroeder M., Fractals, Chaos, Power Laws, 6 Ed.
reliable than the one presented (= 86.46%) in [2], New York: W.H. Freeman and Company, 2006.

191

VIPIMAGE15_Book.indb 191 9/8/2015 7:45:04 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

M5L: A web-based Computer Aided Detection system for automated

search of lung nodules in thoracic Computed Tomography scans

A. Traverso & M. Agnello

Dipartimento di Scienza Applicata e Tecnologia, Politecnico di Torino, Italy
Istituto Nazionale di Fisica Nucleare Sezione di Torino, Italy

P. Cerello, M. Saletta & S. Bagnasco

Istituto Nazionale di Fisica Nucleare Sezione di Torino, Italy

C. Peroni & E. Fiorina

Dipartimento di Fisica, Universita’ degli Studi di Torino, Italy
Istituto Nazionale di Fisica Nucleare Sezione di Torino, Italy

M.E. Fantacci
Dipartimento di Fisica, Universita’ degli Studi di Pisa, Italy
Istituto Nazionale di Fisica Nucleare Sezione di Pisa, Italy

A. Retico
Istituto Nazionale di Fisica Nucleare Sezione di Pisa, Italy

E. Lopez Torres
CEADEN, Havana, Cuba
Istituto Nazionale di Fisica Nucleare Sezione di Torino, Italy

ABSTRACT: M5L, a Web-based fully automated Computer-Aided Detection (CAD) system for the
automated detection of lung nodules in thoracic Computed Tomography (CT), is based on a multi-thread
analysis with two independent CAD subsystems, the lung Channeler Ant Model (lungCAM) and the
Voxel-Based Neural Analysis (VBNA), and on the combination of their results. The M5L performance,
extensively validated on 1043 CT scans from 3 independent data-sets, including the full LIDC/IDRI data-
base, is homogeneous across the databases: the sensitivity is about 0.8 at 6–8 False Positive findings per
scan, despite the different annotation criteria and acquisition and reconstruction conditions. In order
to make CAD algorithms and their results available to users without requiring the installation of CAD
software or new hardware for CAD computations, the project has proposed a Cloud SaaS (Software as a
Service) approach composed by three main building blocks: a front-end web which handles the workflow,
image upload, CAD results notification and direct on-line annotation of the exam by the radiologist; the
OpenNebula-based cloud Iaas (Infrastructure as a Service) batch farm allocates virtual computing and
storage resources; the M5L CAD provides the nodule detection functionality.

1 INTRODUCTION sensitive image way for the detection of small pul-

monary nodules: low dose high resolution CT-
Lung Cancer is one of the main public health based screening trials are regarded as a promising
issues in developed countries, accounting for technique for detecting early-stage lung cancers
about 19% and 28% of cancer-related deaths in (Team et al. 2011). The identification of early
Europe (Ferlay et al. 2010) and the United States stage pathological Region of Interests (ROIs) in
of America (Jemal et al. 2009), respectively, with a low dose high resolution CT scans is a very diffi-
five-year survival rate of only 10–16% (Jemal et al. cult task for radiologists, considering as first main
2010). Lung cancer is commonly manifesting itself issue the high number (300/400) of noisy slices
as non-calcified pulmonary nodules. Computed to be analyzed. In order to support radiologist,
Tomography (CT) has been shown to be the most researchers have started developing CAD methods

193

VIPIMAGE15_Book.indb 193 9/8/2015 7:45:04 PM

 to be applied to CT examinations. Several studies 2.1 THE lungCAM CAD
(Das et al. 2006) (Brochu et al. 2008) (Matsumoto
The lungCAM structure follows a standard
et al. 2008) reported an improvement in the sensi-
approach: the pre-processing stage (equalisation
tivity of radiologists when assisted by CAD sys-
and lung volume segmentation) is followed by a
tems. In addition, CAD systems act as detection
search for Regions Of Interest (ROIs), an analytic
rates equalizers between observers of different
filter and a neural classifier. Before starting the
level of experience (Brown et al. 2005). The most
current analysis, CT scans in DICOM standard
common approach in hospitals, in order to make
format are pre-processed to reduce the noise con-
CAD algorithms available in clinical routine, is the
tribution: each 2D slice is analysed with a Savitzky-
deployment of standalone workstations, usually
Golay filter (Rajagolopan, 2003) that provides
equipped with a provider dependent Graphic User
noise reduction without loss of resolution. From
Interfaces (GUI). This approach presents several
then on, every step of the lungCAM algorithm is
drawbacks, such as the high fixed cost of the soft-
intrinsically 3-dimensional.
ware licenses or the difficulty in upgrading the
software release. Furthermore, the computational
2.1.1 Lung segmentation
power required by CAD computations is often
The lung segmentation (De Nunzio et al. 2011)
very high (increasing with the complexity of the
proceeds according to four main steps:
algorithms) and unpredictable (for example dur-
ing large screening campaigns when a big amount • Analysis of the CT Hounsfield Unit level distri-
of data need to be analyzed as faster as possible), bution and evaluation of the intensity threshold
often requiring new powerful and more expensive to be applied in the following stages.
hardware to be bought with impacts on cost for • 3D region growing of the lung volume with the
the hospitals. The emerging of Cloud computing, detected threshold.
accessible via secure Web protocols, solves almost • Wave-front algorithm for the definition of the
at all the previous issues. A further important lung surface on the inner side and the removal
reason favors the SaaS (Software as a Service) of the trachea and the main bronchi.
approach: the possibility of combining different • Morphological closing with a cylinder from the
CADs with no particular effort. It was shown out-side in order to include pleural nodules and
that the combination of multiple CAD algorithms close the holes left by vessels.
increases the overall sensitivity (van Ginneken et al.
2010). This paper is aiming at presenting the M5L 2.1.2 ROI hunting
on-demand CAD system developed within the The segmentation algorithm is performed with the
MAGIC-5 collaboration. The MAGIC-5 project, Channeler Ant Model (CAM) (Cerello et al. 2010)
funded by the Istituto Nazionale di Fisica Nucle- based on Virtual Ant Colonies. The ant life cycle
are (INFN, Italy) and Ministero della Universita’ e is a sequence of atomic time steps, during which
della Ricerca (MIUR, Italy) is aiming at: the behaviour is determined by a set of rules that
control the pheromone release, the movements
• Developing and combining CAD algorithms for
and the variations of the ant energy, a parameter
lung nodules identification.
related to breeding and death. The lung internal
• Making CAD algorithms available to users with-
structures are segmented by iteratively deploying
out requiring the installation of dedicated soft-
ant colonies in voxels with intensity above a pre-
ware or high cost hardware.
defined threshold (anthills). Ants live according
• Validating the system inside the clinical rou-
to the model rules until the colony extinction: the
tine in hospitals, studying the impact of CAD
pheromone deposition generates pheromone maps.
algorithms in the overall performance of the
By an iterative thresholding of pheromone maps
radiologists.
a list of ROI candidates is obtained. ROIs with
a radius larger than 10 mm are post-processed in
2 THE ALGORITHMS order to disentangle nodules attached to internal
lung structures like vessels and bronchi. When
M5L is the combination of two independent CAD no more voxels meet the condition to become an
subsystems: the Channeler Ant Model (CAM) and anthill, the information provided by the global
the Voxel-Based Neural Approach (VBNA). These pheromone map is analysed.
two algorithms have a common start line, which is
the parenchymal volume segmentation using a 3D 2.1.3 Filtering and classification
region growing algorithm, which produces the sep- Ideally, any computable quantity which is expected
aration of trachea, bronchi and lungs. The results to show a different pattern for true nodules and
are saved as mask of bits and used to perform fur- false candidates would be a useful feature for the
ther steps. neural network. However, the choice to select a

194

VIPIMAGE15_Book.indb 194 9/8/2015 7:45:04 PM

 small number of features for the neural classifier intensities are thresholded at a fixed value. Vessels
training aims at optimising the generality and and airways are not in included in the segmented
keeping the performance stable as the validation lung at this stage since their volume is outside the
data-set size increases. segmented lung volume. To include them without
Selected features are shown in Figure 1. The modifying the pleura surface morphology a com-
average number of ROIs after the nodule hunting, bination of morphological operators (dilation and
depending on the number of slices, ranges between erosion) is applied. Finally, the logical OR opera-
several hundreds to few thousands per CT scan, a tion between the so obtained mask and the original
number far too large to be used as input for a neu- lung mask provides the final mask. The identified
ral network classifier. The vast majority of find- lung mask is used for CADI, whereas its boundary
ings is easily rejected with an analytic filter based is used for CADJP.
on correlations between the radius, the sphericity
and the fraction of voxels connected to the lung 2.2.2 ROI hunting for internal nodules
mask. A Feed Forward Neural Network (FFNN) In the CADI, the internal nodules are modelled as
was selected as nodule candidate classification spherical objects with a Gaussian profile, following
method. The training sample was made of 5 and the approach proposed in (Li et al. 2003). To detect
69 CTs from the ANODE09 and LIDC/IDRI this kind of objects (Retico et al. 2008), a dedicated
databases, respectively. The training was carried Dot-Enhancement (DE) filter, determining local
on in cross-validation mode. The FFNN configu- characteristics of each voxel by the eingeinvalues
ration was defined as follows: 13 input neurones, 1 of the Hessian matrix, is implemented. To enhance
hidden layer with 25 neurones and 1 neurone in the the sensitivity of this filter to nodules of different
output layer, representing the probability of the sizes, a multi-scale approach combining the DE
finding to be relevant. function with Gaussian smoothing is performed
following prescriptions by (Li et al. 2003).
2.2 The VBNA CAD
2.2.3 ROI hunting for juxta-pleural nodules
The VBNA CAD system deals differently with In the CADJP (Retico et al. 2009) pleura surface
internal and juxtapleural nodules, by means of normals (evaluated using the triangular mesh
two dedicated procedures: CADI for internal and representing the pleura surface) are constructed
CADJP for juxtapleural nodules (Retico et al. 2008) and each voxel is assigned a score proportional
(Retico et al. 2009) (Camarlinghi et al. 2011). Both to the number of normals intersecting in it. Since
are three-step procedures. The first step consists in the evaluation of the normal intersections in the
the lung segmentation; the second step consists in real 3D space is a complex and computationally
the ROI (Region Of Interest) hunter and performs intensive operation, it is implemented in the voxel
the candidate nodule selection; the third step con- space. To deal with noise, cylinders with Gaussian
sists in the FP reduction. profile are considered instead of segments (Paik
et al. 2004). This information is collected in the
2.2.1 Segmentation score matrix S (x, y, z). The local maxima of the
The 3-dimensional segmentation algorithm is 3D matrix S (x, y, z) are the juxtapleural candidate
based on four main steps. Once the scans have nodule locations.
been isotropically resampled, to separate the low-
intensity lung parenchyma from the high-intensity 2.2.4 Classification
surrounding tissue (fat tissue and bones), the voxel In order to classify the candidate nodule findings
obtained in the previous step, an original proce-
dure, the Voxel-Based Neural Approach (Gori
et al. 2007), performs the reduction of the number
of FPs in the lists of internal and juxtapleural
candidate nodules. First, a ROI including voxels
belonging to the candidate nodule is defined from
each location provided by the previous step. The
basic idea of the VBNA is to associate with each
voxel of a ROI a feature vector defined by the
intensity values of its 3D neighbours (in this case
5 × 5 × 5 intensity values) and the eigenvalues of
the gradient matrix and of the Hessian matrix.
Figure 1. List of features extracted from the nodule Support Vector Machines (SVM), by which have
output mask. Features labeled with the asterisk were not been obtained better results, are implemented for
used in the classification stage. the classification procedure. The training sample

195

VIPIMAGE15_Book.indb 195 9/8/2015 7:45:04 PM

 scan reaches 80% which, given the size and hetero-
geneity of the dataset, is quite remarkable (Torres
et al. 2015).
In the case of ANODE09 the FROC curves are
shown in Figure 3.

3 THE M5L WEB-BASED

ON-DEMAND SYSTEM

In order to make CAD algorithms and their results

available to radiologists through a user-friendly
web interface, without limiting the users to install
standalone code on their workstations and in order
to guarantee an adaptable computational power to
Figure 2. FROC curves of the lungCAM and VBNA run CAD algorithms, the project has built a system
subsystems ad of their M5L combination on the LIDC
which is basically composed by the following sub-
test validation dataset (949 CT scans).
systems:
• The Web front-end: The Web front-end is
thought as a service to help doctors and review-
ers in the world to exchange imaging studies and
compare diagnoses on the same studies. The
front-end handles the part of CT submission
for analysis, the possibility to insert on-line the
medical review of the study case and to access to
CAD results.
• The Cloud computing back-end: the cloud
computing backend is though as a service to
guarantee flexibility in the available comput-
ing resources. It allows to allocate computing
resources (i.e. virtual machines) according to the
user request. The back-end handles the part of
algorithms computation.
Figure 3. FROC curves of the lungCAM and VBNA
subsystems ad of their M5L combination on the 3.1 The WEB front-end
ANODE09 test validation dataset (50 CT scans).
The service is available as a web application acces-
sible with every browser from desktop and mobile
was made of 69 CTs from the LIDC/IDRI data- devices. Having proper credentials, DICOM images
base. At the end of this step, each ROI is assigned can be uploaded to the remote repository and
a degree of suspicion averaging the score of all the reviewers can insert their medical diagnosis and
voxels belonging to it. see other reviewers ones. The notification system
is based on emails which basically advertises the
submitters when CAD results are ready and adver-
2.3 Combination and results
tise the reviewer that a new case has been associ-
To obtain the combined M5L result the following ated to him for review. The entire front-end web
matching criterion has been used: a CAD finding has been developed using the free and open source
is considered a true positive if its Euclidean dis- tool DRUPAL, an open source content manage-
tance from the center of he lesion annotated by the ment platform powering millions of websites and
radiologists is less than 1.5 times the radius of the applications (Coombs 2009). Drupal core is mod-
annotated lesion. The results have been evaluated ular, defining a system of hooks and callbacks,
in terms of FROC (Free-response Receiver Oper- which are accessed internally via an API (Applica-
ating Characteristic) curves. tion Programming Interface), allowing third-party
Figure 2 shows the results obtained for the lung- contributed modules. We have essentially built two
CAM and VBNA separate subsystems and for the different modules, and the corresponding user pro-
combined M5L on the 949 scans of the LIDC test files which are briefly described in the following
dataset (949 scans). The M5L sensitivity at 8 FP/ sections.

196

VIPIMAGE15_Book.indb 196 9/8/2015 7:45:04 PM

 3.1.1 Submission module 3.2.1 M5L elastic cluster at INFN torino cloud
This module is conceived to be used by a technical The M5L CAD web service is hosted by the INFN
radiologist (which acts as submitter role), uploads Torino Computing Center, witch is a Tier-2 of
in zip format the CT study to be submitted and LHC (Large Hadron Collider) Computing Grid.
selects a radiologist which will review the study In order to reduce the manpower effort to man-
inserting the annotation. All CAD computations age the Tier-2, support an increasing number of
will be performed in a separated cloud-back end, applications and offer computational power to
this feature allows the user to submitted in sequence small research projects with specific needs, a Pri-
the cases without having to wait for CAD results. vate Cloud infrastructure has been created. The
Before being submitted for computation, all cases facility is managed by OpenNebula, a free and
are subject to further checks, such as re-anonymise open-source Cloud Management Platform which
the case if not yet. allows hardware and virtual infrastructure con-
trol and monitoring, adding the possibility of
3.1.2 Review module virtual machine lifecycle management (Milojičić
This module is conceived to be used by a radi- et al. 2011). OpenNebula orchestrates storage,
ologist. Accessing with proper credentials (as network, virtualisation, monitoring, and security
reviewer role) he/she can insert the medical anno- technologies to deploy computing services as vir-
tation of cases assigned to him/her for review dur- tual machines on distributed infrastructures. The
ing the submission process. A notification system, resources used by the M5L are one physical host
based on e-mail, tells the radiologist when CAD for the web-server and several virtual machines for
results will be ready. He will be able to access the computational power. The VMs are deployed
the results in different formats such as DICOM in a specific sandbox (IaaS) with an internal private
Structured Report, HTML, PDF and review his virtual network and a virtual router with a public
annotation according to these results. An addi- address for the external access and the communica-
tional plugin for the medical imaging viewer Osi- tion with the web server (that has another public
rix has been developed to see CAD marks directly IP). At this moment we are allowed to deploy up to
on the exams. 18 VMs, with a total of 48 cores. In order to achieve
the capability to scale up or down the resources
according to the workload, we decided to create
3.2 The cloud back-end
an elastic cluster, composed by one head node and
One of the issues when running CAD algorithms many workers, using CernVM Online (Buncic et al.
and combining their results is that the required 2010). With these tools, it is possible to contextual-
computing power is large and often unpredictable. ise the VMs, define the use of resources and enable
Furthermore, if the tool is to be used in poten- (between other options) HTCondor and Elastiq.
tially massive screening campaigns, which can be HTCondor is a specialised workload manage-
concentrated in a given period of the year, some ment system (batch system) for compute-intensive
flexibility is required in the available computing jobs. Users submit their serial or parallel jobs to
resources. This can be referred to as the scientific HTCondor, HTCondor places them into a queue,
elastic computing paradigm, in which two require- chooses when and where to execute the jobs based
ments need to be satisfied: upon a policy, carefully monitors their progress,
and ultimately informs the user upon completion.
• An application must be able to easily and quickly
Elastiq, on the other hand, is a lightweight Python
expand its resources during peak times.
daemon that allows a cluster of virtual machines
• Resources should never get wasted (i.e. at any
running a batch system to scale up and down auto-
given time the amount of idle resources should
matically. Each CT uploaded on the web service
be minimal).
is copied on the head node of the cluster and the
In the past, there have been some attempts to M5L CAD execution is controlled by HTCondor.
use an approach similar to the GRID infrastruc- If some CTs to be examined waits in the HTCon-
ture used in high energy physics to overcome those dor queue over a defined period of time, new work-
problems (Bellotti et al. 2007) (Lamanna 2004). ers will be deployed up to the limits of resources.
The main issue with GRID computing is the rigid- Conversely if a worker is idle over a defined period
ity, complexity of the structure and the man power of time, it will be deleted. In this way when no
required to manage the system. Furthermore, this exam it’s uploaded we occupy only a small part of
solution does not fit the majority of Medical Phys- the resources available, freeing the remaining for
ics projects, that require a custom environment. others users of the cloud. At the moment we are
For previous reasons, the use of Cloud Comput- using VMs with 6 cores and 15 GB of RAM. Only
ing solutions is progressively growing (Mell and the head node is always active. Stress tests are in
Grance 2011). progress to define the best configuration and the

197

VIPIMAGE15_Book.indb 197 9/8/2015 7:45:06 PM

 prove the solidity of the system. With this configu- Brochu, B., C. Beigelman-Aubry, J. Goldmard, P. Raffy, P.
ration we are able to complete the analysis of a sin- Grenier, & O. Lucidarme (2008). Thorax-évaluation de
gle CT in nearly 30 minutes, and process up to 42 l?impact d?un systéme cad sur la performance des radi-
exams simultaneously. ologues pour la détection des nodules pulmonaires sur
des examens scanographiques multicoupes du thorax.
Brown, M.S., J.G. Goldin, S. Rogers, H.J. Kim, R.D. Suh,
M.F. McNitt-Gray, S.K. Shah, D. Truong, K. Brown,
4 CONCLUSIONS J.W. Sayre, et al. (2005). Computer-aided lung nodule
detection in ct: Results of large-scale observer test1.
The project has developed two CAD algorithms, Academic radiology 12(6), 681–686.
and their combination, for automatic lung nod- Buncic, P., C.A. Sanchez, J. Blomer, L. Franco, A. Haru-
ules identification. The validation has been per- tyunian, P. Mato, & Y. Yao (2010). Cernvm–a virtual
formed over the biggest public available database software appliance for lhc applications. In Journal of
with an overall sensitivity of about 80% at 6–8 FP Physics: Conference Series, Volume 219, pp. 042003.
IOP Publishing.
per scan. This represents a very remarkable result, Camarlinghi, N., M. Fantacci, I. Gori, & A. Retico
not only for the high sensitivity reached, but also (2011). A computer-aided detection system for lung
because this represents the only work with a valida- nodules in ct images.
tion performed over such a big database. Further- Cerello, P., S.C. Cheran, S. Bagnasco, R. Bellotti, L.
more, the project has proposed a new approach to Bolanos, E. Catanzariti, G. De Nunzio, M.E. Fan-
make CAD algorithms and CAD results available tacci, E. Fiorina, G. Gargano, et al. (2010). 3-d object
to users without requiring the installation of soft- segmentation using ant colonies. Pattern Recognition
ware or hardware. In addition, the computational 43(4), 1476–1490.
solution based on a cloud computing back-end Coombs, K. (2009). Drupal done right. Library journal
134(19), 30–32.
allows to solve all issues described in Chapter 4 Das, M., G. Mhlenbruch, A.H. Mahnken, T.G. Flohr,
regarding the need of powerful resources for CAD L. Gndel, S. Stanzel, T. Kraus, R.W. Gnther, & J.E.
computations. The prototype system has already Wildberger (2006). Small pulmonary nodules: Effect
been tested by some Italian hospitals with good of two computeraided detection systems on radiolo-
results. One of the final goals of the project is to gist performance 1. Radiology 241(2), 564–571.
investigate the usage of CAD as second reader De Nunzio, G., E. Tommasi, A. Agrusti, R. Cataldo, I.
inside clinical routine, meaning that the radiolo- De Mitri, M. Favetta, S. Maglio, A. Massafra, M.
gist will see CAD results only after having com- Quarta, M. Torsello, et al. (2011). Automatic lung seg-
pleted the annotation of the case. He/She will be mentation in ct images with accurate handling of the
hilar region. Journal of digital imaging 24(1), 11–27.
then able to to review his annotation after hav- Ferlay, J., H.-R. Shin, F. Bray, D. Forman, C. Mathers, &
ing seeing these results. The comparison of the D.M. Parkin (2010). Estimates of worldwide burden
annotations (before and after CAD) will allow of cancer in 2008: Globocan 2008. International jour-
to evaluate the impact of CAD into the perform- nal of cancer 127(12), 2893–2917.
ance of the radiologist. In order to achieve these Gori, I., F. Bagagli, M.E. Fantacci, A.P. Martinez, A.
goals the project has started a collaboration with Retico, I. De Mitri, S. Donadio, C. Fulcheri, G. Gar-
the radiology department of the IRCCS (Institute gano, R. Magro, et al. (2007). Multi-scale analysis of
for Cancer Research) in Candiolo (Turin, Italy). lung computed tomography images. Journal of Instru-
The project has decided to adopt a perspectival mentation 2(09), P09007.
Jemal, A., R. Siegel, E. Ward, Y. Hao, J. Xu, & M.J. Thun
approach, acquiring exams of patients undergo- (2009). Cancer statistics, 2009. CA: a cancer journal
ing chest CT during the daily activity of the hos- for clinicians 59(4), 225–249.
pital (with a rate of about 50 CT scans per week). Jemal, A., R. Siegel, J. Xu, & E. Ward (2010). Cancer sta-
We have decided to adopt this approach because tistics, 2010. CA: a cancer journal for clinicians 60(5),
it will allow to have a validation with a data-set up 277–300.
to 1–2 thousands CTs per year, one order greater Lamanna, M. (2004). The lhc computing grid project
than data-set found in literature (Das et al. 2006) at cern. Nuclear Instruments and Methods in Phys-
(Brochu et al. 2008). ics Research Section A: Accelerators, Spectrometers,
Detectors and Associated Equipment 534(1), 1–6.
Li, Q., S. Sone, & K. Doi (2003). Selective enhancement
filters for nodules, vessels, and airway walls in two-
REFERENCES and threedimensional ct scans. Medical physics 30(8),
2040–2051.
Bellotti, R., P. Cerello, S. Tangaro, V. Bevilacqua, Matsumoto, S., Y. Ohno, H. Yamagata, D. Takenaka, &
M. Castellano, G. Mastronardi, F. De Carlo, K. Sugimura (2008). Computer-aided detection of
S. Bagnasco, U. Bottigli, R. Cataldo, et al. (2007). lung nodules on multidetector row computed tom-
Distributed medical images analysis on a grid infra- ography using three dimensional analysis of nodule
structure. Future Generation Computer Systems 23(3), candidates and their surroundings. Radiation medicine
475–484. 26(9), 562–569.

198

VIPIMAGE15_Book.indb 198 9/8/2015 7:45:06 PM

 Mell, P. & T. Grance (2011). The nist definition of Team, N.L.S.T.R. et al. (2011). Reduced lung-cancer mor-
cloud computing. Milojičić, D., I.M. Llorente, & tality with low-dose computed tomographic screening.
R.S. Montero (2011). Open nebula: A cloud manage- The New England journal of medicine 365(5), 395.
ment tool. IEEE Internet Computing (2), 11–14. Torres, E.L., E. Fiorina, F. Pennazio, C. Peroni, M.
Paik, D.S., C.F. Beaulieu, G.D. Rubin, B. Acar, Saletta, N. Camarlinghi, M. Fantacci, & P. Cerello
R.B. Jeffrey Jr, J. Yee, J. Dey, & S. Napel (2004). Sur- (2015). Large scale validation of the m5l lung cad
face normal overlap: a computer-aided detection on heterogeneous ct datasets. Medical physics 42(4),
algorithm with application to colonic polyps and lung 1477–1489.
nodules in helical ct. Medical Imaging, IEEE Transac- van Ginneken, B., S.G. Armato, B. de Hoop, S. van
tions on 23(6), 661–675. Amelsvoortvan de Vorst, T. Duindam, M. Niemeijer,
Retico, A., P. Delogu, M.E. Fantacci, I. Gori, & A.P. K. Murphy, A. Schilham, A. Retico, M.E. Fantacci,
Martinez (2008). Lung nodule detection in low-dose et al. (2010). Comparing and combining algorithms
and thinslice computed tomography. Computers in for computer-aided detection of pulmonary nodules
biology and medicine 38(4), 525–534. in computed tomography scans: the anode09 study.
Retico, A., M.E. Fantacci, I. Gori, P. Kasae, B. Golosio, Medical image analysis 14(6), 707–722.
A. Piccioli, P. Cerello, G. De Nunzio, & S. Tangaro
(2009). Pleural nodule identification in low-dose and
thinslice lung computed tomography. Computers in
biology and medicine 39(12), 1137–1144.

199

VIPIMAGE15_Book.indb 199 9/8/2015 7:45:06 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Non-intrusive and calibration free visual exploration analysis in children

with Autism Spectrum Disorder

Dario Cazzato & Francesco Adamo

Faculty of Engineering, University of Salento, Lecce, Italy

Giuseppe Palestra
Department of Computer Science, University of Bari, Italy

Giulia Crifaci, Paola Pennisi & Giovanni Pioggia

Insitute of Clinical Physiology, National Research Council of Italy, Messina, Italy

Liliana Ruta
Department of Developmental Neuroscience, Stella Maris Scientific Institute, Calambrone, Pisa, Italy

Marco Leo & Cosimo Distante

National Institute of Optics, National Research Council of Italy, Arnesano (LE), Italy

ABSTRACT: Assistive technology is a generic system that is used to increase, help or improve the func-
tional capabilities of people with disability. Recently, its employment has generated innovative solutions
also in the field of Autism Spectrum Disorder (ASD), where it is extremely challenging to obtain feed-
back or to extract meaningful data. In this work, a study about the possibility to understand the visual
exploration in children with ASD is presented. In order to obtain an automatic evaluation, an algorithm
for free gaze estimation is employed. The proposed gaze estimation method can work without constrains
nor using additional hardware, IR light sources or other intrusive methods. Furthermore, no initial cali-
bration is required. These relaxations of the constraints makes the technique particularly suitable to be
used in the critical context of autism, where the child is certainly not inclined to employ invasive devices.
In particular, the technique is used in a scenario where a closet containing specific toys, that are neatly
disposed from the therapist, is opened to the child. After a brief environment exploration, the child will
freely choose the desired toy that will be subsequently used during therapy. The video acquisition have
been accomplished by a Microsoft Kinect sensor hidden into the closet in order to obtain both RGB and
depth images, that can be processed by the estimation algorithm, therefore computing gaze tracking by
intersection with data coming from the well-known initial disposition of toys. The system has been tested
with children with ASD, allowing to understand their choices and preferences, letting to optimize the toy
disposition for cognitive-behavioural therapy.

1 INTRODUCTION research in developmental psychology. Signs of

autism are not reliably present at birth, but emerge
Autism is a neurodevelopmental disorder strongly through a process of diminishing, delayed, or
characterized by deficits in social interaction and atypical development of social-communication
impaired understanding of the mental states of behaviours, starting between the ages of 6 and
others (Siegel, Vukicevic, & Spitzer, 1990), a dys- 12 months. Intervention and support should be
function that persists even in people with autism individualized and, if appropriate, multidimen-
who have IQs in the normal range. Autistic Spec- sional and multidisciplinary (Lai MC et al., The
trum Disorders (ASD) is no longer a rare impair- Lancet, 2014).
ment with an extremely poor prognosis, but a Aesthetic preferences of subjects with ASD
multi-factorial disorder whose qualitative impair- are very different (Bogdashina, 2003). The pre-
ments can be mitigated with early intensive inter- ferred toy of a child can be intolerable for another
vention. Worldwide population prevalence is about child. Often, to understand whether an object
1%. Early diagnosis of ASD is a central topic of is attracting or not the attention of an autistic

201

VIPIMAGE15_Book.indb 201 9/8/2015 7:45:06 PM

 child can be impossible because autistic children 2007). Many approaches to technology-enhanced
can use gestures or vocalization to express their intervention rely on educational methods shown to
needs by protoimperative gestures, but usually result in good outcomes and can be used to specify
they don’t communicate objects of shared inter- design principles needed for engineering successful
est by protodeclarative gestures (Baron-Cohen, S., technology-enhanced intervention tools (Tentori
1989; Curcio, F., 1978). Also in pre-verbal chil- M, Hayes G, 2010).
dren, absence of protodeclarative gestures is con- Considering the scarcity of such specialized
sidered a discriminating items between infants medical practitioners compared to the volume
who had later been diagnosed with autism and of patients, along with considering the high-cost
typically developing infants (Clifford et al. 2007). involved, there is a strong motivation for investigat-
Baron-Cohen et al. (1996) showed that absence of ing technological alternatives to solve the problem
protodeclarative pointing, gaze-monitoring and (Sivalingam et al., 2012). From the other side, com-
pretend play in children 18 months olds, in 83.3% puter vision technology has a unique opportunity
of cases predicts a future diagnosis of autism. In to impact that study of children’s behaviour, by
Modified Checklist for Autism in Toddlers, proto- providing a means to automatically capture behav-
declarative gestures are considered a key behaviour ioural data in a non-invasive manner and analyse
(Robins et al. 2009). behavioural interactions. Computational sensing
Autistic subjects show atypically gaze behav- and modelling techniques can play an important
iours: in real social situations, subjects with ASD role in the capture, measurement, analysis, and
show reduced salience of eyes and increased sali- understanding of human behaviour; this research
ence of mouths, bodies and objects (Klin et al. area is called Behaviour Imaging.
2002; Shic et al. 2011) and look significantly less In clinical settings, assessments of behaviour
at the partner in dyadic interaction rather than are typically based on the direct observation of a
typically development children (Noris et al. 2012). child by an experienced clinician. The ability to
In artificial social situations, (for example when automatically measure behavioural variables using
they see a cartoon or a movie with social actors, computer vision-based sensing could be valuable
or in static photos, or in virtual reality, etc.) they in enabling the collection of behavioural data on
attend to characters’ faces for less time that typi- a large scale without requiring substantial human
cally development subjects (Riby et Hancock effort. Behaviour imaging technology can play
2009; Riby et Hancock 2008) and show less the several roles in support of a screening instrument.
centre of subjects’ faces rather than control group It can provide cost-effective tools for managing
(Trepagnier et al. 2002). large collections of video and other data sources
Visual atypical behaviours also affect the way recorded during screening sessions. In particu-
in which subjects with ASD observe the scene; lar, it can enable summarization, content-based
Shic et al. (2011) showed that children with ASD, retrieval, visualization, and comparison of obser-
in comparison to typically development subjects, vational data across populations and over time, to
are more attracted by background object. Many an extent that is not feasible using conventional
studies show an abnormal exploration of object manual methods (Rehg, 2011).
stimuli (Ozonoff et al. 2008); the work of Sasson In this work, a study about a possibility to
et al. (2008), for example, showed that, compared understand the visual exploration in children with
with typically development subjects, subject with ASD is presented. In order to obtain an automatic
ASD are more perseverative in fixations of details evaluation, an algorithm for free (i.e. without con-
of images and that their fixations are more detail strains nor using additional hardware, IR light
oriented, while Hutman et al. (2012) showed that sources or other intrusive methods) gaze estima-
infants later diagnosed with ASD are more atten- tion is employed. Furthermore, no initial calibra-
tive to non-social stimuli during interactions with tion is required. In particular, the technique is used
an unfamiliar experimenter and that they shift in a scenario where a closet containing specific
attention among visual targets less frequently than toys, that are neatly disposed from the therapist,
high and low-risk typically developing children. is opened to the child. After a brief environment
By a simple preferential looking task, Pierce et al. exploration, the child will freely choose the desired
(2011) showed that 40% of the autistic cross sam- toy that will be subsequently used during therapy.
ple spent greater than 50% of viewing time fixating The video acquisition have been accomplished by
dynamic geometric images rather than dynamic a Microsoft Kinect sensor hidden into the closet in
social images. Several experimental protocols for order to obtain a depth image that can be processed
the diagnosis and understanding of developmen- by the estimation algorithm, therefore computing
tal disorders make use of video footage analysis gaze tracking by intersection with data coming
to measure such elements as response time, atten- from the well-known initial disposition of toys.
tion changes and social interaction (Noris et al., The rest of the manuscript is organized as follows.

202

VIPIMAGE15_Book.indb 202 9/8/2015 7:45:06 PM

 Section 2 introduces the proposed method in terms
of employed system and selected participants. Sec-
tion 3 explores the experimental setup and the sen-
sitive data acquisition phase, necessary to evaluate
results off-line, and results are presented, while
Section 4 discusses them. Finally, Section 5 con-
Figure 1. A block diagram of the proposed method.
cludes the paper.

2 METHODS AND PARTICIPANTS

The employed solution is based on the work of

(Cazzato et al. 2012). A block diagram of the pro-
posed method is showed in Fig. 1.
Method’s workflow is the following: first of
all, images are acquired from a depth sensor, tak-
ing into account both RGB and depth channels.
A search for a face over the input image is then
carried out on the RGB image. A detected face is
then tracked over the time, and a parameterized
and people independent 3D face mask is subse-
quently overlapped on the human face. The face
tracker is based on the Active Appearance Model
(AAM), while the alignment is performed using Figure 2. A scheme of the gaze estimation solution.
Iterative Closest Point (ICP). The depth informa-
tion allows to reconstruct the 3D positions of all of
the tracked points and, therefore, to estimate the computed value, in order to translate the gaze vec-
head pose in terms of yaw, pitch and roll angles. tor to the right position. This way, the proposed
Starting from the average point between the two method can be easily extended to compute the
detected three-dimensional positions of the eyes (possible) intersection with an object of interest in
and using the head pose information, a gaze track the plane. In this case, the virtual plane with lying
is finally estimated as well as its intersection with objects is modelled with all the possible bound-
a vertical plane with regard to the ground and ing boxes of the objects of interest. Therefore, the
passing from the centre of the sensor, representing algorithm is able to determine if the gaze track ray
the observed point. Our gaze estimation method has an intersection with a particular bounding box
works as follows. First of all, the 2D position of or not, determining which object in which frame
the detected eye centre points are taken from the have been observed.
face mask. Note that small occlusions are handled, About participants, the sample consisted
and the eye centre point is always estimated in the of three boys (mean age: 50.3 months, range:
used model when the overlapping with the face 42–55 months), two with Autism Spectrum Dis-
successes. After that, the average value is taken, in order and one with Autistic Disorder, recruited at
order to take a point corresponding more or less the Pervasive Healthcare Center of the Institute
with the nose septum and to use it as the origin of Clinical Physiology of the National Research
of the gaze track. This value is converted into 3D Council of Italy in Messina. The diagnostic assess-
coordinates with regard to a Cartesian coordinate ment was based on the Autism Diagnostic Obser-
system centred inside the sensor. Starting from vation Schedule, Griffith’s Developmental Scale,
the head pose information, two angles are taken, Vineland Adaptive Scale, McArtur Language Test
i.e. ωx and ωy, corresponding respectively to pitch and Child Behaviour Checklist (CBCL).
and yaw. Then, the gaze track is computed and its
intersection with a plane vertical with regard to the
ground and passing from the centre of the sensor is 3 DATA ACQUISITION AND
calculated. The intersection point is computed sep- EXPERIMENTS
arately for each angle and using the same method,
and the Euclidean distance from the sensor can Our method has been employed in a scenario
hereafter be computed. The procedure is showed where a closet containing specific toys, that are
for one angle in Fig. 2. neatly disposed from the therapist, is opened to the
In case of translations on the x and y axes, the child. Fig. 3 shows the disposition of the toys into
vector can be algebraically summed up with the the closet. The closet has size of 85 × 195 cm, but

203

VIPIMAGE15_Book.indb 203 9/8/2015 7:45:06 PM

 only three sectors (in the bottom) have been used, After a brief environment exploration, the child
considering the age of the involved children. Thus, freely chooses the desired toy that will be subse-
the useful part has a size of 85 × 117 cm. quently used during therapy. Concerning the face
In Fig. 3, 9 Areas Of Interest (AOI) are designed. of the child, video acquisition has been accom-
They have been manually computed, since a static plished by a Microsoft Kinect sensor hidden into
disposition has been employed. Each AOI has a the closet in order to obtain a depth image that will
size of 28,3 × 39 cm. They are labelled from 1 to be processed by the estimation algorithm, that will
9, and to each one of them correspond a specific compute gaze tracking by intersection with data
toy. Exception, the cell number 2 will represent the coming from the well-known initial disposition of
Kinect area. Gaze data was extracted for each of toys. Depth image is necessary to estimate the head
these AOI. Fixations can be aggregated at many pose and to know the X, Y, Z world coordinate of
different levels, resulting in a wealth of psycho- the detected facial points. The green LED of the
physical measures. In particular, the following sensor has been darkened, to avoid to be appeal-
parameters have been taken into account: ing to the child and to invalidate our experiment.
The depth sensor was connected on an Ultrabook
• Fixation count: the number of fixation in a spe-
Intel i3 CPU @ 1.8GHz with 4 GB of RAM, work
cific AOI. For our system, a fixation of a toy
at 30fps. The usage of our technique on a com-
is considered to be consistent if at least 15 con-
mon Ultrabook was made in order to facilitate for
secutive frames present an hit on the square in
raw installations like in the topmost shelves of the
exam;
closet. This is a very encouraging result, since it
• First fixation: the first consistent AOI looked by
allows to provide the useful data in real-time.
the patient;
All recordings were suitable for off line analysis
• Sequence: the order of gaze hits into the AOIs;
with our algorithm. For this experiment, one well-
• Selected Toy: the final choice of the patient;
known disposition of the toys is used, and a patient
• Most viewed toy: the cell with the maximum
gaze track is firstly computed and projected on the
number of hits.
image plane; in a further step, its intersection with
the toys’ AOIs have been computed.
For each patient, these values have been
recorded. Furthermore, these values have been
elaborated, and results are summed up in Table 1.
2D visual exploration on a plane has been
tracked over time, and this datum had been related
with the chosen toy by means of the belonging
bounding box. Fig. 4 shows the x and the y visual
in an ordered sequence of frames where detection
occurred, while Fig. 5 shows the computed hit map
for the same patient during the interaction. The
system was tested with three children, showing a
completely different behaviour; in particular, the
system tracked the first child only few frames, due
to his very fast and out of field movements. The
child selected a toy after a very short exploration,
the first fixation was in the AOI 5 and the most
viewed toy was in the AOI 2. There are no relations
between what he watched most and the selected toy.
The second child has a lower number of fixations
than the other children. The selection went toward
the last toy observed, in the opposite direction in

Table 1. Gaze measures during tasks.

Child 1 Child 2 Child 3

Fixation Count 4 3 6
Sequence 5-2-5-7 2-5-6 5-2-3-5-2-1
First Fixation Cell 5 2 5
Selected Toy 7 4 1
Figure 3. An example of toys disposition into the
Most Viewed toy 2 6 1
closet.

204

VIPIMAGE15_Book.indb 204 9/8/2015 7:45:06 PM

 Figure 6. The computed hit map for person #2.

Figure 4. Recorded screen coordinates (X on top, Y on

the bottom) for person #1.

Figure 7. The computed hit map for person #3.

4 DISCUSSION

In 2008, a case study reported by Vismara et Rog-

ers showed a reduction in autistic symptoms thanks
to a novel paradigm for treatment of autism based
Figure 5. The computed hit map for person #1. on a early intervention: the Early Start Denver
Model (ESDM). Dawson et al. (2010) showed the
efficacy of the ESDM for children with autism in
respect to the toys seen previously. In this case, only improving cognitive and adaptive behaviours and
with the usage of our system is possible to capture reducing severity of ASD diagnosis. Rogers and
details and, in this case, to understand the real will Dawson (2010) emphasize the role of motivation
of the child. Finally, in the third video, the system to treat children with the ESDM. To motivate chil-
track different interval of the visual exploration, dren they recommend to put subjects in a situation
observing a large exploration of toys before con- where there are many age-appropriate toys and
centrating on a point and taking the correspond- then observe what children do. In this situation,
ing toy. This child has a higher number of fixations children will probably watch toys more interesting
than the other children. The selected toy was the for them, however the absence of protodeclarative
most viewed, He spent more time to see the chosen gestures in children with autism (Baron-Cohen, S.,
toy than the other. Figures 6 and 7 show the hit 1989; Curcio, F., 1978) can be a problem for the
maps for the person #2 and #3, respectively. therapist. In this contest, gaze estimator can be

205

VIPIMAGE15_Book.indb 205 9/8/2015 7:45:08 PM

 a good instrument to understand where children 5 CONCLUSIONS
have focused its attention. Eye tracking is a useful
methodology for investigating spontaneous priori- Computer vision technology has a unique oppor-
ties and patterns of attention. Relative priorities tunity to impact that study of children’s behaviour,
in attention can be inferred by analysing which by providing a means to automatically capture
regions of a stimulus a participant looks at, for behavioural data in a non-invasive manner and
how long the regions are looked at and when dur- analyse behavioural interactions between children
ing the course of viewing the various regions are and toys. Eye tracking behavioural assessments
looked at. may provide spatial information than face-to-face
This non-invasive method can facilitate our assessments. These tools aid the clinician in the
understanding of underlying cognitive processes behavioural assessment task by providing accurate
involved in object perception and the more com- and objective measurements.
plex challenges related to social functioning in The early detection of developmental disorders
ASD (Boraston and Blakemore 2007). Knowing is key to child outcome, allowing interventions to
where a subject is looking provides a wealth of be initiated that promote development and improve
information regarding his motivations, expecta- prognosis. Research on ASD suggests behavioural
tions, and innate preferences. For this reason, eye markers can be observed late in the first year of
tracking has become a standard tool for cognitive life. Many of studies involved extensive frame-by-
and psychological investigation. However, though frame video observation and analysis of a child’s
the direct examination of the scan path obtained natural behaviour. Although non-intrusive, these
by eye-tracking systems tells us where individuals methods are extremely time-intensive and require
are looking, it does not tell us why they choose to a high level of observer training. With the goal of
look at different locations, and, of course, it does aiding and augmenting the visual analysis capabili-
not tell us what these differences mean. In order ties in evaluation and developmental monitoring
to answer the more difficult question, that of how of ASD, we proposed computer vision tools to
scan patterns should be interpreted, more sophisti- observe specific behaviours related to ASD elic-
cated methods for distilling information from raw ited during toy selection tasks, providing both new
visual trajectories are required. challenges and opportunities in video analysis.
Before concluding, a discussion that considers
the differences between a commercial eye tracker
and the proposed system is reported. First of all, ACKNOWLEDGEMENTS
all the parameters of interest during a visual explo-
ration of the patient have been taken into account. This work has been supported by the “2007–
About them, it was possible to obtain a precise esti- 2013 NOP for Research and Competitiveness for
mation of the number of fixations, the sequence of the Convergence Regions (Calabria, Campania,
points of interest and where the attention goes at Puglia and Sicilia)’’ with code PON04a3_00201.
first. Two parameters that can also be estimated,
but further development would be necessary, are
the time of permanence and the average fixation BIBLIOGRAPHY
length. Although it could be easily extracted add-
ing a temporization on the detected frames, there Baron-Cohen, S., Cox, A., Baird, G., Swettenham, J.,
were still a number of misdetection that could fal- Nightingale, N.A.T.A.S.H.A., Morgan, K.A.T.E. &
sify the estimation, since we cannot automatically Charman, T. (1996). Psychological markers in the
detection of autism in infancy in a large population.
detect what happen in that frames (or it could be The British Journal of Psychiatry, 168(2), 158–163.
rebuilt by interpolation, but its value would not be Baron-Cohen, S. (1989). Perceptual role taking and pro-
reliable for the particular application context in todeclarative pointing in autism. British Journal of
exam). For this reason, we decided to do not report Developmental Psychology, 7(2), 113–127.
these values. From the other hand, advantages of Bogdashina, O. (2003). Sensory perceptual issues in autism
the proposed systems are that the patient was able and Asperger Syndrome: different sensory experiences,
to participate to the session in a completely ecolog- different perceptual worlds. Jessica Kingsley Publishers.
ical way, i.e. without requiring any initial calibra- Boraston, Z. & Blakemore, S.J. (2007). The application
tion, nor constraints about its position. Moreover, of eye-tracking technology in the study of autism.
The Journal of physiology, 581(3), 893–898.
the proposed system is considerably inexpensive Cazzato, D., Leo, M. & Distante, C. (2014). An Inves-
compared to commercial eye tracker solutions. tigation on the Feasibility of Uncalibrated and
This was a very encouraging results, that could Unconstrained Gaze Tracking for Human Assistive
provide a new inexpensive technique for evaluating Applications by Using Head Pose Estimation. Sen-
focus-of-attention in autistic children. sors, 14(5), 8363–8379.

206

VIPIMAGE15_Book.indb 206 9/8/2015 7:45:10 PM

 Clifford, S., Young, R. & Williamson, P. (2007). Assess- Riby, D. & Hancock, P.J. (2009). Looking at movies and
ing the early characteristics of autistic disorder using cartoons: eye-tracking evidence from Williams syn-
video analysis. Journal of autism and developmental drome and autism. Journal of Intellectual Disability
disorders, 37(2), 301–313. Research, 53(2), 169–181.
Curcio, F. (1978). Sensorimotor functioning and commu- Riby, D.M. & Hancock, P.J. (2008). Viewing it differ-
nication in mute autistic children. Journal of autism ently: Social scene perception in Williams syndrome
and childhood schizophrenia, 8(3), 281–292. and autism. Neuropsychologia, 46(11), 2855–2860.
Dawson, G., Rogers, S., Munson, J., Smith, M., Robins, D.L., Fein, D. & Barton, M. (2009). Modified
Winter, J., Greenson, J. & Varley, J. (2010). Rand- Checklist for Autism in Toddlers, Revised, with Fol-
omized, controlled trial of an intervention for toddlers low-Up (M-CHAT-R/F) TM.
with autism: the Early Start Denver Model. Pediatrics, Rogers, S.J. & Dawson, G. (2010). Early Start Denver
125(1), e17-e23. Model for young children with autism: Promoting
Hutman, T., Chela, M.K., Gillespie-Lynch, K. & language, learning, and engagement. New York: Guil-
Sigman, M. (2012). Selective visual attention at twelve ford Press.
months: Signs of autism in early social interactions. Sasson, N.J., Turner-Brown, L.M., Holtzclaw, T.N.,
Journal of autism and developmental disorders, 42(4), Lam, K.S. & Bodfish, J.W. (2008). Children with
487–498. autism demonstrate circumscribed attention during
Klin, A., Jones, W., Schultz, R., Volkmar, F. & passive viewing of complex social and nonsocial pic-
Cohen, D. (2002). Visual fixation patterns during ture arrays. Autism Research, 1(1), 31–42.
viewing of naturalistic social situations as predic- Siegel, B., Vukicevic, J. & Spitzer, R.L. (1990). Using
tors of social competence in individuals with autism. signal detection methodology to revise DSM-III-R:
Archives of general psychiatry, 59(9), 809–816. Re-analysis of the DSM-III-R national field trials for
Lai, M.C., Lombardo, M.V. & Baron-Cohen, S. (2014) autistic disorder. Journal of psychiatric research, 24(4),
Autism. Lancet 383(9920):896–910. 293–311.
Noris, B., Benmachiche, K., Meynet, J., Thiran, J.P. & Shic, F., Bradshaw, J., Klin, A., Scassellati, B. &
Billard, A.G. (2007). Analysis of Head-Mounted Wire- Chawarska, K. (2011). Limited activity monitor-
less Camera Videos for Early Diagnosis of Autism. ing in toddlers with autism spectrum disorder. Brain
In Computer Recognition Systems 2 (pp. 663–670). research, 1380, 246–254.
Springer Berlin Heidelberg. Tentori, M. & Hayes, G.R. (2010, September). Design-
Noris, B., Nadel, J., Barker, M., Hadjikhani, N. & Billard, ing for interaction immediacy to enhance social skills
A. (2012). Investigating gaze of children with ASD in of children with autism. In Proceedings of the 12th
naturalistic settings. PloS one, 7(9), e44144. ACM international conference on Ubiquitous com-
Ozonoff, S., Macari, S., Young, G.S., Goldring, S., puting (pp. 51–60). ACM.
Thompson, M. & Rogers, S.J. (2008). Atypical Trepagnier, C., Sebrechts, M.M. & Peterson, R. (2002).
object exploration at 12 months of age is associated Atypical face gaze in autism. Cyberpsychology &
with autism in a prospective sample. Autism, 12(5), Behavior, 5(3), 213–217.
457–472. Vismara, L.A. & Rogers, S.J. (2008). The Early Start
Rehg, J.M. (2011, June). Behavior Imaging: Using Com- Denver Model: A case study of an innovative practice.
puter Vision to Study Autism. In MVA (pp. 14–21). Journal of Early Intervention.

207

VIPIMAGE15_Book.indb 207 9/8/2015 7:45:10 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Semi-automatic tumor contouring method using PET

and MRI medical images

Szabolcs Urbán
Institute of Informatics, University of Szeged, Szeged, Hungary

László Ruskó
GE Healthcare, Szeged, Hungary

Antal Nagy
Institute of Informatics, University of Szeged, Szeged, Hungary

ABSTRACT: Tumor contouring is a challenging task for physicians especially when functional as well
as anatomical images are incorporated. This paper presents a semi-automated technique to solve this
problem. The proposed method is a variant of the level-set segmentation where the initial region is based
on the functional image and the speed function combines both images. The presented approach allows
the user to balance between the information of two input images. The method was evaluated on registered
head and neck PET-MRI image pairs using manually defined tumor contours as reference. The algorithm
was tested for various types of tumors using different weights to combine the functional and the anatomi-
cal information. The best results showed good correlation with the reference (3% volume difference and
80% DICE similarity in average).

1 INTRODUCTION lesions. That is why there is significant need for

computer assisted tools which make this process
The number of patients living with cancer is more accurate, efficient, and reproducible.
increasing from year to year. 3-dimensional (3D) Large number of tumor contouring methods
medical imaging plays central role in the man- have been published which rely wether on anatomi-
agement of patients with cancer during diagno- cal or functional images, but only a few of them
sis, therapy planning, and monitoring. There are can incorporate both of them, especially PET
various 3D imaging techniques available in clinical and MRI. El Naqa et al. (El Naqa, Yang, Apte,
routine. Computed Tomography (CT), Magnetic Khullar, Mutic, Zheng, Bradley, Grigsby, & Deasy
Resonance Imaging (MRI) and Positron Emission 2007) presented a level-set based method which
Tomography (PET) imaging supplies complemen- uses the mean intensity values from the inside and
tary information about the tumor being examined outside region of the tumor contour. They tested
(AR. 2014). CT and MRI provides anatomical their method on two tumor cases (using PET-CT).
information at high spatial and contrast resolu- On (artificially generated) phantom data they
tion, while PET gives functional information at achieved 90% DSC in average (using PET, CT and
lower spatial resolution. MRI), however the shown input images contained
Tumor contouring is an important task of oncol- almost the same, homogeneous sphere-like object.
ogy because the location and the change of the vol- Bagci et al. first published a tumor segmenta-
ume of the lesion plays crucial role in therapeutic tion approach based on PET/CT images (Bagci,
decisions. The more accurate the tumor quantifi- Udupa, Yao, & Mollura 2012). They represent the
cation is the more reliable is the evaluation of the images in a product lattice and perform random
therapy’s effectiveness. Manual tumor contouring walk image segmentation. In this approach the
is very time consuming and subject to operator foreground and the background is selected based
variability, especially when multiple images shall on the PET image such that background points
be incorporated to define the lesion boundary. It is are only taken from the low PET intensity regions.
also a very challenging task (even for experienced Tumors can be significantly larger in the MRI image
physicians) due to the large variation in localiza- compared to the PET, therefore the background
tion, shape, size, and intensity heterogeneity of the points selected this way can be inaccurate. Later

209

VIPIMAGE15_Book.indb 209 9/8/2015 7:45:10 PM

 they generalized this method to use more than two
images (Bagci, Udupa, Mendhiratta, Foster, Xu,
Yao, Chen, & Mollura 2013). According to their
results this approach performed well for various
clinical cases including MRI-PET scans (93.1%
DSC in average). However these images contained
almost no registration errors and the shown MRI-
PET case contained similar information about the
tumor on each of the input images. Figure 1. Main steps of the method: PET with ini-
tial region in magenta (a), hybrid speed function using
λ = 30% (b), and result contour for level = 23 in MRI (c).
2 THE PROPOSED METHOD

The goal of this work was to develop an interactive In FDG-PET images high uptake values show high
tool that can combine the information of registered glucose metabolism (high intensities), which can be
PET and MRI images to facilitate tumor contour- normal inside some regions (e.g. brain, muscles),
ing in the head and neck region which contains but it usually reflects pathology (tumor, inflam-
various-shaped tumors and the mentioned images mation) in the area of the neck. In the input PET
can contain complementary/contradictoire infor- images the intensities are represented by Standard-
mation. The main idea was to extend a standard ized Uptake Value (SUV). Inside the user-defined
segmentation technique to incorporate functional ROI the maximal (SUV Vmax ) value is calculated and
and anatomical images in a user-friendly way. Our its 40% is used for thresholding the high-uptake
primary goal was to develop a MR-PET tumor region. The largest 3D-connected component of
contouring tool the result of which is consistent the detected region, which can be considered as the
with both input images and the user’s expectations. active part of the tumor, is the basis of the starting
This goal was primarily motivated by discussions region. Due to the lower spatial resolution of the
with physicians working at the field of radiother- functional image, the border of this region cannot
apy planning. They are challenged with defining be always considered as part of the tumor in the
the target volume (i.e. the tumor) when both func- anatomical image (that has much higher spatial
tional and anatomical images are given because resolution). Furthermore the input images were
they have to switch back and forth between the globally registered, which cannot guarantee the
input images when they manually contour the dif- local accuracy of the registration at each tumor.
ferent parts of the tumor. This way they include all To take these observation into account, the initial
regions which may belong to the tumor based on region is defined by the skeleton of the connected
any of the input images. According to our experi- region. The applied 3D skeletonization algorithm
ments with head-and-neck cases fully automated (Lee, Kashyap, & Chu 1994) guarantees the con-
approach is hardly achievable, while the presented nectedness of the result. The generated skeleton
semi-automated technique can facilitate this com- is a thin structure, therefore dilation is applied to
plex task. The proposed method consists of two that. The dilation incorporates the thresholded
main steps. First, an initial region is extracted from uptake region, so that the final skeleton is always
the user-defined Region Of Interest (ROI) based located inside that. The proposed initial region rep-
on the functional image. Then, 3D fastmarching resents the shape of the functionally active region
approach is applied to generate a set of contours, and it is not sensitive for small local registration
from which the user can select one (referred as level) inaccuracy. An example for initial region is shown
that fits both images as well as the user’s expec- in Figure 1a.
tations. The contour propagation is driven by a In order to combine the functional and the ana-
hybrid speed function that incorporates both PET tomical information of the input images a hybrid
and the MRI image intensities such that the weight speed function is defined in the following way.
of the PET (referred as λ) and MRI (referred as First, the intensity range of both PET and MRI
1 − λ ) images can be specified by the user. This way images is normalized to the [0,1] interval, using
the user can balance between the PET and MRI the min-max normalization technique. Since the
information according to the characteristics of the intensity maximum of an MRI images can be
tumor to be contoured. Figure 1 demonstrates the associated with some voxels affected by local imag-
main steps of the algorithm. ing artifacts, we use the largest intensity that is
The segmentation requires a starting region. This higher than 99% of the voxels of the MRI image.
region is the basis of the contour propagation, so it This step is substantial because the input images
must be fully located inside the tumor. In order to involve different information, generated by differ-
get such a region, only the functional image is used. ent acquisition techniques. The functional (PET)

210

VIPIMAGE15_Book.indb 210 9/8/2015 7:45:10 PM

 and the anatomical (T2w water MRI) images used
in this study have similar characteristics. In case of
both image types the higher intensities represent
the tumorous tissue and lower intensities repre-
sent the background. After the normalization the
intensity range is the same for both images, which
makes it easier to combine the two different image
information. Figure 2. Tumor contour using different segmentation
parameters: λ = 30% (a and b) and λ = 70% (c and d);
The hybrid speed function F is defined at each
level = 10 (yellow) and level = 30 (magenta).
voxel in the following way:

1 1
F =λ⋅ I ffn − β ffn + ( − λ)⋅ I an − β aan (1) with the restriction that the speed function can
− −
1+ e
α ffn
1+ e α aan
never change sign. It can defined as follows:

where λ ∈[ 0,1] is the weighting factor; I an and I fn F ( x y z ) | ∇T ( x ) | = 1, x ∈ Ω. (2)

are the normalized voxel intensity value in the ana-
tomical and the functional images, respectively; β where F: speed function (F > 0), T: arrival function,
and α are the parameters of the sigmoid function: Ω: a domain in R n .
its centre and the steepness factor, respectively. The initial curve contains those x-es which
The speed function is defined as the weighted sum belong to the zero level (T ( x ) = 0). The main
of two sigmoid filters applied to the input images advantage of the fast-marching technique is that
voxel by voxel. This approach exploits that the it can be efficiently computed as a narrow-band
meaning of intensity values are similar in both level set. In our case alive points are defined based
input images (i.e. high value represents tumor). on the initial region and narrowband points as its
Therefore, the combined intensities represent the neighbors. The point of this definition is that the
same features which characterize the input images. seed region is always inside the tumor and repre-
The higher the value is in the speed function the sents the shape of the tumor based on the func-
faster the contour propagation is at the given voxel. tional image. Therefore, starting from this region
This way, the contour propagation slows down the whole tumor volume can be reached. The
when it reaches voxels which are located outside above-defined hybrid speed function allows incor-
the tumor according to any of the input images. porating the information from both input images.
Note that the speed function can be defined based Consequently, the contour propagation purely
on other image features like edges if the intensity relies on the joint information. The method gen-
does not characterize the tumor in the anatomi- erates an image, which represents the shortest
cal input image. In such a case the speed is high distances from the seed region at each voxel. The
in the homogeneous regions and low at the region result contour can be extracted by thresholding the
boundaries. distance map with a particular value (level). Simi-
Sigmoid functions are commonly used for better lar to parameter λ there is no single value for level
separation of higher and lower values in the speed that provides the best tumor contour in every cases.
function. The center of the function ( β ) is defined Therefore, the user shall adjust manually the level
as 0.5 and the steepness factor (α ) is set to 0.1 for according to the given tumor. Different (λ, level)
both input images ( fn, an ). Therefore the inten- combinations result in different tumor contours as
sity values of the input images below or above the shown in Figure 2. In this case the tumor is larger
range [ β α 2 β α 2 ] become significantly lower in the anatomical image, therefore using λ = 30%
or higher than the original values, respectively. and level = 30 produced the best contour.
The definition of the weighting factor (λ)
depends on the input images and the tumor to be
contoured. There are cases when the functional 3 RESULTS AND DISCUSSION
image can give more reliable information about
the tumor (e.g. when the tumorous region and its The method was evaluated using 8 registered
surroundings are very heterogenous in the MRI PET/MR image pairs. The test images originated
image). In other cases, the anatomical image allows from 3-modality examinations involving a (FDG)
more precise definition of the tumor contour. Con- PET-CT and a subsequent MRI scan at the Uni-
sequently, the weighting factor shall be set accord- versity Hospital Zurich. The MRI images were
ing to the given tumor contouring problem. acquired with a 3 Tesla MRI scanner (GE Health-
The 3D fast marching method (Sethian 1999) is a care) using multiecho IDEAL gradient-echo imag-
special case of the level-set segmentation approach ing. The test exams covered the head and neck

211

VIPIMAGE15_Book.indb 211 9/8/2015 7:45:11 PM

 region and involved a total number of 15 tumors.
Each MRI case included multiple sequences,
among which the T2 weighted IDEAL water image
was used because the tumor contour was the most
visible in this type of image.
For each case the CT image was registered glo-
bally to the MRI image using rigid registration
technique that was based on mutual-information.
The registration transform was applied to the PET
image that was also resampled using the resolution
and voxel spacing of the MRI image. This way the
corresponding PET information was available at
each voxel of the MRI image. In order to reduce
noise, anisotropic diffusion filter (Perona & Malik
1990) was applied to the MRI images.
In each case all tumors were manually seg-
mented by three physicians, separately. First, they
delineated the tumor contour in the functional
image, then they adjusted the contour based on
the anatomical image taking also the PET image
into account. After all three contours were avail- Figure 3. Effect of the parameters λ and level to the DSC
able they selected the best one and (if needed) accuracy (left), the reference (yellow) and the best result
made some manual adjustments in consensus. (magenta) in PET (center) and MRI (right) images.
The contour that was confirmed by all operators
was used as reference. For the evaluation the Dice
Similarity Coefficient (Dice 1945) and Absolute column the Dice Similarity Coefficient (DSC) is
volume difference were used. The first metric is plot for all parameter combinations (yellow means
used to compare the overlap of two segmentation, high accuracy). The second and third columns
while the second one shows the difference of two show the best contour (magenta) and the reference
volumes, which is very important in the case of (yellow) in a characteristic axial slice of the tumor
tumor assessment. The test cases involved typical in the PET and the MRI images, respectively.
head-and-neck tumors characterized by different In the top row of Figure 3 a small tumor is shown
size, shape, and internal and external heterogene- with similar anatomical and functional informa-
ity, which made them very challenging to contour tion. The best result (DSC: 82%) was obtained
(in contrast to sphere-shaped tumors presented using λ = 70 and level = 16. In this case using larger
in some papers). In some cases the MR and the λ eliminates the over-segmentation due to the
PET provided very different information about the ambiguous MRI information. The segmentation
tumor, which introduced significant inter-observer of a medium sized tumor is displayed in the sec-
variability in manual contouring (DSC 55-90, AVD ond row. The best result (DSC: 83%) was achieved
2-39) based on the modality preference of the phy- using λ = 30 and level = 23. In this case the MRI
sician. The average time of manual contouring was shows more precise boundary information about
15 minutes per tumor. the tumor because some part of it is only visible in
The method has two main parameters which the anatomical image. The third row demonstrates
have significant effect on the final contour: the a difficult case that involves a large heterogeneous
weight factor (λ) that represents the weight of the tumor. In this case the best result (DSC: 74%) was
PET image and the level (level) that is used to gen- generated using λ = 60 and level = 86. Using larger
erate the contour. To demonstrate the effect of the weight for the PET image prevents large over-
parameters the result contour was created using segmentation which occur when the method relies
several (λ, level) combinations. The λ was set from on the very ambiguous MRI information. The
0% to 100% (with 10% stepping) and the value large value of the parameter level was due to the
of level ranged from 5 to 100. The method was large size of the tumor.
applied to each lesion using all parameter combi- According to the accuracy plots of Figure 3
nations and after each test run the result contour various combinations of λ and level can be used
was compared with the reference using standard to generate similar contour (see dark contour rep-
segmentation error metrics. resenting 95% of the maximum). Although, there
Figure 3 shows the segmentation accuracy is always a pair that maximizes the accuracy, very
belonging to the different parameter settings and similar result can be achieved using a different λ
the best result for three typical tumors. In the first with a corresponding level. Thanks to the low

212

VIPIMAGE15_Book.indb 212 9/8/2015 7:45:13 PM

 computation time the result contour can be auto- are set, the volume of interest (i.e. bounding box of
matically generated for several combinations of the the tumor) is defined, the segmentation is executed
two parameters so that the corresponding contour (for all parameter combinations automatically), the
shall be only displayed when the user modifies a result is displayed for the default parameter combi-
parameter. The user can find a good contour in an nation, after adjusting a parameter the contour is
intuitive way by adjusting the two parameters (the immediately updated.
balance between the two image information and the The prototype was applied to each tumor in
size of the contoured region) and having real time the test set. Table 1 shows the accuracy of the
visual feedback on the change of that parameter. best result for each tumor involved in the evalua-
The method guarantees that a larger level generates tion. The table shows the DICE Similarity Coef-
larger contour. This observation together with the ficient (DSC) and the Absolute Volume Difference
smoothness of the plots guarantees that the user (AVD) in addition to the reference lesion volume
can find the best contour in an intuitive way: the and the parameter combination used to generate
user can adjust the λ according to the importance the result. According to the error metrics DSC was
of the input images then adjusting the level param- 80.6% and AVD was 3% in average, which shows
eter to generate a more precise tumor contour. the results correlate well with the reference. Note
The proposed method was integrated into the that average AVD was significantly affected by one
Medical Imaging Interactive Toolkit (MITK) tumor (case 1). In most cases the best result was
framework (Nolden, Zelzer, Seitel, Wald, Mller, associated with lower PET image weight (average
Franz, Maleike, Fangerau, Baumhauer, Maier- λ was 35%). This fact was in line with the observa-
Hein, Maier-Hein, Meinzer, & Wolf 2013) as a tion that water T2w MRI images give more accu-
plug-in (Fig. 4). The implemented workflow con- rate information about the tumor boundaries than
sists of the following steps: the two input images the PET images.

Figure 4. The method integrated into the MITK framework: the generated red contour inside the defined ROI using
λ = 3 and level = 11.

Table 1. Segmentation results.

Cases 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

DSC (%) 74 71 80 83 81 75 83 86 79 78 82 81 81 87 88
λ (%) 60 30 20 30 30 70 30 20 20 50 70 20 30 30 20
Level 96 16 6 16 30 46 23 11 14 26 16 11 13 11 21
Volume (ml) 149.0 8.0 1.9 3.4 23.9 55.0 41.6 3.4 2.2 17.0 1.8 11.7 8.5 8.9 2.8
AVD (%) 25.5 3.5 1.4 0.1 1.4 1.5 1.3 1.3 1.0 1.0 1.7 1.9 1.0 2.9 0.3

213

VIPIMAGE15_Book.indb 213 9/8/2015 7:45:15 PM

 The evaluation demonstrated that the value of We are also grateful for Dr. Zsuzsanna Besenyi,
the parameter level primarily depends on the size of Dr. Zsuzsanna Fejes and Prof. Dr. László Pávics
the tumor. For large tumors higher value was used from the University of Szeged for creating the ref-
because the initial region was always the skeleton erence tumor contours.
of the high intensity PET region. For some results This work was supported by Analitic Health-
the DSC was below 80%. In case 1 and 6 the refer- care Quality User Information Program of the
ence segmentation involved regions which were not National Research, Development and Innovation
fully consistent with the image data: in some parts Fund, Hungarian Government, Grant VKSZ_12-
the contour involved high-uptake regions in the 1-2013-0012.
PET image, but in other parts such regions were
excluded. Furthermore, in case 2 the tumor was
weakly differentiable from its surrounding in the REFERENCES
MRI image. These issues can be justified with the
fact that inter-observer variability was high (men- AR., P. (2014). Recommendations for cross-sectional
tioned before) in the above-mentioned cases. It is imaging in cancer management. The Royal College of
important to note that volume difference was very Radiologists.
low in all cases (except for case 1). Bagci, U., J. Udupa, J. Yao, & D. Mollura (2012). Coseg-
mentation of functional and anatomical images. In
The presented method was implemented in C++ N. Ayache, H. Delingette, P. Golland, and K. Mori
using the Insight Toolkit (Johnson, McCormick, (Eds.), Medical Image Computing and Computer-
Ibáñez, & Consortium 2013) library. The evalua- Assisted Intervention MICCAI 2012, Volume 7512
tion was executed on a workstation with 3.4 GHz of Lecture Notes in Computer Science, pp. 459–467.
processor and 16 GB RAM. The running time of Springer Berlin Heidelberg.
generating a contour was below 0.4 second in all Bagci, U., J.K. Udupa, N. Mendhiratta, B. Foster, Z. Xu,
cases using any parameter configuration (the aver- J. Yao, X. Chen, & D.J. Mollura (2013). Joint segmen-
age running time was 0.13 second). The registration tation of anatomical and functional images: Appli-
(including automated method) took 1–5 minutes/ cations in quantification of lesions from pet, pet-ct,
mri-pet, and mri-pet-ct images. Medical Image Analy-
case. The segmentation (including the parameter sis 17(8), 929–945.
tuning) took about 1–3 minutes/tumor. Dice, L.R. (1945). Measures of the amount of eco-
logic association between species. Ecology 26(3), pp.
297–302.
4 CONCLUSIONS El Naqa, I., D. Yang, A. Apte, D. Khullar, S. Mutic, J.
Zheng, J.D. Bradley, P. Grigsby, & J.O. Deasy (2007).
According to the evaluation the presented segmen- Concurrent multimodality image segmentation by
tation approach allows the user to combine MRI active contours for radiotherapy treatment plan-
and PET information to create a tumor contour ninga). Medical Physics 34(12), 4738–4749.
Johnson, H.J., M. McCormick, L. Ibáñez, & T.I.S. Con-
that is consistent with both input images. Experi- sortium (2013). The ITK Software Guide (Third ed.).
ments with complex head and neck tumors showed Kitware, Inc. In press.
the parameters of the method can be adjusted in a Lee, T., R. Kashyap, & C. Chu (1994). Building skeleton
very intuitive way, which helps the user to find the models via 3-d medial surface axis thinning algo-
most accurate tumor contour in efficient way. The rithms. CVGIP: Graphical Models and Image Process-
presented approach is a straightforward extension ing 56(6), 462–478.
of the level-set segmentation to multi-modality Nolden, M., S. Zelzer, A. Seitel, D. Wald, M. Mller, A.
images. The speed function can be generalized by Franz, D. Maleike, M. Fangerau, M. Baumhauer, L.
incorporating edge-based, textural, or other fea- Maier-Hein, K. Maier-Hein, H.-P. Meinzer, & I. Wolf
(2013). The medical imaging interaction toolkit: chal-
tures of morphological images so that other image lenges and advances. International Journal of Compu-
combinations can be also used with this approach. ter Assisted Radiology and Surgery 8(4), 607–620.
In the future, the usability of the method shall be Perona, P. & J. Malik (1990). Scale-space and edge detec-
confirmed by more extensive clinical study involv- tion using anisotropic diffusion. Pattern Analysis and
ing other image combinations. Machine Intelligence, IEEE Transactions on 12(7),
629–639.
Sethian, J.A. (1999). Level set methods and fast marching
ACKNOWLEDGEMENTS methods: evolving interfaces in computational geom-
etry, fluid mechanics, computer vision, and materials
science. Cambridge University Press.
We would like to thank Prof. Dr. Patrick Veit-
Haibach and Gaspar Delso from the University
Hospital Zurich for providing the input images.

214

VIPIMAGE15_Book.indb 214 9/8/2015 7:45:16 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Automatic detection of bones based on the confidence map

for Rheumatoid Arthritis analysis

K. Radlak, N. Radlak & B. Smolka

Institute of Automatic Control, Silesian University of Technology, Gliwice, Poland

ABSTRACT: In this paper an application of confidence map for bones surface detection in ultrasound
images for Rheumatoid Arthritis is presented. The bone feature descriptor is based on confidence maps,
which are directly related with the likelihood of ultrasound transmission and allows for highly accurate local-
ization of the bone surface. The quality of the confidence map has been significantly improved by applying
the Trimmed Non-Local Means filtering. The results were evaluated on the publicly available MEDUSA
database that contains manually annotated markers of bones surface. This research is a preliminary step in
the development of fully automatic system for the assessment of degree of synovitis inflamation.

1 INTRODUCTION (Kulbacki et al. 2014) from the patients with chronic

arthritis during routine visits at the Rheumatology
Ultrasound (US) imaging is a powerful method of Department of Helse Førde in Norway. Exemplary
visualization of human body structures, which is ultrasound images with manually marked regions
often combined with the Power Doppler (PD) tech- of interest are presented in Figure 1.
nique used to measure the blood circulation in tis- In this work, we focus on applying an ultrasound
sues. Ultrasonography with PD signal has indicated confidence map (Karamalis et al. 2012) for bones
its efficiency, when diagnosing joint inflammation region detection. The confidence map is directly
in Rheumatoid Arthritis (RA) (ØStergaard and related with the likelihood of ultrasound transmis-
Szkudlarek 2005), but it involves some degree of sion. The idea of an ultrasound confidence map
subjectivity. The disease level is scored from 0 to 3 is based on the concept of Random Walk intro-
according to the OMERACT definitions (McQueen duced for image segmentation in (Grady 2006) and
et al. 2003), by human examiners based on their has been addressed in the literature with different
experience or standardized atlases (Terslev et al.
2003, Kulbacki et al. 2014). However, such an evalu-
ation system can cause some discrepancies in the
final diagnosis and treatment. Therefore, the most
challenging task is the elaboration of an automatic
method of assessment of joint inflammation level in
RA using ultrasound images acquired with PD.
The degree of synovitis can be assessed by esti-
mation of the size of the dark gray/black area
above the finger bones or the joint. Consequently,
the task of automatic determination of the level
of synovitis can be considered as a segmentation
problem of dark image regions in the vicinity
of bones and the joint. To determine the proper
orientation in an image, we propose to detect the
bones regions in an image as a reference for the
further analysis to determine the synovitis region.
Another possibility is to estimate the joint localiza-
tion (Wereszczyński et al. 2014), but this approach
is not considered in this work.
To evaluate the novel methodology for automated
assessment of joint synovitis activity, a large set of Figure 1. Ultrasound images on the left and manu-
ultrasound images with manually prepared annota- ally delineated synovitis region (green), bones (red), skin
tions was collected within the MEDUSA project (magenta) and joint (cyan) on the right.

215

VIPIMAGE15_Book.indb 215 9/8/2015 7:45:16 PM

 approaches and in different contexts (Karamalis In (Foroughi et al. 2007) the authors also calculate
et al. 2012). Methods that allow for estimation the probability map based on properties of the
of ultrasound attenuation can be used to detect bones in US images, but finally they use dynamic
shadowed and attenuated areas, thus, providing programing to determine where the bone starts and
information about unreliable image regions. The ends. In (Kowal et al. 2007) the authors propose to
estimation of ultrasound transmission allows algo- use thresholding, morphological operations and the
rithms to detect unreliable regions, like shadows, in detected pixels are merged in an iterative manner.
order to improve the robustness and accuracy of In the papers (Karamalis 2013, Quader et al. 2014),
their underlying methods. the authors propose to use the confidence maps to
The bone feature descriptor (Karamalis 2013) determine the bone surface position. The values of
based on confidence maps allows for highly accu- the confidence map can be interpreted as relative
rate localization of the bones surface in continu- signal strengths at different image locations and we
ous acquisition of US images of a human leg, what expect to observe the stronger reduction of signal
encouraged us to apply the method for the ultra- strength, when the signal passes through the bone.
sound images of fingers. However, the main differ- In our approach, we applied the confidence
ence is the fact, that in the US image of a finger map concept proposed in (Karamalis et al. 2012,
there are two bones, which need to be recognized. Karamalis 2013), but the original US images were
The results obtained when applying the pro- enhanced by our Trimmed Non-Local Means
posed algorithm were evaluated on the MEDUSA (TNLM) filter designed for the reduction of both
project database that contains manually annotated mixed Gaussian and impulsive and speckle noise
markers of bones regions. This research is a pre- (Radlak and Smolka 2013, Radlak and Smolka
liminary step in the development of fully automatic 2014). The comparison of the quality of the confi-
system that will support the diagnosis of RA. dence map with and without the proposed filtering
This paper is organized as follows. The next Sec- method is presented in Figure 2. Another difference
tion provides a brief description of the proposed lies in the fact, that the authors used the series of US
method, Section 3 presents the numerical results images along the femur bone and they can combine
of bone detection accuracy evaluated on automati- the results of detection coming from many images.
cally found and manually marked bone surfaces In the applied framework, the bones regions
utilizing the MEDUSA database and Section 4 are detected similarly to the work of Karmalis
concludes the paper. (Karamalis 2013) using bone confidence localizer,
but without the last step in which the author com-
bines the bone localizer responses from series of
2 BRIEF DESCRIPTION OF THE METHOD US images. In the MEDUSA project, we have only
single images of a finger of each patient.
The automatic bones detection is an important The first step in the proposed method is the cal-
problem in Computer-Aided Orthopedic Surgery, culation of the bones confidence localizer map L
in which precise information about the bone locali- defined as
zation is very useful in the pre-operative planning
i +1 j s i s j s
phase and navigation during surgery (Beitzel et al.
2012), but to the best of our knowledge, no stud- L∗ij ∑ ∑ F ( a, b ) − ∑ ∑ M (a, b ), (1)
a i −s b j −s a i −1 b j − s
ies on bones detection for Rheumatoid Arthritis are
available. The previously published papers are related
with femur bone detection. The bone in ultrasound
imaging appears as a bright and connected ridge
with a shadow region and is much more distinguish-
able from other parts of the US image. However, an
automatic detection is still a challenging task due
to the presence of speckle noise, image low con-
trast, various artifacts and shadowing in ultrasound
images and also due to the fact that most ultrasound
segmentation methods are usually limited to a spe-
cific target (Noble and Boukerroui 2006).
There are several works related with bone detec-
tion in surgery. Jain et al. (Jain and Taylor 2004)
calculate the probability that a pixel belongs to the
bone region using Bayesian Probabilistic Frame-
work based on the features which characterize the Figure 2. Ultrasound confidence map for noisy US
bones as structures with high intensity and high image (on the left) and using the Trimmed Non-Local-
gradient accompanied by multiple reflections. Means filtering (on the right).

216

VIPIMAGE15_Book.indb 216 9/8/2015 7:45:18 PM

 The confidence map M is calculated according to
the method described in (Karamalis et al. 2012).
Then, the obtained map L is thresholded using the
Otsu method (Otsu 1979) and the obtained blobs
are filtered using morphological operations and
labeled. To overcome the difficulties caused by the
discontinuities between the blobs, those for which
the distance between contours is less than the
predefined threshold are merged into one group.
Finally, the two largest groups of blobs are consid-
ered as possible bones. To detect the bones region,
we took the upper boundaries and the final bones
contours were interpolated pixel by pixel using a
piecewise cubic Hermite interpolating polynomial.
The exemplary results of each step of the algo-
rithm are presented in Figure 3.

3 BONE DETECTION ACCURACY

For the quantitative and objective evaluation the

following procedure is proposed. A subset of images
without Power Doppler coming from MEDUSA
project has been chosen. These images were taken
around the joint and there are two bones depicted
in each image. However, due to the fact that some
noise and the shadowing artifacts may occur, the
bones can be less distinguishable. Therefore, all
images were divided into three groups according
to the subjective assessment of visibility of the
bones in the image. These groups are: (a) low qual-
ity images with low contrast between bones region
(110 images), (b) images with single bone that is
visible (404 images), (c) images in which two bones
are visible (429 images). Exemplary images are pre-
sented in Figure 4.
For each image in this dataset the bones con-
tours were annotated by an expert to provide the
necessary ground-truth and allow for automatic
evaluation of the bones localization. Due to the
fact that it is our preliminary work and to give the
opportunity to compare the results, this database
is available for researchers at http://dip.aei.polsl.pl/
medusa upon request.
The contours were represented as sets of points
Figure 3. Exemplary results of the following steps of
A = { 1, , m } and B = { 1 , , m } for manual and
the algorithm performed on the images from MEDUSA automatic bones localization respectively. Direct
project. The rows from (a)–(f) show the temporary results one-to-one comparison of the points sets is not
of the most significant algorithm steps. In the last row possible, as they do not have a direct mapping and
the detected bones are marked as green and manually usually have different cardinality. Therefore, the
annotated contours as red. Hausdorff distance (Huttenlocher et al. 1993) is a
common measure for comparing two non-empty
the sets of feature points. The Hausdorff metric is
⎧L∗ if L∗ij > 0 (2) defined as follows:
Lij = ⎨ ij
⎩0 otherwise,
H (A ) a ( h(A ), h( A )), (3)
where M is the confidence map, F is the image h( A ) a i || a − b ||, (4)
a ∈A b ∈B
filtered using TNLM algorithm, s is the filtering
window size and i j are row and column indices. where || ⋅ || denotes the Euclidean distance.

217

VIPIMAGE15_Book.indb 217 9/8/2015 7:45:18 PM

 Figure 4. Test images.

For image processing applications and pattern

recognition tasks the Modified Hausdorff Distance
(MHD) introduced in (Dubuisson and Jain 1994)
is an appropriate choice to diminish the impact of
noise in the sets (Jesorsky et al. 2001). The MHD
is defined as:
1
hmod (A ) ∑
| A | a ∈A
i || a − b || .
b ∈B
(5)

In this work, we propose to present the detec-

tion accuracy according to the methodology used
in facial landmarks detection (Jesorsky et al. 2001,
Cristinacce and Cootes 2006). A mean error between
detected and manually annotated curves is not too
much informative, therefore the distribution func-
tion of the relative error is used providing a good Figure 5. Cumulative error plots of modified Hausdorff
overview of detection variability within the datasets. measure in pixels for our dataset.
Data proportion on the plot, shows for how many
images from the evaluation dataset the modified
Hausdorff distance is less than a given threshold.
The error of the bones detection using the modi-
fied Hausdorff distance in pixels is presented in
Figure 5. The curves show the accuracy separately
for each of the two bones (the green color for the
first and the red color for the second bone) in the
image for our three datasets with two bones visible
(DB1, DB2), single bones visible (SB1, SB2) and low
quality images (LB1, LB2). As can be observed, the
detection results are much better for the subset with
two visible bones. We also propose to normalize the
modified Hausdorff distance, by dividing the result
by the length of manually annotated bone contour.
These results are presented in Figure 6. The confi-
dence map were created for default parameters sug-
gested in the source code provided by the authors in
(Karamalis et al. 2012). Exemplary results of cor- Figure 6. Cumulative error plots of normalized modi-
rectly detected bones are shown in Figure 7. fied Hausdorff measure for our dataset.

218

VIPIMAGE15_Book.indb 218 9/8/2015 7:45:21 PM

 Figure 7. Exemplary results of correctly detected bones in ultrasound images and corresponding confidence and
bone localizer maps.

Figure 8. Some failure examples of the bone detection and corresponding confidence and bone localizer maps.

Our algorithm produces also some errors and 4 CONCLUSION

some examples are presented in Figure 8. These
cases are extremely difficult due to fact that in some In this paper an application of confidence map for
images the skin surface is represented by strong bones region detection in ultrasound images has
edges or the edges produced by bones are not so been presented. The bone feature descriptor is based
distinguishable. This algorithm can be improved on the confidence map that is directly related with the
by analyzing where the values of the confidence likelihood of ultrasound transmission and allows for
map are significantly reduced, but this issue will be accurate localization of the bones in US images. The
considered in the future work. quality of the confidence map has been significantly

219

VIPIMAGE15_Book.indb 219 9/8/2015 7:45:22 PM

 improved by the application of Trimmed Non-Local Karamalis, A., W. Wein, T. Klein, & N. Navab (2012).
Means filtering. The results were evaluated on the Ultrasound confidence maps using random walks.
MEDUSA project database that contains manually Medical Image Analysis 16(6), 1101–1112.
annotated markers of bones surface. Kowal, J., C. Amstutz, F. Langlotz, H. Talib, & M.G.
Ballester (2007). Automated bone contour detection in
This research is an initial step in the develop- ultrasound B-mode images for minimally invasive reg-
ment of fully automatic system that will support istration in computer-assisted surgeryan in vitro eval-
the assessment of the degree of synovitis inflama- uation. The International Journal of Medical Robotics
tion in Rheumatoid Arthritis. and Computer Assisted Surgery 3(4), 341–348.
Kulbacki, M., J. Segen, P. Habela, M. Janiak, W. Knieć,
M. Fojcik, P. Mielnik, & K. Wojciechowski (2014).
ACKNOWLEDGMENT Collaborative tool for annotation of synovitis and
assessment in ultrasound images. In Computer Vision
The research leading to these results has received and Graphics, Volume 8671 of Lecture Notes in Com-
funding from the Norwegian Financial Mecha- puter Science, pp. 364–373. Springer International
nism 2009–2014 under Project Contract No. Pol- Publishing.
Nor/204256/16/2013. Ultrasound images were McQueen, F., M. Lassere, J. Edmonds, P. Conaghan,
C. Peterfy, P. Bird, P. O’Connor, B. Ejbjerg, M. Klarlund,
created for the MEDUSA project at Section for
N. Stewart, P. Emery, R. Shnier, H. Genant, &
Rheumatology; Department for Neurology, Rheu- M. stergaard (2003). OMERACT rheumatoid arthritis
matology and Physical Medicine, Central Hospital, magnetic resonance imaging studies. Summary of
Førde, Norway. OMERACT 6 MR imaging module. The Journal of
Rheumatology 30(6), 1387–1392.
Noble, J. & D. Boukerroui (2006). Ultrasound image seg-
REFERENCES mentation: a survey. IEEE Transactions on Medical
Imaging 25(8), 987–1010.
Beitzel, J., S.-A. Ahmadi, A. Karamalis, W. Wein, & ØStergaard, M. & M. Szkudlarek (2005). Ultrasonogra-
N. Navab (2012). Ultrasound bone detection using phy: A valid method for assessing rheumatoid arthri-
patient-specific CT prior. In Annual International tis? Arthritis & Rheumatism 52(3), 681–686.
Conference of the IEEE Engineering in Medicine and Otsu, N. (1979). A threshold selection method from
Biology Society (EMBC), pp. 2664–2667. graylevel histogram. IEEE Trans. Syst. Man Cybern
Cristinacce, D. & T.F. Cootes (2006). Feature detection 9(1), 62–66.
and tracking with constrained local models. In Proc. Quader, N., A. Hodgson, & R. Abugharbieh (2014).
BMVC, pp. 95.1–95.10. Confidence weighted local phase features for robust
Dubuisson, M.-P. & A. Jain (1994). A modified Hausdorff bone surface segmentation in ultrasound. In Clinical
distance for object matching. In Proceedings of the Image-Based Procedures. Translational Research in
12th IAPR International Conference on Pattern Recog- Medical Imaging, Volume 8680 of Lecture Notes in
nition, Volume 1, pp. 566–568. Computer Science, pp. 76–83. Springer International
Foroughi, P., E. Boctor, M. Swartz, R. Taylor, & G. Fich- Publishing.
tinger (2007). P6d-2 ultrasound bone segmentation Radlak, K. & B. Smolka (2013). Trimmed non-local
using dynamic programming. In IEEE Ultrasonics means technique for mixed noise removal in color
Symposium, pp. 2523–2526. images. In IEEE International Symposium on Multi-
Grady, L. (2006). Random walks for image segmentation. media (ISM), pp. 405–406.
IEEE Transactions on Pattern Analysis and Machine Radlak, K. & B. Smolka (2014). Adaptive non-local
Intelligence 28(11), 1768–1783. means filtering for speckle noise reduction. In Compu-
Huttenlocher, D., G. Klanderman, & W. Rucklidge ter Vision and Graphics, Volume 8671 of Lecture Notes
(1993). Comparing images using the Hausdorff dis- in Computer Science, pp. 518–525. Springer Interna-
tance. IEEE Transactions on Pattern Analysis and tional Publishing.
Machine Intelligence 15(9), 850–863. Terslev, L., S. Torp-Pedersen, A. Savnik, P. von der Recke,
Jain, A.K. & R.H. Taylor (2004). Understanding bone E. Qvistgaard, B. Danneskiold-Samse, & H. Bliddal
responses in B-mode ultrasound images and automatic (2003). Doppler ultrasound and magnetic resonance
bone surface extraction using a Bayesian probabilistic imaging of synovial inflammation of the hand in
framework. In Proc. SPIE 5373, Medical Imaging 2004: rheumatoid arthritis: A comparative study. Arthritis &
Ultrasonic Imaging and Signal Processing, pp. 131–142. Rheumatism 48(9), 2434–2441.
Jesorsky, O., K.J. Kirchberg, & R. Frischholz (2001). Wereszczyński, K., J. Segen, M. Kulbacki, P. Mielnik,
Robust face detection using the Hausdorff distance. M. Fojcik, & K. Wojciechowski (2014). Identifying
In Proceedings of the Third International Conference a joint in medical ultrasound images using trained
on Audio- and Video-Based Biometric Person Authenti- classifiers. In L. Chmielewski, R. Kozera, B.-S. Shin, and
cation, AVBPA ’01, pp. 90–95. K. Wojciechowski (Eds.), Computer Vision and Graphics,
Karamalis, A. (2013). Ultrasound Confidence Maps and Volume 8671 of Lecture Notes in Computer Science, pp.
Applications in Medical Image Processing. Ph.D. thesis. 626–635. Springer International Publishing.

220

VIPIMAGE15_Book.indb 220 9/8/2015 7:45:26 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Improved computer recognition of Fuhrman grading system in analysis

of Clear-Cell Renal Carcinoma

M. Kruk & J. Kurek

Warsaw University of Life Sciences, Warsaw, Poland

S. Osowski
Warsaw University of Technology, Military University of Technology, Warsaw, Poland

R. Koktysz
Military Institute of Medicine, Warsaw, Poland

ABSTRACT: The paper presents the improved system of recognition of the Fuhrman grading in the
analysis of clear-cell renal carcinoma. The previous system was described in (Kruk et al. 2014). The pro-
posed improvement is based on the new segmentation method which gives better results. A redundant
wavelet transformation is used for denoising the image and enhancing the edges in multiple levels of
resolution. The image gradient is estimated using the wavelet transformation. The watershed transform
is then applied to the obtained gradient image, thanks to this the segmented regions that do not satisfy
specific criteria are removed.

1 INTRODUCTION nuclei are about 15 microns in diameter. They may

be visible at high magnification. At grade 3 the
A Clear-Cell Renal Carcinoma (CC-RCC) is a very nuclear contours are even more irregular in size
common renal disease in adults aged ≥ 50 years. and shape. The nuclear diameters can approach
The incidence of renal cell carcinoma has gradu- 20 microns, and the nucleoli are readily seen. The
ally increased each year over the past 30 years. The size of cells at grade 4 exceeds 20 micrometers, and
tumors show a solid, nesting, and tubular pattern the pleomorphic and hyperchromatic nuclei, and
of growth with tumor cells arranged in nests and prominent nucleoli in a minority of cells, are usu-
separated from each other by an extensive net- ally observed.
work of delicate sinusoidal vascular channels. The The main task of this paper is to improve an
most popular and widely used system for grading automatic computer system which was described in
CC-RCC is a nuclear grading system described in (Kruk et al. 2014) that would evaluate the Fuhrman
1982 by Fuhrman (Fuhrman et al. 1982). grade of renal cells with the accuracy comparable
The Fuhrman grading scheme describes 4 to that of human experts. To solve the problem,
nuclear features for assigning a specific grade: we apply the segmentation algorithm described in
nuclear size, nuclear shape, chromatin pattern, (Jung et al. 2002) which gives better results than
and size of the nucleoli. It defines a scale of 1–4, standard watershed algorithm. To perform clas-
where grade 1 carries the best prognosis and grade sification we use the Support Vector Machine
4 the worst. The grade is strictly correlated with (SVM) classifier based on the data representing
the stage of development of illness. Nuclear grade over 100 patients. To counteract the problem of
has been shown to be independent on the tumor biasing the score, we use 3 medical experts. Each
type as a prognostic factor, but its value in specific expert performed the manual estimation of the
histological subtypes of renal cell carcinoma is still Fuhrman grade of cells for the same set of images.
in question. In the learning phase of the system we have used
The grading schema of CC-RCC is based on only the cells for which class recognition was the
the microscopic image of the neoplasm cells with same in the assessment of all experts. In this way
Hematoxylin and Eosin (H&E) staining. Grade 1 we incorporate the average knowledge and expe-
tumors have the round, uniform nuclei with incon- rience of different experts in the learning process
spicuous or absent nucleoli. Nuclear contours at of the classifier, thereby reducing the interobserver
grade 2 are more irregular than in grade 1; the and intraobserver variabilities.

221

VIPIMAGE15_Book.indb 221 9/8/2015 7:45:26 PM

 2 MATERIALS in (Jung et al. 2002) and is based on the wavelet
method of image denoising and edge enhancement
All medical materials used in experiments were in multiple resolutions. In this approach, the gradi-
gathered from Warsaw Military Institute of Health ents of the denoised and enhanced image are esti-
Services. 70 microscopic images representing all mated using the wavelet transform, and then the
4 grades were prepared and then used to learn the watershed transform is applied to the obtained gra-
system. In the verification of the established sys- dient image. In these experiments we have applied
tem we used the additional 62 patients of different Daubechies wavelet function db4. and 4 levels of
Fuhrman grade of the illness and analyzed 94 ran- decomposition. Wavelet transformation improves
domly selected images representing these patients. the robustness of the watershed transformation.
The microscopic digital images of Hematoxylin The watershed implementation uses the immersion
and Eosin (H&E) stained kidney neoplasm cells of simulation approach. A post-processing stage is
the appropriate tumor area were collected. They finally applied to remove over-segmented regions
were acquired in a magnification of 400 × through with small areas, and to merge erroneously seg-
an Olympus BX-61 microscope and registered with mented regions, that are separated by weak bor-
an Olympus DP-72 camera in RGB format at a ders after denoising and enhancement.
resolution of 2070 × 1548. The example of such The example of an original image is shown in
images is presented in Figure 1. Figure 2a and the result of its segmentation is
depicted in Figure 2b.
3 METHODS

The proposed system contains five main steps:

(1) segmentation of nuclei, (2) generation of
numerical descriptors, (3) evaluation and selec-
tion of descriptors, (4) application of the classifier
determining the Fuhrman grade, (5) saving results
to the database.

3.1 The segmentation algorithm

The previous segmentation algorithm was based
on the gradient method of nuclei detection, instead
of the often-used automatic thresholding such
as Otsu method (Gonzales et al. 2008). However,
its main disadvantage is too high sensitivity to
the local extrema, hence sometimes the over seg-
mentation has been observed. The proposed
new approach to segmentation was described

Figure 1. Example of the microscopic image of kidney

tissue. Figure 2. Original image (a) and segmented nuclei (b).

222

VIPIMAGE15_Book.indb 222 9/8/2015 7:45:26 PM

 3.2 The set of numerical descriptors 3.3 Features selection
All segmented nuclei are described by the numerical Feature selection has been performed by applying
features used as the input matrix for classifier. The Fisher measure (Duda et al. 2003):
large set of features has been defined by authors
and then feature selection process was applied Sffc1c 2 = mean (c1) mean
ea (c2
c ) / (σ c1
c + σc2 )
to evaluate each feature and avoid these features
without any value in classification task. The evalu-
where mean(c1), mean(c2) are the mean of feature
ation method was described in next subsection.
f in the first and second class, and σc1,σc2 are the
The numerical features used in our experiment are
standard deviations of the feature in class 1 and 2,
divided into the following groups:
respectively.
• Texture Haralick features: The following 13 tex- Fisher measure arranges the features from the
ture features described in (Huang 2010) have most important (the highest value of measure) to
been generated. the least important. To find their proper popula-
1. Energy tion we have to set appropriate value of threshold.
2. Contrast On the basis of some introductory experiments we
3. Correlation have found the best results corresponding to the
4. Sum of variances value of threshold equal 1.
5. Inverse difference moment In the numerical experiments of classifica-
6. Entropy tion we have applied the strategy one-versus-all
7. Information measures of correlation (Scholkopf & Smola, 2002). In this approach we
8. Sum average have to build 4 classifiers, each responsible for rec-
9. Sum entropy ognition of one Fuhrman scale (Fuhrman scale
10. Sum variance associated with a class).
11. Difference variance Table 1 shows the sets of features selected for
12. Contrast each classification system arranged in this way.
13. Difference entropy The numbers in the table indicate the numbering
• Morphometric features: of features described earlier. As we can see the
14. Area population of selected features depends on the
15. Major axis length class and extends from 14 (class 1 versus all) to 30
16. Minor axis legth (class 2 versus all).
17. Eccentricity
18. Convex area 3.4 Classification system
19. Equivalent diameter: ed = 4 ∗ Area /pi
20. Solidity: solidity Area / Convex area As a classifier we have used Support Vector
21. Perimeter Machine of the Gaussian kernel (Scholkpf &
• Features describing colors: Smola, 2002). To deal with a problem of many
22. Relative sum of pixels of red components: classes, we applied the one versus all approach. In
sr = R /R + G B this approach we train as many local two-class rec-
23. Relative sum of pixels of green components: ognition classifiers as is the number of classes. On
sg = G / R + G B the basis of the results of all classifiers the final
24. Relative sum of pixels of blue components: winner for each observation is found by applying
sb = B / R + G B the majority competition (Scholkopf & Smola,
• Features from histogram: 2002). In learning the classifier systems we have
25. Mean of histogram used 3537 images of nuclei which were earlier
26. Standard deviation of histogram
27. Kurtosis of histogram
• Modified features: Four texture features Table 1. Selected features.
described in (Gianazza et al. 2010; Young IT
Class 1 2 5 10 11 12 14 15 16 18 19 21 24 25 26
et al. 1986) have been defined:
vs all
28. Heterogenity
Class 2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
29. Homogenity vs all 16 17 18 19 20 21 22 23 24 25 26 27
30. Clump 28 29 30
31. Condensation Class 3 1 2 3 4 5 6 8 9 10 11 12 13 14 15 16 17
vs all 18 19 21 22 23 24 25 26 27 28 29 30
In this way 31 numerical features have been Class 4 2 4 5 6 7 9 10 11 12 13 14 15 16 17 18
generated. vs all 19 20 21 22 23 24 25 26 27 28

223

VIPIMAGE15_Book.indb 223 9/8/2015 7:45:29 PM

 Table 2. The population of cells of different classes tak-
ing part in numerical experiments.

Class 1 2 3 4

Number of cells 1193 1167 808 369

separated by the segmentation algorithm and then

annotated by human experts. The population of all
classes is presented in Table 2.
The number of cells forming classes differ sig-
nificantly, from 1193 (class 1) to 369 (class 4). This
is due to the availability of cells representing these
classes in the available data base of the Military
Institute of Health Services.

Figure 3. The graphical results of classification

4 RESULTS system: a).

The results of experiments will be given in the numer-

ical and graphical forms. Figure 3 presents the anno- Table 3. Confusion matrix of the Fuhrman grade
tated cells corresponding to the image of Figure 2. recognition of the previous system.
The recognized cells have been described by num-
bers, where number 1 referes to Fuhrman grade 1, Grade 1 Grade 2 Grade 3 Grade 4
number 2 to grade 2 and number 3 to grade 3.
The numerical results correspond to the accu- Grade 1 0.967 0.025 0.008 0
racy, sensitivity and precision and compare the Grade 2 0.035 0.942 0.015 0.008
new values to the previous results presented in Grade 3 0.006 0.048 0.916 0.030
(Kruk et al. 2014). The confusion matrices of rec- Grade 4 0.010 0.048 0.099 0.843
ognition of all Fuhrman grades of cells in the pre-
vious and our improved system is shown in Table 3
and 4. It is easy to observe that most mistakes are Table 4. Confusion matrix of the Fuhrman grade rec-
made between two adjacent classes. This problem ognition of the improved system.
is very hard to omit because the border between
two classes is rather fuzzy. Grade 1 Grade 2 Grade 3 Grade 4
It is easy to observe that the improved system
has generated better results (higher values of Grade 1 0.978 0.015 0.007 0
diagonal terms and smaller values of off-diagonal Grade 2 0.030 0.951 0.011 0.008
elements). Similar improvement of sensitivity and Grade 3 0.005 0.041 0.922 0.032
precision is shown in Table 5. Grade 4 0.010 0.038 0.091 0.861
Slides of images representing all investigated
cases were reevaluated independently by 3 experts
blind to the previous scores. Each slide was
assigned with respect to Fuhrman grade accord- Table 5. Comparison of class sensitivity and precision
of the previous and improved versions of the Fuhrman
ing to the opinion of the particular expert. The grade recognition system.
results of expert assessments were compared to the
outcome of our computerized Automatic System Fuhrman
(AS). The comparison was done on the basis of grade Grade 1 Grade 2 Grade 3 Grade 4
Cohen’s kappa methodology (Carletta, 1996).
Kappa is widely accepted and used in the field of Previous 96.7% 94.2% 91.6% 84.3%
content analysis. It is interpretable, allows different sensitivity
results to be compared, and suggests a set of diag- Actual 97.8% 95.1% 92.2% 86.1%
nostics in cases where the reliability results are not sensitivity
good enough for the required purpose. The results Previous 94.99% 88.01% 88.25% 95.69%
precision
are presented in Table 6.
Actual 95.4% 89.9% 90.01% 96.21%
In all cases the Kappa values were larger for our precision
improved system than for the inter experts results.

224

VIPIMAGE15_Book.indb 224 9/8/2015 7:45:29 PM

 Table 6. Cohen Kappa values for the results of recogni- Huang P.W., Lai Y.H. (2010). Effective segmentation
tion of 4 Fuhrman grades. classification for HCC biopsy images. J Pattern Rec-
ognition 43:1550–156.
AS Expert 1 Expert 2 Expert 3 Duda R.O., Hart P.E., Stork P. (2003). Pattern classifica-
tion and scene analysis, Wiley, New York.
AS – 0.4341 0.5242 0.4730 Fuhrman S.A., Lasky L.C., Limas C. (1982). Prognostic
Expert 1 0.4341 – 0.3660 0.2499 significance of morphologic parameters in renal cell
Expert 2 0.5242 0.3660 – 0.2476 carcinoma. Am. J. Surg. Pathol. 6:655–663.
Expert 3 0.4730 0.2499 0.2476 – Gianazza E., Chinello C., Mainini V., Cazzaniga M.,
Squeo V., Albo G., Signorini S., Di Pierro S.S.,
Ferrero S., Nicolardi S., van der Burgt Y.E.,
Deelder A.M., Magni F. (2012). Alternations of the
5 CONCLUSION serum peptidome in renal cell carcinoma discrimi-
nating Benin malignant Sidney tumors, J Proteom
76:125–140.
The paper shows that improvement of segmenta- Gonzalez R., Woods R., 2008: Digital Image Processing.
tion algorithm significant has influence on results New Jersey, Prentice Hall.
of classification system. Huang, M., & Haralick, R.M. (2010). Recognizing Pat-
The important advantage of the improved sys- terns in Texts. Pattern Recognition and Machine
tem is its repeatability of scores, which is in great Vision, 6, 19.
contrast with the human expert results, signifi- Jung, C.R., & Scharcanski, J. (2002). Robust watershed
cantly dependent on the particular choice of the segmentation using the wavelet transform. In Compu-
expert and his/her mental and physical condition ter Graphics and Image Processing. Proceedings. XV
Brazilian Symposium on (pp. 131–137). IEEE.
in the time of assessment. Moreover, the our sys- Kruk M., Osowski S., Markiewicz T., Słodkowska J.,
tem allows a reduction of time required for the Koktysz R., Kozłowski W., Świderski B., (2014). Com-
image analysis in comparison to the human expert. puter approach to recognition of Fuhrman grade of
Thanks to this the acceleration of the research in cells in clear-cell renal cell carcinoma. Analytical and
this area is possible. Quantitative Cytology and Histolgy 36(3):147–160.
The main task for the future is to find the Schölkopf, B., & Smola, A.J. (2002). Learning with
method to recognize the most important area in kernels: Support vector machines, regularization,
whole specimen. Now the main region of inter- optimization, and beyond. MIT press.
est must be indicated by human expert to further Young I.T., Verbeek P.W., Mayal B.H. (1986). Charac-
terization of chromatin distribution in cell nuclei.
processing. Cytometry 7:467–474.

REFERENCES

Carletta J. (1996). Assessing agreement on classification

tasks: The kappa statistic. Computational Linguistics
22:249–254.

225

VIPIMAGE15_Book.indb 225 9/8/2015 7:45:30 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

A proof of concept of an augmented reality system for Nuss surgery

A. Ferreira
ICVS/3B’s—PT Government Associate Laboratory, Braga/Guimarães, Portugal

P. Morais
ICVS/3B’s—PT Government Associate Laboratory, Braga/Guimarães, Portugal
INEGI, Faculdade de Engenharia, Universidade do Porto, Porto, Portugal

S. Queirós
ICVS/3B’s—PT Government Associate Laboratory, Braga/Guimarães, Portugal
Algoritmi Center, School of Engineering, University of Minho, Guimarães, Portugal

F. Veloso
ICVS/3B’s—PT Government Associate Laboratory, Braga/Guimarães, Portugal

N.F. Rodrigues
ICVS/3B’s—PT Government Associate Laboratory, Braga/Guimarães, Portugal
DIGARC, Polytechnic Institute of Cávado and Ave, Barcelos, Portugal

J. Correira-Pinto
ICVS/3B’s—PT Government Associate Laboratory, Braga/Guimarães, Portugal

J.L. Vilaça
ICVS/3B’s—PT Government Associate Laboratory, Braga/Guimarães, Portugal
DIGARC, Polytechnic Institute of Cãvado and Ave, Barcelos, Portugal

ABSTRACT: Pectus Excavatum (PE) is the most common congenital chest wall deformity, affecting
1 in 400 live births. This deformity is commonly corrected using the minimally invasive Nuss procedure,
where a bar is positioned under the sternum. Although recent procedure advances based on patient-
specific prosthesis were proposed, correct bar placement is still challenging.
In this work, we propose a novel augmented reality system to guide the surgeon during PE bar place-
ment. This system combines a 3D sensor with a projector to superimpose the thoracic ribs cage on the
chest wall of the patient, thus indicating the optimal insertion and bar placement points.
This system was validated in three different scenarios: 1) simulated chest surface models; 2) 3D printed
phantom; and 3) 3D commercial thoracic phantom. An error of 3.93 ± 3.44 mm, and 3.08 ± 1.57 mm were
obtained in the first and second experiments, respectively. In the final experiment, visual assessment of
the result proved that a high similarity was obtained between the projected model and the real ribs cage
position.
Overall, the proposed system showed high feasibility with low error, proving that 3D projection of the
ribs on the patient’s chest wall may facilitate PE bar insertion and ultimately provide useful information
to guide Nuss procedure.

Keywords: augmented reality; pectus excavatum; Nuss surgery

1 INTRODUCTION births (Koumbourlis 2009). The most common

treatment of this condition is the Minimally Inva-
Pectus Excavatum (PE) is characterized by a sive Surgery (MIS) introduced in 1998 by Donal
depression of the anterior chest wall. It is the Nuss, which consists in inducing remodelling of
most common congenital chest wall deformity, the chest wall to a normal shape by inserting a bar
accounting for 90% of all congenital chest wall behind the sternum. This procedure has become
abnormalities and affecting around 1 in 400 live well accepted by pediatric and thoracic surgeons

227

VIPIMAGE15_Book.indb 227 9/8/2015 7:45:31 PM

 due to its technical simplicity, reproducibility and version 2 sensor and a projector Epson EH-TW550.
long-term functional and cosmetic results. Despite The Kinect is a physical device with depth sensing
allowing the successful PE correction in 93%–96% technology, a built-in color camera and infrared
of the cases, the Nuss procedure has an overall (IR) emitter, enabling 3D real-time scanning of the
complication rate of 33%, namely associated to patient’s chest wall. To guarantee that the patient’s
persistent pain after operation, pneumothorax, ribs cage is projected in the right place, both Kinect
seromas, infection and bar migration (Fallon et al. and projector are calibrated using a gray code
2013). Such complications are mainly related to the projection. Then, the 3D model obtained from
conventional bar modelling process, which consists pre-operative Computed Tomography (CT) images
in the intra-operative manual modelling of the bar, segmentation is registered with the patient’s chest
and the lack of surgery planning. wall and ultimately projected over him. Figure 1
Vilaça et al. (2013) proposed a system to auto- presents the flowchart of the current system.
matically bend a personalized bar for PE correc-
tion based on the patient’s anatomy and tools for
2.1 Kinect/projector calibration
virtual simulations of the postsurgical outcome
(Moreira et al. 2012, Vilaça et al. 2011). Similarly, Since two independent modules (Kinect sensor and
Zhao et al. (2014) proposed a framework for chest projector) are used to create an accurate projection
modelling and personalized surgical planning for of the chest wall, a calibration step should be per-
PE. Although the abovementioned frameworks formed to define a common world between them.
present several advantages for the patient through The proposed system uses gray code patterns to
an optimal deformity correction with superior obtain the dense point correspondences between
final cosmetic result, the bar placement task is still the projector and Kinect images (Fig. 1.a). Each
dependent of the surgeon, which could be difficult pixel in the projected patterns encodes the corre-
due to: 1) loss of visual drive; 2) reduced depth sponding projector image coordinates. The pat-
perception; 3) limited angle of view; and 4) limited terns are observed by the Kinect and decoded to
touch sensation (Pessaux et al. 2014). obtain the correspondences (Schmalz et al. 2011).
Augmented Reality (AR) is a computer-based These correspondences are used to transform
technology that can display virtual information the patient’s CT information from the Kinect’s
in the real environments, such that virtual objects coordinate system to the projector’s coordinate
appear to coexist in the same space as the real system.
world. This technology has been used in medi- Note that gray code patterns recognition is an
cal procedures such as neurosurgery, laparoscopy, active calibration method that makes usage of
otolaryngology, orthopaedics and maxillofacial self-identifying patterns that can be shown by the
surgery. Since these systems can create virtual trans- projector. As such, the tedious manual identifica-
parency of the patient anatomy and consequently tion of marks required by a passive calibration
project it at the body surface, an increase of the sur- method become unnecessary (Schmalz et al. 2011).
geon vision is obtained, which facilitates the surgi-
cal procedure and, consequently, reduces the rate of
2.2 Virtual information registration
procedural complications (Okamoto et al. 2015).
In this work, we propose a system to solve the To perform a markerless registration of the virtual
aforementioned problems in minimally invasive PE information with the patient, the proposed system
procedures based on an augmented reality system uses the Kinect sensor to acquire the 3D model of
to superimpose the patient-specific ribs cage on the the patient’s chest wall (Khoshelham and Elberink
patient’s chest wall. Such system will increase the 2012). Then, the chest wall surface, segmented
safety and accuracy of the Nuss surgery by pro- from the pre-operative CT images using a simple
viding important visual information directly on the threshold, is registered with the Kinect’s informa-
patient and, therefore, allowing the identification tion (Fig. 1.b, c, d). This registration is based on the
of the correct sites for insertion of the bar, while Iterative Closest Point algorithm (ICP). However,
following the steps established during the pre- a first alignment between the two point sets is per-
operative planning. In this way, a reduction of the formed using the Sample Consensus Initial Align-
possible procedure complications, due to the incor- ment (SAC-IA) algorithm proposed by Rusu et al.
rect bar placement, is expected. (Rusu et al. 2009).
The SAC-IA performs fast searches in an
exhaustive Fast Point Feature Histograms (FPFH)
2 SYSTEM OVERVIEW correspondence space to find a good alignment
solution, which can be further refined using the
The projector-based AR system for Nuss surgery ICP algorithm (Rusu et al. 2009). The FPFH are
assistance is composed by a Microsoft Kinect the optimization of the Point Feature Histograms,

228

VIPIMAGE15_Book.indb 228 9/8/2015 7:45:31 PM

 Figure 1. System overview: (a) Microsoft Kinect Sensor/Projector calibration with gray code patterns; (b) Segmen-
tation of the chest wall, rib bones, cartilages and sternum from pre-operative Computed Tomography (CT) data; (c)
Patient’s chest wall 3D scanning obtained with Kinect; (d) Alignment of the 3D CT models with the Kinect model; (e)
Projection of the rib bones, cartilages and sternum on top of the patient’s chest wall.

a robust multi-dimensional features that describe Consequently, the ICP algorithm is used to refine
the local geometry around a point for 3D point the alignment.
clouds datasets. After estimation of the FPFH for The ICP is a popular algorithm employed to
the Kinect and CT point clouds, the SAC-IA pro- register two sets of point clouds X, P, by itera-
ceed as follow: tively matching points in one set to the clos-
est points in another set, consequently refining
i. Select sample points from one 3D model while
the transformation between the two sets in
making sure that their pairwise distances are
order to minimize the distance between both
greater than a user-defined minimum distance;
point clouds. Then, given two corresponding
ii. For each of the sample points, find a list of
points whose histograms are similar to the
point sets: {x , ,xxn } P = { p pn }, the
ICP algorithm computes the translation t and
sample points’ histogram. From these, select
rotation R that minimizes the sum of the squared
one randomly which will be considered as sam-
error:
ple points’ correspondence;
iii. Compute the rigid transformation defined by N
the sample points and their correspondences, 1 p
∑ xi
2
and compute an error metric for the point E (R t ) = R i
Rp t , (1)
N p i =1
cloud that computes the quality of the trans-
formation (Rusu et al. 2009).
where xi and pi are the corresponding points (Besl
These three steps are repeated, and the transfor- and McKay 1992). To compute the transformation
mation that presents the best error metric is stored between X and P, the algorithm iterate the follow-
and used to roughly align the two point clouds. ing steps:

229

VIPIMAGE15_Book.indb 229 9/8/2015 7:45:31 PM

 i. Application of the optimal transformation Then, the proposed 3D registration method was
estimated in the previous iteration; applied between the two model sets, where the set
ii. Find the closest point in the second set, for with white Gaussian noise was used as target while
each point in the first transformed point set; the second set was used as the registration model.
iii. Update the point matches by discarding Figure 2 shows the mean distance error between
outliers; the registered models and the original CT model at
iv. Compute the transformation using the updated different SNRs.
point matches, until convergence of the esti-
mated transformation (Zhang 2014).
3.2 Phantom test
Finally, after alignment between the CT infor-
In order to validate the calibration between projec-
mation with the Kinect 3D data, the other CT
tor and Kinect, a 3D printed phantom (Fig. 3.a)
segmentation structures, namely bones, cartilages
was used. The current phantom consists on a
and sternum, can be transformed to the projector’s
set of cylinders with different radii and heights.
coordinates and overlaid in the patient’s chest wall
Note that this phantom was initially created in
(Fig. 1.e).
a Computer-Aided Design (CAD) tool and then
printed through a 3D-printing process.
3 EXPERIMENTS AND RESULTS Since the Kinect is fixed, only the surface directly
in sight of it are acquired (see Fig. 3.b). As such,
To validate the proposed system, three different one needs to perform the registration of the phan-
scenarios are used. First, a set of translated and tom’s complete virtual model with the acquired
rotated 3D chest wall models were registered with phantom’s partial 3D model. After registration,
a set of chest walls. Second, a 3D printed object is the top surfaces of each cylinder were projected
used as a phantom model to validate the proposed on the phantom (Fig. 3.b), allowing to measure the
system. Finally, preliminary results with a com- displacement of the centers from the phantom’s
mercial thoracic phantom are also presented. cylinders and the reprojected surfaces.
As a final remark, one should note that dif-
ferent projection distances were used (1000 and
3.1 3D registration
1500 mm) to prove that an accurate result can be
A 3D ideal model of a chest wall was used to obtained independently of the focal distance.
assess the accuracy of the proposed 3D registra- Regarding the results, a mean reprojection error
tion method. As such, it was created a set of novel of 3.08 ± 1.57 and 6.87 ± 1.35 mm at 100 and
models through the addition of white Gaussian 150 cm distance, respectively, were obtained.
noise on the ideal model. Note that the white noise
varied from 25 to 35 dB of Signal-to-Noise Ratio
3.3 Thoracic model test
(SNR). With these distorted models, we intend to
mimic the 3D models obtained with the Kinect, The proposed system was also tested with an
which is susceptible to have some noise due to the ABDFAN thoracic model (commercial phantom).
technology limitations and lighting conditions. Initially, a 3D CT acquisition of the phantom
Regarding the ideal model, a set of translations model was performed. Using this dataset, a 3D
and rotations (30° to 270°) were performed to sim- ribs surface and a 3D chest surface were extracted
ulate misalignment between the ideal model and by applying a simple threshold on the CT image.
simulated model. It is important to note that the Similarly, the proposed AR system was applied
translation performed in the second set of 3D mod- and a 3D point set of the chest surface from the
els shifted them in a manner that no overlap occurred commercial phantom was acquired by the Kinect
with the original model on the original position. sensor. This point set is aligned with the segmented

Figure 2. Mean distance error of the 3D registration using targets with different SNR of white Gaussian noise: (a)
25 dB, 30 dB and 35 dB.

230

VIPIMAGE15_Book.indb 230 9/8/2015 7:45:33 PM

 equal to or higher than 30 dB, the registration
method presents good results with a mean error
distance around 5 mm, being in accordance with
previous studies available in literature (Almhdie
et al. 2007). Since the 3D models obtained with
Kinect does not present a SNR with such low
magnitude, the developed registration method is
expected to present a good performance for the
intended application.
Figure 3. 3D printed phantom for system accuracy test: Regarding the second experiment, the results
(a) 3D virtual model, (b) test result at 1500 mm distant
from the phantom’s base.
obtained with the 3D printed phantom indicates
that the system have a low reprojection error, infe-
rior to 7 mm, in the different positions. This fact is
important to ensure that the system will perform as
projected, even in different surgical setups. As the
common distance between ribs is approximately
15 mm (Yilmaz et al. 2014), the reported low error
magnitude also shows that the present system can
represent the intercostal space with the necessary
accuracy. Thus, the results corroborate that the
described AR system can project useful informa-
tion to assist surgeons placing the pectus bar. It is
also important to refer that the system has better
performances when is placed closer to the projec-
tion target. This is explained by the operation of the
Kinect, which can acquire a 3D surface with more
details and vertices when is closer to the target.
The commercial thoracic model test allowed to
evaluate the performance of the proposed system
in a setup closer to the real intended application.
By observing Figure 4, it is possible to support the
utility of the system during the Nuss surgery, since
the projection of the ribs cage on top of the model
provides valuable visual information of the ribs
location to the surgeons. The access to this infor-
Figure 4. Projection of the rib cage on top of a Kyoto mation during Nuss surgery will assist the correct
Kagaku ABDFAN thoracic model. placement of the PE bar, which presents poten-
tial to overcome some complications like persist-
ent pain after operation, pneumothorax, seromas,
3D chest wall surface using the ICP strategy, defin- infection and bar migration.
ing an optimal transformation between them. As
a final step, this optimal transformation is used to
project the CT 3D ribs surface onto the phantom 5 CONCLUSION
model. Note that a previous calibration between the
Kinect and projector was performed, as described The proposed AR system promises to be useful
in Section 2.1. on surgical procedures like the Nuss surgery, pro-
Figure 4 shows the projection result, where the viding valuable information throughout PE bar
3D ribs model is superimposed on the commercial placement. In this work, the proposed AR system
phantom. was validated in three different experiments, prov-
ing the feasibility and accuracy of the presented
system.
4 DISCUSSION In the near future, the described augmented
reality system will be tested in clinical practice.
Figure 2 presents the registration error with differ- Given the experimental results obtained so far, it is
ent SNR of white noise. As expected, lowest SNR expected to have a good performance, helping the
increase the registration errors. Moreover, a mini- surgeons to identify the location of the ribs and
mum SNR of 30 dB should be used to obtain an perform the surgical procedure as pre-operatively
acceptable result. Once the target presents a SNR planned.

231

VIPIMAGE15_Book.indb 231 9/8/2015 7:45:33 PM

 REFERENCES Rusu, R.B., N. Blodow, & M. Beetz (2009). Fast Point
Feature Histograms (FPFH) for 3D registration. In
Almhdie, A., C. Léger, M. Deriche, & R. Lédée (2007). IEEE International Conference on Robotics and Auto-
3d registration using a new implementation of the icp mation, 2009. ICRA’09., pp. 3212–3217. IEEE.
algorithm based on a comprehensive lookup matrix: Schmalz, C., F. Forster, & E. Angelopoulou (2011).
Application to medical imaging. Pattern Recognition Camera calibration: active versus passive targets.
Letters 28(12), 1523–1533. Optical Engineering 50(11), 113601–113601–10.
Besl, P.J. & N.D. McKay (1992). A method for regis- Vilaça, J.L., A.H. Moreira, L. Pedro, N. Rodrigues, J.C.
tration of 3-d shapes. IEEE Transactions on Pattern Fonseca, A. Pinho, & J. Correia-Pinto (2011). Virtual
Analysis and Machine Intelligence 14(2), 239–256. simulation of the postsurgical cosmetic outcome in
Fallon, S.C., B.J. Slater, J.G. Nuchtern, D.L. Cass, E.S. patients with pectus excavatum. In SPIE Medical
Kim, M.E. Lopez, & M.V. Mazziotti (2013). Compli- Imaging, pp. 79642 L–79642 L. International Society
cations related to the nuss procedure: Minimizing risk for Optics and Photonics.
with operative technique. Journal of Pediatric Surgery Vilaça, J.L., P.L. Rodrigues, T.R. Soares, J.C. Fonseca,
48(5), 1044–1048. A.C. Pinho, T. Henriques-Coelho, & J. Correia-Pinto
Khoshelham, K. & S.O. Elberink (2012). Accuracy and (2013). Automatic prebent customized prosthesis
resolution of kinect depth data for indoor mapping for pectus excavatum minimally invasive surgery
applications. Sensors 12(2), 1437–1454. correction. Surg Innov 21(3), 290–296.
Koumbourlis, A.C. (2009). Pectus excavatum: Yilmaz, M., C.A. Dickman, & N.R. Crawford (2014).
Pathophysiology and clinical characteristics. Paediatric Morphometric analysis of the thoracic cage in adults:
Respiratory Reviews 10(1), 3–6. Anatomical considerations in relation to neurosurgi-
Moreira, A.H., P.L. Rodrigues, J. Fonseca, A. Pinho, N. cal thoracoscopic procedures. Journal of Neurological
F. Rodrigues, J. Correia-Pinto, & J.L. Vilac¸a (2012). Sciences (Turkish) 31(1), 107–114.
Pectus excavatum postsurgical outcome based on Zhang, Z. (2014). Iterative Closest Point (ICP). Computer
preoperative soft body dynamics simulation. In SPIE Vision: A Reference Guide, 433–434.
Medical Imaging, pp. 83160 K–83160 K. International Zhao, Q., N. Safdar, C. Duan, A. Sandler, & M. Linguraru
Society for Optics and Photonics. (2014). Chest Modeling and Personalized Surgical
Okamoto, T., S. Onda, K. Yanaga, N. Suzuki, & Planning for Pectus Excavatum, Volume 8673 of Lec-
A. Hattori (2015). Clinical application of navigation ture Notes in Computer Science, Book section 64, pp.
surgery using augmented reality in the abdominal 512–519. Springer International Publishing.
field. Surgery Today 45(4), 397–406.
Pessaux, P., M. Diana, L. Soler, T. Piardi, D. Mutter, &
J. Marescaux (2014). Robotic duodenopancreatec-
tomy assisted with augmented reality and real-time
fluorescence guidance. Surgical Endoscopy 28(8),
2493–2498.

232

VIPIMAGE15_Book.indb 232 9/8/2015 7:45:35 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Usage of mobile devices in a bone fracture reduction process

J.R. Jiménez, F. Paulano, J.M. Noguera & J.J. Jiménez

Graphics and Geomatics Group of Jaen, Universidad de Jaén, Jaén, Spain

ABSTRACT: The market share of mobile devices has spectacularly increased in a few years. The novel
characteristics of this kind of devices: intuitive interfaces and everywhere/everytime access, has promoted
its widespread use in many different scenarios. In addition, medical trainees and specialist are familiar-
ized with them. On the other hand, wearable devices are a very promising new line of products even for
medical applications. In recent years, many computational techniques have been proposed to help the
specialist in the fracture reduction process. However, due to its intrinsic complexity and its high techno-
logical requirements, this research line is still open. The increasing computational power and the birth of
new technology favor the chances of automating more tasks reducing risks and surgery time exposition.
Nevertheless, the use of mobile devices is not widespread in this specific medical context. In this paper,
we briefly introduce the possibilities of mobile and wearable devices in a fracture reduction process from
a global perspective. Different areas of research have to be taken into account to get a practical overview
of the problem and the connections among them. Specifically the focus is placed on the visualization of
medical data, the interaction, collaborative aspects and augmented reality possibilities.

Keywords: mobile devices; bone fracture reduction; 3D medical images; visualization; computer sup-
ported cooperative work; user interaction; augmented reality

1 INTRODUCTION Furthermore, a related new promising line of

products is consolidating its technology and fea-
Their growing capabilities, their everywhere/every- tures: wearable devices. This new kind of gadget
time access and their friendly user interface have keeps one of the main features of handheld devices
favored the proliferation of mobile devices in the its everywhere/everytime access.
last years. In fact, market sales data confirm that Yet, to the best of our knowledge we have not
this new line of products has surpassed the sales found proposals that cover totally or partially a
of traditional desktop PCs and laptops. This is a bone fracture reduction process in a mobile device.
long-term tendency based on current users behav- This paper reviews the possibilities and challenges
ior and preferences (Gartner 2013). of using mobile devices and the like in the proc-
Mobile devices have become ubiquitous in esses and activities related with a bone fracture
medical-care professional settings (Kho, Henderson, reduction. We place together the bone fracture
Dressler, & Kripalani 2006, Whipple, Allgood, & reduction issues and those novel characteristics
Larue 2012, Berkowitz, Kung, Eisenberg, Donohoe, of this kind of devices that somehow can be of
Tsai, & Slanetz 2014) and therefore medical trainees help. This pursuing leads us to different areas of
and physicians have already been using their mobile research that we briefly introduce. Consequently,
devices for a decent amount of time. For this reason the set of topics associated with the scope of
they are already familiarized with the natural multi- this study conveys us to devise human-computer
touch interfaces and interaction metaphors that interaction, augmented reality, visualization and
these platforms provide (Botha, Preim, Kaufman, Computer Supported Cooperative Work (CSCW)
Takahashi, & Ynnerman 2012), and it seems natural challenges, see Figure 1.
to use them for as many tasks as possible (Butson, The paper is structured as follows: Section 2
Tamm, Jain, Fogal, & Kruger 2013). briefly introduces the common stages and activi-
In addition, this new trend has not passed unno- ties related with a bone fracture reduction. Based
ticed to the research community that is increasingly on these stages, next sections emphasize the tech-
taking advantage of the mobile devices GPU- niques based on mobile devices that can be applied
enable capacities in different fields such as 3D to some of them, their requirements and challenges.
visualization techniques, Augmented Reality (AR), Specifically, Section 3 reviews the visualization of
and interaction methods (Noguera & Torres 2013, 3D medical images, Section 4 details the possibili-
Noguera & Jimenez 2015). ties of utilizing augmented reality in a bone fracture

233

VIPIMAGE15_Book.indb 233 9/8/2015 7:45:35 PM

 Figure 1. Research areas of interest related with the
topics of this study: bone fracture reduction activities
and mobile devices features.

reduction process and Section 5 highlights the cor-

responding human computer interaction require-
ments when part of the activities are developed Figure 2. Activities and resources classified according
with a mobile device and Section 6 focuses on the to their position with respect to surgery.
conditions and difficulties to design adequate col-
laborative environments. Finally, Section 7 discusses
among the different issues and concludes the paper. scan make these tasks harder (Paulano, Jiménez, &
Pulido 2013). The resulting 3D model can be a cloud
of points, a surface or a soup of triangles depending
2 THE BONE FRACTURE on the problem to be solved. Secondly, the next step
REDUCTION PROCESS consists of virtually reducing the fracture. In this sec-
ond task, bone fragments are placed in their correct
The main stages of a bone fragment reduction position. Some methods have been proposed in order
process are: diagnosis, planning and intervention. to deal with this process (Willis, Anderson, Thomas,
The activities related with this process can also be Brown, & Marsh 2007) (Chowdhury, Bhandarkar,
classified according to their position with respect Robinson, & Yu 2009). In order to avoid that frag-
to the surgery as: pre-surgery, intra-surgery, post- ment loses their correct position, the fracture need
surgery (Jaramaz, Hafez, & DiGioia 2006), see to be stabilized by including fixation devices. Sev-
Figure 2. Next, we briefly describe the most com- eral works propose to perform this process in a vir-
mon activities and tasks related with a bone frag- tual way (Hu, Li, Qiao, Liu, Ji, & Ye 2011). Finally
ment reduction. the reduced fracture can be analyzed to check the
Computer Tomography (CT) is the most com- goodness of the stabilization. For this purpose, finite
mon method to acquire 3D images in order to element analysis is used most of the times (Blessy,
distinguish bone tissue. Collective or individually, Anburajan, & Ramkumar 2011). An analysis can be
the diagnosis of a bone fracture is usually accom- performed to determine the goodness of a virtual
plished by directly using this kind of images. reduction or as a post-surgery activity.
One of the most important activities in order Furthermore, a continuous training program for
to increase the chances of a successful surgery is medical specialists and practitioners improves the
planning. An adequate planning can reduce surgery common knowledge about new methodologies and
time and minimize risks, favoring a faster recovery technology at hand. A decent variety of cases are
(Sikorski & Chauhan 2003, Sugano 2003). Firstly, required in order to provide a satisfactory training
the planning of a bone fracture reduction requires program. This necessity is a relevant issue especially
generating 3D models. This task is achieved by seg- regarding bone fractures. As a result, a virtual gen-
menting the bone tissue and label all the fragments erator of bone fractures is an important challenge.
(Paulano, Jiménez, & Pulido 2014). The special Regarding intra-surgery activities it is not a
features that present the fractured bone in a CT novelty the use of systems related with Computer

234

VIPIMAGE15_Book.indb 234 9/8/2015 7:45:35 PM

 Assisted Surgery (CAS) or tele-operating. The main
goal are to shorten the surgery time, to reduce the
radiation exposure of the patient, to increment the
precision and to increase the knowledge of what
the surgeon is doing. Real time navigation systems
for intramedullary nails or robot-guided surgery
are good examples of these systems.

3 VISUALIZATION OF 3D MEDICAL
IMAGES

As it has been previously commented 3D CT Figure 4. An example of a 2D visualization.

images are the most recommended to distinguish
bone tissue. The CT scan technique allows acquir-
ing a set of 2D images representing a 3D model of
a part of a body. The value stored in a cell of one
of these image represents an intensity. A specific
range of these intensities represents bone tissue.
Consequently, by using a transfer function it is
possible to remove some components (tissue) and
obtain an image where almost nothing but bones
are visible, see Figure 3. In fact, a bone has dif-
ferent components that are stored with different
intensities. Two components can be distinguished
in a bone: cortical (outer part) and trabecular
(inner part) tissue. As a consequence, in the case
of bone fragments, sometimes it is not straightfor-
ward to define a valid transfer function.
This section is focused on introducing the
main strategies to visualize a 3D medical image in
order to obtain a diagnostic. There are two main
approaches to resolve this visualization problem:
a 2D and a 3D approach. A 2D visualization
demands few computational requirements and,
as a consequence, it is more applicable to mobile
devices. A 2D visualization is based on the dis-
placement of three planes along their correspond- Figure 5. Examples of CT 3D medical images on a
mobile device.
ing axes. The result, is a composite image that is
valuable to specialists, see Figure 4.
On the other hand, direct volume rendering tech-
niques have traditionally been preferred to get a 3D
visualization of 3D medical images (Hadwiger, Kniss,
Rezk-Salama, Weiskopf, & Engel 2006). The first
attempts to visualize 3D medical images in a mobile
device date back to 2005 (Lamberti & Sanna 2005,
Zhou, Qu, Wu, & Chan 2006). In these initial stud-
ies, the low graphics capacity of handheld devices
impeded an independent rendering of 3D images. As
a consequence, a server was mandatory to fulfill this
task and submit the resulting images to the device.
Since the arrival of GPU-enabled mobile
devices the whole rendering process can be carried
out within their own graphics processors (Campo-
alegre, Brunet, & Navazo 2012, Movania & Feng
Figure 3. The same 3D medical images visualized with dif- 2012, Noguera, Jiménez, Ogáyar, & Segura 2012,
ferent transfer functions on a mobile device. Figure (b) uses Noguera & Jiménez 2012). A few examples are
an adequate transfer function to distinguish bone tissue. depicted in Figure 5.

235

VIPIMAGE15_Book.indb 235 9/8/2015 7:45:36 PM

 However, the power limitation of these devices techniques to develop a preoperative reconstruction
in comparison with their desktop PC counter- plate design system for unilateral pelvic and acetabu-
part severely limits the size of the volume and the lar fracture reduction and internal fixation surgery.
required Frames Per Second (FPS) for a real inter- Regarding mobile devices, generally speaking,
active user experience. (Vázquez Alcocer & Balsa the use of tablets or smartphones are not advis-
Rodríguez 2012, Noguera & Jimenez 2015). In spite able during surgery. However, wearable devices,
of depending on a reliable network bandwidth and such as smart glasses, can fit in this activity and
a powerful server in order to provide a good serv- offer interesting novelties and opportunities as a
ice to the client interaction demands, the referred computer aid system during surgery. Nevertheless,
limitations of mobile devices are one of the rea- special tests have to be passed in order to meet the
sons that justify the current necessity of client- required policies and regulations.
server architectures in real environments.
In fact, the mobile volume rendering research
area is still open and it is focused on improving the 5 HUMAN COMPUTER
performance and the compression techniques in INTERACTION ISSUES
order to be able to increase the dimension of the
volume in a real interactive and useful application In order to enrich the visualization of a bone frac-
(Noguera & Jimenez 2015). ture model, medical specialists usually demand
navigational features to mark the model with some
useful measures such as angles, distances, etc., and
4 AUGMENTED REALITY SYSTEMS to set correspondences between fragments. In this
sense, Park et al. (Park, Kim, & Ihm 2008) pro-
Augmented Reality (AR) consists of a mix visuali- posal allows the user to draw vector data by using
zation of live real world and artificially generated the touch-screen for annotation purposes. On the
objects or media. In the context of fracture reduc- other hand, Jeong et al. (Jeong & Kaufman 2007)
tion, it would be the mixture of real patient images presented an interactive virtual colonoscopy and
and extra images and information generated during let the user to enter the volume.
the planning or even in real time during surgery. Additionally, the reduction process requires
In the last decade, many developments have manipulating the fragments: selection, rotations
taken place in the application of AR techniques and translations. We refer the reader to the work
in surgical procedures. Bockholt et al. (Bockholt, of Telkenaroglu et al. (Telkenaroglu & Capin 2013)
Bisler, Becker, Muller-Wittig, & Voss 2003) use AR for an extended review of touch-screen user inter-
techniques to overlap preoperative scanned image action techniques. The selection of the tools to han-
data and results of the planning to the operation dle bone fragments in the mobile device requires
field. Pandya and Auner (Pandya & Auner 2005) considering the material of the surrounding tissue,
superimpose live video with the synchronized blood vessels, etc. In addition, it is also relevant
CT-derived segmented objects of interest. Fischer the response to the user regarding the goodness
et al. (Fischer, Eichler, Bartz, & Straß er 2007) of the union between two or more fragments or
present a hybrid tracking method for medical AR. an adequate metaphor to indicate the proximity
The final pose estimation takes the initial infrared among them. Special care has to be taken with
tracking data as well as salient features in the cam- these aspects as, by using touch-screen devices, a
era image into account. Soler et al. (Soler, Nico- finger partially hides the neighbor area. Similar
lau, Fasquel, Agnus, Charnoz, Hostettler, Forest, considerations apply to the manipulation of fixa-
Mutter, & Marescaux 2008) utilize an AR system tion devices. Observe that this is still an open field
that superimpose preoperative data onto the real of research due to the intrinsic difficulties of a user
view of the patient during a laparoscopic proce- interaction with a touchscreen device and the con-
dure. The system allow tracking instruments and tinuous increment of the number of applications.
improve pathology targeting. Liao et al. (Liao, In fact, this task of manipulating bone fragments
Inomata, Sakuma, & Dohi 2010) develop an image is complex even in case of using haptic devices
overlay navigation system for open MRI-guided within a full virtual reality equipment.
surgery. For that purpose, they present image cali-
bration and markers-based registration methods
for patient-to-image registration. 6 COLLABORATIVE SYSTEMS
Regarding bone fracture reduction, Yao et al.
(Yao, Pang, Liu, & Hu 2009) presents an aided The interface of mobile devices facilitates the inter-
planning and training interactive system frame for active visualization of medical images. Thus these
defective bone fracture reduction surgery. Shen devices could help the specialist to visualize and dis-
et al. (Shen, Chen, Guo, Qi, & Shen 2013) use AR cuss the results of a planning, the medical images

236

VIPIMAGE15_Book.indb 236 9/8/2015 7:45:39 PM

 of the patient, the previously identified bone frag- entirely managed in a handheld device. However,
ments or even their expected final position. some automatic tasks that demand extra computa-
The ubiquity of mobile devices is also a key fac- tional power can be deployed in a server by using
tor for their use in the planning/training of a frac- a distributed architecture model. For instance, the
ture reduction. The special features of this type analysis of a composition, the segmentation of
of devices make them useful for cooperative visu- a bone fragment, the identification of a fracture
alization and interaction, the so-called Computer zone, etc. It is important to remember that the
Supported Cooperative Work (CSCW). A mobile computational power of a mobile device will con-
CSCW solution (Johnson 2013) would allow the tinuously be behind its desktop PC counterpart as
clinician to access medical imaging at the patient it is battery dependent and must focus on energy
bed-site while discussing and sharing images with saving instead of developing pure computational
colleagues in different locations, or even consult the power (Noguera & Jimenez 2015).
images directly in the operating theater. In the con- It is also expected than in a close future the use
text of a bone fracture reduction process a mobile of wearable devices (such as watches, smart glasses
CSCW should provide a comprehensive user inter- and the like) will not be uncommon. These new
face to facilitate a truly cooperation and the use- pervasive devices will likely open up a new range
fulness of the shared information. This approach of possibilities still to be developed. For instance,
would involve the synchronization with a remote in this paper, we have introduced the possibility of
server, where all the patient data would be stored. using smart glasses during surgery.
Nonetheless, at this point it is important to Concluding, we have presented the reduction
remember the essential aim of the CSCW research process problem and the related areas of research
area (Fitzpatrick & Ellingsen 2013): to design in order to propose adequate solutions based on
enhanced collaborative environments by tech- mobile devices. Besides, the increasing number of
nological means. The experience in this field of specific journals and magazines devoted to hand-
research tells us that the technology by itself held and wearable devices issues supports the rel-
does not guarantee an improvement on the col- evance of this new research area.
laborative experience (Tang & Carpendale 2008).
A detailed knowledge of the final socio-technical
system and the impact over all the stakeholders is ACKNOWLEDGEMENTS
always required to success.
This work has been partially supported by the
Spanish Ministerio de Ciencia e Innovacións and
7 DISCUSSION AND CONCLUSIONS the European Union (via ERDF funds) through
the research project TIN2011-25259.
In this paper, we have briefly reviewed the oppor-
tunities of using mobile devices as a technologi-
cal help in a bone fracture reduction process. REFERENCES
The versatility and ubiquity properties of mobile
devices make them very attractive. We have Berkowitz, S.J., J.W. Kung, R.L. Eisenberg, K.
described this process and its main activities and Donohoe, L.L. Tsai, & P.J. Slanetz (2014). Resident
tasks. Afterwards, we have mainly disclosed the ipad use: Has it really changed the game? Journal of
chances of mobile and wearable devices for diag- the American College of Radiology 11(2), 180–184.
nosis, training, composition of bone fragments Blessy, C., M. Anburajan, & S. Ramkumar (2011, April).
and computer aid during surgery. Additionally, Evaluation of mandibular reconstruction plate using
Finite Element Analysis. In 2011 3rd International
we have summarized the requirements regard- Conference on Electronics Computer Technology,
ing human computer interaction issues and for Volume 3, pp. 344–349. IEEE.
designing effective collaborative environments. Bockholt, U., A. Bisler, M. Becker, W. Muller-Wittig, &
We note that medical systems demand especial G. Voss (2003). Augmented reality for enhancement
legal regulation assessments in order to be certi- of endoscopic interventions. In Virtual Reality, 2003.
fied for clinical usage. Proceedings. IEEE, pp. 97–101. IEEE Comput. Soc.
The bone fracture reduction is a complex process Botha, C.P., B. Preim, A. Kaufman, S. Takahashi, & A.
composed of different activities. This fact becomes Ynnerman (2012). From individual to population:
especially true in case of complex fractures with Challenges in Medical Visualization. ArXiv e-prints.
Butson, C.R., G. Tamm, S. Jain, T. Fogal, & J. Kruger
several fragments. This is the reason why in spite of (2013, January). Evaluation of interactive visualiza-
the number of medical apps in the market (iMedi- tion on mobile computing platforms for selection of
calApps) none of them cover the entire reduction deep brain stimulation parameters. IEEE Transac-
process. Those activities that require everytime/ tions on Visualization and Computer Graphics 19(1),
everywhere user interaction are more alike to be 108–117.

237

VIPIMAGE15_Book.indb 237 9/8/2015 7:45:39 PM

 Campoalegre, L., P. Brunet, & I. Navazo (2012). Interactive Noguera, J. & J. Torres (2013). Interaction and visuali-
visualization of medical volume models in mobile zation of 3d virtual environments on mobile devices.
devices. Personal and Ubiquitous Computing, 1–12. Personal and Ubiquitous Computing 17(7), 1485–1486.
Chowdhury, A.S., S.M. Bhandarkar, R.W. Robinson, & Noguera, J.-M. & J.-R. Jimenez (2015). Mobile volume
J.C. Yu (2009, July). Virtual multi-fracture craniofa- rendering: Past, present and future. Visualization and
cial reconstruction using computer vision and graph Computer Graphics, IEEE Transactions on PP(99), 1–1.
matching. Computerized medical imaging and graph- Pandya, A. & G. Auner (2005). Simultaneous augmented
ics: the official journal of the Computerized Medical and virtual reality for surgical navigation. In Fuzzy
Imaging Society 33(5), 333–42. Information Processing Society, 2005. NAFIPS 2005.
Fischer, J., M. Eichler, D. Bartz, & W. Straß er (2007, Annual Meeting of the North American, pp. 429–435.
January). A hybrid tracking method for surgical aug- Park, S., W. Kim, & I. Ihm (2008). Mobile collaborative
mented reality. Computers & Graphics 31(1), 39–52. medical display system. Computer Methods and Pro-
Fitzpatrick, G. & G. Ellingsen (2013). A review of 25 grams in Biomedicine 89(3), 248–260.
years of cscw research in healthcare: Contributions, Paulano, F., J. Jiménez, & R. Pulido (2013). Trends on
challenges and future agendas. Computer Supported identification of fractured bone tissue from CT images,
Cooperative Work (CSCW) 22(4–6), 609–665. pp. 263–269. CRC Press.
Gartner, I. (2013). Gartner says worldwide pc, tablet and Paulano, F., J.J. Jiménez, & R. Pulido (2014, May). 3D
mobile phone combined shipments to reach 2.4 bil- segmentation and labeling of fractured bone from CT
lion units in 2013. http://www.gartner.com/newsroom/ images. The Visual Computer 30(6–8), 939–948.
id/2408515. Shen, F., B. Chen, Q. Guo, Y. Qi, & Y. Shen (2013).
Hadwiger, M., J.M. Kniss, C. Rezk-Salama, D. Weiskopf, & Augmented reality patient-specific reconstruction
K. Engel (2006). Real-time Volume Graphics. Natick, plate design for pelvic and acetabular fracture surgery.
MA, USA: A.K. Peters, Ltd. International Journal of Computer Assisted Radiology
Hu, Y., H. Li, G. Qiao, H. Liu, A. Ji, & F. Ye (2011). and Surgery 8(2), 169–179.
Computer-assisted virtual surgical procedure for Sikorski, J.M. & S. Chauhan (2003). Computer-assisted
acetabular fractures based on real CT data. Injury orthopaedic surgery: Do we need caos? Journal of
42(10), 1121–1124. Bone & Joint Surgery, British Volume 85-B(3), 319–323.
iMedicalApps. http://www.imedicalapps.com [accessed Soler, L., S. Nicolau, J.-B. Fasquel, V. Agnus, A. Charnoz,
17 July 2015]. J.M. Hostettler, C. Forest, D. Mutter, & J. Marescaux
Jaramaz, B., M. Hafez, & A. DiGioia (2006, Sept). (2008). Virtual reality and augmented reality applied
Computer-assisted orthopaedic surgery. Proceedings to laparoscopic and notes procedures. In Biomedical
of the IEEE 94(9), 1689–1695. Imaging: From Nano to Macro, 2008. ISBI 2008. 5th
Jeong, S.-J. & A.E. Kaufman (2007, July). Interactive IEEE International Symposium on, pp. 1399–1402.
wireless virtual colonoscopy. Visual Computer 23(8), Sugano, N. (2003). Computer-assisted orthopedic surgery.
545–557. Journal of Orthopaedic Science 8(3), 442–448.
Johnson, D. (2013). Mobile support in cscw applications Tang, C. & S. Carpendale (2008). Evaluating the deploy-
and groupware development frameworks. International ment of a mobile technology in a hospital ward. In
Journal of Interactive Mobile Technologies (iJIM) 7. Proceedings of the 2008 ACM Conference on Com-
Kho, A., L. Henderson, D. Dressler, & S. Kripalani puter Supported Cooperative Work, CSCW ’08, New
(2006). Use of handheld computers in medical edu- York, NY, USA, pp. 205–214. ACM.
cation. Journal of General Internal Medicine 21(5), Telkenaroglu, C. & T. Capin (2013, October). Dual finger
531–537. 3d interaction techniques for mobile devices. Personal
Lamberti, F. & A. Sanna (2005). A solution for dis- Ubiquitous Comput. 17(7), 1551–1572.
playing medical data models on mobile devices. In Vázquez Alcocer, P. & M. Balsa Rodríguez (2012).
SEPADS’05, Stevens Point, Wisconsin, USA, pp. 1–7. Practical volume rendering in mobile devices. In
World Scientific and Engineering Academy and Soci- Proc. International Symposium on Visual Computing,
ety (WSEAS). Volume 7431 of Lecture Notes in Computer Science
Liao, H., T. Inomata, I. Sakuma, & T. Dohi (2010, June). (LNCS), pp. 708–718. Springer Verlag.
3-D augmented reality for MRI-guided surgery using Whipple, E.C., K.L. Allgood, & E.M. Larue (2012).
integral videography autostereoscopic image overlay. Third-year medical students’ knowledge of privacy
IEEE transactions on bio-medical engineering 57(6), and security issues concerning mobile devices. Medical
1476–86. Teacher 34(8), e532– e548.
Movania, M.M. & L. Feng (2012). Ubiquitous medical Willis, A., D. Anderson, T. Thomas, T. Brown, & J.L.
volume rendering on mobile devices. In Information Marsh (2007). 3D reconstruction of highly fragmented
Society (i-Society), 2012 International Conference on, bone fractures. In Medical Imaging 2007: Image
pp. 93–98. Processing. Proceedings of the SPIE, pp. 65121.
Noguera, J. & J. Jiménez (2012). Visualization of very Yao, Y., X.-x. Pang, T. Liu, & Q.-x. Hu (2009). Design
large 3d volumes on mobile devices and WebGL. In of a prototype for augmented reality defective bone
WSCG Communication Proceedings, pp. 105–112. repair simulation system. In 2009 First International
Noguera, J., J. Jiménez, C. Ogáyar, & R. Segura (2012). Workshop on Education Technology and Computer Sci-
Volume rendering strategies on mobile devices. In ence, pp. 66–70. IEEE.
International Conference on Computer Graphics The- Zhou, H., H. Qu, Y. Wu, & M. Chan (2006). Volume visu-
ory and Applications (GRAPP 2012). Rome (Italy), alization on mobile devices. In 14th Pacific Conference
pp. 447–452. on Computer Graphics and Applications, pp. 76–84.

238

VIPIMAGE15_Book.indb 238 9/8/2015 7:45:39 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Automated peritumoral edema segmentation in preoperative

brain MRI scans

E. Binaghi
Dipartimento di Scienze Teoriche e Applicate, Università degli Studi dell’Insubria, Varese, Italy

P. Melchiorre & L. Romitelli

Università degli Studi dell’Insubria, Varese, Italy

S. Balbi
Dipartimento di Biotecnologie e Scienze della Vita, Università degli Studi dell’Insubria, Varese, Italy

D. Lattanzi
Scuola di Specializzazione in Neurochirurgia, Ospedale di Circolo e Fondazione Macchi, Varese, Italy

ABSTRACT: Peritumoral brain edema is a frequently encountered phenomenon that strongly contrib-
utes to neurological signs and symptoms. This paper presents a fully automated procedure based on
Support Vector Machine for the accurate and efficient edema delineation in brain MRI imagery. The
segmentation procedure is conceived fully automatic and specifically oriented to non-volumetric data to
facilitate the insertion in clinical practice. The major efforts in building a robust classifier were concen-
trated in the extraction and selection of a set of significant features from multispectral brain MR images.
Preliminary experimental results obtained by processing in-house collected data, are encouraging creat-
ing the premises for an effective use of the automated segmentation procedure in brain tumor clinical
studies. To see if the proposed supervised method can be considered an alternative to other contemporary
approaches, the results obtained were compared with those obtained by an unsupervised region-based
segmentation procedure based on Graph Cut.

1 INTRODUCTION giving rise to a variety of approaches. Most of the

work has been focused on the automated delinea-
Magnetic Resonance (MR) imaging plays an impor- tion of the whole pathological area or the active
tant role in the diagnosis of brain pathologies. In tumor area basing on semi-automatic or fully
brain tumor clinical studies it represents the first automated methods (Gordillo et al. 2013).
adopted exam to forward a diagnostic hypothesis, In recent years we assist to a wide spread utili-
typically based on tumor imaging features includ- zation of Machine Learning procedures successful
ing active tumor area, necrosis and edema, usually for multimodal MRI segmentation. From a view-
in MRI sequences with no or slight enhancement point of reproducibility, unsupervised methods are
after gadolinium injection (Duffau 2005). Accurate clearly desirable, but most of them still present low
and efficient segmentation of tumor and surround- accuracy (Gordillo 2013). Supervised approaches
ing tissues of the brain in the MRI imagery is fun- circumvent the problem of explicitly and analyti-
damental in order to select the most appropriate cally describing the parts to be segmented lying to
treatment, to plan the best surgical approach and to a learning stage the charge of inducing the classi-
envisage possible alternative solutions supporting fier from supervised data available. Parametric and
surgery, when the resection appears not at once indi- nonparametric methods based on statistical, neu-
cated (Clark 1995, Kaus et al. 2001). Segmentation ral and fuzzy frameworks are proposed showing in
accomplished through a complete manual tracing general a competitive behavior in capturing com-
is a difficult, time consuming task usually affected plex multivariate relationships in the data (Bezdek
by strong intra- and inter-variation (Warfield et al. et al. 1993, Gordillo et al. 2013, Bauer et al. 2013).
2004, Binaghi et al. 2014a). Support Vector Machine (SVM) methods (Vapnik,
The problem of automatically segmenting MRI 1995) have shown excellent performances in MRI
brian tumor images continues to be investigated segmentation studies aimed at identifying a variety

239

VIPIMAGE15_Book.indb 239 9/8/2015 7:45:39 PM

 of neurological conditions (Bauer et al. 2013, Let vector x ∈ Rn denote a pattern to be classi-
Binaghi et al. 2014b). fied, and let scalar y∈ {−1,+1} denote correspond-
In the present work we focus the attention on per- ing class label. Let {(xi, yi), i = 1, 2 … l} a supervised
itumoral brain edema segmentation. Peritumoral training set of l elements for a two-class classifica-
brain edema is a frequently encountered phenom- tion problem, with xi ∈ X ⊆ Rn and yi ∈ {−1,+1}.
enon that strongly contributes to neurological signs Considering the case of linearly separable data, the
and symptoms. The role of peritumoral edema as solution consists in the construction of the decision
a prognostic factor has been investigated in sev- function fw, b (x) = sgn(gw, b(x)) with g w, b(x) = wt x + b
eral studies that have led to different conclusions that separates training examples and correctly clas-
depending on the pathology under study. A quanti- sifies an input pattern x that is not necessarily from
tative evaluation of the edema from MR imagery the training set. SVM classifier defines the Maximum
is of basilar relevance to formulate surgical indica- Margin Separating Hyperplane that causes the larg-
tions in elderly patients harboring intracranial men- est separation between “borderline” examples from
ingiomas (Greenberg et al. 1999, Caroli et al. 2005). the two classes. Mathematically, this hyperplane can
In such a critical context the availability of auto- be found by minimizing the cost function:
mated segmentation procedure specifically oriented
to the accurate and efficient edema detection and 1 2
J (w ) = w (1)
delineation is of great importance to easily conduct 2
retrospective, large scale studies and to insert the
segmentation task in the clinical practice. subject to
We address the segmentation task by proposing
a supervised strategy based on SVM. The super- wT xi + b ≥ +1 for yi = +1
vised classification makes use of multichannel or (2)
intensities of Post Gadolinium T1-weighted (T1c) wT xi + b ≤ −1 for yi = −1
and T2-weighted FLAIR image (T2f). Additional
textural and contextual features are considered and
For nonlinear classification the function.
increasingly added during the experimentation in
order to select the best configuration. The segmen-
tation procedure is fully automatic and oriented to g′ wtϕ ( x ) b (3)
non-volumetric data characterized by poor inter-
slice spacing to facilitate the insertion in current is specified in which the non liner operator ϕ (.) is
clinical practice. introduced to map the input pattern x into a higher
dimensional space. In case of non linear classifica-
tion a non-negative slack variable ξ is introduced,
2 THEORETICAL BACKGROUND in order to have a tolerance to errors. The cost
function to be minimized become:
SVM is a classification algorithm based on kernel l
1
methods (Vapnik 1995, Schoelkopf & Smola 2002) J ( w, ξ ) w C ∑ ξi (4)
able to map the original parameter vectors into a 2 i =1
higher dimensional feature space through a kernel
function. Classes which are non-linearly separable subject to
in the original space can be linearly separated in the
higher dimensional feature space. Thus SVM is capa- wtϕ ( xi
yi (w b) ξi with ξi ≥ 0 i , , l (5)
ble of solving complex nonlinear classification prob-
lems. The learning algorithm is based on the choice C is a constant value that expresses the balance
of current worst-classified patterns. These training between margin and error and must be provided
examples are the most difficult to classify and dur- a priori. Linearly non separable data are analyzed
ing most of the training period are on the wrong side with kernel functions such as higher order polyno-
of the current decision boundary. At the end of the mials and Gaussian RBF.
training stage, these patterns will be support vectors.
Suykens (2002) proposed a new formulation
of SVM by adding a Least Squares (LS) term in 3 FULLY AUTOMATED EDEMA
the original formulation of the cost function. This SEGMENTATION
modified version of the original Vapnik’s SVM
formulation avoid solving a more difficult quad- The SVM model learns from a set of labeled image
ratic programming problem and only requires the elements the invariant common properties of the
solution of a set of linear equations thus reducing edema and not-edema classes. The trained SVM
significantly the computational complexity. classifier acts as a dichotomizer; it receives in input

240

VIPIMAGE15_Book.indb 240 9/8/2015 7:45:39 PM

 a multidimensional pattern extracted from T1c and
T2f MR elements and assigns labels to elements
never seen during the training phase.
Features composing the multidimensional pat-
ters are extracted from the co-registered imaging
data. The extraction of an appropriate set of local
features is a crucial task strongly affecting the per-
formances of the overall segmentation strategy.
In addition to the image intensities in the two
MR modalities, we consider other significant fea-
tures to describe quantitatively neighbor relation-
ships and texture of the images. Texture analysis
covers a wide range of techniques based on first-
and second order image texture parameters (Tuce- Figure 1. Axial slice of one patient. From left to right:
rian & Jain 1998). the T1c and T2f scans.
In more detail, the following features are con-
sidered: multimodal intensities from T1c and T2f
scans (It1c, It2f), first order texture features: mean 2Tp
DC = (6)
(M), variance (Var), skewness (S), kurtosis (K) 2Tp Fp + Fn
and entropy (E) computed on T1c and T2f scans,
intensities in neighborhoods of multimodal vox-
with Tp = True Positive, Tn = True Negative,
els (I1-I26). All the proposed features have been
Fp = False Positive, Fn = False Negative.
analyzed systematically in order to determine the
combination that is most appropriate for the classi-
fication task (see section 4). The features have been 4.2 Results
normalized to have zero mean and unit variance.
Several experiments were conducted in which dif-
ferent configurations of the segmentation proce-
4 EXPERIMENTS dure were considered varying:
– parameters involved in the SVM model
The segmentation method was experimented on
– training and test data
a in-house collected multispectral datasets of 10
– features composing multidimensional patterns.
patients. Each dataset is composed of T1c and T2f
scans. The T1c is acquired using a 3D sequence After a trial and error procedure the value of the
characterized by 1 mm isotropic voxels, the inter- standard deviation (σ) for Gaussian RBF kernel
slice spacing of 1 mm and the slice thickness of was chosen as 0.5 in the SVM classifier configured
1 mm; the T2f sequence includes an in-plane reso- as soft-margin LS model.
lution between 0.75 and 0.81 mm and slice thick- Different configurations of the classification
ness of 5 mm. The spacing between slice is 6 mm for procedure were also evaluated varying the number
cases 1,3,6,7,8,10 and of 6.5 mm for cases 2,4,5,9. and the source of training and test examples ran-
Figure 1 shows an axial slice of one patient with domly selected from the ground truth masks. We
the two MRI modalities considered. considered patient-specific (intra-patient) analy-
The masks obtained by the automated segmen- sis and inter-patient analysis. In patient-specific
tation were compared with the masks obtained analysis the training and test data were obtained
through a manual segmentation of the T2f images. from the ground truth masks of the image to be
Manual segmentations were performed by a team segmented. Inter-patient analysis was performed
of experts with the support of an interactive soft- training on several patients, and testing in one
ware tool for slice-by-slice labeling. Manual seg- never seen during the training phase.
mentations provided by the experts were fused by Different configurations of the multispectral
applying a Majory Voting procedure. pattern presented in input to the SVM model have
been also considered in the experiments basing on
4.1 Evaluation metrics the following selected feature sets:
Performances were evaluated by comparing spatial – intensity values from T1c and T2f scans
distribution of the masks obtained by the auto- – intensity values from T1c and T2f scans + texture
mated segmentation with spatial distribution of measures
reference masks manually segmented by experts. – intensity values from T1c and T2f + gray scale val-
The adopted similarity measure is the Dice Coef- ues of neighborhoods voxels of each scan includ-
ficient (DC) defined as follows (Bioux et al. 2000): ing incrementally 6, 18, and 26 neighbors (I1-I26).

241

VIPIMAGE15_Book.indb 241 9/8/2015 7:45:40 PM

 The configuration that optimizes the balance obtained by performing inter-analysis on the avail-
between performance accuracy, computational able data sets are also reported. In this case Dice
efficiency, robustness/generalisation capabilities, coefficients are obtained by averaging the individ-
was intensity values from T1c and T2f scans + tex- ual leave-one-out cross-validation results.
ture measures computed on 6 neighborhoods voxels. With the intra-patient analysis, the Dice coeffi-
The multidimensional pattern is then structured as cient is 0.84. Performances obtained with the inter-
follows: patient analysis slightly decrease obtaining a Dice
coefficient equal to 0.82. Results obtained confirm
⎡ xt1c xt 2 f ⎤ with the good and robust behaviour of the segmenta-
i ⎣ ⎦
tion procedure that maintains acceptable perform-
xt c ⎡ I i t1c ,M
M i
t c Var S i t c K i t1c , E i t1c ⎤⎦
V i t1c ,S (7)
⎣ ances without requiring patient specific training.
xt ⎡I i
,M i
,V
,V
Var i
ar t i
,S t ,K
,K i
,E i ⎤ Figure 2 presents the segmentation result for one
f ⎣ t f t f f f t f t f ⎦
slice in case of inter-patient analysis. The edema
and used a training set of 10.000 labeled samples region is well delineated although few omission
randomly chosen within the ground truth masks and commission errors are visible in some areas.
in the percentage of 70% in edema region, 30% in
the remaining brain area. Table 1 shows perform-
ance values obtained by averaging Dice coefficients 5 COMPARISON ANALYSIS
obtained by the SVM model trained on patient
specific data (intra-analysis). Performances values To see if the proposed supervised method can be
considered an alternative to other contemporary
approaches, the results obtained were compared
Table 1. Dice similarity coefficients (mean and with those obtained by the unsupervised region-
standard deviation) for the datasets under study. based segmentation procedure based on graph cuts
(Boykov & Funka-Lea 2006). The s/t Graph Cuts
Dice coefficient framework offers a globally optimal object extrac-
tion method for N-dimensional images. Graph cuts
Intra-patient analysis 0.84 ± 0.05 optimize a segmentation energy function that com-
Inter-patient analysis 0.83 ± 0.07 bines boundary regularization with regularization
of regional properties.
The application of Graph Cut requires the iden-
tification of object and background prototypes with
which to initialize the overall segmentation process.
Usually this task is accomplished through an
interactive session in which users manually select
seeds on the image. Recently there have been some
proposals exploiting solutions for automatically
initializing Graph Cut-based segmentations of
biomedical images (Santle et al., 2012).
The present study uses the max-flow/min-cut
algorithm (Boykov and Kolmogorov 2004) as opti-
mization framework and adopts an automated ini-
tialization procedure based on k-means clustering
algorithm.
Figure 2 presents the segmentation results for
one axial slice obtained by the SVM and Graph
Cut. The SVM segmentation produces a mask with
a higher level of underestimation compared to the
mask produced by the Graph Cut segmentation.
In Table 2 the means of the DC scores obtained
by the CEM-Graph Cut and SVM when segment-
ing Edema part for the datasets under study are
reported. SVM performances are obtained by aver-
Figure 2. Segmentation results of one axial slice
aging results of the inter- and intra-analysis. The
obtained by the interpatient analysis. First row, from Dice coefficient obtained by the Graph Cut model
left to right: original T1c image, original T2f image; sec- over the available data set is equal to 0.87. The two
ond row, from left to right: edema segmentation mask methods show comparable performances. Graph-
obtained by the automated procedure, truth mask. Cut slightly prevails, however the comparison must

242

VIPIMAGE15_Book.indb 242 9/8/2015 7:45:41 PM

 Bezdek JC, Hall LO, Clarke LP. 1993. Review of MR
image segmentation techniques using pattern recogni-
tion. Medical Physic 20(4):1033–1048.
Binaghi, E., Pedoia, V., Balbi S. 2014a. Collection and
fuzzy estimation of truth labels in glial tumour seg-
mentation studies. Computer Methods in Biomechanics
and Biomedical Engineering: Imaging and Visualisation
2:1–15.
Binaghi E., Omodei M., Pedoia V., Balbi S., Lattanzi D.
and Monti E. 2014b. Automatic Segmentation of MR
Brain Tumor Images using Support Vector Machine
in Combination with Graph Cut. In Proc. of the Inter-
national Conference on Neural Computation Theory
and Applications (NCTA-2014), Rome, 22–24 Ottobre
2014, SciTePress.
Bouix, S., Martin-Fernandez, M., Ungar, L.,
Koo, M.N.M.S., McCarley, R.W., Shenton, M.E.,
Figure 3. Segmentation results of one axial slice. First 2007. On evaluating brain tissue classifiers without a
row, from left to right: original T1c image, original T2f ground truth. NeuroImage, 36:1207–24.
image; second row, from left to right: edema truth mask, Boykov, Y., & Kolmogorov, V., 2004. An experimen-
segmentation mask obtained by SVM and by Graph Cut. tal comparison of min-cut/max-flow algorithms for
energy minimization in vision., PAMI, 26:1124–37.
Boykov, & Funka-Lea, G., 2006. Graph cuts and effi-
Table 2. Mean Dice Scores (DS) obtained by cient n-d image segmentation. Int. J. Comput. Vision,
the CEM-Graph Cut and SVM (averaging results 70:109–131.
of the inter- and intra-analysis), when segment- Caroli, M., Locatelli M., Prada F., Beretta F., Martinelli-
ing Edema part for the datasets under study. Boneschi F., Campanella R., Arienta C. 2005. Surgery
for intracranial meningiomas in the elderly: a clinical–
Dice coefficient radiological grading system as a predictor of. J Neuro-
surg, 102:290–294.
Graph Cut 0.87 ± 0.09 Clarke, L.P., Velthuizen, R., Camacho, M., Heine, J.,
SVM 0.84 ± 0.06 Vaidyanathan, M., Hall, L., Thatcher, R., Silbiger,
M.S. 1995. MRI segmentation: methods and applica-
tions. Magn Reson Imaging 13(3):343–36.
Duffau, H. 2005. Lessons from brain mapping in sur-
take into account that, differently from the super- gery for low-grade glioma: insights into associations
vised method, Graph Cut is semi-automated and between tumour and brain plasticity. Lancet Neurol
information provided by the users for each image 4(8):467–487.
to be processed, may improve performances thus Gordillo, N., Montseny E., Sobrevilla, P. 2013. State of
reducing stability and reproducibility of the results. the art survey on MRI brain tumor segmentation
Magnetic Resonance Imaging 31:1426–1438.
Greenberg, H., Chandler, W., Sandler, H. 1999. Brain
6 CONCLUSIONS Tumors, Oxford: Oxford University Press.
Kaus, M.R., Warfield, S.K., Nabavi, A., Black, P.M.,
The objective of the present study was to develop a Jolesz, F.A., Kikinis R. 2001. Automated segmentation
supervised segmentation procedure based on SVM of MRI of brain tumors. Radiology 218:586–591.
Santle, K. Camilus, Govindan, V.K., 2012. ‘A Review on
specifically oriented to peritumoral edema deline- Graph Based Segmentation.’ IJIGSP 4(5):1–13.
ation in non-volumetric MRI data. As seen in our Schoelkopf, B., Smola, A. 2002. Learning with kernels:
experimental context the procedure is competitive support vector machines, regularization, optimization,
with other approaches. It shows a robust behaviour and beyond, Cambridge: MIT Press.
and it is oriented to the application in clinical prac- Suykens, J.A.K., Van Gestel, T., De Brabanter, J.,
tice. Future plans contemplate the acquisition of De Moor, B., Vandewalle, J., 2002. Least Squares Sup-
new data with which we will be able to perform a port Vector Machines. Singapore: World Scientific Pub-
more significant inter-patient analysis and to bet- lishing Co.
ter analyse the generalisation capabilities of the Tuceryan, M. & Jain, A. 1998. Texture analysis. Inc.
River Edge, NJ: World Scientific Publishing Co.
trained segmentation model. Vapnik, V.N. 1995. The Nature of Statistical Learning
Theory, New York: Springer-Verlag.
REFERENCES Warfield, S.K., Zou, K.H., Wells, W.M. 2004.
Simultaneous truth and performance level estima-
Bauer, S., Wiest, R., Nolte, L-P., Reyes, M. 2013. A sur- tion (STAPLE): an algorithm for the validation of
vey of MRI-based medical image analysis for brain image segmentation. IEEE Trans Med Imaging 23(7):
tumor studies. Phys. Med. Biol. 58:R97–R129. 903–921.

243

VIPIMAGE15_Book.indb 243 9/8/2015 7:45:41 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Evaluation of segmentation techniques for wound area identification

Guanghua Zhang
Chongqing University of China, Chongqing, China
Graphics and Imaging Laboratory, University of Girona, Girona, Spain

Pau Xiberta, Anton Bardera & Imma Boada

Graphics and Imaging Laboratory, University of Girona, Girona, Spain

Angel Romero
Faculty of Infirmary, University of Girona, Girona, Spain

ABSTRACT: One of the key parameters that has to be considered in wound treatment is the size of the
wound. Usually, this is monitored by an expert at regular intervals. To carry out this process, image-based
approaches can be applied. Generally, these approaches start applying a segmentation technique to isolate
the wound from the rest of the image. Due to the importance of segmentation, our aim is to evaluate
the performance of four segmentation techniques (grabcut, snakes, region growing and Support Vector
Machine (SVM)) on pressure ulcer images. In our study, we have considered twenty images captured fol-
lowing normal clinical practice and under non-controlled conditions which makes the processing more
difficult. The images have been manually segmented by two experts and the obtained results have been
compared with the automatic segmentation obtained with the different methods. From the comparison, we
can conclude that both snakes and grabcut methods clearly behave better than region growing and SVM.

1 INTRODUCTION region (Thawer, Houghton, & Woodbury 2002, G.

Gethin & S. Cowman 2006).
A pressure ulcer, a type of wound, is a localized Beside these manual techniques, there are also
injury to the skin or underlying tissue usually over more sophisticated methods based on electronic
a bony prominence, as a result of pressure, or devices such as the planimeter that uses a transpar-
pressure in combination with shear. Once a pres- ent film placed over the wound while the margins are
sure ulcer has developed, immediate treatment is traced, and computer calculates the area after dig-
required and the treatment strategies vary accord- itally scanning the tracing (Mayrovitz & Soontupe
ing to the stage of the wound and the purpose of 2009). There are also more complex and expensive
the treatment. Generally, an expert periodically vision-based technologies that use stereophotogram-
examines the ulcer progress to monitor the admin- metry or structured lighting to obtain wound images
istered treatment. This monitoring is based on (Krouskop, Baker, & Wilson 2002, Plassman & Jones
visual inspection and different wound parameters 1998). These images are reconstructed using a com-
are analyzed, amongst them, shape, size, depth and puter to produce a 3-D model of the wound. The
tissue type (Gethin 2006). wound border is then traced on the computer image,
Different methods are used to measure pressure and the computer software determines the area and
ulcer shape and size. The simplest one assumes volume of the wound. More recently, Sprigle et al.
that the wound is rectangular in shape and, using (Sprigle, Nemeth, & Gajjala 2011) proposed a new
a ruler, the clinician measures wound surface by device to photo-document and measure the wound
multiplying the greatest length by the perpen- area that uses four laser diodes arranged in a square
dicular greatest width (Langemo, J, D. Hanson, with the camera lens lying in its center. These lasers
S. Hunter, & P. Thompson 2008). This method are used to calculate wound area and skew. For a
overestimates wound area and accuracy tends to review and comparison of wound measurement
diminish as the wound size increases. A more accu- techniques see (M. Bilgin & U.Y. Gunes 2014,
rate method traces the wound perimeter on a grad- Ahmad-Fauzia, Khansab, Catignanic, Gordillo,
uated clear plastic or acetate paper placed over the Send, & Gurcan 2015).
wound and calculates the wound area by determin- In this context, it is also common the use of digital
ing the number of squares contained in the wound cameras to record wound progress. Several techniques

245

VIPIMAGE15_Book.indb 245 9/8/2015 7:45:42 PM

 have been proposed to measure wound parameters 2.1 Snakes
from taken images and also to automatically classify
Snakes method (Kass, Witkin, & Terzopoulos
them. These methods definition is challenging due
1988) is an edge-based segmentation technique.
to the variability of image parameters such as illu-
From an initial user-defined contour, also called
mination or patient position that makes the method
snake, the method iteratively updates this con-
reproducibility difficult. Focusing on image-based
tour in order to minimize its total energy. The
methods, a key step of all them is the application
total energy is composed of two parts: the internal
of a segmentation method that after image cap-
energy that depends on contour shape (smoother
ture identifies the wound area on the image. Once
the contour, smaller the internal energy) and the
it has been identified other techniques such as color
external energy that depends on the energy field,
treatment, feature extraction and classification
usually defined from the image gradient. The con-
techniques are applied to determine wound param-
tour reaches minimum at the local energy extreme
eters (B. Song & A. Sacan 2012, Veredas, Mesa, &
that correspond to the image edges in case of image
Morente 2010, Kolenik & Fexa 2004, M. Burns, J.
gradient is used. In our framework, we have used
Enderle, Rosow, & Zhu 2002, Weber, N. Watermann,
the implementation provided
J. Jossinet, J. Byrne, J. Chantrey, S. Alam, K. So,
by the opencv library (Bradski 2000). This method
Bush, S. OKane, & E.T. McAdams 2010, Sundeep-
requires as input parameters: α that represents the
Kumar & Eswara-Reddy 2014). Due to the impor-
weight of continuity energy, β that represents the
tance of segmentation techniques in these wound
weight of curvature energy, and γ the weight of
processing pipelines, in this paper, we evaluated the
image energy. For the tests we set α = 0.1, β = 0.4 ,
performance of four different segmentation methods:
and γ = 0.5 . To apply the method the user interac-
grabcut, snakes, region growing and SVM on pres-
tion has to draw an initial contour in the image. In
sure ulcer images. The methods are tested on twenty
our case, the initial contour is defined considering
images and the obtained results are compared with
20 points to ensure that the wound is included in
wound area segmentation performed by experts.
this initial contour. For the test (see section 3) we
set the number of iterations at 1000. The first col-
umns of Figure 1 and 2 illustrate two examples of
2 WOUND SEGMENTATION the initial contour and the obtained results.
There is a wide variety of image segmentation
2.2 Region growing
methods in the literature (Gonzalez & Woods
2006). To carry out our study, we have selected Region growing is a region-based technique. From
four segmentation methods to cover edge-based, an initial seed point, neighbour pixels are added
region-based, graph-based, and clustering-based into the region depending on a similarity criterion,
segmentation techniques, respectively. Next, we typically based on a given threshold. The process
describe these segmentation techniques. is iterated on until there are not new inclusions.

Figure 1. From left to right the user interaction and the obtained segmentation with snakes, region growing, grab cut,
SVM and SVM-neigh techniques.

246

VIPIMAGE15_Book.indb 246 9/8/2015 7:45:42 PM

 Figure 2. From left to right the user interaction and the obtained segmentation with snakes, region growing, grab cut,
SVM and SVM-neigh techniques.

In our implementation, the method computes the The third columns of Figure 1 and 2 show two
standard deviation of the image intensity on a win- examples of selected regions by the user and the
dow centered in the seed point. Then, the neigh- obtained results.
bour pixels are included if their intensity is in the
range [ k k ], where I is the intensity of
2.4 Support Vector Machine (SVM)
the seed point, k is a parameter of the method, and
σ is the standard deviation computed in the initial SVM method is a clustering-based technique that
step. Thus, the method has two parameters: the classifies each pixel using the Support Vector
size of the window where the standard deviation is Machine algorithm (Cortes & Vapnik 1995) from
computed and k. We have used the a training set, which is given by the user. For each
implementation provided by the opencv library pixel, a vector of features is obtained and used by
(Bradski 2000). In this method, the user interac- the SVM algorithm to perform the classification.
tion with the image simply consists in the selection In our implementation, the RGB color values and
of the seed point and the adjustment of k and the the distance to the lesion center (given by the user)
window size to find out the best result. e set σ are used as features. In order to do not have fea-
between 10 and 50. The second columns of Figure tures with different range of values, the distance
1 and 2 show two examples of the seed placement to the lesion center has been divided by the image
and the obtained results. diagonal and multiplied by 255. In this way, every
feature takes values in the range [0,255]. In our
framework, we have used the SVM() implemen-
2.3 Grabcut
tation provided by the opencv library (Bradski
The grabcut method (Rother, Kolmogorov, & 2000). The parameters of this implementation are:
Blake 2004) is a graph-based technique and, in par- the four selected features corresponding to R, G,
ticular, based on graph cuts. To start the process, and B values and the pixel to the center of selected
the user defines a bounding box around the object foreground distance. We have also implemented
to be segmented. Then, the method estimates the the SVM method adding local information. For
color distributions of the target object and the each pixel, we have considered the R, G, and B val-
background using a Gaussian mixture model. This ues from the top, bottom, left and right neighbour
model is used to construct a Markov random field pixels obtaining a feature vector with 12 values.
over the pixel labels, with an energy function that We denote this method SVM-neigh. The user
prefers connected regions with the same label, and interaction for SVM method consists in the defi-
running a graph cut based optimization to infer nition of some rectangular regions corresponding
their values. Since this estimate is more accurate to either background or foreground wound pixels.
than the one taken from the bounding box, this The fourth and fifth columns of Figure 1 and 2
two-step procedure is repeated until convergence. show two examples of selected regions by the user
To correct the estimates the user can point out and the obtained results with the SVM and SVM-
misclassified regions and rerun the optimization. neigh, respectively.
The method also corrects the results to preserve
edges. In our framework, we have used the grabCut
method from the opencv library (Bradski 2000). 3 EXPERIMENTAL SETUP
This implementation does not require any param-
eter. The user interaction for this method consists This study was done in collaboration with clini-
in drawing a square contour containing the wound cians from the Faculty of Infirmary from the
to get an initial segmentation and also selecting University of Girona. For the experiments we
the background and foreground with some points. used 20 images of 2304 × 3072) resolution stored

247

VIPIMAGE15_Book.indb 247 9/8/2015 7:45:45 PM

 in JPEG format (see Fig. 3). Clinicians captured TP
these images following normal clinical practice and P= .
TP + FP
under non-controlled conditions which makes the
processing more difficult. We developed a tool to Recall indicates the proportion of the lesion that
capture the ground truth. Clinicians use this tool has been segmented, that is
to trace the wound boundary on the image. From
this trace the tool computes a mask where black TP
and white pixels represent background and wound R= .
TP + FN
area, respectively.
All testing images were manually segmented Since both measures are complementary (typi-
by two experts and also using the implemented cally when one grows, the other decreases), the har-
methods. We compared the masks obtained from monic mean of both, also known as F-score, is also
manual segmentation with the masks obtained by used to evaluate the results, since it is a combina-
each method and classified pixels in four groups: tion of both. F-score is given by
False Positives (FP), True Positives (TP), False
Negatives (FN), and True Negatives (TN). An P R
example is presented in Figure 4. From the quan- F -score = 2 .
tity of pixels at each group, the measures precision P R
(P) and recall (R) can be computed. Precision is
given by the proportion of the segmented pixels
that are correctly classified as lesion, that is, 4 RESULTS

The obtained results are presented in Table 1, where

the mean and standard deviation of precision (P),
recall (R), and F-score are shown. As it can be seen,
the best results are achieved by the snakes and
grabcut methods, while region growing, SVM, and
SVM-neigh achieve much poorer results. We have
also shown the inter-operator variability, that has
been obtained by considering the operator 1 as the
ground truth and analyzing the manual segmen-
tations of operator 2 in terms of precision, recall
and F-score. As it can be seen, all the methods still
behave poorer than manual edition.
With respect to the method interactivity, snakes
requires the definition of an initial contour that,
Figure 3. The 20 images of the wound database used in our implementation, is done by manually defin-
in this study. ing a polygon of 20 points. After this, the user

Figure 4. Masks of manual segmentation and the automatic ones are compared and points are labelled as true and
false positive or true and false negative.

248

VIPIMAGE15_Book.indb 248 9/8/2015 7:45:47 PM

 Table 1. Segmentation results for the four methods in terms of precision (P),
recall (R), and F-score.

Method P R F-score

Snakes 0.922 ± 0.065 0.915 ± 0.083 0.915 ± 0.058

Region growing 0.866 ± 0.132 0.644 ± 0.186 0.713 ± 0.139
Grabcut 0.896 ± 0.100 0.947 ± 0.076 0.914 ± 0.063
SVM 0.838 ± 0.157 0.690 ± 0.271 0.704 ± 0.217
SVM-neigh 0.869 ± 0.081 0.841 ± 0.080 0.851 ± 0.579
Inter-operator 0.966 ± 0.045 0.937 ± 0.056 0.950 ± 0.033

can define some parameters that can be optimally automate wound measurement from images. Wound
set, for most of cases, after about 10 iterations. segmentation is a key step of these methods. Due to
The total time to obtain the final result is about the importance of segmentation in these approaches
50 seconds. The grabcut algorithm also achieve we have evaluated four segmentation techniques and
very good results, but the interaction takes more compared with manual segmentation. From our
time than snakes, since the method is much more tests we observed that snakes and grabcut methods
computationally demanding. The final result is obtain the best results, which are slightly less accu-
achieved in this case in about 2 minutes. Despite rate than manual edition. As future work, we will
the poor results of the region growing, its interac- improve the user interface to easily interact with the
tion is very fast and a final result can be obtained method parameters, we will integrate the methods
in 30 seconds. SVM and SVM-neigh are the most into an e-learning tool and we will investigate new
computationally demanding methods, but the image features to improve the SVM method results.
manual interaction is not very high. In this case,
the total time is about 2 minutes.
In Figure 4, we show three different cases that ACKNOWLEDGMENTS
represent different wound shapes: a regular one,
an irregular one with concavities, and a lengthened This work was supported by the Spanish Govern-
one, respectively. From these results we can observe ment (Grant No. TIN2013-47276-C6-1-R) and by
that the the snakes method performs better with the Catalan Government (Grant No. 2014-SGR-
regular shapes. Snakes method assumes a smooth 1232). Guanghua Zhang received financial sup-
contour and, when the wound has an irregular or port from the European Commission (Erasmus
lengthened shape, the method does not achieve Mundus project Techno, ref. 204323-1-2011-1-FR-
correct results. Region growing method has a bad EMA21).
performance in all the cases which is due to irregu-
larities on the lesion colour but not on the shape.
Grabcut method has a good performance in all the REFERENCES
cases and irregular shapes do not affect the results.
Moreover, the interaction is almost the same, inde- Ahmad-Fauzia, M., I. Khansab, K. Catignanic,
pendently of the wound shape. If we compare SVM G. Gordillo, K. Send, & M. Gurcan (2015). Computerized
and SVM-neigh, we can observe that in all the segmentation and measurement of chronic wound
cases considering neighbour information leads to images. Computers in Biology and Medicine, 74–85.
Bilgin, M. & U.Y. Gunes (2014). A comparison of 3
better results. Although SVM and SVM-neigh are wound measurement techniques. J Wound Ostomy
not affected on the contour shape themselves, in Continence Nurs 40(6), 590–593.
the case of lengthened shapes, the pixels of wound Bradski, G. (2000). The OpenCV Library. Dr. Dobb’s
extremes are misclassified. This is due to the fact Journal of Software Tools.
that the distance to the central point of manual Burns, M. J. Enderle, E. Rosow, & Q. Zhu (2002). Devel-
interaction is considered as a SVM feature. opment of a wound assessment system for quantita-
tive chronic wound monitoring. In Proc. IEEE Annual
North-east Bioeng. Conference.
5 CONCLUSIONS Cortes, C. & V. Vapnik (1995). Support-vector networks.
Machine Learning 20(3), 273–297.
Gethin, G. (2006). The importance of continuous wound
The standard practice for the treatment of wounds measuring. Wounds 2(2), 60–68.
includes monitoring the size of the wound at regu- Gethin, G. & S. Cowman (2006). Wound measurement
lar intervals. Different methods based on image- comparing the use of acetate tracings and visitrak dig-
processing techniques have been proposed to ital planimetry. J Clinical Nursing 15, 422–27.

249

VIPIMAGE15_Book.indb 249 9/8/2015 7:45:50 PM

 Gonzalez, R. & R. Woods (2006). Digital Image Process- Sprigle, S., M. Nemeth, & A. Gajjala (2011). Iterative
ing (3rd Edition). Prentice-Hall. design and testing of a hand-held, non-contact wound
Kass, M., A. Witkin, & D. Terzopoulos (1988). Snakes: measurement device. Journal of Tissue Viability.
Active contour models. International Journal of Com- Song B. & A. Sacan (2012). Automated wound
puter Vision 1(4), 321–331. identification system based on image segmentation
Kolenik, M. & A. Fexa (2004). Segmentation of wounds and artificial neural networks.
in the combined color-texture feature space. SPIE Sundeep-Kumar, K. & B. Eswara-Reddy (2014). Wound
Med. Imaging 5370, 549–556. image analysis classifier for efficient tracking of
Krouskop, T., R. Baker, & M. Wilson (2002). A non con- wound healing status. Signal and Image Processing
tact wound measurement system. Journal of Rehabili- 5(2), 15–27.
tation Research and Development 39(3), 337–46. Thawer, H., P. Houghton, & G. Woodbury (2002). A com-
Langemo, D., J.A. J, D. Hanson, S. Hunter, & parison of computer-assisted and manual wound size
P. Thompson (2008). Measuring wound length, width, measurement. Ostomy Wound Manage (48), 46–53.
and area; which technique? Adv Skin Wound Care Veredas, F., H. Mesa, & L. Morente (2010). Binary tissue
21(1), 42–47. classification on-wound images with neural networks
Mayrovitz, H. & L. Soontupe (2009). Wound areas by and bayesian classifiers. IEEE Trans. Medical Imaging
computerized planimetry of digital images: accuracy 29(2), 410–27.
and reliability. Advances in skin and wound care 22(5), Weber, S., N. Watermann, J. Jossinet, J. Byrne, J. Chantrey,
222–229. S. Alam, K. So, J. Bush, S. OKane, & E.T. McAdams
Plassman, P. & T. Jones (1998). Mavis: a non-invasive (2010). Remote wound monitoring of chronic ulcers.
instrument to measure area and volume of wounds. IEEE Trans. Inf. Technol. Biomed 14(2), 371–77.
Medical Engineering and Physics 20(5), 332–38.
Rother, C., V. Kolmogorov, & A. Blake (2004). Grab-cut-
interactive foreground extraction using iterated graph
cuts. ACM Transactions on Graphics (SIG-GRAPH)
23(3), 309–314.

250

VIPIMAGE15_Book.indb 250 9/8/2015 7:45:50 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Automated image segmentation based on multiobjective optimization

and machine learning

S.M. Shontz
The University of Kansas, Kansas, USA

J.S. Tahara
BSMP Student, The University of Kansas, Kansas, USA
BSMP Student, The University of Sorocaba, Sorocaba, Brazil

D.O. McLaurin
CD-adapco

D.J.L. Colbry
Michigan State University, Michigan, USA

B. Parikh
The University of Kansas, Kansas, USA

ABSTRACT: Image segmentation is a challenging problem due to sensor-based noise, artifacts caused
by motion, and partial obstruction of objects. Although numerous algorithms for image segmentation
have been invented, there is no single algorithm which can be used to successfully segment all images.
Similarly, there is no fully automated image segmentation tool despite recent progress in this direction.
In this talk, we present a machine learning tool for automated segmentation of a set of images. Our
technique first uses an existing image segmentation algorithm to semi-automatically segment the first
image in the data set. It then proceeds by using our multiobjective optimization technique to match the
segmented shapes in the initial image to their most similar shapes in the second image in the data set.
Optimal shapes are determined based upon factors such as color, geometry, motion, and texture. This
process is then repeated in order to segment the remaining images in the data set. Machine learning is used
throughout the optimization procedure in order to determine the most appropriate values for the weights
of the relevant terms in the objective function for a set of images. We present results obtained from testing
our algorithms on various sets of biological and medical images. In addition, we present results on the
scalability of our method.

Keywords: automated image segmentation; multiobjective optimization; machine learning; unsuper-

vised algorithm

1 INTRODUCTION ing methods, and algorithms based on compres-

sion) have been invented for image segmentation;
Image segmentation is the process of dividing pixels however, there is no single algorithm which can be
in an image into categories based on shared visual used to successfully segment all images. Similarly,
characteristics. Such characteristics may include there is no fully automated image segmentation
the color, geometry, shape, texture, or other key tool despite recent progress in this direction (e.g.,
features. Image segmentation is often an important Chuang 2014, Walsh & Skala 2014).
first step for applications in that the process parti-
tions the image into subimages that are easier to
analyze. Image segmentation is a challenging prob- 2 METHODOLOGY
lem due to sensor-based noise, artifacts caused by
motion, and objects being only partially visible due In this talk, we present a machine learning tool for
to obstructions. Numerous algorithms (e.g., level set automated segmentation of a set of images. Our
methods, model-based methods, graph partition- technique first uses an existing image segmentation

251

VIPIMAGE15_Book.indb 251 9/8/2015 7:45:50 PM

 algorithm to semi-automatically segment the first
image in the data set. The method then proceeds by
using our multiobjective optimization technique to
match the segmented shapes in the initial image to
their most similar shapes in the second image in the
data set. Optimal shapes are determined based upon
factors such as color, geometry, motion, and texture.
A segmentation for the image is achieved by construct-
ing a weighted combination of these segmentations.
For example, in order to segment the image
according to color, the images are converted from
RGB to HSV. Three separate segmentations of the
image are obtained according to the hue, satura-
tion, and value. The final image which is segmented
Figure 1. Input flower image to be segmented.
according to color is given according to the follow-
ing formula

f w1, w2 , w3 ) = w1(hue
( huenew huessegmented
egmented )
2

+ w2 ( saturationnew
− satura
r tionsegmented )2
+ w3 (valuenew
− valuesegmented )2 .

Here a least-squares formulation is used to

determine the segmentation according to hue, sat-
uration, and value, and w1, w2, and w3 are weights
which yield a final segmentation of the image
based on a convex combination of the individual
segmentations. This least-squares formulation can Figure 2. Input flower image which is the basis for the
be extended in a straightforward manner in order multiobjective optimization.
to obtain segmentations of the images according
to other factors such as geometry, motion, and
texture.
This process is then repeated in order to segment
the remaining images in the data set. Machine
learning is used throughout the optimization pro-
cedure in order to determine the most appropriate
values for the weights of the relevant terms in the
objective function for a set of images.

3 EXPERIMENTAL RESULTS

We present some of the results of our automated

image segmentation based upon the color of the
image. Figure 1 shows an input flower image which Figure 3. Manual segmentation for the flower image
is to be segmented according to color. Figure 2 and shown in Figure 2. The manual segmentation is also
its manual segmentation are used as input to the input to the multiobjective optimization.
automated image segmentation algorithm. The
segmentations of the flower image in Figure 1 that
are generated according to the hue, saturation, and Note that while the structure of the flower in
value are shown in Figures 4, 5, and 6. The final Figure 7 is recognizable, its segmentation can be
segmentation of the flower image which results improved using other factors, such as geometry,
from setting w1 = 0.3, w2 = 0.3, and w3 = 0.4 in the motion, and texture, as the basis for the segmenta-
multiobjective optimization formula is shown in tion. Incorporation of machine learning can also
Figure 7. be used to improve upon the results.

252

VIPIMAGE15_Book.indb 252 9/8/2015 7:45:50 PM

 Figure 4. Segmentation of flower image in Figure 1 Figure 7. Final segmentation of flower image in
according to hue. Figure 1.

REFERENCES

Charbonneau, A. 2013. Image provided as part of a study

to quantify phenotypic differences between weedy and
native radish populations, Michigan State University,
Work in progress.
Chuang, H.-C. 2014. Automated Image Segmentation
for Analysis of Materials Microstructures, Ph.D. The-
sis, Purdue University).
Cocosco, C.A. et al. 1997. BrainWeb: Online interface
to a 3D MRI simulated brain database, Neuroimage
5(4): 2/4, S425, Proc. of the 3rd International Confer-
ence on Functional Mapping of the Human Brain,
Copenhagen. See http://www.bic.mni.mcgill.ca/brain-
web/.
Figure 5. Segmentation of flower image in Figure 1 Dyer F. 2013. Image provided as part of a behavior study
according to saturation. of eye movements. The most recent publication is
Ingle et al. 2012.
Ingle N. et al. 2012. Coordination of Vision and Action
in Chameleons, Proc. of the 49th Animal Behavior
Society Annual Meeting, Albuquerque, USA.
Walsh, A.J. and Skala M.C. 2014. An automated image
processing routine for segmentation of cell cytoplasms
in high-resolution autofluorescence images, Proc.
SPIE 8948, Multiphoton Microscopy in the Biomedi-
cal Sciences XIV, 89481M.
West J. 2004. Application of the Level Set Method to
Hydrocephalus: Simulating the Motion of the Ventri-
cles, M.S. Thesis, University of Waterloo.

Figure 6. Segmentation of flower image in Figure 1

according to value.

In our talk, we will present additional results

obtained from testing our automated image segmen-
tation tool on various sets of biological (Charbon-
neau 2013, Dyer 2013, Ingle et al. 2012) and medical
images (Cocosco et al. 1997, West 2004). In addition,
we present results on the scalability of our method.

253

VIPIMAGE15_Book.indb 253 9/8/2015 7:45:53 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Fractal descriptor on holographic images of cervical cells

Mona Mihailescu, Eugen I. Scarlat & Irina Alexandra Paun

Applied Sciences-Physics Department, University Politehnica Bucharest, Bucharest, Romania

Irina Grigorescu
Computer Science and Information Technology Department, University Politehnica Bucharest, Bucharest, Romania

Roxana Radu
LOTUS Hospital SRL, Romania

Oana Tatiana Nedelcu

Institute of Microtechnology, Bucharest, Romania

ABSTRACT: This paper proposes a method to statistically characterize the proprieties of cervical cells
extracted from samples labeled as normal and abnormal such as the ones affected by pathological modi-
fications or subjected to various processes like electroporation. The characterization is based on fractal
descriptors applied on 3D images obtained employing digital holographic microscopy technique. Fractal
descriptions are using the measures of the dilated volume and surface of the holographic phase images of
the cells. The method is a promising tool to discriminate among the normal and abnormal cervical cells
by exploiting the whole image of the cell not only limited zones or random cross sections, with potential
to be automatically implemented.

1 INTRODUCTION (Mihailescu et al. 2011), leukocytes classification

(Scarlat et al. 2013), analysis of red blood cells
The scope of this study is to characterize the 3D stored in standard conditions for different periods
phase images of the cervical cells using fractal (Moon et al. 2013, Bhadury et al. 2014), of stressed
descriptors. These images are obtained after recon- and unstressed red blood cells (Park et al. 2011,
struction of the holograms recorded using Digital Mihailescu et al. 2012), of anisocytosis stages in a
Holographic Microscopy (DHM) technique. red blood cells population (Mir et al. 2013).
Various methods are currently in place to dis- The difficulties appear when optical changes
criminate distinct species of cells, or to character- from different sources are mixing in a single phase
ize slightly different features inside the same cell map thus shadowing the cause under investigation.
species. The classical techniques are based on chro- For example, there are cells with no well-defined
matographic markers able to identify the key differ- nucleus shape or with no nucleus at all so that such
ences between cells such as to separately counting silhouette is not sufficient for in-depth analysis. In
the subspecies of interest and to establish, for these cases more complex instruments are needed
example, the differential leukocyte formulae, the from both the perspective of image taking devices
weight of immature red blood cells, or the ratio and the processing software. For example, cervi-
of normal to abnormal cells that help the clinical cal cells subjected to electroporation process are
diagnostic in the Papanicolaou (PAP) test, etc. suffering optical changes at spatial scales that are
Recently, the chromatographic techniques are differing from the ones caused by pathological
going to be replaced with the more sophisticated reasons.
methods based on phase imaging, with the advan- Fractals are powerful tools dedicated to finding
tage of using no markers; this aspect is preferred the intricate rules between each pixels of the cell
because the influence of the markers on the cells image and its relevant optical and morphological
characteristics is partially unknown. These images features. Fractal descriptor (Bruno et al. 2008) is
achieved enough spatial resolution to reveal sub- the instrument which uses not a single (mean) value
cellular features in order to help the wholeblood of the fractal dimension over the whole scale but
analysis (Marquet et al. 2005), red blood cell iden- a set of values over distinct portions of the scale.
tification like mature-immature red blood cells For any spatial surface expressing the values of the

255

VIPIMAGE15_Book.indb 255 9/8/2015 7:45:54 PM

 fractal dimension computed under different scales with holograms acquisition in a home-made
of analysis, these descriptors provide a complete experimental setup based on the Mach-Zehnder
mapping of the complexity of the image, from the interferometer with additional identical objectives
local details until the global pattern distribution. in both arms. The angle between the reference and
Here, the descriptors use algorithms to quan- the object beams could be trimmed to improve the
titatively describe the properties embedded in lateral resolution Δrrl while maintaining a conven-
the content of 3D reconstructed digital image ient depth range (Wang et al. 2008)
of the cervical cell such as to ensure the detec-
tion of the relevant optical changes that allow for 2λ
grouping them in normal and abnormal cells. Δrrl = , (1)
πA

where A is the numeric aperture and λ is the laser

2 CERVICAL CELLS wavelength. A CCD camera (Pike F421, 7.4 μm
pixel pitch, 2048 × 2048 pixels) grabs the hologram
The cervical cells are harvested in hospital con- and send it to the computer unit.
ditions (LOTUS Hospital), respecting ethical In the case of our system, we used 40× objec-
and legal regulations, and hygienic rules as well. tives with numerical aperture of 0.6 at a wave-
Cervical cells harvested through the standard pro- length of λ = 632.8 nm. The transverse resolution
tocol are subjected for liquid cytology PAP testing is, as in classical optical microscopy, diffraction-
performed using the PAP Cyto fast kit. Typical nor- limited (Cuche et al. 1999). For the present setting,
mal samples consist of cervical cells which include a lateral resolution of Δrrl = 0.7 μm was achieved in
strongly-individualized nuclei with small dimen- good agreement with the Rayleigh’s criterion for
sions (4–10 μm) as compared to the whole cell coherent illumination (Born et al. 1999). It is dem-
(30–80 μm) and with significant coloration with onstrated that the lateral resolution is independent
respect to the embedding cytoplasm. In Figure 1 on the recording distance (Doblas et al. 2015).
are the images of a normal and an abnormal cell The reconstruction of the 3D object image is
marked with chromatographic agents, recorded in performed digitally using an algorithm based on
bright field microscopy. It is clearly observed that the scalar diffraction theory, Fresnel approxima-
the abnormal cells have larger nucleus than the tion (KoalaR). In Figure 2 left are presented the
normal ones, the coloration intensity is weaker, holograms at the level of single cell for a normal (a)
and they exhibit a hallo around the nucleus (as and an abnormal cell (b) where the fringe displace-
is depicted in standard procedure (Solomon et al. ment is proportional to the phase shift introduced
2002)).
According to the PAP criteria the result would
be negative or positive depending on the number
and the weight of the abnormal cells identified in
the smear.

3 HOLOGRAM AQUISITION
AND RECONSTRUCTION

Digital Holographic Microscopy (DHM) technique

allows for 3D imaging of cells in their natural,
marker free environment. The procedure starts

Figure 1. Typical normal cervical cell (a) and abnormal Figure 2. Hologram of a cervical cell (left) and the
cervical cell (b) in a marked smear on glass plate (Nikon corresponding phase image of the reconstructed object
Ti-U microscope, 40× objective). (right) for a) normal cell, b) abnormal cell.

256

VIPIMAGE15_Book.indb 256 9/8/2015 7:45:54 PM

 by the cell in the optical path relative to the sub- normal cells the cross section profile exhibits a sharp
strate. The 3D reconstructed phase maps of the peak corresponding to the refractive index of the
same cells are in the Figure 2 right. The axial reso- compact nucleus (Fig. 3a), differing from the cases
lution is given by the reconstruction numerical pro- of abnormal cells (Fig. 3b) with diffused nucleus
cedure which can simulate the Fresnel transform at inside the cytoplasm or modified membranes.
distances in nanometers range. Consequently, in While the diameters of the nuclei of the normal
the reconstruction process, DHM provides multi- cells are between 4–10 μm, for abnormal cells they
focus quantitative phase contrast imaging with an are larger than 15 μm. The range of the phase shift
axial resolution up to approximate 5 nm (Kemper introduced by the cells along the optical path is
et al. 2007). also different for normal and abnormal cells: it is
The holograms are processed taking into of the order of 100 degrees in the first case but no
account the whole area of the cell according to more than tenths of degrees in the second case. The
a standard procedure consisting in the follow- phase shift distribution over the cell profile accord-
ing steps: 1) numerical filtering of the diffraction ing to Figure 3 is assumed to be bimodal; in the
orders in the Fourier domain to drop off the twin present work the normal cells are considered when
image, 2) numerical filtering inside the first diffrac- the Ashmann’s coefficient DA is larger than two
tion order to eliminate the noise due to parasitic
fringes from the experimental setup, 3) tilt and DA > 2, (2)
aberration corrections, followed by three specific
steps devoted to this kind of cells, i.e. 4) profile such as the distributions are clearly separated
extraction and measurements (phase shift and (Ashmann et al. 1994).
diameters), 5) implementation and running of the Each holographic image H∈[1:p] × [1:q] → I
dilation and packing algorithm, and 6) statistical has different values for p, and q pixels number in
characterization of the cells. The last three steps transverse directions, but the same Imax = 256 phase
are implemented in MATLAB and are applied for levels. The phase image is converted in a three-
all cells following the same rules. dimensional surface S in such a way that each pixel
An enough large number of holographic images with coordinate (x, y) in the image is mapped onto
were grabbed and reconstructed from both normal a point in the 3D space with coordinates (x, y,
and abnormal samples, followed by the extraction of z(x,y)), where z is the pixel phase shift, that is:
1D cross sectional profiles passing through the cor-
responding nucleus (Fig. 3). One can observe that for S = {i, j, k(i, j)|(i, j) ∈ [1:p] × [1:q]}, (3)
and

k(i, j) = {1, 2, …, Imax}|k = H(i, j)}, (4)

where Imax corresponds to the maximum phase shift
in the image. The dimensionless quantities result
via the normalization of p, q and 2π respectively.

4 FRACTAL DESCRIPTORS

A fractal object is a geometrical set of points in a

multidimensional topological space exhibiting a
structure characterized by a theoretically infinite
degree of self-similarity, each part of the object
corresponding to a copy of the whole. The fractal
dimension is the most used measure to describe such
a fractal object. Since in the real-world the set of
points S is a digital images represented in a discrete
and finite space, several approximation methods
have been proposed in the literature (Falconer 1986).
Such methods aim to compute a measure of self-
similarity and complexity of the object by generaliz-
Figure 3. Cross sections through the phase map of a ing the definition of the Euclidean dimension. Thus
normal (a) and an abnormal (b) cervical cell. The section the object is measured by a rule unit with the same
plane is passing through the nucleus surrounded by the topological dimension of the object. The length r of
cytoplasm. this unit is ranged along an interval to compute the

257

VIPIMAGE15_Book.indb 257 9/8/2015 7:45:57 PM

 number of units N(r) necessary to cover the object.
In this context, the fractal dimension is given by

log N (r )
d ∝ − lim . (5)
r→0 log r

Fractal descriptors extend the fractal dimension

concept by computing d under different scales of
analysis. Here the descriptors exploit the fractal
properties of the volume VS(i,j,k) and the corre-
sponding external frontier surface ΣS(i,j,k) of the
digital holographic images S(i,j,k) when covering it
with structural elements as suggested by Falconer
Figure 4. The principle of covering the surface with
(Falconer 1986). disjoint balls.
Our analysis applies these descriptors on the 3D
reconstructed phase images containing single cell
structure. Simultaneous use of volume and surface
is necessary because distinct cells can exhibit the
same volume but different surfaces.
In the Bouligand–Minkowski method, any ele-
ment of S is dilated by a spheric ball s(r) with vari-
able radius r. For each r > 0, the set s(r) is defined
to be the r-neighborhood of S, i.e. the set of all
points in R3 which are at distance less than r from S
(or equivalently, s(r) is the union of all the spheres
of radius r which are centered at a point in S). The
dilation of S by sr is Sr = S ⊕ s(r ) with the dilated
volume Vr V (S ⊕ s r ) , and the dilated frontier
surface Σ r = Σ (S ⊕ s r ) .
Computationally, the dilating spheres s(r) are
centered on the pixels of S with increasing radii
following a scaling rule Figure 5. The principle of filling the volume with dis-
joint balls.
ρ = {rmin, r1, r2, …, rmax}, (6)

where the radii are computed according to the In the present work the descriptors (8) and (8′)
Euclidean metric are used together following a version of the method
used by Da Silva Oliveira et al. (Da Silva Oliveira
rijk i2 + j2 k2 , (7) et al. 2014):

with i, j, k given by Eqs. (3)–(4).

D∑V = ⎨
(
⎧⎪ log N Σ s rijk),
(
log NV s rijk ) ⎫⎪⎬. (9)
For the frontier case the dynamic of the fractal log rijk log rijk
descriptor DΣ is computed using Eq. (5) according ⎩⎪ ⎭⎪
to the number of disjoint balls of radii given by Eqs.
(6)–(7) that are covering the surface Σr (see Fig. 4): The algorithm for the packing of spheres onto
the surface or inside the volume initiates with ran-

( ) ⎫⎪⎬.
domly positioning of the first sphere followed by
⎧⎪ log N Σ s rijk
a neighborhood strategy. For the following posi-
D∑ = ⎨ (8)
log rijk tions concentric spheres around this first position
⎩⎪ ⎭⎪ are considered at increasing distances such as no
overlapping are allowed between the candidate
Similarly, the descriptor DV for volume case is element and the all other already placed spheres.
the number of disjoint balls filling the volume Vr If the candidate does not overlap with any of the
(see Fig. 5): previously placed spheres, then it will be kept and
the procedure moves forward with one step. The
DV = ⎨
(
⎧⎪ log NV s rijk ) ⎫⎪⎬. (8′)
algorithm stops when all possible candidates from
log rijk a given distance are outside the cell’s boundary.
⎪⎩ ⎪⎭ For every radius of the structural element the

258

VIPIMAGE15_Book.indb 258 9/8/2015 7:45:57 PM

 number of covering spheres is averaged over all
position of the first element. The averaged values of
the numbers of spheres at the given radius are pro-
vided automatically at the end of the algorithm.
The procedure was equally applied for all inves-
tigated cells. MATLAB codes were written sepa-
rately for surface covering and volume filling and
run for all values of the radii of spheres given by
Eq. (6). In this case, the background surrounded
the cell is set to zero values.

5 RESULTS Figure 7. Typical descriptor (volume case DV): normal

cells (green), abnormal cells (red), and pyramid shape
For the present study the sources of the normal cells (blue).
were the samples qualified as negative by the result
of the PAP smear analysis, while the sources of the
abnormals were the samples qualified as positive. Table 1. Estimated local slopes.
The study further focuses on single cell analysis per- 
formed on cells extracted from negative samples and Local slope (− μ ± σμ))
subsequently selected to fulfill Eq. (2); if so, they Descriptor/
cell type Small Big
were labeled as normal. Similarly the abnormal label
was assigned to the cells extracted from the positive DΣ/normal 2.105 ± 0.004 2.308 ± 0.003
samples that did not fulfill Eq. (2). 100 normal and DΣ/abnormal 2.139 ± 0.007 2.561 ± 0.012
100 abnormal cells were selected in this way. DV/normal 2.569 ± 0.006 2.037 ± 0.007
For every cell the local fractal dimension is com- DV/abnormal 2.703 ± 0.009 2.392 ± 0.020
puted by estimating the linear slope μ with the
variance table for the fitted model (ANOVA) under
the constraints of statistical significance of p-value
and R2 coefficient. By imposing p-value < 5% irrespective the descriptor, the cells behave con-
and R2 > 95%, the fractal descriptor DΣ (Eq. (8)) tradictory: the local slope increases at big scale in
for abnormal cells exhibits a critical point that the surface descriptor and decreases in the volume
splits the abscissa interval in two different regions descriptor. However, the quadratic approximation
with distinct slopes corresponding to small scale over the whole scale (small and big) is negatively
2√3 ≤ rijk ≤ 15√3, and big scale 15√3 < rijk ≤ 45√3 (see shaped in surface description and positively shaped
Fig. 6). A pyramidal shape is inserted as bench- in the case of volume description.
mark (the blue curves in Fig. 6 and Fig. 7). These The estimated local slopes for the investigated
regions remain valid for the volume descriptor DV regions are given in Table 1. All the data are presented
(Eq. (8′), see Fig. 7). as mean values and standard errors of the means σμ.
Differing from the pyramid shape which exhibits The trends of the mean values of the local frac-
a diminishing slope (in absolute values) at big scale tal dimensions are opposite in the descriptors when
moving from small to big scale, therefore the meas-
ures could be concatenated to improve the discrim-
inating power of the method.
Since we are interested in evaluating the differ-
ences among normal and abnormal cells, a popular
measure to address the issue is the Bhattacharyya
distance (Wallin et al. 2015). Assuming Gaussian dis-
tributions of slopes for
 both normal and abnormal
cells G ( μ ,σ ) with σ = 10σμ the distances increase
from small scale to big scale; the largest distance is
in the case of the surface descriptor, at big scale (see
Table 2). Conversely, since the smallest distance is
found at small scale in the surface description, it
would be an additional argument to concatenate the
Figure 6. Typical descriptor (surface case DΣ): normal descriptors as suggested by Eq. (9).
cells (green), abnormal cells (red), and pyramid shape The small distances at small scales are partially
(blue). the effect of noise that affects the images below the

259

VIPIMAGE15_Book.indb 259 9/8/2015 7:46:01 PM

 Table 2. Bhattacharyya distances REFERENCES
between the distributions of slopes
of normal and abnormal cells. Ashman K.M. et al., 1994, Detecting bimodality in astro-
nomical datasets, Astron. J. 108 2348–2361.
Scale Bhaduri B. et al., 2014, Optical Assay of Erythrocyte Func-
tion in Banked Blood, Scientific Reports, 4, (6211).
Descriptor Small Big Born M., Wolf E., eds., 1999, “Principles of Optics”,
(Cambridge University Press, 467–472.
DΣ 0.118 1.392 Bruno O.M. et al., 2008, Fractal dimension applied to
DV 0.416 0.928 plant identification, Information Sciences 178 (12),
2722–2733.
Cuche E. et al., 1999, Simultaneous amplitude-contrast
and quantitative phasecontrast microscopy by numeri-
diffraction limit where operate the coverings at the cal reconstruction of Fresnel off-axis holograms, Appl.
lower values of the radii in the vicinity of rmin (see Opt. 38, 6994–7001.
Da Silva Oliveira et al., 2014, Enhancing Fractal Descrip-
Eq. (6)).
tors on Images by Combining Boundary and Interior of
The distributions of the slope and consequently Minkowski Dilation, Physica A, 393, 2623–2635.
the distributions of the local fractal dimension are Doblas A. et al., 2015, Study of spatial lateral resolution
more compact in the case of normal cells in the in off-axis digital holographic microscopy, Opt. Comm.
sense of smaller kurtosis whatever the volume or 352, 63–69.
the surface description. Falconer K.J., 1986, “The Geometry of Fractal Sets”,
Cambridge Univ. Press, N.Y.
Kemper B. et al., 2007, Digital Holographic Microscopy
6 CONCLUSIONS A New Method for Surface Analysis and Marker-Free
Dynamic Life Cell Imaging. Biophotonics 2, 41–47.
KOALA http://www.lynceetec.com/content/view/497/193/.
The cervical cells were analyzed using an image Marquet P. et al., 2005, Digital holographic microscopy: a
processing method based on fractal descriptors in noninvasive contrast imaging technique allowing quan-
order to reveal the intricate elements able to classify titative visualization of living cells with subwavelength
the cervical cells in normal and abnormal categories. axial accuracy, Opt. Lett. 30, 468–470.
An algorithm based on fractal descriptors was devel- Mihailescu M. et al., 2011, Automated Imaging Identi-
oped and applied on the reconstructed holographic fication and Counting of Similar Cells From Digital
3D images which contain the phase shift introduced Hologram Reconstruction, Applied Optics, 50 (20),
by the cell in the optical path (unmarked cell). 3589–3597.
Mihailescu M. et al., 2012, Diffraction pattern study
Due to the sharp delimitation of the normal nuclei
for cell type identification, Optics Express, 20, (2),
evidenced in a bimodal distribution of the cross 1465–1474.
sectional phase profile, the dynamics of the local Mihailescu M. et al., 2010, Wavelet filters for program-
slope—equivalently, the local fractal dimension—in mable object investigation in digital holographic micros-
surface and volume descriptions of the cell images copy, Journ. Optoel. Adv. Mat., 12 (1), 94–99.
was presented. The surface descriptor at big scale Mir M. et al., 2010, Blood screenind using diffraction
and the volume descriptor at small scale have greater phase cytometry, J. Biomed. Opt. 15, 2, 027016, 2010.
sensitivity to phase changes exhibited by the abnor- Moon I. et al., 2013, Automated quantitative analysis of
mal cells, therefore is recommended to use them 3D morphology and mean corpuscular hemoglobin in
human red blood cells stored in different periods, Opt.
together. However the small-big scale partition and
Expr. (21), 25, 30947–30957.
the establishing a threshold for Bhattacharyya dis- Park YK. et al., 2011, Measurement of the nonlinear elas-
tance are subjects to more research. ticity of red blood cell membranes, Phys. Rev. E, 83,
The characterization of holographic images with 051925.
fractal descriptors is a promising tool to detecting Scarlat E.I. et al., 2013, Discriminating the main repre-
information about the phase changes of the cervi- sentatives of the white blood cell species on the basis of
cal cells to the advantage of taking into account the fractal properties of the DHM phase profile, UPB.
the entire cell not only a random cross section. Sci. Bull., A, 75 (2), 147–154.
Moreover the method is marker free and suitable Solomon D. et al., 2002, The 2001 Bethesda system.
terminology fo rreporting results of cervical cytology,
for automate analysis.
Jour. Americ. Medical Assoc., 287, 2114–2119.
Wallin, J. et al., 2015, Latent modeling of flow cytometry
cell populations. arXiv preprint arXiv:1502.04058.
ACKNOWLEDGEMENT Wang H. et al., 2008, Simple and robust digital hologra-
phy for high-resolution imaging, Chin. Opt. Lett., 6,
This research was financed by the Romanian Author- (3), 165.
ity for Scientific Research and Innovation contract
UEFIS-CDI, PN-II-PT-PCCA no. 30/2014.

260

VIPIMAGE15_Book.indb 260 9/8/2015 7:46:02 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

An integrated two Time-of-Flight camera system to determine knee

flexion movement: Comparison with standard motion analysis system

Emeric Veron-Tocquet
Laboratoire de Traitement de l’Information Médical, LaTIM-INSERM U1101, France
Université de Bretagne Occidentale, France

Valérie Burdin
Laboratoire de Traitement de l’Information Médical, LaTIM-INSERM U1101, France
Institut Mines Telecom-Telecom Bretagne, France

Joel Savéan
Laboratoire de Traitement de l’Information Médical, LaTIM-INSERM U1101, France
Université de Bretagne Occidentale, France

Julien Leboucher & Olivier Rémy-Néris

Laboratoire de Traitement de l’Information Médical, LaTIM-INSERM U1101, France
Université de Bretagne Occidentale, France
CHU Brest, France

ABSTRACT: Medical rehabilitation therapies depend on the accurate motion of bone structures,
which is often impossible to acquire without using invasive markers. Current clinical studies that evaluate
measurements of motion use skin markers even though its drawbacks due to skin motion are well-known.
Recent developments in motion analysis use markerless Time-of-Flight (ToF) cameras that can acquire
three dimensional data. This study was focused on integrating two synchronized ToF cameras to evaluate
knee flexion movement and compare the findings with standard Vicon system data. A rigid structure was
developed and the best relative location of ToF cameras on this structure relative to the patient was
analyzed. In the experimental process, the rigid structure was placed in the motion analysis lab and
standard skin markers were placed on the lower leg of a volunteer. The volunteer performed knee flexion
movements while lying on a bed and knee flexion data was simultaneously acquired by both the Vicon
system as well as the integrated ToF cameras during nine camera placement combinations. The RMS
error between the two measurement methods ranged from 5.61° to 9.64° depending on camera locations.

1 INTRODUCTION of the optoelectronic cameras system is time con-

cerning for clinicians and patients.
Motion analysis is used in many cases of Recently, new technologies have appeared for
rehabilitation: comprehension of bone deformities, acquiring scene in 3 dimensions and without
diagnosis of a deficient function, post-surgical markers. This include structured light cameras,
follow-up evaluation. Methods that provide motion multi-view RGB cameras, and Time-of-Flight
information are mainly based on marker position (ToF) cameras. The ToF cameras system is an
determined by an acquisition system. One of the interesting way to acquire dynamic movement
most popular, in clinical diagnosis, is the optoelec- with acceptable accuracy compared with RGB
tronic system (e.g. Vicon motion analysis system). cameras (low accuracy) [4] and structured light (no
This kind of system acquires 3D positions of each dynamic acquisition) [5]. A preliminary study for
marker placed, on the patient’s skin representing determining knee flexion movement with a single
anatomical landmarks. These methods provide an ToF camera provided encouraging results [6]. In
accuracy within 1 mm [1] for marker position but this study, angle differences were compared between
skin artifacts appear due to the relative displace- flexion angle measured with single ToF camera and
ment between bones and skin [2], [3]. Furthermore, flexion angle calculated with actual bones positions
patient instrumentation and preliminary calibration on a cadaveric lower limb. The maximum difference

261

VIPIMAGE15_Book.indb 261 9/8/2015 7:46:02 PM

 calculated was 9° and the RMS, for three trials, was aluminiboxes to receive the plane lasers (Fig. 2).
8.03°, 5.90° and 5.60°. Reinschmidt and colleagues We installed them to obtain
 the vertical and hori-
[7] compared bone locations with the segment zontal axis of cameras (x and y ) when laser were
location by markers on the skin to estimate flexion projected on a wall. Lasers had two degrees of
angle. They found RMS error equal to 5.3° with a freedom in rotation, by this way we were able to fix
maximal difference of 7.9°. them in the right position to obtain a cross formed
In the current study, the surface coverage of the by the two projected line lasers. The center of the
ToF was increased by using a synchronized two- cross corresponded the center of the camera.
ToF camera system. The aim of this study was to The alignment of cameras was a two-step
determine the accuracy of knee flexion movement process. First the alignment of the center of the
using two-ToF system as well as to bring out the two cameras was performed (Fig. 3-b) followed by
best position of cameras in reference with each the alignment of camera axes (Fig. 3-c).
other and to the patient.
2.2 Experimental process
2 MATERIALS AND METHODS A one single healthy male volunteer participated
in this study. The volunteer was laid supine on a
A structure was build to mount the ToF cameras in bed that was placed under the structure with the
order to acquire the knee flexion movement in the
transvers plane. The structure was made of metal
ladders to which a one-meter-radius semi circle bar
was attached (Fig. 1-a). This arc was marked with
angle graduations in order to mark the position of
the two ToF-cameras (Fig. 1-b, where the angle α
represented angle between the two cameras). Each
camera was fixed on a 3-dimensions articulated
pliers, which permitted to orient them in the
same plane (plane formed by the arc). Figure 1-c
represents the operating structure with the angle
between cameras (α) and the angle between the
two arcs (θ, that represents the position of the
two-cameras system relative to the patient).

2.1 Laser calibration method

We developed a system based on plane lasers
to calibrate the cameras together. We created
Figure 2. Laser in their box with two degrees
 of freedom

in rotation, which permitted to align the x and y of
cameras and obtain the center of camera by the cross
performed by them.

Figure 1. a) Ladder and the static semi-circle fixed

on. b) Definition of α angle between the two cameras.
c) Represents the operating structure with the angle
between cameras (α) and the angle between the two Figure 3. Method to align the two cameras together.
arcs (θ, that represents the position of the two-cameras a) Cameras not aligned. b) Center of cameras aligned.
system relative to the patient). c) Axes and center of cameras aligned.

262

VIPIMAGE15_Book.indb 262 9/8/2015 7:46:02 PM

 knee located directly under the plan of arc. Using
this set-up, nine tests were performed using a
combination of α and θ in degrees:

α { ; ; } d θ ={ ; ; }.

In each test, two consecutive flexion movements

were performed. To obtain the same coordinate
system for the two cameras views, rigid transfor-
mations were applied. One rotation of α angle to Figure 4. Left: visualization and identification of
obtain in the same coordinate system clouds points markers position captured by Vicon system. Right: flexion
and a translation corresponding to the translation angles calculated with Vicon: (α vicon ).
between the two cameras.

2.3 Vicon system measurement

In addition to the ToF camera measurements, the
volunteer was instrumented with marker visibility
to follow the flexion movement with the Vicon
optoelectronic system. The Vicon system was
as our gold standard. Markers were placed on
thigh (3 femur markers), calf (3 tibia markers),
ankle (2 markers), knee (2 condyle markers) and
Figure 5. Shows, in the sagittal plan, these two points
hip (2 markers). The frequency of acquisition was
clouds in the same coordinate system, after filtering
120 Hz. to delete uninteresting points. Two colors appear,
corresponding to points acquired by each camera. Left:
view of the two points clouds in the sagittal plan. Camera
2.4 Synchronization of the two systems 1 in red and Camera 2 in blue. Right: flexion angles along
Because of difficulties to synchronize the Vicon time calculated with two-ToF cameras: (α ToF ).
system with any other systems which is not
designed by the same company, an in-house system 3.2 Data from the two-ToF cameras
for synchronization was created. Inspired from the
synchronization method in movie studio in order ToF cameras gave two different views of the
to synchronize sound and image, an inflatable bal- leg during flexion movement. Each view was
loon was used for this purpose. A reflective marker represented by a point cloud. After rigid rotation
was stuck in the center of the hand of a manipula- and translation, depending on α and θ values, two
tor and the balloon was put on it. The manipulator point clouds in the same coordinate system were
placed his hand with the marker and the balloon obtained. A first filtering was applied to these sets
under the structure. The balloon was visible by of points to only keep points corresponding to the
ToF cameras and covered the marker visibility moving leg. By this mean we removed all other
from Vicon cameras. As soon as the volunteer points, corresponding to the bed, the ground, the
started moving his legs, the manipulator poked ladders, and the other leg which did not move
the ballon with a pin. The ballon disappeared from (Fig. 5).
the ToF camera views and the reflective marker A second filtering was applied on the point
became visible by the Vicon system. With this cloud corresponding to the leg. This point cloud
method, an accurate start signal was obtained for was split in two different point clouds set corre-
each acquisition system and their synchronization sponding to the femur and the tibia. From each
was possible. of them the principal inertial axis was determined
followed by the angle between these two axes (cor-
responding to femur and tibia) to obtain flexion
angles measured by ToF cameras, α ToF .
3 RESULTS

3.1 Data from the Vicon system 3.3 Measurement errors

Using skin markers of tibia and femur, the position Using the synchronization method between the
and orientation of both tibia and femur segments Vicon system and ToF camera system, it was pos-
were known (Fig. 4). Using this information the sible to compare the flexion curves as calculated
angle of flexion α vicon was plotted against time. by two systems (Fig. 6). The curv e of differences

263

VIPIMAGE15_Book.indb 263 9/8/2015 7:46:09 PM

 economical option for future clinical or surgical
procedures. RMS errors obtained were highly
dependent on camera positions and provided
valuable insights in determining the optimal camera
positions during in vivo assessments. When {α;
θ} = {120; 60}, the camera number 1 was on the left
side of the leg, which resulted in lowest RMS error.
A low angle of incidence between infrared lights
of the ToF and the acquired surface decreased the
accuracy of measurements [6]. This phenomenon
was confirmed by Chibrando and colleagues [8]
who reported inacceptable angle measurements for
incident angles smaller than 45°. When a camera
was on one side of the leg, incidences of rays were
constant during all the movement and close of 90°.
When a camera was over the leg with a flexion
close to 140° of flexion, the incidence of light rays
became smaller than 20° due to the orientation of
femur and tibia and the angle measurements were
Figure 6. Comparison of flexion angles measured, less accurate: for example, at {α; θ} equal to {60;
α = 60° and θ = 0°. 30}. Thus, keeping the ToF camera on one side of
the leg permitted us to obtain accurate results. The
second camera was needed to increase the acquired
Table 1. Error and RMS error calculated between ToF surface and to increase the measurement accuracy.
cameras system and Vicon system for different positions Minimum and maximum errors varied from
of ToF cameras. −13.05° to 15.21° but they were extrema values.
(α; θ) [Min; Max] (deg) RMS Error (deg) No correlation was found relative to the cam-
eras position. As it appeared on Figure 6, the
(60; 0) [−11,82; 14,36] 6.19 high values of error appeared when the maxi-
(60; 30) [−11,67; 15,21] 9.64 mum of flexion was reached (140°) and when the
(60; 60) [−13,05; 14,21] 9.10 leg returned to the extension. To understand the
(90; 0) [−10,75; 12,53] 5.73 errors better, knee movement was split into flex-
(90; 30) [−11,26; 14,49] 6.22 ion and extension cycles and data in each cycle
(90; 60) [−10,89; 14,19] 7.15 was analyzed separately. For all the nine tests, the
(120; 0) [−8,66; 10,32] 5.82 minimum and maximum errors calculated dur-
(120; 30) [−11,10; 13,74] 7.32 ing the flexion phase (0° to 140°) were −7.34° and
(120; 60) [−9,21; 9,83] 5.61 6.42°, respectively for test {α; θ} = {90; 30} and {60,
60}. These values were smaller than those obtained
during the extension (Min and Max in Table 1 were
obtained during extension cycles). We justify this
between α ToF and α was also plotted (Fig. 6) difference between flexion and extension phases,
Vicon
for each test. by the muscle contraction effect. During the move-
Furthermore, the difference error at each time ment, quadricep was active during extension phase
interval and the RMS error for the entire flexion and the hamstring was passive. During the flex-
curves was also determined, which was more rep- ion, hamstring became active and quadricep pas-
resentative of accuracy of our system. sive. Thus, the surface shape was greatly modified
This analysis was performed for each test with between both the movement phases. The differ-
different values of α and θ angles between ToF ence in the point clouds shape could probably have
cameras and between cameras and patient. The impacted the calculus of femur and tibia axes. For
results are summarized in Table 1. the Vicon system, the skin markers were probably
impacted by the soft tissues artifacts.

4 DISCUSSION
5 CONCLUSIONS
This study successfully implemented an integrated
ToF system to determine knee flexion angle and This markerless ToF camera system provides
in doing so illustrated the feasibility of using mul- valuable insights and feasibility in its use as a
tiple ToF camera system as a viable, quick, and rehabilitative diagnostic tool. The best RMS

264

VIPIMAGE15_Book.indb 264 9/8/2015 7:46:11 PM

 errors for this system were low when best camera [3] A. Leardini, L. Chiari, U. Della Croce, A.
positions were considered. These errors (5.61) were Cappozzo (2005), “Human movement analysis using
in fact in line with the known accuracy of the opto- stereophotogrammetry, Part 3: Soft tissue artifact
electronic system (5.3) with skin motion artifact assessment and compensation”, Gait & posture,
21(2):212–225.
[7]. Thus, results found in this study encouraged us [4] German K.M. Cheung, Simon Baker, and Takeo
to perform test on cadaveric leg with intracortical Kanade (2003), “Shape-from-silhouette of articu-
pins in femur and tibia followed by a Vicon system. lated objects and its use for human body kinematics
This way, the gold standard system will be the best estimation and motion capture”, In Proceedings of the
in order to acquire the actual positions of bones IEEE computer society conference on Computer Vision
and to compare with flexion angles measured by and Pattern Recognition (CVPR’03), Washington
our two-ToF synchronized cameras system. DC, 77–84.
[5] C. Rocchini, P. Cignoni, C. Montani, P. Pingi and
R. Scopigno (2001), “A low cost 3D scanner based
on structured light”, Computer Graphics Forum,
ACKNOWLEDGEMENT 20(3):299–308. DOI: 10.1111/1467–8659.00522.
[6] E. Veron-Tocquet, J. Leboucher, V. Burdin, J. Savean,
We would like to thank Bhushan Borotikar, PhD O. Remy-Neris (2014), “A study of accuracy for a
for the assistance in the language corrections and single Time of Flight camera capturing knee flexion
to Mathieu Lempereur, PhD for the use of the movement”, In Proceedings of the IEEE Healthcare
Vicon system. Innovation Conference (HIC), Seattle WA, 339–342.
DOI: 10.1109/HIC.2014.7038944.
[7] C. Reinschmidt, A.J. Van Den Bogert, B.M. Nigg,
REFERENCES A. Lundberg, N. Murphy (1997), “Effect of skin
movement on the analysis of skeletal knee joint
motion during running”, Journal of Biomechanics,
[1] M. Windolf, N. Gtzen, M. Morlock (2008),
30(7):729–732.
“Systematic accuracy and precision analysis of
[8] F. Chiabrando, D. Piatti, F. Rinaudo (2010),
video motion capturing systems–exemplied on
“SR-4000 ToF camera: further experimental tests
the Vicon–460 system”, Journal of Biomechanics,
and first applications to metric surveys”, Intern.
41(12):2776–2780.
Archives of Photogrammetry, Remote Sensing and
[2] F. Leboeuf, C. Schwartz, S. Brochard,
Spatial Information Sciences, 38(5):149–154.
M. Lempereur, V. Burdin, O. Remy-Neris (2012),
“Relevance of skin deformation to track the scap-
ula during forward humeral elevations”, Computer
Methods in Biomechanics and Biomedical Engineer-
ing, 15(1):365–367.

265

VIPIMAGE15_Book.indb 265 9/8/2015 7:46:11 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Mechanics–based analysis of the left atrium via echocardiographic

imaging

S. Gabriele, L. Teresi & V. Varano

Università Roma Tre, Roma, Italy

P. Nardinocchi, P. Piras, G. Esposito, P.E. Puddu & C. Torromeo

Sapienza Università di Roma, Roma, Italy

A. Evangelista
Ospedale San Giovanni Calibita Fatebenefratelli—Isola Tiberina, Roma, Italy

ABSTRACT: A mechanics–based analysis of data from three-dimensional speckle tracking echocardi-

ography can help to infer useful clinical data about the state of myocardium. Precisely, we recently conjec-
tured that principal strain lines in the left ventricle change in presence of heart disease, and used data from
healthy subjects and patients to support our conjecture. Here, we set the first steps of our analysis relative
to the left atrium, describing our method of acquisition, and discussing the key mechanical characteris-
tics delivered by the echocardiographic device for healthy subjects and patients affected by hypertrophic
cardiomyopathy.

1 INTRODUCTION and discussed for the human LV to the correspond-

ing LA. Precisely, as for any LA of our database,
The recent techniques of visualization realized we also have the corresponding LV, and both eco-
by Three–Dimensional Speckle Tracking–based cardiographic data are acquired in the same cardiac
motion–detecting Echocardiography (3DSTE) cycle, we can plan a crossed analysis which can take
make possible, non invasively, to generate a class advantage of the knowledge of the mechanics of
of data which can be studied through a post– both LA and LV chambers of the heart. We focus
processing mechanics–based analysis. In the last the analysis on LA data corresponding to healthy
years, our group proposed two different methods subjects and Hypertrophic Cardiomyopathy (HCM)
of analysis, which may also be integrated one patients, and preliminarily discuss the mechanics of
with the other: the first generates a Matlab–based LA on the basis of the mechanical variables deliv-
tool denominated S–Lab which can automatically ered by the echocardiographic device.
analyze a large amount of data and identify both Next steps involve the application of our S–Lab
principal strain lines and values in the epicardial tool on the acquired LA, to set a crossed analy-
and endocardial layers of the Left Ventricle (LV); sis based on the endocardial and epicardial prin-
the second borrows tools from Modern Shape cipal strain values in healthy subjects and HCM
Analysis and investigates human left ventricular patients.
heart morphological changes in time among a
group of subjects [1, 2, 3, 4]. Both the analyses
aim to identify indicators of early LV dysfunc- 2 METHOD
tions, which are only based on non–invasive
techniques. The technique is based on data obtained by 3DSTE
As it is well known, also Left Atrium (LA) func- which is an application of pattern–matching tech-
tion is an important clinical variable, and recent nology to ultrasound cine data, based on the track-
studies conferred credit to echocardiographically ing of the ‘speckles’ in a 3D volume; speckles are
derived LA mechanics, to non invasively assess LA disturbances in ultrasounds caused by reflections
compliance, and asked for the development of spe- in the ultrasound beam: each structure in the body
cific software for LA analysis [5]. has a unique speckle pattern that moves with tis-
We wish to make some progress towards address- sue (see [4, 6] for further details). As when LV is
ing this question, extending the analysis proposed the object of the study, the operator manually

267

VIPIMAGE15_Book.indb 267 9/8/2015 7:46:11 PM

 approves LA shape and identifies the endocardial
border starting from the mitral annulus. From
here, the acquisition process may start automati-
cally through the identification of the epicardial-
border of the LA wall, and of the 16 segments
which portend the LA. The segments numbered
from 1 to 6 correspond to the basal segments:
Basal Antero–Septum (BAS), Basal Anterior (BA),
Basal Lateral (BL), Basal Posterior (BP), Basal
Inferior (BI), Basal infero–Septum (BS); the seg-
ments from 7 to 12 to the middle segments: Middle
Antero–Septum (MAS), Middle Anterior (MA),
Middle Lateral (ML), Middle Posterior (MP),
Middle Inferior (MI), Middle infero–Septum
(MS); the segments from 13 to 16 to the apical seg-
ments: Apical Anterior (AA), Apical Lateral (AL),
Apical Inferior (AI), Apical Septal (AS).
Figure 1. Wall–motion tracking. The position of many
speckle patterns is tracked along a cardiac cycle; those
2.1 Subjects data are then processed to evaluate specific characteristic
The study was performed under the approval of of the overall LA motion. Here, the field of longitudinal
the review board of “Dipartimento di Scienze strain is displayed in a 5 plane view; color-code scale rep-
Cardiovascolari, Respiratorie, Nefrologiche, resents longitudinal strain, positive.
Anestesiologiche e Geriatriche, Sapienza Univer-
sità di Roma”, and in accordance with the ethical
guidelines of the Declaration of Helsinki. Written 2.2 The protocol of measurement
informed consent was obtained from each subject.
From April 2012 to May 2014, a total of 67 indi- The protocol of measurement is based on a discre-
viduals were enrolled and divided into two main tisation of the LA analogous to the one adopted
groups (see Table 1 for descriptive statistics). The for the LV: the LA is divided through 36 planes
first group consisted of 45 healthy subjects with- perpendicular to the longitudinal axis of the LA;
out a history of hypertension or cardiac disease; each plane is then divided into 36 sections along
they all had normal ECG and blood pressure the circumference; this subdivision gives rise to
below 140/90 mmHg. The second group included 36×36 points on both the endocardial and the epi-
22 HCM patients (see Table 1 for 3DSTE details). cardial surfaces of the LA and determines, for any
In all 67 individuals enrolled, echocardiographic subject, the distance between estimation points.
examinations were performed by an Aplio–Artida It has to be noted that endocardial and epi-
ultrasound system (Toshiba Medical Systems Co, cardial surfaces of LA are extremely close one
Tochigi, Japan). Full-volume ECG-gated 3D data to another; due to it, we expect to find almost
sets were acquired from apical positions using a the same kind of mechanical information from
1−4 MHz 3D matrix array transducer to visualize the analysis of the corresponding data. At the
the entire LA in a volumetric image (see [7] and end, the LA geometry is identified by a cloud of
Fig. 1). 1296×2 points i.e. for endocardial and epicardial
geometry, whose motion is followed along the car-
diac cycle (see Fig. 1).
Table 1. Descriptive statistics ± standard deviation for
the groups under study.
3 ANALYSIS
Descriptive parameter Control = 45 HCM = 22

Age 39.2 ± 8.4 48.2 ± 12.6

Here we set the keystones for a mechanics–based
Sex (males/females) 29/15 14/8
analysis of the left chambers, based on the ecocar-
Beat rate 77.3 ± 15 73 ± 12
diographic data collected for healthy subjects and
HCM. LA deformations (longitudinal, circumfer-
EDV 39.67 ± 14.2 90.1 ± 51.6
ential and radial) during cardiac revolution have
ESV 20.1 ± 7.9 59.2 ± 48.6
been studied in the Literature [5, 8, 9] via 3DSTE.
EF 49.3 ± 0.07 39.6 ± 0.12
However, there were relatively few 3DSTE investi-
Interventricular 8.55 ± 1.4 18.13 ± 4
septum gations whereby LA function was assessed in both
normal and HCM patients.

268

VIPIMAGE15_Book.indb 268 9/8/2015 7:46:11 PM

 Interestingly, for any LA of our database, we
also have the corresponding LV, and both ecocar-
diographic data are acquired in the same cardiac
cycle. This situation allows us to plan an analysis
which can take advantage of the knowledge of the
mechanics of both LA and LV chambers of the
heart.

3.1 Collective behaviour at homologous times

To estimate the collective behaviour of the Figure 3. Average values of LA circumferential strains
vs, homologous times for each healthy subjects (left) and
group, we need firstly to refer to echocardiogra- HCM patients.
phy frames corresponding to the same mechani-
cally and electrically meaningful times, that is, to
homologous times. In Figure 2, we show a typical
LA endocardial volume (red line) vs. time pattern
so as it appears in an echocardiographic image,
whereas black line shows a typical ECG pat-
tern; both of them are associated to any 3DSTE
registration. Blue line shows the corresponding
LV endocardial volume of the same subject.
Black dots on the red and blue curves identify
the homologous times, which are used to make
physiologically appropriate comparisons among
subjects. Figure 4. Average values of LA longitudinal strains vs,
homologous times for each healthy subjects (left) and
HCM patients.
3.1.1 Crossed analysis for healthy
subjects and HCM patients
The values of LA circumferential and longitudi-
nal strains in our control and HCM groups have corresponding to the control group and the right
been measured through 3DSTE as averages on all panel to the HCM group.
the 16 LA segments, at the six homologous times. We remark as the intra–group comparisons of
Then, using these data, we evaluated an aver- the strains vs. time patterns are significative only
age value vs. homologous times for each subject, due to the established correspondence of the same
only considering the middle segments, and repre- mechanically and electrically meaningful times,
sented it in Figures 3 and 4, with the left panels which identify the homologous times.
The direct comparison of the mean value of cir-
cumferential and longitudinal strains for the con-
trol and HCM group is shown in Figure 5.

4 CONCLUSIONS AND FUTURE

DIRECTIONS

We presented a mechanics–based analysis of LA

3D data echocardiographically obtained from
healthy subjects and HCM patients. Interestingly,
for any LA of our database, we also had the corre-
sponding LV, and both ecocardiographic data were
acquired in the same cardiac cycle. This situation
will allow us in the next future to plan a crossed
analysis which can take advantage of the knowl-
Figure 2. Six electrophysiologically homologous times edge of the mechanics of both LA and LV cham-
are identified along the cardiac cycle: R peak, end T wave,
ESV, mitral valve opening, onset of diastasis, Q wave.
bers of the heart as well as to detect the collective
Blue and red lines show a typical time course of endo- behaviour of the two groups of subjects using the
cardial volume for LV and LA, respectively, and black S–Lab tool proposed and discussed in (4).
line shows a typical ECG pattern so as they appear in a Therein we conjectured that the inflation–
3DSTE registration. induced dilation due to blood pressure is more

269

VIPIMAGE15_Book.indb 269 9/8/2015 7:46:13 PM

 Figure 5. Six electrophysiologically homologous times are identified along the cardiac cycle: R peak, end T wave,
ESV, mitral valve opening, onset of diastasis, Q wave.

Figure 6. Primary Strains (PS) measured by 3DSTE along the cardiac cycle in one representative healthy subject
(left panel) from our series of 45 subjects and in one patient with primary pressure overload (right panel) from our
series of 22 HCM patients at endocardial level: it is evident that at end-systolic homologous time (around 400 ms)
the patient presents greatly reduced PS in both LV and LA levels, indicating a reduced capacity to contrast intra–
chamber pressures, a possible consequence of reduced elasticity due to intrinsic heart muscle pathology typical of
HCM. Interestingly enough, in comparison to the representative healthy subject of the left panel, the reduced elastic-
ity indexed by greatly lower PS of the endocardium is dimensionally similar at LV and LA levels in the representative
HCM patient shown in the right panel.

effective for subendocardial layers, which dilat- values which correspond to the circumferential
ing reduce the circumferential shortening induced PSL are smaller.
by muscle contraction. Better is the capacity of
the elastic response of the endocardial surface, REFERENCES
smaller is the dilation induced by blood pressure.
This concept is well represented by endocardial [1] Piras, P., Evangelista, A., Gabriele, S., Nardinocchi, P.,
Primary Strain (PS) lines and values, which rep- Teresi, L., Torromeo, C., Schiariti, M., Varano, V.,
resent at a point of the endocardial surface the Puddu, P.E., 2014. 4D-analysis of Left Ventricu-
directions where the smallest are to be expected. lar Heart Cycle Using Procrustes Motion Analysis.
In this sense, we interpreted the primary strain val- PLoS ONE 9(1): e86896. (doi:10.1371/journal.
ues measured by 3DSTE along the cardiac cycle pone.0086896).
as shown in Figure 6. our representative healthy [2] Evangelista, A., Gabriele, S., Nardinocchi, P., Piras, P.,
subject shows a PS value which denotes a larger Puddu, PE., Teresi, L., Torromeo, C., Varano, V.,
2014. A comparative analysis of the strain-line pat-
shortening than the corresponding value shown by tern in the human left ventricle: experiments vs
the representative HCM patient. It means that the modeling. Computer Methods in Biomechanics and
capacity to contrast blood pressure is reduced in Biomedical Engineering/Imaging & Visualization.
patients with with primary pressure overload due Published online: Jun 23, 2014. (doi:10.1080/216811
to intrinsic heart muscle pathology, hence the PS 63.2014.927741).

270

VIPIMAGE15_Book.indb 270 9/8/2015 7:46:14 PM

 [3] Gabriele, S., Nardinocchi, P., Varano, V., 2013. [12] Helle-Valle, T., Crosby, J., Edvardsen, T., Lyseggen, E.,
Evaluation of the strain–line patterns in a human Amundsen, B.H., Smith, H.J., Rosen, B.D.,
left ventricle: a simulation study. Computer Meth- Lima, J.A.C., Torp, H., Ihlen, H. and Smiseth, O.A.,
ods in Biomechanics and Biomedical Engineering 2005. New noninvasive method for assessment of
18(7), 790–798. left ventricular rotation: speckle tracking echocar-
[4] Evangelista, A., Gabriele, S., Nardinocchi, P., Piras, P., diography. Circulation 112, 3149–3156.
Puddu, PE., Teresi, L., Torromeo, C., Varano, V., [13] Maffessanti, F., Nesser, H.J., Weinert, L., Steringer-
2015. Non-invasive assessment of functional strain Mascherbauer, R., Niel, J., Gorissend, W., Sugeng, L.,
lines in the real human left ventricle via speckle Lang, R.M. and Mor-Avi, V., 2009. Quantitative
tracking echocardiography. Journal of Biomechan- evaluation of regional left ventricular function using
ics 48, 465–471. three-dimensional speckle tracking echocardiogra-
[5] Vieira, M.J., Teixeira, R., Gonzalves, L., Gersh, B.J., phy in patients with and without heart disease. Am.
2014. Left Atrial Mechanics: Echocardiographic J. Cardiol. 104, 1755–1762.
Assessment and Clinical Implications. Journal of [14] Cerqueira, M.D., Weissman, N.J., Dilsizian, V.,
the American Society of Echocardiography 27(5), Jacobs, A.K., Kaul, S., Laskey, W.K., Pennell, D.J.,
463–478. Rumberger, J.A., Ryan, T., Verani, M.S., 2002.
[6] Yeung, F., Levinson, S.F., Parker, K.J., 1998. Multilevel Standardized myocardial segmentation and nomen-
and motion model-based ultrasonic speckle tracking clature for tomographic imaging of the heart: a
algorithms. Ultrasound Med. Biol. 24, 427–441. statement for healthcare professionals from the Car-
[7] Evangelista, A., Nardinocchi, P., Puddu, P.E., diac Imaging Committee of the Council on Clini-
Teresi, L., Torromeo, C., Varano, V., 2011. Torsion cal Cardiology of the American Heart Association.
of the human left ventricle: Experimental analysis Circulation 105, 539–542.
and computational modeling. Progress in Biophys- [15] Madeo, A., Piras, P., Re, F., Gabriele, S.,
ics and Molecular Biology, 107(1), 112–121. Nardinocchi, P., Teresi, L., Torromeo, C.,
[8] Onishi, N., Kawasaki, M., Tanaka, R. et al., 2014. Chialastri, C., Schiariti, M., Giura, G., Evange-
Comparison between left atrial features in well– lista, A., Dominici, T., Varano, V., Zachara, E.,
controlled Hypertensive patients and normal sub- Puddu, P.E., 2015. A new 4D trajectory-based
jects assessed by three dimensional speckle tracking approach unveils abnormal LV revolution dynamics
echocardiography. Journal of Cardiology 63, in hypertrophic cardiomyopathy. PLoS One 10(4),
291–295. e0122376.
[9] Ancona, R., Comenale Pinto, S., Caso, P., et al., [16] Torromeo, C., Evangelista, A., Pandian, NG.,
2014. Left atrium by echocardiography in clinical Nardinocchi, P., Varano, V., Gabriele, S.,
practice: from conventional methods to new echocar- Schiariti, M., Teresi, L., Piras, P., Puddu, PE., 2014.
diographic. Scientif World Journal 2014, 451042. Torsional correlates for End Systolic Volume Index
[10] Pedrizzetti, G., Kraigher-Krainer, E., De Luca, A., in adult healthy subjects. International Journal of
Caracciolo, G., Mangual, j.o., Shah, A., Toncelli, L., Applied Science and Technology 4(4), 11–23.
Domenichini, F., Tonti, G., Galanti, G.,
Sengupta, P.P., Narula, J., Solomon, S., 2012.
Functional Strain-line Pattern in the Human Left
Ventricle. Phys. Rev. Lett. 109, 048103.
[11] Goffinet, C., Chenot, F., Robert, A., Pouleur, A.C., le
Polain de Waroux, J.B., Vancrayenest, D., Gerard, O.,
Pasquet, A., Gerber, B.L. and Vanoverschelde, J.L.,
2009. Assessment of subendocardial vs. subepicardial
left ventricular rotation and twist using two dimen-
sional speckle tracking echocardiography compari-
son with tagged cardiac magnetic resonance. Eur.
Heart J. 30, 608–617.

271

VIPIMAGE15_Book.indb 271 9/8/2015 7:46:15 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

A new thick-walled conical model of the Left Ventricle

B. Leśniak-Plewińska, S. Cygan, J. Żmigrodzki & K. Kałużyński

Institute of Metrology and Biomedical Engineering, Faculty of Mechatronics, Warsaw University
of Technology, Warsaw, Poland

ABSTRACT: In this paper, we proposed a numerical left ventricle model with simplified conical
geometry. The numerical simulation of the model deformation is carried out using the Yeoh constitu-
tive model, hexahedral elements, and nonlinear geometry (Abaqus CAE, Simulia). To validate the model
utility, the displacements and strains of some representative points are compared with clinical data. The
proposed computational LV model can simulate the LV wall deformations to some extent. Although, it
cannot fully mimic the strain distribution and absolute peak strain values obtained in clinical practice, a
physical model approximating the LV shape with a cone and deformed in a dedicated measurement stand
will be a profitable source of reliable data for the validation of myocardial strain imaging modalities.

1 INTRODUCTION the heart) and its variation from one LV site to

the other. Measurements with implanted markers
The heart has been described as a “complex three- have demonstrated the existence of both circum-
dimensional fibre-wound structure with mechani- ferential transmural and longitudinal transmural
cal properties that are nonlinear, anisotropic, time shear strains (Waldman et al. 1985). Furthermore,
varying and spatially inhomogeneous” (Greenbaum the Magnetic Resonance Imaging (MRI)-based
1981) attributing to the complex multidimensional measurements have shown that the magnitude of
deformation field in the Left Ventricle (LV), with the endocardial twist is significantly higher than
the apex rotating counterclockwise (when viewed the epicardial twist (Buchalter et al. 1990). These
from the apex) relatively to the atrioventricular findings can result from the structure of the myo-
plane (the base) and the base moving along the cardium and the geometry of the LV.
long axis (Sengupta et al. 2006). Despite the development of noninvasive myo-
Characterization of global and regional left cardial strain imaging techniques, the “ground
ventricular deformation is the target of numerous truth” about the cardiac deformation pattern
studies (D’hooge et al. 2002, Jasaityte et al. 2013, is still not well known, as all measurement tech-
Langeland et al. 2004). Most of them consider the niques have limitations. Therefore, validation of
entire myocardial wall thickness without distinction existing and new strain imaging methods in the
between any particular layers of the myocardium. in vivo settings is very difficult. Models capable of
However, based on experimental and clinical obser- reproducing deformation patterns similar to those
vations, transmural heterogeneity of a myocardial encountered in clinical practice may provide reli-
mechanical function is commonly accepted. The able data for the development and evaluation of
mechanisms for this are not fully understood. the myocardial strain imaging modalities. With the
Changes in myocardial electro-mechanical proper- advances in numerical modeling and simulations
ties, fiber orientations as well as ventricular geome- as well as non-invasive imaging modalities, patient-
try and shape may be significant. Numerous studies specific models of ventricular biomechanics can be
have measured the characteristics of the transmural created to validate the strain imaging modalities.
distribution of myocardial deformation. The focus However, many challenges still have to be over-
of the studies has been mainly on the transmural come to estimate model parameters from available
gradient of the longitudinal, radial and circum- patient data (the heart geometry in unloaded state,
ferential deformation, demonstrating the lowest the myofibre anatomy, and the material properties
strain in subepicardial and the highest in suben- on patients (Krishnamurthy et al. 2013).
docardial layer (Adamu et al. 2009, D’hooge et al. Although physical LV models have its limita-
2006, Leitman et al. 2010, McCulloch et al. 1991). tions, an experimental setup and phantom capa-
Fever studies concerned the transmural character- ble of producing deformation patterns similar
istics of the human left ventricular twist (defined to those encountered in clinical practice is still a
here as angular rotation relative to the base of profitable source of reliable data for the validation

273

VIPIMAGE15_Book.indb 273 9/8/2015 7:46:16 PM

 of myocardial strain imaging modalities as well
as numerical simulations. The physical model and
laboratory stand may deliver actual data for FE
modeling, that, on the other hand, may be cross-
referenced with deformation pattern estimated
based on physical experiments.
Various heart phantoms have been used to study
and validate cardiac measurements (Claessens et al.
2004, Langeland et al. 2004, Jia et al. (2006), Lesniak-
Plewinska et al. 2010, Smith & Rinaldi 1989; Visser
et al. 2003, Zhu et al. 2014). These phantoms have
many limitations. They are all passively deformed.
Moreover, they are either made of inappropri-
ate material, thin-walled or using the experimen-
tal conditions (pressure waveform, stroke volume,
and heart rate) which are far from physiological.
Only one of them can be twisted (Zhu et al. 2014). Figure 1. Model geometry and dimensions (in mm).
Unfortunately, the turnable model has a mechanical
connection to the apex that makes the apical views
unfeasible. Moreover, it has limited applicability to examinations were carried out. A cylindrical sample
model physiological heart motion—the stress pat- with 34.26 mm in diameter and 28.85 mm in height
tern reported for the model bears no analogy to that was made from a liquid solution of 10 mass% (m%)
of the physiological ventricle (Zhu et al. 2014). PVA (Sigma Aldrich, St. Louis, USA) and 10 m%
The authors have already developed a meas- glycerin (Chempur, Piekary Slaskie, Poland). This
urement stand with a dual chamber heart model solution was physically solidified in a laboratory
shaped as thick-walled truncated ellipsoid (Lesniak- freezer (ZLN 74, Pol-Eko, Poland) by two cycles of
Plewinska et al. 2010). Unfortunately, the twisting freezing at −25°C for 24 hours and thawing for the
motion has not been implemented in the system. LV next 24 hours. The mechanical examinations were
models with simplified geometry usually have cylin- carried out on an MTS Qtest/10 material testing
drical, spherical or ellipsoidal shape. Azhari et al. machine (MTS, USA) with deformation ranging
(1992) and Chaudhry et al. (1996) reported that a up to 30% of the original sample size.
thick conical shell can better approximate the non-
uniformity of heart motion, especially the transmu-
2.2 Simulation
ral and longitudinal nonuniformity of the twisting
motion. Therefore, the authors deiced to construct Numerical model of the phantom geometry (Fig. 1),
new measurement stand with the LV model simpli- created using the Autodesk Inventor 2012 software
fied to the conical shape. (Autodesk Inc., USA), was exported in CAD for-
This work is a part of the ongoing project on the mat to the FE simulation software Abaqus 6.13
effect of selected echocardiographic data acquisi- (Simulia, USA). The model was meshed with the
tion parameters on accuracy of the myocardial seed (maximum element edge length) of 2 mm,
deformation estimation. The paper presents pre- which resulted in 69890 linear hexahedral elements
liminary results of FE simulations aimed to inves- of type C3D8H, defined by 76347 nodes.
tigate the transmural deformation patterns for A fixed cylindrical coordinate system was created
model with conical geometry. by specifying the origin in the center of the circu-
lar surface of the flange (point O), Z-axis running
along the main symmetry axis of the model and the
2 MATERIAL AND METHODS
R-axis perpendicular to the Z-axis (Figs. 2 and 4).
The strain-stress relationship of PVA cryogel
2.1 LV model
was modeled with 3-rd order Yeoh strain energy
The LV model was designed as a thick circular- potential definition as an isotropic hyperelastic
conical shell with two additional fixing elements: material based on strain-stress relation derived
a cylindrical collar at the base and circular flange from the experiments and with Poisson’s ratio of
at the apex end (Fig. 1). 0.45. Evaluation of the chosen material constitutive
The physical model will be made of Poly(vinyl model has been carried out using built-in functions
alcohol) hydrogel (PVA) prepared by freezing- of Abaqus software, providing the model and veri-
thawing cyclic processing. The strain-stress relation fying the stability of the material model.
for the PVA cryogel is nonlinear. To obtain a reliable Deformation of the conical part was forced by
material model of the PVA cryogel, mechanical applying: (i) pressure load to the internal surface

274

VIPIMAGE15_Book.indb 274 9/8/2015 7:46:16 PM

 varied in time, and their value changes were repre-
sented by the same relative amplitude curve that
shape has been derived from the LV volume curve
(Konturek 2013) (Fig. 2). The maximal amplitudes
of the loads were: 18000 Pa (pressure), 14 mm
(base translation along Z axis), 0.6 rad (apex rota-
tion around Z axis).
The simulation was carried out in steps using
non-linear geometry option turned on (Nlgeom)
since the process includes large deformations.
As the result of FE simulation, values of nomi-
nal strains at all nodes in cylindrical coordinate
systems were returned.
According to the Abaqus documentation the
nominal strain is defined as:
3

Figure 2. Coordinate system and fixing surfaces.

εN ∑ ( λi − ) ⋅ ni ⋅ niT (1)
i =1

where V=√(F ⋅ FT) is the left stretch tensor, λi are

the principal stretches, and ni are the principal
stretch directions in the current configuration. The
principal values of nominal strain are the ratios
of change in length to length in the reference con-
figuration in the principal directions (Abaqus 6.13
documentaion, Abaqus Analysis User’s Manual,
1.2.2 Conventions).
A set of nodes lying in the plane of symmetry of
model in the unloaded state was selected (Fig. 4) as
Figure 3. Loads time curve: base translation along Z corresponding to particular layers in segments of
axis (in mm), apex rotation around Z axis (in radians), myocardium e.g. epi-, mio-, and endocardial layers
and pressure (in Pa).
in basal, mid and apical segment.
Three nominal strain components in cylindri-
cal coordinate system: axial, circumferential, and
radial as well as twist in the nine selected were
exported from Abaqus for further processing in
Matlab 2014b (Mathworks, USA). Angular dis-
placements of nodes lying on the outer (epi), mid-
dle (mio) and inner (endo) conical surface were
used to calculate the twist as difference in rotation
magnitude between the given node and the corre-
sponding basal node.

Figure 4. Nodes of the mesh for the plane of symmetry

in cylindrical coordinates. White circle—nodes used to 3 RESULTS
calculate twist along the model wall. Thick black dots—
nodes selected as corresponding to particular layers in The time curves of the radial, circumferential, and
segments of myocardium e.g. epi-, mio-, and endocardial axial strain components resulting from FE simu-
layers in basal, mid and apical segment. lations for nodes corresponding to the epi-, mio-,
and endocardial layers in basal, mid, and apical
segments are presented in Figure 5.
of the this part, (ii) base translation along main A transmural gradient may be seen for circum-
axis of the model symmetry (Z axis) by linear shift ferential and radial strain components with the
of rigid tubes glued to the inner and outer surface lowest values at the epicardial and highest at the
of the cylindrical collar (Fig. 2), and (iii) apex endocardial nodes for each segment (basal, mid,
rotation around main axis of the model symmetry and apical) during the entire cycle.
(Z axis) by rotation of one rigid ring glued to the Axial strain component increases from the epi-
circular surfaces of the flange (Fig. 2). The loads cardial to the endocardial layer only in the case

275

VIPIMAGE15_Book.indb 275 9/8/2015 7:46:16 PM

 Figure 5. Time curves of radial (a), circumferential (b), and axial (c) strain components resulting form FE simula-
tions for nodes corresponding to epi- (-*-), mio- (- -), and endocarial (-•-) layers in basal (left), mid (central), and apical
(right) segments. Abscissa normalized to one cycle, ordinate unitless.

of apical segment. The opposite trend may be The twist along the model wall for the nodes
observed for the basal segment. In the mid seg- located at the outer (epi) and inner (endo) surface
ment, axial strain component for the epi- and myo- of the conical part is presented in Figure 6.
cardial layers are almost identical and higher than A twist gradient may be observed across the model
for the endocardial layer. wall with twist values for the endocardial layer higher

276

VIPIMAGE15_Book.indb 276 9/8/2015 7:46:17 PM

 that for the epicardial one over the entire length of 4 CONCLUSIONS
the LV model wall. The twist gradient increases from
the base towards the apex of the model (Fig. 6). In this study, we have shown that the proposed
The maximal values of the nominal strain com- computational LV model can simulate the LV wall
ponents obtained in the selected nodes and the deformations to some extent.
twist for basal nodes are summarized in Table 1. Comparing the obtained results to that obtained
in clinical practice with the absolute peak strain
values summarized in Table 2, the main differences
were observed between the in situ axial and the in vivo
longitudinal component. This may result from the
fact that in clinical applications the longitudinal
component is calculated as tangential to the wall
curvature while the axial component is calculated as
parallel to model main symmetry axis (axis Z).
The in silico absolute peak values of circumfer-
ential and radial strain components for basal seg-
ment are much lower than the results reported by
physicians, what may be explained by the fact that
Figure 6. The maximal twist along the in silico model the outer and inner surfaces of the fixing flange
wall for the nodes located at the epi- (-*-) and endocardial where immobilized. At the same time, in the case
(-•-) layer. of mid and apical segments good quantitative
agreement was found between the numerical model
and the clinical reports.
Table 1. Maximal absolute values of the nominal strain In addition, the proposed conical model exhibits
components and twist values obtained in the selected nodes. the transmural twist gradient increasing from base
FE simulation
towards apex, with the endocardial twist exceeding
Strain Wall the epicardial one, what is in agreement with the
component [%] segment Epi Mio Endo other studies (Buchalter et al. 1990).
Although, the proposed model cannot fully
Axial Basal 3.4 15.7 24.7 mimic the strain distribution and absolute peak
Mid 15.2 14.9 11.2 strain values obtained in clinical practice, a physical
Apical 14.4 12.2 9.0 model approximating the LV shape with a cone and
Circumferential Basal 3.1 6.9 21.1 deformed in a dedicated measurement stand will be
Mid 11.5 20.4 40.5 a profitable source of reliable data for the validation
Apical 15.2 24.5 46.5 of myocardial strain imaging modalities. However,
Radial Basal 4.8 15.3 28.1 further investigations are necessary to optimize the
Mid 15.7 20.2 26.4
Apical 11.8 16.8 23.5
way the physical model will be fastened in the meas-
Twist [°] 10.6 11.0 12.3 urement stand, as it seems to be one of the major
reasons for the non-compliance with the clinics.

Table 2. Maximal absolute strain and twist values obtained in clinical practice.

Clinical data*
Strain Wall
component [%] segment Epi Mio Endo

Longitudinal Basal 19.2 ± 3.1 19.8 ± 3.2 20.3 ± 4.8

Mid 19.4 ± 3.0 21.3 ± 3.1 23.5 ± 3.6
Apical 18.9 ± 3.9 25.0 ± 4.5 33.5 ± 5.3
Circumferential Basal 15.6 ± 6.7 22.0 ± 8.0 32.4 ± 8.4
Mid 19.4 ± 6.4 21.3 ± 6.1 23.5 ± 9.1
Apical 18.9 ± 7.6 25.0 ± 4.9 33.5 ± 11.2
Radial Basal 39.0 ± 12.9 – 47.8 ± 16.9
Mid – – –
Apical 32.6 ± 12.5 – 39.4 ± 16.4
Twist [°] 8.4 ± 3.7 9.7 ± 4.1 12.5 ± 5.7

*Longitudinal, Circumferential (Leitman et al. 2010), Radial (Chan-Dewar et al. 2010),

Twist (Goffinet et al. 2009).

277

VIPIMAGE15_Book.indb 277 9/8/2015 7:46:18 PM

 ACKNOWLEDGEMENTS Jasaityte, R., Heyde, B., D’Hooge, J. 2013. Current state
of three-dimensional myocardial strain estimation
This study was supported by the Norwegian using echocardiography J Am Soc Echocardiog. 26(1):
Research Programme operated by the National 15–28
Jia, C., Kim, K., Kolias, T.J., Rubin, J.M.,
Centre for Research and Development in Poland O’Donnell, M. 2006. Left ventricular phantom
under the Norwegian Financial Mechanism 2009-14 with pulsatile circulation for ultrasound strain rate
(Project Contarct No: Pol-Nor/209584/19/2013). Imaging. IEEE Ultrasonics Symposium 2–6 Oct.
2006. Vancouver, BC.
Konturek, S. 2013 Konturek Fizjologia człowieka.
REFERENCES Podręcznik dla studentów medycyny Elsevier Urban &
Partner. Wrocław.
Adamu, U., Schmitz, F., Becker, M., Kelm, M., Krishnamurthy, A., Villongco, C.T., Chuang, J.,
Hoffmann, R. 2009. Advanced speckle tracking Frank, L.R., Nigam, V., Belezzuoli, E., Stark, P.,
echocardiography allowing a three-myocardial layer- Krummen, D.E., Narayan S., Omens, J.H.,
specific analysis of deformation parameters. Eur J McCulloch, A.D., Kerckhoffs, R.C.P. 2013. Patient-
Echocardiogr 10: 303–308. specific models of cardiac biomechanics. J Comput
Azhari, H., Buchalter, M., Sideman, S., Shapiro, E., Phys. 244: 4–21
Beyar, R. 1992. A conical model to describe the nonu- Langeland, S., D’hooge, J., Claessens, T., Claus, P.,
niformity of the left ventricular twisting motion. Ann Verdonck, P., Suetens, P., Sutherland, G.R., Bijnens, B.
Biomedl Eng 20(2): 149–65. 2004 RF-based two-dimensional cardiac strain estima-
Buchalter, M.B., Weiss, J.L., Rogers, W.J., Zerhouni, E.A., tion: a validation study in a tissue-mimicking phan-
Weisfeldt, M.L., Beyar, R., Shapiro, E.P. 1990. tom IEEE Trans. UFFC. 51: 1537–46.
Noninvasive quantification of left ventricular rota- Leitman, M., Lysiansky, M., Lysyansky, P., Friedman, Z.,
tional deformation in normal humans using magnetic Tyomkin, V., Fuchs, T., Adam, D., Krakover, R.,
resonance imaging myocardial tagging. Circulation. Vered, Z. 2010. Circumferential and Longitudinal
81(4): 1236–44. Strain in 3 Myocardial Layers in Normal Subjects and
Chan-Dewar, F., Oxborough, D., Shave, R., Gregson, W., in Patients with Regional Left Ventricular Dysfunc-
Whyte, G., George, K. 2010. Left ventricular myo- tion J Am Soc Echocardiog 23(1): 64:70.
cardial strain and strain rates in sub-endocardial and Lesniak-Plewinska, B., Cygan, S., Kaluzynski, K.,
sub-epicardial layers before and after a marathon Eur D’hooge, J., Zmigrodzki, J., Kowalik, E.,
J Appl Physiol. 109(6): 1191–6. Kordybach, M., Kowalski, M. 2010. A dual-chamber,
Chaudhry, H.R., Bukiet, B. and Davis, A.M. 1996. thick-walled cardiac phantom for use in cardiac
Stresses and strains in the left ventricular wall approx- motion and deformation imaging by ultrasound.
imated as a thick conical shell using large deformation Ultrasound Med Biol. 36(7): 1145–56.
theory. J Biol Syst. 4: 353–72. McCulloch, A.D., Omens, J.H. 1991. Non-homogeneous
Claessens, T., Segers, P., Verdonck, P. 2004. A new thick- analysis of 3-dimensional transmural finite deforma-
walled hydraulic model of the left heart for the assess- tion in canine ventricular myocardium. J Biomech
ment of blood-wall interaction using ultrasound. 24(7): 539–48.
IEEE Proc Computers in Cardiology 31: 333 6. Sengupta, P.P., Korinek, J., Belohlavek, M., Narula, J.,
D’hooge, J., Konofagou, E., Jamal, F., Heimdal, A., Vannan, M.A., Jahangir, A., Khandheria, B.L. 2006.
Barrios, L., Bijnens, B., Thoen, J., Van de Werf, F., Left ventricular structure and function: basic science
Sutherland, G., Suetens, P. 2002 Two-dimensional for cardiac imaging. J Am Coll Cardiol. 48: 1988–2001.
ultrasonic strain rate measurement of the human Smith, S.W., Rinaldi, J.E. 1989. Anthropomorphic cardiac
heart in vivo IEEE Trans UFFC. 49: 281–6. ultrasound phantom. IEEE Trans BME 10: 1055–8.
D’hooge, J., Missant, C., Claus, P., Rex, S., Eroglu, S., Visser, J.J., Sokole, E.B., Verberne, H.J., Habraken, J.B.,
Rademakers, F.E., Wouters, P. 2006. Gradients in van de Stadt, H.J., Jaspers, J.E., Shehata, M.,
Radial Strain Across the Cardiac Wall Studied with Heeman, P.M., van Eck-Smit, B.L. 2003. A realistic
Epicardial Ultrasound IEEE Ultrasonics Symposium 3-D gated cardiac phantom for quality control of
2–6 Oct. 2006. Vancouver, BC. gated myocardial perfusion SPECT: the Amsterdam
Goffinet, C., Chenot, F., Robert, A., Pouleur, A-C., gated (AGATE) cardiac phantom. Eur J Nuclear Med
le Polain de Waroux, J-B., Vancrayenest, D., Molecular Imag 31: 222–8.
Gerard, O., Pasquet, A., Gerber, B.L., Waldman, L.K., Fung, Y.C., Covell, J.W. 1985. Transmural
Vanoverschelde, J-L. 2009. Assessment of subendo- myocardial deformation in the canine left ventricle.
cardial vs. Subepicardial left ventricular rotation and Normal in vivo three-dimensional finite strains. Circ
twist using twodimensional speckle tracking echocar- Res. 57(1): 152–63.
diography: comparison with tagged cardiac magnetic Zhu, Y., Luo, X.Y., Gao, H., McComb, C., Berry, C.
resonance Eur Heart J 30: 608–17. (2014) A numerical study of a heart phantom model,
Greenbaum, R.A., Ho, S.Y., Gibson, D.G., Becker, A.E., International J Comput Math. 91: 7, 1535–1551.
Anderson, R.H. 1981. Left ventricular fiber architec-
ture in man. Brit Heart J. 45(3): 248–63.

278

VIPIMAGE15_Book.indb 278 9/8/2015 7:46:18 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Augmented Reality in radiofrequency ablation of the liver tumors

Lucio Tommaso De Paolis & Francesco Ricciardi

Department of Engineering for Innovation, University of Salento, Lecce, Italy

ABSTRACT: Minimally Invasive Surgery is a surgery technique that provides evident advantages for
the patients, but also some difficulties for the surgeons. In medicine, the Augmented Reality (AR) technol-
ogy allows surgeons to have a sort of “X-ray” vision of the patient’s body and can help them during the
surgical procedures. In this paper we present an application of Augmented Reality that could be used as
support for a more accurate preoperative surgical planning and also for an image-guided surgery. The AR
can support the surgeon during the needle insertion for the Radiofrequency Ablation of the liver tumours
in order to guide the needle and to have an accurate placement of the surgical instrument within the
lesion. The augmented visualization can avoid as much as possible to destroy healthy cells of the liver.

1 INTRODUCTION Medical images (CT or MRI) associated to the

latest medical image processing techniques could
The actual trend in surgery is the transition from provide an accurate knowledge of the patient’s
open procedures to minimally invasive interventions. anatomy and pathologies. This information can be
Using this surgery technique visual feedback to the used to guide surgeons during the surgical proce-
surgeon is only possible through the laparoscopic dure in order to improve the patient care.
camera. Furthermore direct palpation of organs is The Augmented Reality (AR) technology has
not possible. Minimally Invasive Surgery (MIS), the potential to improve the surgeon’s visualization
such as laparoscopy or endoscopy, has changed during a MIS surgical procedure. AR technology
the way to practice the surgery. It is a promising refers to a perception of a physical real environment
technique and the use of this surgical approach is whose elements are merged with virtual computer-
nowadays widely accepted and adopted as a valid generated objects in order to create a mixed reality.
alternative to classical procedures. The merging of virtual and real objects in an AR
The use of MIS offers to surgeons the possibil- application has to run in real time. Virtual objects
ity of reaching the patient’s internal anatomy in a have to be aligned (registered) with real world struc-
less invasive way. This reduces the surgical trauma tures. Both of these requirements guarantee that
for the patient. The diseased area is reached by the dynamics of real world environments remain
means of small incisions made on the patient body unchanged after virtual data has been added.
called ports. Specific instruments and a camera are In medicine, the use of AR technology makes
inserted through these ports. The surgeon uses a possible to overlay virtual medical images of the
monitor to see what happens in the surgical field organs on the real patient. This allows the surgeon
inside the patient body. to have a sort of “X-ray vision” of the patient’s
With the use of MIS the patient has a shorter internal anatomy. This technology can “augment”
hospitalizations, faster bowel function return, surgeon’s perception of the real world with the use
fewer wound-related complications and a more of information gathered from patient’s medical
rapid return to normal activities. These benefits images. The use of AR in surgery produces a better
have contributed to accept these surgical proce- spatial perception and a reduction in the duration
dures. If the advantages of this surgical method of the surgical procedure.
are evident on the patients, this technique has In order to obtain a correct alignment of virtual
some limitations for the surgeons. First of all, the and real objects is needed a registration procedure
imagery is in 2D and with this limitation the sur- that can be accomplished using an optical tracker.
geon needs to develop new skills and dexterity in Many research groups are now focusing on the
order to estimate the distance from the anatomi- development of systems that assists surgeons dur-
cal structures. In addition, we have to consider that ing the treatment of the liver tumours.
he had to work in a very limited workspace. Some Nicolau et al. present a real-time superimpo-
modern systems use two cameras to offer a 3D sition of virtual models over the patient’s abdo-
view to the surgeon but this technology is useful men in order to have a three dimensional view of
when the working volume is not too small. the internal anatomy. The authors have used the

279

VIPIMAGE15_Book.indb 279 9/8/2015 7:46:18 PM

 developed system in an operating room and to it is possible to know the right position and the
reduce the influence of the liver breathing motion orientation of the surgical instruments used dur-
they have tried to simulate the patient’s breathing ing the surgical procedure. The tracking system is
cycle (Nicolau et al. 2009). also used in order to permit the overlapping of the
LiverPlanner is a virtual liver surgery planning virtual organs on the real ones providing an aug-
system developed at Graz University of Technology mented visualization of the scene.
that combines image analysis and computer graphics
in order to simplify the clinical process of planning
liver tumour resections. The treatment planning stage 3 AR FOR THE RF ABLATION
enables the surgeon to elaborate a detailed strategy OF THE LIVER TUMORS
for the surgical intervention and the outcome of pre-
operative planning can then be used directly for the Hepatic cancer is one of the most common solid can-
surgical intervention (Reitinger et al. 2004). cers in the world. Hepatocellular Carcinoma (HCC)
Maier-Hein et al. present a system developed is the most common primary hepatic cancer.
for computer-assisted needle placement that uses The Liver Radiofrequency Ablation (RFA) is a
a set of fiducial needles to compensate for organ minimally invasive treatment for liver cancer used
motion in real time; the purpose of this study was since 1980’s. It consists in the placement of a nee-
to assess the accuracy of the system in vivo (Maier- dle inside the liver parenchyma in order to reach
Hein et al. 2008). the centre of the tumour lesion. When the lesion
Stüdeli et al. present a system that provides sur- center is reached, an array of electrodes is extracted
geon, during placement and insertion of RFA nee- from the tip of the needle and it is expanded in the
dle, with information from pre-operative CT images tumour tissue. From these electrodes is injected in
and real-time tracking data (Stüdeli et al. 2008). the tumour tissue a radiofrequency current that
The aim of this paper is to present an AR sys- causes tumour cell necrosis for hyperthermia (the
tem that could be used as support for a more accu- local temperature is higher than 60°C and cancer
rate surgical preoperative planning and also for cells are more sensitive to heat than normal cells).
image-guided surgery. The application can support One problem in using radiofrequency tumour
the surgeon during the needle insertion in the radi- ablation technique is the correct placement of the
ofrequency ablation of the liver tumours. needle. To ensure a maximum efficacy of the treat-
ment the needle has to reach the centre of the
tumour.
2 3D MODEL OF THE ORGANS Today surgeons use ultrasound, CT or MNR
images acquired during the needle placement in
A reconstruction of the 3D model of the ana- order to correctly direct the needle towards the cen-
tomical and pathological structures of the patient tre of the tumour. The use of these two-dimensional
is required in order to improve the standard images makes the insertion procedure very difficult
slice view. An efficient 3D reconstruction of the and requires sometimes more than one insertion.
patient’s organs is generated by applying some seg- In addition, the surgeon, in order to destroy all
mentation and classification algorithms to medical tumoral cells, applies the RFA on an extended area
images (CT or MRI). The grey levels in the medical of the liver. In this way a large number of healthy
images are replaced by colors associated to the dif- cells are also destroyed. This practice can cause
ferent organs (Yoo 2004). to the patient a number of other different con-
Nowadays there are different software used sequences. To reduce this problem is of primary
in medicine for the visualization and the analy- importance to reach the centre of the tumour.
sis of scientific images and the 3D modeling of A guidance system of the needle in tumour abla-
human organs; Mimics (Mimics www.materialise. tion procedures can be obtained using Augmented
com), 3D Slicer (Slicer www.slicer.org), ParaView Reality technology. The purpose of this AR appli-
(ParaView www.paraview.org), and OsiriX (OsiriX cation is to provide a guidance system that can
Imaging Software www.osirix-viewer.com) play an help the surgeon during the needle insertion in liver
important role among these tools. RFA. The position and orientation of the ablation
In our application we have used 3D Slicer for the tool are measured using some reflective spheres
building of the 3D model of the patient’s organ; that are detected by an optical tracker in order to
3D Slicer is a multi-platform open-source software measure in real time position and orientation of
package for visualization and image analysis. the real ablation tool.
In the developed AR platform it is also necessary With the superimposition of the virtual mod-
to use an optical tracker that is able to detect some els of the patient’s anatomy (liver, vessels, biliary
intentionally introduced retro-reflective spheres ducts, cancer, etc.) exactly where are the real ones,
(fiducials) placed on the surgical tools. In this way it is possible to make the needle placement task

280

VIPIMAGE15_Book.indb 280 9/8/2015 7:46:18 PM

 less difficult. In this way the surgeon has a sort of the location and orientation of the surgical instru-
x-ray 3D vision of the patient internal anatomy. ments used during the surgical procedure.
To achieve a correct augmentation it is necessary In the first prototype of the system we have
to have a perfect correspondence between the vir- used the Polaris Vicra optical tracker, a commercial
tual organs and the real ones. This is very important device composed from two IR cameras that pro-
in an image-guided surgery application because a vides precise and real-time spatial measurements
very small error in the registration phase can cause of the location and orientation of an object or tool
serious consequences on the patient due to a bad within a defined coordinate system (NDI www.
alignment of the virtual organs on the real ones. ndigital.com). After the test in the operating room
The registration phase is one of the most delicate we changed the tracking system to ensure surgeons
steps in an AR system for surgery and is obtained more freedom of movement. For this reason we
using some fiducial points. These points have been decided to use the Bonita Vicon tracking system
defined on the patient body and identified on the (VICON www.vicon.com). In particular, we used
patient tomographic images. In this way the appli- four IR cameras positioned at the four edges of the
cation can establish the transformation between operating bed. A specific library has been devel-
the “real” world and “virtual” one. oped, tested and integrated inside IGSTK frame-
If the patient position can change during sur- work for interfacing with the Bonita Vicon tracker.
gery it is necessary to fix on the patient body a At the base of the functioning of IGSTK there is
tracker tool that permits to estimate its new posi- the use of a state machine that allows increasing the
tions and orientations. safety and robustness of the toolkit (Cleary 2009).
The use of a state machine, in fact, allows limiting
and controlling the possible behaviours of applica-
4 DEVELOPED APPLICATION tion in order to ensure that it is always in a formal
state planned in the design phase. This guarantees a
The application is provided of an user interface reproducible and deterministic behaviour that elimi-
designed to be simple and functional at the same nates the risk of application design formal errors.
time. In the left side of that interface we placed the When a user sends a request for an action
application control panel. On the right-top win- through a call of a public method, this request is
dow we show 3D models and the augmented real- translated into an input for the state machine. The
ity scene. On the right-bottom there are the three state machine taking into account the current state
smaller windows where axial, coronal and sagittal and the sent input changes its state as expected in
views of CT dataset are placed. the design phase. In any case, the object will always
The application offers to surgeon the possibility be in a known state because each type of behaviour
to study the case study before going in operating has been programmed.
room. There is the possibility to apply a clipping The IGSTK tracker component communicates
modality that permits the surgeon to dissect the with the tracker to obtain the position and orien-
model and study its internal structure changing tation relative to each tool that is present in the
the opacity of the organs. The dissection could be acquisition volume.
made along the three principal axes of CT images.
The application features described till now are
part of what we consider the pre-operative plan- 5 APPLICATION TESTING
ning task. During this task the surgeon can use the
application to study the pathology in a more sim- The first test of application was made on a dummy
ple and natural way than that provided by simple and we tried to overlap the 3D models of a real
CT slice visualization. patient’s organs on the dummy. This test was
For the navigation and augmentation task are not successful because there was a misalignment
devoted to the surgery room. Here the surgeon between the dummy and the virtual models due
needs to use the optical tracker and to carry out to a different thoracic girth between patient and
the registration task. When the registration process dummy. In Figure 1 is shown an augmented visu-
is complete, a virtual ablation tool is shown in 3D alization over a dummy.
view. It is coupled with the real ones and follows To overcome this problem we decided to make
its movements. a preliminary qualitative test of application in the
In the developed AR platform we use an opti- operating theatre. This test is designed to evaluate
cal tracker. This tool is able to detect some retro- the application uncertainty and also to highlight
reflective spheres intentionally introduced in the all the possible issues related to a live use of the
surgical scene. These spheres are placed on the sur- developed platform in the operating room.
gical tools. They provide within a defined coordi- The test was carried out on a patient operated
nate system the real-time spatial measurements of in open surgery with the tumour located on the

281

VIPIMAGE15_Book.indb 281 9/8/2015 7:46:18 PM

 test the correct position of the devices close to the
operating table has been defined and a precise defi-
nition of the more appropriate fiducial points used
for the registration phase has been decided.
Surgeons judged that the volume of interaction
of the NDI Vicra optical tracker was not sufficient.
For this reason we decided to use the VICON Bon-
ita tracker in order to obtain a bigger volume of
interaction. A specific library was developed and
tested for interfacing with this new tracker. This
library was integrated within the IGSTK frame-
work. This new tracker should be tested in the lab-
oratory to evaluate its measure uncertainty and the
in the operating room to evaluate if the new volume
of interaction can be sufficient for the surgeons.
Anyway surgeons were excited for the obtained
Figure 1. Augmented visualization over a dummy. results and for the potentiality of the augmented
visualization in terms of precision of the needle
placement.
surface of the liver; in that surgical case the sur- We are planning to design a quantitative test in
geon was able to see the correct position of the operating room to evaluate and measure the uncer-
tumour and touch it. tainty of the needle placement. We are also taking
Before a traditional needle insertion, we carried into account the possibility to include in the devel-
out the registration phase in order to overlap the oped system the simulation of the virtual model
virtual organs on the real ones. The surgeon veri- deformations due to the breathing of the patient.
fied that a correct overlap with tumour and real
organs was obtained.
The test doesn’t add risks for the patient, if REFERENCES
compared with traditional RFA surgery, because
the application was not used to guide the needle Cleary, K., P. Cheng, A. Enquobahrie, Z. Yaniv, “IGSTK:
insertion. Anyway the patient was informed on the The Book”, May 29, 2009.
nature of the experiment and signed an informed Maier-Hein, L., A. Tekbas, A. Seitel, et al., “In vivo
consent form. accuracy assessment of a needlebased navigation
system for CT-guided radiofrequency ablation of
the liver”, Medical Physics, 2008, Vol. 35, No. 12,
5385-5396, 0094-2405.
6 CONCLUSIONS AND FUTURE WORK
Mimics Medical Imaging Software, Materialise Group,
http://www.materialise.com.
In this paper we present a guidance system for nee- Nicolau, S., A. Garcia, X. Pennec, L. Soler, X. Buy, A.
dle placement in radiofrequency ablation of the Gangi, N. Ayache, and J. Marescaux, “An augmented
liver tumour. reality system for liver thermal ablation: Design and
The application has been tested previously on a evaluation on clinical cases”, Elsevier, 2009.
dummy and afterwards a first test has been carried NDI Polaris Vicra, http://www.ndigital.com.
out in the operating room during an open surgery OsiriX Imaging Software, http://www.osirix-viewer.com.
procedure for the liver tumour resection. ParaView, http://www.paraview.org.
Reitinger, B., A. Bornik, R. Beichel, G. Werkgartner, and
The results of the laboratory test were not as
E. Sorantin, “Tools for augmented reality based liver
expected because there was a difference in thoracic resection planning”, Proceedings of the SPIE Medical
girth between patient and dummy. Imaging 2004: Visualization, Image-Guided Procedures,
We decided to move to the operating room to and Display, pages 88–99, San Diego, February 2004.
test the application in a qualitative manner on a Stüdeli, T., D. Kalkofen, P. Risholm, et al., “Visualization
real patient. The test in the operating room has tool for improved accuracy in needle placement
also the aim to highlight all the possible issues of during percutaneous radio-frequency ablation of
the augmented visualization in the operating sce- liver tumors”, Medical Imaging 2008: Visualization,
nario and to evaluate the correct overlap of virtual Image-Guided Procedures, and Modeling, Pts 1 and 2,
Vol. 6918, B9180-B9180, 0277-786X.
models of the organs on real ones.
Slicer, http://www.slicer.org.
The test in the operating room produced inter- VICON, Bonita, http://www.vicon.com/products/bonita.
esting results for all the aspects related to the con- html.
straints of the particular environment. Operating Yoo, T.S., “Insight into Images: Principles and Practice
rooms are overloaded of systems and devices and for Segmentation, Registration, and Image Analysis”,
an efficient use of space is mandatory. In this first A.K. Peters, Ltd, 2004.

282

VIPIMAGE15_Book.indb 282 9/8/2015 7:46:18 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Anthropomorphic robot forefinger virtually simulated

H.J. Rabiela, B.V. González & G.D.A. Miranda

Metropolitan Autonomous University, Azcapotzalco, México City, México

ABSTRACT: In this paper the virtual simulation of an anthropomorphic robot forefinger, analyzed and
synthesized as a four-link mechanism open chain is proposed. The metacarpal bone is considered fixed;
the proximal, medial and distal phalanges are links in movement of said mechanism. The five degrees of
freedom of the system are reproduced without interference between the links at the corresponding kin-
ematic pairs and using a unique angular parameter to generate the different geometries. The simulation
is parameterized using a spreadsheet and graphic soft ware. After describing the advantages of virtual
simulation their results are presented, which show the possibility of studying the movement of the system
in different circumstances without higher costs. Additionally, application possibilities are shown in the
care of diseases of the fingers.

1 INTRODUCTION

In recent years, technological development has

made possible a new era in the progress of science.
The consolidation of robotic design is an example.
In the last century, the idea of building a mechani-
cal system capable of emulating the movement of
parts of the human body has changed from dream
to reality; progress in creating new mechanisms
that mimic human locomotion, like: Shadow
Robot Company, Robonaut Arm, Robonaut
Hand, Dexhand, LWR Hand and others it show.
Virtual reality, important technology in the life Figure 1. Forefinger.
cycle management of a product, offers solutions
in reduced time including “testing and tuning”.
It is in the virtual environment in which the kin-
ematics simulation of the forefinger is presented,
considering five degrees of freedom. The viability
of the system is verified through the kinematic
and static analysis for different positions of the
system components ensuring no interference
between them.

2 NOTATION

The kinematic model of the forefinger of the right

hand it is shown, with the longitudinal axis of the
bones in the direction of the x axis, in the Figure 1.
Figure 2 shows the hand, corresponding to the
forefinger shown in Figure 1.
To differentiate the forefinger bones, each line
shown with different thickness. The kinematic
lengths of the bones (links) are parameterized. The
parameters used for the kinematic lengths of each
link (bone) are: Figure 2. Hand for the forefinger shown in Figure 1.

283

VIPIMAGE15_Book.indb 283 9/8/2015 7:46:20 PM

 L1 = length kinematic link 1 (metacarpal bone). ranges of angular motion. In Figures 3, 4, 5 and 6
A value of 60 mm was used. links system (bones) are shown.
L2 = length kinematic link 2 (proximal phalanx The links have cylindrical or spherical ends to
bone). A value of 45 mm was used. allow the five degrees of freedom.
L3 = length kinematic link 3 (middle phalanx Cover the bones, not included, it was considered
bone). A value of 30 mm was used. of 12 mm thickness along the y axis.
L4 = length kinematic link 4 (distal phalanx bone).
A value of 16 mm was used.
The angles of rotation between the links con-
verging on a kinematic pair are a function of α.
α = Minimum angle of rotation in a kinematic
pair. A value of 0.01 degrees for each and every
one of the five degrees of freedom was used. The
rotation angles are multiples of α; for example, to
indicate an angle of 45 degrees is used: 4500 * α.
Other variables used are:
r = Radio container. A value of 50 mm was used.
Θ3 = Angle of the longitudinal axis of the link 3
with respect to the x axis.

Figure 3. Metacarpal bone link 1.

3 COORDINATE SYSTEMS

Whether the inertial coordinate system (x, y, z)

located at the beginning of the link 2, see Figures 1
and 2. A non-inertial reference system (x3, y3, z3)
is fixed at the beginning of the link 3. Other non-
inertial reference system (x4, y4, z4) is fixed at the
start of link 4.
Coordinate system (x3, y3, z3) and (x4, y4, z4)
move respectively with links 3 and 4.

4 DEGREES OF FREEDOM

Defined coordinate systems, the five degrees of

freedom of the system are [1, 2, 3]:
• Rotation of the phalanges around the x axis. Figure 4. Proximal phalanx link 2.
• Flexion-extension of the proximal phalanx with
respect to metacarpal bone around the y axis,
with an average range of 80 degrees.
• Flexion-extension of the medial phalanx with
respect to the proximal phalanx about the y3
axis, with an average range of 100 degrees.
• Flexion-extension of the distal phalanx with
respect to the medial phalanx about the y4 axis,
with an average range of 65 degrees.
• Abduction-adduction of the phalanges around
the z axis, with an average range of 20 degrees.

5 SYSTEM COMPONENTS

For the design of system components, it was tried

to reproduce the shape of the actual bones of a
forefinger, with minor variations to achieve greater Figure 5. Medial phalanx link 3.

284

VIPIMAGE15_Book.indb 284 9/8/2015 7:46:21 PM

 Figure 7. System in its initial position.

Figure 6. Distal phalanx link 4.

The bones were designed with a maximum

diameter of 10 mm.

6 SIMULATION OF THE DEGREES

OF FREEDOM

The system in its initial position is shown in

Figure 7. The system, after a rotation about x axis is
shown in Figure 8. The system, after three flexion- Figure 8. Rotation of the phalanges around the x axis.
extension movements (rotation about y4, y3 and y
axes), is shown in Figure 9. The system, after an
abduction-adduction motion (rotation around the
z axis) between the phalanges and the bone meta-
carpal is shown in Figure 10.

7 PRACTICAL APPLICATION

When a person suffers an accident, the effects are

unpredictable. In case of loss of the phalanges of
all the fingers of his right hand, with prosthesis
of the phalanges of his forefinger should grab a
container.
The prosthesis must be programmed to achieve
this objective. In Figure 11 the prosthesis with flex-
ion between phalanges and between the proximal
phalanx and the metacarpal bone is shown. Figure 9. Movements of flexion-extension.
If we consider a container 38 mm in diameter,
See Figures 11 and 12, the lines representing each
link should grab a container of 25 mm radius, due F r − L2 (2)
to the thickness of the bones and its cover.
If the slope of the link 1 is zero and the slope of G ( xD + r ) + r G 2 xD )2 + 2rxD
the link 2 is equal to infinity; the slope of the links 3 m4 = (3)
and 4 can be calculated. The results are shown in ( xD )2 + 2rxD
the following equations: G r (4)
D

F 2 r2 zD L2 − L3 senθ3 (5)
m3 = (1)
2rF xD L3 cos θ3 (6)

285

VIPIMAGE15_Book.indb 285 9/8/2015 7:46:24 PM

 Figure 10. Movement abduction-adduction.

Figure 13. Explode system.

8 ADVANTAGES OF VIRTUAL
SIMULATION

The advantages of virtual simulation systems are:

• Makes it possible to reproduce the dynamic
response.
• Makes it possible to reproduce the correspond-
ing processes of assembly and disassembly. See
Figure 13.
• Enables the prior analysis of their operation.
• Facilitates analysis and synthesis from the point
of view of kinematics.
• Analogous systems with different parameters
Figure 11. Finger around a container. can be simulated.
• You can check the compliance of the design
constraints in its stages of manufacture, opera-
tion and maintenance.

9 ANALYSIS OF RESULTS

The simulation of different degrees of freedom was

possible without interference between the links of
the system in its different images of positions, lim-
iting these to the following values:
• Rotation around the x axis of ± 45 degrees.
Figure 8 illustrates one of these movements.
• Abduction-adduction of ± 30 degrees. Figure 10
illustrates one of these movements.
• Flexion-extension between metacarpal bone and
proximal phalanx of +90 degrees and −30 degrees.
In Figure 9 the flexion of +90 degrees is
illustrated.
• Flexion-extension between the proximal phalanx
Figure 12. Forefinger around of a container. and the middle phalanx of +135 degrees and zero
degrees. In Figure 9 the flexion the +135 degrees
is illustrated. The negative flexion between
θ3 = arctg
t ( m3 ) (7) these two phalanges was considered humanly
impossible.
Since the spreadsheet is checking files (graph- • Flexion-extension between the medial pha-
ics) components and assembly; independent lanx and the distal phalanx of +45 degrees
parameters can be varied to analyze their impact and −30 degrees. In Figure 9 the flexion
on the results. the +45 degrees is illustrated.

286

VIPIMAGE15_Book.indb 286 9/8/2015 7:46:27 PM

 forefinger bones enables better diagnosis and
therefore a better prognosis.
Boutonniere deformity and swan neck deform-
ity can be resolved with treatments of modern
medicine and, in critical cases, complemented by
solutions of general mechanics and kinematics in
particular. See Figures 15 and 16.

10 CONCLUSIONS

• The movements in the kinematic pairs can be

programmed, for any position of the system
components in real time.
• The three-dimensional virtual and dynamic sim-
ulation of a system enables deeper analysis, to
synthesize and optimize it.
• Forefinger design will enable the capacity of
Figure 14. Detail of Figure 9. this subsystem prosthesis body exceed those of
a human subsystem average real.
• Additionally, these capabilities may be maintained
for longer time, depending on the proper selection
of materials also made with computer support.

11 FUTURE WORK

Some of future works are given below:

• Generating physical prototypes, of the compo-
nent parts of the system.
Figure 15. Boutonniere deformity.
• Assembly, the generated component parts, to get
the actual prototype system.
• Instrumentation and control the real system.

ACKNOWLEDGEMENTS

The authors acknowledge the institutional support

of the Autonomous Metropolitan University and
the Azcapotzalco Unit in general and of the Divi-
sion of Basic Sciences and Engineering and the
Department of Energy in particular.

REFERENCES
Figure 16. Swan Neck deformity.
[1] Neil A. Davidoff, Andris Freivalds (August 4,
1993). “A graphic model of the human hand using
An iteratively refined design was necessary to CATIA”. Department of Industrial and Management
make possible the above limits, without interfer- Systems Engineering, Pennsylvania State University,
ence between the links converging on different kin- University Park, PA 16802, USA.
ematic pairs. The limits obtained are higher than [2] Jung Won Kang, Hyun Seok Hong, Bong Sung Kim,
average limits of a person. In Figure 14, detail of Myung Jin Chung, “Work Assistive Mobile Robots
Assisting the Disabled in a Real Manufacturing
Figure 9 in the kinematic pair between the proxi-
Environment”, International Journal of Assistive
mal and medial phalanges, it can be seen that a Robotics and Mechtronics, Volume 8 Number 3,
larger positive flexion angle of 135 degrees may September 2007.
cause interference between these bones. [3] J.K. Salisbury, M.T. Mason, “Robot Hands and the
Deeper knowledge of the behaviour from Mechanics of Manipulation”, MIT Press, Cambridge,
the point of view of the kinematic joints of the MA, 1985.

287

VIPIMAGE15_Book.indb 287 9/8/2015 7:46:29 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

CT based identification problem for the multicompartment

model of blood perfusion

E. Rohan, V. Lukeš & J. Brašnová

Department of Mechanics and NTIS, Faculty of Applied Sciences, The European Centre of Excellence,
University of West Bohemia in Pilsen, Plzeň, Czech Republic

ABSTRACT: This paper deals with modelling the tissue blood perfusion and focuses on the identifica-
tion of the model parameters using the patient-specific data obtained using the standard CT, or MRI
investigations. The research is aimed to develop a complex model which would assist in planning the liver
surgery. Recently we proposed a multicompartment model of the blood perfusion in liver which serves as
the feed back for simulations of dynamic CT investigation (Rohan, Jonášová, & Lukeš 2014). The flow
can be characterized at several scales for which different models are used. Flow in larger branching vessels
is described using a simple 1D model based of the Bernoulli equation with correction terms respecting
the pressure losses due to the dissipation. This model is coupled through point sources/sinks with a 3D
model describing multicompartment flows at the lower hierarchies of the perfusion trees penetrating to
the parenchyma. The compartments in the liver tissue are associated with segments which confine the
flow to subdomains within the organ, and hierarchies which reflect the flow complexity on branching
vascular trees of the portal and hepatic veins. For simulations of the CT perfusion test, a model of the
dynamic transport of the contrast fluid in the compartments was developed; the time-space distribution
of the so-called tissue density can be computed and compared with the measured data obtained form the
CT. To find suitable values of the perfusion model parameters, we formulate an optimization problem
where the objective function expresses the difference between the standard CT images and the corre-
sponding perfusion maps of the contrast agent concentration. This is computed by solving the perfusion
problem to obtain a steady distribution of the blood pressure at all the compartments, and the transport
problem describing the contrast fluid saturation. The optimization problem can be solved using a suitable
gradient-based method. For this the sensitivity analysis formulae were derived using the adjoint system
method. A simplified identification problem was implemented and solved numerically to show viability
of the proposed approach.

1 INTRODUCTION 2 FORMULATION OF THE STATE

PROBLEM
The blood flow is featured by a complex hierarchi-
cal system of blood vessels comprising arteries, 2.1 Structure of compartments
smaller arteries, arterioles, pre-capillary vessels and in a perfused tissue
capillaries. There the blood is filtrated and partici-
We consider a part of tissue occupying the domain
pates in the metabolism process, giving the oxygen
Ω ⊂ R3. The compartments are established as con-
to the tissue and collecting the waste. The ascending
nected parts of the vasculature with a given range
hierarchical system of veins is arranged similarly to
of characteristic vessel cross-sections; such a range
the arterial system, so that the blood circulation at
is indicated by a hierarchy index. Each compart-
the tissue level can be represented by two penetrat-
ment is associated with a segment Ωe Ω at the
ing but separated trees which are connected only at
level of the whole organ (such as liver). Moreover,
the capillary level. Detailed modelling of this part
different groups of the vascular system are distin-
can be based on the homogenization approach, as
guished, namely the arterial and venous trees form
proposed by Rohan & Cimrman (2010). This is an
separate groups. To summarize, any compart-
idealization which neglects shunts existing between
ment belongs to either the arterial, or the venous
the two trees, however, it is the essential scheme of
group, and is associated with a hierarchy and with
the blood flow which should be reflected by any
a segment.
model of the blood perfusion. In the case of liver
We approximate the perfusion at the level of
tissue, vascular geometry at the microlevel was
tissue parenchyma using a macroscopic model
studied by Debbaut et al. (2014).

289

VIPIMAGE15_Book.indb 289 9/8/2015 7:46:31 PM

 describing parallel flows in multiple compartments, The transport of the contrast fluid is described
(Cimrman & Rohan 2007). as the solution of the following problem: Given
The model involves pressures { pi }i , associated { i }i and { pi }i , for a given initial conditions
to each compartment i 1, …, i . Any i-th compart- { i (t = 0, x)})}i { 0i ( x )}i given for x ∈Ωi , find
ment occupying domain Ωi can be saturated from { i (t, x )}i such that, for all compartments i 1, … i ,
an external source (or drained by y a sink); in gen-
eral, the local source/sink flux F i can be defined ∂S i
which describes the fluid exchange with the upper φi
∂t
+ ∇ ⋅ (S i i ) ∑ Z ij (S )J ji
(macroscopic) hierarchies of the perfusion trees. j
Alternatively, the pressure can be prescribed in = Sin f+i +S
S i f−i x Ωi , t > 0 (3)
a given subdomain Σ i Ωi which may represent
junctions with upper hierarchies of the perfusion where
system described by the 1D flow model on branch-
ing networks; in practice, Σ i can be formed by ⎧Si if J ji > 0,
“small” balls, | | << | |. ⎪
Z ij ( ) = ⎨ (4)
j
⎩⎪S if J ji ≤ 0.
2.2 Flow problem
For the weak formulation of the multicompartment To simplify expressions we may introduce cor-
Darcy flow problem the following admissibility responding index sets I+i { j | J ji > } and
sets are needed: V i = {q H 1(Ωi ) | q = p i ∂ i } the Ii {j | J ji ≤ } which, however, are defined
space of test pressures V0i = {q H 1(Ωi ) | q = 0 on point-wise in Ω according to the variation of fluxes
∂∑i}, and = V V 1 ⊕  ⊕ V i . The problem reads, J ji ( x ), x ∈Ωi ∩ Ω j.
as follows: Find p = ( p1, p2 , , p i ) whereby
pj j
, j = 1, , i satisfy
3 IDENTIFICATION PROBLEM BASED
ON KNOWN CONCENTRATION
∫Ω \ Σ
i i
i
pi qi ∫Ω \ Σ
i i
∑ G ij ( pi p j )q i
j
We assume that the total concentration defined in
=∫ F i qi , ∀qi ∈V
V0i , (1)
Ωi \ Σ i (2) can be measured, i.e. c ( x t ) is known for x ∈Ω
and t [ 0,T ] . We define the objective function
for all compartments i 1,…, i . The summation
T 2
∫0 ∫Ω ∑ φ S
in the second term takes only over nonvanishing Φ(α ) i i
− c dxdt. (5)
perfusion coefficients, G ij . The perfusion velocity i
in one compartment is given by the Darcy law and
involvingg the permeability tensors K i = ( K kkli ) , thus, In fact, ∂α ( , , ) = 0, since Φ in (5) depends
wi K i ∇ppi . The flux between different compart- explicitly just on S, thus Φ Φ( ).
ments is computed as J ji = G ij ( pi − p j ) . All param- The identification problem is formulated as an
eters K i ( x ) and G ij ( x ) depend on the local volume optimization problem, see (6), where the unknown
fractions φi ( x ) . parameters to determine are optimization vari-
ables α (α k ), k = , , m + n. We shall consider
2.3 Contrast fluid transport α being associated with the unknown parameters
The blood saturation by the tracer, denoted by Si, ) { }l ( x )), x ∈Ω. Obviously we assume
({G }k ( x ),
is associated with the i-the compartment. Clearly, the problem is discretized using the Finite Ele-
values of Si are restricted by i ∈[ 0,1]. The tracer ment (FE) method, so that x is confined to a nodal
partial concentration is introduced using the vol- positions of the FE mesh. Above k = k( k i j , x ) is
ume fraction φ i of the vessels for each compart- the multiindex numbering of coupled compart-
ment i by c i φ i S i (no summation). This is the ments with enabled flux between i and j at a
equivalent quantity of the so-called tissue density position x i ; l = l (i , x ) is associated with the
measured by the CT investigation. The total appar- compartment index i and x ∈Ωi . We assume
ent concentration (the gray level) is then given as either Gi j ≡ 0, or Gi j Gi j ( k ). We consider only
such couples ( j ), for which Gi j ≡/ 0. A natural
choice is to assume Gi j Gα k , k 1, , m , with a
C ∑ ci = ∑ φ i S i , (2)
constant G > 0 and α k ( x ) ∈R is a Lipschitz con-
i i
tinuous function defined for x ∈(( i \Σ i ) ∩ (Ω j \ j ).
where the summation is taken over all locally over- In analogy, K i K iα l , l m + 1, …, m + n. The
lapping compartments. We shall use the abbrevia- “reference parameters” are estimated using the
tion S = ( 1, , i ). analysis of the vascular trees according to a method

290

VIPIMAGE15_Book.indb 290 9/8/2015 7:46:31 PM

 pursuing the works by Huyghe & Campen (1995b) ∂ψ j
and Hyde et al. (2013). + w j ⋅ ∇ψ j + f−jψ j − ∑ J i j (ψ j − ψ i )
∂t i ∈I j +

3.1 The optimization problem = φ j ⎛ ∑ φ kS k − c ⎞ ,

⎝ k ⎠
Find α ∈ A ⊂ R m + n such that:
g J i j = Gi j ( p j − pi ) and wi
recalling K i ∇ppi .
Φ( ) → min ⎫ 2. Find λ j ∈ V0 j , j 1 i such that
⎪
p = p( ) solves ( ) ⎬ (6)
T
S = S( ) solves ( ) ⎪⎭ aαj ⎛⎝ q, λ j ⎞⎠ ∑ bαij ⎛⎝ q, λ j − λ i ⎞⎠ ∫0 aα ⎛⎝ q S ψ
j j j⎞
⎠ dt
d
i j

where A is a given admissibility set.

T
(
− ∑ ∫ bαij q,ψ j Zi j S − ψ i Z ij (S ) dt ,
0
)
The following bilinear forms and the linear i j
functional are employed:
for all q ∈VV0 j .
a ( p, q ) The total sensitivity of Φ w.r.t. to the opti-
∫Ω
i i
∇pp ⋅ ∇q ,
i Σi mization parameters α for any admissible state
b ( p, q ) ∫
ij
G ij pq , g i (q) = ∫ F i q. ( p( ) S ( p )) ∈ V × S can now be evaluated using
Ω Ωi \ Σ i
i i
the following formula:
Further, by bαij ( p, q ) the dependence of G ij ( x )
∂ αtot ( , ( p,, ), ( λ ,ψ )) = ∂αtot α, , )
on α k is indicated; recall that index k = k(
k (ii j , x ) is
⎛ ⎞
associated with the two compartments. Similarly,
aαi ( p, q ) depends on α l which is associated with =∑ ⎜ i ⎛ i
⎜ ∂α aα ⎝ p , λ ⎠ +
i⎞
∑
∂α αij ⎛⎝ pi − p j , λ i ⎞⎠ ⎟⎟
i ⎜⎝ j ≠i ⎟⎠
K i (x ) . T
The Lagrange function associated with (6) is + ∑∫ ∑
0
∂α bαij ⎛⎝ pi − p j , Z ij (SS )ψ i ⎞⎠ dt
defined by i j ≠i
T

L(α ( p, ) ( λ ,ψ ))
) Φ(α ) 0 ∫
− ∂α aαi ⎛⎝ pi , S iψ i ⎞⎠ dt. (8)

⎛ ⎞
+ ∑ aαi ⎛⎝ pi , λ i ⎞⎠ ∑ bαij ⎛⎝ pi p j , λ i ⎞⎠ − g i ( λ i )⎟
i ⎝ j i ⎠ 4 SIMPLIFIED PROBLEM
T ⎛ ∂S i i i ⎞ i
+∑∫ ∫0 ⎜ ∂t + ∇ ⋅ ( S ) ψ dtdx
d We consider a simplified problem which was
Ωi ⎝ ⎠ implemented and solved to examine some poten-
i
tial difficulties arising in this kind of inverse prob-
T
+ ∑ ∑ ∫ bαij ⎛⎝ pi − p j , Z ij ( )ψ i ⎞⎠ dt lems. The liver perfusion is described by a reduced
0
i j ≠i multicompartment model which incorporates the
T vascular tree of the portal vein only, Figure 1(a).
−∑∫ ∫Ω (Sin f+ + S
i i i
f−i )ψ i dxdt , Upon decomposing the tree in 2 compartments,
0 i
i the hierarchical flow is represented by three com-
partments which are defined in terms of two per-
where λ i and ψ i are the Lagrange multipliers; meabilities, 1, K 2 and two inter-compartment
λ i ∈V
V0i and ψ j = 0 at t T . perfusion coefficients, G12 and G23. The 3rd com-
partment is degenerate, i.e. the pressure p3 = 0 is
3.2 Optimality conditions prescribed. The flow is described by only y two pres-
sure fields, p ( p1, p2 ), such that p j j
j = 1, 2.
The optimality conditions associated with feasibil- Thus, the state problem is given by the following
ity of the state problem read: 2 equations:
δS δS δp δp=0, (7)
∫Ω K ∇p ⋅ ∇q + ∫ΩG2 ( )q1 = ∫Ω F 1q1,
1 1 1 1 1
− 2

for admissible variations δ S and δ p. From (7) we

∫Ω K ∇p ⋅ ∇q + ∫Ω (G1 ( − ) + G3 p )q = 0,
2 2 2 2 2 1 2 2 2
obtain the adjoint problem which can be solved in (9)
two steps:
1. Find {ψ j }, j 1 i such that ψ j (T ) = 0 and for all q1 q 2 ∈ V0 = L2 ( ). No Dirichlet conditions
{ψ } satisfy (for x Ω, t ∈]0,T [ ),
j
are needed, so that also V j = L2 ( ) j = 1, 2.

291

VIPIMAGE15_Book.indb 291 9/8/2015 7:46:41 PM

 the distribution of fluid sources of the 1st com-
partment; in fact, F 1 is a given function with local-
ized supports to small subdomains ω k which are
situated at connection points of terminal branches
of a truncated perfusion tree describing the upper
hierarchies of the portal vein.

4.1 Identification problem for G21

The permeability tensors K i , given for any com-
partment i can be computed “pointwise” by
the averaging procedure proposed by Vankan
et al. 1997 using the theory derived in (Huyghe &
Campen 1995a), (Hyde et al. 2013). Although Gi j
can be computed as well, in this case the procedure
is more cumbersome, requiring additional fluid
problems to be solved. Therefore, we propose to
obtain relevant values of Gi j by solving an inverse
problem. In our simplified problem we shall assume
that the correct inter-compartmental fluxes J can
be measured, so that G21 is computed by minimiz-
ing the following objective function:
2
ϕ (α ) = ∫ G21 (α ))(( pi pj)− J ,
Ω

where J is the desired data, i.e. a “measured” flux

between the compartments 1 and 2. The identifica-
tion problem reads, as follows: Find α ∈A such that

ϕ (α ) → min
p = p(α ) solves ( ), (10)

where A is a given admissibility set. The state problem

(9) is approximated using the FE method. Therefore
α (α k ) is a piece-wise constant function, given
by values α k on each element Ω k ⊂ Ω, k 1, , m .
Then, G21 ( x ) = G kα k for x ∈Ω k is established using
G k which for a given tree, see Figure 1, decomposed
in two compartments is computed using the aver-
aging method (Hyde et al. 2013). Using the same
approach, the permeability tensors K 1, K 2 are
defined by an averaging procedure at each element
Ω k, however, these parameters are fixed, not consid-
ered for the identification. For a given FE approxi-
mation and J k given by constants on elements Ω k,
the objective function can be rewritten, as follows:
Figure 1. Identification of spatial distribution of per-
fusion coefficients G21 ( x ) using parameters α. Perfusion tree m 2
of the hepatic vein (a), FE model (b, c, d)—a part displayed ϕ (α )=∑∫ Jk( ) − J k , (11)
(planar cut). Initial (b) and optimized (c) fluxes J(x) at finite Ωk
k =1
elements. Optimal parameters α kopt at elements Ωk (d).
where J k ( x ) = α k G k ( p ( x
x)) − p ( x )) .

This model assumes a given constant pressure

4.2 Sensitivity analysis
associated with the parenchyma sinusoids, which
enables to solve a “half-problem”; only the flow Since the contrast fluid transport is not involved
through the saturating vasculature of the portal in (10), also the sensitivity analysis simplifies. The
vein is concerned. Let us recall that F1 represents adjoint state (λ1,λ2) ∈ (L2(Ω))2 is the solution of

292

VIPIMAGE15_Book.indb 292 9/8/2015 7:46:47 PM

 ∫Ω K ∇
1 1
⋅ ∇q1 + ∫ 1
2
( 1
− 2 ) q1
Ω

= −2∫ G21 (
Ω
( 1
− 2
) − J )G21q1,
∫Ω K ∫Ω (G1 ( ) + G32 ) q2
2
∇ 2
⋅ ∇q 2 2 2
− 1 2

(
= 2∫ G21 ( p1 − p2 ) J G21q 2 ,
Ω
) (12)

for all q1 q 2 ∈ V0 = L2 ( )). We recall that G21 G12

depends on α , whereas K 1, K 2 and G32 are fixed. The
total gradient of ϕ defined in (11) is computed by

∂αtotk φ , p) = 2∫ (α k G k ( pi − p j ) − J k )G k ( pi − p j )
Ωk
+∫ G k ( p − p )( λ − λ ) ,
1 2 1 2
Ωk

so that the k-th component is evaluated using inte-

grals on Ω k.

4.3 Numerical results

The problems (9) and (12) have been implemented
in the SfePy code (Cimrman and et al. 2011) using
the FE method. The liver geometry was recon-
structed using anatomically correct data, how-
ever the portal venous tree was generated using
the Global Constructive Optimization method,
as reported in (Lukeš, Jiřík, Jonášová, Rohan,
Bublík, & Cimrman 2014) using the algorithm by
Georg, Preusser, & Hahn. The FE model included
322 elements. Using the approach employed by Figure 3. Simulation of the contrast fluid saturation
Hyde et al. (2013) the initial guess of the perfusion Si (Ω,t) in the 3 compartments, i = 1, 2, 3, for t = 3,50
coefficients G21 G was obtained for each element. sec. The black-and-white pictures are the reconstructed
For this auxiliary flow problems were solved on concentrations which correspond to the “tissue density”
segments of the perfusion tree. The desired fluxes J measured by the CT test.
were computed by solving a reduced flow problem
on the perfusion tree, see e.g. (Jonášová, Bublík, &
Vimmr 2014). The initial values α kini := 0.5, thus the optimal values were expected α kopt ≈ 1, although,
obviously, the fluxes J defined in (11)2 in the sense
of optimization problem (10) may not be coher-
ent with the definition of J based on a different
flow problem. Thus, the example reported in this
paper was intended rather to study some numerical
aspects of the identification problem.
In Figure 1(b,c), the initial and optimized fluxes
defined in (11)2 are displayed, whereas the opti-
mal values of α k on elements Ω k are depicted in
Figure 1(c). The objective function was reduced by
≈ 25% of its initial value, only 25 iterations by a
gradient SQP method were needed, see Figure 2.

5 CONCLUSIONS

The complete model describing the CT exami-

Figure 2. Optimization problem (10): objective func- nation of the liver perfusion, thus, including the
tion decreasing in iterations. contrast agent transport, has been implemented

293

VIPIMAGE15_Book.indb 293 9/8/2015 7:46:52 PM

 in the SfePy code (see (Cimrman & et al. 2011)). REFERENCES
Both the portal and hepatic vein vasculatures were
reconstructed using artificial trees (Georg et al.). Cimrman, R. & et al. (2011). SfePy home page. http://sfepy.
So defined portal and hepatic vein segments ena- org. Software, finite element code and applications.
bled simulation of the upper-level hierarchical flow Cimrman, R. & E. Rohan (2007). On modelling the par-
using the “1D” model (Jonášová, Bublík, & Vimmr allel diffusion flow in deforming porous media. Math.
and Computers in Simulation 76, 34–43.
2014). In Figure 3, we illustrate the flow and the Cimrman, R. & E. Rohan (2012). On identification of the
tracer distributions governed by the equations (1) arterial model parameters from experiments applica-
and (3). The flow in the lower-level hierarchies was ble “in vivo”. Mathematics and Computers in Simula-
approximated using the Darcy-type flow model tion 80, 1232–1245.
with 3 compartments: one associated with the por- Debbaut, C., P. Segers, P. Cornillie, C. Casteleyn, M.
tal vein structure, the second with the hepatic vein Dierick, W. Laleman, & D. Monbaliu (2014). Analyzing
structure, and the third one representing the lower- the human liver vascular architecture by combining
most level of the parenchyma corresponding to the vascular corrosion casting and micro-ct scanning: a
liver lobes. The mesh of the FEM discretization feasibility study. J. of Anatomy 224(4), 509–517.
Georg, M., T. Preusser, & H.K. Hahn. Global con-
comprised 11019 tetrahedral elements, approxi- structive optimization of vascular systems. Technical
mately 7300 degrees of freedom associated with Report: Washington University in St. Louis.
the pressure in all compartments were involved Huyghe, J.M. & D.H. Campen (1995a). Finite defor-
in the computations. The model was coupled with mation theory of hierarchically arranged porous
the upper-level “1D” flow through 28 junctions of solids—i. balance of mass and momentum. Int. J.
which 14 were the sinks and 14 were the sources Engrg. Sci. 33(13), 1861–1871.
defining the domain Σ1 and functions F i . Huyghe, J.M. & D.H. Campen (1995b). Finite deforma-
We have formulated the optimization problem tion theory of hierarchically arranged porous solids,
aimed at identification of the unknown model part I., II. Int. J. Engrg. Sci. 33(13), 1861–1886.
Hyde, E.R., C. Michler, J. Lee, A.N. Cookson,
parameters defined pointwise in the whole domain, R. Chabiniok, D.A. Nordsletten, & N.P. Smith (2013).
(Cimrman & Rohan 2012), namely the permeabili- Parameterisation of multi-scale continuum perfusion
ties K i for each compartment and the perfusion models from discrete vascular networks. Med Biol Eng
parameters Gi j controlling the fluid flow between Comput 51, 557–570.
different compartments. The state problem consists Jonášová, A., O. Bublík, & J. Vimmr (2014). A compara-
of two parts: the perfusion problem, involving the tive study of 1d and 3d hemodynamics in patient-
above mentioned unknown parameters, and the specific hepatic portal vein networks. Appl. and Comp.
contrast fluid transport problem which involves Mech. 8, 177–186.
the perfusion velocities. The sensitivity formula Lukeš, V., M. Jiřík, A. Jonášová, E. Rohan, O. Bublík, &
R. Cimrman (2014). Numerical simulation of liver
(8) provides the gradient of the objective func- perfusion: from CT scans to FE model. Proc. of
tion w.r.t. all parameters; since the adjoint state the 6th European Conference on Python in Science
technique is used, the numerical complexity of the (EuroSciPy 2014).
sensitivity does not increases proportionally to the Rohan, E. & R. Cimrman (2002). Sensitivity analysis and
number of the optimization variables, (Rohan & material identification for activated smooth muscle.
Cimrman 2002). Computer Assisted Mechanics and Engineering Science
The optimization problem (6) is nonconvex, 9, 519–541.
therefore, one may hope to compute local minimiz- Rohan, E. & R. Cimrman (2010). Two-scale modelling
ers only. This deteriorates efficiency of the iden- of tissue perfusion problem using homogenization
of dual prous media. Int. J. for Multiscale Comput.
tification, although the simplified identification Engrg. 8, 81–102.
problem (10) reported in section 4 proved viability Rohan, E., A. Jonášová, & V. Lukeš (2014). Complex
of the proposed approach. The implementation of hierarchical modeling of the dynamic perfusion test:
the full problem (6) is now being tested, new numer- application to liver. In 11th World Congress on Com-
ical issues appear and will have to be sorted out. putational Mechanics (WCCM XI).
Vankan, W., J. Huyghe, J. Janssen, A. Huson,
W. Hacking, & W. Schreiner (1997). Finite element
ACKNOWLEDGMENT analysis of blood flow through biological tissue. I.
Jour of Engrg. Sci. 35(4), 375–385.
This research is supported by project NT 13326
of the Ministry of Health of the Czech Republic
(CR), in part by project GAČR 13-00863S of the
Sci. Foundation of the CR. Publication enabled
due to project LO1506 (MŠMT) of CR.

294

VIPIMAGE15_Book.indb 294 9/8/2015 7:46:56 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

A study on discrimination of SIFT feature applied to binary images

Insaf Setitra
Research Center on Scientific and Technical Information Cerist, Algeria
University of Science and Technology Houari Boumediene, Algeria

Slimane Larabi
University of Science and Technology Houari Boumediene, Algeria

ABSTRACT: Scale Invariant Feature Transform (SIFT) since its first apparition in 2004 has been (and
still is) extensively used in computer vision to classify and match objects in RGB and grey level images
and videos. However, since the descriptor used in SIFT approach is based on gradient magnitude and
orientation, it has always been considered as texture feature and received less interest when treating binary
images. In this work we investigate the power of discrimination of SIFT applied to binary images. A theo-
retical and experimental studies show that SIFT can still describe shapes and can be used to distinguish
objects of several classes.

1 INTRODUCTION The remainder of the paper is organized as fol-

lows: We describe in section 2 the descriptor, in
Description of a shape is very important for many section 3 we describe our shape retrieval approach
applications. Usually, accuracy of further process- followed in section 4 by the shape classification.
ing such as matching and classification relies on the We present and discuss results in section 3.
description of the shape. Morphological features
are usually used to describe binary shapes. Shape
Contexts [10], Graph Transduction [14], Inner Dis- 2 SHAPE DESCRIPTOR
tance [11], and full and partial matching [12] are
some examples of descriptors. In this work we ana- In [7], scale invariant feature transform has been
lyze a descriptor that has been extensively used for proved to be robust to affine transformations (scale
gray level shapes but not for binary shapes. Scale and orientation) and slight changes in illumination.
Invariant Feature Transform proposed by David In [8] its robustness has been proved mathemati-
Lowe (2004) [7] is commonly known to be a texture cally. Although the descriptor was successfully
feature and is used to describe gray level shapes applied to tasks of classification, matching, and
in terms of gradient orientation and magnitude. reconstruction for color and gray level images, it
However, previous to computing gradient orienta- lacks application to binary images. Theoretical
tion and magnitude, the shape is convoluted several background and experiments shows that SIFT can
times so that to keep strong edges. This cue is very also be applied to binary images. We describe in
important to describe binary shapes and makes the what follows steps of computation of the descrip-
feature suitable for them. tor and the effect of each step on a binary image
The most relevant work dealing with SIFT on description.
binary images is devoted to hand gestures classi-
fication [13] by the use of SIFT feature applied to
2.1 Keypoint localization
binary masks. Authors claim to improve accuracy
of classification and to decrease time processing The most distinctive feature in a binary image is
since the descriptor is not sensitive to illumination its contour, and more precisely curvatures usually
changes and less information is needed to quantify contained in the edges of the image. In order to
the keypoints. describe binary shapes we first localize these fea-
In this work, we enlarge the concept of discrimi- tures or keypoints.
nation of SIFT feature on several shapes. We first To do so, let us consider an original binary image
give a review of SIFT computation and study its I(n × m). The image is convolved c times using a
effect on binary image. Experiments are conducted Laplacian of Gaussian filter (we describe how c is
in order to verify robustness of the descriptor on computed below). The expression of Laplacian of
both retrieval and classification. Gaussian Filter is given by:

295

VIPIMAGE15_Book.indb 295 9/8/2015 7:46:56 PM

 1 ⎡ x 2 + y2 − x + 2y ⎤ maximum or minimum over its 18 neighbors
2 2

LoG ( x y )=− 4 ⎢1 − e 2σ
⎥ (1) defined above, it is considered potential maxima
πσ
σ ⎣ 2σ 2 ⎦ and its coordinates are saved.
Once potential maxima over all scales defined, a
where x and y are respectively row and column coor- further refinement is performed in order to eliminate
dinates in the shape to be convolved and σ is the weak edges. This is done by suppressing from each
standard deviation of the Gaussian distribution. convoluted image Li + 1 potential maxima which are
Using the Laplacian of Gaussian besides being an less than an input threshold (measure of stability).
accurate edge detector due to its precision in detect- Maxima detected in this step are returned as
ing zeros-crossings, has many other advantages. output of this phase and are called keypoints. Each
First, the Laplacian of Gaussian is the second deriv- maxima is defined by its row coordinate r, column
ative of an image convolved with a Gaussian which coordinate c and its radius rad which is computed
prevents from noise in the image. Secondly, as the as follows:
filter is pre-calculated in advance, processing time is
reduced. Convolving the image several times allows
to define the zeros-crossing over different scales. rad σ k i −1 × 2 (4)
Number of convolutions c is computed as
follows: Changes in scale allows to see the image across
different resolutions. The next step is to describe
keypoints detected in this step.
gσ ⎤
⎡ log c − log
c=⎢ ⎥ (2)
⎢ log k ⎥
2.2 Keypoint description
where ⎡ is the ceiling function which maps a real A binary image can be defined as a matrix which
number to its smallest following integer, σ and σc can have only two possible values typically relative
are respectively initial and final standard deviation to black and white. Depending on the image rep-
of the Laplacian of Gaussian Filter and are given resentation, those two values can be either 0 and
as input parameters. log is the natural logarithm of 255 or 0 and 1. In a sake of simplicity we use the
base 2 and k is the scale step. second representation in this paper.
At each of the c convolutions applied to I, σ is Description of keypoints localized in the first
updated as follows: step is done by first, computing the gradient mag-
nitude and direction of the original image I around
σ i = σ × k i −1 (3) its keypoints and quantifying orientations weighted
by the magnitude to form the final descriptor. We
where 1 ≤ i ≤ c is a counter ranging from 1 to c. The describe the whole process in what follows.
initial image I is convoluted with the new σ using First gradient of the original image I:
the formula 1. ( x ( ∂I//∂ ) I y = ( ∂I /∂y )) is computed. Hence,
Once the initial image I is convoluted c times, Gradient of an image can have either −1, 0 or
local maxima are defined and their coordinates 1 value in both directions.
(row, column indices and radius) are returned. The Gradient magnitude and Gradient orienta-
maxima is defined as follows: tion are computed. As gradient orientation is
Let Li, Li + 1 and Li + 2 be three adjacent images based on the gradient magnitude, 8 Gradient
computed as follow: orientations can then take up to eight values:
− 3π , − π , − π , π , 0, 3π , π , π .
LoG ( x y; 4 2 4 4 2 4
Li i ) ∗ I, Li = LoG
L G (x y i +1 ) I
Once orientations of the gradient for the whole
Li LoG ( x y; i +2 ) ∗ I image computed, the second step is to get for each
keypoint its window computed as follows:
The local maxima is first computed on the neigh-
borhood of Li + 1 which is the center of the current xo ⎡⎢ max( , r rad
add ) ⎤⎥ , yo ⎡⎢ max( , c − rad ) ⎤⎥
3 − tuples of convoluted images (Li, Li + 1, Li + 2).
More precisely, each pixel p(x, y) in Li + 1 is com- xe = ⎡⎢ min
i (r + rad , n ) ⎤⎥ ye = ⎡⎢ min(c + radd , m ) ⎤⎥
pared to its 8 neighbors. If p is either minimum or
maximum, it is considered as a potential maxima where xo, yo, xe, ye are respectively the (row, col-
of the inner scale and its coordinates are stored. umn) of the beginning respectively end of the key-
Then, p is compared to its 18 neighbors in a 3×3 point window.
windows in Li and Li + 2 centered at (x, y). r, c and rad are row, column coordinates and
Comparisons are done the same way as in the radius of the keypoint. n and m are the two dimen-
inner scale i.e. if the central pixel in Li + 1 is either sion sizes of the image I.

296

VIPIMAGE15_Book.indb 296 9/8/2015 7:46:56 PM

 The next step is to divide that window into 16 Step 4: Construct the vote matrix F. F takes
blocks arranged in a 4×4 manner (4 blocks hori- indices of column for each line in D where D is
zontally and 4 blocks vertically). Number of pixels minimum. The vector F reflects then to which
in each block depends on the size of the window image in the dataset, a keypoint in B looks more
and is rounded to its nearest integer. Histogram similar. Number of lines in F is equal to n (n is
orientation of each block is then computed. number of keypoints of B) and number of col-
umns to 1.
Step 5: The last step is to compute number of
3 SHAPE RETRIEVAL
occurrences of each image index in F (values in F
which range from 1 to s). V is then a s × 1 matrix
Previous steps allow us to describe binary shapes
which contains for each image index, number of its
in a form of a set of keypoints. Each keypoint is
occurrences in F. V is then sorted in a increasing
a 128×1 vector. Matching two shapes is then per-
order and the first index represent the best match-
formed by matching each keypoint of the first
ing image among all s images.
image to one keypoint of the second image by
Step 6: The ancient dataset is updated where the
computing minimal distance between them. In the
best matching image found in step 5 is removed. If
following are the notation used for matching fol-
the dataset is empty or number of images(y) which
lowed by the matching process.
should be matched is reached, then the retrieval is
stopped. Otherwise steps from 1 to 6 are repeated
3.1 Notation used
for the query shape and the new dataset.
Let B be a n × 128 matrix representing all n key-
points of an image query I and Ba be a na × 128
matrix representing all na keypoints of an image Ia 4 SHAPE CLASSIFICATION
of a dataset. In order to get the best match of I
over all Ia’s images where a ranges from 1 to s (s is Classification aims to classify patterns from a
number of images in the dataset), best match of B dataset using a set of patterns for training. In our
to one Ba is computed and the process is repeated context, the dataset is comprised of images and is
to all remaining images. divided into two subsets: a training set and a test
set. The training set is used to train the classifier
3.2 Retrieval by matching and the test set is using for tests. Classification of
keypoints independently images using SIFT is very difficult because of the
Step 1: The first match of B to Ba is done by first high dimensionality of SIFT feature and the non-
computing the angles between each keypoint of B fixed feature representation. Classification using a
to each keypoint of Ba and get the minimum value majority vote such as done previously is very time
for a best match. Angles between keypoints is com- and memory consuming since each key point of
puted as follow: each image is stored and compared over all key
points of all images of the training set. A good
compromise between accuracy and computations
Anglesa = acos(
acos B Bat )
(5) will be to have a new representation of image
features with a fixed length. The bag of features
where acos is the inverse cosine, Bat is the transpose method seems to be a good solution. In what fol-
of Ba and * is the dot product of the two matrices. lows, we describe briefly how we use Bag of Fea-
Step 2: Angles will be a n × na matrix where each tures in classifying binary images using sift. The
line is relative to a keypoint in B and each column is first step is to quantize SIFT features and build
relative to a keypoint in Ba. The matrix Angles is then visual vocabulary using K-means and is described
sorted column wise in increasing order. The mini- in subsection 4.1
mum distance is then the first column of Angles. Then, for a new instance, the second step is to
Step 3: Step 1 and 2 are repeated to all Ba where a classify it using one of supervised learning classifi-
ranges from 1 to s (number of images in the dataset). ers. We choose the Nearest Neighbor classification
For each Ba, Anglesa are concatenated vertically so and describe it in subsection 4.2.
that to form the matrix D of all minimum angles
between keypoints of B and Ba (1 ≤ 1 ≤ s). D is then
4.1 Building visual vocabulary using K-means
D = Angles1 Angles
g 2 .. Angless SIFT features of images of training are concate-
(6) t
nated into one 128 × d matrix M where d ∑ ni ,
where || is a concatenation operator. Each line in D i =1
represents a keypoint and each column the mini- ni is number of keypoints of image Ii and t is the
mum angle with a keypoint in the ath image. number of images of the training set.

297

VIPIMAGE15_Book.indb 297 9/8/2015 7:46:58 PM

 Rows of the matrix M are then clustered using image is attributed to the test image. We use
K − means clustering algorithm described in two distances, Euclidean and χ 2. χ 2 distance is
algorithm 1. described as follows:
Result of clustering is then l stabilized clusters, Let HIi be the feature vector representing the
each cluster is represented by its center which is a image Ii. Note that This feature if the normalized
one of the 128 dimensional vector of keypoints histogram of visual words of the image Ii. Let HTi
and represents the visual vocabulary. Each image be one of the feature vector representing on of the
of the training and test set is then represented by training examples. χ 2 distance is described as:
a histogram where number of bins is l and each
bin stands for number of keypoints nearest to one l
HI i − HTi 2
of the l centers. Figure 1 represents an example of χ2 = ∑ (7)
Bag of Words representation. i =1 HTi

4.2 Training and classification using bag of words where l is the dimension of the feature vector equal
to number of clusters of the K − means algorithm.
After having Bag of Features descriptor of each
of training and test images, the distance between
Bag of Features of test images and training images 5 EXPERIMENTAL RESULTS
is computed, for each test image, the minimal dis-
tance is chosen, and label of the nearest training 5.1 Shape retrieval accuracy
of the dataset ETH-80
Retrieval experiments are done on the dataset
ETH-80 [9]. The dataset comprises 8 categories
each of which has 400 images. We choose ran-
domly 10 images of each category which results
in 80 images in the dataset prepared for retrieval.
Then, we chose randomly an image from each
category for retrieval. We use the bullseye score
for retrieval accuracy measured as the percent
of correct retrievals in the top 20 ranks. Table 1
shows retrieval accuracy of retrieval. The first
column represents the query shape, the second
column represents shapes retrieved in the order
of similarity i.e. similar shapes start in the top left
of the column and are ordered line wise. Names of
shapes are the same as in the original database1.
The third column stands for the processing time.
The table shows that best rates are for shapes with
few curvatures whereas weak rates are observed
for shapes with high curvatures. This is because
keypoints in curvatures are repetitive in many
shapes. However, processing time for shapes with
few curvatures is higher. Besides, shapes with few
curvatures are most often matched to shapes with
few curvatures, examples include: apple, pear and
tomato. However, some shapes of high curvatures
are confused with shapes with few curvatures.
The reason is that, many keypoints in a shape
can be matched to only one keypoint in another
shape and distance is minimal. The opposite is
also true. One solution could be to match a key-
point in a query shape to only one keypoint of a
shape in the dataset. We let this improvement to
a future work.
Figure 1. Example of Bag of Features representation
of an image I (a) the image I to be represented by the
visual vocabulary, (b) Visual Vocabulary represented by
1
its centers, (c) Bag of Features of the image I. http://people.csail.mit.edu/jjl/libpmk/samples/eth.html

298

VIPIMAGE15_Book.indb 298 9/8/2015 7:46:58 PM

 Table 1. Retrieval accuracy on ETH-80 dataset. In a second experiment, we choose a small ratio
of the test set (30%) and fix it for all percentages
Query Bullseye Processing of training. Table 3 represents this experiment.
image score (%) time (ms) Size of the training set in both cases do not exceed
Apple20 100 135.58 70% which represent the classical percentage used
Car4 50 53.69 in most classification experiments (2/3 training
Cow2 50 55.67 and 1/3 test). We analyze classification accuracy
Cup15 100 133.21 of RGB images and their binary masks (binary
Dog31 50 88.36 images) using the same parameters of the code
Horse20 80 82.72 book (same number of iterations of K-means
Pear20 100 87.36 clustering, same number of features in the visual
Tomato36 100 107.81 vocabulary, same distance measure).
Table 2 and 3 show that classification accu-
racy of RGB images in most cases outperform
Table 2. Overall classification accuracy on ETH-80 the one of the binary images of the original RGB
dataset by varying size of training set and test set. images. However, the difference is slight and in
one case, classification of binary images outper-
Accuracy % Accuracy % formed classification of RGB images. In order
to analyze more deeply the classification results,
Training % Test % Binary RGB
we moreover generate for each ratio confusion
20 80 83.01 90.22 matrices. We generate more than 60 confusion
40 60 93.91 96.79 matrices by varying the size of both training and
60 40 99.03 98.71 test set, however, we kept only the most signifi-
cant ones which correspond to ratios of Table 2.
Confusion matrices show that classification
accuracy is enhanced in RGB images compared
Table 3. Overall classification accuracy on ETH-80
dataset by varying size of training set and fixing size of to binary images, however, the difference is still
test set. slight. Also, in some cases, classification of RGB
images. Some interesting points can be seen in the
Accuracy % Accuracy % confusion matrices: Confusion of RGB images
can occur even when the shape is different, exam-
Training % Test % Binary RGB ple of such a confusion can be seen in Figure 2
where horses have been confused with cows. This
10 30 77.77 84.61 confusion is less apparent in the binary images
20 30 84.82 88.35
and accuracy of horse classification can reach
30 30 90.49 93.05
40 30 92.20 94.87
100% for binary images while it does not for RGB
50 30 94.23 95.51 images. Such errors occurred for horses with same
60 30 96.68 97.75 colors as cows and dogs and when some animals
70 30 99.14 99.25 have same skin (texture). The same observation
can be made for classes “Dog” and “Cow”. For
all other classes, classification accuracy of RGB
images is slightly higher than classification accu-
5.2 Classification accuracy
racy of the same binary images.
of the dataset ETH-80
Experiments on the dataset ETH-80 are divided
into two parts. The first part concern classification
accuracy of RGB images, the second part concerns
classification accuracy of the same binary images.
Extensive experiments of ETH-80 are done
by dividing the whole dataset into training and
classification. Division is done following several
percentages from 10 to 70% where this percent-
age represents ratio of training set with the whole
dataset. In a first experiment, we vary size of both
training and test set where sum of their two ratios
is equal to 100%. By doing this, we insure not to Figure 2. Confusion matrices for ETH-80 with training
include images of the training set into the test size 20%, test size 80% (right matrix binary images, left
set. Table 2 represents accuracy of this variation. matrix RGB images).

299

VIPIMAGE15_Book.indb 299 9/8/2015 7:46:59 PM

 [2] Learning Context-Sensitive Shape Similarity by
Graph Transduction, Xiang Bai and Yang, Xingwei
and Latecki, L.J. and Wenyu Liu and Zhuowen Tu,
Pattern Analysis and Machine Intelligence, IEEE
Transactions on, 2010.
[3] Xiang Bai and Cong Rao and Xinggang Wang,
Shape Vocabulary: A Robust and Efficient Shape
Representation for Shape Matching, Image Process-
ing, IEEE Transactions on, 2014.
[4] Haibin Ling and Jacobs, D.W., Using the inner-dis-
tance for classification of articulated shapes, Com-
Figure 3. Confusion matrices for ETH-80 with training puter Vision and Pattern Recognition, 2005. CVPR
size 40%, test size 60% (right matrix binary images, left 2005. IEEE Computer Society Conference on,
matrix RGB images). [5] J.F. Nunes, P.M. Moreira, J.M.R.S. Tavares., Simple
and fast shape based image retrieval, Computational
Vision and Medical Image Processing: VipIMAGE
2011.
[6] Bouagar, Saliha and Larabi, Slimane, Efficient
descriptor for full and partial shape matching Mul-
timedia Tools and Applications, 2014.
[7] Lowe, David G., Distinctive Image Features from
Scale-Invariant Keypoints, Int. J. Comput. Vision,
2004.
[8] Jean-Michel Morel and Guoshen Yu, Is SIFT scale
invariant?, Inverse Problems and Imaging, 2011.
[9] Leibe, B. and Schiele, B., Analyzing appearance and
contour based methods for object categorization
Figure 4. Confusion matrices for ETH-80 with training Computer Vision and Pattern Recognition, 2003.
size 60%, test size 40% (right matrix binary images, left Proceedings. 2003 IEEE Computer Society Confer-
matrix RGB images). ence on.
[10] Belongie, S. and Malik, J. and Puzicha, J., Shape
matching and object recognition using shape con-
6 CONCLUSION texts, Pattern Analysis and Machine Intelligence,
IEEE Transactions on, 2002.
We have presented in this work the Scale Invari- [11] Haibin Ling and Jacobs, D.W., Shape Classifica-
ant Feature Transform SIFT feature and its appli- tion Using the Inner-Distance, Pattern Analysis and
cation to binary images. We showed theoretically Machine Intelligence, IEEE Transactions on. 2007,
[12] Bouagar, Saliha and Larabi, Slimane, Efficient
how the feature can be applied to binary image
descriptor for full and partial shape matching, Mul-
and enhance the study with experiments both on timedia Tools and Applications, 2014.
retrieval and classification. The study concludes [13] Lin, Wei-Syun and Wu, Yi-Leh and Hung, Wei-
that SIFT can be applied to binary shapes. We Chih and Tang, Cheng-Yuan, A Study of Real-
am in further works to improve the matching and Time Hand Gesture Recognition Using SIFT on
apply the approach on binary masks derived from Binary Images, Advances in Intelligent Systems and
background subtraction (15). Applications—Volume 2, 2013.
[14] Xiang Bai and Xingwei Yang and Longin Jan Latecki
and Wenyu Liu and et al., Learning Context-Sensitive
Shape Similarity by Graph Transduction, 2010.
REFERENCES [15] Setitra, I. and Larabi, S., Pattern Recognition
(ICPR), 2014 22nd International Conference on,
[1] Veltkamp, R.C., Shape matching: similarity measures Background Subtraction Algorithms with Post-
and algorithms, Shape Modeling and Applications, processing: A Review, 2014.
SMI 2001 International Conference on. 2001.

300

VIPIMAGE15_Book.indb 300 9/8/2015 7:46:59 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Time-constrained detection of colored objects on raw Bayer data

António J.R. Neves, Alina Trifan & José Luís Azevedo

Department of Electronics and Telecommunications Engineering, Intelligent Systems and Intelligent Robotics
Laboratory, Institute of Electronics and Informatics Engineering of Aveiro, University of Aveiro, Portugal

ABSTRACT: In most image processing applications, the raw data acquired by an image sensor passes
through a demosaicing algorithm in order to obtain a full color image. The demosaicing step can take time
and resources that are of great importance when developing real-time applications. Most of the times, this
is done with single purpose of rendering the images into a viewable format for humans. However, most of
nowadays sensors allow the retrieval of images in raw format and processing the data as is. In this paper
we present a study on the direct usage of raw Bayer data for real-time color object detection, in the sce-
nario of autonomous robot soccer. The experimental results that we provide prove that the efficiency of
the chosen object detection algorithm is not lowered and that several improvements of the autonomous
robotic vision system used in this study can be noted: the bandwidth needed to transmit the data between
the digital camera and the PC is reduced, allowing, in some models, to increase the maximum frame rate.
Moreover, a decrease in the delay between the perception and action of the autonomous robot was meas-
ured, when using raw image data for processing the environment.

1 INTRODUCTION viewable format. However, in most of them it is

possible to retrieve images in a raw format allowing
In digital cameras, a Color Filter Array (CFA) is a the user to demosaic them using software, rather
mosaic of tiny color filters placed over the pixels than using the camera’s built-in firmware.
of an image sensor to capture color information. An example of a raw image acquired by the
They are needed because the typical photosensors camera and the corresponding interpolated RGB
detect light intensity with little or no wavelength version is presented in Figure 1. As we can see in
specificity, and therefore cannot separate color the image on the left, the raw image data is a single
information (Nakamura 2005). channel image where the value of each pixel cor-
A Bayer filter mosaic is a type of CFA for arrang- responds to the information about a specific color,
ing RGB color filters on a square grid of photo- depending on the position of the filters in the sen-
sensors. Its particular arrangement is used in most sor, as described before.
single-chip digital image sensors used in digital In this paper we study the effect of using directly
cameras, camcorders, and scanners to create a color the raw data from an image sensor for real-time
image. The filter pattern is 50% green, 25% red and
25% blue, usually called BGGR, RGBG, GRGB,
RGGB, etc. depending on the position of the filters.
In most of the image processing applications,
the raw data acquired by the image sensor passes
through a demosaicing algorithm in order to
obtain a full color image (Kimmel 1999). This is
a digital image process used to reconstruct a full
color image from the incomplete color samples
output from an image sensor overlaid with a CFA.
There are several well known algorithms, namely
Simple Interpolation (Malvar et al. 2004), Variable
Number of Gradients (Chang et al. 1999), Pixel
Grouping (Chang and Tan 2006) or Adaptive
Figure 1. In a) an example of a RAW image acquired
homogeneity-directed (Hirakawa et al. 2005). by the camera (grayscale image containing the RGB
Most modern digital cameras acquire images information directly from the sensor—Bayer pattern).
using a single image sensor overlaid with a In b) the image a) after the interpolation of the missing
CFA, so demosaicing is part of the processing information for each pixel in order to obtain a complete
pipeline required to render these images into a RGB image.

301

VIPIMAGE15_Book.indb 301 9/8/2015 7:47:00 PM

 color object detection. The experiments are con- one byte each. For example, considering the RGB
ducted on an autonomous soccer robot available color space, the index of the table to be chosen for
at our research unit (Neves et al. 2010, Silva et al. a specific triplet is obtained as
2011)1. We provide algorithms for image process-
ing and object detection using the raw data directly idx
d R << |G 8 | B,
included in a computer vision library developed
within our research group and denoted UAVi- being the << the left shift operator and | the bit-
sion (Trifan et al. 2015)2. Based on the experimental wise OR. The raw data (Bayer pattern) is also RGB
results obtained and presented in this paper, we can information, however, the positions in the image
conclude that the performance of the object detec- where the R, G and B information are located
tion is not affected by the use of the raw image data. change. These positions are also calculated only
Moreover, by using directly the raw data, we reduce once using a LUT in order to obtain a faster access
the bandwidth needed to transmit the data between during color classification.
the digital camera and the processing unit, allow- Each bit in the table entries expresses if one of
ing, in some camera models, to increase the maxi- the colors of interest (in this case: white, green,
mum frame rate available due to processing and blue, yellow, orange, red, blue sky, black, gray—no
bandwidth limitations when the demosaicing is per- color) is within the corresponding class or not.
formed on the camera. We provide also experimen- A given color can be assigned to multiple classes at
tal results showing a decrease in the delay between the same time. For classifying a pixel, first the value
the perception and action of the autonomous robot, of the color of the pixel is read and then used as an
when using raw image data for object detection. index into the table. The 8-bit value then read from
This paper is structured in 4 sections, first of the table is called the “color mask” of the pixel.
them being this Introduction. Section 2 presents the Taking into consideration a CFA BGGR, the
algorithm implemented for colored object detection LUT for the position of the R, G and B compo-
used in this study. We present experimental results nents for each pixel is built based on the following
in Section 3. Section 4 concludes the paper. algorithm:

2 COLORED BALL DETECTION

ALGORITHM

In order to implement a time efficient vision sys-

tem it is necessary to develop all the algorithms
and data structures to be used. Taking this into
consideration, we developed a computer vision
library denoted UAVision (Trifan et al. 2015) that
incorporates software for image acquisition from
digital cameras supporting different technologies,
as well as several modules for camera calibration,
blob formation, which stands at the basis of the
object detection, and image transfer using TCP
communications.
For fast color classification, color classes are
defined through the use of a Look-Up Table
(LUT). A LUT represents a data structure, in this
case an array, used for replacing a runtime compu-
tation by a basic array indexing operation.
This approach has been chosen in order to
save significant processing time. The LUT can be
applied to images in full RGB color or raw image
data (Bayer format) and allows the conversion to
an index image (image of labels).
The table consists of 16,777,216 entries (224,
8 bits for R, 8 bits for G and 8 bits for B) with

1
http://robotica.ua.pt/CAMBADA
2
http://sweet.ua.pt/an/uavision/

302

VIPIMAGE15_Book.indb 302 9/8/2015 7:47:01 PM

 The pipeline of the ball detection procedure can
be resumed as follows: after having acquired an
image, this is transformed into an image of labels
using the LUTs previously built. This image of
color labels, also designated as index image, will be
the basis of all the processing that follows.
The index image is scanned using one of three
types of scan lines or search lines (circular, radial
or linear). The information about colors of inter-
est or transitions between colors of interest is Run
Length Encoded (RLE). Blobs are formed by
merging adjacent RLEs of a color of interest.
The blobs that are of the color of the ball have
to meet the following validation criteria before
being labeled as ball. First, a mapping function,
that has been experimentally designed, is ussed for
verifying a size-distance ratio of the blob (from the
robot). This is complemented by a solidity meas-
ure and a width-height ratio validation, taking into
consideration that the ball has to be a round blob.
The validation is done taking into consideration
the detection of the ball even when it is partially
occluded. More details about all the steps of this
algorithm can be found in (Trifan et al. 2015).

3 EXPERIMENTAL RESULTS

In order to experimentally study the effect of using

directly the raw data from an image sensor to detect
colored objects, several experiments were con-
ducted using an autonomous soccer robot. These
robots use a catadioptric vision system formed by
an Ethernet camera point towards a mirror. This
vision system should detect, in useful time, the ball, Figure 2. Ball detection after being kicked by the robot.
the limits of the field and the field lines, the goal On top, a graphic showing the ball detection on full RGB
images after demosicing and on bottom the ball detection
posts and the other robots that are on the field.
on raw image data.
In Figure 2 we present the ball position detected
by the developed vision system after the robot per-
forms a kick, considering two different configura-
tions. In one case, the vision system was processing
directly the raw image data acquired from the
camera and in the other one the full RGB data
was processed. As we can see in both graphics, the
ball is correctly detected for all its trajectory, both
when we process the raw image data and when the
processing is performed in the full RGB image.
Moreover, we can see that the robot can detect the
ball until more than 10 meters, which is an interest-
ing and important result in this application (see an
example of the images acquired by the robots with
some marks over the detected objects in Fig. 3). Figure 3. Images acquired by the omnidirectional
Two game scenarios have been tested using vision system with the result of the color-coded object
detection. On the left, the ball was at one meter of the
the CAMBADA autonomous mobile robots. In robot. On the right, the ball was at 10 m from the robot.
each one, two experiments have been made. In Blue circles mark the white lines, the white circles mark
one experiment, the vision system processed the the black obstacles and the magenta circles mark the
full RGB image after demosicing for object detec- orange blobs that passed the validation thresholds and
tion (balls, field lines used in robot localization are considered ball.

303

VIPIMAGE15_Book.indb 303 9/8/2015 7:47:01 PM

 and obstacles). In the other experiment, the raw
image data was processed.
In Figure 4 we present a graphic with the result
of the ball detection when the ball is stopped at
a given position (the central point of the field, in
this case) while the robot is moving. The graphic
shows a consistent ball detection with both con-
figurations while the robot is performing a tour
around the field. The field lines are also properly
detected, as it is proven by the correct localization
of the robot in all the experiments.
The second scenario that has been tested is
illustrated in Figure 5, where both the robot and
the ball are moving. The robot is making a tour
around the soccer field, while the ball is being
sent across the field. In all these experiments, no
false positives were observed and the ball has been
detected in more than 90% of the images acquired

Figure 5. Ball detection when both the robot and the

ball are moving. On top, a graphic showing the ball detec-
tion on full RGB images after demosicing and on bottom
the ball detection on raw image data.

by the robot. Most of the times when the ball was

not detected was due to the fact that it was hidden
by the bars that hold the mirror of the omnidirec-
tional vision system.
We can observe a different robot trajectory in
the tours performed by the robots when the vision
system processed the full RGB images, when
comparing with the scenario when the vision sys-
tem processed the raw image data. Since the con-
trol parameters were adjusted to the vision system
processing the raw image data, the robot has a
different behavior in this situation. This is also an
important observation that we will explain right
Figure 4. Ball detection with the robot moving around
after.
the field and the ball stopped in the middle of the field. In addition to the good performance in the detec-
On top, a graphic showing the ball detection on full RGB tion of objects, both in terms of number of times
images after demosicing and on bottom the ball detection that an object is visible and detected and in terms
on raw image data. of error in its position, the vision system must also

304

VIPIMAGE15_Book.indb 304 9/8/2015 7:47:03 PM

 perform well in minimizing the delay between the This is the normal working mode of every robot
perception of the environment and the reaction of that uses vision as main sensor.
the robot. It is obvious that this delay depends on An important result is the delay between the per-
several factors, namely the type of the sensor used, ception and reaction of the robot. We compared
the processing unit, the communication channels the delay when acquiring frames in RGB and when
and the actuators, among others. accessing the raw data. Based on the graphics pre-
Some digital cameras do not provide frame sented in Figure 7, we measured an improvement
rates above 30 fps in RGB, but allow the access to of 23% of the reaction time at 50 fps when using
the raw data of the sensor at higher frame rates, raw data. If the camera used by the robot could
mostly due to processing and bandwidth restric- deliver raw data, these results prove that it is more
tions when the demosicing algorithm runs on the advantageous.
camera. Moreover, it is expected that the conver- Using the LUTs described below and the scan-
sion between the raw sensor data and a specific lines approach for blob detection as described
color space in the camera or in the device driver on in (Trifan et al. 2015) the delay will not be influ-
the computer side will take some time. Therefore, enced by the scene complexity, as presented in the
we performed the following experiments for meas- results of that work. Using a different algorithm
uring the delay between perception and reaction of
the robot accessing directly the raw data.
The delay measurement system used in this work
is based on a microcontroller board connected
to the vision processing PC through a USB port
(Fig. 6). The microcontroller periodically turns on
a LED, during a fixed period of time, and meas-
ures the time that the whole vision system takes to
detect it in the acquired image. The measured time
includes camera acquisition time, image data trans-
fer time and vision processing time. In the setup
used for these experiments, the camera is config-
ured with an exposure time of 5 ms and Gigabit
Ethernet connection that can provide for our setup
(cable, network card, computer and operating sys-
tem) 80 MB/s, far from the theoretical 125 MB/s,
has been used. The vision software is a simplifi-
cation of the one described in (Trifan et al. 2015)
since we do not have to detect the obstacles and
white lines and takes, on average, 6 ms to detect the
led. The vision system detects the led on and when
it happens, sends a message to the microcontroller.

Figure 7. Histograms showing the delay between percep-

tion and action on the vision system of the CAMBADA
robots. On top, the camera is acquiring RGB frames and
Figure 6. The main blocks used in our measurement is working at 50 fps (average = 34.2 ms, max = 48 ms,
setup. The microcontroller periodically turns on a LED, min = 29 ms). On the bottom, the camera acquiring sin-
during a fixed period of time, and measures the time that gle channel frames, returning the raw data of the sensor
the whole vision system takes to detect it in the acquired (Bayer pattern) (average = 26.4 ms, max = 34 ms, min =
image. 23 ms).

305

VIPIMAGE15_Book.indb 305 9/8/2015 7:47:04 PM

 in another application, the delay time will be Aveiro and was partially supported by FEDER
proportional to the processing time of the algo- through the Operational Program Competitive-
rithm used and the scene complexity. Regarding ness Factors—COMPETE and by National Funds
external conditions, namely illumination or other through FCT—Foundation for Science and Tech-
factors, we use in our robots algorithms that guar- nology in a context of a PhD Grant (FCT refer-
antee the same properties of the acquired images ence SFRH/BD/85855/2012) and by National
(Neves et al. 2014). Funds through FCT—Foundation for Science and
The processing time of each captured image, Technology, in the context of the projects UID/
both in raw data and full RGB color, using the CEC/00127/2013 and Incentivo/EEI/UI0127/2014.
developed vision system built using the UAVision
library is considerably low. The full execution of
the pipeline, from image acquisition to object
detection after validation (balls, lines and obstacles)
only takes at most 10 ms, allowing thus a framerate
of 100 fps if the digital camera supports it and if REFERENCES
other control or artificial intelligence processes can
run in parallel. The time results have been obtained Chang, E., S. Cheung, & D.Y. Pan (1999). Color filter
array recovery using a threshold-based variable
on a computer with a Intel Core i5-3340M CPU number of gradients. In Proceedings of SPIE, Volume
@ 2.70 GHzs 4 processor, processing images with 3650, pp. 36.
resolutions of 1024 × 1024 pixels. In the implemen- Chang, L. & Y. Tan (2006). Hybrid color filter array
tation of these vision systems we did not use multi- demo-saicking for effective artifact suppression.
threading. However, image classification as well as J. Electronic Imaging 15(1), 013003.
the next steps can be parallelized if needed. Hirakawa, K., S. Member, & T.W. Parks (2005). Adaptive
homogeneity-directed demosaicing algorithm. IEEE
Trans. Image Processing 14, 360–369.
4 CONCLUSIONS Kimmel, R. (1999). Demosaicing: image reconstruc-
tion from color ccd samples. Image Processing, IEEE
Transactions on.
Most of digital cameras used in robotic applica- Malvar, H.S., L. wei He, & R. Cutler (2004). High-
tions allow the retrieval of images in raw format quality linear interpolation for demosaicing of bayer-
and in some other camera models the demosaic- patterned color images. In Proceedings of the IEEE
ing algorithm if performed by the device driver in International Conference on Speech, Acoustics, and
the processing unit. However, as far as we know, Signal Processing.
no previous study have been done before explor- Nakamura, J. (2005). Image Sensors and Signal Processing
ing the direct usage of raw data in colored object for Digital Still Cameras. Boca Raton, FL, USA: CRC
detection. Press, Inc.
Neves, A., J. Azevedo, N.L.B. Cunha, J. Silva, F. Santos,
In this paper we have presented a study on the G. Corrente, D.A. Martins, N. Figueiredo, A. Pereira,
direct usage of raw Bayer data for real-time color L. Almeida, L.S. Lopes, & P. Pedreiras (2010, January).
object detection and the experimental results that CAMBADA soccer team: from robot architecture to
we provide prove that the efficiency of colored multiagent coordination, Chapter 2, pp. 19–45. Vienna,
object detection algorithm is not lowered and sev- Austria: I-Tech Education and Publishing.
eral improvements can be noted: the bandwidth Neves, A.J.R., A. Trifan, & B. Cunha (2014). Self-
needed to transmit the data between the digital calibration of colormetric parameters in vision sys-
camera and the PC is reduced; a the delay between tems for autonomous soccer robots. In S. Behnke,
the perception and action of the autonomous M.M. Veloso, A. Visser, and R. Xiong (Eds.),
RoboCup 2013: Robot Soccer World Cup XVII,
robot, when using raw image data for processing Volume 8371 of Lecture Notes in Computer Science,
the environment was improved by more that 20%. pp. 183–194. Springer.
In what concerns future work, the next step will Silva, J., N. Lau, A.J.R. Neves, J. Rodrigues, & J.L. Azevedo
be to explore the usage of raw Bayer data in feature (2011, March). World modeling on an MSL robotic
detection and classification of objects that do not soccer team. Mechatronics 21(2), 411–422.
have a dominant color. Trifan, A., A.J.R. Neves, B. Cunha, & J.L. Azevedo
(2015). Uavision: a modular time-constrained library
for soccer robots. In R.A.C. Bianchi, H.L. Akin,
ACKNOWLEDGEMENTS S. Ramamoorthy, and K. Sugiura (Eds.), RoboCup
2014: Robot Soccer World Cup XVIII, Volume 8992
of Lecture Notes in Computer Science, pp. 490–501.
This work was developed in the Institute of Elec- Springer.
tronic and Telematic Engineering of University of

306

VIPIMAGE15_Book.indb 306 9/8/2015 7:47:05 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Image scanning techniques for speeded-up color object detection

Alina Trifan, António J.R. Neves & Bernardo Cunha

Department of Electronics and Telecommunications Engineering, Intelligent Systems and Intelligent Robotics
Laboratory, Institute of Electronics and Informatics Engineering of Aveiro, University of Aveiro, Portugal

ABSTRACT: Nowadays there are many computer vision algorithms dedicated to solve the problem of
object detection, from many different perspectives. Many of these algorithms take a considerable process-
ing time even for low resolution images. Research areas that are more concerned about real-time capabili-
ties of a system, such as the case of robotics for example, are focusing on adapting the existing algorithms
or developing new ones that cope with time constraints. Color object detection algorithms are often used
in time-constrained application since the color of an object is an important clue for its detection. In
addition, color segmentation algorithms have a smaller computational cost than generic object detection
algorithms. Most of the color object detection algorithms include a step of image scanning in search of
pixels of the color of interest. In this paper we present a study on the influence of an image scanning pat-
tern while searching for pixels of a certain color versus the image resolution in the results of three color
object detection algorithms. We present results comparing different scanning approaches versus different
resolution images for color object detection algorithms based on blob formation, contours detection and
region growing clustering. Our experiments show that using search lines for color segmentation, followed
by blob formation provides faster and more accurate results.

1 INTRODUCTION (Neves et al. 2010)1 and the digital vision system

implemented using the UAVision computer vision
Object detection with small computational cost and library2 (Trifan et al. 2014), we present results of
processing time is a necessity of some of the most color object detection algorithms using optimal
diverse domains, such as: robotics, video surveillance configurations of scan lines versus different image
and traffic analysis, among others. With current resolutions. Moreover, we complement these
advances in technology and the decrease of prices results by comparing their detection efficiency and
in image sensors and video cameras, nowadays time processing with two other color detection
digital vision systems can acquire images at higher algorithms based on contours detection and region
resolutions than 1 Megapixel and can work at high growing clustering, for which some of the methods
frame rates. This implies a considerable data size of the OpenCV Library3 have been used.
that has to be processed in a very small period of This paper is structured in 5 sections, first of
time when real-time operations and data processing them being this Introduction. Section 2 presents
are needed. For example, for a camera working at the approach taken for image scanning. In Section 3
50 fps and acquiring images of 1024 × 1024 pixels, we succintly present the algorithms and computer
the total processing time of any given real-time vision libraries used in this study. We present
application should not exceed 20 ms. experimental results in Section 4 and Section 5
A common solution for reducing the processing concludes the paper.
time of an object detection algorithm is to sample
the data, to process only parts of an image or to
2 SCANLINES
acquire and process smaller resolution images.
However, these procedures come with a cost in the
Most algorithms for color object detection, just as the
accuracy of the object detection results. By not
name implies, perform a color segmentation step in
processing all the available information, one might
which pixels of the same color are grouped together.
leave out important data and introduce assump-
tions that can tackle the final result of the detection
algorithms. In this paper we present a study based 1
http://robotica.ua.pt/CAMBADA.
on the real-time scenario of robotic soccer. Using 2
http://sweet.ua.pt/an/uavision/.
sequences acquired by a robotic soccer player 3
http://docs.opencv.org.

307

VIPIMAGE15_Book.indb 307 9/8/2015 7:47:05 PM

 The amount and characteristics of these scanlines
depend on the parameters (start and end positions,
distance—vertical, horizontal or angular between
scanlines, etc).
Each scanline object is used by the RLE Module
that produces a list of occurrences of a specific
color. Depending on the number of scanline objects
created in a specific vision system, we will have the
same number of RLE lists produced. Then, the Blob
Module is able to use all these lists to combine RLEs
and form blobs based on several features that are
presented with more detail in (Trifan et al. 2014).
The OpenCV library (Bradski and Kaehler 2008)
provides two alternative algorithms that have been
used in this study for comparison of the results.
Figure 1. Examples of different types of scanlines:
The first algorithm that has been used is a region
a) horizontal scanlines; b) vertical scanlines; c) circular
scanlines; d) radial scanlines. growing algorithm (Gonzalez and Woods 2001), in
which, using methods from the OpenCV library,
the image is scanned horizontally and every time a
For color segmenting an image, most algorithms use pixel of the color of interest is found, it is consid-
the approach of scanlines. A scanline, or a search line, ered to be the seed for the clustering algorithm. The
is a list of pixel positions in the image, whose shape second approach was using the same library for con-
depends on its type. That is, an image can be scanned tour detection (Canny 1986) and scanning the pixels
in search for a color of interest under different direc- inside each contour. If the pixels belonging to a given
tions, for example either horizontally, vertically, radi- contour are of the color of interest, that contour is
ally or circulary. In the robotic soccer competitions validated as being a blob of the color of interest.
promoted by the RoboCup Federation4 the environ- For all of the three algorithms, for fast color
ment is color coded and many participating research classification, color classes are defined through
teams use the approach of scanlines while searching the use of a Look-Up Table (LUT). A LUT repre-
for one of the colors of interest (Trifan et al. 2012), sents a data structure, in this case an array, used for
(Neves et al. 2008), (Rofer et al. 2011), (Garcia et al. replacing a runtime computation by a basic array
2012), (Akin et al. 2014). indexing operation. Each bit in the table entries
For the purpose of this study, we have used expresses if one of the colors of interest (in the case
sequences acquired by the vision system of the of robotic soccer: white, green, blue, yellow, orange,
robotic soccer agents of CAMBADA team and the red, blue sky, black, gray—no color) is within the
UAVision library (Trifan et al. 2014). UAVision is a corresponding class or not. A given color can be
modular computer vision library developed within assigned to multiple classes at the same time. For
our research group for color object detection. We classifying a pixel, first the value of the color of the
have used the Scanlines Module of the library in pixel is read and then used as an index into the table.
order to test the impact of the scanning pattern The 8-bit value then read from the table is called the
and sparsness in the result of the color object “color mask” of the pixel. This image of color labels
detection algorithms. (Fig. 2 (b)), will be the basis of all the processing
In Figure 1 the three different types of scanlines that has been previously described. Using an image
supported by this module are illustrated: linear, of labels not only saves computational time but it
circular and radial. also introduces less noise than using for example a
grayscale version of the original image.
3 ALGORITHMS

Three different algorithms for color object detec-

tion have been used for the purpose of this study.
First, we used the Blob Module of the UAVision
library, that uses scanlines to scan the image and
then the color information is run-length encoded by
the RLE Module. The Scanlines Module allows the
definition of an object that is a vector of scanlines.
Figure 2. a) Original image. b) Corresponding 8-bit
image of color labels. c) “Painted” image of labels for
4
http://www.robocup.org/. visual understanding purpose only.

308

VIPIMAGE15_Book.indb 308 9/8/2015 7:47:05 PM

 4 EXPERIMENTAL RESULTS blob detection, we have reached the conclusion that
the use of radial scan lines provide a better result in
In this study we present results of the three algorithms terms of ball accuracy. For this experiment, we have
used for color object detection and we evaluate the used a configuration of the radial scan lines of 2880
effect of the scanning pattern and sparsness, as well scanlines, disposed on four levels 3. This configu-
as image resolution in the accuracy and processing ration has been considered optimal by the authors
time of the detection. since it is the best compromise between number of
We focus the results on the ball detection since visited pixels and number of repeated pixels. For
the obstacle and field line detections are only sup- the other two algorithms, we perform a linear (hori-
ported by the UAVision library. The region growing zontal) scan in search for pixels of interest.
and contour detection algorithms are not suitable The table that we are presenting shows that at any
for more generic object detection since for example of the chosen resolutions, the blob detection algo-
field lines do not have a contiguous contour, nor rithm performs faster and the maximum processing
can be merged into a colored cluster. time of a frame that was measured was 10 ms, while
In Table 1 we present the time processing of for example, the contour detection algorithms can
each of the algorithms previously described. take a maximum time of 35 ms per frame.
We have tested several sequences at three different Table 2 shows the maximum distances at which
resolutions: 1024 × 1024, 512 × 512 and 256 × the ball is being detected by each of these algo-
256 pixels. After having performed several iterative rithms. The blob detection algorithms achieves
runs of the object detection algorithm based on correct results for balls found at a distance of up to
12 m, even for the smallest chosen resolution, while
the results of the contours detection algorithm
and region growing clustering algorithm slightly
degrade with distance.
Table 3 presents the percentages of correct ball
detection for each of the three algorithms. In the
video sequences that have been used for tests the
robot rotates around itself and the ball is placed in
front of it at distances from 1 m to 12 m. These
sequences have a total of 5040 frames, out of which,
the ball is present in 4580 frames. The situation in
which the ball is not present in a frame is due to the
hardware architecture of the vision system (Fig. 4).
When the ball is aligned with any of the four sup-
portive bars of the vision system, it cannot be seen
in the image as it is fully hidden by the bar. The blob
detection algorithm has a detection rate of 100% for
full resolution images and the decrease of the resolu-
tion is followed by a slight decrease in the performance
Figure 3. Illustration of the four levels of radial scan-
lines. In this image, there are only 90 scanlines distributed
Table 2. Maximum distance from the robot at which
on four levels for a better understanding of each of the
the ball is correctly detected by each of the algorithms.
levels. 2880 scanlines are used for the detection of the
ball in this study, but with this number of scanlines, the
Blobs Contours Region growing
reader cannot clearly distinguish the four levels.
1024 × 1024 12 m 12 m 11 m
Table 1. Average and maximum time processing, in 512 × 512 12 m 12 m 11 m
miliseconds, for each of the three algorithms studied in 256 × 256 12 m 11 m 11 m
this paper.

Region Table 3. Percentages of correct ball detection for the

Blobs Contours growing three algorithms.

Mean Max Mean Max Mean Max Blobs Contours Region growing

1024 × 1024 8.72 10 17.16 35 8.86 21 1024 × 1024 100% 68.5% 73.1%
512 × 512 2.9 7 5.16 16 2.03 7 512 × 512 95.7% 63.7% 69%
256 × 256 1 5 1.02 3 1 1 256 × 256 85.7% 24.6% 53.9%

309

VIPIMAGE15_Book.indb 309 9/8/2015 7:47:06 PM

 Table 5. Percentages of correct ball detection for the
three algorithms for balls found at 9 m or more from the
robot.

Blobs Contours Region growing

1024 × 1024 100% 67.6% 68.8%

512 × 512 78.4% 58.8% 66.2%
256 × 256 44.5% 11.5% 11.2%

Figure 4. Close-up of the vision system hardware of the

robots.

Table 4. Percentages of correct ball detection for the

three algorithms for balls found at 6 m or more from the
robot.

Blobs Contours Region growing

1024 × 1024 100% 63.8% 70%

512 × 512 91% 62.8% 64.7%
256 × 256 70% 16.3% 22.9%

of the algorithm. Out of the three algorithms, the one

based on contour detection is not only the heavi-
est from a computational point of view, but it also
performs worse that the other two. Moreover, for the
smallest resolution, the contours algorithm can only
detect the ball in a quarter of the frames.
All three algorithms can detect the ball up to dis-
tances of 11 m (12 m in some cases). However, for
some of the algorithms, the detection rate at longer
distances is very small. Tables 4 and 5 present the
detection accuracy for balls that are found further
than 6 m and 9 m, respectively, from the robot.
Figures 5, 6, 7 show the ball detection accuracy
for the three algorithms, for each of the three reso-
lutions. The graphics illustrate the position of the
detected balls, relative to the robot, in world coor-
dinate at the ground level. The algorithm used for
converting ball position in pixels to world coordi-
nate has been developed within our research group
and can be found in (Neves et al. 2011).
The four “valleys” that can be noted in all of Figure 5. Graphics representing the ball detection for
the graphics represent four bars that support the the three resolutions using the blob algorithm.

310

VIPIMAGE15_Book.indb 310 9/8/2015 7:47:06 PM

 Figure 6. Graphics representing the ball detection Figure 7. Graphics representing the ball detection for
for the three resolutions using the contours detection the three resolutions using the region growing clustering
algorithm. algorithm.

vision system in the hardware architecture of the results are even less reliable for balls found at 9 m
robot. When the ball is aligned with any of these or more from the robot. In this case, only 10% of
bars, the ball is not visible. the balls are correctly detected.
For images of 256 × 256 pixels, the contours The region growing clustering algorithm delivers
detection algorithms provides quite poor results. slightly better results, compared to the contours
When the ball is found at 6 m or more from the detection algorithm but it is still far from achieving
robot, it is only detected in 16% of the cases. The the detection rate of the blob detection algorithm.

311

VIPIMAGE15_Book.indb 311 9/8/2015 7:47:09 PM

 ACKNOWLEDGEMENTS

This work was developed in the Institute of

Electronic and Telematic Engineering of University
of Aveiro and was partially supported by FEDER
through the Operational Program Competitiveness
Factors—COMPETE and by National Funds
through FCT—Foundation for Science and
Technology in a context of a PhD Grant (FCT
reference SFRH/BD/85855/2012) and in the
context of the projects UID/CEC/00127/2013 and
Incentivo/EEI/UI0127/2014.

REFERENCES
Figure 8. Ball detection using the blob algorithm at
12 m. The detected ball is marked with a magenta circle. Akin, H.L., T. Mericli, E. Ozukur, C. Kavaklioglu, &
B. Gokce (2014). Cerberus Team Description. RoboCup
2014, Joao Pessoa, Brazil.
Bradski, G. & A. Kaehler (2008, September). Learning
These results prove that working with images at OpenCV (First ed.). O’Reilly.
full resolution can provide optimal results in a Canny, J.F. (1986, November). A computational ap-
small amount of time when using an appropriate proach to edge detection. IEEE Transactions on Pat-
scanning pattern of the image. For applications tern Analysis and Machine Intelligence 8 (6).
where detection accuracy and total lack of false Garcia, D., E. Carbajal, C. Quintana, E. Torres,
positives is required, the algorithm based on radial I. Gonzalez, C. Bustamante, & L. Garrido (2012).
scanlines and blob formation is a suitable solution Borregos Team Description. RoboCup 2012, Mexico
since its detection rate is 100% and it can be used City, Mexico.
with sensors working at 100 fps, if the hardware Gonzalez, R.C. & R.E. Woods (2001). Digital Image
Processing (2 ed.). Addison-Wesley Longman Pub-
allows it. A smaller resolution can lead to an even lishing Co.
smaller processing time, for applications where fast Neves, A., J. Azevedo, N.L.B. Cunha, J. Silva, F. Santos,
performance is more crucial than having 100% G. Corrente, D.A. Martins, N. Figueiredo, A. Pereira,
correct detection. L. Almeida, L.S. Lopes, & P. Pedreiras (In Vladan
In Figure 8 we present an image with the ball Papic (Ed.), Robot Soccer, 2010). CAMBADA soc-
correctly detected by the blob algorithm at a dis- cer team: from robot architecture to multiagent
tance of 12 m from the robot. coordination, Chapter 2. I-Tech Education and Pub-
The results that have been presented in this lishing, Vienna, Austria.
paper have been obtained on a computer with an Neves, A.J.R., D.A. Martins, & A.J. Pinho (2008, April).
A hybrid vision system for soccer robots using
Intel Core i5-3340M CPU @ 2.70 GHz 4 proces- radial search lines. In Proc. of the 8th Conference
sor, which is the main processing unit of the soccer on Autonomous Robot Systems and Competitions,
robots used in the experiments. Portuguese Robotics Open—ROBOT-ICA’2008,
Aveiro, Portugal, pp. 51–55.
Neves, A.J.R., A.J. Pinho, D.A. Martins, & B. Cunha
5 CONCLUSIONS (2011, mar). An efficient omni directional vision
system for soccer robots: from calibration to object
In this paper we have presented a study about the detection. Mechatronics 21 (2), 399–410.
influence of image scanning in the pipeline of a Rofer, T., T. Laue, C. Graf, T. Kastner, A. Fabisch, &
C. Thedieck (2011). B-Human Team Description.
color object detection digital vision system. Using RoboCup 2011, Istanbul, Turkey.
the real-time scenario of robotic soccer and the Trifan, A., A. Neves, N. Lau, & B. Cunha (2012). A
digital vision system of an autonomous robot, we modular real-time vision system for humanoid robots.
presented results for three different algorithms for In Proceedings of IS&T/SPIE Electronic Imaging
object detection, using different scanning patterns 2012.
and different image resolutions. The results Trifan, A., A.J.R. Neves, B. Cunha, & J.L. Azevedo
presented show that when reducing the image (2014). Uavision: a modular time-constrained library
resolution, object detection algorithms tend to lower for soccer robots. In Proc. of RoboCup 2014.
their performance, due to loss of information.

312

VIPIMAGE15_Book.indb 312 9/8/2015 7:47:13 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Improving deep neural networks classification by preprocessing images

H. Erdmann
Department of Research and Technology, Boolabs, São Paulo, Brazil

F.T. Ito
Department of Computer Science, Universidade Federal de Sao Carlos, Brazil

D. Takabayashi
Department of Research and Technology, Boolabs, São Paulo, Brazil

D.N. dos Santos

Department of Research and Technology, Boolabs, São Paulo, Brazil
Department of Electrical Engineering, Centro Universitário da FEI, São Paulo, Brazil

ABSTRACT: The amount of images used to train a classifier has a great impact in the algorithm’s per-
formance. In the domain of e-commerce, most of the dataset is formed by studio images and we demon-
strate in this work that by multiplying the amount of images with different transformations can provide a
significant boost on the overall performance of the classifier. Moreover, we list the challenges encountered
in such task and present the improvements obtained with a classifier trained with additional images that
were synthetically generated by applying several transformations on the original dataset.

1 INTRODUCTION as MNIST (Lecun and Cortes, LeCun et al. 2001),

Image Net (Deng et al. 2009, Russakovsky et al.
To classify an image correctly is a task with 2014) and LabelMe (Russell et al. 2008), composed
applications in several areas related to digital image of, mostly, natural images. However, the common
processing. It can be done by extracting features that images found in the market and e-commerce sites are
describe the image and compare them with the mean doctored ones, with white backgrounds and perfect
features of the known classes. The simplest classi- illumination. Approaches such as Global Contrast
fication techniques are binary classifiers, which are Normalization (GCN) (Graham and Sutter 2000)
used to define if an image belongs or not to a certain and Zero Components Analysis (ZCA) Whitening
class. Moreover, image classification algorithms are (Bell and Sejnowski 1997b, Bell and Sejnowski 1997a)
usually developed for a specific set of images because fail in these situations, and cannot be used directly.
the objective of these algorithms is strongly related Deep Learning (DL) is an area of machine learning
to their applications. In this work we focus on a that relies on high-level, hierarchized abstractions to
multi-classification algorithm using state-of-the-art create models for specific functions (Krizhevsky et
deep neural networks. It is widely known that the al. 2012a, Schmidhuber 2014). The theory behind
amount of images used to train a classifier has a Deep Learning gained traction circa 2000/2003,
direct effect on it’s final results (Fuxman et al. 2009). with the advent of powerful graphic cards and new
In the domain of e-commerce, most of the dataset works on the field. One of the prominent algo-
is formed by studio images and we demonstrate rithms for Image Classification fits the DL profile:
how we improved our classifier by multiplying the the Convolutional Neural Network (Sermanet et al.
amount of images with different transformations. 2012, Smirnov et al. 2014). In this paper, we discuss
The task of object recognition is a great challenge the aforementioned approaches to preprocessing and
in computer vision and has been the focus of their results in studio images, along with other tech-
thorough research by many authors for many years. niques to expand image datasets and to find mean-
There are many applications and algorithms towards ingful features to use in classifiers. A Convolutional
this goal. Mainly, these applications make use of Neural Network is used to get practical results from
natural images. There are many image datasets that these preprocessed techniques on an image dataset
researchers can use to compare their work, such composed by e-commerce products.

313

VIPIMAGE15_Book.indb 313 9/8/2015 7:47:16 PM

 2 RELATED WORK

The preprocessing of image data for machine

learning application has been extensively covered
by several different authors (Kotsiantis and
Kanellopoulos 2006, Förstner 2000), as using the
image itself pixel by pixel as an instance can be
unwieldy at times. A 256 × 256 colored image, for
example, yields 196608 intensity values: training a
classifier with this data and no Graphic Processing
Unit (GPU) takes weeks.
Finding a shorter representation without
losing too much information is the goal of some
preprocessing algorithms (Bell and Sejnowski 1997b, Figure 1. In (a): original image; In (b): 36 differ-
Krizhevsky and Hinton 2011, Farabet et al. 2013). ent images obtained with perspective transformations
However, with the help of a GPU, we can fully applied to the original image.
utilize the abstraction levels provided by CNNs and
Deep Learning with complete images. In Farabet
et al. (Farabet et al. 2013), horizontal flipping, rota- simulate different situations (Yaeger et al. 1996,
tion and scaling were used to decrease the bias that Simard et al. 2003). Moreover, affine transforma-
low-level features can bring to deep classification. tions keep distance ratios (midpoints of lines and
This is further explored in Simard et al. (Simard their relative proportions are unaltered) and col-
et al. 2003), where distortions and translations linearity (points residing in lines keep their former
were used to expand the MNIST database and positions on them). Composed by the translation,
optimize classification in many algorithms. Finding rotation, scaling and shearing operations, this
appropriate features for image matching and group of transforms can focus on different parts of
classification is one of the challenges that each data- the image to increase the classification accuracy by
set poses to the task (Dollar et al. 2007), as distribu- a few percents. An example of perspective trans-
tion, characteristics and size influence the speed and formations can be viewed in Figure 1.
quality of it. Studio and product images have few On the other hand, the perspective transformation
works in the literature, and preprocessing and fea- projects points onto the image plane along lines that
turemining on these is a relatively unexplored area. emanate from a single point, called the center of pro-
The Deep Learning classifier we use is well known jection, changing the focus of the image. This type
in image classification (Sigaud and Wilson 2007, of transform does not preserve line parallelism, and
Arel et al. 2010, Zeng et al. 2013). Optimal accuracy is used to create distortions in images. These tech-
was reached in many competitions by Deep Learn- niques are used in a variety of image classification
ing related approaches, and the Large Scale Visual tasks that involve low-level features (i.e. raw pixels)
Recognition Challenge (ILSVRC) thrives with to increase the feature space and reduce bias.
similar classification techniques. The specific archi- In an image dataset, instances can have high con-
tecture we use, the Convolutional Neural Network, trast and illumination variation: this damages the
has numerous applications and detailed papers and classification process and can incur unnecessary
results (Ranzato et al. 2011, Wu et al. 2014, Le et al. bias. To soften the difference between images, matrix
2011, Krizhevsky et al. 2012a, Le 2013). normalization and whitening algorithms can be
applied before training. In Deep Learning, they are
essential for optimization, but in non-DL algorithms
3 METHODOLOGY these have little to no effect on classification.
Normalization is a process that decreases the
The methodology used on this work consists of variation in ranges between matrixes. In image
some preprocessing techniques followed by the processing, it is used to create consistency in data-
classification at the end of the pipeline. The pre- sets with natural scenes in a wide spectrum of illu-
processing stage consists of four transformations mination sources (Cirean et al. 2011, Makwana et
that are detailed here: a. Affine transformations; b. al. 2011). Effectively, this process spreads out the
Perspective transformations; c. Image whitening; pixel ranges on images to enhance contrast and
d. Image normalization. edges, amplifying corners and form features. Many
Affine and perspective transformations can normalization techniques exist in computer vision.
be used to increase generality in image classifiers In this paper we use the Global Contrast Normali-
where a common approach is to alter the refer- zation (GCN), a common preprocessing approach
ential camera position to distort the image and to image classification (Goodfellow et al. 2013,

314

VIPIMAGE15_Book.indb 314 9/8/2015 7:47:17 PM

 Springenberg and Riedmiller 2013). It consists 2000, Arel et al. 2010, Schmidhuber 2015), and the
in subtracting from each pixel intensity value the convolutional neural network in particular shows
mean of all pixels in the image, then dividing by great results in visual classification. These hierar-
their standard deviation. Whitening algorithms are chical networks reduce the image by filtering and
processes that decorrelate neighboring pixels by reducing it in each iteration, differing from the
transforming the data until the covariance matrix usual neural network approach by sharing weights
is equal to the identity matrix. The pixels from an between nodes in the same layer and spatial pool-
image are usually highly correlated, and thus many ing for simplification.
points are redundant in training and classification. A conventional Neural Network (NN) consists
Whitening is vital in the preprocessing step of algo- of many simple processors connected, called neu-
rithms that use raw images as data to enhance the rons, each one producing a sequence of values of
identification of higher-order features. activations. Input neurons are activated by sen-
In Bell et al. (Bell and Sejnowski 1997b, Bell and sors perceiving the environment, other neurons are
Sejnowski 1997a), several whitening techniques are activated by means of weighted connections from
discussed and explained. Mainly, they find a decor- previous activated neurons. Some neurons can
relation matrix and multiply the desired data by it to influence the network by firing actions. Learning
transform the covariation matrix into the diagonal, or credit assignment is about finding weights
then factoring it to produce the identity. Among that make the NN exhibit desired behavior, such
them, the Principal Component Analysis (PCA) as driving a car. Depending on the problem and
and the Zero-phase Component Analysis (ZCA) have how the neurons are connected, such behavior may
been used in classification tasks by various authors. require long causal chains of computational stages,
The PCA is applied to a matrix X by using the where each stage transforms (often in a non-linear
following values from the covariance matrix: mul- way) the aggregate activation of the network. Deep
tiplying it by the transpose of the eigenvectors, Learning is about accurately assigning credit across
scaled by the inverse square root of the eigenvalues. many such stages (Schmidhuber 2015).
However, this transform is not unique as multiply- A Convolutional Neural Network (CNN) has
ing the whitened data by an orthogonal matrix will several different layers for each operation, but the
yield a result with covariance matrix equal to the basic model has a convolutional layer that sepa-
identity. The ZCA consists in finding the canonical rates the image in many sub-matrixes called feature
whitening by multiplying the PCA transform by an maps, a pooling layer that groups pixels and an acti-
orthonormal matrix. This transform is unique and vation layer. These feature maps are each the result
shows the closest resemblance to the original data. of a filter passed throughout the input matrix, and
On the other hand, the ZCA consists in finding the the activation layer can be chosen from a variety of
canonical whitening by multiplying the PCA trans- functions, such as sigmoids or a Rectified Linear
form by an orthonormal matrix. This transform is Unit. This network is fed with squared pixel patches
unique and shows the closest resemblance to the surrounding each pixel in each image. The label for
original data. Both PCA and ZCA approaches can each patch is the label of the center pixel.
be viewed in Figure 2.
After all the preprocessing, the images are sub-
jected to a Convolutional Neural Network (CNN) 4 EXPERIMENTS
classifier. Neural networks have been used in sev-
eral classification tasks (Hunt et al. 1992, Zhang In our experiments, we used an image object data-
base with roughly two million instances obtained
from e-commerce sites and web crawlers. Numerous
products, from kitchen appliances to tires and cell-
phone accessories, were gathered to form it. To
reduce the scope of classification, we restrained
the size of this database to include only appliances
and related items, totalling 10 classes. All images
were standardized to 256 × 256 pixels, with 3 color
channels. To create a Convolutional Neural Network,
we used the Deep Learning framework Caffe (Jia et
al. 2014). The network layers are based on the refer-
ence CaffeNet (Krizhevsky et al. 2012b).
On each essay, we used a different preprocess-
Figure 2. In (a): image after application of PCA ing approach. First, we use the database without
transform; In (b): image after application of the ZCA any augmentation. Then, a boosted database with
transform. affine transformations (rotations, translations

315

VIPIMAGE15_Book.indb 315 9/8/2015 7:47:18 PM

 with zoom and horizontal flipping) applied to all itself and measures the error rate of the preliminar
images. Finally, we use the ZCA whitening algo- results in 1000, 2000, 3000, 4000 and 5000 iterations,
rithm in all images in the boosted database and giving the accuracy value (100%—error rate).
train with that. A fourth database containing other Tables 2, 4 and 6 refer to the test data taken from
classes of objects was used to further measure the web crawlers and made of natural scenes. This test
image augmented classification accuracy. uses the snapshot classifiers that were obtained
The classifiers are first tested for assertivity using in the training phase and measures the error rate
a random sampling from the database. Then, with when choosing the prediction of the most likely
a second database composed of images mined with classes. The top-1 error rate takes in account only
web crawlers and classified by hand, test the model one prediction and the top-5, a set with the five
for the top-1 and top-5 error rates. We separated most probable classes.
resulting classifiers by their number of iterations, Table 7 is an elaborate description of the results
from 1000 to 5000 in 1000 steps. The first experiment from the last classifier by class. Each class has a
was done in order to set a baseline for comparing our super-class (parent), the number that represents the
techniques results and has a test sampling set com- class and the name of the class. We measured the
posed of 7,5% of the total images. On the following error rates of the last classifier with 5000 iterations
experiments this quantity was raised to 25%. and separated the accuracy and recall per class.
Tables 1, 3 and 5 refer to the training data
and the accuracy results on each number of Table 4. Classifier’s performance for the testing data
training iterations. The classifier takes snapshots of with boosted dataset. The gain in both top-1 and top-5
metrics is around 8%.
Table 1. Classifier’s performance
for the training data. Boosted database—testing data

Original database—training data Iterations Top-1 Top-5

Iterations Assertivity 1000 25.12% 76.04%

2000 44.09% 84.37%
1000 82.00% 3000 40.37% 86.11%
2000 87.02% 4000 47.31% 85.19%
3000 89.70% 5000 47.04% 88.44%
4000 90.75%
5000 90.67%
Table 5. Classifier’s performance for the training
data with boosted and whitened dataset.
Table 2. Classifier’s performance for the testing data.
Boosted + ZCA database—training data
Original database—testing data
Iterations Assertivity
Iterations Top-1 Top-5
1000 75.09%
1000 30.32% 76.03% 2000 83.26%
2000 30.33% 76.21% 3000 85.48%
3000 32.01% 73.33% 4000 86.33%
4000 35.24% 76.19% 5000 87.29%
5000 34.02% 80.44%

Table 6. Classifier’s performance for the testing data

Table 3. Classifier’s performance for with boosted and whitened dataset. A slight improve-
the training data with boosted dataset. ment was obtained.

Boosted database—training data Boosted + ZCA database—testing data

Iterations Assertivity Iterations Top-1 Top-5

1000 77.09% 1000 39.25% 89.19%

2000 83.00% 2000 42.92% 88.31%
3000 84.53% 3000 47.70% 94.17%
4000 86.00% 4000 52.18% 90.95%
5000 87.00% 5000 44.43% 92.42%

316

VIPIMAGE15_Book.indb 316 9/8/2015 7:47:18 PM

 Table 7. Detailed information describing accuracy and in the maximum number of iterations previously
recall for Top-1 and Top-5 for each class. set, without any tangible results.
The dataset without studio images reached great
Boosted database with extra classes—detailed test data accuracy rating. We state here that studio images
Recall cannot be doctored with the current famous pre-
processing techniques and increase a Convolu-
Class Top-1 Top-3 Top-1 Top-3 tional Neural Network classifier accuracy directly:
extra steps must be made to ensure that the white
Tires 0.78% 0.78% 1.00% 1.00% background does not interfere.
Car mats 0.40% 0.60% 0.40% 0.60% Some possible ways of minimizing the back-
Wine cellars 0.73% 0.82% 0.82% 0.91% ground’s importance in the picture are the use of
Freezers 0.82% 1.00% 0.82% 0.91% image segmentation algorithms such as selective
Microwaves 0.64% 0.73% 0.73% 1.00% search (Sande 2011, Uijlings et al. 2013) or DPM
Dishwashers 0.55% 0.73% 1.00% 1.00% (Felzenszwalb et al. 2010, Trulls et al. 2014) to
Ovens 0.45% 0.82% 0.73% 1.00% extract only the object from the image and create
Cooktops 0.50% 0.50% 0.38% 0.62% features from it; Or inserting dummy backgrounds
Stoves 0.30% 0.40% 0.90% 1.00% from natural images behind objects. This can
Washmachines 0.00% 0.18% 0.64% 1.00% lead to some bias if the background list is small;
Drywashers 0.45% 0.73% 1.00% 100% Also, edge detection filters may eliminate some of
Dryers 0.55% 1.00% 0.64% 1.00% the white interference, as regions with that back-
Centrifuges 0.30% 0.60% 0.90% 0.90% ground will have no edges and no useful informa-
Refrigerators 0.20% 0.50% 0.70% 1.00%
tion. Illumination will also be discarded. Another
Watches 0.12% 0.36% 0.27% 0.55%
improvement is the creation of a tree of classifiers
Suitcases 0.30% 0.30% 1.00% 1.00%
where the top most classifiers are more generic as
Keys 0.64% 0.64% 1.00% 1.00%
the bottom most classifiers are more specific. Also,
using binary classifiers for each class in conjunc-
tion with the main classifier as a pool can add to
The first preprocessing stage led to significant the generality of the solution.
increase in accuracy, but in the training of the last We first evaluated the classifiers performance
classifier for the database with the preprocessing against a image dataset without any preprocessing.
plus the ZCA transformations the classifier did not In this scenario, we obtained an accuracy of 80%
converge in the maximum 5000 iterations. Testing within the top-5 predictions, which means that one
the same approach with a database composed of of the 5 best predictions yielded by the classifier
natural images, however, yielded better results than for a image is correct. On the other hand, for the
the second classifier. top-1, the classifier performed with 34% accuracy.
By training the classifier again with the preproc-
essing techniques described earlier, we were able to
5 CONCLUSIONS improve the classification to a 92% accuracy on the
top-5 and 44% accuracy on the top-1. This means
In our experiments, we first evaluated the classi- an improvement of 15% in the top-5 and 30% in
fiers performance against a image dataset without the top-1 predictions.
any preprocessing. In this scenario, we obtained an The first and second classifiers yielded results as
accuracy of 80% within the top-5 predictions, which expected, as primary preprocessing algorithms such
means that one of the 5 best predictions yielded as affine transformations do not heavily depend on
bythe classifier for a image is correct. On the other the color composition of the image and can give the
hand, for the top-1, the classifier performed with classifier a good amount of generalization. But the
34% accuracy. By training the classifier again with third classifier did not even converge in the maxi-
the preprocessing techniques described earlier, we mum number of iterations previously set, without
were able to improve the classification to a 92% any tangible results. The dataset without studio
accuracy on the top-5 and 44% accuracy on the images reached great accuracy rating. We state here
top-1. This means an improvement of 15% in the that studio images cannot be doctored with the cur-
top-5 and 30% in the top-1 predictions. rent famous preprocessing techniques and increase
The first and second classifiers yielded results a Convolutional Neural Network classifier accu-
as expected, as primary preprocessing algorithms racy directly: extra steps must be made to ensure
such as affine transformations do not heavily that the white background does not interfere.
depend on the color composition of the image and Some possible ways of minimizing the back-
can give the classifier a good amount of generaliza- ground’s importance in the picture are the use of
tion. But the third classifier did not even converge image segmentation algorithms such as selective

317

VIPIMAGE15_Book.indb 317 9/8/2015 7:47:18 PM

 search (Sande 2011, Uijlings et al. 2013) or DPM Graham, N. & A. Sutter (2000). Normalization: Contrast-
(Felzenszwalb et al. 2010, Trulls et al. 2014) to extract gain control in simple (Fourier) and complex (non-
only the object from the image and create features Fourier) pathways of pattern vision.
from it; Or inserting dummy backgrounds from Hunt, K.J., D. Sbarbaro, R. Zbikowski, & P.J. Gawthrop
(1992). Neural networks for control systems—A
natural images behind objects. This can lead to some survey.
bias if the background list is small; Also, edge detec- Jia, Y., E. Shelhamer, J. Donahue, S. Karayev, J. Long, R.
tion filters may eliminate some of the white interfer- Girshick, S. Guadarrama, & T. Darrell (2014). Caffe:
ence, as regions with that background will have no Convolutional architecture for fast feature embed-
edges and no useful information. Illumination will ding. arXivpreprintarXiv:1408.5093.
also be discarded. Another possible improvement Kotsiantis, S.B. & D. Kanellopoulos (2006). Data
that we are currently working on is the creation of a preprocessing for supervised leaning. International
tree of classifiers where the top most classifiers are Journal of ... , 1–7.
more generic and the following classifiers on the tree Krizhevsky, A. & G. Hinton (2011). Using Very Deep
Autoencoders for Content-Based Image Retrieval.
are more specific. Also, using binary classifiers for ESANN.
each class in conjunction with the main classifier as Krizhevsky, A., I. Sutskever, & G.E. Hinton (2012a).
a pool can add to the generality of the solution. ImageNet Classification with Deep Convolutional
Neural Networks. Advances In Neural Information
Processing Systems, 1–9.
REFERENCES Krizhevsky, A., I. Sutskever, & G.E. Hinton (2012b).
Imagenet classification with deep convolutional neural
Arel, I., D.C. Rose, & T.P. Karnowski (2010). Deep networks. In F. Pereira, C. Burges, L. Bottou, and K.
Machine Learning -A New Frontier in Artificial Intel- Weinberger (Eds.), Advances in Neural Information
ligence Research [Research Frontier]. Processing Systems 25, pp. 1097–1105. Curran
Bell, A. & T. Sejnowski (1997a). Edges are the “independ- Associates, Inc.
ent components” of natural scenes. Advances in neural Le, Q.V. (2013). Building high-level features using large
information processing systems, 831–837. scale unsupervised learning. In 2013 IEEE Interna-
Bell, A.J. & T.J. Sejnowski (1997b). The ‘independent tional Conference on Acoustics, Speech and Signal
components’ of natural scenes are edge filters. Vision Processing, pp. 8595–8598.
Research 37(23), 3327–3338. Le, Q.V., R. Monga, M. Devin, G. Corrado, K. Chen,
Cirean, D., U. Meier, & J. Schmidhuber (2011). A M.A. Ranzato, J. Dean, & A.Y. Ng (2011). Build-
committee of neural networks for traffic sign ing high-level features using large scale unsupervised
classification. In Proceedings of the International Joint learning Dataset constructions. ICML, 1–10.
Conference on Neural Networks, pp. 1918–1921. LeCun, Y., L. Bottou, Y. Bengio, & P. Haffner (2001).
Deng, J., W. Dong, R. Socher, L.-J. Li, K. Li, & L. Fei-Fei Gradient-based learning applied to document recog-
(2009). ImageNet: A Large-Scale Hierarchical Image nition. In Intelligent Signal Processing, pp. 306–351.
Database. In CVPR09. IEEE Press.
Dollar, P., Z. Tu, H. Tao, & S. Belongie (2007). Feature Lecun, Y. & C. Cortes. The MNIST database of hand-
mining for image classification. In Proceedings of written digits.
the IEEE Computer Society Conference on Computer Makwana, R., V. Thakar, & N. Chauhan (2011, Nov).
Vision and Pattern Recognition. Evaluation and analysis of illumination normalization
Farabet, C., C. Couprie, L. Najman, & Y. Lecun (2013). methods for face recognition. In Image Information
Learning hierarchical features for scene labeling. Processing (ICIIP), 2011 International Conference on,
IEEE Transactions on Pattern Analysis and Machine pp. 1–6.
Intelligence 35(8), 1915–1929. Ranzato, M., Q.V. Le, R. Monga, M. Devin, K. Chen,
Felzenszwalb, P.F., R.B. Girshick, & D. McAllester G.S. Corrado, J. Dean, & A.Y. Ng (2011). Build-
(2010). Cascade object detection with deformable part ing high-level features using large scale unsupervised
models. In Proceedings of the IEEE Computer Society learning. In International Conference in Machine
Conference on Computer Vision and Pattern Recogni- Learning, pp. 38115.
tion, pp. 2241–2248. Russakovsky, O., J. Deng, H. Su, J. Krause, S. Satheesh,
Förstner, W. (2000). Image Preprocessing for Feature S. Ma, Z. Huang, A. Karpathy, A. Khosla,
Extraction in Digital Intensity, Color and Range M. Bernstein, A.C. Berg, & L. Fei-Fei (2014). Imagenet
Images. In Geomatic Method for the Analysis of Data large scale visual recognition challenge.
in the Earth Sciences, Volume 95, pp. 165–189. Russell, B., A. Torralba, K. Murphy, & W. Freeman
Fuxman, A., A. Kannan, A.B. Goldberg, R. Agrawal, P. (2008). Labelme: A database and web-based tool for
Tsaparas, & J. Shafer (2009). Improving classification image annotation. International Journal of Computer
accuracy using automatically extracted training data. Vision 77(1–3), 157–173.
Proceedings of the 15th ACM International Confer- Sande, K.V.D. (2011). Segmentation as selective search
ence on Knowledge Discovery and Data Mining (2009) for object recognition. ... Vision(ICCV), 2011...(2),
33(2), 1145. 1879–1886.
Goodfellow, I.J., D. Warde-Farley, M. Mirza, A. Schmidhuber, J. (2014). Deep Learning in Neural Net-
Courville, & Y. Bengio (2013). Maxout Networks. works: An Overview. arXivpreprintarXiv:1404.7828,
arXiv preprint, 1319–1327. 1–66.

318

VIPIMAGE15_Book.indb 318 9/8/2015 7:47:18 PM

 Schmidhuber, J. (2015). Deep learning in neural networks: Uijlings, J.R.R., K.E.A. Van De Sande, T. Gevers, &
An overview. NeuralNetworks 61, 85–117. Published A.W.M. Smeulders (2013). Selective search for object
online 2014; based on TR arXiv:1404.7828 [cs.NE]. recognition. International Journal of Computer Vision
Sermanet, P., S. Chintala, & Y. LeCun (2012). Convolu- 104(2), 154–171.
tional neural networks applied to house numbers digit Wu, C., W. Fan, Y. He, J. Sun, & S. Naoi (2014). Hand-
classification. Proceedings of International Conference written Character Recognition by Alternately Trained
on Pattern Recognition ICPR12, 10–13. Relaxation Convolutional Neural Network. 2014 14th
Sigaud, O. & S.W. Wilson (2007). Learning classifier sys- International Conference on Frontiers in Handwriting
tems: A survey. SoftComputing11(11), 1065–1078. Recognition, 291–296.
Simard, P., D. Steinkraus, & J. Platt (2003). Best practices Yaeger, L., R. Lyon, & B. Webb (1996). Effective Training
for convolutional neural networks applied to visual of a Neural Network Character Classifier for Word
document analysis. Seventh International Conference on Recognition. Nips 9, 807–813.
Document Analysis and Recognition, 2003. Proceedings. Zeng, X., W. Ouyang, & X. Wang (2013). Multi-stage
Smirnov, E.A., D.M. Timoshenko, & S.N. Andrianov Contextual Deep Learning for Pedestrian Detection.
(2014). Comparison of Regularization Methods for 2013 IEEE International Conference on Computer
ImageNet Classification with Deep Convolutional Vision, 121–128.
Neural Networks. AASRI Procedia 6, 89–94. Zhang, G.P. (2000). Neural networks for classification:
Springenberg, J. & M. Riedmiller (2013). Improving a survey. In IEEE Transactions on Systems, Man,
Deep Neural Networks with Probabilistic Maxout and Cybernetics, Part C (Applications and Reviews),
Units. arXiv:1312.6116, 1–10. Volume 30, pp. 451–462.
Trulls, E., S. Tsogkas, I. Kokkinos, A. Sanfeliu, &
F. Moreno-Noguer (2014). Segmentation-aware
deformable part models. In Computer Vision
and Pattern Recognition (CVPR),2014 IEEE
Conferenceon, pp. 168–175. IEEE.

319

VIPIMAGE15_Book.indb 319 9/8/2015 7:47:19 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Surface reconstruction of bone fragments: A comparative study

F. Paulano, J.J. Jiménez & J.R. Jiménez

Graphics and Geomatics Group of Jaén, Universidad de Jaén, Jaén, Spain

ABSTRACT: The generation of surface models of bone fragments is useful for many medical applica-
tions. On the one hand, bone fragment models can be visualized in a more effective way. On the other
hand, simulation and analysis techniques usually need surface models as input. Simulation and analysis
processes require surface models to have certain properties and features that depend on the methods used
for these processes. Therefore, the suitability of the method utilized to generate the surface will depend on
the simulation or analysis performed. In this paper, the suitability of two algorithms that generate bone
fragment surface models from point clouds is tested. A proven 2D region growing based method enables
the segmentation of bone fragments from CT stacks. Then a new approach to extract contour points and
estimated normals is used in a pre-process step. The experiments performed allow extracting the benefits
and drawbacks of each method applied to different bones and fracture types. Moreover, the quality of
the obtained results is analyzed focusing on the special features of the data obtained from CT stacks rep-
resenting bone fractures. To the best of our knowledge, nowadays studied methods are not widespread in
the generation of surfaces to represent bone fragments.

Keywords: bone fragments; surface reconstruction; 3D medical images; bone fracture reduction;
segmentation; contour extraction; poisson; ball-pivoting

1 INTRODUCTION (Paulano, Jiménez, & Pulido 2014). Thus the

contours obtained for each slice are very irregular
Computational techniques are frequently used to in fracture zones. All these features influence the
help the specialist in the treatment of bone injuries choice of a method for generating surfaces.
and diseases (Jaramaz, Hafez, & DiGioia 2006). In this paper, a study is performed to test the
These techniques usually require 3D models of bone suitability of surface generation methods to model
fragments as input. In most cases the Marching bone fragments. In a first step, bone fragments
Cubes (MC) algorithm (Lorensen & Cline 1987) is are segmented from CT scans. Then contours are
used in order to obtain isosurfaces from CT scans. extracted from the segmented regions. The point
Due to the complexity showing the bone tissue in clouds used as input for surface generation are
a CT stack (Paulano, Jiménez, & Pulido 2014), the composed of the points of the contours. Finally
models obtained are usually huge and just have top- surface generation methods are tested and their
ological information. Therefore, these models often main advantages and disadvantages are shown.
need to be post-processed before being used. Alter- Specifically, Ball-pivoting (Bernardini, Mittleman,
natively, other methods can be used to generate Rushmeier, Silva, & Taubin 1999) and Poisson
mesh surfaces. The selection of a method depends (Kazhdan, Bolitho, & Hoppe 2006) surface genera-
on the problem to be solved and the features that tion methods are considered. The results obtained
the surface is expected to have. show the performance of each method depending
The generation of surface models from the infor- on the bone and the fracture type. In addition, the
mation available in medical images is a complex task. adequacy and quality of the results are analyzed
In addition, the morphology of bone fragments focusing on the special features of the bone frac-
increases the complexity of the problem. The inner ture data obtained from CT stacks. Specifically, the
part of the bone is composed by trabecular tissue. performed study focuses on point distribution and
This type of tissue is very heterogeneous, thus seg- fracture zones. The paper is structured as follows.
mentation methods usually obtain holes. To make Section 2 describes the studied methods and the
the process more complicated the shape of the procedure followed in order to generate the surface
bone fragments is very irregular and trabecular tis- models. Section 3 shows the results of the study
sue appears in the outer part of the bone due to using different criteria. Finally section 4 discusses
the fracture (Paulano, Jiménez, & Pulido 2013) the obtained results and concludes the paper.

321

VIPIMAGE15_Book.indb 321 9/8/2015 7:47:19 PM

 2 METHODOLOGY internal triangles due to the anatomy of trabecular
tissue. As an alternative, we propose to generate
Nowadays, most of the published studies that work contours after the segmentation process and then
with bone fragment models use MC to generate remove internal contours in 2D. Finally normals
them. These models are usually huge and contain are estimated using the topological information of
noise due to the morphology of trabecular tissue. the contours. This process is explained in detail in
As an alternative, two different methods have been next paragraphs.
considered for this study: Ball-pivoting (Bernardini, To begin with, all the bone fragments are segmented
Mittleman, Rushmeier, Silva, & Taubin 1999) and from CT scans. For this purpose, a semi-automatic
Poisson surface generation (Kazhdan, Bolitho, & method is used (Paulano, Jiménez, & Pulido 2014).
Hoppe 2006). The method is based on 2D region growing and only
Ball-pivoting is a geometric algorithm that trian- requires that the user places a seed in the first slice in
gulates a point cloud by traversing a ball of a spe- which each bone fragment appears. The method is
cific size over the surface. The algorithm starts with also able to deal with erroneously joined fragments.
an initial triangle and test if the ball is able to reach In order to split them the method uses an isolated
a point from each triangle edge. If so, the point region growing algorithm, hence additional seeds can
reached is added to the model. This procedure is be required for this special case.
repeated until the ball is not able to reach any points The segmentation algorithm generates a region
from the current edges. This method performs well for each bone fragment in each slice. Then the con-
if the distance between points is homogeneous. tour of each region is extracted. For this purpose,
Otherwise some points could not be triangulated an approach similar to the proposed by Pulido et al.
and the obtained model would be incomplete. is applied (Pulido, Paulano, & Jiménez 214). First,
Poisson surface reconstruction requires as input the marching squares algorithm is utilized in order
a set of points and their associated normals. This to generate contours from the segmented regions
method formalizes the surface approximation as (Ho, Wu, Chen, Chuang, & Ouhyoung 2005). This
a Poisson problem. Both the points and normals algorithm can be considered a 2D adaptation of
are interpreted as samples of an indicator function MC. Using a divide-and-conquer approach, the
that is 0 outside a model and 1 inside it. In order algorithm studies each cell composed of four points
to approximate the indicator function trivariant in a slice to detect boundaries. The algorithm does
B-splines are used. This algorithm is robust against not ensure that the generated contours are closed,
noise in the input data. Nevertheless the approxi- but this is not relevant in this case since the goal is
mation causes that the original points of the con- to generate point clouds.
tours are not preserved. Due to the noise present in the regions because
The generation of surfaces requires point clouds of the trabecular tissue, inner contours can be
that represent the outer part of each bone frag- obtained. These inner contours are removed using
ment. Therefore these point clouds need to be the point in polygon algorithm by (Feito & Torres
extracted from medical images. Most of the pro- 1997), hence only the outer contour of each frag-
posed works use MC to generate isosurfaces from ment is extracted in each slice. This algorithm
medical images. This method could be used to gen- allows determining if a contour is inside other
erate point clouds and estimate the normal associ- contours without the need to perform complex
ated to each point. However, MC generates many calculations, such as solving a system of equations.

Figure 1. Point clouds extracted from CT scans. These points are used as input for the surface reconstruction algo-
rithms. A—Spiral fracture of distal fibula. B—Simple fracture of patella. C—Comminuted fracture of radius.

322

VIPIMAGE15_Book.indb 322 9/8/2015 7:47:19 PM

 After removing internal contours, the points of (see Fig. 1). These CT stacks contain fragments
each external contour are joined in order to form of different bones and fracture types. The resolu-
the point cloud used as input by the surface gen- tion of all images is 512 × 512. The results show
eration algorithms (see Fig. 1). the performance of each method depending on the
In addition to the outer contours, the Poisson features of both the bone and the fracture.
surface reconstruction algorithm also requires The first CT stack shows a spiral fracture
the normal of each point. These normals are of distal fibula (see Fig. 1, A). The fracture
estimated in 2D for each point using topological generated two fragments of similar size. The stack
information from contours. For each three con- has 173 slices with dimensions 159 × 159 mm
secutive points Pi–1, Pi and Pi+1 in a contour, the and 0.625 mm spacing. The next three datasets
normal associated to Pi is estimated as the sum of contain simple fractures in which a small and a big
the normals of the two segments formed by Pi–1 – fragment were generated. The second stack rep-
Pi and Pi – Pi+1. The third component of each nor- resents a simple fracture of patella (see Fig. 1, B)
mal is defined by the plane of the slice. Despite of and contains 157 slices. The dimensions of each
this estimation, normals are accurate enough to slice are 156 × 156 mm and the separation between
estimate the surface curvature. Once point clouds slices is 0.625 mm. The third stack represents a sim-
have been generated, surface generation methods ple fracture of calcaneus and contains 205 slices.
can be applied. The separation between slices is 0.625 mm and the
dimensions of each slice are 173 × 173 mm. The
fourth stack contains a simple fracture of astra-
3 RESULTS galus and it is formed by 156 slices with dimen-
sions 150 × 150 mm and with 0.625 mm spacing.
In this study the suitability of two different methods The fifth CT stack contains a comminuted fracture
to generate surfaces of bone fragments is tested: of distal tibia and it is formed by 173 slices with
Ball-pivoting and Poisson. These methods have dimensions 159 × 159 mm and with 0.625 mm
been widely used to generate surfaces for different spacing. The fracture generated 3 fragments of
applications (Campos, Garcia, & Nicosevici 2011) irregular shape. The last stack shows a comminuted
(Wiemann, Annuth, Lingemann, & Hertzberg fracture of radius that contains 5 fragments (see
2015). Nevertheless, their use is not widespread Fig. 1, C). The stack has 134 slices with dimensions
in the generation of surfaces to represent bone 96 × 96 mm and 1.25 mm spacing.
fragments. Table 1 shows the size of the models obtained
In order to test the performance of the two using the two studied methods. Ball-pivoting does
studied methods, they were used to generate not modify the original points of the contours.
surfaces from 6 CT stacks previously processed Although there is no excessive difference in size
to extract bone fragments contours in each slice between the models generated by the two methods,

Table 1. Number of vertices and faces generated for each of the datasets.

Original point cloud Poisson reconstruction Ball-pivoting

Dataset Fragment Points Points Triangles Points Triangles

Fibula 1 18919 13500 26988 18919 20837

Fibula 2 16501 14863 29722 16501 19924
Patella 1 11734 8177 16350 11734 21558
Patella 2 994 999 1994 994 1099
Calcaneus 1 24315 23546 47088 24315 42952
Calcaneus 2 1866 1599 3194 1866 3283
Astragalus 1 22053 22764 45508 22053 37677
Astragalus 2 4740 4087 8162 4740 8603
Tibia 1 89889 74475 149094 89889 101292
Tibia 2 19696 17879 35742 19696 23890
Tibia 3 30320 26000 52012 30320 35464
Radius 1 21965 16829 33650 21965 26860
Radius 2 2352 1608 3212 2352 4117
Radius 3 1772 1337 2670 1772 2068
Radius 4 529 610 1212 529 646
Radius 5 717 1011 2018 717 949

323

VIPIMAGE15_Book.indb 323 9/8/2015 7:47:19 PM

 the number of points of the models generated by proximity of the points in the same slice (see Fig. 2,
the Poisson reconstruction method is slightly lower B). Comminuted fractures generate more than two
but the number of triangles is higher in most cases fragments. These fractures are commonly caused by
in comparison with Ball-pivoting. Both methods high-energy traumas, hence bone can be deformed
obtain good results in the reconstruction of sim- and small fragments are usually generated. There-
ple fractures in which bone fragments have similar fore, both methods have the same problems before
size. Poisson reconstruction generates better mod- mentioned for small fragments. Furthermore, one
els from a visual point of view (see Fig. 3, A). How- of the stacks tested has 1.25 mm spacing. This
ever, Ball-pivoting preserves the original features of causes that the distance between points is quite het-
the model since does not modify the original points erogeneous. In this case, Ball-pivoting is not able to
of the contours (see Fig. 2, A). This is an advan- connect many of the triangles, hence the generated
tage when models are used in processes that utilize models contain holes and sometimes are incom-
bone features, such as virtual bone fracture reduc- plete (see Fig. 2, C). Poisson also has difficulty in
tion. Other simple fractures lead to the generation modeling the most detailed areas when the distance
of a small fragment and a big fragment. Avulsed between slices is high (see Fig. 3, C).
fracture is one of these cases. Poisson reconstruc- Additionally, the adequacy and quality of the
tion method does not respect the shape of the small obtained results are analyzed depending on the
bone fragment. In addition, the method has diffi- special features of the data obtained from CT
culties in respecting the shape of the bone at the top scans, which differs from the data not-related with
and the bottom of the big fragment (see Fig. 3, B). bone fractures.
On the other hand, Ball-pivoting retains the shape The distribution of the points generated from a
of small fragments but sometimes does not gener- CT stack depends on the resolution and the size of
ate correct triangles in small fragments due to the the 2D images, and the distance between slices. Since

Figure 2. Surfaces generated using the Ball-pivoting algorithm. A—Spiral fracture of distal fibula. B—Simple frac-
ture of patella. C—Comminuted fracture of radius.

Figure 3. Surfaces generated using the Poisson algorithm. A—Spiral fracture of distal fibula. B—Simple fracture of
patella. C—Comminuted fracture of radius.

324

VIPIMAGE15_Book.indb 324 9/8/2015 7:47:19 PM

 Table 2. Point distribution for each CT stack tested. Resolution of all slices is 512 × 512.

Size of Distance between Spacing between

slice (mm) points (mm) slices (mm) Ratio

Fibula 159 × 159 0.3105 0.625 2.0126

Patella 156 × 156 0.3047 0.625 2.0513
Calcaneus 173 × 173 0.3379 0.625 1.8497
Astragalus 150 × 150 0.293 0.625 2.1333
Tibia 159 × 159 0.3105 0.625 2.0126
Radius 96 × 96 0.1875 1.25 6.6667

the resolution of all the tested images is 512 × 512,

the only variables are the size of the images and the
distance between slices. Table 2 shows the value of
both variables for all tested CT stacks. For the first
5 CT stacks the distance between slices approxi-
mately doubles the distance between points in the
same slice. Despite the fact that both algorithms
perform better if the points are uniformly distrib-
uted, they obtain acceptable results in these 5 cases.
The distribution of the last CT stack is more het-
erogeneous because the distance between slices is
more than 6 times higher than the distance between
points in the same slice. In this situation, Poisson
reconstruction performs better than Ball-pivoting.
Poisson only has difficulties in the more detailed
areas, such as the fracture zone. Otherwise, Ball-
pivoting is not able to properly triangulate in most
cases, hence the obtained models contain many
holes and incorrect triangles. Figure 4. Two views of the same fracture zone. Model
One of the most important applications of was generated using Poisson reconstruction. Original
the generation of bone fragment models from points are in red. Differences in position can be appreciated
CT images is the simulation of a bone fracture between original and generated points.
reduction. In recent years, some methods have been
proposed in order to help specialists in the preoper-
ative planning of this surgery (Fürnstahl, Székely, 4 DISCUSSION AND CONCLUSIONS
Gerber, Hodler, Snedeker, & Harders 2012). Using
these methods, specialists can know the number of In this paper a comparison between two well-
fragments and their correct position before the real known methods to generate surfaces of bone frag-
intervention; hence surgery time can be reduced ments has been performed. This experiment has
(Sugano 2003). Some of the proposed methods try previously required extracting point clouds repre-
to virtually reduce bone fractures by calculating, senting the outer part of each bone fragment from
matching and registering fracture zones (Zhou, CT scans. In order to segment bone fragments
Willis, Sui, Anderson, Brown, & Thomas 2009). from CT stacks, the method proposed in (Paulano,
Therefore, it is important that the reconstruction Jiménez, & Pulido 2014) has been used. The
method does not modify the shape of the fracture method has been able to segment bone regions and
zone. Since Poisson reconstruction generate new to separate bone fragments. Then contours have
points and does not respect the original points of been extracted and point normals have been esti-
the contour (see Fig. 4), these models are not suit- mated using a new approach. Finally, some tests
able for being used in a virtual fracture reduction have been performed with the aim of knowing the
process. On the other hand, Ball-pivoting not only benefits and shortcomings of each method when
keeps the original points of the contours, but also generating models of different bones and fracture
adds topological information to the models that types.
can be helpful during the bone fracture reduction The performed tests have shown the advantages
process. and disadvantages of each method when generating

325

VIPIMAGE15_Book.indb 325 9/8/2015 7:47:20 PM

 surfaces of fractured bones. Poisson surface Feito, F.R. & J.C. Torres (1997). Inclusion test for general
reconstruction obtained visually good results, but polyhedra. Computers & Graphics 21(1), 23–30.
does not respect the original shape of the bone Fürnstahl, P., G. Székely, C. Gerber, J. Hodler,
fragments. This can be a drawback in simulation J.G. Snedeker, & M. Harders (2012, April). Computer
assisted reconstruction of complex proximal humerus
processes that require accuracy, such as virtual fractures for preoperative planning. Medical image
fracture reduction. In addition small details in bone analysis 16(3), 704–20.
fragments can be removed. Ball pivoting obtained Ho, C.-c., F.-c. Wu, B.-Y. Chen, Y.-Y. Chuang, & M. Ouh
visually acceptable results in most cases and did young (2005). Cubical marching squares: adaptive
not modify the original position of the points. feature preserving surface extraction from volume
However, the method could not assure that all the data. Computer Graphics Forum 24(3), 537–545.
points are used to build the mesh because points Jaramaz, B., M. Hafez, & A. DiGioia (2006, Sept).
are not always uniformly distributed. Therefore, Computer-assisted orthopaedic surgery. Proceedings
incomplete results could be obtained. of the IEEE 94(9), 1689–1695.
Kazhdan, M., M. Bolitho, & H. Hoppe (2006). Poisson
According to the obtained results, the perform- surface reconstruction. In Proceedings of the fourth
ance of the two methods is related to the quality of Eurographics symposium on Geometry processing,
the CT stacks. Both methods obtain better results Cagliari, Sardinia, Italy, pp. 61–70. Eurographics
if the points are homogeneously distributed. For Association.
that purpose, the distance between slices should Lorensen, W.E. & H.E. Cline (1987). Marching cubes:
be smaller, increasing the radiation to the patient. A high resolution 3D surface construction algorithm.
Therefore, the development of new data acquisi- In ACM Siggraph Computer Graphics, Volume 21, pp.
tion techniques that reduce radiation would make 163–169. ACM.
improve the results obtained by both reconstruc- Paulano, F., J. Jiménez, & R. Pulido (2013). Trends on
identification of fractured bone tissue from CT images,
tion algorithms. On the other hand, results of the pp. 263–269. CRC Press.
segmentation also influence the obtained models. Paulano, F., J.J. Jiménez, & R. Pulido (2014, May).3
Due to the fracture, trabecular tissue is in the outer Dsegmentation and labeling of fractured bone from
part of the bone. The irregularity of this type of CT images. The Visual Computer 30(6–8), 939–948.
tissue and the low resolution of the CT stack make Paulano, F., J.J. Jiménez, & R. Pulido (2014). Identification
difficult to correctly identify the boundary of bone of fractured bone tissue from ct images. Computer
fragments, especially when two fragments appear Methods in Biomechanics and Biomedical Engineering:
joined by their fracture zones in the CT image. Imaging & Visualization.
This may lead to the generation of wrong surfaces Pulido, R., F. Paulano, & J.J. Jiménez (214).
Reconstruction & interaction with 3d simplified
in the fracture area. bone models. In WSCG 2014 Communication Papers
Proceedings, pp. 321–327.
Sugano, N. (2003). Computer-assisted orthopedic sur-
ACKNOWLEDGEMENTS gery. Journal of Orthopaedic Science 8(3), 442–448.
Wiemann, T., H. Annuth, K. Lingemann, &
This work has been partially supported by the J. Hertzberg (2015). An extended evaluation of open
Ministerio de Economía y Competitividad and the source surface reconstruction software for robotic
European Union (via ERDF funds) through the applications. Journal of Intelligent &Robotic Systems
research project TIN2011-25259. 77(1), 149–170.
Zhou, B., A. Willis, Y. Sui, D. Anderson, T. Brown, &T.
Thomas (2009, September).Virtual 3D bone fracture
reconstruction via inter-fragmentary surface align-
REFERENCES ment. In IEEE Workshop on 3D Digital Imaging and
Modeling (3DIM), pp. 1809–1816. IEEE.
Bernardini, F., J. Mittleman, H. Rushmeier, C. Silva, &
G. Taubin (1999, Oct). The ball-pivoting algorithm for
surface reconstruction. Visualization and Computer
Graphics, IEEET transactions on 5(4), 349–359.
Campos, R., R. Garcia, & T. Nicosevici (2011, June).
Surface reconstruction methods for the recovery of 3d
models from underwater interest areas. In OCEANS,
2011 IEEE—Spain, pp. 1–10.

326

VIPIMAGE15_Book.indb 326 9/8/2015 7:47:22 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Towards a robust patch based multi-view stereo technique

for textureless and occluded 3D reconstruction

B. Haines & L. Bai

Department of Computer Science, University of Nottingham, Nottingham, UK

ABSTRACT: Patch based reconstruction methods have been and still are one of the top performing
approaches to 3D reconstruction to date. Their local approach to refining the position and orientation
of a patch, free of global minimisation and independent of surface smoothness, make patch based meth-
ods extremely powerful in recovering fine grained detail of an objects surface. However, patch based
approaches still fail to faithfully reconstruct textureless or highly occluded surface regions; thus, though
performing well under lab conditions, they deteriorate in industrial or real world situations. They are also
computationally expensive. Current patch based methods generate point clouds with holes in texturesless
or occluded regions that require expensive energy minimisation techniques to fill and interpolate a high
fidelity reconstruction. Such shortcomings hinder the adaptation of the methods for industrial applica-
tions where object surfaces are often highly textureless and the speed of reconstruction is an important
factor. This paper presents on-going work towards a multi-resolution approach to address the problems,
utilising particle swarm optimisation to reconstruct high fidelity geometry, and increasing robustness to
textureless features through an adapted approach to the normalised cross correlation. The work also aims
to speed up the reconstruction using advances in GPU technologies and remove the need for costly feature
based initialisation and expansion. Through the combination of these enhancements, it is the intention of
this work to create denser patch clouds even in textureless regions within a reasonable time. Initial results
show the potential of such an approach to construct denser point clouds with a comparable accuracy to
that of the current top-performing algorithms.

1 INTRODUCTION Global based methods perform reconstruction by

defining a global objective on the surface; hence,
High performance 3D reconstruction via Multi-View minimising an energy functional based on the
Stereo (MVS) has received much interest from both photo-consistency score over the entire surface.
academia and industry. Seitz et al. (2006a) and Jensen Vogiatzis et al. (2007), model the problem as a graph
et al. (2014) both showed that MVS algorithms can cut, extracting the globally optimal solution to a
achieve reconstruction accuracies paralleling that of weighted minimal surface. The energy functional is
prohibitively expensive laser scanners. However, to biased towards the minimal surface, smoothing out
achieve robustness for a wide range of input imagery, high frequency details in favour of lower surface
such works demand high processing times limiting area, as such a user defined ballooning term must
their adoption into the public domain. be added. Such graph cut methods struggle to
Most MVS algorithms define a surface and check capture high frequency surface variations whilst
its fidelity via its photo-consistency value throughout also reconstructing thin structures due to the over-
the image set. This value is normally computed using smoothing induced by the ballooning term, volume
similarity metrics such as the Normalised Cross resolution and minimal surface requirement.
Correlation (NCC) or the Sum of Squared Differences Level set methods presented by Cremers
(SSD). The issue plaguing reconstruction algorithms and Kolev (2011), or mesh based deformation
of this sort is that the search space for the lowest cost frameworks of Esteban and Schmitt (2004) and
photo-consistent surface is not only extremely large Hiep et al. (2009), extract the locally minimal
but also extremely costly to evaluate. Large high reso- solution to the globally defined energy functional
lution image sets, with textureless regions, occlusion by iteratively deforming the surface based on its
and lighting assumptions make the matching process local derivatives. If the energy function is convex
highly complex. then these can also converge to the globally optimal
3D reconstruction via MVS can roughly be solution. These methods evolve based on two terms
divided into two categories, global and local. in the energy functional, a data-term measuring

327

VIPIMAGE15_Book.indb 327 9/8/2015 7:47:22 PM

 the photo-consistency and a regularisation term, • A multi-resolution approach creating denser
enforcing smoothness constraints on the surface. patch clouds in textureless image regions.
Along with high computational cost, the regu-
larisation term often leads to an overly-smoothed
solution lacking sharp features. 2 NORMALISED CROSS CORRELATION
Local methods focus on refining only a small
patch of the surface. A planar patch, p, is created The NCC metric returns the inner product between
with position, c(p), and normal, n(p). The view with two image patches, reflecting a measure of similarity
the shallowest angle to the patch normal observes between them. The metric is very popular for com-
this patch best, becoming the patches reference view, paring image patches due to its invariance to both
R(p). This patch is then confined to lie on a viewing additive and multiplicative global illumination
ray through the reference views camera centre. changes of the form I α I β , where α is an
To refine the patch to better fit the surface its additive and β a multiplicative global illumination
back projection in the image set is sampled over change. This allows for two images that where taken
a μ ∗ μ pixel window, and its photo-consistency at different times under different illuminations to
score is evaluated. The goal of refinement is to be compared as if they were taken instantaneously
find the optimal position and normal in which the under the same lighting conditions.
photo-consistency score is lowest. To optimise the
patch parameters Habbecke and Kobbelt (2007) 1
suggests using the Gauss-Newton algorithm γ ( ) = 1− ∑
3 | V ( p )//R( p ) | υ ∈V ( p )//R ( p )
to solve the non-linear least squares problem,
whereas Furukawa and Ponce (2010a) suggests ∑ g(( R ( p ) , Iυ ,c ) (1)
using a conjugate gradient method. Both methods c ∈{ , g ,b}
rely on derivative based optimisation and hence a
precise initialisation of the patch is very important The photo-consistency value γ(p) of a patch p
to avoid convergence to an incorrect local minima. visible in the set of images V(p), with R(p) as the
Furukawa and Ponce (2010a) proposed the reference view, is shown in Equation 1. The NCC
well-known PMVS algorithm which begins by for the patch window projected into images I1 and
initialising a sparse patch cloud through feature I2, for the colour channel, c, is denoted by g(I1,
matching within the image set. These matches I2, c). Each colour channel computes a NCC score,
can be robustly refined and then expanded to and each view computes a score with the reference
initialise new patches close to those just created. view. The score is then averaged out and converted
Following expansion a filtering step is performed to a minimisation problem to compute the final
to remove erroneous patches that occur at depth photo-consistency score.
discontinuities, occlusion or textureless areas. The
reliance on feature based initialisation, expansion 2.1 Limitations of NCC
and filtering limits the extent at which patches can
expand into textureless regions; therefore, highly The NCC performs very well in highly textured
textured regions are densely covered and texture- circumstances due to its invariance to illumination
less regions, or regions seen only by a few views, changes, however when using it to compare con-
are sparsely covered, if at all. Chen et al. (2013) tinuous coloured patches, its value is undefined.
replaced the derivative based refinement stage with Given the presence of image noise it is unlikely
a Particle Swarm Optimisation (PSO) following a that textureless surface patches will be completely
similar matching, expansion and filtering process. constantly coloured in either the reference or
This technique showed promise in filling texture- comparison views. However, in such cases the NCC
less areas but still does not allow for fully dense still fails to return useful information relating to
patch cloud creation for objects with large texture- similarity of the patches. Rather it returns a meas-
less areas, where no matching can be obtained, or ure of the noise correlation that does not relate
datasets with a low number of input views. to the underlying patch similarity. Using an NCC
This paper proposes an alternate approach to based similarity metric to refine textureless patches
patch based MVS with the following contributions: fails, as the minimum score does not relate to the
optimum patch parameters.
• An improved NCC photo-detail-consistency
metric allowing better handling of textureless
2.2 Photo-detail-consistency
image channels.
• A PSO approach to patch refinement avoiding The photo-consistency measure defined in
the need for precise feature based initialization Equation 1 can suffer similar anomalous results
and expansion. when comparing highly textured patches as it does

328

VIPIMAGE15_Book.indb 328 9/8/2015 7:47:22 PM

 in the textureless case. If the NCC is computed 3 PARTICLE SWARM OPTIMISATION
per image channel, and one or more of the image
channels have low texture, these channel will return Derivative based optimisation methods such as
incorrect measures. This effectively skews the simi- gradient or conjugate gradient decent, Newton and
larity score when summed and averaged with the Quasi-Newton methods are popular techniques for
results from the highly textured channels. Further optimising a functions parameters. They are often
to this, when a patch is projected into an image highly effective when correctly initialised, in most
where the angle between the patch and image plane cases requiring only a small number of iterations to
is high, bilinear interpolation is used to sample the converge. However, in cases of poor initialisation
patch. In cases where the patch axes fail to cover or a noisy cost function such methods suffer from
sufficiently many pixels in the image based match- a tendency to get stuck in local minima, where the
ing window, bilinear interpolation returns similar derivatives of the cost function become zero. As
values, creating a textureless effect. such a careful and costly initialisation or the need
to run the method over multiple seed points is
1
γ ( ) = 1− ∑ wυ
3 | V ( p )//R( p ) | υ ∈V ( p )//R ( p )
required, reducing their accuracy and efficiency.
PSO is a derivative free population based
stochastic optimisation method for continuous non-
∑ wc g ( R ( ) , Iυ , ) (2) linear functions, introduced by Kennedy (2010).
c ∈{ , g ,b}
Inspired by the social behaviours of animals such
as birds flocking or fish schooling, it can be used to
Equation 2 proposes adaptively weighting the
iteratively minimise or maximise the solution to a
NCC score based on weights wc and wv. We define
given candidate function, using a population based
this new similarity metric as the photo-detail-
search strategy with deterministic and probabilistic
consistency. wc, computes the amount of texture
rules. All particles in a swarm move through the
per image channel, and wv, weights the angle of
search space looking for the best solution and
the image plane to the patch normal. wc prevents
updating their position using a velocity defined in
textureless colour channels contributing to the
Equation 5.
similarity metric. Thus it is close to zero when the
texture on the image channel is low and high when
there is sufficient texture. vi ( ) ω v i (t ) + ϑ p ∗ rand () ∗ ( pbest i xi (t ))
+ ϑ g ∗ rand () ∗ (gbest − x (t ))
i
−t 2 / 2σ c2
wc 1 e (3) + ϑ l ∗ rand () ∗ (lbest
( lbest i xi (t ))
(lb
+ ϑ n ∗ rand () ∗ (nbest
( nbest i xi (t )) (5)
where t = ||∇IR(p),c(x, y)||
In Equation 3, wc is computed per patch, p, where The i-th particle of the swarm, at the t-th itera-
∇IR(p),c(x, y) is the gradient of the colour channel, tion, is updated based on its current position, xi(t),
c, for its reference image, IR(p). This is taken at the and velocity, vi(t). An inertia factor, ω, is used to
image point (x, y), that is the projection of the patch help stop the effects of stagnation. Stagnation
center, c(p), into its reference image. In practice the occurs when a particles velocity becomes zero as
detail coefficients of the Haar Wavelet Transform its position coincides with the current best solu-
from the multi-resolution approach presented tion, all other particles will be drawn towards it.
in Section 4 are used to approximate the image If no other particle improves its solution on the
gradient. way towards the current particle this may lead to
premature convergence, where no further improve-
if ( p ) ⋅ d (υ ) > 0 ment can be made.
⎧0
wυ = ⎨ − n ( p )⋅vd (υ ) / 2σ υ2 (4) The evolution of each particle in the swarm is
⎩1 − e if ( p ) ⋅ d (υ ) ≤ 0 also controlled by different social factors: its per-
sonal best visited location, pbesti, the global best
To help reduce the influence of views with location found over the entire swarm, gbest, the
sharper viewing angles, wv is calculated per view best position found by its local neighbourhood in
using the patches normal, n(p), and the current terms of its index into the swarm, lbesti, and finally
views principle view direction, vd(v). Factors σc the best position found in terms of its nearest spa-
and σv from Equations 3 and 4 respectively are cial neighbours, nbesti. The influence of the per-
tunable parameters controlling the strength of sonal, global, local and nearest social factors are
the filtering; smaller values represent stricter weighted by ϑp, ϑg, ϑl and ϑn respectively. Finally
filtering. In our experiment σc and σv are fixed rand() simulates the stochastic nature of flocking,
at 0.2. generating a random number in the range [−1, 1].

329

VIPIMAGE15_Book.indb 329 9/8/2015 7:47:23 PM

 Setting ϑp, ϑg = 2 and ϑl, ϑn = 0, results in the coordinates, θ and φ. In order to constrain the
traditional particle swarm optimisation introduced search space without the requirements of feature
by Kennedy (2010). Setting ϑp, ϑl = 2.05 and ϑg, matching, the depth range is set to the patches
ϑn = 0, defines the improved Standard Particle intersection of its viewing ray with the scenes
Swarm Optimisation (SPSO) presented by Bratton bounding volume. This is easily found using a
and Kennedy (2007). Setting ϑp, ϑg, ϑl, ϑn = 1, results simplex algorithm to compute the convex hull of
in Pongchairerks and Kachitvichyanukul (2005) the viewing volume. If silhouettes are available the
Global Local Nearest PSO (GLN-PSO) structure. visual hull can be obtained and the depth range
Mutation operations from Wang et al. (2007) can can be further restricted to lie within the front and
be applied, reguardless of the social structures back intersection of the viewing ray to the visual
used, to further avoid the effects of stagnation. hull. This information can be extracted for every
pixel, regardless of whether it contains a feature
match. This also guarantees that the search space
3.1 PSOs application in MVS
must contain the correct depth position, regardless
In the context of MVS the cost function in PSO of how deep any surface concavity may be.
will become the per-patch photo-detail-consistency The patch normal search space, expressed in
measure. As this is computationally expensive spherical coordinates relative to its reference view,
to evaluate we implement our PSO algorithm is restricted such that θ, φ ∈ [0, π]. This ensures
on the GPU using OpenCL to achieve real time that the patch does not become back facing in the
performance, similarly to the synchronous version reference view. Having such wide search space on
presented by Zhou and Tan (2009). the patch normal allows for sharp grazing angles
Chen et al. (2013) originally proposed the use of of the patch against the reference view. Scaling up
GLN-PSO to optimise a patches parameters inside the patch to a μ ∗ μ matching window in the refer-
the framework of PMVS. By using the sparsely ence view should go some way to account for this
matched features as initialisation they were effect, whilst still allowing correct patch refinement
able to restrict the optimisation search space to for these conditions.
nearby depth and normal values, thus performing Given that PSO refinement of a patch only
optimisation similarly to gradient decent. Their requires knowledge of an approximate depth and
results proved effective but due to their reliance normal range, there is no need to perform the match-
on feature detection, expansion and filtering their ing and expansion procedures needed for precise
approach is limited in terms of expanding into initialisation of derivative based methods. Instead
larger texturless regions. We look to build on their for each pixel in the silhouette we can automatically
work by removing the reliance on feature matching generate a search space and refine a patch, follow-
and widening the search space of PSO, hoping to ing a patch per pixel approach. From only a coarse
achieve a similar result without the requirement of search space initialisation, in highly textured areas,
feature matching. PSO performs comparatively to derivative based
Although GLN-PSO can have superior per- methods, converging on the same optimia.
formance in terms of finding the optimal solution Using our implementation of GPU based PSO
in less iterations, its reliance on the global best we are able to refine a single 5 × 5 patch in roughly
particle and neighbouring best particles positions 2–4 ms, depending on the number of iterations of
are not easily adapted to a GPU based algorithm. the swarm. No doubt this speed could be decreased
Knowledge of the global best particle would require by a more heavily optimised kernel, however this
running a reduction operation on all the particles has served well as a proof of concept. Increasing
to find the current best solution per iteration of the the patch size deteriorates the run time and also
swarm. Then each particle would require a slow begins to over-smooth the patch cloud. As larger
global memory read to access this global best posi- patches approximate a larger surface area, reduc-
tion over each kernel. Further to this calculating ing the patches ability to reconstruct high fre-
a particles spacial neighbours also requires high quency surface details.
levels of computation that is not easily parallelized Figure 1 compares the results of running PMVS,
to the GPU. Therefore we opt to excluded these the current top performing patch based method,
high computational force operations only using relying on gradient decent and expansion from
personal and local forces in the form of SPSO. precisely matched features, against our PSO based
Mutation operations are also randomly applied to approach for the sparse ‘Dino’ benchmark avail-
add further robustness. able from Seitz et al. (2006b). Source code for the
The search space for patch refinement can be PMVS algorithm is available from Furukawa and
relatively large, but it is of low dimensionality, one Ponce (2010b).
for the patch depth, λ, along its viewing ray and Figure 1(a) shows that PMVS failed to create as
two for the patch normal expressed in spherical dense a set of patches, hence is unable to express

330

VIPIMAGE15_Book.indb 330 9/8/2015 7:47:24 PM

 as well as in-place, requiring no extra memory
than is already required to store the images, which
is important when dealing with large, high resolu-
tion image sets.

4.1 Wavelets for PSO patch refinement

In theory patch based methods are only limited in
accuracy by the resolution of the input imagery.
However, in practice the size of the patch and its
corresponding matching window in image space
can impact its ability to reconstruct high frequency
surface details. Klowsky et al. (2012) presented a
study on how patch size affects the correspond-
ing frequencies that can be reconstructed. They
Figure 1. Reconstruction results for the sparse Dino show that adapting the patch size to the surface
dataset. frequency is incredibly important in defining sur-
face details. Smaller patches require less samples
and computation, and can reconstruct higher
high frequency surface details such as the variation frequency details, however, they’re less stable and
on the fins of the ‘Dino’ model. In low textured susceptible to errors in refinement due to lack of
areas the patch cloud contains large holes, where texture variance over the samples of the patch.
patches are filtered out due to incorrect refine- This limitation occurred in the PSO method pre-
ment, particularly note the dinosaur’s back and sented above, where refining small patches over
legs. Figure 1(b) shows a much denser patch cloud texturless regions meant that the global minimum
created by our PSO refinement, using the patch per of the photo-detail-consistency function no longer
pixel approach. In highly textured areas it is able to corresponds to the correct patch parameters.
capture the same degree of accuracy but at a sig- Increasing the patch size creates a more stable
nificantly higher resolution. However in low tex- and less noisy patch cloud filling larger textureless
tured areas where the variation over the patch only regions, however, this comes at a cost of deteriorat-
expresses noise, our method also produces incor- ing the run time and over smoothing the patch cloud.
rectly refined patches. Rather than filter these out Therefore, using the correct patch size for accuracy
and create holes, to help resolve incorrect refine- and computational gains is a core problem in a patch
ment a multi-resolution approach is proposed in based method, choosing an optimal patch size is a
the following section. challenging problem. Habbecke and Kobbelt (2007)
suggest using the texture variance to adapt the patch
size, however, this can lead to high computational
4 MULTI-RESOLUTION cost as the number of surface samples grows with
the patch size. Here we adopt a multi-resolution
The lifted Haar wavelet transform, Sweldens and approach to achieve a similar effect, without the cost
Schröder (2000), is a popular tool for signal process- of increasing the surface samples.
ing and multi-resolution analysis. It takes an image The Haar wavelet transform compresses large tex-
and removes high frequency components storing tureless areas into smaller ones, reducing the number
them in detail coefficients. The high frequency com- of textureless pixels that describe the underlying sur-
ponents are then removed from the input image, face of a patch. Hence, lower resolution approxima-
forming a lower resolution approximation image, tion images induce textured regions and a stabilising
which in turn is stored in approximation coef- effect in a similar way to increasing the extent of
ficients. The original image can be reconstructed the patch, without increasing the cost of sampling
by applying the inverse wavelet transform, com- and computing the photo-detail-consistency metric.
bining detail coefficients with the approximation Therefore, the PSO algorithm can be run at a lower
coefficients. The approximation image can also be resolution, where patches are more robust to texture-
further decomposed until a desired level is reached less areas. This can be seen in Figure 2, showing the
forming a full multi-resolution decomposition. results of PSO refinement at multiple resolutions.
Classically this transform is applied using high and In textureless regions, patch clouds generated from
low pass filters in the frequency domain, however lower resolution images do not normally suffer the
in a second generation lifted wavelet framework, same problems as their higher resolution counter-
these filters can be expressed in the spatial domain. parts. In particular, note how the highly texturless
This allows the transform to be performed faster region over the dinosaurs back is stably recovered at

331

VIPIMAGE15_Book.indb 331 9/8/2015 7:47:24 PM

 Figure 2. PSO results over the multi-resolution scales from coarse to fine resolutions respectively. Initial low resolution
scales show robustness to large textureless regions whereas fine scales show the ability to reconstruct high frequency
information but also suffers from erroneously refined patches.

lower but not higher resolutions, where this region To improve the performance of the NCC, a photo-
becomes steadily more uniform as the resolution detail consistency measure is defined, weighting
increases. the NCC based on the patches texture and view-
Once the patch cloud is stably initialised at the ing angle. To increase the point clouds density
lowest level the image resolution can be raised a PSO patch refinement scheme is run per pixel
and the results propagated upwards. PSO is run allowing for only approximate initialisation and
at the higher resolution using the previous resolu- removing the need for feature based matching
tions optimised result as a velocity factor added to and expansion. Finally a multi-resolution scheme
Equation 5. This allows for reconstruction of sur- using the wavelet transform is proposed, propagat-
face details not expressed in the lower resolution ing refined patch results for textureless surfaces up
images. Further to this we can decide not to run through the resolutions, where consistency meas-
PSO refinement for a higher resolution pixel if the ures start to fail. The experimental results show
detail coefficients of the wavelet transform reflect a the improved patch clouds and the potential for
non-textured patch. In these cases, rather than run more accurate MVS reconstruction in textureless
PSO, we smooth the patch parameters to obtain a regions through propagation of data from lower
consistent patch cloud. Figure 2 shows promise that resolutions.
with correct propagation the refinement of patches
in textureless regions can be improved significantly.
REFERENCES
5 CONCLUSIONS Bratton, D. & J. Kennedy (2007). Defining a standard for
particle swarm optimization. In Swarm Intelligence
This paper has presented on-going work towards Symposium, 2007. SIS 2007. IEEE, pp. 120–127.
a multi-resolution patch based approach to MVS. IEEE.

332

VIPIMAGE15_Book.indb 332 9/8/2015 7:47:24 PM

 Chen, Z., W.-C. Chen, & P.-Y. Sung (2013). A novel Klowsky, R., A. Kuijper, & M. Goesele (2012). Modulation
3d dense reconstruction with high accuracy and transfer function of patch-based stereo systems. In
completeness. In Multimedia and Expo Workshops Computer Vision and Pattern Recognition (CVPR),
(ICMEW), 2013 IEEE International Conference on, 2012 IEEE Conference on, pp. 1386–1393. IEEE.
pp. 1–4. IEEE. Pongchairerks, P. & V. Kachitvichyanukul (2005). Non-
Cremers, D. & K. Kolev (2011). Multiview stereo and homogenous particle swarm optimization with multi-
silhouette consistency via convex functionals over ple social structures.
convex domains. Pattern Analysis and Machine Intel- Seitz, S.M., B. Curless, J. Diebel, D. Scharstein, &
ligence, IEEE Transactions on 33(6), 1161–1174. R. Szeliski (2006a). A comparison and evaluation
Esteban, C.H. & F. Schmitt (2004). Silhouette and stereo of multi-view stereo reconstruction algorithms.
fusion for 3d object modeling. Computer Vision and In Computer vision and pattern recognition, 2006
Image Under-standing 96(3), 367–392. IEEE Computer Society Conference on, Volume 1,
Furukawa, Y. & J. Ponce (2010a). Accurate, dense, and pp. 519–528. IEEE.
robust multiview stereopsis. Pattern Analysis and Seitz, S.M., B. Curless, J. Diebel, D. Scharstein, &
Machine Intelligence, IEEE Transactions on 32(8), R. Szeliski (2006b). Multi-view stereo evaluation web
1362–1376. page, http://vision.middlebury.edu/mview/data.
Furukawa, Y. & J. Ponce (2010b). Patch based multi-view Sweldens, W. & P. Schröder (2000). Building your own
stereo. wavelets at home. In Wavelets in the Geosciences,
Habbecke, M. & L. Kobbelt (2007). A surface-growing pp. 72–107. Springer.
approach to multi-view stereo reconstruction. In Com- Vogiatzis, G., C. Hernández, P.H. Torr, & R. Cipolla
puter Vision and Pattern Recognition, 2007. CVPR’07. (2007). Multiview stereo via volumetric graph-cuts
IEEE Conference on, pp. 1–8. IEEE. and occlusion robust photo-consistency. Pattern Anal-
Hiep, V.H., R. Keriven, P. Labatut, & J.-P. Pons (2009). ysis and Machine Intelligence, IEEE Transactions on
Towards high-resolution large-scale multi-view stereo. In 29(12), 2241–2246.
Computer Vision and Pattern Recognition, 2009. CVPR Wang, H., Y. Liu, & S. Zeng (2007). A hybrid particle
2009. IEEE Conference on, pp. 1430–1437. IEEE. swarm algorithm with cauchy mutation. In Swarm
Jensen, R., A. Dahl, G. Vogiatzis, E. Tola, & H. Aanæs Intelligence Symposium, 2007. SIS 2007. IEEE, pp.
(2014). Large scale multi-view stereopsis evaluation. 356–360. IEEE.
In Computer Vision and Pattern Recognition (CVPR), Zhou, Y. & Y. Tan (2009). Gpu-based parallel particle
2014 IEEE Conference on, pp. 406–413. IEEE. swarm optimization. In Evolutionary Computation,
Kennedy, J. (2010). Particle swarm optimization. In Ency- 2009. CEC’09. IEEE Congress on, pp. 1493–1500.
clopedia of Machine Learning, pp. 760–766. Springer. IEEE.

333

VIPIMAGE15_Book.indb 333 9/8/2015 7:47:28 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Forming tool inspection using fiber-optic sensor heads

S. Matthias, M. Kästner & E. Reithmeier

Insitute of Measurement and Control, Leibniz Universität Hannover, Hannover, Germany

P. Sieczkarek & A.E. Tekkaya

Institut für Umformtechnik und Leichtbau, Technische Universität Dortmund, Dortmund, Germany

ABSTRACT: In-situ measurements of tools allow inspecting a running metal forming process. This
enables a fast adaption of the process and helps reducing costs. However, the requirements on the
measurement systems are high, as the measurement duration needs to be low and the available space
inside the machines is limited. Commercially available measurement systems are unable to operate in
these conditions. An application in metal forming is presented followed by the introduction of a novel
design for an optical measurement system based on the fringe projection profilometry technique. The new
design employs flexible fiber bundles to transport the images of the fringe projection system to a compact
sensor head which is based around gradient-index lenses. The pattern generation unit and camera are
housed in a base unit, which can be placed outside of the machine. While this new approach allows
flexible positioning of the sensor head, it creates challenges for the pattern evaluation algorithms and the
optical design of the system. The laser-based optical setup will be presented followed by measurements
of the tool of the aforementioned forming process. To evaluate the applicability of the new measurement
system, the determined tool parameters will be compared to reference measurements obtained by a tactile
Hommel T8000 system. In combination with a positioning device, which is currently being developed, the
new measurement system will be used to obtain wear information of the running forming process in order
to control the process quality loop in the future.

1 INTRODUCTION acquire geometry data. Different approaches are

currently being researched, which use passive ster-
A new metal forming process is being researched eometry, structured light, confocal techniques or
within the CRC/TR 73 collaborative research center the time-of-flight principle (Geng & Xie 2014).
funded by the German Research Foundation (DFG) This paper describes an exemplary sheet-bulk
(Merklein, Allwood, Behrens, Brosius, Hagenah, metal forming process based on a multi-axis press,
Kuzman, Mori, Tekkaya, & Weckenmann 2012). followed by the introduction of a new endoscopic
The aim of the sheet-bulk metal forming process measurement system. Based on reconstruction by
is to reduce the required number of forming steps triangulation and structured light, the new fringe
to manufacture complex geometries. Additionally, projection system is able to obtain areal measure-
the production of more sophisticated parts with ments of a specimen’s geometry in the form of a
reduced weight or improved stability is made pos- 3D point cloud. Based on this data, further analy-
sible. The scope of research covers the evaluation sis, such as comparisons against CAD reference
of different materials, surface treatments and tool data, can be carried out. The new system features
geometries. For the production of gearings or a laser and an adjustable pattern generator for
similar elements, high forces are necessary in order fast measurements, as well as exchangeable sensor
to achieve proper molding. To be able to monitor heads to be able to adapt to different measurement
wear on the forming elements in the limited space tasks. Based on the description of the forming tool
of the forming presses, measurement devices are for the presented process, measurements of tool
needed which can be positioned in the limited features are shown. Results from the new system
space of the forming machines. Many commercially are compared to reference data obtained with a
available tactile and optical measurement devices tactile profile sensor. Potential benefits of obtain-
are unable to operate under these conditions. In ing real-time information on tool wear with the
order to overcome these limitations, a combination new system for the evaluation of important process
of endoscopy techniques with contact-free optical parameters, such as lubrication or forming trajec-
measurement principles can be used in order to tories, are discussed.

335

VIPIMAGE15_Book.indb 335 9/8/2015 7:47:28 PM

 2 FORMING PROCESS the current research focusses on a reduction of the
high tool wear.
The development of flexible forming technologies With the objective of a near-net-shape produc-
aims to manufacture load-adapted and weight- tion, a forming process is dependent on reliable,
optimized components by a high integration of dimensionally accurate tools. The desired process-
functional elements. A suitable approach to reach ing capabilities are targeting a value of 10 μm
this goal is the incremental sheet-bulk metal forming (IT7). The knowledge of the process-related high
process. In addition to reducing the forming forces tool wear suggests that in-situ measurements of the
by decreasing the surface contact between tool and tool geometry are desirable for a possible industrial
workpiece, the process offers high flexibility. The application in the future. Particularly in the context
incremental procedure allows the manufacturing of of incremental manufacturing, which features a
near-net-shape parts with small functional elements, high flexibility for the prototype segment or small
such as gearings. However, high positioning accura- batch production, much higher cycle times occur.
cies are required under load. With a specially devel- These provide enough time for a tool measurement
oped multi-axis press, the application of different during the running process. Besides the control
processes, such as embossing or rolling, is made pos- of the tool geometry, also a complete inspection
sible with almost arbitrary trajectories (Sieczkarek, of the process would be conceivable. The aspired
Kwiatkowski, Ben Khalifa, & Tekkaya 2013). geometrical accuracy of the produced components
Shaping the gearings and tooth structures by is affected by disturbances, such as strain, lubri-
a bulk metal forming process offers significant cation, number of forming stages, elastic spring-
advantages over conventional machining. A con- back of the setup and different material properties.
tinuous fiber orientation in the formed teeth leads With in-situ measurements, a target-performance
to improved mechanical properties. In addition to comparison of wear effects could be conducted. In
the positively usable effect of strain hardening, no the future, direct adjustment of the targeted com-
pores are exposed in the material, which are often mand variables of the quality control loop can be
the origin of cracks. achieved with this additional information.
The incremental forming of tooth structures
on the sheet edge (Fig. 1) requires thin forming
tools, which need to be of the same or only slightly 3 MEASUREMENT SYSTEM
higher thickness compared to the sheet-metal
parts. This is related to the necessary chambering In order to be able to perform measurements inside
of the workpiece to prevent a lateral material flow the limited space of a forming press, very compact
and to achieve the best possible shape of the tooth dimensions of the sensor head are required. The
structure. The high local contact pressure leads to newly developed endoscopic fringe projection
a reduced tool life as well as to a tool deformation achieves a very compact design by separating the
(Sieczkarek, Kwiatkowski, Tekkaya, Krebs, Kerst- sensor head from the pattern generation and cam-
ing, Tillmann, & Herper 2013). In order to achieve era unit with fiber bundles. These imaging fibers
higher accuracies as well as stable forming tools, are supplied by Fujikura and feature a length of
1000 mm at a diameter of 1.7 mm and a resolution
of 100,000 pixels. A minimum bending radius of
200 mm allows independent positioning of the sen-
sor head and the base station.
The base station is designed with overlapping
sides to safely enclose the laser illumination for
the pattern generator. A single-mode laser with a
wavelength of 532 nm is used for its good proper-
ties for fiber incoupling, as well as high light inten-
sities with an optical output power of 40 mW.
Challenges arise from interference patterns, the so
called speckle, when measuring technical surfaces
with a laser based projector. The contrast of these
interference patterns can easily exceed the contrast
of the fringe patterns, leading to high noise in the
measurements. In order to reduce the effect of
speckle, a former design of the projector beam path
employed a rotating diffuse or plate, leading to tem-
porally changing speckle patterns. By integrating
Figure 1. Schematic of the incremental forming process. over the camera’s exposure duration, speckle can

336

VIPIMAGE15_Book.indb 336 9/8/2015 7:47:28 PM

 be mostly compensated. However, the motor for In order to perform metric measurements,
the rotating motion initiates vibrations. Therefore, the system is calibrated with a planar calibration
the rotating diffusor has been displaced by a laser pattern according to a mixed pinhole and black-
speckle reducer from Optotune, which performs box model (Matthias, Kästner, & Reithmeier 2014).
the movement of the diffusor plate with electro- Measurements with 12 phase-shift patterns for the
active polymers, resulting in a significant reduction reconstruction of an absolute phasemap currently
of undesired vibrations (Blum, Büeler, Grätzel, require a projection time of less than 3 seconds.
Giger, & Aschwanden 2012). Another factor to be Using the inverse fringe projection principle,
considered for the projector is the uniformity of geometry deviations from a reference object can
brightness of the fringe patterns. The single-mode be detected with a single image (Pösch, Vynnyk, &
laser exhibits a typical Gauss-beam profile. For the Reithmeier 2013). This reduces the needed time for
purpose of generating a homogenous profile, two tool inspection to less than one second.
micro-lens arrays supplied by SUSS MicroOptics
are used (Voelkel & Weible 2008). A mirror reflects
the laser beam onto the active pixel matrix of a Dig- 4 TOOL MEASUREMENTS
ital Micro-mirror Device (DMD) (Hornbeck 1996).
Due to the switching angle of ±12 of the individual Figure 3 shows a drawing of the forming tool with
active mirrors, accurate adjustment of the beam the features to be captured. To obtain reference
path is necessary in order to focus the generated surface data, the forming tools are first meas-
fringe patterns into the optical fiber bundles. ured using a Hommel T8000 tactile measurement
The detector part of the base unit consists of a device. By moving a ruby ball over the specimen’s
monochrome Point Grey CCD-camera with a sensi- surface, a very accurate 2D-profile is acquired.
tivity maximum at about 530 nm to complement the The feature parameters are derived from the sur-
laser light source. Precise synchronisation of camera face profiles by fitting a line to the data points of
and projector is achieved by triggering image capture the tooth sides, base and top. In comparison, the
with the hardware output of the DMD-controller. endoscopic fringe projection system captures 3D
At the far end of the projector and camera fiber surface data using absolute phase measurements.
bundle, gradient-index lenses are directly attached As 3D point data is acquired, planes are fitted to
to the planar fiber surface. The imaging rod lenses the respective tool surfaces to derive the angles as
are manufactured by GRINTECH and are 2 mm well as the tooth height. To improve the optical
in diameter and less than 6 mm in length. Depend- characteristics of the tool’s surface, the tool is cov-
ing on the desired measurement volume and work- ered with a thin layer of diffuse spray for the fringe
ing distance, lenses with a focal length of either projection measurements.
10 mm or 20 mm are used. For the 10 mm optics a Table 1 lists the results measured for two dif-
measurement volume of about 3 × 3 × 3 mm2 can ferent forming tools and the difference of the
be achieved. A view of the tool through the endo- results obtained with the fringe projection system
scope with a 20 mm lens can be seen in Figure 2. compared with the tactile reference results. Only

Figure 2. Photo of the tool captured through the

endoscope. Figure 3. Drawing of the forming tool features.

337

VIPIMAGE15_Book.indb 337 9/8/2015 7:47:28 PM

 Table 1. Comparison of feature measurements of the forming tool.

Fringe projection Tactile Hommel Difference optical/

Reference system T8000 tactile

Feature Tool 1 Tool 2 Tool 1 Tool 2 Tool 1 Tool 2 Tool 1 Tool 2

Height left 3.38 mm 3.38 mm 3.42 mm 3.36 mm 3.42 mm 3.33 mm 0.00 mm 0.03 mm
Height right 3.38 mm 3.38 mm 3.40 mm 3.38 mm 3.45 mm 3.38 mm −0.05 mm 0.00 mm
Angle left 110.0° 110.0° 110.5° 109.6° 110.1° 109.9° 0.4° −0.3°
Angle right 110.0° 110.0° 110.6° 110.8° 109.9° 109.8° 0.7° 1.0°

is possible to detect and assess the extend of wear

on the tool to decide a tool change in the running
process based on these measurement, once charac-
teristic deformations are categorized.

5 CONCLUSION AND OUTLOOK

An application for in-situ measurements in metal

forming has been introduced. To overcome the high
requirements in terms of measurement time and
space restrictions, the design of a new measurement
system based on fringe projection profilometry and
endoscopy techniques has been introduced. The
capabilities of the device have been demonstrated
Figure 4. Deviation analysis of a damaged forming by tool measurements of the presented forming
tool. processes. It could be shown that the system meets
the accuracy requirements by comparisons to refer-
ence data obtained by tactile measurements.
small disparities between the two devices are vis- To enable operation of the fiber-based fringe
ible, with a maximum difference of 50 μm for the projection system in the forming press, a position-
tooth height and a maximum deviation of 1.0 for ing system is being devised. This will enable captur-
the tooth angles. Apart from measurement uncer- ing real-time measurement data from the running
tainties, the differences may arise from the vary- process. In order to further assess the applicability
ing measurement principles, with the tactile device of the endoscopic measurement system, compari-
capturing 2D-profile data compared to 3D-surface sons with a geometry standard will be performed.
data obtained by the fringe projection system. For Analysis of deformations of characteristic tool
example deformations on the tooth tip plane or features will help to decide tool changes if certain
a tilt to the sides can not be registered with a 2D thresholds are surpassed.
profile. Additionally, some small differences might
arise from the diffuse spray used to cover the tool
surface to improve the optical measurements. ACKNOWLEDGEMENT
Therefore, the performance of the system will be
assessed by measurements of a geometry standard The authors would like to thank the German
with precise tolerances in the future. Research Foundation (DFG) for funding the
Figure 4 shows a deviation analysis of a meas- projects B6 “Endoscopic geometry inspection”
urement of a damaged tool compared to the CAD and A4 “Incremental Forming” within the Col-
reference data with the evaluation software GOM laborative Research Center (CRC)/TR 73.
Inspect. The color encodes the distance to the next
reference surface. The number of data points in this
REFERENCES
measurement is 18657. It can be seen that the tip of
the gearing elements shows large deformations. The Blum, M., M. Büeler, C. Grätzel, J. Giger, &
height of the teeth is reduced by 0.6 mm, resulting M. Aschwanden (2012). Optotune focus tunable lenses
in a push of material to the sides. By obtaining 3D and laser speckle reduction based on electroactive pol-
information about the deformation of the tool, ymers. In SPIE MOEMS-MEMS, pp. 825207–825207.
analysis of material flow is made possible. Also it International Society for Optics and Photonics.

338

VIPIMAGE15_Book.indb 338 9/8/2015 7:47:29 PM

 Geng, J. & J. Xie (2014). Review of 3-d endoscopic sur- Pösch, A., T. Vynnyk, & E. Reithmeier (2013). Fast detec-
face imaging techniques. Sensors Journal, IEEE 14(4), tion of geometry defects on free form surfaces using
945–960. inverse fringe projection. Journal of the CMSC 8(1),
Hornbeck, L.J. (1996). Digital light processing and mems: 10–13.
An overview. In Advanced Applications of Lasers in Sieczkarek, P., L. Kwiatkowski, N. Ben Khalifa, & A.E.
Materials Processing/Broadband Optical Networks/ Tekkaya (2013). Novel five-axis forming press for the
Smart Pixels/Optical MEMs and Their Applications. incremental sheet-bulk metal forming. Key Engineering
IEEE/LEOS 1996 Summer Topical Meetings:, pp. Materials 554, 1478–1483.
7–8. IEEE. Sieczkarek, P., L. Kwiatkowski, A.E. Tekkaya, E. Krebs,
Matthias, S., M. Kästner, & E. Reithmeier (2014). Cam- P. Kersting, W. Tillmann, & J. Herper (2013). Innova-
era and projector calibration of an endoscopic fringe tive tools to improve incremental bulk forming proc-
projection system. In Proceedings of 11th IMEKO esses. Key Engineering Materials 554, 1490–1497.
Symposium LMPMI2014. Voelkel, R. & K.J. Weible (2008). Laser beam homogeniz-
Merklein, M., J. Allwood, B.-A. Behrens, A. Brosius, ing: limitations and constraints. In Optical Systems
H. Hagenah, K. Kuzman, K. Mori, A. Tekkaya, & Design, pp. 71020J–71020J. International Society for
A. Weckenmann (2012). Bulk forming of sheet metal. Optics and Photonics.
CIRP Annals-Manufacturing Technology 61(2),
725–745.

339

VIPIMAGE15_Book.indb 339 9/8/2015 7:47:30 PM

This page intentionally left blank
 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

Navigation of robotics platform using advanced image processing

navigation methods

L. Beran, P. Chmelar & L. Rejfek

Faculty of Electrical Engineering and Informatics, University of Pardubice, Pardubice, Czech Republic

ABSTRACT: This paper deals with methods suitable for movable robotics platform navigation. We can
divide these methods into suitability for indoor navigation, outdoor navigation or its combination. At
present, the navigation methods of robotics platforms, suitable for outdoor spaces, are based on GNSS
(Global Navigation Satellite System). Accurate position estimation is a big advantage for this type of
navigation. Unfortunately this type of navigation is not suitable for indoor spaces and industrial areas,
mainly because of the weak signal and low position accuracy. An alternative navigation system is neces-
sary for these solutions—system independent on satellite navigation. Our research is focused on movable
robotics platform navigation using advanced image processing methods. Descriptions of these sophisti-
cated methods are presented in this paper.

Keywords: monocular camera vision; image processing; visual odometry; robot navigation; surface
detection

1 INTRODUCTION of sensors—accelerometers and gyroscopes. Non-

stationary offset and a high noise level is typical for
At present, the systems for automatic space map- cheap Inertial Measurement Units (IMU). These
ping are widely distributed in many fields of robotic properties lead to inaccurate position estimation in
navigation. This automatic mapping system needs an integration process. For estimated path correc-
a precision navigation system. In the most cases, tion, it is necessary to use some correction system.
places are unknown or dangerous; like ruins or In the paper (Beran et al, 2014) a correction sys-
caves for autonomous space mapping. Navigation tem using a geometrical transformation based on
types for these platforms are different for indoor or Direct Cosine Matrix (DCM) is shown. Even a
outdoor spaces. Navigation based on Global Navi- small offset error leads to a big mistake in position
gation Satellite System (GNSS) with an example determination which grows quickly with time. This
such as GPS, GLONASS and in future Galileo, as problem was described in detail in paper (Beran et
precision navigation system are mainly used. The al, 2014). This type of navigation can be used for
benefit of using GNSS navigation systems is the navigation of a robotics platform.
achieved accuracy, typically better than 15 meters. A modern method of navigation called Visual
When the augmentation system is used, the accu- Odometry using advanced image processing can
racy is typically better than 3 meters in position be used for navigation of a mobile platform (Yang
determination. Such an augmentation system is et al, 2005), (Campbell et al, 2005). The visual
called European Geostationary Navigation Over- odometry method is known from the eighties, but
lay Service (EGNOS) and in the US Wide Area massive penetration of this method is connected
Augmentation Service (WAAS). When the GPS with development of fast Digital Signal Processors
signal phase measurement is used, the accuracy is (DSP). In our research, we are focused on the appli-
at the centimetre level (Mcnamara, 2004), (Jonas & cation of these modern advanced image processing
Mandlik, 2013). Satellite navigation is unsuitable methods. In our case, the Scale Invariant Feature
for our purpose, because the signal has a lot of Transform (SIFT) method (Lowe, D.G., 1999) and
drop outs in buildings, tunnels, or mines and we Speeded-Up Robust Features (Rodriguez & Aouf,
are focused on indoor navigation. 2010), (Bay et al, 2008), (Nemra et al, 2013), can be
It is possible to use a navigation system based used. Both modern methods are based on extrac-
on Inertial Navigation System (INS) navigation tion of pairs of interesting points from image
in an indoor space. This type of navigation sys- and template. It is possible to calculate the pixel
tem is based on the fusion of two different types distances from differences between these pairs of

341

VIPIMAGE15_Book.indb 341 9/8/2015 7:47:30 PM

 interesting points. The SURF method is optimized pixels of input image I(i, j). This sum is defined as
and faster than the SIFT method. We used a result a square between point x and its origin. This inte-
from the paper (Suaib et al, 2014) and we focus on gral image can be calculated using (2).
the SURF method in our research.
I Σ ( x ) = ∑ i = 0 ∑ j = 0 I (i j )
i x j y
(2)
2 ADVANCED IMAGE
PROCESSING METHOD Interesting points in the image are extracted by
using a Hessian Matrix. This matrix is created like
2.1 Speed Up Robust Features (SURF) a second-order partial derivation of function. This
The SURF method is based on extraction of inter- matrix is calculated using equation (3).
esting points from an image. This method is also used
for description of these points. The big advantage in ⎡ Lxx ( x, y ) Lxy ( x yy,σ ) ⎤
comparison with the SIFT method is speed (Suaib H ( x, y,σ ) = ⎢ ⎥ (3)
et al. 2014). Higher speed processing is achieved by ⎢⎣ Lxy ( x, y ) Lyy ( x yy,σ ) ⎥⎦
using the integral convolution method and approxi-
mation of the Gaussian second order derivative. where x is coordinates of pixel; σ is pixel scale;
This approximation allows speed-up. The equation Lxx(x, y, σ) is convolution of the second-order
of the Gaussian function is shown in (1). derivation of the Gaussian function with original
image I in pixel x
(x +y )
G ( x, y σ )
1
e
−
2σ 2 (1) Det ⎡⎣ H ( x yy,σ ) ⎤⎦ Lxx ( x, y ) Lyy ( x yy,σ )
2πσ
σ 2
− Lxy ( x, y,σ )
2
(4)
where x and y are pixel coordinates in the image, σ
is a parameter of Gaussian function. where Det[H(x, y, σ)] is a determinant of the Hes-
In Figure 1 a discrete and approximate sian matrix.
second-order derivation of the Gaussian function The descriptors must be independent in rotation.
is shown. The second-order partial derivation of This rule is provided by setting the orientation in all
Gaussian function derivate in the y-direction is important points. The decision about orientation is
shown on left side; the Gaussian function derivate provided by using a sum of Haar Wavelet responses.
in the xy-direction is shown on right side. The response is written into a histogram with six bin
Every interesting point in integral image I∑(x) length (1 bin = 60°). The orientation is chosen by the
at position x = (x, y) is defined as a sum the all maximal value in this histogram. It is possible to write
the descriptor of each interesting point using (5).

z (∑dx
d ∑dy
d ∑ dx d )
d ∑ dy (5)

where z is the important point; dx and dy are

sub-regions.

3 EXPERIMENTAL RESULT

In our research we construct two workplaces. The

first workplace has a static camera perpendicular
to the ground. This camera has a static scale and
we are moving the background. The second work-
place was constructed as a user-controlled movable
platform. On both platforms a SURF algorithm
was implemented.

3.1 Static platform

The static platform consists of a stationary
Figure 1. Discrete (up) and approximation (down) of construction with fixed camera and a movable
second order derivation of Gaussian Function (Bay et al, background. During our testing we mounted a
2008). camera on the print head of 3D printer. The 3D

342

VIPIMAGE15_Book.indb 342 9/8/2015 7:47:30 PM

 printer has high precision of position estima-
tion. The maximal error of position estimation is
0.25 mm. This value is sufficient for our purpose.
The SURF algorithm is used for template track-
ing. The template is created from the first captured
image by cutting out the centre of image. This
template is tracked in the field of view. The trace
of tracking and results are shown in Figure 2.
The black colour is used for a tracked trace and
the white square with white dot in the centre, is the
cropped template from the first image.
The more information about static platform and
results of measurement is presented in (Beran et al, Figure 3. Radio-controlled platform for testing visual
2015). odometry.

3.2 Movable platform

In our research we construct a mobile robot-
ics platform. At present, the robotics platform
is radio-controlled by a user, but in future, we
are planning autonomous control of this platform.
The movable platform is based on a Raspberry
PI. The parameters of this platform are writ-
ten in Table 1. At present, the Raspberry is on a
vehicle designed only for capturing a video. The
camera is recording a floor with defined angle.
Captured images are stored into an SD-Card for
post-processing. Post-processing is provided by
Matlab Software. The template is continuously Figure 4. Demonstration of detection left curve using
cropped from the center of the previous image. visual odometry.

In Figure 3, our radio-controlled robotic plat-

form is shown. The algorithm sequentially detects
a position. The template is cropped from the cen-
tre of the previous image. The algorithm calcu-
lates interesting points in the template and also in
the image, where many pairs of these interesting
points are found. The pixel position is calculated,
from distance between centre of previous image
and searched centre, by algorithm.
In Figure 4, the detected trace using the SURF
algorithm is shown. The white square with white
point in the centre is detected position of the
template in the next image. The black square with
Figure 2. Tracked template using SURF algorithm.
black point is the original position of the template.
The pixel distance is calculated as a difference
between these points.
Table 1. Parameters of movable platform.

Type Parameters 4 MAPPING CAMERA COORDINATES

Raspberry Broadcom BCM2836 ARM Cortex A7, The geometrical transformation is a transforma-
PI 2 4 × 900 MHz, 1 GB LPDDR2 SDRAM tion from the camera coordinates to real coor-
Camera OV05647-G04A, Pixel size 1.4 × 1.4 μm, dinates. This transformation is based on simply
Frame rate: 1080p (30 fps), 960p (45 fps),
mathematics equations. First, it is necessary to
720p (60 fps), 640 × 480 (90 fps),
320 × 240 (120 fps), lens chiev array (24°). measure a distance between ground and camera.
Also it is necessary to know a camera’s vertical

343

VIPIMAGE15_Book.indb 343 9/8/2015 7:47:31 PM

 field of view. This vertical field is calculated from where width and height is resolution of camera
trigonometric equation.
In Figure 5, our camera physical settings are V 1
depicted. At the beginning it is possible measure f = ⋅ (9)
2 tan(VFOF /2 )
these variables: H, α and L.
When a height H of camera was measured, we
can measure camera view angle α
tan ( β ) =
( − )⋅ ⋅ tan (VFOV / )
(10)
⎛H⎞ V
α = tan −1 ⎜ ⎟ (6)
⎝ L⎠
When focal length is not known, angle β can be
where α is an angle between the principal ray and established using VFOV. These relationships are
ground; H is a distance between the camera and written in equations (7), (9) and (10).
ground; L is a distance between camera and princi-
pal ray of camera mapped to the ground. δ =α+β (11)
Also it is possible calculate an angle β using H
angle VFOF (Vertical Field of View). The rela- y= (12)
tan δ
tionship between focal length f and field of view is
shown in Figure 6.
H cos ( )
Pv − (V /2 ) z= (13)
tan ( β ) = (7) i ( )
sin
f
where y is coordinates of point P; z is depth of
where β is an angle between captured point P at observed point.
CCD and the principal ray; Pv is vertical position
of point P at CCD; V is vertical resolution of chip
(in our case 240 rows); f is focal length.
When the Horizontal Field of View (HFOV) is 5 RESULTS
not known, this angle can be calculated using (8).
In our research we are testing two platforms. The
−1 ⎛
⎛ VFOV ⎞ Width ⎞ first platform is statically constructed and the
HFOV = 2 ⋅ tan ⎜ tan ⎜ ⋅ (8)
⎝ ⎝ 2 ⎟⎠ Height ⎟⎠ results from this platform are shown in (Beran
et al, 2015). The second platform is movable. In
this platform the scene is captured by the camera
with the defined angle. All parameters of configu-
ration of our system are written in Table 2.
In Table 3 and Table 4, the sample of meas-
urements are written. All measurements have
fifteen repetitions of the same path. For better
readability, the first five results of these measure-
ments are written. In this research a five different
path lengths were tested and results are written in
Table 5.
In Figure 7, the corrected estimated path using
Figure 5. Mapping a camera coordinates to ground. geometrical transformation is shown. In the image,
the black square is a template and the white square

Table 2. Configuration of movable platform.

Parameter Value

VFOV 60°
HFOV 35, 98°
H 17.5 cm
L 44 cm
α 21.68°
Figure 6. Relationship between angle β and focal Width × height 320 × 240 pix
length f.

344

CH54-IA9441.indd 344 9/9/2015 7:34:00 PM

 Table 3. Results of estimated path—length 3 m. SURF was presented. Also an implementation of
this method on static and movable robotics plat-
Direct drive—path length 3 m forms is described. The first platform works with
a static camera and movable background. The
Measure Std.
nr. 1 2 3 4 5 Mean dev. second platform is user-controlled platform. We
are planning to construct an autonomous robotics
X [m] 0,05 −0,07 0,05 0,02 0,08 0,0328 0,052 platform in the future. Our platform is based on a
Y [m] 3,05 2,97 3,05 3,09 3,11 3,05 0,114 camera and Raspberry PI. The scene is captured
into an internal memory, and all calculations are
evaluated as post-processing in Matlab software.
Table 4. Results of estimated path—length 6 m. The algorithm searches interesting points of the
template in an image. For this purpose it is nec-
Direct drive—path length 6 m essary to have a sharp image using a camera with
fast shutter. The pixel distance is calculated from
Measure Std. pairs of these interesting points. Every fifth to
nr. 1 2 3 4 5 Mean dev. fifteenth image is selected in Matlab to calculate
interesting points. The frame step is selected by
X [m] 0,02 −0,05 −0,03 0,1 0,04 0,026 0,099 the real speed of a movable platform, at present,
Y [m] 6,05 5,96 6,15 6,13 6,09 6,07 0,157 manually. In this case we are using a high camera
framerate (120 fps) and image decimation which
will decrease time consumption. The algorithm
Table 5. Statistical results of estimated path. also provides geometrical transformations which,
after recalculation, transform this pixel distance
1.5 m 3m 5m to real distance in centimeters. From the results,
Path
length X Y X Y X Y
it is possible to recognize a higher estimated value
than real distance. In following research this higher
Mean 0,052 1,52 0,0328 3,05 0,13 5,1 value must be corrected. At this time, the algorithm
Std.dev. 0,0056 0,09 0,052 0,114 0,088 0,14 calculates with direct drive only. But in future the
some correction system will be implemented in all
types of paths. This correction will be similar to the
Direct Cosine Matrix transformation as described
in paper (Beran et al, 2014). In this paper geometri-
cal transformation for direct paths was described.
In the near future, a correction system to get a cor-
rected estimated path will be implemented. For
this purpose is necessary to use a fusion of visual
odometry with high precision gyroscopes. This
system, based on visual odometry, will be imple-
mented as a fusion of Inertial Measurement Unit
(IMU) to Autonomous Research Exploration Sys-
tem (ARES). Also, it will cooperate with a system
called Optical Measurement Device (Chmelar &
Dobrovolny, 2013), (Chmelar et al, 2014).

Figure 7. Demonstration of estimated corrected path.

ACKNOWLEDGMENT

is a detected previous position of the template. At The research was supported by the Internal Grant
the top of this image, estimated differences between Agency of University of Pardubice, the project
black and white centres are shown in centimetres. No. SGSFEI 2015003.
At first, this distance is calculated in pixel distance
and later, it is transformed to real coordinates.
REFERENCES
6 CONCLUSION Bay, H. et al, 2008, Speeded-Up Robust Features
(SURF), In: LEONARDIS, Computer Vision and
In this paper a modern navigation method based Image Understanding: 9th European Conference on
on an advanced image processing method called Computer Vision, 2008, s. 346–359.

345

VIPIMAGE15_Book.indb 345 9/8/2015 7:47:35 PM

 Beran, L. & Chmelar, P. & Dobrovolny, M., 2014, Navi- Lowe, D.G., 1999, Object recognition from local
gation of robotic platform with using inertial meas- scale-invariant features, Computer Vision, 1999. The
urement unit and Direct Cosine Matrix, Proceedings Proceedings of the Seventh IEEE International Con-
Elmar—International Symposium Electronics in ference, vol.2, no., pp. 1150, 1157 vol. 2.
Marine 2014, pp. 87–90. Mcnamara, J., 2004, GPS for dummies. Hoboken, NJ:
Beran, L. & Chmelar, P. & Rejfek L., 2015, Navigation Wiley, p. 57.
of Robotics Platform using Monocular Visual Odom- Nemra, A. et al, 2013, Adaptive Iterative Closest SURF
etry, Radioelektronika (RADIOELEKTRONIKA), for visual scan matching, application to Visual odom-
2015 25th International Conference. etry,” Systems and Control (ICSC), 2013 3rd Interna-
Campbell, J. et al, 2005, A Robust Visual Odometry tional Conference, pp. 927.
and Precipice Detection System Using Consumer- Rodriguez, D. & Aouf, N., 2010, Robust Harris-
grade Monocular Vision, Robotics and Automation, SURF features for robotic vision based navigation,”
2005. ICRA 2005. Proceedings of the 2005, vol., no., Intelligent Transportation Systems (ITSC), 2010 13th
pp. 3421, 3427. International IEEE Conference, pp. 1160.
Chmelar, P. & Dobrovolny, M., 2013 The Fusion Suaib, N.M. et al, 2014, Performance evaluation of
of Ultrasonic and Optical Measurement Devices feature detection and feature matching for stereo
for Autonomous Mapping, Radioelektronika visual odometry using SIFT and SURF,” Region 10
(RADIOELEKTRONIKA), 2013 23rd International Symposium, 2014 IEEE, vol., no., pp. 200, 203.
Conference, pp. 292,296. Yang, C.H. & Maimone, M. & Matthies, L., 2005 “Visual
Chmelar, P. & Beran, L., & Kudriavtseva, N., 2014, odometry on the Mars Exploration Rovers,” Systems,
Projection of point cloud for basic object detection, Man and Cybernetics, vol. 1, no., pp. 903.
ELMAR (ELMAR), 2014 56th International
Symposium
Jonas, M. & Mandlik, M., 2013 “Influence of the GNSS
satellite elevation to the position error,” Radioele-
ktronika 2013 23rd International Conference, pp. 119,
124.

346

VIPIMAGE15_Book.indb 346 9/8/2015 7:47:36 PM

 Computational Vision and Medical Image Processing V – João Tavares & Natal Jorge (Eds)
© 2016 Taylor & Francis Group, London, ISBN 978-1-138-02926-2

SNIP: Smile—neutral facial display intensity predictor

K. Nurzynska
Institute of Informatics, Silesian University of Technology, Poland

B. Smolka
Institute of Automatic Control, Silesian University of Technology, Poland

ABSTRACT: Many affective disorders can influence the capacity of emotion expression and
understanding. For instance, in the case of depression, the frequency and intensity of smiling is diminish-
ing when the disease proceeds. Analysing these changes by a psychologist could bring information about
the effectiveness of the prescribed medical treatment. Therefore, this work describes a system which auto-
matically analyses video recordings from sessions with patients and determines the length and intensity
of smile.
The proposed system exploits a standard database to train the classifier, which is later used to dis-
tinguish between the smiling and neutral facial expression depicted in the video frames. The obtained
information is presented graphically, indicating the class membership and temporal smile intensity. The
performed experiments revealed that the classification accuracy exceeds 85%. Moreover, the case studies
show, that the variations in smile magnitude enable better insight into the smile analysis.

1 INTRODUCTION popular in the medicine and psychology. Hence,

no matter how useful the findings are, it is impos-
Patients suffering from some mood disorders sible to follow the proposed methodology in every-
have problems to correctly recognize or express day use. Therefore, a system, which automatically
emotions. When schizophrenia is considered, the annotates the frames whether they refer to smiling
accurate understanding of others emotions is trou- or neutral face with additional information about
blesome (Kohler et al. 2008, Souto et al. 2013). The smile intensity is of high practical importance.
ability of autistic persons to recognize the genuine The paper addresses the problem of automatic
and posed smile is impaired, because they usually classification between smiling and neutral facial
tend to omit the eye contact (Boraston et al. 2008). displays with additional estimation of the smile
Next, the research presented in (Scherer et al. intensity. The SNIP system uses pre-trained linear
2013, Girard et al. 2014) show that in the case of Support Vector Machine (SVM) to decide to which
depression, the changes in smile intensity and its class each frame from the video can be assigned,
frequency correlate with the patient’s mental state
and can be used as a measure of effectiveness of
the therapy. For comparison, the measurements
of cardiovascular changes, e.g. cardiac interbeat
interval, finger pulse amplitude, etc., bring no
effect (Tsai et al. 2003).
Recording sessions with patients to analyse
the behavioural changes were commonly used in
research described by (Reed et al. 2007, Girard
et al. 2014, Hurley and Frank 2011). The gath-
ered material proved to be very useful in diagnosis
process and mental state evaluation of the patient.
The biggest problem is the necessity to annotate
each frame in the video, which is a tiresome effort.
Moreover, this work should be performed by the
trained experts with expertise in the Facial Action
Coding System (FACS), which was developed
by (Ekman and Friesen 1978) and became very Figure 1. Main window of the SNIP system.

347

VIPIMAGE15_Book.indb 347 9/8/2015 7:47:36 PM

 then basing on the response from classifier the which summarises an image content in a form of
smile intensity is determined. The main software a histogram. A histogram bin contains the number
window graphically presents all the relevant infor- of similar texture patches in the image, whereas
mation in Figure 1. the texture characteristics itself are used to define
The paper is structured as follows. Section 2 dis- which bin it belongs to.
cussed previous work concerning the recognition In order to calculate the descriptive value,
between smiling and neutral facial displays. Next in a monochromatic image, denoted as I, is consid-
Section 3, the Local Binary Patterns (LBP) is pre- ered. For each pixel with intensity gc I ( x y ) a
sented as a tool to describe the facial expression. circular neighbourhood is sampled in P points on
The system work flow is given in Section 4 and a radius R following the equation:
results of the performed experiments are presented
in Section 5. Finally, the conclusions are drawn in xp = x + R (2 p/P
/P)
P ) y p = y R sin(
i ( p P ), (1)
Section 6.
where p 0 P − 1, resulting in an ordered
sequence of grey-scale values gp. More information
2 STATE OF THE ART may be derived from the differences between the
sampled points gp and the central value gc, however
The emotional conditions depicted in human face that would result in a big range of necessary codes
are changing dynamically, therefore the first attempt to describe the textural information. Therefore,
to the smiling and neutral facial display recogni- only the sign is exploited in the definition of the
tion were based on the facial muscles movements generic LBP:
recorded in the image sequence and described by
the optical flow (Mase 1990, Bartlett et al. 1999, P −1
Essa and Pentland 1997). Many approaches use PP ,R ( x y ) =
LBP ∑ s( g p − gc ) ⋅ 2 p , (2)
texture operators for frame description in order to p=0
discriminate between basic emotions (for instance
sadness, anger, fear, happiness, etc.) (Cohn et al. where
1999, Cohn et al. 1998, Pantic and Rothkrantz
2000, Shan et al. 2009). But there are also works ⎧1, z ≥ 0,
which exploit extreme learning machine as a s( z ) = ⎨ (3)
classifier (An et al. 2014) or Gabor filters and ⎩0, z < 0.
GentleSVM (Wu et al. 2011).
This operator proved to be very descriptive and
Different approaches do not concentrate on the
found many applications in image understanding
whole facial display, but use only some part of it.
domain (Brahnam et al. 2014), yet in some cases
The active shape model was applied to follow the
the large length of the generated histogram is
movement of the lips in the image sequence (Lanitis
a severe drawback of this method (for P = 8 the
et al. 1997). Similarly, the research of (Freire et al.
LBP histogram length is 256) as it influences the
2009, Freire et al. 2009) concentrates on the lips
computation costs remarkably. Moreover, it was
describing the changes by texture operators, while
noticed (Pietikäinen et al. 2011) that some pat-
(Dibeklioğlu et al. 2010) examines the eyes.
terns comprise more information than others. The
The attempt to estimate the smile intensity was
distinction is based on the number of transition
made previously by (Shimada et al. 2010) and
between 0s and 1s when treating the LBP code as a
was also mentioned in the review of smile detec-
sequence of bits. The discerning codes, called ‘uni-
tion (Whitehill et al. 2009). Although, the pre-
form’ patterns (ULBP), have up to two such tran-
sented solution is more complex when compared
sitions, e.g. 00110000, 01111100, and are stored in
to our approach, the results enable to distinguish
separate histogram bins. All other codes share a
the moment of transition between emotions, yet
common bin. This upgrade shortens the histogram
smile intensities measured during smile period are
length to P(P − 1) + 3.
almost constant. In contrast, the SNIP system esti-
mates the smile intensity more accurately, which
enables to characterize the character and variation 4 SNIP SYSTEM SET-UP
of smile.
The SNIP determines between the smiling and
neutral facial displays in a sequence of images.
3 LOCAL BINARY PATTERNS The classifier is trained once, using the standard
facial display database and then applied for each
Local Binary Patterns (LBP) (Ojala et al. 2001, face detected in the video. The training and testing
Ojala et al. 1996) is a type of textural feature processes both require to perform following steps:

348

VIPIMAGE15_Book.indb 348 9/8/2015 7:47:37 PM

 Figure 2. Data flow in the SNIP system. On the right, the feature vectors from the standard Cohn-Kanade database
used to train the classifier are presented. On the left, is the input frame from the UVA-NEMO database. The arrows
show successive processing steps.

presented approach assumes a division of the image

into several sub-images as it is depicted in Figure 3.
According to previous research (Nurzynska and
Smolka 2014) the best results are achieved when
the original image is divided into 7 columns and
10 rows and the final descriptor is a concatenation
of ULBP histograms obtained for each sub-image.
Figure 3. Descriptor calculation. From the left: original
image, with marked sub-image grid; ULBP histograms
retrieved from each sub-image; ULBP concatenated his- 4.2 Training database
togram obtained from the sub-images.
Since it is possible to determine single muscle
activation during expression of each emotion,
• Convert image/frame to a grey-scale. (Ekman and Friesen 1978) defined Action Units
• Determine a region of the face in the image/ (AU), which correspond to the movement of the
frame using the face detector described in smallest possible set of muscles activated simultane-
(Kawulok and Szymanek 2012). ously. FACS defines which set of AUs is necessary
• Resize the area to the standard resolution of to present a particular facial gesture. As mentioned
112 × 150 pixels. before, this approach is commonly used to annotate
• Calculate the descriptor for facial display. the experimental data, therefore a natural conse-
quence is its application for training purposes. The
The data flow in the SNIP system is depicted in
database prepared according to these demands is the
Figure 2.
Cohn-Kanade AU-Coded Facial Expression Data-
base (Cohn et al. 1999). It contains image sequences
4.1 Descriptor calculation
from neutral to target display of basic emotions like
A lot of work concerning the smile description was joy, surprise, anger, fear, disgust, and happiness. The
conducted by (Ekman and Friesen 1978), who con- image resolution is 640 × 480 pixels with 8-bit preci-
cluded that the movement of two muscles is suffi- sion for grey-scale values. For training, frames from
cient to generate a Duchenne smile (the only one 82 sequences were chosen which depict the neutral
of smile types assumed to reflect pleasure) on the and smiling expression. In the paper we refer to this
human face. These are the orbicularis oculi, which data as Cohn-Kanade database.
raises the cheek and gathers the skin inwards from
around the eye socket and the zygomatic major
4.3 Support vector machine
which pulls the lip corners upwards towards the
cheekbones. Since the movement is present only in Support Vector Machine (SVM) method was applied
several parts of the face, constructing a descriptor for the recognition between the smiling and neutral
based on the whole image would not be satisfac- facial displays. Using the training data, a model for
tory, as the crucial information gets blended by the two-class problem solution is created. The exem-
regions where no change is noticeable. Therefore, the plary objects are treated as points in the space in

349

VIPIMAGE15_Book.indb 349 9/8/2015 7:47:38 PM

 which a dividing gap is looked for. If necessary, the the highest error ratio was noticed for the change
points are transformed by non-linear kernel in order between the emotional states (96.56% of the wrong
to better discriminate between objects from differ- classification) where frames labelled with neutral
ent categories. However, for considered problem, facial display were recognized as smiling expres-
the linear kernel proved to achieve the best results sion. The annotating person reported that in these
(Nurzynska and Smolka 2014). The goal is to obtain parts of the videos it was difficult to clearly decide
a gap as broad as possible, while the division plane is which emotion is presented, therefore it might
placed in the middle. Then in the testing phase, the be assumed that some wrong annotations took
examples are mapped to the same space. Depending place. To conclude, the system enables satisfactory
on which side of the cutting plane they are situ- classification between the emotions, the only prob-
ated, its membership to the category is defined. lem is the estimation of point in time where the
Additionally, the distance from the division plane change between the neutral and smiling facial dis-
defines the strength of belonging to one category. play occurs, as it might be shifted in some videos
These values are normalized in the range 0–1 to plot when compared to others.
the smile intensity variation over the time. Figure 4 shows the results for a video present-
ing the change between neutral, smiling, and once
again neutral expressions for spontaneously smiling
5 RESULTS person. In the middle part of the image, the graph
presents the smile intensity indicated by a red line
Verification of the classification accuracy and cor- and category annotation resulting from the classifier
relation of intensity estimation by the system with exhibited by black (neutral) and white (smile) colors.
visual data was performed on three datasets. First The top row shows every second frame recorded
of them (UVA-NEMO dataset) concentrates on when the transition between neutral and smiling
providing movies for smile truthfulness descrip- expression took place, whereas the bottom row col-
tion, therefore only basic test was possible. The sec- lects images describing the change from the smiling
ond (FEEDB dataset) consists of posed expression into the neutral facial display. Additionally, frames
and is treated as a reference for results gathered for for the most neutral expression (frame No. 19) and
UVA-NEMO. Whereas the third dataset supplied the highest intensity of smile (frame No. 199) are
movies with varying intensity of presented emo- presented. This case study confirms the accuracy in
tion, what enabled better insight into correct smile classification between the emotional states.
intensity prediction.
5.2 Case II: FEEDB—multimodal database
5.1 Case I: UVA-NEMO database of facial expressions and emotions
The experiments were performed on the UVA- The subset of 62 movies presenting happy and
NEMO database (Dibeklioğlu et al. 2012) which laugh emotion from Multimodal database of Facial
consists of 1240 videos presenting the spontaneous Expressions and Emotion (FEEDB) (Szwoch 2013,
and deliberate smiles. The data were collected for
400 subjects (56% male and 44% female) with age
rangingfrom 8 to 76. The frame resolution is 1920 ×
1080 pixels for color images, recorded at the speed
of 50 frames per second. The number of frames in
videos vary depending on the facial gesture length.
Moreover, it is not assured that each image sequence
starts and finishes with the neutral facial display.
In order to verify whether the elaborated lin-
ear SVM classifier is accurate for smiling and
neutral facial display recognition, its classifica-
tion results were compared to frames manually
annotated by a person without prior experience in
emotion description. For the experiment 100 vid-
eos were selected, their length varied from 91 to
540 frames. For total 21,022 frames the face was Figure 4. Video presenting transition between neutral—
smiling—neutral expressions. The middle part visualizes
correctly detected in 20,642 images, where 16,249 the change of the smile intensity over the time (red line),
correspond to the smiling facial display and 4,394 with additional annotation for neutral (black back-
describe the neutral emotion. In general, the sys- ground) and smiling expressions (white). Top and bot-
tem accuracy achieved 91.41% of correct emotion tom rows present every second frame for the moment of
classification. Analysis of the results showed that shift between emotions.

350

VIPIMAGE15_Book.indb 350 9/8/2015 7:47:40 PM

 Szwoch 2014) was exploited for verification of the between emotional states during speaking mostly
SNIP system performance. The database con- occurring in case of presentation of emotion with
sists of movies representing 30 different emotions low intensity. Rows 2nd and 3rd in Table 1 show the
per subject presented by the undergraduates and influence of incorporating speech into the emotion
graduates students of both genders. The record- recognition. One can see that the data with uttered
ings contain not only the color visual information sentences caused more errors, yet the difference is
stored with 640 × 480 pixel resolution frames gath- not significant. Finally, last three rows summarize
ered with 25–30 frame per second speed but also the influence of smile intensity on the emotion
the depth map. Yet, in presented research only the recognition. As it was supposed the most difficult
visual information is exploited. The length of the case is when the emotion display has low inten-
considered movies varied from 11 to 220 frames sity (83%), whereas for medium and high emotion
with average movie length of 120 frames (6560 expression 86% and 87% were recorded respec-
frame count in total). tively. Yet, it should be pointed out, than in many
The aim of this experiment was to repeat the cases most frames of the video presenting the low
procedure described in case study I using different intensity of happiness was assigned to neutral class
dataset. Hence, similarly the results of automatic by the system and annotating person. Another
classification were compared with the manual interesting phenomenon, which was noticed due to
annotations to give the overview of SNIP system the possibility of smile intensity visualization was
accuracy. For these data the correct classification
ratio was 89.76% and as in previous example most
errors occurred for frames depicting transition Table 1. Correct emotion classification into smiling and
neutral one for movies depicting happiness in DaFEX
from neutral to the smiling state and vice versa. database.
This high performance confirms the ability of emo-
tional state recognition by the presented system. Description Efficiency [%] No. of movies

All data 85.48 144

5.3 Case III: DaFEx database
No utterance 86.44 48
In this experiment the DaFEx database (Battocchi Utterance 84.94 96
et al. 2005, Battocchi et al. 2005) was used. It con- High 87.17 48
sists of recordings prepared by 8 actors (4 women, Medium 86.00 48
and 4 men) performing 6 Ekman’s emotions (anger, Low 83.08 48
disgust, fear, happiness, sadness, and surprise)
along with a neutral facial display recording. Each
emotion was acquired in 6 separate sessions, where
two of them presented the emotion only, while in
four additional sentence was uttered by the subject.
Moreover, each session consisted of three record-
ings with varying emotion strength: low, medium,
and high. To perform the experiment, sequences
depicting only happiness were exploited, that gave
144 movies in total (35870 frames). The frame res-
olution is 360 × 288 pixels for color images. The
number of frames in videos vary from 147 to 532
depending on the facial gesture length.
Similarly as in previous case studies, the result of
classification of the frames into smiling and neu-
tral groups is compared with manual annotations
performed by the person without prior experience.
Table 1 presents achieved correct classification
accuracy ratio considering some characteristics
of the dataset. First of all, the general smiling ver-
sus neutral facial display classification efficiency
is given in the top row. The score 85% is satisfac-
tory, however it is lower than the result in the pre- Figure 5. Examples of smile intensity estimation (red
vious cases. There are several reasons which could line) for time series of various smile intensity presented
deteriorate the classification accuracy: Firstly, the without utterances by subject No. 5 in series No. 3. Black
low image resolution. Secondly, introduced utter- background corresponds to neutral expression, while
ance in the sequences. Finally, unclear transition white to smiling one.

351

VIPIMAGE15_Book.indb 351 9/8/2015 7:47:41 PM

 the subjectivity of the emotion intensity presenta- Battocchi, A., F. Pianesi, & D. Goren-Bar (2005).
tion. For one actor the highest smile intensity is on A first evaluation study of a database of kinetic facial
the level of medium or even low level of the other expressions (DaFEx). In Proc. 7th Intern. Conf. Multi
subject. On the other hand, when comparing the modal Interfaces, ICMI ’05, New York, NY, USA,
pp. 214–221. ACM.
same person in similar circumstances, the assumed Battocchi, A., F. Pianesi, & D. Goren-Bar (2005, Oct).
change in smile intensity is easily visible on the The properties of DaFEx, a database of kinetic
plots generated by SNIP system as is presented in facial expressions. In Proc. 1st Intern. Conf. Affective
Figure 5. Computing and Intelligent Interaction, pp. 558–565.
Boraston, Z., B. Corden, L. Miles, D. Skuse, & S. Blakemore
(2008). Brief report: Perception of genuine and posed
6 CONCLUSIONS smiles by individuals with autism. Journal of Autism
and Developmental Disorders 38(3), 574–580.
This work presents a system for automatic classifica- Brahnam, S., L. Jain, L. Nanni, & A. Lumini (2014).
Local binary patterns: new variants and applications.
tion between smiling and neutral facial display with Springer.
estimation of the smile intensity. The system needs Cohn, J., A. Zlochower, J. Lien, & T. Kanade (1999).
a standard database for classifier training before its Automated face analysis by feature point tracking has
application to automatically annotate each frame of high concurrent validity with manual FACS coding.
the input video. As a result, the user receives infor- Psychophysiology 36(2), 35–43.
mation about the smile intensity changes within the Cohn, J.F., A.J. Zlochower, J.J. Lien, & T. Kanade (1998).
whole video with additional classification of each Feature-point tracking by optical flow discrimi-
frame for those presenting smile or not. nates subtle differences in facial expression. In Proc.
The performed research revealed that the correct of the 3rd. Intern. Conf. on Face & Gesture Recog.,
Washington, DC, USA, pp. 396–401.
classification ratio between neutral and smiling Dibeklioğlu, H., A. Salah, & T. Gevers (2012). Are
facial display is above 85% for DaFEx, 89% for you really smiling at me? Spontaneous versus posed
FEEDB, and 91% for UVA-NEMO databases. enjoyment smiles. In A. Fitzgibbon, S. Lazebnik,
However, in the last case when excluding from P. Perona, Y. Sato, and C. Schmid (Eds.), Computer
the calculation situation of transition between Vision ECCV 2012, Volume 7574 of Lecture Notes in
the emotion display it reaches almost 99%. The Computer Science, pp. 525–538. Springer.
case studies show, that it is possible to derive the Dibeklioğlu, H., R. Valenti, A.A. Salah, & T. Gevers
smile intensity information from the SVM classfier. (2010). Eyes do not lie: Spontaneous versus posed
The performed experiments confirmed that not smiles. In Proc. of the Intern. Conf. on Multimedia,
MM ’10, New York, NY, USA, pp. 703–706. ACM.
only binary distinction between smiling and neu- Ekman, P. & W. Friesen (1978). Facial action coding
tral facial display is possible but also the graduation system: a technique for the measurement of facial
in smile intensity can be expressed in this manner. movement. Consulting Psychologists Press.
The authors expect, that the presented functional- Essa, I. & A. Pentland (1997, Jul). Coding, analysis, inter-
ity of the system will find an application in medical pretation, and recognition of facial expressions. IEEE
treatment of patients with mood disorders and facil- Trans Pattern Anal Mach Intell 19(7), 757–763.
itate the work of psychologists and psychiatrists. Freire, D., L. Anton, & M. Castrillon (2009). A study
for the self similarity smile detection. In J.e.a. Fierrez
(Ed.), Biometric ID Management and Multimodal
Communication, Volume 5707 of Lecture Notes on
ACKNOWLEDGEMENTS Computer Science, pp. 97–104. Springer.
Freire, D., M. Castrillon, & O. Deniz-Suarez (2009). Smile
This work has been supported by the Polish National detection using local binary patterns and support vec-
Science Centre (NCN) under the Grant: DEC- tor machines. In VISSAPP(1)’09, pp. 398–401.
2012/07/B/ST6/01227 and was performed using the Girard, J., J. Cohn, M. Mahoor, S. Mavadati, Z. Hammal, &
infrastructure supported by POIG.02.03.01-24- D. Rosenwald (2014). Nonverbal social withdrawal
099/13 grant: GeCONiI—Upper-Silesian Center in depression: Evidence from manual and automatic
for Scientific Computation. analyses. Image Vision Comput. 32(10), 641–647.
Hurley, C. & M. Frank (2011). Executing facial control
during deception situations. Journal of Non verbal
Behavior 35(2), 119–131.
REFERENCES Kawulok, M. & J. Szymanek (2012, March). Precise
multi-level face detector for advanced analysis of
An, L., S. Yang, & B. Bhanu (2014). Efficient smile detec- facial images. Image Processing, IET 6(2), 95–103.
tion by extreme learning machine. Neurocomputing Kohler, C., E. Martin, N. Stolar, F. Barrett, R. Verma,
149, 354–363. C. Brensinger, W. Bilker, R. Gur, & R. Gur (2008,
Bartlett, M., J. Hager, P. Ekman, & T. Sejnowski (1999). October). Static posed and evoked facial expressions
Measuring facial expressions by computer image of emotions in schizophrenia. Schizophrenia Research
analysis. Psychophysiology 36(2), 253–263. 105(1–3), 49–60.

352

VIPIMAGE15_Book.indb 352 9/8/2015 7:47:43 PM

 Lanitis, A., C. Taylor, & T. Cootes (1997, Jul). Automatic Shan, C., S. Gong, & P. McOwan (2009, May). Facial
interpretation and coding of face images using flexible expression recognition based on local binary patterns:
models. IEEE Trans Pattern Anal Mach Intell 19(7), A comprehensive study. Image Vision Comput. 27(6),
743–756. 803–816.
Mase, K. (1990). An application of optical flow— Shimada, K., T. Matsukawa, Y. Noguchi, & T. Kurita
extraction of facial expression. In IAPR Workshop on (2010). Appearance-based smile intensity estimation
Machine Vision Applications, pp. 195–198. by cascaded support vector machines. In R. Koch and
Nurzynska, K. & B. Smolka (2014). Optimal classification F. Huang (Eds.), ACCV Workshops (1), Volume 6468
method for smiling vs. neutral facial display recogni- of Lecture Notes in Computer Science, pp. 277–286.
tion. Journal of Medical Information and Technologies Springer.
23, 87–94. Souto, T., A. Baptista, D. Tavares, C. Queirós, &
Ojala, T., M. Pietikäinen, & D. Harwood (1996). A com- A. Marques (2013). Facial emotional recognition in
parative study of texture measures with classification schizophrenia: preliminary results of the virtual real-
based on featured distributions. Pattern Recognition ity program for facial emotional recognition. Revistade
29(1), 51–59. Psiquiatria Clinica 40(4), 129–134.
Ojala, T., M. Pietikäinen, & T. Mäenpää (2001). Szwoch, M. (2013, June). Feedb: A multimodal database
A generalized local binary pattern operator for mul- of facial expressions and emotions. In 6th Intern. Conf.
tire solution gray scale and rotation invariant texture Human System Interaction, pp. 524–531.
classification. In Proc. 2nd Intern. Conf. on Advances Szwoch, M. (2014). On facial expressions and emo-
in Pattern Recognition, London, UK, pp. 397–406. tions RGB-D database. In S. Kozielski, D. Mrozek,
Springer. P. Kasprowski, B. Maysiak-Mrozek, and D. Kostrzewa
Pantic, M. & L. Rothkrantz (2000). Automatic analysis (Eds.), Beyond Databases, Architectures, and Struc-
of facial expressions: The state of the art. IEEE Trans tures, Volume 424 of Communications in Computer and
Pattern Anal Mach Intell 22, 1424–1445. Information Science, pp. 384–394. Springer.
Pietikäinen, M., G. Zhao, A. Hadid, & T. Ahonen (2011). Tsai, J., N. Pole, R. Levenson, & R. Munoz (2003,
Computer vision using local binary patterns. In Com- February). The effects of depression on the emotional
putational Imaging and Vision, Volume 40, pp. 13–9. responses of Spanish-speaking Latinas. Cultur Divers
Springer. Ethnic Minor Psychol. 9(1), 49–63.
Reed, L., M. Sayette, & J. Cohn (2007). Impact of depres- Whitehill, J., G. Littlewort, I. Fasel, M. Bartlett, &
sion on response to comedy: A dynamic facial cod- J. Movellan (2009, Nov). Toward practical smile detec-
ing analysis. Journal of Abnormal Psychology 116(4), tion. IEEE Trans Pattern Anal Mach Intell 31(11),
804–809. 2106–2111.
Scherer, S., G. Stratou, M. Mahmound, J. Boberg, Wu, Q., X. Shen, & X. Fu (2011). The machine knows
J. Gratch, A. Rizzo, & L.-P. Morency (2013, what you are hiding: An automatic micro-expression
April). Automatic behavior descriptors for psycho- recognition system. In S. DMello, A. Graesser,
logical disorder analysis. In IEEE Conf. on Auto- B. Schuller, and J.-C. Martin (Eds.), Affective Comput-
matic Face and Gesture Recognition, Shanghai, ing and Intelligent Interaction, Volume 6975 of Lecture
China. Notes in Computer Science, pp. 152–162. Springer.

353

VIPIMAGE15_Book.indb 353 9/8/2015 7:47:43 PM

This page intentionally left blank
Computational Vision and Medical Image Processing. VipIMAGE 2015 contains invited
lectures and full papers presented at VIPIMAGE 2015 - V ECCOMAS Thematic Conference on
Computational Vision and Medical Image Processing (Tenerife, Canary Islands, Spain, 19-21
Computational Vision and
Medical Image Processing
October, 2015). International contributions from 19 countries provide a comprehensive coverage
of the current state-of-the-art in the fields of:

3D Vision;
Computational Bio- Imaging and Visualization;

Medical Image Processing

Computational Vision and
Computational Vision;
Computer Aided Diagnosis, Surgery, Therapy and Treatment;
Data Interpolation, Registration, Acquisition and Compression;
Industrial Inspection;
Image Enhancement;
Image Processing and Analysis;
VIPIMAGE 2015
Image Segmentation;
Medical Imaging;
Medical Rehabilitation;
Physics of Medical Imaging;
Shape Reconstruction;
Signal Processing;
Simulation and Modelling;
Software Development for Image Processing and Analysis;
Telemedicine Systems and their Applications;
Tracking and Analysis of Movement and Deformation;
Virtual Reality.

Computational Vision and Medical Image Processing. VipIMAGE 2015 will be useful
to academics, researchers and professionals in Biomechanics, Biomedical Engineering,
Computational Vision (image processing and analysis), Computer Sciences, Computational
Mechanics, Signal Processing, Medicine and Rehabilitation.

Sponsored by

Joao Manuel R.S.Tavares

R.M. Natal Jorge
EDITORS
an informa business