3-D Computer Vision: Principles, Algorithms and

Applications 1st Edition Yu-Jin Zhang pdf

download

https://ebookmeta.com/product/3-d-computer-vision-principles-
algorithms-and-applications-1st-edition-yu-jin-zhang/

Download more ebook from https://ebookmeta.com

We believe these products will be a great fit for you. Click
the link to download now, or visit ebookmeta.com
to discover even more!

Computer Vision Algorithms and Applications 2nd Edition

Richard Szeliski

https://ebookmeta.com/product/computer-vision-algorithms-and-
applications-2nd-edition-richard-szeliski/

A Selection of Image Processing Techniques: From

Fundamental to Research Front 1st Edition Yu-Jin Zhang

https://ebookmeta.com/product/a-selection-of-image-processing-
techniques-from-fundamental-to-research-front-1st-edition-yu-jin-
zhang/

Computer Vision for X Ray Testing Imaging Systems Image

Databases and Algorithms Domingo Mery

https://ebookmeta.com/product/computer-vision-for-x-ray-testing-
imaging-systems-image-databases-and-algorithms-domingo-mery/

The Irresistible Attraction of Gravity A Journey to

Discover Black Holes Rezzolla

https://ebookmeta.com/product/the-irresistible-attraction-of-
gravity-a-journey-to-discover-black-holes-rezzolla/
Pina Colada Poison Charlotte Gibson Mystery 4 Jasmine
Webb

https://ebookmeta.com/product/pina-colada-poison-charlotte-
gibson-mystery-4-jasmine-webb/

Artificial Intelligence and Machine Learning for

Healthcare Vol 2 Emerging Methodologies and Trends
Intelligent Systems Reference Library 229 1st Edition
Chee Peng Lim (Editor)
https://ebookmeta.com/product/artificial-intelligence-and-
machine-learning-for-healthcare-vol-2-emerging-methodologies-and-
trends-intelligent-systems-reference-library-229-1st-edition-
chee-peng-lim-editor/

Get It Done Now Own Your Time Take Back Your Life 2nd
Edition Brian Tracy

https://ebookmeta.com/product/get-it-done-now-own-your-time-take-
back-your-life-2nd-edition-brian-tracy/

The Beauty Of Murder Mysteries from the Antique Store 2

Sydney Tate

https://ebookmeta.com/product/the-beauty-of-murder-mysteries-
from-the-antique-store-2-sydney-tate/

Middle Eastern Cooking Discover Tasty Middle Eastern

Food with Easy Middle Eastern Cooking 2nd Edition
Booksumo Press

https://ebookmeta.com/product/middle-eastern-cooking-discover-
tasty-middle-eastern-food-with-easy-middle-eastern-cooking-2nd-
edition-booksumo-press/
The Economics of Ecology Exchange and Adaptation
Anthropological Explorations 1st Edition Donald C. Wood

https://ebookmeta.com/product/the-economics-of-ecology-exchange-
and-adaptation-anthropological-explorations-1st-edition-donald-c-
wood/
Yu-Jin Zhang

3-D Computer
Vision
Principles, Algorithms and Applications
3-D Computer Vision
Yu-Jin Zhang

3-D Computer Vision

Principles, Algorithms and Applications
Yu-Jin Zhang
Department of Electronic Engineering
Tsinghua University
Beijing, China

ISBN 978-981-19-7579-0 ISBN 978-981-19-7580-6 (eBook)

https://doi.org/10.1007/978-981-19-7580-6

Jointly published with Publishing House of Electronics Industry

The print edition is not for sale in China (Mainland). Customers from China (Mainland) please order the
print book from: Publishing House of Electronics Industry.

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore
Pte Ltd. 2023

Jointly published with Publishing House of Electronics Industry, Beijing, China

This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or
dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publishers, the authors, and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publishers nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
herein or for any errors or omissions that may have been made. The publishers remain neutral with
regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface

This book is a specialized textbook that introduces the basic principles, typical
methods, and practical techniques of 3-D computer vision. It can provide an
advanced/second course service of computer vision for graduate students of related
majors in higher engineering colleges and universities, after studying 2D Computer
Vision: Principles, Algorithms and Applications.
This book mainly covers the higher level of computer vision from the selection of
materials. This book is self-contained, mainly for information majors, but also takes
into account learners of different professional backgrounds, and also considers the
needs of self-study readers. After learning the fundamental content of computer
vision in this book, readers can carry out scientific research and solve more and even
difficult specific problems in practical applications.
This book pays more attention to practicality in writing. Considering that com-
puter vision technology has been involved in many professional fields in recent
years, but many working people are not specialized in computer vision technology,
this book does not emphasize the theoretical system too much, minimizes the
formula derivation, and focuses on commonly used techniques. This book has
many sample questions and uses intuitive explanation to help readers understand
abstract concepts. A subject index list is given at the end of the book, and those
corresponding terms are marked in bold in the text.
This book provides a large number of self-test questions (including hints and
answers). In terms of purpose: on the one hand, it is convenient for self-study to
judge whether they have mastered the key content; on the other hand, it is also
convenient for teachers to carry out online teaching and strengthen teacher-student
interaction during lectures. The types of questions are multiple-choice questions,
which can be easily judged by a computer. In terms of content, many questions
express the basic concepts in a different way, supplementing the text, so that learners
can deepen their understanding. Some questions list descriptions that are similar but
not the same, or even have opposite meanings. Through dialectical thinking of pros
and cons, learners can also deeply understand the essence. The hints have been
provided for all self-test questions, allowing readers to obtain more information to

v
vi Preface

further identify the meaning of the questions. At the same time, each question can be
said to be divided into two levels in this way. Readers can complete the self-test after
reading the hints to show that they basically understand it, and to complete the self-
test without looking at the hints indicates that they have an even better grasp.
From the structure of this book, there are 12 chapters in total, plus one appendix,
answers to self-test questions, and subject index. Under these 15 first-level headings,
there are a total of 103 second-level headings (sections), and there are 141 third-level
headings (subsections) underneath. The book has a total of about 500,000 words
(including pictures, drawings, tables, formulas, etc.) and a total of (numbered)
228 figures, 22 tables, and 566 formulas. In order to facilitate teaching and learning,
this book provides a total of 68 examples of various types and 157 self-test questions
(all with hints and answers). In addition, there are a list of more than 100 directly
related references and a list of more than 500 subject terms for indexing, at the end of
the book.
This book can consider three aspects from the knowledge requirements of the
prerequisite courses: (1) Mathematics: including linear algebra and matrix theory, as
well as basic knowledge about statistics, probability theory, and random modeling;
(2) Computer science: including the mastery of computer software technology, the
understanding of computer structure system, and the application of computer pro-
gramming methods; (3) Electronics: On the one hand, the characteristics and prin-
ciples of electronic equipment; on the other hand, circuit design and other content. In
addition, the book 2D Computer Vision: Principles, Algorithms and Applications
can be counted as the discipline prerequisite of this book.
Thanks to the editors of the publisher for carefully composing the manuscript,
seriously reviewing, and attentively modifying.
Finally, the author thanks his wife Yun HE and daughter Heming ZHANG for
their understanding and support in all aspects.

Beijing, China Yu-Jin Zhang

Contents

1 Computer Vision Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Human Vision and Characteristics . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Visual Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Brightness Properties of Vision . . . . . . . . . . . . . . . . . . 3
1.1.3 Spatial Properties of Vision . . . . . . . . . . . . . . . . . . . . . 6
1.1.4 Temporal Properties of Vision . . . . . . . . . . . . . . . . . . . 8
1.1.5 Visual Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.2 Computer Vision Theory and Framework . . . . . . . . . . . . . . . . . 13
1.2.1 Reaserch Goals, Tasks, and Methods of Computer
Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.2.2 Visual Computational Theory . . . . . . . . . . . . . . . . . . . 14
1.2.3 Framework Problems and Improvements . . . . . . . . . . . 20
1.3 Three-Dimensional Vision System and Image Technology . . . . . 23
1.3.1 Three-Dimensional Vision System Process . . . . . . . . . . 23
1.3.2 Computer Vision and Image Technology Levels . . . . . . 24
1.3.3 Image Technology Category . . . . . . . . . . . . . . . . . . . . 25
1.4 Overview of the Structure and Content of This Book . . . . . . . . . 27
1.4.1 Structural Framework and Content of This Book . . . . . 27
1.4.2 Chapter Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 29
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.1 Linear Camera Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.1.1 Complete Imaging Model . . . . . . . . . . . . . . . . . . . . . . 38
2.1.2 Basic Calibration Procedure . . . . . . . . . . . . . . . . . . . . 40
2.1.3 Internal and External Parameters . . . . . . . . . . . . . . . . . 41
2.2 Non-Linear Camera Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.2.1 Type of Distortion . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

vii
viii Contents

2.2.2 Calibration Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

2.2.3 Classification of Calibration Methods . . . . . . . . . . . . . 48
2.3 Traditional Calibration Methods . . . . . . . . . . . . . . . . . . . . . . . . 49
2.3.1 Basic Steps and Parameters . . . . . . . . . . . . . . . . . . . . . 49
2.3.2 Two-Stage Calibration Method . . . . . . . . . . . . . . . . . . 51
2.3.3 Precision Improvement . . . . . . . . . . . . . . . . . . . . . . . . 56
2.4 Self-Calibration Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
2.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 61
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3 Three-Dimensional Image Acquisition . . . . . . . . . . . . . . . . . . . . . . . 67
3.1 High-Dimensional Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.2 Depth Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.2.1 Depth Image and Grayscale Image . . . . . . . . . . . . . . . 69
3.2.2 Intrinsic Image and Non-Intrinsic Image . . . . . . . . . . . 70
3.2.3 Depth Imaging Modes . . . . . . . . . . . . . . . . . . . . . . . . 72
3.3 Direct Depth Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.3.1 Time-of-Flight Method . . . . . . . . . . . . . . . . . . . . . . . . 74
3.3.2 Structured Light Method . . . . . . . . . . . . . . . . . . . . . . . 77
3.3.3 Moiré Contour Stripes Method . . . . . . . . . . . . . . . . . . 79
3.3.4 Simultaneous Acquisition of Depth and Brightness
Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.4 Stereo Vision Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.4.1 Binocular Horizontal Mode . . . . . . . . . . . . . . . . . . . . . 85
3.4.2 Binocular Convergence Horizontal Mode . . . . . . . . . . . 89
3.4.3 Binocular Axial Mode . . . . . . . . . . . . . . . . . . . . . . . . 92
3.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 95
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4 Video Image and Motion Information . . . . . . . . . . . . . . . . . . . . . . . 101
4.1 Video Basic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.1.1 Video Expression and Model . . . . . . . . . . . . . . . . . . . 102
4.1.2 Video Display and Format . . . . . . . . . . . . . . . . . . . . . 104
4.1.3 Color TV System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.2 Motion Classification and Representation . . . . . . . . . . . . . . . . . 108
4.2.1 Motion Classification . . . . . . . . . . . . . . . . . . . . . . . . . 109
4.2.2 Motion Vector Field Representation . . . . . . . . . . . . . . 109
4.2.3 Motion Histogram Representation . . . . . . . . . . . . . . . . 111
4.2.4 Motion Track Description . . . . . . . . . . . . . . . . . . . . . . 113
4.3 Motion Information Detection . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.3.1 Motion Detection Based on Camera Model . . . . . . . . . 114
4.3.2 Frequency Domain Motion Detection . . . . . . . . . . . . . 118
4.3.3 Detection of Movement Direction . . . . . . . . . . . . . . . . 121
Contents ix

4.4 Motion-Based Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

4.4.1 Motion Detection-Based Filtering . . . . . . . . . . . . . . . . 123
4.4.2 Motion Compensation-Based Filtering . . . . . . . . . . . . . 124
4.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 130
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5 Moving Object Detection and Tracking . . . . . . . . . . . . . . . . . . . . . . 135
5.1 Differential Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5.1.1 Calculation of Difference Image . . . . . . . . . . . . . . . . . 136
5.1.2 Calculation of Accumulative Difference Image . . . . . . . 138
5.2 Background Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
5.2.1 Basic Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5.2.2 Typical Practical Methods . . . . . . . . . . . . . . . . . . . . . . 140
5.2.3 Effect Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
5.3 Optical Flow Field and Motion . . . . . . . . . . . . . . . . . . . . . . . . 145
5.3.1 Optical Flow Equation . . . . . . . . . . . . . . . . . . . . . . . . 145
5.3.2 Optical Flow Estimation with Least Square Method . . . 146
5.3.3 Optical Flow in Motion Analysis . . . . . . . . . . . . . . . . . 147
5.3.4 Dense Optical Flow Algorithm . . . . . . . . . . . . . . . . . . 150
5.4 Moving Object Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
5.4.1 Kalman Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
5.4.2 Particle Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
5.4.3 Mean Shift and Kernel Tracking . . . . . . . . . . . . . . . . . 160
5.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 164
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
6 Binocular Stereo Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
6.1 Stereo Vision Process and Modules . . . . . . . . . . . . . . . . . . . . . 170
6.1.1 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . 170
6.1.2 Image Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
6.1.3 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
6.1.4 Stereo Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
6.1.5 3-D Information Recovery . . . . . . . . . . . . . . . . . . . . . 172
6.1.6 Post-Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
6.2 Region-Based Stereo Matching . . . . . . . . . . . . . . . . . . . . . . . . 173
6.2.1 Template Matching . . . . . . . . . . . . . . . . . . . . . . . . . . 174
6.2.2 Stereo Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
6.3 Feature-Based Stereo Matching . . . . . . . . . . . . . . . . . . . . . . . . 185
6.3.1 Basic Steps and Methods . . . . . . . . . . . . . . . . . . . . . . 186
6.3.2 Scale Invariant Feature Transformation . . . . . . . . . . . . 190
6.3.3 Dynamic Programming Matching . . . . . . . . . . . . . . . . 192
6.4 Error Detection and Correction of Parallax Map . . . . . . . . . . . . 194
6.4.1 Error Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
x Contents

6.4.2 Error Correction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

6.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 198
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
7 Monocular Multiple Image Recovery . . . . . . . . . . . . . . . . . . . . . . . 205
7.1 Photometric Stereo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
7.1.1 Light Source, Scenery, Lens . . . . . . . . . . . . . . . . . . . . 207
7.1.2 Scene Brightness and Image Brightness . . . . . . . . . . . . 208
7.1.3 Surface Reflection Characteristics and Brightness . . . . . 213
7.2 Shape from Illumination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
7.2.1 Representation of the Surface Orientation
of a Scene . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
7.2.2 Reflectance Map and Brightness Constraint
Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
7.2.3 Solution of Photometric Stereo . . . . . . . . . . . . . . . . . . 222
7.3 Optical Flow Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
7.3.1 Optical Flow and Motion Field . . . . . . . . . . . . . . . . . . 226
7.3.2 Solving Optical Flow Equation . . . . . . . . . . . . . . . . . . 229
7.4 Shape from Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
7.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 240
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
8 Monocular Single Image Recovery . . . . . . . . . . . . . . . . . . . . . . . . . 247
8.1 Shape from Shading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
8.1.1 Shading and Orientation . . . . . . . . . . . . . . . . . . . . . . . 248
8.1.2 Gradient Space Method . . . . . . . . . . . . . . . . . . . . . . . 250
8.2 Solving Brightness Equation . . . . . . . . . . . . . . . . . . . . . . . . . . 253
8.2.1 Linearity Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
8.2.2 Rotational Symmetry Case . . . . . . . . . . . . . . . . . . . . . 256
8.2.3 The General Case of Smoothness Constraints . . . . . . . . 258
8.3 Shape from Texture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
8.3.1 Monocular Imaging and Distortion . . . . . . . . . . . . . . . 260
8.3.2 Orientation Restoration from the Change of Texture . . . 263
8.4 Detection of Texture Vanishing Points . . . . . . . . . . . . . . . . . . . 271
8.4.1 Detecting the Vanishing Point of Line Segment
Texture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
8.4.2 Determine the Vanishing Point Outside the Image . . . . 273
8.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 276
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
9 Three-Dimensional Scenery Representation . . . . . . . . . . . . . . . . . . 283
9.1 Local Features of the Surface . . . . . . . . . . . . . . . . . . . . . . . . . . 284
9.1.1 Surface Normal Section . . . . . . . . . . . . . . . . . . . . . . . 284
Contents xi

9.1.2 Surface Principal Curvature . . . . . . . . . . . . . . . . . . . . 285

9.1.3 Mean Curvature and Gaussian Curvature . . . . . . . . . . . 286
9.2 Three-Dimensional Surface Representation . . . . . . . . . . . . . . . . 288
9.2.1 Parameter Representation . . . . . . . . . . . . . . . . . . . . . . 288
9.2.2 Surface Orientation Representation . . . . . . . . . . . . . . . 291
9.3 Construction and Representation of Iso-surfaces . . . . . . . . . . . . 294
9.3.1 Marching Cube Algorithm . . . . . . . . . . . . . . . . . . . . . 295
9.3.2 Wrapper Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 298
9.4 Interpolating Three-Dimensional Surfaces from Parallel
Contours . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
9.4.1 Contour Interpolation and Tiling . . . . . . . . . . . . . . . . . 300
9.4.2 Problems That May Be Encountered . . . . . . . . . . . . . . 302
9.4.3 Delaunay Triangulation and Neighborhood Voronoï
Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
9.5 Three-Dimensional Entity Representation . . . . . . . . . . . . . . . . . 306
9.5.1 Basic Representation Scheme . . . . . . . . . . . . . . . . . . . 306
9.5.2 Generalized Cylinder Representation . . . . . . . . . . . . . . 309
9.6 Key Points and References for Each Section . . . . . . . . . . . . . . . 311
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
10 Generalized Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
10.1 Matching Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
10.1.1 Matching Strategies and Categories . . . . . . . . . . . . . . . 318
10.1.2 Matching and Registration . . . . . . . . . . . . . . . . . . . . . 321
10.1.3 Matching Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 323
10.2 Object Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324
10.2.1 Measure of Matching . . . . . . . . . . . . . . . . . . . . . . . . . 324
10.2.2 Corresponding Point Matching . . . . . . . . . . . . . . . . . . 327
10.2.3 String Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
10.2.4 Matching of Inertia Equivalent Ellipses . . . . . . . . . . . . 329
10.2.5 Shape Matrix Matching . . . . . . . . . . . . . . . . . . . . . . . 331
10.3 Dynamic Pattern Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
10.3.1 Matching Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
10.3.2 Absolute Pattern and Relative Pattern . . . . . . . . . . . . . 334
10.4 Graph Theory and Graph Matching . . . . . . . . . . . . . . . . . . . . . 336
10.4.1 Introduction to Graph Theory . . . . . . . . . . . . . . . . . . . 336
10.4.2 Graph Isomorphism and Matching . . . . . . . . . . . . . . . . 340
10.5 Line Drawing Signature and Matching . . . . . . . . . . . . . . . . . . . 343
10.5.1 Contour Marking . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
10.5.2 Structural Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . 344
10.5.3 Labeling via Backtracking . . . . . . . . . . . . . . . . . . . . . 346
10.6 Key Points and References for Each Section . . . . . . . . . . . . . . . 347
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
xii Contents

11 Knowledge and Scene Interpretation . . . . . . . . . . . . . . . . . . . . . . . . 355

11.1 Scene Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
11.1.1 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
11.1.2 Attribute Hypergraph . . . . . . . . . . . . . . . . . . . . . . . . . 358
11.1.3 Knowledge-Based Modeling . . . . . . . . . . . . . . . . . . . . 360
11.2 Logic System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362
11.2.1 Predicate Calculation Rules . . . . . . . . . . . . . . . . . . . . . 362
11.2.2 Inference by Theorem Proving . . . . . . . . . . . . . . . . . . 366
11.3 Fuzzy Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
11.3.1 Fuzzy Sets and Fuzzy Operations . . . . . . . . . . . . . . . . 370
11.3.2 Fuzzy Reasoning Method . . . . . . . . . . . . . . . . . . . . . . 372
11.4 Scene Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
11.4.1 Bag-of-Words/Feature Model . . . . . . . . . . . . . . . . . . . 375
11.4.2 pLSA Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
11.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 383
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
12 Spatial-Temporal Behavior Understanding . . . . . . . . . . . . . . . . . . . 389
12.1 Spatial-Temporal Technology . . . . . . . . . . . . . . . . . . . . . . . . . 390
12.1.1 New Research Field . . . . . . . . . . . . . . . . . . . . . . . . . . 390
12.1.2 Multiple Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
12.2 Spatial-Temporal Interest Point Detection . . . . . . . . . . . . . . . . . 392
12.2.1 Detection of Points of Interest in Space . . . . . . . . . . . . 392
12.2.2 Detection of Points of Interest in Space and Time . . . . . 393
12.3 Spatial-Temporal Dynamic Trajectory Learning and
Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
12.3.1 Automatic Scene Modeling . . . . . . . . . . . . . . . . . . . . . 396
12.3.2 Path Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
12.3.3 Automatic Activity Analysis . . . . . . . . . . . . . . . . . . . . 403
12.4 Spatial-Temporal Action Classification and Recognition . . . . . . 405
12.4.1 Motion Classification . . . . . . . . . . . . . . . . . . . . . . . . . 405
12.4.2 Action Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . 407
12.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 414
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418

Appendix A: Visual Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419

A.1 Shape Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
A.2 Spatial Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
A.2.1 Nonvisual Indices of Depth . . . . . . . . . . . . . . . . . . . . . 421
A.2.2 Binocular Indices of Depth . . . . . . . . . . . . . . . . . . . . . 422
A.2.3 Monocular Indices of Depth . . . . . . . . . . . . . . . . . . . . . 424
Contents xiii

A.3 Motion Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427

A.3.1 The Condition of Motion Perception . . . . . . . . . . . . . . . 427
A.3.2 Detection of Moving Objects . . . . . . . . . . . . . . . . . . . . 429
A.3.3 Depth Motion Detection . . . . . . . . . . . . . . . . . . . . . . . 429
A.3.4 Real Motion and Apparent Motion . . . . . . . . . . . . . . . . 430
A.3.5 Correspondence Matching of Apparent Motion . . . . . . . 430
A.3.6 Aperture Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
A.3.7 Dynamic Indices of Depth . . . . . . . . . . . . . . . . . . . . . . 433
A.4 Key Points and References for Each Section . . . . . . . . . . . . . . . 434
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434
Answers to Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435
Chapter 1 Computer Vision Overview . . . . . . . . . . . . . . . . . . . . . . . . 435
Chapter 2 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
Chapter 3 Three-Dimensional Image Acquisition . . . . . . . . . . . . . . . . . 436
Chapter 4 Video Image and Motion Information . . . . . . . . . . . . . . . . . 437
Chapter 5 Moving Object Detection and Tracking . . . . . . . . . . . . . . . . 437
Chapter 6 Binocular Stereo Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
Chapter 7 Monocular Multiple Image Recovery . . . . . . . . . . . . . . . . . 439
Chapter 8 Monocular Single Image Recovery . . . . . . . . . . . . . . . . . . . 439
Chapter 9 Three-Dimensional Scene Representation . . . . . . . . . . . . . . 440
Chapter 10 Scene Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
Chapter 11 Knowledge and Scene Interpretation . . . . . . . . . . . . . . . . . 441
Chapter 12 Spatial-Temporal Behavior Understanding . . . . . . . . . . . . . 442
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
Chapter 1
Computer Vision Overview

Computer vision is an information subject/discipline that uses computers to realize

the functions of human vision system (HVS). This book mainly introduces the high-
level content of computer vision, which can be used as a textbook for in-depth study
of computer vision. Readers can read this book after studying 2D Computer Vision:
Principles, Algorithms and Applications.
The process of human vision (in brief, vision) can be regarded as a complex
process from sensation (obtaining the image by the 2-D projection of the 3-D world)
to perception (gaining the content and meaning of the 3-D world from the 2-D
image) process. The ultimate goal of vision in a narrow sense is to make a mean-
ingful explanation and description of the scene for the observer, and in a broad sense,
it is also to make a behavior plan based on these explanations and descriptions, as
well as the surrounding environment and the wishes of the observer. Computer
vision is artificial vision or human-made vision. It is to use computers to realize
human visual functions, hoping to make meaningful judgments about actual targets
and scenes based on the perceived images.
The sections of this chapter are arranged as follows:
Section 1.1 introduces the characteristics of human vision, the brightness properties
of vision, the spatial properties of vision, and the temporal properties of vision, as
well as makes some discussions on visual perception.
Section 1.2 discusses the research purpose, research tasks, and research methods of
computer vision. It also introduces the visual computational theory proposed by
Marr in more detail. Moreover, a combined presentation for some improvement
ideas is provided.
Section 1.3 gives a general introduction to the 3-D vision system that obtains 3-D
spatial information and realizes the understanding of the scene. It compares and
discusses the layers of computer vision and image technology, leading to the
main content of this book.
Section 1.4 presents the structure of the book and gives the brief summaries of each
chapter.

© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 1
Y.-J. Zhang, 3-D Computer Vision, https://doi.org/10.1007/978-981-19-7580-6_1
2 1 Computer Vision Overview

1.1 Human Vision and Characteristics

Computer vision (also called artificial vision or human-made vision) is developed on

the basis of human vision. There are two meanings here. One is that computer vision
needs to realize the functions of human vision, so it could imitate the system
structure and functional modules of human vision; the other is that computer vision
needs to extend the functions of human vision, so it could use the features of human
vision to improve the efficiency and effectiveness of these functions.
The following first gives an overview of the functional characteristics of human
vision and then discusses some important visual brightness, spatial, and temporal
properties.

1.1.1 Visual Characteristics

First compare vision with some related concepts.

1.1.1.1 Vision and Other Sensations

It is generally believed that humans have five senses of vision, hearing, smell, taste,
and touch and the corresponding sensory organs, for obtaining information from the
objective world. Among them, vision provides humans with most of the data they
receive; in other words, humans often rely more on vision than other senses when
they acquaint the world. For example, the input information obtained by humans
from eye observation often reaches several million bits, and the data rate during
continuous viewing can exceed tens of million bits per second. The human brain has
more than 1010 cells/neurons, some of which have more than 10,000 connections
(or synapses) with other neurons. It is estimated that the amount of visual informa-
tion the brain receives from the eyes is at least two orders of magnitude larger than all
the information obtained from other sense organs.

1.1.1.2 Vision and Computer Vision

Although computer vision needs to realize the functions of human vision, there are
still differences between human vision and computer vision. Human vision first
receives light stimulation in a certain wavelength range from the external environ-
ment through the sensory organs (eyes) of the visual system to the retina and then
encodes and processes the perceptive organs (brain or cerebral visual cortex) of the
visual system to obtain subjective feelings. Therefore, vision involves not only
physics and chemistry but also psychophysiology. Computer vision mainly relies
on photoelectric conversion for image acquisition, then uses processing and analysis
Another Random Scribd Document
with Unrelated Content
Gutenberg™ electronic works, and the medium on which they may
be stored, may contain “Defects,” such as, but not limited to,
incomplete, inaccurate or corrupt data, transcription errors, a
copyright or other intellectual property infringement, a defective or
damaged disk or other medium, a computer virus, or computer
codes that damage or cannot be read by your equipment.

1.F.2. LIMITED WARRANTY, DISCLAIMER OF DAMAGES - Except for

the “Right of Replacement or Refund” described in paragraph 1.F.3,
the Project Gutenberg Literary Archive Foundation, the owner of the
Project Gutenberg™ trademark, and any other party distributing a
Project Gutenberg™ electronic work under this agreement, disclaim
all liability to you for damages, costs and expenses, including legal
fees. YOU AGREE THAT YOU HAVE NO REMEDIES FOR
NEGLIGENCE, STRICT LIABILITY, BREACH OF WARRANTY OR
BREACH OF CONTRACT EXCEPT THOSE PROVIDED IN PARAGRAPH
1.F.3. YOU AGREE THAT THE FOUNDATION, THE TRADEMARK
OWNER, AND ANY DISTRIBUTOR UNDER THIS AGREEMENT WILL
NOT BE LIABLE TO YOU FOR ACTUAL, DIRECT, INDIRECT,
CONSEQUENTIAL, PUNITIVE OR INCIDENTAL DAMAGES EVEN IF
YOU GIVE NOTICE OF THE POSSIBILITY OF SUCH DAMAGE.

1.F.3. LIMITED RIGHT OF REPLACEMENT OR REFUND - If you

discover a defect in this electronic work within 90 days of receiving
it, you can receive a refund of the money (if any) you paid for it by
sending a written explanation to the person you received the work
from. If you received the work on a physical medium, you must
return the medium with your written explanation. The person or
entity that provided you with the defective work may elect to provide
a replacement copy in lieu of a refund. If you received the work
electronically, the person or entity providing it to you may choose to
give you a second opportunity to receive the work electronically in
lieu of a refund. If the second copy is also defective, you may
demand a refund in writing without further opportunities to fix the
problem.
1.F.4. Except for the limited right of replacement or refund set forth
in paragraph 1.F.3, this work is provided to you ‘AS-IS’, WITH NO
OTHER WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR ANY PURPOSE.

1.F.5. Some states do not allow disclaimers of certain implied

warranties or the exclusion or limitation of certain types of damages.
If any disclaimer or limitation set forth in this agreement violates the
law of the state applicable to this agreement, the agreement shall be
interpreted to make the maximum disclaimer or limitation permitted
by the applicable state law. The invalidity or unenforceability of any
provision of this agreement shall not void the remaining provisions.

1.F.6. INDEMNITY - You agree to indemnify and hold the Foundation,

the trademark owner, any agent or employee of the Foundation,
anyone providing copies of Project Gutenberg™ electronic works in
accordance with this agreement, and any volunteers associated with
the production, promotion and distribution of Project Gutenberg™
electronic works, harmless from all liability, costs and expenses,
including legal fees, that arise directly or indirectly from any of the
following which you do or cause to occur: (a) distribution of this or
any Project Gutenberg™ work, (b) alteration, modification, or
additions or deletions to any Project Gutenberg™ work, and (c) any
Defect you cause.

Section 2. Information about the Mission

of Project Gutenberg™
Project Gutenberg™ is synonymous with the free distribution of
electronic works in formats readable by the widest variety of
computers including obsolete, old, middle-aged and new computers.
It exists because of the efforts of hundreds of volunteers and
donations from people in all walks of life.
Volunteers and financial support to provide volunteers with the
assistance they need are critical to reaching Project Gutenberg™’s
goals and ensuring that the Project Gutenberg™ collection will
remain freely available for generations to come. In 2001, the Project
Gutenberg Literary Archive Foundation was created to provide a
secure and permanent future for Project Gutenberg™ and future
generations. To learn more about the Project Gutenberg Literary
Archive Foundation and how your efforts and donations can help,
see Sections 3 and 4 and the Foundation information page at
www.gutenberg.org.

Section 3. Information about the Project

Gutenberg Literary Archive Foundation
The Project Gutenberg Literary Archive Foundation is a non-profit
501(c)(3) educational corporation organized under the laws of the
state of Mississippi and granted tax exempt status by the Internal
Revenue Service. The Foundation’s EIN or federal tax identification
number is 64-6221541. Contributions to the Project Gutenberg
Literary Archive Foundation are tax deductible to the full extent
permitted by U.S. federal laws and your state’s laws.

The Foundation’s business office is located at 809 North 1500 West,

Salt Lake City, UT 84116, (801) 596-1887. Email contact links and up
to date contact information can be found at the Foundation’s website
and official page at www.gutenberg.org/contact

Section 4. Information about Donations to

the Project Gutenberg Literary Archive
Foundation
Project Gutenberg™ depends upon and cannot survive without
widespread public support and donations to carry out its mission of
increasing the number of public domain and licensed works that can
be freely distributed in machine-readable form accessible by the
widest array of equipment including outdated equipment. Many
small donations ($1 to $5,000) are particularly important to
maintaining tax exempt status with the IRS.

The Foundation is committed to complying with the laws regulating

charities and charitable donations in all 50 states of the United
States. Compliance requirements are not uniform and it takes a
considerable effort, much paperwork and many fees to meet and
keep up with these requirements. We do not solicit donations in
locations where we have not received written confirmation of
compliance. To SEND DONATIONS or determine the status of
compliance for any particular state visit www.gutenberg.org/donate.

While we cannot and do not solicit contributions from states where

we have not met the solicitation requirements, we know of no
prohibition against accepting unsolicited donations from donors in
such states who approach us with offers to donate.

International donations are gratefully accepted, but we cannot make

any statements concerning tax treatment of donations received from
outside the United States. U.S. laws alone swamp our small staff.

Please check the Project Gutenberg web pages for current donation
methods and addresses. Donations are accepted in a number of
other ways including checks, online payments and credit card
donations. To donate, please visit: www.gutenberg.org/donate.

Section 5. General Information About

Project Gutenberg™ electronic works
Professor Michael S. Hart was the originator of the Project
Gutenberg™ concept of a library of electronic works that could be
freely shared with anyone. For forty years, he produced and
distributed Project Gutenberg™ eBooks with only a loose network of
volunteer support.

Project Gutenberg™ eBooks are often created from several printed

editions, all of which are confirmed as not protected by copyright in
the U.S. unless a copyright notice is included. Thus, we do not
necessarily keep eBooks in compliance with any particular paper
edition.

Most people start at our website which has the main PG search
facility: www.gutenberg.org.

This website includes information about Project Gutenberg™,

including how to make donations to the Project Gutenberg Literary
Archive Foundation, how to help produce our new eBooks, and how
to subscribe to our email newsletter to hear about new eBooks.