Computer Vision

CSC-455
Today’s Lecture

 Introduction to Computer Vision

 Concepts behind Computer based
Recognition and feature Extraction.
 Applications of Computer Vision
 Overview of Early, Intermediate
and High Level Vision.
What is Computer Vision?

Field of computer science that works on enabling

computers to see, identify and process images in the
same way that human vision does, and then provide
appropriate output.
Textbook and Reference Books
Marks Distribution

5 Quizzes* (Zero Marks for Cheating)

5 Assignments** (MATLAB Programming) (Zero Marks for

Plagiarism)

Project (5 Marks)

Sessional, Terminal.
• *Quizzes will be conducted on 2nd Class of Week (3rd Week, 5th Week, 9th Week, 11th Week, 14th Week of

Course)
• **Class Participation Marks (Questions and Review)
Why study computer vision?

Vision is useful: Images and video are everywhere!

Personal photo albums Movies, news, sports

Surveillance and security Medical and scientific images

Concepts behind Computer based Vision
and Feature Extraction
To perceive the story behind the picture

What we see What a computer sees

Source: S. Narasimhan
Vision as measurement device

Real-time stereo Structure from motion Multi-view stereo

NASA Mars Rover

Pollefeys et al. Goesele et al.

 Object Recognition
sky

building

flag

face
banner
wall
street lamp
bus bus
cars
slide credit: Fei-Fei, Fergus & Torralba
Scene and context categorization
• outdoor
• city
• traffic
•…

slide credit: Fei-Fei, Fergus & Torralba

Features based Training

• Quantize local features into a “vocabulary”, describe

images as histograms of “visual words”, train classifiers to
recognize scenes based on these histograms.

J
Key Stages
Image Morphological
Restoration Processing

Image Segmentation
Enhancement

Image Representation
Acquisition & Description

Object
Problem Domain Recognition
Colour Image Image
Processing Compression
Key Stages
Image Morphological
Restoration Processing

Image Segmentation
Enhancement

Image Representation
Acquisition & Description

Object
Problem Domain Recognition
Colour Image Image
Processing Compression
Key Stages
Image Morphological
Restoration Processing

Image Segmentation
Enhancement

Image Representation
Acquisition & Description

Object
Problem Domain Recognition
Colour Image Image
Processing Compression
Key Stages
Image Morphological
Restoration Processing

Image Segmentation
Enhancement

Image Representation
Acquisition & Description

Object
Problem Domain Recognition
Colour Image Image
Processing Compression
Key Stages
Image Morphological
Restoration Processing

Image Segmentation
Enhancement

Image Representation
Acquisition & Description

Object
Problem Domain Recognition
Colour Image Image
Processing Compression
Key Stages
Image Morphological
Restoration Processing

Image Segmentation
Enhancement

Image Representation
Acquisition & Description

Object
Problem Domain Recognition
Colour Image Image
Processing Compression
Key Stages
Image Morphological
Restoration Processing

Image Segmentation
Enhancement

Image Representation
Acquisition & Description

Object
Problem Domain Recognition
Colour Image Image
Processing Compression
Key Stages
Image Morphological
Restoration Processing

Image Segmentation
Enhancement

Image Representation
Acquisition & Description

Object
Problem Domain Recognition
Colour Image Image
Processing Compression
Key Stages
Image Morphological
Restoration Processing

Image Segmentation
Enhancement

Image Representation
Acquisition & Description

Object
Problem Domain Recognition
Colour Image Image
Processing Compression
Key Stages
Image Morphological
Restoration Processing

Image Segmentation
Enhancement

Image Representation
Acquisition & Description

Object
Problem Domain Recognition
Colour Image Image
Processing Compression
Course Outline

 From Image Processing to Computer Vision:

 low, mid and high-level processes Before 2nd Sessional

Before Terminal

Low Level Mid Level High Level Process

Input: Attributes/Image
Process Input: Process Input: Output: Understanding
Image Image
Examples: Scene
Output: Image Output: Attributes understanding, Object
Examples: Hough recognition, kNN/CNN.
Examples: Noise Transform,
removal, image Segmentation,
sharpening, Split Multiview Geometry
& Merge, & Recognition
Image Processing Computer Vision
Histogram.
Why study Computer Vision?

 Fast-growing collection of useful applications

 building representations of the 3D world
from pictures
 automated surveillance (who’s doing what)
 movie post-processing
 face finding
 Various deep and attractive scientific mysteries
 how does object recognition work?
 Greater understanding of human vision
Applications of Computer Vision
Interdisciplinary Uses
Optical character recognition (OCR)

Technology to convert scanned docs to text

• If you have a scanner, it probably came with OCR software

Digit recognition, AT&T labs

http://www.research.att.com/~yann/ License plate readers

Source: S. Seitz
Face detection and smile detection

• Many new digital cameras now detect faces and smile.

• Snapchat face filters

Source: S. Seitz
Face tracking

Source: S. Seitz
Face recognition

Who is She?

Source: S. Seitz
Vision-based biometrics

Source: S. Seitz
Login without a password

Fingerprint scanners on many new

laptops, other devices

Source: S. Seitz
Object recognition (in mobile phones)
Word
e.g., Google Lens Leafsnap.com Lens.com
Face makeovers (Website)
Special effects: shape capture

The Matrix movies, ESC Entertainment, XYZRGB, NRC

Pirates of the Carribean, Industrial Light and Magic

Source: S. Seitz
Smart cars

Source: S. Seitz Slide content courtesy of Amnon Shashua

Robotics

NASA’s Mars Spirit Rover http://www.robocup.org/

http://en.wikipedia.org/wiki/Spirit_rover

Source: S. Seitz
Examples: Sign Language/Gesture Recognition

Gesture Recognition

British Sign Language Alphabet

Facial Expression
Augmented Reality and Virtual Reality

MS HoloLens, Oculus, Magic Leap,

ARCore / ARKit
Interactive Games: Kinect
Beyond Visible (i.e. Thermal Imaging)

• Gamma-ray and X-ray: medical and astronomical

applications
• Infrared (thermal imaging): near-infrared and far-infrared
• Microwave imaging:
• Radio-frequency: MRI and astronomic applications
Operate in infrared
frequency
Summary of Applications
Problem Domain Application Input Pattern Output Class

Document Image Optical Character Document Image Characters/words

Analysis Recognition
Document Internet search Text Document Semantic categories
Classification
Document Junk mail filtering Email Junk/Non-Junk
Classification
Multimedia retrieval Internet search Video clip Video genres

Speech Recognition Telephone directory Speech waveform Spoken words

assistance
Natural Language Information extraction Sentence Parts of Speech
Processing
Biometric Recognition Personal identification Face, finger print, Iris Authorized users for
access control
Medical Computer aided Microscopic Image Healthy/cancerous cell
diagnosis
Military Automatic target Infrared image Target type
recognition
Industrial automation Fruit sorting Images taken on Grade of quality
conveyor belt
Bioinformatics Sequence analysis DNA sequence Known types of genes
Why study Computer Vision?

 Vision is useful
 Vision is interesting
 Vision is difficult
 Half of primate cerebral cortex is devoted to
visual processing
 Achieving human-level visual perception is
probably “AI-complete”
 Challenges or Opportunities: Images are
confusing, but they also reveal the structure of the
world through numerous cues
 Our job is to interpret the cues!
Challenges in Computer Vision
Challenges: viewpoint variation

Michelangelo 1475-1564 slide credit: Fei-Fei, Fergus & Torralba

Challenges: illumination

image credit: J. Koenderink

Challenges: Scale

slide credit: Fei-Fei, Fergus & Torralba

Challenges: Motion
Challenges: object intra-class variation

slide credit: Fei-Fei, Fergus & Torralba

Category Specific Object Reconstruction
Kar, Tulisiani, Carreira & Malik
Basis Shape Models
Depth cues: Linear perspective
Depth cues: Aerial perspective
Shape cues: Texture gradient
Grouping cues: Similarity (color, texture, proximity)
 Bottom line

Perception is an inherently ambiguous problem

– Many different 3D scenes could have given rise to a particular 2D picture

Possible solutions
– Bring in more constraints (more images)
– Use prior knowledge about the structure of the world
Need a combination of different methods
Image source: F. Durand
Course overview

I. Early vision: Image formation and processing

II. Mid-level vision: Grouping and fitting
III. Multi-view geometry
IV. Recognition
V. Advanced topics
I. Early vision

Basic image formation and processing

Cameras and sensors

Light and color

* =
Linear filtering
Edge detection
Feature extraction: corner and blob detection
 II. Mid-level vision

Fitting and grouping

Fitting: Least squares

Hough transform RANSAC
Alignment
III. Multi-view geometry

Stereo Epipolar geometry

Tomasi & Kanade (1993)

Affine structure from motion Projective structure from motion

IV. Recognition

Patch description and matching Clustering and visual vocabularies

Bag-of-features models
Classification

Sources: D. Lowe, L. Fei-Fei

V. Advanced Topics

Segmentation Face detection

Articulated models Motion and tracking

Self Study & Tasks

• MATLAB Installation
• MATLAB tutorial
References
 Some Slide material has been taken from Dr M. Usman Akram Computer Vision
Lectures
 CSCI 1430: Introduction to Computer Vision by James Tompkin
 Statistical Pattern Recognition: A Review – A.K Jain et al., PAMI (22) 2000
 Pattern Recognition and Analysis Course – A.K. Jain, MSU
 Pattern Classification” by Duda et al., John Wiley & Sons.
 Digital Image Processing”, Rafael C. Gonzalez & Richard E. Woods, Addison-Wesley,
2002
 Machine Vision: Automated Visual Inspection and Robot Vision”, David Vernon,
Prentice Hall, 1991
 www.eu.aibo.com/
 Advances in Human Computer Interaction, Shane Pinder, InTech, Austria, October
2008
 Computer Vision A modern Approach by Frosyth
 http://www.cs.cmu.edu/~16385/s18/