CMPE 403

Fundamentals of Image Processing

Burcu KIR SAVAŞ, PhD.

Image types and color

 Course Overview
– Introduction
– What is image processing?
– Image types and color
– Point operations
– Basic Density Conversion Functions
– Histogram Synchronizationg
– Frequency Domain Techniques
Midterm exam
– Wavelets
– Gradients, edges, contours
– Image segmentation
– Image smoothing
image formation
• What determines the brightness of an image pixel?
Light source
properties
Surface
Sensor characteristics shape and orientation

Exposure

Surface reflectance
Optics properties
Slide credit: L. Fei-Fei
digital camera

A digital camera replaces film with a sensor

array

• Each cell in the array is light-sensitive

diode that converts photons to
electrons
• http://electronics.howstuffworks.com/digital-
camera.htm

Slide credit: S. Seitz

digital images

Slide credit: D. Hoiem

 digital images
• Sample the 2D space on a regular grid
• Quantize each sample (round to nearest integer)

• Image thus represented as a matrix of integer values.

2D

1D

Slide credit: K. Grauman, S. Seitz

image representation
• Digital image: 2D discrete function f
• Pixel:The raster
Smallest image
element (pixel
of an image matrix)
f(x,y)
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93

Slide credit: M. J. Black

TODAY

• Perception of color and light

• Color spaces
Why does a visual system need color?
 Why does a visual system need
color?

• To tell food and mates.

• To distinguish material changes from shading changes.
• To group parts of one object together in a scene.
• A persons appearance looks normal/healthy.

Slide credit: W. Freeman

 What is color?
• Colour, also spelled, Color is the result of interaction
between physical light in the environment and our visual
system
• Color is a psychological property of our visual experiences
when we look at objects and lights,
not a physical property of those objects or lights
(S. Palmer, Vision Science: Photons to Phenomenology)
 #thedress
• What is the color of the
dress?

• blue and black

• white and gold
• blue and brown

• What #thedress tell about

our color perception?

11
 #thedress
• Let’s take averages

two pieces averages basic pattern

of the dress

12
 #thedress
• The dress in the photograph

13
 #thedress
• Consider the dress is in shadow.

• Your brain remove the blue cast,

and perceive it as white and gold.
14
 #thedress
• The dress in the photograph

15
 #thedress
• Consider the dress is in bright light.

• Your brain perceive the dress as a

darker blue and black

16
 #thedress
• Answer:

• The dress is actually blue and black.

17
 Brightness perception

Consider two pieces of paper, one

black and one white.

Let's say we were outside in bright

sunlight (instead of being in this dingy
lecture hall).

What color do you imagine they would

look like?

Still black and white.

Edward Adelson
 Brightness perception

The image of the black paper

outdoors is actually more
intense than the image of white
paper indoors.

Why does the black paper

outdoors still look black even
though it is physically more
intense?

Edward Adelson
Brightness perception

21
Land’s Experiment (1959)

• Cover all patches except a blue rectangle

• Make it look gray by changing illumination
• Uncover the other patches
Color Constancy
We filter out illumination variations
Slide credit: S. Narasimhan
Land’s Experiment (1959)

• Cover all patches except a blue rectangle

• Make it look gray by changing illumination
• Uncover the other patches
Color Constancy
We filter out illumination variations
Slide credit: S. Narasimhan
 Color Cube Illusion

Content © 2008 R.Beau Lotto

 Color perception

Content © 2008 R.Beau Lotto

 Color perception

Content © 2008 R.Beau Lotto

 Color perception

Content © 2008 R.Beau Lotto

Image Brightness (Intensity)
Scene
e' (l ) Irradiance

b' q (l )
r ' (l )
Image p ' (l ) Scene
Intensity Reflectance

• Monochromatic Light : (l = li )
b' ( x, y) = r ' ( x, y) e' ( x, y) q(li ) = 1

NOTE: The analysis can be applied to COLORED LIGHT using FILTERS

Slide credit: S. Narasimhan
 Color and light
• Color of light arriving at camera depends on
– Spectral reflectance of the surface light is leaving
– Spectral radiance of light falling on that patch

• Color perceived depends on

– Physics of light
– Visual system receptors
– Brain processing, environment

• Color is a phenomenon of human perception;

• it is not a universal property of light
Slide credit: K. Grauman, S. Marschner
Color
White light: composed of
about equal energy in all
wavelengths of the visible
spectrum Colo
r

Newton 1665
Slide credit: B. Freeman, A. Torralba, K. Grauman
Electromagnetic spectrum
• Light is electromagnetic radiation
– exists as oscillations of different frequency (or, wavelength)

Human Luminance Sensitivity Function

Slide credit: A. Efros

 The Physics of light
Any source of light can be completely described
physically by its spectrum: the amount of energy emitted
(per time unit) at each wavelength 400 - 700 nm.

Relative
# Photons
spectral
(per power
ms.)

400 500 600 700

Wavelength (nm.)
Slide credit: A. Efros 36
© Stephen E. Palmer, 2002
 The Physics of light
Some examples of the spectra of light sources
.

A. Ruby Laser B. Gallium Phosphide Crystal

power
# Photons

# Photons
power

Rel.
Rel.

400 500 600 700 400 500 600 700

Wavelength (nm.) Wavelength (nm.)

C. Tungsten Lightbulb D. Normal Daylight

power
power
# Photons

Photons
#Rel.
Rel.

400 500 600 700 400 500 600 700

Slide credit: A. Efros © Stephen E. Palmer, 2002
 The Physics of light

Some examples of the reflectance spectra of surfaces

% Light Reflected

Red Yellow Blue Purple

400 700 400 700 400 700 400 700

Slide credit: A. Efros Wavelength (nm) © Stephen E. Palmer, 2002
Image formation
• What determines the brightness of an image pixel?
Light source
properties
Surface
Sensor characteristics shape and orientation

Exposure

Surface reflectance
Optics properties
Slide credit: L. Fei-Fei
Color mixing
Cartoon spectra for color names:

Credit: W. Freeman
 Additive color mixing

Colors combine by adding color spectra

Mixing the three primaries or a secondary with its opposite primary
color produces white light.

Light adds to black.

Credit: W. Freeman
Subtractive color mixing

Colors combine by multiplying color spectra.

Pigments remove color

from incident light
(white).

Credit: W. Freeman
Interaction of light and surfaces
• Reflected color is the
result of interaction of
light source spectrum
with surface
reflectance

Slide credit: A. Efros

 Reflection from colored surface
[Stone 2003]

Slide credit: S. Marschner

https://www.dkfindout.com/us/science/light/seeing-color/
The Eye

• Iris - colored annulus with radial muscles

• Pupil - the hole (aperture) whose size is controlled by the iris
• Lens - changes shape by using ciliary muscles (to focus on
objects at different distances)
• Retina - photoreceptor cells
Slide credit: S. Seitz
 The eye as a measurement device
• We can model the low-level
behavior of the eye by thinking
of it as a light-measuring machine
– its optics are much like a camera
– its detection mechanism is also
much like a camera
• Light is measured by the
photoreceptors in the retina
– they respond to visible light
– different types respond to different
[Greger et al. 1995]

wavelengths

• The human eye is a camera!

Slide credit: S. Marschner

 Layers of the retina

Slide credit: S. Ullman

Eye Movements
• Saccades
– Can be consciously controlled. Related to perceptual
attention.
– 200ms to initiation, 20 to 200ms to carry out. Large
amplitude.
• Microsaccades
– Involuntary. Smaller amplitude. Especially evident during
prolonged fixation. Function debated.
• Smooth pursuit – tracking an object

Slide credit: J. Hays

 Receptors Density - Fovea

The fovea itself is the central portion of the macula, which is

responsible for central vision. A large proportion of the
striate cortex is devoted to processing information from the
fovea.

Slide credit: S. Ullman

Human Photoreceptors

Images: Foundations of Vision,

by Brian Wandell, Sinauer Assoc., 1995 Slide Credit: B. Freeman and A. Torralba
Human eye photoreceptor spectral
sensitivities

Images: Foundations of Vision,

by Brian Wandell, Sinauer Assoc., 1995

Slide Credit: B. Freeman and A. Torralba

 Two types of light-sensitive receptors

Cones cone-shaped less sensitive

operate in high light color vision

Rods rod-shaped highly sensitive

operate at night gray-scale vision

Rods are responsible for intensity, cones for color perception

Rods and cones are non-uniformly distributed on the retina

Images by Shimon Ullman

Slide credit: A. Efros
Rod / Cone sensitivity
 Physiology of Color Vision
.

Three kinds of cones:

RELATIVE ABSORBANCE (%) 440 530 560 nm.

100
S M L

50

400 450 500 550 600 650

WAVELENGTH (nm.)

• Ratio of L to M to S cones: approx. 10:5:1

• Almost no S cones in the center of the fovea
Slide credit: A. Efros © Stephen E. Palmer, 2002
 Color perception
M L
Power

Wavelength
Rods and cones act as filters on
the spectrum
Q: How can we represent an
– To get the output of a filter,
entire spectrum with 3
multiply its response curve by
the spectrum, integrate over all numbers?A: We can’t! Most of
wavelengths the information is lost.
• Each cone yields one number

Slide credit: S. Seitz

 Digital images
• Sample the 2D space on a regular grid
• Quantize each sample (round to nearest integer)

• Image thus represented as a matrix of integer values.

2D

1D

Slide credit: K. Grauman, S. Seitz

 Color Images: Bayer Grid

• Estimate RGB
at G cells from
neighboring values

http://www.cooldictionary.com/
words/Bayer-filter.wikipedia
Slide credit: S. Seitz
Digital color images

Color images, RGB

color space

R G B
Slide credit: K. Grauman
Color spaces: RGB
• Single wavelength primaries
• makes a particular monitor RGB standard
• Good for devices (e.g., phosphors for monitor), but not
for perception
RGB color matching functions

Slide credit: K. Grauman, S. Marschner

 Color spaces: RGB
Default color space
0,1,0

R
(G=0,B=0)

1,0,0 G
(R=0,B=0)

0,0,1

Some drawbacks B
(R=0,G=0)
• Strongly correlated channels
• Non-perceptual
Image from: http://en.wikipedia.org/wiki/File:RGB_color_solid_cube.png Slide credit: D. Hoiem
Color spaces: CIE XYZ
• Standardized by CIE (Commission Internationale de
l Eclairage, the standards organization for color science)
• Based on three imaginary primaries X, Y, and Z
– imaginary = only realizable by spectra that are negative at
some wavelengths
– separates out luminance: X, Z have zero luminance, so Y tells you
the luminance by itself
CIE XYZ Color matching functions

Slide credit: K. Grauman, S. Marschner

Color spaces: CIE XYZ
• Standardized by CIE (Commission Internationale de
l Eclairage, the standards organization for color science)
• Based on three imaginary primaries X, Y, and Z
– imaginary = only realizable by spectra that are negative at
some wavelengths
– separates out luminance: X, Z have zero luminance, so Y tells you
the luminance by itself

Slide credit: K. Grauman, S. Marschner

Perceptually organized color spaces
• Artists often refer to colors as tints, shades, and tones
of pure pigments
– tint: mixture with white
– shade: mixture with black tints
pure
– tones: mixture with white color
black and white

[after Foley et al.]

– gray: no color at all grays
(aka. neutral) shades
black
• This seems intuitive
– tints and shades are inherently related to the pure color
• same color but lighter, darker, paler, etc.

Slide credit: S. Marschner

Perceptual dimensions of color
• Hue
– the kind of color, regardless of attributes
– colorimetric correlate: dominant wavelength
– artist s correlate: the chosen pigment color
• Saturation
– the colorfulness
– colorimetric correlate: purity
– artist s correlate: fraction of paint from the colored tube
• Lightness (or value)
– the overall amount of light
– colorimetric correlate: luminance
– artist s correlate: tints are lighter, shades are darker

Slide credit: S. Marschner

Color spaces: HSV
• Hue, Saturation, Value
• Nonlinear – reflects topology of colors by coding
hue as an angle

Image from mathworks.com Slide credit: K. Grauman

Color spaces: HSV
Intuitive color space

H
(S=1,V=1)

S
(H=1,V=1)

V
(H=1,S=0)

Slide credit: D. Hoiem

 Color spaces: YCbCr
Fast to compute, good for
compression, used by TV
Y=0 Y=0.5
Y
(Cb=0.5,Cr=0.5)

Cr

Cb
Cb (Y=0.5,Cr=0.5)

Y=1

Cr
(Y=0.5,Cb=05)

Slide credit: D. Hoiem

 Color spaces: YCbCr
Fast to compute, good for
compression, used by TV
Y=0 Y=0.5
Y
(Cb=0.5,Cr=0.5)

Cr

Cb
Cb (Y=0.5,Cr=0.5)

Y=1

Cr
(Y=0.5,Cb=05)

Slide credit: D. Hoiem

Distances in color space

• Are distances between points in a color space

perceptually meaningful?

Slide credit: K. Grauman

 Perceptually uniform spaces
• Two major spaces standardized by CIE
– designed so that equal differences in
coordinates produce equally visible
differences in color CIE XYZ
– by remapping color space so that just-
noticeable differences are contained by
circlesà distances more perceptually
meaningful.
– LUV: earlier, simpler space; L*, u*, v*
– LAB: more complex but more uniform:
L*, a*, b*
– both separate luminance from CIE u’v’
chromaticity
– including a gamma-like nonlinear
component is important
Slide credit: K. Grauman, S. Marschner
Color spaces: L*a*b*
“Perceptually uniform”* color space

L
(a=0,b=0)

a
(L=65,b=0)

b
(L=65,a=0)

Slide credit: D. Hoiem

Color spaces: L*a*b*
“Perceptually uniform”* color space

L
(a=0,b=0)

a
(L=65,b=0)

b
(L=65,a=0)

: measured white point

Slide credit: D. Hoiem
Most information in intensity

Only intensity shown – constant color

Slide credit: D. Hoiem
Most information in intensity

Original image
Slide credit: D. Hoiem
Back to grayscale intensity
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93

Slide credit: D. Hoiem

Today
• Perception of color and light
• Color spaces
Next week
• Point operations
• Histogram processing
Reading Assignment
• Watch Beau Lotto’s TED talk on “Optical illusions
show how we see”
• Prepare a 1-page summary of the talk
• Due on 17st of October
Programming assignment
• Colorizing the Prokudin-Gorskii photo collection
• A warm-up exercise
• Main steps:
1. Divide the input image into three equal parts
corresponding to RGB channels.
2. Align the second and the third parts (G and R channels)
to the first one (B channel).
Prokudin-Gorskii's Russia in Color
• Russia circa 1900
• One camera, move the film with filters to get 3 exposures

Slide credit: F. Durand

Prokudin-Gorskii's Russia in Color
• Digital restoration

Slide credit: F. Durand

Emir Seyyid Mir Mohammed Alim Khan, the Emir of Bukhara, ca. 1910.
Self-portrait on the Karolitskhali River, ca. 1910.
A metal truss bridge on stone piers, part of the Trans-Siberian Railway,
crossing the Kama River near Perm, Ural Mountains Region, ca. 1910.
On the Sim River, a shepherd boy, ca. 1910.
Peasants harvesting hay in 1909. From the album "Views along the
Mariinskii Canal and river system, Russian Empire", ca. 1910.