Shanghai Jiao Tong University

Instructor: Xu Zhao

Computer Vision Class No.: AU7005

Spring 2023

Xu Zhao @ Shanghai Jiao Tong university

Lecture 3: Color
 Contents
❖ Color and color space
๏ Radiometry and photometry

๏ CIE XYZ space

๏ Color constancy

๏ sRGB and beyond

❖ Camera imaging pipeline

๏ A typical color imaging pipeline

๏ White balance etc.

Slides are provided courtesy of ICCV 2019 Tutorial:

“Understanding Color and the In-Camera Image Processing

Pipeline for Computer Vision”

- by Dr. Michael S. Brown @York University - Toronto

 Color

❖ Def Color (noun): The property possessed by an

object of producing different sensations on the eye
as a result of the way it reflects or emits light.

Oxford Dictionary
 Color is perceptual
❖ Color is not a primary physical property on an object
❖ Red, Green, Blue, Pink, Orange, Atomic Tangerine,
etcetcBaby Pink, etc. .
❖ These are words we assign to human color sensations

 Subjective terms to describe color

❖ Hue
❖ Name of the color (yellow,red,
blue,green,...)
❖ Value/Lightness/Brightness
❖ How light or dark a color is.
❖ Saturation/Chroma/Color
Purity
❖ How “strong” or “pure” a
color is. 6

Where do “color sensations” come from?

7
White light through a prism

8
Biology of color sensations

9
 Cones and rods
❖ We have additional light sensitive cells called rods that
are not responsible for color. Rods are used in low-light
vision.
❖ Cone cells are most concentrated around the fovea of
the eye.

10

 Spectral power distribution (SPD)

❖ We rarely see monochromatic
light in real world scenes.
Instead, objects reflect a wide
range of wavelengths. This
can be described by a
spectral power distribution
(SPD) shown right. The SPD
plot shows the relative
amount of each wavelength
reflected over the visible
spectrum.

11
 SPD relation to color is not unique
❖ Due to the accumulation effect of the cones, two different SPDs
can be perceived as the same color (such SPDs are called
"metamers").

12
 Tristimulus color theory
❖ Before the biology of cone cells was understood, it was
empirically known that only three distinct colors (primaries)
could be mixed to produce other colors

❖ Thomas Young (1803), Johann Wolfgang von Goethe (1810),

Hermann Grassman (1853), James Maxwell (1856) all explored
the theory of trichromacy for human vision

13

Tristimulus color theory

14
 Radiometry vs. photometry
❖ Radiometry
❖ Quantitative measurements of radiant energy
❖ Often shown as spectral power distributions (SPD)
❖ Measures either light coming from a source (radiance) or light falling
on a surface (irradiance)
❖ Photometry/ colorimetry
❖ Quantitative measurement of perceived radiant energy based on
human’s sensitivity to light
❖ Perceived in terms of “brightness” (photometry) and color
(colorimetry)

15

 Quantifying color
❖ We still need a way to quantify color & brightness
❖ SPDs go through a “black box” (human visual system) and are perceived
as color
❖ The only way to quantify the “black box” is to perform a human study

16

Experiments for photometry

17
Result of the flicker experiments

18
CIE (1924) Photopic luminosity function

19
Radiometric vs. photometric units

20
 Colorimetry
❖ Based on tristimulus color theory, colorimetry attempts to
quantify all visible colors in terms of a standard set of primaries

21
CIE RGB color matching

22
CIE RGB color matching

23
CIE RGB results

24
CIE RGB results

25
 CIE 1931 XYZ
❖ In 1931, the CIE met and approved defining a new canonical
basis, termed XYZ that would be derived from Wright-Guild’s
CIE RGB data
❖ Properties desired in this conversion:
❖ White point defined at X=1/3, Y=1/3, Z=1/3

❖ Y would be the luminosity function (V(λ))

❖ Quite a bit of freedom in selecting these XYZ basis

❖ In the end, the adopted transform was:

26

CIE 1931 XYZ

27
CIE XYZ 3D plot

28
 Using CIE 1931 XYZ functions
❖ We now have a canonical color space to describe SPDs
❖ Given an SPD, I(λ), we can compute its mapping to the CIE XYZ
space

❖ Given two SPDs, if their CIE XYZ values are equal, then they are
considered the same perceived color,
❖ i.e. I1 (λ), I2 (λ) → (X1, Y1, Z1) = (X2, Y2, Z2) [ perceived as the same

color ]
So . . we can quantitatively describe color!

29



SPD to CIE XYZ example

30
 Usefulness of CIE 1931 XYZ

❖ CIE XYZ space is also considered “device independent” –

the XYZ values are not specific to any device
❖ Electronic devices (e.g. cameras, flatbed, scanners,
printers, displays) can compute mappings of their device
specific values to the corresponding CIE XYZ values.
❖ This provides a canonical space to match between
devices (at least in theory).

31

Luminance-chromaticity space (CIE xyY)

❖ CIE XYZ describes a color in terms of linear combination

of three primaries (XYZ)
❖ Sometimes it is useful to discuss color in terms of
luminance (perceived brightness) and chromaticity (we can
think of as the hue-saturation combined)
❖ CIE xyY space is used for this purpose

32

 Deriving CIE xyY

❖ Project the CIE XYZ values onto the X=1, Y=1, Z=1 plane

33
 CIE xy chromaticity diagram
❖ This gives us the familiar
horseshoe shape of
visible colors as a 2D plot. Note
the axis are x & y.
❖ Point “E” represents
where X=Y=Z have equal
energy (X=0.33, Y=0.33,
Z=0.33) CIE XYZ “white point”

34





What is perhaps most amazing?
❖ 90 years of CIE XYZ and it is all based on the
experiments by the “standard observers”
❖ How many standard observers were used? 100, 500,
1000?

35

CIE XYZ is based on 17 standard observers

36
 Color constancy
❖ In a real scene, an object’s SPD is a combination of the its
reflectance properties and scene illumination
❖ Our visual system is able to compensate for the
illumination

37

 CIE standard illuminants

❖ CIE established several “synthetic” SPDs that serve as
proxies for common real illuminants
❖ Illuminant A: tungsten-filament lighting (i.e. a standard light-bulb)
❖ Illuminant B: noon sunlight
❖ Illuminant C: average daylight
❖ Illuminant D series:represent natural daylight at various color temps
(5000K, 5500K, 6500K), generally denoted as D50, D55, D65
❖ Illuminant E: idea equal-energy illuminant with constant SPD does not
represent any real light source, but similar to D55
❖ Illuminant F series: emulates a variety of fluorescents lamps (12 in total)

38

CIE standard illuminants

39
 Color constancy
❖ Color constancy (also called chromatic adaptation) is the
ability of the human visual system to adapt to scene
illumination
❖ This ability is not perfect, but it works fairly well
❖ Image sensors do not have this ability (it must be
performed as a processing step, i.e. “white balance”)

40

 White point

❖ A white point is a CIE XYZ or CIE xyY value of an ideal

“white target” or “white reference”
❖ This is essentially an illuminants SPD in terms of CIE XYZ/
CIE xyY
❖ We can assume the white reference is reflecting the
illuminant
❖ The idea of chromatic adaptation is to make white points
the same between scenes

41

White point in CIE xy chromaticity

42
 CIE XYZ and RGB
❖ While CIE XYZ is a canonical color space, images/devices
rarely work directly with XYZ
❖ XYZ are not real primaries
❖ RGB primaries dominate the industry
❖ We are all familiar with the RGB color cube

43

Device specific RGB values

44
Trouble with RGB

45
Standard RGB (sRGB) – Rec.709

46
 CIE XYZ to sRGB conversion

❖ D65 is taken as the white-point

❖ This is the linear-sRGB space
❖ sRGB also specifies a gamma correction of the values
❖ The CIE refers this as the Recommendation 709 color
space – or Rec.709
47

 sRGB gamma curve

❖ Actual formula is a bit complicated, but effectively this

is gamma (I’ = 255*I(1/2.2)), where I’ is the output
intensity and I is the linear sRGB ranged 0-1, with a
small linear transfer for linearized sRGB values close to
0 (not shown in this plot).
48
Stevens' power law

49
Standardization: NTSC/PAL

51
CIE XYZ ↔ NTSC/sRGB

52
CIE XYZ: The grandmother of color spaces

53
 CIE LAB space
❖ CIE LAB space (also written as CIE L*a*b*) was
introduced as a perceptually uniform color space
❖ Why?
❖ CIE XYZ provides a means to map between a physical
SPD (radiometric measurement) to a colorimetric
measurement (perceptual)
❖ However, a uniform change in CIE XYZ space does
result in an uniform change in perceived color
difference (see diagram)
❖ CIE Lab transforms CIE to a new space where color (and
brightness) differences are more uniform.
54

CIE LAB space

55
 CIE 1976 LAB
❖ Considering the MacAdam experiments and the Steven's power-law, CIE LAB
was derived in 1976 by applying various transformations to the CIE XYZ values
that result in the following:
❖ L* represents a perceptual brightness measure between 0-100
❖ L* is a non-linear transformation of the Y component of CIE XYZ.
❖ L is approximately a cube root of Y (directly from Steven's power law)
❖ a* and b* (often range ±100)
❖ Both have similar non-linear transformations applied, and represent
approximately:
❖ a* values lying along colors related to red and green
❖ b* values lying along colors related to yellow and blue
❖ a*=b*=0 represents neutral grey colors
❖ NOTE: CIE LAB requires the white-point to be specified for the
transformation. The default white-point is D65.
56

CIE LAB

57
 Y'UV, Y'IQ, Y'CrCb
❖ These spaces are color decompositions that separate the RGB space into a
"brightness-like" component and chrominance (color) components.
❖ The Y in these color spaces are not defined on linear-sRGB or linear- N TSC
❖ They are defined on the gamma encoded sRGB and NTSC color spaces
❖ These Y are referred to as “Luma”, not Luminance
❖ It should be written as Y' but they are typically written as only Y

58

Standard color spaces are great

59
 Integrated signal processor (ISP)
❖ You will hear the term "ISP" associated with camera pipelines

❖ An ISP is dedicated hardware used to process the sensor image to produce the
final output (JPEG image) that is saved on your device

❖ The ISP is usually integrated as part of a system on a chip (SoC) that has other
modules
❖ Companies such as Qualcomm, HiSilicon, Intel (and more) sell ISP chips
❖ An ISP can be customized by the customer (Samsung, Huawei, LG, Apple,
etc)
❖ Note that it is also possible to perform operations common on an ISP on your
device's CPU and GPU

60






A typical color imaging pipeline

❖ This diagram represents the steps applied on a typical consumer camera

pipeline. ISPs may apply these steps in a different order or combine them in
various ways.
61
 Camera RGB sensitivity

❖ The color filter array (CFA) on the camera filters the light into three sensor-
specific RGB primaries
62
Sensor raw-RGB image

spectral response

63
 Sensors are linear to irradiance
❖ Camera sensors are decent light measuring devices
❖ If you double the amount of light hitting a sensor's pixel, the
digital value output of that pixel will double

64

IMPORTANT: raw-RGB sensor images are not in a standard

color space

65
 Displaying raw-RGB images
❖ Inserting a raw-RGB image in your slides, research paper, etc
will result in strange colors.
❖ Why? Our devices (computers, printers, etc) expect the
image to be in a standard color space like sRGB.

66

A typical color imaging pipeline

❖ This diagram represents the steps applied on a typical consumer camera

pipeline. ISPs may apply these steps in a different order or combine them in
various ways.
67
 ISO signal amplification (gain)
❖ These Imaging sensor signal is amplified and digitized
❖ Amplification to assist A/D conversion
❖ Need to get the voltage to the range required to the desired digital output
❖ This gain is used to accommodate camera ISO settings
❖ Gain to signal applied on sensor
❖ Note – gaining the signal also gains image noise

68

A typical color imaging pipeline

❖ This diagram represents the steps applied on a typical consumer camera

pipeline. ISPs may apply these steps in a different order or combine them in
various ways.
69
Color mapping/colorimetric stage
❖ This step in the IPS converts the sensor raw-RGB values to a device
independent color space

70
 Two step procedure
(1) Apply a white-balance correction to the raw-RGB values
(2) Map the white-balanced raw-RGB values to CIE XYZ

71

How does white balance (WB) work?

72
White balance (computational color constancy)

❖ The challenging part for white-balance is determining the

proper white-balance setting!
❖ Users can manually set the white balance
❖ Camera specific white-balance matrices for common
illuminations
❖ These can be manually selected by the user
❖ Otherwise auto white balance (AWB) is performed
❖ In computer vision, we often refer to AWB as "illumination
estimation”
❖ Since the hard part is trying to determine what the
illumination in the scene is.
73

WB manual settings

74
Examples of manual WB matrices

75
AWB is not easy

76
 Color manipulation
❖ This is the stage were a camera applies its "secret sauce" to
make the images look good
❖ This procedure can be called by many names:
❖ Color manipulation
❖ Photo-finishing
❖ Color rendering or selective color rendering
❖ Yuv processing engine
❖ DSLR will often allow the user to select various photo-
finishing styles
❖ Smartphones often compute this per-image
❖ Photo-finishing may also tied to geographical regions!
77

A typical color imaging pipeline

❖ This diagram represents the steps applied on a typical consumer camera

pipeline. ISPs may apply these steps in a different order or combine them in
various ways.
79
 Exif metadata
❖ Exchangeable image file format (Exif)
❖ Created by the Japan Electronics and Information Technology Industries
Association (JEITA)
❖ Associates meta data with images
❖ Date/time

❖ Camera settings (basic)

❖ Image size, aperture, shutter speed, focal length, ISO speed,

metering mode (how exposure was estimated)
❖ Additional info (from in some Exif files)

❖ White-balance settings, even matrix coefficients of white-balnace

❖ Picture style (e.g. landscape, vivid, standard, portrait)

❖ Output color space (e.g. sRGB, Adobe RGB, RAW)

❖ GPS info

❖ More...

80

sRGB/JPEG is slowly being replaced

❖ sRGB was developed for monitors in
the 1990s – it is an old standard.
❖ High Efficient Image Encoding (HEIC)
❖ Better compression than JPEG

❖ Apple iPhone has started to use HEIC

to replace JPEG
❖ HEIC supports multiple color spaces.
Apple uses Display P3 – a variation on
a Digital Cinema Initiative P3 space.
❖ The P3 gamut is 25% wider than sRGB
❖ There is also a gamma encoding similar
to sRGB.
❖ Pixel 4 and other Android devices are
will support this color space soon.

81