Optical architectures and technologies for

AR, VR and MR HMDs.

Part 2/2

Short course
SC_1218

Bernard Kress
Partner Optical Architect
Microsoft HoloLens

[email protected]

Slide # 1
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Some of the challenges paving the road down to
the ultimate MR experience:

1) FOV
2) Angular resolution
3) Weight/ size and CG
4) Eyebox
5) Non conflicting 3D cues
6) Vision impairment correction
7) Immersion experience
8) Sensor fusion

Slide # 2
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Some of the challenges paving the road down to
the ultimate MR experience:

1) FOV
2) Angular resolution
3) Weight/ size and CG
4) Eyebox “Human centric”
5) Non conflicting 3D cues
6) Vision impairment correction
optical design
7) Immersion experience
8) Sensor fusion

Slide # 3
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Some of the challenges paving the road down to
the ultimate MR experience:

1) FOV
2) Angular resolution
3) Weight/ size and CG
4) Eyebox
5) Non conflicting 3D cues
6) Vision impairment correction
7) Immersion experience
8) Sensor fusion

Slide # 4
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 The human visual system is complex and limited:
FOV
Binocular vision
(120°)

Color discrimination
(60°)
Limit of visual field
Symbol rec. (+50° / -70°)
(40°)
Eye rotation range
Text rec.
(+25° / -30°)
(20°)
Fovea (3°)

Unconstrained eye rotation range

(+15° / -20°)

Monocular peripheral vision

(220°)

Slide # 5
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 The human visual system is complex and limited:
Binocular FOV overlap and stereo fusion

90 deg
Binocular field
75 deg

60 deg Fixed foveated region

(40-50 deg full angle)
45 deg

30 deg Right field

Left field
15 deg

45 deg Fovea
(3 deg full angle)
60 deg
75 deg
90 deg
Parafovea

Slide # 6
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 The combination of unaltered see through FOV and display FOV create the MR immersion
experience.

Oculus CV1 VR Google Glass Magic Leap One HoloLens V2

No see through 220 deg See through 70 deg see through 220 deg see through
Display D-FOV: 110 deg 15 deg display D-FOV 50 deg display D-FOV 52 deg display D-FOV

The perceived FOV for the user is subjective, and related to the quality of the virtual image.

Slide # 7
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Some of the challenges paving the road down to
the ultimate MR experience:

1) FOV
2) Angular resolution
3) Weight/ size and CG
4) Eyebox
5) Non conflicting 3D cues
6) Vision impairment correction
7) Immersion experience
8) Sensor fusion

Slide # 8
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Display Peripheral vision = all areas expect the Macula
Average eye resolution in arcmin

Macula
Macula

14

Angular Resolution (arcmin)

12

10

85° 60° 30° 10° 10° 30° 60° 8

Blind spot 4
Fovea
2
Macula
Paracentral 0 5.0 10.0 15.0 20.0 25.0 30.0
area
Left eye Right eye Viewing angle from center (+/- deg)

Slide # 9
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Human centric MTF optimization

MTF for three different fields as a function of cycles per +/-1/8th D through focus MTF for the same 3 fields at a
mm at a specific focus specific frequency (cycles per mm)
MTF (%)

MTF (%)
+/-1/8D

Cycles/mm Focus (mm)

Slide # 10
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Color perception in human eye

Slide # 11
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 12
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 LCA in human visual system

LCA as ray traced in eye Image on retina Image “seen” by viewer

Longitudinal eye chromatism
Blue Green Red
Longitudinal chromatism (Diopters)

1.0 Blue Green Red

0.0

2.0 D
Focus location

-1.0 Red
Red
Green

Off-axis direction
Blue
Green
-2.0

Blue
0.4 0.5 0.6 0.7
Wavelength (microns)

Slide # 13
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 1.0
Mostly photopic vision
Modulation Transfer Function (MTF) 2mm
2.8mm
0.8 3.8mm 1 arcmin angular resolution or 60 pixels per deg
5.8mm (PPD) … or 20/20 vision

0.6
Reversal point for high spatial frequencies in high
brightness
0.4

30% MTF level

0.2

0.0
0 10 20 30 40
50 Spatial frequency (cycles per degree)

Slide # 14
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 High DPI Displays: Do We Really Need Them?
Curved, foldable, bendable?

The Galaxy S7 has a 5.1” display, a resolution of 2560×1440 which leads to 576 PPI in density, for 3.7M pixels
The Huawei P9 has a 5.2” display, a resolution of 1920×1080 which leads to 425 PPI in density, for 2M pixels
The iPhone 6s has a 4.7” display, a resolution of 1334×750 which leads to 326 PPI in density, for 1M pixels

Slide # 15
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 16
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Foveated rendering (software foveation)
The human eye as two distinct vision systems:
1) Foveal vision (sharp and detailed)
2) Peripheral vision (lacks fidelity)

The difference in characteristics of the two systems enable

foveated rendering, which seek to increase rendering
performance by lowering image quality in the periphery.

Slide # 17
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Unfoveated scene
Slide # 18
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Foveated scene, gaze is on clock
Slide # 19
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Optical foveation is the key to human high resolution “experience”

+ +

Entire FOV is low No foveation, but Gaze contingent

Fixed central foveated Gaze contingent optical foveation
resolution entire FOV is high foveated rendering on a Fixed central optical foveation on 2
rendering on a single on 2 displays (high/low res)
resolution single display displays (high/low res)
display

GPU rendering Number of pixels required Perceived resolution by human eye

Slide # 20
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Some of the challenges paving the road down to
the ultimate MR experience:

1) FOV
2) Angular resolution
3) Weight/ size and CG
4) Eyebox
5) Non conflicting 3D cues
6) Vision impairment correction
7) Immersion experience
8) Sensor fusion

Slide # 21
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Optical backbone of an MR headset

Slide # 22
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Gaze tracking

6DOF Head
tracking

Gesture Display Imaging Combiner

Exit pupil
sensing Engine optics Optics
expansion

Depth mapping

Slide # 23
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Mechanicals ,thermals,…

Driving electronics:
GPU, SOC, connectivity, power,…
Optical systems

Display Combiner
engine Sensors
engine
Visor, Rx

Slide # 24
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Display tech: Panel displays

Emission
cone
angles
Tuned by
Tuned by backlight illumination NA Tuned by Tuned by
Lambertian illumination NA illumination NA Lambertian Tuned by geometry

DLP MEMS mu-OLED mu-iLED

LTPS- LCD or IPS-LCD Flat or curved AMOLED panel HTPS LCD LCoS
reflective panel microdisplay panel microdisplay panel
panel with backlight (can be transparent) Microdisplay Panel microdisplay panel on
on Si backplane or on Si backplane
(transparent) (transparent) Si backplane
glass/plastic sheet or glass (LPTS)

Micro-display panels
Panel displays (smart phone type)

Require Silicon Backplane

Require illumination engine Emissive displays

Slide # 25
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Example of amplitude panel displays

0.75” LCoS Si backplanes 2K muOLED panel iLED arrays on 4 inch

on 8 inch wafer (Kopin) wafers

Single 5.5 inch LCD panel Dual 3.8inch OLED panels

0.75” DLP imager panel (Ti) ½” 1080p phase LCoS panel

0.35” amplitude nHD LCoS (HoloEye)
panels

Slide # 26
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Display Brightness at panel 5M nits

Size of display engine

2-3M nits
FS: Free Space illumination
BL: Back-Lit illumination
FT: Front-Lit illumination 1.5M nits
LBS: Laser Beam Scanner

700 nits

>500K nits
Min diffractive
waveguide combiner 300K nits

40K nits
Min geometric
waveguide combiner
5K nits

Min free space 1-2K nits

combiner (also VR)

Mu-OLED HTPS BL LCD FT LCOS BL LCOS Mu-iLED BL DLP FS LCOS LBS FS DLP

Slide # 27
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 1 m2 area
1 m radius

Illuminance Luminance
1 ft radius 1 sr solid angle

1 ft2 area

Luminous flux
(Lumens) Candela/m2
Luminous
intensity
(Candelas)
1 candela

1 Lumen/ft2
(1 fc or FootCandel) Lumens/m2

1 Lumen/sr 1 Lux =
1 Lumen/m2

Slide # 28
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 RGB

RGB
R

G RGB B RGB

B R G
R G B R G
B

Flat waveguide hologram type

X-Cube type Folded prisms type Linear prisms type

Slide # 29
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 LED edge
lit guide

PBS

LCD

LCoS LCoS

Backlit LCoS Backlit LCD Frontlit LCoS

Slide # 30
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Examples of scanning display technologies

Typical single or dual axis MEMS scanner Fiber piezo scanner SAW waveguide scanner

Scanned image

Incoupler SAW scanner

Scanning fiber Laser illumination

Lightguide Outcoupler

Piezo tube
actuator Fiber
Electrodes

Slide # 31
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Wobulation using a switchable window position (Optotune AG)

Wobulation using a switchable illumination source

DLP pixels
Multiple
illumination LEDs

Example of 2x RGB
LED illumination die
(total of 6 LEDs, only 3
on at a time)

First illumination LED on Second illumination LED on

(first sets of pixels ON) (second inter-digited sets of pixels ON)

Slide # 32
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 2D fast 1D fast/slow 1D fast/slow 1D slow 2D slow 2D slow 2D slow
MEMS mirror MEMS mirrors MEMS mirrors MEMS mirror MEMS mirror MEMS mirror MEMS mirror
Single source / color Single source / color Multiple sources / color 1D array/ staggered 2D panel display 2D panel display 2D panel display
(laser/VCSEL) (Laser/VCSEL) (Laser/VCSEL) (i-LED, OLED array) (wobbulatiojn) (stitching) (wobbulation and stitching
for optical foveation)

Slide # 33
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Popular light engines: LCOS and DLP

Spatially
demultiplexe
d Exit pupils

DLP chip

Relay
PBS optics

LCoS
Polarization
LEDs
recycling and
homogenization LED
optics
PBS
birdbath Dual TIR
LEDs Overlappin prisms
Relay g exit
optics pupils
LCoS

Folded LCoS illumination Inline LCOS PBS illumination DLP based optical engines
(HoloLens V1 - 2016) (Magic Leap One - 2018) (Digilens / Avegant - 2014)

Slide # 34
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 HoloLens H1 light engine

Slide # 35
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Waveguide combiner H1 and H2

Optical engines linked to waveguide combiner Optical engine ray trace Packaged optical engines

Birdbath
optics

LCoS light
engines

Exit pupil

Waveguide
combiner

Exit pupils

Lasers

Laser scanner
light engine Slow mirror
and Exit
pupil

Exit pupil

Waveguide Exit pupil

combiner

Fast mirror

Slide # 36
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 On the VR front

Slide # 37
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 VR hardware roadmap (2016-2019)

HTC Vive Focus Pro Oculus Quest

Samsung Odyssey
Oculus CV1
Window MR Oculus GO

2019: Standalone inside-out 6DOFsensors

2016: Tethered outside-in 2017: Tethered inside-out

2018: Standalone 3DOF
6DOF sensors 6DOF sensors

2019: High end PC tethered inside-out 6DOF

HTC Vive Cosmos HTC Vive Pro

Oculus Rift S

HTC Vive Window MR

(Lenovo, ACER, HP, Dell, Asus) Vive Focus

Slide # 38
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Reducing the VR lens size/weight

Convex/convex lens Hybrid Fresnel, entire aperture Hybrid Fresnel, partial aperture Dual Fresnel shells
Oculus DK2, Sony Playstation VR,… Oculus CV1, HTC Vive,.. Oculus, Google Daydream (concepts) Wearality Inc.
2013-17 2016-18 2018-19 2016

Slide # 39
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Reducing lens to display distance (smaller form factor)

Single panel display

Single micro-display panel Flat micro-display panel

Dual partial image panels
(on curved surface)
MLA based VR Light
field display

Cemented pancake Plastic/glass air pancake

Pancake lenses (cemented and in air)

(Glasses by DloDlo, Luci, e-magin OLED goggles, Wide-5 by
Conventional VR lenses Multipath lenses Nvidia Corp. UNCC
FakeSpace, Google, 3M, etc…)
(ThinEyes by Limbak) NTE light field display.
2015
2017 2014

Slide # 40
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Near – to – eye (NTE) VR optics roadmap

Display

Display
Fractional display
Fractional display

Light field rendering scenes

Compound lens
Single lens (hybrid) Single catadioptric lens 2D compound lens array MicroLens array (MLA)
(refractive or hybrid)
Oculus, Vive, Microsoft MR Pancake, ThinEyes ThinEyes, Panasonic VR Lusovu Lisplay Nvidia/UNCC MLA Light field

Reducing form factor of headset by optimizing the lens / display architecture

Slide # 41
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Pancake lenses and other polarization lens compounds are not new!

Slide # 42
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Video see through VR products

Intel project Alloy Lynx-AR Varjo XR-1 “bionic display”

Visette 45SXGA HTC Vive Focus 2.0
Vuzix Wrap 1200

StereoLabs video add-on ZED Mini video add-on

Microsoft add-on on Samsung Odyssey MR

Slide # 43
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Minimizing weight and size of combiners:
1: Free space combiner optics

Slide # 44
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Google glass – in media bird bath architectures

50/50 splitter Glass PBS LED backlight

Metallic reflective
coating on lens
LCOS

Slide # 45
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 “See-around” display architectures

Optional coating Tapered lightguide

Micro-display
TIR prism or coated
prism
Side view Top view

Lens “ribbed” surfaces to reduce reflections

Vuzix m100
Kopin Solos “see around” tapered Olympus “see around” opaque lightguide opaque smart glass
Kopin “pupil” prism / lens hidden in
transparent lightguide upper frame

Slide # 46
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Flat half tone combiner designs

Display (phone panel, LCOS/OLED or CRT!) Apache Helicopter AH64 Integrated Helmet And Display
Sight System (IHADSS) – with mini CRT display
Collimation
lens(es) ODG R7 with Micro-OLED display

Visor

50/50
beam splitter

Slide # 47
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 48
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 “Bug eye” AR combiner designs

FOV: 60 deg
Eyebox: 10mm
Eye relief: 17mm

Slide # 49
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Single large curved visor combiner designs

META 2 DreamGlass

Smart phone display panel

LeapMotion North Star ref. design

Visor / MIRA Prism
curved
half tone
reflector

Slide # 50
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Air birdbath combiner designs

ODG R9 Nreal
Display (phone panel, LCOS/OLED)

Field
lens(es) Curved
partial
mirror
Visor

Chinese OEM display engines

Lenovo AR

50/50 beam
splitter

Slide # 51
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Triple reflector design (IMMY).
FOV up to 50 deg

Microdisplay
IMMY

Mirror
Half
tone
mirror

Slide # 52
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Simple half tone curved reflector design
FOV down to 15 deg

Lens
Raptor (Everysight)
Microdisplay
Half tone mirror region

Slide # 53
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Hybrid curved combiner designs

Toshiba Smart Glass WearVue (Hybrid refractive/reflective Fresnel)

Temple mount projector

(LCD, LCOS, OLED, iLED)

Visor / Projector
curved
half
tone
reflect
or
Glass Up Smart glasses (Hybrid Diffractive/refractive Fresnel)

Fresnel reflector array

inside prescription lens.
Casting or hybrid injection
molding/casting with half
tone reflective surface on Projector Combiner region
Fresnel

Slide # 54
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Laser scanners and holographic combiner designs
Intel Vaunt
Temple mount laser/VCSEL, 2x1D or 2D
MEMS scanner Display engine

Hybrid curved
holographic
reflector

Laser / VCSEL MEMS projector

Reflective hologram
cast inside prescription lens.
Photopolymer hologram is tuned North Focals
for specific spectral and angular
bandwidths (Bragg selectivity).

Slide # 55
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 56
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 57
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Freeform prism combiner designs

ThirdEye
Lenovo DaystAR

NED Glass X2 Chinese OEM display engines

Slide # 58
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Example of curved TIR prism combiner using 5 bounces, 4 of which Example of curved combiner using 4 bounces,
are TIR. No compensator (2012) 3 of which are TIR
(Google Glass, 2014)
US Patent # 20120162549A1

TIR reflection
Coated Compensator
Flat TIR Camera
TIR reflection reflector
Coated Micro display
asphere
TIR reflection

Flat TIR

Conical TIR IPD adjust swivel

Micro display
Cylinder refractor

Slide # 59
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 HMD type Free space combiner Example Display engine FOV Size / weight Advantages Inconvenients
architecture
Smart glasses / See-around opaque prism combiner Kopin Solos, Vuzix MuOLED, LCoS <2=0 deg Small Cheap Not see through
smart eyewear m100,

Cemented bird-bath combiners Google Glass V1/V2 LCoS, HTPS LCD <= 15 deg Small Small/ compact Parasitic reflections

Single visor reflector (monocular smart Raptor EverySight MuOLED <= 20 deg Minimal Small form factor Small FOV, small eyebox
glass)

Hybrid volume holographic reflector Intel Vaunt, ”focals’ MEMS scanner <= 20 deg Minimal (minimal Best form factor Small eyebox
by North temple projector)

Hybrid reflector Toshiba WearVue, LCoS <= 15 deg Minimal (large Small form factor Large projector
Glass-Up temple projector)

Augmented Flat tilted partial reflector ODG R7, Apache MuOLED, Green CRT <= 20 deg Medium Simple design Tilted plate in front of eye
reality binocular AH64

Single TIR bounce freeform prism Lenovo DayStAR, LCoS <= 50 deg Medium but heavy Compact / low cost Thick and Heavy
ThirdEye, NEDGLass

Multiple off-axis reflectors IMMY Inc. OLED <= 50 deg Medium Good color, large Pupil swim
FOV, compact

MLA based plastic shell NTE combiner Lusovu, various,… Transparent OLED, <150 deg Curved, compact Curved, no light Low resolution, low contrast.
iLED sheets projector, UH FOV

Mixed reality Single visor reflector (binocular AR) Meta 2, DreamGlass, Cell phone panels <= 90 deg Large Cheap, large FOV, Need smart phone display, bulky
Mira, LeapMotion ref. good color

Off-axis in air birdbarth combiner ODG R9, Nreal, MuOLED, LCoS <= 52 deg Medium Compact and light, Low efficiency
Lenovo AR good color

Slide # 60
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Conventional ophthalmic lens puck casting process

UV cured and thermally Custom generated back Polishing and hard Edging to lens frames
Lens puck casting
annealed puck surface (diamond turning) coating and IPD and final
(UV curing and thermal annealing)
mounting, inspection and
dispensing

Back Front
surface surface

Base
curve

Slide # 61
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Moore Nanotech FG350 / FG650 DTM Precitech Freeform DTM Sumitomo injection molding machines

Slide # 62
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Inserts aligned with gate and
Separate diamond turned freeform metal insert Final mold block
runner in mold

“MoldFlow” FE modeling Injection molded parts Final elements with gate cut out
showing stress points

Slide # 63
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 64
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 LUXeXcell additive manufacturing of optics Ophthalmic and Fresnel lenses MLAs, diffusers and non imaging optics

Slide # 65
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 UV casting in glass molds Polymer film insertion in cast for various base surfaces

Slide # 66
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Required surface figures (Peak to Valley - PV) for 1 wave error (l) on the incoming wavefront (a is AOI)

Refraction:

PV<=l/((n-1)cos(a)) Reflection in air:

≅ 2l/cos(a)
PV<=l/(2cos(a))

TIR or reflection in material

through coating:

PV<=l/(2ncos(a))

Slide # 67
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Minimizing weight/size/CG on combiners

2: Waveguide combiner optics:

The ultimate form factor for AR combiners?

Slide # 68
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Flat single mirror outcoupler designs

Curved half tone reflector Flat HOE reflector

TIR guiding TIR guiding

Microdisplay Microdisplay

Epson Moverio BT300 and BT 30 C binocular smart glasses Konica Minolta monocular

Slide # 69
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Curved single mirror outcoupler designs

Flat waveguide Micro-Optical Smart Glasses (1995): Curve waveguide Zeiss “Tooz” Smart Glasses (2015):
Two bounded shells with coated prism extractor. Two bounded curved shells with coated Fresnel structures and freeform injector.

LCD microdisplay

OLED
microdisplay

Embedded
coated prism Embedded coated
extractor Fresnel extractor

Slide # 70
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Architecture continuum for mirror based combiners

Single half tone / holographic mirror Multiple half tone mirrors Fractured metal mirrors

Flat waveguide, flat mirror,

Lumus LOE
lens in optical engine LetinAR CES2018

Epson Moverio BT100

Optinvent prisms

Flat waveguide, flat

inverted mirror, lens Concepts and patents
in birdbath
Google Glass V1 & V2
LetinAR CES2019

Flat waveguide, lenses

Concepts and patents Concepts and patents
in curved mirrors
Konica Minolta (holographic lens)
Epson Moverio BT300

Curved combiner,
lenses and Concepts and patents
compensators in Zeiss Tooz smart glass
curved mirrors
LetinAR concept
Luminit (Holographic lens)

Slide # 71
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Some of the challenges paving the road down to
the ultimate MR experience:

1) FOV
2) Angular resolution
3) Weight/ size and CG
4) Eyebox
5) Non conflicting 3D cues
6) Vision impairment correction
7) Immersion experience
8) Sensor fusion

Slide # 72
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 The difficult design balance between
FOV, angular resolution and eye box size

Slide # 73
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Image intensity
Left display line intensity Right display line intensity
100%

Lower
vignetting limit
(%)
Lateral eye movement
direction
Eye pupil
Eye box dimension

Slide # 74
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Eye box region Partial field vision

Single lens VR Single reflector AR Simple waveguide AR Stitched fields waveguide Demultiplexed field AR
(Oculus, HTC Vive, …) (Meta 2, Dreamglass,…) (Hololens V1, ML One,…) AR waveguide (Lumus LOE,
(HoloLens V2, LetinAR,…)
WaveOptics,…)

Field 1 Field 1

All fields

Field 2 Field 2

EB range
EB range EB range EB range EB range

Max ER

Slide # 75
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Luminance 10-6 10-4 10-2 1 10+2 10+4 10+6 10+8
(Cd/m2)
Pupil size (mm) 7.9 7.5 6.1 3.9 2.5 2.1 2.0 2.0

Vision mode Scotopic (starlight) Mesopic Photopic (office light to sunlight)

(moonlight)

Eyebox (mm) Eyebox (mm) Perceived Eyebox (mm)

Eye relief (mm) Display FOV (deg) Eye pupil diameter (mm)

Slide # 76
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Prescription lens
Combiner (waveguide)

Cornea

Pupil

VD
VD: Vertex Distance (Rx lens)
ER 1 ER 1: Eye Relief to Cornea (wear)
ER 2: Eye Relief to Pupil (optics)
ER 2

Slide # 77
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Age Female IPD Male IPD Total IPD Min/Max IPD
group (mm) (mm) (mm) (mm)
20-30 59.2 61.5 60.3 49-70
31-50 62.0 64.5 63.0 55-72
51-70 62.3 65.7 63.8 52-76
71-89 62.1 63.1 62.7 49-74

Slide # 78
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Design exercise: UX and ID constrains limiting achievable eyebox

Design space window

Thickness of combiner

Aesthetic constraint

Waveguide combiners w EPE

Mechanical stability constraint

Min IPD Full IPD

coverage coverage Eye box size

Slide # 79
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Eye box scales with eye relief (or vertex distance)

Design space window

Non pupil forming

Eye box size

architectures

Full IPD coverage

Pupil forming
architectures

Min IPD coverage

Vertex distance Aesthetic Eye relief /

constraint
vertex distance
Slide # 80
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Eyebox is shared with FOV

Eye box

FOV
Eye box size

Eye box

FOV

FOV

Slide # 81
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Eyebox is modulated by eye pupil size

Eye box size

50% vignetting definition

Eye pupil diam

Slide # 82
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Major design problem for HMDs: large eyebox / FOV and small footprint

Fundamental optical problem: Etendue conservation (Lagrange invariant)

Microdisplay size * NA in image space = Exit pupil size * FOV

Microdisplay size * NA in image space < Eye box * FOV

Can Law of Etendue be

compromised to increase Yes, fortunately, there
the exit pupil (thus are many ways to
enlarging the eyebox) tweak the
without altering display Law of Etendue
size, NA and FOV? to accommodate the
human visual system

Slide # 83
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 How to make the eye box large?
Fundamental optical problem: Etendue conservation
Microdisplay size * NA in image space = Exit pupil size * FOV

“Microdisplay size * NA in image space” <= Exit pupil size * FOV

1) IPD adjust 2) Pupil expansion 3) Pupil replication 4) Pupil steering 5) Display tiling 6) Lens movement 7) Pupil switching

Slide # 84
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Increasing the ridiculously small eye box in retinal scanners

North Focals Smart Glass: HoloLens V2 MR headset:

Intel Vaunt Smart Glass
Single 2D MEMS mirror laser scanner with Dual MEMS mirrors laser scanner
Single 2D MEMS mirror VCSEL scanner
4 exit pupils generation with 2D waveguide exit pupil replication
with 4 exit pupils generation
QD Laser Smart Glass: by multiplexed VCSEL colors/holograms by multifaceted prism
Single 2D MEMS mirror laser scanner
direct retinal scan

Single exit pupil

4 exit pupils

2D array of exit pupils

Large eyebox, natural depth of focus

Retinal scanners, small eyebox, synthetic large depth of focus

Slide # 85
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 86
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 All in one shell design

Partial transmission sparse emissive

display (muOLED, iLED, …)

Transparent curved plastic shell

Partially reflective MLA array

Reflection through coating or
Bragg selectivity

Lenses are see-through without distortion

FOV, eyebox and angular resolution are (Holographic, index matched reflective or
built synthetically through redundancy of Fresnel, or diffractive)
display and custom lens off-axis

Slide # 87
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Lusovu “Lisplay” flat holographic MLA
prototype (Portugal)

Slide # 88
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Future versions?

Slide # 89
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 90
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Waveguide combiners and the eyebox
expansion technique

Slide # 91
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Pupil expansion in waveguides is not a new concept!

Holographic coupler waveguide combiners Grating couplers waveguide combiners Partial mirrors waveguide combiners

Slide # 92
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Single dimension exit pupil expansion in waveguide combiners: mirrors

Sony 1D pupil expansion waveguide

Lumus LOE 1D pupil expansion waveguide Optinvent LOE 1D pupil expansion based on photopolymer reflective
combiner based on embedded partially reflective waveguide combiner based on half hologram couplers (SED 100A)
mirrors tone embedded microprism arrays

Slide # 93
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Single dimension exit pupil expansion in waveguide combiners: volume holograms

Original Sony Smart Glasses Univet SwimAR

using Sony SED 100A waveguide using Sony SED 100A waveguide

Slide # 94
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Various third party implementations of the Lumus LOE combiner

Daqri smart helmet Flex smart helmet

Lumus DK Vision

Lumus DK50 AR glasses Daqri AR glasses Atheer Labs AR glasses

Slide # 95
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 K vector diagram Input pupil K vector diagram Input pupil
EPE Resulting eyebox Resulting eyebox
with FOV generated by dimension with FOV generated by dimension EPE
replicat dimension generated by dimension generated by
display engine generated by display engine generated by replicat
ion waveguide combiner waveguide combiner
display engine display engine ion
ky/k0
schem ky/k0

schem
e e

NO
EPE kx/k0

NA
kx/k0

NA

K vector diagram Input pupil K vector diagram Input pupil

EPE Resulting eyebox Resulting eyebox
with FOV generated by dimension with FOV generated by dimension EPE
replicat dimension generated by dimension generated by
display engine generated by display engine generated by replicat
ion waveguide combiner waveguide combiner
display engine display engine ion
ky/k0
schem ky/k0

schem
e e

1D
EPE kx/k0 kx/k0

K vector diagram Input pupil

EPE Resulting eyebox
with FOV generated by dimension
replication dimension generated by
display engine generated by
scheme waveguide combiner
display engine
ky/k0

2D Best
EPE kx/k0 FOV/EPE/EB
compromises

Slide # 96
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 The problem with 1D pupil expansion
Horizontal exit pupil,
Vertical exit pupil, Square exit pupil,
vertical 1D EPE
horizontal 1D EPE 2D EPE

Square pupil
EPE

EPE
Pupil

EPE Pupil

EPE
Lumus DK50 Lumus DK Vision Digilens Crystal

Slide # 97
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Various implementations of ”2D EPE in grating waveguide couplers”

Slide # 98
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Example of 2D EPE waveguide combiners

Vuzix Blade (grating) Digilens Crystal (holographic) EnhancedWorld – WaveOptic Phlox (grating)

HoloLens V2 (grating)
Magic Leap One (grating)

Slide # 99
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 2D EPE expansion in HoloLens V1 (2015)

Collimated field (FOV) normally

incident to guide, from light ky/k0 Single input pupil over which
TIR in-coupling by I-E
engine, and collimated image is located at infinity
outcoupled field. Edge located input
grating coupler RGB LED
LCOS
display
R-E engine

I-E
ANGLE LIMIT
MODE LIMIT

O-E
kx/k0 Single 90 deg
redirection
grating
PROPAGATING

MIN TIR

Out-coupling grating
forming the eyebox

Field redirected HoloLens V1 display system incorporating a

Field out-coupled downwards by R-E 2D exit pupil replication waveguide
by O-E Replicated exit pupils forming a single image
at infinity over the entire eyebox

K vector diagram

Slide # 100
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 2D EPE expansion in HoloLens V2 (2019)

Centrally located input

grating coupler
ky/k0

Laser MEMS display

Central I-E engine
Left R-E Right R-E

Left “butterfly”
Redirection grating
kx/k0
Right “butterfly”
Common O-E Redirection grating

Out-coupling grating
forming the eyebox

Replicated exit pupils forming a single image HoloLens V2 display system incorporating a
at infinity over the entire eyebox “Butterfly” waveguide combiner architecture
K vector diagram

Slide # 101
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Field expansion in waveguide

ky/k0

O-E processing all FOV over

O-E processing dropping unused FOV at
entire eyebox (strong O-E size
eyebox edges reducing O-E size
requirements)
requirements

R-E R-E

I-E I-E
kx/k0

O-E O-E

Slide # 102
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Guide thickness

Left Center field Right Left Center field Right

field field field field

Partially out-coupled right field

Thin guide Thick guide

Right Right
field field

Center Center
field field

Left Left
field field

I-E Ø and
I-E Ø and
input pupil Ø
input pupil Ø

Slide # 103
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 The core of waveguide combiners:
in- and out- coupler elements

Slide # 104
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Surface relief couplers: A zoo!
In-coupler modes Out-coupler mode

Blazed gratings

Slanted gratings

Binary gratings

Analog surface relief

gratings/CGHs

Slide # 105
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Optimization of 3D nanostructures to reach the desired bandwidths
(angular and spectral)

Grating period

Duty cycle
TIR condition angular range
n=1.53

Top Unoptimized grating structures/indices

n_top

Diffraction efficiency
slant

Top
cover
Depth
n_grat n=1.53
Optimized grating structures
Bottom slant n_base
base

Front slant Back slant

n=1.70
Optimized grating structures / indices
n_guide

Central design angle

Angle of incidence over grating

Slide # 106
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 107
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 On the need for rigorous EM diffraction models for waveguide combiner gratings

Rigorous EM
« MACRO « MICRO « NANO Extended scalar
OPTICS » OPTICS » 100%
L OPTICS »
100

Diffraction efficiency (%)

m 0 order « p » pol Scalar
Feature size x2

0 order « s » pol
+1 order «s» pol
+1 order scalar
« MICRO
ELECTRONICS » 50% +1 order «p» pol
10 m

1 m
0 order scalar
100 nm « NANO
ELECTRONICS
L/l
10
» 0%
nm
100nm 1 m 10 l 1.0 2.0 3.0 4.0 5.0
m

Slide # 108
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 First implementations of slanted grating waveguides (early 1990s):
Optical clock broadcasting in MCM (Eureka FOTA)
(Jari Turunen, Pierre Chavel,…)

Slide # 109
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 110
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Extension to 2D EPE by Nokia (circa 2005)

Slide # 111
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 TiO2 ALD on oxide, or resin can be a
good alternative to high index NIL.

Slide # 112
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Volume holographic couplers in waveguide combiners

Thin transmission Thin Reflection Thick reflection/transmission

2-3 m (H-PDLC) 16-70m (photopolymer) 100-500m photopolymer
Digilens Covestro, DuPont, Polygrama Akonia/Apple

Slide # 113
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Example of Kogelnik modeling of holographic waveguide combiner
Kogelnik angle/spectrum efficiency plots in reflection mode
for holographic waveguide coupler
Reconstruction wavelength in nm (light engine illumination)

Photopolymer mean index n=1.53, dn=0.03, thickness = 16 m

Lower FOV angle Upper FOV angle Reference beam is normal to substrate, object beam is tilted at 50 deg
for laser line for laser line

17 deg

20nm
Resulting angular bandwidth (FOV) of holographic coupler for
single laser line spectrum at 550nm (>1 nm spectral bandwidth)

34 deg

Lower FOV angle for lower Upper FOV angle for upper
part of the LED spectrum part of the LED spectrum
Resulting angular bandwidth (FOV) of holographic coupler for
LED spectrum around 550nm (with 20nm spectral bandwidth)

Incoming angle in degrees (FOV in air from light engine)

Slide # 114
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 LightTrans example of 2D EPE waveguide grating combiner simulation module

Slide # 115
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 The cascaded angular and spectral bandwidths

Image formed by display engine Waveguide space (TIR) Image over eyebox

Spectral
dimension
RGB I-E spectral bandwdith TIR reflection R-E spectral bandwdith O-E spectral bandwdith Image
LEDs color

Angular
dimension
Initial I-E angular bandwidth TIR condition R-E angular bandwidth O-E angular bandwidth Image
FOV FOV

Leaky
Redirection Extracted
Display In-coupler TIR outcouple
coupler image
engine r

Multiple
interactions

Slide # 116
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 RWG couplers

Close-up on
Resonant Waveguide Grating (RWG) layer on top of thin RWG (<1 m)
combiner lightguide substrate Light guide not to scale

Slide # 117
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Design gymnastic in using couplers in the third dimension

Sandwiched Transmissive couplers Reflective couplers T+R dual L+R

/ buried couplers couplers

Slide # 118
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Diapersive and achromatic waveguide coupler technologies

Refractives Diffractives / Hybrid Phase multiplexed Achromatic

Cascaded reflective
holographic (T+R) Diffractive/refractive holographic (T+R) Metasurfaces (T+R)

G R B R+G+B R+G+B
R+G+B
B G
R

B R+G+B
R+G+B R+G+B R+G+B R+G+B R+G+B
G R+G+B
R+G+B R+G+B
R

Dispersive couplers Achromatic couplers

Slide # 119
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Waveguide coupler Operation Reflective Transmission Efficiency Lensed out- Spectral Color Dynamicall Polarization Mass Company/Product
tech coupling coupling modulation coupler dispersion, uniformity y tunable maintaining production

Embedded mirrors Reflective Yes No Complex No Minimal Good No Yes Slicing, Lumus Ltd.
coatings coating, DK 50
polishing,

Micro-prisms Reflective Yes No Coatings No Minimal Good No Yes Injection Optinvent SaRL.
molding ORA

Surface relief slanted Diffractive Yes Yes Depth, Duty Yes Strong Needs comp. Possible with No NIL (wafer, Microsoft
grating cycle, slant LC plate) HoloLens, Vuzix
Inc, Nokia,…

Surface relief blazed Diffractive Yes No Depth No Strong Needs comp. Possible with No NIL (wafer, Magic Leap One,
grating LC plate)

Surface relief binary Diffractive Yes Yes Depth, Duty Yes Strong Needs comp. Possible with No NIL (wafer, Magic Leap One
grating cycle LC plate)

Multilevel surface Diffractive Yes Yes Depth, Duty Yes Strong Needs comp. Possible with Possible, but NIL WaveOptics Ltd,
relief grating cycle LC difficult (wafer,plate) BAE. Dispelix.

Thin photopolymer Diffractive Yes Yes Index swing Yes, but Strong Needs comp. Possible with No Contact print Sony Ltd,
hologram difficult shear roll to roll TruelifeOptics Ltd,

H-PDLC volume Diffractive No Yes Index swing Yes, but Strong OK Yes No Exposure Digilens Corp.
holographic difficult (electrical)( (MonoHUD)

Thick photopolymer Diffractive Yes Yes Index swing Yes, but Minimal OK No No Multiple Akonia Corp
hologram difficult exposure (now Apple Inc.)

Resonant Diffractive Yes Yes Depth. Duty Yes Can be NA Possible with Possible Roll to roll NIL CSEM / Resonannt
Waveguide Grating cycle mitigated LC Screens

Metasurface coupler Mostly Yes Yes Various Yes Can be Needs comp. Possible with Possible NIL (wafer, Metalenz Corp.
diffractive mitigated LC plate)

Slide # 120
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Spectral dispersion compensation in waveguide couplers

l0 ± dl l0+dl
l0-dl l0 l0-dl l0

Slide # 121
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Focus dispersion in waveguide combiners
Exit pupils:
Input pupil: Focus spread through eyebox
Single wavelength,
Image in near field

Slide # 122
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Getting back the image in the near field

No lens: Static external lenses: Lensed out-coupler

Input pupil: Image at infinity Image in near field Image in near field
Image at infinity

-1/2D lens

-1/2D
coupler

+1/2D lens

Slide # 123
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 RGB waveguide combiner architectures

Slide # 124
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 What about when using multiple colors? HoloLens solution
Spatially Recombined
multiplexed color image
input pupils

Parasitic
transmission
Slanted gratings as input and diffraction order
output couplers

Zero order

Zero order
Reflection diffraction order
forming the eyebox

Slide # 125
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 What about when using multiple colors? Magic Leap solution

Spatially color de-

multiplexed entrance
pupils
Reflection diffraction
order forming the
eyebox

Binary phase gratings with depth

Metal coated reflective
modulation as out-couplers
blazed gratings as in put Parasitic transmission
couplers diffraction order

Recombined color image

Slide # 126
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 One color per guide: 3 guides for RGB
Input color image from
light engine (image
pupil at infinity)

Reconstructed image at infinity

with all 3 recombined colors
Blue

Green

Red

Slide # 127
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Low hanging fruit: 2 guides for RGB
First guide: B + left of G FOV Second guide: Right of G + R FOV
ky/k0 ky/k0
Input Input

kx/k0 kx/k0

Output
Output

First guide

Second guide

Slide # 128
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Using a single guide to combine RGB

Attempting at coupling RGB FOV in single Coupling reduced RGB FOV in single guide
guide over max allowed TIR angle range using max allowed TIR range

ky/k0 ky/k0
Input Input

B and R FOV Full reduced

cropped, full G RGB FOV
FOV guided guided and out-
coupled

kx/k0 kx/k0

Output
Output

Slide # 129
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 SRG (Surface Relief Gratings) fabrication

Element type Fabrication type

Refractive Grinding
(Efficiency: Theoretical/practical / Gray scale litho.
100% - 100%)
Diamond turning
Blazed
/ Gray scale litho.
(100% - 90%)

Saw-tooth
(90% - 85%) Diamond turning

Amplitude
Binary Lithography
(< 10% - < 8%)

Binary Binary Lithography

(40.5% - 35%)
Quartenary
(81.0% - 70%) Multiulevel Lithography

Sinusoidal Interference
(33.3% - 25%) lithogrpahy

Slide # 130
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Binary
element

Quartenary
element

8 levels
element

16 levels
element

Analog surface
relief element

Slide # 131
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Multilevel lithography fab and gray scale lithography

Multilevel lithography Direct write method Gray scale lithography

Single exposure
Analog dosage
First Patterning
Single development
masking/etch (e- or laser)

After 1st
pass/resist dev.
Second Single etch
masking/etch
After 2nd
pass/resist dev.
Resulting quasi-analog surface relief
Third
After 3rd
masking/etc pass/resist dev.
h

Final proportional
etch into substrate

Slide # 132
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Example of fabricated SRGs

Refractive and diffractive microlens arrays fabricated by gray scale lithography Multilevel diffractive elements (Quarternary)

Slanted gratings fabricated by interference Binary CGHs fabricated by optical lithography and dry etching
lithography and directional dry etching

Slide # 133
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Nano Imprint Lithography for SRGs

EVG® HERCULES®NIL UV-NIL Track System (wafer) Canon® FPA-1100NR2 NIL (wafer) EVG®7200 LA Smart NIL (plate) EVG® 750 R2R hot embossing roll to roll

Slide # 134
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Example of Vuzix’s replicated grating waveguides

Slide # 135
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 What else can one do with waveguide combiners?

Slide # 136
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Potential future applications of waveguide combiners : smart windows

Smart window concepts based on waveguide combiner AR display overlay smart IR imaging smart window
window (eye tracking, face recognition)
with grating in-out couplers
IR flood

“Welcome back Mrs Smith”

Outcoupler / CMOS
Projector / camera
incoupler

LCPV solar cell Passive IR management

smart window smart window
Summer
sun
PV cell

Winter High angle summer IR

sun rays mostly reflected or
TIR to fluid line

Part of visible to
1.1mu trapped in
all directions by Most summer Low angle winter IR
TIR to lateral PV IR rays reflected rays transmitted
cells strips
PV cell
Solar co-generation
(fluid line IR heating for AC)

Slide # 137
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Gen 3 and Gen 5 NIL nanostructures replication

Slide # 138
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 What about getting closer to the eye?

Slide # 139
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 The future of wearable displays: Closer and closer to the brain

Visual cortex implant

VR headset

AR/MR headsets Smart retinal implant

Smart Glasses / Smart intra-ocular lens

smart eyewear
Smart Contact lens

Slide # 140
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Smart contact lenses

Diabetes management through glucose level contact lens sensor

(Google / Novartis and KIST 2013)

Glaucoma sensing through MEMS on soft contact lens (Sensimed

AG Triggerfish for IOP pressure sensor -2012, and Stanford U.
intraocular IOP pressure sensor with sensing channel)

Minimalistic displays
(individual RGB LEDs on contact lens)
Prof. Babak Parviz, UW, 2010

Liquid Crystal contact lens for real time Presbyopia correction

Google / Verily – 2015,
IMEC – Cmst - Ghent Univ.

Liquid Crystal intraocular lens for real time Presbyopia correction

Slide # 141
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 AR display on a contact lens?
NTE implementation Contact lens
implementation Prototype examples

Display+MLA on contact lens Lusovu “Lisplay”, Portugal

Display+MLA on NTE lens

MLA based NTE and contact lens architectures (using semi-

transparent emissive display – iLED, muOLED)

Projection Emissive display on lens Innovega / Emacula (Seattle, WA)

Display polarization coating
display on
reflective
diffuser on lens
Hybrid glasses/contact lens display architecture (polarization
diversity)

See through polarization coating

LETI-Grenoble / MIPS Mulhouse

Integrated on NTE lens Integrated on contact lens

Active waveguide array and holographic pixel extractors

architecture – laser lit
(multibeam interference)

Curved waveguide Curved waveguide

phase panel in NTE phase panel in contact Microsoft Research (free space version)
lens lens

Laser edge-lit curved shell LC phase panel waveguide (digital

holography - CGH)

Slide # 142
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Some of the challenges paving the road down to
the ultimate MR experience:

1) FOV
2) Angular resolution
3) Weight/ size and CG
4) Eyebox
5) Non conflicting 3D cues
6) Vision impairment correction
7) Immersion experience
8) Sensor fusion

Slide # 143
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 3D display immersion in NTE displays

Slide # 144
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 IPD
60 64 68
Natural vision

Stimulus distamce (m)

Accommodation reflex (spherical diopters)
2 Fixed focus 0.5

Accommodation reflex (meters)

display (at 2m)

Vergence
Accommodation
Conflict 0.5 Far Arms 2
field length

“VAC”

0.1 10
Disparity limits inducing
nausea (blur)

Disparity limits inducing diplopia ∞

0
10 20 30
Convergence reflex (prism diopters)

Slide # 145
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 VAC conflict in fixed focus stereo displays

Stereo image focus is set at 2m

2.0 Near field (-1/2 D offset from infinity)

1.5
interface region
Close up hand/display interaction

1/2D VAC conflict limit region

VAC (Diopters)

1.0
1/4 D VAC conflict limit region
Display

0.5

0.25

0.0
0.25m 0.5m 1.33m
0m 1m 2m 3m 4m 5m
Image distance (m)

Slide # 146
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 The problem of arm’s length display interaction

85cm reach

57cm reach

47cm reach

Wheelchair 5th percentile Caucasian adult female 5th percentile Asian adult female 95th percentile Caucasian adult male

Slide # 147
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Display immersion as a 2D concept (FOV) Display immersion as a 3D concept

FOV (deg)
FOV (deg)
Immersion increases

< 1m (near field) to far field

> 1m (far field)

Slide # 148
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Stereo display with fixed focus

Binocular FOV
Right eye view of nose Left eye view of nose Monocular FOV B
Brow ridge Positive parallax
C C
AA B B
Field of Nose bridge A
stereo
overlap Stereo overlap FOV Display
image
plane
(zero

Display focus
Approx 40
Tip of parallax)
deg Negative parallax
nose left display right display C
C C
Chee A B A B
k

Lateral stereoscopic field of

Approxview
90 deg left eye right eye

Slide # 149
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Real point source Single scene rendering
(no digital display) Fixed focus display (required stereo disparity)

Virtual image point

(at infinity)
Real world Stereo imaging
(focus at infinity Display panel
but stereo plane
disparity
in near field)

Left eye Right eye

Slide # 150
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Mitigating VAC conflict with tunable lenses

Focus tuning (varifocal) Multiple focus rendering (multifocal)

Full scene rendering on continuous focus tuning Partial scenes rendering on different planes

VAC conflict
Full scene on single plane Partial scenes on multiple planes
VAC conflict

Image distance Image distance

Slide # 151
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Focus change: Lens movement

Display translation Fixed focus MLA translation

Virtual image focus tuning

Virtual image focus tuning

Display
Display

Global LC based MLA tuning Individually addressed LC based MLAs

Per pixel depth virtual image tuning

Virtual image focus tuning

Display

Display
Slide # 152
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Focus change: Display movement

Stanford (G. Wetzstein) varifocal Oculus Half Dome varifocal prototype 3M variable focus compound air-gap Pancake lens
prototype with external motor based on (D.Lanman) with integrated motors (Timothy Wong, 3M)
Gear VR (2015) (2018)

Slide # 153
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 154
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 155
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 156
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Binary near/far field focus switch: exit pupil switching (Magic Leap One)
Near field LEDs switched ON Far field LEDs switched ON
R1 R1

G2 G2
B1 G1 R2 G1
R2 B2 B2 B1

G1 R2
G1 B1 R2 B1
R1 B2
R1 B2

G2 G2
Cover plate
Far field red plate
2.775 mm

Near field red plate

Far field blue plate
Near field blue plate
Far field green plate
Near field green plate
Cover plate

Slide # 157
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Varifocal implementation in free space AR combiners
Static display
A: lens or B: +/- Focus C: +/- Focus tuning engine
display tuning
translation

Tunable free
space combiner
(no EPE)

Tunable display
engine
D: - Focus
tuning

Avegant Multifocal AR (temporal light field) UNCC deformable membrane combiner varifocal AR prototype

Static free space

combiner
(no EPE)

Slide # 158
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 159
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Varifocal implementation in waveguide combiners

+/- Focus
tuning

Display Display
Display engine engine Lumus / DeepOptics Liquid Crystal lens Varifocal
engine unpol. S pol.
(+)
focus
tuning
(-)
(-)
focus
focus
tuning
tuning

P polarizer

S pol
lens
Waveguide Waveguide Waveguide
combiner combiner combiner
w/o EPE w EPE w EPE

Slide # 160
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Fractional light field imaging and digital holographic imaging

Large number of light field rendering

2x light field 3x light field 5x light field or digital “per pixel depth”
Fixed focus display Fixed focus display Fixed focus display fixed focus digital hologram
Virtual image
point
Display panel
plane

Slide # 161
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 VAC mitigation in digital holographic imaging

Typical phase pattern to be

injected in phase panel

Resulting per pixel depth

image in near field

Slide # 162
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Example of holographic display technologies (phase panels)

HoloEye Pluto (2014) VividQ (UK) Typical Fresnel Example of multi depths
phase panel based on holographic CGH pattern on with Fresnel CGH in
LCoS technology display headset phase panel VividQ headset
(2019)

Slide # 163
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 164
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 LetinAR PinMirror Display (extended DOF display)

Slide # 165
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Binary focus
Switch
(optical blur) Focus range potential
Accommodation focus depth

Continuous
Varifocal display
(Digital blur) Focus range potential
Accommodation focus depth

Multifocal
“temporal light field”
(optical blur)
Focus range potential
Accommodation focus depth

Retinal Display
Scan NTE
(no blur) Focus range potential
Accommodation focus depth

Holographic
display NTE
(optical blur) Focus range potential
Accommodation focus depth

Slide # 166
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 VAC Display Illuminati Free EPE VAC Depth Blur Eye tracking Accommodat Limitations Company/
mitigation engine on engine space Waveguide planes ion product/prototype
combiner combiner experience s

Binary Orthogonal pol. Orthogonall Yes No Partially 2 Optical Not required Realistic Requires dual pol. Various prototypes
focus plane states display y polarized solved display or fast pol. and demonstrators
engine switch

Dual focus Any Spatially Yes Yes Partially 1 Digital Vergence tracker Realistic Requires multiple Magic Leap One
switch multiplexed solved waveguide stacks and
multiple illumination

Continuous Any Any Yes, with Yes with large Solved 1 Digital Vergence tracker Realistic (digital) Requires large Tr. Lumus/DeepOptics,
varifocal internal external tunable lenses with Lemnis, Oculus Half
Transmissio transmission waveguide EPE Dome
n / Refl. lenses.
lenses

Multifocal DLP, laser Any Yes No Solved 2<N<8 Optical Pupil tracker for Realistic Refresh rate hit. Avegant Corp. AR
scanner, fast late stage Requires fast tunable
LCOS, … occlusion lens and display
rendering

Focal Display with Any Yes No Solved 2<N<8 Optical Pupil tracker for Realistic Refresh rate hit Oculus prototype
surfaces additional phase late stage Requires phase panel
panel occlusion
rendering

Retinal Single or dual Laser/VCS Yes No (OK only if Not solved analog No blur No required Very large Small eyebox. QD laser, Intel Vaunt,
Scan mirror laser EL no EPE) (unrealistic) Parasitic phase Focals by North,
Display scanner objects. Best w text in Brother airscouter,…
smart glasses

Light Fields High res. 2D Any Yes No Partially - Optical Not required (can Realistic Spatial resolution hit Nvidia, Leia, Stanford
display display w MLA solved increase tensor display
resolution)

Holographi Phase panel Laser/VCS Yes No Solved N>=2 Optical Pupil tracker for Realistic Heavy real time Holoeye, Eyeway,
c display (LC or MEMS) EL late stage calculation Vivid-Q, Microsoft
occlusion requirements Research, prototypes
rendering

Slide # 167
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Some of the challenges paving the road down to
the ultimate MR experience:

1) FOV
2) Angular resolution
3) Weight/ size and CG
4) Eyebox
5) Non conflicting 3D cues
6) Vision impairment correction
7) Immersion experience
8) Sensor fusion

Slide # 168
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Myopia (- power)
Presbyopia

Hyperopia (+ power)

Blind spot

-6D to +4D

-8.0 -6.0 -4.0 -2.0 0.0 2.0 4.0 6.0

Ametropia (Diopters)
Slide # 169
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Google Glass “Explorer edition”

A) Combiner after Rx lens

Display is corrected by Rx lens

Zeiss “Tooz” glasses

B) Combiner in Rx lens
Display is partially
compensated by Rx lens base
curvature and refractive index

“Focals” by North

C) Combiner on Rx lens base

Display is affected only by base
curvature of Rx lens

Vuzix “Blade”

D) Combiner before Rx lens

Display is no affected by Rx lens

Kopin “Pupil”

E) Combiner inside Rx lens

frame
Display is no affected by Rx lens

Slide # 170
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Lumus DK40 prescription inserts HoloLens V1 prescription inserts

Slide # 171
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Some of the challenges paving the road down to
the ultimate MR experience:

1) FOV
2) Angular resolution
3) Weight/ size and CG
4) Eyebox
5) Non conflicting 3D cues
6) Vision impairment correction
7) Immersion experience
8) Sensor fusion

Slide # 172
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Central foveated area with
stereo overlap and VAC
mitigation

Left Right
monocular monocular
peripheral peripheral
area area

Slide # 173
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Early work: peripheral patching with LEDs

SparseLightVR SparseLightAR

Slide # 174
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 First solution: enlarging the FOV with a single display system
Wide 5 from Fakespace VR (2006)
150 deg FOV
InfinitEye 210 deg VR headset prototype (2013) StarVR 210x130 deg FOV VR headset (2015)
Single large hybrid Fresnel lens Single large hybrid Fresnel lens

Pimax 8K display (2017) with 220 deg FOV

Single large hybrid Fresnel lens XTAL VR headset by VRgineers (2018) Valve VR (135 deg FOV), 2019
with non Fresnel wide FOV lenses at 180
deg FOV

Slide # 175
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Second solution: dual display systems

Dual panel/microdisplay foveated

Dual lens Panasonic VR
VR architecture from Varjo
(refractive for foveated area Dual freeform prism combiners with compensator (Finland)
and Fresnel for peripheral) 150 deg FOV H total. Stereo overlap on central prism.
SA Photonics Inc.

Slide # 176
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 SA Photonics’m (Los Gatos, CA) dual freeform prism (muOLED) display HUGE H FOV AR headset

Slide # 177
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 178
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Video See Through (Intel project Alloy) Optical See Through pixel occlusion (Prof. Hong Hua, U of A)

Slide # 179
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Some of the challenges paving the road down to
the ultimate MR experience:

1) FOV
2) Angular resolution
3) Weight/ size and CG
4) Eyebox
5) Non conflicting 3D cues
6) Vision impairment correction
7) Immersion experience
8) Sensor fusion

Slide # 180
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Sensors Fusion Actions Display

IPD sensor
Pixel occlusion

3DOF IMUs
Stereo disparity maps
6 DOF HeT
3DOF lateral cameras
Head gaze direction Specific action

Hologram occlusion
Depth map sensor(s) Gesture sensing 3D rendering
Hologram world locking
Thermal IR / night
vision / low light cams Optical foveation
Eye gaze direction 3 DOF HeT
Distortion comp.
Vergence
Focus/accommodation
Late Stage
Eye trackers Pupil position Pupil swim comp. Reprojection (LSR)

Wink/blink sensor Specific action

Display pipeline
Iris recognition

Vitals sensors Display engine

(heart beat, skin temp, Centralised database
sweat, pupil, EEG…)

Display brightness,
Ambient Light Sensor
active dimming

On/off head sensor Display On/Off

Slide # 181
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Depth map sensors

Structured illumination Time Of Flight (TOF) Stereo imaging

Distorted fringe

Object to be Signal out A

scanned

A
Signal in
Projector
Camera B
Image of distorted
fringe under B
different angle of
view

Distance
Triangulation base

Parallax

Slide # 182
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Kinect Azure TOF RGBD sensor

Slide # 183
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Eye tracking (ET) technologies

Imaging based ET:

Feature based ET: glints retinal and pupil imaging
4 successive Purkinje reflection
(the second glint – P2- forming the glints) Examples of IR –LED integration in a VR headset (Pupil Labs on Retinal imaging ET (high resolution)
HTC Vive)

P4 Transmitted ray
to retina
P3
IR LEDs
P2
P1
IR Camera
Incoming IR ray
Examples of gaze direction showing glint to pupil distance changes
Examples of IR –LED integration in MR systems (Magic Leap One, left, HoloLens V2 right)

IR LEDs

IR LEDs IR camera

IR Camera

Slide # 184
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Lensless eye tracking (ET) detectors

Rambus’ lensless spiral grating based eye tracking camera Ad Hawk’s lensless MEMS based eye tracker camera

Slide # 185
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Using same optical trains for ET imaging and sensing

AR/VR Imaging: infinite conjugate Bi-directional OLED display/imaging Bi-directional display/imaging prism combiner
(Uwe Vogel, Fraunhoffer Institute Dresden) (Hong Hua, U of A)

microDisplay

ET sensing: finite conjugate

IR sensor array Detail of

OLED /
detector
pixel
arrays on
a single
panel

Slide # 186
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Use waveguide for ET IR illumination (and also imaging maybe)

Slide # 187
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Gesture sensors

Solis radar chip from Google ATAP group Leap Motion IR gesture sensor
Finger gesture sensing

Slide # 188
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 6DOF tracking and SLAM

Slide # 189
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Product review: HoloLens V1

Slide # 190
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 191
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Hardware choices for HoloLens V1

Slide # 192
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 HoloLens Sensor Bar 4 Environment
Depth Understanding
Camera Cameras

2MP Photo /
HD Video Camera

Slide # 193
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 194
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 HoloLens Optics and IMU

IMU

HD 16:9 Light Engines

See-Through Lenses (waveguides)

Slide # 195
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 196
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 HoloLens MLB (Main Logic Board)

 Windows 10
 Custom-built
Microsoft Holographic Processing Unit (HPU 1.0)
 64GB Flash
 2GB RAM (1GB CPU and 1GB HPU)
 x86 architecture

Slide # 197
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 198
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 HoloLens Spatial Sound

also 4 microphones for speech/beamforming

Slide # 199
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Tri guide RGB WMA (Waveguide Module Assembly)
based on Surface Relief Grating (SRG)

Slide # 200
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 SRG coupling operates both in transmission as well as reflection modes

Slide # 201
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 The Optical Module
Assembly
(DMA + WMA)

Slide # 202
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Finalized OMA

Slide # 203
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Hololens V1 : 3 plates, 3 different periods for 3 different colors, same angle in guide
Max FOV in X and Y dictated by TIR in guide

Slide # 204
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Sensor fusion architecture in HoloLens V1
MIPI CSI
POP LPDDR3 POP LPDDR3
Depth

Mics x 4

I2C
Temp Sense
Headphones
CODEC I2S
MIPI CSI

Speakers x 2
PCle HPU EnvCam x 4

I2C
Temperature
Sense SPI
Intel
Cherry Trail IMU
Off the Shelf WIFI / BT
PCle
UART SOC
MIPI DSI MIPI DSI
DISPLAY
System
Custom Buttons / GPIO
LEDs Display x 2

MIPI CSI
SDIO
Photo/Vid
eMMC I2C

ALS
SOC Power ASIC Power
Rails Rails
Sensor
Gas Guage Power
PCM Rails
Power Button

Charger
MicroUSB

Slide # 205
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 HPU (Holographic Processing Unit): Chip Plot

• TSMC 28nm HPC DDR3 IO

PLL PLL
o Logic Gates: ~65M, SRAM : ~8 MB s
PLLs
s Compute
Package: 12mm x 12mm BGA, 0.4 mm pitch

PLL
o

s
o DRAM package: 1GB LPDDR3, 0.4 mm pitch
Compute

Accelerators and SRAM

• Sensor aggregator with environment and gesture
Compute
processing Compute

Fabric, Shared
• Computed output is very compact, e.g. fully
processed pose data to Host SOC
• 24 Tensilica DSP cores with custom TIE instructions Compute Compute
o “Tensilica Instruction Extensions” allows new
instructions to be added directly to the ISA Compute
• Compute

IO
Flexible DMA and fixed-function accelerators
o Stand-alone accelerators, as well as
Compute
o Accelerators tightly coupled with the DSPs Compute

IO
• Algorithms accelerated up to 200x over pure SW

PCIe
implementation Compute
• Low power (less power budget than Host SOC)
Compute
MIPI
 Slide # 207
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Video: HoloLens head tracking technology.

6DOF relocalization (IMU + lateral camera couples):

Correct drift through loop closure

Revisit same location later
Share between different users

Slide # 208
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Video: Depth sensing through IR TOF sensor

Slide # 209
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 VideoSurface reconstruction

Slide # 210
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Video: Semantic recognition in 3D scanning

Slide # 211
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Video: Gesture recognition

Slide # 212
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Video: Gesture understanding

Slide # 213
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Product review: HoloLens V2

Slide # 214
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 215
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Microsoft Silicon Architecture and Development Slide # 216
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Holographic
Silicon at the Heart of HoloLens 2
Processing Unit
(HPU) 2.0
Application
Processor

Depth
camera

Eye tracking
cameras
Head tracking
cameras

Fit system
Front Enclosure Back Enclosure Slide # 217
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 218
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 DISPLAY

Slide # 219
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 FULLY ARTICULATED HAND TRACKING

Slide # 220
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 SPATIAL MAPPING

Slide # 221
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 AZURE KINECT

Slide # 222
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 WINDOWS HELLO AND EYE-TRACKING

Slide # 223
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 224
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 UNIVERSAL FIT

Slide # 225
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 PIVOTING VISOR

Slide # 226
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 The V2 light
engine: dual
RGB lasers /
dual 1D MEMS
scanner (fast /
slow)

Slide # 227
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 HoloLens V2: 2 plates (RG-GB) and larger FOV by FOV splitting method (butterfly)
Exit pupil from DMA
ky/k0

DOE1

DOE2-a DOE2-b

Partial Partial
FOV Incoming FOV FOV
kx/k0
DOE3

Recombi
ned FOV

Slide # 228
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Hololens V1 : 2 plates, 2 different periods for 3 different colors, green truncated & shared
Max FOV in X and Y dictated by TIR in guide

Slide # 229
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 HPU (Holographic Processing Unit): Chip Plot
2nd Gen HoloLens HPU : includes DNN for AI
 Holographic Processor Unit 2.0

• TSMC 16FF+ 79 mm2 die

• 15x15 JEDEC POP, 0.4mm ball pitch

• 123M gates, 2 Billion Transistors

• 125Mbits SRAM

• Sept 2016 Tapeout,

“first time right” silicon

Microsoft Silicon Architecture and Development Slide # 231

SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Partial List, Holographic Processing Unit 2.0 Workloads

• Fully Articulated Hand Tracking

• Spatial Audio

• 6DOF tracking

• Hologram Stability

• Eye Tracking

• Spatial Reconstruction, Spatial Mapping

• Scene Understanding (e.g. find floors, walls, ceilings)

Slide # 232
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Hologram Stability

Slide # 233
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 DRAM Scheduler

HPU 1.0: Display Path

• 20% QOS bandwidth Display x 2
• 50% Utilization

Internal Sensors
DRAM Scheduler
HPU2.0:
• 50% QOS bandwidth Head Tracking (6DOF) x 4

• 75% utilization
IMU

Eye Tracking x 2

Depth

Microphones x 5
HPU 2.0

Slide # 234
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Product review: Magic Leap One

Slide # 235
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 236
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Lots of patents on stacked waveguide combiners and variable focus elements

http://www.patentsencyclopedia.com/assignee/magic-leap-inc/

Slide # 237
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Finally, a headset, pod and clicker unveiled
(12/2017)

Slide # 238
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 ML1 display architecture
In a nutshell:
a) Single plate LEDs (total of 6, only 3 ON at the a time) are
intentionally spatially demultiplexed, producing 6 spatially
demultiplexed collimated exit pupils (which are the 6
entrance pupils for the 6 waveguide combiner plates).
b) Imager is a field sequential (RGB) OMNIVISION LCOS
panel (0.37 inch @ 720p). LCOS/LEDs illumination Collimatio
n optics
optics includes conventional PBS.
c) There is no polarization recycling
d) Image is located at infinity over all input pupils.

LEDs
e) All 6 incouplers are 100% reflective blazed gratings.
f) Field is propagated by TIR in the various RGB guides PBS
while still being collimated (images at infinity)
g) DOE2 and DOE3 are simple binary gratings, very
shallow (depths ranging from 20nm to 60nm). Not
coated. LCOS
h) DOE3 (exit couplers) have an additional lensing
functionality to push the far field image at either 2m or
40cm in front of the user’s eyes.

Slide # 239
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 ML1 optical engine teardown

LCOS

LEDs

PBS

Collimation optics

Cover glass /
waveguides

Slide # 240
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 ML1 LED illumination
As scanned before plates (plates contain extra color
filters)
6 LED plane (RRGGBB)

lR=636nm

lb=457nm lG=525nm

dlb: 32nm dlG: 43nm dlR: 27nm

Slide # 241
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 6 RR-GG-BB LEDs Collimation optics Resulting 6 pupils overlapping with 6 DOE1s
plane

Near field R

Collimation optics
Far field G
Near field B Near field G
Far field B
Far field R

Far field R
Near field B Far field B
Near field G
Near field R

Far field G

Slide # 242
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 6 exit pupils overlapping 6 DOE1 injection gratings (reflective blazed gratings covered by 50nm of Al).
All exit pupils have also a color filter to clean up the spectrum.

The colors on this picture show the RRGGBB color filters, not any diffraction from gratings

Slide # 243
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 ML1 waveguide combiner stack:
8 plates stack, 6 high index grating combiner plates and 2 lower index cover plates

Total thickness is 8x0.330 + 7x0.25 = 2.815mm (Eye Tracking plate is extra)

Each color plate is 330mu thick

Air gap between plates is 25 mu
The 6 color plates are 1.7 index
Top and bottom cover plates are 1.5 index
Grating resin index is 1.53
All 8 plates have front and back AR coatings
Each DOE1 has a top color filter

World side
Cover plate
Far field red plate
2.775 mm

Near field red plate

Far field blue plate
Near field blue plate
Far field green plate
Near field green plate
Cover plate
Eye side
Slide # 244
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 ML1 input pupils on waveguide stack

Grating resin index is 1.53 6 individual spatially de-multiplexed entrance pupils to the waveguide stack.
All 8 plates have front and back AR coatings No pupil overlap allows for in-coupling gratings (DOE1) to be totally reflective
Each DOE1 has a bottom color filter Single side nano-imprint (DOE1 reflective, DOE3 transmissive).
Near and far field plates are interleaved, near field plates loser to the eye

Cover plate
Far field red plate
2.775 mm

Near field red plate

Far field blue plate
Near field blue plate
Far field green plate
Near field green plate
Cover plate

Each DOE1 is 0.9x1.2mm

1.2mm
0.9mm

Slide # 245
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 ML1 DOE1, DOE2 and DOE3 gratings for 2D EPE

DOE1 are all 6 pure reflective (Al coated)

DOE2 has 6 increasing etching depths, right to left
DOE3 has 6 increasing etching depths, top to down
DOE1
Depth increase from 20nm to 60nm (shallow!)
DOE2
DOE2 and DOE3 apertures are touching

DOE3

Slide # 246
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 ML1 grating structures
DOE1: DOE2: DOE3:
Blazed grating with 30% mesa to reduce shadowing Binary gratings with depth modulations Binary gratings with depth modulations
Bottom sputtered with Aluminum (30nm) Index is only 1.53 (no nanofillers). Substrate is 1.7 Index is only 1.53 (no nanofillers). Substrate is 1.7
Bottom color filter to clean up spectrum DOE2 works in reflection although transparent DOE3 works in reflection although transparent
Blazed angle is 27 deg, side wall is 79 deg Bottom AR coating Bottom AR coating
Grating depths vary between 20nm and 60nm Grating depths vary between 20nm and 60nm

Color plate DOE1 DOE2 DOE3

Red grating pitches @ 636 nm 460nm 336nm 460nm +/ 2.1 nm (far field)
460nm +/- 6.5 nm (near field)
Green grating pitches @ 525nm 380nm 280nm 380nm +/- TBD

Blue grating pitches @ 457nm 330nm 235nm 330nm +/- TBD

Grating orientation (deg) 79 32 -14

Side wall angles (deg) 27/79 90 90

Slide # 247
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Course conclusion

Slide # 248
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Today’s optical toolbox available
Dn
to the optical engineer
1.0

Diffractives
0.5 Fresnel/
micro-optics Refractives/GRIN/
Waveguides reflectives
0.1
Volume
0.01 holograms

D/l
0.01 0.1 1 10 100

Slide # 249
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Tomorrow’s newly stocked toolbox
Dn

1.0
Photonic
crystals
SBW gratings Diffractives
/resonant waveguide Fresnel/ micro-
0.5 refractives
gratings Refractives/GRIN/
GP / PB micro-grin
Waveguides reflectives
0.1 holograms, optics
DOEs/HOEs
Surface on curved surfaces
0.01
plasmons
D/l
0.01 0.1 1 10 100

DnRe DnIm

-0.5 Parity Time

Metamaterials, symmetry optics
Metasurfaces (gratings)
D/l
-1.0
Slide # 250
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 251
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 252
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 253
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20
 Slide # 254
SPIE Short Course SC_1218 – SPIE AR/VR/MR 20, Moscone Center, SF, CA. 02/20