Adv. Opt. Techn.

2021; 10(3): 145–232

Review article

Vladan Blahnik* and Oliver Schindelbeck

Smartphone imaging technology and its

applications
https://doi.org/10.1515/aot-2021-0023 1.3 Tomorrow’s smartphones ............................. 147
Received May 6, 2021; accepted June 29, 2021; 2 Mobile imaging ..................................................147
published online August 10, 2021 2.1 Market development ..................................... 148
2.2 Supply chain ................................................ 149
Abstract: Thanks to their portability, connectivity, and
3 Brief history and milestones of smartphone
their image performance – which is constantly improving –
imaging technology .......................................... 150
smartphone cameras (SPCs) have been people’s loyal
4 Physical properties and requirements of
companions for quite a while now. In the past few years,
Smartphone photography ................................ 150
multicamera systems have become well and truly estab-
4.1 Camera form factor and image sensor size ... 150
lished, alongside 3D acquisition systems such as time-of-
4.2 Image sensor resolution ................................151
flight (ToF) sensors. This article looks at the evolution and
4.3 Optical resolution and required aperture ..... 154
status of SPC imaging technology. After a brief assessment
4.4 Portrait photography: Perspective, bokeh, and
of the SPC market and supply chain, the camera system and
depth of field ................................................ 155
optical image formation is described in more detail. Sub-
4.5 Étendue and photographic exposure ........... 159
sequently, the basic requirements and physical limitations
4.6 David versus Goliath: The pros and cons
of smartphone imaging are examined, and the optical
of miniaturization ........................................ 160
design of state-of-the-art multicameras is reviewed along-
4.7 SPC lenses: How good are they actually? ..... 162
side their optical technology and manufacturing process.
5 The multicamera system in modern
The evolution of complementary metal oxide semi-
smartphones ..................................................... 164
conductor (CMOS) image sensors and basic image pro-
6 Optical System Design .......................................165
cessing is then briefly summarized. Advanced functions
6.1 Optical design structure of a smartphone
such as a zoom, shallow depth-of-field portrait mode, high
lens ...............................................................165
dynamic range (HDR), and fast focusing are enabled by
6.2 Optical design imaging performance ........... 172
computational imaging. Optical image stabilization has
6.3 Extreme wide-angle lenses ........................... 175
greatly improved image performance, enabled as it is by
6.4 Tele lenses .................................................... 176
built-in sensors such as a gyroscope and accelerometer.
6.5 Periscope tele lenses and alternative tele
Finally, SPCs’ connection interface with telescopes, mi-
concepts ........................................................176
croscopes, and other auxiliary optical systems is reviewed.
7 Zoom .................................................................. 177
Keywords: computational imaging; digital camera; multi- 7.1 Hybrid zoom in multicamera systems .......... 178
camera; optical design; photography; smartphone. 7.2 Optical zoom systems ...................................180
8 Opto-mechanical layout and manufacturing ....182
CONTENTS 8.1 Plastic lenses: Key for miniature opto-
1 Evolution of the mobile phone imaging system for mechanical layout ........................................182
the mass consumer market .............................. 146 8.2 Opto-mechanical layout ............................... 182
1.1 From mobile phone to smartphone .............. 146 8.3 Active optical assembly ................................ 183
1.2 Smartphones today ...................................... 147 8.4 Tolerancing and yield analysis ..................... 184
8.5 Wafer-level manufacturing ........................... 184
8.6 Anti-reflection coating for plastic lenses ...... 186
*Corresponding author: Vladan Blahnik, Carl Zeiss AG, Carl-Zeiss-
9 Image sensor ..................................................... 186
Straße 22, 73447 Oberkochen, Germany,
E-mail: [email protected]
10 Image processing ............................................. 189
Oliver Schindelbeck, Carl Zeiss AG, Carl-Zeiss-Straße 22, 73447 11 Noise and noise reduction .............................. 190
Oberkochen, Germany, E-mail: [email protected] 12 Focusing ............................................................ 191
146 V. Blahnik and O. Schindelbeck: Smartphone imaging

12.1 Autofocus methods: Contrast and phase So when we think of smartphones, why do we usually
detection .......................................................193 associate them with Apple’s iPhone if it wasn’t launched
12.2 Optical System changes focus position ........ 194 until 2007? The multimedia features of the first iPhone were
12.3 Focusing mechanisms: Voice coil motors and not incredibly impressive. It featured only a 2 MP camera,
other concepts ............................................. 196 with no autofocus or flash, and no front-facing camera. The
13 Image stabilization ............................................197 display only achieved a resolution of 480 × 320 pixels and it
13.1 Hand-shaking and image blur ...................... 197 did not even support the current third generation (3G)
13.2 Optical image stabilization mobile standard. Many smartphones that had come out
implementations .......................................... 200 years before actually outranked it. However, what was new
14 Dynamic range .................................................... 55 and appealing about it were the smartphone’s controls: it
14.1 HDR imaging ................................................ 201 featured a single button and a capacitive touchscreen of-
14.2 Lens flare and ghosts ...................................204 fering precise and intuitive control. The slide-out electronic
15 Portrait mode .................................................... 206 keyboard replaced the many buttons that mobile phones
15.1 3D depth acquisition technology ..................209 previously had, allowing for a much larger screen. And the
15.2 Simulation of lens bokeh: Camera 3D point phone itself responded to users’ actions – thanks to the
spread function .............................................. 68 ambient light, acceleration and position sensors. But even
15.3 Portrait look: a quality evaluation ................ 214 more important than the multimedia features was the
16 Image performance specification and test .......216 interaction between man and machine – a phenomenon
16.1 Lab evaluation during R&D .......................... 218 the iPhone rendered “smart.” But it was not until later
16.2 Evaluation of image quality in the imaging iPhone generations that the technical specifications were
pipeline ......................................................... 219 slowly brought into line with the state of the art.
17 Smartphone camera interface with telescopes, In 2007, Apple accompanied the release of the iPhone
microscopes, and accessory lenses .................. 220 with its dedicated App Store. This gave many small
18 Summary and outlook ...................................... 224 companies and individual developers the chance to
References ............................................................... 226 contribute to programs and market themselves without
having to worry about running their own sales opera-
tions. A service that probably helped Apple earn more
than its hardware did. This created a major dynamic in
1 Evolution of the mobile phone the software landscape. Furthermore, a key strategic
move that proved crucial for Apple’s success was its de-
imaging system for the mass
cision to partner with mobile network operators and offer
consumer market data packages alongside its devices. This gave users the
feeling that they were not making a costly mistake by
1.1 From mobile phone to smartphone purchasing from Apple. It was only then that users could
really start surfing the mobile web largely worry-free and
“A smartphone is a mobile phone that includes actually start using their smartphones to their full
advanced functionality beyond making phone calls and advantage.
sending text messages. Most smartphones have the capa- With its open operating system Android, Google was
bility to display photos, play videos, check and send Apple’s only competitor who proved to be a match for its
e-mail, and surf the Web.” [1]. iOS operating system in the long run. The Android oper-
As per this definition, one could posit that the ating system requires and enables a highly standardized
Personal Communicator launched by IBM in 1992, which hardware environment which, in turn, allows numerous
was capable of sending and receiving emails and faxes, other manufacturers to easily develop and offer smart-
was in fact the first-ever “smartphone.” The subsequently phones running on the Android system. While Android’s
launched Nokia Communicator and the Blackberry market share is far higher than that of iOS, it is very frag-
devices from RIM offered a kind of mobile office for one’s mented and split between multiple manufacturers, each of
pocket, and became an essential part of every manager’s whom have made their own adjustments to it. During the
briefcase. In 2005, Nokia offered its 770, a mobile com- initial phase in particular – but also to this day – this
puter with a large, capacitative touchscreen which, fragmentation makes it difficult to ensure reliable security
in hardware terms, is very close to what we call a smart- concepts and a consistent user experience, which is where
phone today. Apple comes out on top.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 147

Figure 1: The combination of sensors (e.g.

gyroscope, proximity sensor, GPS,
microphone, and camera), processing units
(CPU, GPU, etc.) and connections (e.g. Wi-Fi
and Bluetooth) create a whole world of ap-
plications.

1.2 Smartphones today transferred to external computing centers in the cloud.

This means smartphones are becoming less of a high-
Today, smartphones are much more than just compact performance minicomputer – and more a kind of platform
pocket knives boasting a series of nifty functions: their mix to manage how data is distributed to numerous subsystems
of sensors (“sensing”), data processors (“processing”), and like screens, eyeglasses, watches, cameras, health sensors,
their connectivity (“connecting”) create whole new func- and audio playback functions, thereby making all of these
tions and processes that no small device had managed to devices smart, too.
offer before – even in conjunction with several different So the smartphone itself is likely to take a backseat to
devices and plenty of time and effort (Figure 1). them all. For instance, smartphones will no longer need
Many new smartphone applications – such as navigation their own (large) screen, so it can be integrated into another
systems and augmented reality – are now possible thanks to a device, such as a smartwatch. And the lower processing
combination of sensing, processing and connectivity. Details power requirements mean that miniaturization can
on the status and evolution of these fundamental technolo- continue advancing.
gies can be found in the literature: For “miniature smart- Essentially, users will prefer to interact with these sub-
phone camera (SPC) sensors” see refs. [2, 3]; for processing systems, and smartphones will fade into the background.
and system on a chip: [4, 5] and connectivity [6]. This will lead to the different spheres of our lives being
Nowadays, it is much more likely for people to leave the seamlessly integrated, i.e. the time we spend at work, at
house without their wallet or car keys than without their home or on the go.
phone. That’s because the smartphone has redefined the lives
of a whole generation of people. Some have even grown up
with smartphones, which has completely changed their 2 Mobile imaging
values and habits. 30 years ago, most young people wanted a
driver’s license and their own car. As well as being a status The first cameras appeared in around 1840, and the poor
symbol, a car was their way of getting from A to B and staying sensitivity of the film used back then made them very bulky
in touch. Today, young people often view cars as a necessary indeed. They were anything but portable. Nevertheless,
evil, while the smartphone is seen as a gateway to the world continuous improvements in the sensitivity of photo-
and to one’s friends. The modern-day status symbol is pocket- graphic film by several orders of magnitude enabled sig-
sized. If you ask people today what item they couldn’t live nificant reductions in the size of the camera, which went
without, it’s not just young people who would choose their some way toward improving its portability. But the real
smartphone. The impact of smartphone’s on society has been breakthrough to making them portable came with the
discussed in several papers (e.g. [7, 8]). launch of roll film in compact cameras. This opened up
photography to large swathes of the population. The
original Leica camera released in 1924 became the icon of
1.3 Tomorrow’s smartphones this new technology. And while Leica, ZEISS, and other
premium manufacturers turned their attention to sophis-
So what might tomorrow’s smartphone be like? ticated photography equipment, others set their sights
New, faster mobile standards are allowing for ever squarely on the mass market – like the affordable Brownie
more data-intensive and storage-intensive functions to be cult camera from Kodak and other low-cost point-and-
148 V. Blahnik and O. Schindelbeck: Smartphone imaging

shoot cameras that featured plastic lenses as early as the released in recent years, the connection with the integrated
1950s. Alongside these were 1930s manufacturers like sensors, and highly advanced image processing have
Minox, who advanced camera miniaturization even further ensured that the image quality delivered by SPCs matches
[9]. But it was not until mobile phone photography that the that achieved with full-frame cameras [10, 11].
market began seeing new developments left, right, and
center, and the technology was brought to the masses.
2005 saw a very small number of mobile phones with built- 2.1 Market development
in cameras, and five years later people could not imagine
their phones without them. Today, more than half the Around the turn of the millennium, starting with the
world’s population owns a camera that they almost never transition from analog to digital cameras, the entire camera
go anywhere without. market has continued to grow exponentially. Since 2010,
We have grown accustomed to being able to the compact camera market has been shrinking consider-
take photos anytime, anywhere. And now, in 2021, some 2 ably, and by 2020 it had dwindled by more than a factor of
trillion (2000 billion) photos will likely be taken each 10. Sales of top-quality photo cameras with large image
year – which is twice as many compared with five years formats (full-frame, APS-C) with interchangeable lenses
ago, and five times as many compared with 10 years ago. have fallen by around half (Figure 2). Conversely, sales of
And upwards of 90% of them are taken using a camera-equipped smartphones have continued to rise. The
smartphone. quality of SPCs has been increasingly enhanced with high-
The image quality is actually very similar to that ach- resolution image sensors, image stabilization, and multi-
ieved with full-frame single-lens reflexes (SLRs) or system camera systems.
cameras across many photography scenarios. Some pho- Figure 3 shows how smartphone sales have developed
tographs can even boast resolutions of over 50 megapixels, (cf. Figure 3; data source: [12, 13]. The dashes indicate the
and films a resolution of up to 8K (8192 × 4608 pixels) at smartphones with a built-in camera. From 2002 to 2011, the
frame rates of 48 photo/s, which is much higher than those number of smartphones sold quadrupled. And since 2011,
on modern, high-resolution smartphone displays. Smart- that number has continued to rise each year, but is
phones with 512 GB of memory can hold hundreds of distinctly more moderate than it was for a few years after its
thousands of photos, however, the standard practice is to initial release. Today, around 200 times more smartphones
save a large volume of the images centrally, in a cloud, and are sold per year than photo cameras with interchangeable
then access them via a computer, tablet or other device on lenses. A similar trend can be seen in the compact camera
demand. Modern smartphones are thinner than 1 cm, segment. Over the past decade, smartphone sales have
weigh less than 200 g and easily fit into our pockets. risen by more than a factor of 20 over those of compact
A key factor for the success of camera phones – “the cameras – so SPCs have largely replaced the traditional
camera is fully integrated into the phone” – is something point-and-shoot variety. This is less true if we consider the
many manufacturers have tried to circumvent time and ratio of SLR or system cameras with interchangeable lenses
again, usually for technical reasons, by offering clip-on
additions, slide-on lenses, etc. This has resulted in these
systems becoming a niche product used by only a small
number of people.
And when it comes to the many sensors integrated into
a smartphone, in recent years in particular the camera
system and 3 dimension (3D) image acquisition have
improved in leaps and bounds. In 2016, the dual camera
made inroads into the smartphone world with Apple’s
iPhone 7+. Alongside the option of calculating high-
resolution depth maps using two or more cameras, lenses
were released with double the focal length compared with
the long-established standard wide-angle lens. In recent
years, even longer focal lengths as well as ultra-wide-angle
lenses have been launched. They are known as “zoom
lenses” and can continuously zoom in on subjects by
interpolating the camera images. The multicameras Figure 2: Camera sales since 2004. Data source: ref. [206].
 V. Blahnik and O. Schindelbeck: Smartphone imaging 149

Figure 3: Worldwide smartphone and feature

phone sales with breakdown of suppliers
(source: Gartner). Since 2015, smartphones
and feature phones have been separated in
statistics. Feature phones have been added
here, and the vast majority of them also
have cameras. Estimated number of SPCs
shown by blue dashed trend line. For 2021 a
forecast is given.

as compared with SPCs. For technological reasons, system Today, the major image sensor suppliers are Sony
and SLR cameras still outdo smartphones in a number of (revenue in this segment approx. $7 bn), Samsung ($2.5 bn)
features, which has resulted in a far lower substitution with and Omnivision ($1.5 bn). The majority of lens manufac-
smartphones for this segment. Here, the smartphone is not turers is based in China, Taiwan, and Japan, and include
a veritable competitor, but rather a welcome addition as
the “camera that is always to hand.”
Recently, traditional digital system cameras (DSCs)
and system cameras have adopted a number of functions
from smartphones (e.g. network connection and digital
workflows), which appears to have allowed the segment to
somewhat stabilize and recover.
The smartphone business is under tremendous pres-
sure to innovate. The latest features that customers want
need to be implemented in just a few months. So the market
is highly dynamic. In a few short years, many companies
are seeing their revenue grow exponentially, but shrink
again just as quickly. Nokia, Motorola, and Blackberry
Limited (previously Research in Motion, and most recently
TCL) are examples of such fallen giants.

2.2 Supply chain

The majority of smartphone manufacturers does not nor-

mally develop and build their own camera modules, which
usually comprise an image sensor, lens, actuators, and
sensors (Figure 4). They rely on a large network of Figure 4: Smartphone camera (SPC) module: Top left: Camera
specialist optics and module developers with whom they module cross-section, below: Lens barrel including voice-coil motor,
specify, develop and then supply new camera modules. bottom right: Image sensor module.
150 V. Blahnik and O. Schindelbeck: Smartphone imaging

Largan ($2 bn), Sunny Optical ($1 bn), and Kantatsu. In- name the actual pioneers. It is not uncommon for de-
tegrators (module assembly) include LG Innotek, Semco, velopments to be repeated until, after little initial success,
and Foxconn (Sharp) (approx. $3 bn in segment revenue they eventually established themselves in the long term.
each) (Data from ref. [14]. Examples of this include 3D dual cameras from 2011 (LG
The market is growing all the time, buoyed by the trend Optimus 3D, HTC Evo 3D, Sharp Aquos SH80F), some even
toward multi-camera systems. While the figure stood at with a 3D display as a forerunner of the general trend since
around 1.5 billion camera modules in 2010, this number 2016 in a slightly different form (smaller stereo base and
had almost quadrupled by 2020, to reach 5.5 billion. In calculation of depth maps instead of stereo images), or the
addition to being built into smartphones, tablets and lap- Nokia Lumia 1020 from 2012 with a very large 41 MP image
tops, small camera modules are also developed and pro- sensor and pixel binning, which has been revived in many
duced for automobile, security/surveillance, medical, high-end camera systems since 2018.
drones, and industrial applications, as well as in robotics. In very simple terms, one can say that we are currently
Smartphones hold the majority of the market share, at over in the third era of smartphone imaging developments: 2002
80% [15]. Total global revenue for mobile device camera to around 2010 was the era of the “MegaPixel Race”, and
modules currently stands at around 30 billion dollars. At 2008 to around 2016 the era of coupling the camera system
the same time, the manufacturers have been under to advanced software algorithms and smartphone sensors
tremendous price pressure, which has increased consid- like the gyroscope or accelerometer (Imaging Apps, Noise
erably in recent years. Reduction, HDR; Image Stabilization). As of 2016 we have
In modern smartphones, the camera system is often been in the MultiCamera and Computational Imaging Era
the most expensive function in a smartphone [16]. All with 3D sensing heading towards Augmented Reality.
camera modules, from high-end smartphones together
(multicamera system and front-facing camera on screen
side), cost between 20 and 80 dollars. That equates to be- 4 Physical properties and
tween 6 and 20 dollars per camera. 40% of those costs can
be attributed to the image sensor, 30% to the optics incl. requirements of Smartphone
image stabilization and autofocus, and 30% to the module photography
(module and assembly).
Besides the rising number of multicamera systems, We consider practical photographic requirements to derive
and those on the front and back of a smartphone, modules the basic technical specifications of a SPC. We then
are increasingly being integrated for the purposes of hu- compare the physical dimensions of these miniaturized
man identification (or “biological recognition”): optical optics with a SLR or system camera for a large-image
fingerprint modules, iris recognition systems, 3D struc- format.
tured light, 3D time-of-flight (ToF) and lidar systems [17] all
contain one or several camera imaging modules. They are
usually smaller in size and simpler – i.e. with 3 or 4 lens 4.1 Camera form factor and image sensor
elements – compared to a smartphone’s main camera size
module, e.g. the standard wide-angle camera on the rear
usually contains 6 or 7 lens elements. Consumer demand for the smartphone that is “always to
hand” means that the camera should be encased in a very
flat housing 7–10 mm thick, so it requires a miniaturized
3 Brief history and milestones of camera. Minus the dimensions for the housing and the
sensor board, the depth of a cell phone optic must not
smartphone imaging technology exceed 5 or 6 mm.
The “relative flatness factor” (r) of the optical design
In Table 1 we give a chronological list of milestones in
L
mobile phone imaging. One could extend this list indefi- r= (1)
⊘im
nitely. The aim is more to give some examples of which
developments have had a lasting impact on mobile imag- together with the overall length (L) determines the still
ing system evolution than to claim completeness or to feasible full image diagonal ⊘im.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 151

Table : Milestones in smartphone imaging.

 Camera phone patent by Nokia employees ref. []

 P. Kahn connected a digital camera with his mobile phone and wirelessly shared a picture of his newly born daughter with  people
around the world
 In November the world’s first mobile camera phone is released, the Sharp J-SH with . MP; only distributed in Japan
 Launch of third-generation wireless mobile communication standard G with UMTS (Universal Mobile Telecommunication System) with
data rates of  kBit/s enabling picture transfer in mere seconds
 Nokia : first mobile phone with camera in Europe: . MP color camera with first color display
Sanyo SPC- first camera mobile phone (. MP) in US market
 NTT DoCoMo Mova iS: first mobile phone autofocus camera and exceeding  MP (. MP)
 Sony Ericsson Z: first mobile phone with front-facing camera supporting video calls (front and back camera both . MP)
 Nokia N with  MP camera (ZEISS) with autofocus, LED flash, video; unique swivel design with camcorder feeling
 Sony Eriksson Ki with . MP and Xenon Flash; Nokia N also launches . MP; Nokia launches integration of Flickr to upload photos
 Apple's iPhone revolutionizes operating concept via seamless touchscreen and one button only; birth of the “smartphone”; becomes
new standard system layout copied by all other suppliers
 Launch of app platform, and open development system iOS
Android becomes non Mac platform
 Many phone cameras with  MP; Samsung SCH-B with  MP
 Consumer mass produced backside-illuminated CMOS from Sony (Exmor R) improves low light performance by a factor of two
 Apple's iPhone  with gyroscope for precise orientation determination
 Nokia PureView  followed by Lumia  feature  MegaPixel Sensor with Pixel Binning to  MP “high-end digital zoom”
 Nokia N first smartphone featuring optical image stabilization
 Rise of Dual Cameras (beginning of multicamera systems in smartphones): iPhone + with different focal lengths:  mm, and  mm
equivalent focal length for hybrid zoom and portrait mode
Huawei P (Leica) with identical FOV, RGB/monochrome sensor for increased resolution, lower noise and portrait mode
LG G: Standard wide angle combined with extreme wide angle (° FOV) for hybrid wide-angle zoom
 Samsung Galaxy S with Dual Pixel Autofocus
 HEIF compression format in Apple iPhone  + halves space required by pictures and film
 Samsung Galaxy Note  dual cam with synchronized optical image stabilization for both cameras
 Sony Xperia ZX with Super Slow-Motion Video >  frames/s
 LG V with real Color Grading (instead of simple color filters)
 iPhone X with D structured light depth sensor, e.g. for face recognition
 Huawei Honor View  D Time-of-Flight depth sensor
 Huawei P Pro and OPPO Reno × feature  mm periscopic tele lens to extend hybrid zoom range
 Apple iPhone  and other flagship smartphones released with wireless communication standard G

The image sensor should be as large as possible so that several high-end smartphone models (e.g. Huawei P40,
as much light as possible falls on a pixel. This reduces Xiaomi Mi 10+, vivo X60 Pro +).
fundamental disadvantages such as image noise, a
reduced dynamic range or longer exposure times and thus
motion blur. The factor r depends essentially on the field of 4.2 Image sensor resolution
view (FOV) of the lens and its layout (standard upright,
periscopic, etc.), which we will discuss in more detail in Smartphones typically offer a standard image resolution of
Section 6. For wide-angle lenses, r = 0.83 is typical, which 12 megapixels (12 MP). SPC image sensors have an aspect
means that an image diagonal of 6 mm is obtained with an ratio of 4:3, in contrast to the aspect ratio of 3:2 for full-
overall length of 5 mm. For example, the image sensor of format (36 × 24 mm2) or the APS-C format (approx.
the Apple iPhone 6 has a sensor diagonal of 6 mm. With a 23.6 × 15.7 mm, depending on camera manufacturer). The
slightly larger construction depth of around 8 mm, i.e. with more square format is a compromise so that a picture taken
slightly thicker smartphones or mostly a slightly protrud- in landscape format does not appear too narrow when
ing camera housing, as well as complex 7-lens designs, for viewed in portrait format.
which r = 0.65 is possible, an image diagonal of more than The image sensor formats are usually referred to as
12 mm can be achieved. Such image sensor sizes of the “inch values.” The aforementioned image sensor that
main camera (standard wide-angle) are integrated into measures 6 mm in the diagonal would be referred to as
152 V. Blahnik and O. Schindelbeck: Smartphone imaging

1/3”. Unfortunately, the conversion of the unit 1” = 2.54 mm manufacturers, under the names “Tetracell technology”
does not itself lead to the actual dimensions of the sensor: (Samsung), “Quad-Bayer” (Sony), and “4-Cell” (OmniVi-
25.4 mm·1/3 = 8.3 mm does not correspond to any of the sion). These are arrangements with pixel clusters or
actual side dimensions, i.e. 6 mm in the diagonal, 4.8 mm “macro-pixels” of 4 or 9 pixels (Figure 5). The standard
in width, or 3.6 mm in height. The inch specification was output images are 48 MP/4 = 12 MP for a typical 48 MP
taken from old Vidicon video tubes from the 1950s and sensor with a 4-pixel cluster or 108 MP/9 = 12 MP for the
corresponded to the outer glass diameter of the photo- 9-pixel cluster at 108 MP (e.g. in the Oppo Reno 3). Mac-
electric front surface. The sensor diagonal corresponds to ropixels do of course have enhanced light sensitivity
about 2/3 of the inch value, but only roughly, and sensor compared to a single pixel. This could also be obtained by
sizes are unfortunately not exactly proportionally scalable just using larger pixels. However the benefit of using a pixel
according to the inch values. In other words, the absolute cluster is flexibility and additional advanced features. The
inch sizes are antiquated and misleading. So it is better to image data of the multicell sensors can be read out in
determine the exact sensor dimensions in millimeters. various ways: By selecting different sensitivities or expo-
Mobile phone camera image sensors were developed sure times of the pixels, the dynamic range can be
between 2002 and 2010, raising pixel counts from 0.3 MP to increased (see Section 14) or the noise can be reduced
around 12 MP. Keeping the same image sensor size of about through pixel binning, or a very high-resolution image can
6 mm in the diagonal, this corresponds to a reduction in be output. A clear distinction should therefore be made
pixel pitch from about 6 μm down to 1 μm. The pixel race between sensor pixels, i.e. the pixels available when
had come to an end by around 2012 and even began partly recording, and the number of pixels used to display the
moving in reverse in the following years since many SPC images.
suppliers had recognized that pixel amounts far greater Figures 6 and 7 show the trends of pixel numbers of
than 12 MP are not beneficial due to practically unused main rear camera and front camera. In the years after 2016,
resolution. What is more, handling with reasonable pixel shrinking was pushed further down, reaching 0.7 μm
amount of image data and better image noise and dynamic by 2020: First, so that the additional cameras of different
range were consequences of the reversed pixel race [18]. focal lengths that are being launched can achieve the
Nonetheless, it should be mentioned that as of 2018 an desired resolution of 8 or 12 MP (standard Bayer Pattern).
increasing number of main cameras in modern SPCs have Second, the aforementioned multi-cell sensors of the main
featured image sensors with a high number of pixels: camera: Mastering the production of the image sensors of
48 MP, 64 MP or even 108 MP. This suggests a very high such small pixels in such high volumes through mass
resolution, especially, since these sensors are often production with a high yield is an additional challenge
referred to as “high resolution” in product marketing. compared to the small sensors with the same pixel pitch.
However, this is misleading: The arrangement of the color Even the “typical image resolution of 12 MP” in today’s
pixels is usually not made up of adjacent Bayer patterns. images captured using a smartphone is seldom exhausted
The number of pixels in the final image is typically around in terms of the physiological limits of the human eye under
12 MP as standard, which is also effectively supported by typical viewing conditions. Most of the images are viewed
the optical performance of the lens. Most of these image directly on the smartphone and rarely exceed the size of a
sensors are not standard Bayer sensors, but rather PC monitor or TV. With the resolution of the eye of around
“multicell sensors” as they are called by the ϑres = 0.3 mrad or 1 arc s [19], the result for the typical

Figure 5: Image sensor architectures: Standard Bayer pattern and multi-cell configurations.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 153

Figure 6: Main camera pixel count trend: As

of 2016 most cameras have had about 12 MP
(10–13 MP line). Currently >32 MP sensors
are increasingly used: These are mainly
Quad-Bayer pattern sensors, so while the
spatial resolution does not increase, they
do offer improved HDR capabilities.

Figure 7: Corresponds to Figure 6, but

regarding the front camera. The main trend
of 5–8 MP cameras is being supplemented
by > 32 MP sensors (mainly larger Quad-
Bayer pattern, which improves HDR and
other properties).

observer distance of around s = 25–40 cm under optimal than 2000 × 1000 pixels are required. Experimental studies
conditions (perfectly still image) produces an object reso- confirm this (e.g. [20]). The resolution of smartphone dis-
lution that is just barely distinguishable, of: plays corresponds roughly to this HD resolution, typically
around 2300 × 1080 pixels; higher-resolution displays like
Δxres = s ⋅ ϑres ≈ 330 mm ⋅ 0.3 mrad = 0.1 mm. (2)
that of the Sony Xperia with 3840 × 1644 pixels are the
This corresponds to a smartphone display measuring absolute exception here. Significantly more pixels are only
140 × 70 mm, so even with a large 6.2”. display, only necessary if you want to create a poster from the picture and
1400 × 700 pixels can be effectively resolved. With optimal look at it up close or if you enlarge image sections. This also
viewing conditions – i.e. bright ambient light and a applies to VR applications in which scenes on the smart-
completely stationary smartphone without hand tremors, phone screen are viewed significantly enlarged “under a
and even closer eye relief to the display – at best no more magnifying glass” (Samsung Gear VR, ZEISS VR One, etc.).
154 V. Blahnik and O. Schindelbeck: Smartphone imaging

Such observations on the screen – which are enlarged With SPCs, and all digital cameras in general, the im-
by a factor of 3 or separately on a screen with a 4K age quality is specified via the contrast of (quasi) periodic
projector – benefit from a resolution of beyond 12 MP. structures, which in turn is related to the modulation
transfer function (MTF). The concept of this linear transfer
function opens up the possibility of combining the com-
4.3 Optical resolution and required aperture ponents of the digital imaging chain (optics, sensor, and
image processing – as long as they are not correlated) in
In order for the resolution of the image sensor to be used, order to calculate the overall contrast transferred (for a
the quality of the optical image as viewed through the lens comprehensive analysis see ref. [21]). From the pixel pitch
must be good enough. Let us now consider the number of of the image sensor we have:
pixels of the image sensor and the resolution required for 1
Nyq = (6)
the optical image. Let us continue with the example of the 2p
image sensor measuring 6 mm in the diagonal and 12 MP:
which is the Nyquist frequency (Nyq) in the horizontal and
The side lengths in the aspect ratio 4:3 are 4.8 × 3.6 mm. At
vertical direction; in our example, p = 1.2 μm, we have
12 MP, i.e. 4000 × 3000 pixels, this corresponds to a pixel μm
1000 mm
Nyq = 2⋅1.2μm ≈ 416 mm
lp
. This is the smallest period that the
pitch of 1.2 μm.
image sensor can in principle still resolve or sample as such.
The point spread function diameter produced by the
However, this only applies to a limited extent: As shown in
lens must be close to the pixel pitch of 1.2 μm in order not to
Figure 8, a periodic intensity distribution close to the Nyq is
limit the performance of the optics. The diameter of the Airy
mapped with a vastly differing contrast depending on the
diffraction disk of the ideal image is
exact position relative to the pixel grid; in the worst case even
⊘Airy = 2.44 λK (3) with no contrast at all. For this reason, it does not make sense
to specify the image performance for Nyq itself, but rather
where λ denotes the wavelength, that is approx. 0.4–
starting at around Nyq/2 and higher spatial frequencies,
0.7 μm for the visible light range, and the f-number K is
where the deviations of the integrated sampled signal from
related by K = 1/(2 NAʹ) to the numerical aperture NA′ on the
the original signal become increasingly smaller (Figure 9).
image. An appropriate relationship between the diameter
The system contrast is often displayed simultaneously with
of the Airy spot ⊘Airy and the sensor pixel pitch p is to
different fine structure periods, such as Nyq/8, Nyq/4, Nyq/2.
choose
For p = 1.2 μm, these would be the spatial frequencies 52, 104,
⊘Airy = 2 ⋅ p (4) and 208 lp/mm.
The product of the optics MTF and sensor MTF has an
Then, a significant portion of the light distribution,
increasingly statistical character near the Nyq because of
namely a relative encircled energy of 0.73, is inside a
the position dependence.
square-shaped pixel. Further, if we consider that the in-
tensity is transferred to a grey value distribution by the
photo conversion curve and opto-electronic conversion
function, the “effective encircled brightness” to give a
name to the gray value distribution, is in fact >0.8. We set
the corresponding “critical f-number” Kcrit as a require-
ment for the lens:
p
K crit = (5)
1.22 ⋅ λ
This would be Kcrit = 1.8 (or the aperture ratio f/1.8) Figure 8: Illustration of capturing a periodic optical intensity
with the pixel pitch p = 1.2 μm. Most of today’s standard distribution with a periodic pixel array. The periodicity of the
wide-angle lenses for smartphones meet this requirement. intensity corresponds to the Nyq . The grid period is exactly 2 pixels
However, “telephoto lenses” (normal focal lengths and here. The periodic intensity distribution (blue) generates a signal on
the pixel array, which is represented here by the gray values of the
short/long portrait focal lengths) taken using modern dual
individual boxes (“white pixels”: strong light signal, “black pixels”:
or multiple camera systems fall below this requirement, no light). In (a) the pixel array distribution perfectly matches the
with f-stops of 2.4 or more and often even smaller pixels of intensity distribution. (b) A structure shift of half a pixel results in a
usually less than 1 μm, e.g. p = 0.8 μm. completely homogeneous distribution on the image sensor.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 155

are “diffraction-limited.” This means that with the same

image resolution (relative to the respective pixel size) and
the same lens aperture with an ideal aberration-free lens,
the contrast is weaker.
The aberrations of SPC lenses are so small that stop-
ping down would lead to a weaker contrast. Therefore,
and to minimize the complexity by dispensing with
Figure 9: Same as Figure 8, but spatial frequency is Nyq/2. moving parts, SPC lenses are not even equipped with iris
diaphragms. The exposure is adjusted solely via the
In contrast to most full-frame reflex or system cameras, exposure time and the International Organization for
a SPC does not contain an optical low-pass filter, which Standardization (ISO) sensitivity via the read-out ampli-
suppresses the moiré effect, i.e. disruptive low-frequency fier on the image sensor. But with full-frame lenses, the
beats caused by periodic structures, e.g. a finely checked contrast is usually weaker, with a more open aperture
shirt, which are insufficiently scanned by the equidistant than when you stop down. Often the maximum contrast is
pixel grid. This is due, first, to the fact that optical low-pass obtained at around f/4, f/5.6, or f/8, before the contrast
filters are usually implemented using birefringent struc- decreases with further stopping down to the diffraction
tures with a thickness of around 2 or 3 mm, which is un- limit. The loss of contrast with an open aperture is due to
acceptable for SPCs for reasons of space, and which are the fact that larger aberrations are allowed in favor of a
also relatively expensive, and second, because the point simpler, more compact design. Exceptions are premium
spread function of the SPC lens is already larger than a lenses such as the ZEISS Otus or SIGMA Art series, which
pixel and therefore has a low-pass effect. already have maximum contrast at f/1.4 and f/2, but at the
An ideal incoherent optical system transfers informa- expense of a significantly more complex optical design,
tion up to a limited spatial frequency of size, and weight.
Resolution is not independent from signal-to-noise
NA′ 1
νmax = 2 = (7) ratio of the image sensor; a more rigorous analysis can be
λ λK
found in ref. [23].
For f/1.8 and green light of wavelength λ = 0.55 μm, the
transferred spatial frequency limit is νmax = 0.55 μm⋅1.8
1
≈
1000 lp/mm. The contrast of a periodic signal decreases ac- 4.4 Portrait photography: Perspective,
cording to the MTF of the optical system with increasing bokeh, and depth of field
spatial frequency. With ideal incoherent imaging with a ho-
mogeneous circular pupil corresponding to ref. [22] A popular “allrounder lens” is the classic standard wide-
angle lens with a focal length of 35 mm for the full-format
MTFideal (ν)
(abbreviated “ff”) 36 × 24 mm2, i.e. a full image diagonal of
⎪
⎧ ⎡ ⎤ √̅̅̅̅̅̅̅̅
⎪
⎪ ⊘im, ff = 362 + 242 mm = 43.3 mm. An equivalent focal
⎪
⎪ ⎢ ⎥
⎪
⎪2⎢ ν ν ⎥
⎪
⎪ ⎢arc cos( ) − √ ̅̅̅̅̅̅̅̅ ̅ ⎥, ν ≤ 2νmax
length of 35 mm was also typical in the early days of
⎪
⎨π⎣ νmax 2⎦
smartphone imaging, but this changed after a couple of
=⎪ ν
νmax 1 − (νmax )
⎪
⎪ years to about 28 mm (more detailed data are given in
⎪
⎪
⎪
⎪ Section 6.1). This corresponds to a full diagonal field of
⎪
⎪
⎪
⎩ 0, else view (FOV), related to focal length and image field size by
(8) ⊘im, ff /2
tan(FOV/2) = (9)
f′
This function falls in a monotonous, almost linear way,
towards larger spatial frequencies (Figure 10). The same of FOV = 2 arctan(⊘im, ff /( 2f′)) = 2arctan(43.3 mm/56 mm)
figure also shows the contrast for spatial frequencies rele- ≈75° . The 28 mm focal length equivalent for the image sensor
vant for SPC (pixel pitch of 0.8 or 1.2 μm, corresponding to with a 6 mm image diagonal is an actual focal length of the
about 200 or 300 lp/mm) compared to the finest structures SPC of
typically observed with full-frame cameras (40 lp/mm).
′
f eq
The corresponding number of pixels on the image sensor is f′ = ⊘im, SPC (10)
⊘im, ff
similar for these values, approx. 12 MP. This makes it clear
that SPCs are physically limited by their size alone. They so (28 mm/43.3 mm) × 6 mm = 3.9 mm or about 4 mm.
156 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 10: MTF of ideal lens. The achievable

contrast in the Nyquist/2 spatial frequency
is smaller for miniature SPC systems.

For a “close-up portrait”, in which a face (height Typical normal portrait distances with a vertical object
from chin to crown approx. 30 cm) almost fills the ver- side length of 0.72 m are about twice as large, i.e. 0.8 m. To
tical image field, with a 36 cm vertical section, the dis- have people completely in the picture, yob = 2.16 m, again
tance to the object (measured from the entrance pupil) is about three times as much, i.e. 2.4 m object distance. So
about you can still see a lot in the picture in most cramped situ-
2yob ations such as indoors, in a group photo or when dining at
s= f (11) the same table with others.
2yim, SPC
For reasons of perspective, wide-angle lenses are not
where 2yim is the total length of the side of the vertical, that well suited for portraits: To fully depict a person, one has to
is the “short direction” of the 4.8 × 3.6 mm image sensor. get very close up, at feq = 28 mm – the aforementioned
The object distance is s = 360 mm/3.6 mm·4 mm = 400 mm, 0.4 m. Then the nose will be 10–20% of the object distance
i.e. 0.4 m. in front of the ears, so it is imaged magnified and leads to
Since 0.4 m is also a typical distance at which one deformations on the face shown. Classic portrait focal
holds a smartphone, including for video calls or selfies, one lengths have an equivalent focal length of approx. 85 mm
can often choose an equivalent focal length of 28 mm for that is about three times longer. A face looks much more
the front camera. pleasant in a perspective with a portrait lens (see Figure 11).

Figure 11: Portrait with “portrait lens” of feq = 85 mm (left) shows much less “perspective deformation” compared to a feq = 28 mm wide-angle
lens (right).
 V. Blahnik and O. Schindelbeck: Smartphone imaging 157

Figure 12: Image taken with a smartphone (left) and a DSLR (right) with a lens with the same field of view (FOV) (feq = 28 mm or FOV 75°) and
same f-number (f/2.2).

The pictures in Figure 11 were shot with a full-frame By comparing the triangles in the image space, the ratio of
digital single-lens reflex (DSLR): At a large aperture (here the radius of the defocused point image rspot to the position of
f/2.2) the person is detached from the background due to a the defocus in the image space (sʹF − sʹ) is obtained:
shallow depth of field. A direct comparison of the two
r Ø 1
lenses in Figure 11 shows that a light source in the back- ⃒⃒ spot ⃒⃒ = AP = (12)
⃒⃒s′ − s′ ⃒⃒ 2s′F 2K
F
ground, the defocused point spread function, is larger with
a longer focal length. It can be shown that the spot diam- The spot radius is thus
⃒⃒ ′ ⃒
eter with the same f-number and the same image format ⃒⃒s − s′ ⃒⃒⃒
F
scales approximately in the ratio of the focal lengths. r spot = . (13)
2K
With SPC lenses, however, the depth of field is large.
The whole scene, including the background, looks sharp For an optical system, represented by its entrance and
(see Figure 12). exit pupils, the focusing conditions are from ref. [24], where
The size of the point spread in the depth – and from this mp denotes the pupil magnification:
the depth of field – can be calculated using a geometrical 1 mp 1
model: According to Figure 13 the variables s, sʹ denote the − + ′ = (14)
mp s s f
object and image distance with respect to an out-of-focus
object point, relative to the entrance and exit pupil of the and
optical system, respectively; sF and sʹF denote the corre- 1 mp 1
sponding distances for which the lens is focused, and f the − + = (15)
mp sF s′F f
focal length.

Figure 13: Lens represented as black box by

entrance and exit pupil and definition of
parameters with respect to focus (F ) and
out-of-focus (not indexed) distances to
calculate spot diameter 2rspot.
158 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 15: Portrait: Proportions and sizes.

Figure 14: Entrance pupil (EP) and exit pupil (AP) of a standard wide-
angle smartphone lens (drawn with respect to on-axis imaging). The
position results from the intersection of the (extended) chief ray in
the object or image space with the optical axis; the size of the pupil ∅rel.spot, ∞
in each case is obtained by lengthening the marginal rays in the f2 |s − sF | f2 (19)
= lim =
object or image space up to this position. s→∞ ∅im K (f /mp + s)(f /mp + sF ) ∅im KsF

With the definition of the f-number, K = ∅fEP , and the

respectively. Replacing the image distances sʹ, sʹF in Eq. magnification (approximated for large object distances)
(13) by object distances s, sF with Eqs. (14) and (15) yields: m = sFf+f ≈ sfF we obtain:
f2 |s − sF | ∅EP
rspot = (16) ∅rel.spot, ∞ = m (20)
2 K (f /mp + s)(f /mp + sF ) ∅im

Pupil magnification mp depends on the specific optical For a portrait motif as in Figure 15, the diameter of the
design. For wide-angle SPC lenses (Figure 14) the value of object field, i.e. the distance between opposite corners, is
mp is typically between 0.5 and 1. around 700 mm (more precisely 666 mm for 4:3 format,
If the object distances are much larger than the focal 722 mm for 3:2 format):
length, which applies to SPC lenses, since s is at least about ∅im
20 times as large as f, then the term f/mp in the denominator ∅ob, Portrait = 700 mm = (21)
m
of Eq. (16) can be neglected. The diameter of the image’s
circle of confusion Øspot = 2rspot is then: Inserting (21) in (20) yields that the relative diameter of
the circle of confusion only depends on the diameter of the
f 2 |sF − s| entrance pupil:
Øspot = (17)
K sF s
∅EP ∅EP
∅rel.spot, ∞ = = (22)
In order to compare the imaging of lenses with ∅ob, Portrait 700mm
different image formats, we define the relative circle of
confusion (based on the image diagonal) instead of the With this formula you can immediately estimate the size of
absolute circle of confusion in units of length: the highlights far in the background for our standardized
portrait situation: With a long full-frame portrait lens of
∅spot 2/135 mm, the diameter of the entrance pupil is
∅rel.spot = (18)
∅im ∅EP = Kf = 135 2mm = 67.5 mm, and thus the relative circle of
This value directly indicates the spot size in the defo- confusion 67.5 mm/700 mm, i.e. about 10%.
cused area as it appears in the photo. With a wide-angle SPC lens, the entrance pupil is only
For a very distant background, the relative circle of ∅EP = Kf = 4 1.7
mm
= 2.3 mm and the relative circle of confu-
confusion will be: sion is only 0.3%.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 159

Inserting the expression f

sF, hyp = +f (28)
f ∅im 2K 1
2 )
tan(FOV
∅EP = = (23) 1500
K 2Ktan(FOV/2)
The hyperfocal distance of two lenses with the same
and replacing NA′ = 21K and using the abbreviation cFOV =
1 FOV but different focal lengths, i.e. lenses with the same
tan( FOV/2)
in (22) yields:
equivalent focal length, scales directly with the focal
cFOV length (or with the sensor format). For an SPC standard
∅rel. spot, ∞ = ∅im NA´ (24)
700 mm wide-angle lens with a focal length of 4 mm and an aper-
ture of f/2 for a sensor with a diagonal Øim = 6 mm,
For a given FOV, the relative diameter of the circle of sF, hyp = 2 m, i.e. the image is sharp from 1 m to infinity.
confusion is therefore directly proportional to the With the equivalent full-frame lens, on the other hand,
product of the image field diameter and the numerical from 7 m to infinity. The autofocus is therefore only
aperture of the image. For a constant transmission required for close range, e.g. to photograph documents.
within the pupil and field, the product of field area and The typical close-range distance of standard wide-angle
squared NA, lenses in smartphones is approx. 80–100 mm with a
π 2 2 magnification of approx. 1:20. The object area of
G= ∅ NA′ (25)
4 im 96 × 72 mm shown on a 4.8 × 3.6 mm is only slightly larger
than a standard 85 × 55 mm business card. The depth
is the optical systems etendue. Other names for etendue are
of field is then only about ±3 mm. An autofocus with
“throughput,” “collecting power,” or “ΑΩ product.” This
the appropriate accuracy is required to achieve this (see
means that the relative circle of confusion is proportional
Section 12).
to the root of the etendue value:
The physical – and for the photographer, the creative –
cFOV √̅̅ limitations imposed by the large depth of field are over-
∅rel.spot, ∞ = G (26)
700 mm come in modern smartphones in “portrait mode” with the
depth determined stereographically and/or with high-
In other words: To achieve the same background cir- resolution 3D sensors via the image. And then, according to
cle of confusion, the etendue of the small lens must be the depth distance from the focal plane, the image is
equal to that of a full-format camera lens. For an image “calculated out of focus” (see Section 15).
diameter that is 7× smaller, the NA would have to be a
factor of 7× larger. Of course, this is no longer possible
compared to high-aperture camera lenses, e.g. with f/1.4, 4.5 Étendue and photographic exposure
i.e. NA´ = 1/(2·1.4) = 0.36, the SPC lens would have to be
f/0.2 or NA´ = 2.5. In the previous Section 4.4 we established a direct
Depth-of-field formulas are obtained from the equa- connection between the bokeh and the etendue. Étendue is
tions for the size of the geometric spot (circle of confusion) very important for photography because it defines the
if a threshold value Øthres is defined for the spot size for exposure control. In addition to the amount of light
which the image still appears “sharp” and resolved ac- described by the etendue that reaches the image plane
cording to the object distances [25, 26]. from the object space, the photographic exposure H is
Specifically, one obtains the hyperfocal distance to controlled by the sensitivity of the film or the image sensor
f2 and the exposure time T. Exposure Control in practical
sF, hyp = +f (27) photography is described as follows:
KØthres

With regard to this focus distance, the image is sharp from 1 2

H∼ISO × ( ) × T (29)
sF, hyp /2 to infinity. For many normal viewing conditions, K
the threshold value of the circle of confusion is chosen as “ISO” refers to the sensitivity of the image sensor.
1/1500 of the image diagonal: Øthres = Øim /1500. With Essentially, the sensitivity depends on the “area of a pixel”
because this determines how many photons strike each
2 )
tan(FOV = Ø2imf we obtain:
time. For a 4:3 aspect ratio sensor the horizontal side length
160 V. Blahnik and O. Schindelbeck: Smartphone imaging

of the image sensor is √̅̅̅4 ̅

⊘ = 45⊘im = 0.8⊘im and the
42 +32 im 4.6 David versus Goliath: The pros and cons
short side length correspondingly √̅̅̅3 ̅
2 2 ⊘im = 0.6⊘im , such
4 +3
of miniaturization
that the total sensor area is 0.48 ⋅ ⊘2im . Division by the total
number of pixels yields the surface area of a pixel to 0.48 ⋅ From the photographic imaging equations above, we may
⊘2im /(#Pixel). The ISO value also indicates how efficiently a derive scaling rules for a direct comparison between
“large,” e.g. a full-format (36 × 24 mm2) DSLR or system
pixel absorbs light and converts it into an electrical voltage
cameras and a much smaller SPC (Figure 16). The com-
(see Section 9). The desired exposure time in an SPC is
parison shall be done for the same photographic situation:
achieved by pure electronic control of the image sensor,
A scene, e.g. a portrait of a person or a landscape, taken
while in larger-format cameras a mechanical shutter is
with the same field of view from the same position (the
usually available, sometimes integrated with the iris
position of the entrance pupil, to be precise). The same
aperture, and combined with electronic read-out of the
content of the 3D scene is projected in an identical
image sensor. A high level of sensitivity enables a shorter
perspective (we assume the lens distortions to be small).
exposure time, which means that it is useful for capturing
The crop factor
fast-moving subjects. On the other hand, high sensitivity
leads to increased image noise. ⊘im, ff 43.3 mm
c= = (30)
In contrast to traditional photography, an SPC does not ⊘im, SPC ⊘im, SPC
use an iris to vary the f-number, because this would lead to
between a full-format camera sensor and an SPC sensor is
loss of resolution (see previous Section 4.3).
between approx. c = 3.5 to 12, depending on the SPC sensor
A classical photography rule of thumb is the “sunny 16
size, which in turn specifically depends on the lens FOV
rule”: On a sunny day with an ISO100 film and f/16 stop the
(Section 6). We assume that the image sensor contains an
required exposure time is about 1/100 s.
equal number of pixels, e.g. 12 MP. Consequently, the pixel
All three contributions are scaled logarithmically to
pitch is larger by a factor of c for the full-format sensor.
base 2, i.e. ISO100, ISO200, ISO400, … or the f-stop
Furthermore, we assume the lens f-number K (or the
numbers according to the square root of 2 between the
NA of the image) to be equal as well. We can think of the
f-stops, i.e. K = 2, 2.8, 4, etc. so that 1/K2 doubles per step
same lens and image sensor, but just scaled down by a
and finally for the exposure time T = 1/100, 1/50, 1/25, etc.
factor of c. For the scaled system (denoted by a bar) all
Then it is easy to deduce from the “sunny 16 rule” that an
angular quantities remain the same, namely the lens
increase of, say, 3 f-stops in aperture to f/5.6 enables a
f-number (or NA) and field of view
23 = 8-times shorter exposure – that is, 1/800 s – or an ISO of
50 and 1/400 s. K=K (31)
Meanwhile, for an SPC the number of photons per pixel
is inversely proportional to the square of the crop factor, c2,
assuming both the full-frame camera and SPC have an
equal number of pixels. For a crop factor of 7, an SPC pixel
therefore receives about 50 times fewer photons. This
corresponds to 5–6 exposure values (EV), 6 EV corresponds
to 26 = 64, which is a significant disadvantage when
shooting in low light and/or with fast-moving objects. Note
that due to the photographer’s shaking hand, long expo-
sure times are also critical in still photographs. This
contribution has been significantly reduced in SPC through
optical and electronical image stabilization (Section 13).
Improvements in the image sensor technology – like
“binning,” “deep trench isolation,” “dual conversion
gain,” and “higher quantum efficiency” have also helped
to close the gap to some extent in terms of these funda- Figure 16: Cross-section of a smartphone dual-camera module
mental disadvantages compared to full-frame cameras [27]. versus a DSLR camera lens.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 161

FOV = FOV (32) artistic shallow depth of field, which most passionate pho-
tographers see as a significant disadvantage.
while all lengths (e.g. focal length, sensor, and lens The scaling of optical resolution is a little more
diameter) scale inversely with the crop factor: complex. For this consideration, lens aberrations should
f = f /c (33) be included, since for full-format lenses at large aper-
tures those tend to significantly limit optical resolution.
⊘im = ⊘im /c (34) Now it is straightforward to see in a spot diagram, which
describes the ray deviations on the image plane, and that
L = L/c. (35) aberrations scale down linearly due to length scaling.
That is according to 1/c on an absolute scale, when a lens
The geometrical scaling on optical system lengths, di-
is downscaled. But since pixel size also scales down by
ameters, surface area and volume/weight are straightfor-
the same factor 1/c, ray aberrations remain unchanged
ward (Tables 2 and 3). The weight and volume reduction in
on the scale of a pixel. However, an ideal lens is limited
particular are immense, e.g. for a crop factor of 7 by a factor
by diffraction, that is the Airy spot size, namely 2.44·λ·K.
of 343; reducing weights of full-format camera equipment
Since the f-number K does not change, the resolution
will be transformed from a few kilograms to just a few grams.
does not change either on an absolute scale, but the Airy
Another aforementioned benefit of miniaturization,
spot size does change on the scale of the smaller pixel
also due to length scaling, is the decreased minimum op-
when downscaled to miniature size. Consequently, as
tical distance (assuming the identical focusing mechanism
soon as lens performance becomes diffraction-limited
of the original and miniature lens). This enables the user to
while downscaling the size of a lens, the resolution will
focus on much smaller objects.
drop linearly according to 1/c.
The major disadvantages of miniaturization arise from
Lohmann [28] proposed how the spot size of geometric
the fact that a pixel sees much less light, in our assumed
contribution – describing the lens aberration, and an (ideal)
model by a factor of 1/c2 proportional to the pixel area,
diffraction contribution – can be approximately combined
resulting either in c2 larger exposure time or higher ISO
to describe the effect of lens scaling on resolution:
sensitivities, the latter resulting in increased noise.
Depth-of-field scales linearly with focal length (see A′p = λ2 K 2 + (1/c)2 ξ 2 (36)
equation on hyperfocal distance) meaning that the hyper-
focal distance is a factor of c further away from the miniature Apʹ denotes the spot area and ξ 2 the second moment of
lens. For practical photography this is an advantage, since the ray deviation distribution. In Figure 17 this scaling rule
the probability increases that the image quality will not is exemplified together with a wave-optical calculation of
deteriorate due to focusing errors. However, as mentioned the point spread function (PSF) (including both aberrations
previously, miniature lenses are not capable of creating an and diffraction).

Figure 17: Scaling down the same lens by a factor of 7 and the resulting PSF on the scale of a pixel (red box in graphs on right-hand side).
162 V. Blahnik and O. Schindelbeck: Smartphone imaging

Table : Scaling of physical parameters with crop factor. Advantages task which to some degree is realized in current smart-
of miniature cameras in blue; advantages of large-format cameras in phone multi-camera systems [31, 32]. Multicamera sys-
black; depth of field can be seen as both an advantage and a
tems are capable of performing stereographic 3D depth
disadvantage.
acquisition. We can thus conclude that an attempt has
Parameter Scaling factor been made to compensate for all disadvantages marked
in Table 2 by means of shooting multiple pictures (with
Length /c
the same or several different cameras) and computa-
Diameter /c
Sensor area /c tionally combining those images.
Volume /c
Weight /c
Minimum optical distance /c
Etendue/throughput /c
4.7 SPC lenses: How good are they actually?
Exposure time c
Low light noise c We want to follow up on the discussion of aberration per-
Object resolution (PSF size relative to pixel size) ( + (k·c)). formance of the previous Section 4.6 by asking: How good are
Out-of-focus spot diameter /c SPC lenses corrected actually? How good would they be if
Depth-of-field c
scaled up to the size of a full format (36 × 24mm2) lens?
Conversely, what happens if they would be scaled further
Multi-camera systems containing many lenses in down in size?
parallel are a straightforward concept for smartphone We take a SPC lens f/1.9 wide-angle (FOV 75°) design
imaging [29, 30]: They are thin in order to fit into the tight for a 1/3.3” image sensor. The image diagonal of 5.4 mm is
camera housing, while the effective image sensor area almost exactly 8× smaller compared to a full format image
increases. Combining the images of these cameras in sensor (diagonal 43.3 mm). In Figure 18 the polychromatic
order to improve image performance in different di- MTF is shown of the original SPC lens, denoted as “ff/8,
rections (e.g. noise reduction, HDR) is a computational original” (“ff” means “full format”), and scaled up in size

Figure 18: MTF of SPC lens “ff/8, original” scaled up (up to a factor of 8 to full format “ff”) and down (down to a factor 16 to “ff/128”)
consecutively by a factor of 2 in each step. The image field size is a factor of 2 different in each step (see scale on abscissa). Spatial frequencies
are scaled accordingly. The spectral relative weights of the polychromatic MTF are 1 (405 nm), 3 (436 nm), 12 (486 nm), 24 (546 nm), 24
(587 nm), and 11 (656 nm).
 V. Blahnik and O. Schindelbeck: Smartphone imaging 163

The performance level of the full frame upscaled

version of the SPC is excellent. The MTF performance is
comparable with excellent full format lenses like the
ZEISS Otus 1.4/28 mm (when compared at SPC aperture
f/1.9) [33].
Looking on the other side of lens scaling: When the
lens size is scaled down the overall performance severely
drops. The diffraction contribution in Lohmann’s scaling
law becomes dominant. This is obvious when the lens is
scaled down by a factor of 2 (“ff/16”). The Strehl ratio is a
common measure to evaluate lens performance in com-
parison to the diffraction limit [34]. Figure 20 shows the
Strehl ratio for the PSF in the center of the field: At
original size (“ff/8”) the SPC lens is rather diffraction-
Figure 19: Summary of Figure 18. MTF-performance of original SPC limited (S = 0.88), but not when scaled to full frame
(“ff/8”) and upscaled and downscaled lens design evaluated at Nyq/ format (S = 0.12). The Strehl ratio approaches its
2, Nyq/4, Nyq/8 for each size. While at original size diffraction limits maximum value 1 for scaled down versions but as
the performance only moderately (moderate performance improve- mentioned is not capable to support the corresponding
ment by upscaling), downscaling lead to severely limited perfor-
sensor resolution: This means the performance is prin-
mance due to diffraction limitation.
cipally limited for yet smaller lens sizes. Indeed for
actual lenses in current SPC multicameras the diffraction
by a factor of 2 (“ff/4”), 4 (“ff/2”) and 8 (“ff”)). In order not limitation is actually limiting the performance: Unlike in
to compare apples and pies the spatial frequency is corre- the current analysis with aperture f/1.9 and field diam-
spondingly scaled down in each consecutive step by a eter 5.4 mm both aperture and field are smaller than that,
factor of 2. A common choice (at ZEISS and other lens e.g. f/3.4, ⊘im = 4.2 mm for tele lenses, and therefore
suppliers) of spatial frequencies for MTF evaluations at predominantly diffraction-limited.
full format is 10, 20 and 40 lp/mm. Correspondingly the 8
times smaller SPC lens is evaluated at 8× larger spatial
frequencies 80, 160, 320 lp/mm. (If these spatial fre-
quencies correspond to Nyq/8, Nyq/4, Nyq/2 the pixel
pitch p = 1/(2 Nyq) of the SPC is p = 0.78 μm and for the full
format camera p = 6.25 μm, respectively).
In Figure 18 the MTF performance is shown and sum-
marized in Figure 19 and Table 3. As can be seen the MTF at
original size is only slightly smaller compared to the
upscaled version of the lens. According to the Lohmann’s
scaling law presented in Section 4.6 this means that the
overall performance at the original size (5.4 mm image
diagonal) with respect to imaging with 0.78 μm pixel pitch
is only moderately limited by diffraction compared to the Figure 20: Strehl ratio at the center of field for (the scaled versions
aberration level of the lens design. of) the SPC lens.

Table : Summary of MTF according to Figure  and corresponding pixel size (p = /(Nyq) and image field radius ymax).

Full format ff/ ff/ ff/ ff/ ff/ ff/

Scale rel. to SPC     / / /

MTF @ Nyq/ . . . . . . .
MTF @ Nyq/ . . . . . . 
MTF @ Nyq/ . . . . .  
Pixel size (μm) . . . . . . .
ymax (mm) . . . . . . .
164 V. Blahnik and O. Schindelbeck: Smartphone imaging

5 The multicamera system in

modern smartphones
For a long time, only one standard wide-angle lens was
used in mobile phone photography. Up to about 2006 it
typically had an equivalent focal length of 35 mm
(FOV ≈ 60°), and later of 28 mm (FOV ≈ 75°). Today the
standard wide-angle camera as a component of the
multi-camera system is often called the “main camera.”
Besides its importance in everyday photography, this is
mainly due to its relatively superior performance
compared to other camera lens focal lengths. This is
based on the feasibility of achieving extremely flat form
factors (see Section 6) at high apertures. Since 2016 and
the launch of the Apple iPhone 7+, the number of rear
cameras has been steadily increasing from dual, triple,
quad, and now penta. In high-end smartphones, the
standard wide-angle lens is supplemented by lenses with
Figure 21: Sony Xperia 1II multi-camera system. Cameras (top to a shorter focal length (corresponding to an FOV of
bottom): ultrawide angle 16 mm, F2.2, 12 MP, Tele 70 mm, F2.4, around 120°) and longer focal lengths such as 55 mm,
12 MP, 3D iToF for depth acquisition, standard wide-angle 24 mm, 70 mm or even 125 mm. In addition, there is a 3D depth
F1.7 12 MP (main camera). Courtesy of Sony. sensor, e.g. based on ToF measurement, which is
capable of generating real-time depth maps of a scene
Conversely, it can be said that it makes little sense to over a wide FOV (see Section 15.1). Figure 21 shows an
achieve pixel resolutions far below 0.7–0.8 μm, which is example of a multicamera system.
the current state of the art for image sensor CMOS tech- Today, almost all smartphones are equipped with two
nology. To do this, it would have to be possible to imple- rear cameras as standard, including mid- and low-end
ment optical designs with an even larger aperture such as smartphones. The number of cameras continues to grow,
about f/1. both on the rear and the front. The multicamera market

Smartphone Camera Module Sales

6000

5000
Sold Modules [Million pcs]

4000

3000

2000

1000

0
2015 2016 2017 2018 2019 2020 2021 2022
front 2nd 0.8 2 49 47 80 93 106 122
front 1387.8 1498.5 1525 1545.1 1543.5 1594.8 1641.4 1684.1
rear 4th 1 7 9 12 22
rear 3rd 26 197 369 465 528
rear 2nd 1.4 78.8 266.8 537 926 1112 1279 1394
Figure 22: Camera sales per module in
rear 1833.2 1910.93 1965.8 1932.6 1918.5 1948.1 1947.2 1997.6
million units (Source: TSR; 2021/22 are
rear rear 2nd rear 3rd rear 4th front front 2nd predictions).
 V. Blahnik and O. Schindelbeck: Smartphone imaging 165

Table : Sample data from a high-end SPC system consisting of four rear cameras and one front camera.

Front cam Extreme wide cam Wide cam Tele cam Folded tele cam

Diagonal field-of-view ° ° ° ° °

Equivalent focal length  mm  mm  mm  mm  mm
Sensor size (sensor diagonal) . mm (/.")  mm (/") . mm (/.") . mm (/.") . mm (/.")
Sensor pixel #  MP  MP  MP  MP  MP
Pixel pitch . μm . μm . μm . μm  μm
f-Number . . . . .
Focal length . mm . mm . mm . mm  mm
Minimum optical distance  cm; fixed focus  cm  cm  cm  cm
Image stabilisation / OIS OIS / OIS

trend is shown in Figure 22. In the past the front “selfie The structure of the optical system (Figure 24) is largely
camera” usually offered a low resolution, as a fixed focus determined by the required miniaturization. The SPC must fit
camera with a small sensor and small aperture to in- into a flat case about 8–9 mm thick. Subtracting the housing
crease the depth of field. In recent years, however, the and image sensor thickness, the overall length of the lens
standard wide-angle lens has evolved into a larger must therefore not be longer than about 5–6 mm. The diag-
camera with a high-resolution image sensor (often onal of the image sensor should be as large as possible in
multicell to improve HDR) and an autofocus. High-end order to reduce the disadvantages of small image sensors
smartphones usually have a dual front camera. Often, described above (image noise, dynamic range, etc.). At the
the standard wide-angle lens is provided in tandem with same time, the aperture of the lens must be relatively large,
an ultrawide camera for handheld panorama selfies. about f/2 or larger, so that the system resolution is not limited
Since about 2018 there has usually been a 3D sensing with image pixel sizes of around 1 μm.
camera for face recognition on the front, just next to the With an FOV of around 80°, SPC wide-angle lenses
visual cameras. achieve a form factor of construction length (from front
Table 4 shows sample technical data from a 4-rear- lens to image plane) to the image sensor diagonal of
camera and 1-front-camera system. L
Figure 23 shows the FOV of each camera, together with r= ≈ 0.65 … 0.85. (37)
⊘im
the minimum optical distance (MOD) and the depth of field
at MOD. The object field diameter (⊘ob = 0.15 m) is also With an overall length of L = 6 mm, this enables an image
shown for the front camera’s MOD. As you can see, the tele sensor diameter of up to ⊘im = 9 mm. If all means are
lenses’ captured object diameter at their corresponding exhausted, such as minimum board and housing thick-
MODs is larger (about ⊘ob = 0.5 m). Similarly, for the front ness, and the camera is protruding slightly from the
camera the fixed focus distance is ⊘ob = 0.6 m, such that a housing, some current high-end models exceed the im-
person’s face is well within this object region. age sensor diameter of the wide-angle main camera by
12 mm. Despite the highly miniaturized design, they are
only a crop factor of around 4 away from full-frame im-
6 Optical System Design age sensors.
With spherical lens shapes, there is no classic lens type
6.1 Optical design structure of a with such a large aperture of around f/2 that has such a
smartphone lens small overall length-to-image-diameter ratio. The Biogon is
a well-known classic spherical ultracompact high-aperture
The opto-mechanical design of and the manufacturing wide-angle lens. Steinich and Blahnik [35] compare the
technology for smartphone lenses are very different Biogon with a mobile phone lens with the same aperture,
from those of classic photo lenses. The use of highly and FOV, scaled to the same image diagonal: The image
aspherical plastic lenses, the miniaturization, and the performance of the SPC optics turns out to be even better
completely automated production in quantities of several than that of the Biogon, despite the Biogon being about
millions can hardly be compared with classic optics twice as long. Figure 25 shows this with another example:
manufacturing. For the 2/35 mm Biogon made of spherical lenses, r = 1.58,
166 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 23: Multi-camera system according to

data in Table 4: FOV, MOD, DoF @MOD of
the corresponding camera lens. The object
field diameter for the main rear and front
cameras has also been drawn (0.15 and
0.6 m, respectively.).

the early 1950s in the form of simple meniscus lenses or

doublets, and a correspondingly small aperture of f/15,
as in the Kodak Kodet [36]. In 1957 Eastman Kodak used
molded plastic lenses in the rangefinder of their Signet
cameras and since 1959 an f/8 triplet lens made of plas-
tic. Kodak established a standard in the consumer cam-
era market for the following decades. Until the 1970s, the
technological advantages of plastic had not yet been
exhausted: the lenses remained spherical, due in part to
the still-low computing power of mainframes for lens
design calculations. That changed in the 1970s. The
innovative Polaroid SX70 camera from 1972 even had a
freeform surface in the rangefinder [37]. Precisely man-
Figure 24: Structure of a wide-angle lens: The light coming from a
faraway object enters the lens at a cover glass (approx. 0.2 mm
ufactured molded aspheres have been used in many
thick), then passes through the plastic lens elements and an projectors and photo cameras since the 1970s, such as
infrared (IR) filter (thickness approx. 0.2–0.3 mm) before finally the triplet shown in Figure 26 or the Kodak Ektar 1.9/
arriving at the image sensor. The fixed lens stop is usually placed at 25 mm built in the Kodak Ektramax, also with a plastic
the lens entrance. asphere. These lenses can be regarded as early fore-
runners of modern SPC lenses, although the aspheres
while the 1.9/28 mm aspherical plastic lens achieves still had low asphericities, which was in part due to the
r = 0.83. The contrast of the SPC lens is even higher, the limited computing power available at the time for optical
peripheral light intensity drop is lower (both due to the lack design [38].
of vignetting with the SPC lens) and the distortion is From the doublets or triplets in the early days of
comparably very good (<1%), as are the chromatic aber- smartphone photography, the number of lens elements
rations (<1.5 μm color fringes). was continuously increased to further increase the nu-
The key to the flat form factor is to use extreme merical aperture and thus the optical resolution, as
plastic aspheres. The use of plastic lenses did not emerge dictated by pixel shrinking. In 2020, there were often 7 lens
for SPC until after 2000, but has already been in use since elements in the standard wide-angle lens, which were
 V. Blahnik and O. Schindelbeck: Smartphone imaging 167

Figure 25: Comparison of different wide-angle lenses: (a) 2/28 for SLR camera, (b) classic 2/35 mm for (mirrorless) rangefinder camera,
(c) modern 2/35 mm for mirrorless system camera, and (d) SPC lens; all systems are scaled to the same image size (to true scale the SPC lens d)
is by a factor of 4–10 smaller in length and diameter). The ratio of the overall length (vertex of the first lens to the image plane) to the sensor
diagonal is (a) r = 2.63, (b) 1.58, (c) 1.39, and (d) 0.83.

closely packed one after the other. Figure 29 shows a started being added. A study of SPC lens patent literature
chronological history from 2004 to 2020. Until 2016, SPCs [39] shows this same trend shifted forward in time: For
were practically exclusively equipped with standard wide- many years only FOVs of around 60° and 75° were
angle lenses. Since 2016 lenses with other focal lengths considered; but now the optical designs are very diverse in

Figure 26: Comparison of optical designs:

Early aspherical plastic consumer camera
lens from 1974 with one aspherical lens
element and one early (2004) and modern
(2020) mobile phone camera wide-angle
lens. The material type is abbreviated as
“g” for glass and “p” for plastics.
168 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 27: Patent literature on mobile phone camera optical design [39]: F-number and FOV of about 750 mobile phone camera lens patents
versus patent publication day. Courtesy of Luxin Nie.

an FOV range of about 20°–150°. At the same time the lenses is rarely dealt with in the literature, as in refs.
f-number decreased to support pixel shrinking (Figure 27). [41–44] or in the work done by the group of José Sasian [45].
Consequently, optical design has become more and more The optical designer David Shafer wrote the following
complex, with the number of lens elements continuously in his study: “A new theory of cell phone lenses” [46]: “My
increasing (Figure 28). Meanwhile there are optical design conclusion from this study is that the usual cell phone
patents which feature 9 lens elements [40]. designs with very wiggly aspherics are using extremely
In Figure 29 the evolution in the diversity of optical high-order aberrations to compensate for uncorrectable
designs is shown in terms of typical configurations of lower-order aberrations – due to a nonoptimum third and
multicamera system SPCs. fifth order distribution inside the designs.”
The optical design of SPC lenses amounts to a para- In contrast to classic camera lens designs consisting of
digm shift in the fundamental layout of camera lenses. spherical lens elements there are excellent configurations,
Despite its widespread use today, the optical design of SPC e.g. the Double-Gauss type and Triplet variants, (e.g. [47–49])

Figure 28: Number of lens elements versus patent publication day (left); wide-angle lens designs with 3–8 lens elements (right). Courtesy of
Luxin Nie.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 169

Figure 29: Evolution of smartphone optical design beginning with standard wide-angle lenses of increasing aperture and complexity. FOV was
extended towards the extreme wide-angle and tele ranges.

which are able (within a certain parameter range of f-number where the lens component’s refractive powers are φj and
and FOV) to completely remove third and fifth-order aberra- refractive indices nj. Consequently, the lens must consist
tions. Let us have a look at the optical design structure of the of both positive and negative components. The Biogon
Biogon and then compare it with the design of a corre- lens structure consists of negative outer lens elements
sponding SPC lens with the same FOV: The Biogon is based (meniscus shape) and a positive inner group (Figure 30b).
on a strictly concentric structure with respect to both the outer With this structure the chief ray is bent to a smaller angle
lens elements and the inner positive lens group (Figure 30a). inside the lens, which is beneficial because it leads to
The stop is placed in the center of this system, which results in smaller aberration contributions. Another advantage of
the rays entering the system at a certain angle of incidence the negative outer elements is that the off-axis entrance
and also exiting the system at this same angle. A strictly pupil becomes larger, improving the relative illumina-
concentric system delivers consistent image quality from a tion. Asymmetrical aberration types, namely distortion,
spherically curved object surface to a spherically curved im- coma, and lateral chromatic, are eliminated by the quasi-
symmetric arrangement around the stop in the center of
age surface. For a curved image sensor with simple mono-
the system, because the aberration contributions in the
centric lenses, excellent image performance is feasible [50].
front lens part occur with opposite signs at the corre-
For SPC lenses it has been shown that curved image sensors
sponding position at the rear and thus cancel each other
can result in lenses with one f-stop superior f-number and
out (Figure 30c). Longitudinal and higher-order chro-
comparable aberration performance [51]. In particular,
matic aberrations are corrected by an arrangement of
extreme wide-angle lenses would benefit significantly from
low-dispersion glasses for the outer negative lens ele-
curved image fields. For a plane image sensor, however, the
ments and achromats of the inner positive elements
Petzval condition must be fulfilled,
(Figure 30d). Spherical aberration and astigmatism are
φj the remaining aberrations to be corrected by fine-tuning
∑ =0 (38)
j nj all lens parameters through optimization, especially the
170 V. Blahnik and O. Schindelbeck: Smartphone imaging

lens radii. At the time of the invention [52] these calcu-

lations were extremely time-consuming since computers
were not available yet.
Now let us compare the optical designs of the Biogon
and SPC lens: both have a field of view of about 80°
(Figure 31) and are similar in that the ray angle at the
entrance of the lens is about the same as at the lens exit.
Conversely, for SPC half of the system structure from the
aperture to the image plane is sufficient: the aberrations, in
particular distortion and coma, are corrected by the
aspheres. With spherical elements in such an arrangement
with the diaphragm in the front area, the distortion would
be difficult to correct; with the strongly aspherical
design though, relaxation of the distortion specification
(typically <1%) and subsequent digital correction would
bring almost no advantage. Within the SPC lens the chief
ray path generally runs along a straight line. Despite a
much larger aperture (f/1.7 compared to f/4.5), the SPC lens
is considerably shorter.
With wide-angle lenses for SLR cameras, one is forced
to use a retrofocus type because of the space required for
the folding mirror between the last lens element and the
image sensor, i.e. negative refractive power is required in
the front and negative power in the rear of the lens [53].
This enables a certain symmetry and the lens construction
becomes considerably more complex (see Figure 25a). Figure 31: Comparison of a Biogon (4.5/21 mm) with an SPC lens
Modern photo lenses for mirrorless cameras are also (1.7/25 mm) scaled to same image size.

Figure 30: Structural features of the classic

Biogon lens type: (a) Monocentric layout, b)
(−, +, −) refractive power to correct for field
curvature, (c) symmetry to the central stop,
and (d) achromatization.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 171

increasingly asymmetrical (see Figure 25c). To correct the optical axis. The first term alone results in different
distortion, curvature of field and astigmatism, aspherical conic shapes depending on the value of the conical con-
lenses are often placed directly in front of the image stant: k = 0: sphere, −1 < k < 0: ellipsoid with main axis on
plane. These very compact systems are favored thanks to the optical axis (prolate spheroid), k = −1: paraboloid,
the availability of inexpensive aspheres through molding k < −1: Hyperboloid. In Figure 32 the description of the
processes and also by the considerably greater computing asphere of a typical SPC “w-shaped” lens element surface
power needed for optimization in optical design. Never- is shown.
theless, the classic symmetrical lens types, all of which In the case of SPC lenses, usually even polynomial
were created before computers were used in the 1950s, coefficients up to about the 10th order are used for the first 2
can still be found in many of today’s products. to 3 moderate aspherical lens elements and up to the
For the optical design of SPC optics, the classic design 14th or 16th order for the extremely strongly aspherically
rules according to Seidel’s third order theory [54] are no curved near-field lenses, i.e. a4, a6, to a14 or a16. Some
longer applicable. High-order aberrations are used here in optical designers also use odd orders: a3, a5, etc. (odd
order to reduce low-order aberrations. All lens surfaces are polynomials). In addition, there are surface descriptions
aspherical. The spherical basic shape corresponding to the with orthogonal function systems such as the Forbes
radius of curvature in the center of the surface is shown in polynomials [55], which have advantages in the conver-
blue in comparison. On the rear, the deviations from the gence of the optimization [56] and also in the desensitiza-
spherical shape are very large. On the last surfaces it is tion of the system [57] (Figure 33).
obvious that this asphere cannot be obtained from The low aberration orders are largely compensated for
manufacturing techniques like grinding and polishing with high-order aspheres, but residual high-order aberra-
starting from a spherical basic shape (Figure 33). The tions remain (as can be seen in the aberration graphs in
standard surface description of aspheres is: Figure 36). When optimizing SPC lenses, the pupil and the
field must therefore be sampled sufficiently, otherwise the
c r2
z= √̅̅̅̅̅̅̅̅̅̅̅̅ + a4 r4 + a6 r6 + · · · · (39) image performance between the optimized field points
1 + 1 − (1 + k)c2 r2
threatens to drop sharply.
Here, c denotes the curvature at the apex of the sur- In contrast to a lens for a large-format camera, SPC optics
face, k the conical constant and r the radial distance from must hardly have any vignetting by lens edges or other

Figure 32: Right surface of a “w-shaped” SPC lens element. The coefficient values are given on the bottom right. Middle top: The actual
asphere shape deviates strongly from the corresponding lens center spherical curvature. Top right: Conic constants are often used for SPC
asphere description with considerable contribution. Middle bottom: Typically the separate functions (monomials) take on large values at the
edge.
172 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 33: SPC optical design. (a) The local radius of curvature in the center of the lens is shown in blue for comparison. In particular, the last
lens elements in front of the image plane deviate significantly from the spherical shape. The refractive power of a lens in the center of the lens,
whether positive or negative, is also shown. However, the last two lens elements have a strong field-dependent effect with (locally) very
different refractive powers and deflections on the beam. (b) The local curvature variation causes a change in the refractive power along each
field-dependent light path. Positive (+), neutral (o), and negative (−) power is indicated in the figure for the field near lens elements 4, 5 and 6.

apertures. This would be tantamount to reducing the aperture angle as they enter the lens: With a standard wide-angle
and would lead to a loss of resolution towards the image lens, this is approx. ±35°–40° when entering the lens; the
corners. In the case of an SPC, and unlike in DSLR lenses, the typical chief ray angle in the image corner on the image
light is usually not blocked by any field stop or lens edge. plane is also around 35°. The mechanism described in the
With mobile phone wide-angle lenses, the geometric previous paragraph (same refractive power in the anterior
light path to the corner of the image is more than 20% pupil area, then more negative and then positive for the
longer than it is to the center of the image. The design field edge compared to the center of the image) ensures, in
structure ensures that the size of the numerical aperture addition to the relative aperture size, that the chief rays are
remains almost the same up to the edge of the image, which bent considerably more. The result is a nonlinear course of
is necessary to keep the diffraction-limited resolution the angle of incidence towards the field edge: Rising from
almost constant up to the corner of the image: The lens the center of the image, stagnating in the field zone and
elements in the middle of the lens between the pupil and edge. This means that the position of the exit pupil is not in
field considerably change their refractive power depending the same place for all field points, but rather for field points
on the image field height. This can be seen particularly well at the edge of the field much further in the front.
in Figure 33 on lens element 4 due to the more negative In Figure 34 lenses of different focal lengths are shown:
refractive power at the edge of the field, the light bundle “Super wide-angle” (FOV 120°), wide-angle (75°), normal
initially becomes more divergent. On the rear of the cell tele (38°), and periscope tele (27°). The chief ray in the
phone optics, especially with the “w-shaped” last lens standard wide-angle lens essentially runs along a line
element, the refractive power at the edge of the field is through the lens. For the telephoto and extreme wide-angle
positive, while it is clearly negative in the center of the lens. types, however, there is global bending of the chief ray
As a result of these differences in refractive power, the passing through the lens (Figure 35). This characteristic is a
aperture at the field edge becomes significantly larger than reason why standard wide-angle lenses are the most
it would be with a corresponding conventional spherical compact lens type among SPC lenses. Therefore, the image
optic. With these extreme asphericities of the last lens el- sensor used for a standard wide-angle lens is the largest
ements of the SPC optics, one can no longer speak of pos- within the multicamera system.
itive or negative lenses: the refractive power varies over the
field height. Zhuang et al. [58] propose a systematic design
approach based on different types of such significantly 6.2 Optical design imaging performance
curved field lenses.
A characteristic of the standard wide-angle lenses is Optical image performance evaluations of camera lens
that the chief rays strike the image plane at a similarly high design commonly include:
 V. Blahnik and O. Schindelbeck: Smartphone imaging 173

Figure 34: State-of-the-art optical design of today's typical SPC lenses with different focal lengths (Courtesy of ref. [59]: (a) Extreme wide-
angle [60], (b) standard wide-angle of 28 mm, (c) Tele 65 mm [61], and (d) periscope Tele 90 mm [62].

Figure 35: Comparison of chief ray path

through extreme wide, wide, and tele
lenses.

– MTF (vs. field or FOV and vs. spatial frequencies and shows MTF versus spatial frequencies up to the cut-off
vs. distance with respect to image plane) frequency 2 NA´/λ = 1/(λ·f-number), the third MTF through
– Distortion (radial distortion, TV distortion, and image focus for Nyq/4.
distortion grid plot) From these graphs it follows that the image performance
– Relative illumination of all lenses is diffraction-limited near the image center and
– Aberration (e.g. spot diagrams, ray aberration curves, drops off for the wide-angle lenses (but not dramatically as
chromatic focus, and lateral shift) can be seen in the MTF versus field graphs for the relevant
– Angle of incidence at image sensor spatial frequencies). The through-focus region with very high
contrast is roughly only about ±10 μm for all lenses.
In Figure 36 the image performance of the 4 designs are The next graph is a distortion graph on the 4:3 aspect
shown: “Extreme wide,” “wide,” “tele,” and “periscope ratio image field showing that distortion of all lenses is not
tele.” The corresponding module sizes fit into a 6 mm noticeable (typically <1%). The barrel-type distortion of the
available length as constrained by smartphone thickness extreme wide-angle lens is about 20%. In many camera
(the standard wide-angle lens is extended slightly further modules this distortion is not corrected by software, although
outside the housing). The spatial frequencies for the first there are apps available that can correct this distortion. Note
three graphs on MTF are chosen with reference to a 12 MP that it depends on the scene in question whether there is a
image sensor, that is 4000 × 3000 pixels. Since the image need to correct this for the barrel distortion: Just photo-
sensor sizes are different (5.8; 10; 4.4; 4.4)mm the “effective graphing a 2D plane like a chess board fish-eye distortion is of
pixel pitches” are also different (1.16, 2.0, 0.88, and 0.88) course undesired; but when photographing a group of people
μm and therefore so too is the corresponding Nyq. As barrel distortion compensates for the “egghead effect,” also
explained earlier “effective pixel pitches” are not actual known as “perspective distortion” [63].
pixel pitches on the image sensor but refer to the “macro- Relative illumination drops more towards the corner of
pixel” size of multicell sensors. The first graph shows the image field for wide-angle lenses. The intensity drop-
polychromatic MTF versus image field for spatial fre- off, sometimes called “shading,” is corrected by software.
quencies Nyq/2, Nyq/4, and Nyq/8. The second graph Although shading is not apparent in pictures taken using
174 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 36: Optical design performance for lenses given in Figure 34: extreme wide-angle f/2.1, FOV 120°, wide-angle f/1.7, FOV 75°, tele f/2.8,
FOV 38°, and periscope tele f/2.8, FOV 27°.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 175

smartphones it gives rise to increased noise sensitivity by 1 high-contrast edge” under photographic worst-case situa-
or 2 EV for wide-angle photography. tions [64].
The angle-of-incidence graph shows the chief ray and
marginal ray angles in the image plane. In order to avoid
further light loss the chief ray incidence angle is usually 6.3 Extreme wide-angle lenses
limited to about <35°: In addition to the lenses’ relative
illumination, which includes a natural geometrical loss ac- Extreme wide-angle lenses have been used in many high-
cording to approximately cos4(AOI), namely cos4(35°)≈0.45 end SPCs with FOVs of around 120°–130° since around
(which is already included in the relative illumination 2018. A distortion of around 20% is permitted. In this way, a
graph), the oblique incidence on the image sensor results in short flatness factor of r = L/⊘im < 1 is achieved. Due to the
further intensity losses (included in the software-corrected additional negative “bending lens elements” at the front, r
shading). In addition to the improvements obtained by is larger compared to a standard smartphone wide-angle
back-side-illuminated image sensors and specific architec- lens. At the expense of a slightly larger r factor of about 1, it
ture, e.g. deep-trench structures (see Section 9), those losses is possible to remove the distortion completely (examples
can be further minimized by slightly shifting the micro for FOV 116° in ref. [65]). In the front part of the lens there
lenses according to the incidence angle (that is a slightly are usually one or two lens elements in front of the dia-
smaller micro lens array compared to the pixel array). phragm, which are strongly aspherical with a lens curva-
The ray aberration graphs in the final line of Figure 36 ture that rises rapidly towards the edge of the lens and
show the specific aberrations on the tangential and sagittal which bend the steep angles at 60° such that the angles at
image plane for different wavelengths versus field (image the diaphragm are reduced to less than 40°. Starting here,
center on bottom, image corner on top). The scale is only the beam path is comparable to that of a compact standard
±2.5 μm. The aberration curves are very “wiggly” as residual wide-angle lens. Accordingly, the structure of the system
aberrations of the compensation principle of lower-order and the chief ray angles of approx. 35° on the image plane
aberrations by higher-order aberrations through usage of are similar.
high-order aspherical lens surface deformations. Chromatic Optical designs of extreme fish-eye wide-angle lenses
aberrations can be physically evaluated e.g. by image simu- with an even larger FOV of up to around 150° [66] and also
lations of edge or line spread functions versus field and over 160° are possible with about 50% distortion and a
through focus. From that the “color fringe widths” can be similar form factor [65]. Such camera lenses may well enter
evaluated defining the “number of colored pixels along a the smartphone market soon.

Figure 37: Cross-section of dual SPC with standard wide-angle lens (1.8/28 mm) and “tele lens” (2.4/56 mm). Due to the smaller aperture and
smaller sensor size the optical resolution of the tele lens is weaker, i.e. the ratio of Airy spot diameter to pixel size is larger (bottom graph).
176 V. Blahnik and O. Schindelbeck: Smartphone imaging

6.4 Tele lenses With this layout, the optical performance is severely
limited as the focal length increases (Figure 38).
Fulfilling the overall length requirement is even more The ratio of the overall length to the image diameter
difficult for longer focal lengths, such as those on the r = L/Øim increases and the size of the image sensor inev-
market for dual systems with a (hybrid) “optical zoom.” itably decreases due to the limitation of the overall length.
Therefore, a compromise is usually made with these lenses According to the lens data in the example in Figure 34 with
by using smaller image sensors and smaller apertures – a 65 mm equivalent focal length r = 1.32, which means that
both at the expense of optical resolution (Figure 37). The the image sensor size is 0.85/1.32 ≈ 0.64 smaller than the
difficulties of compact telephoto lenses are due to the standard wide-angle lens with the same overall length.
following: The longer the focal length, the smaller the That means that if we assume the image sensor size of the
required telefactor (TF): 28 mm wide-angle lens is 6 mm, then that of the 65 mm lens
is only 4 mm. In addition, the telephoto only has an aper-
L
TF = (40) ture of f/2.5 compared to f/1.7 of the wide angle: This means
f
that the diffraction-limited resolution is weaker by a factor
TF < 1 can only be achieved if the refractive power is of 1.7/2.5 = 0.68. Overall, this leads to a smaller number of
positive in the front part of the lens and negative in the rear “optical pixels” (meaning Airy spots per sensor area) by a
part. The smaller the TF, the more positive or negative factor of (0.64 × 0.68)2 ≈ 0.19.
refractive power is required, and the larger aberrations are
introduced. Greater refractive powers lead to greater lens
curvatures and these in turn lead to greater aberrations. 6.5 Periscope tele lenses and alternative
tele concepts

The problem of the short telephoto lens can be avoided by

rotating the optics by 90° in the housing. This can be
achieved with a 45° mirror or with a 45° prism mirror (see
Figures 34, 39 and 40): Then the optics can be much longer.
However, this “periscope layout” has another limita-
tion: The mirror must of course be smaller than the depth of
the housing and thus also the entrance pupil of the lens:
This means that the longer the focal length of the lens, the
f-number K = f/ØEP increases and thus the diffraction-
limited resolution becomes weaker. An approximately
realistic entrance pupil diameter of ØEP = 4 mm means that
even with a relatively small aperture of f/3.4 a focal length
of only f = K·ØEP = 3.4·4 mm = 13.2 mm can be achieved. For
a still realistic image diameter of 4.4 mm, this corresponds
to an equivalent focal length of 130 mm. These data are
Figure 38: For lenses with an equivalent focal length of 56 mm, f > L,
a telephoto design is required with a refractive power distribution of
close to what is feasible with this concept and can be found
(+, −). with similar data in some current multi-lens cameras and

Figure 39: Standard tele lens (left) and periscope tele lens (right).
 V. Blahnik and O. Schindelbeck: Smartphone imaging 177

Figure 40: Periscope Tele lens [67]. f/3, f = 14.55 mm, ⊘im = 5 mm,
corresponding to a FOV = 19.8°.

patent literature [67] (Figure 40), [68–70]. Alternative, but

Figure 41: Rotationally symmetric catadioptric design [73], f/2.4
catadioptric periscope layouts enabling the desired form
with central obscuration of 1/3 of the pupil. For an overall length
factor were proposed by Araki et al. [71]. An intermediate L = 6 mm, f = 14.4 mm (that is tele factor TF = L/f = 0.42),
image is used to get sufficient space for folding and ⊘im = 3.6 mm, corresponding to a FOV = 13.6°.
enabling a large aperture.
Carles and Harvey [72] take advantage of the fact that
much more space is available perpendicular to the narrow 7 Zoom
housing depth, so that the entrance pupil can be made
rectangular with a larger f-number (and therefore deliver Due to space limitations, it is very difficult to implement
better resolution) in the long direction. The contrast can be good zoom systems in smartphones. There were various
azimuthally homogenized by using several systems or approaches to implementing zooms in cell phones early
rotating the system in combination with a fusion of the on. Figure 42 shows an overview based on some products.
images of these subsystems. Since there were very good optical zoom systems for
Another alternative are catadioptric layouts with two compact digital system cameras (DSC) as early as 2000,
mirrors in the front part [73, 74] or with several reflections attempts have been made again and again to integrate
between the mirrors [75, 76]). This layout allows for very these in various forms in the mobile phone, e.g. in a
small telefactors of TF < 0.5 (Figure 41) or even less for compact, foldable layout as in the Nokia N93 or as an
layouts with multiple reflections at the front mirrors. In extendable zoom as in DSCs (e.g. Samsung Galaxy s4). Of
principle, this allows a very large entrance pupil diameter course, digital zoom was also available very early on in cell
to be achieved and therefore a high aperture ratio. Aperture phones, albeit in a very modest resolution due to the image
ratios as large as f/1 are possible, but only with a corre- sensors, which were already limited in resolution: The
spondingly large central obscuration, i.e. the objective Nokia Pureview 808 and subsequently the Nokia 1020 were
pupil is a narrow ring. This leads to a distinct loss of pioneering and ahead of their time. They featured a huge
contrast in the lower spatial frequencies. In principle, very 41 MP “Supersensor” (standard for high-quality smart-
fine structures would be displayed with a higher contrast, phones at that time were 5–8 MP) for their time, and
which is irrelevant for smartphone dimensions because therefore also very high image quality even with a 3× zoom.
this high resolution cannot be used due to the available And finally, since 2016, hybrid zooms through multi-
pixel sizes. Catadioptric lenses are known in photography, camera systems using lenses of different focal lengths have
especially as telephoto lenses with very long focal lengths. become a standard. Every year, smartphones are equipped
They are extremely short compared to standard telephoto with more and more individual cameras, thus increasing
lenses. In addition to the loss of contrast, many photog- the zoom range. This trend is likely to continue, but that
raphers do not appreciate the “donut bokeh” caused by the does not mean that alternative concepts are dead: After the
obscuration as much, i.e. the noticeable ring-shaped out- “super digital zoom” from Nokia from 2012/13 had no
of-focus highlights. This would play less of a role with SPC successor for a few years, wide-angle modules with very
lenses, but out-of-focus lines tend to take the form of bulky image sensors have been around since 2018. They
double lines. measure around 12 mm in diameter and are fitted as
178 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 42: Evolution and dead ends: Different zoom concepts in Smartphones.

⊘2im,
standard in high-end smartphones. With a multicell sensor NPoptics ≈ 0.32 (44)
λ2 K 2
architecture, these image sensors usually contain more
than 50 MP, sometimes even more than 100 MP. Figure 43 shows this “optical resolution in Mega-
Pixels” NPoptics for the aforementioned patent lens
examples.
7.1 Hybrid zoom in multicamera systems
In the following simplified consideration, we include
the digital zoom and the image sensor resolution in order to
The combination of different fixed focal lengths listed in
roughly estimate the total resolution of a hybrid multi-
the previous sections is referred to as the “zoom” in modern
camera zoom system over the entire focal length range. The
smartphone multicamera systems [77].
hybrid multicamera system consists of several cameras
As explained in the previous sections, the achievable
with lenses of different focal lengths: f1, f2, etc. Digital zoom
optical resolution of SPC lenses is heavily dependent on the
reduces the resolution according to the cropped FOV. Since
specific optical design, which in turns depends on an
this FOV directly scales with the focal length (f), the value
equivalent focal length. To simplify matters, we assume a
of NPoptics, the resolution drops according to:
diffraction-limited optical resolution of resoptics = 0.5 ⋅ ⊘Airy
(see Section 4.3), that is: f02
NPoptics digital zoomed = NPoptics, 0 (45)
f2
resoptics = 1.22 ⋅ λ ⋅ K (41)
where f0 denotes the focal length of the lens corresponding
and counts all these resolved areas on the image sensor
to NPoptics = NPoptics,0.
surface (4:3 aspect ratio with full diagonal ⊘im)
Figure 44 shows NP versus the equivalent focal length
π
Aim = 0.48 ⋅ ⋅ ⊘2im (42) for the camera module consisting of four cameras with
4
feq = 15, 28, 56, and 125 mm, as defined in Table 4: The blue
Then the number of “optical pixels” (i.e. resolved areas) curve shows the contribution of optics to the resolution (as
NP is just discussed), together with digital zoom from a shorter
focal length to the adjacent longer one. The red curve
Aim
NPoptics = π (43) shows the number of “macro pixels” NPsensor on the image
4
res2optics
sensor that is e.g. a 4-pixel of a Quad-Bayer sensor taken as
so one macropixel. The number of macropixels NPsensor is
 V. Blahnik and O. Schindelbeck: Smartphone imaging 179

Figure 43: NPoptics, the number of optical resolution areas on the image sensor, for different lenses from the patent database with FOV 20–
144° corresponding to equivalent focal lengths 8–125 mm. Beyond feq = 125 mm we can assume the same entrance pupil and image sensor
size, but the corresponding feq, due to the fundamental periscope space constraint.

given by the total number of pixels on the image sensor system is worse than the corresponding resolutions of the
NPsensor,total divided by the number of pixels within the tele cameras, since the digitally zoomed-in pixel resolution
cluster of pixels building a “macropixel” NPmacro-pixel: is clearly worse for feq > 56 mm. So overall the tele lenses
NPsensor, total improve the image resolution for the considered setup. The
NPsensor = (46) white line in Figure 44 shows the minimum “optical pixel
NPmacro−pixel
number” and “sensor pixel number,” that is the effective
The number of pixels per macropixel is namely pixel number in this simplified consideration:
NPmacro-pixel = 1 for a standard Bayer sensor and
NPeffective = min(NPoptics digital zoomed , NPsensor ) (47)
NPmacro-pixel = 4 or 9 for 2 × 2- and 3 × 3-multi-cell sensor,
respectively. A well-balanced system should have about According to the graph the effective resolution in the
the same resolution of optics and image sensor. This is zoom range 15–125 mm varies between about 2 and 12 MP.
roughly the case for extreme wide-angle and standard tele A more detailed analysis requires a simulation of
camera lenses. However, for the wide main camera the all the steps of the digital imaging chain, which also
optical resolution is clearly better than the image sensor includes demosaicing, image enhancement, the lens
resolution, so the sensor limits the overall resolution. The aberrations and the specific algorithms how images are
opposite is true for the periscopic long tele camera lens: fused by multiple cameras [78]. To a certain degree image
Here the optical performance of the lens clearly limits the performance can be improved through image fusion in the
overall resolution. Note that the optical performance of the common FOV of both cameras. However, this requires
wide-angle lens is also better than the optical performance some cumbersome joined camera module calibration
of tele and periscope tele lenses even at the same effective and is increasingly difficult at close range, due to the
focal length, that is when zoomed in digitally. However, the parallax of the images caused by the spacing of the camera
actual resolution of the complete wide-angle camera modules.
180 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 44: Optical (blue curves) and sensor (red curves) resolution of the multi-camera system according to Table 4 while zooming in with
digital zoom until longer focal length camera takes over (without image fusion).

7.2 Optical zoom systems smartphone, the realized image performance of an optical
zoom system can decrease significantly at long focal
Optical zooms have never truly caught on in the smart- lengths, as shown in the following example:
phone market. Since major manufacturers such as Nokia In 2006 Nokia joined forces with ZEISS to launch a true
(N93) and Samsung (G810, Galaxy K Zoom, Galaxy S4 optical zoom in the N93 model (Figure 42a). The mobile
Zoom) have made some attempts on the market in the past phone was designed to be “camcorder-like,” with a rotat-
and manufacturers continue to make announcements on able display and lens arrangement alongside the housing.
optical zooms, we will deal with this topic in this section. Figure 45 shows the type of system used: The almost line-
A classic optical zoom changes the imaged object arly moved “variator” changes the overall focal length and
frame, i.e. the FOV, by changing the focal length (f) of the the front group of optics, the “compensator, compensates
system. In most zoom systems, the change in focal length is for the focus deviation with a small nonlinear movement.
done by changing the distance between the lens elements The required movements of the optical groups of around
or groups of lens elements. The total focal length changes 10 mm driven by voice coil motors (VCMs) took a couple of
because it depends not only on the individual focal lengths seconds. With an overall length of 20 mm and a 6 mm
of its subgroups, but also on their distances. For example, image sensor diagonal, the aperture drops from f/3.3 in the
in the case of two optical groups with focal lengths (f1, f2) wide angle to f/5.9 with a long focal length and thus also
the total focal length (f) is given by: the image sharpness due to the diffraction limitation.
For similarly tight space constraints, Kreitzer and
1 1 1 d
= + − (48) Moskovich [79] describe a 28–85 mm zoom lens design with
f f1 f1 f1 f2
a continuously high f/2 aperture in a periscope design. This
In contrast to “digital zoom,” “optical zoom” is often aperture for the entire zoom range is made possible with an
associated with “lossless image performance” over the intermediate image, which becomes smaller and smaller
entire focal length range. However, this is not necessarily with increasing system focal length, but at the price of a
the case. Especially for the tight space constraints of a length of almost 60 mm and 17 lens elements.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 181

Figure 45: 3× standard zoom 3.3–5.9/32–90 mm. Figure 46: Zoom lens design of a typical compact camera, an
equivalent 2.8–4.7/24–200 mm zoom. The structure has six
movable groups, some of which have both zoom and focus
Excellent image performance can be achieved with the functions, and is highly complex [82].
relatively loose space constraints of a compact digital camera
(DSC). In the 2000s, due to the strong DSC consumer market, was far inferior with regard to its digital zoom. While in the
the R&D activities on optical zoom lens design were quite 2010s image quality with SPC steadily improved – and in
significant. DSC zoom systems are highly complex and make many everyday situations hardly compromised vis-a-vis
use of digital-optical co-optimization. There is a large base of DSCs – the DSC market declined. In response to this, several
patent literature on compact camera zooms (e.g. [80, 81]). DSC camera makers substantially increased the zoom range
The ratio between the overall length and the image of DSCs to more than 20× and even up to 50× – which is
sensor size in the example in Figure 46 is approx. r = 4.5– practically unachievable for SPCs – in an attempt to defend
5.5, i.e. much larger than with the fixed focal lengths, but their market position. These attempts were unsuccessful.
can be mechanically reduced to a ratio of r = 2–3. The opto- The large R&D departments of smartphone manufacturers
mechanical design of the camera lens including the zoom put a lot of effort into further improving their cameras,
curves is complex and time-consuming. A significant pro- which for most everyday situations became at least equal to
portion of lens elements, sometimes about half of all lens the image quality of DSCs.
elements, are aspherical. In addition, digital aberration Although DSC sales today are smaller than SPC sales
corrections are made: In the wide-angle range, a distortion by about a factor of 200, several smartphone makers have
of approx. 20–30% in the design is permitted and digitally implemented DSC-like optical zooms in their smartphones.
corrected, which significantly reduces the size; the image is However, attempts to sell a “DSC with phone functionality”
cropped somewhat depending on the zoom. Finally, as is proved unsuccessful due to the bulky shape of the
usual with many zooms with a large zoom factor, the smartphone.
aperture in the long focal length area is reduced to limit the To conclude this chapter: in principle, opto-mechanical
diameter in the front area, and the overall length. zooms are feasible and brought to market by large manu-
Until around 2010, the compact digital camera market facturers such as Samsung and Nokia, but without a sus-
was steadily growing (Figure 2). The image quality of digital tainable trend. The hybrid multicamera zoom is widely
system cameras (DSCs) was clearly superior. In particular, accepted today (2021) and this is expected to remain so in the
the zoom functionality of compact camera zooms of DSCs years to come, and to be supplemented by further camera
was completely missing in mobile phones, respectively, and modules, e.g. fish-eye wide-angle lenses.
182 V. Blahnik and O. Schindelbeck: Smartphone imaging

8 Opto-mechanical layout and

manufacturing
8.1 Plastic lenses: Key for miniature opto-
mechanical layout

As already shown in comparison to classic design forms,

plastic has been used to create the distinctly aspherical lens
shape and it is the key to the small depth of SPC optics (for a
detailed analysis, see ref. [83]). In addition, the high
complexity of the structural shapes that can be manufac-
tured with plastic makes it possible to implement not only
the complex lens shape but also the mechanical mount in
the same component (Figure 47). The reproduction accuracy
of the components is in the sub-micrometer range. Special
noncontact interferometric measurement technology is
required to measure the small, complex components, often
at steep angles of incidence [84]. Besides noncontact
metrology, contact-mode measurement devices are also in
use, e.g. the Panasonic UA3-P or Werth VideoCheck UA. Due
to the very good reproducible component shape accuracy, Figure 48: SPC lens mount concept [86].
the mechanical mount concept is also very simple: The
plastic lenses can be stacked directly on top of each other in individual lens elements are often matched to one another
a barrel. from the injection molding cavities.
The combination of “optics” and “lens-bearing
mount” in a single component is a key to mastering the
8.2 Opto-mechanical layout extreme manufacturing tolerances in mass production.
Figure 48 shows an example of a mount concept. The lens
The lenses are pressed into the plastic barrel either directly elements are positioned directly on top of each other on the
one behind the other or with spacers, and many are almost flat plastic mounting rings, often in a Sandwich with ring
fully automated [85]. No adjustments have been made stops to prevent straylight.
here, i.e. no measurements are carried out during the as- The accuracy requirements of SPC lenses are extremely
sembly process. MTF measurements are only carried out high. First of all, for very small lenses, the sensitivities also
once assembly is complete. To improve quality, the scale directly with the size difference of the system: For the
same lens that is 7 times smaller for a camera with the same
number of pixels, all position and geometry tolerances,
measured in units of length, are also a factor of 7 smaller. In
addition, due to their extreme shape, especially those in
the rear part of the SPC lens, the aspheres are even more
sensitive to decentering or tilting of lens elements. Critical
tolerances are typically decentered and surface accuracy
with tolerances for high-end modules of about +/−1 μm,
sometimes even less. Detailed analysis of sensitivities,
tolerancing, and yield can be found in the references given
in Section 8.5.
Figure 47: Lens elements of a standard wide-angle lens measured SPC modules are mostly used in several different
against a millimeter scale: First line: 7 aspherical plastics lens ele-
smartphone models, sometimes over several years, so that
ments, the largest “w-shaped” lens element close to the sensor
(pictured left), the first lens element close to the pupil (pictured quantities are in the range of millions, often tens of mil-
right); below: Very thin straylight discs; last line 2 mounting ele- lions. This mass production enables low manufacturing
ments positioned in between the last lens elements. costs of the aspherical lenses. The lens elements are
 V. Blahnik and O. Schindelbeck: Smartphone imaging 183

usually all made of plastic. They are manufactured using sensitivity, which is one to two orders of magnitude higher,
an injection molding process at temperatures of around both in terms of the refractive index dn/dT and the
90–170 °C. Depending on the type of plastic, it is either expansion coefficient. However, because of the small size,
injected or pressed into precise molds while still in liquid these high sensitivities are much less significant than with
form. The melting temperatures are lower than in the cor- a full-frame lens: the thermally induced wavefront defor-
responding aspheric pressing process for glasses. Plastics mation scales directly with the size, i.e. it is about a factor
are not as rigid and stable as glass [87]. A lens is manu- of 7 smaller than with a full-frame lens. The dominating
factured after about a minute. The cost of making the aberration type caused by temperature change is usually a
precise molds is in the tens of thousands of dollars. If the focus shift which is compensated for by the camera mod-
production runs with high yield, then the tool costs quickly ules with autofocus. To a lesser extent, other aberrations
pay for themselves. such as field curvature can also arise with temperature
The disadvantages of plastics are the relatively low changes. Systematic passive thermal compensation stra-
refractive indices with a large dispersion (Figure 49), which tegies for plastic lenses have been around for some time
complicates the optical design somewhat [88]. Precise now [91]. In practice, like when skiing in low-temperature
information can be obtained from plastic optics suppliers, environments, the situation is often somewhat relaxed, as
e.g. OGN, Zeonex (http://www.ogc.co.jp/e/products/fluoren/ the smartphone is usually worn close to the body, and due
okp.html, http://www.zeonex.com/Optics.aspx. html). The to the waste heat from the electronic components in the
disadvantageous dispersion and refractive index properties closed housing.
could be mitigated in the future with the use of nano-
composites [89, 90].
There are some technological limitations, for example
there are no cemented lenses like there are photo lenses
8.3 Active optical assembly
made of glass. Another general disadvantage is the thermal
With active optical alignment (AOA, sometimes abbrevi-
ated to AA), the lenses are aligned with the image sensor
and glued in with UV adhesive [92, 93]. The AOA runs fully
automatically and takes just a few seconds. Today the cycle
takes around 2–5 s per module, having been continuously
reduced over the years. During the assembly process on the
sensor, the barrel is aligned in the degrees of freedom:
centering x/y, tilt x/y and focus distance (Figure 51) until
the specified spatial frequency response (SFR) values are
reached simultaneously over the entire image field
(Figure 50). The assembly accuracies of the robotic ma-
chines are in the order of ±1 μm for x/y/z position and about
±0.005° for the angular position ϑx, ϑy, ϑz. Modules that
have not reached the specification within a certain period
Figure 49: Abbe diagram (refractive index n at λ = 587.6 nm versus
of time are marked as “scrap” and rejected. The quality, i.e.
Abbe-number νd = (nd − 1)/(nF − nC)) for optical glass (white circles)
and optical plastic (red circles). the measured MTF values, is continuously monitored over

Figure 50: Typical test chart for MTF

evaluation during active optical alignment:
slanted edge charts [94]. The contrast
transfer function of the individual module
and the function of the autofocus or the
setting of the fixed focus are also checked
separately using these charts.
184 V. Blahnik and O. Schindelbeck: Smartphone imaging

long periods of time and the process is interrupted if the the complete system optics/sensor, while the term MTF is
yield threatens to fall short of the target. commonly used for the optics qualification only. The yield
Leading suppliers of these robotic machines include for the optics module output inspection is typically around
ASM, HVS, IsMedia, and Pioneer, who usually supply 50%, while for active optical alignment of optics to image
different types of machines for the complete smartphone sensor, a yield greater than 95% or more is aimed for
packaging process (electronics assembly), among other because of the significantly higher component costs.
equipment. There are also dedicated machines for the as- In the case of SPC lenses, reject analyses, or to put it in
sembly of dual- or multi-camera modules which adjust, in 6 the common positive term, yield analyses, are an essential
axes, the optical axis between the camera modules and part of the development work [96, 97]. These are done in
then glue the modules into a single housing. the final stages of optical design with Monte Carlo analyses:
The optical designer uses his/her sensitivity analyses to
set tolerances for the lens elements (radius, thickness,
8.4 Tolerancing and yield analysis aspherical deviations or deviations in the refractive index,
and dispersion of the plastic) and their relative positioning
The manufacturability and desensitization of the errors in assembly (decentering, tilting, spacing deviations,
manufacturing and assembly tolerances of a lens requires etc.) and probability distributions of those individual errors
detailed knowledge of the manufacturing technologies on (e.g. Gaussian or “top hat” distributions matching typical
the part of the optical designer. “Yield optimization” is a statistical distributions of the manufacturing data). Then
central topic of product development in close coordination he/she “rolls” many different systems, where each compo-
with technologists in production, often at different pro- nent’s tolerance assumes a random value according to the
duction locations, e.g. in China or Taiwan at the same time. assumed probability distribution. The result is an ensemble
If you set the permitted deviations from the theoretically of many different realizations of the system with slightly
achievable image performance too low, you run the risk of different radii, aspherical deviations, refractive indices, etc.
disappointing customers with poor-quality products. If you For all these systems MTF data gives a statistical distribution
set the limits too tight, you risk large rejects or long delivery which can be evaluated for production yields with respect to
times. a system MTF specification (Figure 52). With respect to the
Finally, during the production process the quality of final active optical alignment of the mounted lens to the
the optics is qualified on the basis of MTF values [95]. Tri- image sensor the compensation of aberrations is taken into
optics ref. [212] is a supplier of MTF measurement equip- account [98] (Figure 53). In the optical design process, these
ment. This happens during the final inspection done by the analyzes are preceded by various methods of optimizing as-
optics module supplier for the system integrator, and built performance by desensitizing lens aberration [99–101].
partly during the receiving inspection performed by the
system integrator. The optics are not yet connected to the
image sensor. The system integrator then actively adjusts 8.5 Wafer-level manufacturing
the optics to the image sensor (previous Section 8.3). The
final image performance of the SPC is qualified with SFR SPC wafer-level manufacturing has been seen as a very
measurements. SFR is the common notation of the MTF of promising endeavor for low-cost mass production, especially
when it comes to thin devices [102, 103]. Although it has been
believed for some time that wafer-level manufacturing will
replace assembly-based manufacturing [104] the break-
through is still pending. Nevertheless, in recent years with
the rise of new modules like 3D acquisition systems, some
parts have now been produced through wafer-level mass
manufacturing. It will be interesting to see whether the
technology will take off in the next few years.
Related to lithographic production are micro-
electromechanical systems (MEMS)-based sensors (e.g.
gyroscope, accelerometer) as well as photonic chips
(photonic integrated circuits (PICs)), which might play a
role in consumer smartphones in the future. Multiple
Figure 51: Active optical alignment of lens module to image sensor. photonic functionalities in the form of building blocks
 V. Blahnik and O. Schindelbeck: Smartphone imaging 185

Figure 52: MTF vs. yield at different field positions and azimuths.

Figure 53: Yield-analysis incl. lens tolerances only (left graph); compensation by active alignment improves the yield from 87.3 to 98% for the
final camera lens qualification.

are integrated into a (usually semiconductor) substrate. vehicles. Optical functionalities (e.g. switchable light
Semiconductor manufacturing technology like lithog- sources or sensors) can be co-integrated and combined
raphy, epitaxial growth and etching is used for produc- with electronics and MEMS. As for electronically inte-
tion. Scalable, low-cost mass production makes PICs a grated chips, PICs promise to be robust and energy effi-
promising candidate, e.g. for lidar sensors in selfdriving cient [105].
186 V. Blahnik and O. Schindelbeck: Smartphone imaging

8.6 Anti-reflection coating for plastic lenses sensors, which were already being used in scientific ap-
plications from the 1970s on, became widely available. This
Anti-reflection coatings are especially important in scenes invention [111] was honored with the Nobel Prize in 2009.
with a large dynamic range, especially in the presence of From the beginning, SPC contained CMOS sensors, and
bright light sources. Residual light reflections at lens and now almost all digital consumer cameras also feature
image sensor surfaces can result in unwanted straylight or them (as of around 2010 onwards). The metal-oxide-
“ghosts” on the image plane (see Section 14.2). semiconductor (MOS) active pixel sensor (APS) was
According to the principles of the invention of Smakula developed by Tsutomu Nakamura at Olympus in 1985 [112].
[106] at Carl Zeiss in Jena, reflections on optical surfaces The CMOS active pixel sensor (CMOS sensor) “camera-on-
can be reduced through the destructive interference caused a-chip” was later developed by Eric Fossum and his team in
by a single or multilayer coating by properly choosing the the early 1990s [113–116]. A detailed description of digital
layer thicknesses and material refractive index. In today’s image sensors can be found in refs. [27, 117].
camera lenses, as well as SPC lenses, the AR coatings are A CMOS sensor is a matrix of semiconductor photodiodes
multilayer coatings typically consisting of a succession of 2 that detects the irradiance distribution on the sensor surface.
or 3 materials of thicknesses of about 10 to a few 100 nm. According to the irradiance distribution on the sensor chip
Usually a low (n < 1.5) and a high (n > 2) refractive index and the exposure time (T), electrons are generated as charge
material is used in succession. A multilayer AR coating carriers in the individual photodiodes and converted by ca-
typically consists of 6–8 layers. pacitors a voltage is finally generated [118]. The voltage is
Because of different adhesion, lower melting temper- amplified and digitized resulting in a digital value, e.g. a
atures and other physical differences, the coating process number between 0 and 255 for an 8-Bit image.
for plastic cannot easily be transferred to glass [87]. Many The probability of whether a photon, which enters the
low-index materials, like magnesium fluoride, are not sensor finally generates an electron in the photo-electric
suited to AR coatings due to poor mechanical properties of layer is the quantum efficiency (QE) (often also denoted
layers deposited without substrate heating and the high- “η”), that is 0 ≤ QE ≤ 1. QE depends e.g. on transmission
tensile growth stress. Alternative materials of standard (coating and material absorption) and geometry of micro
materials like MgF2 (n = 1.38) include SiO2 (n = 1.45), lens above the pixel and the entire light path past the 3D
leading to minor compromises in performance or greater electrical circuits. In addition to the sensor architecture, QE
effort, e.g. more layers being required. Nevertheless, also depends on the wavelength of the light, the angle of
comparable reflectivities, such as in glass AR coatings incidence and the numerical aperture of the incident light.
like <0.5% or less within the visible spectrum and a wide Novel technologies have been introduced to overcome
range of incidence angles, are possible and common in the problems of CMOS sensors with very small pixel sizes.
mass production. Over the past 10–15 years, there has been an improvement
And when it comes to glass, vapor deposition or in quantum efficiency of miniature CMOS sensors, espe-
sputter processes are preferred for plastic lens AR coatings cially in the chip architecture [119], from QE = 0.3–0.4 to
[107–109]. Due to the significantly deformed surface ge- QE > 0.7. For example the photon absorbing active silicon
ometries in SPC lenses, for a large range of different angles layer (epitaxic layer) was increased by about a factor of 2
of incidence, a uniform coating thickness is even more over the past 10 years. With “deep trench isolation” (TSMC
difficult. Of course, complex machining kinematics such as patent [120]; Omnivision patent [121]) walls are built be-
planetary gearings are out of the question in the mass tween the pixels enhancing the QE and reducing cross-talk.
production of the small lenses massively parallelized and With stacking technology the light sensitive rear illumi-
certain inhomogeneities in the coating thickness are nated photodiode array is separated from the electronics. A
accepted. Recently an alternative new process, atomic valuable resource of trends and developments on digital
layer deposition (ALD), was applied to the vivo X60 Pro for image sensors is the “Image Sensors World” Blog by Vla-
performance improvement due to more uniform coating dimir Koifman ref. [209]. For a comprehensive summary of
thicknesses [110]. very small pixel pitch CMOS evolution see ref. [122]. The
majority of the image sensors have been made as back-side
illuminated sensors (BSI) [123] since around 2010, which
9 Image sensor have a higher sensitivity than a front-side illuminated
sensor because the light at the BSI travels along a shorter,
The digital photography era started at the end of the last undisturbed path from the micro lens to the photoelectric
century, when charged coupled device (CCD) image layer (Figure 54).
 V. Blahnik and O. Schindelbeck: Smartphone imaging 187

Figure 54: Front-side and back-side illuminated CMOS sensors.

Each individual photodiode is provided with its own sometimes different color models such as CYGM or, more
electronic circuit, specifically a readout amplifier, and can recently, RYYB. Since each pixel is now only sensitive to one
be read out individually at each XY coordinate. Together, color, more precisely a specific spectral range, the missing
the cell and the electronic circuits form a pixel (picture color information must be estimated from the signal of the
element). And with the wiring, it represents the cell of a neighboring pixels (Figure 55). There are different interpola-
pixel, which is why only a portion of the cell surface is tion models for this. These are considerably more complex
sensitive to light. A microlens is attached in front of each than a bilinear interpolation of neighboring pixels of the same
cell, i.e. a microlens array over the entire sensor. This col- color, with case-sensitive weighted averaging over larger
lects as much of the incident light as possible, including areas, also considering the brightness of pixels of different
that which would otherwise hit the electronics on the light- colors [125]. The interpolation routines can also vary in a
sensitive photodiode, and also avoids shadowing within context-sensitive way in the image field, e.g. on high-contrast
the cell structure. The voltage level on the individual edges compared to quasi-homogeneous image areas. Of
photodiodes, and thus the image signal, depend solely on course, an interpolation calculation cannot reliably deter-
the respective brightness and the exposure time. mine the missing signal, e.g. the red and green components at
A color signal is obtained by placing a color filter directly the position of a diode behind a blue filter, and thus the
below the microlens in front of each individual cell. Similar to correct color and brightness value. In practice, clearly visible
the human eye, color information is detected with 3 different artifacts are seldom recognizable, and then only when the
types of color sensors. Bryce Bayer developed and patented image is greatly magnified. One consequence is that color
this concept while working at Kodak in the 1970s [124]. Each cameras with a Bayer or other color mask in the individual
pixel transmits only a limited spectral range, for example in spectral ranges – and thus overall – have a lower resolution
the red–green–blue (RGB) color model with a red, green or than monochrome cameras. Especially since only 25% of all
blue filter. The spatial arrangement of the individual color pixels are sensitive to blue and red, respectively, and 50% are
filters is often implemented using the Bayer mask as RGGB. sensitive to green. For this reason, there are monochrome
Some cameras also use different arrangements and camera modules for some SPC multicamera systems with

Figure 55: Interpolation (demosaicing) of a raw image.

188 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 56: Rolling shutter effect on fast-moving objects: Picture taken from inside a moving car; rolling shutter effect significantly tilts road
sign toward the foreground (left); moving propeller appears highly deformed due to rolling shutter effect (right). Images taken with SPC
cameras. (Left-hand image courtesy of Richard Gerlich.)

subsequent fusion of the high-resolution image from the already feature an electronic “global shutter” (for high-
monochrome camera with the color information from the speed slow-motion pictures in smartphones: Sony (2017)).
second camera (e.g. Huawei P9). Silicon has a monotonically increasing sensitivity from
In contrast to high-quality DSLRs or mirrorless system blue towards IR. Due to this strong absorption of silicon in
cameras, in which a mechanical shutter leads to the fast, IR and in order to limit the complexity of the RGB filters, an
almost simultaneous exposure of all pixels, exposure additional IR band pass filter is used to block the residual
control with SPC is carried out purely electronically. This significant portion of light in IR.
means that here, in contrast to DSLR, exposure is done Figure 57 shows the spectral response of the sensor’s
during the readout. In the case of CMOS sensors, this RGB filters, including the transmission of the IR filter for
means that the individual photodiodes are not exposed the standard cameras working in visible-shortwave-
and read out at the same time, so runtime effects do occur infrared (VIS). Please note that for special camera mod-
(rolling shutter effect, Figure 56). As a result, rolling ules like for 3D face or iris recognition, the IR cut-off may
shutters are especially limiting in slow motion mode. be at, e.g., 840 nm or 950 nm. In general, the lens trans-
However, there are new CMOS image sensor developments, mission and the external window glass absorbs some light
especially for industrial machine vision applications, that and varies with wavelength. The transmission of these

Figure 57: Relative sensitivity of a diode

behind a red, green or blue filter of the
CMOS sensor of an SPC. The transmission
of a typical IR filter is also shown.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 189

contributions depends especially on the specific coatings The abscissa is shown in exposure values (EV). The zero
as well as on the spectral material transmission of the lens value was chosen here so that it is close to the saturation
elements [126]. value, so that the noise value is 10 EV, i.e. a factor of 210
The relation between the physical intensity of the light below saturation. Here the exposure levels result as the
(horizontal axis) received by the image sensor and the logarithm of two of the number of incident photons divided
output digital numeric value (vertical axis) is described by by their value when saturation occurs. The change by +1 EV
the so-called opto-electronic conversion function (OECF, corresponds to an increase in exposure by a factor of 2. The
[127]). The OECF is often simply designated as ‘response ratio of maximum and minimum signal is called the dy-
curve’ or ‘characteristic curve.’ The exact term for “in- namic range.
tensity” in the image plane is “irradiance” as used in An analog-to-digital converter converts the voltage
radiometry, which is the radiation power per area, generated by the diode into a digital value. Simple, and
measured in Watts per square meter (W/m2). For visible especially older, SPCs use 8-bit converters to display sig-
light, the Commission Internationale de l’Eclairage (CIE)- nals from 0 to 255. Newer and high-quality SPCs and
standardized luminous efficiency function of the human camera models also use 10-, 12-, or 14-bit converters.
eye at daylight is used to define the corresponding photo- However, a higher bit value in the analog-digital converter
metric variable called “luminance”, which is measured in does not necessarily lead to better image quality. This de-
Candelas per square meter (cd/m2). pends entirely on factors such as the noise behavior of the
Let us consider a single photodiode: To generate a image sensor and the image motif.
measurable signal, there must be a minimum brightness on
the pixel (minimum signal). This corresponds to the noise
limit. The relationship between the number of electrons 10 Image processing
generated in the diode and the luminance is described by
the photoconversion curve (see Figure 58). The diode The basic image processing function on a digital camera is
signal rises from the noise limit. With current high-end used to create the most natural image possible of the
smartphone CMOS sensors this is between 1 and 2 electrons subject, and perhaps enhance it somewhat in terms of
until it saturates (maximum signal). The number of contrast or color rendition. In recent years in particular,
generated electrons is called full well capacity (FWC) and and promoted by social platforms such as Instagram and
thus represents the capacity of the potential well of the Snapchat, the cosmetic improvement and alienation of
diode. Typical values for the current CMOS sensors under photos of people taken on smartphones has become
consideration are 4000–5000 electrons. The FWC of good established and grown in importance. In Asia, smart-
SLR cameras is more than one order of magnitude higher. phones without “beautifying modes” are practically un-
sellable. What is more, they even influence the general
ideal of beauty today. Moreover, the Association of German
Plastic and Aesthetic Surgeons has noted that an
increasing number of people are opting for cosmetic sur-
gery in order to recreate this heavily filtered and distorted
“Insta look” in real life [129].
Advanced image processing often combines or fuses
several images from the same or different camera modules,
including 3D sensors, and uses them to generate a better
image. This is known as computational imaging, or
computational photography. With computational imaging,
the raw images can also be "encoded," e.g. in the form of
subfields or subapertures of the light field. Alternatively,
coded or phase-distorted apertures can be deployed in
combination with deconvolution, e.g. to extend the depth
Figure 58: Typical camera response function (sensor signal; dashed of field (EDoF). (For a broad overview of computational
black) and various tone value curves (image signal; other curves)
imaging, see ref. [130]. Current research and development
(Courtesy of ref. [128]). All these tone value curves provide very
different images. Please note: Due to the definition of EV, the x axis in mobile imaging focuses on artificial intelligence, ma-
is a logarithmic axis and the almost linear photoconversion curve is chine learning and other software improvements. Mean-
shown as “curved.” while, augmented reality applications are driving further
190 V. Blahnik and O. Schindelbeck: Smartphone imaging

improvements in 3D acquisition and image recognition. The image is then usually output in an 8-bit JPG file. So,
Great hope and large investments, e.g. from companies like only 8 bits of information are available per pixel, whereby
Magic Leap, continue to be poured into superior interfaces the associated color component is still known due to the
like augmented reality (AR) glasses [131]. Bayer mask.
Before an image is saved, a great deal of work needs to Different tone value curves are generally used for
be done by the camera’s image processor. In addition to different photographed scenes to optimize the image.
demosaicing, the photoconversion curve is fitted into an These can consider an overall increase or decrease in the
8-bit brightness scale (for each of the 3 color channels) by contrast or brightness, or only in the dark areas of the im-
means of a tone value curve, as is required for the image to age. The image quality has been significantly improved by
be displayed on the usual screens (smartphone screen, TV, adapting specific image processing algorithms to the
PC monitor, etc.) or for the purposes of a photo print. As respective exposure parameters (ISO, exposure time) as
with all photographic film, the tone value curve considers well as the properties of the recorded scene (brightness,
the logarithmic sensitivity curve of the eye. A simple linear color, structure, etc.). To do this, operations and algorithm
rescaling, especially of a large dynamic range, would not parameters must be saved for a large parameter space.
approximate human perception. The distinction between The quality of the individual algorithms, such as
brightness tones, as well as other human senses such as demosaicing, tone value adjustments, noise removal, etc.
hearing, touch, weight estimates, etc., scales according to (e.g. [135]) alone are no guarantee of a good image result.
the Weber–Fechner law [132, 133] rather than according to a Since most of the image processing operations influence
logarithmic scale, from which one can also mathematically each other, the order of the individual steps and the indi-
(formally) show that this minimizes the maximum errors in vidual weights of the operations in the consecutive steps
distinguishing between signal differences [134]. (Figure 59) are at least as important, if not more so.
Furthermore, other color adjustments are usually
made, including white balance. The image is sharpened
and the noise is reduced. In principle, these are competing 11 Noise and noise reduction
operations, where the trick is to apply greater sharpening
to medium or low spatial frequencies to which humans are Small pixels are less sensitive to light than larger ones. As a
particularly responsive and to remove noise (convolution) result, SPCs have a lower signal-to-noise ratio. In addition,
moderately, on high-frequency areas according to the there is the low full well capacity value for very small
noise amplitudes. This has its limits, especially with com- pixels, which reduces the dynamic range. Compared to
plex motifs for low-light shots. DSLRs, these are all significant disadvantages, but with
In recent years an increasing number of SPCs has been some SPCs they are partly compensated for by image pro-
outputting the images in RAW format with a typical bit cessing and, recently, also by improved sensor technology,
depth of 10, 12, or 14 bits for further processing, e.g. on a e.g. deep trench isolation, binning, dual conversion gain,
computer. As a rule, however, the images are compressed. and higher quantum efficiency [27].

Figure 59: Example of (basic) image

processing in SPC. The diagram is
simplified as often several iterations are
done, image data is exchanged with data
bases, different images are composed
(different times, different modules, …), etc.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 191

The noise of the electronics plays a minor role in to- mentioned earlier, a typical focusing range for a standard
day’s contact image sensor (CIS) in reasonably good wide-angle camera is an object distance of between 0.1 m
lighting conditions. Photon noise dominates, which affects and infinity. The requirement as regards the focusing
all cameras. However, the user often does not notice accuracy can be deduced from the number of “depth
increased noise directly, as the image processing in the ranges” within the complete focusing range. The first
camera results in software-based noise reduction. Such depth range is the aforementioned hyperfocal depth
image corrections irrevocably smooth out small image range, i.e. 2.25 m to infinity. And the second depth range
details, i.e. the resolution becomes smaller and structures can be determined by setting 2.25 m as the far distance and
with less contrast, such as human skin (Figure 60), appear calculating the corresponding focusing distance, and
unnatural. This loss in image quality cannot be compen- from that deriving the lower limit of the depth range. The
sated for in subsequent image processing. However, the hyperfocal distance can be approximated very accurately
image recognition processes – which are getting better and as (Section 4.4; Eq. 27), neglecting the very small second
better – improve the image quality thanks to different noise summand):
filters, which are used depending on the image content,
f2 f2
and sometimes at individual points on an image. sF, hyp = :h = = (49)
KØthres KrØim

The relative size of the PSF spot size where the image
can be considered “sharp” is about r = 1/1500. The near and
12 Focusing far distances (snear and sfar) corresponding to the focusing
distance sF are:
The depth of field achieved by mobile phone lenses is
significantly greater than that of full-frame camera lenses. sF
snear = (50)
For a long time, the rule of thumb was that with SPCs for 1 + shF
object distances of >1 m, i.e. for most everyday situations, sF
sfar = (51)
no focusing is required. This has changed somewhat in 1 − shF
recent years due to the larger focal lengths available in We introduce the index (j) to define the successive depth
multi-cameras. A high-end standard wide-angle lens, ranges, starting with j = 1 from the hyperfocal depth region
FOV = 75°, for an image diameter of ⊘im = 12 mm has a focal from infinity to h/2. We can find the focus positions sF, j of
length of f = 7.82 mm. For an aperture ratio of f/1.7 the the adjacent depth regions by setting
hyperfocal distance is 4.5 m. That is, if the lens is focused to
an object distance of 4.5 m, the image is sharp for all object snear, j = sfar, j+1 (52)
distances between 2.25 m to infinity. sF, j sF, j+1
Specifically, that is s = s and solving for
To implement a focusing mechanism, the key pa- 1+ F,h j 1− F,hj+1

rameters are the required focusing range and accuracy. As sF, j+1 yields:

Figure 60: Image of a human hand in weak ambient light, taken with a DSLR (left) and an SPC (right): As a result of the distinct, software-
enabled noise reduction, the skin's structure is reproduced with low contrast and thus appears unnatural.
192 V. Blahnik and O. Schindelbeck: Smartphone imaging

sF, j f f
sF,j+1 = s (53) mF, j = = (2j + 1) (56)
1 + 2 F,h j 1
h h
2j+1

Recursive calculation yields the following focusing As can be seen in Table 5 the magnification grows
distances for adjacent depth ranges: nearly linearly with j.
1 1 1 From sF, j = 2j+1
1
h = sMOD we get the total number of
sF, 0 = h, sF, 1 = h, sF, 2 = h, sF, 1 = h, · · · ,
1 3 5 7 depth ranges:
sF, j = h, · · · (54)
2j + 1 h
J= (57)
Table 5 shows the depth ranges within the complete 2sMOD
focusing range of 0.1 m to infinity. The depth range quickly For the considered SPC lens data we get J = 2sMOD
h
=
decreases towards small focusing distances: at sF = 0.9 m
the region (−15 cm, +23 cm) to the foreground and back- 2 ( 100 mm )
1 4500 mm
≈ 23.
ground, respectively, is still sharp; at minimum focusing A fairly reasonable accuracy for focus positioning is
distance sF = 0.1 m the depth range is extremely small, only having about three focus steps per depth range (instead of
(−22 μm, +23 μm). one focus step, just to allow some margin for a safe spec-
Expressing the consecutive depth ranges instead of the ification in continuous operation). Consequently, within
distance (sF) in terms of the magnification the complete focusing range about 3 J ≈ 70 focus positions
s′F f should be resolved. As we will see in the following section
mF = = (55)
sF sF + f the total movement of the lens actuator to focus an object
from infinity to MOD (0.1 m) is about 0.28 mm. So the
we find with the expression we just obtained sF, j = 2j+1 1
h:
required positioning accuracy is equal to 0.28 mm/
that for the nonmacro distance (sF >> f), that is mF ≈ sF , the
f
70 = 4 μm. Specifically, with ΔmF = mF, j+1 − mF, j = hf and
depth range number (j) is linearly related to the 2
magnification h = KrØ
f
im
the required focusing accuracy is:

f KrØim K
ΔmF, acc ≈ = = (58)
3h 3f 2250tan(FOV/2)
Table : Depth ranges of an SPC standard wide-angle lens (f/.,
FOV=°, ⊘im =  mm).
For K = 1.7 and FOV = 75° the required accuracy is
ΔmF, acc ≈ 0.001.
Depth range # Sfar SF Snear mF /mF
Note that the required accuracy – as expressed with
 inf . . . .
respect to magnification – depends on the f-number K and
 . . . . .
 . . . . . FOV only, and not on size-related quantities of the camera
 . . . . . lens (e.g. on focal length f). This is a consequence of
 . . . . . relating accuracy to magnification only: As we will see in
 . . . . . Section 12.2 the required actual distances that optical
 . . . . .
groups need to move in order to bring an object into focus
 . . . . .
 . . . . . depend very much on the actual size of the lens.
 . . . . . Not all mobile phone camera modules have a
 . . . . . focusing device: Often front cameras will have a fixed
 . . . . .
focus, which is positioned at a typical face distance to-
 . . . . .
 . . . . . ward the person operating the smartphone. There are
 . . . . . also “depth map cameras” which, with a smaller image
 . . . . . format and a larger f-number, provide greater depth of
 . . . . . field (but poorer resolution) in order to create stereo-
 . . . . .
scopic depth maps in conjunction with a main camera
 . . . . .
 . . . . . module. Until around 2005, many cameras with poor
 . . . . . resolution in early generations of SPCs still offered
 . . . . . no focusing, but this all changed with increasing
 . . . . .
resolution.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 193

“Focusing” comprises three aspects: In the digital age, the availability of image data as a
– An autofocus system, i.e. the automatic determination number matrix for structured objects makes it possible to
of the target focus position directly evaluate the contrast based on the brightness
– Modification of the optical imaging system so that an variations in the read-out image. In order to ensure the
object focuses at a different distance optimal image position, the focus position must be
– Mechatronic implementation of the focus drive changed, and the contrast evaluated using the focusing
mechanism until the contrast is at its maximum. The large
There is no manual focusing option on a smartphone number of measurements required, and the associated
(except for the special creative modes on a few smart- time is the disadvantage of contrast autofocus. In addition,
phones or some apps enabling this feature), unlike there the focus initially runs beyond the optimal focus point and
is on most cameras for large image formats. This opens then swings back again before finally settling at the
up the possibility of “creative photography,” in which optimal focus point, which is perceived as unpleasant,
image areas outside the main subject can be selectively especially when making films (“overshooting oscilla-
focused. For the sake of simpler operation and because tions”). It is also possible for the focus movement to
of the large depth of field of the SPC, manual focusing is initially move in the wrong direction (“bad direction
not very useful. In addition, “creative photography” has move”) (Figure 61). Another disadvantage is that contrast
also been added to the portrait mode function in recent autofocus becomes error-prone or fails for low-contrast
years, and even enables users to change the focus posi- objects, especially in low-light conditions. The big advan-
tion and depth of field of a scene, even after the picture tage is the compact design of the camera due to the elim-
has been taken. ination of the folding mirror and the lenses due to the
simpler correction, thanks to the close distance to the im-
age plane (see Section 6.1). Typical shooting lags can range
anywhere from 0.5 s under good conditions to about 2 s in
12.1 Autofocus methods: Contrast and low-contrast conditions.
phase detection Due to the very small depth required by thin smart-
phones, the DSLR principle is of course out of the question
We will first consider autofocus systems; see also ref. [136]. for SPC. From the very beginning, practically all AF mobile
With classic cameras, a distinction is made between reflex cameras have had a contrast autofocus. In recent years,
cameras and rangefinder cameras. Autofocus systems for the contrast autofocus in high-end SPCs has been supple-
SLR cameras have been around since the 1980s. With SLRs, mented by types of “phase detection auto focus” (PDAF)
the focus is measured using a “phase contrast measure- pixels, and in some cases even completely replaced. “Phase
ment.” The position of light rays is measured on a separate contrast pixels” as split or dual pixels first appeared in
image sensor that is positioned conjugate to the camera around 2008 with the rise of professional mirrorless system
image sensor, which it reaches via a mirror attached to the cameras like the Sony Nex and, later, the Sony α7. This was
beam splitter. The position of this light beam in the later extended to groups of 4 pixels, with PDAF aiming to
objective pupil is precisely known and the deviation from detect the actual focus position. The principle can be seen as
the focus can be quantitatively determined using triangu- a “light version” of SLR phase contrast: Instead of scanning
lation. The great advantage of phase contrast measurement a very small area of the lens pupil as with an SLR camera, the
is that you know exactly what the target shift is from a amount of light that passes through a part, about half of the
single measurement in order to get to the optimal focal pupil, is evaluated here. In 2014 Sony applied masks in front
point. The disadvantage is that you need a folding mirror of the photoelectric layer in the image sensor [137], mostly in
between the lens and the image plane. The space required 4 different orientations, e.g. to differentiate between “left,
for this results in a bulkier, heavier camera, and in larger right, above, and below” (or rotated by 45° to this arrange-
lenses. Wide-angle lenses in particular are becoming ment) which partially blocks the light. Choi et al. [138] pre-
considerably larger and more complex because retrofocus sent a geometrical model and analysis. With this PDAF
lenses are needed in order to ensure the required distance principle with masked pixels, approx. 5–10% of the total
between the last lens element and the image plane pixels are used as focus pixels and are not available as
(Figure 25a). Due to their intrinsically asymmetrical struc- normal sensor pixels but leave behind “blind spots” that
ture (negative front group and positive rear group), these have to be interpolated. More recent solutions use two
lenses are much more difficult to correct than lenses a separate neighbored pixels, a “photodiode twin”, below a
small distance away from the image plane. common microlens [139]. This concept was developed by ON
194 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 61: Typical curve of actual focus position (green) and contrast (blue) for contrast autofocus process when focus is changed to another
best focus position.

Semiconductor (Aptina) and applied, e.g. to iPhones, start- e.g. 12 MP, and the image is saved. The image stack is then
ing in 2016 with Sony image sensors. Here all pixels can be removed from the buffer memory. Alternatively, the “ani-
used for imaging as well as PDAF. It does not suffer from mation” can also be saved as a “live image” or “motion
blind spots. The entire image sensor can consist of these still” at the expense of storage space.
“Dual Pixel AF” such as in the Samsung Galaxy S7 and, in
principle, distances can be measured over the entire image
field. A detailed analysis is given by Kobayashi et al. [140]. 12.2 Optical System changes focus position
Towards outer field regions, however, PDAF becomes
increasingly problematic due to the oblique incidence of There are different ways to implement optical focusing.
light since the angle of incidence is about 35° in the image With almost all camera lenses, either the entire lens (“total
corners. A cooptimization of pixel architecture, microlens lens focusing”) or one or more individual optical groups
design and data processing can be supported by ray-tracing- (“floating element focusing”) is moved along the optical
and wave-calculation-based image simulations [141]. axis. With SPC lenses, however, with the exception of a few
In addition to the focus detection mechanisms long periscope telephoto lenses, focusing is almost exclu-
mentioned on the actual image sensor, high-end SPCs usu- sively done through total lens focusing.
ally also contain active distance measuring systems, ToF, or Another optical focusing concept involves changing
lidar, with continuously improving spatial resolution. By the focal length of the lens either by deforming the lens –
combining different distance measuring systems, accuracy usually achieved with liquid lenses – or by using Alvarez–
can be improved, especially in situations in which a certain Lohmann manipulators. The latter consist of a pair of
measuring system falls short. aspherical components that can be moved laterally toward
We must also mention one reason why SPC focusing one another [142, 143]. A MEMS-driven implementation is
has become faster still: In most SPCs, when the trigger is described by Zhou et al. [144]. A liquid lens can be realized
released, instead of a single picture, a whole stack of pic- by electrowetting, whereby two immiscible liquids with
tures is taken in a somewhat reduced resolution. This is different refractive indices are placed in a cell (e.g. a cy-
essentially a short film in HD or 4K resolution. As soon as lindrical volume) and the curvature of the boundary be-
the object is in focus, the SPC switches to a high resolution, tween them is varied electrically [145]; alternatively, in a
 V. Blahnik and O. Schindelbeck: Smartphone imaging 195

liquid crystal lens, cells of birefringent liquid crystal ma- This difference is the distance that the lens and the image
terial form a variable gradient index lens [146, 147]. Liquid plane have to move in relation to one another so that the image
lenses were first commercially integrated in a smartphone remains sharp. In practice, both with SPC and with SLR cam-
in the Xiaomi Mi Mix in 2021. eras, the image sensor is fixed and the lens is moved forward in
The required lens focusing distance for focusing be- order to focus on an object that is closer to the lens (Figure 62).
tween two object distances is determined using the imag- Therefore, the entire lens is moved. With many modern DSLR
ing equation or system camera lenses, more complex focusing mechanisms
1 1 1 are used, whereby one or more individual optical groups are
− + = (59) moved within the lens [148].
s s′ f
It is noteworthy that, according to this equation, the
where s denotes the object distance and s´ the image dis- distance required for focusing scales almost quadratically
tance from the front or rear principal plane: For an infinite with the focal length, and not in an approximate linear
object distance the focal length (sʹ) on the image is equal to fashion. This means that two lenses with the same equiv-
the focal length (f), so sʹ∞ = f. The minimum optical alent focal length (or the same FOV) but differently sized
distance (MOD) is obtained from (59): s′MOD = 1+ssMOD
MOD
/f ′
. The image sensors have to move significantly different dis-
difference between these image distances, as expressed by tances in order to focus from infinity to the same close
object distances, is distance. We consider this particular photographic situa-
tion for an SPC and a DSLR:
sMOD f f2
Δs′MOD = s′MOD − f = −f =− (60) A typical close-range distance for an SPC is about
sMOD + f sMOD + f 100 mm. Thus, for the standard wide-angle lens of an SPC

Figure 62: To focus on a close object, the entire

lens is moved forward. The movement range is
relatively small, only about 0.56 mm.
196 V. Blahnik and O. Schindelbeck: Smartphone imaging

(f = 7.8 mm, equivalent on an image sensor with a diagonal mechanism used: For longer focal lengths, more space is
of 12 mm with feq = 28 mm in full format) required for the focus movements.
2
f
−Δs′MOD = sMOD
2
In the comparison above between the SPC and full-
+f = 100+7.8 ≈ 0.56 mm, while with the full-
7.8

2 frame system camera with the same FOV, a close distance

frame camera −Δs′MOD = 80+2828
≈ 7.2 mm. In this example, a
of 100 mm (measured from the entrance pupil and not, as is
path distance almost 13 times longer is required for a crop
commercially common, but not useful for this direct com-
factor of only 3.6. (A DSLR with a magnification of approx.
parison from the image plane) corresponds to a magnifi-
1:3, the user is already in the “macro photography range”;
cation of approx. 1:15 for smartphones, and for system
meanwhile, for more moderate close-range distances, the
cameras about 1:4. This means that a much larger distance
path scales almost quadratically with the focal length).
range for the lens design must be optimized, which
Alternatively, the focus offset can also be specified as a
significantly increases the opto-mechanical complexity of
function of magnification instead of the object distance. Now,
the system. This necessitates internal focusing, preferably
with the help of the imaging Eq. (59), the magnification is
with two moving groups to achieve sufficient image per-
s′ f formance, whereby the movement of one group is usually
m= = (61)
s s+f nonlinear.

or s = f (m1 − 1) = f 1−m
m
and inserted above
−f 2 −mf 2
12.3 Focusing mechanisms: Voice coil
Δs′MOD = = f ( 1−m)+mf = −mf .
f 1−m+f
m motors and other concepts
That means that for a lens with a focal length (f) the
focus movement distance directly scales with magnification: VCMs are almost exclusively used in SPCs as drives for the
Δs′MOD = −mf (62) focus movement of the lens relative to the image sensor.
Voice coil actuation exploits the interaction between a
In other words: the focus scale is linear with respect to current-carrying coil winding and the field generated by a
magnification. permanent magnet. The coil and the magnet, one in front of
From this equation it also follows that for a given the other, are attached to two sides of the camera module’s
magnification the required lens movement focusing dis- housing (Figure 63). When a current is applied to the coil
tance directly scales with focal length. For many “normal” the interaction between the fixed and the magnetic fields
DSLR lens series, i.e. with the exception of “close focus” or electrically generated by the coil creates a Lorentz force
macro lenses, the typical close distance magnification is that enables the camera body to move by a distance that is
about 1:10. Accordingly, for longer focal lengths you have directly proportional to the current applied. Detailed ex-
to go back a long way, e.g. 10 times as much for a 280 mm planations are given in refs. [149–151, 210, 211].
telephoto lens as compared with a 28 mm wide angle. This Alternative actuator types for focus movement are
applies accordingly to floating element focused lenses, but stepper motors, piezo motors [152], and MEMS. A compar-
of course also specifically – depending on the focusing ison of different actuator concepts is offered by Murphy

Figure 63: Camera components: 1. Housing

top cover, 2. spring element for gimbal
movement, 3. lens Barrel, 4. AF VCM and
housing with image sensor, 5. multiply
folded connector cable, 6. gimbal voice
coils, 7. gimbal housing, 8. camera
housing, and 9. housing cover. Courtesy of
vivo. Gimbal components are related to a
new generation of image stabilization; see
Section 13.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 197

et al. [153]; and novel VCM concepts by Hsieh and Liu [154]. and the camera’s position. Unlike in a DSLR the trigger di-
An implementation of MEMS in a SPC and comparison with rection is the same as the camera orientation. SLRs are
VCM is given in ref. [155]. Silicon-based MEMS actuation designed for the user’s hands, while a smartphone is pri-
could be a viable alternative for VCM [156] in the future. marily designed to fit in the pocket. The weight of the SLR
VCM have issues with predictability in its location due also helps to reduce hand-shaking.
to directional hysteresis, typically approx. 10 μm – that is, Electronic image stabilization (EIS) and optical image
in the order of image depth of field, temperature de- stabilization (OIS) are integrated in most high-end smart-
pendencies, and coil resistance variation. This necessitates phones. Often OIS and EIS are used simultaneously. Some
open-loop control with multiple adjustments before the of the modules of one camera system may contain OIS
focus is correct. while others do not, e.g. front cameras may not contain
EDoF by computational imaging was also considered OIS. The first mobile phone with EIS was the LG Viewty in
as a promising option for smartphones [157]. According to 2007 (a 5 MP high-end camera). For EIS the image frame is
the method proposed by Dowski and Cathey [158] a 3rd slightly cropped and, according to a photographer’s hand-
order profile aspherical (alternative phase functions are shaking as measured with the on-board gyroscope and
also possible) extends the DoF at the expense of contrast, accelerometer, compensated for by frame shift. EIS results
which can, essentially, be recovered by deconvolution with in much better moving image performance. However, the
the a priori data of the phase mask. With this method a movement during one-frame exposure is not compensated
factor of about 2 DoF extension in image space is achiev- for, which results in blur. This blur can be reduced some-
able, which in turn translates to a significant extension of what through deconvolution of the integral PSF for a frame
DoF in object distance equal to about the minimum optical exposure [160], which is nevertheless a computationally
distance: MOD = 30 cm to infinity. That is less than a intensive task. Another disadvantage of EIS is the reduc-
standard autofocus with barrel shift (MOD approx. 10 cm). tion of FOV and the fact that the image stabilization range
Autofocus achieved by moving the lens to a desired focus is limited to the FOV portion, which is taken into account
position is obsolete. However, deconvolution struggles for EIS, such that any larger hand-shaking amplitudes
with noise, which is omnipresent in low light: A significant result in residual errors.
noise level results in unrecoverable contrast. EDoF also All these disadvantages do not arise with OIS, at least
tends to produce image artefacts. There were a few EDoF in an idealized consideration that all hand-shaking is in
mobile phone cameras on the market like the Nokia E52 situ ideally compensated for – that is, that all movement is
with a 3.2 MP camera in 2009. Nokia’s marketing depart- instantaneously compensated for such that the camera
ment named it the “full-focus camera.” Even though so- does not move at all. In fact, there are some residual ab-
phisticated computational imaging methods were used, errations since the degrees of freedom used in the OIS
consumers regarded the camera as rather a cheap alter- system (e.g. relative movement of the image sensor) are not
native to a real autofocus camera. the same as the actual movement of the camera (e.g. an
angular pitch or yaw displacement of the complete cam-
era). The first OIS in smartphone imaging was released in
2012 by Nokia, in its N920.
13 Image stabilization
As the image resolution of mobile phone cameras improved, 13.1 Hand-shaking and image blur
it became more and more obvious that their imaging per-
formance was fundamentally limited by human hand- A tremor is not a pathology; it is a common physiological
shaking during image exposure [159]. Particularly in low phenomenon present in all humans. It’s an involuntary
light but also indoors, photography quality suffered signif- oscillatory movement of body parts directly caused by
icantly, especially when compared with full-frame cameras. muscles contracting and relaxing repetitively [161].
This was due to the intrinsically low etendue and a lower Everyone has a typical movement pattern, plus un-
FWC, and therefore much longer required exposure times of systematic variations. Simon [162] empirically analyzes the
the squared crop factor (approx. factor 50 for same f-stop). A typical type and amount of hand-shaking in test subjects
compensation by increasing ISO sensitivity results in addi- and their effects on the drop in image quality. Due to the
tional noise. Furthermore, in smartphone photography, omnipresent human tremor, both video performance and
camera handling is much less stable. Often the lever is large, photography worsen significantly, especially in low light
as there is a long distance between the trigger pressure point (Figure 64).
198 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 64: Picture taken with the same SPC without (right) and with OIS (left).

The key for EIS and OIS is the availability of on-board The first components (f ′ dϑx and f ′ dϑy ) are due to tremor
MEMS-based miniature sensors. Smartphone gyroscopes pitch and yaw, respectively. They do not depend on the
and accelerometers are lithographically manufactured as distance of an object to the camera. The sensitivity increases
complex 3D structures on silicon wafers that are only about towards a larger focal length (fʹ), that is towards tele lenses.
0.5 mm in diameter. Since differential capacities are used Those components are measured by the gyroscope.
in a comb-drive actuator design, the sensitivities of these ′ ′
The second components ( fs dx and fs dy) are due to
sensors are linear – unlike standard capacitors [163] – and
have a high-voltage sensitivity, leading to low power hand-shaking translation of the camera and are measured
consumption [164]. by the accelerometer. They are larger at close distances to
In order to model the effect of tremors on image per- the camera, e.g. an object distance of 0.1 m is 10× more
formance, 6 degrees of freedom of camera movement need sensitive than an object at 1 m.
to be considered: 3 translation directions denoted (x, y, z), The third components (x dϑz and y dϑz ) are due to the
and 3 angular directions (pitch, yaw, roll) denoted ϑx, ϑy, ϑz, roll: this is effectively a rotation around the optical axis and
respectively (Figure 65). results in azimuthal-oriented blur, which increases linearly
For a hand-shaking translation (dx, dy, dz) and a hand- in a radial direction from the image center. Since the hand-
shaking tilt (dϑx, dϑy, dϑz) of the camera during exposure shaking components (dϑx, dϑy, dϑz) are typically similar in
time, the image point (xʹ, yʹ) of an object point (x, y) at a amplitude, the roll-induced blur in the image corner is
distance (s) from the lens with a focal length (f) is displaced purely geometrically related to the yaw- and pitch-induced
by following the distance (dxʹ, dyʹ) (see Figure 66): blur offsets: Due to the relation tan(FOV/2) = ⊘im /(2f ′ ) a
standard wide-angle lens with an FOV of 75° has a factor of
f′
dx′ = f ′ dϑy + dx + ydϑz (63) about 0.75 smaller blur in the image corner compared to
s yaw- and pitch-induced blur. Correspondingly, for a tele of
f′ FOV = 25° it is 0.25× smaller. However, note that although
dy′ = f ′ dϑx + dy + xdϑz (64)
s the sensitivities are smaller this component is highly
 V. Blahnik and O. Schindelbeck: Smartphone imaging 199

Figure 65: Definition of parameters:

Translation (x, y, z) and tilt components (ϑx,
ϑy, ϑz).

Figure 66: Definition of distance of camera to

object point (s) and image distance
(approximately equal to focal length ( f ) for
nonmacro distances) to formulate the
equations for tremor-induced image shift
(dxʹ, dyʹ).

relevant, since most OIS systems are not able to compen- and their effect on the image blur component Δxʹ in mi-
sate for this component. crometers, assuming that fʹ = 4 mm and FOV = 75°. Ac-
Table 6 shows typical hand-shaking according to Si- cording to the detailed statistical data, also collected by
mon’s study for an exposure time of T = 0.4 s (considerably other authors [165, 166], the tremor-induced tilts of any
simplifying the statistical analysis performed in ref. [162] person in any “usual” (non-action) situation is clearly
200 V. Blahnik and O. Schindelbeck: Smartphone imaging

Table : Typical hand-shaking-induced errors and their effect on Δϑx, rms (T) ≈ 0.37 ⋅ T 0.62 (65)
image blur for mobile phone photography with an exposure time of
. s (Data from ref. []). According to this equation’s exponent, the typical
hand-shaking blur is not a purely random walk (expo-
Mean Δx′ [μm], Δx′ [μm], nent = 0.5), it also contains some systematic components
handshake s=m s = . m
(exponent = 1). Figure 67 shows the corresponding effect
Δϑx .° Pitch, yaw . . of exposure time on image blur with dx′ = ′fdϑy and
Δx . mm Decenter . .
fʹ = 4 mm.
Δϑz .° Roll . .
For interior images, exposure times with SPCs are
typically in the order of around 1/10 s, night scenes around
1 s (often ISO sensitivity is automatically increased signif-
below 1°. Accordingly, the range of most OIS is in the range icantly to prevent this): These situations clearly benefit
of 0.5–1°. from image stabilization, as image blur is in the order of 10
The y components (Δϑy, Δy) are typically similar in pixels (Figure 67). On the other hand, daylight photog-
magnitude to the x components (statistically, not individ- raphy with exposure times of 1/100 s or less is far less
ually). For “normal” object distances like 1 m the angular critical.
hand-shaking components (pitch, yaw, roll) clearly domi-
nate, whereas for close-distance photography (0.1 m) de-
center components dominate (Table 6). At about 0.3 m the 13.2 Optical image stabilization
contributions are comparable. implementations
Obviously, according to these data – and without any
image stabilization – the hand-shaking-induced blur by far There are several different compensation mechanisms
exceeds the nominal camera lens performance and signifi- (Figure 68): In a first-order approximation a tilt and shift of
cantly limits image performance. The effect is smaller at the barrel is equivalent and therefore also the correction of
smaller exposure times, as is typical in bright light envi- a hand-shaking tilt or decenter. For a fʹ = 4 mm a 1° tilt
ronments: Simon has also given an empirical formula for the corresponds to about a 70 μm image shift. In most SPCs,
dependance of integrated hand-shaking tilt rms OIS is implemented by a compensating movement of the
T
Δwx, rms (T) = ∫0 d( wx −<w
dt
x >)
dt on exposure time in seconds: entire lens barrel in x, y with a voice coil motor [167]. In a

Figure 67: Typical image blur due to angular

hand-shaking versus exposure time.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 201

Figure 68: Different OIS systems: (a) Barrel decenter (most common), (b) image sensor decenter (in the SLR world this is referred to as body
image stabilization [BIS]), and (c) “gimbal.”

few systems the barrel is tilted. Recently, alternative objects and brightness variations in the interior indistin-
movements have been introduced. In the Apple iPhone 12 guishable. If the exposure is increased to show the interior,
Pro released in 2020, the image sensor is actively moved. then the outer area is completely overexposed, that is, it is
Also in 2020, vivo launched a “gimbal system” that tilts the homogeneously white and therefore no longer resolved in
complete barrel together with the fixed-to-barrel image the dynamic range shown (Figure 69).
sensor (vivo X51). In combination with the wide-angle The reason behind this deficiency is the limited dy-
camera with a large 3° amplitude, it also supports “action namic range of image sensors in relation to the range of
cam video shooting” situations. irradiances in a scene.
Several mechanical design parameters of OIS, e.g. the As long as there are no very bright light sources (like
most common one with a VCM-driven barrel decenter, are the sun) or spotlights in the field of view or its immediate
critical for OIS performance as well as eliminating un- vicinity, the dynamic range of typical outdoor scenes is in a
wanted side-effects: The stiffness of the springs determines range of about 9–12 EV and rarely lies above 14 EV
the correctable amplitude frequency distribution, e.g. a (Figure 70). However, with strong light sources, the dy-
very stiff spring enables high-frequency corrections but namic range may lie significantly above 20 EVs [170].
exhibits smaller amplitudes at low frequencies. The me- Dynamic range is usually referred to as low ISO
chanical adjustment of yoke to moving barrel also needs to sensitivity (i.e. ISO 100). It is important to mention that DR
be balanced: A tight adjustment increases friction and decreases as the ISO value increases, typically with 0.6–
hysteresis, while a loose one – that is, a large clearance – 1.1 EV per ISO step. For example, the dynamic range with a
may lead to parasitic tilts. A more detailed description of high ISO sensitivity of 6400 mostly ranges between 6 and
recent optical image stabilization technology can be found 9 EV, with smartphones achieving even lower values. The
in a white paper from ST Microelectronics [168]. DxOMark website provides a comprehensive database of
many cameras with measured data of the dynamic range.
Figure 71 shows a development in the dynamic range of
different cameras over the past years, according to mea-
14 Dynamic range surements taken by DxOMark showing that the dynamic
range of image sensors is constantly increasing. For today’s
14.1 HDR imaging high-end, full-frame digital consumer cameras, the dy-
namic range, i.e. the range of distinguishable luminance, is
In Section 10, we looked at the display of brightness, e.g. as about 14–15 EV, while SPCs range from about 10–12 EV.
an 8-bit image, that is as 28 = 256 different brightness levels. Moreover, quick sequences of exposure times are
Now, different irradiances of the real physical object space realized; they can be subsequently processed in HDR im-
can only be recognized as distinguishable if the camera is age processing programs in ever better quality, and also
able to record all brightness values of a scene simulta- converted to HDR images on the camera itself. Meanwhile,
neously, as distinguishable values. This is usually not the many digital cameras, including mobile phone cameras,
case. If you take a picture outside on a sunny day from calculate HDR images from an automatic sequence of
within a room and set the exposure to the outside area, different exposure times. The algorithms on how to obtain
then the inside area will be completely dark and make all an extended dynamic range image from a sequence of
202 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 69: Indoor and outdoor areas are not simultaneously captured within the limited dynamic range of the camera. (Source: Reference
[169]).

Figure 70: Typical dynamic ranges of real-life scenes [170].

 V. Blahnik and O. Schindelbeck: Smartphone imaging 203

Figure 71: Dynamic range of digital cameras. SPCs are marked in blue in this graph. Source: DXOMark.

images taken with different exposures have been devel- extend over a very large value range (e.g. 16 bits, i.e.
oped in the 1990s [171, 172]. This replaced methods of 216 = 65536 tonal values, or more) are reduced to a much
manual local tone mapping, as practiced by the famous smaller brightness range (often 8 bits, i.e. 256 tones). In the
photographer Ansel Adams for his illustrated book “The HDR process, several images with different exposure
Print” by locally exposing negatives, with a holistic process values are recorded to represent a situation with a high
[173]. The tonal values of these HDR recordings, which contrast between light and dark areas, each of which then

Figure 72: From a series of 4 pictures taken at different exposures (left) and a calculated HDR image (right).
204 V. Blahnik and O. Schindelbeck: Smartphone imaging

contain information from overexposed or underexposed 14.2 Lens flare and ghosts
areas (Figure 72). This information is artificially combined
to form a new image in which the very high tonal range is A common HDR situation is when the sun or another very
compressed to a lower one [174–176]. bright light source is in or just outside the frame. In this case
When it comes to smartphone photography, it is the dynamic range often exceeds 25 or 30 EV (factor of
helpful that the brightness scale can be quickly adjusted by 1,000,000 or much higher). Then, in addition to the image
tapping on the main subject on the display. The dynamic sensor, the lens quality is crucial: residual light reflection on
range can be increased using the HDR function, which can lens surfaces from a bright light source may superpose or
usually be selected separately. The scene is then recorded even cover up parts of the image. This may even happen for a
several times with different exposure times and combined multilayer coated surface (<0.2% reflectivity) if the reflection
for the extended dynamic range according to the bright- is considered in combination with the image sensor (approx.
ness range shown. 5% reflectivity): The double reflex is a factor of
Due to the much longer exposure time required, this 0.002 × 0.05 = 0.00001 = 1/10,000 weaker than the bright
technique is unsuitable for fast-moving subjects and more light source, which corresponds to about 2−14 or −14 EV.
prone to loss of resolution due to hand shaking. The Since the maximum irradiance within a normal scene is also
problem with dynamic situations has been mitigated with about 15 EV smaller than the bright light source, the reflected
multicell sensors because they enable exposure times of ghost of the light source may appear very bright on the pic-
different lengths to be parallelized with different pixel ture. Whether or not the ghost is apparent depends very much
clusters (Figure 73). on if it is almost in focus or out of focus (Figure 74).

Figure 73: Multi-cell sensor in normal binning mode and HDR mode.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 205

Figure 74: Ghosts' paths of out-of-focus and almost in-focus ghost images. The in-focus section appears very bright on the image (Source:
Reference [169]).

The number of lens surface-surface reflections inside The reflections within the lens and in interactions with
the lens with n surfaces is the image sensor not only affect images with local high-
lights, but also every scene with a large dynamic range: The
1
n + (n − 1) + … + 2 + 1 = n (n + 1). (66) false light from the bright areas in the image overlays the
2
darker areas and reduces the macro contrast there.
The exact position, shape, color, and intensity of the In addition to using good hardware, i.e. AR coatings or
individual ghost images produced at the various optical straylight-blocking rings, the straylight performance of a
lens surfaces are each dependent on the optical design of lens can be optimized as early on as the optical design
the lens. With large apertures, local light concentrations phase (e.g. [169, 177, 178]) to avoid the aforementioned in-
usually have aberrated shapes: Caustics can occur and focus ghosts. With ghost ray trace analysis these ghosts can
may be crescent-shaped (Figure 75, left). In smartphone be identified and potentially eliminated or reduced later on
imaging, sometimes “egg-shaped” in color, often purple, in the design process through modifications on the optical
structures around very bright light sources are especially surface shapes (see Figure 76). With powerful computer-
characteristic (Figure 75, right). These are due to reflections aided design (CAD) software like ref. [207] or ref. [205];
that occur in between the micro lens array above the image straylight analysis can be extended to include the detailed
sensor combined with the IR filter and possibly also with opto-mechanical layout of the system. For a comprehen-
other lens surfaces. The purple color arises from higher sive analysis of the straylight effects of smartphone lenses,
coating reflectivity in blue and red compared to green. see reference [179].

Figure 75: Straylight in smartphone photography. Left: Radial structures produced by roughness on the lens stop, and ghosts created by
internal reflections. Right: Ghosts created by internal reflections from the image sensor micro lens array combined with IR filter. Courtesy of
Richard Gerlich.
206 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 76: Reflections on different pairs of optical surfaces. Most of the reflex paths are rather uncritical because they are severely defocused
or do not reach the image plane at all, such as 9a). Critical are those reflections that arrive almost in focus on the image plane from a larger
aperture area (b), (c). (d) Shows that the IR filter must be well coated, since otherwise – with very bright light sources – ghost images adjacent
to the desired lens PSF will appear.

For SPC lenses’ ghost ray trace analysis the fields must such as in portraits of people, but also for portraits of pets,
be sufficiently well sampled, since bright ghosts may etc. This is done to generate purely on a context-by-context
appear inside a very small portion of the full FOV. This is basis, and is based in particular on the shape of edges (face
due to the distinct local differences in reflection direction at contours), to create simplified depth maps. Or it is com-
the wiggly lens surfaces. In particular, surfaces exceeding bined with 3D acquisition to improve depth maps.
the total reflection angle can cause problems if they acci- From the normal photo, which is naturally sharp over a
dentally reflect the light near to the focus. The local vari- large depth range on smartphones, and the depth map
ation of straylight can easily be observed when rotating the created at the same time (Figure 78), a “shallow depth of
lens relative to a very bright light source (Figure 77). This is field look” is generated by computationally blurring out-of-
essentially also the situation in which SPCs are tested in the focus regions according to the depth map data. Mathemat-
lab under standardized conditions. ically, this corresponds to a (local) convolution with a depth-
dependent point spread function. The depth dependent PSF
is stored in the memory either as a function or as a
comprehensive data set. In principle, these calculated im-
15 Portrait mode
ages could resemble the real images of a DSLR if these 3D
The dual or multicameras and 3D depth sensors installed in PSF data – which is dependent on both the depth and the
smartphones in recent years make it possible to determine selected focus – match the physical PSF data of the DSLR
depth information in high resolution, i.e. depth maps. lens. The depth map should accurately reproduce the scene
Software development, e.g. through machine learning, in all its details: Incorrectly recognized depths can lead to
coupled with access to image databases, has made great very unpleasant artifacts. In addition to imitating the DSLR
strides in recent years and is used in subject recognition, camera look, one can do something in this way that one
 V. Blahnik and O. Schindelbeck: Smartphone imaging 207

Figure 77: Straylight and ghost lab analysis performed by rotating a camera relative to a bright light source (relative angle indicated in the
graph) for three different SPCs. All SPCs feature standard wide-angle lenses with an FOV of about 80°, which can be seen as the light source in
the corner of the image at 40° rotation. The ghost distribution depends on the specific optical design: overall quality is very different, and the
strength and color of the ghosts depends very much on the coating’s spectral and angular-dependent reflectivity.
208 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 78: Photo and associated depth map

of an SPC.

Figure 79: Synthesis of “DSLR-like” shallow depth of field and change in focusing distance (not feasible in DSLRs “after the fact”).

cannot of course do with a DSLR: change the sharpness level ∅EP ∅EP
∅rel. spot, ∞ = = . (67)
with a single image after the fact (Figure 79). ∅ob, Portrait 700 mm
The (artistic) “isolation” of a person against a blurred
background is very popular among ambitious photogra- A portrait photo with SPC multicamera realistic lens
phers using large image sensors. With a DSLR, portraits are data at an equivalent focal length of 85 mm requires f/3
usually taken with lenses of medium or long focal lengths with f = 8.8 mm (85 mm/CF, where the crop factor is
with a high aperture. The background of the image taken about 43.3 mm/4.5 mm = 9.6), resulting in an entrance
with an SPC is hardly blurred, as the depth of field of an pupil diameter ⊘EP = 8.8 mm/3 ≈ 3 mm. This gives
SPC is many times greater than that of the DSLR due to the a relative background blur spot diameter of only
much smaller image format. ⊘rel.spot,∞ ≈ 0.42%.
We derived the relative size of the circle of confusion, On the other hand a classic DSLR portrait lens (full-
the diameter of the PSF, in the out-of-focus background frame 1.4/85 mm) has an entrance pupil diameter of 85 mm/
earlier, in Section 4.4; Eq. (22) as follows: 1.4 ≈ 60 mm and a relative background blur spot diameter of
 V. Blahnik and O. Schindelbeck: Smartphone imaging 209

⊘rel. spot,∞ ≈ 8.7%, which is more than 20 times as large as for sensor such that the raw image consists of many small,
the SPC. partly overlapping, fields of view of a scene. From corre-
lation analysis between adjacent fields of view, disparity –
and therefore depth – can be determined. The setup can be
15.1 3D depth acquisition technology chosen such that either only a few or many capture the
same object details within the micro image frames. The
Since SPCs have been equipped with several cameras, the more images are involved the better the quality of the depth
distance can be determined stereoscopically on the com- estimation, which however is at the expense of spatial
mon image portion of the cameras. With current multi- resolution. A consequence of combining micro lenses with
cameras, this is often the given FOV of the standard tele- a much smaller focal length compared to the main lens
photo camera (feq ≈ 55–75 mm) and the corresponding crop results in small achievable disparities (typically <1%) and
of the wide-angle camera. therefore inferior depth resolution compared to multi-
With dual cameras, distances to objects can of course cameras separated by base lengths in the order of the im-
only be determined for the directional component of the age sensor size or higher. Another feature of such lightfield
neighboring cameras and not in the direction perpendic- cameras is an extended depth of field – especially for the
ular to it. By using more cameras, like the 5 lenses of the Raytrix system that uses micro lenses with different focal
Nokia 9 Pureview from 2019 (Figure 80), the disparity can lengths [185].
be captured along different directions, which improves the From the similar triangles in Figure 82 we derive the
quality of the depth map. following relations between the lenses’ focal lengths (we
When calculating images from several cameras, the assume both lenses have both same f and FOV), the object
amount of occluded areas (Figure 81) and areas with un- points distances s1 and s2 and the base distance (b) between
usable image information – e.g. due to reflections – is the cameras:
reduced. In addition, the correlation analysis to determine
s1 f
the disparity becomes more robust as the amount of over- = (68)
b y1
lapping image data increases [181–183]. Classic multiview
stereo processing is computationally expensive and usu- s2 f
= (69)
ally done on a desktop PC. It is very challenging to perform b y2
this on a mobile platform such as a smartphone. It involves
image feature extraction, feature matching, camera cali- The difference in image position on the second cam-
bration and pose estimation, dense depth estimation, era’s image plane is the disparity d = y2 − y1:
surface reconstruction, and texturing. Another multi- 1 1
d = fb( − ) (70)
aperture depth-acquisition method is lightfield imaging, as s2 s1
realized in the Lytro Illum Camera [184] or by Raytrix [185].
Here, a conventional lens is combined with a micro lens Now with f = tan(yFOV/2)
max
we express the disparity relative
array, which is positioned closely in front of an image to the image frame as d/ymax:

Figure 80: Miniature multi-camera systems. Left: Nokia N9 with 5 camera lenses with equal focal lengths; right: PiCam from Pelican Imaging
[180].
210 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 81: Raw images from the 70 mm

cameras (marked in red) of the Light L16
during a close-up indoor shot. The dispar-
ities of the individual images are large here,
greater than disparities typical of SPC
multicameras, as these are usually placed
close together. As a result, there are large
occluded areas in the background, i.e. not
visible by both cameras at the same time.

Figure 82: The disparity is the change in the

relative distance on the image sensor of two
objects at different distances, as seen from
two different perspectives.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 211

d b 1 1 able to compare (correlate) them with one another in

= ( − ) (71)
ymax tan(FOV/2) s2 s1 different image fields. A depth estimation is impossible for
unstructured objects, e.g. a clear blue sky or periodic ob-
A very far (“infinite”) distant object point appears at
jects. The depth of these image areas can only be assigned
the same position on both image sensors. In this case
using assumptions or trained by machine learning: For
1/s1 = 0 and the distance s = s2 is determined by:
example, “evenly light-blue areas in the upper part of the
b image are probably sky and are assumed to be somewhere
s= d
(72)
ymax
tan(FOV/2) in the distance. The quality of the depth map, determined
stereoscopically or multiscopically, is generally highly
Since b, d, ymax and FOV are all known from the dependent on the contrast and therefore also on the light
lens and camera data and the cameras are calibrated intensity and spectral distribution of the illumination in the
(positioning, axis-orientation), inside the common FOV scene (Figure 85).
distances can be determined by the disparity evaluation in Incorrect depth data lead to unnatural blurring on the
absolute units. image and at the edges between the foreground and
A typical baseline distance (b) between SPC modules is background. The quality of the portrait calculated in the
about 1–2 cm. For a baseline distance of b = 10 mm and an smartphone ultimately depends on the quality of the depth
FOV 44° normal tele lens a relative disparity of d/ map. So, if the depth of the image areas is incorrectly
ymax = 0.354 (that is, 3.54% of the distance between image estimated, the blurred image appears unnatural in this
center and corner) corresponds to an object distance of area. Figure 86 shows a portrait: At first glance, the person
700 mm, which is a common portrait distance for this FOV. looks nicely “isolated” from the background. If one takes a
Correspondingly, a 2.5-m object distance gives a relative closer look, one can see that areas at the edge of the head
disparity of d/him = 0.01, that is 1% of the relative image are incorrectly assigned to the background. Even very fine
height which is still in the order of 20 pixels. A roughly 10 m object structures in the foreground, such as fine hair, often
distance disparity is in the order of 2 pixels, so depth be- cannot be accurately depicted. For normal viewing on a
comes indistinguishable, which is of no practical concern smartphone screen, these errors are mostly inconspicuous
since the through depth PSF of a lens is practically and completely sufficient for the purpose.
invariant – this also applies when photographing at usual Sound face recognition requires the acquisition of
portrait distances (see next subchapter). With multi- three-dimensional depth data. For this purpose, ToF sen-
camera systems, depth resolution decreases as distance sors are mostly used in SPC, in which distances are
from the lens increases. That is, the distance between the measured simultaneously at different positions, measuring
“depth planes” increases (Figure 83). In general, depth the propagation time of IR light from the light source and
resolution depends on the depth of field of the lenses and back to the receiver. ToF does not encounter the same
on their actual focus position as well (Figure 84). problems as stereoscopic distance measurement, i.e. the
All stereoscopic distance measurement methods dependence on high-contrast structures and occlusions.
fundamentally require structured objects in order to be On the other hand, the resolution is currently much

Figure 83: The position of the “depth planes”

depends on the base distance (b1, b2, …).
More cameras lead to more depth planes
and will increase the depth resolution.
212 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 84: Depth resolution depends on the f-number, as well as on the sensor size and pixel resolution on the actual focusing position: For
regions near the focus plane, the depth resolution increases due to better lateral resolution.

Figure 85: The quality of the stereoscopic depth estimate is highly dependent on the contrasts in the image and thus also on the intensity,
direction and spectral characteristics of the illumination: Despite nominally sufficient depth resolution, there are problems with depth
detection in the example shown for the moderately illuminated indoor shot (right), while the outdoor depth map shows very good, high-
resolution depth detection (left).

smaller, at around 240 × 180 pixels. Figure 87 shows the 3D depth acquisition cameras that augment the tradi-
system layout and the principle [186]. tional front-facing camera are now fitted as standard in
 V. Blahnik and O. Schindelbeck: Smartphone imaging 213

Figure 86: Failures with depth detection (picture taken with the iPhone 7+ in portrait mode).

high-end smartphones. Some smartphones use a struc-

tured light camera system for face recognition. In most
cases, a time-of-flight system is used. This component
provides the depth data of objects in the distance, between
approximately 0.15 and 1 m in real time. 3D acquisition
systems are also used on the rear side in combination with
the rear camera system. The light source is typically a
vertical cavity surface-emitting laser (VCSEL); for a concise
presentation on principle, research and applications see
ref. [187] at a power of about 150 mW in near IR light, e.g.
λ = 940 nm. In order to deduce the time of flight that is
temporally pulsed or continuous-waved, modulated light
is required and a multiple pixel system (e.g. consisting of 2
or 4 pixels) for a single direction within the FOV, where the
pixels detect in a mutually phase-shifted way in order to
recover the travel time of light [188, 189]. The dimension of
one of these pixels is about 3 μm. As for usual imaging, a
lens is required to focus the light on the sensor. The lens’
field of view is usually similar to the standard wide-angle
Figure 87: Time-of-flight sensor for 3D acquisition. imaging lens, e.g. a 75–80° fully diagonal FOV. The
214 V. Blahnik and O. Schindelbeck: Smartphone imaging

aperture is also similar, e.g. f/2 in order to support the This equation is true for “normal imaging.” That is, if
required lateral resolution. The sensor is about 4.5 mm in we do not consider macro distances. The spot diameter
diameter. The working distance range is about 0.2–2 m. dependence is shown in Figure 88 for different focusing
distances.
The imperfections of the bokeh created by a lens re-
veals much more information about a lens than the
15.2 Simulation of lens bokeh: Camera 3D sharpness it delivers when in focus. In photography the
point spread function term “confused,” as “bokeh” translates from Japanese,
relates to light beams which no longer come together at a
The equation of the diameter of the geometrical PSF, the single point, and so are out of focus [190]. A non-uniform
“circle of confusion,” through depth was given in Section intensity distribution within an out-of-focus highlight
4.4; Eq. (17). The diameter of the circle of confusion relative source is due to lens aberrations, i.e. overcorrected low-
to the image diagonal with respect to the focus distance (sF) order spherical aberration gives rise to an “donut shape”
and the distance from the lens (s) is:
that has highlighted edges at the spots in the image fore-
f2 |sF − s| ground whereas in the background the distribution is
Ørel.spot = (73)
⊘im K sF s brighter in the spot center (Figure 89).

Figure 88: Relative diameter of the circle of

confusion versus object distance from the
lens for different focusing distances sF
f 2 |sF −s|
according to Ørel.spot = ⊘im K sF s . The lens is
a f/1.7 standard wide-angle, feq = 28 mm
(real f = 7.8 mm), for an image sensor
diameter of ⊘im = 12 mm.

Figure 89: Effect of spherical aberration on out-of-focus spot distribution.

 V. Blahnik and O. Schindelbeck: Smartphone imaging 215

Figure 90: Through-focus intensity

distribution in the presence of
overcorrected spherical aberration.

Figure 91: Out-of-focus highlights of lenses: (a) Overlapping out-of-focus highlights, (b) “cat’s-eye bokeh” due to vignetting, (c) “edgy bokeh”
due to iris stop (diaphragm with 9 blades), (d) Softar bokeh (Softar filter scatters light out of nominal light path), (e) “donut bokeh” (spherical
aberration, also red color fringe due to chromatic aberration), (f) “Christmas ball bokeh” (spherical aberration), and (g) “fine structured
bokeh”, bokeh on the right sometimes called “onion ring bokeh” (optical manufacturing induced residual surface deformations).

The effect is due to the redistribution of intensity in the In addition to the PSF, depending on the depth and
presence of aberrations (Figure 90). Furthermore, local FOV for the respective focusing position, yet another
lens surface defects, e.g. polishing artefacts, may appear as parameter is important for the appearance of the bokeh
spatially high-frequent structures. (especially in comparison to the “natural bokeh” of a DSLR
As high-aperture camera lenses vignetted due to lens): the light intensity of the respective source point. This
mount constraints [148] the bokeh spots of these lenses is not a problem as long as the dynamic range is not
looks increasingly like a cat’s eye (Figure 91c)), i.e. towards exceeded, but as soon as it is exceeded, that is to say
the image corners. This “cat’s eye” is created by the lens’ overexposed, it is unknown whether it is exceeded only
field stops in the front and rear part of camera lenses. There slightly or by several orders of magnitude. While for a real
are many more attributes, like chromatic aberrations, lens the irradiance of a (local) light source is physically
apodization, and the digital filter components, which lend redistributed inversely proportional to the surface area of
the bokeh of a specific lens a characteristic look (Figure 91). the out-of-focus spot, for the synthetic bokeh of an SPC the
216 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 92: Left: Normal shot captured with a smartphone; right: Shot captured with a full-frame camera and a highly stopped-down lens (f/16)
to give the photo a rich depth of field.

Figure 93: Left: Shot captured with a smartphone (iPhone X) in portrait mode; right: Shot captured with a full-frame system camera (Sony α7
with ZEISS Batis 2/40).

spot brightness must be guessed as the relative irradiance The face is usually noisy due to the low ambient light.
is unknown. And the background is initially just as noisy in the real
recorded image but is smoothed out by convolution to create
the bokeh effect. The side effect is that there is a difference in
15.3 Portrait look: a quality evaluation noise between the person portrayed and the background,
which is perceived as unpleasant. You can compensate for
The quality of synthetic bokeh has been continuously this difference in noise by artificially adding noise to the
improved for SPCs in recent years. However, it is very background and/or removing noise from the face area.
difficult to approximate full-frame camera quality due to In addition, very bright highlights are problematic
the issues we discussed above (depth map quality, real- because they are overexposed in the real photo. As soon
istic bokeh is computationally expensive) and because of as all color channels (R, G, and B), are overexposed, a
the inferior low-light capabilities and reduced dynamic colored light source will appear white. The color of the
range. Figures 92 and 93 show a challenging low-light source, that is a difference between RGB as it would
scene for computational bokeh with bright local light appear within the dynamic range, cannot be recovered.
sources. Consequently, the computed blurred background will be
Obtaining a high-quality bokeh computationally in incorrectly displayed as white (Figure 94). This may not
SPCs is especially challenging in low-light scenes with very happen when taking a picture with a full-frame camera:
bright background highlights for several reasons: The overexposed area of a bright light spot is distributed
 V. Blahnik and O. Schindelbeck: Smartphone imaging 217

Figure 94: Detailed look at a bokeh (here:

Out-of-focus spots of several bright light
sources): Physical bokeh of a full-frame
camera and high-aperture lens (left) versus
the synthetic bokeh of a typical SPC (please
note: there are significant quality differ-
ences between different SPC suppliers).

Figure 95: Lens bokeh of a classic camera

lens for a modern SPC’s portrait mode.
Image Credits: vivo.

over a much larger area due to the defocusing, and de- Bokeh quality tests have helped to push and guide
creases in intensity according to the area it occupies. If the smartphone suppliers to systematically improve synthetic
resulting intensity is less than the saturation value of the bokeh. As of 2018, DXO offers a test setup with critical
dynamic range, the bright light spot no longer appears objects for high-resolution depth acquisition in topologi-
oversaturated as a uniformly bright spot, but as a distri- cally enclosed areas like holes or tiny eyelets (Figure 96).
bution and additively overlaps with the structures in With structured stripes, which are placed diagonally
the area, possibly also with neighboring, defocused into the depth, it must be checked whether the natural
highlights. sharpness transition in the depth is reproduced continu-
However, the quality of synthetic bokeh has improved ously and naturally.
significantly at some smartphone manufacturers in recent The evaluation criteria are based on the ideal of the
years. Figure 95 shows a portrait image with the vivo X60 natural bokeh of a full-frame photo camera. The test
Pro+ with a “vintage look” based on real lens data. The criteria are as follows: 1. subject background segmenta-
original optical construction data of a famous classic tion; 2. repeatability; 3. blur gradient smoothness; 4.
35 mm format camera lens, the Carl Zeiss Biotar 2/58 mm bokeh shape; 5. equivalent aperture; 6. noise consistency.
from 1936, are used to create the bokeh. Criteria 1 to 3 relate to depth map quality, while 3 to 5
218 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 96: DXO computational bokeh test

(Source: Reference [191]).

relate to the 3D PSF model, and criterion 6 to the afore- 16.1 Lab evaluation during R&D
mentioned “noise” uniformity by any additional image
processing. The focus of these tests is to objectively check measurable
optical and sensor properties. A typical specification for
the development of a camera module can be between 20
and 120 pages. Most of the requirements are related to
16 Image performance electronic functions. The image quality of the module is
specification and test assessed using the following tests:
– Resolution and contrast
Image quality tests are extensive and done at different – Color reproduction
stages in the development and production processes: – Field of view and distortion
– Lab evaluation during R&D – Dynamic range
– Qualification in mass production – Auto exposure (AE)
– Qualification of the image quality, including signal – Autofocus (AF)
and image processing in the smartphone – Auto white balance (AWB)
– Lens shading
We mentioned qualification in mass production earlier (in – Color shading
Section 8.3). We will now briefly summarize other tests – Flare
performed during R&D and production. – Ghosts
 V. Blahnik and O. Schindelbeck: Smartphone imaging 219

Figure 97 shows several standardized test charts for these Not only are the objectively measurable and assessable
evaluations. criteria that have already been tested during module
In addition to these optical performance parameters, development checked, numerous subjective quality pa-
numerous other tests and inspections must be carried out rameters are also taken into account. In addition to the
by the module manufacturer. These include: technical test charts, test images of natural objects, special
– Material tests (RoHS) image arrangements, and people (Figure 98) are created for
– Dust and environmental tests evaluation purposes.
– Shock and vibration In order to for the subjective evaluations to be as
– Continuous run, lifetime objective as possible, the respective evaluations are per-
– Electromagnetic compatibility formed by several trained persons according to precisely
– Electronic tests e.g. software and sensor interfaces to specified test and evaluation procedures (lighting, viewing
the camera, drives (VCM, OIS), etc. time, etc.) and converted into key figures.
Since the development of the image processing soft-
ware takes place practically until the device is sold (and
16.2 Evaluation of image quality in the beyond), it is extremely important to be able to assess as
imaging pipeline early as possible which deficiencies can still be remedied
during regular development and which weak points still
During development of the signal processing and imaging require special treatment. The camera device must then be
software, the camera modules and the entire imaging chain released on key date X, even if the final quality can only be
are evaluated and verified. estimated at this point in time.

Figure 97: Examples of test charts. Courtesy of Image Engineering ref. [208].
220 V. Blahnik and O. Schindelbeck: Smartphone imaging

Figure 98: Examples of subjective motifs.

While there are different and mostly standardized 17 Smartphone camera interface
methods for measuring and evaluating individual param-
eters, there are no recognized standards for the overall with telescopes, microscopes,
evaluation of a camera system. and accessory lenses
In recent years, DXOMark (https://www.dxomark.
com/) has established itself in the broad public percep- Let us consider another practical aspect: With the SPC
tion as a commercial provider of tests. DXOMark is an one can very easily take pictures through a telescope
excellent resource for tests for many different (SPCs and (Figure 99), binoculars or a microscope, simply by placing
for numerous reports on image quality evaluation. Un- the SPC on the eyepiece instead of the eye. Additional
fortunately, the test procedures, evaluation matrix and optics for connecting a smartphone and telescope, binoc-
weighting, especially when evaluating subjective image ulars or a microscope to one another are not required.
parameters, are not freely accessible and therefore only
partially comprehensible and reproducible from the
outside.
The nonprofit organization VCX (https://vcx-forum.
org/) aims to bridge this gap across multiple manufacturers
and providers by developing a comprehensive test meth-
odology and adapting it to the constantly changing
requirements [192]. Great importance is attached to trans-
parency and traceability. Any manufacturer or supplier can
join this consortium, work on further developing the test
procedures, and subsequently apply the test procedures
themselves.
In order to guarantee the objectivity and neutrality of
the evaluations, the implementation of “official” tests
with ranking is exclusively reserved for manufacturer- Figure 99: Image of the moon taken with SPC through the telescope
independent test laboratories certified by VCX. at the Aalen observatory.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 221

This is possible because the FOV and entrance pupil scopes, binoculars, or microscopes is often 60–70° (image
diameter of an SPC are similar to that of the human eye and circle diameter) and is therefore completely captured by
thus also the eyepieces that are designed for it (Figure 100): the SPC wide-angle optics (FOV approx. 70–80°).
With the 28 mm-equivalent SPC wide-angle lens, the However, good freehand photos are a matter of
entrance pupil diameter is around 2–4 mm, like that of the patience and luck: The tolerances for the positioning
day vision of the human eye (night vision approx. 5–7 mm (centering and distance) of the SPC entrance pupil are
depending on age). The captured image field angle of smaller than one 10th of a millimeter, otherwise we have to
typical high-quality eyepieces on telescopes, spotting accept a loss in sharpness. There are some precise adapter

Figure 100: The smartphone's exit pupil diameter in daylight conditions of approx. 2–4 mm is similar to the entrance pupil diameter of the
human eye. As the entrance pupil of the smartphone is also at the front of the device, it can be placed right at the desired position of the optical
instruments' eyepieces' exit position. Typical fields of view of high-end spotting scopes or astronomical telescopes or microscopes are very
similar to the FOV of the smartphones' standard wide-angle lens of approx. 75°.

Figure 101: Adapter for attaching a

smartphone to a spotting scope.
222 V. Blahnik and O. Schindelbeck: Smartphone imaging

solutions available (Figure 101). Usually these adapters are Sometimes a connection to a camera with a longer
only suitable for a limited number of eyepieces and only for focal length would be more attractive because the entrance
certain smartphones. pupil is larger, and the image field appears more magni-
It is important that the entrance pupil of the camera fied. It is not uncommon for the internal image processing
lens can be brought exactly to the position of the exit pupil of the camera system to thwart the bill. This is because the
of the eyepiece. This is almost always the case with the SPC “telephoto range” also requires the wide-angle camera to
because the entrance pupil is at the front. However, the be active and not covered, as the software needs the image
entrance pupil on full-frame cameras is usually located from the wide-angle camera to display the two images in
deep inside the lens. If the distance from the front lens to the telephoto FOV in order to merge them.
the entrance pupil exceeds the eye clearance of the The modern integrated multicamera systems have
eyepiece – in the case of eyepieces designed for spectacle replaced the accessory lenses which were on the market
wearers this distance is about 18 mm – then the image field for many years. These auxiliary optics, or converter
is vignetted. This means that one can only see a limited lenses, cause a moderate change in the focal length:
area of the image field. For direct observation with an Typical converter lenses achieve a telephoto factor of
eyepiece, a certain amount of field curvature can be toler- 2–3× at angular magnification or 0.6× for distortion-free
ated because this can be compensated for by the accom- wide-angle conversion. Even at these moderate conver-
modation of the eye while the fovea wanders through the sion factors quite significant effort is required, e.g. usage
field of view. The amount of compensable focus of a of aspherical lens elements, to achieve high-contrast
50-year-old human is about 2–3 diopters, but strongly de- and distortion-free image performance (Figure 104).
pends on the humans age [193]. Now a field curvature Furthermore, there are converters for extremely wide
corresponding to 1–2 diopters clearly deteriorates off-axis FOV fish-eye images. It is practically impossible to
image quality when viewed through a camera on an image implement larger angular magnification or demagnifi-
sensor. cation in good quality: The afocal design [194] must be in

Figure 102: Scheme of afocal lens: Galilei and Kepler Type (left) and optical design example of a 4× converter Galilei and Kepler Type (right).
 V. Blahnik and O. Schindelbeck: Smartphone imaging 223

the Galileo design, i.e. with a positive and negative lens object structures of around 10 μm. With this magnification,
group but without an intermediate image, so that the it is inevitable that the object is very close, only about 20–
image on the display does not appear upside down, as 30 mm in front of the cover glass, in front of the smart-
with a Kepler type, which contains an intermediate im- phone, and that the smartphone housing covers a lot of
age (Figure 102). ambient light. The locally mounted, laterally offset flash is
The optical lenses available on the market were not unsuitable for uniform and natural lighting. This is a major
connected to the SPC with any electronic interfaces to disadvantage compared to macro lenses for large image
enable the system to recognize that the image would have formats, with a magnification of up to 1:1, i.e. an object field
to be turned upside down electronically when connecting. of 43.2 mm in diameter and an approx. 10-μm resolution,
The diameter of a Galileo attachment lens increases almost where the working distance between lens and object is
proportionally to the TF and is unacceptably large – at much larger. Here, the external lens attachment for SPC in
around 4× – and delivers good image performance at the Figure 103 (left) is still a good compromise: Ambient light
same time (Figure 102 (right)). For larger telefactors it is reaches the object from the side through the diffusely
cheaper to use Kepler systems [195]. scattering glass ring. The thickness of the glass ring cor-
While afocal lens attachments have practically been responds exactly to the necessary focus distance, so the
replaced by integrated camera lenses with the corre- system is placed on it when taking a picture. The layout
sponding focal lengths, smartphone-integrated macro with two lens groups enables high-contrast, distortion-free
lenses are still a rarity, such as in the Nokia 8.3. Such macro images. Furthermore, via an adjustment mechanism for
lenses achieve magnifications of around 1:6 to 1:8, i.e. they the distance between the two lens groups an additional
image an object field of around 25–35 mm and can resolve manual focusing mechanism can be implemented, which

Figure 103: Auxiliary optics for SPCs. Macro

close-range optics, and wide-angle and
telephoto converters.

Figure 104: Optical design of accessory lenses shown in Figure 103: Vario-Proxar 40–80, Mutar 0.6×, and 2×, respectively, (drawn on same
scale) [196–198]. The SPC lens is represented by an ideal lens with a 4 mm focal length (thin blue lines).
224 V. Blahnik and O. Schindelbeck: Smartphone imaging

enables the photographer to shoot sharp pictures within a Structure recognition in microscope applications, e.g.
certain magnification range [198]. when analyzing bacteria and parasites in drinking water or
Microscope attachment systems go one step further in clinical pictures in blood samples, has significantly
terms of object resolution: Such systems are described by improved in recent years thanks to machine learning.
Switz et al. [199–201]. There are also some commercially Based on sample data, and possibly also on system settings
available systems, e.g. https://diple.smartmicrooptics. such as lighting mode or depth stack, neural networks
com/. learn relationships with which features in image data can
One can easily achieve very good imaging performance be better identified.
by placing a smartphone lens directly in front of the main There are several accessory products for healthcare,
camera (see Figure 105). This is possible because the dia- monitoring and diagnosis, like Shack–Hartmann sensors
phragm is in front of the first lens. A nearly diffraction- for eye defect analysis, spectroscopes, and microscopes
limited image is then obtained over a large part of the im- [202, 203]. Ultimately, finding everyday applications for a
age field, because each lens is well corrected, and the beam large number of people will decide whether those func-
path is collimated as in the original system. Then one has a tionalities will be integrated into smartphones or not.
1:1 magnification, the resolution of which is essentially During the COVID-19 pandemic, some smartphones – like
only limited by the sensor pixel size due to the high optical the Huawei Honor Play 4 Pro 5G – have been equipped with
quality. Since the pixels are about 1 μm in size, the object IR thermal sensors for quick human body temperature
detail resolution is about 2 μm, which is close to the reso- measurement. Overall, medical and healthcare applica-
lution of an ordinary professional light microscope tions benefit from the improved capabilities of smart-
(Figure 106). The object field is about 6 mm in diameter, phones [204]. FLIR offers thermal IR images that can be
which is comparable with commercial light microscopes. used to identify heat leaks in buildings. However, this
Smartphone microscopy is not exclusively an add-on functionality is limited to a niche market.
feature anymore, as beginning of 2021 in OPPO Find ×3
Pro a camera module with a magnification around 1:1 has
been integrated. 18 Summary and outlook
Almost nobody would have expected the excellent quality
of today’s SPCs to be achievable 20 years ago when the
first miniature cameras were integrated into mobile
phones. They have almost completely replaced compact
cameras and have slowed professional camera system
sales (Section 2.1). Today the quality of the recordings
with SPCs is hardly distinguishable from professional
cameras in many everyday situations, especially when
there is plenty of ambient light. In more difficult condi-
tions such as low-light situations or to capture fast-
moving subjects, however, there are still significant
differences, but these will continue to become more minor
in the future. The basic physical disadvantages of the
miniaturization of a photo camera are (Section 4):
(1) For optical image formation diffraction limits to ach-
ieve very high pixel resolutions
(2) Very large depth of field, which is undesirable in cre-
ative photography (e.g. portraits)
(3) Small pixel size results in low light per pixel: increased
noise or longer exposure times
(4) Due to space limitations in a smartphone body, tele
lenses, and optical zoom systems are often not possible
Figure 105: Mini microscope consisting of two smartphone lenses,
image sensor, specimen sample with cover glass, in front of a light
source (e.g. area light emitting diode (LED)) for uniform illumination In Sections 4.3, 4.4 and 4.5 we show that the first 3 disad-
of the specimen. vantages can be traced back to a single parameter, the
 V. Blahnik and O. Schindelbeck: Smartphone imaging 225

Figure 106: Comparison of the image quality

of a smartphone microscope with that of a
commercial microscope. The smartphone
resolves structures just a few micrometers
in size.

optical systems etendue. In addition, starting from the Without the user noticing, a single image is created
current system architecture with scaling laws, we discuss through multiple recordings, being processed in various ways
the consequences of a further reduction in the size of the depending on the recording situation or camera mode. In the
systems or of pixel sizes (Sections 4.6 and 4.7). Accord- meantime, artificial intelligence algorithms identify image
ingly, a further reduction in the pixel size below the current content through the connection to image databases linked
level of 0.7 μm would have to increase the complexity of the with position recognition via global positioning system (GPS)
optical system considerably due to the necessary increase and change or insert them virtually into the recorded image,
in numerical aperture. But, this is contrary to miniaturi- or remove disruptive objects. In portrait mode, the large depth
zation. With the current system concept, it is therefore not of field, as dictated by physics, is computationally reduced
to be expected that significantly better optical performance by convolution of the 3D data of the scene with a depth-
will be achievable in the future. dependent point spread function according to the depth map
The efforts of a branch of industry with strong growth of the scene (Section 15). Dynamic range can be extended by
that is closely interlinked (Section 2.2) led to dynamic exposure bracketing, that is combining multiple images with
technology development (briefly summarized in Section 3) different exposure times to form an HDR image. This function
to reach the limits of the physical limitations mentioned has been further improved in recent years with multicell
and to circumvent them in some cases. Computational image sensors, in which individual pixels are controlled in
imaging in particular was responsible for shifting the different ways, e.g. with regard to the exposure time (Section
physical limits, which was spurred on by the considerable 14). Optionally, the effective pixel area can also be increased
increase in computing power on mobile platforms, greatly in order to reduce noise in low-light situations. Advances in
improved algorithms, and hardware extensions such as the architecture of CMOS image sensors (Section 9) or context-
multicamera systems and 3D acquisition systems (e.g. ToF and situation-sensitive noise suppression algorithms in im-
sensors). Identity verification and augmented reality ap- age processing (Sections 10 and 11) also help in a low-light
plications will be a motor to further improve imaging environment. The subjects of image sensors and image pro-
including 3D acquisition and position recognition sensors cessing are presented here rather succinctly with reference to
in the years to come. the very extensive literature.
226 V. Blahnik and O. Schindelbeck: Smartphone imaging

We go into more detail about the optical system and accelerometers in smartphones as well as in the refine-
optical technologies (Sections 6–8). The wide-angle lens of ment of the actuators.
the main camera has become more and more complex and Finally, in Section 17, SPCs’ connection interface with
powerful over the years and, with a ratio of track length to telescopes, microscopes, and other auxiliary optical sys-
sensor diagonal of only about 2/3, achieves a resolution tems is reviewed. Due to the similarity of the SPC with the
that is almost diffraction-limited with high apertures of human eye in terms of pupil position, size, and field of
about f/1.6 and thus supports image sensor resolutions of view, the connection is usually possible without significant
almost 1 μm. This very compact size, which is far superior to loss of quality. With the connection to powerful image
comparable classic optical designs made of glass, is made processing and extensive image databases, the images can
possible by optical designs with extreme plastic aspheres. be processed and interpreted and then sent or shared via
In the past few years, more and more lenses with other the connectivity interfaces.
focal lengths have been integrated and supplement the
standard wide angle as a super wide angle up to around Acknowledgement: The authors would like to thank Dr.
FOV 120° or tele lenses either of conventional and peri- Norbert Kerwien for his valuable comments.
scope layout with field-of-view up to approx. 16°. Due to a
smartphone’s space constraints, both extreme wide-angle
and tele systems do not achieve the same high optical References
resolution as a standard wide-angle lens. Correspondingly,
in the hybrid zoom systems consisting of several lenses of [1] Techterms, 2021. Available at: https://techterms.com/definition/
different focal lengths, the camera resolution constantly smartphone.
decreases towards very large or very small fields of view [2] M. B. del Rosario, S. J. Redmond, and N. H. Lovell, “Tracking the
evolution of smartphone sensing for monitoring human
(Section 7.1). This is a disadvantage compared to many real
movement,” Sensors (Basel, Switzerland), vol. 15, no. 8,
optical zoom systems for SLR or system cameras which, pp. 18901–18933, 2015.
unfortunately, cannot be scaled to the space restrictions of [3] S. S. Saini, A. Sridhar, and K. Ahluwalia, “Smartphone optical
smartphones (Section 7.2). sensors,” Opt. Photonics News, vol. 30, no. 2, pp. 34–41, 2019.
The enabler of miniaturization is the production [4] K. C. Ginny and K. Naik, “Smartphone processor architecture,
technology of the complex components, i.e. the strongly operations, and functions: current state-of-the-art and future
outlook: energy performance trade-off,” J. Supercomput., vol. 77,
aspherical lenses with integrated mounts, which can be
pp. 1377–1454, 2021.
mass produced in plastic with high precision and repro- [5] P. M. Singh and M. Kumar, “Evolution of processor architecture in
ducibility (Sections 8.1 and 8.2), paired with a fully auto- mobile phones,” Int. J. Comput. Appl., vol. 90, 2014, https://doi.
mated assembly and adjustment process for the lens and org/10.5120/15564-4339.
the active alignment of lens to image sensor with precise [6] M. E. Khaddar and M. Boulmalf, “Smartphone: the ultimate IoT
and IoE device”, in Smartphones from an Applied Research
robotics and fast measurement technology (Section 8.3)
Perspective, Chapter 7, N. Mohamudally ed. 2017, https://doi.
based on extensive yield optimization in the optical design org/10.5772/intechopen.69734.
process (Section 8.4). The costs for high-end SPCs are well [7] M. Rather and S. Rather, “Impact of smartphones on young
below $20 per camera module. Some parts and compo- generation,” Libr. Philos. Pract., vol. 2384, 2019.
nents, such as time-of-flight cameras, are increasingly be- [8] M. Sarwar and T. R. Soomro, “Impact of smartphone’s on society,”
ing manufactured lithographically using wafer-level Eur. J. Sci. Res., vol. 98, pp. 216–226, 2013.
[9] M. Moses and J. Wade, Spycamera – The Minox Story, 2nd ed,
technology. It is possible that even more lithographically
Denver CO, Hove Foto Books, 1998.
manufactured components, including photonic integrated [10] DXO, Disruptive Technologies: Mobile Imaging taking
circuits, will be used in smartphones, and linked to elec- Smartphone Cameras to Next Level, 2018a. Available at:
tronic circuits in the future. https://www.dxomark.com/disruptive-technologies-mobile-
In Section 12 requirements for focusing accuracy and imaging-taking-smartphone-cameras-next-level/.
[11] DXO, Smartphones versus Cameras: Closing the Gap on Image
distance determination are derived. The entire lens,
Quality, 2020. Available at: https://www.dxomark.com/
driven by voice coil motors, is moved forward for close- smartphones-vs-cameras-closing-the-gap-on-image-quality/.
up focus. Motion blurring is reduced with EIS and OIS. [12] Counterpointresearch, 2021. Available at: https://www.
OIS has improved the image quality of lower-light im- counterpointresearch.com/global-smartphone-share/.
[13] Gartner Newsroom. Available at: https://www.gartner.com/.
ages by compensating for hand tremors by enabling
[14] P. Cambou and J.-L. Jaffart, Status of the Camera Module
longer exposure times (Section 13). OIS was made Industry 2019, Yole Développement, 2019. Available at:
possible by integrating MEMS gyroscopes and https://www.systemplus.fr/wp-content/uploads/2019/04/
 V. Blahnik and O. Schindelbeck: Smartphone imaging 227

YD19006_Status_Camera_Module_Industry_2019_WLO_Yole_ [36] P. L. Ruben, “Design and use of mass-produced aspheres at

Developpement_Sample.pdf. Kodak,” Appl. Opt., vol. 24, no. 11, 1985. https://doi.org/10.
[15] P. Cambou, Smartphone Camera Trends, Yole Développement, 1364/ao.24.001682.
2020. Available at: https://www.youtube.com/watch? [37] W. T. Plummer, “Unusual optics of the Polaroid SX-70 land
v=fR4KXZ1zjw0. camera,” Appl. Opt., vol. 21, pp. 196–202, 1982.
[16] D. Yang, S. Wegner, and A. Cowsky, Apple iphone 11 Pro Max [38] R. Altman and J. Lytle, “Optical design techniques for polymer
Teardown, 2019. Available at: https://www.techinsights.com/ optics,” Proc. SPIE, vol. 237, p. 380, 1980.
blog/apple-iphone-11-pro-max-teardown. [39] L. Nie, “Patent review of miniature camera lenses and a brief
[17] Yole, Rear ToF to Become Main Smartphone Camera Growth comparison of two relative design patterns,” Master Thesis,
Engine, 2019. Available at: http://image-sensors-world. College of Optical Sciences, University of Arizona, 2017.
blogspot.com/2020/02/yole-rear-tof-to-become-main- [40] W.-Y. Chen, “Optical lens system, image capturing unit and
smartphone.html. electronic device,” US 2020/0393653 A1, 2020.
[18] R. L. Baer, “Resolution limits in digital photography: the looming [41] J. Bareau and P. Clark, “The optics of miniature digital camera
end of the pixel wars,” in Imaging Systems, OSA technical Digest modules,” in International Optical Design, Technical Digest (CD)
(CD) (Optical Society of America, 2010), paper ITuB3, 2010. (Optical Society of America, 2006), paper WB3, 2006.
[19] G. Westheimer, “Visual acuity and hyperacuity,” in Handbook of [42] R. Bates, “The modern miniature camera objective: an
Optics, Volume III Vision and Vision Optics, M. Bass, Ed., New evolutionary path from the landscape lens,” Adv. Opt. Technol.,
York, McGraw-Hill, 2010. vol. 2, no. 1, pp. 13–20, 2013.
[20] S. Triantaphillidou, J. Smejkal, E. Fry, and C. Hsin Hung, “Studies [43] P. Clark, “Mobile platform optical design,” Proc. SPIE,
on the effect of megapixel sensor resolution on displayed image International Optical Design Conference, vol. 9293, p. 92931M,
quality and relevant metrics,” in Proceedings of the IS&T 2014.
International Symposium on Electronic Imaging, 2020, p. 170. [44] P. Clark, “Lens design and advanced function for module
Available at: https://t1p.de/MegaPixelEffect. cameras, chapter 1,” in Smart Mini-Cameras, T.V. Galstian, Ed.,
[21] R. D. Fiete, Modeling the Imaging Chain of Digital Cameras, Boca Raton, FL, CRC Press, 2014.
Bellingham, SPIE Press, Tutorial Text, 2010. [45] J. Sasian, Introduction to Lens Design, Cambridge, Cambridge
[22] J. W. Goodman, Introduction to Fourier Optics, Greenwood University Press, 2019.
Village, CO, Roberts & Co., 2005. [46] D. Shafer, A New Theory of Cell Phone Lenses, 2019. Available at:
[23] C.-L. Tisse, F. Guichard, and F. Cao, “Does resolution really https://de.slideshare.net/operacrazy/a-new-theory-of-cell-
increase image quality?,” Proc. SPIE 6817, Digital Photography, phone-lenses.
vol. IV, 2008, Art no. 68170Q. [47] H. Gross, F. Blechinger, and B. Achtner, “Photographic lenses,”
[24] M. Berek, Grundlagen der praktischen Optik, Berlin, de Gruyter, in Handbook of Optical Systems, vol. 4, Survey of Optical
1930. Instruments 4, Weinheim, Wiley-VCH, 2008.
[25] S. Ray, Applied Photographic Optics, 3rd ed. Waltham, Focal [48] R. Kingslake, Lens Design Fundamentals, Cambridge, Academic
Press, 2002. Press, 1978.
[26] D. A. Rowlands, “Physics of digital photography,” in IOP Series [49] W. Smith, Modern Lens Design, New York, McGraw-Hill, 2004.
in Emerging Technologies in Optics and Photonics, R.B. Johnson, [50] I. Stamenov, I. P. Agurok, and J. E. Ford, “Optimization of two-
Ed., IOP Publishing Limited, 2020. glass monocentric lenses for compact panoramic imagers:
[27] U. Teubner and H. J. Brückner, Optical Imaging and Photography, general aberration analysis and specific designs,” Appl. Opt.,
Berlin/Boston, De Gruyter Verlag, 2019. vol. 51, pp. 7648–7661, 2012.
[28] A. W. Lohmann, “Scaling laws for lens systems,” Appl. Opt., vol. [51] D. Reshidko, and J. Sasian, “Optical analysis of miniature lenses
28, no. 23, pp. 4996–4998, 1989. with curved imaging surfaces,” Appl. Opt., vol. 54, no. 28,
[29] D. J. Brady and N. Hagen, “Multiscale lens design,” Opt. Express, pp. E216–E223, 2015.
no. 13, pp. 10659–10674, 2009. https://doi.org/10.1364/oe.17. [52] L. Bertele, “Five component wide-angle objective,” US2721499A,
010659. 1951.
[30] M. W. Haney, “Performance scaling in flat imagers,” Appl. [53] H. H. Nasse, Distagon, Biogon und Hologon. ZEISS Camera
Optics, vol. 45, no. 13, p. 2901, 2006. Lenses Technical article, 2011. Available at: https://lenspire.
[31] A. Brückner, “Multiaperture cameras, chapter 8,” in Smart Mini- zeiss.com/photo/app/uploads/2018/02/en_CLB41_Nasse_
Cameras, T.V. Galstian, Ed., Boca Raton, FL, CRC Press, 2014. LensNames_Distagon.pdf.
[32] G. Carles, G. Muyo, N. Bustin, A. Wood, and A. R. Harvey, [54] L. Seidel, “Ueber die Theorie der Fehler, mit welchen die durch
“Compact multi-aperture imaging with high angular resolution,” optische Instrumente gesehenen Bilder, behaftet sind, und über
J. Opt. Soc. Am. A, vol. 32, pp. 411–419, 2015. die mathematischen Bedingungen ihrer Aufhebung,” in
[33] ZEISS Lens Data Sheet Otus 1.4/28, 2021. Available at: https:// Abhandlungen der naturwissenschaftlich-technischen
www.zeiss.com/content/dam/consumer-products/downloads/ Commission bei der Königl, München, Bayerischen Akademie
photography/datasheets/en/otus-lenses/datasheet-zeiss- der Wissenschaften in München, 1857, pp. 227–267.
otus-1428.pdf. [55] G. Forbes, “Shape specification for axially symmetric optical
[34] M. Born and E. Wolf, Principles of Optics, Cambridge, Cambridge surfaces,” Opt. Express, vol. 15, pp. 5218–5226, 2007.
University Press, 2019. [56] G. W. Forbes and C. Menke, “Optical design with orthogonal
[35] T. Steinich, and V. Blahnik, Optical Design of Camera Optics for surface descriptions,” Proc. SPIE, vol. 8884, p. 88841C, 2013.
Mobile Phones, 2012. Available at: http://www.degruyter.com/ [57] B. Ma, K. Sharma, K. P. Thompson, and J. P. Rolland, “Mobile
view/j/aot.2012.1.issue-1-2/aot-2012-0002/aot-2012-0002.xml. device camera design with Q-type polynomials to achieve
228 V. Blahnik and O. Schindelbeck: Smartphone imaging

higher production yield,” Opt. Express, vol. 21, [78] G. Shabtay, E. Goldenberg, O. Gigushinski, N. Cohen, Dual
pp. 17454–17463, 2013. aperture zoom digital camera. US 9.185.291 B1, 2015.
[58] Z. Zhuang, X. Dallaire, J. Parent, P. Roulet, and S. Thibault, [79] M. Kreitzer and J. Moskovich, “Optical design of a smartphone
“Geometrical-based quasi-aspheric surface description and zoom lens,” Proc. SPIE, Zoom Lenses VI, vol. 11106, p. 111060D,
design method for miniature, low-distortion, wide-angle camera 2019.
lens,” Appl. Opt., vol. 59, pp. 8408–8417, 2020. [80] T. Kato, A. Oohata, and H. Hagiwara, “Zoom lens and imaging
[59] H. J. Brückner, V. Blahnik, and U. Teubner, “Maximale apparatus,” US9235036B2, 2014.
Bildqualität aus Minikameras: Smartphonekameras, Teil 1: [81] H. Yamanaka, M. Kanai, M. Sueyoshi, and M. Hosoi, “Enhanced
Kameraoptik,” Physik in unserer Zeit, vol. 51, no. 5, p. 236, 2020. variable power zoom lens,” US 8.451 549, 2013.
[60] C.-C. Lin, C.-Y. Chen, Y.-T. Tseng, K.-T. Yeh, and H.-H. Huang, [82] K. Li and Kato, “Zoom lens,” US 9 176 308 B2, 2014.
“Optical Imaging module, image capturing apparatus and [83] M. P. Schaub, The Design of Plastic Optical Systems, Bellingham,
electronic device,” US20190250380, 2019. SPIE Press, 2009.
[61] M. Dror, E. Goldenberg, and G. Shabtay, “Miniature telephoto [84] Y. Yu and G. Berger, Fast and Highly Accurate Metrology Tool for
lens assembly,” US 9568712 B2, 2017. Smart Phone Lens Mold Characterization, 2016. Available at:
[62] Y.-T. Tseng, T.-Y. Hsieh, and H.-H. Huang, “Photographing lens https://www.taylor-hobson.com.de/-/media/
assembly, image capturing unit and electronic device,” ametektaylorhobson/files/learning-zone/application-notes/
US20190094500A1, 2019. a150—smart-phone-lens-mold_highres_en.pdf.
[63] B. Hönlinger and H. H. Nasse, Distortion. ZEISS camera lenses [85] J. Y. Song, T. H. Ha, C. W. Lee, J. H. Lee, and D. H. Kim,
technical article, 2009. Available at: https://lenspire.zeiss. “Development of minimized-assembly system for camera phone
com/photo/app/uploads/2018/04/Article-Distortion- lens module,” in Proceedings of the 11th euspen International
2009-EN.pdf. Conference – Como, 2011. Available at: https://www.euspen.eu/
[64] V. Blahnik, D. Gaengler, and J.-M. Kaltenbach, “Evaluation and knowledge-base/ICE11252.pdf.
analysis of chromatic aberrations in images,” Proc. SPIE, Optical [86] C. F. Lin, C.-H. Tsai, and M.-T. Chou, “Imaging lens module and
Design and Engineering IV, vol. 8167, p. 81670G, 2011. mobile terminal,” US 2016/0231526 A1, 2016.
[65] P.-L. Hsu, and S.-Y. Yang, “Optical photographing lens [87] S. Bäumer, Handbook of Plastic Optics, Weinheim, Wiley-VCH
assembly, imaging apparatus and electronic device,” US Verlag, 2005.
20190271831A1, 2019. [88] H. Gross, “Fundamentals of technical optics,” in Handbook of
[66] L.-M. Chen and H. H. Huang, “Imaging optical lens assembly, Optical Systems, vol. 1, Weinheim, Wiley-VCH, 2005.
imaging apparatus and electronic device,” US2019/0170970 A1, [89] K. Weber, D. Werdehausen, P. König, S. Thiele, M. Schmid,
2019. M. Decker, P. W. De Oliveira, A. Herkommer, and H. Giessen,
[67] C.-Y. Liao and W.-Y. Chen, “Imaging lens assembly, image “Tailored nanocomposites for 3D printed micro-optics,” Opt.
capturing unit and electronic device,” US20210048630A1, 2021. Mater. Express, vol. 10, pp. 2345–2355, 2020.
[68] P.-L. Hsu and W.-Y. Chen, “Imaging lens assembly, image [90] D. Werdehausen, Nanocomposites as Next-Generation Optical
capturing unit and electronic device,” US20210018725, 2021. Materials. Fundamentals, Design and Advanced Applications,
[69] D. H. Jeong, H. S. Yoo, J. I. Lee, and J. K. Kim, “Portable electronic Berlin, Springer Series in Materials Science, 2021.
device, optical imaging system, and lens assembly,” [91] K. Straw, “Control of thermal focus shift in plastic-glass lenses,”
US20200409065A1, 2020. Proc. SPIE, vol. 237, p. 386, 1980.
[70] Y. Yao, and Y. Shinohara, Folded Lens System, Patent US 2019/ [92] K. Bitzer and A. By, “Active alignment for cameras in mobile
0196148 A1, 2019. devices and automotive applications,” in IEEE Electronics
[71] K. Araki, T. Tanaka, M. Sekita, K. Kimura, N. Nanba, H. Saruwatari, Packaging Technology Conference, EPTC, 2010, pp. 260–264.
and T. Akiyama, “Optical apparatus,” US6426841B1, 2002. [93] D. Wilson, Automating Optical Alignment of Camera Modules,
[72] G. Carles and A. R. Harvey, “Multi-aperture imaging for flat 2017. Available at: http://www.novuslight.com/
cameras,” Opt. Lett., vol. 45, pp. 6182–6185, 2020. automatingoptical-alignment-of-camera-modules_N3530.html.
[73] F. Dai, J. Wenren, and J. Yang, “Optical imaging system,” US [94] Imatest, Imatest Documentation by Norman Koren, 2009.
20190187446, 2019. Available at: https://www.imatest.com/docs/Imatest%
[74] L.-M. Chen, and H. H. Huang, “Optical photographing system and 20Documentation.pdf.
electronic device,” US10877244B1, 2020. [95] D. Winters, “Image quality testing improves as cameras
[75] E. Tremblay, R. A. Stack, R. L. Morrison, and E. F. Ford, “Ultrathin advance,” Photonics Spectra, vol. 48, no. 1, pp. 66–70, 2014,
cameras using annular folded optics,” Appl. Opt., vol. 46, no. 4, Available at: https://trioptics.com/wp-content/uploads/2019/
p. 463, 2007. 08/Articles_Image_Quality_Testing_improves_as_Camera_
[76] E. Tremblay, R. A. Stack, R. L. Morrison, J. H. Karp, and E. F. Ford, Advance_Photonics_Spectra_0114-2.pdf.
“Ultrathin four reflection imager,” Appl. Opt., vol. 48, no. 2, [96] R. Bates, “Performance and tolerance sensitivity optimization of
p. 343, 2009. highly aspheric miniature camera lenses. Optical system
[77] I. Yedid, The Evolution of Zoom Camera Technologies in alignment, tolerancing, and verification IV,” Proc. SPIE, vol.
Smartphones, Corephotonics White Paper, 2017. Available at: 7793, pp. 779302, 2010.
https://corephotonics.com/wp-content/uploads/2017/09/ [97] S. Jung, D.-H. Choi, B.-L. Choi, and J. H. Kim, “Tolerance
Corephotonics-White-Paper_The-Evolution-of-Zoom-Camera- optimization of a mobile phone camera lens system,” Appl. Opt.,
Technologies-in-Smartphones_Aug-2017.pdf. vol. 50, no. 23, 2011, https://doi.org/10.1364/AO.50.004688.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 229

[98] F. E. Sahin, “Lens design for active alignment of mobile phone [117] J. Nakamura, Image Sensors and Signal Processing for Digital
cameras,” Opt. Eng., vol. 56, no. 6, 2017, Art no. 065102. Still Cameras, Boca Raton, CRC-Press, 2005.
[99] L. Carrión-Higueras, A. Calatayud, and J. Sasian, “Improving as- [118] I. Takayanagi, “CMOS image sensors,” in Image Sensors and
built miniature lenses that use many aspheric surface Signal Processing for Digital Still Cameras, J. Nakamura, Ed.,
coefficients with two desensitizing techniques,” Opt. Eng., Boca Raton, FL, CRC Press, 2006, pp. 143–177.
vol. 60, no. 5, 2021. https://doi.org/10.1117/1.oe.60.5. [119] R. Fontaine, “The state-of-the-art of mainstream CMOS image
051208. sensors,” in International Image Sensor Workshop, 2015.
[100] J. Joo, “Design of a miniaturized imaging system for as-built [120] S.-Y. Chen, C.-C. Chuang, J.-C. Liu, and D.-N. Yaung, “Image
performance,” Graduate Theses-Phys. Opt. Eng., vol. 8, 2020. sensor with deep trench isolation structure,” US2012/0025199
https://scholar.rose-hulman.edu/dept_optics/8. A1, 2012.
[101] J. P. McGuire, and T. G. Kuper, “Approaching direct optimization [121] Y. Qian, H.-C. Tai, D. Mao, V. Venezia, and H. E. Rhodes, “Image
of as-built lens performance,” Proc. SPIE, Novel Optical Systems sensor having dark sidewalls between color filters to reduce
Design and Optimization XV, vol. 8487, p. 84870D, 2012. optical crosstalk,” US 2012/0019695 A1, 2012, .
[102] T. Hayes, “Next-generation cell phone cameras,” Opt. Photon. [122] M. Lapedus, Scaling CMOS Image Sensors, 2020. Available at:
News, vol. 23, no. 2, pp. 16–21, 2012. https://semiengineering.com/scaling-cmos-image-sensors/.
[103] E. Wolterink and K. Demeyer, “WaferOptics(R) mass volume [123] H. Yamanaka, “Method and apparatus for producing ultra-thin
production and reliability,” Proc. SPIE, vol. 7716, p. 771614, 2010. semiconductor chip and method and apparatus for producing
[104] M. Zoberbier, S. Hansen, M. Hennemeyer, D. Tönnies, ultra-thin back-illuminated solid-state image pickup device,”
R. Zoberbier, M. Brehm, A. Kraft, M. Eisner, and R. Voelkel, US7521335 B2, 2009.
Wafer Level Cameras – Novel Fabrication and Packaging [124] B. E. Bayer, “Color imaging array,” U.S. Patent 3 971 065, 1976.
Technologies, 2009. Available at: https://www.researchgate. [125] H. Malvar, L.-W. He, and R. Cutler, “High-quality linear
net/publication/266046128_Wafer_level_cameras_-_novel_ interpolation for demosaicing of Bayer-patterned color images.
fabrication_and_packaging_technologies. Acoustics, speech, and signal processing, 1988,” in
[105] J. Lapointe, M. Gagné, M.-J. Li, and R. Kashyap, “Making smart ICASSP-88., 1988 International Conference on. 3. iii-485, 2004.
phones smarter with photonics,” Opt. Express, vol. 22, [126] C.-Y. Lee, S.-W. Hyun, Y.-J. Kim, and S.-W. Kim, “Optical
pp. 15473–15483, 2014. inspection of smartphone camera modules by near-infrared
[106] A. Smakula, “Verfahren zur Erhöhung der Lichtdurchlässigkeit low-coherence interferometry,” Opt. Eng., vol. 55, no. 9, 2016.
optischer Teile durch Erniedrigung des Brechungsexponenten https://doi.org/10.1117/1.OE.55.9.091404.
an den Grenzflächen dieser Teile,” Deutsches Reichspatent, [127] ISO 14524, Photography, Electronic Still-picture Cameras,
vol. 685, p. 767, 1935. Methods for Measuring Opto-Electronic Conversion Functions
[107] L. Martinu and J. E. Klemberg-Sapieha, “Optical coatings on (OECFs), Geneva, Switzerland, Vernier, 2009.
plastics,” in Optical Interference Coatings, N. Kaiser, and [128] H. J. Brückner, V. Blahnik, and U. Teubner, “Tricksen für gute
H. Pulker, Eds., Berlin/Heidelberg, Springer-Verlag, 2003. Bilder: Smartphonekameras, Teil 2: Bildsensor und
[108] U. Schulz, U. Schallenberg, and N. Kaiser, “Antireflection -verarbeitung,” Physik in unserer Zeit, vol. 51, no. 5, p. 290,
coating design for plastic optics,” Appl. Opt., 2002. https:// 2020b.
doi.org/41.3107-10.10.1364/AO.41.003107. [129] VDÄPC, Pressemitteilung der Vereinigung der deutschen
[109] U. Schulz, “Review of modern techniques to generate ästhetisch-plastischen Chirurgen 2019, 2019. Available at:
antireflective properties on thermoplastic polymers,” Appl. https://www.vdaepc.de/pressemitteilung-statistik-
Opt., vol. 45, pp. 1608–1618, 2006. 2019-trends-der-aesthetisch-plastischen-chirurgie/ [Press
[110] P. Paul, K. Pfeiffer, and A. Szeghalmi, “Antireflection coating on release of the Association of German Aesthetic Plastic
PMMA substrates by atomic layer deposition,” Coatings, 2020. Surgeons 2019].
https://doi.org/10.64.10.3390/coatings10010064. [130] J. N. Mait, G. W. Euliss, and R. A. Athale, “Computational
[111] W. S. Boyle and G. E. Smith, “Charge coupled semiconductor imaging,” Adv. Opt. Photon., vol. 10, pp. 409–483, 2018.
devices,” The Bell Systems Technical Journal, vol. 49, [131] B. C. Kress, “Digital optical elements and technologies
pp. 587–593, 1970. (EDO19): applications to AR/VR/MR,” Proc. SPIE, Digital Optical
[112] K. Matsumoto, T. Nakamura, A. Yusa and S. Nagai, “A new MOS Technologies, vol. 11062, p. 1106222, 2019.
phototransistor operating in a non-destructive readout mode,” [132] G. T. Fechner, Elemente der Psychophysik, vol. 2, Leipzig,
Jpn. J. Appl. Phys., vol. 24, no. 5A, p. L323, 1985. Breitkopf und Haertel, 1860.
[113] E. R. Fossum, “CMOS image sensors: electronic camera-ona- [133] E. H. Weber, “Der Tastsinn und das Gemeingefühl,” in
chip,” IEEE Trans. Electron. Dev., vol. 44, no. 10, Handwörterbuch der Physiologie, vol. 3, Braunschweig, 1850.
pp. 1689–1698, 1997. [134] R. Portugal and B. Svaiter, “Weber-fechner law and the
[114] E. R. Fossum, “Active pixel sensors—are CCDs dinosaurs?,” CCD’s optimality of the logarithmic scale,” Minds and Machines, vol.
and Optical Sensors III, Proc. SPIE, vol. 1900, pp. 2–14, 1993. 21, pp. 73–81, 2011.
[115] S. Mendis, S. E. Kemeny, R. Gee, B. Pain, and E. R. Fossum, [135] R. C. Gonzalez and R. E. Woods, Digital Image Processing,
“Progress in CMOS active pixel image sensors,” Charge- Upper Saddle River, NJ, Prentice Hall, 2008.
Coupled Devices and Solid State Optical Sensors IV, Proc. SPIE, [136] P. Robisson, J.-B. Jourdain, W. Hauser, C. Viard, and F. Guichard,
vol. 2172, pp. 19–29, 1994. Autofocus Measurement for Imaging Devices, IS & T, 2017.
[116] S. Mendis and E. R. Fossum, “CMOS active pixel image sensor,” [137] Shimoda, K. and Fujii, S.. “Imaging device and imaging
IEEE Trans. Electron Devices, vol. 41, no. 3, pp. 452–453, 1994. apparatus,” US 8,817,166 B2, 2014.
230 V. Blahnik and O. Schindelbeck: Smartphone imaging

[138] B.-S. Choi, J. Lee, S.-H. Kim, S. Chang, J. Park, S.-J. Lee, and [155] A. A. N. Galaom, “Integration of a MEMS-based autofocus actuator
J.-K. Shin, “Analysis of disparity information for depth into a smartphone camera,” Master thesis, Mechanical and
extraction using CMOS image sensor with offset pixel Industrial Engineering University of Toronto, 2016. Available at:
aperture technique,” Sensors, vol. 19, p. 472, 2019. https://tspace.library.utoronto.ca/bitstream/1807/92573/3/
[139] A. Morimitsu, I. Hirota, S. Yokogawa, I. Ohdaira, M. Matsumura, Galaom_Ahmed_201611_MAS_thesis.pdf.
H. Takahashi, T. Yamazaki, H. Oyaizu, Y. Incesu, M. Atif, and [156] B. T. Faez and B. M. Ridha, “Autofocus camera using MEMS
Y. Nitta, A 4M pixel full-PDAF CMOS image sensor with 1.58 μm actuation of image sensor,” WO2017106955, 2017.
2×1 on-chip micro-split-lens technology, 2015. Available at: [157] H.-Y. Sung, P.-C. Chen, C.-C. Chang, C.-W. Chang, S. Yang, and
Imagesensors.org. H. Chang, Mobile Phone Imaging Module with Extended Depth
[140] M. Kobayashi, M. Ohmura, H. Takashi, T. Shirai, K. Sakurai, of Focus Based on Axial Irradiance Equalization Phase Coding,
T. Ichikawa, H. Yuzurihara, and S. Inoue, “High-definition 2011.
and high-sensitivity CMOS image sensor with all-pixel image [158] E. R. Dowski and W. T. Cathey, “Extended depth of field through
plane phase-difference detection autofocus,” Jpn. J. Appl. wave-front coding,” Appl. Opt., vol. 34, pp. 1859–1866, 1995.
Phys., vol. 57, no. 10, 2018. https://doi.org/10.7567/jjap.57. [159] B. Golik and D. Wueller, “Measurement method for image
1002b5. stabilizing systems,” Proc. SPIE, Digital Photography III, vol.
[141] G. Chataignier, B. Vandame, and J. Vaillant, “Joint 6502, p. 65020O, 2007.
electromagnetic and ray-tracing simulations for quad-pixel [160] O. Šindelář and F. Šroubek, “Image deblurring in smartphone
sensor and computational imaging,” Opt. Express, vol. 27, devices using built-in inertial measurement sensors,”
pp. 30486–30501, 2019. J. Electron. Imag., vol. 22, no. 1, 2013, Art no. 011003.
[142] L. W. Alvarez and W. E. Humphrey, “Variable power lens and [161] K. T. Wyne, “A comprehensive review of tremor,” JAAPA, vol. 18,
system,” US Patent 3 507 565, 1970. no. 12, pp. 43–50, 2005, quiz 57-8.
[143] A. W. Lohmann, “A new class of varifocal lenses,” Appl. Opt., [162] E. Simon, “Optical image stabilization for miniature cameras,
vol. 9, no. 7, pp. 1669–1671, 1970. chapter 7,” in Smart Mini-Cameras, T.V. Galstian, Ed., Boca
[144] G. Zhou, H. Yu, and F. S. Chau, “Microelectromechanically- Raton, FL, CRC Press, 2014.
driven miniature adaptive Alvarez lens,” Opt. Exp., vol. 21, no. 1, [163] C. Acar and A. Shkel, MEMS Vibratory Gyroscopes. Berlin,
pp. 1226–1233, 2012. Springer, 2009.
[145] B. Berge, “Liquid lens,” in Chapter 5 in Smart Mini-Cameras, T. [164] S. Sinha, S. Shakya, R. Mukhiya, R. Gopal, and B. Pant, Design
V. Galstian, Ed., Boca Raton, FL, CRC Press, 2014. and Simulation of MEMS Differential Capacitive Accelerometer,
[146] T. V. Galstian, “Electrically variable liquid crystal lenses, 2014, https://doi.org/10.13140/2.1.1074.8809.
chapter 6,” in Smart Mini-Cameras, T.V. Galstian, Ed., Boca [165] F. Xiao, A. Silverstein, and J. Farrel, “Camera-motion and
Raton, FL, CRC Press, 2014. effective spatial resolution,” Proc. SPIE, vol. 7241, pp. 33–36,
[147] T. V. Galstian, O. Sova, K. Asatryan, V. Presniakov, 2006.
A. Zohrabyan, and M. Evensen, “Optical camera with liquid [166] F. Xiao, J. E. Farrell, P. B. Catrysse, and B. Wandell, “Mobile
crystal autofocus lens,” Opt. Express, vol. 25, imaging: the big challenge of the small pixel,” Proc. SPIE,
pp. 29945–29964, 2017. Digital Photography V, vol. 7250, p. 72500K, 2009.
[148] V. Blahnik, About the Irradiance and Apertures of Camera [167] C.-W. Chiu, P. C. P. Chao, N.-Y. Y. Kao, and F.-K. Young,
Lenses, 2014. Available at: http://www.zeiss.com/content/ “Optimal design and experimental verification of
dam/Photography/new/pdf/en/cln_archiv/cln_en_web_ magnetically actuated optical image stabilization system for
special_irradiance_and_apertures.pdf. cameras in mobile phones,” J. Appl. Phys., vol. 103, 2008, Art
[149] C.-L. Hsieh, C.-S. Liu, and C.-C. Cheng, Design of a 5 Degree of no. 07F136.
Freedom-Voice Coil Motor Actuator for Smartphone Camera [168] F. La Rosa, M. C. Virzì, F. Bonaccorso, and M. Branciforte,
Modules, Sensors and Actuators: Physical, New York, Elsevier, Optical Image Stabilization, ST Microelectronics,
2020. White Paper, 2017. Available at: https://www.st.com/
[150] L. Lu, “Voice coil motors for mobile applications. Chapter 3,” in content/ccc/resource/technical/document/white_paper/
Smart Mini-Cameras, T.V. Galstian, Ed., Boca Raton, FL, CRC c9/a6/fd/e4/e6/4e/48/60/ois_white_paper.pdf/files/ois_
Press, 2014. white_paper.pdf/jcr:content/translations/en.ois_white_
[151] H. Wu, C. Chou, and S. Shyiu, “Voice coil driving auto-focus lens paper.pdf.
module,” US20097612957, 2009. [169] V. Blahnik and B. Voelker, About the Reduction of Reflections in
[152] Y. Chou, “Camera module with piezoelectric actuator,” Camera Lenses. How T*-coating Made Glass Invisible, 2016.
US20100271541Al, 2009. Available at: https://pixinfo.com/wp-content/uploads/2016/04/
[153] M. Murphy, M. Conway, and G. Casey, “Lens drivers focus on en_About-the-reduction-of-reflections-of-camera-lenses.pdf.
performance in high-resolution camera modules,” Analog [170] M. Steinbach, “Development of a method to measure the
Dialogue, pp. 40–11, 2006. Veiling Glare contributions of the lens and the sensor and its
[154] C.-L. Hsieh and C.-S. Liu (2020). Design of a voice coil motor influence on the limit of the dynamic range when shooting
actuator with L-shape coils for optical zooming smartphone movies,” Bachelor Thesis, Cologne University of Applied
cameras. IEEE Access, vol. 8, pp. 20884–20891. Sciences, 2015.
 V. Blahnik and O. Schindelbeck: Smartphone imaging 231

[171] P. E. Debevec and J. Malik, “Recovering high dynamic range [192] D. Wueller, U. Artmann, V. Rao, G. Reif, J. Kramer, and F. Knauf,
radiance maps from photographs,” in Siggraph Conference, “VCX: an industry initiative to create an objective camera
July 1997, 1997. module evaluation for mobile devices,” Electronic Imaging, vol.
[172] S. Mann and R. W. Picard, “On being ´undigital´ with digital 2018, no. 5, pp. 1–5, 2018.
cameras: extending dynamic range by combining differently [193] A. Duane, “Studies in monocular and binocular accommodation
exposed pictures,” Proc. IST, pp. 422–428, 1995. with their clinical applications,” Am. J. Ophthalmol. Ser., vol. 3
[173] A. Adams, The Print. New Edition of 1950 from Little, Brown and 5, pp. 865–877, 1922.
Company. New York, Boston, Little, Brown and Company, 1980. [194] W. B. Wetherell, “Afocal lenses,” in Applied Optics and Optical
[174] A. Darmont, High Dynamic Range Imaging – Sensors and Engineering, vol. X, R.R. Shannon, and J.C. Wyant, Eds.,
Architectures, Bellingham, Washington USA, SPIE Press, 2012. Orlando, FL, Academic Press, 1987, pp. 109–192 also: Afocal
[175] J. J. McCann and A. Rizzi, The Art and Science of HDR Imaging, systems, in Handbook of Optics, ed. M. Bass, Vol. 1, 2.1–2.23,
New York, Wiley, 2012. McGraw-Hill, New York (1995).
[176] E. Reinhard, E. Ward, S. Pattanaik, P. Debevec, W. Heidrich, and [195] V. Blahnik and O. Schindelbeck, “Optical lens system for a front
K. Myszkowski, High Dynamic Range Imaging, New York, lens in front of a camera module of an electrical terminal,”
Elsevier, Morgan Kaufmann, 2010. US20180210173A1, 2018.
[177] R. H. Abd El-Maksoud, and J. Sasian, “Modeling and analyzing [196] V. Blahnik and H. Mehnert, “Optical lens system for a wide-
ghost images for incoherent optical systems,” Appl. Opt., vol. angle afocal optic in front of a camera module,”
50, no. 15, pp. 2305–2315, 2011. DE102016004455A1, 2017.
[178] E. Fest, Straylight Analysis and Control, Bellingham, SPIE Press, [197] V. Blahnik and D. Gängler, “Optical lens system for afocal tele-
2013. optical attachment in front of a camera module,”
[179] J. Jur, Straylight Analysis of a Mobile Phone Camera, 2016. DE102016004454A1, 2017.
Available at: https://www.slideshare.net/JordanJur/stray- [198] V. Blahnik, N. Lippok, and T. Steinich, “Optical lens system for a
light-analysis-of-a-mobile-phone-camerarevd. macro attachment in front of a camera module,”
[180] K. Venkataraman, D. Lelescu, J. Duparré, and A. McMahon, DE102016004457A1, 2017.
“PiCam: an ultra-thin high performance monolithic camera [199] R. Diederich, R. Wartmann, H. Schadwinkel, and R. Heintzmann,
array,” in Conference: ACM Transactions on Graphics, “Using machine-learning to optimize phase contrast in a low-
Proceedings of SIGGRAPH Asia, vol. 32, 2014. cost cellphone microscope,” PLoS One, 2018. https://doi.org/
[181] Y. Furukawa and C. Hernández, “Multi-view stereo: a tutorial,” 10.1371/journal.pone.0192937.
Foundations and Trends in Computer Graphics and Vision, vol. [200] M. B. Schäfer, D. Reichert, K. Stewart, A. Herkommer,
9, 2015. https://doi.org/10.1561/0600000052. C. Reichert, and P. Pott, “Smartphone-based low-cost
[182] B. Krishnamurthy and A. Rastogi, “Refinement of depth maps microscope with monolithic focusing mechanism,” Curr. Dir.
by fusion of multiple estimates,” J. Electron. Imag., vol. 22, no. Biomed. Eng., vol. 4, no. 1, p. 267270, 2018.
1, 2013, Art no. 011002. [201] N. A. Switz, M. V. D’Ambrosio, and D. A. Fletcher, “Low-cost mobile
[183] R. Szeliski, Computer Vision: Algorithms and Applications, phone microscopy with a reversed mobile phone camera lens,”
Berlin, Springer-Verlag, 2010. PLOS ONE, vol. 9, no. 5, 2014, Art no. e95330.
[184] R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, [202] V. Lakshminarayanan, J. Zelek, and A. McBride, “Smartphone
“Light field photography with a hand-held plenoptic camera,” science,” Eye Care Med. Opt. Photonics News, vol. 26, no. 1,
Technical Report CTSR 2005-02, CTSR, 2005. pp. 44–51, 2015.
[185] C. Perwass and L. Wietzke, “Single lens 3D-camera with [203] S. Majumder and M. J. Deen, “Smartphone sensors for health
extended depth-of-field,” Proc. SPIE, vol. 8291, 2012. https:// monitoring and diagnosis,” Sensors (Basel), vol. 19, no. 9,
doi.org/10.1117/12.909882. p. 2164, 2019.
[186] M. Hansard, S. Lee, O. Choi, and R. Horaud, Time-of-Flight [204] B. Hunt, A. J. Ruiz, and B. W. Pogue, “Smartphone-based
Cameras: Principles, Methods and Applications, Berlin, imaging systems for medical applications: a critical review,”
Springer, 2012. Journal of Biomedical Optics, vol. 26, no. 4, 2021. https://doi.
[187] K. Iga, “VCSEL: born small and grown big,” Proc. SPIE, Vertical org/10.1117/1.jbo.26.4.040902.
External Cavity Surface Emitting Lasers (VECSELs) X, vol. 11263, [205] ASAP, CAD-Software. Available at: https://www.breault.com/
p. 1126302, 2020. software/about-asap.
[188] S. Foix, G. Alenyà, and C. Torras, “Lock-in Time-of-Flight (ToF) [206] CIPA, Camera and Imaging Products Association. Available at:
cameras: a survey,” IEEE Sensors Journal, vol. 11, no. 3, 2011. https://www.cipa.jp/stats/dc_e.html.
https://doi.org/10.1109/jsen.2010.2101060. [207] FRED, CAD-Software. Available at: https://photonengr.com/
[189] L. Li, “Time-of-flight camera—an introduction,” Technical white fred-software/.
paper, Texas Instruments, 2014. Available at: https://www.ti. [208] Image Engineering, Solutions to Test Image Quality. Available
com/lit/wp/sloa190b/sloa190b.pdf. at: https://www.image-engineering.de/.
[190] H. H. Nasse, Depth of field and bokeh. ZEISS Camera [209] Image Sensor World Blog. News and Discussions About Image
Lenses Technical Article, 2010. Available at: https://lenspire.zeiss. Sensors, by Vladimir Koifman. Available at: Image-sensors-
com/photo/app/uploads/2018/04/Article-Bokeh-2010-EN.pdf. blogspot.com.
[191] DXO, Test of Computational Bokeh, 2018b. Available at: [210] C. S. Liu and P. D. Lin, “A miniaturized low-power VCM actuator
https://www.dxomark.com/evaluating-computational-bokeh- for auto-focusing applications,” Opt Express, vol. 16, no. 4,
test-smartphone-portrait-modes/. p. 2533, 2008.
232 V. Blahnik and O. Schindelbeck: Smartphone imaging

[211] C. S. Liu, P. D. Lin, P. H. Lin, S. S. Ke, Y. H. Chang, and J. B. Horng, Consumer Products division and worked on developing lenses for
“Design and characterization of miniature auto-focusing voice photography, cinematography, industrial applications and later also
coil motor actuator for cell phone camera applications,” IEEE binoculars and spotting scopes. He worked also on developing digital-
Trans. Magn., vol. 45, no. 1, p. 155, 2009. optical systems and features such as 3D acquisition & imaging
[212] Trioptics, Metrology for Miniature Lenses and Cameras. systems, digital-optical co-optimization or machine vision, partly for
Available at: https://trioptics.com/products/imagemaster- consumer imaging devices in close cooperation with ZEISS partners
pro-image-quality-mtf-testing/. like Nokia and Sony. Since 2018, he has been a Senior Optical
[213] K.-P. Wilska, R. Paajanen, M. Terho, and J. Hämäläinen, “Device Systems Engineer at Optical Design & Concepts for ZEISS' Corporate
for personal communication, data collection and processing Research and Technology working on different topics, such as LIDAR
and circuit board,” Finnish patent FI942334A, 1994. systems for autonomous driving, high-precision interferometry, and
3D scanning systems.

Oliver Schindelbeck
Bionotes Carl Zeiss AG, Carl-Zeiss-Straße 22, 73447
Oberkochen, Germany
Vladan Blahnik [email protected]
Carl Zeiss AG, Carl-Zeiss-Straße 22,
Oberkochen, Germany
[email protected]

Oliver Schindelbeck earned his Dipl.-Ing. (BA) in Mechanical

Engineering from the Cooperative State University Heidenheim. In
1992 he started working at the Carl Zeiss Eyeglass Lens division as a
Vladan Blahnik studied physics and wrote his PhD thesis on partially production technology developer and planner. From 2002 to 2005 he
coherent optical imaging theory at the Technical University of assumed the role of project manager, to optimize the manufacturing
Braunschweig (Germany) and the Optical Sciences Center in Tucson, processes in the central optics production facility of Carl Zeiss AG.
Arizona (USA). He joined ZEISS in 2001 as Lead System Engineer and Since 2005 he has been based at ZEISS' Consumer Products division
Project Leader for lithography optics, a role that saw him working on as a project leader, quality assurance manager and innovation
the development of projection lenses and illumination systems for manager for digital consumer imaging systems. Today, as Senior
lithographic scanners for microchip mass production. Examples Smartphone Technology Manager, he focuses on the development of
include the NA0.93, 193nm optical system for ASML's TWINSCAN mobile imaging solutions, collaborating with ZEISS' industrial
XT:1400. In 2008, he became Head of Optical Design at ZEISS' partners who include Sony and vivo.