Real-time aerial multispectral imaging solutions

using dichroic filter arrays

Eric V. Chandler, David E. Fish

Pixelteq, Inc., 16080 Table Mountain Parkway, Golden, CO USA 80404

ABSTRACT

The next generation of multispectral sensors and cameras needs to deliver significant improvements in size, weight,
portability, and spectral band customization to support widespread commercial deployment for a variety of purpose-
built aerial, unmanned, and scientific applications.

The benefits of multispectral imaging are well established for applications including machine vision, biomedical,
authentication, and remote sensing environments – but many aerial and OEM solutions require more compact,
robust, and cost-effective production cameras to realize these benefits. A novel implementation uses micro-
patterning of dichroic filters into Bayer and custom mosaics, enabling true real-time multispectral imaging with
simultaneous multi-band image acquisition. Consistent with color camera image processing, individual spectral
channels are de-mosaiced with each channel providing an image of the field of view. We demonstrate recent results
of 4-9 band dichroic filter arrays in multispectral cameras using a variety of sensors including linear, area, silicon,
and InGaAs. Specific implementations range from hybrid RGB + NIR sensors to custom sensors with application-
specific VIS, NIR, and SWIR spectral bands. Benefits and tradeoffs of multispectral sensors using dichroic filter
arrays are compared with alternative approaches – including their passivity, spectral range, customization options,
and development path. Finally, we report on the wafer-level fabrication of dichroic filter arrays on imaging sensors
for scalable production of multispectral sensors and cameras.

CURRENT HYPERSPECTRAL & MULTISPECTRAL APPROACHES

The objective with spectral imaging is not only to acquire the spatial information of an object or scene, but also to
extract the spectral information associated with each particular spatial location within the frame. By definition, a
monochrome sensor – commonly employed in many scientific grade imaging systems – cannot distinguish between
incoming light of different colors. Obtaining this spectral information requires dispersing the light across the sensor,
sacrificing one axis of spatial information

(hyperspectral imaging), or pre-filtering the light prior to reaching the detector and acquiring a series of spectrally-
selective images (multispectral imaging).

In a way, the spectral information can be considered a third data “dimension,” similar to physical depth. With a
single two-dimensional detector, gathering this information requires projecting acquisition into a third dimension –
time. In hyperspectral imaging, one spatial axis and the spectral information are acquired simultaneously, and the
second spatial axis is obtained by acquiring sequential images. In multispectral imaging, both spatial axes are
acquired simultaneously, while the spectral information is sequentially acquired over time.

Each imaging methodology has drawbacks and tradeoffs, the relevance of which varies with application.
Hyperspectral imaging can provide substantial spectral information and uninterrupted spatial information with
sufficient image acquisition speed. Significant back-end computing is required to process, store, and transmit the
sheer volume of information obtained from a hyperspectral system – essentially a full spectrum for each spatial

Airborne Intelligence, Surveillance, Reconnaissance (ISR) Systems and Applications XI, edited by Daniel J. Henry,
Davis A. Lange, Dale Linne von Berg, S. Danny Rajan, Thomas J. Walls, Darrell L. Young, Proc. of SPIE
Vol. 9076, 90760L · © 2014 SPIE · CCC code: 0277-786X/14/$18 · doi: 10.1117/12.2053493

Proc. of SPIE Vol. 9076 90760L-1

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/16/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

 location. Situations where power consumption or weight is a concern, such as aerial observation, will limit the use of
hyperspectral imaging.

Alternately, multispectral imaging performed via the aforementioned method cannot produce temporally
simultaneous images – in moving applications, this will require post-processing to ensure proper spatial overlap,
reducing the amount of useable data. Further, the spectral window of each image is determined by the properties of
the pre-filter, so the full spectrum for each spatial location is not available.

However, there are a sizeable number of applications wherein the object of interest is stationary and the spectral
bands are pre-determined. In these situations, multispectral imaging can provide excellent spectral contrast with
high-spatial resolution images. The case of subcutaneous bruising, shown in Figure 1, is not discernible in the
visible spectrum (left). In the near-infrared (NIR), both the presence and spatial extent of the bruise is immediately
apparent (right) [1,2]. This information is of significant value to medical personnel seeking non-invasive and rapid
methods of screening patients.

Figure 1: A subcutaneous bruise imaged at high spatial resolution in the visible (left) and NIR (right).

Though hyperspectral and multispectral imaging systems are highly capable, neither lends itself to facile and rapid
deployment, often requiring a more significant degree of integration into the end device. Further, both methods
require an active component, be it spatial or spectral scanning.

COLOR FILTER ARRAYS

By carefully choosing how spatial and spectral information are acquired and parsed, passivity and speed-of-
acquisition concerns can be allayed. Modern red-green-blue cameras are a step toward high-speed multispectral
imaging, in that a repeating (Bayer) pattern of absorbing dyes is deposited onto the image sensor [3]. In this manner,
spatial information is traded for spectral information, but the device remains passive, and the “multispectral” image
acquisition rate is not decreased. Further, the different spectral bands observe the same spatial scene, though
“missing” pixels for each spectral band are computationally interpolated.

While the spectral and spatial components of this kind of device are passive, such an approach still falls short of true
multispectral imaging. The quantity of absorbing dyes is extremely limited – typically to red, blue, green, cyan,
magenta, and yellow. The transmission bands are quite broad, and out-of-band blocking is comparatively poor –
leading to significant cross-talk and relatively weak signal-to-noise ratio.

Proc. of SPIE Vol. 9076 90760L-2

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/16/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

 DICHROIC FILTER ARRAYS

Broadening this imaging methodology to true multispectral imaging requires a novel approach. Custom dichroic
filters are deposited into an array matched to the image sensor via semiconductor-style lithographic techniques.
These arrays can be directly deposited onto the sensor itself at the wafer level, or applied to a “cover” glass that is
then bonded to the sensor [4]. Figure 2 is an example of a Bayer pattern of short-wave infra-red (SWIR) filters
applied to an InGaAs image sensor via glass bonding (false color applied). Figure 3 displays the transmission bands
of the three SWIR filters.

22 μm

22 μm
.
.
3 μm

Figure 2: A Bayer pattern of SWIR dielectric filters with false coloring applied –
in this case 22μm square pixels with a 3μm gap between.

SWIR Bayer Pattern Filter - Transmittance

100%
Transmission [%]

80%
60%
40%
20%
0%
1000 1100 1200 1300 1400 1500 1600 1700 1800
Wavelength [nm]

Figure 3: A short-wave pass, band-pass, and long-wave pass filter were applied in the SWIR filter array.

The advantages of a process of this kind are three-fold. First, multispectral imaging at arbitrary wavebands is now
possible, limited only by the typical constraints of dielectric filtering. Secondly, the filter pattern is not limited to the
traditional Bayer arrangement – the filters can be deposited in any repeating pattern across the sensor. Lastly, the
pattern can be matched to many varieties of sensors, for use in a wide range of cameras and sensor material types,
such as silicon or InGaAs. Linear and two-dimensional array sensors are all candidates for array filtering.

Proc. of SPIE Vol. 9076 90760L-3

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/16/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

 The spectral range of this patterned approach is defined by the range of the base sensor – dielectric filtering is
possible from the ultraviolet (UV) to the SWIR. Further, multiple filter types are possible – short-wave pass, long-
wave pass, and bandpass filtering are all tractable. The properties of the filtering are limited only by the required
feature sizes.

Beyond the exchange of direct spatial information for spectral simultaneity, a secondary restriction of this approach
is the spectral rigidity of the final device. Unlike tunable multispectral systems, once the filter array has been applied
to the sensor – either directly or by bonding glass – the spectral properties cannot be altered. As such, the dichroic
filter array approach finds the greatest utility as an end-device in specific applications where the spectral bands are
well established.

Of the discussed multispectral imaging methods, dichroic array filtering serves best in applications where devices
are required in volume. Multiple wafers can be processed simultaneously, producing spectrally selective sensors in
quantity. An additional benefit of direct wafer processing is that light is coupled more efficiently into the device, due
to the reduced index mismatch between the sensor material and the external environment.

APPLICATIONS

As previously stated, dichroic filter patterning yields a compact, lightweight, passive imaging device – ideal for
aerial imaging. A SWIR patterned sensor following a Bayer pattern, as in Figure 2, was recently flight-tested by
Advanced Coherent of San Diego, CA, USA. The sensor had a resolution of 640 x 512 with a pixel pitch of 25
microns [5]. An RGB camera (Allied Vision GT6600C) was employed to acquire visible images to correspond
spatially with the data obtained with the SWIR camera. The RGB camera possessed a 29MP TrueSense KAI-29050
sensor, at 6600 x 4400 resolution with 5.5 micron pixel pitch. Both cameras were nadir positioned, or aligned
normal to the Earth’s surface. Images were acquired along the flight path, and were stitched together in real-time by
a FluxData Intelligent SWIR hardware platform.

An example of a single flight pass is shown in Figure 4, wherein a series of 20 snapshot images acquired from the
patterned SWIR sensor have been stitched together to form a continuous frame. The images were acquired with the
flight path beginning at the right-hand side of the composite image. The length of the flight path was approximately
4 kilometers. The plane hosting the imaging platforms traveled at a speed of 80 knots, and at an altitude of 1500
feet. The PT GUI software was used to generate a continuous map of the flight range via onboard, real-time
processing. This software methodology provides the capability of instant object-of-interest designation via false-
coloring.

Figure 4: The contiguous image of a residential area, generated in a single flight-path, with the Bayer-patterned SWIR sensor.

The composite RGB image of the residential area is shown in Figure 5. A number of different terrain and material
types were captured by both the SWIR and RGB cameras, to explore how vegetation, soil, and artificial materials
respond differently within the wavebands of the SWIR sensor.

Proc. of SPIE Vol. 9076 90760L-4

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/16/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

 29 MP RGB (Allied Vision)

Figure 5: A composite RGB image, taken in-flight, of residential open space.

Figure 6 is an overlay of the SWIR data with the RGB images. It is readily apparent that the artificial turf of the
athletic fields possesses a very different spectral signature than the surrounding vegetation. Further, the replanted
vegetation in front of the buildings to the northeast of the playing fields also registers as synthetic. Indeed, SWIR
systems are often employed for materials evaluation – such as plastics sorting in recycling plants. Additionally, the
larger trees exhibit spectral differences from the surrounding vegetation, though the cause was not evaluated.
29 MP RGB (Allied Vision)

Figure 6: A composite RGB+SWIR image, taken in-flight, of residential open space.

The artificial turf stands in contrast to the surrounding natural vegetation.

Proc. of SPIE Vol. 9076 90760L-5

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/16/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

 SUMMARY

Dichroic filter arrays applied to monochrome imaging sensors expand the reach of real-time multispectral imaging
into previously inaccessible application spaces. Lightweight, compact, and passive, this technology naturally lends
itself to aerial and remotely deployed platforms. The scalable processing enables sensors to be fabricated in volume.
Advances in processing technology allow for real-time analysis and image synthesis. Between the small size and the
developing image processing techniques, the contrast improvement obtained in the SWIR can be paired with visible
cameras. The resultant display is intuitive for the end user, and can be customized to the application.

With the available spectral range spanning the UV through the SWIR, possible uses include aerial sensing for
agricultural, real-time authentication or material detection, forensic analysis, and live security screening. Especially
powerful is the capability to instantly discriminate between organic and inorganic materials, regardless of visible
appearance. This has implications both for live, real-time security screening – so as to defeat attempts at disguise –
and also for food processing, so as to identify and segregate out potential contaminants.

Dichroic filter arrays are uniquely capable to deliver real-time multispectral imaging in virtually the same footprint
as corresponding monochrome snapshot acquisition cameras. Wafer level processing supports a scalable production
process - providing cost-effective solutions for application-specific sensors and cameras. This combination of
compact real-time imaging + scalable volume costs is making a wide variety of aerial and remote multispectral
applications feasible technically and commercially.

REFERENCES

[1] Eichenholz, J., Barnett, N., Juang, Y., Fish, D.E., Spano, S., Lindsley, E., Farkas, D., “Real time Megapixel
Multispectral Bioimaging,” Appl. Note, www.pixelteq.com/resources

[2] Wang, F., Behrooz, A., Morris, M., Adibi, A., “High-contrast subcutaneous vein detection and localization using
multispectral imaging,” J. Biomedical Optics, 18(5), 050504 1-3, 2013.

[3] FujiFilm Electronic Materials Co., “Color Mosaic for Image Sensor”,
http://www.fujifilmusa.com/products/semiconductor_materials/image-sensor-color-mosaic/cmy/index.html

[4] Buchsbaum, P., Morris, M., “Method for Making Monolithic Patterned Dichroic Filter Detector Arrays for
Spectroscopic Imaging”, US. Patent No. 6,638,668, 2001.

[5] Dougherty, J., Jennings, T., Snikkers, M., “Compact camera technologies for real-time false-color imaging in the
SWIR band”, Proc. SPIE 8899, Emerging Technologies in Security and Defense; and Quantum Security II; and
Unmanned Sensor Systems

Proc. of SPIE Vol. 9076 90760L-6

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/16/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx