Image Processing Using FPGAs Donald G Bailey Editor download
Image Processing Using FPGAs Donald G Bailey Editor download
Editor download
https://ebookmeta.com/product/image-processing-using-fpgas-
donald-g-bailey-editor/
https://ebookmeta.com/product/design-for-embedded-image-
processing-on-fpgas-2nd-edition-donald-g-bailey/
https://ebookmeta.com/product/digital-image-processing-using-
matlab-4th-edition-rafael-c-gonzalez/
https://ebookmeta.com/product/embedded-microprocessor-system-
design-using-fpgas-uwe-meyer-baese/
https://ebookmeta.com/product/the-fairness-doctrine-and-the-
media-steven-j-ssmmons/
NSW Civil Procedure Handbook 2017 Carol Webster
https://ebookmeta.com/product/nsw-civil-procedure-
handbook-2017-carol-webster/
https://ebookmeta.com/product/sin-and-soil-12-1st-edition-anya-
merchant/
https://ebookmeta.com/product/international-negotiation-and-
mediation-in-violent-conflict-the-changing-context-of-
peacemaking-1st-edition-chester-a-crocker/
https://ebookmeta.com/product/atmosphere-architecture-cinema-
thematic-reflections-on-ambiance-and-place-1st-edition-michael-
tawa/
https://ebookmeta.com/product/unproduction-studies-and-the-
american-film-industry-1st-edition-james-fenwick/
Starting a Business All In One For Dummies The Experts
At Dummies
https://ebookmeta.com/product/starting-a-business-all-in-one-for-
dummies-the-experts-at-dummies/
Journal of
Imaging
Image Processing
Using FPGAs
Edited by
Donald G. Bailey
Printed Edition of the Special Issue Published in Journal of Imaging
www.mdpi.com/journal/jimaging
Image Processing Using FPGAs
Image Processing Using FPGAs
Editorial Office
MDPI
St. Alban-Anlage 66
4052 Basel, Switzerland
This is a reprint of articles from the Special Issue published online in the open access journal
Journal of Imaging (ISSN 2313-433X) from 2018 to 2019 (available at: https://www.mdpi.com/
journal/jimaging/special issues/Image FPGAs).
For citation purposes, cite each article independently as indicated on the article page online and as
indicated below:
LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. Journal Name Year, Article Number,
Page Range.
c 2019 by the authors. Articles in this book are Open Access and distributed under the Creative
Commons Attribution (CC BY) license, which allows users to download, copy and build upon
published articles, as long as the author and publisher are properly credited, which ensures maximum
dissemination and a wider impact of our publications.
The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons
license CC BY-NC-ND.
Contents
Donald Bailey
Image Processing Using FPGAs
Reprinted from: Journal of Imaging 2019, 5, 53, doi:10.3390/jimaging5050053 . . . . . . . . . . . . 1
Fahad Siddiqui, Sam Amiri, Umar Ibrahim Minhas, Tiantai Deng, Roger Woods,
Karen Rafferty and Daniel Crookes
FPGA-Based Processor Acceleration for Image Processing Applications
Reprinted from: Journal of Imaging 2019, 5, 16, doi:10.3390/jimaging5010016 . . . . . . . . . . . . 5
Paulo Garcia, Deepayan Bhowmik, Robert Stewart, Greg Michaelson and Andrew Wallace
Optimized Memory Allocation and Power Minimization for FPGA-Based Image Processing
Reprinted from: Journal of Imaging 2019, 5, 7, doi:10.3390/jimaging5010007 . . . . . . . . . . . . . 27
Andrew Tzer-Yeu Chen, Rohaan Gupta, Anton Borzenko, Kevin I-Kai Wang and
Morteza Biglari-Abhari
Accelerating SuperBE with Hardware/Software Co-Design
Reprinted from: Journal of Imaging 2018, 4, 122, doi:10.3390/jimaging4100122 . . . . . . . . . . . . 91
Zhe Wang, Trung-Hieu Tran, Ponnanna Kelettira Muthappa and Sven Simon
A JND-Based Pixel-Domain Algorithm and Hardware Architecture for Perceptual Image
Coding
Reprinted from: Journal of Imaging 2019, 5, 50, doi:10.3390/jimaging5050050 . . . . . . . . . . . . 164
v
About the Special Issue Editor
Donald G. Bailey received his Bachelor of Engineering (Honours) degree in Electrical Engineering
in 1982, and Ph.D. degree in Electrical and Electronic Engineering from the University of Canterbury,
New Zealand, in 1985. From 1985 to 1987, he applied image analysis to the wool and paper industries
of New Zealand. From 1987 to 1989, he was a Visiting Research Engineer at University of California,
Santa Barbara. Prof. Bailey joined Massey University in Palmerston North, New Zealand, as Director
of the Image Analysis Unit in November 1989. He was a Visiting Researcher at the University of
Wales, Cardiff, in 1996; University of California, Santa Barbara, in 2001–2002; and Imperial College
London in 2008. He is currently Professor of Imaging Systems in the Department of Mechanical
and Electrical Engineering in the School of Food and Advanced Technology at Massey University,
where he is Leader of the Centre for Research in Image and Signal Processing. Prof. Bailey has spent
over 35 years applying image processing to a range of industrial, machine vision, and robot vision
applications. For the last 18 years, one area of particular focus has been exploring different aspects
of using FPGAs for implementing and accelerating image processing algorithms. He is the author
of many publications in this field, including the book “Design for Embedded Image Processing on
FPGAs”, published by Wiley/IEEE Press. He is a Senior Member of the IEEE, and is active in the
New Zealand Central Section.
vii
Preface to ”Image Processing Using FPGAs”
Over the last 20 years, FPGAs have moved from glue logic through to computing platforms.
They effectively provide a reconfigurable hardware platform for implementing logic and algorithms.
Being fine-grained hardware, FPGAs are able to exploit the parallelism inherent within a hardware
design while at the same time maintaining the reconfigurability and programmability of software.
This has led to FPGAs being used as a platform for accelerating computationally intensive tasks. This
is particularly seen in the field of image processing, where the FPGA-based acceleration of imaging
algorithms has become mainstream. This is even more so within an embedded environment, where
the power and computational resources of conventional processors are not up to the task of managing
the data throughput and computational requirements of real-time imaging applications.
Unfortunately, the fine-grained nature of FPGAs also makes them difficult to programme
effectively. Conventional processors have a fixed computational architecture, which is able to provide
a high level of abstraction. By contrast, on an FPGA, it is necessary to design not only the algorithm
but also the computational architecture, which leads to an explosion in the design space complexity.
This, coupled with the complexities of managing the concurrency of a highly parallel design and the
bandwidth issues associated with the high volume of data associated with images and video, has
led to a wide range of approaches and architectures used for realising FPGA-based image processing
systems. This Special Issue provides an opportunity for researchers in this area to present some of
their latest results and designs. The diversity of presented techniques and applications reflects the
nature and current state of FPGA-based design for image processing.
Donald G. Bailey
Special Issue Editor
ix
Journal of
Imaging
Editorial
Image Processing Using FPGAs
Donald G. Bailey
Department of Mechanical and Electrical Engineering, School of Food and Advanced Technology,
Massey University, Palmerston North 4442, New Zealand; [email protected]
Abstract: Nine articles have been published in this Special Issue on image processing using
field programmable gate arrays (FPGAs). The papers address a diverse range of topics relating
to the application of FPGA technology to accelerate image processing tasks. The range includes:
Custom processor design to reduce the programming burden; memory management for full frames,
line buffers, and image border management; image segmentation through background modelling,
online K-means clustering, and generalised Laplacian of Gaussian filtering; connected components
analysis; and visually lossless image compression.
Keywords: field programmable gate arrays (FPGA); image processing; hardware/software co-design;
memory management; segmentation; image analysis; compression
2. Contributions
Programming an FPGA to accelerate complex algorithms is difficult, with one of four approaches
commonly used [1]:
• Custom hardware design of the algorithm using a hardware description language, optimised for
performance and resources;
• implementing the algorithm by instantiating a set of application-specific intellectual property
cores (from a library);
• using high-level synthesis to convert a C-based representation of the algorithm to
synthesisable hardware; or
• mapping the algorithm onto a parallel set of programmable soft-core processors.
The article by Siddiqui et al. [1] took this last approach, and describes the design of an efficient
16-bit integer soft-core processor, IPPro, capable of operating at 337 MHz, specifically targetting the
dataflow seen in complex image processing algorithms. The presented architecture uses dedicated
stream access instructions on the input and output, with a 32-element local memory for storing pixels
and intermediate results, and a separate 32-element kernel memory for storing filter coefficients
and other parameters and constants. The exploitation of both data-level parallelism and task-level
parallelism is demonstrated through the mapping of a K-means clustering algorithm onto the
architecture, showing good scalability of processing speed with multiple cores. A second case study of
traffic sign recognition is partitioned between the IPPro cores and an ARM processor, with the colour
conversion and morphological filtering stages mapped to the IPPro. Again, the use of parallel IPPro
cores can significantly accelerate these tasks, compared to conventional software, without having to
resort to the tedious effort of custom hardware design.
Garcia et al. [2] worked on the thesis that the image processing operations which require random
access to the whole frame (including iterative algorithms) are particularly difficult to realise in FPGAs.
They investigate the mapping of a frame buffer onto the memory resources of an FPGA, and explore
the optimal mapping onto combinations of configurable on-chip memory blocks. They demonstrate
that, for many image sizes, the default mapping by the synthesis tools results in poor utilisation, and is
also inefficient in terms of power requirements. A procedure is described that determines the best
memory configuration, based on balancing resource utilisation and power requirements. The mapping
scheme is demonstrated with optical flow and mean shift tracking algorithms.
On the other hand, local operations (such as filters) only need part of the image to produce an
output, and operate efficiently in stream processing mode, using line buffers to cache data for scanning
a local window through the image. This works well when the image size is fixed, and is known in
advance. Two situations where this approach is less effective [3] are in the region of interest processing,
where only a small region of the image is processed (usually determined from the image contents at
run-time), and cloud processing of user-uploaded images (which may be of arbitrary size). This is
complicated further in high-speed systems, where the real-time requirements demand processing
multiple pixels in every clock cycle, because, if the line width is not a multiple of the number of pixels
processed each cycle, then it is necessary to assemble the output window pixels from more than one
memory block. Shi et al. [3], in their paper, extend their earlier work on assembling the output window
to allow arbitrary image widths. The resulting line buffer must be configurable at run-time, which is
achieved through a series of “instructions”, which control the assembly of the output processing
window when the required data spans two memory blocks. Re-configuration only takes a few clock
cycles (to load the instructions), rather than conventional approach of reconfiguring the FPGA each
time the image width changes. The results demonstrate better resource utilisation, higher throughput,
and lower power than their earlier approach.
When applying window operations to an image, the size of the output image is smaller than
the input because data is not valid when the window extends beyond the image border. If necessary,
this may be mitigated by extending the input image to provide data to allow such border pixels to be
calculated. Prior work only considered border management using direct form filter structures, because
the window formation and filter function can be kept independent. However, in some applications,
transpose-form filter structures are desirable because the corresponding filter function is automatically
pipelined, leading to fewer resources and faster clock frequencies. Bailey and Ambikumar [4] provide
a design methodology for border management using transpose filter structures, and show that the
resource requirements are similar to those for direct-form border management.
An important task in computer vision is segmenting objects from a complex background. While
there are many background modelling algorithms, the complexity of robust algorithms make them
difficult to realise on an FPGA, especially for larger image sizes. Chen et al. [5] address scalability issues
with increasing image size by using super-pixels—small blocks of adjacent pixels that are treated as a
single unit. As each super-pixel is considered to be either object or background, this means that fewer
2
J. Imaging 2019, 5, 53
models need to be maintained (less memory) and fewer elements need to be classified (reduced
computation time). Using hardware/software co-design, they accelerated the computationally
expensive steps of Gaussian filtering and calculating the mean and variance within each super-pixel
with hardware, with the rest of the algorithm being realised on the on-chip CPU. The resulting system
gave close to state-of-the-art classification accuracy.
A related paper, by Badawi and Bilal [6], used K-means clustering to segment objects within video
sequences. Rather than taking the conventional iterative approach to K-means clustering, they rely
on the temporal coherence of video streams and use the cluster centres from the previous frame as
initialisation for the current frame. Additionally, rather than waiting until the complete frame has
been accumulated before updating the cluster centres, an online algorithm is used, with the clusters
updated for each pixel. To reduce the computational requirements, the centres are updated using a
weighted average. They demonstrate that, for typical video streams, this gives similar performance to
conventional K-means algorithms, but with far less computation and power.
In another segmentation paper, Zhou et al. [7] describe the use of a generalised Laplacian of
Gaussian (LoG) filter for detecting cell nuclei for a histopathology application. The LoG filters detect
elliptical blobs at a range of scales and orientations. Local maxima of the responses are used as
candidate seeds for cell centres, and mean-shift clustering is used to combine multiple detections
from different scales and orientations. Their FPGA design gave modest acceleration over a software
implementation on a high-end computer.
Given a segmented image, a common task is to measure feature vectors of each connected
component for analysis. Bailey and Klaiber [8] present a new single-pass connected components
analysis algorithm, which does this with minimum latency and relatively few resources. The key novelty
of this paper is the use of a zig-zag based scan, rather than a conventional raster scan. This eliminates the
end-of-row processing for label resolution by integrating it directly within the reverse scan. The result is
true single-pixel-per-clock-cycle processing, with no overheads at the end of each row or frame.
An important real-time application of image processing is embedded online image compression
for reducing the data bandwidth for image transmission. In the final paper within this Special Issue,
Wang et al. [9] defined a new image compression codec which works efficiently with a streamed image,
and minimises the perceptual distortion within the reconstructed images. Through small local filters,
each pixel is classified as either an edge, a smooth region, or a textured region. These relate to a
perceptual model of contrast masking, allowing just noticeable distortion (JND) thresholds to be
defined. The image is compressed by downsampling; however, if the error in any of the contributing
pixels exceeds the visibility thresholds, the 2 × 2 block is considered a region of interest, with the
4 pixels coded separately. In both cases, the pixel values are predicted using a 2-dimensional predictor,
and the prediction residuals are quantised and entropy-encoded. Results typically give a visually
lossless 4:1 compression, which is significantly better than other visually lossless codecs.
3. Conclusions
Overall, this collection of papers reflects the diversity of approaches taken to applying FPGAs to
image processing applications. From one end, using the programmable logic to design lightweight
custom processors to enable parallelism, through overcoming some of the limitations of current
high-level synthesis tools, to the other end with the design of custom hardware designs at the
register-transfer level.
The range of image processing techniques include filtering, segmentation, clustering, and
compression. Applications include traffic sign recognition for autonomous driving, histopathology,
and video compression.
3
J. Imaging 2019, 5, 53
References
1. Siddiqui, F.; Amiri, S.; Minhas, U.I.; Deng, T.; Woods, R.; Rafferty, K.; Crookes, D. FPGA-based processor
acceleration for image processing applications. J. Imaging 2019, 5, 16. [CrossRef]
2. Garcia, P.; Bhowmik, D.; Stewart, R.; Michaelson, G.; Wallace, A. Optimized memory allocation and power
minimization for FPGA-based image processing. J. Imaging 2019, 5, 7. [CrossRef]
3. Shi, R.; Wong, J.S.; So, H.K.H. High-throughput line buffer microarchitecture for arbitrary sized streaming
image processing. J. Imaging 2019, 5, 34. [CrossRef]
4. Bailey, D.G.; Ambikumar, A.S. Border handling for 2D transpose filter structures on an FPGA. J. Imaging
2018, 4, 138. [CrossRef]
5. Chen, A.T.Y.; Gupta, R.; Borzenko, A.; Wang, K.I.K.; Biglari-Abhari, M. Accelerating SuperBE with
hardware/software co-design. J. Imaging 2018, 4, 122. [CrossRef]
6. Badawi, A.; Bilal, M. High-level synthesis of online K-Means clustering hardware for a real-time image
processing pipeline. J. Imaging 2019, 5, 38. [CrossRef]
7. Zhou, H.; Machupalli, R.; Mandal, M. Efficient FPGA implementation of automatic nuclei detection in
histopathology images. J. Imaging 2019, 5, 21. [CrossRef]
8. Bailey, D.G.; Klaiber, M.J. Zig-zag based single pass connected components analysis. J. Imaging 2019, 5, 45.
[CrossRef]
9. Wang, Z.; Tran, T.H.; Muthappa, P.K.; Simon, S. A JND-based pixel-domain algorithm and hardware
architecture for perceptual image coding. J. Imaging 2019, 5, 50. [CrossRef]
c 2019 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).
4
Other documents randomly have
different content
when there were friends present at his house and all historians were
absent, he would declaim in thunderous tones—
"Would you believe it, gentlemen, there are in France, at the present
moment and of our generation and rank, historians who take it into
their heads to copy the style of the veterans, Berruyer, Catrou and
Rouille? Yes, in each line of their modern battles they will tell you
that thirty thousand men were cut in pieces, or that they bit the
dust, or that they were left lying strewn upon the scene. How behind
the times these youngsters are! The other day, one of them, in
describing the battle of Austerlitz, wrote this sentence: 'Twenty-five
thousand Russians were drawn up in battle upon a vast frozen lake;
Napoléon gave orders that firing should be directed against this lake.
Bullets broke through the ice and the twenty-five thousand Russians
BIT THE DUST!'"
THE CENTAUR
CHAPTER XI
Then God took pity on him. He did not restore his health or hope,
his youth, beauty and loves in this life; those three illusions vanished
all too soon: but God granted him the gift of tears. And he thanked
God for it. Towards the close of the year 1829, the disease made
such progress that Rabbe resolved he would not live to see the
opening of the year 1830. Thus, as he had addressed God, as he
had addressed his soul, so he now addresses death—
DEATH
"Thou diest! Thou hast reached the limit to which all things
comes at last; the end of thy miseries, the beginning of thy
happiness. Behold, death stands face to face with thee! Thou
wilt not longer be able to wish for, nor to dread it. Pains and
weakness of body, sad heart-searchings, piercing spiritual
anguish, devouring griefs, all are over! Thou wilt never suffer
them again; thou goest in peace to brave the insolent pride of
the successful evil-doer, the despising of fools and the abortive
pity of those who dare to style themselves good.
"The deprivation of many evils will not be an evil in itself; I have
seen thee chafing at thy bit, shaking the humiliating chains of
an adverse fate in despair; I have often heard the distressing
complaints which issued from the depths of thy oppressed
heart.... Thou art satisfied at last. Haste thee to empty the cup
of an unfortunate life, and perish the vase from which thou wast
compelled to drink such bitter draughts.
"But thou dost stop and tremble! Thou dost curse the duration
of thy suffering and yet dost dread and regret that the end has
come! Thou apprisest without reason or justice, and dost
lament equally both what things are and what they cease to be.
Listen, and think for one moment.
"In dying, thou dost but follow the path thy forefathers have
trodden; thousands of generations before thee have fallen into
the abyss into which thou hast to descend; many thousands will
fall into it after thee. The cruel vicissitude of life and death
cannot be altered for thee alone. Onward then towards thy
journey's end, follow where others have gone, and be not afraid
of straying from it or losing thyself when thou hast so many
other travelling companions. Let there be no signs of weakness,
no tears! The man who weeps over his own death is the vilest
and most despicable of all beings. Submit unmurmuringly to the
inevitable; thou must die, as thou hast had to live, without will
of thy own. Give back, therefore, without anxiety, thy life which
thou receivest unconsciously. Neither birth nor death are in thy
power. Rather rejoice, for thou art at the beginning of an
immortal dawn. Those who surround thy deathbed, all those
whom thou hast ever seen, of whom thou hast heard speak or
read, the small number of those thou hast known especially
well, the vast multitude of those who have lived formerly or
been born or are to be born in ages to come throughout the
world, all these have gone or will go the road thou art going.
Look with wise eyes upon the long caravan of successive
generations which have crossed the deserts of life, fighting as
they travel across the burning sands for one drop of the water
which inflames their thirst more than it appeases it! Thou art
swallowed up in the crowd directly thou fallest: but look how
many others are falling too at the same time with thee!
"Wouldst thou desire to live for ever? Wouldst thou only wish
thy life to last for a thousand years? Remember the long hours
of weariness in thy short career, thy frequent fainting under the
burden. Thou wast aghast at the limited horizon of a short,
uncertain and fugitive life: what wouldst thou have said if thou
hadst seen an immeasurable, inevitably long future of weariness
and sorrow stretch before thy eyes!
"O mortals! you weep over death, as though life were
something great and precious! And yet the vilest insects that
crawl share this rare treasure of life with you! All march towards
death because all yearn towards rest and perfect peace.
"Behold! the approach of the day that thou fain wouldst have
tried to bring nearer by thy prayers, if a jealous fate had not
deferred it; for which thou didst often sigh; behold the moment
which is to remove the capricious yoke of fortune from the
trammels of human society, from the venomous attacks of thy
fellow-creatures. Thou thinkest thou wilt cease to exist and that
thought torments thee.... Well, but what proves to thee that
thou wilt be annihilated? All the ages have retained a hope in
immortality. The belief in a spiritual life was not merely a dogma
of a few religious creeds; it was the need and the cry of all
nations that have covered the face of the earth. The European,
in the luxuries of his capital towns, the aboriginal American-
Indian under his rude huts, both equally dream of an immortal
state; all cry to the tribunal of nature against the
incompleteness of this life.
"If thou sufferest, it is well to die; if thou art happy or thinkest
thou art so, thou wilt gain by death since thy illusion would not
have lasted long. Thou passest from a terrestrial habitation to a
pure and celestial one. Why look back when thy foot is upon the
threshold of its portals? The eternal distributor of good and evil,
our Sovereign Master, calls thee to Himself; it is by His desire
thy prison flies open; thy heavy chains are broken and thy exile
is ended; therefore rejoice! Thou wilt soar to the throne of thy
King and Saviour!
"Ah! if thou art not shackled with the weight of some
unexpiated crime, thou wilt sing as thou diest; and, like the
Roman emperor, thou wilt rise up in thy agony at the very
thought, and thou wouldst die standing with eyes turned
towards the promised land!
"O Saint Preux and Werther! O Jacob Ortis! how far were you
from reaching such heights as that! Orators even to the death
agony, your brains alone it is which lament; man in his death
throes, this actually dying creature, it is his heart that groans,
his flesh that cries out, his spirit which doubts. Oh! how well
one feels that all that hollow philosophising does not reassure
him as to the pain of the supreme moment, and especially
against that terror of annihilation, which brought drops of sweat
to the brow of Hamlet!
"One more cry—the last, then silence shall fall on him who
suffered much."
Thus, poor Rabbe, after all thy philosophy, sifted as fine as ripe
grain; after all thy philosophising; after many prayers to God and
dialogues with thy soul, and many conversations with death, these
supreme interlocutors have taught thee nothing and thy last thought
is a doubt!
Rabbe had said he would not see the year 1830: and he died during
the night of the 31 December 1829.
Now, how did he die? That gloomy mystery was kept locked in the
hearts of the last friends who were present with him. But one of his
friends told me that, the evening before his death, his sufferings
were so unendurable, that the doctor ordered an opium plaster to be
put on the sick man's chest. Next day, they hunted in vain for the
opium plaster but could not find it....
On 17 September 1835, Victor Hugo addresses these lines to him:—
À ALPHONSE RABBE
CHAPTER XII