ME NU

DE E P LE ARNI NG KE RAS AND T E NSORFLOW OB JE CT DE T E CT I ON OPE NCV T UT ORI ALS T UT ORI ALS

Region proposal object detection with

OpenCV, Keras, and TensorFlow
by Adrian Rosebrock on July 6, 2020

Click here to download the source code to this post

6:40

In this tutorial, you will learn how to utilize region proposals for object detection using OpenCV, Keras,
and TensorFlow.
Today’s tutorial is part 3 in our 4-part series on deep learning and object detection:

Part 1: Turning any deep learning image classi er into an object detector with Keras and
TensorFlow

Part 2: OpenCV Selective Search for Object Detection

Part 3: Region proposal for object detection with OpenCV, Keras, and TensorFlow (today’s tutorial)

Part 4: R-CNN object detection with Keras and TensorFlow

In last week’s tutorial, we learned how to utilize Selective Search to replace the traditional computer
vision approach of using bounding boxes and sliding windows for object detection.

But the question still remains: How do we take the region proposals (i.e., regions of an image that
could contain an object of interest) and then actually classify them to obtain our nal object
detections?

We’ll be covering that exact question in this tutorial.

To learn how to perform object detection with region proposals using OpenCV, Keras, and
TensorFlow, just keep reading.
 Looking for the source code to this post?
JUMP RIGHT TO THE DOWNLOA DS SEC TION

Region proposal object detection with OpenCV, Keras, and

TensorFlow
In the rst part of this tutorial, we’ll discuss the concept of region proposals and how they can be used
in deep learning-based object detection pipelines.

We’ll then implement region proposal object detection using OpenCV, Keras, and TensorFlow.

We’ll wrap up this tutorial by reviewing our region proposal object detection results.

What are region proposals, and how can they be used for object
detection?

Figure 1: OpenCV’s Selective Search applies hierarchical similarity measures to join regions and eventually form
the nal set of region proposals for where objects could be present. (image source)

We discussed the concept of region proposals and the Selective Search algorithm in last week’s tutorial
on OpenCV Selective Search for Object Detection — I suggest you give that tutorial a read before you
continue here today, but the gist is that traditional computer vision object detection algorithms relied
on image pyramids and sliding windows to locate objects in images and varying scales and locations:
There0:13
are a few problems with the image pyramid and sliding window method, but the primary issues
are that:

1 Sliding windows/image pyramids are painfully slow

2 They are sensitive to hyperparameter choices (namely pyramid scale size, ROI size, and window
step size)

3 They are computationally ine cient

Region proposal algorithms seek to replace the traditional image pyramid and sliding window
approach.

These algorithms:

1 Accept an input image

2 Over-segment it by applying a superpixel clustering algorithm

3 Merge segments of the superpixels based on ve components (color similarity, texture similarity,
size similarity, shape similarity/compatibility, and a nal meta-similarity that linearly combines the
aforementioned scores)

The end results are proposals that indicate where in the image there could be an object:
 Figure 2: In this tutorial, we will learn how to use Selective Search region proposals to perform
object detection with OpenCV, Keras, and TensorFlow.

Notice how I’ve italicized “could” in the sentence above the image — keep in mind that region proposal
algorithms have no idea if a given region does in fact contain an object.

Instead, region proposal methods simply tell us:

“ Hey, this looks like an interesting region of the input image. Let’s apply our more computationally
expensive classi er to determine what’s actually in this region.

Region proposal algorithms tend to be far more e cient than the traditional object detection
techniques of image pyramids and sliding windows because:

Fewer individual ROIs are examined

It is faster than exhaustively examining every scale/location of the input image

The amount of accuracy lost is minimal, if any

In the rest of this tutorial, you’ll learn how to implement region proposal object detection.

Con guring your development environment

To con gure your system for this tutorial, I recommend following either of these tutorials:

How to install TensorFlow 2.0 on Ubuntu

 How to install TensorFlow 2.0 on macOS

Either tutorial will help you con gure your system with all the necessary software for this blog post in a
convenient Python virtual environment.

Please note that PyImageSearch does not recommend or support Windows for CV/DL projects.

Project structure
Be sure to grab today’s les from the “Downloads” section so you can follow along with today’s tutorial:

→ Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

1. $ tree
2. .
3. ├── beagle.png
4. └── region_proposal_detection.py
5.
6. 0 directories, 2 files

As you can see, our project layout is very straightforward today, consisting of a single Python script,
aptly named region_proposal_detection.py for today’s region proposal object detection
example.

I’ve also included a picture of Jemma, my family’s beagle. We’ll use this photo for testing our OpenCV,
Keras, and TensorFlow region proposal object detection system.

Implementing region proposal object detection with OpenCV,

Keras, and TensorFlow
Let’s get started implementing our region proposal object detector.

Open a new le, name it region_proposal_detection.py , and insert the following code:

→ Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

1. # import the necessary packages
2. from tensorflow.keras.applications import ResNet50
3. from tensorflow.keras.applications.resnet50 import preprocess_input
4. from tensorflow.keras.applications import imagenet_utils
5. from tensorflow.keras.preprocessing.image import img_to_array
6. from imutils.object_detection import non_max_suppression
7. import numpy as np
8. import argparse
9. import cv2

We begin our script with a handful of imports. In particular, we’ll be using the pre-trained ResNet50
classi er, my imutils implementation of non_max_suppression (NMS), and OpenCV. Be sure to
follow the links in the “Con guring your development environment” section to ensure that all of the
required packages are installed in a Python virtual environment.

Last week, we learned about Selective Search to nd region proposals where an object might exist.
We’ll now take last week’s code snippet and wrap it in a convenience function named
selective_search :

→ Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

11. def selective_search(image, method="fast"):
12. # initialize OpenCV's selective search implementation and set the
13. # input image
14. ss = cv2.ximgproc.segmentation.createSelectiveSearchSegmentation()
15. ss.setBaseImage(image)
16.
17. # check to see if we are using the *fast* but *less accurate* version
18. # of selective search
19. if method == "fast":
20. ss.switchToSelectiveSearchFast()
21.
22. # otherwise we are using the *slower* but *more accurate* version
23. else:
24. ss.switchToSelectiveSearchQuality()
25.
26. # run selective search on the input image
27. rects = ss.process()
28.
29. # return the region proposal bounding boxes
30. return rects

Our selective_search function accepts an input image and algorithmic method (either
"fast" or "quality" ).

From there, we initialize Selective Search with our input image (Lines 14 and 15).

We then explicitly set our mode using the value contained in method (Lines 19-24), which should
either be "fast" or "quality" . Generally, the faster method will be suitable; however, depending
on your application, you might want to sacri ce speed to achieve better quality results.

Finally, we execute Selective Search and return the region proposals ( rects ) via Lines 27-30.

When we call the selective_search function and pass an image to it, we’ll get a list of bounding
boxes that represent where an object could exist. Later, we will have code which accepts the bounding
boxes, extracts the corresponding ROI from the input image, passes the ROI into a classi er, and
applies NMS. The result of these steps will be a deep learning object detector based on independent
Selective Search and classi cation. We are not building an end-to-end deep learning object detector
with Selective Search embedded. Keep this distinction in mind as you follow the rest of this tutorial.

Let’s de ne the inputs to our Python script:

 → Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

32. # construct the argument parser and parse the arguments
33. ap = argparse.ArgumentParser()
34. ap.add_argument("-i", "--image", required=True,
35. help="path to the input image")
36. ap.add_argument("-m", "--method", type=str, default="fast",
37. choices=["fast", "quality"],
38. help="selective search method")
39. ap.add_argument("-c", "--conf", type=float, default=0.9,
40. help="minimum probability to consider a classification/detection")
41. ap.add_argument("-f", "--filter", type=str, default=None,
42. help="comma separated list of ImageNet labels to filter on")
43. args = vars(ap.parse_args())

Our script accepts four command line arguments;

--image : The path to our input photo we’d like to perform object detection on

--method : The Selective Search mode — either "fast" or "quality"

--conf : Minimum probability threshold to consider a classi cation/detection

--filter : ImageNet classes separated by commas that we wish to consider

Now that our command line args are de ned, let’s hone in on the --filter argument:

→ Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

45. # grab the label filters command line argument
46. labelFilters = args["filter"]
47.
48. # if the label filter is not empty, break it into a list
49. if labelFilters is not None:
50. labelFilters = labelFilters.lower().split(",")

Line 46 sets our class labelFilters directly from the --filter command line argument. From
there, Lines 49 and 50 overwrite labelFilters with each comma delimited class stored organized
into a single Python list.

Next, we’ll load our pre-trained ResNet image classi er:

→ Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

52. # load ResNet from disk (with weights pre-trained on ImageNet)
53. print("[INFO] loading ResNet...")
54. model = ResNet50(weights="imagenet")
55.
56. # load the input image from disk and grab its dimensions
57. image = cv2.imread(args["image"])
58. (H, W) = image.shape[:2]
Here, we Initialize ResNet pre-trained on ImageNet (Line 54).

We also load our input --image and extract its dimensions (Lines 57 and 58).

At this point, we’re ready to apply Selective Search to our input photo:

→ Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

60. # run selective search on the input image
61. print("[INFO] performing selective search with '{}' method...".format(
62. args["method"]))
63. rects = selective_search(image, method=args["method"])
64. print("[INFO] {} regions found by selective search".format(len(rects)))
65.
66. # initialize the list of region proposals that we'll be classifying
67. # along with their associated bounding boxes
68. proposals = []
69. boxes = []

Taking advantage of our selective_search convenience function, Line 63 executes Selective

Search on our --image using the desired --method . The result is our list of object region
proposals stored in rects .

In the next code block, we’re going to populate two lists using our region proposals:

proposals : Initialized on Line 68, this list will hold su ciently large pre-processed ROIs from our
input --image , which we will feed into our ResNet classi er.

boxes : Initialized on Line 69, this list of bounding box coordinates corresponds to our
proposals and is similar to rects with an important distinction: Only su ciently large regions
are included.

We need our proposals ROIs to send through our image classi er, and we need the boxes
coordinates so that we know where in the input --image each ROI actually came from.

Now that we have an understanding of what we need to do, let’s get to it:

→ Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

71. # loop over the region proposal bounding box coordinates generated by
72. # running selective search
73. for (x, y, w, h) in rects:
74. # if the width or height of the region is less than 10% of the
75. # image width or height, ignore it (i.e., filter out small
76. # objects that are likely false-positives)
77. if w / float(W) < 0.1 or h / float(H) < 0.1:
78. continue
79.
80. # extract the region from the input image, convert it from BGR to
 81. # RGB channel ordering, and then resize it to 224x224 (the input
82. # dimensions required by our pre-trained CNN)
83. roi = image[y:y + h, x:x + w]
84. roi = cv2.cvtColor(roi, cv2.COLOR_BGR2RGB)
85. roi = cv2.resize(roi, (224, 224))
86.
87. # further preprocess by the ROI
88. roi = img_to_array(roi)
89. roi = preprocess_input(roi)
90.
91. # update our proposals and bounding boxes lists
92. proposals.append(roi)
93. boxes.append((x, y, w, h))

Looping over proposals from Selective Search ( rects ) beginning on Line 73, we proceed to:

Filter out small boxes that likely don’t contain an object (i.e., noise) via Lines 77 and 78

Extract our region proposal roi (Line 83) and preprocess it (Lines 84-89)

Update our proposal and boxes lists (Lines 92 and 93)

We’re now ready to classify each pre-processed region proposal ROI:

→ Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

95. # convert the proposals list into NumPy array and show its dimensions
96. proposals = np.array(proposals)
97. print("[INFO] proposal shape: {}".format(proposals.shape))
98.
99. # classify each of the proposal ROIs using ResNet and then decode the
100. # predictions
101. print("[INFO] classifying proposals...")
102. preds = model.predict(proposals)
103. preds = imagenet_utils.decode_predictions(preds, top=1)
104.
105. # initialize a dictionary which maps class labels (keys) to any
106. # bounding box associated with that label (values)
107. labels = {}

We have one nal pre-processing step to handle before inference — converting the proposals list
into a NumPy array. Line 96 handles this step.

We make predictions on our proposals by performing deep learning classi cation inference (Line
102 and 103).

Given each classi cation, we’ll lter the results based on our labelFilters and --conf
(con dence threshold). The labels dictionary (initialized on Line 107) will hold each of our class
labels (keys) and lists of bounding boxes + probabilities (values). Let’s lter and organize the results
now:

→ Launch Jupyter Notebook on Google Colab

 Region proposal object detection with OpenCV, Keras, and TensorFlow
109. # loop over the predictions
110. for (i, p) in enumerate(preds):
111. # grab the prediction information for the current region proposal
112. (imagenetID, label, prob) = p[0]
113.
114. # only if the label filters are not empty *and* the label does not
115. # exist in the list, then ignore it
116. if labelFilters is not None and label not in labelFilters:
117. continue
118.
119. # filter out weak detections by ensuring the predicted probability
120. # is greater than the minimum probability
121. if prob >= args["conf"]:
122. # grab the bounding box associated with the prediction and
123. # convert the coordinates
124. (x, y, w, h) = boxes[i]
125. box = (x, y, x + w, y + h)
126.
127. # grab the list of predictions for the label and add the
128. # bounding box + probability to the list
129. L = labels.get(label, [])
130. L.append((box, prob))
131. labels[label] = L

Looping over predictions beginning on Line 110, we:

Extract the prediction information including the class label and probability (Line 112)

Ensure the particular prediction’s class label is in the label lter, dropping results we don’t wish to
consider (Lines 116 and 117)

Filter out weak con dence inference results (Line 121)

Grab the bounding box associated with the prediction and then convert and store (x, y)-coordinates
(Lines 124 and 125)

Update the labels dictionary so that it is organized with each ImageNet class label (key)
associated with a list of tuples (value) consisting of a detection’s bounding box and prob (Lines
129-131)

Now that our results are collated in the labels dictionary, we will produce two visualizations of our
results:

Before applying non-maxima suppression (NMS)

After applying NMS

By applying NMS, weak overlapping bounding boxes will be suppressed, thereby resulting in a single
object detection.

In order to demonstrate the power of NMS, rst let’s generate our Before NMS result:
 → Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

133. # loop over the labels for each of detected objects in the image
134. for label in labels.keys():
135. # clone the original image so that we can draw on it
136. print("[INFO] showing results for '{}'".format(label))
137. clone = image.copy()
138.
139. # loop over all bounding boxes for the current label
140. for (box, prob) in labels[label]:
141. # draw the bounding box on the image
142. (startX, startY, endX, endY) = box
143. cv2.rectangle(clone, (startX, startY), (endX, endY),
144. (0, 255, 0), 2)
145.
146. # show the results *before* applying non-maxima suppression, then
147. # clone the image again so we can display the results *after*
148. # applying non-maxima suppression
149. cv2.imshow("Before", clone)
150. clone = image.copy()

Looping over unique keys in our labels dictionary, we annotate our output image with bounding
boxes for that particular label (Lines 140-144) and display the Before NMS result (Line 149). Given
that our visualization will likely be very cluttered with many bounding boxes, I chose not to annotate
class labels.

Now, let’s apply NMS and display the After NMS result:

→ Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

152. # extract the bounding boxes and associated prediction
153. # probabilities, then apply non-maxima suppression
154. boxes = np.array([p[0] for p in labels[label]])
155. proba = np.array([p[1] for p in labels[label]])
156. boxes = non_max_suppression(boxes, proba)
157.
158. # loop over all bounding boxes that were kept after applying
159. # non-maxima suppression
160. for (startX, startY, endX, endY) in boxes:
161. # draw the bounding box and label on the image
162. cv2.rectangle(clone, (startX, startY), (endX, endY),
163. (0, 255, 0), 2)
164. y = startY - 10 if startY - 10 > 10 else startY + 10
165. cv2.putText(clone, label, (startX, y),
166. cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 255, 0), 2)
167.
168. # show the output after apply non-maxima suppression
169. cv2.imshow("After", clone)
170. cv2.waitKey(0)

Lines 154-156 apply non-maxima suppression using my imutils method.

From there, we annotate each remaining bounding box and class label (Lines 160-166) and display
the After NMS result (Line 169).
Both the Before NMS and After NMS visualizations will remain on your screen until a key is pressed
(Line 170).

Region proposal object detection results using OpenCV, Keras,

and TensorFlow
We are now ready to perform region proposal object detection!

Make sure you use the “Downloads” section of this tutorial to download the source code and example
images.

From there, open up a terminal, and execute the following command:

→ Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

1. $ python region_proposal_detection.py --image beagle.png
2. [INFO] loading ResNet...
3. [INFO] performing selective search with 'fast' method...
4. [INFO] 922 regions found by selective search
5. [INFO] proposal shape: (534, 224, 224, 3)
6. [INFO] classifying proposals...
7. [INFO] showing results for 'beagle'
8. [INFO] showing results for 'clog'
9. [INFO] showing results for 'quill'
10. [INFO] showing results for 'paper_towel'

Figure 3: Left: Object detections for the “beagle” class as a result of region proposal object detection with
OpenCV, Keras, and TensorFlow. Right: After applying non-maxima suppression to eliminate overlapping
bounding boxes.

Initially, our results look quite good.

If you take a look at Figure 3, you’ll see that on the left we have the object detections for the “beagle”
class (a type of dog) and on the right we have the output after applying non-maxima suppression.
As you can see from the output, Jemma, my family’s beagle, was correctly detected!

However, as the rest of our results show, our model is also reporting that we detected a “clog” (a type
of wooden shoe):

Figure 4: One of the regions proposed by Selective Search is later predicted incorrectly to have a “clog” shoe in
it using OpenCV, Keras, and TensorFlow.

As well as a “quill” (a writing pen made from a feather):

Figure 5: Another region proposed by Selective Search is then classi ed incorrectly to have a “quill” pen in it.

And nally, a “paper towel”:

 Figure 6: Our Selective Search + ResNet classi er-based object detection method (created with
OpenCV, TensorFlow, and Keras) has incorrectly predicted that a “paper towel” is present in this
photo.

Looking at the ROIs for each of these classes, one can imagine how our CNN may have been confused
when making those classi cations.

But how do we actually remove the incorrect object detections?

The solution here is that we can lter through only the detections we care about.

For example, if I were building a “beagle detector” application, I would supply the --filter beagle
command line argument:

→ Launch Jupyter Notebook on Google Colab

Region proposal object detection with OpenCV, Keras, and TensorFlow

1. $ python region_proposal_detection.py --image beagle.png --filter beagle
2. [INFO] loading ResNet...
3. [INFO] performing selective search with 'fast' method...
4. [INFO] 922 regions found by selective search
5. [INFO] proposal shape: (534, 224, 224, 3)
6. [INFO] classifying proposals...
7. [INFO] showing results for 'beagle'
 Figure 7: While Selective Search has proposed many regions that might contain an object, after
classi cation of the ROIs, we’ve ltered for only the “beagle” class so that all other classes are
ignored.

And in that case, only the “beagle” class is found (the rest are discarded).

Problems and limitations

As our results section demonstrated, our region proposal object detector “only kinda-sorta worked” —
while we obtained the correct object detection, we also got a lot of noise.

In next week’s tutorial, I’ll show you how we can use Selective Search and region proposals to build a
complete R-CNN object detector pipeline that is far more accurate than the method we’ve covered
here today.

What's next? I recommend PyImageSearch University.

Course information:
13 total3:52
classes • 21h 2m video • Last updated: 4/2021
★★★★★ 4.84 (128 Ratings) • 3,690 Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and
deep learning.

Do you think learning computer vision and deep learning has to be time-consuming,
overwhelming, and complicated? Or has to involve complex mathematics and equations? Or
requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things
to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change
education and how complex Arti cial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch
University, the most comprehensive computer vision, deep learning, and OpenCV course
online today. Here you’ll learn how to successfully and con dently apply computer vision to
your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll nd:

✓ 13 courses on essential computer vision, deep learning, and OpenCV topics

✓ 13 Certi cates of Completion

✓ 21h 2m on-demand video

 ✓ Brand new courses released every month, ensuring you can keep up with state-of-the-art
techniques

✓ Pre-con gured Jupyter Notebooks in Google Colab

✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no
dev environment con guration required!)

✓ Access to centralized code repos for all 400+ tutorials on PyImageSearch

✓ Easy one-click downloads for code, datasets, pre-trained models, etc.

✓ Access on mobile, laptop, desktop, etc.

CLICK H ERE TO J OIN PYIM AG ES EA RCH U N IV ERS ITY

Summary
In this tutorial, you learned how to perform region proposal object detection with OpenCV, Keras, and
TensorFlow.

Using region proposals for object detection is a 4-step process:

1 Step #1: Use Selective Search (a region proposal algorithm) to generate candidate regions of an
input image that could contain an object of interest.

2 Step #2: Take these regions and pass them through a pre-trained CNN to classify the candidate
areas (again, that could contain an object).

3 Step #3: Apply non-maxima suppression (NMS) to suppress weak, overlapping bounding boxes.

4 Step #4: Return the nal bounding boxes to the calling function.

We implemented the above pipeline using OpenCV, Keras, and TensorFlow — all in ~150 lines of code!

However, you’ll note that we used a network that was pre-trained on the ImageNet dataset.

That raises the questions:

What if we wanted to train a network on our own custom dataset?

How can we train a network using Selective Search?

And how will that change our inference code used for object detection?
I’ll be answering those questions in next week’s tutorial.

To download the source code to this post (and be noti ed when future tutorials are published here
on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on
Computer Vision, OpenCV, and Deep Learning. Inside you'll nd my hand-picked tutorials,
books, courses, and libraries to help you master CV and DL!

Your email address

DOWN LOA D TH E C ODE!

About the Author

Hi there, I’m Adrian Rosebrock, PhD. All too often I see developers, students, and
researchers wasting their time, studying the wrong things, and generally struggling to get
started with Computer Vision, Deep Learning, and OpenCV. I created this website to show
 you what I believe is the best possible way to get your start.

Previous Article:

OpenCV Selective Search for Object Detection

Next Article:

R-CNN object detection with Keras, TensorFlow, and Deep Learning

Comment section

Hey, Adrian Rosebrock here, author and creator of PyImageSearch. While I love hearing
from readers, a couple years ago I made the tough decision to no longer o er 1:1 help over
blog post comments.

At the time I was receiving 200+ emails per day and another 100+ blog post comments. I
simply did not have the time to moderate and respond to them all, and the shear volume of
requests was taking a toll on me.

Instead, my goal is to do the most good for the computer vision, deep learning, and OpenCV
community at large by focusing my time on authoring high-quality blog posts, tutorials, and
books/courses.

If you need help learning computer vision and deep learning, I suggest you refer to my
full catalog of books and courses — they have helped tens of thousands of developers,
students, and researchers just like yourself learn Computer Vision, Deep Learning, and
OpenCV.

Click here to browse my full catalog.