Power ai image-pipeline

2018 IBM Systems
Technical University
August
São Paulo
Image processing pipelines for
mobile devices
Paulo Queiroz
Systems Consultant
IBM Systems Lab Services

Session summary
— Image Processing Pipeline
— Web APIs to capture Image
– WebRTC / JavaScript Canvas
— Upload Image to the server
– WebSocket
— Pre-process the Image
– Reconstruct the image
— Use the Neural Network
– Which network or framework to use ?
– Multi-threading
— Process the image at the remote end
© Copyright IBM Corporation 20182
Using the WebCam on the Browser
To
Upload Images to a GPU enabled Server
Pre-process the image at the server
Use a Neural network to detect objects
within the image
Send back the result to the Browser

Web APIs to capture images
Most modern Web browsers support HTML5, which thanks to the community supports a
new Multimedia method called WebRTC ( Real time communication ).
The WebRTC API works on top of the tag <video> and can be called throught the method
navigator.mediaDevices.getUserMedia as shown bellow:
navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia;
navigator.mediaDevices.getUserMedia({ video: { width: 800, height: 600 }, audio: false }).then(getUserMedia_success).catch(general_error_message);
navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia;
navigator.mediaDevices.getUserMedia({ video: { width: 800, height: 600 }, audio: false }).then(getUserMedia_success).catch(general_error_message);

WebRTC vs Server
WebRTC is designed to stablish PTP communication between browsers.
Therefore in order to capture the video frames a second method is used, this method is
used throught the <canvas> tag
var fps = 30;
function getUserMedia_success(stream) {
source_video.src = window.URL.createObjectURL(stream);
source_video.play();
window.stream = stream;
}
detection_loop = setInterval(function(){ send_frame_to_server(); },
Math.floor(1000/fps));
var fps = 30;
function getUserMedia_success(stream) {
source_video.src = window.URL.createObjectURL(stream);
source_video.play();
window.stream = stream;
}
detection_loop = setInterval(function(){ send_frame_to_server(); },
Math.floor(1000/fps));
Video - OK
Draw video frame on canvas
Serialize video frame
Upload the image to server
Detection Loop

Pre-process the image – Client Side
In order to upload the image to the server, it has to be structured in a way that can be
understood by the server.
This can be done as shown bellow:
function send_frame_to_server()
{
if ( window.stream != undefined )
{
var capture_obj = document.getElementById('person_face');
capture_ctx = capture_obj.getContext('2d', { alpha: false });
capture_ctx.drawImage(source_video, 0, 0, video_width, video_height);
var img_url = capture_obj.toDataURL('image/jpeg');
var sp = img_url.split(',');
var obj_to_send = { 'frame' : unescape(sp[1]), 'frame_type' : sp[0] };
var json_string = JSON.stringify( obj_to_send );
send_socket.send(json_string);
}
}
function send_frame_to_server()
{
if ( window.stream != undefined )
{
var capture_obj = document.getElementById('person_face');
capture_ctx = capture_obj.getContext('2d', { alpha: false });
capture_ctx.drawImage(source_video, 0, 0, video_width, video_height);
var img_url = capture_obj.toDataURL('image/jpeg');
var sp = img_url.split(',');
var obj_to_send = { 'frame' : unescape(sp[1]), 'frame_type' : sp[0] };
var json_string = JSON.stringify( obj_to_send );
send_socket.send(json_string);
}
}

Upload the data to the server
In order to make the image upload work with the lowest possible, the WebSocket method
is preferred.
var server_name = window.location.hostname;
var send_socket_url =
'wss://'+server_name+':4556/socket_recv';
var send_socket = new WebSocket(send_socket_url);
send_socket.onmessage = handle_bboxes;
var server_name = window.location.hostname;
var send_socket_url =
'wss://'+server_name+':4556/socket_recv';
var send_socket = new WebSocket(send_socket_url);
send_socket.onmessage = handle_bboxes;
One of the key differences between REST ( PUT/POST )
and websocket is that the later is stateful and the former is
stateless.

Work at the server side
At the server side Python is being used to stablish the websocket conection and process
the image.
Tornado has been choosen due it‘s non-blocking IO capabilities, however almost all
current frameworks has non-blocking IO support.
import os, io, sys, uuid
import tornado.ioloop, tornado.web, tornado.httpserver, tornado.websocket
from pq_tornado_classes.socket_recv import socket_recv
from pq_tornado_classes.main_page import main_page
# Default variables and objects
http_port = 4556
def main():
global http_port
my_name = sys.executable
print("starting http server, port : "+str(http_port))
context = ('ssl/preg.crt', 'ssl/preg.key')
http_server = tornado.httpserver.HTTPServer(tornado_main(),
ssl_options={ "certfile": "ssl/preg.crt", "keyfile": "ssl/preg.key" })
http_server.listen(http_port)
tornado.ioloop.IOLoop.current().start()
#### main stuff
if __name__ == "__main__":
main()
import os, io, sys, uuid
import tornado.ioloop, tornado.web, tornado.httpserver, tornado.websocket
from pq_tornado_classes.socket_recv import socket_recv
from pq_tornado_classes.main_page import main_page
# Default variables and objects
http_port = 4556
def main():
global http_port
my_name = sys.executable
print("starting http server, port : "+str(http_port))
context = ('ssl/preg.crt', 'ssl/preg.key')
http_server = tornado.httpserver.HTTPServer(tornado_main(),
ssl_options={ "certfile": "ssl/preg.crt", "keyfile": "ssl/preg.key" })
http_server.listen(http_port)
tornado.ioloop.IOLoop.current().start()
#### main stuff
if __name__ == "__main__":
main()
class tornado_main(tornado.web.Application):
def __init__(self):
handlers = [
(r'/socket_recv', socket_recv),
(r"/", main_page)
]
settings = dict(
template_path=os.path.join(os.path.dirname(__file__), "template"),
static_path=os.path.join(os.path.dirname(__file__), "static"), login_url="/login",
debug=True, cookie_secret=str(uuid.uuid5(uuid.NAMESPACE_DNS, 'localhost.localdomain')),
)
tornado.web.Application.__init__(self, handlers, **settings)
class tornado_main(tornado.web.Application):
def __init__(self):
handlers = [
(r'/socket_recv', socket_recv),
(r"/", main_page)
]
settings = dict(
template_path=os.path.join(os.path.dirname(__file__), "template"),
static_path=os.path.join(os.path.dirname(__file__), "static"), login_url="/login",
debug=True, cookie_secret=str(uuid.uuid5(uuid.NAMESPACE_DNS, 'localhost.localdomain')),
)
tornado.web.Application.__init__(self, handlers, **settings)

WebSockeet Python Server
On Tornado, what differenciate a standard REST class froma Websocket type is the
WebSocketHandler, as highlighted on the code block
import io, base64, cv2, json
import tornado.websocket
import numpy as np
from aux_utils.frame_handler import frame_handler
from aux_utils.pq_object_detection_tensorflow import protocol_handler as object_detection_queue_server
from aux_utils import pq_logger, encode_transmission
from multiprocessing import Process, Queue
class socket_recv(tornado.websocket.WebSocketHandler) :
def check_origin(self, origin):
return(True)
import numpy as np
return(True)
def open(self):
self.logger = pq_logger(name='tornado : socket recv', stdout=False)
self.object_detect_queue_in, self.object_detect_queue_out = Queue(), Queue()
self.object_detect_inference_process = Process(target=object_detection_queue_server,
args=(self.object_detect_queue_in,self.object_detect_queue_out,
'pb/faces_cnh/inference_final/frozen_inference_graph.pb',
'pb/faces_cnh/inference_final/mscoco_label_map.pbtxt'))
self.object_detect_inference_process.start()
# Load general frame handler
self.inference_servers = [[ self.object_detect_queue_in, self.object_detect_queue_out ]]
self.video_proc = frame_handler(logger = self.logger, inference_servers = self.inference_servers, fps = 10,
size_for_inference = -1, )
self.set_nodelay(True)
self.logger.info('WebSocket open')
def open(self):

WebSockeet Python Server
On Tornado, what differenciate a standard REST class froma Websocket type is the
WebSocketHandler, as highlighted on the code block
import numpy as np
return(True)
import numpy as np
return(True)
def open(self):
def open(self):
def on_close(self):
self.object_detect_inference_process.terminate()
self.logger.info('WebSocket closed')
pass
def on_close(self):
self.object_detect_inference_process.terminate()
self.logger.info('WebSocket closed')
pass

WebSockeet Python Server – Process incoming messages
To each image that arrive a async response is sent, by sending the reply this way the CPU
utilization at the server becomes more efficient
async def on_message(self, message):
data = json.loads(message)
cur_session = data['session']
cur_frame = data['frame_seq']
send_frame = data['send_frame']
raw_frame = io.BytesIO(base64.b64decode(data['frame']))
arr_frame = np.asarray( raw_frame.getbuffer(), dtype=np.uint8)
img_frame = cv2.imdecode( arr_frame, cv2.IMREAD_COLOR)
self.video_proc.process_image(img_frame)
bboxes = list(self.video_proc.get_adjusted_bbox_info(use_ext_info=True))
ret_obj = { 'frame_id' : cur_frame, 'session' : data['session'], 'bboxes' : bboxes}
await (self.write_message(json.dumps(ret_obj)))
async def on_message(self, message):
data = json.loads(message)
cur_session = data['session']
cur_frame = data['frame_seq']
send_frame = data['send_frame']
raw_frame = io.BytesIO(base64.b64decode(data['frame']))
arr_frame = np.asarray( raw_frame.getbuffer(), dtype=np.uint8)
img_frame = cv2.imdecode( arr_frame, cv2.IMREAD_COLOR)
self.video_proc.process_image(img_frame)
bboxes = list(self.video_proc.get_adjusted_bbox_info(use_ext_info=True))
ret_obj = { 'frame_id' : cur_frame, 'session' : data['session'], 'bboxes' : bboxes}
await (self.write_message(json.dumps(ret_obj)))
Parse JSON String
Decode Image ( base64 )
Reconstruct image
Send to neural Network
Send back json string

What about this frame Handler stuff ?
This frame handler class is a helper class that assist into frame process and bondary boxes
processing.
On a traditional process if combined neural networks are wanted, they must be processed in
sequence, within the same thread
The frame handler helps into split the work across multiple threads through Queue() or network
connections ( remotely )
Adjust image to the wanted size
Send adjusted image to inference Queue()
Adjust and process bondary boxes
Inference process
Thread 0
Thread 1
Thread N
OpenMP

Sample Inference process
def protocol_handler(inq,outq, tf_checkpoint, tf_labels, tf_device = '/device:GPU:0',
tf_gpu_mem_factor = 0.2) :
logger = pq_logger(stdout = True, name = 'Object detection queue handler')
# /cpu:0
# /device:GPU:0
ob_detect = pq_object_detection(tf_chkpoint = tf_checkpoint,
tf_labels = tf_labels, tf_device = tf_device,
tf_gpu_mem_factor = tf_gpu_mem_factor, logger = logger)
while True :
message = inq.get()
# If got commands.... process them
if isinstance(message,dict) :
if 'cmd' in message :
if message['cmd'] == PQ_INFERENCE_SERVER_GET_TYPE :
outq.put(PQ_INFERENCE_SERVER_OBJ_DETECT)
# Find Faces
elif message['cmd'] == PQ_OB_COMMAND_DETECT_OBJECTS :
outq.put(ob_detect.detect_objects(message['data']))
def protocol_handler(inq,outq, tf_checkpoint, tf_labels, tf_device = '/device:GPU:0',
tf_gpu_mem_factor = 0.2) :
logger = pq_logger(stdout = True, name = 'Object detection queue handler')
# /cpu:0
# /device:GPU:0
ob_detect = pq_object_detection(tf_chkpoint = tf_checkpoint,
tf_labels = tf_labels, tf_device = tf_device,
tf_gpu_mem_factor = tf_gpu_mem_factor, logger = logger)
while True :
message = inq.get()
# If got commands.... process them
if isinstance(message,dict) :
if 'cmd' in message :
if message['cmd'] == PQ_INFERENCE_SERVER_GET_TYPE :
outq.put(PQ_INFERENCE_SERVER_OBJ_DETECT)
# Find Faces
elif message['cmd'] == PQ_OB_COMMAND_DETECT_OBJECTS :
outq.put(ob_detect.detect_objects(message['data']))

Which Network should be used
The network to be chosen usually is tied to the amount of objects and latency tolarated by
the application

Back to the client side
Once the server send the bondary boxes back to the client, it‘s time to process them and
render on the frame
function handle_bboxes(stuff)
{
var json_obj = JSON.parse(stuff.data);
var ctx_back = render_pic_obj.getContext('2d', { alpha: false });
ctx_back.drawImage(local_img, 0, 0);
for ( bbox_id = 0 ; bbox_id < json_obj['bboxes'].length; bbox_id++ )
{
bb = json_obj['bboxes'][bbox_id];
label = bb;
x = label['main_cord'][0];
y = label['main_cord'][1];
w = label['main_cord'][2] - x;
h = label['main_cord'][3] - y;
ctx_back.strokeStyle = "rgb("+label['main_color'][2]+","+label['main_color'][1]+","+label['main_color'][0]+");";
ctx_back.strokeRect(x, y, w, h);
}
}
function handle_bboxes(stuff)
{
var json_obj = JSON.parse(stuff.data);
var ctx_back = render_pic_obj.getContext('2d', { alpha: false });
ctx_back.drawImage(local_img, 0, 0);
for ( bbox_id = 0 ; bbox_id < json_obj['bboxes'].length; bbox_id++ )
{
bb = json_obj['bboxes'][bbox_id];
label = bb;
x = label['main_cord'][0];
y = label['main_cord'][1];
w = label['main_cord'][2] - x;
h = label['main_cord'][3] - y;
ctx_back.strokeStyle = "rgb("+label['main_color'][2]+","+label['main_color'][1]+","+label['main_color'][0]+");";
ctx_back.strokeRect(x, y, w, h);
}
}

Thank you!
Paulo Queiroz
Systems Consultant
pqueiroz@br.ibm.com
+55 0800 701 4262
ibm.com
Please complete the Session
Evaluation!

Notices and disclaimers
• © 2018 International Business Machines Corporation. No part of
this document may be reproduced or transmitted in any form
without written permission from IBM.
• U.S. Government Users Restricted Rights — use, duplication
or disclosure restricted by GSA ADP Schedule Contract with
IBM.
• Information in these presentations (including information relating
to products that have not yet been announced by IBM) has been
reviewed for accuracy as of the date of initial publication
and could include unintentional technical or typographical
errors. IBM shall have no responsibility to update this information.
This document is distributed “as is” without any warranty,
either express or implied. In no event, shall IBM be liable for
any damage arising from the use of this information,
including but not limited to, loss of data, business
interruption, loss of profit or loss of opportunity.
IBM products and services are warranted per the terms and
conditions of the agreements under which they are provided.
• IBM products are manufactured from new parts or new and used
parts.
In some cases, a product may not be new and may have been
previously installed. Regardless, our warranty terms apply.”
• Any statements regarding IBM's future direction, intent or
product plans are subject to change or withdrawal without
notice.
• Performance data contained herein was generally obtained in a
controlled, isolated environments. Customer examples are
presented as illustrations of how those
• customers have used IBM products and the results they may have
achieved. Actual performance, cost, savings or other results in other
operating environments may vary.
• References in this document to IBM products, programs, or services
does not imply that IBM intends to make such products, programs or
services available in all countries in which IBM operates or does
business.
• Workshops, sessions and associated materials may have been
prepared by independent session speakers, and do not necessarily
reflect the views of IBM. All materials and discussions are provided
for informational purposes only, and are neither intended to, nor
shall constitute legal or other guidance or advice to any individual
participant or their specific situation.
• It is the customer’s responsibility to insure its own compliance
with legal requirements and to obtain advice of competent legal
counsel as to the identification and interpretation of any
relevant laws and regulatory requirements that may affect the
customer’s business and any actions the customer may need to
take to comply with such laws. IBM does not provide legal advice
or represent or warrant that its services or products will ensure that
the customer follows any law.
16

Notices and disclaimers continued
• Information concerning non-IBM products was obtained from the suppliers of
those products, their published announcements or other publicly available
sources. IBM has not tested those products about this publication and cannot
confirm the accuracy of performance, compatibility or any other claims
related to non-IBM products. Questions on the capabilities of non-IBM
products should be addressed to the suppliers of those products. IBM does
not warrant the quality of any third-party products, or the ability of any such
third-party products to interoperate with IBM’s products. IBM expressly
disclaims all warranties, expressed or implied, including but not limited
to, the implied warranties of merchantability and fitness for a purpose.
• The provision of the information contained herein is not intended to, and
does not, grant any right or license under any IBM patents, copyrights,
trademarks or other intellectual property right.
• IBM, the IBM logo, ibm.com and [names of other referenced
IBM products and services used in the presentation] are
trademarks of International Business Machines Corporation,
registered in many jurisdictions worldwide. Other product and
service names might be trademarks of IBM or other
companies. A current list of IBM trademarks is available on
the Web at "Copyright and trademark information" at
: www.ibm.com/legal/copytrade.shtml.
• .
17

Power ai image-pipeline

Recommended

More Related Content

What's hot (20)

Similar to Power ai image-pipeline (20)

Recently uploaded (20)

Power ai image-pipeline