If PIP is being used within the

presentation window, be sure it is

positioned here.

Make sure all text, images, and

logos are positioned safely away
from this non-visible area.

Design and Deploy Collaborative Robot

Applications Using MATLAB

YJ Lim Ankur Bose

Technical Robotics Product Lead Embedded Systems Dev Lead
MathWorks, Natick, MA MathWorks, Bangalore, India
[email protected] [email protected]

© 2023 The MathWorks, Inc.

1
Webinar Presenters

YJ Lim, Ph.D. Ankur Bose

Principal Technical Robotics Product Sr. Embedded System Dev Lead +
Lead Robotics Hardware Support Team
Lead
• 20+ years Robotics and Mechatronics
• 8+ years of experience in Embedded
experience
Systems
• Main Expertise
• Main Expertise
• Cobots and Compliant Actuators
• UAV algorithm design
• AMRs and Mobile Manipulators
• Robot Manipulators connectivity

2
3
1st Webinar: Design and Deploy 2nd Webinar: Design Industrial 3rd Webinar: Applying AI to
Collaborative Robot (Cobot) Robot Applications from Enable Autonomy in Robotics Using
Applications Using MATLAB Perception to Motion MATLAB

Today (September 13) September 28 October 12

4
Polling

5
Agenda

Introduction
– Cobots
– How to program Cobots?

Core Components to design Cobotic Applications

– AI-driven robotics system design workflow
– Perception, Planning, Control, and Deployment

User Case: Intelligent Bin Picking

– Bin Picking Setup
– Connect and control UR cobots

Summary

6
Agenda

Introduction
– Cobots
– How to program Cobots?

Core Components to design Cobotic Applications

– AI-driven robotics system design workflow
– Perception, Planning, Control, and Deployment

User Case: Intelligent Bin Picking

– Bin Picking Setup
– Connect and control UR cobots

Summary

7
Evolution of ADAS and Autonomous Driving Car Technologies

L5 Full Automation Self-Driving Car

L4 High Automation Self-Driving & Human-Driven Car

Auto Pilot: Road Train

Conditional
L3 Auto Pilot: Parking
Automation
Auto Pilot: Highway
L2 Partial Automation Auto Pilot: Traffic Jam Assist
Junction Assist

AEB-VRU(Cyclist)

Driver AEB-VRU (Pedestrian)

L1
Assistance AEB-Vehicle (City/Inter-Urban)
Lane Keep Assist/Lateral Support
ACC
L0 No Automation FCW, LDW
2010 2014 2016 2018 2020 2025 2030

J3016 8
Evolution of Industrial Robotics Technologies

L5 Full Autonomy • Fully autonomous in all situations

L4 High Autonomy • Fully autonomous in certain situations or defined areas AI-powered

Robots
Conditional • Control all monitoring of the environment
L3 • Require a human’s attention and intervention
Autonomy

• Use sensory input from environment to make decisions Cobots

L2 Partial Autonomy • Requires pre-registration and clear instruction

• Traditional industrial robots

• Single automated operation
L1 Automation • Repeated motion for specific tasks with high precision and Traditional Industrial
speed
• Not use environment information Robots
• Not capable of sensing or adapting to changes in environment

L0 No Automation Human operate machines; No robots are involved

1961 2008 2020+/-

9
 Cobots are a key trend in factory automation

AI AI-powered
AI will allow the realization of a lot more
Robots
advanced applications with cobots.*

Katja Butterweck (ABB, Global Product Specialist)

Cobots

Traditional Industrial
Robots The arrival of cobots has broken down traditional
barriers to adoption and inspired thousands of SMEs
to deploy automation for the first time.*
Joe Campbell (Universal Robots)
1961 2008 2020+/-

*: source from “Tech Briefs”

10
 Cobots
Top 5 Industrial Applications of Cobots*

PICK AND
PLACE
Question?
• Do you teach a robot with the
manufacturer’s teach
pendant? ASSEMBLY

Done by
• Do you teach with offline off-line
programming? programming
MACHINE
TENDING

Done by online WELDING

programming (e.g.,
Teach Pendant)

PACKAGING &
PALLETIZING
*: source from Universal Robots
11
 Top 5 Industrial Applications of Cobots

Online Programming PICK AND

Offline Programming
PLACE

ASSEMBLY

MACHINE
TENDING

• Traditional Teach Pendant WELDING

Programming • Flexible Offline Programming

• Hand Guiding or Teaching by • AI Solutions

Demonstration PACKAGING &
PALLETIZING

12
 Offline Programming Technical Algorithms and Simulation

70% Cobot Applications Top 5

Machine
Example Technologies
Application Areas Pick and Place Assembly Welding
Welding Palletizing
Palletizing Painting
Painting Sanding
Sanding Inspection
Inspection
Tending
6-D Pose Estimation
Joint-Space Motion Planning
Task-Space Motion Planning
Coverage Planning
Task-Space Motion Control
Force / Impedance Control
Visual Servoing
Mobile Manipulation

Photorealistic simulation
Collision / Force Simulation

Typically Applied Sometimes Applied

: AI solutions
13
 Offline Programming Technical Programming and Simulation

70% Cobot Applications Top 5

Machine
Example Technologies
Application Areas Pick and Place Assembly Welding
Welding Palletizing
Palletizing Painting
Painting Sanding
Sanding Inspection
Inspection
Tending
6-D Pose Estimation
Joint-Space Motion Planning
Scalable for other robot brands and models / Applications
Task-Space Motion Planning
Coverage Planning
Task-Space Motion Control
Force / Impedance Control
Visual Servoing
Mobile Manipulation

Photorealistic simulation
Collision / Force Simulation

Typically Applied Sometimes Applied

: AI solutions
14
LINK

15
Agenda

Introduction
– Cobots
– How to program Cobots?

Core Components to design Cobotic Applications

– AI-driven robotics system design workflow
– Perception, Planning, Control, and Deployment

User Case: Intelligent Bin Picking

– Bin Picking Setup
– Connect and control UR cobots

Summary

16
Cobot for part sorting

Simulation Hardware Camera

Detector

17
System Level Simulation and Testing
Platform

Robot Model

AI model

Perceive Plan Control

18
Autonomous system design
workflow

Autonomous algorithms
AI
Perceive Plan Control
Autonomous system design
workflow

System
Modeling

Autonomous algorithms

Perceive Plan Control

Deploy

Platform
 Deep Learning with MATLAB

Perceive
21
Complete perception pipeline with domain specific functions

Pre-processing Post-processing

AI Model

Object
Point Cloud Matching with 6 DoF Object
RGB-D Image Detection with
Reconstruction CAD models Pose
YOLOv3

24
 Pose estimator – Primary Component Analysis (PCA)

Post-processing

AI Model

Object
Matching with 6 DoF Object
Detection with
CAD models Pose
YOLOv3

25
 MathWorks offers many perception tools with cobot applications

Camera Calibration Fault & Anomaly Detection

Visual SLAM & 3D Vision Point Cloud Processing

Computer Vision Toolbox (mathworks.com) 26
Autonomous system design
workflow

System
Modeling

Planning

Perceive Plan Control

Deploy

Platform
 Plan Motion Planning

Path Planning Trajectory Generation

Plan a collision-free path from

a starting configuration to a Create a time-based
goal configuration control sequence

1 2 3
Sense Identify Parts Execute a Pick-and-Place Task

Pick up the
Scan Detect Parts Estimate Pose Select a part Place the part
part

28
 Performant Robotics Algorithms
Path Planner
- ex: Optimization-based motion planner

camera

Perceive Robotics System Toolbox

TM

manipulatorRRT
manipulatorCHOMP

Plan
Doc Page
Navigation Toolbox

plannerBiRRT
Control plannerRRT
plannerRRTStar
plannerHybridAStar
plannerAStarGrid
PlannerPRM
…

Ebook 29
AI model for motion planning?

AI

Define takeoff and landing Exploit AI sampler to inform Path planned

locations sample-based path planning in a feasible time

Deep Learning Toolbox

Navigation Toolbox
31
 Add Capability with Constraints & Performance with AI
AI + Sampling-Based Planning AI + Optimization-Based Planning

Path Length
without DL: 1.8 m

Path Length
with DL: 0.8 m

Better Paths in Tricky Shorter, More Efficient

Faster Planning Speed
Scenarios Paths 32
Autonomous system design
workflow

System
Modeling

Control

Perceive Plan Control

Deploy

Platform
Advanced Control Designs for Optimal System Behavior

Model Predictive Control (MPC) to

• estimate future plant output
• solve an optimization problem
Perceive • account for constraints in plant outputs,
etc

Control Logic for

• Supervisory control
Plan • Task scheduling
• Fault management

Multi-agent control to
Control • Assign and coordinate actions

AI in Controls

34
Control
Design control logic for robots

Perceive

Plan

Control

Stateflow
35
 Reinforcement Learning Toolbox
Case study: picking with reinforcement learning Deep Learning Toolbox
Simscape Multibody

Contact
Robot Define RL
forces Training Validation
modeling agent
modeling

36
Autonomous system design
workflow

System
Modeling

Perceive Plan Control

Deploy

Platform
MATLAB and Simulink Code Generation

Code
Generation

38
Deploy to target with zero coding errors

CPU
Any CPU oneDNN ARM Compute
No Library needed Library Library

GPU
Code
Generation

µC

FPGA

39
 MATLAB and Simulink Code Generation

Generate efficient C++ code that plugs easily into

your C++ architectures
• Create ROS specific
folder structure,
MATLAB and resources files, and
C++ Software
Simulink components
Architecture libraries
• Generate C++ code

• Compile (Catkin /
Colcon)

• Launch ROS node

41
Deploy to Jetson device as CUDA ROS node

Other ROS
nodes

CUDA ROS node

CUDA
ROS Code .cpp
CMakeLists.txt

Generation

NVIDIA® Jetson™
42
Agenda

Introduction
– Cobots
– How to program Cobots?

AI in Robotics
– Why Use AI for Robotics (Cobots)?
– AI-driven robotics system design workflow

User Case: Intelligent Bin Picking

– Bin Picking Setup
– Connect and control UR cobots

Summary

43
User Case – Intelligent Bin Picking

45
Intelligent bin picking setup

Depth Camera

UR5e Cobot

Bin

Cobot stand

46
What do we need to solve for Intelligent bin picking

Connectivity
Perception Planning/Control
Deployment
DL-Based Object Detection Hardware connection established
manipulatorRRT computes
trained on real images for flat, with UR support package
collision-free paths online
spaced objects
contopptraj optimizes trajectory Camera feedback and EE control
given kino-dynamic constraints integrated using ROS
PCA for Pose estimation using
YOLO results Trajectories deployed to robot as
Results previewed with they are provided by the planner
rigidbodytree.show()

Robotics System Toolbox Robotics System Toolbox

Robotics System Toolbox
Computer Vision Toolbox ROS Toolbox
47
How MATLAB talks with UR Cobots: ROS

MATLAB with
ROS Universal_Robots_
Robotics System Toolbox and
Controller ROS_Driver
ROS Toolbox

Robot Operating System (ROS)

UR Cobots

48
Role of ROS middleware: From Camera to Cobot

Publish Raw image

Subscribe
Trajectory
Linux Host

ROS
Subscribe Publish
Raw image Trajectory

Publish
Subscribe
Pose
Pose

Pose Motion
Estimation Planning

49
Perception: Use Deep Learning to train the YOLO network
Start

Arrange the objects in bin

RGB image as
Subscribe to RGB Camera images over seen by Camera
ROS
NO

Are camera images NO

more than 400?
YES
Train the YOLO network

Learning andYESloss
rate satisfactory? Object
identification by
YES YOLO after
training
Use training network to identify object

Training complete 50
Perception: Pose estimation using Point cloud
Start

Place objects in bin

Obtain Point Cloud images from camera

Extract the bin base plane in the point

cloud

Extract the points in the point cloud not

part of bin base plane but lie within
bounding box

Apply PCA on each object point cloud to

identify pose

Pose estimation
complete

51
Perception: Estimated pose of objects

52
Motion Planning and trajectory generation

manipulatorRRT

Constrained Time-Optimal Path Parametrized

(TOPP) Trajectory

53
Connectivity with UR Cobot from MATLAB

The Robotics System Toolbox Support Package for Universal Robots UR Series Manipulators
allows user to connect to and control Universal Robots Cobots over ROS.

54
Set UR Robot with MathWorks URCap

56
Getting started with communication and control of UR5e cobot

57
Intelligent bin picking Examples in the support package

58
Intelligent bin picking with Simulink

LINK Intelligent Bin Picking in Universal Robot and Gazebo from Simulink

59
Intelligent bin picking with Industrial Parts

60
Intelligent bin picking – future enhancements to watch out for

▪ Support for ROS2

– This will allow communication with UR
Cobot over the new ROS2 protocol

▪ Send trajectories over URScript

– Using URScript language we can send
commands directly to UR controller

▪ Bin picking for randomized distribution

– Use Deep Learning based Image processing
capability from Computer Vision Toolbox to
pick and sort object in randomized heap.

61
Agenda

Introduction
– Cobots
– How to program Cobots?

AI in Robotics
– Why Use AI for Robotics (Cobots)?
– AI-driven robotics system design workflow

User Case: Intelligent Bin Picking

– Bin Picking Setup
– Connect and control UR cobots

Summary

62
 Musashi Seimitsu Industry Co.,Ltd.
Detect Abnormalities in Automotive Parts

MATLAB use in project:

▪ Preprocessing of captured images
▪ Image annotation for training
▪ Deep learning based analysis
– Various transfer learning methods
(Combinations of CNN models, Classifiers)
– Estimation of defect area using Class Activation Map
(CAM)
– Abnormality/defect classification
Automated visual inspection of 1.3 million bevel ▪ Deployment to NVIDIA Jetson using GPU Coder
gear per month

“ Using camera connection, preprocessing, and various pretrained models in MATLAB enabled
us to work on the entire workflow. Through discussions with consultants, our team gained
many tips for solving problems, growing the skills of our engineers.
“
63
Core of robotics and autonomous systems design +
AI

Perceive Plan Control

Computer vision /
Mission planning Motion control
Image processing

Localization, mapping Path tracking

/ SLAM Motion planning

Sensor fusion and Behavior planning Control logic

tracking

Deep learning Reinforcement

networks Learning

64
 MATLAB and Simulink as an integrated development environment from
perception to motion
System
Engineering • Support a full suite of
curated algorithms for
Autonomy Algorithms
robotics
Perceive Plan Control
• Design and evaluate AI
applications
Connect &
Deploy
• Simulate and test Cobot
behaviors within an
Platform integrated development
environment

66
Learn More – Cobots / Manipulators Robotics Examples

Examples

67
Learn More – Cobots / Manipulators Robotics Hardware Support

UR Support Package►

UR Intelligent Bin Picking (MATLAB)►

Learn More►

UR Intelligent Bin Picking (Simulink)► 68

Key Takeaways

• Perceive environment for advanced robot

applications using Deep Learning in
MATLAB

• Teach robot motions using inverse

kinematics and motion planning

• Control robot motion for safe interaction

• Connect and deploy robots using MATLAB

via ROS

69
 YJ Lim, Ph.D. Ankur Bose
Principal Technical Robotics Product Robotics Hardware Support Dev
Lead Lead
[email protected] [email protected]

© 2023 The MathWorks, Inc. MATLAB and Simulink are registered trademarks of The
MathWorks, Inc. See www.mathworks.com/trademarks for a list of additional trademarks.
Other product or brand names may be trademarks or registered trademarks of their
respective holders.
70
71