G Model

CIRP-1905; No. of Pages 4

CIRP Annals - Manufacturing Technology xxx (2019) xxx–xxx

Contents lists available at ScienceDirect

CIRP Annals - Manufacturing Technology

jou rnal homep age : ht t p: // ees .e lse vi er . com /ci r p/ def a ult . asp

Digital twin driven human–robot collaborative assembly

Arne Bilberg, Ali Ahmad Malik *
University of Southern Denmark, Sønderborg, Denmark
Submitted by Leo Alting (1), Copenhagen, Denmark

A R T I C L E I N F O A B S T R A C T

Keywords: This paper discusses an object-oriented event-driven simulation as a digital twin of a flexible assembly
Man-machine system
cell coordinated with a robot to perform assembly tasks alongside human. The digital twin extends the
Robot
use of virtual simulation models developed in the design phase of a production system to operations for
Digital twins
real-time control, dynamic skill-based tasks allocation between human and robot, sequencing of tasks
and developing robot program accordingly. The methodology combines lean methods of manual
assembly in human–robot collaboration paving path towards flexible human–robot work teams. The
study is validated with an industrial case study involving dexterous assembly tasks.
© 2019 Published by Elsevier Ltd on behalf of CIRP.

1. Introduction than ever. An aspect of it is human–robot collaboration (HRC) that

strives to use the best skills of humans and robots in a combination
Modern manufacturing landscape is increasingly getting to achieve a common task e.g. an assembly process which is a
complexed and today’s production plants are becoming multi- combination of dexterity, intelligence and sensing skills.
plexes of highly complex electro-mechanical elements. The The HRC work environment is complex and dynamic. Since
complexity of these systems refers to the information content these systems are supposed to offer high product variety, they need
that proliferates along the system’s lifecycle phases i.e. design, to be able to continuously extend and adapt to various configura-
development and operation. The advancement in virtualization, tions during their operation. It is important that such systems are
sensing technologies and computing power has evolved the quickly validated during design, development and operation.
concept of digital twins to design, develop and control complex Safety is another vital consideration in designing HRC workspace
systems by offering a safe-virtual-space for testing and validation. [4,5]. New approaches to design and control HRC systems therefore
A digital representation that mirrors the real-time operating need to accommodate high complexity, safety, and efficiency.
conditions of a physical system is defined as a digital twin (DT) Realization of final products is greatly dependent upon
[1]. The concept of digital twins originated from a hardware twin assembly processes where end-product is achieved by integration
developed in NASA’s Apollo program consisting of two identical of parts and components. With growing demand of product variety
space vehicles. A different approach to the concept of DT as by the end-users, various stages of product realization are
‘Conceptual Ideal for PLM’ in relation to product lifecycle identified for their potential to enable variety such as design,
management (PLM) was presented at University of Michigan fabrication, assembly, sales, or through adjustment during the
[2]. The approach was consisted of an idea that every system can be usage phase. However, assembly is considered the most important
a subset of two systems i.e. the physical system existing in the aspect of manufacturing to enable cost effective product variety.
physical world, and a virtual system existing in the virtual space But it requires that assembly systems are designed and operated to
containing all (necessary) information of the physical system. The handle variety oriented production [6].
bi-directional relation of physical artefacts with virtual models can Besides the significance of assembly work in manufacturing,
enable efficient product design, manufacturing and service the application of collaborative robots in variety-oriented assem-
throughout the system’s life cycle [3]. bly systems is very limited. The challenges are manifold whilst
As DTs are perceived to manage systems’ complexity in various safety of humans and frequent reconfigurations in the assembly
domains of sciences, significant focus has been observed to deal work are of significance. This paper presents a simulation based
with manufacturing-complexity driven by the need of customiza- digital-twin for the control of a human–robot collaborative
tion and decentralized controls of production environments. To assembly cell. A generic model of using DTs in HRC assembly is
cope with the present-day manufacturing challenges the ideation first presented. The issues solved by presented digital twins for
of man-machine systems is on the rise for much closer interaction HRC assembly are: (1) Rapid skills-based workload balancing
between human and robot for product variety; (2) Dynamic
workload balancing during operation to account for human
factors; (3) Trajectory planning and generating robot control
* Corresponding author.
E-mail address: [email protected] (A. Malik). program.

https://doi.org/10.1016/j.cirp.2019.04.011
0007-8506/© 2019 Published by Elsevier Ltd on behalf of CIRP.

Please cite this article in press as: Bilberg A, Malik AA. Digital twin driven human–robot collaborative assembly. CIRP Annals -
Manufacturing Technology (2019), https://doi.org/10.1016/j.cirp.2019.04.011
G Model
CIRP-1905; No. of Pages 4

2 A. Bilberg, A.A. Malik / CIRP Annals - Manufacturing Technology xxx (2019) xxx–xxx

2. State of the art

The desire for increasing manufacturing flexibility and produc-

tivity is a key driver for research in human–robot collaboration
especially in assembly systems [7]. HRC offers the opportunity to
increase the degree of production automation without having to
lose flexibility. A keynote paper “Cooperation of humans and
machines in assembly lines” [7] concluded that for closer
cooperation of humans and robots, besides ensuring safety of
the worker, newer methods of programming the HRC systems will
be required.
HRC is seen as a complex biomechatronic architecture from
design perspective and the design decisions are extensively
supported by computer-aided simulation models for validation
and failure assessments [8]. As the design and operation are
traditionally disconnected life-cycle phases from data-usage point
of view, these simulation models gradually get useless along the
system’s life cycle as the system evolves from design to operation.
These CAD (Computer Aided Design) based simulation models
contain maximum details of the desired performance of the
production system and have the potential to be integrated at all
levels of system’s life span [9,10]. By using sensing technologies,
and internet connectivity, these simulation models can be
connected to the physical events in operation phase forming a
digital twin [11]. The efficacy of digital twins can accommodate the
use of already developed simulation models from design phase to
the operation phase for intelligent control of complex systems e.g.
HRC. During operation, the digital twin can escalate productivity
Fig. 1. Digital twins in HRC assembly.
by offering the methods for fast reconfiguration of an HRC
assembly with each production changeover and can lead to new
offerings. A reference model for implementation of a digital twins potential, thereby distributing the tasks into manual and robotic
in manufacturing was proposed by Schleich et al. [3]. It ensures the tasks.
model properties such as scalability, interoperability, expansibility Module-2 is a library of function-blocks that are representing
and fidelity throughout the system’s life cycle. the assembly tasks and trigger the task events. Connected to the
The concept of DT as connecting a virtual simulation model with function-blocks is an event-based simulation of human and robot
a real control-system and a real production-system has been which is a representation of the actual production system. The
discussed by Khan et al. [11]. The authors presented an offline simulation executes each task according to event trigger-signals
approach for comparing a virtual simulation model against a received from function-blocks and generates operation plans
specification model that is receiving sensor signals from a physical calculating estimated cycle times. The simulation already has
system. The comparison reveals the operational success or failure defined key-positions (pick and place points), therefore minimum
based on conformity or non-conformity of specification model and information is needed to develop robot program. Module-3 is a
the simulation model. During the design phase of HRC, the decision engine that receives the data from simulation, assigns
usability of a disconnected digital twins has been discussed in Ref. appropriate resource for a balanced production schedule and
[12] for layout planning, reachability of human and robot for generates a control program. Module-4 is the control section for
various locations and human ergonomic analysis. the operator and the robot. The robot program is sent to the robot
A wide range of DTs based solutions in manufacturing are controller for execution on the robot. The human task-instructions
available in literature but the efforts to implement them for can be communicated at the screen in front of the operator.
assembly flexibility with human–robot interaction is rather
limited. A holistic approach needs to be addressed for integrating
operational activities in HRC with product data, simulations and a 4. Digital twins for human–robot collaboration
control system. This can utilize the fidelity of design related CAD
data in event-based simulation to support assembly process 4.1. Digital twins for tasks distribution and work-load balancing
planning, dynamic work-load balancing and workspace optimi-
zation attributed towards production space dynamics. This section forms a method to make a dynamic work-load
balancing of the assembly tasks based on the skills of humans and
3. System overview robots. Each task is a set of various activities e.g. picking the part,
moving the part to assembly location, and mounting the part in
The basis for an event-driven simulation-based digital twin required orientation. Since humans and robots have distinct skills,
acting as a counterpart of the human–robot assembly system is various factors can influence the suitability of a task for the robot
illustrated in Fig. 1. e.g. physical and geometrical properties of the components,
For this study, the DT advantages for HRC are focussed at skill- assembly characteristics, safety implications, feeding and joining
based tasks distribution between human and robot, generating an method involved. Method to quantify the automation complexity
optimized robot trajectory, balancing the workload during of assembly tasks has been discussed by Samy and ElMaraghy [13]
production, and generating robot program. This dynamic control which takes care of assembly attributes contributing to automa-
of human–robot collaboration is evaluated especially for variant tion complexity and a qualitative score as a representation for
oriented production environments. automation potential is generated.
The system consists of four modules. Module-1 is an engine that All assembly tasks in a cycle are evaluated for ease of
decomposes a product assembly into various tasks and evaluates automation against various attributes (Fig. 2). A unique score is
them based on physical properties and assembly characteristics. assigned representing the ease of automating an assembly task
Each task is then assigned a numerical score for their automation given the complexity of an HRC environment.

Please cite this article in press as: Bilberg A, Malik AA. Digital twin driven human–robot collaborative assembly. CIRP Annals -
Manufacturing Technology (2019), https://doi.org/10.1016/j.cirp.2019.04.011
G Model
CIRP-1905; No. of Pages 4

A. Bilberg, A.A. Malik / CIRP Annals - Manufacturing Technology xxx (2019) xxx–xxx 3

Fig. 3. Feedback in HRC assembly cell for signalling of critical events.

The comparison of the simulated signals with measured ones

help to identify the failure mode. The measured signals are
compared continuously with simulated signals. In case of a
deviation, the data is sent to the simulation for necessary
optimization. A temporary instruction is communicated to the
robot, which surpasses the original program. As soon as the system
returns to the normal state, the robot program is reverted to
original tasks.
For example, if the work-in-process at location 6 increases, it is
taken as a blockage, and a signal is initiated to the control system. It
redefines the control logic and instructs the robot to leave its
original tasks and start performing subsequent tasks on the
product at work-in-process (originally defined for the operator),
thus saving time for the human.
As soon the work-in-process decreases the robot resumes
performing its original tasks. Further, with every changeover,
based on the cycle times at predecessor and successor stations of
the HRC station, the simulation re-schedules the tasks to balance
the assembly-line.

4.3. Online optimization of robot trajectories

Fig. 2. Digital-twins supported assembly tasks distribution in HRC. Based on the allocation of tasks and human engagement with
the robot, an appropriate level of human–robot collaboration is
The data is then sent to the task-execution program (function- determined. The levels are based on the variables of time and
blocks) that is communicating with the simulation. The simulation space-sharing and are a trade-off between cycle-times and safety
makes the first run of the process to find accurate cycle times for implications. It is important that the HRC system is made safe for
each robotic and human task. Once the cycle times are evaluated, fellow human(s) for which several approaches are available both in
the final tasks assignment is performed to balance the workload of theory and practice. As per ISO15066, speed and separation
the assembly process. The balancing of the workload is performed monitoring need to be implemented in industrial HRC. This will
based on the logic that determines the task precedent constraint, cause the robot to have varying speeds based on the distance from
ease of automation (starting from highest in a descending order) the human and in some situations, a complete halt would require if
and availability of the resource (i.e. human and robot) and assigns the operator gets too close to the robot. Industrial 3D camera such
the task to the appropriate resource. This data is then sent again to as SafetyEye can be used for this objective [14].
the simulation to generate final control program. A Kinect sensor is used to monitor human positions and the
presence inside the work space. This study focuses on the potential
4.2. Digital twins enabled workload monitoring of DT in HRC to track the interference-volumes between human
and robot (Fig. 4), and the frequency of interferences. This data is
Manual assembly cells are prone to time variability in cycle times. then utilized to optimize robot trajectories periodically to
The human factors e.g. instability of humans with respect to work accommodate for the locations where human often enters. The
rate, skill level, motivation and failure-sensitivity for complex historic data of human positions will make simulation to get self-
processes are reasons for the possible time variability. Additionally, learned about constantly occurring human–robot collisions and
changeover times can also create imbalance in the line. Human generate robot trajectories free from possible human intervention.
factors continue to play their role in man-machine systems.
A Lean tool “Yo-i-don” in Toyota Production System is used to 4.4. Generating robot control program
take care of human-variability in assembly cells. The Japanese
words (Yo-i-don) are translated as "ready, set, go,". It is The purpose of this step is to generate program instructions for
synchronization of the processes for balancing the work-pace by the robot to perform the assigned tasks. The current industrial
coordination of the fellow operators to make sure that everyone approaches to teach collaborative robots are through teach-
finishes the tasks in given time window thereby minimizing the
delays. If an operator at a workstation is getting delayed, a bulb
lights-up alerting the fellow operator to help the colleague in
completing the task. Once the delayed tasks are completed the
whole production system will resume.
In the proposed DT driven assembly cell, “Yo-i-don” is
implemented to take care of human variability. Fig. 3 shows the
points where different sensors are used to signal the control
system for critical events happening at the assembly cell. Fig. 4. Volume of robot trajectories and possible human collision.

Please cite this article in press as: Bilberg A, Malik AA. Digital twin driven human–robot collaborative assembly. CIRP Annals -
Manufacturing Technology (2019), https://doi.org/10.1016/j.cirp.2019.04.011
G Model
CIRP-1905; No. of Pages 4

4 A. Bilberg, A.A. Malik / CIRP Annals - Manufacturing Technology xxx (2019) xxx–xxx

pendants, hand-guiding and offline-programming. For each of Based on the task precedent constraint, the ease of automation
these methods, in assembly with multiple robotic tasks, several and availability of the resource, the tasks are assigned to the
key points are needed to be taught to the robot for which appropriate resource. Assembly of tasks 1, 2, 3 and 4 are pick &
significant manual work is required. Additional refinements are place and screwing tasks and are assigned to robot. The tasks 5, 6,
needed to take care of the interferences with human and other 7 and 8 are kept manual. The sequence is sent to the simulation.
auxiliary equipment. Based on pre-defined key-locations a simulation is developed
The DT method has key positions for the robot already saved in generating a control program.
the simulation model. Once the tasks assignment is completed and The proposed assembly station receives sub-assembly from
robotic tasks are identified, the tasks are simulated in the virtual preceding station, and robot mounts parts 1, 2, 3 and 4. After this,
environment to generate robot trajectory. The intermediate the subassembly turns around and is transferred to the operator.
positions are generated automatically, and robot avoids any Operator picks-up parts 5, 6 and 7, forms a sub-assembly and
possible collision with the equipment. These paths are then saved assembles it on the main-assembly followed by the screwing task.
in an SRC file which is loaded to the robot. Additionally, for The work in-process monitoring (to take care of human factors) is
dynamic changes, the robot instructions are passed in form of a validated with the task 8 where the robot can perform the screwing
message that surpasses original robot program. task if there is increased work in process. Frequent occurring
human-positions are integrated with the simulation model to form
5. An assembly case study an obstacle-envelope to optimize the robot trajectories.

The case used in this study is a sub-assembly of a linear- 6. Conclusion

actuator as shown in Fig. 5. The product is currently assembled as a
manual process and is investigated for HRC production. The realization of digital twins is progressing but is facing
The simulation used in this study is a 3-dimensional virtual challenges due to insufficient synchronization possibilities be-
simulation model of human–robot cooperation. Tecnomatix tween physical and digital spaces and the lack of high-fidelity
Process Simulate software is selected and is used as a simulation simulation models at various levels of details. In this paper a
tool during the experiment. The software helped in three digital-twin driven human-robot assembly system is presented for
directions: (1) designing the digital-twin as a 3D virtual industrial application of HRC in variant oriented assembly systems.
representation of the HRC workstation for the experiment (3D Results show that the approach supports the notion of automation
models of the equipment are designed in CAD software and are while maintaining assembly flexibility. The method can extend the
imported in the simulation software), (2) estimating cycle times industrial applicability of HRC for newer as well as legacy assembly
and defining the robot trajectory, and (3) communicating with cells. The future work will consider the seamless integration of
function blocks for tasks execution. The virtual simulation model is different modules with minimum manual intervention.
built in four layers of modelling (creation of 3D CAD objects),
physics (placement of CAD objects in the scene), behaviour
(kinematics of robots and human), and rule (assembly process References
sequence), as suggested by Tao and Zhang [15].
[1] Rosen R, von Wichert G, Lo G, Bettenhausen KD (2015) About the Importance of
The virtual model of robot is a pre-defined kinematic model of Autonomy and Digital Twins for the Future of Manufacturing. IFAC-PapersOn-
UR-10 robot. The Human package of the software is used to develop Line 48:567–572.
a digital human for process verification. The digital human is a [2] Kahlen F-J, Flumerfelt S, Alves A (2016) Transdisciplinary Perspectives on
Complex Systems: New Findings and Approaches, Springer.
deformable mesh technology based human figure representing [3] Schleich B, Anwer N, Mathieu L, Wartzack S (2017) Shaping the Digital Twin for
visually accurate body shapes to virtually map out complex tasks. Design and Production Engineering. CIRP Annals—Manufacturing Technology
The computer used for deploying the experimental setup is 66:141–144.
[4] Pedrocchi N, Vicentini F, Matteo M, Tosatti LM (2013) Safe Human–Robot
configured with an Intel Core i7 processor and 16 GB RAM running Cooperation in an Industrial Environment. International Journal of Advanced
on Windows 10 operating system. Another computer, connected Robotic Systems 10:1–27. SAGE Publications.
via ethernet to simulation-computer, maintains a communication [5] Zanchettin AM, Ceriani NM, Rocco P, Ding H, Matthias B (2016) Safety in
Human–Robot Collaborative Manufacturing Environments: Metrics and Con-
with the robot and is equipped with Intel 2.7 GHz Core i7 CPU and trol. IEEE Transactions on Automation Science and Engineering 13:882–893.
12 GB RAM running on Windows 7. [6] Hu SJ, Ko J, Weyand L, ElMaraghy H, Lien T, Koren Y, et al (2011) Assembly
System Design and Operations for Product Variety. CIRP Annals—Manufacturing
Technology 60:715–733.
[7] Krüger J, Lien TK, Verl A (2009) Cooperation of Human and Machines in
Assembly Lines. CIRP Annals—Manufacturing Technology 58:628–646.
[8] Maurice P, Padois V, Measson Y, Bidaud P (2017) Human-Oriented Design of
Collaborative Robots. International Journal of Industrial Ergonomics 57:88–102.
[9] Bilberg A, Alting L (1991) When Simulation Takes Control. Journal of
Manufacturing Systems 10:179–193.
[10] Hehenberger P, Bradley D (2016) Mechatronic Futures: Challenges and Solutions
for Mechatronic Systems and their Designers, Springer.
[11] Khan A, Dahl M, Falkman P (2018) Digital Twin for Legacy Systems: Simulation
Model Testing and Validation. IEEE 14th International Conference on Automation
Science and Engineering (CASE), 421–426.
[12] Malik AA, Bilberg A (2018) Digital Twins of Human Robot Collaboration in a
Production Setting. Procedia Manufacturing 17:278–285.
[13] Samy S, ElMaraghy H (2010) A Model for Measuring Products Assembly
Complexity. International Journal of Computer Integrated Manufacturing
23:1015–1027.
[14] Makris S, Karagiannis P, Koukas S, Matthaiakis A-S (2016) Augmented Reality
System for Operator Support in Human–Robot Collaborative Assembly. CIRP
Annals—Manufacturing Technology 65:61–64.
[15] Tao F, Zhang M (2017) Digital Twin Shop-Floor: A New Shop-Floor Paradigm
Towards Smart Manufacturing. IEEE Access 5:20418–20427.
Fig. 5. Use case and experimentation.

Please cite this article in press as: Bilberg A, Malik AA. Digital twin driven human–robot collaborative assembly. CIRP Annals -
Manufacturing Technology (2019), https://doi.org/10.1016/j.cirp.2019.04.011