AR_experiments_Robot_Industry_4.0
AR_experiments_Robot_Industry_4.0
Abstract—The role of human in the Industrie 4.0 vision is still such as PLC (Programmable Logic Controllers) or robots, to
considered as irreplaceable. Therefore, user interfaces of cyber- management of the production at the SCADA (Supervisory
physical systems involved in the production automation need to Control and Data Acquisition) level. Therefore this paper
be well designed and taking into consideration the industrial
application requirements. With the advances in augmented and focuses only on specific parts. Selected activities involve the
virtual reality data visualization and novel interaction techniques presence of the user in the factory shop floor. Specifically,
like mid-air gestures, these approaches seem to be suitable for these activities involve mainly maintenance tasks, but also
integration into the industry environment. This paper describes monitoring and planning tasks can be included. At this level,
the implementation of an augmented reality application for devices used for user interaction and data visualization are mo-
smart glasses with mid-air gestures and smart phone with touch
interaction to compare and evaluate the usage of such interfaces bile devices (e.g., phones/tablets) and wearable smart glasses
in a production cell comprising an industrial robot. (e.g., look-through glasses). From the data presentation point
of view, augmented reality (AR) and virtual reality (VR) ap-
I. I NTRODUCTION plications provide data in form of standardized graphical user
Current trends in Industrie 4.0 vision [1] show that the interfaces, human-machine interfaces or graphical information
human interaction in Cyber-physical systems (CPS) cannot as a part of AR or VR environment.
be eliminated but, in opposite, it should be supported and The objective of this paper is to describe the experiences
emphasized. Apparently, humans cannot be replaced by robots with development of an AR application for augmentation of an
in some specialized product processing tasks, so the cooper- industrial robot. The paper focuses on the description of the
ation between humans and robots [2] needs to be supported. application requirements and prototype implementation. The
Users are also shifting from repeated manual work to more initial version of the user interface is evaluated and results are
specialized roles. Such roles, as mentioned by Gorecky et al. discussed.
[3], cover mainly servicing and maintenance of manufacturing The rest of the paper is organized as follows: section II
plants, but also monitoring, planning, and simulation of pro- discusses related AR applications and experiments. Section III
duction processes. To conclude that, human-system interaction describes the requirements and the architecture for the devel-
must be developed as an integral part of CPS with respect to oped AR application. Section IV describes the implementation
both user tasks but also human abilities and limitations. details of the AR application, including hardware and software
The application of human factors in the development of components. In section V, the qualitative evaluation of the AR
products is intensively used in the human-computer interaction application is described and results from the evaluation are
field. Methodologies like user-centered design (UCD) [4] that discussed in section VI. Finally, section VII rounds up the
include the usage of prototyping tools, representation of target paper with conclusion and points out some future work.
user groups in form of persons or task modeling, help to take
the user into account in every step of the product development. II. R ELATED W ORK
The usage of these approaches increases the chance that the VR and AR applications may be used with many types
user interface and interaction will be usable, therefore improve of displays, such as head-mounted displays (HMD), hand-
the performance and cause fewer user errors. Initial research held displays, spatial see-through displays or projectors. A
in the field of human-centered design in the context of CPS comprehensive list of current technologies is provided by Bim-
already started in the form of usability engineering in projects ber [8] and van Krevelen [9]. HMD and see-through glasses
like ARVIKA [5]. Additionally, Romer et al. [6] recently allow AR/VR data visualization while preserving empty hands.
showed how to develop applications for the Industrie 4.0 Moreover, they can be used in stereoscopic mode, which
environment using the UCD methodology. Also Valdeza et allows better illusion of 3D augmented visualization in space.
al. [7] described how to use additional usability methods for Hand-held displays require that at least one hand holds the
decreasing the complexity of user interfaces applications in display. Spatial displays are similar to hand-hold displays or
Industrie 4.0 environment. see-through glasses, but they are usually bigger and attached,
The diversity of the activities in Industrie 4.0 vision is very so no hands need to be used to hold them. On the other hand,
big, spanning from the interaction with individual elements, their usage is limited by their position. Projectors are used
177
FM
Axis 3
FM Axis 2
FM MM
MM
Axis 1
MM
Fig. 2. User using Epson Moverio smart glasses with AR application and
Leap Motion controller. Fig. 3. Robot visualization in base pose with multi-target marker composed
of three markers (MM) and with four frame markers (FM). Axis 1, 2, and 3
are depicted, other four axis are hidden for simplicity.
B. AR Applications
The AR applications for all platforms were developed
using Unity3d [20]. This framework allowed to implement
the structure of the AR application according to the planned Fig. 5. Various types of specific point highlight visualization on the robot. In
architecture, including marker detection and client-server ar- the left image the point is highlighted by 3D arrows. In the middle image the
point is highlighted by leading line going from the left side of the screen to
chitecture. the down right point. In the right image the point is not directly highlighted
Based on the literature search and manual for the ABB IRB but the state of the point is shown by text message.
1400 maintenance, a basic set of tasks that will be provided in
our AR application was detected. The tasks cover mainly the
interaction with the robot from near distance (up to 6 meters). – Highlighting specific points of the robot by 3D
The list of implemented visual and interaction functions is arrows.
listed as following and showed in Fig. 4 and in Fig. 5: – Navigation to invisible points on the robot using
• Data visualization
leading line.
– Providing instructions using text.
– Icon visualization, e.g., detection of safe zone.
• Basic interaction using touch control and mid-air gestures
– Robot visualization using a virtual reality model, out-
line augmentation of the robot and no visualization – Manipulation with robot by adjusting axis values.
of virtual robot using depth mask. – Navigation in menu structures.
– Highlighting robot parts by color. The AR application for HMD was developed for Epson
178
1) ”You will see several types of augmented robot (outline,
virtual robot, real robot) together with highlighted part
of the robot. Evaluate, which type of visualization is
most suitable for you.”
2) ”You will see three types of a specific point highlighting
(by 3D arrow, leading line and text). Evaluate suitability
of the highlighting.”
3) ”Use particular interaction technique (touch or Leap
motion gestures) to manipulate virtual robot to the
predefined position.”
4) ”Using particular interaction technique (touch or Leap
motion gestures), evaluate basic set of gestures for
Fig. 6. AR application run on Android smart phone. navigation in maintenance task wizard.”
Participants performed the tasks with one device setup, then
with the other. First half of the participants started with smart
glasses and second half started with Android smart phone.
At the beginning of each part, the users were given time to
experience the usage of the devices.
Two types of the data were collected and analyzed during
the evaluation, a) feedback given by participants from task
execution, and b) evaluation of expected functionality from
the observing application. The participants’ feedback was
collected as immediate feedback expressed mainly vocally
during the task execution. After completing the tasks, the
participants were interviewed to summarize the use of the
Fig. 7. Evaluation application running on PC. Various settings of the client applications.
applications are set by buttons on the right side. Virtual robot in the middle
reflects the visualization perceived by user of client application. VI. R ESULTS AND D ISCUSSION
Findings observed during the evaluation of AR applications
can be divided into two groups. The first group is related to
Moverio BT-200 smart glasses (see Fig. 2), which consist of
technical aspects and reflects technical findings. The second
two see-through displays for augmented reality and one cam-
group focuses on subjective evaluation and observation of
era on the side for environment sensing. The AR application
participants issues. In following subsections the major findings
was used in stereoscopic mode. In case of hand-held displays
are discussed.
the AR application was developed for Android devices and
the application was run in normal 2D mode (see Fig. 6). For A. Limitation of the camera for frame marker detection
evaluation of the AR application usage, the PC AR application While the multi-target marker detection together with ex-
was developed and used in 2D AR mode (see Fig. 7). The PC tended tracking used for tracking of the user in the scene works
application was implemented so that it is merging the client very well on hand-held devices, we have observed problems
and server layers into one, also providing operator with the with the detection and tracking with smart glasses. From the
functions to control the course of experiment and show how technical point of view, the marker detection can be separated
the user is performing and what he/she can see and do. in two different tasks. The first is the detection and recognition
The interaction with the client applications was done using of marker. The second is the marker tracking. For the first
touch interaction on mobile devices and using Leap motion phase, we evaluated that camera had to be closer to the marker
sensor with smart glasses. The Leap motion controller was not and also the marker should be still. For the second case (we
directly connected to the smart glasses, but it was connected to suppose successful marker recognition from first phase) the
PC, due to the availability of drivers, without any significant camera or marker can move up to some distance, where marker
impact on the user interaction experience. is lost. So the participant is limited by a distance of the camera
from the marker and also by the speed of movement (faster
V. AR A PPLICATION E VALUATION
moves causes tracking lost). The second criteria (speed) can
The prototype AR application was evaluated in the small be refined by higher camera frame rate (FPS); on the other
scale cell setup. The goal was to qualitatively analyze initial hand, the detection algorithms need to process more frames so
feedback of the test participants on individual types of the the hardware (CPU) limits the processing speed. Both factors
robot augmentation. In the study, there were 6 participants, influence the tracking stability on smart glasses.
who work as researchers and Ph.D. students in the field of The following table shows the size of frame markers in pix-
CPS. All participants were given all of the following tasks: els depending on the camera distance and camera resolution.
179
The used frame marker size was 47 × 47mm, and consists of tablet (FOV 60°), 1.6m
9 black or white squares separated by spaces of the same size. glasses (FOV 23°)
Based on our testing, we can say that a marker of this size 2.6m
is successfully recognized, if the marker resolution is at least 1.6m
2mm/px. For example with a Tablet Nexus 10, we were able
0.6m
to detect and recognize markers from 0.98m distance and than
the marker was tracked up to 1.5m. With smart glasses, the
first detection was closer to the marker (0.6m) and tracking
was lost at 1m distance. We observed that the marker detection
is also dramatically affected by lighting conditions and camera
contrast.
Epson (640x480px) Tablet (2560x1920px) 2.6 m 1.6 m
2.6m 0.47px/mm 0.36px/mm
22px/marker 17px/marker
1.6m 0.76px/mm 1.06px/mm
36px/marker 50px/marker
0.6m 2.12px/mm 3.04px/mm
100px/marker 143px/marker
The detection and tracking of multi-target markers had
better performance mainly due to the markers size (A3 format)
and availability of extended tracking. Successful detection Fig. 8. Comparison of visible area depending on the camera distance. The
starts at 2.5m distance for tablet and at 1.5m for glasses, and FOV of Moverio glasses (23◦ ) is depicted by cones a) and b) from distances
1.6m and 2.6m respectively. The close distance of glasses (0.6m) is not shown
were tracked up to 4m and 2.5m respectively. in bottom image. The c) cone corresponds to device with larger FOV (tablet).
From the participants’ subjective point of view, we saw In top figure are depicted visible areas for tablet in 1.6m distance and for
that the unstable glasses tracking caused issues with correct glasses from 2.6m, 1.6m, and 0.6m distances respectively (areas without white
overlay).
alignment of augmented data in real world and users had to
pause the tasks in order to re-detect markers. This issue had
impact on trust of the participants to AR visualization. augmentation, augmentation of spots by arrows, leading line
B. Limited field of view and text.
The used AR glasses had a narrow field of view (23◦ ), i.e. 1) Visualization styles: The evaluation of the visualization
they covered only small part of human binocular vision system style of the robot showed that users preferred to see the robot
(114◦ and 190◦ including monocular sector [21]). This fact either outlined or in full virtual reality. Only one user wanted
is limiting for real applications in two ways. First, the user to see invisible robot with highlighted parts only. The most
had to stay far away from augmented object for overall view probable reason is the fact that users wanted to be sure about
(approximately 3m in our case, see Fig. 8), thereby increasing matching parameters of the real and virtual robot.
the probability of user collision with the environment, which 2) Arrows: There were no issues with the usage of the
can be limiting in real factory environment. Second, in case arrows. All the users found the spots where arrows were
of close detail, the user had to be informed about model pointing. The only case when the user missed the point was
changes and had to be guided (for example by line of sight) to when the marker tracking was lost but the image of the robot
look at other part of augmentation. Up to our knowledge the was still shown, but not aligned. We also saw that when the
limited field of view is similar for all other optical see-through arrow was occluded by the robot, the user had difficulty to find
displays available currently on the market. it, because he did not know, how many of arrows are shown.
From participants’ point of view, we observed the difference 3) Leading line: The leading line was visible, but no user
between limitations of the visualization in tasks 1-2 and 3-4. fully understood its meaning and behavior. There was no clear
In first two tasks the user did not complain about not seeing difference between start and end point, so the users were not
the complete robot. However, in latter two tasks they did. The sure where it is pointing. Missing indication of the end point
difference may be the fact that in the first two tasks they only also caused that when the line went through the robot, it was
observed the robot; however, in second two tasks they had to not clear whether the intersection point is end point or not.
additionally control the robot. Another reason may be the fact 4) Text visualization: The text visualization was accepted
that in the first tasks they observed the augmented robot but well by all users. In case of smart glasses, due to the FOV
in the second tasks they interacted only with the virtual robot. limitation, the area for the text has to be very limited. Partic-
ipants suggested maximum 1 line of text with 30 characters
C. Visualization techniques as acceptable. In case of smart phone, the amount of text can
The evaluation of the visualization techniques had focused be approximately 3x more, due to the bigger FOV and better
on four major parts, namely visualization styles of robot screen resolution.
180
D. Interaction techniques evaluation R EFERENCES
The evaluation of interaction styles compared the usage of [1] H. Kagermann and W. Wahlster and J. Helbig, “Securing the future of
Leap motion gestures and touch interaction. German manufacturing industry: Recommendations for implementing
the strategic initiative INDUSTRIE 4.0,” ACATECH – German National
1) Touch interaction: This interaction technique was easily Academy of Science and Engineering, Tech. Rep., 2013.
understandable to the participants, even though it was not [2] J. Krüger, T. K. Lien, and A. Verl, “Cooperation of human and machines
implemented optimally. Participants preferred the selection of in assembly lines,” CIRP Annals-Manufacturing Technology, vol. 58,
no. 2, pp. 628–646, 2009.
axis and manipulation with objects in general using direct [3] D. Gorecky, M. Schmitt, M. Loskyll, and D. Zuhlke, “Human-machine-
manipulation with parts of the robot. interaction in the industry 4.0 era,” in Proccedings of the 12th IEEE
2) Leap motion gestures: The users were able to use hand International Conference on Industrial Informatics (INDIN’14). IEEE,
2014, pp. 289–294.
movement gestures to rotate the robot. They appreciated the [4] I. DIS, “9241-210 2010. ergonomics of human system interaction-
speed of the movement compared to touch interaction, but part 210: Human-centred design for interactive systems. standard,”
were not satisfied by the quality of extended finger count International Organization for Standardization, Tech. rep. International,
2010.
detection and changes in hand count detection. These issues [5] W. Friedrich, D. Jahn, and L. Schmidt, “Arvika-augmented reality for
caused that some of the users were not able to finish the development, production and service.” in Proceedings of the Interna-
interaction with Leap motion. tional Symposium on Mixed and Augmented Reality (ISMAR’02), 2002,
pp. 3–4.
[6] T. Römer and R. Bruder, “User centered design of a cyber-physical
VII. C ONCLUSION support solution for assembly processes,” Procedia Manufacturing,
This paper describes the development of an AR application vol. 3, pp. 456–463, 2015.
[7] A. C. Valdeza, P. Braunera, A. K. Schaara, A. Holzingerb, and
that augments an industrial robot for shop floor tasks like M. Zieflea, “Reducing complexity with simplicity-usability methods for
maintenance or cooperative work of human and robot. The industry 4.0,” in Proceedings of the 19th Triennial Congress of the IEA,
AR application was designed and developed for smart glasses vol. 9, 2015.
[8] O. Bimber and R. Raskar, “Modern approaches to augmented reality,”
with Leap motion air-gestures and mobile phones with touch in ACM SIGGRAPH 2006 Courses. ACM, 2006, p. 1.
interaction. For visualization of the robot, the AR application [9] D. Van Krevelen and R. Poelman, “A survey of augmented reality
is using outline, virtual or no visualization of the robot. For technologies, applications and limitations,” International Journal of
Virtual Reality, vol. 9, no. 2, p. 1, 2010.
the visualization of the spots on the robot, the AR application [10] S. Ong, M. Yuan, and A. Nee, “Augmented reality applications in
is using 3D arrows visualization, leading line visualization and manufacturing: a survey,” International Journal of Production Research,
text visualization. vol. 46, no. 10, pp. 2707–2742, 2008.
[11] N. Garcia, D. Espindola, G. Schroeder, C. Steinmetz,
Six participants performed 4 tasks for evaluating the AR and C. E. Pereira, “Cyber physical systems data visual-
application and interaction techniques. We observed that lim- ization using augmented reality in industrial automation,”
itations of the markers detection and limitations of the field https://www.researchgate.net/publication/276081154, accessed: 2016-
02-01.
of view using smart glasses caused usability issues of the AR [12] D. Reiners, D. Stricker, G. Klinker, and S. Müller, “Augmented reality
application for robot augmentation in our setup. Participants for construction tasks: doorlock assembly,” Proceedings of the IEEE and
wanted to have complete knowledge about the virtual robot ACM International Workshop on Augmented Reality (IWAR’98), vol. 98,
no. 1, pp. 31–46, 1998.
representation in the AR application and therefore selected [13] F. De Crescenzio, M. Fantini, F. Persiani, L. Di Stefano, P. Azzari,
the virtual robot or outline of the robot visualization style. and S. Salti, “Augmented reality for aircraft maintenance training
Additionally, they were satisfied with the 3D arrow visualiza- and operations support,” Computer Graphics and Applications, IEEE,
vol. 31, no. 1, pp. 96–101, 2011.
tion and text visualization, but they refused the leading line [14] A. Olwal, J. Gustafsson, and C. Lindfors, “Spatial augmented reality
concept of spot highlighting. In case of interaction with the on industrial cnc-machines,” in Electronic Imaging 2008. International
AR application, the touch interaction was evaluated as more Society for Optics and Photonics, 2008, pp. 680 409–680 409.
[15] J. Zhou, I. Lee, B. Thomas, R. Menassa, A. Farrant, and A. Sansome,
familiar then mid-air gestures. On the other hand, mid-air “In-situ support for automotive manufacturing using spatial augmented
gestures were considered as faster, but some gestures with reality.” International Journal of Virtual Reality, vol. 11, no. 1, 2012.
extended fingers were not correctly detected, which caused [16] R. Penna, M. Amaral, D. Espindola, S. Botelho, N. Duarte, C. E.
Pereira, M. Zuccolotto, and E. Morosini Frazzon, “Visualization tool for
failure of task completion. cyber-physical maintenance systems,” in Procceding of the 12th IEEE
Future work is devoted to perform further qualitative and International Conference on Industrial Informatics (INDIN’14). IEEE,
quantitative evaluation in environments where an AR ap- 2014, pp. 566–571.
[17] D. B. Espı́ndola, L. Fumagalli, M. Garetti, C. E. Pereira, S. S. Botelho,
plication is connected to the robot controller, so that the and R. V. Henriques, “A model-based approach for data integration
visualization can properly visualize robot states and position. to improve maintenance management by mixed reality,” Computers in
Industry, vol. 64, no. 4, pp. 376–391, 2013.
ACKNOWLEDGMENT [18] V. Paelke and C. Röcker, “User interfaces for cyber-physical systems:
Challenges and possible approaches,” in Design, User Experience, and
This research has been (partially) supported by the Tech- Usability: Design Discourse. Springer, 2015, pp. 75–85.
nology Agency of the Czech Republic under the research [19] “Vuforia sdk,” https://developer.vuforia.com/, accessed: 2016-02-01.
program TE01020415 (V3C - Visual Computing Competence [20] “Unity3d,” https://unity3d.com/, accessed: 2016-02-01.
[21] I. P. Howard and B. J. Rogers, Binocular Vision and Stereopsis, ser.
Center). This work is also (partially) funded by the Operational Oxford psychology series. New York: Oxford University Press, 1995,
Programme for Competitiveness and Internationalisation – vol. 29.
COMPETE 2020 and by FCT – Portuguese Foundation for
Science and Technology.
181