Conference on Autonomous and

Robotic Construction of Infrastructure

June 2-3 Ames, IA Iowa State University of Science and Technology

Proceedings of the 2015

Conference on Autonomous
and Robotic Construction
of Infrastructure
Edited by David J. White, Ahmad Alhasan,
and Pavana Vennapusa
About CEER Notice
The mission of the Center for Earthworks Engineering 7KHFRQWHQWVRIWKLVUHSRUWUHÀHFWWKHYLHZVRIWKH
Research (CEER) at Iowa State University is to be the authors, who are responsible for the facts and the
nation’s premier institution for developing fundamental accuracy of the information presented herein. The
knowledge of earth mechanics, and creating inno- RSLQLRQV¿QGLQJVDQGFRQFOXVLRQVH[SUHVVHGLQWKLV
vative technologies, sensors, and systems to enable publication are those of the authors and not necessar-
rapid, high quality, environmentally friendly, and eco- ily those of the sponsors.
nomical construction of roadways, aviation runways,
railroad embankments, dams, structural foundations, This document is disseminated under the sponsorship
IRUWL¿FDWLRQVFRQVWUXFWHGIURPHDUWKPDWHULDOVDQG of the U.S. DOT UTC program in the interest of infor-
related geotechnical applications. mation exchange. The U.S. Government assumes no
liability for the use of the information contained in this
About MTC
document. This report does not constitute a standard,
The Midwest Transportation Center (MTC) is a region- VSHFL¿FDWLRQRUUHJXODWLRQ
al University Transportation Center (UTC) sponsored
E\WKH86'HSDUWPHQWRI7UDQVSRUWDWLRQ2I¿FHRI The U.S. Government does not endorse products
the Assistant Secretary for Research and Technology or manufacturers. If trademarks or manufacturers’
(USDOT/OST-R). The mission of the UTC program names appear in this report, it is only because they
is to advance U.S. technology and expertise in the are considered essential to the objective of the docu-
many disciplines comprising transportation through ment.
the mechanisms of education, research, and technol-
Quality Assurance Statement
ogy transfer at university-based centers of excellence.
Iowa State University, through its Institute for Trans- The U.S. Department of Transportation (U.S. DOT)
portation (InTrans), is the MTC lead institution. provides high-quality information to serve Govern-
ment, industry, and the public in a manner that pro-
About InTrans
motes public understanding. Standards and policies
The mission of the Institute for Transportation (In- are used to ensure and maximize the quality, objec-
Trans) at Iowa State University is to develop and tivity, utility, and integrity of its information. The U.S.
implement innovative methods, materials, and tech- DOT periodically reviews quality issues and adjusts its
QRORJLHVIRULPSURYLQJWUDQVSRUWDWLRQHI¿FLHQF\VDIHW\ programs and processes to ensure continuous quality
reliability, and sustainability while improving the improvement.
learning environment of students, faculty, and staff in
WUDQVSRUWDWLRQUHODWHG¿HOGV
ISU Nondiscrimination Statement
Iowa State University does not discriminate on the
basis of race, color, age, ethnicity, religion, national
origin, pregnancy, sexual orientation, gender identity,
genetic information, sex, marital status, disability, or
status as a U.S. veteran. Inquiries regarding non-
GLVFULPLQDWLRQSROLFLHVPD\EHGLUHFWHGWR2I¿FHRI
Equal Opportunity, Title IX/ADA Coordinator, and Af-
¿UPDWLYH$FWLRQ2I¿FHU%HDUGVKHDU+DOO$PHV
,RZDHPDLO
HRRI¿FH#LDVWDWHHGX
 Technical Report Documentation Page

1. Report No. 2. Government Accession No. 3. Recipient’s Catalog No.

4. Title and Subtitle 5. Report Date

Proceedings of the 2015 Conference on Autonomous and Robotic Con- June 2015
struction of Infrastructure
6. Performing Organization Code

7. Author(s) 8. Performing Organization Report No.

Editors: David J. White, Ahmad Alhasan, and Pavana Vennapusa
9. Performing Organization Name and Address 10. Work Unit No. (TRAIS)
Center for Earthworks Engineering Research
Institute for Transportation
11. Contract or Grant No.
Iowa State University Research Park
2711 S. Loop Drive, Suite 4700
Ames, IA 50010-8664
12. Sponsoring Organizations 13. Type of Report and Period Covered
Midwest Transportation Center U.S. Department of Transportation Proceedings
2711 S. Loop Drive, Suite 4700 OĜce of the Assistant Secretary for 14. Sponsoring Agency Code
Ames, IA 50010-8664 Research and Technology
1200 New Jersey Avenue, SE DTRT13-G-UTC37
Caterpillar, Inc.
P.O. Box 1875 Washington, DC 20590
Peoria, IL 61656
15. Supplementary Notes
Visit www.intrans.iastate.edu and www.ceer.iastate.edu/carci/proceedings/ for pdfs of this and other publications.
16. Abstract
This is a proceedings of 21 papers presented at the 2015 Conference on Autonomous and Robotic Construction of
Infrastructure at Iowa State University, Ames, Iowa, on June 2–3, 2015. The conference provided information on
how autonomous robotics systems are being applied in practice, discussed emerging technologies, and described
theories that will no doubt shape the future of this ęeld. The papers cover a range of topics including mobile robotic
operations, visual analysis, terrain modeling, simulations inspired by natural processes, multi-dimensional modeling,
3D printing, sensors, data processing, and new applications for construction technologies like photogrammetry. An
underlying theme for many of the papers is data analytics—trying to make sense of and bring value to the deluge
of data now being generated during construction operations (e.g., machine telematics, digital photos, weather data,
sensors, etc.).

17. Key Words 18. Distribution Statement

automated machine guidance—autonomous construction—earthworks
engineering—robotic construction—transportation construction
19. Security Classięcation (of this 20. Security Classification (of this 21. No. of Pages 22. Price
report) page) 2 NA
Unclassięed. Unclassified.

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized

Proceedings of the 2015 Conference on Autonomous
and Robotic Construction of Infrastructure
June 2015

Edited by
David J. White
Ahmad Alhasan
Pavana Vennapusa

Sponsors
Caterpillar, Inc.
Iowa Department of Transportation
Center for Industrial Research and Service, Iowa State University
Department of Civil, Construction, and Environmental Engineering, Iowa State University
Midwest Transportation Center, Iowa State University
Proceedings of the 2015 Conference on Autonomous
and Robotic Construction of Infrastructure
‹,RZD6WDWH8QLYHUVLW\

Copyright and reprint permission: Abstracting is permitted with credit to the author and source. Copying
LQGLYLGXDOSDSHUVLQZKROHRULQSDUWIRUQRQSUR¿WFODVVURRPHGXFDWLRQDOSXUSRVHVLVSHUPLWWHGZLWKFUHGLW
to the author and source. For all other purposes, all rights, title, and interest, including copyright, in indi-
vidual papers herein are owned by the author(s) of the individual papers.

/LEUDU\RI&RQJUHVV,661

,6%1

An electronic version of the proceedings is available online,

http://www.ceer.iastate.edu/carci/proceedings/.

The accuracy and validity of the text and data presented in the papers are the responsibility of the various
authors and not of the Center for Earthworks Engineering Research, the Institute for Transportation, Iowa
6WDWH8QLYHUVLW\RURWKHUVSRQVRUVRIWKH&RQIHUHQFHRQ$XWRQRPRXVDQG5RERWLF&RQVWUXFWLRQRI
Infrastructure.

0DUFLD%ULQN3URGXFWLRQ(GLWRU
Institute for Transportation
Iowa State University
6/RRS'ULYH6XLWH
$PHV,$

ACKNOWLEDGMENTS
The Center for Earthworks Engineering Research would like to thank the following organizations for spon-
VRULQJWKH&RQIHUHQFHRQ$XWRQRPRXVDQG5RERWLF&RQVWUXFWLRQRI,QIUDVWUXFWXUHDQGWKHUHODWHG
proceedings of papers:

Caterpillar, Inc.

Center for Industrial Research and Service, Iowa State University

Department of Civil, Construction, and Environmental Engineering, Iowa State University

Iowa Department of Transportation

Midwest Transportation Center, Iowa State University

86'HSDUWPHQWRI7UDQVSRUWDWLRQ2I¿FHRIWKH$VVLVWDQW6HFUHWDU\IRU5HVHDUFKDQG7HFKQRORJ\

iv
Contents
vii Foreword
David J. White

1 Integrated Mobile Sensor-Based Activity Recognition of Construction Equipment

and Human Crews
 5H]D$NKDYLDQ/LDQQH%ULWRDQG$PLU%HK]DGDQ

21 Terrestrial Laser Scanning Roughness Assessments for Infrastructure

Ahmad Alhasan and David J. White

34 Autonomous Construction of Separated Artifacts by Mobile Robots Using SLAM

and Stigmergy
 +DGL$UGLQ\6WHIDQ:LWZLFNLDQG)UDQFHVFR0RQGDGD

47 A Comparison of Automated and Semi-Automated Compressed Earth Block

Presses Using Natural and Stabilized Soils
 -HGDGLDK)%XUURXJKV7RGG65XVKLQJDQG&3KLOOLS5XVFKH

56 Virtual Operator Modeling Approach for Construction Machinery

 <X'X0LFKDHO&'RUQHLFK%ULDQ/6WHZDUG(ULF5$QGHUVRQ/DZUHQFH).DQHDQG
 %ULDQ-*LOPRUH

67 Vision-Based Machine Pose Estimation for Excavation Monitoring and Guidance

 &KHQ)HQJ.XUW0/XQGHHQ6X\DQJ'RQJDQG9LQHHW5.DPDW

85 A User-centered Approach to Investigate Unmanned Aerial System (UAS)

Requirements for a Department of Transportation Applications
Masoud Gheisari and Javier Irizarry

105 Simulation of Biologically-Inspired Control Algorithms for Teams of Ground

Vehicles
 &KULVWRSKHU*RRGLQ=DFKDU\3UHYRVWDQG%HUWUDQG/HPDVVRQ

112 Case Study on 3D Modeling and AMG Practices

Fangyu Guo, Yelda Turkan, and Charles T. Jahren

118 A Formalism for Utilization of Autonomous Vision-Based Systems and Integrated

Project Models for Construction Progress Monitoring
 .HYLQ.+DQ-DFRE-/LQDQG0DQL*ROSDUYDU)DUG

 ([SORUDWRU\6WXG\RQ)DFWRUV,QÀXHQFLQJ8$63HUIRUPDQFHRQ+LJKZD\
Construction Projects: as the Case of Safety Monitoring Systems
 6XQJMLQ.LPDQG-DYLHU,UL]DUU\

149 The Current State of 3D Printing for Use in Construction

 0HJDQ$.UHLJHU%UXFH$0DF$OOLVWHU-XOLDQD0:LOKRLWDQG0LFKDHO3&DVH

v
159 Bridge Structural Condition Assessment using 3D Imaging
 6LPRQ/DÀDPPH<HOGD7XUNDQDQG/LDQJ\X7DQ

171 Bluetooth Low Energy Sensing Technology for Proximity Construction

Applications
JeeWoong Park, Yong Cho, and Willy Suryanto

180 A Design Framework for Off-road Equipment Automation

 %ULDQ/6WHZDUG/LH7DQJDQG6KXIHQJ+DQ

197 Discrete Element Modelling (DEM) For Earthmoving Equipment Design and
Analysis: Opportunities and Challenges
Mehari Tekeste

207 Impacts of Automated Machine Guidance on Earthwork Operations

 3DYDQD.59HQQDSXVD'DYLG-:KLWHDQG&KDUOHV7-DKUHQ

 5RERWLF+\EULG'DWD&ROOHFWLRQ6\VWHP'HYHORSPHQWIRU(I¿FLHQWDQG6DIH+HDY\
Equipment Operation
 &KDR:DQJDQG<RQJ.&KR

 'HVLJQLQJ'LJLWDO7RSRJUDSK\2SSRUWXQLWLHVIRU*UHDWHU(I¿FLHQF\ZLWKD
Primitives and Operators Approach
Caroline Westort

237 Applicability and Limitations of 3D Printing for Civil Structures

Mostafa Yossef and An Chen

247 Time-Optimal Path Planning for Automated Grain Carts

 0HQJ]KH=KDQJDQG6RXUDEK%KDWWDFKDU\D

vi
Foreword
By David J. White, Richard L. Handy Professor of Civil, Construction, and Environmental Engineering and Director,
Center for Earthworks Engineering Research, Iowa State University of Science and Technology

Infusion of autonomous and robotic operations into infrastructure construction holds promise for
VLJQL¿FDQWO\LPSURYLQJWKHTXDOLW\HI¿FLHQF\DQGVDIHW\RIEXLOGLQJ$PHULFD¶VLQIUDVWUXFWXUH'XH
to the complex developments required in the areas of theory, data analytics, machine systems,
RSHUDWRUWUDLQLQJVSHFL¿FDWLRQVGHYHORSPHQWDQGVHQVRUVLWLVHQYLVLRQHGWKDWSDUWQHUVKLSVEH-
WZHHQDFDGHPLDLQGXVWU\DQGJRYHUQPHQWDJHQFLHVDUHQHHGHGWRPDNHVWUDWHJLFDQGHI¿FLHQW
DGYDQFHPHQWVLQWKHQHDUWHUPWRUHDOL]HPD[LPXPEHQH¿W

Although there are efforts focused on the topic of robotics for manufacturing, seemingly little
DWWHQWLRQKDVEHHQJLYHQWRWKHIXOOVSHFWUXPRIWHFKQRORJLHVVHQVRUVV\VWHPVVSHFL¿FDWLRQV
training, visualization, and data analytics needed for automating the in situ manufacturing of
our infrastructure. Robotics (including co-robotic operations) and autonomous equipment op-
erations hold great potential for advancement. Even a small improvement in productivity could
JHQHUDWHDVLJQL¿FDQWUHWXUQJLYHQWKHPRUHWKDQELOOLRQDQQXDOLQYHVWPHQWLQLQIUDVWUXFWXUH
construction. Unfortunately, infrastructure construction has only limited theory established for
process control.

7KH&RQIHUHQFHRQ$XWRQRPRXVDQG5RERWLF&RQVWUXFWLRQRI,QIUDVWUXFWXUHDW,RZD6WDWH
University was organized to discuss application and theory for advancing autonomous and
robotic-guided equipment to improve productivity, quality, reliability, and safety for civil infrastruc-
ture construction and maintenance. Unlike some conferences that focus on practice or theory
and are dominated by one particular discipline, this conference provided information on how
autonomous robotics systems are being applied in practice, discussed emerging technologies,
DQGGHVFULEHGWKHRULHVWKDWZLOOQRGRXEWVKDSHWKHIXWXUHRIWKLV¿HOG

Focusing attention on this topic can be used to leverage needed investment in people and
facilities to better anchor this area as a “core” national research focus. Given that this area of
research and development covers a spectrum of needs, participants were invited to bridge the
gaps within and between academia and industry.

Papers presented at the conference are collected in this proceedings. They cover a range of
topics including mobile robotic operations, visual analysis, terrain modeling, simulations inspired
E\QDWXUDOSURFHVVHVPXOWLGLPHQVLRQDOPRGHOLQJ'SULQWLQJVHQVRUVGDWDSURFHVVLQJDQG
new applications for construction technologies like photogrammetry. An underlying theme for
many of the presentations was data analytics—trying to make sense of and bring value to the
deluge of data now being generated during construction operations (e.g., machine telematics,
digital photos, weather data, sensors, etc.). These papers show that there is tremendous inter-
est in this topic and the future is bright.

:KHQZHORRNEDFNDQG\HDUVIURPQRZ,EHOLHYHZHZLOOVHHWKDWWKHSDUWLFLSDQWV
in this conference continued their tremendous and exciting work to advance autonomous and
robotics operations, that new collaborations were formed, and that new developments have in-
GHHGPDGHVLJQL¿FDQWFRQWULEXWLRQVWRLPSURYLQJKXPDQOLYHVE\EHWWHUPHQWRIWKHLQIUDVWUXFWXUH
construction processes.

vii
Integrated Mobile Sensor-Based Activity Recognition
of Construction Equipment and Human Crews

Reza Akhavian
Lianne Brito
Amir Behzadan
Department of Civil, Environmental, and Construction Engineering
University of Central Florida
12800 Pegasus Dr.
Orlando, FL 32816
[email protected]
[email protected]
[email protected]

ABSTRACT
Automated activity recognition of heavy construction equipment as well as human crews can
contribute to correct and accurate measurement of a variety of construction and infrastructure
project performance indicators. Productivity assessment through work sampling, safety and
health monitoring using worker ergonomic analysis, and sustainability measurement through
equipment activity cycle monitoring to eliminate ineffective and idle times thus reducing
greenhouse gas emission (GHG), are some potential areas that can benefit from the integration
of automated activity recognition and analysis techniques. Despite their proven performance
and applications in other domains, few construction engineering and management (CEM)
studies have so far employed non-vision sensing technologies for construction equipment and
workers’ activity recognition. The existence of a variety of sensors in ubiquitous smartphones
with evolving computing, networking, and storage capabilities has created great opportunities
for a variety of pervasive computing applications. In light of this, this paper describes the latest
findings of an ongoing project that aims to design and validate a ubiquitous smartphone-based
automated activity recognition framework using built-in accelerometer and gyroscope sensors.
Collected data are segmented to overlapping windows to extract time- and frequency domain
features. Since each sensor collected data in three axes (x, y, z), several features from all three
axes are extracted to ensure device placement orientation independency. Finally, features are
used as the input of supervised machine learning classifiers. The results of the experiments
indicate that the trained models are able to classify construction workers and equipment
activities with over 90% overall accuracy.

Key words: activity recognition—big data analytics—construction engineering—machine

learning—mobile sensor

INTRODUCTION
With the ever-growing infrastructure projects, demands for information and automation
technologies in the architecture, engineering, construction, and facility management (AEC/FM)
industry is rapidly increasing. 3D printing in design and construction of affordable houses
(Krassenstein 2015), drones for various applications in construction jobsites (Irizarry et al.
2012), and Micro-Electro-Mechanical Systems (MEMS) sensors for tracking and monitoring of
construction resources (Akhavian and Behzadan 2015) are some of the examples of how latest

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
cutting-edge technology serves the AEC/FM industry. The latter, in particular, has ample
unrivalled potentials for use in day-to-day operations due to the existence of various sensors in
a major technology platform carried by almost everyone these days, the smartphone.

Ubiquity, affordability, small size, and computing power of mobile phones, equipped with a host
of sensors have made them ideal choices for tracking and monitoring of construction resources.
In particular, these built-in sensors can provide invaluable information regarding the
performance, safety, and behavior of construction workers and equipment in the field. For
example, activity analysis using inertial measurement unit (IMU) sensors including
accelerometer and gyroscope can help evaluate the time spent on interconnected construction
tasks. Such information results in better understanding and potential improvement of the
processes involved. Moreover, effective and timey analysis and tracking of the construction
resources can help in productivity measurement, progress evaluation, labor training programs,
safety and health management, and greenhouse gas emission (GHG) and fuel consumption
analysis in construction projects.

This paper presents the results of a smartphone sensors-based machine learning platform for
accurate recognition and classification of activities performed by construction equipment and
workers. Specifically, process data is collected using smartphone built-in accelerometer and
gyroscope from construction equipment and human crews. Certain data pre-processing is then
performed to prepare the data as the training input of supervised machine learning classifiers.
The outputs of the classifiers are the labels of various activities carried out by the construction
resources and detected using this framework.

LITERATURE REVIEW
Human Activity Recognition using Sensors
The initial efforts on human activity recognition date back to the late ’90s (Foerster et al. 1999).
During the last 15-16 years, the use of MEMS sensors has increased tremendously for
acquiring knowledge on humans’ activities. With the growing demand in activity recognition for
different fields of research and practice, the cost of system implementation and prototyping has
also decreased, which makes the technology more affordable and accessible to businesses and
the industry. Most of the activity recognition research during the past decade has been
conducted using wired or wireless accelerometers attached to different parts of a subject’s
body. It has been proven that the accuracy of the recognition algorithm improves as sensors are
attached to more than one part of the body, while a disadvantage of this approach is that the
presence of multiple asynchronous sensors each communicating through a separate interface
make the data collection process computationally inefficient and tedious, while ergonomically
obtrusive and uncomfortable for the subject. In a study by Frank et al. (2010) MEMS-based
IMUs attached to the performer’s waist were used to present four different algorithms and
recognize daily activities such as standing, running, walking, and falling. Even though most of
the activities were recognized with an accuracy of 93-100%, the rate of accuracy of detecting
falling was 80%.

Activity recognition has been widely used in the medical field primarily for elderly patient
monitoring. In a study by Gupta and Dallas (2014), waist-mounted triaxal accelerometer was
used to classify gait events into six daily living activities including run, jump, walk, sit and
transitional events including sit to stand and stand to kneel. While most of the previous studies
measured the activities with accelerometer sensor, some studies added gyroscope sensor to
improve the accuracy of classification. For instance, in a study aiming at accurate and fast fall

Akhavian, Brito, Behzadan 2

detection, Li et al. (2009) proved that using both accelerometer and gyroscope data for activity
recognition yields in more accurate results than using accelerometer only. They reported
specific improvement in classifying dynamic transitions rather than static postures and in
determining whether the transitional movement was intentional. In another study, Frank et al.
(2010) used MEMS-based IMU to train four different classification algorithms that recognized
daily activities such as standing, sitting, walking, running, jumping, falling, and lying. In these
and most other similar studies, the target activities for classification are daily routine tasks.
However, activities in a construction jobsite are relatively more complex and involve interactions
between multiple resources (human and equipment crews), which highlights the need for
rigorous research in this area within the AEC/FM domain.

Design and development of a small-size, unobtrusive, and low-cost data collection scheme has
been one of the most important challenges in improving the accuracy of activity recognition in a
more affordable and robust manner (Lara and Labrador 2013). During the last decade and with
the advancement of smartphones, a major paradigm shift occurred in the data collection
scheme for human activity recognition. As mentioned before, nowadays smartphones include a
wide variety of computationally powerful sensors. Many researchers seized this opportunity to
build more pervasive human activity recognition systems. Kwapisz et al. (2011) conducted a
study on 29 Android smartphone users that utilized accelerometer sensor to monitor daily
activities such as walking, jogging, ascending stairs, descending stairs, sitting, and standing for
a period of time. The most accurate classification results were achieved in two relatively
straightforward activities: sitting and jogging (compared to walking, standing, and climbing
up/down the stairs). Again, even though the target activities were recognized with 90% accuracy
in most cases (except for climbing up/down the stairs), recognizing such daily and
straightforward activities is relatively easier than those carried out in a construction site, for
instance. Dernbach et al. (2012) published the results of their study aimed at measuring simple
and more complex daily activities using built-in accelerometer and gyroscope sensors in
smartphones. Simple activities included biking, climbing stairs, driving, lying, running, sitting,
standing, and walking while complex activities included cleaning (i.e. wiping down the kitchen
counter top and sink), cooking (i.e. heating a bowl of water in the microwave and pouring a
glass of water from a pitcher in the fridge), medication (i.e. retrieving pills from the cupboard and
sorted out a week’s worth of doses), sweeping (i.e. sweeping the kitchen area), washing hands,
and watering plants. Their results showed that while an accuracy of around 90% was achieved
for simple activities, more complex activities were best classified with less than 50% accuracy.
Moreover, in all activity types, adding gyroscope data to accelerometer data improved the
results by 10-12%. Another research that used smartphone accelerometer and gyroscope
sensors was conducted by Wu et al. (2012). In this research an iPod Touch was used for data
collection measuring 13 different daily activities (e.g. stair climbing, jogging, sitting) from 16
human subjects. Using only the accelerometer data, the accuracy in recognizing these activities
ranged between 50% to 100%, while adding the gyroscope data helped improve the results by
another 3.1% to 13.4%.

Non-Human Subject Activity Recognition using Sensors

In addition to human activity recognition, classification of activities performed by construction
equipment is investigated in this research. To the best of the authors’ knowledge, very little
research has been conducted aiming at non-human subject activity recognition since the
application areas are rare and limited. For example, in transportation research and practice, the
use of sensors for transportation mode detection, urban planning improvement, targeted
advertising, and guidance systems has recently gained traction (Hemminki et al. 2013; Wang et
al. 2010; Zheng et al. 2010). Although most of these studies primarily relied on global

Akhavian, Brito, Behzadan 3

positioning data (GPS) data for recognizing activities and in particular, transportation modes,
some of them have used accelerometer data for more precision. In one study, Reddy et al.
(2010) used GPS and accelerometer to classify stationary, walking, running, biking and
motorized transportation. In their work, classification mainly relied on GPS speed for detecting
motorized mode and no distinction was made between different motorized modalities. However,
in another study aiming at improving this framework, Hemminki et al. (2013) used only
accelerometer data to detect if the subject is stationary or on a motorized transportation and to
classify the motorized transportation modalities into bus, train, metro, tram, or car. Using some
improved algorithms for estimating the gravity component of the accelerometer measurements
and feature extraction, they achieved higher precision and recall than the combined GPS and
accelerometers approach while eliminating the high power consumption of the GPS sensor. In
addition to transportation systems management and planning, a few construction engineering
and management (CEM) research studies targeted non-human subject activity recognition that
are summarized in the next Subsection.

Activity Recognition in Construction

Considering the dynamic and complex environment of most construction project sites, being
able to control and measure the efficiency of construction resources is vital to the overall
performance of the project in terms of time and financial resources. Moreover, by monitoring
workers and equipment activities, catastrophes that include safety and health issues as well as
many lawsuits could be prevented. Within the CEM research domain, vision-based systems are
used in most existing object recognition and tracking frameworks. For example, Brilakis et al.
(2011) proposed a framework for vision-based tracking of construction entities. Their
methodology requires calibration of two cameras, recognition of construction resources and
identification of the corresponding regions, matching the entities identified in different cameras,
two-dimensional (2D) tracking of the matched entities, and finally calculating the 3D
coordinates. This and similar vision-based approaches, although provide promising results for
recognition and tracking of construction equipment, still require much computation in each one
of the aforementioned steps. In another study, an image processing methodology was adopted
for idle time quantification of hydraulic excavators (Zou and Kim 2007). The level of detail (LoD)
of the framework, however, was limited to detection of only idle and busy states of a hydraulic
excavator. For the purpose of learning and classification of labor and equipment actions, the
concept of Bag-of-Video-Feature-Words model was extended into the construction domain
(Gong et al. 2011). This technique uses unsupervised learning for classification, and only
considers frequency of feature occurrence for classification. Another vision-based framework
was proposed by Rezazadeh Azar and McCabe (2012) for dirt-loading cycles in earthmoving
operations that depends on the location of equipment which requires the algorithm to be
modified for every new jobsite.

Construction workers’ activity recognition using vision-based systems has been also the subject
of some other studies. For example, 3D range image cameras were used for tracking and
surveillance of construction workers for safety and health monitoring (Gonsalves and Teizer
2009; Peddi et al. 2009). Gonsalves and Teizer (2009) indicated that if their proposed system is
used in conjunction with artificial neural network (ANN), results would be more robust for
prevention of fatal accidents and related health issues. In their study on construction workers’
unsafe actions, Han and Lee (2013) developed a framework for 3D human skeleton extraction
from video to detect unsafe predefined motion templates. All of these frameworks, although
presented successful results in their target domain, require installation of multiple cameras (up
to 8 in some cases), have short recognition distance (maximum of 4 meters for Kinect) and

Akhavian, Brito, Behzadan 4

require direct line of sight for implementation. Such shortcomings have served as a major
motivation to investigate alternative solutions that can potentially alleviate these problems.

Non-vision-based (a.k.a. senor-based) worker activity analysis has recently gained popularity
among CEM researchers. Cheng et al. (2013) used ultra-wide band (UWB) and Physiological
Status Monitors (PSMs) for productivity assessment. However, the LoD in recognizing the
activities was limited to identification of traveling, working, and idling states of workers and could
not provide further insight into identified activities. In another set of research studies aiming at
construction equipment activity analysis to support process visualization, remote monitoring and
planning, queueing analysis, and knowledge-based simulation input modeling, the authors
developed a framework by fusing data from ultra-wide band (UWB), payload, and orientation
(angle) sensors to build a spatio-temporal taxonomy-based reasoning scheme for activity
classification in heavy construction (Akhavian and Behzadan 2012, 2013, 2014).

As one of the first accelerometer-based activity recognition studies, Joshua and Varghese
(2011) developed a work sampling framework for bricklayers. The scope of that study, however,
was limited to only a single bricklayer in a controlled environment. Moreover, their proposed
framework used accelerometer as the sole source of motion data. Also, the necessity of
installing wired sensors on the worker’s body may introduce a constraint on the worker’s
freedom of movement. In another study, Ahn et al. (2013) used accelerometers to classify an
excavator operations into three modes of engine-off, idling, and working. Further decomposition
of these activities, however, was not explored in their study.

METHODOLOGY
In this study, data are collected using mobile phone accelerometer and gyroscope sensors.
Collected raw sensory data are segmented into windows containing certain number of data
points. Next, key statistical features are calculated within each window. Furthermore, each
segment is labeled based on the corresponding activity class performed at the time identified by
the timestamp of the collected data. In order to train a predictive model, five classifiers of
different types are used to recognize activities performed in the data collection experiments.
Figure 1 depicts the steps from data collection to activity recognition.

Figure 1. System architecture for activity recognition of construction resources using

mobile phones

Akhavian, Brito, Behzadan 5

Data Collection
Data collection is performed using commercially available data logger applications for iOS
and/or Android devices. Accelerometer sensors measure the acceleration of the device. The
reading can be in one, two, or all three axes of x, y, and z. The raw data is represented as a set
of vectors and returned together with a timestamp of the reading. Gyroscope is a sensor that
measures the rotation rate of the device by detecting the roll, pitch, and yaw motions of the
smartphone about the x, y, and z axes. Similar to accelerometer, readings are presented as
time-stamped vectors. When the mobile device is attached to construction equipment or
worker’s body involved in different activities, these two sensors generate different (and distinct)
patterns in their transmitted signals.

Data Preparation
A major step before transforming raw data into the input features for machine learning
algorithms is removing noise and accounting for missing data. When collecting data for a long
period of time, it is possible that the sensors temporarily freeze or fail to properly collect and
store data for fractions of a second to a few seconds and in return, compensate for the missing
data points by collecting data in a rate higher than the assigned frequency. In such cases, a
preprocessing technique to fill in for missing data points and removing redundant ones can help
insuring a continuous and orderly dataset. Also, since the raw data are often collected with a
high sampling rate, segmentation of the data helps in data compression and prepares data for
feature extraction (Khan et al. 2011). If segmentation is performed considering an overlap
between adjacent windows, it reduces the error caused by the transition state noise (Su et al.
2014). The length of the window size depends on the sampling frequency and the nature of
activities targeted for classification from which data is collected (Su et al. 2014).

Feature Extraction
Feature is an attribute of the raw data that should be calculated (Khan et al. 2011). In data
analytics applications, statistical time- and frequency-domain features generated in each
window are used as the input of the training process (Ravi et al. 2005). The ability to extract
appropriate features depends on the application domain and can steer the process of retaining
the relevant information. Most previous studies on activity recognition used almost the same set
of features for training the models and classification of activities (Shoaib et al. 2015).

Data Annotation
Following data segmentation and feature extraction, the corresponding activity class labels
should be assigned to each window. This serves as the ground truth for the learning algorithm
and can be retrieved from the video recorded at the time of the experiment.

Supervised Learning
In supervised learning classification, class labels discussed in Data Annotation (above) are
provided to the learning algorithms to generate a model or function that matches the input (i.e.
features) to the output (i.e. activity classes) (Ravi et al. 2005). The goal is to infer a function
using examples for which the class labels are known (i.e. training data). The performance of this
function is evaluated by measuring the accuracy in predicting the class labels of unseen
examples. Researchers have used different types of supervised classification methods for
activity recognition (Kim et al. 2013; Reddy et al. 2010; Sun et al. 2010). Details of the
supervised learning classification algorithms are discussed in the next Section.

Akhavian, Brito, Behzadan 6

Model Assessment
In order to determine the reliability of the trained model in detecting new examples of activity
classes, part of the training dataset is used for testing the model. It is recommended that the
test set is independent of the training set, meaning that the data that are used for testing have
not been among the training data. For example, randomly chosen 10% of the training data can
be left out so that the training is performed on the remaining 90% of the data. Assessment of the
model provides an opportunity for its fine-tuning so that certain variables (e.g. regularization
factor to prevent over-fitting in ANNs) in the algorithm can be revised to yield the best possible
model.

Activity Recognition
Once the model is trained and its parameters are finalized, it can be used for recognizing
activities for which it has been trained. While data is being collected to determine the activities
according to a trained classifier, such data can be stored in a dataset repository and be added
to the existing training data, so that the model is further trained with a richer training dataset.

DETAILS OF THE CLASSIFICATION ALGORITHMS

In the presented research, five different classification techniques are used in order to
systematically evaluate their performance in accurately detecting construction activities. In
particular, ANN, decision tree, K-nearest neighbor (KNN), logistic regression, and support vector
machine (SVM) are employed for classification. Decision tree, KNN, and SVM have been
previously used for activity recognition (Kose et al. 2012; Ravi et al. 2005; Yan et al. 2012) so
they are selected in this study to evaluate their performance for classifying construction
activities. However, ANN and logistic regression were examined to a much lesser extent
(Staudenmayer et al. 2009).

Artificial Neural Network (ANN)

An ANN trained based on the experiments’ data follows a simple pattern of one input, one
hidden, and one output layer. The number of input layer units is set to the number of extracted
features. The hidden layer consists of p=25 units. The number of units for the output layer is
equal to the number of activity classes, n in each case. Considering the large feature space and
in order to prevent over-fitting, regularization was used. Using a regularization parameter, the
magnitude of the model weights decreases, so that the model will not suffer from high variance
to fail to generalize to the new unseen examples (Haykin et al. 2009). The activation function
(i.e. hypothesis) used for minimizing the cost function in the training process is a Sigmoid
function shown in Equation 1,

( )= (1)

in which ( ) is the activation function (i.e. hypothesis), is a matrix of model weights (i.e.
parameters), and is the features matrix. In this study, in order to minimize the cost function,
the most commonly used ANN training method, namely feed-forward backpropagation is used.
Considering a set of randomly chosen initial weights, the backpropagation algorithm calculates
the error of the activation function in detecting the true classes and tries to minimize this error by
taking subsequent partial derivatives of the cost function with respect to the model weights
(Hassoun 1995).

Akhavian, Brito, Behzadan 7

Decision Tree
Decision tree is one of the most powerful yet simplest algorithms for classification (Bishop
2006). The decision tree method that is used in this research is classification and regression
tree (CART). CART partitions the training examples in the feature space into rectangle regions
(a.k.a. nodes) and assigns each class to a region. The process begins with all classes spread
over the feature space and examines all possible binary splits on every feature (Bishop 2006). A
split is selected if it has the best optimization criterion which is the Gini diversity index in this
research, as shown in Equation 2,

( )=1 (2)

in which is the Gini index, is the fraction of items labeled with value and is the number of
classes. The process of splitting is repeated iteratively for all nodes until they are pure. A node
is considered pure if it contains only observations of one class, implying a Gini index of zero, or
that there are fewer than 10 observations to split.

K-Nearest Neighbor (KNN)

Similar to the decision tree and unlike the ANN, KNN is a simple algorithm. Training examples
identified by their labels are spread over the feature space. A new example is assigned to a
class that is most common amongst its K nearest examples considering the Euclidean distance
that is used as the metric in this research, and as appears in Equation 3,

( ) ( ) ( ) ( ) ( ) ( )
= ( ) +( ) + +( )
(3)

in which is the Euclidean distance, is an existing example data point which has the least
distance with the new example, is the new example to be classified, and is the dimension
of the feature space.

Logistic Regression
Logistic regression is a type of regression problems in which the output is discretized for
classification (Hastie et al. 2009). Logistic regression seeks to form a hypothesis function that
maps the input (i.e. training data) to the output (i.e. class labels) by estimating the conditional
probability of an example belonging to class k given that the example actually belongs to the
class k. This is accomplished by minimizing a cost function using a hypothesis function and
correct classes to find the parameters of the mapping model (Hastie et al. 2009). The
hypothesis function used in this research is the same as the activation function introduced in
Equation 1 (the Sigmoid function) and thus the cost function to minimize is as shown in
Equation 4,

( )= () () () ()
[ log + 1 log 1 ] (4)

in which ( ) is the cost function, m is the number of training examples, ( ) is the ith training
example, and ( ) is the corresponding correct label. Once the cost function is minimized using
any mathematical method such as the Gradient Decent (Hastie et al. 2009) and parameters are
found, the hypothesis will be formed. In multi-class classification, the one-versus-all method is

Akhavian, Brito, Behzadan 8

used to determine if a new example belongs to the class k (Hastie et al. 2009). Therefore,
considering k classes, k hypothesis functions will be evaluated for each new example and the
one that results in the maximum hypothesis is selected.

Support Vector Machine (SVM)

Compared to decision tree and KNN, SVM is considered as a more powerful classification
algorithm. Although it has been widely used in vision-based pattern recognition and
classification problems, some researchers (Bishop 2006) used it for classifying daily activities
and thus its performance is also assessed in this research. In a nutshell, SVM tries to maximize
the margin around hyperplanes that separate different classes from each other. SVM can
benefit from a maximum margin hyperplane in a transformed feature space using kernel
function to create non-linear classifiers. The kernel function used for non-linear classification in
this research is Gaussian radial basis function (rbf) which has been successfully applied in the
past to activity recognition problems (Ravi et al. 2005). Further description of SVM models are
out of the scope of this study but can be found in (Bishop 2006).

VALIDATION EXPERIMENTS
In this Section, the description and details of two separate experiments conducted using
construction equipment and workers in order to validate the designed activity recognition
methods are provided.

Front-End Loader Activity Recognition

In this experiment, smartphones were placed inside the equipment cabin for data collection. At
any time, two smartphones were simultaneously used to guarantee the uninterrupted storage of
data. It must be noted that since data collection and feature extraction is done using tri-axial
data, results do not depend on the placement orientation of the data collection device.
Moreover, potential significant correlation between each pair of axes is reflected in three of the
extracted features, thus guaranteeing capturing any distinguishable feature related to the
placement orientation of the data collection devices. In order to fully automate the process of
data collection, low-cost near field communication (NFC) RFID smart tags were also used (Want
2006). NFC tags were glued to the device holder (i.e. suction cup attached to the side window of
the cabin) to automatically launch the data logger application once the smartphone was placed
in the holder. A JOHN DEERE 744J front-end loader was employed for data collection. All
experiment operations were fully videotaped for later activity annotation and labeling, and visual
validation. Figure 2 shows how data collection devices were mounted and secured inside the
target equipment cabin.

Figure 2. Smartphones mounted inside the front-end loader cabin

Akhavian, Brito, Behzadan 9

Data Collection
Data was collected using commercially available data logger applications for iOS and Android
devices. The sampling frequency was set at 100 Hz. Among different modes of data collected in
this study, it was observed that acceleration (i.e. vibration) values resulted from different
equipment motions had the highest degree of volatility. Several sensor manufacturers have
recommended that a bandwidth of 50 Hz be used for normal-speed vibration and tilt sensing
applications. Therefore, in this research, and considering the Nyquist criterion in signal
processing (Lyons et al. 2005), the sampling frequency was set at twice this value or 100 Hz.
This bandwidth guaranteed that no significant motion was overlooked and at the same time, the
volume of recorded data was not prohibitively large. Data was stored with comma separated
value (CSV) format for processing in Microsoft Excel. The logger applications provided time-
stamped data which facilitated the synchronization of data and video recordings. As mentioned
earlier, GPS data was not directly used in data mining processes employed in this study and
was only collected to demonstrate the potential of acquiring high accuracy positional data for
such context-aware applications. Figure 3 shows snapshots of the collected accelerometer,
gyroscope, and GPS data.

Figure 3. Snapshots of collected sensory data

Data Processing and Classification

Raw data must be first represented in terms of specific features over a window of certain data
points. In this research, mean, variance, peak, interquartile range (IQR), correlation, and root

Akhavian, Brito, Behzadan 10

mean error (RMS) are the statistical time-domain features that were extracted from data.
Moreover, signal energy was picked as the only frequency-domain feature since it had already
shown positive discrimination results in previous studies (Figo et al. 2010; Khan et al. 2011) for
context recognition using accelerometer data. These 7 features were extracted from both
accelerometer and gyroscope data corresponding to each of the x, y, and z axis. Since both
sensors return tri-axial values (x, y, z), a total of 42 (i.e. multiplication of 7 features from 2
sensors in 3 axes) features were extracted. The size of the window depends on the sampling
frequency and thus, varies for different applications. However, it should be selected in such a
way that no important action is missed. This can be achieved by overlapping consecutive
windows. Previous studies using accelerometer for context recognition have suggested a 50%
overlap between windows (Ahn et al. 2013; Darren Graham et al. 2005; DeVaul and Dunn
2001). Time-domain features can be extracted using statistical analysis. However, the
frequency-domain feature (i.e. signal energy) should be extracted from the frequency spectrum
which requires signal transformation. In this study, fast Fourier transform (FFT) was used to
convert the time-domain signal to the frequency-domain. In order to be computationally efficient,
FFT requires the number of data points in a window to be a power of 2. Data was initially
segmented into windows of 128 data points with 50% overlap. Therefore, given a sampling
frequency of 100 Hz, each window contained 1.28 seconds of the experiment data. A sensitivity
analysis presented in Section 4.5 provides more detail about the process of selecting the proper
window size. The entire data analysis process including feature extraction was performed in
Matlab.

Among all extracted features, there are some that may not add to the accuracy of the
classification. This might be due to the correlation that exists among the collected data and
consequently extracted features, since many actions result in a similar pattern in different
directions and/or different sensor types (i.e. accelerometer vs. gyroscope). Therefore, in order to
reduce the computational cost and time of the classification process, and increase its accuracy,
a subset of the discriminative features is selected by filtering out (removing) irrelevant or
redundant features (Pirttikangas et al. 2006). In this study, two filtering approaches are used:
ReliefF and Correlation-based Feature Selection (CFS). ReliefF is a weighting algorithm that
assigns a weight to each feature and ranks them according to how well their values distinguish
between the instances of the same and different classes that are near each other (Yu and Liu
2003). CFS is a subset search algorithm that applies a correlation measure to assess the
goodness of feature subsets based on the selected features that are highly correlated to the
class, yet uncorrelated to each other (Hall 1999).

Using CFS, irrelevant and redundant features were removed which yielded 12 features (out of
42). These features were than ranked by ReliefF using their weight factors. The first 12 features
selected by ReliefF were compared to those selected by CFS and the 7 common features in
both methods were ultimately chosen as the final feature space. Table 1 shows the selected
features by each filter as well as their intersection.

A learning algorithm can be supervised or unsupervised depending on whether or not different

classes are labeled for training. Although unsupervised methods can be employed for
equipment action recognition (Gong et al. 2011), supervised learning algorithms provide better
performance for this purpose (Golparvar-Fard et al. 2013). This is mainly due to the fact that
action classes of a piece of equipment consist of some classes with limited number of
instances. This creates an imbalanced set of classes (caused by large differences between the
number of instances in some classes) that can very likely lead to over-fitting in unsupervised

Akhavian, Brito, Behzadan 11

learning classification. Among several supervised learning methods those that follow more
complex algorithms may seem more accurate in classification.

Table 1. Selected features by CFS and ReliefF and their intersection (A: Accelerometer,
G: Gyroscope)
Filter Selected Features Common
Selected
Features
CFS A_mean_x, A_mean_y, A_mean_z, A_peak_x, A_iqr_y, G_mean_z
A_iqr_z, A_correlation_z, A_rms_z, G_mean_x, A_mean_x
G_mean_y, G_mean_z, G_variance_x G_mean_x
A_mean_y
A_mean_z
A_iqr_z
ReliefF G_mean_z, A_mean_x, G_mean_x, A_peak_z, A_peak_x
A_mean_y, A_correlation_y, A_correlation_x,
A_mean_z, A_iqr_z, A_peak_x, A_peak_y, G_rms_z

However, the choice of the learning algorithm is highly dependent on the characteristics and
volume of data. As a result, a “single” best classifier does not generally exist and each case
requires unique evaluation of the learning algorithm through cross validation (Goldberg 1989).
Therefore, a number of learning algorithms are tested in this research to compare their
performance in classifying actions using sensory data.

In this experiment, classification was performed by labeling the classes in different LoDs. The
first set of training and classification algorithms is applied to three classes namely Engine Off,
Idle, and Busy. Next, the Busy class is broken down into two subclasses of Moving and
Scooping, and Moving and Dumping, and so on. As stated earlier, for action classification, five
supervised learning methods were used: 1) Logistic Regression, 2) K-NN, 3) Decision Tree, 4)
ANN (feed-forward backpropagation), and 5) SVM. Using different classifiers reduces the
uncertainty of the results that might be related to the classification algorithm that each classifier
uses.

Results of Equipment Activity Recognition

For each LoD, five classifiers were trained. Training and testing were performed through
stratified 10-fold cross validations. In a k-fold cross validation the dataset is divided into k sets of
equal sizes, and classifiers are trained k times, each time they are tested on one of the k folds
and trained using the remaining k íIROGV0RUHRYHULQWKHVWUDWLILHGk-fold cross validation,
each fold contains almost the same proportions of classes as in the whole dataset. The mean
accuracy is reported as the accuracy of each class. Result of the classification performance for
each case (i.e. LoD) is presented in Table 2 in terms of overall classifier accuracy. As shown in
Table 2, ANNs had the best relative overall accuracy among all five classifiers in all the LoDs.
Moreover, although the 3-class level the accuracy gets to as high as 98.59%, the highest
accuracy for the 4-classe level is 81.30% which is less than that of the 5-class level, which is
86.09%.

Akhavian, Brito, Behzadan 12

Table 2. Overall accuracy of classifiers for each LoD
Classifier Accuracy
(%)
ANN 98.59
DT 97.40
3 KNN 97.65
Classes
LR 96.93
SVM 96.71
ANN 81.30
DT 81.21
4 KNN 80.51
Classes
LR 77.58
SVM 78.03
ANN 86.09
DT 73.78
5 KNN 84.20
Classes
LR 84.42
SVM 78.58

As stated earlier, construction equipment activity recognition has been previously explored
through vision-based technologies. Gong et al. (2011) reported an overall accuracy of 86.33%
for classification of three action classes of a backhoe. In a more recent study, Golparvar-Fard et
al. (2013) achieved 86.33% and 76.0% average accuracy for three and four action classes of an
excavator, respectively, and 98.33% average accuracy for three action classes of a dump truck.
Although the target construction equipment are different in each case and action categories
varies in these studies, the developed framework in this study that uses IMUs for the first time
for construction equipment action recognition shows promising results when compared to
existing vision-based systems that have been the subject of many research studies in the past
few years.

Construction Workers’ Activity Recognition

In a separate set of experiments, an outdoor construction workspace was created where
different activities were performed by multiple workers. These activities included sawing,
hammering, turning a wrench, loading sections into wheelbarrows, pushing loaded
wheelbarrows, dumping sections from wheelbarrows, and returning with empty wheelbarrows.
Activities were performed in 3 different categories in order to assess certain circumstances (as
described later) in the outcome of classification. A Commercially available armband was used to
secure a smartphone on the upper arm of the dominant hand of each worker. Recent research
on the selection of accelerometer location on bricklayer’s body for activity recognition has
shown that according to the body movement of the worker while performing different bricklaying
activities, among 15 potential locations for wearing an accelerometer, the lower left arm and the
upper right arm are the two best locations that yield the highest information gain (Joshua and
Varghese 2013). In this study, the lower arm was not selected for recognition of the activities of
interest since it precludes convenient execution of some activities. Consequently, the selection
of the upper arm was expected to provide accurate and consistent results compared to other
locations on the body. Figure 4 shows some snapshots of the construction workers wearing

Akhavian, Brito, Behzadan 13

mobile phones on their upper arms while performing assigned activities in the experiments
conducted in this research.

Figure 4. Snapshot of construction worker data collection experiments using mobile

phones

Data Collection and Logging

Technical details of the data collection in terms of the mobile applications, sensors used
(accelerometer and gyroscope), and sapling frequency (100 Hz) remained the same as the
equipment data collection experiment described in the previous Section. Construction workers
were asked to do their assigned activities for a certain period of time while waiting for a few
seconds in between each instance of their assigned activities. Each activity was performed by
two subjects for later user-independent evaluations. One subject performed only sawing. In this
case, the goal of activity recognition was to differentiate between the time they were sawing and
the time they were not sawing. Another subject performed hammering and turning a wrench. In
this case, the activity recognition was intended to detect the time they were hammering, the time
they were turning the wrench, and the time there were not doing any of the two activities.
Finally, the last subject was responsible for pushing the wheelbarrow and loading/unloading the
sections. Therefore, the activities to be recognized in this case were loading sections into a
wheelbarrow, pushing a loaded wheelbarrow, dumping sections from a wheelbarrow, and
returning with an empty wheelbarrow. Also, the entire experiment was videotaped for data
annotation.

Data Processing and Classification

Classifications are conducted in 3 activity categories. Category 1 includes only one
distinguishable activity, sawing, to assess the performance of the classifiers in detecting value-
adding versus non-value-adding instances in a pool of accelerometer and gyroscope data. The
result of classification in this category contributes to the overall performance of the developed
activity recognition system when used for productivity measurement. In this category, sawing is
categorized against idling. Category 2 includes instances of consecutive hammering and turning
a wrench as two adjacent activities with almost similar corresponding movements of the
worker’s arm. These two activities are also classified against idling to assess the accuracy of
the developed activity recognition system in differentiating between activities that produce
similar physical body motions. Finally, in category 3, four activities that produce different
distinguishable body movements are categorized. These activities include loading sections into

Akhavian, Brito, Behzadan 14

a wheelbarrow, pushing a loaded wheelbarrow, dumping sections from a wheelbarrow, and
returning an empty wheelbarrow, that were also categorized against idling. Similar to the
equipment experiments, 128 data points were segmented in one window and 50% overlap for
the adjacent windows is considered. In this experiments, the extracted features are as follow:
mean, maximum, minimum, variance, RMS, IQR, and correlation between each two pairs of
axes comprised the seven time-domain features and spectral energy and entropy were the two
frequency domain features. Considering data collection in three axes of the two sensors and
nine independent features extracted per sensor per axis, a total of 54 features were extracted
from all collected data.

Results of Workers’ Activity Recognition

The performance of the classifiers is assessed in two ways. First, the training accuracy of each
classifier was calculated. This means that all collected data points were used for both training
and testing which provided an overall insight into the performance of a host of classification
algorithms in recognizing construction worker activities using accelerometer and gyroscope
data. Next, a more robust approach in evaluation of classifiers was adopted. In particular, 10-
fold stratified cross validation was used and the results of the 10 replications of the training and
testing were averaged out to report the overall accuracy.

The classification accuracies are reported for 3 activity categories listed in Table 3. The
following activity codes are used in reporting the results: in the first category, activity sawing
(SW) and being idle (ID) are classified. In the second category, activities hammering (HM),
turning a wrench (TW), and being idle (ID) are classified. Finally, in the third category
classification is performed on the activities loading sections into wheelbarrow (LW), pushing a
loaded wheelbarrow (PW), dumping sections from wheelbarrow (DS), returning an empty
wheelbarrow (RW), and being idle (ID). Table 3 shows the results of training and 10-fold cross
validation classification accuracy the subject performing activities of category 1.

Table 3. Category 1 activities classification accuracy

Decision Logistic
Category 1 ANN KNN SVM
Tree Regression
Training 100.00 99.36 98.08 98.72 98.19
10-fold CV 96.77 96.06 95.95 96.05 96.91

According to Table 3, over 99% training accuracy was achieved in category 1 using ANN
classifier. This confirms the hypothesis that IMU data pertaining to a single activity performed by
different workers contain highly distinguishable patterns. However, training accuracy is not an
appropriate measure to assess the ability of using such data for new instances of the same
activity. Nevertheless, the stratified 10-fold cross validation results confirm that regardless of the
nature of classification algorithm, a single activity can be recognized with over 96% accuracy
using all five classifiers.

Since it is very likely that a construction worker performs more than one highly distinguishable
activity at a time, activities performed in category 2 are designed such that they produce almost
the same physical arm movement. Table 4 shows the training and 10-fold cross validation
classification accuracy results of both subjects performing activities of category 2.

Akhavian, Brito, Behzadan 15

Table 4. Category 2 activities classification accuracy
Decision Logistic
Category 2 ANN KNN SVM
Tree Regression
Training 98.62 97.07 93.81 88.14 87.28
10-fold CV 93.19 85.83 87.80 86.42 85.34

Similar to category 1, the training accuracies are high particularly for the ANN classifier and the
decision tree. While CART decision trees are not very stable and a small change in the training
data can change the result drastically as appears in the outcome of the 10-fold cross validation,
ANN presents an average of around 90% accuracy for both subjects. This is while all other
classification methods performed almost the same with a slight superiority of KNN relative to the
other algorithms. This result is particularly important considering the fact that the two activities in
category 2 (i.e. hammering and turning a wrench) produce almost similar physical movements in
a worker’s arm.

In the third category, a mixture of different distinguishable activities performed by typical

construction workers is included to evaluate the performance of the developed activity
recognition system in recognizing them. Table 5 shows the training and 10-fold cross validation
classification accuracy results of both subjects performing activities of category 3.

Table 5. Category 3 activities classification accuracy

Decision Logistic
Category 3 ANN KNN SVM
Tree Regression
Training 94.80 97.11 95.75 90.37 85.82
10-fold CV 92.01 87.95 90.75 90.75 84.42

According to Table 5, again decision tree gained a high accuracy in training while as expected;
its performance is not the same in 10-fold cross validation evaluation. However, except for the
decision tree and SVM, all other classifiers, namely ANN, KNN, and logistic regression resulted
in around 90% average accuracy for both subjects. Similar to the other two categories, the
feedforward back-propagation implementation of the ANN resulted in the highest accuracy
among all.

SUMMARY AND CONCLUSIONS

The goal of the research presented in this paper was to investigate the prospect of using built-in
smartphone sensors as ubiquitous multi-modal data collection and transmission nodes in order
to detect detailed construction equipment activities. In spite of its importance, automated
recognition of construction worker and equipment activities on the jobsite has not been given
due attention in CEM literature. The discovered process-level knowledge can provide a solid
basis for different applications such as productivity improvement, safety management, and fuel
use and emission monitoring and control. In addition, this methodology can serve as a basis for
activity duration extraction for the purpose of construction simulation input modeling.

In order to study construction equipment activity recognition, a case study of front-end loader
was used to describe the methodology for action recognition and evaluate the performance of
the developed system. In doing so, several important technical details such as selection of
discriminating features to extract different LoDs for classification, and choice of classifier to be

Akhavian, Brito, Behzadan 16

trained were investigated. Results indicated that different equipment actions generate distinct
data patterns (i.e. signatures) in accelerometer and gyroscope data. In the 3-class level a high
accuracy of as 98.59% can be achieved using ANN. With the same choice of classifier, 81.30%
and 86.09% accuracy was achieved for 4- and 5-class levels.

In case of workers’ activity recognition, built-in sensors of ubiquitous smartphones have been
employed to assess the potential of wearable systems for activity recognition. Smartphones
were affixed to workers’ upper arms using armbands, and accelerometer and gyroscope data
were collected from multiple construction workers involved in different types of activities. The
high levels of training accuracies achieved by testing several classification algorithms including
ANN, decision tree, KNN, logistic regression, and SVM confirmed the hypothesis that different
classification algorithms can detect patterns that exist within signals produced by IMUs while
different construction tasks are performed. Through 10-fold stratified cross validation, algorithms
were trained with 90% of the available data and the trained models were tested on the
remaining 10%. In different categories of activities, around and over 90% accuracy was
achieved. This promising result indicates that built-in smartphone sensors have high potential to
be used as integrated data collection and activity recognition platforms in construction
environments.

FUTURE WORK
A potential direction for future work in this research will be to explore whether the results
achieved so far can be used for automatically extracting process knowledge such as activity
durations and precedence logic for the purpose of ubiquitously updating and maintaining
simulation models corresponding to field operations. In addition, another branch of future work
rooted in the current research is automated identification of unsafe workers’ postures in
physically demanding construction activities. Work-related Musculoskeletal Disorder (WMSD),
back, knee, and shoulders injuries are among the most common injuries that can be prevented
or reduced by complying with Occupational Safety and Health Administration (OSHA) or the
National Institute for Occupational Safety and Health (NIOSH) standards and rules (NIOSH
2015; OSHA 1990).

ACKNOWLEDGMENTS
The authors would like to acknowledge the help and support of Hubbard Construction, for
providing access to active construction facilities for equipment data collection experiments. Any
opinions, findings, conclusions, and recommendations expressed in this paper are those of the
authors and do not necessarily reflect the views of Hubbard Construction. Also, the authors are
grateful to the construction engineering research laboratory students who assisted in the data
collection process.

REFERENCES
Ahn, C. R., S. Lee, and F. Peña-Mora. 2013. "The Application of Low-Cost Accelerometers for
Measuring the Operational Efficiency of a Construction Equipment Fleet." Journal of
Computing in Civil Engineering.
Akhavian, R., and A. H. Behzadan. 2012. "An Integrated Data Collection and Analysis
Framework for Remote Monitoring and Planning of Construction Operations." Advanced
Engineering Informatics 26 (4):749–761.

Akhavian, Brito, Behzadan 17

Akhavian, R., and A. H. Behzadan. 2013. "Knowledge-Based Simulation Modeling of
Construction Fleet Operations Using Multimodal-Process Data Mining." Journal of
Construction Engineering and Management 139 (11):04013021.
Akhavian, R., and A. H. Behzadan. 2014. "Evaluation of Queuing Systems for Knowledge-
Based Simulation of Construction Processes." Automation in Construction 47:37-49.
Akhavian, R., and A. H. Behzadan. 2015. "Construction Equipment Activity Recognition for
Simulation Input Modeling Using Mobile Sensors and Machine Learning Classifiers."
Advanced Engineering Informatics.
Bishop, C. M. 2006. Pattern Recognition and Machine Learning. Vol. 4: springer New York
Brilakis, I., M. W. Park, and G. Jog. 2011. "Automated Vision Tracking of Project Related
Entities." Advanced Engineering Informatics 25 (4):713-724.
Cheng, T., J. Teizer, G. C. Migliaccio, and U. C. Gatti. 2013. "Automated Task-Level Activity
Analysis through Fusion of Real Time Location Sensors and Worker's Thoracic Posture
Data." Automation in Construction 29:24-39.
Darren Graham, L., S. D. Smith, and P. Dunlop. 2005. "Lognormal Distribution Provides an
Optimum Representation of the Concrete Delivery and Placement Process." Journal of
construction engineering and management 131 (2):230-238.
Dernbach, S., B. Das, N. C. Krishnan, B. L. Thomas, and D. J. Cook. 2012. Simple and
Complex Activity Recognition through Smart Phones. Intelligent Environments (IE), 2012
8th International Conference on.
DeVaul, R. W., and S. Dunn. 2001. "Real-Time Motion Classification for Wearable Computing
Applications." 2001, project paper,
http://www.media.mit.edu/wearables/mithril/realtime.pdf.
Figo, D., P. C. Diniz, D. R. Ferreira, and J. M. Cardoso. 2010. "Preprocessing Techniques for
Context Recognition from Accelerometer Data." Personal and Ubiquitous Computing 14
(7):645-662.
Foerster, F., M. Smeja, and J. Fahrenberg. 1999. "Detection of Posture and Motion by
Accelerometry: A Validation Study in Ambulatory Monitoring." Computers in Human
Behavior 15 (5):571-583.
Frank, K., M. J. V. Nadales, P. Robertson, and M. Angermann. 2010. Reliable Real-Time
Recognition of Motion Related Human Activities Using Mems Inertial Sensors.
Proceedings of the 23rd International Technical Meeting of the Satellite Division of the
Institute of Navigation.
Goldberg, D. E. 1989. Genetic Algorithms in Search, Optimization, and Machine Learning. Vol.
412: Addison-wesley Reading Menlo Park
Golparvar-Fard, M., A. Heydarian, and J. C. Niebles. 2013. "Vision-Based Action Recognition of
Earthmoving Equipment Using Spatio-Temporal Features and Support Vector Machine
Classifiers." Advanced Engineering Informatics 27 (4):652-663.
Gong, J., C. H. Caldas, and C. Gordon. 2011. "Learning and Classifying Actions of Construction
Workers and Equipment Using Bag-of-Video-Feature-Words and Bayesian Network
Models." Advanced Engineering Informatics 25 (4):771-782.
Gonsalves, R., and J. Teizer. 2009. Human Motion Analysis Using 3d Range Imaging
Technology. Proceedings of the 26th International Symposium on Automation and
Robotics in Construction, at Austin, Texas.
Gupta, P., and T. Dallas. 2014. "Feature Selection and Activity Recognition System Using a
Single Tri-Axial Accelerometer."
Hall, M. A. 1999. "Correlation-Based Feature Selection for Machine Learning." The University of
Waikato.
Han, S., and S. Lee. 2013. "A Vision-Based Motion Capture and Recognition Framework for
Behavior-Based Safety Management." Automation in Construction 35:131-141.
Hassoun, M. H. 1995. Fundamentals of Artificial Neural Networks: MIT press

Akhavian, Brito, Behzadan 18

Hastie, T., R. Tibshirani, J. Friedman, T. Hastie, J. Friedman, and R. Tibshirani. 2009. The
Elements of Statistical Learning. Vol. 2: Springer
Haykin, S. S., S. S. Haykin, S. S. Haykin, and S. S. Haykin. 2009. Neural Networks and
Learning Machines. Vol. 3: Pearson Education Upper Saddle River
Hemminki, S., P. Nurmi, and S. Tarkoma. 2013. Accelerometer-Based Transportation Mode
Detection on Smartphones. Proceedings of the 11th ACM Conference on Embedded
Networked Sensor Systems.
Irizarry, J., M. Gheisari, and B. N. Walker. 2012. "Usability Assessment of Drone Technology as
Safety Inspection Tools." Journal of Information Technology in Construction (ITcon)
17:194-212.
Joshua, L., and K. Varghese. 2011. "Accelerometer-Based Activity Recognition in Construction."
Journal of Computing in Civil Engineering 25 (5):370-379.
Joshua, L., and K. Varghese. 2013. "Selection of Accelerometer Location on Bricklayers Using
Decision Trees." Computer-Aided Civil and Infrastructure Engineering 28 (5):372-388.
Khan, M., S. I. Ahamed, M. Rahman, and R. O. Smith. 2011. A Feature Extraction Method for
Realtime Human Activity Recognition on Cell Phones. Proceedings of 3rd International
Symposium on Quality of Life Technology (isQoLT 2011). Toronto, Canada.
Kim, T.-S., J.-H. Cho, and J. T. Kim. 2013. "Mobile Motion Sensor-Based Human Activity
Recognition and Energy Expenditure Estimation in Building Environments." In
Sustainability in Energy and Buildings, 987-993. Springer.
Kose, M., O. D. Incel, and C. Ersoy. 2012. Online Human Activity Recognition on Smart
Phones. Workshop on Mobile Sensing: From Smartphones and Wearables to Big Data.
Krassenstein, E. 2015. "Renderings & Details Unveiled for Extraordinary 3d Printed Home in
New York." http://3dprint.com/. Accessed May 8, 2015. http://3dprint.com/59753/d-
shape-3d-printed-house-ny/.
Kwapisz, J. R., G. M. Weiss, and S. A. Moore. 2011. "Activity Recognition Using Cell Phone
Accelerometers." ACM SigKDD Explorations Newsletter 12 (2):74-82.
Lara, O. D., and M. A. Labrador. 2013. "A Survey on Human Activity Recognition Using
Wearable Sensors." Communications Surveys & Tutorials, IEEE 15 (3):1192-1209.
Li, Q., J. A. Stankovic, M. A. Hanson, A. T. Barth, J. Lach, and G. Zhou. 2009. Accurate, Fast
Fall Detection Using Gyroscopes and Accelerometer-Derived Posture Information.
Wearable and Implantable Body Sensor Networks, 2009. BSN 2009. Sixth International
Workshop on.
Lyons, G., K. Culhane, D. Hilton, P. Grace, and D. Lyons. 2005. "A Description of an
Accelerometer-Based Mobility Monitoring Technique." Medical engineering & physics 27
(6):497-504.
NIOSH. 2015. "Safety & Prevention." Accessed March 30, 2015.
http://www.cdc.gov/niosh/topics/safety.html.
OSHA. 1990. "Excavation Final Rule." Safety and Health Administration,
Peddi, A., L. Huan, Y. Bai, and S. Kim. 2009. Development of Human Pose Analyzing
Algorithms for the Determination of Construction Productivity in Real-Time. Construction
Research Congress at Seattle, WA.
Pirttikangas, S., K. Fujinami, and T. Nakajima. 2006. "Feature Selection and Activity
Recognition from Wearable Sensors." In Ubiquitous Computing Systems, 516-527.
Springer.
Ravi, N., N. Dandekar, P. Mysore, and M. L. Littman. 2005. Activity Recognition from
Accelerometer Data. AAAI.
Reddy, S., M. Mun, J. Burke, D. Estrin, M. Hansen, and M. Srivastava. 2010. "Using Mobile
Phones to Determine Transportation Modes." ACM Transactions on Sensor Networks
(TOSN) 6 (2):13.

Akhavian, Brito, Behzadan 19

Rezazadeh Azar, E., and B. McCabe. 2012. Vision-Based Recognition of Dirt Loading Cycles in
Construction Sites.
Shoaib, M., S. Bosch, O. D. Incel, H. Scholten, and P. J. Havinga. 2015. "A Survey of Online
Activity Recognition Using Mobile Phones." Sensors 15 (1):2059-2085.
Staudenmayer, J., D. Pober, S. Crouter, D. Bassett, and P. Freedson. 2009. "An Artificial Neural
Network to Estimate Physical Activity Energy Expenditure and Identify Physical Activity
Type from an Accelerometer." Journal of Applied Physiology 107 (4):1300-1307.
Su, X., H. Tong, and P. Ji. 2014. "Activity Recognition with Smartphone Sensors." Tsinghua
Science and Technology 19 (3):235-249.
Sun, L., D. Zhang, B. Li, B. Guo, and S. Li. 2010. "Activity Recognition on an Accelerometer
Embedded Mobile Phone with Varying Positions and Orientations." In Ubiquitous
Intelligence and Computing, 548-562. Springer.
Wang, S., C. Chen, and J. Ma. 2010. Accelerometer Based Transportation Mode Recognition
on Mobile Phones. Wearable Computing Systems (APWCS), 2010 Asia-Pacific
Conference on.
Want, R. 2006. "An Introduction to Rfid Technology." Pervasive Computing, IEEE 5 (1):25-33.
Wu, W., S. Dasgupta, E. E. Ramirez, C. Peterson, and G. J. Norman. 2012. "Classification
Accuracies of Physical Activities Using Smartphone Motion Sensors." Journal of medical
Internet research 14 (5):e130.
Yan, Z., V. Subbaraju, D. Chakraborty, A. Misra, and K. Aberer. 2012. Energy-Efficient
Continuous Activity Recognition on Mobile Phones: An Activity-Adaptive Approach.
Wearable Computers (ISWC), 2012 16th International Symposium on.
Yu, L., and H. Liu. 2003. Feature Selection for High-Dimensional Data: A Fast Correlation-
Based Filter Solution. ICML.
Zheng, Y., Y. Chen, Q. Li, X. Xie, and W.-Y. Ma. 2010. "Understanding Transportation Modes
Based on Gps Data for Web Applications." ACM Transactions on the Web (TWEB) 4
(1):1.
Zou, J., and H. Kim. 2007. "Using Hue, Saturation, and Value Color Space for Hydraulic
Excavator Idle Time Analysis." Journal of computing in civil engineering 21 (4):238-246.

Akhavian, Brito, Behzadan 20

Terrestrial Laser Scanning Roughness Assessments
for Infrastructure

Ahmad Alhasan
Department of Civil, Construction and Environmental Engineering
Iowa State University
394 Town Engineering Building
Ames, IA 50011-3232
Email: [email protected]

David J. White, Ph.D., P.E.

Department of Civil, Construction and Environmental Engineering
Iowa State University
422 Town Engineering Building
Ames, IA USA 50011-3232
Email: [email protected]

ABSTRACT
Road roughness is a key parameter for controlling pavement construction processes and for
assessing ride quality of both paved and unpaved roads. This paper describes algorithms used
in processing three-dimensional (3D) stationary terrestrial laser scanning (STLS) point clouds to
obtain surface maps of point wise indices that characterize pavement roughness. The backbone
of the analysis is a quarter-car model simulation over a spatial 3D mesh grid representing the
pavement surface. Two case studies are presented, and results show high spatial variability in
the roughness indices both longitudinally and transversely (i.e., different wheel path positions). It
is proposed that road roughness characterization using a spatial framework provides more
details on the severity and location of roughness features compared to the one-dimensional
methods. This paper describes approaches that provide an algorithmic framework for others
collecting similar STLS 3D spatial data to be used in advanced road roughness characterization.

Keywords: 3D laser Scanning—road roughness—algorithms

INTRODUCTION
Road surface roughness increases vehicle operation and travel delay costs (Gao and Zhang
2013; Ouyang and Madanat 2004); reduces vehicle durability (Bogsjö and Rychlik 2009; Oijer
and Edlund 2004); and reduces ride quality and structural performance (Al-Omari and Darter
1994). Structural performance diminishes faster on rough roads because rough features
increase dynamic stresses that accelerate structural deterioration (Lin et al. 2003). Accurate
evaluation of pavement roughness levels and modes is a key factor in optimizing maintenance
decisions (Chootinan et al. 2006; Kilpeläinen et al. 2011; Lamptey et al. 2008) and a leading
indicator in construction quality assurance/quality control (QC/QA).

In 1986, the International Roughness Index (IRI) was introduced as a time stable pavement
roughness measurement (Sayers et al. 1986; Sayers et al. 1986). Since then IRI has been
widely used because it empirically correlates with ride quality and vehicle operating costs (Gao

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
and Zhang 2013; Ouyang and Madanat 2004; Tsunokawa and Schofer 1994). IRI is calculated
by mathematically simulating the quarter-car dynamics and accumulating the quarter-car
suspension response induced by variations in a vertical profile. Current teste method for
measuring longitudinal profiles using an inertial profiler (ASTM E950 / E950M-09) is most suited
for collecting data in one or two or even few profiles, however it is hard to synchronize these
profiles to test the transverse variability in elevation at a specific point. Karamihas et al. (1999)
investigated the variability in the IRI values for different profiles across the road, and reported
that two profiles across the lane are not representative of the entire lane and that drivers typical
wander laterally within a range of 50 cm (20 in).

Another limitation when reporting summary indices with fixed analysis interval, is the ability to
detect localized features, Swan and Karamihas (2003) proposed reporting IRI continuously by
applying a moving average to the suspension response. Studies have shown that local features
affect rider comfort, pavement stresses, and cause most vehicle fatigue damage (Bogsjö and
Rychlik 2009; Kuo et al. 2011; Oijer and Edlund 2004; Steinwolf et al. 2002).

Unlike paved roads, there is a lack of a common set of criteria to evaluate unpaved roads, many
local agencies use visual inspection to estimate an IRI value (Archondo-Callao 1999; Namur
and de Solminihac 2009; Walker et al. 2002). Some agencies combine visual inspection with
direct measurement of defects (e.g., pothole depth, corrugation spacing) (Soria and Fontenele
2003; Woll et al. 2008). Several studies have pointed out the importance of precise assessment
of unpaved road conditions using indirect data acquisition methods such as unmanned aerial
vehicles (UAV), ground penetrating radar (GPR), and accelerometers to help transportation
agencies decide whether to maintain or upgrade these roads (Berthelot et al. 2008; Brown et al.
2003; Zhang 2009; Zhang and Elaksher 2012).

Recent developments in laser scanning techniques and light detection and ranging (LIDAR)
sensing have motivated researchers and practitioners to adopt these technologies due to the
accurate and rich data measurements. Recent studies (Fu et al. 2013; Tsai et al. 2010; Zalama
et al. 2011) have demonstrated the effectiveness of laser scanning and LIDAR in identifying
geometrical features of interest (e.g., cracks, bumps). Also, recent studies have investigated the
applicability of using stationary three dimensional (3D) laser scanning techniques in obtaining
IRI by selectively extracting track profiles (Chang and Chang 2006) or by analyzing the surface
to develop spatial roughness maps (Alhasan et al. 2015).

This study will introduce an overview of stationary laser scanning approach and the practical
needs for applications in road roughness assessment. Also a discussion of a procedure and
associated algorithms that can be used in processing 3D point clouds obtained for paved and
unpaved sections, to obtain spatial surface maps of rectified slopes (RS) and IRI values across
road sections. A brief review of frequency based analysis (i.e. Fast Fourier transform, FFT and
Continues wavelet transform) will be introduced as well. Frequency based approaches can
reveal the sources of road roughness. The backbone of the analysis approaches described
herein is a quarter-car model.

DATA COLLECTION USING STATIONARY LASER SCANNER

LIDAR systems measure information (spatial coordinates and color) of a 3D space and store the
information in a 3D point cloud. The term LIDAR is generic, and includes airborne laser
scanning technologies, mobile scanners mounted on vehicles, and stationary laser scanners or
stationary terrestrial laser scanners (STLS), where the laser scanner is fixed at a station with

Alhasan and White 22

known coordinates and based on the distance between the scanner and the detected points, a
geospatially referenced 3D point cloud is constructed. Trimble CX 3D STLS system was used in
two case studies to acquire 3D laser scans. The position accuracy of a single point is 4.5 mm at
30 m and drops to 7.3 mm at 50 m. The distance accuracy is 1.2 mm at 30 m and drops to 2
mm at 50 m. Figure 1 shows the scanner set-up.

Figure 1. Trimble CX 3D laser scanner set-up.

Scanning process starWVE\DFTXLULQJIXOOVFDQWKDWFRYHUVWKHࡈ[ࡈYLHZ7KLVVFDQ

produces a mother file that includes common targets (Figure 2) to be used for registration in the
post processing. Area scans are then conducted to acquire denser point clouds for the areas of
interest. The size of the targets affect the density of the full scan, where smaller targets require
high density full scans to capture them. However, the density of the area scans can vary
depending on the application. In the two case studies, criterion specified to analyze roads
roughness from the point cloud was a maximum spacing of 100 mm in both spatial directions
(longitudinal and transverse) between two consecutive points in the region of interest. By
several trials it was found that a density of 35 mm at 100,000 mm for the area scans produced
sufficiently dense clouds to satisfy the 100 mm spacing criterion.

Alhasan and White 23

 Figure 2. Reference point (a) spherical targets and (b) flat black and white target.

After acquiring the data, the scan should be registered in a specialized software. Many activities
can be performed using software packages, such as registration, fitting geometries. Registration
of clouds is done by identifying common targets (spheres or flat targets as shown in Figure 2)
appearing in the full scans of each consecutive station that share a common spatial domain with
other stations. These targets are used as benchmarks to geospatially reference scans by
matching the common target locations in each scan, and thus stitching the scans to produce a
full 3D cloud. Figure 3 shows examples of the registered point clouds. The variation in color
indicates the material reflectivity.

Figure 3. Registered point cloud.

After registration, the point clouds are cleaned of unnecessary data points, where the sections
of interest should be separated from areas beyond the edges of the sections and noise from any
passing vehicles that might appear in the scans. The final data points left after cleaning can be

Alhasan and White 24

exported in ASCII file format; these files contain the x, y, and z coordinates of the points in the
section under consideration.

DATA PROCESSING
Visioning Algorithms
To use the road surface in simulations developed algorithms require a uniformly spaced grid the
longitudinal direction. To achieve that a mesh grid is formed with grid elements that have a
predefined x and y edge dimensions. The grid centre elevation is calculated as the average of
all cloud points falling in that grid region. All points are rotated and translated to a local
coordinate system corresponding to the longitudinal and transvers axes (Zalama et al. 2011).
Transformation for processing along horizontal curves can be achieved by constructing a
curvilinear local coordinate system; however, for simple geometries without curves the point
cloud is rotated globally, where the x axis corresponds to the longitudinal direction and the y
axis to the transverse direction. Vertical slopes are corrected by subtracting the z elevation
along a quadratic fit from the z coordinate of the corresponding transformed points (Alhasan et
al. 2015). Figure 4 shows a pavement section point cloud data after processing in the visioning
algorithm.

Figure 4. Point cloud data after processing in the visioning algorithm

for a pavement section.

Roughness Evaluation Algorithms

Evaluation of roughness described herein is based on the responses of the quarter-car model
described in (ASTM E1926-08). Figure 5 presents a schematic of the quarter-car model. Where
Ms and Mu are the sprung and unsprung masses respectively, ks and kt are the suspension and
tire spring coefficients respectively, and cs is the suspension damping rate.

Alhasan and White 25

 Figure 5. Quarter-car model.

The dynamics of the system can be described by four first-order differential equations presented
in a matrix form (Sayers and Karamihas 1996) (Equations 1—4):

X AX  B h ps
(1)
Where;

X >z s z s z u z u@
T
(2)

ª 0 1 0 0 º
« c c »
« k2 k2 »
A « 0 0 0 1 »
« k2 c k1  k 2 c»
« P   »
¬ P P P¼ (3)

B >0 0 0 k1 P @
T
(4)

Where hps is the elevation of the profile after applying the moving average smoother;
zs and
zu
z z
are the elevations of sprung and unsprung masses; s and u are elevation time derivatives of
sprung and unsprung masses; k1 is the tire spring coefficient divided by the sprung mass, k2 is

Alhasan and White 26

the suspension spring coefficient divided by the sprung mass, c is the suspension damping rate
divided by the sprung mass, and P is the ratio of unsprung to sprung mass. Two approaches
can be followed to solve this system, a finite difference solution that solves the system in
stepwise fashion and a frequency domain solution (Fourier analysis). The outcome of the
solution is the rectified slope profile, which is the time rate at which the sprung mass is moving
in the z direction relative to the unsprung mass. This index reflects the suspension rate for a
quarter-car model.

Finite Difference Algorithms

The differential equation described in Equation 1 can be solved in state-space form as shown
Equation 5 (Murray et al. 1994; Sayers and Karamihas 1996). FORTRAN code is provided in
(ASTM E1926-08) to solve this system. The code can be translated to MATLAB code and used
to simulate the quarter-car model over each longitudinal strip in the grid developed from the
point cloud. A longitudinal strip is defined as a sequence of grid elements in the x direction
having one y coordinate. The longitudinal strips are used as an input profile for simulations. This
approach leads to a surface roughness map where the rectified slope is a function of the
variable x and the fixed coordinate y. Because each profile is simulated independently,
transverse effects such as rolling and pitch are not included.

X i S X i 1  P h ps (5)

Where;
A'x v
S e
P A1 S  I B (6)
X, A, hps, and B are predefined in Equations 1—4, 'x is the spacing between profile points, v is
the assumed speed of the model, and I is the identity matrix. The finite difference algorithm is
implemented in many commercial packages and proved efficiency when used in analysing long
profiles. However, the algorithm suffers from long memory effects, where the rectified slope at a
point is affected by the elevation of previous points back to several meters, reaching 10 meters
in some cases, and thus the starting point affects the results when analysing short profiles
(Swan and Karamihas 2003). The critical length of the profile depends on several factors, the
smoothness of the profile to be analysed and the sampling frequency of the data acquisition
system.

Frequency domain algorithms

Frequency analysis is another approach to solve the described dynamic system. This approach
namely depends on transforming the state equations from spatial to frequency or quasi-
frequency domain. The analytical description of quarter-car model is best explained in terms of
random signal analysis, where fast Fourier transformation (FFT) can be a helpful tool for such
analysis. The basic assumption in this approach is that any profile can be decomposed into
building sinusoids, for infinite precision the signal should be decomposed to an infinite number
of sinusoids, however only finite number can be used to describe a signal. Results for the FFT
analysis can be characterized by examining the height amplitude versus spatial frequency plots
(Alhasan et al. 2015), or by applying a band-pass filter (Liu and Herman 1999) that
characterizes the analytical solution of the quarter car model in frequency domain, this filter
results in rectified slope profile.

Alhasan and White 27

Wavelet analysis techniques were developed independently in different fields (i.e., pure
mathematics, physics, and engineering) to overcome the time-frequency resolution issue in
Fourier analysis (Boggess and Narcowich 2009; Daubechies 1992). This issue results from the
assumption that sines and cosines, which are infinitely periodic functions, are building blocks for
any function. This assumption induces uncertainty in the analysis, where high resolution cannot
be achieved simultaneously in both frequency and space domains. Wavelets, which can be
thought of as wave pulses that translate in the spatial domain and can change size (i.e., dilate
or shrink), are used as building blocks to overcome the time-frequency resolution issue.
Wavelets come in families, and for each family there is a wavelet, the “mother wavelet,” and a
scaling function, a “father wavelet,” although some families do not include scaling functions.
Wavelet dispersion is controlled by a scale factor ‘a’, where larger scale factors correspond to
dilated waves and smaller scale factors correspond to shrunken waves.

Wavelet transform projects (transforms) the profile on a wavelet function at different scales
(layers of details) to result in wavelet coefficients describing the correlation between the signal
and the wavelet function. This decomposition allows examining the location of features with
certain frequency bands with known effect on vehicle response, and thus provides a valuable
tool to determine localized features (Alhasan et al. 2015).

CASE STUDIES
Road roughness of two road sections was evaluated, the sections include a 55 m long rural
unpaved road and a 58.8 m long HMA overlay over a jointed plain concrete pavement (JPCP)
pavement. Figures 6a and 6b show the rectified slope map for rural road and the paved road
respectively.

For the unpaved road the left lane extends between stations 0 and 3 in the transverse distance,
and the right lane extends between stations 0 and -3. This map reveals great details, for
instance the central region is unsystematically rough compared to the rest of the scan area,
however a localized rough region appears in the left lane between stations 50 and 55, which
corresponds to a loose pile of aggregate. The paved road surface map includes an
approximately 3 m wide lane that extends between stations 0 and 4. The map shows the
severity and location of the reflective cracks in the transvers direction.

Alhasan and White 28

 Figure 6. Absolute rectified slope surface maps for (a) unpaved and (b) paved roads.

The average of rectified slope values along each profile is defined as its IRI, since the map was
generated by simulating a quarter-car model moving at a speed of 80 km/h. Figure 7 shows the
IRI values versus width for both roads, it can be noticed that IRI values are highly variable. Due
to the high variability in IRI values it is proposed to base the conclusions regarding surface
roughness on the IRI versus width plots, and if a single summary index is needed median would
be a more proper way to report overall IRI values than the average. The advantage of using the
median is robustness to outliers.

Alhasan and White 29

 Figure 7. IRI versus width for both road sections included in the study.

To investigate the reason for low IRI values for the unpaved section, two profiles were
transformed to frequency space, one profile corresponding to maximum IRI value and another
profile corresponding to minimum IRI value. Each profile was decomposed into its constituent
spectrum using the FFT (Figure 8). Presenting the results as height amplitude versus spatial
frequency clearly shows different components of amplitude and frequency. The threshold for
“smooth” was set at amplitude less than 0.4 mm. “Unsystematically rough” is defines as
amplitude greater than 0.4 mm, but variable over a range of spatial frequencies. “Corrugation” is
defined as amplitude with a central peak of greater than 0.7 mm and a spatial frequency of 1.5
to 2.5 (1/m). By examining the results this way, the source of roughness can be distinguished.
And it can be seen that highest amplitudes are due to corrugations, however the quarter-car
filter attenuates these frequencies, which results in substantially lower IRI values.

Figure 8. FFT for road profiles in (a) smooth surface with corrugations,
(b) unsystematically rough surface with corrugations.

Alhasan and White 30

SUMMARY AND CONLUSIONS
This paper introduces the frame work of quantitative techniques for evaluating the surface
roughness of paved and unpaved roads. Methods for evaluating the surface roughness can be
set to two main categories, finite difference simulation and frequency domain analysis. The key
findings from this research are:

x Terrestrial laser scanning is a promising technology to assess a range of surface

conditions for unpaved roads.
x 2-D Surface roughness maps were developed using the information obtained from the
laser scanner.
x Algorithms used in producing 2-D roughness maps are semi-automated, and further
developments are expected to introduce fully automated algorithms that can process the
data directly after scanning.
x IRI values are highly variable across the road section, thus it is hard to define the
appropriate profile to be used as the representative profile.
x The proposed analysis technique can be used to identify localized rough features.
x Finite difference simulations are not suitable for short profiles.
x Filtering of the profiles in Fourier space as a tool to get the spatial roughness maps are
more suitable compared to finite difference algorithms for analyzing short profiles,
however wavelet analysis provide a more robust approach to analyze short profiles and
identify localized features.
x At this stage, high resolution terrestrial laser scans are time consuming and require
trained personnel; however, newer terrestrial laser scanners will be able to reduce the
data acquisition time significantly due to faster scanning rates and longer ranges.

REFERENCES
ProVAL: Profile Viewing and Analysis Software, version 3.4 (2013),The Transtec Group, Inc.
available from www.RoadProfile.com.
Al-Omari, B., and Darter, M. I. (1994). "Relationships between international roughness index
and present serviceability rating." Transportation Research Record(1435).
Alhasan, A., White, D. J., and De Brabanter, K. (2015). "Continuous wavelet analysis of
pavement profiles." Automation in Construction, Under review.
Alhasan, A., White, D. J., and De Brabanter, K. (2015). "Quantifying Unpaved Road Roughness
from Terrestrial Laser Scanning." 2015 TRB 94th Annual MeetingWashington, D.C.
Alhasan, A., White, D. J., and De Brabanter, K. (2015). "Spatial pavement roughness from
stationary laser scanning." International Journal of Pavement Engineering, Under review.
Archondo-Callao, R. (1999). "Unpaved roads roughness estimation by subjective evaluation."
The World Bank.
ASTM E950 / E950M-09 "Standard Test Method for Measuring the Longitudinal Profile of
Traveled Surfaces with an Accelerometer Established Inertial Profiling Reference."ASTM
International, West Conshohocken, PA, 2008, www.astm.org.
ASTM E1926-08 "Standard Practice for Computing International Roughness Index of Roads
from Longitudinal Profile Measurements."ASTM International, West Conshohocken, PA,
2008, www.astm.org.
Berthelot, C. F., Podborochynski, D., Stuber, E., Prang, C., and Marjerison, B. (2008).
"Saskatchewan Case Studies of Network and Project Level Applications of a Structural
Asset Management System." Proc., 7th International Conference on Managing
Pavement Assets. TRB Committee AFD10 on Pavement Management Systems,
Transportation Research Board, Washington, DC Retrieved October, 2011.

Alhasan and White 31

Boggess, A., and Narcowich, F. J. (2009). A first course in wavelets with Fourier analysis, John
Wiley & Sons.
Bogsjö, K., and Rychlik, I. (2009). "Vehicle fatigue damage caused by road irregularities."
Fatigue & Fracture of Engineering Materials & Structures, 32(5), 391-402.
Brown, M., Mercier, S., and Provencher, Y. (2003). "Road maintenance with Opti-Grade®:
Maintaining road networks to achieve the best value." Transportation Research Record:
Journal of the Transportation Research Board, 1819(1), 282-286.
Chang, J.-R., and Chang, K.-T. (2006). "Application of 3 D laser scanning on measuring
pavement roughness." ASTM Journal of Testing and Evaluation, 34(2), 83-91.
Chootinan, P., Chen, A., Horrocks, M. R., and Bolling, D. (2006). "A multi-year pavement
maintenance program using a stochastic simulation-based genetic algorithm approach."
Transportation Research Part A: Policy and Practice, 40(9), 725-743.
Daubechies, I. (1992). Ten lectures on wavelets, SIAM.
Fu, P., Lea, J. D., Lee, J. N., and Harvey, J. T. (2013). "Comprehensive evaluation of automated
pavement condition survey service providers' technical competence." International
Journal of Pavement Engineering, 14(1), 36-49.
Gao, H., and Zhang, X. (2013). "A Markov-Based Road Maintenance Optimization Model
Considering User Costs." Computer-Aided Civil and Infrastructure Engineering, 28(6),
451-464.
Karamihas, S. M., Gillespie, T. D., Kohn, S. D., and Perera, R. W. (1999). "Guidelines for
longitudinal pavement profile measurement: final report." 126p + appendices.
Kilpeläinen, P., Jaakkola, M., and Alanaatu, P. (2011). "Development of a control system for a
multipurpose road repairing machine." Automation in Construction, 20(6), 662-668.
Kuo, C. M., Fu, C. R., and Chen, K. Y. (2011). "Effects of Pavement Roughness on Rigid
Pavement Stress." Journal of Mechanics, 27(1), 1-8.
Lamptey, G., Labi, S., and Li, Z. (2008). "Decision support for optimal scheduling of highway
pavement preventive maintenance within resurfacing cycle." Decision Support Systems,
46(1), 376-387.
Lin, J.-D., Yau, J.-T., and Hsiao, L.-H. (2003). "Correlation analysis between international
roughness index (IRI) and pavement distress by neural network." Proc., 82nd annual
meeting in January, 2003.
Liu, C., and Herman, R. (1999). "Road profile, vehicle dynamics, and ride quality rating." Journal
of transportation engineering, 125(2), 123-128.
Murray, R. M., Li, Z., Sastry, S. S., and Sastry, S. S. (1994). A mathematical introduction to
robotic manipulation, CRC press.
Namur, E., and de Solminihac, H. (2009). "Roughness of Unpaved Roads Estimation and Use
as an Intervention Threshold." Transp. Res. Record(2101), 10-16.
Oijer, F., and Edlund, S. (2004). "Identification of transient road obstacle distributions and their
impact on vehicle durability and driver comfort." Vehicle System Dynamics, 41, 744-753.
Ouyang, Y., and Madanat, S. (2004). "Optimal scheduling of rehabilitation activities for multiple
pavement facilities: exact and approximate solutions." Transportation Research Part A:
Policy and Practice, 38(5), 347-365.
Sayers, M. W., Gillespie, T. D., and Paterson, W. D. (1986). Guidelines for conducting and
calibrating road roughness measurements.
Sayers, M. W., Gillespie, T. D., and Queiroz, A. (1986). "The international road roughness
experiment. Establishing correlation and a calibration standard for measurements."
Sayers, M. W., and Karamihas, S. M. (1996). "Interpretation of road roughness profile data."
Sayers, M. W., and Karamihas, S. M. (1996). "Interpretation of road roughness profile data."
University of Michigan Transportation Research Institute.

Alhasan and White 32

Soria, M., and Fontenele, E. (2003). "Field Evaluation of Method for Rating Unsurfaced Road
Conditions." Transportation Research Record: Journal of the Transportation Research
Board, 1819(-1), 267-272.
Steinwolf, A., Giacomin, J., and Staszewski, W. (2002). "On the need for bump event correction
in vibration test profiles representing road excitations in automobiles." Proceedings of
the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering,
216(4), 279-295.
Swan, M., and Karamihas, S. M. (2003). "Use of a ride quality index for construction quality
control and acceptance specifications." Transportation Research Record(1861), p. 10-
16.
Tsai, Y. J., Wu, J., Wang, Z., and Hu, Z. (2010). "Horizontal roadway curvature computation
algorithm using vision technology." Computer-Aided Civil and Infrastructure Engineering,
25(2), 78-88.
Tsunokawa, K., and Schofer, J. L. (1994). "Trend curve optimal control model for highway
pavement maintenance: Case study and evaluation." Transportation Research Part A:
Policy and Practice, 28(2), 151-166.
Walker, D., Entine, L., and Kummer, S. (2002). "Gravel-Paser Manual: Pavement Surface
Evaluation and Rating." Wisconsin Transportation Information Center.
Woll, J. H., Surdahl, R. W., Everett, R., and Andresen, R. (2008). "Road Stabilizer Product
Performance: Seedskadee National Wildlife Refuge." Federal Highway Administration.
Zalama, E., Gómez-García-Bermejo, J., Llamas, J., and Medina, R. (2011). "An effective texture
mapping approach for 3D models obtained from laser scanner data to building
documentation." Computer-Aided Civil and Infrastructure Engineering, 26(5), 381-392.
Zhang, C. (2009). "Monitoring the Condition of Unpaved Roads with Remote Sensing and Other
Technology." Geographic Information Science Center of Excellence, South Dakota State
University.
Zhang, C., and Elaksher, A. (2012). "An Unmanned Aerial Vehicle-Based Imaging System for
3D Measurement of Unpaved Road Surface Distresses1." Computer-Aided Civil and
Infrastructure Engineering, 27(2), 118-129.

Alhasan and White 33

Autonomous Construction of Separated Artifacts by
Mobile Robots Using SLAM and Stigmergy

Hadi Ardiny, Stefan Witwicki, and Francesco Mondada

Robotic Systems Laboratory
École Polytechnique Fédérale de Lausanne
Lausanne 1015, Switzerland
Email: [email protected], [email protected], [email protected]

ABSTRACT
Autonomous mobile robots equipped with arms have the potential to be used for automated
construction of structures in various sizes and shapes, such as houses or other infrastructures.
Existing construction processes, like many other additive manufacturing processes, are mostly
based on precise positioning, which is achieved by machines that have a fixed mechanical link
with the construction and therefore relying on absolute positioning. Mobile robots, by nature, do
not have a fixed referential point, and their positioning systems are not as accurate as fixed-
based systems. Therefore, mobile robots have to employ new technologies and/or methods to
implement precise construction processes.

In contrast to the majority of prior work on autonomous construction that has relied only on
external tracking systems (e.g., GPS) or exclusively on short-range relative localization (e.g.,
stigmergy), this paper explores localization methods based on a combination of long-range self-
positioning and short-range relative localization for robots to construct precise, separated
artifacts in particular situations, such as in outer space or in indoor environments, where
external support is not an option.

Achieving both precision and autonomy in construction tasks requires understanding the
environment and physically interacting with it. Consequently, we must evaluate the robot’s key
capabilities of navigation and manipulation for performing the construction in order to analyze
the impact of these capabilities on a predefined construction. In this paper, we focus on the
precision of autonomous construction of separated artifacts. This domain motivates us to
combine two methods used for the construction: 1) a self-positioning system and 2) a short-
distance relative localization. We evaluate our approach on a miniature mobile robot that
autonomously maps an environment using a simultaneous localization and mapping (SLAM)
algorithm; the robot’s objective is then to manipulate blocks to build desired artifacts based on a
plan given by a human. Our results illuminate practical issues for future applications that also
need to integrate complex tasks under mobile robot constraints.

Keywords: autonomous construction—miniature mobile robot—self-positioning

system—precise construction—unknown environments

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
I. INTRODUCTION
Construction automation is an interesting field focused on applying automating processes to
reduce the cost of construction and/or to increase operational efficiency. Developments in
robotics sciences have recently led to the use of various robotic platforms to achieve
construction automation objectives, although fully automated construction is still a dream of civil
engineers. Robotic developments have shown that robots could potentially perform construction
tasks where human presence is impossible, undesirable, or intensively expensive, for instance,
construction in hazardous areas after natural or man-made disasters such as earthquakes and
nuclear accidents; construction under difficult physical conditions such as undersea or outer
space locations; and construction where in area that are not readily accessible to humans or
that require an initial structure to prepare the environment for human arrival. Robots can be
used to build these structures for particular situations in the autonomous mode without explicit
human intervention or with some levels of planning interaction conducted with a human
supervisor.

Generally, a robot performing autonomous construction has to adapt itself to the sensed
environment, make decisions regarding the execution of its task, and replan when its task is not
executable. Mobile robots represent one type of robotic system that could be used for
construction automation. Applying mobile robots to construction opens new approaches in this
field. For instance, building large structures without being confined by dimensions is a challenge
for current technologies; for example, we might need huge and expensive fixed-based
fabricating systems (e.g., 3D printers) to build giant structures. Capabilities of mobile robots,
however, allow them to create objects without fixed-base system constraints (e.g., size of the
printer’s frame constraint). Similar to social insects, such as ants, a group of mobile robots can
work cooperatively, as a collective system, to efficiently build large-scale.

In contrast to these advantages, mobile robots, by nature, do not have a fixed referential point
and their positioning systems are not as accurate as fixed-based systems. Existing construction
processes, like many other additive manufacturing processes, are mostly based on precise
positioning, which is achieved by machines that have a fixed mechanical link with the
construction and rely on absolute positioning. Therefore, mobile robots have to compensate for
this weakness with new technologies and/or methods to supply precision for construction
processes. Although equipping robots with external tracking systems (e.g., GPS, camera)
provides an accurate positioning system, but we aim to implement and study localization
methods based on self-positioning system to autonomously handle construction tasks,
especially where external tracking systems is are difficult to access and are expensive (e.g.,
undersea). On the other hand, the accuracy of self-positioning system is not sufficient to handle
construction processes; therefore, we aim to combine it with short-range relative localization to
provide the required precision for construction of structures spatially separated from one
another, which we refer to here as separated artifacts.

In this paper, our goal is to develop a construction system by which robots are able to build
separated artifacts. We evaluate our approach with a miniature autonomous mobile robot and
simple blocks in unknown environments. The robot’s objective is to build the artifacts using both
the simultaneous localization and mapping algorithms (SLAM) and stigmergy based on a
human-prescribed blueprint. In fact, it employs SLAM using the LIDAR scanner to autonomously
map an environment and determine the current robot position. The robot's end effector is
equipped to many IR-sensors that allow it to sense previously placed blocks in order to place
subsequent blocks; this approach is commonly referred to as stigmergy [1].

H.Ardiny, S. Witwicki, and F. Mondada 35

In Section II, related work is discussed. The scenario and assumptions, robot hardware, and
control are provided in Section III. The results and discussion are presented in Section IV.
Finally, in Section V, we conclude for this study.

II. RELATED WORKS

In this section, we are primarily reviewing research that employs mobile robots for autonomous
construction based on positioning systems, robot platforms, and materials. Obviously, precise
positioning systems are necessary for most systems to support construction processes. At
present, an external localization system could be employed to provide accurate systems for
construction. In the research conducted at ETH Zurich, four quadcopters are exploited to
construct a brick-like tower. They benefit from an external real-time camera system to guide
robots to pick up and deposit objects according to the given blueprint [2]. Lindsey et al. [3] used
a team of quadcopters to assemble the cubic structures with particular self-assembling rods .
The VICON motion tracking system is used to estimate the position and orientation of the picked
objects and aerial vehicles states. It provided position feedback at 150 Hz with marker position
accuracy on the order of a millimeter. Moreover, VICON is used by ground robots to build roofed
structures [4].

Another general external system is the GPS used by a robotic excavator with centimeter
resolution allowing them to determine position accurately and then to control the motion of the
robots [5]. In [6], the ROCCO robot was developed to assemble heavy blocks in industrial
buildings with standardized layouts. It was equipped with digital angular encoders and an
external global position sensor (telemeter) correct error. In [6], a method was demonstrated in
simulation by which robots are able to build 2D structures of desired shapes by blocks. A robot
acts as a stationary beacon to help other robots find its position. In [7], robot placed blocks of
alternating color along a straight line starting with a pre-placed seed block located underneath a
beacon. Although using external system improves the positioning system capability, many
additional localization devices are required, which might be impossible or very expensive to
provide, for example, in outer space or undersea construction. In contrast to these works, the
robot is completely autonomous in our work and does not rely on any motion-capture systems
or external localization systems.

However, some robots applied short-range relative localization for construction. Werfel et al. [8]
present 3D collective construction in which enormous numbers of autonomous robots build
large-scale structures. They employed a ground mobile robot that was inspired by activity of
termites. Robots climb on the structure to drop passive solid blocks on top of it. They just use
six active infrared (IR) sensors to recognize white stripes on the blocks and then determine their
path and final destination. Novikov et al. [9] have built 3D shape structures by using deposition
amorphous material with mobile heads. This method allows an object to be printed independent
from its size of the object.

Stroupe et al. [10] present construction by two platform robots SRR and SRR2K in an outdoor
environment. Each rover is holonomically equipped with a forward-facing stereo pair of cameras
and a four degree-of-freedom arm. A triple-axis force-torque sensor on the gripper helps the
rover maintain coordination for transporting and placing rods. This model provides high-
precision manipulator placement by comparing the observed position of beam markers on the
end effector with the obtained kinematics position of the end effector.

H.Ardiny, S. Witwicki, and F. Mondada 36

In some research, self-alignment methods have been used to tackle substance alignment and
attachment restrictions. For instance, bricks are made from expanded foam, with physical
features to achieve self-alignment and magnets for attachment [11]. In [12], self-alignment cubic
modules are used to build structures. The special assembler robots manipulate and transport
these modules. In [13], a novel robot used bidirectional geared rods and connectors to build a
truss structure. In conclusion, using only the short-range relative localization limits construction
to a local place because robots need to look at the local configuration of the building material to
determine where to add additional materials.

Magnenat et al. [14] used a miniature autonomous robot with a magnetic manipulator to grasps
ferromagnetic self-alignment blocks. This robot also has the LIDAR and camera on top. It used
the odometry and laser data to perform SLAM and employed the front camera and proximity
sensors to provide the required data for dropping blocks. The goal of this research was to use
ten blocks to build a simple tower. We advance this research by studying the precision of using
both a short-range relative localization and a self-positioning system for separated artifacts.

III. SYSTEM DESCRIPTION

A. Scenario and Assumptions
As overviewed in the Introduction, the goal of this experiment is to build separated artifacts
based on a blueprint by a human in an unknown environment. We assume that the initial
position of the robot is also the location of a block repository. We put a new block at the
repository for each step of construction. The robot has to detect the block and align itself with
respect to the block to able to pick up the block at the correct position of gripper.

The arena is a 200 cm × 100 cm rectangle with a flat surface that contains few obstacles. Note
that the environment is unknown for the robot, and SLAM is used to inform the robot of its
position and to map the environment for path planning as well. The artifacts, as illustrated later,
are composed of simple polystyrene blocks with an attached stripe of ferromagnetic metal on
the lower part of the body (Figure 1). Each block is 6 cm in length, 6 cm in width, and 18 cm in
height, and it weighs approximately 20 g. The size and weight of the block are chosen to satisfy
the requirements of the robot’s gripper and the LIDAR.

After grasping the block, the robot moves toward the destination point. The path-planning
algorithm determines the global path to the destination. It also sets the local path planning
during its movements to avoid collision with dynamic obstacles and to correct the path based on
the robot’s improved position. The first block of the artifact will be dropped after the robot fine-
tunes its position using the accurate movement behavior. 1 The robot returns and takes a new
block from the repository. Now, the robot is ready to drop the second block of the artifact. The
robot drops the second block beside the first block using stigmergy. In this section, we
explained the construction scenario and assumptions. In the next sections, we first describe the
robot hardware and then provide the details of the control system, including low-level behaviors
and control architecture.

1
See section 3-3-2

H.Ardiny, S. Witwicki, and F. Mondada 37

 Figure 1. Polystyrene blocks are Figure 2. The MarXbot consists of the
used for the experiments. gripper, LIDAR, computer board, base
modules.

B. Robot platform
For the experiment, we used a miniature and modular robot called marXbot [15] (Figure 2). The
robot is 17 cm in diameter and 18 cm in height. It consists of four modules as follows:

Base: The non-holonomic base has 2 degrees of freedom (DOF), a 38Wh lithium polymer
battery, and 24 proximity sensors.

Gripper: The 3 DOF magnetic manipulator consists of a magnetic switchable device to grasp
ferromagnetic objects. This module has 20 proximity sensors for the alignment usage.

Computer board: This module includes the main computer based on a 533MHz Freescale
i.MX31 with 128MB of RAM and running LINUX.

LIDAR: The 360° laser distance scanner (Neato LIDAR < $100) perceives walls and obstacles.

C. Control system
I) Control architecture
The control architecture, shown in Figure 3, consists of several layered modules. At the top, the
builder planner serves to execute an overall construction of separated artifacts by generating a
sequence of high-level sub-goals. These sub-goals either take the form of target poses for robot
movement, which are delegated to the navigation block, or block manipulation sub-goals (such
as pickup and place), which are delegated to the middle planner.

H.Ardiny, S. Witwicki, and F. Mondada 38

Navigation is implemented through a collection of ROS nodes, including navigation
(Move_base 2) and SLAM (Hector_mapping 3) packages. The SLAM package receives the data
from the LIDAR to map the environment and localize the robot. The navigation package obtains
the map and positioning information from SLAM to direct and set the reliable paths for the
mobile robot from the current position to the goal position. To physically move the robot,
navigation in turn invokes low-level behaviors that have been pre-programmed using the
ASEBA software architecture [16], which runs directly on the robot’s microcontrollers.

The middle planner, implemented using state-of-the-art AI planning technologies [17], renders
the robot fully autonomous in its manipulation of blocks in the environment. For each
manipulation goal (pickup, drop, place-adjacent-to), the robot accesses a (pre-computed)
conditional plan that iteratively selects among low-level task executions (e.g., approach-to-
within-manipulation-distance, align-gripper-angle, grasp-block). Each of these low-level tasks is
implemented as a finite state controller that, at 5Hz, senses using the gripper infrared and
actuates the treel 4 and gripper motors. Due to sensory inaccuracies and environmental
imperfections, the low-level tasks are not deterministic in their effects. For instance, a brief mis-
measurement of infrared distances caused by fluctuations in ambient lighting could cause the
align-gripper-angle controller to over-rotate the gripper such that the robot loses sight of the
block that it is aligning with. When such unintended effects occur, the robot's conditional plan
gracefully recovers by selecting the next appropriate task. In essence, the conditional plan
composes a complex and dynamic sequences of tasks that is theoretically guaranteed [18] to
eventually bring the robot to its manipulation goal.

Figure 3. Control architecture in which green and blue boxes represent hardware and
software layers, respectively.

2
Move_base is a 2D navigation stack that receives information from sensors and a goal pose and then directs the
mobile base by determining safe speed and reliable paths.
3
Hector_mapping is a SLAM algorithm using LIDAR systems such as the Hokuyo UTM-30LX. The system has
been used on unmanned ground robots, unmanned surface vehicles.
4
Treel refers to a combination of a wheel and a track.

H.Ardiny, S. Witwicki, and F. Mondada 39

II) Low-level behaviors
Pick-up: The pick-up behavior has to accurately grasp blocks because a misaligned block will
add errors to all subsequent operations. As illustrated in Figure 4, the middle planner generates
a sequence of movements. At the beginning, the robot turns to face the repository. It uses both
the ROS navigation and accurate movement behavior to align itself at the right angle; then it
starts to find the nearest block (1). The robot then uses front infrared sensors of the gripper to
tune the distance in respect to the blocks (2). Next, it rotates 90 degrees rightward, and the
gripper rotates in the opposite direction (3), which situation helps the robot align itself laterally.
Using the front infrared sensors of the magnetic gripper also helps the robots to align the block
at the center of the gripper (4). When the block is centered, the robot performs a 90-degree
leftward rotation while the gripper rotates in the opposite direction (5). It then moves forward and
rotates a few degrees at a time to touch the block at the corner of the magnet position (6). When
the robot touches the blocks, it grasps it and lifts up the gripper; this time the block is well
aligned (7).

Traveling: This behavior consists of raising the gripper in order that the attached block and
gripper do not interfere with the robot movements and SLAM.

Figure 4. The movement sequence to pick Figure 5. The movement sequence to

up a block. drop a new block adjacent to a placed
one.

Accurate movement: This behavior occurs in two modes to move the robot precisely. When
the robot approaches a destination point, this behavior moves the robot to put it in the precise

H.Ardiny, S. Witwicki, and F. Mondada 40

position using a PID controller. In the first mode, it rotates and moves to correct its position in
the X axis. The robot then rotates 90 degrees and moves to correct its position in the Y
direction. Finally, it rotates to reach the goal yaw angle. In the second mode, the robot only
rotates to correct its yaw angle. Depending on the situation in the planner request, the robot will
use either the first or second mode.

Drop/Place-Adjacent-To: This behavior consists of dropping a block either in the first place of
the artifact or directly adjacent to existent blocks. To build an artifact, the robot has to drop its
first block. Thus, it simply lowers the gripper after finding the precise position and then
disengages its magnetic switchable device. It then lifts its manipulator slightly and moves back
for a short distance. For the remaining blocks of the artifact, the robot goes toward the artifact
(1) and scans for it using the magnetic gripper’s IR sensors. The robot computes the distance to
the artifact and moves accordingly to tune its position (2). Then it rotates 90 degrees while it
rotates the gripper in the reverse direction with the same angular speed. In this position (3), the
robot aligns itself laterally and then finds the left edge of the placed block to drop the new one
exactly beside it (4). The robot performs a 90-degrees leftward rotation while the gripper rotates
in the opposite direction (5). It then moves forward, lower, and rotates the gripper by a few
degrees at a time to avoid collision with the other blocks (6). It then lowers its gripper slightly
and moves back a short distance. Finally, the robot tilts the gripper and moves forward to push
and line up the block (7).

IV. RESULTS AND DISCUSSTION

We employed the marXbot for the two types of experiments. First, the robot places three blocks
based on given positions to study the precision of SLAM. In the second type, we employed both
stigmergy and SLAM to build several separated artifacts, each made up of composed by several
blocks. Five trials were carried out for each type of experiment. Figure 6 illustrates the
construction of artifacts through some snapshots at the different steps. After finishing artifact
construction, we took a photo to measure construction performance. We extracted the red color
through image-processing methods to compare the performance of each experiment with the
ideal block arrangement. In the ideal block arrangement, we assumed SLAM and stigmergy
were perfect, and artifacts were being built exactly based on the given map. Using this model,
we can measure the translational and rotational errors of the blocks on the plane as illustrated in
Figure 7. The graphs in Figure 9 and Figure 10 show these absolute errors for the two
experiments. In the first experiment, the robot only drops blocks B1, B4, B8 (the first block of
each artifact) using SLAM. The important error of placement based on SLAM, of one third of the
block size, shows that we cannot use this positioning technique to build artifacts made of
several blocks. If the robot employs only SLAM, the important positioning error will cause
collisions among blocks or gaps between blocks of the artifacts. Nevertheless, the robot can find
the approximate position of construction and then apply other methods (e.g., stigmergy) to
successfully accomplish the construction.

For stigmergy, the robot uses infrared sensors to align itself with respect to the blocks that are
already part of the artifact. Because the artifacts are not fixed to the ground, the dropping
operation may cause a displacement of the existing blocks. As illustrated in Figure 10, the
average errors of stigmergy for blocks B4 and B8 (first block of the second and third artifacts)
are different from the errors measured after using only SLAM (see Figure 9). This shows that
the sigmergy action moved these blocks. This means, for instance, that when the robot pushed
the block to align it, it also pushed previously-placed blocks because of the errors in stigmergy
positioning. Indeed, if the robot could apply a perfect stigmergy, we would expect to see the
same precision for other blocks of the artifact as we saw for the first block.

H.Ardiny, S. Witwicki, and F. Mondada 41

 Figure 6. Ideal arrangement of the blocks is Figure 7. The processed image of the
shown for the three artifacts. A comparison second artifact and measuring
of built artifacts with this ideal arrangement translational and rotational errors for
provides the construction performance the fourth block; d shows the distance
results. The robot starts from the (0,0) point between center of the squares, and ș
to take a block from the repository. The shows the angle between the lower
numbers on the blocks point to the edges of the squares. Green color
numerical order for the construction. also depicts the overlap area between
the fourth block and the second
artifact.

Figure 8. Image sequence of artifact construction

H.Ardiny, S. Witwicki, and F. Mondada 42

Overall, stigmergy could improve or weaken the precision. If the artifacts were fixed to the
ground, the robots could be able to use force sensing to push the blocks without influencing the
precision of the next dropping steps. Moreover, the minor difference of the translational and
rotational blocks errors for each artifact shows that the building artifacts are not sensitive to the
final shape or to the number of blocks used in these experiments.

Figure 9. The left graph shows the translational error and the right graph shows the
rotational error when the robot drops the first, fourth, and eighth blocks using just SLAM.

Figure 10. The left graph shows the translational error, and the right graph shows the
rotational error when the robot is supposed to build the artifact with SLAM and
stigmergy.

Finally, we measured the surface of each ideal artifact that is occupied by blocks placed by the
robot. Ideally, the overlap percentage between the blocks and the given blueprint has to be a
100%. Figure 11 shows the overlap percentage for two construction types: single separated
blocks or separated multiple-block artifacts. Note the average percent of the single-block
construction is 57.88%, but it is 73.53% in artifact construction. This increase of performance
does not mean that multiple-blocks artifacts are placed more precisely, because adjacent blocks
can compensate for the positioning error for the global covering of the artifact surface. This

H.Ardiny, S. Witwicki, and F. Mondada 43

increase of performance only shows that stigmergy enables the construction of more coherent
artifacts.

Figure 11. The left graph depicts the overlap percentage for dropping blocks based on
the using SLAM, and the right graph depicts the overlap percentage for building the
artifact based on using SLAM and stigmergy.

This section provided an analysis of the precision achieved by a mobile robot in building
separated artifacts using SLAM and stigmergy. The SLAM algorithm was used as a localization
and mapping method by a miniature mobile platform in an unknown environment. In our
experiment, we have a translational error of about 21 mm (one-hundredth of the diagonal of the
environments) in a static environment using a miniature robot with a low-cost LIDAR. It is
difficult to evaluate how this error will scale in a real environment because the SLAM accuracy
depends on many factors; environment dimensions will clearly impact precision, as larger
distances will generate larger measurement errors. The quality and quantity of landmarks also
impacts the estimation of the position by the SLAM algorithm. A dynamic environment can also
cause a loss of precision, as dynamic obstacles can hide interesting landmarks. Finally the
quality of the sensor for distance measurement impacts the whole system directly. Obviously,
there is a need to further develop the sensory system and SLAM algorithms for complex
artifacts in real and large environments. Despite a lot of progress in the past decades in the
SLAM field, high-precision applications are still challenging. For the time being, SLAM could
help to find approximate construction sites and then the robot could use other methods to follow
the construction, such as stigmergy as used in this research.

In this research, we applied stigmergy based on a pure IR-sensing system while the mechanical
stigmergy can be more suitable to place the blocks. Today, companies are designing and
manufacturing prefabricated components to increase construction speed and efficiency. New
prefabricated components could be designed and made for robotic use in automated
construction. For example, components with male–female connectors allow for automatic
assembly in a more robust way [11]. Developing construction methods based on mechanical
stigmergy or using force-sensing systems could provide a new way to place components in a
more reliable and precise way.

Autonomous construction is also a complex application in which many failures can occur. These
failures can propagate from one step to another, for instance, if the robot incorrectly grasps the

H.Ardiny, S. Witwicki, and F. Mondada 44

block, it could destroy the built structures. Thus, it is important to detect and correct faults. We
performed fault detection for middle planner, which is responsible for the planning low-level
behaviors (pick up, drop, place-adjacent-to); however, we need to improve the builder planner to
take high-level decisions for failures caused during construction.

V. Conclusion

This paper presents an autonomous construction system for building separated artifacts with
simple blocks. We used a miniature mobile robot that autonomously mapped an environment
using a SLAM algorithm and then manipulated blocks to build desired and separated artifacts.
Our approach was based on the combination of two methods: a self-positioning system (SLAM)
to find the construction place in an unknown environment and stigmergy to build coherent
artifacts. The control system allowed the robot to perceive and pick up the block, move toward
the construction place in an unknown environment, and drop the block based on a human-
prescribed blueprint. We observed that, even in an ideal environment, positioning using SLAM is
not sufficiently precise. This task still requires improvement in sensing technology. We also
observed that stigmergy allow the creation of coherent constructions. The process analyzed in
this paper, based on mobile blocks and sensing stigmergy, could be improved by having blocks
fixed when dropped and employing mechanical stigmergy.

As a result, in future works, we are focusing on developing robot hardware and improving the
SLAM algorithm. We also plan to develop stigmergy and use force-sensing systems for
mechanical stigmergy. Thanks to stigmergy and hardware development, robots could be used
to build complex artifacts such as a multi-layer wall with prefabricated components.

REFERENCES
[1] G. Theraulaz and E. Bonabeau, “Coordination in distributed building,” Science, vol. 269,
no. 5224, pp. 686–8, Aug. 1995.

[2] J. Willmann et al. “Aerial robotic construction towards a new field of architectural
research,” Int. J. Archit. Comput., vol. 10, no. 03, pp. 439–460, 2012.

[3] Q. Lindsey, D. Mellinger, and V. Kumar, “Construction of Cubic Structures with

Quadrotor Teams,” in Robotics: Science and Systems, 2011.

[4] S. Wismer, G. Hitz, M. Bonani, A. Gribovskiy, and S. Magnenat, “Autonomous

construction of a roofed structure: Synthesizing planning and stigmergy on a mobile robot,” in
IROS, 2012, pp. 5436–5437.

[5] D. A. Bradley and D. W. Seward, “The Development, control and operation of an

autonomous robotic rxcavator,” Intell. Robot. Syst., no. 138257, pp. 73–97, 1998.

[6] E. Gambao, C. Balaguer, A. Barrientos, R. J. Saltarén, and E. A. Puente, “Robot

assembly system for the construction process automation,” in International Conference on
Robotics and Autonomation, April 1997, pp. 46–51.

H.Ardiny, S. Witwicki, and F. Mondada 45

[7] J. Wawerla, G. S. Sukhatme, and M. J. Mataric, “Collective construction with multiple
robots,” in International Conference on Intelligent Robots and Systems, October 2002, pp.
2696–2701.

[8] J. Werfel, K. Petersen, and R. Nagpal, “Distributed multi-robot algorithms for the
TERMES 3D collective construction system,” RSJ Int. Conf. Intell. Robot. Syst. (IROS),, 2011.

[9] P. Novikov, S. Maggs, D. Sadan, S. Jin, and C. Nan, “Robotic positioning device for
three-dimensional printing,” CoRR, vol. abs/1406.3, pp. 1–14, 2014.

[10] A. Stroupe et al. “Sustainable cooperative robotic technologies for human and robotic
outpost infrastructure construction and maintenance,” Auton. Robot., vol. 20, no. 2, pp. 113–
123, Apr. 2006.

[11] J. Werfel, K. Petersen, and R. Nagpal, “Designing collective behavior in a termite-

inspired robot construction team,” Science, vol. 343, no. 6172, pp. 754–8, Mar. 2014.

[12] Y. Terada and S. Murata, “Automatic modular assembly system and its distributed
control,” I. J. Robot. Res., vol. 27, no. 3–4, pp. 445–462, Mar. 2008.

[13] F. Nigl, S. Li, J. Blum, and H. Lipson, “Structure-reconfiguring robots: autonomous truss
reconfiguration and manipulation,” IEEE Robot. Autom. Mag., vol. 20, no. 3, pp. 60–71, 2013.

[14] S. Magnenat, R. Philippsen, and F. Mondada, “Autonomous construction using scarce

resources in unknown environments - Ingredients for an intelligent robotic interaction with the
physical world.,” Auton. Robot., vol. 33, no. 4, pp. 467–485, 2012.

[15] M. Bonani et al. “The marXbot, a miniature mobile robot opening new perspectives for
the collective-robotic research,” in 2010 IEEE/RSJ International Conference on Intelligent
Robots and Systems, 2010, pp. 4187–4193.

[16] S. Magnenat and P. Rétornaz, “ASEBA: a modular architecture for event-based control
of complex robots,” IEEE/Asme Trans. mechatronics, vol. 16, no. 2, pp. 321–329, 2011.

[17] S. Witwicki and F. Mondada, “Circumventing robots’ failures by embracing their faults: a
practical approach to planning for autonomous construction,” in The 29th AAAI Conference on
Artificial Intelligence, 2015, pp. 4298–4299.

[18] C. Muise, V. Belle, and S. A. McIlraith, “Computing contingent plans via fully observable
non-deterministic planning,” in The 28th AAAI Conference on Artificial Intelligence, 2014, pp.
2322–2329.

H.Ardiny, S. Witwicki, and F. Mondada 46

A Comparison of Automated and Semi-Automated
Compressed Earth Block Presses Using Natural and
Stabilized Soils

Jedadiah F. Burroughs
Geotechnical and Structures Laboratory
US Army Engineer Research and Development Center
3909 Halls Ferry Road
Vicksburg, MS 39180
[email protected]

Todd S. Rushing
Geotechnical and Structures Laboratory
US Army Engineer Research and Development Center
3909 Halls Ferry Road
Vicksburg, MS 39180
[email protected]

C. Phillip Rusche
Geotechnical and Structures Laboratory
US Army Engineer Research and Development Center
3909 Halls Ferry Road
Vicksburg, MS 39180
[email protected]

ABSTRACT
This study compared the use of two compressed earth block press machines and the properties
of compressed earth blocks made with natural and stabilized soils. The Vermeer BP714 Block
Press uses a hydraulically driven, two-stage compression process to produce compressed earth
blocks with consistent density and dimensions similar in shape to concrete masonry units. The
AECT Impact 2001A uses a hydraulically driven, fully automated single stage compression
process to produce modular compressed earth blocks with simple prismatic geometries at a rate
of 240 blocks per hour. This study focused on the advantages and disadvantages of using each
press, as well as the compressive strength development of compressed earth blocks made with
selected soils and soil stabilizers. Recommendations for the suitability of each machine for
different soil conditions are given.

Key words: adobe—compressed earth block—construction materials—hydraulic press—

soil stabilization

PROBLEM STATEMENT
It is estimated that one third of the world’s population lives in earthen structures. For many
centuries, adobe blocks have been used as the main earthen construction element. The
process for making adobe blocks is a time-consuming and inefficient process that can last in
excess of three weeks (Dominguez 2011). In an effort to more efficiently study this traditional

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
construction material, the US Army Engineer Research and Development Center has used
compressed earth blocks (CEBs) as surrogates for adobe blocks.

Compressed earth blocks are earthen construction elements produced using a mechanized
hydraulic ram to compact soil into a mold. The soil is molded in a relatively dry state (generally
less than 10% moisture content) resulting in blocks that are strong and weather resistant. Since
CEBs have very consistent geometry, they can be readily stacked to form walls of structures.
Doors, windows, fixtures, and utilities can all be incorporated in similar ways to masonry
construction. Earthen structures made using CEBs are inexpensive, versatile, and energy
efficient. For these reasons, CEBs have become a popular construction technique in
disadvantaged areas. CEB machines are commercially available to produce blocks in a range of
sizes and shapes.

One example is the AECT Impact 2001A. This automated machine produces CEBs at a rate of
four blocks per minute that are 12-inches long, 6-inches wide, and between 2-inches and 4.5-
inches tall. Each block weighs approximately 15 to 20 lbs depending on soil type, block
thickness, moisture content, etc. Soil is loaded into a hopper situated above a single stage
compression ram. All soil particles must be finer than ½ inch to pass the screen covering the
hopper. Soil is manually loaded into the hopper by means of buckets, shovels, etc. A full hopper
will produce between eight and nine blocks. Once soil is loaded, the diesel engine is started and
block production is initiated by pressing the start button. Soil is dropped from the hopper into a
rectangular cavity that sets the length and width dimensions at 12-inches and 6-inches
respectively. The depth of the cavity is determined by the position of a hydraulic ram and is set
by the user. Once soil has been loaded, the cavity is covered by an automatically sliding tray
and pressed by a vertical ram. The end of the compression stage is selected by the operator,
and can be either a set dimension or a maximum forming pressure. For the AECT Impact
2001A, the maximum hydraulic pressure that is applied is 3000 psi. If the maximum pressure is
reached prior to reaching the required thickness, compression is stopped. The block is then
pressed out of the mold vertically and pushed out of the machine. Because the compressive
force is limited, blocks of varying thickness can be produced from a single run of material.
Variability in moisture throughout a soil can cause changes in the amount of soil that fills the
cavity and thus influence final block dimensions. Blocks can be produced with initial soil
moisture contents from approximately 5% to 20%. In general, wetter soil produces thinner
blocks. Block production continues as long as there is adequate soil filling the hopper. The
AECT Impact 2001A and example blocks are shown in Figure 1. This trailer mounted system
can either be towed with a standard two inch ball hitch or moved easily on site by hand.

Another commercially available CEB machine is the Vermeer BP714. This semi-automated
machine produces blocks of uniform size, approximately 14-inches long, 7-inches wide, and 4-
inches tall, with interlocking geometric features and through-holes at a rate of three blocks per
minute. With the through-holes, the Vermeer BP714 produces CEBs that are reminiscent of
concrete masonry units (CMUs). Each block weighs approximately 20 lbs, which varies
depending on soil type, moisture content, and additives. The through-holes align when stacked
and can be fitted with reinforcing steel or conduit and/or filled with mortar or grout for further
strengthening. Similar to the AECT Impact 2001A, soil is loaded into a screened hopper. The
hopper on the Vermeer BP714 holds enough soil to produce approximately 14 blocks. Rather
than being fully automated, the Vermeer BP714 is controlled by three control levers and
functions in a semi-automated fashion. One lever controls a vertical ram used for filling the soil
cavity and compressing to a fixed thickness. One lever controls a horizontal ram used to
maneuver the dirt tray to move soil from the hopper to the compression cavity. The third lever

Burroughs, Rushing, Rusche 48

controls two tapered, vertical pins, nominally three inches in diameter, that push through the
block from beneath to form the through-holes and compresses the block to consistent forming
pressures. After the block is formed, the vertical ram presses the block out of the mold, and the
horizontal ram pushes the block out of the machine. The Vermeer BP714 is shown in Figure 2.
This press is affixed to skids and can be easily moved with a forklift or pallet jack of sufficient
capacity.

Figure 1. AECT Impact 2001A machine and example blocks

Figure 2. Vermeer BP714 and example blocks

Since compressed soil is susceptible to erosion by water, CEBs are usually produced using soil
that has been mixed with a stabilizer to increase durability. Soil stabilization techniques are well
known for a variety of engineering purposes, and known techniques can be applied to CEBs.
Portland cement added at about 6% to 8% by volume is a common soil stabilizer that undergoes
a hydraulic reaction that binds soil particles. Many other additives are used for soil stabilization
including polymers, petroleum byproducts, microorganisms and their secretions, enzymes,
fibers, minerals, pozzolans, and industrial waste products such as fly ash.

Burroughs, Rushing, Rusche 49

RESEARCH OBJECTIVES
The objective of this study was to compare the functionality of both the Impact 2001A and
Vermeer BP714 using identical soil and stabilizers. By using similar soils and stabilizers, the
effects of the CEB presses themselves could be directly studied. Recommendations for the use
of each machine and its suitability with different soil types were also developed. The effects of
each machine and block geometry on the compressive strength development of the blocks were
also studied.

RESEARCH METHODOLOGY
For the purposes of this study, a clayey sand (SC) from central Louisiana was chosen for use.
Soil classification data is given in Table 1. This soil was chosen due to its sand and clay
contents. Previous studies had shown that soils with greater than 50% sand content and at least
10% clay performed better than lower sand or clay soils (Dominguez 2011). Also, the plasticity
index is given as 20 indicating low plasticity clay particles. High plasticity clays tend to be more
highly expansive when exposed to moisture, which can negatively affect the performance of
compressed earth blocks. Also given in Table 1 is soil characterization data on three other soil
types for reference. SM Blend is very similar to what is found in most arid climates. NP
designates that the soil is non-plastic. Non-plastic soils have little to no natural cohesion, so
binders are required to produce even low quality CEBs. Buckshot clay is a high plasticity, very
expansive clay. While soils such as this tend to produce CEBs with satisfactory green
properties, the CEBs tend to lose structure and stability due to changes in humidity. Agricultural
loam is representative soil for most agricultural fields. While materials such as this do exhibit
some natural cohesion, the high silt content leads to low strength CEBs. For stabilized soils,
Type I portland cement was chosen as the stabilizing agent. The addition of portland cement at
6-8% by volume is a common stabilizer of soils because it undergoes a hydraulic reaction to
bind soil particles together. Various forms of portland cement are also available in most areas
throughout the world, so studying its use in CEBs can have wide-reaching implications.

Table 1. Soil characterization data

Soil Sand (%) Silt (%) Clay (%) PI
Louisiana clay 54 24 22 20
SM Blend 60 38 2 NP
Buckshot clay 3 18 79 64
Agricultural loam 15 63 22 11

Soil was excavated approximately one week prior to production of CEBs. The soil was stored in
a climate controlled area to minimize moisture content changes. The soil was used at its in-situ
moisture content. As it was field excavated, the soil contained many large clumps that were
much too large to pass the screens on either machine’s hopper. In an effort to eliminate the
majority of the clumps and to better homogenize the material, the soil was tilled and placed in a
paddle mixer prior to use. The soil after being removed from the paddle mixer is shown in Figure
3. Even after tilling and mixing with the paddle mixer, the soil still has some large clumps. Many
of these large clumps were broken down by hand prior to adding the material to the CEB
machines. The paddle mixer used in this study is shown in Figure 4. For stabilized blocks,
portland cement was added at a rate of 7% by volume while the material was mixing in the
paddle mixer. The addition of dry powdered portland cement helped to remove moisture from
the soil and allow the mixing action to eliminate more clumps. Soil before and after the addition
of portland cement is shown in Figure 5. The soil after the addition of portland cement is much
more uniform than the soil without binders.

Burroughs, Rushing, Rusche 50

 Figure 3. Louisiana clay prior to CEB production

Figure 4. Paddle mixer

The first part of the study focused on the production of blocks with natural soil. Approximately 35
blocks were made using each machine and stored in a climate controlled area exposed to the
ambient atmosphere. The second part of the study focused on the production of blocks with soil
stabilized with portland cement. Approximately 35 blocks were produced with stabilized soil and
wrapped in plastic and stored in a climate controlled area. These blocks were wrapped to
minimize the loss of moisture needed to react with the added portland cement.

Burroughs, Rushing, Rusche 51

 Figure 5. Louisiana clay before (right) and after (left) adding 7% portland cement by vol.

After blocks had been allowed to dry/cure for a specified time, the full CEBs were tested using a
400-kip hydraulic press. Full blocks were tested per the recommendations of the machine
manufacturers. Due to the irregular surface profile of CEBs produced with the Vermeer BP714,
custom compression plates were used to evenly transfer the load across the cross-section. At
least 10 blocks were tested for each soil or soil-cement combination at seven, 14, and 28 days
to help increase statistical accuracy and reliability. These ages are standard testing ages for
cementitious materials and can be used to show strength development over time.

KEY FINDINGS
The production of blocks varied dramatically between machines and soils. The AECT Impact
2001A press is much more user friendly for first time users. After cranking the diesel engine, all
that was necessary to produce blocks was the addition of soil to the hopper and pressing the
start button. First time users of the Vermeer BP714 have to learn the correct operation of the
hydraulic levers to produce blocks. Incorrect sequencing of the control levers can lead to poor
block consolidation or repetitive compression. As a result, initial block production is a much
slower process. Periodically, as soil continues to be compacted with the Vermeer BP714, the
compression pins need to be cleaned to function properly. Once a consistent operating
procedure is learned, the larger hopper on the Vermeer BP714 allows for more blocks to be
made before additional soil is needed.

For natural soil, the production of blocks was much easier with the AECT Impact 2001A. The
soil was very sticky in its natural condition and did not readily fall from the hopper when using
the Vermeer BP714. This caused problems in block production because too little material was
filling the press cavity. To alleviate some of these issues, soil was manually added to the cavity
and rodded to aid in consolidation. No such problems arose using the AECT Impact 2001A.
Once portland cement was added, both machines functioned at much higher efficiency. With
less clumps being present and the soil being drier overall, the material much more readily filled
the compression cavities of both machines. The manual addition and rodding of soil with the
Vermeer BP714 was not needed with stabilized soil.

Burroughs, Rushing, Rusche 52

Table 2-Table 4 below show the compressive strength data for each material/machine
combination at each of the three ages.

Table 2. 7-day compressive strength data

7-day Compressive Strength
Soil/Machine
Samples Avg. (psi) St. Dev. (psi) COV (%)
AECT Impact 2001A Natural Soil 12 549 80 14.6
Vermeer BP714 Natural Soil 11 258 84 32.4
AECT Impact 2001A Stabilized Soil 12 709 80 11.3
Vermeer BP714 Stabilized Soil 12 790 128 16.2

At seven days, the AECT Impact 2001A CEBs with natural soil are over twice as strong as the
same blocks made in the Vermeer BP714. This is partly because of the production difficulties
when using the Vermeer machine with natural soil. The CEBs were most likely not compacted
as well as ideal. The stabilized soil tested with nearly identical strength regardless of machine.
This is indicative of the consistency of the soil after the addition of portland cement.

Table 3. 14-day compressive strength data

14-day Compressive Strength
Soil/Machine
Samples Avg. (psi) St. Dev. (psi) COV (%)
AECT Impact 2001A Natural Soil 12 612 136 22.2
Vermeer BP714 Natural Soil 12 368 97 26.5
AECT Impact 2001A Stabilized Soil 12 972 204 21.0
Vermeer BP714 Stabilized Soil 12 773 154 19.9

After 14 days, once again the AECT Impact 2001A CEBs with natural soil exhibited much
greater compressive strength than the blocks made with the Vermeer BP714. Blocks made with
stabilized soil in the AECT Impact 2001A showed a nearly 300 psi increase over the seven days
since the previous test age. The blocks made with stabilized soil in the Vermeer BP714 showed
no strength increase between seven and 14 days. This was an unexpected result attributed to
the moisture retained in the different block types. The Vermeer blocks were much wetter in
appearance and feel at 14 days than were blocks made with the AECT machine.

Table 4. 28-day compressive strength data

28-day Compressive Strength
Soil/Machine
Samples Avg. (psi) St. Dev. (psi) COV (%)
AECT Impact 2001A Natural Soil 12 746 49 6.6
Vermeer BP714 Natural Soil 12 415 43 10.3
AECT Impact 2001A Stabilized Soil 12 1251 193 15.5
Vermeer BP714 Stabilized Soil 15 1603 218 13.6

Minimal strength increase was seen between 14 and 28 days with natural soil CEBs. The
maximum average compressive strength of the Vermeer natural soil blocks was lower than the

Burroughs, Rushing, Rusche 53

initial average compressive strength of the AECT natural soil blocks. This further showed the
difference in quality of the blocks made with natural soil. The CEBs made with stabilized soil in
the Vermeer BP714 showed a significant strength increase between 14 and 28 days. As
previously mentioned, this is attributed to the rate of drying of the blocks. The significant
strength final strength difference of 352 psi between the stabilized soil blocks showed the value
of having a two stage compression process. The additional compression stage allowed for more
thorough compression of the blocks than the single stage compression used by the AECT
Impact 2001A. The AECT stabilized soil CEBs also showed a strength increase between 14 and
28 days, albeit at a much lower rate than the Vermeer BP714 stabilized soil blocks. Figure 6
shows the compressive strength development of each soil and machine combination versus
time.

Figure 6. Mean compressive strength development versus time

CONCLUSIONS
The data recorded from the natural soil blocks indicated that the AECT Impact 2001A can be
used to produce blocks with significant quality advantages over blocks made with Vermeer
BP714 when the soil used may be in less than desirable conditions. The data recorded from the
stabilized soil blocks showed that, for more uniform and free flowing soil, the Vermeer BP714
produced higher quality blocks than the AECT Impact 2001A. The Vermeer BP714 has a two-
stage compression process that allows for more uniformly consolidated blocks, rather than the
one-stage compressive action of the AECT Impact 2001A. However, the AECT Impact 2001A is

Burroughs, Rushing, Rusche 54

much more forgiving of imperfect soil conditions due to the solid rectangular construction of the
blocks. Because blocks produced with the Vermeer machine have through-holes, the soil must
readily be able to fill around the large pins in the mold. No such problems arise with the AECT
machine, so more clumpy soils are more easily used. As a result of the AECT machine being
more automated than the Vermeer machine, less training time is needed prior to block
production. The through-holes produced by the Vermeer machine allow for construction
advantages and easier handling of the blocks. While the waste can be recycled, there is much
less of it in the production of blocks with the AECT Impact 2001A. Figure 7 shows blocks as
they exit the two machines. Note all of the excess loose soil that is on the top of the Vermeer
block and along the exit chute. This loose material is not present on the AECT block. The AECT
blocks are ready-made for stacking, whereas the Vermeer blocks must be brushed off prior to
stacking. Periodically, this loose material can be added back to the hopper, provided it has not
be contaminated with foreign material, extra water, etc. In conclusion, the AECT Impact 2001A
is a more user-friendly machine that produces perfectly functional CEBs with a wider variety of
soils. The Vermeer BP714 requires more preprocessing of the soil and slightly more operating
effort than the AECT Impact 2001A. However, these added efforts allow the Vermeer BP714 to
produce more robust and consistent CEBs.

Figure 7. Blocks exiting the AECT Impact 2001A (left) and Vermeer BP714 (right).

ACKNOWLEDGEMENTS
The authors would like to acknowledge the Transatlantic Division of the US Army Corps of
Engineers for funding this research study. Additionally, we could not have completed this study
without the help of our many technicians who worked tirelessly to assure that blocks were
produced and cured in the manner prescribed. Special thanks are given to Mr. Lawrence Jetter
and Mr. Jimmy Allen for their help with the AECT Impact 2001A press and to Mr. Adam de Jong
for his help with the Vermeer BP714 press. Permission to publish is granted by the Director,
Geotechnical and Structures Laboratory.

REFERENCES
Dominguez, T. (2011). Guide G-521: ABCs of Making Adobe Bricks. New Mexico State
University, College of Agricultural, Consumer and Environmental Sciences. Las Cruces,
NM: New Mexico State University.

Burroughs, Rushing, Rusche 55

Virtual Operator Modeling Approach for Construction
Machinery

Yu Du
Industrial and Manufacturing Systems Engineering Department
Iowa State University
0068 Black Engineering Building
Ames, Iowa 50011
[email protected]

Michael C. Dorneich
Industrial and Manufacturing Systems Engineering Department
Iowa State University
3028 Black Engineering Building
Ames, Iowa 50011
[email protected]

Brian L. Steward
Agricultural and Biosystems Engineering Department
Iowa State University
2325 Elings Hall
Ames, Iowa 50011
[email protected]

Eric R. Anderson
C&F Dynamics Systems Modeling
Deere & Company
18600 S John Deere Rd
Dubuque, Iowa 52001
[email protected]

Lawrence F. Kane
A&T Dynamic System Modeling
Deere & Company
1100 13th Ave
East Moline, Illinois 61244
[email protected]

Brian J. Gilmore
Advanced Systems Engineering
Deere & Company
One John Deere Place
Moline, Illinois 61265
[email protected]

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
ABSTRACT
Greater understanding of how highly skilled operators achieve high machine performance and
productivity can inform the development of automation technology for construction machinery.
Current human operator models, however, have limited fidelity and may not be useful for
machinery automation. In addition, while physical modeling and simulation is widely employed
in product development, current operator simulation models may be a limiting factor in
assessing the performance envelope of virtual prototypes. A virtual operator modelling approach
for construction machinery was developed. Challenges to the development of human operator
models include determining what cues and triggers human operators use, how human operators
make decisions, and how to account for the diversity of human operator responses. Operator
interviews were conducted to understand and build a framework of tasks, strategies, cues, and
triggers that operators commonly use while controlling a machine through a repeating work
cycle. In particular, a set of operation data were collected during an excavator trenching
operation and were analyzed to classify tasks and strategies. A rule base was derived from
interview and data analyses. Common nomenclature was defined and is explained. Standard
tasks were derived from operator interviews, which led to the development of task classification
rules and algorithm. Task transitions were detected with fuzzy transition detection classifiers.

Key words: human operator modeling—construction machinery—excavator trenching—

machinery operations

INTRODUCTION
Introducing new product features can impact machine performance goals such as higher
productivity or fuel economy. Virtual design, the process by which new features are modeled
and tested in a simulation environment, is typically conducted early in the design process where
it is less expensive to make changes. While machines have been modeled with a fidelity that
enables robust testing, approaches to operator modelling technology are limited, which in-turn
limits the ability of engineers to make solid comparisons in the virtual prototyping stage between
different design alternatives. Given the tightly coupled, non-linear nature of the sub-system
dynamics in off-road vehicles, combined with a strong human-in-the-loop involvement of
operators, dynamic simulation of the complete vehicle system must include the operator,
environment, and working tasks (Filla et al., 2005).

Expert human operators display several characteristics: humans can adapt quickly to context
using prior experience and training; humans have the ability to integrate contextual cues and
strategies; and expert operators can often outperform automated functions. As human operators
gain experience, their operations progress from a primarily knowledge-based behavior, to rule-
based behavior, and finally to skill-based behavior (Rasmussen, 1983). Knowledge-based
behavior depends on explicitly formulated goals and plans. With more practice, operators
become rule-based, where sequences of action become rules to follow. Eventually, the expert
exhibits skill-based behavior, where much of the action takes place without conscious control
(Rasmussen, 1983). These human characteristics are quite different from those of automated
machine systems.

An automated system can significantly improve consistency of repeated tasks in a stable,

controlled environment which does not have much variation. In the research area of
autonomous vehicles, optimal operation methods were identified, which were used to control the
vehicle autonomously for situations without much variation (Bradley, 1998; Wu, 2003). However,

Du, Dorneich, Steward, Anderson, Kane, Gilmore 57

when the operating environment or conditions change within which an automated system
operates, higher-level machine intelligence technologies, beyond closed-loop control, must be in
place for the autonomous system to adapt to these changes. Developing these types of
behavioral responses for autonomous systems is challenging. A robust automation system with
perception of external cues and use of internal goals, may be able to exhibit adaptive behavior.
For this behavior, expert human operator behavior and decision making process may have
great utility. A virtual operator model aims to capture key behaviors of human operators,
enabling autonomous system to adapt to external environment changes.

A virtual operator model is designed explicitly to be independent of the vehicle model. Without
this independence, operator models that are highly tuned for particular vehicle models must be
retuned when vehicle designs are changed. To avoid the cumbersome nature of this tight
dependency, an operator model should adapt to changes in vehicle capabilities such as
available power or mechanical linkage constraints.

The objective of this work was to develop an approach to virtual operator modeling (VOM). The
VOM is an encapsulated model independent from the machine model with a well-defined
interface. The outputs of the VOM are the control inputs to the construction machine model.
The VOM should simulate human operators’ behavior and decision making to generate
appropriate control inputs for vehicle model simulation. The initial phase of this work is
described in this paper.

The methodology used to inform the VOM design included expert operator interviews, machine
data analysis, and task-based state modelling. The ability to have adaptive VOM will enable
enhanced performance analysis including fuel efficiency, productivity, and component loading
and strategies for robust automation technology development.

RELATED WORK
Preliminary studies of operator modeling found in autonomous vehicle research helps to
develop an understanding of how a virtual operator model can be used to improve autonomous
control systems. Data collection, task analysis and human behavior study techniques can be
combined to gather, organize, and represent how human operators perform certain operations,
supported by their strategies, situation awareness, knowledge, and decision making process.

Operator Modeling Approaches

Two virtual operator modeling approaches, task-oriented and reference-oriented, for
construction equipment, were found in the literature. Task-oriented approaches simulated
machine operations that go through a repeated sequence of tasks to accomplish a high-level
goal. Operation modelling was based on a finite state machine and combination of a finite state
machine and controllers for generating machine control inputs for each task making up the
operation (Filla, 2005; Elezaby, 2011). Generally, a decision making module and a command
conditioning module are needed for virtual operator modeling. The decision-making module
perceives and classifies information. The command conditioning module generates appropriate
control inputs. Validation was limited to the comparison of simulated paths with experimental
paths for different vehicle components (Filla, 2005). By contrast, a reference-oriented approach
simulates machines navigating along predefined paths to accomplish some type of operational
goal. For example, a steering controller was developed for vehicles by compensating path
tracking errors (Norris, 2003).

Du, Dorneich, Steward, Anderson, Kane, Gilmore 58

Operator modeling attempts can be also found in research investigating approaches to enabling
the autonomous operation of construction equipment. Operator strategies and behavior were
studied and used to develop the control module for autonomous trenching activities. For
example, adaptation to different obstacles was realized by using different strategies from
discovered through operator behavior studies. It was determine, for example, that human
operators penetrate and drag the bucket for dense soil, while they penetrate and rotate the
bucket for loose soil (Bradley, 1998). Additionally Wu (2003) developed an approach to
automatic adaptation to different materials for a wheel loader dig cycle by using neural network
and fuzzy logic approaches.

Task Modeling Approaches

Data collection and task analysis are human factors methods, which are commonly used for
human subject-related research. Data collection is normally the first step to start the project,
which collects specific information about the tasks, the system, and operation information of
human operators including communication and controls. Common data collection methods
include interviews, questionnaires, and observations (Stanton & Walker, 2005). Task analysis
methods result in hierarchical task analysis, which decomposes the higher level task into
subtasks with detailed information needed to perform the task. In general, task analysis can be
classified as knowledge-based and entity-relationship-based analysis: knowledge-based
analysis focuses on the knowledge in term of objects to the task; entity-relationship-based
analysis focuses on finding the connection between actions and objects (Dix, Finley, Abowd &
Beale, 2004).

Autonomous Control
Autonomy in semi-controlled environments like those associated with construction or agricultural
application requires specification and generation of human-like behavior. Han et al. (2015)
presented a multi-layered design framework for behavior-based autonomous or intelligent
systems. Intelligent systems have the ability to perceive cues from the environment and
machine and plan processes for adapting to different situations. Blackmore et al. (2007)
proposed a behavior-based approach to agricultural field robotics with the capability to perform
operations in unknown field conditions, which includes integrated human-like adaptation ability
with intelligence for perception, and decision making in robotics.

MODELING METHODS
For this project, the excavator trenching operation was selected as the target construction
machine operation for virtual operator development. Excavator trenching is a very common
construction operation, which contains multiple tasks and deals with multiple situations. During
the operation, an operator needs to make a trench at a predetermined location and orientation
with defined dimensions, and dump the material either in a defined area or into a truck.
Operators tend to work at their maximum ability to finish trenching as soon as possible. To
automate the trenching operation, an autonomous system must mimic human operator behavior
to adapt to different situations or disturbances during the operation.

To develop a virtual operator model that replicates human operator behavior, operator
interviews were first conducted to understand the approach of operators and to collect
information about their behavior. The modeling structure described in Figure 1 describes the
elements of the VOM.

Du, Dorneich, Steward, Anderson, Kane, Gilmore 59

 Figure 1. The virtual operator model interacts with the vehicle model through a well-
defined interface.

The vehicle model provides machine signals like cylinder extension length and velocity. Vehicle
data can be translated into the absolute position and orientation of machine implement elements
such as buckets and booms through a kinematic model of the machine. These dynamic
variables are closely related to the visual cues that the human operator uses for decision
making during the work cycle. The signal flow in this operator model and vehicle model structure
forms a closed loop for simulation.

The virtual operator model consists of three modules: the human perception module, the task
model module, and the control model module. The human perception module is responsible for
acquiring the vehicle model measurements and transforming them into information at the human
perception level that human operators use for decision making. The vehicle model produces
measurements such as cylinder displacements. However, humans perceive the vehicle in terms
of position and location, such as bucket height. A kinematic model translates raw vehicle data
into human-level perceptual cues. The human perception model triggers transitions between
tasks by interpreting these visual cues and sounds to trigger transitions between tasks. The task
model module uses the results from the human perception module to determine the sequence
and status of tasks. The control model module uses task goals, modified by external conditions
like soil condition, environmental condition, to generate control inputs for the vehicle model.

Three steps were followed to develop a modeling approach for virtual operator modeling. The
first step was to interview operators, observe the operation, and acquire machine data.
Secondly, the operation was analyzed to define the tasks and relate machine data to those
tasks. The third step developed the transition detection classifier to identify transitions between
tasks, which led to a state sequence model.

Data Collection
Operator interviews were conducted to gain a deeper understanding of the information needed
and operator behaviors during the trenching operation. An interview protocol was developed to
determine an operator’s operating experience, behavior, strategies, and possible problems

Du, Dorneich, Steward, Anderson, Kane, Gilmore 60

during operation. Three participants with different backgrounds and skill levels participated in
the interviews. With experience from a wide range of different machines, the interviews provided
general information about the trenching operation in general and was not limited to specific
types of machines. The interview protocol was used to guide the interviews, and written notes
as well as audio records were collected from the interviews. In addition, operators reviewed
recorded videos of themselves operating the machinery. They reviewed their operation and
walked through the video in a think-aloud technique (Lewis, 1982; Ericsson & Simon, 1993) with
verbal identification of tasks, needs, goals, strategies, and behavior. This work collected both
descriptive data as well as quantitative data, which enabled a combination of knowledge based
and entity relationship based analysis for accurate task analysis.

Machine data recorded during predefined excavator trenching operations were used for analysis
of operator’s operational behavior. An excavator was equipped with cameras inside the cab and
outside the cab, which captured both video and audio, an eye tracking sensor to record and
track the real-time view field of the operators, and sensors were used to log data produced from
the machinery itself. Machine operation data were collected during the operation with signal
channels of operator inputs, cylinder positions, and relative speed and direction.

Task and Data Analysis

Task analysis was used to represent the trenching operation and to support a detailed
understanding of trenching. Tasks were identified and described based on operator interviews
and operation observations. Perception of cues, selection of strategies, and subtasks was
summarized in detail.

Machine data were analyzed and related to the task analysis, which described the tasks
identified for trenching and specified control input information related to each task. Data analysis
was performed on the machine data to identify different tasks defined from task analysis, which
represents the actual operation information about the sequence, timing information, and control
inputs of the tasks.

Task and data analysis resulted in a qualitative task analysis and a quantitative data-based task
classification, which provided an accurate description of the trenching operation. A task model
was established to represent the trenching operation with detailed information about human
operator behavior, strategies, and control input for each task.

Fuzzy Transition Detection Classifiers

By classifying numerical signals into human perceivable information used for reasoning, fuzzy
transition detection classifiers were developed for successful detection of transitions between
tasks. Transition detection classifiers predicted the transitions between tasks, which can inform
the virtual operator model when to generate control inputs for the next task. Based on machine
data, the dynamic state of excavator implement elements were determined through a kinematic
model, which can be translated to human perceivable descriptive information using fuzzy
transition detection classifiers. Fuzzy transition detection classifiers were used to identify the
transition between tasks based on some of these common cues and triggers that operators use.
To develop the fuzzy transition detection classifier, membership functions were determined to
describe each of signals required for rules, which were classified into different levels with
descriptive language similar to how human operators perceive the signals. A set of rules were
derived based on the classified human perceivable signals to identify transitions between tasks,
which serve as reasoning statements in the fuzzy transition detection classifiers. The outputs

Du, Dorneich, Steward, Anderson, Kane, Gilmore 61

from the fuzzy transition detection classifiers represent the transitions between the tasks. By
successfully identifying the transitions between tasks, the state model was completed, which
represents the current state of the operation and the start of the next state.

MODELING RESULTS
Task and Data Analysis Results
The tasks and sub-tasks of the trenching operation were identified from the operator interviews
and are summarized in a nominal task timeline (see Figure 2). Five main tasks were identified
within the trenching operation: bucket filling, bucket lifting, swing to dump, dumping, and swing
back to trench. The timing of the start and end of each task was estimated through review and
analysis of video of a trenching operation for one of the participants. It was noted in both
interviews and video analysis that the tasks overlap. Task overlap was a consistent theme
among all participants – one participant said that the more expert the operator, the more he or
she can overlap tasks to increase efficiency and reduce cycle time. While the video analysis of
timing provided a qualitative estimation of task overlap, vehicle data analysis (described later in
this section) was used to obtain more precise estimates of task timing.

Figure 2. Task timeline based on human operator interviews. Task start and end times
were calculated based on analysis of videos of trenching operations.

The durations of specific tasks were overlaid on traces of the machine cylinder extension
lengths, swing speed, and operator inputs (see Figure 3). The topmost graph of Figure 3 shows
the cylinder extension positions for the Boom, Arm, and Bucket. The remaining graphs show
operator control inputs for Swing, Boom, Arm, and Bucket. All these signals were used to
identify the five tasks: Bucket Filling, Bucket Lifting, Swing to Dump, Dumping, and Swing to
Trench within two work cycles. Rectanglar bars with different colors were used to show the start
time point and end time point of tasks with task names on them. Timing information of tasks
could be directly read from the diagram.

Du, Dorneich, Steward, Anderson, Kane, Gilmore 62

 Cylinder Extension
Max
Boom Cylinder
Medium Arm Cylinder
Bucket Cylinder
Min
20 25 30 35 40 45
Time in Seconds
Swing Control Input

Max
Swing Control Input
Medium Swing to
Dump Swing to Trench Swing to Dump Swing to Trench
Min
20 25 30 35 40 45
Time in Seconds
Boom Control Input

Max
Boom Control Input
Medium
Bucket Lifting Bucket Lifting
Min
20 25 30 35 40 45
Time in Seconds
Arm Control Input

Max
Arm Control Input
Medium
Bucket Filling Bucket Filling
Min
20 25 30 35 40 45
Time in Seconds
Bucket Control Input

Max
Bucket Control Input
Medium
Dumping Dumping
Min
20 25 30 35 40 45
Time in Seconds

Figure 3. Data Characterization with Task Timing Overlaid on Traces of the Operator
Inputs and Machine Configurations.

Transition Detection
Task transition identification aims to predict when the operator transitions attention from one task
to another based on vehicle state. Measured vehicle signals were classified into human
perceivable descriptive information, which enabled human-like reasoning rules. For example, by
comparing the bucket height to the ground surface, three levels were defined for states
BelowSurface, NearSurface, and AboveSurface.

INALTable 1 contains the example of rules used in the fuzzy transition detection classifiers to
detect the transition between Swing to Trench and Bucket Filling.

With this information, the operator model transitions through the task model and provides
correct reference inputs to the controller and task model. Figure 1 represents the transition
identification result between Swing back to Trench and Bucket Filling. The green line represents
the fuzzy classification result for the transition detection. The blue line represents the ground
truth task for Bucket Filling, which describes when Bucket Filling starts and ends. By
comparison of the traces, the correct detection happens when the green line starts to rise
slightly ahead of blue line, since the goal of the classifier is to predict a transition between tasks.
If the green line raises later than the blue line, the transition is detected late.

Du, Dorneich, Steward, Anderson, Kane, Gilmore 63

INALTable 1. Transition Rules to detect the transition between the Swing and Bucket
Filling tasks.

From Swing to Trench to Bucket Filling:

1. If (BucketHeight is BelowSurface) and (SwingAngle is NearTrench) and
(ExtensionDistance is Extended)
then (BucketFillTransition is BucketFill)
2. If (BucketHeight is BelowSurface) and (SwingAngle is NearTrench) and
(ExtensionDistance is MidRange)
then (BucketFillTransition is BucketFill)
3. If (BucketHeight is BelowSurface) and (SwingAngle is NearTrench) and
(ExtensionDistance is Retracted)
then (BucketFillTransition is BucketFill)
4. If (BucketHeight is AboveSurface)
then (BucketFillTransition is Swing2Dig)
5. If (BucketHeight is NearSurface) and (SwingAngle is NearTrench) and
(ExtensionDistance is Extended)
then (BucketFillTransition is Swing2Dig)

Figure 4. Transition Detection Results between Swing back to Trench and Bucket Filling

Du, Dorneich, Steward, Anderson, Kane, Gilmore 64

By combining of all identified transitions within the trenching operation, the sequence of tasks
and current state of the operation can be represented. Figure 5 visualizes the sequence of the
tasks with information about when each task starts, which can be considered as a state
sequence model. An accurate task sequence is important timing and state information for
control signal generation.

Figure 5. State Sequence derived from Fuzzy Transition Detection Classifiers to

Represent the Transition Time to each Task

DISCUSSION AND CONCLUSION

Virtual operator modelling based on studies of human operator perception, decision making,
and behaviors can be used to improve development of autonomous and robotics systems and
improve evaluation methods in virtual environments. It has a direct benefit for automating
construction activities. Initial work focused on construction vehicle operation; future work will
expand the application area to agricultural operations, using similar approaches. The main
challenges of developing virtual operator models are understanding the ability of human
operators to adapt, modeling operator perception, and modeling operator decision making. This
paper introduced methods used to collect data and identify information needed to start
modelling. A fuzzy logic task transition model has shown promising performance in correctly
identifying transitions between tasks. Expert behavior has in part been identified as the ability of
operators to overlap tasks within an operation. Future work will include developing methods for
identifying overlapping tasks, modeling overlap, and generating control inputs.

ACKNOWLEDGEMENTS
This work was funded by Deere & Company. The opinions expressed herein are those of the
authors and do not necessarily reflect the views of Deere & Company.

Du, Dorneich, Steward, Anderson, Kane, Gilmore 65

REFERENCES
Dix, A., Finley, J., Abowd, G. & Beale, G. (2004). Human-computer interaction. England:
Pearson Education Limited.
Bradley, D., & Seward, D., (1998). The Development, Control and Operation of An Autonomous
Robotic Excavator. Journal of Intelligent and Robotic Systems 21: 73-97, 1998.
Elezaby, A. A. (2011). Virtual Autonomous Operator Model for Construction Equipment
Applications. Dissertation, University of Illinois – Chicago, Chicago, Illinois.
Filla. R. (2005). Operator and Machine Models for Dynamic Simulation of Construction
Machinery. Thesis No. 1189. Linkopings University, Linkoping, Sweden.
Filla, R., Ericsson, A., & Palmberg, J.-O. (2005). Dynamic Simulation of Construction Machinery:
Towards an Operator Model. International Fluid Power Exhibition 2005 Technical
Conference, Las Vegas, (NV), USA, pp. 429-438.
Han, S., Steward, B. L., and L. Tang. (2015). Intelligent Agricultural Machinery and Field
Robots. In Precision Agriculture Technology for Crop Farming, ed. Zhang, Q. CRC Press.
Norris, W. R. (2001). A Design Framework for Qualitative Human-in-the-Loop System
Development. Dissertation, University of Illinois at Urbana Champaign, Champaign, Illinois.
Rasmussen, J. (1983). Skills, rules, and knowledge; signals, signs, and symbols, and other
distinctions in human performance models. Systems, Man and Cybernetics, IEEE
Transactions, (3), 257-266.
Stanton, N. A., & Walker, G. H. (2013). Human factors methods: a practical guide for
engineering and design. Ashgate Publishing, Ltd.
Wu, L., (2003). A Study on Automatic Control of Wheel Loaders in Rock/Soil Loading. Ph.D.
thesis, University of Arizona.

Du, Dorneich, Steward, Anderson, Kane, Gilmore 66

Vision-Based Machine Pose Estimation for Excavation
Monitoring and Guidance

Chen Feng
Kurt M. Lundeen
Suyang Dong
Vineet R. Kamat
Department of Civil and Environmental Engineering
University of Michigan
2350 Hayward Street, 2340 G.G. Brown Building
Ann Arbor, MI 48109-2125
[email protected]

ABSTRACT
The ability to automatically estimate the pose of an articulated machine's base (e.g., tracks or
wheels) and its articulated components (e.g., stick and bucket) is crucial for technical innovations
aimed at improving both safety and productivity in several construction tasks. Vision-based pose
estimation, in which optical cameras monitor fiducial markers to determine their three dimensional
relative pose (i.e., position and orientation), offers a promising low cost alternative to currently
available sensor packages that are non-ubiquitous and cost prohibitive. This paper presents an
overview of the design and implementation of this technique. A computer vision based solution
using a network of cameras and markers is proposed to enable a three-dimensional pose tracking
capability for articulated machines. First, markers are installed on both the machine’s components
and the jobsite. The site markers are installed and surveyed so their poses are known in a project
coordinate frame. Cameras then observe the markers to measure their relative transformations,
and calculate the machine components’ poses in the project frame. Several prototypes have been
developed to evaluate the system’s performance on an articulated excavator. Through extensive
sets of uncertainty analyses and field experiments, the proposed method has been shown to
achieve centimeter level depth tracking accuracy at a 15m range with only two ordinary cameras
and multiple markers, providing a flexible and cost-efficient alternative to other commercial
products that use infrastructure dependent sensors like GPS. A working prototype has been
tested on several active construction sites with positive feedback from excavator operators
confirming the solution’s effectiveness.

Key words: camera—excavator—guidance—marker—pose

INTRODUCTION
The construction industry has long been affected by high rates of workplace injuries and fatalities.
According to the Census of Fatal Occupational Injuries (CFOI) report (Bureau of Labor Statistics,
2013), the construction industry had the largest number of fatal occupational injuries, and in terms
of rate ranked the fourth highest among all industries.

In addition to the safety concerns, there are also increasing concerns of relatively stagnant
productivity rates and skilled labor shortage in the construction industry. For example, recently
the construction sector in the United Kingdom is reported to be in urgent need of 20% more skilled

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
workers and thus 50% more training provision by 2017, to deliver projects in planning
(LCCI/KPMG, 2014).

Excavation is a typical construction activity affected by the safety and productivity concerns
mentioned above. Excavator operators face two major challenges during excavation operations,
described as follows:

First is how to maintain precise grade control. Currently, grade control is provided by employing
grade-checkers to accompany excavators during appropriate operations. Grade-checkers
specialize in surveying and frequently monitor the evolving grade profile. The evolving grade
profile is compared to the target grade profile and this information is communicated by the grade-
checker to the excavator operator. The operator reconciles this information and adjusts the
digging strokes accordingly. This process is repeated until the target profiles are achieved.
Employing grade-checkers is not only dangerous but also results in a significant loss in excavation
productivity due to frequent interruptions required for surveying the evolving profile.

Second is how to avoid collisions to either human workers, buried utilities, or other facilities,
especially when excavator operators cannot perceive the digging machine’s position relative to
hidden obstructions (i.e., workers or utilities) that it must avoid. According to the aforementioned
CFOI report, among all the causes for the 796 fatal injuries in the U.S. construction industry in
2013, the cause of striking by object or equipment comprised 10 percent. This percentage is even
higher in other industries such as agriculture (19%), forestry (63%), and mining (23%). Besides
directly causing fatal injuries on jobsites, construction machines can also inadvertently strike
buried utilities, thus disrupting life and commerce, and pose physical danger to workers,
bystanders, and building occupants. Such underground strikes happen with an average frequency
of about once every six minutes in the U.S. by the nation's leading organization focused on
excavation safety (Common Ground Alliance, 2015). More specifically, excavation damage is the
third biggest cause of breakdowns in U.S. pipeline systems, accounting for about 17% of all
incidents, leading to over 25 million annual utility interruptions (US DOT PHMSA, 2015).

Automation and robotics in construction (ARC) has been extensively promoted in the literature as
a means of improving construction safety, productivity and mitigating skilled labor shortage, since
it has the potential to relieve human workers from either repetitive or dangerous tasks and enable
a safer collaboration and cooperation between construction machines and the surrounding human
workers. In order to apply ARC and increase intelligence of construction machines to improve
either safety or productivity for excavation and many other activities on construction jobsites, one
of the fundamental requirements is the ability to automatically and accurately estimate the pose
of an articulated machine (e.g., excavator or backhoe). The pose here includes the position and
orientation of not only the machine base (e.g., tracks or wheels), but also each of its major
articulated components (e.g., stick and bucket).

When a construction machine can continuously track its end-effector's pose on the jobsites, such
information can be combined together with the digital design of a task, either to assist human
operators to complete the task faster and more efficiently, or to eventually finish the task
autonomously. For example, an intelligent excavator being able to track the pose of its bucket can
guide its operator to dig trenches or backfill according to designed profiles more easily and
accurately with automatic grade-check. This can eventually lead to fully autonomous construction
machines. When construction machines becomes more intelligent, it can be expected to save
time in training operators and thus to mitigate skilled labor shortage and also improve productivity.

Feng, Lundeen, Dong, Kamat 68

On the other hand, when construction machines are aware of the poses of their components at
any time and location on jobsites, combined with other abilities such as the recognition of human
workers' poses and actions, such machines will be able to make decisions to avoid striking human
workers, for example by sending alerts to their operators or even temporarily taking over the
controls to prevent accidents. Thus it will help to decrease the possibilities of those injuries and
fatalities and improve the safety on construction jobsites. Similarly, with continuous tracking of
the pose of its end-effector (e.g., a bucket of an excavator), an intelligent excavator could perform
collision detection with an existing map of underground utilities and issue its operator a warning
if the end-effector's distance to any buried utilities exceeds some predefined threshold.

Thus, from a safety, productivity, and economic perspective, it is critical for such construction
machines to be able to automatically and accurately estimate poses of any of their articulated
components of interest. In this paper, a computer vision based solution using planar markers is
proposed to enable such capability for a broad set of articulated machines that currently exist, but
cannot track their own pose. A working prototype (Figure 1) is implemented and shown to enable
centimeter level excavator bucket depth tracking.

Figure 1. Overview of the SmartDig prototype

The remainder of the paper is organized as follows: Related work is reviewed first. The authors’
technical approach is then discussed in detail. The experimental results are presented afterwards.
Finally, the conclusions are drawn and the authors’ future work is summarized.

PREVIOUS WORK
The majority of the construction machines on the market do not have the ability to track their
poses relative to some project coordinate frames of interest. To track and estimate the pose of an
articulated machine, there are mainly four groups of methods.

First are the 2D video analysis methods, stimulated by the improvement in computer vision on
object recognition and tracking. Static surveillance cameras were used to track the motion of a
tower crane in (Yang, Vela, Teizer, & Shi, 2011) for activity understanding. Similarly in
(Rezazadeh Azar & McCabe, 2012) part based model was used to recognize excavators for
productivity analysis. This type of methods generally require no retrofitting on the machine, but
suffers from both possibilities of false or missed detection due to complex visual appearance on
jobsites and the relative slow processing speed. Although real-time methods exist as in
(Memarzadeh, Heydarian, Golparvar-Fard, & Niebles, 2012; Brookshire, 2014), they either cannot

Feng, Lundeen, Dong, Kamat 69

provide accurate 6D pose estimation, or require additional information such as a detailed 3D
model of the machine.

Second are stereo vision based methods. A detailed 3D model of the articulated object was
required in (Hel-Or & Werman, 1994) in addition to stereo vision. A stereo camera was installed
on the boom of a mining shovel to estimate pose of haul trucks in (Borthwick, Lawrence, & Hall,
2009), yet the shovel’s own pose was unknown. In (Lin, Lawrence, & Hall, 2013) the shovel’s
swing rotation was recovered using stereo vision SLAM, yet the pose of its buckets was not
estimated. This type of methods can be infrastructure independent if with SLAM, yet some
problems (sensitivity to lighting changes or texture-less regions) remain to be resolved for more
robust applications.

Third are laser based methods, e.g., (Duff, 2006; Kashani, Owen, Himmelman, Lawrence, & Hall,
2010; Cho & Gai, 2014), which rooted from the extensive use of laser point clouds in robotics.
This type of methods can yield good pose estimation accuracy if highly accurate dense 3D point
clouds of the machine are observed using expensive and heavy laser scanners. Otherwise with
low quality 2D scanners, only decimeter level accuracy was achieved [9] (Kashani, Owen,
Himmelman, Lawrence, & Hall, 2010).

Finally are angular sensor based methods, such as (Ghassemi, Tafazoli, Lawrence, & Hashtrudi-
Zaad, 2002; Cheng & Oelmann, 2010; Lee, Kang, Shin, & Han, 2012). They are usually
infrastructure independent and light-weight, but the resulting pose estimation is either not
accurate enough or sensitive to changes of magnetic environment which is not uncommon in
construction sites and can lead to large variations in the final estimation of the object poses.
Moreover this type of methods only estimate the articulated machine's pose relative to the
machine base itself, if without the help of sensors dependent on infrastructure that consume
power and need careful maintenance like GPS. However the use of GPS brings several technical
challenges. For example, construction sites in a lot of cases do not have good GPS signals to
provide accurate position estimation when these sites are located in urban regions or occluded
by other civil infrastructure such as under bridges. Sometimes GPS signals could even be blocked
by surrounding buildings on jobsites and thus fail to provide any position estimation (Cui & Ge,
2003). In addition, since the GPS only provides 3D position estimation, to get the 3D orientation
estimation one needs at least two GPS receivers at different locations of a rigid object. When the
object is small, such as a mini-excavator's bucket, the estimated 3D orientation's uncertainty will
be high.

TECHNICAL APPROACH
In this section, different versions of the proposed articulated machine pose estimation system
design are explained first. Then, the process to calibrate this system is described. Finally,
uncertainty analysis is explored for the system with some important observations of the
relationship between the system configuration and its stability, i.e., uncertainty of the estimated
pose.

System Design
As mentioned previously, this computer vision based articulated machine pose estimation solution
relies on a method called marker based pose estimation. Generally, marker based pose
estimation firstly finds a set of 2D geometry features (e.g., points or lines) on an image captured
by a calibrated camera, then establishes correspondences between another set of 2D or 3D
geometry features on a marker whose pose is known with respect to a certain coordinate frame

Feng, Lundeen, Dong, Kamat 70

of interest, and finally estimates the pose of the camera in that coordinate system. If 2D-2D
correspondences are used, the pose is typically estimated by homography decomposition. If 2D-
3D, the pose is typically estimated by solving the perspective-n-point (PnP) problem. Two typical
marker-based pose estimation methods are AprilTag (Olson, 2011) and KEG (Feng & Kamat,
2012) algorithms.

There are two ways of applying marker based pose estimation for poses of general objects of
interest. As shown in Figure 2, one way is to install the calibrated camera 1 rigidly on the object
of interest (in this case, the cabin of the excavator), and pre-survey the marker 1's pose in the
project coordinate frame. The other way is to install the marker 2 rigidly on the object (in this case,
the stick of the excavator), and pre-calibrate the camera 2's pose in the project coordinate frame.
As long as the camera 2 (or the marker 1) stays static in the project coordinate frame, the pose
of the excavator's stick (or the cabin) can be estimated in real-time.

Figure 2. Two examples of basic camera marker configuration

However, these basic configurations don't always satisfy application requirements. For example,
if only the camera 1 and the marker 1 are used, the excavator's stick pose cannot be estimated.
On the other hand when only the camera 2 and the marker 2 are used, once the stick leaves the
field of view (FOV) of the camera 2, the stick's pose becomes unavailable as well. Thus it is
necessary to take a camera's FOV into consideration when designing an articulated machine
pose estimation system. This understanding leads to the camera marker network design
proposed as follows.

Camera Marker Network

A camera marker network is an observation system containing multiple cameras or markers for
estimating poses of objects embedded in this system. It can be abstracted as a graph with three
types of nodes and two types of edges (e.g., Figure 3). A node denotes an object pose (i.e. the
local coordinate frame of that object), which can be a camera, a marker, or the world coordinate
frame. An edge denotes the relative relationship between two objects connected by this edge,
which can be either image point observations for the previously mentioned marker based pose
estimation, or a known pose constraint (e.g., through calibration).

Feng, Lundeen, Dong, Kamat 71

 w Camera i
i Frame
Marker j
j Frame

3 1 1 w World
Frame
Pose
Observation
Pose
3 2 2 Constraint

Figure 3. An example graph of a camera marker network

Thus, if at least one path exists between any two nodes in such a graph, the relative pose between
them can be estimated. In addition, any loop in the graph means a constraint of poses that can
be used to improve the pose estimation. For example, in Figure 3, marker 2's pose in the world
frame can be found via a path through camera 3 whose own pose in the world frame is pre-
calibrated. The marker 2's pose can also be better estimated when observed by the rigidly
connected camera 1 and 2 whose relative pose is pre-calibrated, since a loop is created.

Applying this concept to articulated machine pose estimation results in numerous possible
designs. One of the possible camera marker networks is shown in Figure 4, camera 1 observes
the benchmark while camera 2 observes the stick marker, and the rigid transformation between
the two cameras is pre-calibrated. Thus as long as the two markers stay inside the two cameras'
FOV respectively, the stick's pose in the world frame can be estimated. It is worth noting that this
only illustrate a simple configuration. With more cameras and markers in the network, there are
more chances of creating loops and thus improving pose estimation, especially considering that
surveillance cameras are becoming popular in construction jobsites whose poses can be pre-
calibrated and thus act as the camera 3 in Figure 3.

Camera-Marker Graph

S w

2 1 B

Figure 4. Multiple-camera multiple-marker configuration

Prototypes
Multiple prototypes have been implemented to realize the above described camera marker
network designs. Figure 5 demonstrates one of the early prototypes implementing a single-
camera multiple marker configuration. A mechanical device driven by a synchronous belt was

Feng, Lundeen, Dong, Kamat 72

adopted to map the relative rotation between the excavator bucket and the stick to the relative
rotation between the stick marker and the rotary marker. This implementation enables pose
tracking of the excavator bucket.

Figure 5. Synchronous belt prototype design

The prototype functions by means of two markers. The first marker, termed stick marker, is rigidly
attached to the stick with a known relationship to the bucket’s axis of rotation. The second marker,
termed rotary marker, is attached at a location removed from the vicinity of the bucket. The rotary
marker is constrained with one degree of rotational freedom and a known angular relationship to
the bucket. If the bucket’s geometry is also known, or measured onsite, then all necessary
information is available to deduce tooth pose, as shown in Figure 6.

Stick Marker

Rotary Marker

Figure 6. Bucket tooth pose calculation in the synchronous belt prototype

Due to the potential interference of the rotary marker and any obstructions during excavation, the
above synchronous belt prototype was slightly modified and evolved to the current prototype as
shown in Figure 7. The newer working prototype implements the multiple-camera multiple-marker
configuration similar to Figure 5. Two cameras are rigidly mounted forming a camera cluster. A
cable potentiometer is installed on the bucket’s hydraulic cylinder to track the relative motion of
the excavator bucket and the stick even if the bucket is deep inside the earth. In addition to
possessing a cable potentiometer for measuring linear displacement, the device contains a

Feng, Lundeen, Dong, Kamat 73

microcontroller for signal conversion, a radio for wireless transmission, and a battery for power,
all of which are mounted inside an enclosure for protection.

Figure 7. Cable potentiometer prototype

System Calibration
Three types of calibration are necessary for an articulated machine pose estimation system
implementing the above camera marker network design.

The first type is intrinsic calibration which determines internal parameters (e.g., focal length) of all
cameras in the system. This is done using same methods as in (Feng, Xiao, Willette, McGee, &
Kamat, 2014).

The second type is extrinsic calibration which determines relative poses (e.g. dotted edges in the
graph) designed to be calibrated before system operation. There are two kinds of such poses: 1)
poses of static markers in the world frame, and 2) poses between rigidly connected cameras. The
first kind of poses can be calibrated by traditional surveying methods using a total station. The
second kind of poses, however, cannot be directly surveyed physically since a camera frame's
origin and principal directions usually cannot be found or marked tangibly on that camera. Thus
to calibrate a set of m rigidly connected cameras, a camera marker graph needs to be constructed
as denoted in Figure 8. A set of n markers' poses need to be surveyed in the world frame. Then
when the m cameras observe these n calibration markers, the graph is formed to estimate each
camera's pose in the world frame and thus their relative poses between each other (i.e., edges
with question mark) are calibrated. It is suggested to ensure that multiple loops exist in this graph
to improve the accuracy of the poses to be calibrated. Such loop exists as long as at least two
markers are observed by a same camera simultaneously. It is also worth noting that with enough
many calibration markers, each camera's intrinsic parameters can be further optimized together
with their extrinsic parameters.

1 2 3 …... n

1 ? 2 ? m
Figure 8. A camera marker graph for extrinsic calibration

Feng, Lundeen, Dong, Kamat 74

The third and final type of calibration involves mapping a relationship between the cable
potentiometer output and the pose of the excavator’s bucket. The calibration process requires a
marker to be attached to the side of the bucket for the duration of the calibration process. The
system is then calibrated by moving the bucket through its range of motion and mapping the
potentiometer output to the relative transformation between the stick marker and bucket teeth.
After the calibration process, the marker is removed from the bucket. The system then functions
by measuring stick marker pose and cylinder stroke length, and mapping such measurements to
bucket tooth pose.

Uncertainty Analysis
It is not sufficient to only estimate the pose of an articulated machine. The uncertainty of the
estimated pose is critical for the following reasons. Firstly the uncertainty provides a measure of
the confidence level of the estimated pose, which is necessary for many downstream applications
(e.g., deciding buffer size for collision avoidance). Secondly it serves as a tool for evaluating the
stability of the pose estimation system under different configurations, and thus further guiding to
avoid critical configurations that lead to unstable pose estimation.

To perform uncertainty analysis on the proposed camera marker network pose estimation system,
the system is firstly abstracted as the following state space model:

Z F( X ; Y , C ) (1)

where X is the state vector of the network (usually encodes the poses of nodes in the graph), Z
is the predicted measurement vector containing image coordinates of all the points projected from
markers, Y is the known parameters (usually contains marker points' local coordinates), C is the
calibrated parameters (usually encodes all cameras' intrinsic parameters and all calibrated
poses), and F is the system's observation function parameterized by Y and C, i.e., the camera
perspective projection function.

For example, for a network of a single camera and a single marker, X is a 6 u 1 vector that
encodes the marker's orientation and position in the camera frame; Y is a 3n u 1 vector containing
n marker points' coordinates from surveying; C is a vector of the camera intrinsic parameters. If
another marker is added to this network, Y should be extended with points on the new marker.

Uncertainty Propagation
No matter how complex such a network is and what method is used to get an initial estimate of X
(either PnP or homograph decomposition), the optimized state X̂ can be calculated by the
following least square optimization, i.e., bundle adjustment:

2
ˆ
X arg min Zˆ  F( X; Y, C) (2)
X PZˆ

where PẐ is the a priori covariance matrix of the actual measurements Ẑ , typically assumed as
V u2 I when image coordinates are measured with a standard deviation of V u .

Feng, Lundeen, Dong, Kamat 75

To backward propagate the measurement uncertainty PẐ to the uncertainty of the optimized state
X̂ requires linearization of F around X̂ . Since the error is assumed to come from only the
measurements (the uncertainty in calibrated parameters C can be included in future work, but is
assumed to be negligible in this paper), one can directly apply the results in (Hartley & Zisserman,
2000) to calculate the uncertainty of the optimized states:

PXˆ ( J T PZˆ1J ) 1 V u 2 ( J T J ) 1 (3)

wF
where J is the Jacobian matrix of F evaluated at X̂ .
wX ˆ
X

Uncertainty and Configuration

Equation (3) not only provides a means of evaluating uncertainty of the optimized pose estimation
of a camera marker network, but also provides a tool to predict the system stability at any given
system configuration X before even making any measurements. This is done by evaluating the
Jacobian matrix J of F at that X, and then applying equation (3) to predict the covariance matrix.
It is based on the fact that the aforementioned backward propagation of measurement uncertainty
does not directly rely on specific measurements. In fact it directly relies on the system
configuration X around which the linearization is performed. Thus, when evaluating Jacobian
matrix J at a configuration X, equation (3) yields the theoretically best/smallest pose estimation
uncertainty one can expect at that configuration, which denotes the system stability at that
configuration.

Using this method, some important empirical conclusions on the basic single-camera single-
marker system are found about relationships between system stability and configuration, based
on numerical experiments, which are useful for more complex system design and are listed as
follows. Similar analysis will be performed to multiple-camera or multiple-marker system in future
work.

1. The marker's origin/position in the camera frame, c t m , has the largest uncertainty along a
direction nearly parallel to the camera's line of sight to the marker, i.e., c t m itself. Figure 9
exemplifies this observation at two randomly generated poses between the camera and the
marker.

2. The largest uncertainty of marker's position in the camera frame increases approximately
quadratic to the marker's distance to the camera; compared to which the two smallest
uncertainty's increases are almost negligible. Figure 10 shows a typical example.

3. The largest uncertainty of marker's position in the camera frame increases approximately
linear to the camera focal length; compared to which the two smallest uncertainty's increases
are almost negligible. Figure 11 shows a typical example.

Feng, Lundeen, Dong, Kamat 76

 Figure 9. Largest position error direction

Figure 10. Position error vs marker distance

Figure 11. Position error vs focal length

Feng, Lundeen, Dong, Kamat 77

EXPERIMENTAL RESULTS
Feasibility Experiments
Before implementing the pose estimation system prototypes, a set of experiments were performed
to test the feasibility of marker based pose estimation in different indoor/outdoor construction
environments. In all the experiments, AprilTag (Olson, 2011) was chosen as the basic marker
detection and tracking algorithm.

Firstly, the outdoor detectability of markers was tested. A marker's detectability is a function of
many factors including the marker size, the distance between the marker and the camera,
included angle between the camera viewing direction and the marker plane's normal direction,
and also image resolution. Since the distance between the marker and the camera is the most
critical factor affecting the method's feasibility in real applications, this experiment is performed
by fixing other factors and then gradually increasing the distance of the marker in front of the
camera, until the algorithm fails to detect the marker, and recording the distance. Varying other
factors and repeating this process results in Table 1. One can consult this table to decide how
large the marker should be to fit application need.

Table 1. Outdoor detectability of AprilTag

Marker Angle (degree)
Max Detectable Distance (m)
0 45 0 45
0.2 x 0.2 6.10 4.88 11.28 8.84
0.3 x 0.3 8.23 7.01 14.94 11.58
Marker Size (m2)
0.46 x 0.46 13.41 11.28 25.91 21.64
0.6 x 0.6 19.51 16.46 34.44 30.48
Image Resolution 640 x 480 1280 x 960
Focal Length 850 pixels 1731 pixels
Processing Rate 20 Hz 5 Hz

Secondly, illumination is a critical factor affecting performance of many computer vision

algorithms. The AprilTag algorithm was thus tested under various illumination conditions to
examine its robustness for construction applications. Figure 12 shows successful marker
detection under different indoor/outdoor lighting conditions. These experiments and following
extensive prototype tests proved AprilTag based marker detection method's robustness to
illumination changes.

Figure 12. Marker detection vs illumination

Finally, for uncertainty propagation, one needs to have a prior estimation of the image
measurement noise's standard deviation V u . This is achieved by collecting multiple images under
a static camera marker pose. Repeating this process for different poses and collecting

Feng, Lundeen, Dong, Kamat 78

corresponding image measurement statistics lead to an image measurement covariance matrix
Ȉu , which can be further relaxed to V u2 I to include all the data points. Figure 13 shows that
V u 0.2 pixel is reasonable.

Figure 13. Image measurement noise estimation

Mockup Experiments
As previously mentioned, a multiple-camera multiple-marker articulated machine pose estimation
prototype has been implemented with the application of estimating an excavator's bucket depth
in a project frame, which could be used for automatic excavation guidance and grade control.

The top row of Figure 14(a) shows the camera cluster of the prototype in Figure 1, and different
experiment configurations to test the depth estimate's accuracy. The experiments were setup by
observing the two markers in the bottom row of Figure 14(a) using the two cameras in the cluster
respectively. Then the depth difference between the two markers was estimated using the
proposed method, while the ground truth depth difference between the two marker centers was
measured by a total station with 1 mm accuracy. Figure 14(b) illustrates the configurations of
different sets of such experiments, for comprehensive tests of the method's accuracy under
several system and design variations. The first set varies one of the marker's pitch angle (top row
of the figure). The second set varies its height (bottom-left). The third set varies its distance to the
camera (bottom-middle). And the fourth set varies the number of tags used in that marker (bottom-
right).

Marker Pitch

Marker Height Marker Distance Number of Tags

(a) Setup (b) Different configurations
Figure 14. Mockup experiment setup

Feng, Lundeen, Dong, Kamat 79

Figure 15 shows the absolute depth errors comparing the ground truths with the results from
camera marker network pose estimation, in the above mentioned different sets of prototype
experiments, using the box quartile plot. Note that all errors are less than 2.54 cm, even when
observed from more than 10 meters away. Further experiments showed that the system
worked up to 15 meters.

|Depth Error|<1 in
Marker Pitch Marker Height

|Depth Error|<1 in
Marker Distance illumination

Figure 15. Error vs. configuration

Prototype Experiments
Another experiment was conducted to characterize the performance of the cable potentiometer
prototype. The prototype was installed and tested on a Caterpillar 430E IT Backhoe Loader, as
shown in Figure 16. The experiment involved calibrating the sensor system, placing the backhoe
in random poses, using the sensors to estimate bucket tooth height, and comparing estimates
with ground truth measurements obtained using a total station. The experiment’s pass/fail criterion
was set at 2.5 centimeters (1 inch) of absolute error.

Feng, Lundeen, Dong, Kamat 80

 Figure 16. Calibration of cable potentiometer prototype installed on a backhoe

A total of eight trials were conducted. For each trial, three components of tooth position (x, y, and
z, where y corresponds to the zenith direction) were measured and compared with ground truth
measurements, as shown in Figure 17. Of the twenty-four data points collected, only three points
exceeded the pass/fail criterion of 2.5 centimeters (1 inch) of absolute error. Though the system’s
ability to estimate x-position only marginally met the pass criterion, the system’s performance as
a whole was deemed satisfactory, especially considering that accuracy along the zenith direction
is more important in many excavation applications.

ABS Error
3.5
3.0
2.5
ABS Error (cm)

2.0
1.5
1.0
0.5
0.0
x-position y-position z-position

Figure 17. Cable potentiometer experimental results

The cable potentiometer prototype was also tested on an active construction site, as shown in
Figure 1 and Figure 18. The system was used to assist the operator in trenching operations. A
computer screen was mounted in the operator’s cabin, providing a display of bucket height relative
to a desired trench grade specified by the job plans.

Feng, Lundeen, Dong, Kamat 81

 Figure 18. Cable potentiometer prototype installed on excavator

Shown in Figure 19 is a short section of trench in which the operator conducted a side-by-side
comparison of traditional grading versus grading guided by the sensor system. The resulting
trench depth differences between the manual grade and the guided grade (by the prototype) were
less than 1 inch, which fulfils the need of many construction applications.

Traditional Grading

Sensor Guided Grading

Figure 19. Comparison of sensor guided grading versus traditional grading

CONCLUSIONS AND FUTURE WORK

This paper proposed a vision based pose estimation solution for articulated machines using a
camera marker network. The uncertainty of the network pose estimation is analyzed through
backward propagation of measurement covariance matrix. Based on this, an efficient approach
of evaluating such a pose estimation system's uncertainty at any given state is introduced and
applied to the basic single-camera single-marker system to find some important relationships
between system states and corresponding system uncertainty, which is useful to guide more
complex design. The conducted experiments and a working prototype proved the proposed
solution's feasibility, robustness, and accuracy for real world construction applications.

The authors’ current and planned work in this research direction is focused on continuously
improving the estimation accuracy such as taking the uncertainty of calibrated parameters C into

Feng, Lundeen, Dong, Kamat 82

consideration, and also analyzing uncertainty versus system configuration for more complex
camera marker networks.

ACKNOWLEDGMENTS
This research was funded by the United States National Science Foundation (NSF) via Grants
CMMI-1160937, CMMI-1265733, and IIP-1343124. The authors gratefully acknowledge NSF’s
support. The authors also thank Walbridge Construction Company, Eagle Excavation Company,
and the University of Michigan Architecture, Engineering and Construction (AEC) division for their
support in providing access to construction equipment and job sites for experimentation and
validation. The authors would also like to thank Dr. Manu Akula, Dr. Ehsan Rezazadeh Azar, and
Mr. Nicholas Fredricks for their invaluable help during experiments. Any opinions, findings,
conclusions, and recommendations expressed in this paper are those of the authors and do not
necessarily reflect the views of the NSF, Walbridge, Eagle Excavation, or the University of
Michigan. Suyang Dong and Vineet R. Kamat have a significant financial and leadership interest
in a start-up company named Perception Analytics & Robotics LLC (PeARL), and are the
inventors of technology (proposed to be licensed by PeARL) involved in or enhanced by its use
in this research project.

REFERENCES
Borthwick, J., Lawrence, P., & Hall, R. (2009). Mining haul truck localization using stereo vision.
Proceedings of the Robotics and Applications Conference, 664, p. 9.
Brookshire, J. (2014). Articulated pose estimation via over-parametrization and noise projection.
Cambridge: MIT.
Bureau of Labor Statistics. (2013). Census of Fatal Occupational Injuries (CFOI) - Current and
Revised Data. Retrieved February 3, 2015, from http://www.bls.gov/iif/oshcfoi1.htm
Cheng, P., & Oelmann, B. (2010). Joint-angle measurement using accelerometers and
gyroscopes—A survey. IEEE Transactions on Instrumentation and Measurement, 59(2),
404--414.
Cho, Y., & Gai, M. (2014). Projection-Recognition-Projection Method for Automatic Object
Recognition and Registration for Dynamic Heavy Equipment Operations. Journal of
Computing in Civil Engineering, 28(5), A4014002.
Common Ground Alliance. (2015, March 03). Survey finds that nearly half of homeowners who
plan to dig this year will put themselves and others at risk by not calling 811 before
starting. Retrieved May 18, 2015, from http://commongroundalliance.com/media-
reports/press-releases/survey-finds-nearly-half-homeowners-who-plan-dig-year-will-put
Cui, Y., & Ge, S. (2003). Autonomous vehicle positioning with GPS in urban canyon
environments. IEEE Transactions on Robotics and Automation, 19(1), 15--25.
Duff, E. (2006). Tracking a vehicle from a rotating platform with a scanning range laser.
Proceedings of the Australian Conference on Robotics and Automation, 12.
Feng, C., & Kamat, V. R. (2012). Plane Registration Leveraged by Global Constraints for
Context-Aware AEC Applications. Computer-Aided Civil and Infrastructure Engineering,
28(5), 325-343.
Feng, C., Xiao, Y., Willette, A., McGee, W., & Kamat, V. R. (2014). Towards Autonomous
Robotic In-Situ Assembly on Unstructured Construction Sites Using Monocular Vision.
Proceedings of the 31th International Symposium on Automation and Robotics in
Construction, (pp. 163-170). Sydney, Australia.
Ghassemi, F., Tafazoli, S., Lawrence, P., & Hashtrudi-Zaad, K. (2002). An accelerometer-based
joint angle sensor for heavy-duty manipulators. Proceedings of IEEE International
Conference on Robotics and Automation, 2, pp. 1771--1776.

Feng, Lundeen, Dong, Kamat 83

Hartley, R., & Zisserman, A. (2000). Multiple view geometry in computer vision. Cambridge
University Press.
Hel-Or, Y., & Werman, M. (1994). Model based pose estimation of articulated and constrained
objects. ECCV (pp. 262--273). Springer.
Kashani, A., Owen, W., Himmelman, N., Lawrence, P., & Hall, R. (2010). Laser Scanner-based
End-effector Tracking and Joint Variable Extraction for Heavy Machinery. The
International Journal of Robotics Research, 29(10), 1338-1352.
LCCI/KPMG. (2014). Skills to Build: LCCI/KPMG Construction Skills Index 2014. Retrieved
February 6, 2015, from
http://www.kpmg.com/uk/en/issuesandinsights/articlespublications/pages/construction-
skills-index-2014.aspx
Lee, S., Kang, M.-S., Shin, D.-S., & Han, C.-S. (2012). Estimation with applications to dynamic
status of an excavator without renovation. Gerontechnology, 11(2), 414.
Lin, L.-H., Lawrence, P., & Hall, R. (2013). Robust outdoor stereo vision SLAM for heavy
machine rotation sensing. Machine vision and applications, 24(1), 205--226.
Memarzadeh, M., Heydarian, A., Golparvar-Fard, M., & Niebles, J. (2012). Real-time and
automated recognition and 2D tracking of Construction workers and equipment from Site
video streams. Int. Workshop on Computing in Civil Engineering.
Olson, E. (2011). AprilTag: A robust and flexible visual fiducial system. Proceedings of the 2011
IEEE International Conference on Robotics and Automation, (pp. 3400-3407).
Rezazadeh Azar, E., & McCabe, B. (2012). Part based model and spatial--temporal reasoning
to recognize hydraulic excavators in construction images and videos. Automation in
construction, 24, 194--202.
US DOT PHMSA. (2015). Pipeline Incident 20 Year Trends. Retrieved February 4, 2015, from
http://www.phmsa.dot.gov/pipeline/library/datastatistics/pipelineincidenttrends
Yang, J., Vela, P., Teizer, J., & Shi, Z. (2011). Vision-based crane tracking for understanding
construction activity. Proc. ASCE IWCCE, 258--265.

Feng, Lundeen, Dong, Kamat 84

A User-centered Approach to Investigate Unmanned
Aerial System (UAS) Requirements for a Department
of Transportation Applications

Masoud Gheisari
Building Construction Science Program
Mississippi State University
823 Collegeview St. 132C Howell Building
Mississippi State MS 39762;
[email protected]

Javier Irizarry
School of Building Construction
Georgia Institute of Technology
245 4th Street, NW, Rm 138
Atlanta, GA 30332
[email protected]

ABSTRACT
This user-centered study explored the usability of Unmanned Aerial Systems (UASs) in Georgia
Department of Transportation (GDOT) operations. The research team conducted a series of
semi-structured interviews with subject matter experts in GDOT divisions. Interviews focused on
(1) the basic goals of the operators, (2) their major decisions for accomplishing those goals, and
(3) the information requirements for each decision. Following an interview validation process, a
set of UASs design characteristics that fulfill user requirements of each previously identified
division was developed. And ultimately five reference systems were proposed using a house-of-
quality viewgraph to illustrate the relationships between GDOT tasks and potential UASs aiding
those operations. Each of these five reference systems (Flying Camera, Flying Total Station,
Perching Camera, Medium Altitude & Long Endurance (MALE), and Complex Manipulation) was
then discussed in terms of its Airframe, Payload, Control Station, and System capabilities.
These technical requirements determined would aid in more rapid development of test UASs for
GDOT use as well as advance GDOT’s implementation of UASs to help accomplish the
Department’s goals.

Key words: human-centered technology—unmanned aerial system (UAS)—house of

quality (HoQ)

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
INTRODUCTION
Innovative applications of Unmanned Aerial System (UAS) for improved mapping operations
take advantage of several inherent characteristics of UAV systems. For instance, aerial video,
collected by visible or infrared video cameras deployed on UAV platforms, is rapidly emerging
as a low cost, widely used source of imagery for response to time-critical disaster applications
(Wu and Zhou 2006) or for the purpose of fire surveillance (Wu et al. 2007).
During the past decade, UASs have been applied in a wide range of traffic management,
transportation and construction disciplines related to Departments of Transportation (DOTs),
including traffic surveillance (1,2), traffic simulation (3,4), monitoring of structures (5,6),
avalanche control (7), aerial assessment of road surface condition (8), bridge inspection (9), and
safety inspection on jobsite (10). States such as Virginia (11), Florida (12), Ohio and
Washington (1), Utah (13) were leading UAS application and implementation within their DOTs.
This chapter introduces a wide variety of UAS applications in traffic management, transportation
and construction disciplines related to DOTs. One of the very recent UAS-based projects
supported by a DOT was in the State of Georgia in which a group of researchers from the
Georgia Institute of Technology explored the feasibility of using this leading edge technology in
several divisions and offices within GDOT (14, 15, 16). The goal of this study was aligned with
the Federal Aviation Administration (FAA) goals of efficient integration of UASs into the nation’s
airspace for civilian and public applications. Integrating UAS into GDOT practices might seem
very beneficial at first glance, but the most important issue that was investigated at the first step
in that GDOT-supported project was to see how usable this technology could be within GDOT
divisions and offices.

RESEARCH METHODOLOGY
Having the UAS might seem very useful for most GDOT practices but the very first issue that
should be resolved is whether this technology would be usable for different applications within
GDOT divisions and offices. A usable UAS should be designed firstly by investigating the user
requirements across all divisions and offices of GDOT and then identifying and developing a set
of design characteristics for the UAS based on the previously identified user requirements. Even
when a real UAS is designed based on user requirements; it should be tested using real users
of the system to evaluate its applicability and usability. The work plan of this research has been
illustrated in Figure 1.

Figure 1. Project Methodology

The research team first studied all GDOT divisions and offices to identify those that were
expected to have a potential for using UASs. This analysis was performed by investigating the

Gheisari & Irizarry 86

operations, mission and sets of responsibilities that each division and their internal offices have.
At the same time, previous uses of UASs across all DOTs as well as the current status of
different civilian applications of UASs were investigated. This step resulted in the identification
of four divisions (of the twelve overall divisions of GDOT) with the highest potential for
benefitting from UAS technology; construction, engineering, intermodal, permits and operation.
Semi-structured interviews were conducted with subject matter experts in each identified
division to identify the basic goals of the operators and the information requirements for each
decision. Once the potential uses of UASs across the GDOT divisions were determined, more
than 40 tasks were identified and selected as the tasks with the greatest potential use of UAV in
the near term. The majority of the tasks are centered around collecting data, providing
information, and decision making based on the data. Currently most of the related data are
collected through field personnel. All of the information was validated through feedback from the
interviewees. Finally, this step resulted in the identification of 19 main tasks as well as their
duration, frequency, and environment. Following the interview validation process, a set of UASs
design characteristics that fulfill user requirements of each previously identified division was
developed. The following sections provide more details on the analysis of GDOT divisions and
technical and operational requirements of UASs.

ANALYSIS OF GDOT DIVISIONS

All divisions and offices of the GDOT were studied to identify the divisions with the highest
potential for benefitting from UAV technology. The GDOT consists of 8 divisions; each relating
to a major area of transportation concern as follows: administration, construction, engineering,
finance, intermodal, local grants and field services, program delivery, and permits and
operations.
This analysis is performed by investigating the operations, mission and sets of responsibilities
that each division and their internal offices might have. Having a clear understating of what
other DOTs have done and determining the current status of civilian application of UAVs were
utilized to identify different divisions of GDOT with potential of applying UAVs. Of the twelve
overall divisions of GDOT, four divisions with the highest potential for benefitting from UAV
technology were selected for further investigation (construction, engineering, intermodal,
permits and operation). Figure 2 shows some of the tasks and responsibilities associated with
each of the selected GDOT divisions. Through a series of interviews with employees at the
division and office level, the user requirements of each identified division/office were
investigated.
The Construction Division is responsible for construction contract administration and overseeing
construction projects and permitting in the State of Georgia. It conducts general construction
oversight and also oversees project advertising, letting and awards, and testing of materials.
Furthermore, it inspects and monitors contractual field work, specifies material requirements,
and provides geotechnical services. Interviews were conducted with eight engineers at
management and operational levels. Its goal is to provide the resources necessary to insure the
quality of construction projects by improving decisions made in the field, making information
available for training and to maintain statewide consistency.

Gheisari & Irizarry 87

 Figure 2. GDOT divisions with the highest potential for benefitting from UAV technology

The Engineering Division develops environmental studies, right-of-way plans, construction plans
and bid documents through a cooperative effort that results in project design and
implementation. Moreover, the division is responsible for supporting and maintaining all
engineering software, engineering document management, and state wide mapping. Four
interviews were conducted with persons in charge of activities conducted by the engineering
division.
The main responsibility of the Intermodal Division is to support and facilitate the development
and implementation of intermodal policies, planning, and projects in the highway program and
organize all major statewide non-highway programs for the development of a comprehensive
transportation system. The intermodal division consists of four main programs: (1) aviation
programs is tasked to guide airport development and to assure a safe and well-maintained
system of public-use airports; (2) transit programs provide transit capital and operating
assistance to the urban and all metropolitan planning organizations in Georgia; (3) rail programs
include track inspection and safety investigation for the Georgia rail system in cooperation with
the Federal Railroad Administration; (4) waterways programs maintain the navigability of the
Atlantic Intracoastal waterway and Georgia's deep water ports in Savannah and Brunswick.
Consequently, five interviews were conducted with members from the intermodal division,
including at least one interviewee from each of the above programs.
The Permits and Operations Division ensures a safe and efficient transportation system by
collecting traffic data, addressing maintenance needs (e.g. related to traffic lights) and
regulating the proper use of the state highway system. In order to improve traffic flow and
coordinate traffic engineering, traffic safety, and incident management statewide, the division
collects traffic data (e.g. flow, speed, counts) using a wide range of devices (e.g. video cameras,
microwave sensors, and computer applications pertaining to traffic services). This division
consists of four offices: Transportation Data, Utilities, Traffic Operations, and Maintenance.
Interviews were conducted with seven engineers at management, district and area office levels.

Gheisari & Irizarry 88

IDENTIFICATION OF OPERATIONAL REQUIREMENTS
In order to determine the operational requirements for UAS usage for a specific division that
could benefit from such technology, a user-centered top-down approach was chosen. In this
user focused approach, the overall tasks and user requirements are categorized into various
functions and components, enabling a comprehensive understanding of users’ goals, their
working environment, and decision-making processes.
The data sample comprised of 24 GDOT employees in the major fields of construction,
engineering, intermodal and traffic operations who volunteered to participate in the study. Prior
to the interview, all participants were required to give informed consent that explains to an
individual who volunteers to participate in the study, the goals, processes, and risks involved.
Therefore, each participant was presented with an Informed Consent Form for him or her to
read in agreement to participate in the study. The university’s Institutional Review Board (IRB)
evaluated and approved the study protocol.
Semi-structured interviews were conducted with subject matter experts in each identified
division to identify the basic goals of the operators and the information requirements for each
decision. Once the potential uses of UASs across the GDOT divisions were determined, more
than 40 tasks were identified and selected as the tasks with the greatest potential use of UAV in
the near term. The majority of the tasks are centered around collecting data, providing
information, and decision making based on the data. Currently most of the related data are
collected through field personnel. All of the information was validated through feedback from the
interviewees. Finally, this step resulted in the identification of 19 main tasks as well as their
duration, frequency, and environment. An example is shown in Figure 3, in which “Construction
Site Measurement (Task 04)” is identified as an outdoor, mobile task in the range of 0.5 to 10
miles, and with an approximate duration of 2-3 hours. The information for this frequent task
(frequency is hourly and/or daily) is based upon the validated data resulting from the interview
process. Of the 7 validated data for construction-related tasks, 71% (5 of 7) of the respondents
believed that the task is performed locally. A local task occurs at one location or a job site that
can be best described by the length and width of the area. However, a distributed task occurs
along a road, a river, or a railway. More details about the tasks and responsibilities associated
with each of the selected GDOT divisions can be found in (14).

Figure 3. Construction Division Task Requirement Matrix

Gheisari & Irizarry 89

Based on the records and interviews, the study team created a list of tools used while
performing the identified tasks. The tools (and technology) used by the GDOT personnel can be
classified under eight categories: Measuring equipment are used in various linear, areas, and
counting measurements. Examples include measuring wheel and tape, distance meter, air
meters, scoops, and thermometer. Surveying equipment (e.g. theodolite, total station, GPS
devices, etc.) are used to produce accurate maps and dimensions of the site. Engineering
software tools are used to collect and process observation data from the workplace. The most
important example is “Site Manager” software that is used by construction office. Computer
hardware, specialized software, and communication devices are used together to collect,
analyze, and transmit data collection and communication processes. Basic hand tools and
digital cameras are frequently used by GDOT employees traveling out-of-office. Finally, many of
the tasks performed by the division of permits and traffic operations require vehicle detection
and mobile application tools. Examples include traffic counter, radar gun, range finder, and
turning movement counter.
Figure 4 shows the result operational requirement matrix that includes each division’s operation,
user characteristics, working environment, and technology use. The frequency of occurrence is
calculated based upon the number of tasks performed in each division. To identify missing
information and errors in the matrix and validate its outcomes, this matrix was then taken back
to the subject matter experts who were interviewed. All of the information was validated through
feedback of the interviewees.

Figure 4. Operational Requirement Matrix

Figure 5 shows the created UAV requirement matrix for the identified divisions within GDOT; the
left part is pertaining to the notion of UAV classes, while the right part shows the sensor suites.
Based upon the task classification of being either local or distributed, the related primary
descriptor (a length or an area, respectively), and the task attributes duration and frequency, the
identified tasks can be binned.

Gheisari & Irizarry 90

 Figure 5. Technical Requirement Matrix

TECHNICAL AND OPERATOR REQUIREMENTS

For the technical analysis, a UAS is broken into three components: Vehicle, Control Station, and
System. The Vehicle section groups requirements for the UAV, which are mainly airframe
hardware related requirements (Table 1).
Table1. Technical Requirements; Vehicle
Vehicle Requirements Description
Airframe Ruggedness tries to capture the overall “sturdiness” of the UAV
Airframe Availability is one major factor in the overall availability of the UAV, which in essence
consists of the airframe, the avionics (the “autopilot”), and the payload (i.e.
sensors)
Endurance is an abstract high-level requirement that tries to capture the technical
aspects of available operational time and is mainly driven by the chosen
aircraft category (for example, a rotary or fixed wing vehicle) and the size
Actuated Video Camera requires a video sensor of the airframe to be moveable in any or all of the
three axes of motion, roll, pitch, and yaw
Actuated Non-Video package captures the need to steer other payload sensors, potentially
Sensor Package independent of the video source
Telepresence establishes communication with people in the vicinity of the aircraft usually
through a voice and/or video communication feature
Manipulator and/or usually ranges from simple grappling hooks to fully articulate and very
Effector dexterous robotic arms.

The Control Station groups requirements for hard- and software for the control station utilized by
the UAS operator, however, the software considered is related to the graphical user interface
(GUI) of the interface and does not include specific guidance, navigation, and control (GNC)
aspects. The details of these requirements are discussed in the following table (Table 2).
Guidance, navigation, and control (GNC) requirements are grouped into the System section,
which mainly contains capability features of the reference systems (Table 3)

Gheisari & Irizarry 91

 Table2. Technical Requirements; Control Station
Control Station Description
Requirements
Interactive Object relates to the GUI of the control station and requests a simple interface to
Selection & Identification directly interact with video coming from the UAV
Ruggedness relates to the capability of control station to be capable of being used
outdoors, potentially in harsh conditions
Portability covers the requirement to easily transport the control station

Table3. Technical Requirements; System Requirements

Control Station Description
Requirements
Sense and Avoid allows a UAS to detect cooperative as well as non-cooperative traffic in the
vicinity of the aircraft and conduct evasive maneuvers if a collision might be
imminent
Waypoint Navigation allows a UAS to follow a flight plan through the means to waypoints
Kinematically restricts the possible motion through physical means (e.g. a flexible cable or
Constrained Operations a rigid connector)
Unattended Deployment imposes the requirement for automatic take-off and landing without the need
and Return of an external pilot
High-precision poses the requirement to achieve a navigation solution that is comparable to
Navigation the accuracy achievable through the use of ground surveying methods (e.g.
total station equipment)
Simultaneous Location describes the capability of a system to navigate through a priori unknown
and Mapping (SLAM) environment while building a map of that environment
Advanced Data-link and captures the need for Radio Frequency (R/F) communication capabilities
Networking beyond a simple point-to-point link between the UAV and its control station
Sensor Data Abstraction captures the need for the system to automatically process raw sensor data
and Reduction and only provide the (abstract) result to the operator
Vision Based Data captures the request for any kind of computer vision based algorithm like
Extraction pattern recognition, optical flow, or feature point based processes

The result of the user-centered approach summarized in Figure 2 is a list of operator’s

requirements. The list is further refined by the subject matter experts who were interviewed. The
result of this step was a list of 22 operator requirements (see Table 4).

Table 4. Operator Requirements

# Requirement # Requirement
1 First-Person Video (FPV) Operation 12 Precise navigation
2 Non-FPV Operation 13 Provides survey-quality data
3 Pilot-independent sensor package control 14 Correlate sensor data to plans/drawings
4 Use at construction site 15 "Line-of-sight" clearance measurements
5 Use at traffic signal site 16 Collect floating traffic samples
6 Metro-area range/coverage 17 Allows triggering a "reset"
7 Get point coordinates 18 Interact with inspected element
8 Measure site dimensions 19 Establish contact to people
9 Estimate volumes 20 Eliminate need for human task execution
10 Create digital elevation model 21 Conduct traffic counts
11 Count/track/detect items 22 Data on demand

Gheisari & Irizarry 92

REFERENCE SYSTEMS
Based upon the validated data resulting from the interview process, the researchers used their
expertise to group tasks based upon a “best fit” to potential UAS and created a set of five
reference systems.

System A – Flying Camera

The air unit of System A, the Flying Camera, provides the most basic functionality of placing a
(video) camera anywhere in the accessible space and streaming live video to the operator (see
Figure 6). This system has been discussed in more details at (16)
Usage Scenario: The Flying Camera could be used in any situation where a video or picture is
all that is needed as a data input. The operator would simply start the system, use FPV to frame
the picture or video needed, record the images and finish the task at hand. The data post
processing could be as simple as storing the resulting still photo or video sequence or
interacting with the live video feed from the UAV.
Airframe: The airframe for System A would need to be VTOL capable, fairly robust, and safe to
be used around people, for example a small scale quad rotor with shrouded propellers. The
benefit of a quad rotor over a conventional helicopter would be the reduced mechanical
complexity and hopefully a resulting increase in robustness.
Payload: System A would most likely not carry any special payload beside a video camera.
Actuating the camera could improve the performance, so, in order to maintain a high level of
robustness, a virtual camera tilting could be implemented through several small scale cameras.
The setup could provide a low resolution feed and, upon operator request, high(er) definition
onboard video recording or still photography.
Control Station: The control station would be primarily focused on ruggedness and portability.
A first implementation could be a based on a tablet computer, utilizing on-screen virtual controls,
with the potential to expand the setup with a dedicated game pad style controller or a pair of
video goggles for improved FPV operation. If goggles are used, the GUI interface of the tablet
based control station software would need to be adapted to be compatible to the utilized
controller.
Required Infrastructure: No special infrastructure is necessary to operate System A. However,
there is the need for training, maintenance, and recharging, which needs to be organized. As
the system presumably would be small enough to be transported in a protective case in a car’s
trunk or pickup bed, no special transport equipment should be needed.
Capabilities: The systems capabilities would mainly rely on computer vision based algorithms
performed off board (i.e. with a transmission time delay) and only with the limited computational
power available in a tablet.

Gheisari & Irizarry 93

 )LJXUH+RXVHRI4XDOLW\IRU6\VWHP$ )O\LQJ&DPHUD

*KHLVDUL ,UL]DUU\
System B – Flying Total Station
System B expands upon System A by providing higher quality measurements of the
environment. The main focus of the system is to act as a flying total station, i.e. perform tasks
similar to the ones done by a survey crew, but faster, especially in otherwise unprepared
environments. As the systems’ expected operational radius is limited, the UAV could be
tethered to a power outlet at the control station site (see Figure 7).
Usage Scenario: The flying total station could be used anytime survey data or location data of
survey quality is needed. The system could be brought on scene, maneuvered to the area to be
surveyed (either through FPV video or conventional third person flying), and the survey could be
started. Post processing would presumably be similar to post-processing regular total station
data.
Airframe: System B would need an airframe that is capable of prolonged hover operations,
precise positioning, and a certain level of failure tolerance to protect against the loss of
potentially expensive sensors. These requirements would point towards a multi copter which is
designed for redundancy. This could be a hexa- or octocopter which is sized so that not all
rotors are needed to stay airborne. As people, both cooperative as well as non-cooperative,
would presumably negatively impact the survey process by blocking line of sight, it can be
assumed that the system would be operated in the presence of relatively few cooperative
people, as such reducing the requirements for safety through shrouds, etc.
Payload: The system would presumably carry LiDAR equipment to replicate a total station.
Additionally, altimeter, for example sonar or laser based, could be used to establish a correct
above ground altitude and in reverse determine the elevation of the terrain. Additional onboard
computational power might be required to process the LiDAR data.
Control Station: The control station for System B would most likely be comprised of a powerful
ruggedized laptop as well as a GNSS reference station to establish a differential correction for
the navigation solution.
Required Infrastructure: System B would not need any special communications landing site
infrastructure to be operated. However, due to the size of the UAV as well as the additional
antennas for the control station, the system most likely would be comprised of some larger
crates (to be transportable by pickup or SUV). An alternative would be a dedicated trailer or van.
(For more details on dedicated vehicles, see Section 5.2.6, the infrastructure requirements for
System E.)
Capabilities: The increased computation power both on- as well as offboard would allow the
system to not only utilize vision to detect and identify objects, but also to process video data in
combination with the LiDAR data to perform SLAM-based high precision navigation.
Furthermore, the system should be able to allow working with digital plans and drawings, for
example to check the correct location of construction features, roadway markings, or property
boundaries.

Gheisari & Irizarry 95

 )LJXUH+RXVHRI4XDOLW\IRU6\VWHP% )O\LQJ7RWDO6WDWLRQ

*KHLVDUL ,UL]DUU\ 
System C – Perching Camera
System C is also expands upon System A, but mainly focusing on a prolonged capturing of the
environment. Based upon that, operational endurance is a main application goal of System B.
This could either be achieved through highly efficient flight, which might be hard to achieve
given that a considerable amount of that flight time could be hover or hover-like operations, or it
could be achieved through perching (see Figure 8).
Usage Scenario: Due to the pertinent standby capability perching provides, System C could
mainly be used in two modes: as an ad-hoc deployed UAS for local, on-site inspection or
measurement tasks, or as a deployed-on-demand system. The former usage is comparable to
that of System A. The later usage would imply that System C UAVs have strategically located
fixed “base stations,” which would serve as recharging stations and potentially as
communication towers. A user requiring services provided by System C would use a control
station, request a UAV, and be given control over the closes available air unit. Once the
operator doesn’t need the services any longer, the UAV would be released and return to a base
station.
Airframe: As a result from the perching requirement, the airframe is required to be relatively
small to be able to get to potential perching locations as well as being robust to inadvertent
collisions close to the selected perching location. Furthermore, the airframe needs to be
equipped with a landing gear of sorts to facilitate perching on poles, or traffic signal installations.
These requirements could be realized with a smaller scale hexa-copter or potentially a small
electric conventional helicopter.
Required Infrastructure: The perching capability recommends System C for an extended dual
use: one the one hand as a mobile UAS with the described capabilities, on the other hand as a
static continuously operating camera. A potential scenario could be the deployment of several
dedicated perching locations which could double as a charging or refueling station. If the
System C units then provide a MANET capability, System C units could, for example, replace
the conventional Navigator cameras installed throughout the Atlanta metro area. Resulting from
that, a set of permanent perching locations are needed. These base stations would provide
recharging capabilities, allow easy access for maintenance, and could also double as R/F
communication outlets spanning the MANET utilized by System C.
Payload: Given that perching could limit the achievable attitudes while landed, actuated video
and non-video sensor packages are presumably necessary to compensate for that. Given the
use cases for System C, it seems likely that the system would also provide telepresence
equipment as well as advanced networking capabilities.
Control Station: The control stations for System C could be of several types. Given the
potential use cases, operators should be able to use control stations tailored to them and their
particular needs: HERO personnel, for example, could utilize a rugged tablet computer based
system comparable to a unit used to control System A; GDOT employees working in the Traffic
Control Center could use a software that operates on their desktop computers; traffic
engineering and traffic management personnel could use a laptop. The control station would
provide a FPV interface and a graphical tool for waypoint navigation as well as indicating
measurement areas or regions of interest.
Capabilities: System C would mainly provide vision based capabilities, potentially making use
of dedicated external computation centers to support limited computation power available in
tablet based control stations. The system would expand upon the capabilities of System A,
especially toward traffic related tasks.

Gheisari & Irizarry 97

 )LJXUH+RXVHRI4XDOLW\IRU6\VWHP& 3HUFKLQJ&DPHUD
*KHLVDUL ,UL]DUU\ 
System D – Medium Altitude, Long Endurance (MALE)
Whereas the proposed Systems A through C could be classified as having a local operational
area, System D is designed to expand this to a regional scale. The UAV would allow long
operational usage throughout a county-sized area. The system separates the piloting tasks from
the payload operation and data acquisition tasks, allowing for a very high level of operator
interaction (see Figure 9).
Usage Scenario: System D could operate in two ways, comparable to System C. In the ad-hoc
mode, the UAV would be stationed at an airfield and would get airborne as soon as an operator
requests control over a system. In the data-on-demand scenario, the air unit(s) of System C
would loiter over a dedicated operational area in a low energy standby mode. Once an operator
requests data, the system would relocate to the specific area requested and start to operate its
payload sensors.
Airframe: The airframe of choice for System D would most likely be a fixed wing aircraft design,
sized somewhere in the 2 m to 6 m wingspan regimen. The airframe would provide all
mandated general aviation equipment, for example, aviation band radios, a transponder, and
most likely an Automatic dependent surveillance-broadcast (ADS-B) transceiver, as the system
would have to operate within the (controlled) national airspace, among other general aviation
traffic.
Payload: The system would primarily provide a calibrated pan/tilt/zoom video sensor to provide
high quality aerial photography. The system could augment this with ground scanning LiDAR
systems and directional antennas for advanced networking features. Further sensor equipment
could include microwave based systems to detect traffic movements or near and far infrared
systems to aid in the localization of stranded motorists at night or the detection of wild fires.
Further optional payload capabilities could include R/F relay stations for first responder disaster
response communication systems.
Control Station: The control station for System D would most likely be consisting of several
independent units. One of them would be the payload focused unit available to the payload
focused operators. Related to that would be a highly portable data display unit, available to first
responders on site, which could be used to access the data provided from the system. Separate
from that would be a control station for an external pilot which would aid the system during taxi,
take-off, and landing operations as well as serve as a voice relay when conversing with ATC.
Required Infrastructure: System D would require an airstrip for take-off and landing. Given
that the system should need much smaller take-off and landing distanced, operation out of a
general aviation (GA) airport is not necessary and a dedicated airstrip might mitigate a lot of
integration into GA ground operations. If a dedicated airstrip is chosen, external pilots for take-
off and landing aiding could also have better access to the runway strip. As the system should
provide longer range operations, ground communication stations with dedicated directional
antennas might be necessary and could also be located at the utilized airstrip. Furthermore,
maintenance and refueling opportunities need to be provided.
Capabilities: System D would primarily provide aerial photographic and video data, which
would satisfy the quality requirements of photogrammetric applications. The payload operator
could also make live feeds of the video data available to first responders or HERO units. System
D could also be used as a disaster response communication relay station in case conventional
ground based infrastructure would not be available. System D could also provide Navigator like
traffic sensing capabilities, which could allow temporary traffic data capturing during larger
events outside the conventionally covered areas.

Gheisari & Irizarry 99

 )LJXUH+RXVHRI4XDOLW\IRU6\VWHP' 0$/(
*KHLVDUL ,UL]DUU\ 
System E – Complex Manipulation
An example for this category: This most likely would be a custom made multi-rotor with 8 or
more rotors or an even more special “inverted” helicopter, where the main rotor sits below most
of the airframe. The multi-rotor configuration or the low main rotor configuration would most
likely be necessary to allow an tele-robotics style manipulator to act above the rotor disc(s), as
this most likely would be safest for working under bridges. The system would be transported in a
dedicated van/truck and could potentially be tethered to allow for prolonged operations in hover
and/or while powering the manipulator. The system would most likely have an external (safety)
pilot as we well as a remote operator (in the van/truck) (see Figure 10).
System E allows its operators to completely perform a task through telepresence and tele-
robotics in areas that are either complicated or dangerous to access for humans. As such the
focus of the system shifts from a primarily sensing oriented operation to tasks which include
also portions in which previously inspected elements need to be manipulated.
Usage Scenario: System E is specifically slanted to be used for bridge or other structural
inspection activities. As such the system would be relocated to the current site under inspection
and the operated on site for the duration of the inspection. The main motivation behind System
E is to replace special access equipment needed during complex inspection tasks in order to
get the human inspector(s) to the inspection site. The system would presumable be used by at
least two operators: a dedicated external pilot, primarily responsible for piloting the UAV and
maneuvering the unit to the inspection site and a dedicated payload operator who would focus
on using the manipulator and effectors to conduct the actual inspection task. As the later
presumably would need a feedback device to control the manipulator, the payload operator is
assumed to work out of a dedicated control station vehicle, while the pilot operator is located at
a vantage point that provides good situational awareness of the situation the UAV operates in.
Both operators would have voice communication equipment to coordinate their efforts.
Airframe: The airframe of System E most likely would have to be a custom designed system.
As the system’s tasks include object manipulation, the airframe has to support a manipulator or
effector of sorts which raises the question of the general geometry of the system. The airframe
would have to be able to hover and provide VTOL capabilities, which would mean rotors, but
also provide a large operational range for the manipulator. A possible solution could be a large
scale multi-rotor where the manipulator is mounted above the main rotor disk, which allows
using the system to be used, for example, under bridges. Multirotor configurations are
preferable for such arrangements as it is easier ti build airframe structure through the non-rotor
occupied center. However, fixing this arrangement would limit the system to tasks in which the
manipulator wouldn’t have to be used below the main rotor discs of the multirotor setup.
Switching this to a conventional helicopter setup with a rotor head configuration that allows for
highly negative pitch angles could, in combination with advanced GNC algorithms, be used to
allow such a system to continuously operate in both orientations: manipulator above the main
rotor as well as below. The airframe furthermore would have to be strong enough to support not
only the manipulator, but also all the forces applied through it. Additionally several thrusters
could be needed to provide pushing or pulling forces without major changes in the vehicle
attitude.
Payload: The primary payload of System E would be the manipulator or effector and the
supporting telepresence and situational awareness sensors. These could include a stereo vision
rig, a time-of-flight camera or some similar close range 3D sensor, and LiDAR systems.
Additionally the system potentially would have to carry a selection of grappling interfaces and
probing tools.

Gheisari & Irizarry 101

 )LJXUH+RXVHRI4XDOLW\IRU6\VWHP( &RPSOH[0DQLSXODWLRQ
*KHLVDUL ,UL]DUU\ 
Control Station: The control station most likely would also be split into two parts, one more
tailored towards the needs of an external safety pilot, and more tailored towards the needs of
the main payload operator. The former would need to focus on providing the external pilot with a
good situational awareness of the surroundings of the vehicle, later would need to include a
device to control the installed manipulator and request necessary manipulating forces in
addition to the forces needed to maintain flight.
Required Infrastructure: System E would need a dedicated vehicle. This vehicle would on the
one side serve as the transport vehicle to get the system to the different inspection sites and on
the other side double as the control station once the system is unloaded. The vehicle would
provide a source of power, an indoor workstation for the payload operator, and the required
computer systems. Additionally it would also carry all the other required elements for the
inspection tasks, i.e. any potentially necessary specialized sensors, etc. The vehicle furthermore
would carry traffic control devices to setup and secure the operations.
Capabilities: The system would mainly provide capabilities comparable to a trained human
worker operating out of a bucket truck or a similar reach extending device. To maintain
prolonged operations, the system could be tethered to an external power source and would
need to provide the ability to operate under kinematic constraints, not only from the tether, but
also and especially when latchet onto other objects.

CONCLUSION
After conducting interviews with 24 individual in the four selected GDOT divisions, the research
team identified tasks that could benefit from the use of UAS technology. The majority of the
tasks in GDOT divisions with the highest potential for benefitting from UAS technology are
centered around collecting data, providing information, and decision making based on the data.
Each task is also characterized by particular attributes (e.g. location where the tasks are
performed and the time required to complete a given task) that yield a better understanding of
the environmental conditions. Thus, UAS technical requirements that embed the operational
and technical requirements for development of a potential UAS have been investigated. The
result of this investigation was the identification of five potential systems. Given the issues with
cost related data collection in this study, it is recognized that additional research is needed to
obtain a clearer idea of the economic and intangible benefits of the use UASs for GDOT
operations. A possible departure point would be the selection of construction related tasks. It
would be possible to perform a detailed tasks analysis for a construction jobsite inspection task
to set the base for UAS operator system interface needs. The analysis would include a detailed
assessment of the current practice and shadowing of personnel performing the task. In that way
an estimate of the time and cost of performance could be developed. Based on this analysis, a
potential UAS flight path through a jobsite could be established. Using a staff mounted sensor
suite as a UAS mock-up or an off-the –shelf UAS, sensor data including video would be
collected along the established flight paths. Then, a software replica of the site would be
developed, using the collected data. The system developed would be used in a staged field test
in an access-controlled construction site to validate the simulation results. This activity (and
preparations for it) would include direct coordination with the FAA. The technical requirements
determined would also aid in more rapid development of test UASs for GDOT use as well as
advance GDOT’s implementation of UAS(s) to help accomplish the Department’s goals.

ACKNOWLEDGMENTS
The authors gratefully acknowledge the support provided for this research by GDOT under grant
12-38. The views and opinions presented in this paper are those of the authors do not
necessarily reflect the views, opinions or policy of GDOT.

Gheisari & Irizarry 103

REFERENCES
[1] Coifman, B., M. McCord, R. G. Mishalani, and K. Redmill. Surface transportation surveillance
from unmanned aerial vehicles. Presented at Proc. of the 83rd Annual Meeting of the
Transportation Research Board, Washington, D.C., 2004.
[2] Srinivasan, S., H. Latchman, J. Shea, T. Wong, and J. McNair. Airborne traffic surveillance
systems: video surveillance of highway traffic. Presented at ACM 2nd international workshop
on Video surveillance & sensor networks, New York, NY, 2004.
[3] Puri, A., K. Valavanis, and M. Kontitsis. Generating traffic statistical profiles using unmanned
helicopter-based video data. Presented at International Conference on Robotics and
Automation. , Roma, Italy, 2007.
[4] Coifman, B., M. McCord, R. Mishalani, M. Iswalt, and Y. Ji. Roadway traffic monitoring from
an unmanned aerial vehicle. Presented at Intelligent Transport Systems, 2006.
[5] Rathinam, S., Z. W. Kim, and R. Sengupta. Vision-Based Monitoring of Locally Linear
Structures Using an Unmanned Aerial Vehicle 1. Journal of Infrastructure Systems, Vol. 14,
No. 1, 2008, pp. 52-63.
[6] Frew, E., T. McGee, Z. Kim, X. Xiao, S. Jackson, M. Morimoto, S. Rathinam, J. Padial, and
R. Sengupta. Vision-based road-following using a small autonomous aircraft. Presented at
Aerospace Conference, 2004.
[7] McCormack, E. D., and T. Trepanier. The use of small unmanned aircraft by the Washington
State Department of Transportation.In, Washington State Department of Transportation,
2008.
[8] Zhang, C., and A. Elaksher. An Unmanned Aerial Vehicle%DVHG ,PDJLQJ 6\VWHP IRU '
Measurement of Unpaved Road Surface Distresses1. Computer$LGHG&LYLODQG
Infrastructure Engineering, Vol. 27, No. 2, 2012, pp. 118-129.
[9] Metni, N., and T. Hamel. A UAV for bridge inspection: Visual servoing control law with
orientation limits. Automation in Construction, Vol. 17, No. 1, 2007, pp. 3-10.
[10] Irizarry, J., M. Gheisari, and B. N. Walker. Usability assessment of drone technology as
safety inspection tools. Journal of Information Technology in Construction (itcon), Vol. 17,
2012, pp. 194-212.
[11] Carroll, E. A., and D. B. Rathbone. Using an Unmanned Airborne Data Acquisition System
(ADAS) for Traffic Surveillance, Monitoring, and Management. Presented at ASME 2002
International Mechanical Engineering Congress and Exposition, New Orleans, LA, 2002.
[12] Werner, J. FDOT explores the viability of using unmanned aerial vehicles (UAVs) for traffic
surveillance. Newsletter of the ITS Cooperative Deployment Network [online], 2003.
[13] Barfuss, S. L., A. Jensen, and S. Clemens. Evaluation and Development of Unmanned
Aircraft (UAV) for UDOT Needs.In, UDOT, Utah Department of Transportation, 2012.
[14] Irizarry, J., and E. N. Johnson. Feasibility Study to Determine the Economic and
Operational Benefits of Utilizing Unmanned Aerial Vehicles (UAVs).In, GDOT, Georgia
Department of Transportation, 2014.
[15] Karan, E. P., C. Christmann, M. Gheisari, J. Irizarry, and E. N. Johnson. A Comprehensive
Matrix of Unmanned Aerial Systems Requirements for Potential Applications within a
Department of Transportation. Presented at Construction Research Congress 2014:
Construction in a Global Network, Atlanta, GA, 2014.
[16] Gheisari, M., Karan, E. P., Christmann, H. C., Irizarry, J., & Johnson, E. N. (2015).
Investigating Unmanned Aerial System (UAS) Application Requirements within a
Department of Transportation. In Transportation Research Board 94th Annual Meeting (No.
15-1430).

Gheisari & Irizarry 104

Simulation of Biologically-Inspired Control Algorithms
for Teams of Ground Vehicles

Christopher Goodin, Zachary Prevost

Geotechnical and Structures Laboratory
U.S. Army Engineer Research and Development Center
3909 Halls Ferry Road
Vicksburg, MS 39180
[email protected], [email protected]

Bertrand Lemasson
Environmental Laboratory
U.S. Army Engineer Research and Development Center
3909 Halls Ferry Road
Vicksburg, MS 39180

ABSTRACT
While the software controlling autonomous systems has grown more complex, controlling
systems of robots with multiple autonomous agents remains a challenging research area.
Assigning tasks or roles to each individual agent in the group is time consuming and may
restrict collaborative behaviors. One possible solution to this problem is to enable individual
autonomous agents with relatively simple interactive logic that allows more complex group
behaviors to emerge. We have tested the validity of this method by applying simple social
models derived from collective animal behavior to autonomous unmanned ground vehicles. We
have evaluated the results of this approach using high-fidelity simulations in the Virtual
Autonomous Navigation Environment (VANE). Our initial findings demonstrate the potential of
biologically inspired control algorithms for enabling navigation in groups of autonomous ground
vehicles.

Key words: autonomy—social navigation—HMMWV

INTRODUCTION
The potential for autonomous navigation by unmanned ground vehicles (UGV) holds great
promise for military applications. However, unlike autonomous navigation of civilian vehicles,
which can be operated on previously mapped road networks, military vehicles may require off-
road navigation with little previously known information about the state of the terrain. In addition,
without the benefit of well-marked lanes, road signs, and rules of the road, autonomous
navigation in off-road environments must work in a dynamic, unconstrained environment.

Many of the complexities of autonomous operation in an off-road environment can be avoided

by having the autonomous vehicle simply follow a lead-vehicle driven by human. If the
autonomous vehicle can maintain close contact with the lead vehicle, then high-level
autonomous decision-making, and the problems associated with it, can be avoided. However, in
situations where the ratio of unmanned (follower) to manned (leader) vehicles becomes large,
simple follower behaviors will not produce optimal results for the group.

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
In this work we will demonstrate the feasibility of using biologically inspired control algorithms for
UGV operating in teams with manned ground vehicles. These algorithms work by having each
robot compare its own state to the state of the vehicles (both manned and unmanned) around it.
We use high-fidelity simulations from the Virtual Autonomous Navigation Environment (VANE)
[1] to show that these biologically-inspired algorithms give rise to group behaviors in teams of
manned and unmanned UGV that depend on the ratio of manned to unmanned vehicles.

BACKGROUND
Biologically inspired control
We implemented biologically inspired algorithms for controlling the robotic vehicles. In this work,
the inverse distance weights how individuals perceive one another; closer vehicles are given
more importance than those farther away. While there are additional mechanisms used by
animals to process information, such as selective attention to salient features [2,3], our goal was
to begin by integrating cognitive effects at the simplest level (sensory perception) to explore
their impact on the collective motion of realistically simulated ground vehicles.

In the algorithm used in this work, the robots desired velocity was determined by comparing its
own velocity with all the other vehicles, and weighting the averaged results by the inverse of the
distance calculates velocity. So for vehicle i with velocity the desired velocity is given by
& 1 n
& &
vi '
w
¦ w (v
j 1,i z j
ij j  vi ) (1)

where n is the total number of vehicles and the weights are given by
& &
wij ri  rj (2)
and the total weight factor w is given by
w ¦
wij .
j
(3)

The Virtual Autonomous Navigation Environment

The VANE is a high fidelity, physics-based robotics simulator that features realistic models for
vehicle dynamics, vehicle-terrain interaction, environment, and sensors. Vehicle dynamics is
simulated using the Chrono::Vehicle multi-body dynamics library [4]. Chrono::Vehicle features
detailed models of the suspension element, chassis inertial properties and running gear. In our
simulations, Chrono::Vehicle was coupled to the Ground Contact Element (GCE) tire-terrain
interaction model [5]. The GCE features a nodal-based radial spring tire model coupled to
empirically derived soil models, resulting in realistic torque calculations for the tire in a wide
variety of terrain conditions. Terrains are represented as highly attributed triangular meshes.

Sensor models in the VANE use detailed ray tracing to simulate the reflection of visible and
infrared wavelengths in the environment. This includes environmental light for camera sensors
and active sources such as the lasers in LADAR systems. The VANE also features a detailed
GPS model that accounts for multipath reflections, satellite occlusion, dilution-of-precision, and
atmospheric effects [6].

Goodin, Prevost, and Lemasson 106

EXPERIMENT DETAILS
The VANE simulation used a HMMWV as the test vehicle. The HMMWV was modeled in
Chrono::Vehicle with the double-wishbone suspension in the standard HMMWV configuration.
In order to focus on the interactional aspects of navigation, the terrain was flat and there were
no obstacles. A simulated visualization of the formation of HMMWV is shown in Figure 1.

Figure 1. Visualization of the group of 7 HMMWV

We performed experiments with groups of 4, 7, and 10 vehicles. For each group of vehicles, we
varied the number of vehicles that were autonomous. In the experiments, the autonomous
vehicles were not equipped with any sensing capability, but all the vehicles communicated their
state (position and velocity) to a common server to which all other vehicles subscribed. This was
the only information that the vehicles shared.

The initial configuration of the vehicles was a staggered column, as depicted in Figure 1, with an
average longitudinal spacing of 10 meters and an average lateral spacing of five meters. The
initial position of each vehicle was randomly varied by up to 2.5 meters in any direction, and the
initial orientation was randomly varied by 25 degrees. These ranges were chosen to give the
maximum possible variation to the simulations while ensuring the vehicles did not collide with
each other.

In the experiment, the driven vehicles abruptly changed course after five seconds to simulate an
avoidance maneuver. In the maneuver, the manned vehicles quickly turned to the left while
maintaining formation. Ten trials were run for each combination of manned and unmanned
vehicles shown in Table 1, and the average alignment at 15 seconds was calculated for each
configuration.

To quantify leader-follower coordination between manned and unmanned vehicles we

measured the alignment of the unmanned vehicles with the manned vehicles after 15 seconds,
10 seconds after the turn maneuver was initiated. For manned vehicles, denoted subscript m,
and unmanned vehicles, denoted subscript u, the alignment, Į, of the system is given by
& &
1 vm x vu
D
2nm nu
¦¦
m u
& &
vm vu
(4)

Goodin, Prevost, and Lemasson 107

where nm and nu are the numbers of manned, and unmanned vehicles respectively, and the
factor of 2 is to account for double counting of pairs.

The alignment defined by Equation 4 would have a theoretical minimum of -1 (unmanned

vehicles perfectly anti-aligned with manned vehicles) and a maximum of 1 (all vehicles perfectly
aligned). The alignment of a randomly oriented system of vehicles will be close to 0. Because
the vehicles were nearly aligned in the initial configuration, the system started with Į§1, and the
alignment typically improved steadily until the abrupt movement event, at which point the
alignment decreased. Figure 2 shows the alignment of a group of ten robots, with one driven as
a function of time.

Table 1. Experiment Matrix

Number of Vehicles Number Driven Ratio Alignment Std Error
4 1 25.0% 0.880 0.001
4 2 50.0% 0.933 0.005
4 3 75.0% 0.948 0.002
7 1 14.3% 0.864 0.004
7 2 28.6% 0.901 0.004
7 3 42.9% 0.931 0.003
7 4 57.1% 0.952 0.002
10 1 10.0% 0.852 0.003
10 2 20.0% 0.878 0.004
10 4 40.0% 0.927 0.003
10 5 50.0% 0.956 0.003

RESULTS AND DISCUSSION

The results of the experiment are shown in Table 1. In general, groups with a higher fraction of
manned vehicles had better alignment after the maneuver than those groups with fewer manned
vehicles, as would be expected. Figure 2 compares the average over all ten trials for the
alignments of ten robots with either five or one driven configuration. Note that the maneuver
event occurred at five seconds, and the final alignment was measured at 15 seconds. This plot
demonstrates how the alignment of the group tended to diverge over time in groups with smaller
numbers of manned vehicles.

Goodin, Prevost, and Lemasson 108

 Figure 2. Alignment of a group of 10 robots, with one manned. Data points are sampled
every 0.3 s. and represent means r 1 standard error.

In Figure 3, we show the paths of vehicles from one of the simulations with seven vehicles. In
the left figure, only one vehicle was manned, while in the right figure, four vehicles were
manned. The manned vehicles are denoted with a thick red line, while the unmanned vehicles
are denoted with a thin gray line. The results depicted in Figure 3 show the typical result that
after the maneuver initiated at five seconds, the unmanned systems tended to move as a group,
and that group was influenced by the number and proximity of manned vehicles. Systems with
greater than 50% manned vehicles tended to recover alignment after the maneuver, while
systems with less than 50% recovered with proportionally varying degrees of success.

Figure 3. Comparison of the paths taken by a group of 7 vehicles with six unmanned
systems (left) and 3 unmanned systems (right)

Goodin, Prevost, and Lemasson 109

Figure 4 shows another summary of the data. In this figure, the average alignment of the system
after 15 seconds is plotted against the percentage of manned vehicles. The alignment of the
system increases linearly with increasing percentage of manned vehicles up to 50%, above
50% the increase in alignment levels off. The black line is a fit to the points below 50% manned
with the form
D 0.244 Pm  0.828 (5)
where Pm is the percentage of manned vehicles. The fit has a coefficient of determination of
R2=0.982.

Figure 4. Alignment of the vehicles as a percentage of manned systems. Data points

represent means r 1 standard error.

CONCLUSIONS AND FUTURE WORK

In this work we showed how simple control algorithms inspired by biological systems can be
used to control mixed groups of manned and unmanned vehicles. We demonstrated a method
for determining the influence of the percentage of unmanned systems that used high-fidelity
simulations of the dynamics of wheeled vehicles and were able to determine a clear trend and
threshold for optimizing performance.

The current dynamics agree with theoretical and empirical work on the influence of majority-
minority interactions in the collective movement decisions of groups. When individuals are
unaware of who is in charge, and pool all social cues indiscriminately, followers will either adopt
a new course that is a compromise (Fig. 3, left), or else follow the majority (Fig. 3, right) [7.8]. In
future work we will explore how selective attention to particular features can shift these
dynamics across groups of increasing size [3]. In addition, we will parameterize those
algorithms to allow optimization of the control algorithms for a particular task such as navigation

Goodin, Prevost, and Lemasson 110

or obstacle avoidance. Finally, we will expand the analysis to include heterogeneous groups of
vehicles consisting of trucks of various sizes.

REFERENCES
[1] Goodin, C., George, T. R., Cummins, C. L., Durst, P. J., Gates, B. Q., & McKinley, G. B. (2012). The
Virtual Autonomous Navigation Environment: High Fidelity Simulations of Sensor, Environment, and
Terramechanics for Robotics. In Earth and Space 2012 (pp. 1441-1447).
[2] Ballerini, M., Cabibbo, N., Candelier, R., Cavagna, A., Cisbani, E., Giardina, I., Lecomte, V.,
Orlandi, A., Parisi, G., Procaccini, A., Viale, M., & Zdravkovic, V. (2008) Interaction ruling animal
collective behavior depends on topological rather than metric distance: evidence from a field
study. Proceedings of the National Academy of Sciences, U.S.A.,105(4), 1232-1237.
[3] Lemasson, B. H., Anderson, J. J., & Goodwin, R. A. (2013). Motion-guided attention promotes
adaptive communications during social navigation. Proceedings of the Royal Society B: Biological
Sciences, 280(1754), 20122003.
[4] Tasora, A., & Anitescu, M. (2009). A fast NCP solver for large rigid-body problems with contacts,
friction, and joints. In Multibody Dynamics (pp. 45-55). Springer Netherlands.
[5] Creighton, D. C., McKinley, G. B., Jones, R. A., & Ahlvin, R. B. (2009). Terrain Mechanics and
Modeling Research Program: Enhanced Vehicle Dynamics Module (No. ERDC/GSL-TR-09-8). Engineer
Research and Development Center, Geotechnical and Structures Laboratory.
[6] Goodin, C., Durst, P. J., Gates, B., Cummins, C., & Priddy, J. (2010). High fidelity sensor simulations
for the virtual autonomous navigation environment. In Simulation, Modeling, and Programming for
Autonomous Robots (pp. 75-86). Springer Berlin Heidelberg.
[7] Couzin, I. D., Krause, J., Franks, N. R., & Levin, S. A. (2005). Effective leadership and decision-
making in animal groups on the move. Nature,433(7025), 513-516.
[8] Biro, D., Sumpter, D. J., Meade, J., & Guilford, T. (2006). From compromise to leadership in pigeon
homing. Current Biology, 16(21), 2123-2128.

Goodin, Prevost, and Lemasson 111

Case Study on 3D Modeling and AMG Practices

Fangyu Guo
Department of Civil, Construction, and Environmental Engineering
Iowa State University
136 Town Engineering Building
Ames, IA, 50011
Email: [email protected]

Yelda Turkan
Department of Civil, Construction, and Environmental Engineering
Iowa State University
428 Town Engineering Building
Ames, IA, 50011
Email: [email protected]

Charles T. Jahren
Department of Civil, Construction, and Environmental Engineering
Iowa State University
456 Town Engineering Building
Ames, IA, 50011
Email: [email protected]

ABSTRACT
The adoption of 3D modeling and automatic machine guidance (AMG) are becoming more
popular in the transportation industry. With a 3D model uploaded to an on-board computer
within a piece of heavy construction equipment, operators can easily monitor machine
operations with respect to grad and location or engage the machine to produce the proper grade
automatically. Thus, it provides great convenience and improved productivity for field workers.

AMG and 3D modeling have been identified as enabling technologies for a Civil Integrated
Management (CIM) system. When CIM is implemented, an entire transportation agency and
stakeholder partners share in the use and development of a common data pool that is
accessible to authorized users involved with all phases of a transportation facility life cycle (such
as planning, design, construction, maintenance and rehabilitation) and all departments in the
agency (Administration, finance, operation and others). The concept of CIM was developed and
promoted by the United States Federal Highway Administration in 2013 and was established to
make better use of accurate data and information that results from the utilization of advanced
technologies and/or tools thus to facilitate more effective decision making for transportation
projects.

Using the CIM concept and framework, technologies such as 3D modeling and AMG could be
more efficiently adopted within the full life cycle of a transportation facility. More importantly,
data could be collected and managed systematically in the early phases of a project life cycle so
they could be useful for later phases of the facility lifecycle. The purpose of this study is to
investigate how CIM system could support autonomous construction and vice versa. During a
domestic scan effort, seven state agencies and their contractors collaborated to present their
extensive experiences on certain CIM related practices and tools. In particular, the experiences

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
of the agencies that were under investigation regarding 3D modeling and AMG will be
addressed in this paper. In addition, the benefits and challenges of using 3D modeling and AMG
will also be discussed.

Keywords: civil integrated management (CIM)—3D modeling—automatic machine

guidance (AMG)

PROBLEM STATEMENT
Productivity, quality, and safety are important aspects of any transportation project and are used
as measures of project success. Various technologies and tools have been used to achieve
better productivity, quality, and safety. Adoption of 3D engineered models in the transportation
industry is relatively behind the adoption of Building Information Modeling (BIM) in the building
industry. However, this is now changing; the transportation industry is adopting 3D engineered
models at an increasing rate. In 2010, the adoption rate of 3D engineered models for
transportation projects was 27%, which was increased to 46% in 2012 (McGraw-Hill 2012). The
successful implementation of BIM in the building industry is a good example. It is critical for
transportation industry to investigate the lessons learned from the building industry and start
implementing 3D engineered models and other supporting technologies such as Automatic
Machine Guidance (AMG) in their projects. The implementation of 3D engineered models is
expected to bring benefits such as less rework and change orders (Parve 2013), more effective
communications (Olde and Hartmann 2012), reduced time and costs, and improved quality and
productivity (Myllymaa and Sireeni 2013; Cylwik and Dwyer 2012). Similarly, the major benefits
of using AMG include increased quality and productivity, and reduced time and costs on job
sites (Peyret 2000).

For an agency planning to implement a new technology, it is important to investigate the best
practices and lessons learned from other agencies that already have extensive experiences with
that particular technology. Since Iowa DOT was identified as one of the leading state DOTs for
3D modeling implementation (EDC 2013), this paper presents a case study on Iowa DOT’s best
practices and lessons learned from the use of 3D modeling and AMG.

RESEARCH OBJECTIVES
The objectives of this study are presented as follows:
x Learn Iowa DOT’s progress on the use of 3D engineered models and AMG;
x Study how the agency transitioned from the traditional processes to the new processes
of implementing 3D engineered models and AMG;
x Identify the benefits and challenges of using 3D engineered models and AMG within the
agency;
x Conclude lessons learned from the adoption of 3D engineered models and AMG for the
agency.

RESEARCH METHODOLOGY
The results presented in this paper are obtained from the National Cooperative Highway
Research Program (NCHRP) Domestic Scan project 13-02 - Advances in CIM. The scan project
investigated the current practices related to Civil Integrated Management (CIM) adopted by
various state DOTs. During the project, 3D engineered models and AMG were identified as two
of the enabling technologies for CIM implementation. The scan panel members were composed
of various state DOTs and the Federal Highway Administration (FHWA) personnel, and a

Guo, Turkan, and Jahren 113

subject matter expert (SME). The third author of this paper served as the SME, and the other
two authors assisted the SME during this project. The scan team conducted an initial survey,
called desk scan, to obtain knowledge on CIM and CIM-related technologies and tools. The
amplifying questions were prepared by the scan team and sent to target agencies. The scan
team then spent two weeks in total visiting seven leading state DOTs who had extensive
experiences in CIM-related practices. All day workshops and presentations were provided by
the host agencies and their contractors. It is also important to note here that there were also
intensive interactions during these meetings between the host agencies and the scan team
members. Detailed notes were taken during the meetings for further analysis. The data related
to the implementation of 3D engineered models and AMG by Iowa DOT were extracted and
analyzed in this paper. The results obtained from the coding strategies (Maxwell 2013) are
presented in the following section.

KEY FINDINGS
Progress on 3D Engineered Models and 3D Renderings
Transitioning from 2D to 3D design is one of the main goals of Iowa DOT. Iowa DOT uses 3D
renderings extensively to communicate with public, which allows the public to have a better
understanding about the project and its impact on the surrounding environment. Great amount
of details can be added to 3D renderings, although it might be challenging to add paint lines on
roads. Furthermore, these renderings can be used as a starting point for 3D design. Currently,
they are modeling all their highway projects in 3D using Bentley Corridor Modeler software
package. Newly hired personnel are given proper training on 3D modeling knowledge and skills.
When Iowa DOT first started 3D modeling, 2D cross sections would be developed and used to
build a 3D model. This has been improved over the years. Today, 3D models are developed first
and cross sections are extracted from the 3D models. Model development process has been
standardized within the agency to ensure designers use the same approach to build the model,
so that the same type of information can be obtained from a 3D model. Iowa DOT also adjusted
the proportion of in-house modeling development effort over time. They used to develop all 3D
models in-house, which consumed great amount of their time since they wanted to ensure the
resulting products fit contractors’ needs. Currently, 70% of the 3D models are developed in-
house, and the rest 30% are developed by their consultants. The standardized tools and
templates, which were developed by Iowa DOT, are shared with their consultants. With this
adjustment, time and effort for in-house modeling development is reduced while maintaining the
same model quality. The completed models are typically provided to the contractors before the
bidding process, since design-bid-build is the dominant project delivery method for their
projects. Overall, it can be concluded that 3D models and 3D renderings help improve
communications between Iowa DOT and their contractors and with the public respectively.

Some of the challenges experienced by Iowa DOT in implementing 3D modeling are as follows.
First, copyright issues can be quite challenging. It is hard to determine who can change or
modify which information. Iowa DOT has a copyright policy and the state of Iowa has a
copyright law in place. However, this is not yet included in the consultant contracts. Second, the
entire model development process has to be changed. Attribute data should properly be
assigned to each 3D object in 3D models. In the future, Iowa DOT plans to develop 4D (3D+
time (schedule)) animations for their projects. Having 4D models is expected to give public a
better idea about how the project will look like once it is completed.
Progress on AMG
In the past, contractors who wanted to implement AMG would take 2D project plans to their
consultants and have them build 3D models based on the 2D plans, so that they could use the

Guo, Turkan, and Jahren 114

3D models on their equipment for AMG. This was changed after Iowa DOT started to develop
3D models and provide them for AMG use. Before Iowa DOT start implementing 3D modeling
for machine control, they met with Trimble, Leica, CAT, and other manufacturers to learn about
the current status and discuss the future direction for construction equipment. Based on the
discussions during those meetings, they decided to do a pilot project where they made it
mandatory to use 3D models and AMG. Based on the benefits observed, they continued using
these technologies for other projects. This pushed their grading contractors transition from
traditional equipment to AMG in order to stay in business. Construction equipment is
continuously improved, and it is becoming easier for contractors to use. Operators can easily
adjust the location and movement of certain parts of equipment (such as blade or bucket) based
on the guidance provided by GPS. Construction equipment with a GPS device installed can be
either purchased or rent from dealers. Most contractors in Iowa prefer purchasing equipment to
renting, as most jobs require the use of AMG to some extent.

Based on Iowa DOT’s experiences, stringless paving technology helped increase the quality of
paving greatly. Stakes only need to be placed every 1000 feet when using stringless paving,
which greatly improves the safety in the field. Competition among contractors became more
intense with little changes in bid prices. Although setting up the machine control in the field
increased the upfront project costs, the overall project cost did not change much compared to
traditional process of not using AMG.

From contractor’s perspective, the following benefits were observed when AMG was used:
x Productivity and accuracy are greatly improved with the adoption of AMG
x Owners tend to be more satisfied with the product delivered with AMG.
x Design changes could be integrated easier and faster.
x Problems could be detected early in the process via 3D models that are intended to be
used for AMG.
x Field safety is greatly improved since less people are needed on the job site for AMG-
implemented projects.
Lessons Learned from the Model Centric Design
Iowa DOT learned the following lessons from implementing 3D modeling and AMG for highway
projects.
x Before implementing a new technology or a new approach, it is important to set the
goals first, and then determine how to achieve them.
x It is beneficial to consult with industry practitioners to better understand their needs.
Iowa DOT started to use standard data exchange format (LandXML), naming
convention, color paper, and others based on the feedback they received from their
contractors and consultants.
x It is beneficial to set up some rules to standardize the process and keep those rules
consistent. For example, color plans were adopted about five or six years ago within the
agency. The same color coding system was used for all projects to maintain the
consistency, so that there were no confusions about the color codes.
x It is critical to organize the data or documents (naming convention, file format, etc.) well.
Better organized documents help people perform better when searching and sorting
information.
x It would be beneficial to maintain continuous communication with industry associations.
For example, discussions with the associated general contractors of America (AGC)
helped Iowa DOT determine a future direction that matches the direction of AGC. Also,

Guo, Turkan, and Jahren 115

 annual meetings with AGC keep Iowa DOT up to date with the development and
utilization of the most recent technologies and tools in the industry.
x How the agency approaches training is a key to success. In-time and periodical training
is proven to be the most efficient.

CONCLUSIONS
This paper focused on the implementation of 3D engineered models and AMG by Iowa DOT,
which was identified as one of the leading state DOTs in this area. A brief introduction was
provided on how the agency transitioned from traditional design and construction processes to
new processes where 3D models and AMG are used. Benefits, challenges, and lessons learned
from using these technologies were also discussed. The findings of this study would be
beneficial to the agencies that are planning to implement these technologies within their
organization. Before implementing a new technology, it is important for agencies to consult
other agencies and/or companies with extensive experiences using a particular technology and
discuss it with software or hardware vendors to set up a clear goal and select the best way to
start. Lessons learned from pilot projects can be great guidance for future projects. Benefits and
challenges should also be clearly identified. This may help motivate employees to use the
technologies and tools. Finally, as agencies become more experienced with new technologies
and tools, proper standards should be put in place to improve the overall work efficiency.

ACKNOWLEDGMENTS
The material presented in this paper is based on the presentations and documents obtained
during the Scan 13-02 project - Advances in Civil Engineering Management (CIM), which was
supported by the National Cooperative Highway Research Program (NCHRP) 20-68A US
Domestic Scan Program. Scan team members included John Adam of Iowa DOT, Katherine
Petros and Brian Cawley of the US Federal Highway Administration, Rebecca Burns of
Pennsylvania, Duane Brautigam of Florida DOT, Julie Kliewer of Arizona DOT, John Lobbestael
of Michigan DOT, Stan Burns and Randall R. Park of Utah DOT, David Jeong of Iowa State
University. The scan team was supported by Harry Capers, Jake Almborg, and Melissa Jiang of
Aurora and Associates, P.C.. Their assistance in analyzing the data collected during the site
visits is gratefully acknowledged. We would also like to convey our sincere thanks to Iowa DOT
for their continuous support during this project.

REFERENCES
Cylwik, E. and Dwyer, K. (2012). “Virtual Design and Construction in Horizontal Infrastructure
Projects.” Engineering News-Record,
<http://enr.construction.com/engineering/pdf/News/Virtual%20Design%20and%20Constructi
on%20in%20Horizontal%20Construction-05-03-12.pdf> (May 02, 2015)
Every Day Counts (EDC). (2013). “”3D Engineered Models for Construction Implementation
Plan.” Federal Highway Administration. 1-20.
Maxwell, J.A. (2013). Qualitative Research Design (3rd ed.). Thousands Oaks, CA: SAGE
Publications, Inc.
McGraw Hill Construction. (2012). “The Business Value of BIM.” SmartMarket Report, New
York.
Myllymaa, J., and Sireeni, J. (2013). “Cost Savings by Using Advanced VDC Tools and
Processes, Case Studies from Europe,” Presentation at the 2013 Florida Department of
Transportation Design Training Expo, June 12-14, 2013.
http://www.dot.state.fl.us/structures/designexpo2013/2013ExpoPresentations.shtm (May
02, 2015)

Guo, Turkan, and Jahren 116

Olde Scholtenhuis, L. L., & Hartmann, T. (2012). “An object model to support visualizations and
simulations of subsurface utility construction activities.” In Proceedings of 14th International
Conference on Computing in Civil and Building Engineering.
Parve L. (2013), “Redesigning a complex, critical freeway: Building information modeling helps
deliver cost avoidance and savings for Wisconsin's Mitchell Interchange Project.”
<http://www.rebuildingamericasinfrastructure.com/print-magazinearticle-
redesigning_a_complex__critica-9018.html> (May 02, 2015)
Peyret, F. et al. (2000). “The Computer Integrated Road Construction Project.” Automation in
Construction, 9, 447-461.

Guo, Turkan, and Jahren 117

A Formalism for Utilization of Autonomous Vision-
Based Systems and Integrated Project Models for
Construction Progress Monitoring

Kevin K. Han
Department of Civil and Environmental Engineering
University of Illinois at Urbana-Champaign
205 N. Matthews Ave
Urbana, IL 61874
[email protected]

Jacob J. Lin
Department of Civil and Environmental Engineering
University of Illinois at Urbana-Champaign
205 N. Matthews Ave
Urbana, IL 61874
[email protected]

Mani Golparvar-Fard
Department of Civil and Environmental Engineering & Computer Science
University of Illinois at Urbana-Champaign
205 N. Matthews Ave
Urbana, IL 61874
[email protected]

ABSTRACT
Actual and potential progress deviations during construction are costly and preventable.
However, today’s monitoring programs cannot easily and quickly detect and manage
performance deviations. This is because (1) the current methods are based on manual
observations made at specific locations and times; and (2) the captured progress information is
not integrated with “as-planned” 4D Building Information Models (BIM). To facilitate this process,
construction companies have focused on collecting as-built visual data through hand-held
cameras and video recorders. They have also assigned field engineers to filter, annotate,
organize, and present the collected data in comparison to 4D BIM. However, cost and
complexity associated with collecting, analyzing and reporting operations still result in sparse
and infrequent monitoring and therefore a portion of the gains in efficiency is consumed by
monitoring costs. To address current limitations, this paper outlines a formal process for
automating construction progress monitoring using visual data captured via camera-equipped
Unmanned Aerial Vehicles (UAVs) and 4D BIM. More specifically, for data collection, formal
methods are proposed to identify monitoring goals for the UAVs using 4D BIM and
autonomously acquiring and updating the necessary visual data. For analytics, several methods
are proposed to generate 3D and 4D as-built point cloud models using the collected visual data,
to integrate them with BIM, and to automatically conduct appearance-based and geometrical
reasoning about progress deviations. For reporting, a method is proposed to characterize the
analyzed and identified progress deviations using performance metrics such as the Earned

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
Value Analysis (EVA) or Last Planner System (LPS) concepts. These metrics are then
visualized via color-coding the BIM elements in integrated project models, presented to project
personnel via a scalable and interactive web-based environment. The validation of this
formalism is discussed based on several real-world case studies.

Key words: progress monitoring—vision-based system—unmanned aerial vehicles—BIM

INTRODUCTION
Actual and potential progress deviations during construction of buildings and infrastructure
systems are costly but are preventable. To capture construction performance deviations
accurately and to make effective and prompt decisions based on the captured performance
deviations, frequent and accurate methods are needed for tracking construction progress. If
implemented on a daily basis, such methods can effectively bridge the gap in information
sharing between daily work execution and weekly work planning/coordination, ultimately leading
to improved project efficiencies (Bosché et al. 2014; Yang et al. 2015).

Today’s construction progress tracking methods, however, do not provide the accuracy and
frequency necessary for effective project controls. This is in large due to the labor intensive
processes involved with as-built data collection and analysis (Bae et al. 2014). It is common for
field engineers and superintendents to walk around the site, take photos, and document the
progress of ongoing operations, nevertheless this process is time consuming as it needs to be
conducted per daily task and for all associated construction elements. The ability to perform
monitoring and photographic documentation is also restricted by hard-to-reach areas. This
constraint negatively impacts the completeness of the data collection process. Once this data is
collected (whether it is complete or not), the field engineers filter, annotate, organize, and
present the collected data in comparison to 4D BIM. However, the cost and complexity
associated with the analysis and reporting operations result in sparse and infrequent monitoring
(Bae et al. 2013). Therefore through these activities for project controls, a portion of the gains in
efficiency is consumed by monitoring costs.

Over the past few years, research has focused on addressing current limitations through (1)
automatically generating 3D point cloud models from unstructured images and video sequences
(Brilakis et al. 2011; Golparvar-Fard et al. 2009), (2) aligning the resulting 3D point cloud models
with 4D BIM (Golparvar-Fard et al. 2011), and then (3) methods that can automatically infer the
status of work tasks and their relevant elements using geometry information (Golparvar-Fard et
al. 2012), appearance information (Han and Golparvar-Fard 2015), and via leveraging
formalized construction sequencing knowledge and reasoning mechanisms (Han and
Golparvar-Fard 2014a). While significant progress is achieved, yet many areas in research are
still open which require further investigation.

This paper presents a formal process for automating construction progress monitoring via
images taken with camera-equipped Unmanned Aerial Vehicles (UAVs) and 4D BIM. To
address the current limitations in accuracy and completeness of the data collection, we propose
to build on the emerging practice of using camera-equipped UAVs for collecting close-range site
imagery. Different from these practices, we propose to use 4D BIM to identify monitoring goals
for visual data collection, path planning, and autonomous navigation of the UAV through both
exterior and interior construction scenes. We also present a method that leverages BIM (1) to
improve accuracy and completeness of the 3D image-based point cloud modeling via UAV
captured images, and (2) for a better alignment of the images and the resulting point cloud

119
Han, Lin, Golparvar-Fard
models with the 4D BIM. For analytics, methods are presented to leverage geometry,
appearance, and inter-dependency information among elements together with formalized
construction sequencing knowledge to document progress for work tasks that have
corresponding physical elements. A new system architecture is also proposed for visualizing the
collected images, produced 3D point cloud models, and 4D BIM in a scalable web-based
environment. This environment allows user interactions which is particularly important for
collecting inputs on task constraints and those tasks that do not necessarily have physical
element correspondences. As shown in Figure 1, the resulting integrated project models can be
used to minimize the gap in information communication between work task coordination and
daily task execution. It can also support the identification and removal of work constraints and
root-cause analysis in construction coordination meetings.

Start Integrated Project Model

(Plan + As-built)
Yes, Performance Problem
Prepare Commit to Yes,
Master Weekly Monitor Progress Problems No Report Performance
Schedule Work Plan Work Progress / Deviations? Completed Task Problem

Look- Coordinate Quality No Confirm Task

Inspect Task
Ahead Weekly Issues? Complete
Schedule Work Plan Pull Information

Constraint Analysis Lead and Conduct Work As-Planned

End
Weekly Work Planning Daily Work Execution, and
Master Scheduling
and Coordination Performance Monitoring and Controls

Figure 1. The planning, execution and monitoring of weekly work plans and how
integrated project models generated via images taken from camera-equipped UAVs and
BIM can improve current work flows.

In the following, we provide an overview on the state of the practice and research on using
camera-equipped UAVs and BIM for progress monitoring. Next, we propose a formalized
procedure for leveraging 4D BIM for enhanced data collection, analytics, and communication of
construction progress deviations. A discussion is also provided on how the proposed procedure
and its associated tools can improve current information sharing and enhance construction
coordination processes.

STATE OF PRACTICE AND RESEARCH ON USING CAMERA-EQUIPPED UAVS AND BIM

FOR CONSTRUCTION PROGRESS MONITORING
Data collection
Over the past two years, the application of camera-equipped UAVs for comprehensive visual
documentation of work-in-progress on construction sites has gained significant popularity
among the Architecture/Engineering/Construction and Facility Management (AEC/ FM)
practitioners. Many construction and owner companies have procured their own UAV platforms
for frequent data collection. Several companies have also emerged that provide close range
aerial image data collection as a service. Ideally these UAVs should operate on sites on a daily
basis such that an accurate and complete visual documentation of work in progress can be
achieved.

120
Han, Lin, Golparvar-Fard
To operate autonomously, the UAV operators manually place waypoints on 2D maps and
leverage GPS coordinates for path planning and navigation purposes. The reliance on GPS for
navigation limits autonomous data collection to outdoors and causes major difficulties in dense
urban areas. Also the presence of steel components on sites affect the accuracy of the
magnetometers used on these platforms for navigation purposes, potentially causing safety
issues on construction sites. Placing waypoints manually can also create potential safety
hazards as the 2D maps used for navigation do not reflect the most updated status of the
resources on site, and leave the UAV operators to approximate the 2D location and height of
the new construction and resources such as mobile cranes.

Because the locations where construction progress is expected on project sites are not known
in advance, current best practices require the UAVs to cover the entirety of a site. Since current
batteries are limited to 15-35min operations, covering the entirety of a project site needs
multiple flights and manual supervision by the UAV operators. With the number of images
increasing, the computation time necessary for processing these images, generating 3D point
cloud models, and analyzing work-in-progress also exponentially grows.

While research (Lin et al. 2015a; Siebert and Teizer 2014; Zollmann et al. 2014) has focuses on
leveraging UAV-based images for progress monitoring, the aforementioned challenges are still
remained unexplored. Figure 2 (a) shows a camera-equipped UAV flying onsite, (b) an operator
using the controller to navigate the UAV for data collection and (c) a commercially available
application (DJI Ground Station 2015) for setting the waypoints for UAV navigation.

Figure 2. (a) camera-equipped UAV (b) operator executing data collection task on the
construction site, (c) waypoints setting in commercially available application (DJI Ground
Station 2015).

Visual Analytics for Progress Tracking

Today, there are two dominant practices for leveraging images collected from camera-equipped
UAVs for tracking work in progress:

(1) Generating large panoramic images of the site and superimposing these large-scale high
resolution images over existing maps– While these images provide excellent visuals to ongoing
operation, they lack 3D information to assist with area-based and volumetric-based
measurements necessary for progress monitoring. Also none of the current commercially
available platforms provide a mechanism to communicate who is working on which tasks at
what location.

(2) Producing 3D point cloud models– Over the past decade, the state-of-the-art in image-based
3D modeling methods from computer vision domain has significantly advanced. These

121
Han, Lin, Golparvar-Fard
developments have led to several commercially available platforms that can automatically
produce 3D point cloud models from collections of overlapping images. Several AEC/FM firms
have started to leverage these platforms to produce as-built 3D point cloud models of their
project sites via images taken from camera-equipped UAVs. Nevertheless, today’s practices are
mainly limited to measuring excavation work and stockpiles. This is because highly overlaying
images taken with a top-down view can produce high quality 3D models of these operations.
However creating complete 3D point cloud models for building and infrastructure systems also
requires the UAVs to fly around the structure to capture work in progress. Because there is not
automated mechanism for identifying most informative viewpoints, often the produced 3D point
cloud models are incomplete. Also the state of the art Structure from Motion (SfM) techniques
for image-based 3D reconstruction –as used in (Golparvar-Fard et al. 2012; Golparvar-Fard et
al. 2011) - may distort angles and distances in generated point cloud models. Figure 3 shows an
example of a point cloud model that was generate via highly overlapping images taken around
the perimeter of a construction site. The reconstructed 3D point cloud is up-to-scale and exhibits
problems in completeness and accuracy.

Figure 3. Typical challenges in using Standard SfM techniques for image-based 3D

reconstruction: (a) the UAV’s flight path around the project site for data collection. Here
the location and orientation of the images taken are shown with small pyramids; (b) the
produced 3D point cloud model is incomplete; (c) distortions in angle and distance, and
(d) the reconstructed 3D point cloud is up to scale and is unit less.

Comparing Image-based 3D Point Clouds and BIM

Once the as-built 3D point cloud models are generated and are integrated with as-planned BIM,
the resulting integrated project models can be used to identify progress deviations. The state-of-
the-art methods for identifying these deviations mainly falls into two categories:

(1) Analyzing the physical occupancy of the as-built models: Research on occupancy based
assessment methods use 3D point cloud models and BIM to monitor whether or not structural
elements, Mechanical/Electrical/plumbing (MEP) components, or temporary resources such as
formwork and shoring are physically present in the as-built point cloud model. These methods

122
Han, Lin, Golparvar-Fard
are still challenged with the lack of enough details in (1) BIM and (2) work breakdown structure
of the schedule. The 4D BIM may not have a physical representation for all elements,
particularly for the temporary resources. Hence model-driven monitoring is not possible for all
elements. The lack of formalized construction sequencing knowledge such as steps in
placement of concrete elements (i.e. forming, reinforcing, placing, and stripping) in 4D BIM also
adds to the complexity of the assessments, since without knowing exactly when each element is
expected to be placed, identifying the most updated progress status is not possible.

(2) Analyzing the appearance of the changes of the as-built models: The latest research on
appearance-based assessment of the as-built models superimposes the BIM with the 3D point
cloud models and then back-projects the BIM elements to the images used to generating the
point cloud model. From these back projections, several squared shape image patches are
sampled and used to analyze the observed construction material. The most observed material
from these image patches is then used to infer the most updated status of progress for each
element in 4D BIM (Han and Golparvar-Fard 2015).

While significant advancement in research has been made, still applying these methods to a
full-scale projects requires (1) accounting for the lack of details in 4D BIM, (2) addressing as-
built visibility issues, (3) creating large-scale libraries of construction materials that could be
used for appearance-based monitoring purposes; and (4) methods that can jointly leverage
geometry, appearance, and interdependency information in BIM for monitoring purposes.

Visualization and Information Communication

Today’s platforms for visualizing and communicating work in progress can produce 2D
panoramas and 3D point cloud models (with or without BIM) for highlighting the work in
progress and deviations from plan. The current 2D interfaces build on panoramic images and
present a top-down view of project sites and ongoing operations. These project controls
interfaces also provide tools for annotating the imagery for monitoring work in progress and field
reporting purposes. However the provided functions are not sufficient for practical progress
monitoring and information communication. For example, none of the current interfaces envision
any workflow as to how the visuals can facilitate information sharing and communication.

The 3D interfaces also have a number of challenges that currently prevent their wide spread
application. These interfaces either show the BIM or the 3D point cloud models. Hence it is
difficult to differentiate and highlight the changes between isolated as-planned and as-built
models. Also, the measurement tools in these interfaces are only capable of basic interactive
operations such as 3D volumetric measurements from point cloud models. To facilitate
information sharing and communication, the integrated project models – produced by
superimposing BIM and the 3D point cloud models– should remain accessible by all onsite and
offsite practitioners. Without having access to scalable and interactive web-based systems that
can support such functionalities, it will be difficult to use the integrated project models to support
smooth flow of information among project participants.

Overall, there is a lack of an end-to-end formalized procedure that can take account for visual
as-built data collection, progress monitoring analytics, and visualization. In the following, a
formal procedure is presented that can address current limitations. The opportunities for further
research in each step of the procedure are discussed as well.

123
Han, Lin, Golparvar-Fard
FORMAL METHODS FOR AUTONOMOUS MONITORING OF CONSTRUCTION PROGRESS
This section proposes a procedure for autonomous vision-based monitoring of construction
progress, which consists of 1) data collection using camera-equipped UAVs, 2) vision-based
analytics and comparison with 4D BIM for reasoning about progress deviations, and 3)
performance analysis and visualization. Figure 4 illustrates these steps in detail.

Figure 4. A procedure for autonomous vision-based monitoring of construction progress

Data collection
Figure 5 shows two different procedures for collecting visual data using UAVs for construction
progress monitoring. In the first procedure, the images are directly used to produce 3D and 4D
point cloud models. In the procedure shown in Figure 5b, the 4D BIM is used to assist with the
data collection process. Since the 4D BIM entails information about where changes are
expected to happen on a construction site, it can serve as a great basis for identifying scanning
goals, path planning and navigation. This strategy can also potentially address the current
limitations associated with path planning and navigation of the UAVs in interior spaces and
dense urban areas. This is particularly important as current methods primarily rely on GPS for
UAV control purposes while at indoor scenes or dense urban areas, reliable GPS is not
accessible.

To support a complete and accurate image-based reconstruction – whether BIM is used for data
collection or not– images should be taken with an overlap of 60-70%+. This rule of thumb
guarantees detection of sufficient number of visual features in each image, which is typically
required for standard Structure form Motion procedures. Taking images with such overlaps
require adjusting and controlling the flight path and speed of the UAV. BIM guided data
collection process as discussed in (Lin et al. 2015b) will certainly require less number of images
and can more intelligently assist with choosing informative views necessary for progress
monitoring purposes.

(a) (b)

Figure 5. Two different alternatives for progress monitoring data collection

124
Han, Lin, Golparvar-Fard
Data analytics
Alignment of image and BIM is the first step in vision-based analytics for construction progress
monitoring. In early research (Golparvar-Fard et al. 2012), a pipeline of image-based
reconstruction procedures consisting of Structure from Motion and Multi View Stereo algorithms
were used to generate point clouds and then transform them into the BIM coordinate system.
However, the 3D point cloud models generated through these standard procedures are up to
scale. To recover the scale, a user-driven process is required to select at least three
correspondences between the BIM and the point cloud. Utilizing these correspondences
between the 3D point cloud model and BIM, the least square registration problem can be solved
for the 7 degrees-of-freedom (3 rotation, 3 translation, 1 uniform scale) to transform and scale
the point cloud to the BIM coordinate system. This manual process can be improved by
leveraging BIM as a priori and adopting a constrained-based procedure for image-based 3D
reconstruction which is shown in Figure 6 (Karsch et al. 2014). The results from preliminary
experiments in (Karsch et al. 2014) show that the accuracy and completeness of the image-
based 3D point clouds can be significantly improved with using BIM as a priori (Figure 7).

Figure 6. Leveraging BIM as a priori for as-built modeling purposes

Figure 7. BIM-assisted SfM: enhanced accuracy of overlaying BIM on site images.

125
Han, Lin, Golparvar-Fard
Geometry and appearance-based progress monitoring analysis is the next step of data
analytics. As discussed in section 2 current practices are suffering from challenges of geometry
and appearance-based progress monitoring methods. To formalize the utilization of integrated
project models for progress monitoring purposes and facilitating information flows, the following
practical challenges that should be addressed first:

(1) Lack of detail in as-planned models- For accurate progress monitoring analytics, the as-
planned model should contain a BIM with level of development of 400/450. It should also reflect
daily operational details within the work break down structure (WBS) of the schedule which is to
be integrated with BIM. Nevertheless, in today’s practice of project controls, daily operation-level
tasks are not often reflected in the WBS. Their corresponding elements such as scaffolding and
shoring are also not typically represented in BIM. Hence with using these models, it is not easy
to identify “who does which work at what location”, especially for works related to temporary
structures and for detecting both geometry and appearance based changes. Formalizing
knowledge of construction sequencing and then enhancing the BIM with relevant reasoning
mechanism, can enhance current progress monitoring methods. Figure 8 illustrates an example
of this issue. As can be seen in the figure, the rebars and formwork should have been present in
the as-planned models for accurate assessment of work-in-progress on placement a concrete
foundation wall.

Figure 8. The necessary LoD in BIM that can enable comparison between as-built and as-
planned for progress monitoring (Han et al. 2015)

Figure 9. An example on how formalized knowledge of construction sequencing can

resolve limited visibility issues

(2) Limited visibility- Although taking images from different viewpoints – assisted by the UAVs–
may reduce the challenges associated with no visibility to construction elements, yet any
progress monitoring method should still be able to account for progress reasoning based on

126
Han, Lin, Golparvar-Fard
partial element visibilities. Formalizing knowledge of construction sequencing and integrating
that into BIM through a reasoning mechanism can address this issue to some degree. The
preliminary study conducted in (Han and Golparvar-Fard 2014b) shows that such formalized
knowledge can enhance the performance of vision-based progress monitoring methods by five
to seven percent. Figure 9 illustrates a case with a visibility issue that can be resolved by this
reasoning mechanism.

Visualization and Information Communication

Developing a scalable and interactive web-based platform for visualizing images, point cloud
models, BIM, together with analytical tools– particularly considering the limited memory and
bandwidth on commodity smartphones and tablets– is challenging. The main requirements are:
(1) Interaction, presentation, and manipulation of large-scale point cloud models,
(2) Displaying integrated point cloud model with 4D BIM, and
(3) Offering quick and easy to use analytical and communication tools.

As a first step towards addressing these challenges, (Lin et al. 2015a) introduces a new web
platform that can visualize as-built vs as-planned models. This new platform allows users to
access integrated project models on smartphones and tablets. To present large-scale point
cloud models in a convenient and scalable manner considering the limited memory available in
commodity smartphones and tablets, the point cloud models are structured in form of nested
octrees similar to (Scheiblauer et al. 2015). Figure 10 shows the developed web platform for
visualizing as-built point clouds generated by camera-equipped UAV and as-planned BIM.
Figure 10b in particular shows an example of the nested octree.

This data structure subsamples the point cloud and only shows relevant points depending on
the user viewpoint. Also the number of points projected to the same pixel on the screen will be
reduced according to the level of details chosen by the user. Loading a point cloud with density
of 10 million points takes only approximate to 2 seconds on a standard commodity smartphone.
The manipulation method can also be adjusted to first person or third person view controls. The
platform is built with BIMServer (Beetz et al. 2010) to integrate 4D BIM (Figure.10 g-h) for
visualization and information retrieval purposes. The semantic information necessary for
progress monitoring (e.g. expected construction materials and element inter-dependency
information) can be queried from BIM and presented through the integrated platform. To assist
with documenting work in progress for those tasks that do not have any geometrical
representation, several new tools are also created that allow the point cloud model to be directly
color coded and semantics such as “who does which work at what location” to be accounted for
based on various locations. This location-based method allows the issues associated with level
of detail in BIM to be also addressed, since now the users can push and pull information from
any user-annotated location (with or without a BIM representation). Integrating various
workflows such as quality control as shown in Figure 1 is part of ongoing research.

DISCUSSION
The integrated project model presented in the previous section can bridge the gap in information
sharing between the downstream feedback (onsite activities) and the short-term planning
(coordination meeting). By providing a near real-time visualization of ongoing work, particularly
who does which work at what location, it allows the most updated status of work in progress to
be communicated among all parties on and offsite. The platform shows potential in supporting
easy and quick identification of potential performance problems. It can also support root-cause
analysis discussions in coordination meetings, ultimately leading to a more smooth flow of
production in construction. Instead of measuring and communicating retrospective progress

127
Han, Lin, Golparvar-Fard
metrics such as the EVA or Percentage Plan Complete (PPC) metrics, intuitive and data-driven
communication of work in progress can allow for measuring progress based on more
informative metrics such as task maturity or Task Anticipated (TA) and Tasks-Made-Ready
(TMR) (Hamzeh et al. 2015).

Figure 10. Web platform for visualizing as-built point clouds generated by camera-
equipped UAV and as-planned BIM. (a) as-built point cloud; (b) the nested octree for pre-
processing and visualizing point cloud in a scalable manner; (c) the location and
orientation of the UAV-mounted camera when images were taken derived from the
Structure from Motion algorithm; (d) viewing the point cloud models through one of the
camera view points and texture mapping the frontal face of the camera frustum; (e) area
measurement tool which directly operated on the as-built point cloud; (f) color coding
part of the as-built point cloud for communicating who does which work at what location;
(g) and (h) integrated visualization of the as-built point cloud and the as-planned BIM for
two different construction projects.

128
Han, Lin, Golparvar-Fard
CONCLUSION AND FUTURE WORK
This paper presents a formalized procedure via utilizing images taken by camera-equipped
UAVs and BIM for generating integrated project models for construction progress monitoring
purposes. The various aspects of data collection, analysis, and communication as it pertains to
these integrated project models were discussed. These integrated project models have potential
for enhanced information flow and improving situational awareness on construction projects.
They can also support identifying and removing work constraints in coordination meetings, and
measuring proactive metrics such as task maturity, Task Anticipated (TA), or Tasks-Made-
Ready (TMR). The current limitations in data collection and analysis were discussed as well.
Ongoing research is focused on addressing these limitations as well as conducting pilot projects
on using the proposed procedures for creating integrated project models and validating their
potential in smoothening information flow and improving situational awareness on construction
projects.

ACKNOWLEDGMENTS
This research is financially supported by the National Science Foundation (NSF) Grant CPS
#1446765. Any opinions, findings, and conclusions or recommendations expressed in this
material are those of the authors and do not necessarily reflect the views of the National
Science Foundation. The technical support of the industry partners in providing access to their
sites for data collection and assisting with progress monitoring analytics is also appreciated.

REFERENCES

Bae, H., Golparvar-Fard, M., and White, J. (2013). “High-precision vision-based mobile
augmented reality system for context-aware architectural, engineering, construction and
facility management (AEC/FM) applications.” Visualization in Eng, Springer, 1(1), 1-13.

Bae, H., Golparvar-Fard, M., and White, J. (2014). “Image-Based Localization and Content
Authoring in Structure-from-Motion Point Cloud Models for Real-Time Field Reporting
Applications.” Journal of Computing in Civil Engineering, DOI:
1023 .1061/(ASCE)CP.1943-5487.0000392, B4014008, 637–644.

Beetz, J., van Berlo, L., de Laat, R., and van den Helm, P. (2010). “bimserver. org–An Open
Source IFC Model Server.” Proceedings of the CIB W78 2010: 27th International
Conference–Cairo, Egypt, (Weise 2006), 16–18.

Bosché, F., Ahmed, M., Turkan, Y., Haas, C. T., and Haas, R. (2014). “The value of integrating
Scan-to-BIM and Scan-vs-BIM techniques for construction monitoring using laser scanning
and BIM: The case of cylindrical MEP components.” Automation in Construction, 49, 201-
213, Elsevier.

Brilakis, I., Fathi, H., and Rashidi, A. (2011). “Progressive 3D reconstruction of infrastructure
with videogrammetry.” Automation in Construction, Elsevier, 20(7), 884–895.

DJI Ground Station (2015). http://www.dji.com/info/releases/dji-released-2-4g-bluetooth-

datalink-ipad-ground-station, last accessed 5/18/2015.

129
Han, Lin, Golparvar-Fard
Golparvar-Fard, M., Pena-Mora, F., and Savarese, S. (2009). “D4AR- A 4-Dimensional
Augmented Reality model for automating construction progress data collection, processing
and communication.” Journal of ITCON, 14(1), 129–153.

Golparvar-Fard, M., Peña-Mora, F., and Savarese, S. (2011). “Integrated Sequential As-Built
and As-Planned Representation with Tools in Support of Decision-Making Tasks in the
AEC/FM Industry.” Journal of Construction Engineering and Management. 137(12), 1-21.

Golparvar-Fard, M., Peña-Mora, F., and Savarese, S. (2012). “Automated Progress Monitoring
Using Unordered Daily Construction Photographs and IFC-Based Building Information
Models.” Journal of Computing in Civil Engineering, 10.1061/(ASCE)CP.1943–
5487.0000205.

Hamzeh, F. R., Saab, I., Tommelein, I. D., and Ballard, G. (2015). “Understanding the role of
‘tasks anticipated’ in lookahead planning through simulation.” Automation in Construction,
49, Part A(0), 18–26.

Han, K., and Golparvar-Fard, M. (2014a). “Multi-Sample Image-Based Material Recognition and
Formalized Sequencing Knowledge for Operation-Level Construction Progress Monitoring.”
Computing in Civil and Building Engineering, I. Raymond and I. Flood, eds., American
Society of Civil Engineers, 364–372.

Han, K., and Golparvar-Fard, M. (2014b). “Multi-Sample Image-Based Material Recognition and
Formalized Sequencing Knowledge for Operation-Level Construction Progress Monitoring.”
International Conference on Computing in Civil and Building Engineering, 2014, 364–372.

Han, K., and Golparvar-Fard, M. (2015). “Appearance-based material classification for

monitoring of operation-level construction progress using 4D BIM and site photologs.”
Automation in Construction, Elsevier, 53, 44–57.

Han, K., Lin, J., and Golparvar-Fard, M. (2015). “Model-driven Collection of Visual Data using
UAVs for Automated Construction Progress Monitoring.” Int’l Conference for Computing in
Civil and Building Engineering 2015, Austin, TX June 21-23.

Karsch, K., Golparvar-Fard, M., and Forsyth, D. (2014). “ConstructAide: analyzing and
visualizing construction sites through photographs and building models.” ACM
Transactions on Graphics (TOG), ACM, 33(6), 176.

Lin, J., Han, K., Fukuchi, Y., Eda, M., and Golparvar-Fard, M. (2015b). “Model-Based Monitoring
of Work-in-Progress via Images Taken by Camera-Equipped UAV and BIM.” 2nd
International Conference on Civil and Building Engineering Informatics, Tokyo, Japan.

M. Golparvar-Fard, Pena-Mora, F., Savarese, S., Golparvar-Fard, M., Pena-Mora, F., and
Savarese, S. (2011). “Monitoring changes of 3D building elements from unordered photo
collections.” Computer Vision Workshops (ICCV Workshops), 2011 IEEE International
Conference on, 249–256.

Scheiblauer, C., Zimmermann, N., and Wimmer, M. (2015). “Workflow for Creating and
Rendering Huge Point Models.” Fundamentals of Virtual Archaeology: Theory and
Practice, A K Peters/CRC Press.

130
Han, Lin, Golparvar-Fard
Siebert, S., and Teizer, J. (2014). “Mobile 3D mapping for surveying earthwork projects using an
Unmanned Aerial Vehicle (UAV) system.” Automation in Construction, Elsevier, 41, 1–14.

Yang, J., Park, M.-W., Vela, P. A., and Golparvar-Fard, M. (2015). “Construction performance
monitoring via still images, time-lapse photos, and video streams: Now, tomorrow, and the
future.” Advanced Engineering Informatics, Elsevier. 29 (2), 211–224.

Zollmann, S., Hoppe, C., Kluckner, S., Poglitsch, C., Bischof, H., and Reitmayr, G. (2014).
“Augmented Reality for Construction Site Monitoring and Documentation.” Proceedings of
the IEEE, IEEE, 102(2), 137–154.

131
Han, Lin, Golparvar-Fard
Exploratory Study on Factors Influencing UAS
Performance on Highway Construction Projects: as
the Case of Safety Monitoring Systems
Author 1 Sungjin Kim, 2 Javier Irizarry
School of Building Construction, College of Architecture
Georgia Institute of Technology
280 Ferst Drive, 1st Floor
Atlanta, GA, 30318
Email: [email protected], [email protected]

ABSTRACT
Highways are one of the most important infrastructure systems that support our society.
Infrastructure construction projects, particularly highway projects, have become increasingly
large and complex. It is critical that project managers protect workers from accidents by
providing a safe work environment. Therefore, safety managers should frequently monitor the
worksite conditions and prevent accidents on construction sites. Unmanned Aerial Systems
(UASs) have a great potential to fly and monitor highway construction sites, since the horizontal
layout of these jobsites has few obstacles that disturb the operation of UASs. Comprehensive
literature reviews aim to refine factors influencing the performance of UAS as safety monitoring
systems on the jobsite. A questionnaire survey was developed to analyze UAS implications on
highway projects and critical factors based on the derived potential 29 factors and 17 benefits
that can be performance measure attributes. As a beginning step of understanding UAS in
construction environments, this exploratory research contributes to defining critical factors
influencing the performance of UAS on highway construction environments.

Key words: critical factor analysis—highway construction project—performance

measurement—safety monitoring system—unmanned aerial system (UAS)

INTRODUCTION
Infrastructure construction projects require a significant amount of time, capital and human
resources during the project life cycle. It has also required more advanced construction
management systems to assists project managers with the significant challenges faced in
continuously evolving worksites. One of the most momentous challenges in infrastructure sites
is construction workers’ safety. Even with the progress made in site safety, the laborers are still
exposed to hazardous conditions. The Occupational Safety and Health Administration (OSHA)
in the United States requires that project owners and contractors have a duty to prevent workers
from having fatal accidents on worksites. According to the U.S. Bureau of Labor Statistics, 796
construction-related fatal injuries were occurred in 2013 (BLS 2015). This statistic indicated the
fatality ratio in construction industry is a big concern in the United States.

One of the safety managers’ main responsibilities is to observe workers’ behavior, work
sequencing and the jobsite conditions to prevent workers from having construction-related
accidents. For the last few years, many researchers have developed advanced Information
Technology (IT) based safety monitoring systems to reduce accidents on construction sites. Lin
developed workers’ behavior monitoring system on construction site (Lin et al. 2013) and
Naticchia (2013) developed real-time unauthorized interference control system based on RFID

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
(Naticchia et al. 2013). As an important part of our infrastructure, highway construction projects
have unique characteristics. These horizontal construction projects can extend through long
distances. Hence, they need monitoring systems that can cover safety issues on these
extensive construction worksites and their surrounding environments.

A proposed method to address this issue is the use of Unmanned Aerial System (UAS), which
are remotely operated without an onboard operator. UASs can maximize the efficiency of safety
monitoring in highway construction projects because no vertical obstacles affecting UAS
operations would be present. UASs can easily monitor all site areas by circling around the site
under a safety manager’s control and delivering real-time images and videos. This technology
has been utilized to perform monitoring or inspecting in various industries, for example,
monitoring soil erosion (D’Oleire-Oltmanns et al. 2012) monitoring forest fires (Hinkley and
Zajkowski 2011) and bridge inspection (Eschmann et al. 2012; Morgenthal and Hallermann
2014). In particular, Irizarry developed the concept of UAS-based safety inspection system
during construction (Irizarry et al. 2012).

However, the use of UASs on construction sites is in very early stages and their impact on
construction management tasks is not known and the implication of UASs use in construction is
not fully understood. Although UAS is being researched in various industries around the world,
very few studies on the impact of UAS applications in construction have been carried out. In
order to understand the potential of this technology in construction environments, performance
measurement and evaluation of UASs for tasks, such as safety monitoring, is necessary for
supporting the implementation of UAS technology in all types of construction projects. This
research defines critical factors influencing UAS performance through extensive literature
review, surveys and interviews with construction and IT professionals. This study aims to define
performance measurement factors for the performance of UAS in construction safety monitoring
tasks in highway construction or other infrastructure projects. Furthermore, the main findings will
be able to contribute to our understanding of the potential impact of adoption of UAS technology
in the Architecture, Engineering, Construction and Operations (AECO) industry.

PROBLEM STATEMENT
Zhou (2013) comprehensively reviewed 119 published studies in terms of advanced safety
management technology between 1986 and 2012 (Zhou et al. 2013). Even though the number
of research focusing on IT-Based safety control systems has dramatically increased over two
decades (Figure 1A), the construction accident rate has been still unstable and frequent in the
AECO industry (Figure 1B). In addition, the construction injury rate is a big concern with this
industry having the highest fatality ratio 21 percent from all industries in the United States
(Figure 1C) (BLS 2015) Zhou concluded that future studies should pay more attention to
proactive safety management systems, such as safety monitoring or information systems rather
than the reactive system for achieving the goal of zero accidents on worksites (Zhou et al. 2013).

As one of the proactive safety control systems, Irizarry (2012) suggested the concept of a UAS
safety inspection and monitoring system in the AECO industry. He proposed the UAS could
provide aerial images as well as real-time videos from a range of locations around the jobsite to
safety managers with fast access, since it was equipped with high resolution camera and
Wireless system (Irizarry et al. 2012). Even though UAS research has been of increased
interest to practitioners in the AECO industry, the concept’s application has not been perfectly
investigated. Moreover, researchers have never measured UAS’s effects on various
construction environments. For this reason, measurement and evaluation of the performance of
UAS-based safety monitoring system is needed. In addition, many important factors associated

Kim and Irizarry 133

with UAS technology are not fully understood. Recent studies have aimed to solve existing
technical problems, introduced new technology to improve the current systems, or ignored the
factors influencing the performance of the systems.

Figure 1. IT Safety Research trend and Construction Accidents

Therefore, there are gaps in most practitioners’ understanding of which factors can affect the
performance of UAS system. This paper will answer the research questions stemming from
three main research problems and fill the aforementioned gaps (Table 1.) The goal of this study
is to define critical factors that may contribute or influence the performance of UAS-based
monitoring system through industry-wide surveys.

Table 1. Research Problem Statement

Research Problems Research Questions Research Gaps

How has the UAS concept
AECO Industry is not
been applied in industries
familiar UAS
including AECO?

UASs’ potential applications How can a UAS be applied Research Gaps are
(i.e. Safety Monitoring task) for safety monitoring or
is not understood inspection in construction? (1) What factors can
affect performance of
What are critical factors UAS in case of safety
that may influence the monitoring tasks, (2)
performance of UAS in How they affect, and
(3) What their impacts
Factors that may contribute safety monitoring?
are on the highway
to the performance of UAS
construction worksite
and benefits from UAS have What are potential benefits
not been investigated of UAS in case of safety
monitoring, or the potential
attributes to evaluate the
performance of UAS?

Kim and Irizarry 134

RESEARCH BACKGROUND
Safety Monitoring
Construction environments usually have low quality working-conditions as well as high accident
risk potential. Safety managers are responsible for preventing construction workers from having
any type of construction accident. Toole (2002) determined that most construction accidents
have eight basic causes: (1) Insufficient Safety Training, (2) Weak enforcement of safety, (3)
Insufficient safety equipment (4) Unsafe work method sequencing, (5) Unsafe site conditions,
(6) Improper Safety equipment, (7) Poor attitude toward safety, and (8) workers’ bad behavior
(Toole 2002). The safety manager should pay attention to the above eight causes of fatalities on
construction jobsites. Irizarry (2012) identified the three main characteristics of observation in
safety inspection. First, the observation should be frequently performed. Second, the manager
should be able to observe in person all designated areas and tasks on the jobsite. Third, the
managers should interact with workers on site (Irizarry et al. 2012).

Lee and Kim (2009) developed a mobile safety monitoring system, which is based on hybrid
sensor, in order to reduce the rate of fatal accidents on construction sites. Prabhu (2005)
developed a Tablet PC application to inspect jobsites for safety issues and prevent fatalities.
Leung et al (2008) developed a monitoring system based on network cameras that can be used
in a web environment to observe the quality of construction work. Lin et al (2013) developed a
model for tracking workers’ behavior on a large-dam construction site. Naticchia (2013)
reviewed previous literature and developed a monitoring system based on ZigBee technology in
order to detect unauthorized worker behavior in real-time. Table 2 summarizes previous
research about safety monitoring systems.

Table 2. Current IT-based safety monitoring systems

References System Function Main Information Technology

Reduce dead zones of safety
(Lee et al. Developed and applied Hybrid
management where fatal
2009) on the real jobsite Sensor
accidents, especially falls
Safety inspection, Modeling, performance Tablet PC
(Sunkara 2005)
Fatality Prevention evaluation, case study Application

(Leung et al. Monitoring the quality of Developed the model, Network

2008) construction work Pilot study Camera

Workers’ behavior analysis, Modeling, Application RSSI,

(Lin et al. 2013)
Onsite Tracking on real site ZigBee

(Naticchia et al. Real-time unauthorized Laboratory testing, RFID

2013) interference control Onsite test ZigBee

UAS Applications
Unmanned Aerial System (UAS)
Unmanned Systems have brought great benefits to military missions and most recently to
civilian life. These systems are unmanned hardware platforms equipped to perform data
collection and sometimes processing, and to operate without direct human intervention, such as

Kim and Irizarry 135

unmanned aerial vehicle, satellites, and underwater explorers. In particular, Unmanned Aircraft
System (UAS) has been the focus of potential application in various industry areas in the United
States in recent years. UAS, commonly known as a drone, can be defined as, “A powered aerial
vehicle that does not carry a human operator, uses aerodynamic forces to provide vehicle lift,
can fly autonomously or be piloted remotely” (Newcome 2004). The Federal Aviation
Administration (FAA), the national aviation authority of the United States, has adopted the term,
“UAS (Unmanned Aircraft/Aerial System)” instead of “UAV (Unmanned Aerial Vehicle)”,
because the concept of unmanned aircraft should have ground control station, vehicles, and
other elements for its safe operation.

Federal Aviation Administration (FAA) Regulation in the U.S

Federal Aviation Administration (FAA) has authority over all commercial flight regulations in the
United States National Airspace System (NAS). The FAA defines two types of small UAS
operations in the United States: (1) public operations and (2) civil operations, and they managed
the regulatory environments for each flight type. Therefore, familiarity with these regulations is
very important if the commercial application of UAS on construction environments is to be
understood.

Public operations can be defined as government-related aircraft operations to conduct public

activities including military operations, law enforcement, and research and development. FAA
issues Certificate of Authorization or Waiver (COA) to qualified applicants who want to use UAS
for public operations in the United States. COA is a permission that UAS operators must request
to use UASs for commercial applications (FAA 2015A). Then COA holders can control and
operate UASs legally. Between 2006 and 2012, 791 COA applications were approved out of
897 based on analysis of FAA’s released data. In addition, total 79 organizations have obtained
COAs from the FAA as of February 25th, 2015 (FAA, 2015B). Civil operations can be considered
any operation that cannot meet the required criteria for public operations. The FAA suggests
two methods to grant an authorization for civil operations: Section 333 exemptions and
Airworthiness Certificate. A total of 289 petitioners have been granted Section 333 exemptions
as of May 5th, 2015 (FAA, 2015C).

UAS Applications in various industries

Unmanned Aerial Systems (UASs) were initially utilized in military missions and boarder
observation in the United States (Nisser and Westin 2006). More recently, considerable other
studies have also been devoted to potential applications of UASs in various industries such as
agriculture, meteorological observation, forestry, archeology, and infrastructure (Table 3). In
addition, federal or state agencies in the United States have been operating small UASs with
the goal of law enforcement or surveillance. According to the Association for Unmanned Vehicle
Systems International (AUVSI), the market for UAVs is about $11.3 billion, and the potential
spending will grow to over $140 billion for the next 10 years in the United States (Jenkins and
Vasigh 2013).

UAS Safety Inspection and Monitoring System

Irizarry (2012) suggested the concept of a UAS safety inspection tool and examined two
feasible scenarios for UAS implementation. The UAS can fly all around the construction jobsite
and provide safety managers with real time information such as work sequencing and worker
behavior (Irizarry et al. 2012). A UAS can be equipped with high-resolution cameras, various
sensors including a Global Positioning System (GPS). In addition, they can be operated

Kim and Irizarry 136

autonomously or semi-autonomously with pre-defined flight paths over construction sites. Figure
2 shows the general framework of UAS’s application for monitoring safety issues on
construction sites. First, Heuristic Evaluation (HE) was conducted to evaluate the user interface
of an of-the-shelf and quad copter type UAS (Parrot AR drone). Second, a user participation
experiment was designed to decide the optimal interface size among three alternatives: (1)
Plain screen, (2) Phone size, and (3) Tablet PC. In this experiment, it was concluded that the
combination of drone and user interface with tablet PC size display was adequate to perform
safety inspection related tasks on the construction site. The results motivate the continued study
of UAS technology for safety management related tasks.

Table 3. Recent UAS’s application studies in various industries

Industry References UAS’s Function

(D’Oleire-Oltmanns et al. 2012) Monitoring soil erosion

Agriculture
And Forestry
(Hinkley and Zajkowski 2011) Monitoring forest fires

Observing atmospheric
Meteorology (Reuder et al. 2009)
Boundary Layer
Monitoring daily work of the
(Rinaudo, Chiabrando, Lingua 2012)
excavation
Archeology
Creating 3D models of
(Saleri et al. 2013)
archaeological heritages
(Eschmann et al. 2012; Morgenthal
Bridge inspection
and Hallermann 2014)
Infrastructure
(Eschmann et al. 2012) Building and facility inspection

Figure 2. Concept of UAS-based safety monitoring

Kim and Irizarry 137

Critical Factors Influencing Performance
In the AECO industry, a Performance Measurement System (PMS) is an essential element of
construction management. The use of a simple and well-designed PMS is essential for
supporting the implementation of business strategies such as the application of drone
technology for safety monitoring on the construction site (Sarhan and Fox 2013). It is very
important that the factors affecting performance should be reasonably defined in order to
accurately evaluate performance of a system. Nitithamyong (2006) conducted an empirical
study to define potential factors influencing performance of commercial Web-Based
Construction Project Management Systems (WPMSs). The author defined 42 potential factors
and potential 36 measurements, which can be used for evaluating performance of the WPMS.
These potential elements were further refined from previous literature reviews, interviews with
professionals from 18 construction firms and the results of collected data from the questionnaire
survey of 39 professionals who had practical experience in the AECO industry (Nitithamyong
and Skibniewski 2006).

RESEARCH METHODOLOGY
This exploratory study aimed defining and analyzing factors that may influence the performance
of a UAS-based safety monitoring system was conducted in three main phases and activities as
shown in figure 3. The stages are (1) development of a conceptual factor model, (2) survey
instrument, (3) analyzing and defining factors. The following sections in this chapter describe
the steps of the study in more detail.

Figure 3. Research Work plan

The first phase began with a comprehensive literature review and interviews to develop an initial
factor model. The UAS-based safety inspection concept was used in study (Irizarry et al. 2012),
factor analysis for implementing safety management system was performed (Ismail et al. 2012),
and Nitihamyong and Skibniewski (2006) method for defining success or failure factors and
measures of a web-based project management system was adopted in this study. In addition,

Kim and Irizarry 138

semi-structured interviews with seven professional IT researchers and three construction project
personnel were conducted. The interviewees first watched the visual assets (images and
videos) obtained from a UAS at construction sites on a laptop and were introduced to the UAS
safety monitoring system concept. Then they provided feedback that was used to develop the
factors that may have an effect on the performance of a UAS safety monitoring system on
construction environments. The interviews were conducted on November 2014.

Questionnaire Survey Instrument

The questionnaire Instrument had three parts, including demographic questions, personnel
opinions on UAS and highway construction projects, and the initial set of factors. Demographic
questions were asked to understand the background of survey participants. The second section
required participants to describe their knowledge and experience of UAS as well as uses of a
UAS safety monitoring system on highway construction projects. The respondents were also
requested to indicate if the presented factors would affect the UAS’s performance on highway
construction projects. Questions with three choice options; Yes, No, Do not know, were used
for indicating participants agreement level. Topics in the questions included UAS’s system
characteristics, project characteristics, project team characteristics. This questionnaire survey
was reviewed by Georgia Institution of Technology’s Institutional Review Board (IRB) and
approved for use with human subjects. A total 15 responses have been collected in a three-
week period from mid April to early May 2015.

RESULTS
Developing Conceptual Factor model
A total of 46 potential factors were derived based on the three previous studies and input from
the interview with IT and construction professionals. These factors include 29 potential factors
and 17 potential benefit attributes. Three main factor groups include (1) UAS system features,
(2) Project Team features, and (3) Project Features that may impact the performance of UAS
safety monitoring system in highway construction projects. Particular UAS systems may
influence the performance of UAS, since they have some specifications like controller, battery
and camera. They need to be considered in order to develop an optimal safety monitoring
system. In addition, implementing such a system on construction projects requires various
conditions, such as interest, training, knowledge, or experience, to be considered. This is one of
the most important considerations to define factors for successful performance of UAS. All
control systems for construction management require different functions, environments, and
benefits on construction projects. The project has varied conditions and features so they should
be considered to implement UAS and measure UAS performance. In this paper, the factor of
project type and owner type are defined as highway construction project and owner as
government agency respectively.

In addition, there are several other measures such as benefits of performance that should be
considered when evaluating the performance of UAS-based safety monitoring systems on
highway construction environments. These are anticipated impacts when the UAS safety
monitoring system is implemented. Finally, a conceptual factor model was developed as
described in Figure 4. Table 4 summarizes the potential factors associated with the three factor
groups and Table 5 describes the potential attributes related to the benefits or performance of
UAS safety monitoring system.

Kim and Irizarry 139

 Figure 4. Conceptual Factor Model

Table 4. Potential Factors Influencing UAS Performance

Factor
Factor Description
Group
1. Easy user interface for UAS operation
2. Obtained quality of visual assets (photo, video)
3. Battery life of the UAS
4. UAS system reliability
5. Maximum Visible Angle of the UAS Camera
6. Autonomous flight capability
UAS
7. Communication range for UAS control
System
8. Equipped Sensors (i.e. GPS, Compass etc.)
Features
(13) 9. Flight Stability
10. Features of UAS camera (i.e. Image resolution, video recording capability,
infrared)
11. Available sensors (i.e. Compass, Gyroscope, Accelerometer, etc.)
12. Emergency response system (Failsafe for lost connection, return to ground
control station, etc.)
13. UAS system's easy upgradability without manufacturer's services
1. Project location (urban vs. rural)
2. Project cost
Project
3. Project size
Features
4. Duration of project
(6)
5. Complexity of the construction tasks
6. Sub contractors trades involved
1. Team’s prior experience with UAS
2. Team attitudes toward UAS in construction Projects
3. Team's prior experience with IT-based safety system
4. Team attitudes toward IT in construction projects
Project 5. User type (safety manager or hire pilot)
Team’s
6. Knowledge of UAS safety monitoring system features
Features
7. Adequacy of resources for UAS use as safety monitoring system (i.e. money,
(10)
time, personnel, etc.)
8. UAS control ability of users (safety managers or project managers)
9. Adequacy of training or education for UAS use as safety monitoring system
10. Clear method to measure the performance of UAS safety monitoring system

Kim and Irizarry 140

Questionnaire Survey Results
The targeted respondents of the survey were professionals who are involved in project or
construction management and safety management (i.e. owners, construction managers,
engineers, or safety managers) in the AECO industry. For three weeks, a total of 15 responses
were collected to explore the factors that may impact on the UAS performance in terms of safety
monitoring on highway construction environments. Of these 15 participants, six have over 20
years of experience (40%), three have worked between 11-20 years (20%), and others have
experience of less than 10 years (40%) in the construction industry. The results also show they
have various work positions in construction companies. There are three project managers
(20%), five safety managers (33.33%), two BIM managers (13.33%), one IT director (6.67%),
one scheduler (6.67%) and others including owner and risk manager (20%). Figure 5 depicts
the demographic information of all respondents.

Table 5. Potential Benefit Attributes

Features Attribute Description

1. Reduce bottlenecks in communication during safety control process
2. Effective management of construction workers’ behaviors
3. Inspection of personal protective equipment use
4. Effective monitoring of the whole work site
5. Effective monitoring of hard to access areas on work sites
6. Effective monitoring of traffic and vehicles on work sites
7. Effective monitoring of heavy or hoisting equipment on work sites
8. Reduce time it takes to perform safety monitoring tasks
9. Makes identification of safety issues easier
Potential
10. Reduces number of construction injuries
Performance
Attributes 11. Reduces cost of safety management system
12. Expedites reaction to potential hazards
13. Improves project safety performance
14. Improves overall project performance
15. Enables immediate reporting of potentially dangerous situations
and allows preventive measures to be applied
16. Simplification of documentation process or reduces waste time for
documentation (reduce hardcopy)
17. Enhances and identification and actions regarding potential
hazards on sites

Figure 5. Demographic Information of Respondents

Kim and Irizarry 141

The second section asked respondents to share their views about UAS on highway construction
projects. All of them already had knowledge of the concept of UAS or drone and how the
machine is operated. Specifically, nine (60%) had experience with UAS for hobby (11.1%),
training (11.1%), or utilizing on the construction project (77.8%). They have operated and been
familiar with UAS for less than two years. Figure 6 summarizes that the respondents’ UAS
experiences. 13 out of 15 respondents (80%) had positive opinions about the implications of
UAS safety monitoring system on the highway construction project.

Almost all respondents (80%) had positive expectations of the use of a UAS safety monitoring
system on highway construction environments. They mainly agreed with that a UAS would be a
good fit for safety monitoring on highway construction sites since these are more horizontal and
open sites, and they can extend for miles. In addition, they suggested and recommended that
USAs can access areas humans sometimes cannot directly access, can monitor work flow,
jobsite logistics and material stocking on the jobsite, and have the capability to reduce the costs
safety monitoring. On the other hands, only two (20%) were concerned that the FAA will not
allow legally the use of UAS on the construction site and the battery life of UAS would limit the
effectiveness of UAS for safety monitoring tasks.

Figure 6. UAS Experience of Respondents

Since 40% of respondents had no experience with this tool, this could affect on the reliability of
the result of this analysis. Therefore, this paper has only an exploratory analysis and introduces
preliminary results. Continued research will include field flight test with a small UAS platform
where project personnel will participate in the test and then be asked to provide their opinion
about the factors and performance of UAS’s applications on highway construction projects.

Exploratory Analysis of Factors

Factors that may influence the performance of UASs are derived from the extensive literature
review and the interviews with industry professionals. The last section of the questionnaire
survey asked the respondents to indicate whether they thought the factors would affect the

Kim and Irizarry 142

performance of a UAS-based safety monitoring system on highway construction sites.
Regarding the factors influencing the performance of UASs, all of respondents agreed that the
seven factors of (1) ease of user interface for UAS operation, (2) quality of visual assets, (3)
UAS system reliability, (4) features of UAS camera, (5) team’s prior experience with UAS, (6)
adequacy of resources for UAS use as safety monitoring system, and (7) adequacy of training
or education for UAS use as safety monitoring system should be considered as influencing
factors on the performance of UAS applications in the case of safety monitoring system on
highway construction environments.

However, more than half of respondents did not know or disagreed that two factors, such as (1)
UAS system's easy upgradability without manufacturer's services and (2) sub contractors trades
involved contribute to the performance of the UAS safety monitoring system on highway
construction projects. In addition, one-third of the professionals did not know if duration of
project, complexity of the construction tasks and user type (safety manager or hire pilot) were
related to the considerations for UAS’s performance for safety monitoring tasks on the jobsites
or not. Since the number of response for this survey is small, these factors must be considered
after collecting more data from construction and UAS practitioners. Table 6 provides details on
the magnitude of the impact of the factors based on indications of responses collected from the
respondents.

Regarding the benefits and performance attributes, 14 out of 15 personnel agreed that effective
monitoring of the whole work areas could be achieved with a UAS safety monitoring system on
highway construction projects. The apparent benefits or performance measures are (1) Effective
monitoring of traffic and vehicles on sites, (2) Effective monitoring of heavy or hoisting
equipment on sites, (3) Enhanced identification of actions related to potential hazards on sites
with 87% of respondent in agreement. One explanation is that use of small UAS platform in
highway construction sites can be a potential safety control system and contribute to the
effective monitoring tasks on large worksites. However, the time it takes to perform safety
monitoring and simplifying the documentation process or reducing wasted time for
documentation could not be considered as benefits from utilizing a small UAS. In addition, they
were doubtful that a UAS monitoring system can improve overall project performance or can
reduce number of construction accidents on highway construction worksites. Table 7 illustrates
more details about UAS benefits or performance measures.

A total of 29 potential factors that may influence the performance of UAS monitoring system,
particularly, the case of safety condition monitoring on highway construction environments was
developed. In addition, 17 expected benefits, can be used for evaluating performance of UAS
safety monitoring system on jobsites, are also derived in this paper. Even though a limitation
was revealed during data collection, this paper presents two main preliminary results: First, UAS
system characteristics and inner team’s characteristics can certainly influence the UAS
performance for safety monitoring tasks on highway construction projects. That is why there
were not a lot of real implications and implementations of UAS on real construction sites so the
professionals mostly considered technical factors affecting UAS performance.

Second, the potential benefits from UAS applications on highway construction projects or the
performance measure attributes could not be well focused. It was difficult for many survey
participants to indicate definite answers. For this reason, field tests should be conducted with a
UAS platform on highway construction projects, because the construction project personnel did
not have experiences with UAS for safety monitoring on the jobsite. Through those tests,
additional survey participants who have experience with UAS on highway construction projects
could be included as well as improving the reliability of the collected data for futures studies.

Kim and Irizarry 143

Table 6. Influence of Factor
Results*
Factor
Factor Description Do Not
Group Yes No
Know
1. Easy user interface for UAS operation 1.00 - -
2. Obtained quality of visual assets (photo, video) 1.00 - -
3. Battery life of the UAS 0.80 0.07 0.13
4. UAS system reliability 1.00 - -
5. Maximum Visible Angle of the UAS Camera 0.80 - -
6. Autonomous flight capability 0.80 0.07 0.13
7. Communication range for UAS control 0.87 - 0.20
UAS 8. Equipped Sensors (i.e. GPS, Compass etc.) 0.93 - 0.07
System 9. Flight Stability 0.93 - 0.07
Features 10. Features of UAS camera (i.e. Image resolution, video
1.00 - -
(13) recording capability, infrared)
11. Available sensors (i.e. Compass, Gyroscope,
0.87 - 0.13
Accelerometer, etc.)
12. Emergency response system (Failsafe for lost
0.87 - 0.13
connection, return to ground control station, etc.)
13. UAS system's easy upgradability without
0.47 - 0.53
manufacturer's services
Average: 0.87 0.01 0.12
1. Project location (urban vs. rural) 0.80 0.20 -
2. Project cost 0.80 0.20 -
Project 3. Project size 0.87 0.13 -
Features 4. Duration of project 0.67 0.33 -
(6) 5. Complexity of the construction tasks 0.67 0.33 -
6. Sub contractors trades involved 0.47 0.53 -
Average: 0.71 0.29 -
1. Team’s prior experience with UAS 1.00 - -
2. Team attitudes toward UAS in construction Projects 0.93 0.07 -
3. Team's prior experience with IT-based safety system 0.80 0.13 -
4. Team attitudes toward IT in construction projects 0.73 0.20 -
5. User type (safety manager or hire pilot) 0.67 0.20 0.13
6. Knowledge of UAS safety monitoring system features 0.87 0.07 0.07
Project 7. Adequacy of resources for UAS use as safety
Team’s 1.00 - -
monitoring system (i.e. money, time, personnel, etc.)
Features 8. UAS control ability of users (safety managers or
(10) 0.87 0.07 0.07
project managers)
9. Adequacy of training or education for UAS use as
1.00 - -
safety monitoring system
10. Clear method to measure the performance of UAS
0.87 0.07 0.07
safety monitoring system
Average: 0.87 0.08 0.05
*Converted percentage (%) to Decimal Scale

Kim and Irizarry 144

Table 7. Analysis on Potential Benefits and Performance Attributions
Results*
Descriptions Do Not
Yes No
Know
1. Reduce bottlenecks in communication during safety control
0.60 0.20 0.20
process
2. Effective management of construction workers’ behaviors 0.53 0.07 0.40
3. Inspection of personal protective equipment use 0.73 0.27 -
4. Effective monitoring of the whole work site 0.93 0.07 -
5. Effective monitoring of hard to access areas on work sites 0.80 0.20 -
6. Effective monitoring of traffic and vehicles on work sites 0.87 0.07 0.07
7. Effective monitoring of heavy or hoisting equipment on work sites 0.87 0.07 0.07
8. Reduce time it takes to perform safety monitoring tasks 0.60 0.33 0.07
9. Makes identification of safety issues easier 0.73 0.07 0.20
10. Reduces number of construction injuries 0.60 0.07 0.33
11. Reduces cost of safety management system 0.53 0.20 0.27
12. Expedites reaction to potential hazards 0.67 0.13 0.20
13. Improves project safety performance 0.80 0.07 0.13
14. Improves overall project performance 0.53 0.13 0.33
15. Enables immediate reporting of potentially dangerous situations
0.60 0.20 0.20
and allows preventive measures to be applied
16. Simplification of documentation process or reduces waste time
0.47 0.33 0.20
for documentation (reduce hardcopy)
17. Enhances and identification and actions regarding potential
0.87 0.07 0.07
hazards on sites
Average: 0.69 0.15 0.16
*Converted percentage (%) to Decimal-scale

CONCLUSIONS
The application of UAS in the construction industry is relatively new yet the technology can bring
many benefits to industry practitioners. A UAS can be utilized as a safety monitoring system on
highway construction projects when equipped with high-resolution cameras and an array of
sensors including a Global Positioning System (GPS) and Gyroscope. Highway construction
environments have a big advantage to fly small UAS around the jobsite since that project can
extend long distances without vertical obstructs. Nevertheless, there is doubt of the
effectiveness of UAS safety monitoring system implementation on highway projects because the
factors influencing their performance are not fully understood. Since most practitioners are still
uncertain about the benefits or performance of UAS, measuring performance and improvement
of UAS implementation is very difficult or impossible at the present time.

This exploratory study has discussed the results of a questionnaire to obtain professional ‘s
perceptions about UAS monitoring system of safety conditions on highway construction
environments. The perceptions of 15 industry professionals working in the United States were
collected to analyze the critical factors and performance attributes of the UAS system. As a
result, 29 factors that may influence the performance of UAS safety monitoring and 17 attributes
to evaluate the performance were experimentally analyzed as the initial step of this UAS
research in the AECO industry. The analysis resulted in two main research findings. First, the

Kim and Irizarry 145

practitioners were only familiar with the technical elements of UAS or they just imagined or
expected the relationship between their team characteristics and the UAS implementation on
the project. This is why they almost did not have many experiences with UAS monitoring system
on the construction environment. Second, the project personnel did not know about the
performance attributes and potential benefits from UAS system well. They had difficulty in
exactly indicating whether the described attributes can be benefits, or if they should be
considered in evaluating the performance of UAS. Low sample size in the survey is a limitation
to be considered.

This exploratory study contributes to establish the initial group of factors that may influence the
performance of a UAS-based safety monitoring system in highway construction environments.
In addition, as the beginnings of understanding UAS applications in the construction industry,
the potential benefits of the system and performance measures were identified. It is definitely an
important step to increase UAS implementation as well as clearly a guide the development
UAS-based system strategies in the AECO industry.

Kim and Irizarry 146

REFERENCES

BLS. (2015). “Revisions to the 2013 Census of Fatal Occupational Injuries (CFOI) counts.” BLS,
(August).

D’Oleire-Oltmanns, S., Marzolff, I., Peter, K. D., and Ries, J. B. (2012). “Unmanned aerial
vehicle (UAV) for monitoring soil erosion in Morocco.” Remote Sensing, 4(11), 3390–
3416.

Eschmann, C., Kuo, C., and Boller, C. (2012). “Unmanned Aircraft Systems for Remote Building
Inspection and Monitoring.” Proceedings of the 6th European Workshop on Structural
Health Monitoring, July 3-6, 2012, Dresden, Germany, 1–8.

FAA. (2015, MAY 10A). Public Operations - Certificate of Waiver of Authorization (COA)

FAA. (2015, MAY 10B). Freedom of Information Act Responses

FAA. (2015, MAY 10C). Authorizations Granted Via Section 333 Exemptions.

Hinkley, E. A., and Zajkowski, T. (2011). “USDA forest service–NASA: unmanned aerial
systems demonstrations–pushing the leading edge in fire mapping.” Geocarto
International, 26(2), 103–111.

Irizarry, J., Gheisari, M., and Walker, B. N. (2012). “Usability assessment of drone technology
as safety inspection tools.” Electronic Journal of Information Technology in Construction,
17(September), 194–212.

Ismail, Z., Doostdar, S., and Harun, Z. (2012). “Factors influencing the implementation of a
safety management system for construction sites.” Safety Science, Elsevier Ltd, 50(3),
418–423.

Jenkins, D., and Vasigh, B. (2013). “The economic impact of unmanned aircraft systems
integration in the United States.” (March), 1–40.

Lee, U.-K., Kim, J.-H., Cho, H., and Kang, K.-I. (2009). “Development of a mobile safety
monitoring system for construction sites.” Automation in Construction, Elsevier B.V.,
18(3), 258–264.

Leung, S. W., Mak, S., and Lee, B. L. P. (2008). “Using a real-time integrated communication
system to monitor the progress and quality of construction works.” Automation in
Construction, 17(6), 749–757.

Lin, P., Li, Q., Fan, Q., and Gao, X. (2013). “Real-time monitoring system for workers’ behaviour
analysis on a large-dam construction site.” International Journal of Distributed Sensor
Networks, 2013.

Morgenthal, G., and Hallermann, N. (2014). “Quality assessment of Unmanned Aerial Vehicle
(UAV) based visual inspection of structures.” Advances in Structural Engineering, 17(3),
289–302.

Kim and Irizarry 147

Naticchia, B., Vaccarini, M., and Carbonari, A. (2013). “A monitoring system for real-time
interference control on large construction sites.” Automation in Construction, Elsevier
B.V., 29, 148–160.

Newcome, L. R. (2004). Unmanned aviation: a brief history of unmanned aerial vehicles. Aiaa.

Nisser, T., and Westin, C. (2006). “Human factors challenges in unmanned aerial vehicles
(uavs): A literature review.” School of Aviation of the Lund University, Ljungbyhed.

Nitithamyong, P., and Skibniewski, M. J. (2006). “Success/Failure Factors and Performance

Measures of Web-Based Construction Project Management Systems: Professionals’
Viewpoint.” Journal of Construction Engineering and Management, 132(1), 80–87.

Reuder, J., Brisset, P., Jonassen, M., Müller, M., and Mayer, S. (2009). “The Small Unmanned
Meteorological Observer SUMO: A new tool for atmospheric boundary layer research.”
Meteorologische Zeitschrift, 18(2), 141–147.

Rinaudo, Chiabrando, Lingua, and S. (2012). “Politecnico di Torino Porto Institutional

Repository.”

Saleri, R., Cappellini, V., Nony, N., Pierrot-Deseilligny, M., Bardiere, E., Campi, M., and De
Luca, L. (2013). “UAV photogrammetry for archaeological survey: The Theaters area of
Pompeii.” Digital Heritage International Congress (DigitalHeritage), 2013, 497–502.

Sarhan, S., and Fox, A. (2013). “Performance measurement in the UK construction industry and
its role in supporting the application of lean construction concepts.” Australasian Journal
of Construction Economics and Building, 13, 23–35.

Sunkara, P. (2005). “A Tablet PC Application for Construction Site Safety Inspection and
Fatality Prevention.” Louisiana State University.

Toole, T. M. (2002). “Construction Site Safety Roles.” Journal of Construction Engineering and
Management, 128(3), 203–210.

Zhou, Z., Irizarry, J., and Li, Q. (2013). “Applying advanced technology to improve safety
management in the construction industry: a literature review.” Construction Management
& Economics, 31(6), 606–622.

Kim and Irizarry 148

The Current State of 3D Printing for Use in
Construction

Megan A. Kreiger
Construction Engineering Research Laboratory
U.S. Army Engineer R&D Center
2902 Newmark Dr
Champaign, IL 61822
[email protected]

Bruce A. MacAllister
Construction Engineering Research Laboratory
U.S. Army Engineer R&D Center
2902 Newmark Dr
Champaign, IL 61822
[email protected]

Juliana M. Wilhoit
Construction Engineering Research Laboratory
U.S. Army Engineer R&D Center
2902 Newmark Dr
Champaign, IL 61822
[email protected]

Michael P. Case
Construction Engineering Research Laboratory
U.S. Army Engineer R&D Center
2902 Newmark Dr
Champaign, IL 61822
[email protected]

ABSTRACT
Over the past few years, 3D printing has become a household term, but in the near future this
term may be used to describe method of construction of the house that you’re sitting in. There
are several groups working on developing printers capable of 3D printing building structures.
Researchers, corporations, and makers are working and experimenting with extruding foam,
mortar, concrete, and other materials to print things such as huts, castles, apartments, and
Army structures. Part of this effort is being done by the Engineer Research and Development
Center – Construction Engineering Research Laboratory (ERDC-CERL) in collaboration with
NASA, to create structures for U.S. Army contingency bases using contour crafting. This paper
provides an overview of additive manufacturing techniques applied to the construction of
buildings and building components using extrudable construction materials and discusses the
motivation for research in this field by ERDC-CERL. The use of 3D printing in construction of
structures has the potential to change construction methods and the buildings that surround us
in terms of labor requirements, logistics, energy performance, and speed.

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
Work done by the U.S. Government

Key words: 3D printing—additive manufacturing—large-scale—buildings

INTRODUCTION
Advances in building construction are a result of experience and education of accountable
parties in the industry, which is preceded by a large body of research on the subject.
Uncertainties that arise from the acceptance of alternative practices as well as the
interdependence on professional knowledge, code regulations, and interdisciplinary
relationships leads to complexities that often result in slow adoption of new technologies
(Dubois and Gadde 2001; Khoshnevis 2004; Balaguer and Abderrahim 2008; Blayse and
Manley 2015). Much of the focus in construction advancement in past decades has been placed
on energy efficiency, fostered by related programs such as Energy Star (Energy Star 2013),
LEED (U.S. Green Building Council 2015), and the development of international regulations
(International Code Council 2014; International Code Council 2015). Projects have
conventionally maintained the application of industry accepted construction methods associated
with the use of common building materials (e.g. Concrete Masonry Units, Cast-in-place
concrete, wood, and steel). However, with greater access to technology, research in automated
construction processes has become increasingly popular.

Masonry construction dominates military building projects domestically and internationally. The
costs associated with traditional construction techniques are ever increasing, in large part due to
the high price of transporting materials to job sites and increases in labor costs. Worker
productivity drives both the cost and time to completion of structures built using concrete
masonry units (CMU). For example, consider a small structure measuring 16’ wide, 32’ long and
8’ tall with two doorways built using CMU. Assuming standard grouting requirements and that a
mason with the assistance of one laborer lays 200 CMU blocks per work day using 8”x8”x16”
blocks, the outside walls of the structure could be completed in approximately two days (Scott
2009). The outer walls of a wood framed structure of this size would take at least 1 day to
complete, with a team consisting of three carpenters and one laborer in order to lift heavy
sections without additional equipment (Freund 2004; “Cost to Frame Wall - Zip Code 47474”
2015). A concrete structure generally takes at least 5 days to cure before any construction or
service loads can be applied, in addition to forming, rebar placement, pouring, form removal,
and other requirements (Simmons 2011). These estimates include only the exterior wall
structure construction time and do not include any time for framing the doorway, provisions for
the roof, reinforcement, internal walls of the structure, exterior cladding or finishing. While this
may be acceptable in many cases, there are situations where this process is too slow and
costly.

The U.S. Army has unique requirements when constructing contingency bases in remote and
potentially hostile environments. These requirements include minimizing the number of Soldiers
or civilian contractors required to be on-site, logistics for shipping of construction materials
(typically lumber) and energy-related fuel, and time to occupancy. As the U.S. Army reduces the
size of the force, fewer personnel required for construction frees up Soldier resources from the
construction itself or security duties. Today’s semi-permanent contingency bases typically use
lumber-based construction, with materials often purchased and shipped over large distances.
Concrete is locally available in much of the world and a capability to use cementitious or other
indigenous materials without a need for form work would substantially reduce logistics for
construction materials. In addition to construction materials, tents and wooden structures in
contingency bases do not offer the high levels of energy efficiency found in today’s domestic
and commercial markets. Energy to heat, cool, and ventilate these structures must be

MA Kreiger, BA MacAllister, JM Wilhoit, MP Case. 150

Work done by the U.S. Government

transported in the form of fuel (usually JP-8), resulting in high costs, increased convoy
requirements and higher exposure of personnel to attack. Well-built structures would mitigate
this risk by decreasing energy requirements.

3D PRINTING
3D printing provides a potential solution to U.S. Army contingency base requirements. Although
many people may associate 3D printing with plastic trinkets, biological applications, or even
printing automobiles, it is capable of much more. As is common with regards to construction
from sky scrapers to sports stadiums, two words often apply, “think larger.” The use of large-
scale 3D printing is often overlooked, as it is a relatively new application of this technology. In
the next few years this term may be associated with the buildings around us. Researchers,
corporations, and hobbyists are working and experimenting with extruding foam, mortar,
concrete, and other materials to print building structures, such as, huts, a castle, apartments,
and Army bases or shelters.

While 3D printing or additive manufacturing (AM) technology was developed in the 1980s, the
field has expanded dramatically over the past 5 years. It has had an impact on many fields and
applications in ways that were previously viewed as too costly or impossible using traditional
manufacturing methods.

3D printing is generally used to reduce material requirements, create custom objects, reduce
labor, and for rapid prototyping. 3D printing of building structures also shares these focuses,
with potential to reduce the amount of materials needed to be shipped to the build site, create
unique buildings, reduce the number of laborers required, and provide a capability to print the
buildings in a relatively short amount of time.

The various groups working on 3D printing buildings have different goals. Some are focused on
a more experimental setup, others on creating a new form of technology by making an efficient
system, and others are focused on looking for practical solutions to building construction.
Different materials are used in each project. Some methods focus on the creation of molds for a
building by creating a cavity to fill with an alternate material, whereas others focus on directly
printing the structure with the print materials. There are also a number of projects using small-
scale printers and objects (e.g. building blocks) to build large structures, but relatively few which
attempt to build large sections or the entire building at one time (Williams 2015). There are
multiple research projects being done by groups all over the world in an effort to effectively 3D
print large-scale structures. This paper reviews and summarizes this work in the next section.

The Engineer Research and Development Center (ERDC) is investigating the use of this
method for the construction of contingency base shelters. In cooperation with the University of
Southern California and NASA, researchers have constructed a number of prototypes that use
3D printing technology to test robotic construction methods using locally sourced concrete
materials. ERDC is currently working towards an on-site construction concrete 3D printer that
will be able to print a 16’x32’x8’ structure within 24 hours. A conceptual representation of the
equipment and process is shown in Figure 1.

MA Kreiger, BA MacAllister, JM Wilhoit, MP Case. 151

Work done by the U.S. Government

Figure 1. Concept representation of ACES 3D printer. (ERDC 2015)

CURRENT RESEARCH

Many different methods are implored by AM technologies to construct large-scale structures

including fused-filament fabrication (FFF), powder-bed and inkjet head, and extrusion printing.
While many methods exist for small-scale printers, only a few are relevant to the construction
industry and are described below.

Fused-Filament Fabrication
Fused filament fabrication (FFF), also known by other more proprietary terms, is the most
readily available type of 3D printer in households and businesses due to its maker (a community
dedicated to fabricating objects) and open-source following. It uses a filament-type material
which is heated through an extruder and deposited as a string of material on the build platform.
After the first layer is printed, the printer then repeats the process, building the next layer on the
previous one. An example of this type of deposition process is shown in Figure 2.

Figure 2. A small-scale fused-filament fabrication 3D printer

3D Print Canal House

A factory gantry-based FFF project was started by DUS Architects and partners in Amsterdam.
The project known as the 3D Print Canal House takes the common 3D printer to the next level.
The Kamermaker (Roombuilder) 3D printer builds large polymer-based components, up to 2m x

MA Kreiger, BA MacAllister, JM Wilhoit, MP Case. 152

Work done by the U.S. Government

2m x 3.5m, out of polymer beads instead of the traditional spools. The components are then
trucked to the site or assembled on-site. They are experimenting with multiple materials, but are
focused mainly on polymer-based structures. They are also experimenting with filling their
structure with lightweight foaming eco-concrete to provide structural and insulation properties.
The project is expected to be completed in 2017 (DUS Architects 2015). The methods is limited
to mostly polymers and materials that solidify quickly upon cooling, the major disadvantages of
this method are the material limitations.

Powder-bed and Inkjet Head Printing

Powder-bed printing, also known as binder jetting, is a printer that initially spreads a thin layer of
powder (e.g. sand) across the build area, then immediately dispenses a binder in the areas
where the structure is desired, see Figure 3. There are other similar powder-bed processes (e.g.
laser sintering) which will not be discussed in this paper. After each layer is complete, it spreads
out another thin layer of powder and repeats the process. This process repeats until there is a
complete structure. Once the structure has finished, the binder must be allowed to cure and
then the excess material is removed. This method can print objects that would be nearly
impossible using most other 3D printing methods or traditional construction, because of the
support added by the unbound powder material.

Figure 3. Basic representation of powder-bed and inkjet printing. Gray depicts unbound
powder and blue is the binder combined with the powder. The sand spreading apparatus
is not shown.

D-Shape
D-shape is a factory gantry-based powder-bed 3D printer built in the United Kingdom by
Monolite UK Ltd. This system can print up to 6m x 6m x 6m of architectural structures and
components at a time, which are then trucked to the site. It uses granular material deposition
using a powder bed and inkjet method composed of sand and a binder to create structures. It
works by putting down a full layer of sand, then going over the sand with the print heads while
extruding a binder where the structure is desired. The layers are between 5-10mm. The
structure itself takes about 24 hours to solidify, after which the excess sand must be removed to
show the sandstone-like or marble-like structure underneath. Originally using epoxy or
polyurethane, it now uses an organic binder to make an environmentally friendly structure. The
goals of this project are to increase quality and safety, while decreasing time and costs.

D-shape won top prize in the Change the Course - New York City Waterfront Construction
Competition for innovative construction ideas for waterfront infrastructure ran by the New York

MA Kreiger, BA MacAllister, JM Wilhoit, MP Case. 153

Work done by the U.S. Government

City Economic Development Corporation (NYCEDC) in 2013. (D-Shape 2015; New York City
Economic Development Corporation 2013) One example of the projects D-shape is working on
is the Landscape house. The project is going to be printed in sections, filled with fiber-reinforced
concrete, and put into place. (Crook 2013; EeStairs 2015).

While the method does provide greater freedom for architectural design, a drawback to this
technique is that it requires a large amount of material to work with, as the sand or other powder
has to be built up to the top of the print in the entire build area. In turn the powder which
supports the structure during the curing time requires a labor intensive period of excess material
removal before the print can be used.

Extrusion Printing
The extrusion print method is very similar to FFF, but uses a variety of slurry-like or clay-like
materials to create a structure. This type of 3D printer extrudes a fluid-like material out of a
nozzle and builds layer upon layer. The difference is that because the materials used are not
based on a cooling process for solidification, the cure time is longer than in FFF. This means
that there may be a wait time between layers or after a certain number of layers before
proceeding with the print. It also limits the geometry of the printed structure due to the fluid-like
behavior and the freshly printed state dependent on the material used for the print.

Factory delta-based clay printer:

World’s Advanced Saving Project

The World’s Advanced Saving Project (WASP) started in 2012, uses clay and fluid-dense
materials to print moderately sized structures (World’s Advanced Saving Project 2015). Able to
print in materials such as ceramic and porcelain, the goal of the project is clay houses. The
creations from this group have been known as “mud huts” and utilize local materials to provide
long lasting low-impact structure.

On-site boom arm foam printer:

MIT (Digital Construction platform)

Massachusetts Institute of Technology (MIT) has been developing a truck-mounted boom,
“Digital Construction Platform,” that can be used as a 3D printer. On the boom, there is an
industrial arm with a fast response time, which compensates for the long period oscillations of
the boom. The combined reach of the boom and robotic arm is 80’. The current model can be
used for additive or subtractive manufacturing, by extruding with foam or using an attached
milling head to shape, respectively. This concept is similar to the commonly used method of
insulated concrete forms (ICF), where a foam form is setup and filled with concrete. This system
is complex, but provides multiple benefits including easy transportation, little to no setup time, a
greater freedom of movement, and the potential to continue a print beyond the immediate print
area by repositioning.

On-site swarm concrete printer:

IAAC Minibuilders
A research team at the Institute for Advanced Architecture of Catalonia (IAAC) has created a
unique 3D printing system of 3 types of small robots (foundation, grip, vacuum). Each robot has

MA Kreiger, BA MacAllister, JM Wilhoit, MP Case. 154

Work done by the U.S. Government

a different function, contributing to the creation of a large structure made out of concrete (“Small
Robots Printing Big Structures” 2015). The first robot is a foundation robot, which follows a line
to print the first 20 layers. The grip robot clamps onto the existing material and continues to print
the structure. These robots are capable of printing ceilings and lintels by printing horizontally.
After the structure is in place, vacuum robots are placed on the structure and print extra shells.
This system is capable of very flexible shapes and is not limited by the size of the machine. A
disadvantage of this system is that the building envelope is more limited in complexity and
thickness.

Factory gantry-based concrete printer:

Freeform Project
At the Loughborough University in Leicestershire, U.K., a team in the Civil and Building
Engineering department has been developing a factory gantry-based concrete printer with
multiple industrial partners (Buswell and Austin 2015). This printer creates components by
extruding concrete through a single nozzle, which is used in conjunction with a softer material to
support overhangs and is later removed. The use of a foam-based support material in concrete
printers provides the freedom of design that is typically associated with powder-bed print
methods, while limiting the support to needed areas to reduce waste.

Winsun Co.
In China, Winsun has created a factory gantry-based concrete 3D printer for building
components (Winsun 2015). The printer measures 150m long x 10m wide x 6.6m tall, uses
concrete extrusion to print buildings in sections, then ships the pieces by truck, which are
assembled on-site with a crane. The pieces are generally printed such that the roof is one solid
piece with the walls and needs to be tilted up 90 degrees during the build for proper orientation.
They have built at least 10 houses and are creating many types of smaller structures as well.
Win Sun has created a couple of ‘firsts’ in the 3D printing construction world: the first full-sized
printed structure with a roof and the world’s tallest 3D-printed building at 5-stories high, both
from components. This follows a more traditional approach to building construction, where the
pieces are shipped to the site and assembled. This method still potentially reduces waste and
time, but puts the burden on the shipment of materials to the site.

On-site gantry-based concrete printer:

Backyard Castle
Andrey Rudenko and his collaborators have designed and built a gantry-based 3D printer in his
backyard. This printer is mounted on rails at the build site and is capable of printing concrete
structures up to 12’ tall with each layer of concrete measuring 10mm height and 30mm width.
He printed a castle out of concrete with a building footprint of 3m x 5m (Krassenstein 2014;
Rudenko 2015). This method utilizes on-site construction, but still requires the use of printed
components. The use of on-site construction would be beneficial to construction areas that are
difficult to access.

USC
One of the earliest groups to work on 3D printing of structures was the University of Southern
California (USC), led by Behrokh Khoshnevis. Funded in part by the ERDC-CERL, they have
been working on automated construction by contour crafting (CC) for over 10 years (Khoshnevis

MA Kreiger, BA MacAllister, JM Wilhoit, MP Case. 155

Work done by the U.S. Government

2004; Kamrani and Nasr 2006). Their system is an on-site gantry which is mounted on rails at
the build site. It currently has arguably the most advanced shaping and material handling
system. The extruder is a multi-nozzle print head with a trowel to smooth the layers while
printing. The nozzle itself is 3D printed and has an interruptible material delivery system so it
can stop when needed for short amounts of time. USC was funded by NASA to investigate
using the contour crafting technology to print extraterrestrial structures using lunar material
(Khoshnevis 2012). The contour crafting technology has a large amount of research behind it
and is potentially the most feasible for on-site construction. The multiple print heads speed up
print time while still allowing complicated wall cavity geometry. This technique does come with
some limitations as the design is limited to the capabilities of the multi-nozzle print head.

SHORTCOMINGS
There is a movement toward using 3D printing technology for the construction of building
structures, but there are limitations that must be overcome before widespread use is possible.
The current 3D printed building structures are experimental, as further characterization of print
materials, clarification of construction practices and printing processes, and integration into
current building code regulations is required.

Extrusion printing methods show the most promise for acceptance into the construction industry,
as the materials (i.e. clay and concrete) and method of material placement (i.e. concrete
pumping) are similar to those used currently in the field. With over 10 years of experience in
concrete 3D printing, the work at USC provides a first stop for the current body of extrusion
printing. Other technologies, while not as developed, show much promise. The Digital
Construction Platform at MIT is still relatively early in its development, but shows that a simple
modification to current construction technology, such as a boom crane, will allow for large print
areas. It is unclear at this time whether the technology will support concrete printing or is limited
to lightweight materials such as foam. The IAAC Minibuilders provide for flexibility in design and
construction, but may require a change in skills and knowledge (e.g. robotic controls) that is not
as readily available in on a typical construction site.

The methods implored by Win Sun, the Freeform project, and similar may be more readily
accepted due to its similarity to the current precast concrete industry, requiring very little change
for the construction industry as the house is built in sections using cranes. This method does
require transportation of large concrete sections and the use of large machinery to lift them into
place.

Other methods, such as, fused filament fabrication and powder bed printers are more familiar to
those in the AM community, but may be slow to acceptance in the construction industry. The
FFF method used in the 3D Print Canal House project is set to be completed in 2017, however
the technology relies on quick cooling after an initial heating, which limits the materials to
polymers and other rapid cooling materials. The powder bed printer by D-shape provides
architectural freedom, but requires large amounts of powder material to support the print during
setting, leading to material waste after curing despite the ability to recycle leftover powder.

The 3D printing industry still continues to be a novelty to the general public and applications
may still be too new to be generally accepted. A great deal of work needs to be done to get this
technology ready for use in construction, but when it is ready, it has potential to change the
structures around us in ways never previously thought possible. 3D printing may lead to a world
where construction and architectural design are not limited by conventional methods and where
a unique design is merely printed under the same conditions as a basic structure would be.

MA Kreiger, BA MacAllister, JM Wilhoit, MP Case. 156

Work done by the U.S. Government

FUTURE DIRECTION
Still in its infancy, the 3D printing of buildings is a growing field that still requires much research
before the technology will become a viable option for the everyday job site. Further knowledge
is required in the areas of materials testing and characterization, structural design requirements,
the incorporation into building codes and standards, and the standardization of construction
practices. Concrete materials should be characterized based on typical mix designs, required
particle size based on print head diameter, rheology, and the use of indigenous materials. In an
industry where uncertainties exist there is also a need for material testing standards. On the
structural side, the use of reinforcement will be a major concern, whether it is discontinuous
fibers, fiber meshes, or the more familiar welded wire and reinforcing bars with grouting.
Another structural challenge is the printing of roofs or over voids (e.g. windows, doorways,
mechanical openings), and how the voids are reinforced. On the construction side, research
should be focused on general practices for applying the technology on-site (e.g. site grading
requirements, construction requirements, environmental considerations, scheduling and cost
considerations). Furthermore, the incorporation of the technology in building codes and
standards should be the final step in acceptance.

CONCLUSION
There is a growing effort to be able to construct structures using 3D printing technology. Many
projects are able to create small structures or components of structures currently, but more
needs to be done regarding material behavior and applicability for building structures. Building
codes will need to take into account this new technology before it can be readily used. As
technology advances and research is done in the area, this field will continue to evolve and 3D
printed buildings may be a possibility over the next few years.

ACKNOWLEDGMENTS
Special thanks to Eric Kreiger at Bacon Farmer Workman Engineering and Testing, Inc. for
contributions to this paper. Thanks also to Jason Galtieri, and W. Jacob Wagner at ERDC-
CERL for conversations and input on this paper. The authors would also like to thank Dr.
Joshua Pearce at Michigan Technological University for conversations on 3D printing topics.

REFERENCES
Balaguer, Carlos, and Mohamed Abderrahim. 2008. Trends in Robotics and Automation in
Construction. INTECH Open Access Publisher. http://cdn.intechopen.com/pdfs/5555.pdf.
Blayse, Aletha M., and Karen Manley. 2015. “Key Influences on Construction Innovation.”
Construction Innovation 4 (3): 143–54. Accessed May 18.
Buswell, Richard, and Simon Austin. 2015. “Freeform Construction: Partners.” Loughborough
University. http://www.freeformconstruction.com/partners.php.
“Cost to Frame Wall - Zip Code 47474.” 2015. Homewyse.
http://www.homewyse.com/services/cost_to_frame_wall.html.
International Code Council. 2014. 2015 International Energy Conservation Code. ICC.
Crook, Jordan. 2013. “The World’s First 3D-Printed Building Will Arrive In 2014 (And It Looks
Awesome).” Blog. TechCrunch. January 20. http://techcrunch.com/2013/01/20/the-
worlds-first-3d-printed-building-will-arrive-in-2014-and-it-looks-awesome/.
D-Shape. 2015. “The Technology.” Monolite UK. http://www.d-shape.com/tecnologia.htm.
Dubois, Anna, and Lars-Erik Gadde. 2001. “The Construction Industry as a Loosely Coupled
System - Implications for Productivity and Innovativity.” In . Oslo, Norway.
http://www.impgroup.org/uploads/papers/169.pdf.

MA Kreiger, BA MacAllister, JM Wilhoit, MP Case. 157

Work done by the U.S. Government

DUS Architects. 2015. “3D Print Canal House.” Accessed May 18.
http://3dprintcanalhouse.com/.
EeStairs. 2015. “EeStairs Founding Father of the Landscape House.”
http://www.eestairs.com/en/743_eestairs_founding_father_of_the_landscape_house.htm
.
Energy Star. 2013. “ENERGY STAR Certified Homes, Version 3 (Rev. 07) National Program
Requirments.”
http://www.energystar.gov/ia/partners/bldrs_lenders_raters/downloads/National_Progra
m_Requirements.pdf.
Freund, James F. 2004. “How to Estimate the Cost of Rough Carpentry Framing.” American
Society of Professional Estimators.
http://www.aspenational.org/userfiles/file/Technical%20Papers/2007/TechPaper_May20
07.pdf.
International Code Council. 2015. “Overview of the IgCC.” Accessed May 18.
http://www.iccsafe.org/codes-tech-support/codes/2015-i-codes/igcc/.
Kamrani, Ali, and Emad Nasr. 2006. Rapid Prototyping: Theory and Practice. Springer.
Khoshnevis, Behrokh. 2004. “Automated Construction by Contour Crafting—related Robotics
and Information Technologies.” Automation in Construction, The best of ISARC 2002, 13
(1): 5–19. doi:10.1016/j.autcon.2003.08.012.
Khoshnevis, Behrokh, A. Carlson, N. Leach, M. Thangavelu. 2012. "Contour Crafting Simulation
Plan for Lunar Settlement Infrastructure Buildup." Earth and Space 2012: 1458-1467.
doi:10.1061/9780784412190.155.
Krassenstein, Eddie. 2014. “Architect Plans to 3D Print a 2-Story Home in Minnesota Using a
Homemade Cement Printer.” 3DPrint.com. April 22. http://3dprint.com/2471/3d-printed-
home-in-minnesota/.
New York City Economic Development Corporation. 2013. “NYCEDC Announces Three
Winners of Change the Course - The NYC Waterfront Construction Competition.”
NYCEDC. April 10. http://www.nycedc.com/press-release/nycedc-announces-three-
winners-change-course-nyc-waterfront-construction-competition.
Rudenko, Andrey. 2015. “3D Concrete House Printer.” Total Kustom.
http://totalkustom.com/home.html.
Scott, Rick. 2009. “Estimate the Cost of A Concrete Masonry Unit Wall.” Estimating Today, July.
Simmons, H. Leslie. 2011. Olin’s Construction: Principles, Materials, and Methods. John Wiley
& Sons.
“Small Robots Printing Big Structures.” 2015. Minibuilders. http://iaac.net/printingrobots/.
U.S. Green Building Council. 2015. “Guide to LEED Certification.” USGBC.
http://www.usgbc.org/cert-guide.
Williams, Adam. 2015. “Berkeley Researchers Pioneer New Powder-Based Concrete 3D
Printing Technique.” Gizmag. March 12. http://www.gizmag.com/berkeley-researchers-
pioneer-powder-based-concrete-3d-printing/36515/.
Winsun. 2015. Accessed May 18. http://www.yhbm.com/.
World’s Advanced Saving Project. 2015. “About Us - WASProject.” WASP.
http://www.wasproject.it/w/en/wasp/.

MA Kreiger, BA MacAllister, JM Wilhoit, MP Case. 158

Bridge Structural Condition Assessment using 3D
Imaging
Simon Laflamme
Department of Civil, Construction and Environmental Engineering
Iowa State University
416A Town Engineering
Ames, IA 50011-3232
[email protected]

Yelda Turkan
Department of Civil, Construction and Environmental Engineering
Iowa State University
428 Town Engineering
Ames, IA 50011-3232
[email protected]

Liangyu Tan
Department of Civil, Construction and Environmental Engineering
Iowa State University
427 Town Engineering
Ames, IA 50011-3232
[email protected]

ABSTRACT
Objective, accurate, and fast assessment of bridge structural condition is critical to timely
assess safety risks. Current practices for bridge condition assessment rely on visual
observations and manual interpretation of reports and sketches prepared by inspectors in the
field. Visual observation, manual reporting and interpretation has several drawbacks such as
being labor intensive, subject to personal judgment and experience, and prone to error.
Terrestrial laser scanners (TLS) are promising sensors to automatically identify structural
condition indicators, such as cracks, displacements and deflected shapes, as they are able to
provide high coverage and accuracy at long ranges. However, there is limited research
conducted on employing TLS to detect cracks for bridge condition assessment, which mainly
focused on manual detection and measurements of cracks, displacements or shape deflections
from the laser scan point clouds. TLS is an advance 3D imaging technology that is used to
rapidly measure the 3D coordinates of densely scanned points within a scene. The data
gathered by a TLS is provided in the form of 3D point clouds with color and intensity data often
associated with each point within the cloud. This paper proposes a novel adaptive wavelet
neural network (WNN) based approach to automatically detect concrete cracks from TLS point
clouds for bridge structural condition assessment. The adaptive WNN is designed to self-
organize, self-adapt, and sequentially learn a compact reconstruction of the 3D point cloud. The
architecture of the network is based on a single-layer neural network consisting of Mexican hat
wavelet functions. The approach was tested on a cracked concrete specimen. The preliminary
experimental results show that the proposed approach is promising as it enables detecting
concrete cracks accurately from TLS point clouds. Using the proposed method for crack
detection would enable automatic and remote assessment of bridge condition. This would, in

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
turn, result in reducing costs associated with infrastructure management, and improving the
overall quality of our infrastructure by enhancing maintenance operations.

Key words: laser scanning—bridge condition assessment—crack detection—wavelets—

neural networks

INTRODUCTION
The majority of bridge condition assessments in the U.S. are conducted by visual inspection,
during which a printed checklist is filled by trained inspectors. An inspector must correctly
identify the type and location of each element being inspected, document its distress, manually
record this information in the field and then transcribe that information to the bridge evaluation
database after arriving back at his/her office. This is a complex and time-consuming set of
responsibilities which are prone to error.

Terrestrial laser scanners (TLS) are promising sensors for documenting as-built condition of
infrastructure (Hajian and Brandow, 2012), and they have already been utilized by a number of
state DOTs for this purpose at the project planning phase. Furthermore, TLS technology has
been shown to be effective identifying structural condition indicators, such as cracks,
displacements and deflected shapes (Park et al. 2007; Olsen et al. 2009; Werner and Morris,
2010; Meral 2011; Wood et al. 2012), as they are able to provide high coverage and accuracy at
long ranges. However, there is limited research conducted on employing TLS to detect cracks
for bridge condition assessment, which mainly focused on manual detection and measurements
of cracks, displacements or shape deflections from the laser scan point clouds (Chen 2012;
Chen et al. 2014; Olsen et al. 2013).

The research presented in this paper attempts to automatically detect cracks from TLS point
clouds (Olsen et al. 2009; Anil et al. 2013; Adhikari et al. 2013; Mosalam et al. 2013) for bridge
structural condition assessment. TLS is an advance imaging technology that is used to rapidly
measure the 3D coordinates of densely scanned points within a scene (Fig. 1(a)). The data
gathered by a TLS is provided in the form of 3D point clouds with color and intensity data often
associated with each point within the cloud. Point cloud data can be analyzed using computer
vision algorithms (Fig. 1(b)) to detect structural conditions (Fig 1(c)).

Figure 1. Research Vision

In its raw format, TLS point cloud data contains significant number of data points that is
unstructured, densely and non-uniformly distributed (Meng et al. 2013). Therefore, in machine
learning community, substantial effort has been put in reconstructing 3D shapes from point

Laflamme, Turkan, Tan 160

clouds. Popular reconstruction methods include the utilization of Splines (Gálvez & Iglesias
2012) and partial differential equations (PDE) (Wang et al. 2012), seen as an improvement over
Splines in terms of numbers of parameters. Neural networks have also been proposed, and
demonstrated as superior to PDE-based methods in (Barhak & Fisher 2001).

The overarching goal of this research is to detect 3D shapes from point clouds real-time while
scanning on-site. However, there exist critical challenges in designing a shape reconstruction
algorithm for real-time adaptive scanning, namely:

x The algorithm must adapt sequentially to enable adaptive scanning.

x The representations must be compact to reduce demand on memory. A compact
representation can also facilitate queries over a large database, particularly useful in
extracting prior information in the case of sequential training.
x The number of parameters must remain low to accelerate computational speed. A high
number of parameters would result in a substantial lag in the parameterization process.
x The algorithm must be robust with respect to noise in data, which can be substantial with
TLS-based technologies.

Neural networks have been proposed as candidates for providing robust and compact
representations. In particular, Radial Basis Functions (RBF) neural networks have been applied
to the problem of shape reconstruction (Bellocchio et al. 2013). Compared against traditional
types of neural networks, they provide a better approximation, convergence speed, optimality in
solution and excellent localization (Suresh et al. 2008). Furthermore, they can be trained faster
when modeling nonlinear representations in the function space (Howlett 2001). Recent work has
been published in utilizing sequential RBF networks for reconstructing surfaces from point
clouds (Meng et al. 2013). A self-organizing mapping (SOM) (Kohonen 2001) architecture was
used to optimize node placement, and the algorithm provided good accuracy with minimum
number of nodes.

The authors have developed a sequential adaptive RBF neural network for real-time learning of
nonlinear dynamics (Laflamme & Connor 2009), and returned similar conclusions where the
network showed better performance with respect to traditional neural networks. They also
designed wavelet neural networks (WNN) for similar applications in (Laflamme et al. 2011,
Laflamme et al. 2012). WNN are also capable of universal approximation, as shown in (Zhang &
Benveniste 1992). This particular neural network has also been demonstrated as capable to
learn dynamics on-the-spot, without prior knowledge of the underlying dynamics and
architecture of the input space.

The study presented in this paper proposes a novel adaptive wavelet neural network (WNN)
based approach to automatically detect concrete cracks from TLS point clouds for bridge
structural condition assessment. The adaptive WNN is designed to self-organize, self-adapt,
and sequentially learn a compact reconstruction of the 3D point cloud. The approach was tested
on a cracked concrete specimen, and it successfully reconstructed 3D laser scan data points as
wavelet functions in a more compact format, where the concrete crack was easily identified.
This is a significant improvement over previous TLS based crack detection methods as it does
not require a priori knowledge about the crack or the 3D shape of the object being scanned. It
also enables to process 3D point cloud data faster and detect cracks automatically.
Furthermore, since it is designed to self-organize, self-adapt and sequentially learn a compact
reconstruction of the 3D point cloud, it can easily be adapted for real-time scanning in the field,
which will be investigated in the future using the adaptive WNN approach presented in this
paper.

Laflamme, Turkan, Tan 161

BACKGROUND

Terrestrial Laser Scanning (TLS) Technology

Terrestrial Laser Scanning (TLS) – also known as Light Detection and Ranging (LiDAR) –
enables direct acquisition of 3D coordinates from the surface of a target object or scene that are
visible from the laser scanner’s viewpoint (Alba et al. 2011; Vosselman and Maas 2013; Xiong
et al. 2013). TLS is based on either time-of-flight (TOF) or phase-based technology to collect
range (x, y, z) and intensity data of objects in a scene. The two technologies differ in calculating
the range, while both acquire each range point in the equipment’s spherical coordinate frame by
mounting a laser on a pan-and-tilt unit that provides the spherical angular coordinates of the
point. TOF scanners emit a pulse of laser light to the surface of the target object or scene and
calculate the distance to the surface by recording the round trip time of the laser light pulse.
Phase based scanners measure phase shift in a continuously emitted and returned sinusoidal
wave. Both types of TLS achieve similar point measurement accuracies. They differ in scanning
speed and maximum scanning range. Typically, phase-based TLS achieve faster data
acquisition (up to one million points per second), while TOF-based TLS enables collecting data
from longer ranges (up to a kilometre).

TLS implementation in the AEC-FM Industry

Laser scanning technology enables capturing comprehensive and very accurate three-
dimensional (3D) data for an entire construction scene using only a few scans (Cheok et al.
2002). Among other 3D sensing technologies, laser scanning is the best adapted technology for
capturing the 3D status of construction projects and condition of infrastructure accurately and
efficiently. In a study by Greaves and Jenkins (2007), it is shown that the 3D laser scanning
hardware, software, and services market has grown exponentially in the last decade, and the
Architecture, Engineering, Construction and Facilities Management (AEC-FM) industry is one of
its major customers. This shows that owners, decision makers and contractors are aware of the
potential of using this technology for capturing the 3D as-built status of construction projects and
condition of infrastructure. However, laser scanners’ current adoption rate is very low despite
their tremendous benefits. The major reasons are related to the big size of data they produce
and the long data processing time required.

Laser scanners can output extremely high resolution models, but at a much larger file size and
processing time (Boehler and Marbs, 2003). Despite the remarkable accuracy and benefits,
laser scanners’ current adoption rate in the AEC-FM industry is still low, mainly because of the
data acquisition and processing time and data storage issues. Full laser scanning requires
significant amount of time. Depending on the size of the site, it can take days for large scale
high-resolution shots. Accordingly, resulting data file sizes are typically very large (e.g., a single
high resolution scan file size could be a couple of gigabytes or much larger). Therefore, data
storage and processing are the two biggest factors for the low adoption rates of laser scanners
in the AEC-FM industry.

Thus, there is a need for advanced algorithms that enable automated 3D shape detection from
low resolution point clouds during data collection. This would improve project productivity as
well as safety by reducing the amount of time spent on-site. Importantly, practical applications of
the developed algorithms to field laser scanners will be straightforward since commercially
available laser scanners on the market are generally programmable (Trimble Inc. 2015).

Laflamme, Turkan, Tan 162

RESEARCH METHODOLOGY: ADAPTIVE WAVELET NETWORK
An adaptive wavelet neural network (WNN) has been designed to sequentially learn a compact
reconstruction of the 3D point cloud. The architecture of the WNN is based on a single-layer
neural network, as illustrated in Fig. 2, consisting of Mexican Hat wavelets centered at , with
a bandwidth where each function (or node) can be written as below:

(1)
( )= 1 for = 1,2, … ,

The wavelet network maps the coordinate of point =[ , ] using the following function:

(2)
= ,

where are function weights, and the tilde denotes an estimation.

Figure 2. Single-layer architecture of the wavelet network

The network is self-organizing, self-adaptive, and sequential. The self-organizing feature

consists of the capability to add functions at sparse locations. This is done following Kohonen’s
Self-Organizing Mapping Theory (Kohonen 2001). The self-adaptive feature consists of
adapting the network parameters and to learn the compact representation. Lastly, the
sequential feature refers to the capability of the network to learn at representation while
scanning is occurring, in a sequential way, in opposition to a batch process. This sequential
capability would be used to interact with the 3D scanner in real-time.

The wavelet network algorithm is as follows. First, a new point is queried from the scanner,
along with its associated . The shortest Euclidean distance is computed between the location

Laflamme, Turkan, Tan 163

of the new point and the center of the existing functions for = , , … , .If the shortest
distance is greater than a user-defined threshold , a new function is added at = and
the number of functions increases by 1. Note that this threshold decreases with decreasing
bandwidth , which allows the creation of denser regions where the network resolution is
higher. The weight of the new function is taken as = . Second, if no new function is
added, the estimate is compared against the value , and the network error = is
computed. Third, the network parameters and are adapted using the backpropagation
method (Laflamme et al. 2012):

(3)
=

where = [ , ], and are positive constants representing the learning rate of the network.

EXPERIMENTS AND PRELIMINARY RESULTS

The adaptive wavelet network has been validated on a cracked concrete specimen. The
specimen was scanned using a Trimble TX5 phase-based TLS on a region limited to 50 by 65
mm2 to focus the study on the algorithm itself. A total of 8170 points have been generated. The
specimen is shown in Fig. 3, along with a zoom on the limited region (Fig. 3(b)). Fig. 3 (b) shows
the crack that runs through the region with a wider region (along the first 35.1 mm from the
bottom), and a smaller damage geometry along 9.8 mm and after.

Fig. 4 shows a typical fitting result obtained using 59 nodes. The compact representation
provides a good fit of the 3D point cloud, and includes the damage feature. A study was
conducted on the accuracy of the representation as a function of the number of nodes in the
network, by changing the parameter while keeping all other network parameters constant. The
accuracy was measured in terms of the root means square (RMS) error. Fig. 5 is a plot of the
RMS error as a function of the number of nodes. It also shows the relative computing time
versus the network size. In this case, there is a region in which the algorithm provides an
optimal representation in term of RMS error. The decrease in performance for a higher number
of nodes can be attributed to the network parameters that become mistuned. In particular, when
more nodes are allowed in the network and the initial bandwidth is large, one would expect a
relatively higher training period to obtain an acceptable level of accuracy. The relative
computing time changes linearly with the number of nodes in the network.

Laflamme, Turkan, Tan 164

Figure 3. (a) Specimen (scanned region shown by the dashed rectangle); and (b) zoom on
the scanned region (distances in mm).

Figure 4. (a) Point cloud; (b) Compact representation; and (c) Overlap of point cloud and
representation.

Laflamme, Turkan, Tan 165

 Figure 5. RMS error and relative computing time versus wavelet network size.

While the wavelet network provides an accurate representation of the 3D point cloud, it should
be also capable of extracting key features, such as damage. With this particular example, an
attempt was made to automatically localize the damage and determine its severity. The strategy
consists of identifying regions of wavelets (or nodes) of lower bandwidths, which would indicate
a region of higher resolution, thus the location of a more complex feature (a crack, in this case).
Fig. 6(a) is a wavelet resolution map, which is obtained by computing the average wavelet
bandwidth within a region of the representation. Dark blue areas indicate a high resolution
region, while dark red areas represent low resolution regions. The damage is approximately
localized using this strategy. Next, the crack length and width were estimated by evaluating the
maximum distances along the x- and y-axes within a group of wavelets of low bandwidth. Fig.
6(b) is a plot of the computed crack length and width as a function of the number of nodes. The
approximate crack length is more accurately determined for networks created with a large
number of nodes, but yet yields to an acceptable approximation. The estimated crack width
increases with increasing number of nodes. This is explained by the presence of a high
resolution region around the coordinate [-20, 20], shown in Fig. 6(a), that is perceived as a
crack. A representation created with a large number of functions may over-fit the 3D point cloud.

Laflamme, Turkan, Tan 166

 Figure 6. (a) Wavelet resolution map showing the average wavelet bandwidths for a
representation using 59 nodes (the approximate crack region is shown within the black-
dashed region); and (b) Identified crack length and width based on wavelet resolutions.

CONCLUSIONS
A strategy to sequentially construct a compact representation of a 3D point cloud has been
presented. The representation is wavelet network capable of self-organization, self-adaptation,
and sequential learning. It can be utilized to transform thousands of 3D point cloud data
obtained from a TLS or LiDAR into a small set of functions. The proposed wavelet network has
been demonstrated on a cracked cylindrical specimen. It was shown that the algorithm was
capable of replacing a set of 8170 3D coordinates into a set of 59 functions while preserving the
key features of the scan data, which included a crack. By looking at local regions of high-
resolution wavelets, it is possible to localize these features, and estimate their geometry. While
the promise of automatic damage detection has been demonstrated, the development of more
complex algorithms in future work could lead to more accurate numerical localization and
estimation of damage.

ACKNOWLEDGMENTS
This research is funded by Midwest Transportation Center (Award# 011296-00014). The
authors would like to thank Ahmad Abu-Hawash, Justin Spencer, and Michael Todsen from
Iowa DOT for their continuous support by providing us with data, and for sharing their expertise
and experience during this project. Any opinions, findings, conclusions, or recommendations
expressed in this paper are those of the authors and do not necessarily reflect the views of
Midwest Transportation Center or Iowa DOT.

REFERENCES
Alba M.I., Barazzetti L., Scaioni M. Rosina E., and Previtali M. Mapping infrared data on
terrestrial laser scanning 3D models of buildings, Remote Sensing, 3(9): 1847–1870, 2011.

Anil E.B., Akinci B., Garrett J.H., Kurc O. (2013). Characterization of laser scanners for
detecting cracks for post-earthquake damage inspection. Proceedings of the International
Symposium on Automation and Robotics in Construction and Mining, IAARC, pp 313-320.

Laflamme, Turkan, Tan 167

Adhikari, R. S., Zhu, Z., Moselhi, O., & Bagchi, A. (2013). Automated Bridge Condition
Assessment with Hybrid Sensing. Proceedings of the International Symposium on Automation
and Robotics in Construction and Mining, IAARC, pp.1263-1270.

Barhak, J., & Fischer, A. (2001). Parameterization and reconstruction from 3D scattered points
based on neural network and PDE techniques. Visualization and Computer Graphics, IEEE
Transactions on, 7(1), 1-16.

Bellocchio, F., Borghese, N. A., Ferrari, S., & Piuri, V. (2013). Hierarchical Radial Basis
Functions Networks. In 3D Surface Reconstruction (pp. 77-110). Springer, New York.

Boehler W., Marbs A. (2003). Investigating laser scanner accuracy. Institute for spatial
information and surveying technology. University of Applied Sciences, Mainz, Germany.

Chen S. (2012). Laser Scanning Technology for Bridge Monitoring, Laser Scanner Technology,
Dr. J. Apolinar Munoz Rodriguez (Ed.), ISBN: 978-953-51-0280-9, InTech

Chen, S. E., Liu, W., Bian, H., & Smith, B. (2014). 3D LiDAR Scans for Bridge Damage
Evaluations. Bridges, 10, 9780784412640-052.

Cheok G.S., Leigh S., Rukhin A. (2002). Calibration experiments of a Laser Scanner. Building
and Fire Research Laboratory, National Institute of Standards and Technology, Gaithersburg,
MD, USA.

Gálvez, A., & Iglesias, A. (2012). Particle swarm optimization for non-uniform rational B-spline
surface reconstruction from clouds of 3D data points. Information Sciences, 192, 174-192.

Greaves T., Jenkins B. (2007). 3D laser scanning market red hot: 2006 industry revenues $253
million, %43 growth, SPAR Point Research, LLC 5 (7).

Hajian, H., Brandow, G. (2012). As-Built Documentation of Structural Components for

Reinforced Concrete Construction Quality Control with 3D Laser Scanning. Proceedings of the
2012 ASCE International Conference on Computing in Civil Engineering. ASCE Publications,
pp.253-260.

Howlett, R. J., & Jain, L. C. (Eds.). (2001). Radial basis function networks 1: recent
developments in theory and applications (Vol. 66). Springer.

Kohonen, T. (2001). Self-organizing maps (Vol. 30). Springer Science & Business Media.

Laflamme, S., & Connor, J. J. (2009, March). Application of self-tuning Gaussian networks for
control of civil structures equipped with magnetorheological dampers. In SPIE Smart Structures
and Materials+ Nondestructive Evaluation and Health Monitoring (pp. 72880M-72880M).
International Society for Optics and Photonics.

Laflamme, S., Slotine, J. J. E., & Connor, J. J. (2011). Wavelet network for semi-active control.
Journal of Engineering Mechanics, 137(7), 462-474.

Laflamme, Turkan, Tan 168

Laflamme, S., Slotine, J. E., & Connor, J. J. (2012). Self-organizing input space for control of
structures. Smart Materials and Structures, 21(11), 115015.

Meng, Q., Li, B., Holstein, H., & Liu, Y. (2013). Parameterization of point-cloud freeform
surfaces using adaptive sequential learning RBFnetworks. Pattern Recognition, 46(8), 2361-
2375.

Meral C. (2011). Evaluation of Laser Scanning Technology for Bridge Inspection, MS Thesis,
Civil Engineering Department, Drexel University

Mosalam, K. M., Takhirov, S. M., & Park, S. (2014). Applications of laser scanning to structures
in laboratory tests and field surveys. Structural Control and Health Monitoring, 21(1), 115-134.

Olsen, M. J., Kuester, F., Chang, B. J., & Hutchinson, T. C. (2009). Terrestrial laser scanning-
based structural damage assessment. Journal of Computing in Civil Engineering, 24(3), 264-
272.

Olsen, M. J., Chen, Z., Hutchinson, T., & Kuester, F. (2013). Optical techniques for multiscale
damage assessment. Geomatics, Natural Hazards and Risk, 4(1), 49-70.

Park, H. S., Lee, H. M., Adeli, H., & Lee, I. (2007). A New Approach For Health Monitoring Of
Structures: Terrestrial Laser Scanning. Computer aided civil and infrastructure engineering, 22,
19–30.

Suresh, S., Narasimhan, S., & Sundararajan, N. (2008). Adaptive control of nonlinear smart
base-isolated buildings using Gaussian kernel functions. Structural Control and Health
Monitoring, 15(4), 585-603.

Trimble Inc. Trimble TX5 3D Laser Scanner User Guide. Online:

http://mep.trimble.com/sites/mep.trimble.com/files/Trimble%20TX5%20User%20Guide.pdf

Trimble Inc. Trimble TX5 Automation Interface User Guide. Online:

http://mep.trimble.com/sites/mep.trimble.com/files/marketing_material/Trimble_TX5_Automation
_Interface.pdf

Wang, C., Shi, Z., Li, L., & Niu, X. (2012). Adaptive Parameterization and Reconstruction of 3D
Face Images using Partial Differential Equations. IJACT: International Journal of Advancements
in Computing Technology, 4(5), 214-221.

Werner T., Morris D. (2010). 3D Laser Scanning for Masonry Arch Bridges. Proceedings of FIG
Congress, Facing the Challenges – Building the Capacity.

Wood, R. L., Hutchinson, T. C., Wittich, C. E., & Kuester, F. (2012). Characterizing Cracks in
the Frescoes of Sala degli Elementi within Florence’s Palazzo Vecchio. In Progress in Cultural
Heritage Preservation (pp. 776-783). Springer Berlin Heidelberg.

Vosselman G. and Maas H.-G. Airborne and Terrestrial Laser Scanning, first ed. Whittles
Publishing, Dunbeath, UK, 2010.

Laflamme, Turkan, Tan 169

Xiong X., Adan A., Akinci B., and Huber D. Automatic creation of semantically rich 3D building
models from laser scanner data, Automat. Constr., 31: 325–337, 2013.

Zhang, Q., & Benveniste, A. (1992). Wavelet networks. Neural Networks, IEEE Transactions on,
3(6), 889-898.

Laflamme, Turkan, Tan 170

Bluetooth Low Energy Sensing Technology for
Proximity Construction Applications

JeeWoong Park
School of Civil and Environmental Engineering,
Georgia Institute of Technology,
790 Atlantic Dr. N.W.,
Atlanta, GA 30332-0355, United States,
[email protected]

Dr. Yong Cho

School of Civil and Environmental Engineering,
Georgia Institute of Technology,
790 Atlantic Dr. N.W.,
Atlanta, GA 30332-0355, United States,
[email protected]

Willy Suryanto
School of Civil and Environmental Engineering,
Georgia Institute of Technology,
790 Atlantic Dr. N.W.,
Atlanta, GA 30332-0355, United States,
[email protected]

ABSTRACT
Safety is considered one of the most importance components that need to be successfully
addressed during construction. However, dynamic nature and limited work space of roadway
work zones create highly occupied working environments. This may further result in hazardous
proximity situations among ground workers and construction equipment. In fact, historical
incident statistics prove that the current safety practice has not been effective and there is a
need for improvement in proving more protective working environments. This study aims at
developing a technically and economically feasible mobile proximity sensing and alert
technology and assessing it with various simulation tests. Experimental trials tested the sensing
and alert capability of the technology against its accuracy and reliability by simulating
interactions between equipment and a ground worker. Experimental results showed that the
developed mobile technology offers not only adequate alerts to the tested person in proximity
hazardous situations but also other advantages over the commercial products that may play an
important role in overcoming the obstacles for rapid deployment of new technology in
construction segments.

Key words: Bluetooth low energy—construction safety—construction equipment—

iBeacon—proximity sensing

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
INTRODUCTION
With the development of wireless technology in the last decade, mobile devices have become
an essential component in our daily life being used for multiple purposes. The advancement in
the wireless technology enabled most of the recently produced cars equipped with a Bluetooth
technology. The driver is then able to communicate with his/ her mobile device via a Bluetooth
enabled car, triggering phone calls listening to music and radio without having to making
physical contacts with the device. This has turned our daily activity of driving a car into a much
safer experience, allowing the driver to better focus on the road. According to (Statista, 2015),
the population of smartphone users have rapidly been increasing and the expected number of
population in the U.S. is 183 million, which is more than a half U.S. population (See Figure 1).

Figure 1. The number of smart phone users in the U.S.

Safety is one of the most importance components that need to be successfully addressed
during construction. However, dynamic nature and limited work space of roadway work zones
create highly occupied working environments. This may further result in hazardous proximity
situations among ground workers and construction equipment. 962 deaths of workers were
recorded at a road construction sites from 2003 to 2010. In addition, about 30% of deaths
related to construction 2012 were resulted from being struck by a vehicle. These historical
incident data prove that the current safety practice has not been effective and there is a need for
improvement in proving more protective working environments. Recent industrial (ENR, 2015)
efforts have been found with deploying cameras and motion sensors near the blind spots of a
piece of equipment. Various proximity sensing devices have been discussed and evaluated by
(Ruff, 2007), (Begley, 2006), (Marks and Teizer, 2012) and (Larsson, 2003). Tested and
evaluated systems in the past research require external hardware, such as camera, laser
scanner, tripod, power supply lines, heavy antenna, or tags. These are the major components of
each of the systems to achieve communication between a hazardous source and an object that
is potentially in a dangerous zone. While being major components, these requirements are
barriers that limit the systems’ feasibility and practicality in dynamic construction applications. In
addition to the infrastructure requirement, there are other parameters that are crucial for
assessing a system’s feasibility and practicality, including detection area, cost, maintenance,
accuracy, precision, consistency, alert method, adaptability, required power sources, ease of

JeeWoong Park, Dr. Yong Cho, Willy Suryanto 172

use, ease of deployment. Benefits and limitations of other proximity sensing systems were
discussed in (Castleford et al., 2001; Goodrum et al., 2006; Marks and Teizer, 2013)

Although smart devices are already pervasive and embedded into our society, their potential
uses in construction industry have not been discussed and minimal research and
experimentation have not been conducted with utilizing smart devices, despite the efforts made
with other technologies in the last decade. This study proposes a wireless proximity sensing
technology that utilizes Bluetooth transmitters and mobile devices to create a proximity sensing
and warning system. The authors consider that several characteristics of smart devices, such as
pervasiveness, availability, and familiarity to end users, are the key factors to realize a feasible
and practical technology. In the following sections, an extensive overview of the proposed
system will be discussed, and experimental validation and conclusion will follow.

OBJECTIVE
In adopting a proximity sensing and alerting system, several factors play an important role for a
system to be feasible and pragmatic. They include detection area, cost, maintenance, accuracy,
precision, consistency, alert method, size of infrastructure, adaptability, required power sources,
ease of use, ease of deployment, and others. The main objective of this study is to develop and
validate a proximity sensing system that is economically and technically feasible and practical.
The system should provide minimal infrastructure, adaptability with calibrating ability,
intensifying alerts to reflect the degree of dangerousness, and real-time alerts to pedestrian
workers and equipment operators during hazardous proximity situations. Widely available smart
devices and low-cost Bluetooth transmitters are utilized to create a proximity sensing and alert
system. Through field experimentations, their performance have been tested and assessed in
various aspects.

PROPOSED PROXIMITY SENSING SYSTEM

The proposed proximity sensing system is based on Bluetooth based wireless sensing
technology (iBeacon technology). This system offers various promising characteristics, including
rapid connectivity, ease of deployment, low-cost hardware, minimal infrastructure and ease of
integration with other systems. These characteristics are especially beneficial when considering
adoption into construction industry. The system provides intensifying alerts upon creation of
hazardous incidents to mitigate potential risks that will otherwise be posed upon workers and
equipment. The system is composed of major components to create a basic proximity sensing
and alert system, and of auxiliary components to support the system in different aspects.

System Major Components

The developed system is composed of three main hardware components, including 1) signal
transmitters (Bluetooth transmitter), 2) personnel receivers (Bluetooth enabled mobile device),
3) equipment operator’s receiver (Bluetooth enabled mobile device), and the software
component which provides an user interface and application function on which the system
operates. A signal transmitter (beacon) is called Equipment Protection Unit (EPU) that transmits
Bluetooth signal using Bluetooth Low Energy (BLE). Beacons are to create detection area from
a hazard source to protect workers. Figure 2 describes an example of beacon deployments on a
piece of construction equipment and a worker operating nearby. To create a symmetrical
detected area, several beacons are attached symmetrically around a piece of construction
equipment. A worker on the right of Figure 2 is equipped with a Bluetooth enabled mobile
device, which is also called Pedestrian worker’s Personal Protection Unit (PPU). This PPU
provides intensifying sound alerts and vibration upon breach into a hazardous zone defined

JeeWoong Park, Dr. Yong Cho, Willy Suryanto 173

based on a piece of equipment considered. An equipment operator is also equipped with a
Bluetooth enabled mobile device, which is also called Equipment operator’s Personal Protection
Unit (PPU). When a potential hazardous incident is created, this PPU provides intensifying
alerts and the direction of the incident with respect to the equipment.

Figure 2. Major components of the proposed system

System Auxiliary Components

The proposed system is also composed of several auxiliary components to reinforce the
communication and alert system especially when operating in a harsh working environment and
to provide more features and opportunities for post safety analysis. Additional Bluetooth enabled
devices including a smartwatch and an earpiece can be incorporated into the existing system for
enhanced vibratory and sound alerts. Also, a cloud server is configured for the proposed system
to gather data that indicate a potentially created hazardous situation. Whenever hazard incident
is created, the system can send to a cloud server the creation of the incident for post safety
analysis.

System Component Communication Flow

Previous sections described the roles of each of the components. This section explains the
system work flow and details in each of the flow steps. Figure 3 shows a flow chart for one cycle
of communication of the system. This flowchart illustrates one cycle of the system
communication. After each cycle the flowchart points to “Keep Monitoring”, which basically
indicates that cycles are repeating. Each cycle starts with communication of beacons from a
hazardous source (e.g., a piece of equipment) and a mobile device (e.g., a protected worker).
Based on the user’s distance set-up, the system determines if dangerous zone has been
breached by either the piece of construction equipment or the worker during the dynamic
interaction of the two. When it is determined to be breached, the system proceeds with

JeeWoong Park, Dr. Yong Cho, Willy Suryanto 174

prevention actions to provide an additional chance for the worker and equipment to escape from
the scene. The prevention actions immediately take place without time delay from the hazard
situation detection by the system. Audible alerts and vibration get triggered to the worker’s PPU,
while audible alerts and visualization of the direction of the hazard situation with respect to the
equipment are provided to the equipment operator via his/ her PPU. In addition, the system
support cloud based data collection to allow for post analysis.

Audible
Worker’s
Beacon Alert &
PPU
(EPU) Vibration

Yes Equipment Audible

Preventive
Operator’s Alert &
Action
PPU Visualization
Worker’s Is Hazard
PPU Situation Created?
Send Data
Post analysis
to Cloud

No
Keep
Monitoring

Figure 3. Communication flow for one cycle

Methods
The proposed system is a Bluetooth based system whose communication among devices is
accomplished by radio signal transmission. The transmitted radio signal is recorded and the
recorded Received Signal Strength (RSS) estimates the approximate horizontal distance
between the beacons and receiver. The user can perform calibration to set his/ her desired
distance range at which the system initiates alerts. This calibrated range is then trisected to
provide intensifying alerts to indicate the degree of dangerousness of a created hazardous
situation. Upon the creation of the hazardous situation, the worker’s PPU immediately turns to a
beacon to send out signals to the nearby operator’s PPU. With this signal, alerts and
visualization can be realized on the equipment operator’s PPU.

System Deployment and Calibration

The key aspect in system deployment is to acquire a symmetrical coverage centered from a
piece of construction equipment. To improve the quality of symmetrical coverage, multiple
Bluetooth signal transmitters are deployed around the equipment. This deployment allows the
communication to reply more on beacons that experience the least amount of multipath effects
and signal degradation. In addition, to reduce the amount of signal interference, care should be
taken when placing beacons on a piece of equipment so that the best line of sight is obtained.

Calibration is desired for two major reasons. First, the RSS is dependent on environmental
conditions. For different equipment and environmental conditions, there is no guarantee for the

JeeWoong Park, Dr. Yong Cho, Willy Suryanto 175

radio communication to be in the same quality. Second, each user’s may have a different
desired coverage range. Per user’s need for coverage range, by collecting RSS at a desired
distance for each of the beacons, calibration is performed.

FIELD VALIDATION
To validate the proposed proximity sensing and alert system, a set of experimental trials were
designed and conducted to evaluate the system. To assess the system reliability and
effectiveness, (1) trials were performed at eight different angles centered from a piece of
equipment and (2) two different pieces of equipment were tested, such as a wheel loader and a
dump truck. The design of experimental simulation is to emulate real-time construction roadway
work zone operations. Presented material in this paper is mobile workers and static equipment
situation. Figure 4 shows the test bed and approach angles for worker and equipment
interaction simulations during testing. A worker equipped with a worker’s PPU approaches to a
piece of construction equipment and the alert distance (at which breach into hazard zone is
detected) was recorded for each of the trials. For each angle of the eight angles, 20 trials were
made for statistical data collection.

Figure 4. Test bed and approach angles during trial simulations

For these two different sets of trials, calibration was only performed for the trial with wheel
loader. The purpose of this was to observe the difference in RSS behavior, thus difference in
alert distance accuracy, in different environmental conditions and to confirm the needs of
calibration. In the wheel loader trials, the alert distance was set to 12 meters, and the same
setup was used for the truck trials. Statistically analyzed data for two complete sets of trials are
tabulated in Table 1, and Figure 5 shows plots for the sets of trials. In the simulation with the
wheel loader, the overall average alert distances did not deviate significantly from the set
distance of 12 meters except at 135°. This drop needs be investigated considering various
factors, such as battery, interference with the surroundings, or potential mal-functions of
transmitter. As seen in both Table 1 and Figure 5, overall average alert distances for the dump

JeeWoong Park, Dr. Yong Cho, Willy Suryanto 176

truck simulation are, however, greater than those for the wheel loader simulation. This shows
that the importance of calibration to obtain a desired distance. In this case, the RSS was more
powerful with the truck simulation, and proper calibration should be able to manage this
difference to set the alert distance as desired by the worker. Other than the average alert
distance, the two simulations behaved similarly.

Table 1. Statistical analysis of alert distance and standard deviation

Approach Wheel loader Dump truck
angle Average (m) Std (m) Average(m) Std (m)
0° 15.3 2.7 16.8 3.4
45° 12.2 3.0 17.1 2.5
90° 12.4 3.6 22.6 2.8
135° 5.3 1.8 19.2 1.9
180° 12.4 3.9 17.1 1.6
225° 13.0 3.6 18.8 1.7
270° 15.2 1.0 16.2 1.4
315° 17.3 1.0 12.5 2.9
*std is standard deviation.

Figure 5. Field trials with a wheel loader (left) and a dump truck (right)

Table 2 displays another statistical result showing recall values. Different values of distance at
failure were used to compute recall values as the definition of distance at failure may be
different depending on the application. As Table 2 is read, one should keep in mind that the
average alert distances of the two simulations (wheel loader and dump truck) were different,
and therefore, their direct comparisons should not be made. For the entire trials, the system
performed with less than 3% recall rates for less than five meters distance at failure. Five meter
distance boundary seems reasonable as alerts are simultaneously provided to both the

JeeWoong Park, Dr. Yong Cho, Willy Suryanto 177

pedestrian worker and equipment operator allowing them to stop operations and take a proper
action for avoidance of collision. However, if the lower distance boundary (distance at failure) is
required as high as nine meters, one should consider having a higher desired distance, in this
case higher than 12 meters.

Table 2. Statistical analysis of recall rates

Number of false negative
Distance at
Wheel Loader Truck Total Recall rate
failure (m)
(out of 160) (out of 160) (out of 320)
3 3 0 3 0.9%
5 8 0 8 2.5%
7 25 0 25 7.8%
9 32 2 34 10.6%

CONCLUSION
This study aims at developing a technically and economically feasible mobile proximity sensing
and alert technology and assessing it with various simulation tests. In order to overcome the
barrier of deployment costs, a cost effective system was developed. This system is based on
Bluetooth technology, which is already widely available in most of the recent smart devices.
Also, the system offers minimal infrastructure, ease of deployment, calibration functionality and
adaptability, compared with other similar proximity sensing and alert systems. Experimental
trials were designed and performed to evaluate the proposed proximity sensing and alert
system for its capability to offer real-time situational awareness via alerts to pedestrian workers
and equipment operators working in proximity hazardous situations. Results of the simulated
tests showed that this system was acceptable in providing pedestrian workers and equipment
operators multiple forms of alerts. Upon the detection of a potential hazardous situation,
immediate alerts were provided to both the pedestrian worker and the equipment operator. This
can help to minimize the proximity related accidents by providing an additional chance of time
and space for the workers to escape from the hazardous scenes.

REFERENCES

Begley, R. (2006). "Development of Autonomous Railcar Tracking Technology Using Railroad

Industry Radio Frequencies". Research Opportunities in Radio Frequency Identification
Transportation Applications, 59–60. Retrieved from
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.4645&rep=rep1&type=pdf#p
age=67

Castleford, D., Nirmalathas, A., Novak, D., and Tucker, R. S. (2001). "Optical crosstalk in fiber-
radio WDM networks". IEEE Transactions on Microwave Theory and Techniques, 49(10),
2030–2035. doi:10.1109/22.954826

ENR. (2015). "More Blind-Spot Sensors Make Jobsites Safer". Retrieved April 9, 2015, from
http://enr.construction.com/products/equipment/2015/0216-65279more-blind-spot-sensors-
make-jobsites-safer.asp

JeeWoong Park, Dr. Yong Cho, Willy Suryanto 178

Goodrum, P. M., McLaren, M. A., and Durfee, A. (2006). "The application of active radio
frequency identification technology for tool tracking on construction job sites". Automation
in Construction, 15(3), 292–302. doi:10.1016/j.autcon.2005.06.004

Larsson, T. (2003). "Industrial forklift trucks: Dynamic stability and the design of safe logistics".
Safety Science Monitor, 7(1), 1–14. Retrieved from http://www.diva-
portal.org/smash/record.jsf?pid=diva2:430110

Marks, E., and Teizer, J. (2013). "Method for testing proximity detection and alert technology for
safe construction equipment operation". Construction Management and Economics, 31, 1–
11. doi:10.1080/01446193.2013.783705

Marks, E., and Teizer, J. (2012). "Proximity Sensing and Warning Technology for Heavy
Construction Equipment Operation" (pp. 981–990). Construction Research Congress.
doi:10.1061/9780784412329.146

Ruff, T. (2007). "Recommendations for evaluating and implementing proximity warning systems
on surface mining equipment". Retrieved from http://stacks.cdc.gov/view/cdc/8494/Print

Statista. (2015). "Smartphone users in the U.S. 2010-2018 | Forecast". Retrieved from
http://www.statista.com/statistics/201182/forecast-of-smartphone-users-in-the-us/

JeeWoong Park, Dr. Yong Cho, Willy Suryanto 179

A Design Framework for Off-road Equipment
Automation
Brian L. Steward
Department of Agricultural and Biosystems Engineering
Iowa State University
2325 Elings Hall
Ames, Iowa 50011
[email protected]

Lie Tang
Department of Agricultural and Biosystems Engineering
Iowa State University
2325 Elings Hall
Ames, Iowa 50011
[email protected]

Shufeng Han
John Deere Intelligent Solutions Group
4140 114th Street
Urbandale, IA 50322
[email protected]

ABSTRACT
Design frameworks can be helpful in the development of complex systems needed to automate
machines. Designing autonomous off-road machinery requires having the means for managing
the complexity of multiple interacting systems. A design framework, consisting of four technical
layers, is presented. These layers are (1) machine architecture, (2) machine awareness, (3)
machine control, and (4) machine behavior. Examples of technology advanced in development
efforts of autonomous, robotic platforms for agricultural applications are provided. Linkages
were made to applications in the construction machinery sector. Similarities between agricultural
and construction automation exist in each of the technical layers.

Key words: automation—design framework—perception—machine localization—

machine behavior

INTRODUCTION AND BACKGROUND

Developing autonomous or robotic construction machinery systems can be inspired by similar
technological developments in other sectors. In agriculture, for example, automated agricultural
machinery technology has been under development for almost 50 years. In the 1970s,
electronics for monitoring and control were introduced to agricultural machines, particularly for
controlling the chemical application rate of agricultural sprayers and monitoring the consistent
drop of seeds in a planter. The most significant advance toward agricultural machine autonomy
started in the 1990s when precision agriculture became the key driver for integrating more
sensors and controls into agricultural machinery.

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the authors, who are responsible for the facts
and accuracy of the information presented herein.
Precision agriculture is a management strategy to reduce the management scale from field
scale to sub-field scales – on a meter-by-meter scale or in management zones with similar soils
or topography. Precision agriculture is an approach to intensively managing spatial and
temporal variability of agricultural fields. It is enabled by automation technologies, but there are
many examples around the world where precision agriculture management is practiced under
low technology conditions.

In the large scale agricultural practices of North America, Europe, and Australia, among others,
automated operation of agricultural machines has been relied upon to achieve the goal of
precision agriculture. As an example, variable-rate application of chemical inputs, one of the
major precision agriculture practices, needs the application rate to be changed on-the-go and
sometimes within every square meter of a field. Manual operation and control of the machine is
infeasible under large field conditions. Thus automatic rate control has been implemented on
these machines.

However, while many agricultural machinery operations have some automation technology
embedded in them, there are few examples of autonomous or robotics machines in agriculture.
Based on the experience of the industry with automation technology, the vision of autonomous
and robotic systems has developed nevertheless. The use of small field robots is desirable for
many precision agriculture practices, such as soil sampling, crop scouting, site-specific weed
control, and selective harvesting. Robotic applications are not only desirable, but are also more
economically feasible than conventional systems for some agriculture applications (Pedersen et
al., 2006).

There are several underlying motivations to move to more autonomous and robotic agricultural
field operations. First in many cases, automated processes can achieve a greater precision in
meeting performance specifications than can humans. An automatically guided tractor, for
example, can be driven through a field with smaller deviations from a straight line resulting in
lower overlapped application of inputs and fewer skipped areas which were not properly treated.
Automation thus leads to greater input efficiency. The availability and cost of labor can be
prohibitive in agriculture particularly because of the timeliness requirements of agricultural
processes such as planting and harvesting. Agricultural automation can extend the productivity
of human labor by working in collaboration with humans in a co-robotic fashion. Recent
increases in agricultural productivity have been achieved through increasing equipment size.
Many agricultural machines are now facing limits to larger sizes, and increased size also leads
to soil compaction which has a negative impact on crop yield. Autonomous field robots have
potential to address productivity barriers and reduced soil compaction at the same time through
small vehicle platforms operating in a fleet to accomplish the needed work rates. Such small
vehicles also have the potential to reduce energy inputs (Toledo et al., 2014).

While there are many differences between the agricultural machinery and construction
machinery sectors, there are also several similarities. Similarities between the two sectors
include machinery interaction with media such as soil or biomaterials with uncertain physical
parameters that are spatially and temporally varying. Both sectors need to lower costs and
improve input efficiency, as well as improve performance and productivity. Additionally, in both
cases, human operators interact with the machines, and other people work in close proximity to
the machines, so safety is of primary importance. Construction and agriculture both seek to
minimalize environmental impact while developing an infrastructure that meets human needs.

Steward, Tang, and Han 181

A main goal of agricultural automation then is to increase food production for a growing world
population. Many estimate that food production will need to double by 2050 to keep up with the
demand driven by growing population and affluence. These production increases must occur
while also minimizing negative environmental impacts. In the automation of earthmoving and
road construction machines, as specific examples of construction machinery automation, the
goals are different, but bear some similarity to those of agricultural automation. Construction
automation goals include improved productivity particularly in the earthmoving and material
transport applications (Singh, 2002, 1997). Efficiency is also an important goal. Process and
quality control of a construction process is an important goal as in the applications of roller-
integrated compaction measurements (Vennapusa et al., 2009) or concrete paving (Castro-
Lacouture et al., 2007). Safety of construction workers is also an important goal (Shi et al.,
2005; Dumpert, 2004).

The goal of this paper is to present a design framework developed for autonomous or robotic
machines in agriculture that it might inspire similar thinking about robotic construction machines,
particularly those used for road construction or earthmoving.

DESIGN FRAMEWORK FOR AUTONOMOUS MACHINES

Machine autonomy is the capability of a machine to achieve operational goals independent of
human operation or intervention in an uncertain environment. Thus autonomy requires a much
higher level of complexity and artificial intelligence than observed in machines that are simply
automated. To manage this complexity and to understand where various research and
development projects fit in the context of building autonomous machines, a design framework
has been developed to categorize the required technologies for robotic agricultural machines
(Han et al., 2015). This design framework is general enough to be applied to autonomous
construction machines as well. The framework consists of four different technology layers
naturally dependent on one another (Figure 1). These layers are, starting from the bottom: (1)
machine architecture, (2) machine awareness, (3) machine control, and (4) machine behavior.
Each layer is described below with applications from the agricultural machinery area. Possible
extensions to construction robots are provided.

Machine Architecture
In autonomous systems engineering, an architecture is a means for managing complexity.
Autonomous and robotic machines of necessity are complex systems comprised of various
components and sub-systems; many of which are complex systems themselves. Since
individual humans and teams are limited in their time and resources, as well as their ability to
keep track of details, they need a way to manage system complexity during development. The
principle of abstracting complexity through encapsulation of components and using clearly
defined interfaces to the components is generally what is meant by the phrase “robotic system
or software architecture.”

Steward, Tang, and Han 182

 Figure 1. A four-layer design framework for autonomous machines.

Autonomous robots require, to varying degrees, architecture for both hardware and software.
Just thinking about the components required for a particular robotic application and how those
components are connected to one another and are interacting with one another is a simple
example of system architecture. Potential is excellent for leveraging the work across research
teams through system architectures that can be shared. These architectures can be proprietary
so that development teams can internally manage complexity. Architectures can also be open
and public to facilitate more rapid development across development teams, as well as to
facilitate the interconnectivity of components and sub-systems available on the market.

At the lowest level, an autonomous construction machine is composed of hardware and

software components required for the machine’s function. These components include the
mechanical parts and assemblies making up the machine along with a prime mover and drive
train needed to control and apply power to meet the operational goals of the machine. For
machine automation and eventual autonomy, these hardware components also include the
electronic, sensing, and actuation components required to automate machine functions. A
machine’s system architecture, consisting of both hardware and software, must be in place to
build the higher level layers needed for autonomy. Since the machine must interact with the
physical world, a physical hardware architecture must be in place. For an autonomous
construction robot, the hardware must enable mobility within the job site, as well as provide the
capability to perform operations in an automated manner. The hardware architecture must be
mechatronic – an integration of mechanical, electrical and electronic, fluid power, and
computational systems – to provide the functionality required to support autonomous
operations. The necessary interconnections between systems are needed to communicate both
data and power. Other hardware that must be present are the sensors which transduce physical
into electrical signals and actuators which provide force and motion to interact with the soils or
other materials.

Complementary to the hardware architecture, software and communications architectures must

also be in place so that the development of higher level layer technology can be built on pre-

Steward, Tang, and Han 183

established software components enabling communication and reusing lower level
computational solutions. While different architectures may focus on different aspects of robotic
systems, they tend to provide means for (1) modularizing tasks for processes that are important
to a functioning robot, (2) defining messaging systems and protocols for inter-process
communication, and (3) defining operations that must occur across distributed processes.
Several illustrative robotic system architectures portray key features of architectural thinking that
is needed for autonomous machines.

Kramer and Scheutz (2007) surveyed nine open source robotic development environments, or
system architectures, for mobile robots, and evaluated their usability and impact on robotics
development. Jenson et al. (2014) surveyed available robotic system architectures including
CARMEN, CLARAty, Microsoft Robotics Developer Studio, Orca, Orocos, Player, and ROS.
They also found examples of lesser known architectures which may be more relevant to
agricultural robots, including Agriture, Agroamara, AMOR, Mobotware, SAFAR, and Stanley. Of
these, four architectures, CARMEN, Agroamara, Mobotware, and SAFAR, had field trials for
agricultural applications. However, open source availability was limited and only Mobotware had
been recently updated.

In early efforts to promote architectural thinking about agricultural robots, Blackmore et al.
(2002) proposed a conceptual system architecture for autonomous tractors that consisted of a
set of objects or agents which have well defined narrow interfaces between them. The two types
of agents are processes and databases. A process carries out tasks to achieve a goal. Nine
processes were defined and described: Coordinator, Supervisor, Mode Changer, Route Plan
Generator, Detailed Route Plan Generator, Multiple Object Tracking, Object Classifier, Self-
Awareness, and Hardware Abstraction Layer (Figure 2). Three databases were defined
(Tractor, Implement, and GIS) and are used to store and retrieve data about the machine and its
operational context. This type of architectural thinking could be applied to construction robots as
a means for determining the structure required for construction robots.

The Joint Architecture for Unmanned Ground Systems (JAUGS) has seen some
implementations in agriculture including an autonomous orchard tractor and an autonomous
utility vehicle (Torrie et al., 2002). JAUGS was primarily a standard messaging architecture to
enable components to communicate to one another in a standard manner. Later, JAUGS was
changed to JAUS (Joint Architecture for Unmanned Systems) to be more generally applied to all
types of unmanned vehicles and became a Society of Automotive Engineers standard. The
standard has two parts; the Domain Model which describes the goals for JAUS, and the
Reference Architecture which specifies an architecture framework, a message format, and a
standard message set (Rowe and Wagner, 2008).

The Robotics Operating System (ROS; Open Source Robotics Foundation) is a general open-
source robotic operating system not specific to any application domain. It provides an interface
for passing messages between processes running on different host computing platforms that
make up the computer hardware architecture of a robot. ROS also provides a broad set of
libraries and tools useful for robotics development. Libraries include (1) standard robot message
definitions, (2) the transform library for managing coordinate transform data, (3) a robot
description language for describing and modelling a robot, (4) means for collecting diagnostics
about the state of the robot and (5) packages for common robotics problem such as pose
estimation, localization, and mobile navigation (ROS.org, 2015; Quigley et al., 2009). ROS has
applicability to autonomous construction machines. It has been used as a part of larger system
architectures for agricultural machines.

Steward, Tang, and Han 184

 Figure 2. A conceptual system architecture consisting of ten encapsulated processes,
databases and inter-process messaging for an autonomous tractor (Blackmore et al.,
2002).

FroboMind is a robotic software systems architecture intended to assist in the development of

field robots for precision agriculture tasks (Jenson et al., 2014; Figure 3). FroboMind has a four
part structure, which from lowest to highest levels include: operating system, middleware,
architecture, and components. The Linux operating system Ubuntu was chosen because of its
large distribution and long-term support. ROS was used for the middleware to define the internal
communication structure between processes. The FroboMind architecture level consists of four
modules, which are perception (sensing and processing), decision making (mission planning
and behavior), action (executing and controlling), and safety modules. The component level
consists of software components implemented as ROS packages. FroboMind is open source
and has been used in the development of several agricultural robots. It is not a hard real-time
system, but its “soft” real-time performance appears to be sufficient for agricultural robotics
applications, and would likely be satisfactory for construction applications. While FroboMind was
designed for field robots doing precision agriculture tasks to enable field experiments and more
efficient reuse of existing work across projects, it could also provide an architecture for
construction robots.

Robotic software system architectures provide the means for handling complexity through well-
defined processes and messaging, as well as higher level features, all of which are needed for
construction robots. These architectures also promote reusability, which enables research and
development teams to build on one another’s work and move toward more autonomy in
construction machines.

Machine Awareness
The next design framework layer, machine awareness, is built on the machine architecture layer
which contains the transducers that convert machine and environment signals into electrical
signals. Machine awareness, conversion of sensor signals into knowledge about machine state
and work environment, is fundamental to producing autonomous machine behavior.
Autonomous machines must have awareness of their state and location, the work environment
including objects to be avoided and the shape of the terrain, and properties of the material that
the machine is processing. In addition, they must have machine health awareness. This
machine awareness layer mainly consists of localization and perception technologies.

For off-highway applications in agriculture and construction, localization is often accomplished

through a global navigation satellite system (GNSS) with inertial sensors. However, many
applications require the machine to follow some local path or more efficiently move within a job

Steward, Tang, and Han 185

Figure 3. The FroboMind architecture level consists of perception, decision making, and
action modules along with a separate safety module (Source: Jensen et al., 2014).

site. In these cases, machine localization using relative position sensors has advantages.
Included in this localization sub-layer are sensor fusion methods enabling more robust
localization through complementary sensors. Sensor fusion can extend localization when one of
the sensor signals is lost and can improve localization accuracy when various error sources
exist from any single sensor in the system.

Before a machine can be classified as autonomous, it must perceive its environment to carry out
its tasks effectively and safely. A primary goal of machine perception is machine safeguarding to
ensure safe operation of the machine. Obstacle detection, recognition, and avoidance are
typical steps in machine safeguarding. Perception algorithms and strategies are built on top of
the perception sensors in the hardware architecture to achieve safeguarding functions.

Both agricultural and construction machines need to perceive features of the environment with
which they are interacting. Agricultural machines need to interact with the crop, soil and field
topography to accomplish field operations. Construction machines also need to interact with the
soil and job site terrain. For autonomous operation, machine perception systems are needed
with the following capabilities: localization to determine where the machine is relative to the
world coordinate systems, object recognition of obstacles around the machine, navigation and
collision avoidance so that the machine can safely interact with its environment, and learning
and inference so that the perception system can solve new problems. Han et al. (2015)

Steward, Tang, and Han 186

reviewed perception sensors used for navigation and obstacle detection in autonomous
agricultural vehicles. Sensor technologies that have been investigated for this application
include vision sensors which are used to capture two-dimensional scene information, as well as
stereo vision and active 3D camera that add range information. Laser, Ladar, Lidar, Radar
sensors as well as ultrasonic sensors have been used for range detection and 3D
reconstruction of the scene. Many of these same sensors have been used for construction
automation for object detection, pile shape estimation, elevation measurement across terrains
(Singh, 1997, 2002). Sensors for the physical properties of soils have application in both
agricultural and construction domains (Rossel et al., 2011).

With any machine, failures or breakdowns will occur. Thus, the condition of the machine must
be monitored, and machine health awareness is needed to achieve machine autonomy. In a
human-operated machine, the operator is not only controlling the operation of the machine, but
is also monitoring the machine through visual, audio, or vibration cues to ensure that the
machine is functioning correctly. As machinery is automated, machine condition monitoring,
along with fault detection and diagnosis, also must be automated, although it can still have
some reliance on human intervention when a human operator is present. For driverless,
autonomous machines, machine intelligence to monitor machine health must be in place to
produce machine health awareness with no human assistance – a very big requirement for the
development of these machines. Machine health awareness requires a high degree of
intelligence, perhaps higher than all other requirements for an autonomous machine.

Central to health awareness are technologies often referred to as condition monitoring systems
or fault detection and diagnostic systems. Condition monitoring is typically part of an overall
maintenance strategy for a process, machine or machine system, which will involve a human
manager. It uses signals from a machine acquired with sensors to provide some indication of
the condition of machine components. Based on these signals and their changes over time, with
some signal processing and pattern recognition analysis, managers can make decisions about
what maintenance interventions should be taken and when they should be scheduled.

Implementing a machine health awareness system for an autonomous machine requires several
layers of technology, which can be structured in a format similar to the design framework
presented above (Figure 4). For machine health awareness, there first must be a hardware
layer consisting of sensors that are measuring physical signals known to be related to machine
component condition. Several sensing modes have been used for condition monitoring and will
be described below. Secondly, the signals from the sensors must be preprocessed to remove
abnormal signals and then processed to extract the features that are correlated to machine
component conditions. Next, fault detection applies automatic pattern recognition processes to
determine if a fault has occurred in the system. Generally this step involves finding deviations
from patterns associated with normal operation. Once a fault has been detected, it must be
diagnosed to identify what the fault is and what might have caused it.

The last layer might be the most important for an autonomous machine, i.e., to decide what
action should be taken next and then execute it. Several possible actions can be taken when a
fault occurs, including: (1) initiate a graceful shutdown and remain at current position, (2) stop
operations and move to a designated location for maintenance, (3) stop operations, alert remote
human supervisor for further instructions, or (4) continue operations, and send a warning
message to human supervisor. Blackmore et al. (2002) identified six safety modes similar to
those listed above.

Steward, Tang, and Han 187

Many sensing modes for machine condition monitoring exist (Khodabakhshian, 2013).
Temperature measurement can be used to detect increased friction in bearings that are moving
into a failure mode. Dynamic monitoring of vibration signals or acoustic signals associated with
rotating machines can find vibration signatures related to wear and machine life. Monitoring
internal wear debris or particle contamination of lubricants is another approach. While light
blockage particle sensors are available, they are typically not applied directly to off-road
machines because of cost and robustness limitations. Typically, oil is sampled from the machine
being monitored and analyzed in a laboratory setting, which does not lend itself to autonomous
machine health awareness. However, dielectric spectroscopic sensing technology for oil
contaminants shows promise for an on-line sensor to be used continuously during machine
operation (Kshetri et al., 2013), and could be applied to autonomous machines. Finally,
monitoring machine performance variables, such as power consumption or hydraulic pressure,
and searching for anomalies in those dynamic variables shows promise as a condition
monitoring approach. Craessarts et al. (2010) took this approach applying self-organizing maps
and neural networks to detect failures on a New Holland combine harvester.

The application of these technologies to both autonomous agricultural and construction

machines has limitations. Vibration analysis, for example, is more easily applied to rotating
machinery and is not as suitable to machines involving lateral motion or limited rotational
motion. Others are also better suited for more controlled machine operating environments found
in factories rather than in fields or job sites. While some off-highway machine monitoring
technologies exist, little has been done to more broadly monitor agricultural machine health.

Machine Control

Once an autonomous machine has awareness of its location, environment, and health, the
machine control layer must next be in place. For agricultural applications, machine control is
necessary to navigate the vehicle through the field and to control the implements accomplishing
field operations. In construction applications, the vehicle must be navigated along paths in the
job site and control the soil engaging tools such as buckets or blades or the construction
process such as compacting soil or paving. Agricultural examples of machine control are
presented below to provide insight in how this domain has developed machine control leading
toward machine autonomy.

Navigation control of agricultural machines is highly developed and has progressed through
several generations of automatic guidance technologies as applied to conventional agricultural
vehicles and implements. However, for smaller, next-generation field robots, research questions
exist since robotic vehicle platforms may provide additional degrees of mobility freedom,
through independent four wheel steering (4WS) and four wheel drive (4WD), that can be utilized
for novel navigation control strategies.

The main goal of navigation controls is to automatically guide or steer a vehicle along a path
and to minimize the error between the actual trajectory that the vehicle takes and the desired
path. Automatic guidance of mobile agricultural field equipment improves the productivity of
many field operations by improving field efficiency and reducing operator fatigue. The idea of
automatically guiding vehicles is by no means new, and relevant literature can be found from
several decades back (Parish and Goering, 1970; Grovum and Zoerb, 1970; Smith et al., 1985).
The launching of the Global Position System (GPS) in the early 1990’s led to research

Steward, Tang, and Han 188

 Figure 4. Machine health awareness requires several layers of technology.

investigating the use of GPS as a positioning system for automatic guidance (Larsen et al.,
1994; Elkaim et al., 1997; Griepentrog et al., 2006; Burks et al., 2013). Commercialization of
GPS-based automatic guidance occurred in the first decade of this century, and was adopted
very quickly to become one of the most highly adopted precision agriculture automation
technologies.

Agricultural robotic vehicles operate under environmental uncertainties and time-varying

parameters. External factors such as soil conditions also affect vehicle dynamic characteristics.
Both unpredictable internal perturbations and external disturbances create a great challenge. In
their early work to develop a self-tuning navigation controller for farm tractors, Noh and Erbach
(1993) used a variable forgetting factor in an adaptive steering controller based upon a
minimum variance control strategy to cope with nonlinear time-varying dynamics. More recently,
Gomez-Gil et al. (2011) developed two control laws: one for tracking straight lines and the other
for tracking circular arcs. These control laws were shown to have global asymptotic stability with
no singularity points.

Four-wheel-steering and four-wheel-drive designs provide maneuverability and traction control

advantages to a field robot. Tu (2013) reported on the development of a 4WD/4WS vehicle and
developed a sliding mode control-based robust navigation controller. Sliding mode control has
robustness to parameter perturbations and external disturbances, making it suitable for off-road
environments. Errors of 0.08 m and 0.13 m were observed for straight-line and curved trajectory
tracking, respectively, in field tests (Figure 5).

Navigation control for the guidance of construction equipment has different requirements. Often
3D control is required. For grading, the blade height is controlled along with the path of the
vehicle. However, there may be good cross-collaboration between agriculture and construction
motivated by the example of the agricultural robotics research community. Here, the vision of
autonomous systems opened up investigations into new machine forms which are feasible if a
human operator is no longer required. With the new machine forms come new navigation
control strategies. While this process may play out differently in the construction domain,

Steward, Tang, and Han 189

autonomy does open the machine form design space to the consideration of new forms that are
no longer designed around an operator.

Implement control has also been implemented commercially for various agricultural machine
operations. For example, in the case of liquid chemical application, chemical application rate
control was first developed and commercialized in the late 1970s, upon which variable rate
application systems were developed in the 1990s. Since that time, more and more aspects of
machine operations have been automatically controlled.

Implement control is also available so that the burden on the operator to control implement
settings can be moved to automatic control. This reduces stress and fatigue on the operator and
gives the operator freedom to take on more of a supervisory role of the overall machinery
system. In addition, implement control often leads to the reduction of errors in the field operation
such as turning on or off the seeding at the wrong location and overlap of adjacent swaths or
skips in chemical application. There are many implement control examples for field crop
machines.

(a) (b)

Figure 5. Robust navigation control of a small four-wheel drive/steer agricultural robot (a)
tracking over a U-shaped path (b) (Source: Tu, 2013).

A widely adopted control system application in precision agriculture is automatic section-control,

which ranks second only to automatic guidance technology in terms of its commercial success.
To implement section control, the width of a field sprayer boom is divided into multiple sections,
with individual sections controlled in an on/off fashion. Boom section control can enable more
efficient spraying by reducing pass-to-pass overlap as well as preventing application to off-
target areas. Each section is controlled independently of the rest of the system – based on the
section’s location within the field or canopy. These systems rely on a task computer to control
the desired state of each section based on the section’s location within the field, taken from
accompanying sensors and/or on-board maps. A main benefit of using the boom section-control
technology for a sprayer is the reduction in the of chemical application overlap. The savings
realized through the adoption of section-control technology is mainly based on the number of
control sections and the field shape (Luck et al., 2011).

Steward, Tang, and Han 190

The path to agricultural machine autonomy has occurred in a series of implement control
technologies assisting human operators to accomplish a field operation with great work quality
or higher productivity with lower operator fatigue. Similar applications can also be found in
construction machines such as machine control for intelligent compaction.
Machine Behavior
Machine behavior is the highest layer of the design framework and includes mission planning
and machine supervision. For a robot to be autonomous, it must exhibit behaviors similar to
those observed of humans. Some of these behaviors include planning by determining the best
plan of action to achieve a particular goal and supervision by monitoring the work environment
and making modifications to the planned actions based on new information.

Blackmore et al. (2007) promoted a structure for defining the behaviors field robots need to
perform agricultural operations autonomously. At the highest level, a field operation is the action
that a robot will carry out to meet the needs of a crops’ cultural practices. Within an operation,
certain tasks must be carried out – either deterministic or reactive. Deterministic tasks can be
planned before the operation starts, are goal-oriented to achieve the objective of the operation,
and can be optimized to best draw on the resources available. Reactive tasks are foreseen
responses to uncertain situations that may occur during the operation. They are captured in
terms of behaviors that the robot should do in response to new situations. For example, when
an unknown obstacle is perceived in the current path of the robot, the robot should behave
according to the type of obstacle. If a tree is perceived in the path, the robot could alter its path
to go around it. If an animal is detected in the path, the robot might wait until it moves away, or
produce stimuli to scare the animal away, or stop and seek guidance from a human supervisor.
An example deterministic task is field coverage where the robot covers a field by navigating
through a predetermined coverage path. Several examples of autonomous machine behavior
research in the agricultural domain are presented below.

Optimal Path Planning

For agricultural field operations, the goal is to cover the entire field with that operation, such as
tillage or planting. Determining the best path direction is the main goal of coverage path
planning. Whole fields can usually be covered by straight parallel paths with alternating
directions parallel to the optimal coverage path direction for each given field. Several
approaches to optimal path direction discovery have been investigated. The time and travel over
field surfaces associated with field operations should be minimized within constraints associated
with machine characteristics, field topography, and field operation-specifics characteristics. To
achieve these goals, optimized coverage path planning algorithms are needed for both planar
surfaces and fields with three dimensional terrain features.
Optimized Coverage Path Planning on Planar Surfaces
Research has been done on coverage path planning of planar surfaces, but results have some
limitations in being applied to agricultural fields. Following the longest edge of the field is a
simple strategy, but it is only suitable for fields with simple convex shapes such as a rectangles.
Fabret et al. (2001) framed the coverage path planning problem as a Traveling Salesman
Problem (TSP), and first chose a “steering edge” that provided the direction to guide successive
swaths. In the field headland, characteristic points were then collected. Those points were
connected by lines in the steering direction via an associated graph constructed by a TSP
solver.
Field boundary irregularities must be considered for general coverage path planning solutions.
Field decomposition has potential to further improve the efficiency of field operations before

Steward, Tang, and Han 191

determining the best path directions in fields, particularly those with irregular field boundaries.
The trapezoidal decomposition method has been investigated as an approach to field
decomposition (Berg et al., 1999; Choset et al., 1997). Oksanen and Visala (2009) explored
greedy search algorithms to find coverage paths of planar (2D) field surfaces. Their search
algorithm iteratively found the optimal trapezoidal field decomposition and path direction using a
split and merge strategy.
Jin and Tang (2010) developed an algorithm that optimally decomposed planar fields and
planned optimized operational patterns (Figure 6). Their algorithm used a geometric model
which represented the coverage path planning problem. Their algorithm produced better
solutions than farmers’ solutions and showed good potential to improve field equipment
efficiency on planar fields.

Figure 6. Example results from an optimized coverage path planning algorithm for planar
field surface. The inner polygons indicate non-traversable obstacles (Jin and Tang, 2006,
2010).
Optimized Coverage Path Planning on 3D Terrain
More factors must be considered when optimizing the coverage path over terrain with three-
dimensional (3D) topographic features. The main factors are headland turning, soil erosion, and
skipped area. Jin and Tang (2011) approached the problem by first developing an analytical 3D
terrain model with B-Splines surface fits to facilitate the computation of various path costs. Then
they analyzed different coverage costs on 3D terrains and developed methods to quantify soil
erosion and curving path costs of particular coverage path solutions. Similar to the planar field
approaches, they developed a terrain decomposition and classification algorithm to divide a field
into sub-regions with similar field attributes and comparatively smooth boundaries. The most
appropriate path direction of each region minimized coverage cost. A “Seed Curve” search
algorithm was successfully developed and applied to several practical farm fields with various
topographic features (Figure 7).

Optimized Vehicle Routing

After optimal field decomposition and coverage path planning, the vehicle route, which is the
sequence of an agricultural vehicle following individual paths, can be further optimized to
minimize the distance traveled in headland turning and improve field efficiency when performing
an agricultural operation. Bochtis et al. (2009) developed a mission planner based on an
algorithmic approach where field coverage planning was transformed and formulated, as a
vehicle routing problem (VRP), which was formulated as an integer programming problem.
Through this approach, non-working travel distance was reduced by up to 50% compared to the
conventional non-optimized method. They also incorporated different operational requirements

Steward, Tang, and Han 192

 (a) (b)
Figure 7. Elevation model for a field where terraces and valleys exist (a) along with the
results (b) of a path planning algorithm designed for optimized coverage of a 3D terrain
(Jin and Tang, 2011).

and produced a different field pattern for each particular operation, which were optimal in non-
working travel distance (Figure 8).

(a) (b)
Figure 8. Differences between traditional turning pattern (a) and optimized turning
pattern (b) from an optimized vehicle routing algorithm for a mowing operation (Bochtis
et al., 2009).

CONCLUSIONS
A design framework for autonomous machines was presented in this paper. While the
framework has emerged from the agricultural domain, its generality allows a broader application
to other domains. The layers of machine architecture, awareness, control, and behavior will
need to be developed for autonomous machines in any domain.

ACKNOWLEDGMENTS
This research was supported by Hatch Act and State of Iowa funds, and this paper is a paper of
the Iowa Agriculture and Home Economics Experiment Station, Ames, Iowa.

Steward, Tang, and Han 193

REFERENCES
Berg, M. D., M. V. Kreveld, M. Overmars and O. Schwarzkopf (2000). Computational Geometry.
2nd ed. Springer, New York, NY, USA.
Blackmore, B. S., S. Fountas, and H. Have (2002). Proposed system architecture to enable
behavioral control of an autonomous tractor. In: Zhang, Q. (ed.), Automation Technology
for Off-Road Equipment: Proceedings of the 2002 Conference, ASAE, St. Joseph, Mich,,
USA.
Blackmore, B. S., S. Fountas, S. Vougioukas, L. Tang, C. G. Sørensen, and R. Jørgensen
(2007). Decomposition of Agricultural Tasks into Robotic Behaviours. Agricultural
Engineering International: the CIGR Ejournal, 9.
Bochtis, D. D., S. G. Vougioukas, and H. W. Griepentrog (2009). A mission planner for an
autonomous tractor. Transactions of the ASABE, 52: 1429-1440.
Burks, T., D. Bulanon, K. S. You, Z. Ni, and A. Sundararajan. (2013). Orchard and vineyard
production automation. In: Zhang, Q. and F.J. Pierce (eds), Agricultural Automation:
Fundamentals and Practices, CRC Press, Boca Raton, FL, USA. pp. 149-204.
Choset, H. and P. Pignon (1997). Coverage path planning: the boustrophedon cellular
decomposition. In: Proceedings of International Conference on Field and Service Robotics,
Canberra, Australia.
Craessaerts, G., J. De Baerdemaeker, and W. Saeys (2010). Fault diagnostic systems for
agricultural machinery. Biosystems Engineering, 106: 26-36.
Dumpert, J. (2004). Hardware and software design for a system of autonomous highway safety
markers (Doctoral dissertation, University of Nebraska).
Elkaim, G., M. O’Conner, T. Bell, and B. Parkinson (1997). System Identification and Robust
Control of Farm Vehicles Using CDGPS. Proceedings of ION GPS, 10:1415-1426.
Fabret, S., P. Soueres, M. Taix and L. Cordessed (2001). Farmwork path planning for field
coverage with minimum overlapping. In: Proceedings of 2001 8th IEEE International
Conference, IEEE, Piscataway, New Jersey, USA.
Gomez-Gil, J., J. C. Ryu, S. Alonso-Garcia and S. K. Agrawal (2011). Development and
validation of globally asymptotically stable control laws for automatic tractor guidance.
Applied Engineering in Agriculture, 27: 1099-1108.
Griepentrog, H. W., B.S. Blackmore, and S. G. Vougioukas (2006). Positioning and Navigation.
In: Axel Munack (ed), CIGR Handbook of Agricultural Engineering, Vol. VI Information
Technology, ASABE, St. Joseph, Mich. USA. pp. 195-204.
Grovum, M. A. and G. C. Zoerb (1970). An automatic guidance system for farm tractors.
Transactions of the ASAE 13: 565 -576.
Han, S., Steward, B. L., and L. Tang. (2015). Intelligent Agricultural Machinery and Field
Robots. In Precision Agriculture Technology for Crop Farming, ed. Zhang, Q. CRC Press.
Jin, J. and L. Tang (2010). Optimal coverage path planning for arable farming on 2D surfaces,
Transactions of the ASABE, 53: 283-295.
Jin, J. and L. Tang (2011). Coverage path planning on three-dimensional terrain for arable
farming. Journal of Field Robotics, 28: 424-440.

Steward, Tang, and Han 194

Jensen, K., M. Larsen, S. H. Nielsen, L. B. Larsen, K. S. Olsen, and R. N. Jørgensen (2014).
Towards an open software platform for field robots in precision agriculture. Robotics, 3:
207-234.
Khodabakhshian, R. (2013). Maintenance management of tractors and agricultural machinery:
Preventive maintenance systems. Agricultural Engineering International: CIGR Journal, 15:
147-159.
Kramer, J., and M. Scheutz (2007). Development environments for autonomous mobile robots:
A survey. Autonomous Robots, 22: 101-132.
Kshetri, S., B. L. Steward, and S. J. Birrell (2014). Dielectric Spectroscopic Sensor for Particle
Contaminant Detection in Hydraulic Fluids. ASABE Paper Number 1914249, ASABE, St.
Joseph, Mich. USA.
Larson, W. E., G. A. Nielsen, and D. A. Tyler (1994). Precision navigation with GPS. Computers
and Electronics in Agriculture, 11: 85-95.
Luck, J.D., R.S. Zandonadi, and S.A. Shearer (2011b). A case study to evaluate field shape
factors for estimating overlap errors with manual and automatic section control.
Transactions of the ASABE 54: 1237-1243.
Noh, K-M. and D. C. Erbach. (1993). Self-tuning controller for farm tractor guidance.
Transactions of the ASAE 36: 1583-1594.
Oksanen, T., and A. Visala. (2009). Coverage path planning algorithms for agricultural field
machines. Journal of Field Robotics, 26: 651-668.
Parish, R. L., and C. E. Goering (1970). Developing an automatic steering system for a
hydrostatic vehicle. Transactions of the ASAE, 13: 523–527.
Pedersen, S. M., S. Fountas, H. Have, and B. S. Blackmore (2006). Agricultural robots – system
analysis and economic feasibility. Precision Agriculture, 7:295-308.
Quigley, M., K. Conley, B.Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A.Y. Ng (2009).
ROS: an open-source Robot Operating System. In: ICRA Workshop on Open Source
Software, 3: 5.
ROS (2015). ROS Core Components. Accessed at www.ros.org/core-components. Retrieved on
January 1, 2015.
Rossel, R. V., Adamchuk, V. I., Sudduth, K. A., McKenzie, N. J., & Lobsey, C. (2011). Proximal
soil sensing: an effective approach for soil measurements in space and time. Advances in
Agronomy, 113: 237-282.
Rowe S. and C. Wagner (2008). An Introduction to the Joint Architecture for Unmanned
Systems (JAUS). Technical Report. Cybernet Systems Corporation, Ann Arbor, Mich. USA.
Accessed at www.openskies.net/papers/07F-SIW-089%20Introduction%20to%20JAUS.pdf.
Retreived on Jan. 31, 2015.
Toledo, O. M., B. L. Steward, L. Tang, and J. Gai (2014). Techno-economic analysis of future
precision field robots. ASABE Paper No. 141903313, ASABE, St. Joseph, Mich. USA.
Tu, X. (2013). Robust Navigation Control and Headland Turning Optimization of agricultural
vehicles. Graduate Theses and Dissertation. Iowa State University, Ames, Iowa, USA.
Singh, S. (1997). The State of the Art in Automation of Earthmoving. ASCE Journal of
Aerospace Engineering, 10: 179-188.

Steward, Tang, and Han 195

Singh, S. (2002). State of the art in automation of earthmoving, 2002. Proceedings of the
Workshop on Advanced Geomechatronics.
Shi, J., Goddard, S., Lal, A., Dumpert, J., & Farritor, S. (2005). Global Control of Robotic
Highway Safety Markers: A Real-time Solution. Real-Time Systems, 29, 183-204.
Smith, L. A., R. L. Schafer, and R. E. Young (1985). Control algorithms for tractor-implement
guidance. Transactions of the ASAE 28: 415-419.
Torrie, M. W., D. L. Cripps, and J. P. Swensen. (2002). Joint Architecture for Unmanned Ground
Systems (JAUGS) Applied to Autonomous Agricultural Vehicles. In Zhang, Q. (ed.),
Automation Technology for Off-Road Equipment: Proceedings of the 2002 Conference,
ASAE, St. Joseph, Mich,, USA. pp. 1-12

Steward, Tang, and Han 196

Discrete Element Modelling (DEM) For Earthmoving
Equipment Design and Analysis: Opportunities and
Challenges
Mehari Tekeste, Assistant Professor
Department of Agricultural and Biosystems Engineering
Iowa State University
2356 Elings Hall
Ames, Iowa, 50011
[email protected]

ABSTRACT
Simulation of granular materials (soil, rocks) interaction with earthmoving machines provides
opportunities to accelerate new equipment design and improve efficiency of earthmoving
machine performances. Discrete Element Modelling (DEM) has a strong potential to model soil
and rocks bulk behavior in response to forces applied through interaction with machinery.
Numerical representation of granular materials and methodology to validate and
verify constitutive micro-mechanical models in DEM will be presented. In addition, how DEM
codes can be integrated to CAE tools such as multibody dynamics will also be discussed. A
case study of tillage bar-soil interaction was modeled in EDEM to predict tillage draft force and
soil failure zone in front of tool moving at 2.68-m/sec and depth of 102-mm. The draft force and
soil failure zone was predicted at 10% and 20% error from laboratory measured data.

Key words: soil (geomaterials)—discrete element modelling—off-road machinery—

calibration

BACKGROUND: SIMULATION BASED DESIGN USING DISCRETE ELEMENT MODELLING

Earthmoving machinery product development and verification tests involve interaction of
machine equipment with granular materials (soil, biomass and rock). The geomaterial properties
and their conditions impact the equipment performance in terms of productivity, efficiency and
durability. The traditional and iterative product development cycle of “Design--Physically build
prototype--Field test” is laborious, costly and time demanding. Virtual engineering tools have
potential to accelerate product design engineering and reduce field testing. Automation of
earthmoving system modelling architect (Cannon and Singh, 2002) will need a geomaterial-tool
interaction model component to simulate earthmoving equipment trajectory motion. Simulation
of soil-tool interaction is essential for virtual earth-moving product development and automation
of earth moving operations.

Simulation based tools consist of generating CAD geometry surface mesh, pre-processing,
material model, solver, post-processing and data analytics for engineering decision support.
Modelling geomaterial - tool interaction will require versatile material models, quick and easy
testing methods to generate data for model parameters calibration and numerical tool to solve
geomaterial-tool interaction responses. Generally there are two broad categories of
geomaterial-tool interaction problems 1) load loosening processes for instance in tillage tools,
soil flow from bulldozer blades, soil fill in loader buckets; and 2) load bearing process in soil to
vehicle tractive devices interactions where soil supports vehicle loads and helps generate

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
traction. The desired engineering objectives are generally to reduce energy expenditure during
cutting and tillage processes, maximize soil fill on buckets, easy soil flow from crawler blades,
maximize traction and optimal soil density for growing crops.

Discrete Element Modelling (DEM) has the potential to simulate soil-tool interaction and could
also be integrated in co-simulation with other systems modelling tools such as Finite Element
Analysis (FEA), Multi-Body Dynamics (MBD) and Computational Fluid Dynamics (CFD). DEM
formulation comprises numerical representation of the particle shape and size, assembly of
particles, constitutive micro-mechanics contact laws that defines force vs. displacement
relationships; and every time step contact detection and explicit numerical integration governed
by Newton’s law of motion (Cundall and Strack, 1997). Details in contact laws and their
formulation are available in literature (Walton and Braun, 1986; Luding, 2008; Cleary, 2010;
EDEM, 2011). The DEM contact laws originated based on Hertizian contact theory and now
there are advanced contact models that defines the relationship between forces and
displacement using material normal and tangential stiffness, coulomb friction coefficient,
damping coefficient, rolling resistance coefficient, cohesion/adhesion and bond parameters.

Researchers have shown the predictive capability of DEM to model behaviour of granular
materials since late 70’s after Cundall and Strack (1979). Some of the works related to
earthmoving include DEM modelling of wide cutting blade to soil interaction on scaled
experimental box to predict forces on crawler blades and soil flow in front of the cutting blade
(Shmulevich et al., 2007); simulation of hydraulic excavator digging process using confining
stress dependent DEM cohesive soil model (Obermayr et al., 2014); and DEM modelling of soft
ground cutterhead (4.2-m in diameter) predicting torque performance for different Tunnel Boring
Machine (TBM) cutterhead designs (Mongillo and Alsaleh, 2011).

OPPORTUNITIES AND CHALLENGES FOR USE OF DEM IN EARTHMOVING INDUSTRY

In theory DEM, a particle based modelling technique, is able to simulate particles interaction
with earth moving equipment and makes it ideal fit to integrate the tool into the engineering work
flow for product design and performance. DEM has shown to be a proven research and
development tool in manufacturing and process industry (Favier, 2011) especially where
particles are non-cohesive, resemble to spherical shape and the size of engineering problem
are manageable using desktop multi-core computers. With the continuous increase in
computing power and efficiency in parallel computing, DEM has started to become a useful tool
for large scale applications in mining and construction industries (Cleary, 2010; Mongillo and
Alsaleh, 2011).

This paper illustrates potential opportunities and challenges with DEM for earthmoving machine
virtual protyping construction equipment and automation process. The discussion in this paper
may apply to the interactions of crawler blade (over 3-m X 3-m blade width X blade height for
instance John Deere 1050K and CAT D9), bucket (with loader capacity over 6 M3 (Yd3) for
instance John Deere 844 and CAT 980) and ripper tine with geomaterials ranging from clay-
sized (less than 0.002-mm) to gravel (75-mm).

The challenges with utilizing DEM arise from the difficulty with particle based approximation of
geomaterial and its dynamic behaviour. Geomaterials have spatial-temporal variations in
conditions (wet to dry), wide range of particle sizes (clay to gravel size) and their response to
loading have stochastic bulk response behaviour. Approximation of these realistic geomaterial
type, size and conditions using DEM spheres and solving bulk geomaterial to large size

Tekeste, Mehari Z 198

machine interaction using micro-mechanics contact laws are either computationally prohibitive
or impossible to fit within reasonable engineering work flow. DEM based engineering analysis in
earth moving industry thus demands a methodology that is adaptive towards the desired
systems engineering, existing DEM code requirement and simulation process that will provide
industry value added results.

DEM USE FOR ENGINEERING DESIGN ANALYSIS

DEM simulation work flow for simple geometry motion, granular shape close to spheres and
small number of particles is fairly straight forward. For large scale industrial applications that
require DEM integrated into earth moving product development cycle and transient coupling
with other methods such as FEA, MBD and CFD for systems modelling, DEM simulation work
flow can be looked as a component of system engineering. The system layers proposed in
Figure-1 may apply to earth moving simulation based design and automation. This may capture
the requirements from the application, simulation and granular mechanics know-how to
systematically define DEM shape and size approximation, calibrate DEM particle model and
simulation of application with reduced uncertainty and controlled variance for design and system
optimization.

(1) Defining systems requirement and setup for material model

calibration
• Design and systems application requirements
• Material boundary condition
• Experimental test development
• Upscale benchmark test development (2) (1)
(2) Material model calibration and simplification for application
simulation
• Material model properties fit for the defined requirement
• DEM advance physics and co-simulation
• Computing efficiency (upscaling, simplification of
geometry boundary) (3)
(3) Application product simulation
• Accelerate virtual product development
• Accelerate computing within defined accuracy
• Data analytics and optimization
Figure 1. DEM for simulation in systems engineering

DEM Material Properties Development

The major steps in DEM particle model development consists 1) define the shape and particle
size representation; 2) identify micro-mechanics contact model that captures the expected
geomaterial behaviour (elastic, plastic, cohesive and non-cohesive); and 3) determine the
material model properties.

Shape and Particle Size Approximation

Shape and particle size approximation have strong influence on the dynamics of granular
system and computational effort. The shape of DEM assembly of particles affect packing of
particles (void ratio), shear strength, angle of repose and discharge from hoppers and relative
particle velocity on conveying (Cleary, 2010; Lu et al., 2015).

In DEM codes particle shape is approximated using spheres, glued (clumped) spheres and non-
spherical particles (for instance polyhedron, ellipsoids). Spherical DEM particle representation

Tekeste, Mehari Z 199

has benefit in terms of relatively easier contact detection algorithm, simpler overlap based force-
displacement calculation and reduced computational effort. According to Cleary (2010), the
computational cost with superquadrices was 2 to 3 times compared with spheres. Commercial
DEM codes such as EDEM from DEM solutions UK and Particle Flow Code (PFC) from Itasca
Consulting Group, Inc have overlap spherical capability and rolling resistance DEM models to
numerically reproduce mechanical interlocking effect on dynamics of granular systems. Lee et
al. (2012) used non-spherical primitive shape from polyhedral DEM code. Hohner et al. (2015)
studied various non-spherical particle shape approximation types (polyhedra, multi-sphere
cluster, superellipsoids) on hopper discharge and found out that shear strength of the particle
bed and discharge from hopper were affected by sphericity and aspect ratio but less on fine
scale resolutions of shape surface approximation. The choice of spherical or non-spherical
shape approximation depends on user know how on engineering problem, dimensional
measured properties of the geomaterial, bulk material behaviours, affordable computational
effort and availability of shape library in the DEM code. For most of natural geomaterials
(cohesive and non-cohesive soils) expect gravel, measurement of geometric shape parameters
such as sphericity, angularity, aspect ratio and surface roughness are difficult to obtain and
easily reproduce in DEM primitive shapes. For non-spherical and angular materials that exhibit
mechanical interlocking and affect their initial packing density, it is important to make significant
effort to use non-spherical DEM shape approximation using clumped sphere, non-spherical
primitive shapes or rolling resistance contact models.

Similar to shape representations, particle size approximation to real geomaterial size distribution
also influences DEM bulk material response behaviour and computation effort. The smallest
DEM particle size derives the explicit time step value for DEM calculation. Geomaterial DEM
modelling, it is computationally prohibitive to match equivalent clay, silt and sand size fractions.

DEM particle size scaling is thus necessary and can be done using linear scaling factor of DEM
mean particle size, particle size distribution or ratio number of particle to geometry (wall)
dimension. Lee et al. (2012) applied DEM particle size scaling by a factor of 10 times larger than
the experimentally measured sub-angular uniformly graded fine sand distribution and
normalized by D50. DEM simulation of triaxial compression with polyhedral particle shape, Lee et
al. (2012) successfully reproduced initial packing density and stress-strain of undrained triaxial
compression sand soil test.

Limited studies are available that investigate various DEM shape approximations and particle
scaling methodology and how they relate to achieve the desired quality of dynamics of bulk
material response behaviour and computational effort. Similarly studies are needed on
upscaling methodology from small size simulation used for calibration of DEM model
parameters, shape and size to simulation of large size earth moving applications.

Tekeste, Mehari Z 200

Experimental test and calibration methodology
Simulation of bulk (macro) geomaterial behaviour interaction with equipment using DEM
assembly of spherical or non-spherical particles and micro-mechanics laws can only be
achieved through calibration of the DEM parameters with measured response variables.
Methodology to generate DEM material properties fit for the approximation of bulk material
response to equipment interaction within relatively short time is often more relevant than to
measure individual granular particle to particle or particle to geometry model parameters.
As shown in Figure -1, the steps for system requirement and material properties development
will help to identify experimental tests and measured response variables for model calibration.

Soil mechanical properties (stress-strain, angle of internal friction, cohesion) obtained from
geotechnical ASTM standard tests (tri-axial, shear and others) can be good candidates. Other
experimental tests that reflect in-situ geomaterial behaviour and allow to measure multiple
dependent variables such as angle of repose, deformation, torque and forces will provide
enhanced accuracy for wider granular dynamics behaviour.

DEM calibration process involves first reproducing initial packing density (void ratio) of particle
assembly with estimated initial model parameters. DEM virtual experiments are then conducted
taking the model parameters as independent variables and response properties similar to the
experimental test as dependent variables. Besides to the material model parameters (stiffness
and coefficients), shape and size parameters can be added as independent variables during
calibration process. Reducing the number of model parameters for calibration is always helpful.
For instance in quasi-static engineering systems, determination of coefficient of restitution may
be less important than shear stiffness and friction coefficients. In dynamic system application for
instance in grind milling and transfer chutes where system performance is dependent on
collision energy losses characterization, coefficient of restitution is important. Sensitivity and
optimization scheme will then be deployed to generate calibrated DEM particle model and
properties to be used for application simulation.

Simulation of Application
Upscaling and computational accuracy
The main engineering value from DEM modelling is obtained from simulation of industrial
application using the calibrated DEM particle model. Depending on the simulation domain of
industrial application, the particle size or distribution used in the calibration process may need
scaling before using it for application simulation. Upscaling DEM particle model into large size
simulation will need know-how on particle: geometry system similarity and scale-invariance of
contact models (Feng et al., 2009). Interpretation of DEM results from virtual equipment design
changes may not necessarily eliminate the uncertainty/stochastic natural geomaterial behaviour
and their associated equipment performance. Having DEM results with acceptable variance and
showed value-added trends of improved performance from virtual equipment changes can be
considered successful outcome.

Some engineering design and analysis may require coupling with CFD, FEM and MBD. For
earth-moving application, coupling of DEM with MBD and FEM will be more applicable to
transfer transient loads from geomaterials into rigid or flexible multibody mechanical or hydraulic
driven systems for equipment (excavator buckets, blade) kinematics motion control and
structural stress analysis. DEM coupling with other tools will involve surface element position
mapping, interpolation and synchronization of sampling time (Favier, 2011) that will affect the
stability and accuracy. For automation of earth-moving operation with soil-tool interaction in the

Tekeste, Mehari Z 201

loop with actuator-control algorithms for equipment trajectory as proposed in Cannon and Singh
(2000) can be implemented using DEM coupling technique.

Predicting Tillage Force

Simple tillage-soil interaction was modelled in EDEM Academic (EDEM, 2011) to predict forces
and soil failure. EDEM model was developed with Hertz-Mindlin (with no slip) contact model to
replicate the tillage tool bar from linear soil bin test in Becker (2008). The test parameters from
Becker (2008) are shown in Table-1.

Table 1. Tool dimension and test parameters (Becker, 2008)

Dimension
Parameters
Value
Geometer 42
Width (mm) 50.8
Length (mm) 203.2
Thickness (mm) 12.7
Tool depth (mm) 102
Tool velocity (mm/sec) 268
Density (kg/m^3) 1760
Moisture content (%, d.b.) 9.15

Initial Feasibility Simulation using 5-mm and 10-mm particle size

Assembly of single sphere particle and two sizes (5-mm and 10-mm particle diameter) in
simulation soil box (length = 790-mm, width = 265-mm; and depth = 408-mm) in EDEM 2.7. The
total number in the soil box were 274494 and 36432 with 5-mm and 10-mm, respectively.
Baseline properties shown in Table-2 were used for DEM parameters of particle size sensitive.
Steel geometry was used for the tool and the soil box. The values for coefficient of static friction
was estimated from angle of internal friction coefficient of soil composition and bulk density
reported in Becker (2008). The soil-tool interaction parameter was assumed as 10% lower than
the soil:soil interaction. The other interaction parameters were best guess and quick sensitivity
EDEM runs (not reported). Particle were generated using EDEM factory creator filling the soil
box and compressed at 0.02m/sec to obtain stable and maximum bulk density. The bulk density
after relaxed DEM soil particle assembly for 5-mm and 10-mm was 1700 kg/m3 (3% lower than
the measured bulk density). Simulation was run for 0.19-sec with time step of 1.0e-06 sec using
12- CPU cores. Output variables tool forces (horizontal and vertical) and velocity to predict soil
failure zone in front of tillage tool were sampled at 0.01-sec interval.

Table 2. Base line DEM properties for Hertz-Mendlin (No Slip) (EDEM, 2011)
Poisson’s ratio 0.3
Shear modulus (Pa) 1e+06
Density (kg/m3) 2650
Soil:Soil Interaction
Coefficient of restitution 0.01
Coefficient of static friction 0.36
Coefficient of rolling friction 0.4
Soil:Steel Interaction
Coefficient of restitution 0.01
Coefficient of static friction 0.33
Coefficient of rolling friction 0.2

Tekeste, Mehari Z 202

The predicted horizontal (draft) force from 10-mm DEM particle was higher and closer to the lab
measured (Becker, 2008) draft forces (Mean = 416 N and Standard deviation = 36.9 N) than the
5-mm DEM particle. DEM elapsed computational time with 5-mm was 1.5 times greater than 10-
mm particle size. Thus to understand the sensitivity of Hertz-Mendlin contact model parameters
to predict force and soil failure flow, 10-mm particle size DEM model was used.

Figure 2. DEM predicted Horizontal (draft) force for the rigid flat bar for 5-mm and 10-mm.
The lab measured draft force was 416 N (Standard deviation = 36.9 N).

Sensitivity of Forces and Soil Failure to DEM model parameters

For the sensitivity study of soil:soil and soil:steel interaction parameters, four EDEM runs
represented as base line; HH = High:High; MM= Medium:Medium; and ML = Medium:Low
(Table-3) were simulated. The material parameters of density, poisson’s ratio and shear
modulus; and coefficient of restitution for the soil:soil and soil:steel were kept constant.
Assumption was made the change in coefficient of restitution may not influence the soil flow in
front of tool traveling at 2.68 m/sec.

Table 3. DEM parameters used for sensitivity study (Base line was considered as lowest
value; HH = High High; MM= Medium Medium; and ML = Medium Low showed ranking for
soil:soil and soil:steel interaction ranges).
Sensitivity Classes for EDEM
runs
DEM Parameters Base HH MM ML
Soil:soil Coefficient of static friction 0.36 0.90 0.60 0.60
Soil:soil Coefficient of rolling friction 0.40 0.90 0.60 0.60
Soil:steel Coefficient of static friction 0.33 0.90 0.60 0.30
Soil:steel Coefficient of rolling friction 0.20 0.90 0.60 0.30

The results for predicting horizontal force (draft) and soil failure from tool bar interaction with soil
are shown in Figure 3. Medium Low (ML) values DEM interaction parameters showed good
prediction (10% error) in horizontal force compared to the lab measured value. The soil failure
zone infront of the tool were better predicted with the base line properties at 20% error. The
DEM parameters providing better estimate for force prediction and soil flow are in different
ranges especially on the soil:soil particle interaction parameters. Further study is needed to
optimize DEM parameters for two conflicting objective functions. The shear modulus may affect
the force prediction and should be considered in next optimization steps.

Tekeste, Mehari Z 203

With the sensitivity study of DEM tool-soil interactions, bulk tool-soil interaction response
variables from simple tests can be used to obtain better estimate range of combination of DEM
parameters. The simple test used in this experiment captures soil response behaviour similar to
cultivator sweep (shovel) interaction for tillage operations.

This exercises demonstrates implementation of the value of adaptive system approach and
know-how of the engineering problem for calibration DEM parameters instead of depending on
individual DEM parameters measurement.

(A) DEM predicted Horizontal (draft) force

Soil failure zone

(B) DEM predicted soil failure zone

Figure 3. DEM predicted Horizontal (draft) force and soil failure zone from sensitivity
model parameters of 10-mm particle. The lab measured draft force was 416 N (Standard
deviation = 36.9 N) and soil failure zone was 133-mm.

CONCLUSION AND FUTURE STUDIES

x DEM based simulation of earth-moving equipment interaction with goematerials has
strong potential for off-road machinery industry to support virtual prototyping in design
and process improvements.

x Effective utilization of DEM for earth-moving requires adaptive system approach to

improve the inherent limitations of DEM in terms of trade-off in shape approximation and
particle size definitions versus real geomaterials; calibration methodology from micro-

Tekeste, Mehari Z 204

 mechanics to macro-experimental mechanical behaviour; and scaling principles in
mechanical, geometry and particle size from simple tests to application simulation sizes.

x Simulation of tillage bar-soil interaction was run in EDEM and showed the sensitivity of
Hertz-Mendlin contact model parameters for predicting horizontal tillage tool force and
soil failure zones. This demonstrates the value of adaptive system approach in utilizing
DEM model for tool-soil interaction problems.

x Future research studies are needed on “realistic” geomaterial shape approximate vs.
accuracy of dynamic granular behaviour, development of material tests fit for DEM
calibration purposes, robust calibration and optimization methodology for shape, particle
size and material models; and methods to evaluate application simulation output
uncertainty and variance for earth-moving virtual product development.

REFERENCES
Barker, M.E. (2008). Predicting loads on ground engaging tillage tools using computational fluid
dynamics. PhD. Dissertation. Iowa State University.

Cannon, H. and S. Singh (2000). Models for Automated Earthmoving, In P. Corke and J.
Trevelyan, Editors, Experimental Robotics VI, Lecture Notes in Control and Information
Sciences, Springer Verlag, Sydney. 183-192.

Cleary, P.W. (2010). DEM prediction of industrial and geophysical particle flows. Particuology
10:106-118.

Cundall, P.A. and O.D.L. Strack (1979). A discrete numerical model for granular assemblies.
Geotechnique 29:47-65.

EDEM (2011). EDEM theory reference guide. Edinburgh, UK: DEM Solutions.

Feng Y.T., K. Han, D.R.J. Owen and J. Loughran (2009). On upscaling of discrete element
models: similarity principles", Engineering Computations. 26 (6): 599 - 609

John Favier (2011). Using DEM for Engineering Design and Analysis: Opportunities and
Challenges. 6th International Conference on Discrete Element Modelling (DEM 6) Proceeding.
Golden, Colorda, USA. Pp 36-4.

Hohner D, S. Wirtz and V. Sherer (2015). A study on the influence of particle shape on the
mechanical interactions of granular media in a hopper using the Discrete Element Method.
Powder Technology 278: 286-305.

Itasca Consulting Group, Inc. (2015). PFC (Particle Flow Code in 2 and 3 Dimensions), Version
5.0, Documentation Set. Minneapolis: ICG.

Lee, S. J., Y.M.A. Hashash and E.G. Nezami (2012). Simulation of triaxial compression tests
with polyhedral discrete elements. Computers and Geotechnics 43: 92-100.

Luding S (2008). Cohesive, frictional powders: contact models for tension. Granular Matter 10:
235-246.

Tekeste, Mehari Z 205

Mongillo, G and M. Alsaleh (2011). Discrete Element Method to Predict Soft Ground Cutterhead
performance. In Rapid Excavation and Tunneling Conference Proceedings, 1058-1067. R.
Steve and R. Victor. Society for Mining, Metallurgy & Exploration, Inc. (SME) 12999 East Adam
Aircraft Circle, Englewood, Colorado 80112.

Obermayr M, C. Vrettos, P. Eberhard and T. Dauwel (2014). A discrete element model and its
experimental validation for the prediction of draft forces in cohesive soil. Journal of
Terramechanics 53: 93-104.

Shmulevich I., Z. Asaf and D. Rubinstein (2007). Interaction between soil and a wide cutting
blade using the discrete element method. Soil & Tillage Research 97: 37-50.

Walton, O.R. and R.L. Braun (1986). Stress Calculations for assemblies of inelastic spheres in
uniform shear. Acta Mechanica (63):73-86.

Tekeste, Mehari Z 206

Impacts of Automated Machine Guidance on
Earthwork Operations
Pavana K. R. Vennapusa, David J. White, Charles T. Jahren
Center for Earthworks Engineering Research
Department of Civil, Construction, and Environmental Engineering
Iowa State University, Ames, Iowa
Email: [email protected]; djwhite @iastate.edu; [email protected]

ABSTRACT
Use of automated machine guidance (AMG) that links sophisticated design software with
construction equipment to direct the operations of construction machinery with a high level of
precision, has the potential to improve the overall quality, safety, and efficiency of transportation
construction. Many highway agencies are currently moving towards standardizing the various
aspects involved in AMG with developing the design files to implementing them during
construction. In this paper, two aspects of AMG and their impacts on earthwork operations are
discussed. The first aspect deals with the estimation of earthwork quantities and its impact on
productivity on costs. The second aspect deals with the factors contributing to the overall
accuracy of AMG. These two aspects are discussed in this paper using survey responses from
various AMG users (contractors, agencies, software developers, and equipment manufacturers)
and some experimental test results. Both these aspects are critical to understand during
implementation of AMG as these have productivity and cost implications to the users.

Keywords: automated construction—earthwork quantities—accuracy—specifications—

GPS—AMG

INTRODUCTION
Currently, highway agencies are improving electronic design processes that support
construction with automated machine guidance (AMG) and deliver higher quality products to the
public. Equipment providers are rapidly advancing software tools and machines systems to
increase automation in the design and construction process. Motivation to more widely adopt
AMG processes therefore exists. However, the framework for adoption of AMG into the complex
framework of design to construction has not been fully developed. Technical, equipment,
software, data exchange, liability/legal, training, and other barriers, limits progress with AMG
implementation into construction projects. To address these issues, a national level study was
initiated by the Transportation Research Board as the National Cooperative Highway Research
Program (NCHRP) 10-77 study.

In this paper, two specific aspects of AMG that directly influences the earthwork operations are
discussed. The first aspect deals with earthwork quantity estimation using AMG. The second
aspect deals with accuracy of AMG and the various factors that contribute to errors in the AMG
process. Both these aspects are critical to understand in a practical perspective as these have
productivity and cost implications to the contractors and agencies. In the following, each of
these aspects are separately discussed by presenting results of a national survey conducted
with over 500 participants from agencies, contractors, equipment vendors, software vendors,
and some experimental tests conducted by the authors. Survey results of selected questions
are presented herein for brevity, and all results are available in White et al. (2015).

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
EARTHWORK QUANTITY ESTIMATION
Earthwork pay items are historically objects of great dispute between agencies and contractors.
Proper use of digital information for AMG will likely result in less confusion and more accuracy
than traditional methods of earthwork pay item quantification and payment. According to the
survey responses (see White et al. 2015), a majority of the survey responding contractors
currently use DTMs for estimating quantities, means and methods, constructability, quantity of
the progress of work, and payment. Earthwork pay quantification from AMG must include
mechanisms that all parties in the contract (both the agency-owner and the contractor) can trust.
The efficient use of digital information in AMG applications typically involves creation of a digital
terrain model (DTM) during initial planning, which is then passed to the design phase for
addition of design data in a 3D model. This facilitates efficient computation and measurement of
earthwork quantities for use during the procurement phase (bidding). Finally, the construction
phase involves verification of project as-built quantities.

Impact of AMG on Productivity Gain and Cost Savings

Figure 1 presents responses from contractors and vendors on the impact of AMG on
productivity gain and project cost savings. A majority of the equipment vendors indicated
potential productivity gain of about 40% and potential cost savings of about 25 to 40% using
AMG. On the other hand, a majority of the contractors indicated potential productivity gain of
about 10 to 25% and potential cost savings of about 10 to 25% using AMG. Productivity gain
and cost savings reported in the literature on earthwork construction projects using AMG is also
presented in Figure 1 (Jonasson et al., 2002; Aðalsteinsson, 2008; Forrestel, 2007; Higgins,
2009; Caterpillar, 2006).

Jonasson et al. (2002) reported productivity gain and cost savings information for a fine grading
project using a motorgrader with different position measurement technologies (i.e., ultrasonic’s,
2D and 3D lasers, and GPS). The productivity gain ranged from about 20 to 100% and cost
savings ranged from about 15 to 40%, depending on the position measurement technology
used. The cost savings were due to a reduction in surveying support and grade checking, an
increase in operational efficiency, and a decrease in number of passes. Their study indicated
that the 3D laser systems required a direct line of sight to the equipment while the GPS systems
did not, which resulted in a small increase in fleet productivity and a decrease in unit cost using
GPS guidance systems over 3D laser systems.

Aðalsteinsson (2008) reported results from a field demonstration project conducted using an
excavator to excavate a trench with 1650 cubic yards of sandy gravel material. In his study, the
AMG approach showed a productivity gain of about 25% over a no AMG approach. Caterpillar
(2006) reported results from a field demonstration project conducted in Spain by constructing
two 80 m identical roads: one road with AMG on construction equipment and the other with
similar equipment but using conventional methods and no AMG. AMG was used for bulk earth
moving and fine grading work. An overall productivity increase of about 101%, fuel cost savings
of about 43%, and increased consistencies in grade tolerances were reported for this project.

The results from these field case studies and survey responses indicate that the productivity
gain and cost savings using AMG on earthwork projects can vary significantly (with productivity
gains in the range of 5% to 270% and cost savings in the range of 10% to 70%). This variation
is most likely because of various contributing factors, such as project conditions, materials,
application, equipment used, position measurement technologies used, and operator
experience.

P. Vennapusa, D. White, and C. Jahren 208

 12
3D Laser1/ Notes: 1Fine-grading using CAT 140H motor grader
GPS1 (89 to (Jonasson et al., 2002)
2
10 Ultrasonic1 GPS6 2D Laser1 101%) Trench excavation using CAT 330DL hydraulic
excavator (Aðalsteinsson, 2008)
2 4 3
GPS GPS GPS3 Earth moving and fine grading (general values;
8 GPS5 not project specific) (Forrestel, 2007)
Frequency
4
Contractors (107%) Earth moving and fine grading project - Port
Vendors of Brisbane (Higgins, 2009)
6
GPS7
(265%) 5
Bulk earth moving and subgrade fine grading using
4 CAT D6N dozer (gain in the number of passes;
Caterpillar, 2006)
6
Bulk earth moving using CAT 330D excavator
2 (Caterpillar, 2006)
7
Base course fine grading using CAT 140H motorgrader
0 (gain the number of passes; Caterpillar, 2006)
10 20 30 40 50 >50
Productivity Gain Using AMG (%)
12
GPS5 Notes: 1Fine-grading using CAT 140H motor grader - Overall
GPS2 Ultrasonic1 2D Laser1 3D Laser1 (68%) unit cost (Jonasson et al., 2002)
2
10 Earth moving and fine grading project - Port of
GPS1
GPS4 GPS3 Brisbane (overal cost savings) (Higgins, 2009)
3
8 Bulk earth moving using CAT 330D excavator - fuel
Frequency

cost savings (Caterpillar, 2006)

4
Contractors Bulk earth moving and fine grading using CAT D6N
6 Vendors dozer - fuel cost savings (Caterpillar, 2006)
5
Fine grading using 140H motorgrader - fuel cost
savings (Caterpillar, 2006)
4

0
10 20 30 40 50 >50
Cost Savings Using AMG (%)

Figure 1. Survey Responses by Contractors and Vendors and Productivity Gain and
Potential Cost Savings using AMG, and Data obtained from Field Case Studies

Earthwork Quantity Computation and Measurement

Accuracy of DTMs
Survey results reported by White et al. (2015) indicated that a majority (> 70%) of contractors,
software/hardware vendors, and agencies who responded believe that the number of elevation
data points used in creating the DTM is an important factor in the accuracy of the DTM.
Evaluating the accuracy of DTMs by comparing them to the actual surface is a challenging and
expensive task.

Various interpolation methods are available in the literature for generating contour grid data for
DTMs, which include: (a) inverse distance to power; (b) Kriging; (c) local polynomial; (d)
minimum curvature; (e) nearest neighbor; and (f) triangulated irregular network (TIN). To study
the influence of the number of data points, three different data sets, with 78, 38, and 11 data
points, were captured over a 540 m2 area. The area consisted of a sloping terrain with an
elevation difference of about 3.5 m over 60 m length. DTMs were generated using the six
different interpolation methods described above. DTMs generated from 78 data points are
presented in Figure 2.

The accuracy of each DTM that used 78 data points was evaluated using a cross-validation
technique. This technique involved taking out a known data point from the data set, estimating
the point using the model, and comparing the estimated value with the actual one. This process
was repeated for all 78 data points. An absolute mean error (calculated as the average of
absolute value of the difference between the actual and the estimate value) was then calculated
for each interpolation method, as summarized in Table 1.

P. Vennapusa, D. White, and C. Jahren 209

 Figure 2. DTMs of a 540m2 area using 78 elevation data points using different
interpolation methods: (a) inverse distance to a power; (b) kriging; (c) local polynomial;
(d) minimum curvature; (e) nearest neighbor; (f) TIN

Table 1. Absolute mean error of estimated elevation data based on cross-validation

process using different interpolation methods
Estimated Elevation
Data Interpolation Method Absolute Mean Error (mm)
Inverse distance to power 100
Kriging 20
Local polynomial 70
Minimum curvature 50
Nearest neighbor 40
Triangulated irregular network (TIN) 30

For this data set, results indicated that the Kriging method is the most accurate method with
0.02 m absolute mean error. The TIN method showed a slightly higher absolute mean value of
0.03 m. The grid generated using the Kriging method with 78 data points was then considered
as a “true” representative surface, and it was used as a comparison to the grid data generated
using the other interpolation methods, as summarized in Table 2. The Kriging method produced
absolute mean error of 0.02 m using 38 data points and 0.05 m using 11 data points. The TIN
method produced slightly higher absolute mean error values. Minimum curvature, local
polynomial, and inverse distance to power methods produced greater absolute mean error
values, compared to the TIN method. The nearest neighbor method could not replicate the
surface terrain, as it doesn’t interpolate the data, which is clearly a limitation of the method.

It is important that existing surfaces are portrayed as accurately as possible, so the model can
be passed ahead to the design, estimation, bidding, and construction phases of the project with
high fidelity. A proper understanding of the factors that influence the accuracy of the DTM is
important to understand and must be addressed during the model development phase.

P. Vennapusa, D. White, and C. Jahren 210

Table 2. Absolute mean error of estimated elevation data by comparing Kriged DTM with
79 points with different interpolation methods
Estimated Elevation
Absolute Mean Error (m)
Data Interpolation Method 79 Data Points 38 Data Points 11 Data Points
Inverse distance to power 0.06 0.10 0.11
Kriging 0.00 0.02 0.05
Local polynomial 0.06 0.07 0.07
Minimum curvature 0.03 0.04 0.09
Nearest neighbor 0.06 0.12 0.24
Triangulated irregular network (TIN) 0.01 0.04 0.06

Computation of Earthwork Quantities

Earthwork quantities are traditionally computed using the average-end-area method, which is
based on averaging the areas of two consecutive cross-sections and multiplying the average by
the distance between them (Burch, 2007). A polar planimeter is typically used by surveyors to
measure the area by tracking the boundaries.

Using DTM, the surface-surface method can be used to compute quantities, by overlapping the
existing terrain and the design DTM surfaces. The U.S. Army Corps of Engineers (2004)
provides a detailed explanation of the surface-surface quantity estimation method using TIN
surfaces. Many software applications (including Microstation and Autodesk) now have the
capability to easily compute quantities using the surface-surface method. The accuracy of the
generated DTM, as described above, plays a significant role in the estimated earthwork
quantities. Soil shrink-swell factors also affect to the overall quantity estimation, which are
dependent on the soil type, so they must be selected appropriately (Burch, 2007).

Vonderohe et al. (2010) reported that differences between average-end-area and surface-
surface increases as the cross-section levels increase, although the relationship is not linear. As
the cross-section intervals decrease, the computations become theoretically the same. The
differences are observed to be as great as 5% when 100 ft cross-section intervals are used with
the average-end-area method. Such differences can contribute to significant cost discrepancies
for large projects. The advantage of using DTMs is that earthwork quantities can be computed
“on the fly,” as the model is being developed, and also during construction. Various layers and
volumes that represent various bid items and various costs can be collected and categorized
during the design process. Designed surfaces are accurately portrayed and can be passed
ahead in the AMG process with high fidelity.

Model Enhancement for Construction Purposes

Model enhancement might be necessary during the development process for certain aspects,
such as providing offsets between pavements and subgrades, delineating areas where
equipment operation is excluded, and correcting inconsistencies that are not problematic for
design models but are for AMG. The benefits of this work phase are that the constructor may
discover possible design improvements or design errors in the model, which can end up saving
time and money during construction. The constructor may develop a better understanding of
how to construct the project as the design model is enhanced. The constructor could improve

P. Vennapusa, D. White, and C. Jahren 211

construction productivity and safety by adding exclusion zones for equipment and methods to
track equipment usage during construction.

Model Conversion to AMG Format

It may be possible to develop automatic load counts and infer earthwork or pavement volume or
tonnages moved by equipment using onboard weight detection. A data collection method could
be developed to infer current elevations of partially-completed projects by knowing current
equipment elevations. This information may be used to monitor current earthwork volumes for
partial payment.

Model Conversion to QA/QC Format

QA/QC personnel can potentially use DTM and the final design model to automatically locate
test locations and display results. Elevations of existing surfaces can be obtained quickly and
modeled in 3D to estimate current earthwork and pavement volumes or tonnages for partial
payments. Quality information is processed along with volume information to ensure that partial
payments are made for earthwork or pavement that meets quality requirements.

Limitations
The limitations in all of the above, however, include potentially higher up-front costs for
software, hardware, and highly-trained personnel, and the possible inability to make gut-level
checks for some types of design errors. Downstream personnel may be critical of design
personnel for alternative designs that were not used and documented in unused parts of the
model. Designers may consider inspection of the details of the design process by downstream
personnel to be too invasive of their professional autonomy.

ACCURACY OF AMG PROCESS DURING CONSTRUCTION

The accuracy of the AMG process during construction is primarily influenced by three variables:
(a) position measurement technology; (2) construction processes; and (c) human errors.

Survey responses from surveyors and planners indicated total station surveying (robotic and
conventional) is considered more accurate than GPS and photogrammetric surveying.
Manufacturers and researchers have published the precision and accuracy values of various
position measurement technologies in the technical literature (Peyret et al., 2000; Retsher,
2002; Barnes et al., 2003; Mautz, 2008; and Trimble, 2008).

It does not appear that the effect of construction process and human errors has ever been
thoroughly studied or quantified. Most contractors, vendors, and agency personnel who
responded to the survey questions reported that these variables play a major role in the overall
accuracy of the AMG process.

Position Measurement Technologies

Table 3 provides a summary of accuracy, coverage range, measurement principle, and relative
cost of different position measurement technologies that are typically used in construction
applications. The laser or ultrasonic technologies offer higher vertical (elevation) accuracies
than GPS and have shown success in achieving tighter tolerances on some fine grading
projects (Daoud, 1999). However, laser or ultrasonic technologies have some practical
limitations with use in rain, dust, wind, and snow, and need frequent charging of deep cell

P. Vennapusa, D. White, and C. Jahren 212

batteries (Cable et al., 2009). These technologies also require a direct line of sight between the
control station and the receiver on the equipment, which is why they have not been used on
heavy earth moving equipment, other than motor graders (Jonasson et al., 2000).

GPS-based technologies can overcome the limitations stated above with laser and ultrasonic
technologies, but they don’t offer high vertical accuracy. Peyret et al. (2000) noted that RTK
GPS systems normally have vertical accuracy (±2 cm) or twice the horizontal accuracy (±1 cm).
A vertical accuracy level of ±2 cm is not sufficient for applications such as paving or fine
grading. Another common problem reported with GPS-based technologies is limited availability
of satellites (and, consequently, poor signal attenuation) when operating close to structures,
trees, or underground environments. Currently, the U.S. Air Force is committed to maintaining
availability of 24 operational GPS satellites, 95% of the time (U.S. Air Force 2014) and is
projecting for increased number of satellites in the future. The relative gain in accuracy from an
increased number of satellites may be marginal (Hein et al. 2007), however, AMG users can
expect to increase the chances of having the minimum number of satellites required to achieve
a certain amount of accuracy because of the new additional satellites.

Recent advancements with use of HA-NDGPS with initiatives from FHWA, globally positioned
GDGPS and IGS technologies is providing opportunities to achieve cm level accuracy without
significant on-site investment. U.S. Air Force is currently in the process of developing and
launching a next-generation GPS satellite (GPS III) which will be available for all military and
civilian applications with improved accuracies (U.S. Air Force 2014).

GPS with laser or ultrasonic augmentation offers improved vertical accuracies (2 to 6 mm)
(Trimble, 2008). From recent field studies on concrete paving projects in Iowa, Cable et al.
(2009) found that laser-augmented GPS measurements are somewhat capable of guiding the
paver and controlling elevation to achieve a reasonable profile for low-volume roads, but
recommended that improvements (or fine tuning) in software is required to better control the
elevation that will result in smoother surface profiles.

Construction Process and Human Errors

The overall accuracy of the AMG process includes these construction process parameters: (a)
speed of operation; (b) direction of travel; (c) terrain; and (d) material type and support
conditions (uniformity). These parameters have not been thoroughly studied or documented in
the technical literature and they are application-specific or machine-specific. A statistical
approach to quantify the influence of these factors on the overall accuracy of the AMG process
is presented in White et al. (2015).

The level of impact for each of these factors differs with the application type. Speed of operation
affects AMG accuracy and overall project costs. Increasing speed decreases the ability of
machines to react to error signals and, consequently, reduces the accuracy of the
measurement. However, productivity declines as speed declines, impacting project costs. The
effect of speed of operation is clearly interlinked with the abilities of the position measurement
technology feedback response time. The terrain on a job site can have an impact. Although not
critical for paving and fine grading applications, terrain can be critical for general earthwork and
excavation applications.

The type of material and support conditions under the equipment (whether stable or unstable,
uniform or non-uniform) impacts the overall accuracy. Unstable or non-uniform support

P. Vennapusa, D. White, and C. Jahren 213

conditions under the equipment make it more difficult to maintain control relative to the
reference. This factor can play a critical role in paving and fine grading applications, and may
not be as critical for general earthwork and excavation applications.

Table 3. Summary of Different Position Measurement Technologies

System Accuracy Range User Cost Reference
Conventional GPS (no
Variable, > 5 m Global Low DoD, 2008
corrections)
Assisted GPS (via mobile
Variable, 2 to 10 m Global Low Mautz, 2008
phones)
GPS integrated with INS Variable Global Variable Mautz, 2008
Wide Area Augmentation
1.6 to 3.2 m
System (WASS) or Satellite
horizontal and 4 to Global Low FAA, 2008
Based Augmentation
6 m vertical
System (SBAS)
1 m within 150 km
Nationwide differential GPS ARINC Inc.,
of the broadcast Global Low
(NDGPS) 2008
site
Low –
10 cm horizontal
HA–NDGPS Global currently in FRP, 2012
and 20 cm vertical
development
Global DGPS 10 cm horizontal Global Low NASA, 2014
International GNSS Service <10 cm horizontal
Global Low Moore, 2007
(IGS) and vertical
Moderate to
RTK GPS cm Global Mautz, 2008
high
Barnes et al.,
Locata (pseudolites) 6 mm 2 to 3 km High
2003
Moderate to
Laser- augmented GPS 3 to 6 mm 300 m/line of Trimble, 2008
high
site radius of
Low to
Laser r2 mm laser source Retscher, 2002
moderate
700 m/line of
Robotic total station r2 mm site radius of High Retscher, 2002
source
Immediate Low to
Ultrasonic r1 mm Trimble, 2008
reference moderate
Immediate Moderate to
Ultrasonic augmented GPS r1 mm Trimble, 2008
reference high
Kraut-
Infrared laser 0.1 to 0.2 mm 2 to 80 m High
Schneider, 2006

SUMMARY OF KEY FINDINGS

Impacts of AMG on Earthwork Quantities
x Earthwork pay quantification from AMG must include mechanisms that all parties to the
contract (both the agency-owner and the contractor) can trust.
x The accuracy of the generated DTM plays a significant role in the estimated earthwork
quantities. Experimental test results documented herein indicated that the interpolation
model and the number of data points both affect the accuracy of the DTM.
x Model enhancement might be necessary during the development process for certain
aspects, such as providing offsets between pavements and subgrades, delineating areas

P. Vennapusa, D. White, and C. Jahren 214

 where equipment operation is excluded, and correcting inconsistencies that are not
problematic for design models but are for AMG.
x The results from case studies described in the literature and survey responses indicate that
the productivity gain and cost savings using AMG on earthwork projects can vary
significantly because of various contributing factors, such as project conditions, materials,
application, equipment used, position measurement technologies used, and operator
experience.
x It is important that existing surfaces are portrayed as accurately as possible, so the model
can be passed ahead to the design, estimation, bidding, and construction phases of the
project with high fidelity. A proper understanding of the factors that influence the accuracy of
the DTM is important to understand and must be addressed when developing the model.
Accuracy of AMG Process
x AMG component accuracies is an issue that affects various stages of the process including:
Initial data collection for developing existing surface terrains; development of DTM and EED,
AMG processes, procedures, and end-user competencies, QA/QC reported practices, heavy
and fine grading equipment operations, and paving equipment operations.
x A common problem reported with GPS-based technologies is limited availability of satellites
(and, consequently, poor signal attenuation) when operating close to structures, trees, or
underground environments. Currently, the U.S. Air Force is committed to maintaining
availability of 24 operational GPS satellites, 95% of the time and is projecting for increased
number of satellites in the future. While the relative gain in accuracy from an increased
number of satellites may be marginal (Hein et al., 2007), AMG users can expect to increase
the chances of having the minimum number of satellites required to achieve a certain
amount of accuracy because of the new additional satellites.
x The overall accuracy of the AMG process includes various construction process parameters:
speed of operation, material type and support conditions (uniformity), and terrain. These
parameters have not been thoroughly studied or documented in the technical literature and
they are application-specific or machine-specific.

REFERENCES
ARINC Inc. (2014). NDGPS Assessment Report, Final Report, Prepared by ARINC Inc. for
Operations Research and Development, Federal Highway Administration, McLean, VA.
Aðalsteinsson, D.H. (2008). GPS Machine Guidance in Construction Equipment, BSc. Final
Project Report, School of Science and Engineering, Háskólinn Í Reykjavík University,
Iceland.
Barnes J., Rizos, C., Wang, J., Small, D., Voigt, G., Gambale, N. (2003). “LocataNet: A New
Pseudolite-Based Positioning Technology for High Precision Indoor and Outdoor
Positioning.” Proceedings of the 16th International Technology meeting of the Satellite
Division of the U.S. Institute of Navigation, Portland, OR.
Burch, D. (2007). Estimating Excavation, Craftsman Book Company, Carlsbad, CA.
Cable, J.K., Jaselskis, E.J., Walters, R.C., Li, L., and C.R. Bauer. (2009), “Stringless Portland
Cement Concrete Paving.” Journal of Constr. Engrg. and Mgmt., 135(11), p.1253-1260.
Caterpillar. (2006). Road Construction Production Study, Malaga Demonstration and Learning
Center, Spain <http://www.trimble-productivity.com/media/pdf/ProductivityReport
CATRoadConstruction2006.pdf> (accessed June 2010).
Daoud, H. (1999). “Laser Technology Applied to Earthworks.’’ 16th IAARC/ IFAC/IEEE
International Symposium on Automation and Robotics in Construction, Universidad Carlos
III de, Madrid, Madrid, Spain, Proceedings C. Balaguer, ed., p. 33–40.

P. Vennapusa, D. White, and C. Jahren 215

DoD. (2008). Global Positioning System Standard Positioning Service Performance Standard,
4th Edition, Published by the Positioning, Navigation, and Timing Executive Committee,
Department of Defense, Washington, D.C.
FAA (2008). Global Positioning System Wide Area Augmentation System (WAAS) Performance
Standard, 1st Edition, Published by the Federal Aviation Administration, Washington, D.C.
Forrestel, R. (2007). “3D Models for Machine Guidance Systems,” Presented at the 2007
Highway Engineering Exchange Program International Conference (IHEEP), Albany, NY.
FRP (2012). 2012 Federal Radionavigation Plan, Published by the Department of Defense,
Department of Homeland Security, and the Department of Transportation, National
Technical Information Service, Springfield, VA.
Hein, G., Rodriguez, J., Wallner, T., Eissfeller, B., and P. Hartl. (2007). “Envisioning a Future
GNSS System of Systems—Part 2.” Inside GNSS, 2(2), p. 64–72.
Higgins, M. (2009). “Positioning Infrastructures for Sustainable Land Governance,” Presented at
the FIG-World Bank Conference, Washington, D.C.
Jonasson, S., Dunston, P.S., Ahmed, K., and J. Hamilton. (2000). “Factors in Productivity and
Unit Cost for Advanced Machine Guidance.” Journal of Construction Engineering and
Management, 128(5), p. 367-374.
Kraut-Schneider, R. (2006), Untersuchungen zur Leistungsfähirkeit des Messystems Indoor
GPS, Diploma Thesis, University of Applied Sciences Karlsruhe, Baden-Wuttemberg,
Germany.
Mautz, R. (2008). “Combination of Indoor and Outdoor Positioning.” 1st International Conference
on Machine Control and Guidance, Zurich, Switzerland.
Moore, A. W. (2007). “Innovation: The international GNSS service – Any questions?”, GPS
World, 58-64.
NASA (2014). The Global Differential GPS System, Jet Propulsion Laboratory at the California
Institute of Technology, National Aeronautics and Space Administration (NASA), Pasadena,
CA. <http://www.gdgps.net/> (Accessed December 2014)
Peyret, F. Betaille, D., and Hintzy, G. (2000). “High-precision applications of GPS in the field of
real-time equipment positioning.” Autom. Constr., 9(3), 299-314.
Retscher, G. (2002). “Multi-sensor systems for machine guidance and control.” FIGXXII Intl.
Congress Meeting, April 19-26, Washington, D.C.
Trimble (2008). Trimble construction solutions for heavy and highway. http://www.trimble-
productivity.com/media/pdf/trimbleHeavyHighwaysSolutions.pdf (accessed May 11, 2010).
U.S. Air Force (2014). Space Segment – Constellation Arrangement, National Coordination
Office for space-Based Positioning, Navigation, and Timing, U.S. Air Force, Washington,
D.C. <http://www.gps.gov/systems/gps/space/> (accessed December 2014).
U.S. Army Corps of Engineers (2004). Engineering and Design Hydrographic Surveying: EM
1110-2-1003, Department of the Army, U.S. Army Corps of Engineers, Washington, D.C.
Vonderohe, A, Hintz, C., and A. Hanna, (2010). 3D Design Terrain Models for Construction
Plans and Control of Highway Construction Equipment, CFIRE 02-05, National Center for
Freight & Infrastructure Research & Education, Madison, WI.
White, D.J., Jahrent, C., Vennapusa, P., Westort, C., Alhasan, A., Turkan, Y., Guo, F., Hannon,
J., Sulbaran, T., and Dubree, A. (2015). Use of the Automated Machine Guidance (AMG)
within the Transportation Industry, NCHRP 10-77 Draft Final Report, National Cooperative
Highway Research Program, Washington, D.C. (in review).

P. Vennapusa, D. White, and C. Jahren 216

Robotic Hybrid Data Collection System Development
for Efficient and Safe Heavy Equipment Operation

Chao Wang, Ph.D.

School of Civil and Environmental Engineering
Georgia Institute of Technology
790 Atlantic Dr.
Atlanta, GA, 30332
[email protected]

Yong K. Cho, Ph.D.

School of Civil and Environmental Engineering
Georgia Institute of Technology
790 Atlantic Dr.
Atlanta, GA, 30332
[email protected]

ABSTRACT
This paper introduces a framework of automatic object recognition and rapid surface modeling
to aid the heavy equipment operation in rapidly perceiving 3D working environment at dynamic
construction sites. A custom-designed data acquisition system was employed in this study to
rapidly recognize the selected target objects in a 3D space by dynamically separating target
object’s point cloud data from a background scene for a quick computing process. A smart
scanning method was also applied to only update the target object’s point cloud data while
keeping the previously scanned static work environments. Then the target’s point cloud data
was rapidly converted into a 3D surface model using the concave hull surface modeling
algorithm after a process of data filtering and downsizing to increase the model accuracy and
data processing speed. The performance of the proposed framework was tested at a steel
frame building construction site. The generated surface model and the point cloud of static
surroundings were wirelessly presented to a remote operator. The field test results show that
the proposed rapid target surface modeling method would significantly improve productivity and
safety in heavy construction equipment operations by distinguishing a dynamic target object
from a surrounding static environment in 3D views in near real time.

Key words: segmentation—object recognition—point cloud—laser scanner—site

development

INTRODUCTION
Visibility-related accidents can be easily caused by the interactions between workers,
equipment, and materials. This problem can lead to serious collisions without pro-active
warnings. There have been a number of advances in vision-aid techniques because lacking full
visibility is a major contributing factor in accidents at construction sites. 3D spatial modeling can
help to optimize equipment control [1,24], significantly improve safety [2-3], monitor construction
progress [4], and enhance a remote operator’s spatial perception of the workspace [5-8].
However, the rapid processing of tens of thousand bits of range data in real time is still an
unsolved problem requiring further investigation [9]. Unstructured work areas like construction

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
sites are difficult to graphically visualize because they highly involve unpredictable activities and
change rapidly. Construction site operations require real-time or near real-time information
about the surrounding work environment, which further complicates graphical modeling and
updating.
One commonly used method to obtain the 3D position of an object is based on 3D laser
scanning technology [7,10-11]; this method, however, has some limitations, such as low data
collection speed and low object recognition rates [12]. It has always been a challenge to
recognize specific objects from a 3D point cloud in unstructured construction environments
because it is difficult to rapidly extract the target area from background scattered noises in a
large and complex 3D point cloud.
While rapid workspace modeling is essential to effectively control construction equipment [13],
few approaches have been accepted by the construction industry due to the difficulty of
addressing all the challenges of current construction material handling tasks with the current
sensor technologies. Thus, an innovation in rapid 3D spatial information is necessary to meet
the challenges. The main objective of this paper was to validate a 3D visualization framework to
collect and process dynamic spatial information rapidly at a construction job site for safe and
effective construction equipment operations. Multi-video camera integrated vision-based object
recognition and tracking method has been developed, based on which, a smart laser scanning
method was proposed to reduce data size and scanning time.

LITERATURE REVEW
For the operator to monitor blind spots of the workspace from the cab, a vision-based system
using a single or multiple cameras is an inexpensive option [14]. Brilakis et al. [15] introduced
2D vision-based methods that recognize new overlapping feature points and track them in the
subsequent video stream. To acquire a precise 3D position of objects with additional depth
information, generally two or more cameras generate a stereo view after calibration with known
intrinsic parameters. Park et al. [16] achieved more accurate 3D locations of tracking objects by
projecting the centroids of the tracked entities from two cameras to 3D coordinates.
Notwithstanding the recent advances, there are some known drawbacks of vision-based
techniques in tracking moving equipment at the sites: 1) additional infrastructure is needed to
install and maintain cameras; 2) fixed camera locations have limited view angles and
resolutions, and 3) the results are sensitive to lighting conditions [17].
Laser scanners have been extensively utilized to automatically obtain the “as-is” condition of the
existing buildings [18]; they also can be used to classify and capture a complex heavy
equipment operation as it happens or to provide automated feedback to those who are
conducting the operations [7,17,19]. Teizer et al. presented a methodology for real-time 3D
modeling using Flash LADAR which has a limited measurement range and low accuracy for
outdoor use [3]. Lee et al. proposed an automated lifting-path tracking system on a tower crane
to receive and record data from a laser device [13]. Bosche and Hass registered 3D static CAD
objects to laser-scanned point cloud data [20], which can be utilized to efficiently assess
construction processes. However, most of the algorithms were developed mainly to recognize
and register static objects’ models to point clouds. Few applications have demonstrated the
technical feasibility of registering dynamic models to point clouds in real or near real time.
In the authors’ previous studies [17], a model-based automatic object recognition and
registration method, Project-Recognize-Project (PRP), was introduced to register the CAD
models with the corresponding point cloud of the recognized objects through comparing the
recognized point cloud of the objects with existing CAD models in a database (shown in Figure
1). While the PRP approach provides very detailed, accurate solid models in a point cloud, the
Wang, Cho 218
limitation of this method is that it only works for the objects which have corresponding models in
the database. In this study, a non-model based, surface modeling method is introduced to
automatically recognize and visualize dynamic objects on construction sites.

Figure 1. Model-based object recognition and registration [17]

FRAMEWORK OF THE PROPOSED METHODOLOGY

In Figure 2, the framework of the proposed rapid surface modeling method is illustrated. The
developed data acquisition system is composed of two 2D line laser scanners (80 meter
working ranges at 100Hz scan speed, up to 2.5 sec / 360º scan, 190º for vertical line), a digital
camera and three video cameras with a resolution of 0.25 degrees in a vertical direction and
0.0072 degrees in a horizontal direction. In this system, multiple degree-of-freedom (DOF)
kinematic problems were solved based on the mechanical installation, and 3D point cloud data
and digital image streams can be collected simultaneously. Utilizing this form of flexible design
together with Time-of-flight (TOF) laser scanner working type, higher scanning resolution and
faster scanning rate were obtained, which is more suitable for the complicated dynamic
construction environments.

Figure 2. The framework of the proposed method

Working together with the laser scanner system, the digital video cameras were employed to
capture real time image streams in the jobsite for the operators. Then, the operators can select
Wang, Cho 219
the target objects in the image streams by drawing a bounding box through the developed
graphical user interface. Using a robust local feature detector, Speeded Up Robust Features
(SURF) [21], image-based target object recognition and tracking algorithms were implemented.
Taking the bounding box area as an input, smart scanning and visualization processes were
immediately applied to separately collect and update the data of the target object and the static
site environment. The major two steps of the smart scanning method are: (1) a static jobsite
environment is scanned by the laser system with a very high resolution. Those collected site
data are stored in the memory; and (2) from the second scanning round, only the point clouds in
the dynamic target area specified by the equipment operator are updated separately. In this
process, the size of point cloud data obtained from complex, large construction sites and the
updating time can be significantly reduced.
Table 1. Data size of each step of the proposed methodology
Number of points
Raw data 24,330
Filtered data 24,276
Leaf size = 50 * Resolution* 108
Downsize

Leaf size = 10 * Resolution 1,730

d data

Leaf size = 5 * Resolution 4,674

Leaf size = Resolution 22,138
*
Resolution = 0.06 m.
Surface modeling of point clouds can be more advantageous for the equipment operators over
the point-cloud-only visualization because a surface model of target object can be better
distinguished from a complex point-cloud environment. The concave hull approach is one of the
most widely used surface modeling methods creating a polygon that represents the area
occupied by a set of points. A concave hull better details the shape of the point cloud than the
convex hull does. A concave hull of a set of surfaces is the enclosing concave surfaces with
smallest volume. There are several existing concave hull calculation algorithms [22], however
the efficiency of these algorithms decreases significantly because of high computing queries for
the large size of point cloud data like the ones obtained from construction jobsites.
The raw data size of point cloud data collected from a construction jobsite is quite a large and
as a result, it is challenging to process and visualize these data in real or near real time. In this
study, as shown in Figure 1, the smart scanning technology was employed at the very beginning
of the surface modeling phase in order to significantly reduce the data size of the point clouds
and the surface modeling time. In addition, outliers of the point data were statistically identified
and removed to improve concave hull model accuracy. Then, a data filtering and downsizing
process was conducted to further decrease the number of points. The goal of data downsizing is
to increase the data processing speed by reducing the amount of overly dense data being
processed.

EXPERIMENTS AND RESULTS

Validation of the proposed methodology was implemented on a crawler crane at a building
construction site. The data acquisition system was mounted on a mobile cart, and set up in the
working area of the equipment, especially in its blind spots. It firstly scanned the whole jobsite
and kept the point cloud data in the database; then smart scanning and visualization were
fulfilled based on the selected tracking objects. All tests were benchmarked on an Intel Core i5
Wang, Cho 220
CPU with 4GB RAM on a 64 bit Windows mobile computer. It should be noted that the required
resolution, registration accuracy, and scan rate for successful surface modeling vary based on
the scan range, ambient lighting conditions, properties of the target (e.g., shape, color,
reflectivity, and moving speed), and the number of mounted 2D laser scanner. In this study, all
data were collected from the system with two 2D laser scanners, and the maximum scan speed
of which is 227,500 points/sec. The scan speed could be doubled if four 2D laser scanners were
equipped [17].
Table 2. Processing time of each step of the proposed methodology
Data Downsizing Leaf Size (m)
3 0.6 0.3 0.06
Data Filtering Time (sec.) 0.176 0.176 0.176 0.176
Data Downsizing Time (sec.) 0.004 0.005 0.006 0.008
Į  /HDI6L]H 0.189 0.302 0.446 1.354
Surface Modeling

Į  /HDI6L]H 0.188 0.254 0.410 1.363

Time (sec.)

Į  /HDI6L]H 0.188 0.255 0.410 1.371

Į  /HDI6L]H 0.188 0.253 0.408 1.366
Į  /HDI6L]H 0.188 0.254 0.408 1.365
Į  /HDI6L]H 0.184 0.255 0.407 1.370
Į  /HDI6L]H 0.183 0.254 0.405 1.370
Į  /HDI6L]H 0.184 0.253 0.406 1.373
The whole crane was chosen as a tracking target to track. The collected raw data includes
24,330 points, and its resolution is 0.06 m. The leaf size applied for data downsizing was 0.3 m,
which also applied five times of the resolution. The data size decreased to 24,276 and 4,674
correspondingly after data filtering and downsizing (Table 1  7KHQ  GLIIHUHQW Į YDOXHV ZHUH
assigned to the proposed surface modeling algorithm. Table 2 shows the processing time under
different data downsizing scales and the size of the concave hull segments. Finally, the result
ZLWKĮ  OHDIVL]HZDVWDNen as the output shown to the operator. The total processing time
can be found in Table 3, and it is less than 0.5 seconds. Figure 3 displays a pre-scanned point
clouds of site scene and a scene with a crane model embedded which is continuously updated
at each scan.
Table 3. Total processing time
Process Time (sec.)
Data filtering 0.104
Data downsizing* 0.009
Surface modeling** 0.223
Total processing time 0.336
*
Leaf size = 0.3 m, **Į 1.0 * leaf size

CONCLUSION AND FUTURE STUDY

In this study, a specially designed data capturing system was utilized to collect point cloud data
from multiple laser scanners; while multiple video camera arrays were used to rapidly recognize
and track the selected dynamic construction equipment objects including a backhoe loader and
a crane. The validation of the proposed method was implemented at a real world construction
jobsite. The concave hull of the crawler crane was generated in less than 0.5 seconds, and then
the data were smoothly transferred to the operator in a cabin. The test results indicate that the
Wang, Cho 221
proposed rapid workspace modeling approach can improve the heavy equipment operations by
distinguishing surface-modeled dynamic target objects from the point cloud of existing static
environment in 3D views in near real time.

Figure 3. A crane model embedded in the site environment

While the surface modeling time is sufficiently fast enough for real-time operation, the data
collection time should be carefully configured based on the types of equipment (e.g., size and
moving speed), distance, ambient lighting, and reflectivity.
For future work, the research will continue to improve the resolution of laser scanner data while
reducing data collection time. With an increase in scanning speed, the scanned resolution is
lowered accordingly. To resolve this issue, a smart scanning approach with differentiated scan
speeds will be further developed, to allow faster rotations for the areas to be skipped, and slow
the scan speed for the target areas. Improving surface modeling speed using a higher
performance computer and surface model quality will be another future research focus.

ACKNOWLEDGMENTS
This material is based upon work supported by the National Science Foundation (Award #:
CMMI-1055788). Any opinions, findings, and conclusions or recommendations expressed in this
material are those of the authors and do not necessarily reflect the views of the NSF.

REFERENCES
[1] H. Son, C. Kim, K. Choi, Rapid 3D object detection and modeling using range data from 3D
range imaging camera for heavy equipment operation, Automation in Construction 19(7) (2010)
898-906.
[2] J. Teizer, B. S. Allread, C. E. Fullerton, J. Hinze, Autonomous pro-active real-time
construction worker and equipment operator proximity safety alert system, Automation in
Construction 19(5) (2010) 630-640.
[3] J. Teizer, C. H. Caldas, C. T. Haas, Real-Time Three-Dimensional Occupancy Grid Modeling
for the Detection and Tracking of Construction Resources, ASCE Journal of Construction
Engineering and Management 133(11) (2007) 880-888.
[4] H. Son, C. Kim, 3D structural component recognition and modeling method using color and
3D data for construction progress monitoring, Automation in Construction 19(7) (2010) 844-854.
Wang, Cho 222
[5] Y. Cho, C. Haas, K. Liapi, S. Sreenivasan, A framework for rapid local area modeling for
construction automation, Automation in Construction 11(6) (2002) 629-641.
[6] Y. Cho, C. Haas, S. Sreenivasan, K. Liapi, Error Analysis and Correction for Large Scale
Manipulators in Construction, ASCE Journal of Construction Engineering and Management 130
(1) (2004) 50-58.
[7] Y. Cho, C. Wang, P. Tang, C. Haas, Target-focused local workspace modeling for
construction automation applications, ASCE Journal of Computing in Civil Engineering 26(5)
(2012) 661-670.
[8] Y. Cho, C. Haas, Rapid Geometric Modeling for Unstructured Construction Workspaces,
Journal of Computer-Aided Civil and Infrastructure Engineering 18 (2003) 242-253.
[9] J. Gong, C. H. Caldas, Data processing for real-time construction site spatial modeling,
Automation in Construction 17(5) (2008) 526-535.
[10] P. Tang, D. Huber, B. Akinci, R. Lipman, Automatic reconstruction of as-build building
information models from laser-scanned point clouds: A review of related techniques, Automation
in Construction 19 (2010) 829-843.
[11] D. Huber, B. Akinci, P. Tang, A. Adan, Using laser scanner for modeling and analysis in
architecture, engineering and construction, Proceedings of the Conference on Information
Sciences and Systems (CISS) (2010) Princeton, NJ.
[12] C. Kim, J. Lee, M. Cho, C. Kim, Fully automated registration of 3D CAD model with point
cloud from construction site, 28th International Symposium on Automation and Robotics in
Construction, Seoul, Korea, (2011) 917-922.
[13] G. Lee, H. Kim, C. Lee, S. Ham, S. Yun, H. Cho, B. Kim, G. Kim, K. Kim, A laser-
technology-based lifting-path tracking system for a robotic tower crane, Automation in
Construction 18(7) (2009) 865-874.
[14] T. M. Ruff, Recommendations for evaluating & implementing proximity warning systems on
surface mining equipment, Research Report to the National Institute for Occupational Safety
and Health, Centers for Disease Control, (2007) <
http://www.cdc.gov/niosh/mining/works/coversheet202.html >.
[15] I. Brilakis, M. Park, G. Jog, Automated vision tracking of project related entities, Advanced
Engineering Informatics 25 (2011) 713-724.
[16] M. Park, C. Koch, I. Brilakis, Three-dimensional tracking of construction resources using an
on-site camera system, J. Comput. Civ. Eng. 26(4) (2012) 541-549.
[17] Y. Cho, M. Gai. Projection-Recognition-Projection (PRP) Method for Rapid Object
Recognition and Registration from a 3D Point Cloud, ASCE Journal of Computing in Civil
Engineering (2014) doi: 10.1061/(ASCE) CP.1943-5487.0000332 (in press).
[18] C. Wang, Y. Cho, M. Gai, As-is 3D Thermal Modeling for Existing Building Envelopes Using
a Hybrid LIDAR System, ASCE Journal of Computing in Civil Engineering 27(6) (2013) 645–
656.
[19] Y. Arayici, An approach for real world data modeling with the 3D terrestrial laser scanner for
built environment, Automation in Construction 16 (6) (2007) 816–829.
[20] F. Bosche, C. T. Haas, Automated retrieval of 3D CAD model objects in construction range
images, Automation in Construction 17(4) (2008) 499-512.
[21] H. Bay, T. Tuytelaars, L. V. Gool, SURF: Speeded up robust features, Computer Vision–
ECCV (2008) 404-417.
[22] P. Zhou, Computational Geometry: Analysis and Design on the Algorithms, second edition,
Tsinghua University press, Beijing, 2005.
[23] C. Kim, C. Haas, K. Liapi, C. Caldas, Human-Assisted Obstacle Avoidance System Using
3D Workspace Modeling for Construction Equipment Operation, ASCE J. Comp. in Civ. Engrg.
20(3) (2006) 177-186.

Wang, Cho 223

Designing Digital Topography: Opportunities for
Greater Efficiency with a Primitives and Operators
Approach
Caroline Westort
Department of Landscape Architecture
Iowa State University
156 College of Design
Ames, IA 50010
[email protected]

ABSTRACT
This paper focuses on characterizing proposed human-built topographic forms and describing
them parametrically. Two basic approaches exist for characterizing shape algorithmically:
parametric descriptions, which describe discrete geometries, and non-parametric methods,
which for the most part work on fields. This paper offers a brief overview of the range of
parametric modeling options for topography, a set of criteria that need to be fulfilled for any
successful landform design system, and a primitives and operators approach that offers some
specific advantages in the AMG context.

Key words: DTM—design—landform

INTRODUCTION

Reshaping land to meet societal needs is a complex, disruptive, time consuming, and costly
effort. Industry increasingly relies on Digital Terrain Models (DTMs) as the principle medium for
landform design and the basis for autonomous construction machinery (AMG). However,
controlling DTM geometry remains a non-trivial algorithmic problem. At the project concept
stage and later during construction, existing 3D manipulation methods are cumbersome and
unwieldy, adding to downtime, guesswork and inefficiencies in the field.

Landform Design
Landform as a creative, expressive medium
Landforms may be used as design elements unto themselves, and also to organize and
establish the base upon which other elements may be composed on a site. Landscape
architects and engineers think of topography in both functional quantitative and spatial
qualitative terms [1-3]. Through use of slope, elevation change, convex and concave re-grading,
both subtle and dramatic meanings can be achieved [4].

Landscape designers use a variety of abstractions which help to organize or synthesize their
design of landform, including 'signatures' of contour lines in plan [2, 4, 5]; compositions of
regular geometric solids [6] and flat planes with break lines and transitions [5]; and processes of
land formation: erosion, deposition, scraping and piling, etc. [7, 8]. The topographic condition of
a site is therefore regarded as an essentially plastic one.

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
Through 3D modeling, rendering and visualization, designers try out an assortment of design
alternatives on a topographic surface. Through manipulation of 3D models or images design
alternatives are seen, changed, and analyzed [9]. For environmental designers, visualization
works in close concert with manipulation and quantitative and qualitative analysis. The editing
and analysis tasks can be more closely integrated in the digital medium, as shown in Figure 2.

Figure 2: Iterative design loop for topography that shows how digital methods can
integrate editing and analysis tasks more tightly in the iterative life cycle of a design
project.

Digital landform representation

Landform is challenging to represent in any medium [10] especially computationally where large
scales, fuzzy edges, continuous surfaces, and huge datasets pose particular challenges [11-14].
Digital Terrain in this project is defined as the elevation of the earth's surface above some
reference geoid. Over the last few decades ever-larger quantities of terrain data with higher
accuracy in (x, y) and z have become available to represent existing earth surface geometry.
Digital Terrain Models (DTMs) are an important class of surface models [14] increasingly
becoming the principle medium for landform design and construction [15, 16, 19].

Automated Machine Guidance

Dramatic advances with AMG in recent decades allow a remotely operated bulldozer to
construct a landform from a precise 3D DTM [16-19-30]. Landscape architects and civil
engineers work from existing site survey DTMs to propose and revise many drafts of proposed
topographic surface geometry both on paper and with software to produce a final landform
design, called a grading plan by landscape architects [2], or the design in the engineering
domain [25]. Designers typically hand-off a grading plan to the earthworks contractor only once
so the final construction is often the first and only full-scale built realization of a design that the
team encounters. However, a lot can happen to the design once it is taken into the field and
construction begins. Site conditions, equipment incompatibilities, weather factors, personnel
skill, preference and taste issues are categories of unforeseen factors that can significantly
impact a design once construction gets underway. Thus DTMs often require editing during
construction to avoid downtime in the system and guesswork in the field [19, 20, 21, 25, 30].

Control criteria for DTM geometry over the lifecycle of a grading project
Controlling DTM geometry remains a non-trivial algorithmic problem [31, 32]. Methods for
changing DTM geometry are cumbersome and unwieldy [34], thereby limiting the ability to
creatively explore, modify, and optimize topographic form. These shortcomings yield serious
inefficiencies throughout the lifecycle of a project and result in a need for improved tools for
landform design that fulfill the following criteria:

Westort 225
 3D
Local geometric control
Ease of handling
Quick response time
Quantitative accuracy

What is generally missing is the ability to iteratively revise, update, edit, modify, manipulate –
i.e., “sculpt” DTMs, both before and during the construction phase as a way to insure quality
design.

Approaches to DTM Manipulation

3D model manipulation strategies for topography have evolved chiefly in the CAD and
geographic information system (GIS) digital domains [10], with the former focusing primarily on
the parameterization of objects, the latter on combinations of spatial attribute information with
digital terrain models in several representations (TINS, DTMs, contours, e.g., see [34]) The
simplest and most common of these is the regular tessellation, mesh, displaced as a raster
array. Large raster arrays are reasonable to use for representation of arbitrarily shaped “natural”
existing features such as the earth’s surface, but the manipulation tools available make them
neither an efficient nor expressive medium for design.

Geographic information systems (GIS) have evolved to handle the large datasets characteristic
of landscape, but have mostly focused on the display and analysis of elevation data, rather than
on active tools for manipulating landform. Figure 3 shows the manipulation scopes of action
available in many GISs which allow only either global or ‘local’ changes to a model: a local
change is a change that happens to a single vertex, a global change is one that affects the
entire topographic dataset, e.g., scale changes, which exaggerate the Z value [35].

Figure 3: Local, regional, global scopes of action.

While some operations exist to limit the scope of activity to a mask of pixels or a specific
polygon, these techniques are not especially useful for landform design. An example of a DTM
editor developed to resolve artifacts resulting from elevation data interpolation has been an
interesting approach [36]. As have algorithms for compression [37-42], line of site, shortest path,
drainage [43], multiple observers[40], local maxima and minima, and drainage patterns[44] and
siting from first principles. These approaches, while promising, have focused on data extraction
and algorithmic techniques acting on geospatial data for existing landform geometry of a digital
terrain surface, rather than proposed geometry – the target concern here.

Westort 226
A key advantage of digital/virtual methods is the ability to directly manipulate a 3D
representation [45-49]. The following list, adapted from [50] and [51] summarizes some of the
3D topographic modeling tools available in current industry-standard CAD software packages:
Data Structures
Contours—Landform design using 2D CAD systems relies heavily upon the
representation of 3D form as contour lines; an abstraction well suited to
representation, but notoriously cumbersome for design. It is also a data structure
with key disadvantages: Oversampling along, and under-sampling between lines.
Manipulating contour lines effectively with CAD systems remains a daunting
challenge, requiring spline curves and geometric constraints that have nowhere
yet been satisfactorily packed for—much less mastered by—designers from any
discipline.
B-Rep—Boundary Representations—The surface of an object is described as a
description stored as a list of vertices, lines joining the vertices, and list of faces.
These include Bezuer-spline curves (B-splines), Non-uniform rational basis spline
(NURB) surfaces
Primitives—A set of simple, generic, 3D models (cube, sphere, cylinder, cone,
torus, wedge, lane and others). These primitives can be scaled, translated, and
rotated within the application, often both interactively (such as with a mouse),
and by numerical input.
CSG—Constructive Solid Geometry—An object is represented as a combination
of simple primitives such as cues, spheres, and cylinders. These basic solids are
used as building blocks for more complex objects by means of a system that
uses Boolean combinations (union, intersection, and difference) to describe the
logical operations of adding two objects, subtracting one from another, or
designing the overlap between two objects.
Voxels—Volume/Solid Modeling. Spatial occupancy enumeration divides –
dimensional space into cubic units called voxels, or a 3D pixels.
Operators:
Swept Forms—a 2-dimensional (XY) section ‘swept’ along a third (Z) dimension.
Extrusion—a template is swept in a direction orthogonal to the plane in which it
lies.
Surface of Revolution—a 2D template, closed or open, rotated about an axis.
Skin—the ability to construct a ‘skeleton’ of a form and then wrap a surface skin
around it to create an object
Patches—same as skin except using boundaries as the skeleton.
Curved Patches—while popular remain too computationally intensive to be
justified or affordable for most landform design.
Primitives
What is called for is a specification of landform primitives—mound, swale, plane, for example –
that carry their own parametric definitions and constraints, and so enable ‘regional’ changes,
between local and global in their scope, in which slopes, radii and other dimensions may be
user-specified and propagated (Figure 3). Geometric modeling with pure Euclidean shape
primitives such as cones and cylinders is promising [8], but by itself is too limited for most
landform design, which more often than not involves the design of a continuous surface rather

Westort 227
than of a solid. While some efforts for landscape are promising [51, 53] no unifying ontology, or
organizing Landscape Information Model (LIM) currently exists.

A wide variety of disciplines and scales use a range of terms and parameters to
describe discrete topographic shape, and they vary qualitatively and quantitatively. Moreover,
landform shapes are frequently described in terms of an underlying DTM Data structure. I.e.,
there are contour line specific forms, or signatures, TIN specific data structures. A sampling of
this diversity includes:

x Geomorphological forms, landforms produced by the earth’s natural processes (e.g.

drumlins, eskers, moraines, saddles, valleys, cliffs, glacial forms) [7, 8, 54-58]

x Domain-specific forms, e.g., Contour line signatures [3-5], American Disabilities Act
design specifications [59].

Figure 4: Contour line signatures.[60]

x Project-type specific (e.g., roadways, levees, sand dunes, water management, golf
courses, battlefields, gaming environments, American Disabilities Act (ADA)
compliant features)

Figure 5: Road profile primitive forms [5]

x Individual Project specific forms, A particular project may standardize its own set of
topographic forms for re-use throughout a project.[61]

x Tool-based forms, shovels, rakes, particle guns.

Westort 228
 Figure 6: Example tool-based specific forms.[62]

Operators
Defining a universal set of geometric parameters for landform, coupled to a way to combine the
primitives together, would contribute to dramatically improved geometric control over a DTM
surface. Specification of a set of operations, such as cutting or filling tools, with parametrically
defined shape characteristics (such as angles of slope, depth of fill, etc.) to be performed along
a path. This set of operations is then swept along the path, either over an existing base terrain,
or on a blank surface, and the result is a terrain geometry, which has the desired shape.

Conceptual framework for a primitives and operators approach

A simple and useful way to generalize all topographic forms as four generic shapes derivable
from either a point- or line-based feature. General concavity or convexity are subsequent shape
descriptors.

Mounds – point-based convex surface

Craters – point based concave surface
Berms – line-based concave surface
Swales – line based convex surface.

Figure 7 shows this initial set, with their blade and path shapes abstracted and combined using
a sweep (extrude) algorithm.

Westort 229
 Figure 7: Generic subset of topographic primitives, abstracted as blades and paths
shapes

Operators to generate these sorts of geometries as part of a continuous surface were then
prorammed as a plug-in software to AutoCAD. Figures 8 and 9 show these initial results.

Figure 8. A parameterized berm primitive left.

Figure 9. Boolean combination operators on mound primitives on the right

Westort 230
A stand-alone prototype software was then generated as a generic sculpting tool definition,
Figure 9.T his implementation kept geometric change parameters separate from any underlying
DTM data structure description, Figure 10.

Figure 9: Left Topographic surface sculpting generic tool.[50]

Figure 10: Right Blade and path abstractions as alternative descriptions[63]

Summary & Discussion

A set of blades and path primitives describe proposed topographic form such that the following
criteria for a digital terrain design system are fulfilled: 3D, local geometric control, ease of
handling, quick response time, quantitative accuracy. A primitives and operators formalism
represents an intermediate internal data structure that is independent of the underlying DTM
data structure, and therefore would represent an exciting opportunity for increased efficiency in
the AMG context.

The following are some outstanding challenges that would need to be resolved for this to
happen:

Challenge 1: Primitive shapes can have a wide range of relationships with one another. In
the case of a primitive blade shape that is extruded along a primitive path shape, what is the
desired relationship, symmetrical, static, dynamic with other primitive models. How to provide
interactive handles to the user for setting and varying these shape parameters.

Westort 231
 Figure 11: Symmetrical versus A-Symmetrical blade and path relationships [65]

Challenge 2: Should the relationship between primitives simulate real world on-the-
ground tool behavior or physical phenomena? E.g. shovels, bulldozers, graders, rakes,
etc.? What soil-type, moisture content is assumed? Gravity? Since final DTM geometry is the
priority, and simulation of the manipulation process itself is of secondary importance, those
geometry determining “real-world” parameters which affect final landform shape will be
prioritized.

Figure 12: Left, corner and overlap options.

Figure 13: Middle, rake simulation blade

Figure 14: Right, shovel simulation blade

Challenge 3: Embedding into the underlying terrain – how does the blade-path complex
embed in the underlying DTM surface? Is it an absolute or relative relationship? How are
these relationships parameterized to optimize interactive user control.

Westort 232
 Figure 15: Absolute versus relative relationship of the blade-path-relationship with the
underlying terrain data.

REFERENCES CITED

1. J. Beardsley and N. Grubb, Earthworks and beyond: contemporary art in the landscape:
Abbeville Press, 1989.
2. S. Strom, K. Nathan, and J. Woland, Site engineering for landscape architects: John
Wiley & Sons, 2013.
3. H. C. Landphair and F. Klatt, "Landscape architecture construction," 1979.
4. R. K. Untermann, "Grade easy: an introductory course in the principles and practi ces of
grading and drainage," 1973.
5. P. Petschek, Grading for landscape architects and architects: Walter de Gruyter, 2008.
6. M. Ferraro and R. Mallary, "ECOSITE: A Program for Computer-Aided Landform
Design," Proceedings ACM/SIGGRAPH (NY 1977).
7. R. J. Chorley, R. P. Beckinsale, and A. J. Dunn, History of the Study of Landforms: Or
The Development of Geomorphology: The Life and Work of William Morris Davis vol. 2:
Psychology Press, 1973.
8. B. Etzelmüller and J. R. Sulebak, "Developments in the use of digital elevation models in
periglacial geomorphology and glaciology," Physische Geographie, vol. 41, pp. 35-58,
2000.
9. J. Corner, "Representation and landscape: drawing and making in the landscape
medium," Word & Image, vol. 8, pp. 243-275, 1992.
10. S. M. Ervin, "Digital Terrain Models," Landscape Architecture, Magazine, January, 1994
1994.
11. W. R. Franklin, "Towards a mathematics of terrain."
12. W. R. Franklin, M. Inanc, and Z. Xie, "Two novel surface representation techniques,"
AutoCarto Vancouver, Washington, 2006.
13. W. R. Franklin, Z. Xie, E. Lau, and Y. Li, "Algorithms for terrain and bathymetric sensor
data."
14. R. Weibel and M. Heller, Digital terrain modelling: Oxford University Press, 1993.
15. B. Alsobrooks, "Introduction of 3D Technology & Machine Control Systems," ed.

Westort 233
 16. A. Vonderohe, "Status and Plans for Implementing 3D Technologies for Design and
Construction in WisDOT," Construction and Materials Support Center University of
Wisconsin – Madison Department of Civil and Environmental Engineering WisDOT
Project ID: 0657-45-11, 2009/05//undefined 2009.
17. C. Zhang, A. Hammad, and H. Bahnassi, "Collaborative Multi-Agent Systems for
Construction Equipment Based on Real-Time Field Data Capturing," Journal of
Information Technology in Construction, vol. 14, pp. 204-228, 2009/06//undefined 2009.
18. A. Zogheib, "Autonomous Navigation Tool for Real & Virtual Field Robots," in 1st
International Conference on Machine Control & Guidance 2008, 2008, pp. 1-11.
19. C. D. O. TRANSPORTATION, "Digital Design Environment Guide," Newington,
CTOctobor 2007.
20. F. D. o. T. F. E. C. S. O. (ECSO), "Multi-Line Earthwork for Designers," F. D. o. T. F. E.
C. S. O. (ECSO), Ed., ed, 2007, p. 72.
21. T. Hampton. (2005, 2005) 3D Grade Control Puts Designers Right in the Operator’s
Seat. Enigineering News Record. Available:
http://www.dot.state.mn.us/caes/files/pdf/enr_3d_grade_control_10_05.pdf
22. J. J. Hannon, "NCHRP Synthesis 372 Emerging Technologies for Construction
Delivery," Transportation Research Board 978-0-309-09791-8, 2007 2007.
23. J. J. Hannon, NCHRP Synthesis 385 Information Technology for Efficient Project
Delivery: Transportation Research Board, 2008.
24. J. J. Hannon and D. Townes, "GPS Utilization Challenges in Transportation Construction
Project Delivery," in The construction and building research conference of the Royal
Institution of Chartered Surveyors, Georgia Tech, Atlanta USA, 2007, p. 15.
25. T. Hoeft, "lmproving Construction Efficiencies Through 3D Electronic Design," C. Jarhen,
Ed., ed, 2009, p. 1.
26. D. Kratt, "Design Memorandum NO. 18-05-Electronic Files Submittal with the Final
Contract Plans," ed, 2005.
27. M. Leja and R. Buckley, "Cross-Section Preparation and Delivery Memorandum," ed:
California Department of Transportaion, 2004.
28. A. Z. Sampaio, A. R. Gomes, and J. Prata, "Virtual Environment in Civil Engineering:
Construction and Maintenance of Buildings," in ADVCOMP 2011, The Fifth International
Conference on Advanced Engineering Computing and Applications in Sciences, 2011,
pp. 13-20.
29. D. Sheldon and C. Mason, "A Proposal for Statewide CAD Standards in Iowa," Howard
R. Green Company2009/04//undefined 2009.
30. P. Söderström and T. Olofsson, "Virtual Road Construction – a Conceptual Model," in
W78 Conference, Maribor, Slovenia, 2007, pp. 255-261.
31. X. Wang, M. J. Kim, P. E. D. Love, and S.-C. Kang, "Augmented Reality in built
environment: Classification and implications for future research*," Automation in
Construction, vol. 32, pp. 1-13, July 2013 2013.
32. S. Andrews and S. Geiger, "Unlocking Design Data," presented at the 19th Annual
AGC/DOT Technical Conference, New York, 2005.

33. F. Vahdatikhaki, A. Hammad, and S. Setayeshgar, "Real-time simulation of earthmoving

projects using automated machine guidance," in The 30th ISARC, Montréal, Canada,
2013, pp. 718-730.
34. R. Laurini and D. Thompson, Fundamentals of spatial information systems: Academic
press, 1992.
35. S. M. Ervin and C. Y. Westort, "Procedural Terrain; A Virtual Bulldozer," in Proceedings
CAAD-Futures, International Conference on Computer Aided Architectural Design, 1995.

Westort 234
 36. H. Bär, "Interaktive Bearbeitung von Geländeoberflächen-Konzepte, Methoden,
Versuche," 1996.
37. Representing landscapes : a visual collection of landscape architectural drawings /
edited by Nadia Amoroso. Abingdon, Oxon ; New York: Abingdon, Oxon ; New York :
Routledge, 2012.
38. B. Cutler, W. R. Franklin, and T. Zimmie, "Fundamental Terrain Representations and
Operations: Validation of Erosion Models for Levee Overtopping," in NSF Engineering
Research and Innovation Conference, Honolulu, 2009.
39. R. Fabio, "From point cloud to surface: the modeling and visualization problem,"
International Archives of Photogrammetry, Remote Sensing and Spatial Information
Sciences, vol. 34, p. W10, 2003.
40. W. R. Franklin, M. Inanc, Z. Xie, D. M. Tracy, B. Cutler, and M. V. Andrade, "Smugglers
and border guards: the GeoStar project at RPI," in Proceedings of the 15th annual ACM
international symposium on Advances in geographic information systems, 2007, p. 30.
41. M. Inanc, Compressing terrain elevation datasets: ProQuest, 2008.
42. M. Metz, H. Mitasova, and R. Harmon, "Efficient extraction of drainage networks from
massive, radar-based elevation models with least cost path search," Hydrology and
Earth System Sciences, vol. 15, pp. 667-678, 2011.
43. M. V. Andrade, S. V. Magalhaes, M. A. Magalhães, W. R. Franklin, and B. M. Cutler,
"Efficient viewshed computation on terrain in external memory," GeoInformatica, vol. 15,
pp. 381-397, 2011.
44. T.-Y. Lau and W. R. Franklin, "COMPLETING RIVER NETWORKS WITH ONLY
PARTIAL RIVER OBSERVATIONS VIA HYDROLOGY-AWARE ODETLAP."
45. W. J. Mitchell, "A computational view of design creativity," Modeling Creativity and
Knowledge-Base Creative Design, pp. 25-42, 1993.
46. W. J. Mitchell, The logic of architecture: Design, computation, and cognition: MIT press,
1990.
47. W. J. Mitchell, Computer-aided architectural design: John Wiley & Sons, Inc., 1977.
48. W. J. Mitchell, Digital design media: John Wiley & Sons, 1995.
49. M. McCullough, W. J. Mitchell, and P. Purcell, The electronic design studio: architectural
knowledge and media in the computer era: MIT Press, 1990.
50. S. M. Ervin, Landscape modeling : digital techniques for landscape visualization /
Stephen M. Ervin, Hope H. Hasbrouck. New York: New York : McGraw-Hill, 2001.
51. S. Mealing, Mac 3D: Three-dimensional Modelling and Rendering on the Macintosh:
Intellect Books, 1994.
52. M. Flaxman, "Fundamentals of Geodesign," Proceedings Digital Landscape
Architecture, Buhmann/Pietsch/Kretzel (Eds.): Peer Reviewed Proceedings Digital
Landscape Architecture, Anhalt University of Applied Science, Germany, 2010.
53. S. Ervin, "A system for GeoDesign," Proceedings Digital Landscape Architecture, Anhalt
University of Applied Science, Germany, 2011.
54. T. R. Allen, "Digital terrain visualization and virtual globes for teaching geomorphology,"
Journal of Geography, vol. 106, pp. 253-266, 2008.
55. D. G. Brown, D. P. Lusch, and K. A. Duda, "Supervised classification of types of
glaciated landscapes using digital elevation data," Geomorphology, vol. 21, pp. 233-250,
1998.
56. R. Dikau, "The application of a digital relief model to landform analysis in
geomorphology, Raper J., Three Dimensional Applications in Geographical Information
Systems, 1989, 51-77," ed: Taylor and Francis, London.
57. R. Dikau, "The application of a digital relief model to landform analysis in
geomorphology," Three dimensional applications in geographical information systems,
pp. 51-77, 1989.

Westort 235
 58. J. P. Wilson, "Digital terrain modeling," Geomorphology, vol. 137, pp. 107-121, 2012.
59. R. L. Mace, "Universal design in housing," Assistive Technology, vol. 10, pp. 21-28,
1998.
60. C. W. Harris and N. T. Dines, "Time-saver standards for landscape architecture,"
McGraw-Hili Book Company. New York, 1988.
61. 3*U\ERĞ0.DOHWRZVND8/LWZLQ-03LMDQRZVNL$6]HSWDOLQDQG0=\JPXQW
"Data preparation for the purposes of 3D visualization," Geomatics, Landmanagement
and Landscape, pp. 19-29, 2013.
62. H. L. Nichols, Moving the earth : the workbook of excavation / Herbert L. Nichols, Jr.,
David A. Day. New York: New York : McGraw-Hill, 2010.
63. C. Y. Westort, "Methods for Sculpting Digital Topographic Surfaces," University of
Zurich, 1998.
64. C. Y. WESTORT, "An Explosion Tool for DTM Sculpting," Trends in Landscape
Modeling: Proceedings at Anhalt University of Applied Sciences 2003, p. 35, 2003.
65. C. Y. Westort, "Corner, End, and Overlap “Extrusion Junctures”: Parameters for
Geometric Control," in Digital Earth Moving, ed New York: Springer, 2001, pp. 78-86.
66. C. Y. Westort, Digital Earth Moving: First International Symposium, DEM 2001, Manno,
Switzerland, September 5-7, 2001. Proceedings vol. 1. New York: Springer, 2001.
67. M. D. Johnson, E. C. Holley, F. P. Morgeson, D. LaBonar, and A. Stetzer, "Outcomes of
Absence Control Initiatives A Quasi-Experimental Investigation Into the Effects of Policy
and Perceptions," Journal of Management, vol. 40, pp. 1075-1097, 2014.
68. D. Gergle and D. S. Tan, "Experimental Research in HCI," in Ways of Knowing in HCI,
ed: Springer, 2014, pp. 191-227.

Westort 236
Applicability and Limitations of 3D Printing for Civil
Structures

Mostafa Yossef
Department of Civil, Construction and Environmental Engineering
Iowa State University
Ames, Iowa, 50011
[email protected]

An Chen
Department of Civil, Construction and Environmental Engineering
Iowa State University
Ames, Iowa, 50011
[email protected]

ABSTRACT
Three Dimensional Printing (3DP) is a manufacturing process that builds layers to create a
three-dimensional solid object from a digital model. It allows for mass customization and
complex shapes that cannot be produced in other ways, eliminates the need for tool production
and its associated labor, and reduces waste stream. Because of these advantages, 3DP has
been increasingly used in different areas, including medical, automotive, aerospace, etc. This
automated and accelerated process is also promising for civil structures, including building and
bridges, which require extensive labor. If successful, it is expected that 3D structural printing
can significantly reduce the construction time and cost. However, unlike applications in other
areas, civil structures are typically in large scale, with length or height spanning hundreds of
feet. They are subjected to complex loadings, including gravity, live, wind, seismic, etc.
Therefore, it is challenging to develop suitable printing tools and materials. As a result, although
there are limited 3D printed buildings, 3DP of civil structures is still at a primitive stage. This
papers aims to explore the applicability of 3DP for civil structures. The first part is devoted to a
review of 3DP in different areas, including 3D printed buildings. Based on the state of art, the
weakness and opportunities of 3DP are identified. Finally, future directions for 3DP in civil
structures are discussed.

Key words: 3D Printing—review—civil structures—applicability—limitations

INTRODUCTION
Three Dimensional Printing (3DP) was evolved from automated production, which started in the
early twentieth century. It was first applied in manufacturing and automotive industries.
Recently, its applications were expanded to other industries, including medical, aerospace,
construction, etc. This automated and accelerated process is also promising for civil structures,
including building and bridges, which require extensive labor. However, many factors have
limited its further development. As a result, although there are limited application of 3DP in civil
construction, 3DP of civil structures is still at a primitive stage. This paper first reviews the latest
development of 3DP in construction and other areas. It then identifies the limiting factors and
challenges of 3DP. Finally, future directions of 3DP in civil engineering are discussed.

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
BRIEF HISTORY OF 3D PRINTING
According to 3DPI (2014), 3DP started in the late 1980’s. It was known as Rapid Prototyping
(RP) technology developed by Kodama in Japan. Six years later, Charles Hull invented Stereo
Lithography Apparatus (SLA). In 1987, SLA-1 was introduced as the first commercial RP
system. In 1989, a patent for Selective Laser Sintering (SLS) was issued for Carl Deckard at
University of Texas. Through the 1990’s until early 2000’s, SLS has been developed to focus on
industrial applications such as casting. New terminology, Casting and Rapid Manufacturing
(RM), was introduced for such applications. In 2005, the terminology evolved to include all
processes under Additive Manufacturing (AM). The term Additive Manufacturing (AM) is defined
by ASTM as “a process of joining materials to make objects from 3D model data, usually layer
upon layer” (ASTM Standard 2012). Unlike Subtractive term which means machining away the
material from a block to form the required object. Casting or shaping the material in a mold is
often called Formative process. Table 1 shows a summary of different techniques used in AM
(Buswell et al. 2007).

Table 1 –Summary of AM techniques (Buswell et al. 2007)

Process Description
Stereolithography Liquid photopolymer resin is held in a tank. A flat bed is immersed to a
(SLA) depth equivalent to one layer. Lasers are used to activate the resin and
cause it to solidify. The bed is lowered and the next layer is built.

Fused Deposition Extrudes a narrow bead of hot plastic, which is selectively deposited where
Modelling (FDM) it fuses to the existing structure and hardens as it cools.

Selective Laser Utilises a laser to partially melt successive layers of powder. One layer of
Sintering (SLS) powder is deposited over the bed area and the laser targets the areas that
are required to be solid in the final component.

3D Printing (3DP) Based on inkjet printer technology. The inkjet selectively deposits a liquid
binder onto a bed of powder. The binder effectively ‘glues’ the powder
together.

3DP is based on AM process. It is a process where a 3D model is created using Computer

Aided Drafting (CAD) software. The model is then transferred to the 3D printer as a standard
data known as stereolithography language (STL), where the model is converted into layers that
can be applied consecutively. Each layer is formed where the printer head deposits an
activating agent, and premixes it with a power material. The layers are bonded together
consecutively to form the 3D object. In 2009, the first commercial 3D printer was offered for
sale. In 2012, an alternative 3D printer was introduced at entry level of the market with
affordable price.

APPLICATIONS OF 3D PRINTING
3DP has been increasing used in different areas. Architectural modelling is one of the major
areas that uses 3DP for developing prototypes that facilitate the communication between the
architect and customer. Architect can print now complex structures and color it as well for better
representation (Gibson et al. 2002). In medical area, 3DP is used to create high quality bone

Yossef, Chen 238

transplant, modelling of damaged bones for better fracture analysis (James et al. 1998; Murray
et al. 2008). 3PD can also be used to print complex shapes, such as human tissue or artificial
blood vessels that are used in coronary bypass surgery (Wong and Hernandez 2012). Dentists
are using 3DP to create a plaster model of the mouth or to replace patient’s teeth (van Noort
2012). In aerospace industry, 3DP is used to print airfoils (Thomas et al. 1996). In automotive
field, Song et al. (2002) used RP technology to manufacture the die of an automobile deck part.

3DP FOR CIVIL STRUCTURES

Warszawski and Navon (1998) pointed out the following problems concerning construction
industry: low labor efficiency compared with automated machines, high accident rate, low quality
work due to insufficient skilled workforce, and difficulty of applying control of construction site.
Applying automation or 3DP can overcome these problems.

Automation in construction industry started in terms of robotics [Gambao et al. (2000); Kuntze et
al. (1995); Lorenc et al. (2000); Williams et al. (2004)]. Buswell et al. (2007), (2008) conducted a
review over RM technologies for construction, based on which they developed a Freeform
Construction method. The term of Freeform Construction was defined for methods that deliver
large-scale components for construction without the need of formworks using AM. They
concluded that Freeform Construction could reduce the construction cost and provide freedom
of selecting desired geometry with better performance than traditional method. Lim et al. (2009)
stated that Freeform Construction methods are currently limited to CC (US); Concrete Printing
(UK); and D-shape (Italy).

Khoshnevis (1998) introduced the Contour Crafting (CC) which later become an effective
method of printing 3D houses. Khoshnevis (2004) defined CC as “an additive fabrication
technology that uses computer control to exploit the superior surface-forming capability of
troweling to create smooth and accurate planar and free-form surface”. The idea of CC was to
use two trowels to form a solid planar surface for external edges. Filler material such as
concrete can then be poured to fill the extruded area. They demonstrated that CC can be used
in building structures as shown in Figure 1, where a nozzle is supported by a gantry system
which moves in two parallel lanes. The nozzle is capable of full 6-axis positioning and can
extrude both sides and filler material. CC nozzle can also be used for forming paint-ready
surface, placing reinforcement before pouring concrete, plastering and tiling, plumbing and
installing electrical modules and communication line wiring.

Zhang and Khoshnevis (2013) developed an optimized method for CC machine to efficiently
construct complicated large-scale structures. Extensive research was done to avoid collision
between multiple nozzles. Three approaches were compared, namely: path cycling, buffer zone
path cycling and auxiliary buffer zone. The results indicated that the path cycling and buffer
zone cycling provided the maximum optimization. They concluded that using CC method is
significantly faster than traditional methods and implementation to multi-story building is
possible by climbing as shown in Figure 3.

According to Roodman and Lenssen (1995), the construction industry consume more than 40%
of all raw materials globally. CC can reduce the material waste from 7 tons to almost none for a
single-family home. And the speed of the construction can be increased to one day per house.
Although the ability of using this method in luxury structures or complex structures is still limited,
implementation of CC can help with fast construction of low income housing and emergency
shelter.

Yossef, Chen 239

Despite the many advantages of CC, Lim et al. (2009) listed some limitations of CC as follows.
The mold is not disposed and becomes a part of the wall. CC method requires excessive steps
including molding, installing reinforcement, and placing concrete to build layers up to 20 mm
high. These limitations encouraged them to develop another Freeform Construction method
called Concrete Printing. Similar to 3DP idea, the concrete printing machine has a frame of
5.4m x 4.4 m (footprint) x 5.4 m (height) and a printing head moving on a mobile beam. A 9 mm
nozzle is supported with the printing head to provide the material extrusion. Later, Le et al.
(2012) conducted experimental program to figure out the optimum mix design of a high-
performance fiber-reinforced fine-aggregate concrete for printing concrete.

The 3D printed houses can provide a cheap and efficient homes of low-income families. The
printed houses consist of different printed parts assembled together to form the house. It can
take less than 24 hour to build one house. However, no details are provided about 3DP of
wiring, plumbing and HVAC, etc.

The latest development of 3DP was from WinSun, a Chinese company. They printed five-story
apartment block using 3DP as shown in Figure 4 (Charron 2015). They stated that the houses
were in full compliance with relevant national standards, which overcomes one of the main
issues that face 3D printed houses. WinSun also printed a decorated house as shown in Figure
5.

3DP can also be used for non-conventional structures. DUS, a Dutch architecture company
used 3DP to design facades integrated with solar panels, where the angle of the solar panel
could be optimized automatically for any location. This can eliminate the need of manufacturing
a mold for every different location (Jordan et al. 2015).

Other automation effort was done by the industry sector. For example, Shimizu Corporation in
Japan developed an automated system that included erection and welding of steel-frames,
laying concrete floor planks, installation of exterior and interior wall panels, and installation of
various other units (Yamazaki and Maeda 1998).

Lim et al. (2012) compared CC, D-shape, and Concrete Printing. They concluded that Concrete
Printing could optimize strength prior to manufacturing, which resulted in less material. It could
also create complex concrete shapes without the need of labor-intensive molding as shown in
Figure 2.

CHALLENGES OF 3DP FOR CIVIL STRUCTURES

As described above, 3DP allows for mass production, uses less labor, increases the
construction speed and produces less waste compared to traditional construction methods. 3D
printed structure is a layered structure, which is not new in civil engineering. Concrete Masonry
Unit (CMU) structure is a typical layered structure, where the CMU units are installed by pieces
and bonded together with mortar. The author of this paper has designed CMU buildings up to 13
stories and 125 feet high. The integrity of 3D printed structure is better than CMU structure.
Therefore, 3D printed structure should be able to exceed the height of CMU structure. However,
the tallest building that has been printed so far was the 5-story apartment building. Khoshnevis
(2004) and Buswell et al. (2007) stated multiple issues that slowed down the growth of
automation industry in construction. It can be summarized as follows:
x automated fabrication is often not suitable for large scale products and conventional
design approaches;

Yossef, Chen 240

 x smaller ratio of automated products in comparison with other industries;
x only limited material can be used by automated machines;
x expensive automated machines tend to be unfeasible economically; and
x managerial issues and the increasing pressure towards environmental issues of
construction materials in developing countries (Guthrie et al. 1999).

CONCLUSIONS AND FUTURE DIRECTIONS OF 3DP FOR CIVIL STRUCTURES

It can be shown from this paper that the application of 3D printing in civil engineering is
promising. It can not only help to improve communication among designers by creating
prototypes of the desired projects, but also be used in high-stress performance testing and end-
user applications. Considering the limitations described above, in the authors’ opinion, the
following directions of 3DP for civil structures deserve further attention.

(A)

(B)
Figure 1 - Schematic view of Figure 2 – Complex Concrete Printing
construction of conventional buildings Product
using CC (a) 3D Model (Lim et al. 2012),
(Zhang and Khoshnevis 2013) (b) During printing (Le et al. 2012)

Yossef, Chen 241

 Figure 3 – Construction of multi-story buildings using Contour Crafting method
(Zhang and Khoshnevis 2013)

Figure 4 – A 3D printed five-story Figure 5 – A Decorated 3D printed house

apartment block (Charron 2015) (Charron 2015)

Construction components of significant size are heavy, typically being up to 5 tons. Suitable
equipment needs to be developed in order to lift and move heavy component. However, before
suitable equipment can be developed for large scale structure, in-situ deposit approach, i.e.,
printing lighter parts on site followed by assembly would be an alternative option.

3DP can be especially useful for structures with complex shapes. For example, rubber can be
used to print shock absorbers in a large scale which can help in reducing the seismic effects on
buildings. A prototype is shown in Figure 6.

3DP can also open up a frontier to use new materials. These new materials need to satisfy
specific requirements from 3DP. For example, they need to have proper curing time since the
lower layer needs to support the upper layer. The bonding between different layers should be
strong. These materials also require extensive testing to determine their mechanical properties,
including the properties of the materials, inter- and intra-layers.

Yossef, Chen 242

 Figure 6 - 3D Printed Rubber Shock Absorber (http://3dprinting.co.uk/)

Jordan et al. (2015) stated that automated industry will take over the constructions process. This
requires revising building codes to ensure that additive machines are operating within limits and
meet performance criteria. For example, the 3D printed structure should be able to take
complex loads, including gravity, live, wind, seismic, etc., and satisfy the performance
requirements, such as fire, smoke and toxicity. In addition to that, current safety factors are high
due to the consideration of human mistakes. Such factors can be lowered in case of using
automated machines instead of human workforce.

Development of a complete process from the parametric design until printing the building is
needed to control the whole process and eliminate any wasted time during printing. Khoshnevis
(2004) proposed a planning system that shows each component of future automated system.
Figure 7 shows a brief explanation for the proposed plan.

Further research is also needed in connections for 3D printed structures, where few studies are
available. These include, but not limited to, beam-column, column-footing, wall connections, etc.

Yossef, Chen 243

 Figure 7 – Components of future automated construction
(Khoshnevis 2004)

REFERENCES

3DPI. (2014). “3D Printing History: The Free Beginner’s Guide.” 3D Printing Industry,
<http://3dprintingindustry.com/wp-content/uploads/2014/07/3D-Printing-Guide.pdf> (May
18, 2015).

ASTM. (2012). “F2792. 2012 Standard terminology for additive manufacturing technologies.”
West Conshohocken, PA: ASTM International. www.astm.org.

Yossef, Chen 244

Buswell, R. A., Soar, R. C., Gibb, A. G. F., and Thorpe, A. (2007). “Freeform Construction:
Mega-scale Rapid Manufacturing for construction.” Automation in Construction, 16(2), 224–
231.

Buswell, R. A., Thorpe, A., Soar, R. C., and Gibb, A. G. F. (2008). “Design, data and process
issues for mega-scale rapid manufacturing machines used for construction.” Automation in
Construction, 17(8), 923–929.

Charron, K. (2015). “WinSun China builds world’s first 3D printed villa and tallest 3D printed
apartment building.” 3ders.org, <http://www.3ders.org/articles/20150118-winsun-builds-
world-first-3d-printed-villa-and-tallest-3d-printed-building-in-china.html> (May 18, 2015).

Gambao, E., Balaguer, C., and Gebhart, F. (2000). “Robot assembly system for computer-
integrated construction.” Automation in Construction, 9(5-6), 479–487.

Gibson, I., Kvan, T., and Wai Ming, L. (2002). “Rapid prototyping for architectural models.”
Rapid Prototyping Journal, MCB UP Ltd, 8(2), 91–95.

Guthrie, P., Coventry, S., Woolveridge, C., Hillier, S., and Collins, R. (1999). “The reclaimed and
recycled construction materials handbook.” CIRIA, London, UK.

James, W. J., Slabbekoorn, M. A., Edgin, W. A., and Hardin, C. K. (1998). “Correction of
congenital malar hypoplasia using stereolithography for presurgical planning.” Journal of
Oral and Maxillofacial Surgery, 56(4), 512–517.

Jordan, B., Dini, E., Heinsman, H., Reichental, A., and Tibbits, S. (2015). “The Promise of 3D
Printing.” Thornton Tomasetti,
<http://www.thorntontomasetti.com/the_promise_of_3d_printing/> (May 18, 2015).

Khoshnevis, B. (1998). “Innovative rapid prototyping process makes large sized, smooth
surfaced complex shapes in a wide variety of materials.” Materials Technology, 13(2), 52–
63.

Khoshnevis, B. (2004). “Automated construction by contour crafting—related robotics and

information technologies.” Automation in Construction, 13(1), 5–19.

Kuntze, H.-B., Hirsch, U., Jacubasch, A., Eberle, F., and Göller, B. (1995). “On the dynamic
control of a hydraulic large range robot for construction applications.” Automation in
Construction, 4(1), 61–73.

Le, T. T., Austin, S. A., Lim, S., Buswell, R. A., Gibb, A. G. F., and Thorpe, T. (2012). “Mix
design and fresh properties for high-performance printing concrete.” Materials and
Structures, 45(8), 1221–1232.

Lim, S., Buswell, R. A., Le, T. T., Austin, S. A., Gibb, A. G. F., and Thorpe, T. (2012).
“Developments in construction-scale additive manufacturing processes.” Automation in
Construction, 21, 262–268.

Yossef, Chen 245

Lim, S., Le, T., Webster, J., Buswell, R., Austin, S., Gibb, A., and Thorpe, T. (2009). “Fabricating
construction components using layer manufacturing technology.” Global Innovation in
Construction Conference, Loughborough University.

Lorenc, S. J., Handlon, B. E., and Bernold, L. E. (2000). “Development of a robotic bridge
maintenance system.” Automation in Construction, 9(3), 251–258.

Murray, D. J., Edwards, G., Mainprize, J. G., and Antonyshyn, O. (2008). “Optimizing
craniofacial osteotomies: applications of haptic and rapid prototyping technology.” Journal
RIRUDODQGPD[LOORIDFLDOVXUJHU\࣯RIILFial journal of the American Association of Oral and
Maxillofacial Surgeons, 66(8), 1766–72.

Van Noort, R. (2012). “The future of dental devices is digital.” 'HQWDOPDWHULDOV࣯RIILFLDO

publication of the Academy of Dental Materials, 28(1), 3–12.

Roodman, D. M., and Lenssen, N. (1995). A Building Revolution: How Ecology and Health
Concerns Are Transforming Construction. Worldwatch Institute.

Song, Y., Yan, Y., Zhang, R., Xu, D., and Wang, F. (2002). “Manufacture of the die of an
automobile deck part based on rapid prototyping and rapid tooling technology.” Journal of
Materials Processing Technology, 120(1-3), 237–242.

Thomas, C. L., Gaffney, T. M., Kaza, S., and Lee, C. H. (1996). “Rapid prototyping of large
scale aerospace structures.” 1996 IEEE Aerospace Applications Conference. Proceedings,
IEEE, 219–230.

Warszawski, A., and Navon, R. (1998). “Implementation of Robotics in Building: Current Status
and Future Prospects.” Journal of Construction Engineering and Management, American
Society of Civil Engineers, 124(1), 31–41.

Williams, R. L., Albus, J. S., and Bostelman, R. V. (2004). “Self-contained automated

construction deposition system.” Automation in Construction, 13(3), 393–407.

Wong, K. V., and Hernandez, A. (2012). “A Review of Additive Manufacturing.” ISRN

Mechanical Engineering, 2012, 1–10.

Yamazaki, Y., and Maeda, J. (1998). “The SMART system: an integrated application of
automation and information technology in production process.” Computers in Industry,
35(1), 87–99.

Zhang, J., and Khoshnevis, B. (2013). “Optimal machine operation planning for construction by
Contour Crafting.” Automation in Construction, 29, 50–67.

Yossef, Chen 246

Time-Optimal Path Planning for Automated Grain
Carts
Mengzhe Zhang
Department of Mechanical Engineering
Iowa State University
2025 Black Engineering
Ames, 50011
[email protected]

Sourabh Bhattacharya
Department of Mechanical Engineering
Iowa State University
2025 Black Engineering
Ames, 50011
[email protected]

ABSTRACT
In this paper, we address a motion planning problem for an autonomous agricultural vehicle
modeled as tractor-trailer system. We first present a numerical approach and a primitive-based
approach for computing the time-optimal path based on given static initial and goal
configurations. In the former approach, we define a value function for the entire state space.
The value function is consistent with the time to reach goal configuration, and it is finally used to
compute the optimal trajectory. In the latter approach, based on the regular and singular
primitives, we present an algorithm to construct such primitives and derive the final path.
Subsequently, we extend the results and present a dynamic motion planning strategy to
accommodate the case of mobile target configuration. Finally, simulation results are provided to
validate the feasibility and effectiveness of these techniques.

Key words: motion planning—autonomous—agricultural vehicle

INTRODUCTION
Logistics problem can be described as the management of resources in order to meet specific
requirements, or customers. The resources to be dealt with include physical items such as
materials and tools, as well as abstract items such as information and energy. Logistics problem
can be embodied in many aspects, such as business, economics and agriculture. In this paper,
we address the problem of logistics in path planning for agricultural vehicles.

Currently, crop-harvesting is usually performed by agricultural machines called combines

(combine harvesters). Due to the limited capacities of the combines, a grain cart is involved for
transporting the grains from combines to the depot. If there are sufficient number of grain carts,
each combine can go alongside with a grain cart for unloading. However, if the number of
combines exceeds the number of grain carts, a single grain cart has to serve multiple combines.
Based on the fact that the grain cart is a tractor-trailer system, we explore the problem of time-
optimal path planning for an autonomous grain cart to move from one combine to the next
combine.

Proceedings of the 2015 Conference on Autonomous and Robotic Construction of Infrastructure, Ames, Iowa. © 2015
by Iowa State University. The contents of this paper reflect the views of the author(s), who are responsible for the
facts and accuracy of the information presented herein.
There have been some efforts in the past to address the problem of harvesting in large-scale
farming scenarios. In (Fokkens and Puylaert 1981), a linear programming model for harvesting
operations is presented. The model gives the management results of harvesting operations at
the large scale grain farm. In (Foulds and Wilson 2005) and (Basnet and Foulds 2006),
researchers analyze the scheduling problem of farm-to-farm harvesting operations for hay and
rape seed. These works mainly focus on scheduling harvesting operations from farm to farm. In
contradistinction, our research focuses on the motion planning for an unloading vehicle in a
single field.

Recently, path planning of agricultural machines has received some attention in the research
community. In (Makino, Yokoi, and Kakazu 1999), authors develop a motion planning system,
which integrates global and local motion planning components. In (Ferentinos, Arvanitis, and
Sigrimis 2002), authors propose two heuristic optimization techniques in motion planning for
autonomous agricultural vehicles. In (Ali and Van Oudheusden 2009), authors address the
motion planning of one combine by using integer linear programming formulation. In (Oksanen
and Visala 2009), coverage path planning problem is considered. The presented algorithms not
only aim to find an efficient route, but also ensure the coverage of the whole field. In (Hameed,
Bochtis, and Sørensen 2013), a coverage planning approach is proposed with the consideration
of the presence of obstacles. However, compared with our research, the aforementioned works
do not consider the problem of path planning on unloading vehicles and do not model the
vehicle as a tractor-trailer system.

There has been some previous research to plan optimal trajectories for tractor-trailer model
which are prevalent in farming applications. In (Divelbiss and Wen 1994), authors present an
algorithm to find a feasible path which satisfies the given non-holonomic constraints. In
(Divelbiss and Wen 1997), authors propose a trajectory tracking strategy which controls a
tractor-trailer system moving along a path generated off-line. In (Hao, Laxton, Benson, and
Agrawal 2004), researcKHUV SUHVHQW D GLIIHUHQWLDO IODWQHVVíEDVHG IRUPDWLRQ IROORZLQJ IRU D
tractor-cart moving along with a combine harvester. In (Astolfi, Bolzern, and Locatelli 2004), an
application of using Lyapunov technique is proposed to design the control laws for tractor-trailer
model to follow a prescribed path. Researchers introduce the notion of equivalent size and
propose an approach for path planning based on genetic algorithm in (Liu, Lu, Yang, and Chen
2008). In (Yu and Hung 2012), the tractor-trailer model is regarded as a Dubins vehicle, which
can only move with constant speed and turn with upper bounded curvature. The proposed
algorithm is used to find the shortest path in Dubins Traveling Salesman Problem with
Neighborhoods (Isaacs, Klein, and Hespanha 2011). In (Chyba and Sekhavat 1999) and
(Chitsaz 2013), authors introduce the notion of regular primitive and singular primitive which are
local time-optimal.

The contribution of this paper can be summarized as follows. First, according to the
mathematical model of grain cart, we present a numerical approach, as well as a primitive-
based approach to find the time-optimal solution to the path planning problem in different
situations. To tackle the case of moving target configuration, we further propose a two-stage
motion planning strategy, taking advantage of the previous results. Finally, feasibility and
effectiveness of presented techniques are demonstrated by simulations.

The rest of the paper is organized as follows. In Section II, we present the mathematical models
for both combine and grain cart. In Section III, we present our previous work on the scheduling
of agricultural operations. Based on the scheduling scheme, we formulate the path planning
problem for grain cart in Section IV. In Section V, we present a numerical approach to obtain the

Zhang, Bhattacharya 248

time-optimal path between two given configurations. In Section VI, a primitive-based approach
is considered to solve the problem. In Section VII, we address the dynamic case by proposing a
two-stage motion planning strategy. In Section VIII, simulation results are presented. In Section
IX, we conclude with some future work.

MATHEMATICAL MODELING
In this section, we describe the mathematical models for the combine harvester and the grain
cart.

Combine Harvester
Combine harvester is the machine for harvesting crops, for example, wheat, oats, rye, barley
corn, soybeans and flax. Figure 1 shows a combine at work. In this active mode, the header
cuts the crop and feeds it into threshing cylinder. Grain and chaff are separated from the straw
when crop goes through the concaved grates. The grain, after being sieved, will be stored in the
on-board tank temporarily, and the waste straw is ejected. We use C to denote the maximum
capacity of on-board tank.

Figure 1. Combine harvester

Threshing grain loss is an important issue for combine harvester. For any combine, the quantity
of threshing grain loss greatly depends on the forward speed of the harvester. In (Flufy and
Stone 1983), authors show that automatic control has a better performance than manual control
on the threshing grain loss. The forward speed is controlled to give a level of crop feed
according to the required threshing grain loss. In this paper, we simplify the model and assume
that all the combines posses identical constant speed, denoted as vch .

Since the tank does not hold a large capacity, modern combine usually has an unloading auger
for removing the grains from the tank to other vehicles. For most of the combines, the auger is
mounted on the left side, as shown in Figure 1. At this point, a vehicle has to be on the left side
of the combine to empty the tank. Here we denote the unloading rate of the tank using auger as
ru . So when a combine proceeds with the harvesting and the unloading operations
simultaneously, the unloading rate is ru  rf (ru ! rf ) .

Zhang, Bhattacharya 249

Grain Cart
A grain cart, also known as chaser bin, is a trailer towed by a tractor. In this paper, we use term
grain cart to represent the system including both the tractor and the trailer. Figure 2(a) shows
the appearance of a grain cart. Because of the larger capacity, one can use it to collect grains
from multiple combines so that the combines could work without interruption.

Figure 2. Grain cart

Figure 2(b) shows a grain cart. We model the grain cart as a trailer attached to a car-like robot.
The robot is hitched by the trailer at the center. The equations of motion for the grain cart are as
follows.

§ x · § v cos T ·
¨ ¸ ¨ ¸
y
q ¨  ¸ ¨ v sin T ¸
¨T ¸ ¨ Z ¸
¨ ¸ ¨ ¸
©E ¹ © v sin E  Z ¹

where q ( x, y, T , E )  2 u S1 u S1 is the configuration, u (v, Z )  U [1,1]2 is the control. In

the configuration q , ( x, y ) is the coordinate of robot's center, T is the robot's orientation, E is
the angle between trailer orientation, and the robot orientation. v and Z denote the speed and
angular velocity of the robot, respectively (Chyba and Sekhavat 1999) (Chitsaz 2013).

PREVIOUS WORK
In previous work, we addressed the logistics scheduling problem of grain cart during harvesting
operation. A scheduling scheme was proposed for an arbitrary number of combines with a
single grain cart. Based on the mathematical models and proposed scheme, the grain cart could
serve multiple combines without stopping harvesting operation. In the scheme, the grain cart
was scheduled to serve the combines sequentially, so that it needed to move along a trajectory
from one combine to the next combine. Figure 3 shows an example of the path planning. Based
on the scheduling scheme of N-combine case, we have obtained

Zhang, Bhattacharya 250

 (ru  rf )C  ( N  1)rf C
'T d (1)
rf (ru  rf )(2 N  2)

where 'T denotes the travel time for the grain cart to switch. Since the travel time is
constrained, in this paper we focus on finding the time-optimal solution to the path planning
problem. In the next section, we provide an elaborated problem description.

Figure 3. Path planning of the grain cart

PROBLEM DESCRIPTION
In this section, we formulate the problem. Consider a grain cart moving from one combine to the
next combine, we would like to find the time-optimal solution for the grain cart to perform the
path planning. In other words, given initial configuration, denoted as qi and goal configuration,
denoted as qg , we intend to compute the path from qi to qg minimizing the total travel time. In
the following sections, we first consider the case when initial and final configurations are static,
following which, we extend the result to a dynamic target configuration case.

NUMERICAL APPROACH
In this section, we present a numerical approach to solve the navigation problem between two
given static configurations. Before computing the trajectory, we first define a value function and
establish Hamilton-Jacobi equation based on tractor-trailer model. Then we present an update
scheme for computing the value function. Finally, the trajectory is computed using the obtained
results.

Hamilton-Jacobi Equation
Denote the set of admissible path from the configuration qi as A xi , yi ,Ti , Ei . Given a goal

configuration qg , we define the corresponding value function u : q o ‰ {0} as follows
(Takei, Tsai, Shen, and Landa 2010).

u (q (T )) inf{T : q (t )  A xi , yi ,Ti , Ei , q (T ) qg } (2)

The value function can be regarded as the optimal cost-to-go for the tractor-trailer model with
certain constraints, an initial configuration and a final configuration. By applying dynamic
programming principle for (2), we have

Zhang, Bhattacharya 251

 u (q (t )) inf{u (q (t  't ))  't : q (t )  A xi , yi ,Ti , Ei } (3)

Dividing the terms by 't and taking 't o 0 , we are able to derive

1 inf{ͪu ˜ q :| v | 1,| Z |d 1} (4)

With the equations of motion of the grain cart, Hamilton-Jacobi-Bellman equation is obtained as
follows.

wu wu wu wu wu
1 cos(T )  sin(T )  sin( E )  inf{T(  )} (5)
wx wy wE |Z|d1 wT wE

The last term in (5) can be eliminated by applying bang-bang principle w r1 . Since qi is the
goal configuration which has no cost-to-go, we have u (qg ) 0 . For the points located in the
obstacle or outside the space, we define the cost-to-go to be infinity. In the next section, we
present an update scheme for the defined value function.

Update Scheme
In order to find the time-optimal path satisfying (5), we apply fast sweeping method and propose
an update scheme for the value function u (q ) in the entire space. The basic idea is to employ
the fact that the value function has zero cost-to-go at the goal configuration, and to compute the
value function from the nodes close to the goal configuration, to the nodes at farther positions.

With this in mind, we first set up a four dimensional uniform Cartesian grid with refinement
(hx , hy , hT , hE ) . Let ua ,b ,c ,d u (qa ,b ,c ,d ) u (ahx , bhy , chT , dhE ) be the approximation of the solution
on the grid nodes. Moreover, we discretize Z in the range of [1,1] and further define ua*,b ,c , d as
follows.

ua*,b ,c ,d min Zi [ 1,1]{u (qa ,b ,c ,d  q 't )}  't (6)

where q (cos(chT ),sin(chT ), Zi ,  sin( dhE )  Zi ))T , Zi is the ith element in the discretization and
't is the length of time step. For the value of u (qa ,b ,c ,d  q 't ) , we take the value directly if it is
lying on the presented grid, otherwise it is approximated by taking the average value of the
adjacent nodes.

Finally, the update scheme can be described in the following way.

uan,b1,c ,d min{uan,b ,c ,d , ua*n,b ,c ,d } (7)

where the superscripts denote the iteration. We set up the termination condition of the
computation as follows.

Zhang, Bhattacharya 252

 ( uan,b1,c ,d  uan,b ,c ,dý2 ) 2  ò
ý (8)

where ò ! 0 is an arbitrary number.

Computing Trajectory
By using the obtained value function, we are able to derive the time-optimal path from any initial
configuration qi to the goal configuration qg . According to (5), the control law can be
summarized as follows.

x cos T
y sin T
wu wu (9)
T  sgn(  )
wT wE
E  sin E  T.

Note that the partial derivative in (9) is obtained by applying central difference approximation. In
the computation of trajectory, the values of u which are not on the grid are computed using a
nearest-neighbor interpolation.

The numerical approach computes the time-optimal trajectory efficiently if the corresponding
value function is provided. The main time consumption is in computing the value function of the
final configuration. But in real implementation, one can compute the value function beforehand.
Therefore, the time cost will not influence the real-time operation on path planning. In the next
section, we consider another approach to address the situation when we do not have such
value function.

PRIMITIVE-BASED APPROACH
Related Work
Based on the model presented in Section II, the time-optimal trajectory satisfies Pontryagin
Maximum Principle. In (Chitsaz 2013), authors define adjoint variables O (Ox , O y , OT , OE ) and
the Hamiltonian as follows.

H (q, O , u ) Ox x  O y y  OT T  OE E v(Ox cos T  O y sin T  OE sin E )  Z (OT  OE ) (10)

Additionally, the switching functions are defined as

Iv Ox cos T  O y sin T  OE sin E

(11)
IZ OT  OE

Depending on Iv and IZ , the optimal trajectory, which is called extremal, consists of two
categories, namely, regular and singular. On one hand, an extremal is called regular if the times

Zhang, Bhattacharya 253

at which Iv 0 or IZ 0 have zero measure. On the other hand, an extremal is called
abnormal if it has both Iv { 0 and IZ { 0 . A singular is an extremal which contains a positive
measure along which Iv { 0 or IZ { 0 . The subtrajectories of a regular and a singular extremal
are called regular primitive and singular primitive, respectively.

In the regular primitive, Iv z 0 and IZ z 0 . Based on the state equations, a regular primitive
satisfies

T (t ) Zt  T (t0 )
x(t ) x(t0 )  (v / Z )(sin(T )  sin(T (t0 )))
y (t ) y (t0 )  (v / Z )(cos(T )  cos(T (t0 )))
t  2v  K1 (12)
E (t ) 2(v / Z ) arctan( )
t  K1
2
K1
v  Z tan( E (t0 ) / 2)

where t0 is the starting time instant and t is the passing time.

In singular primitive, we have either Iv { 0 or IZ { 0 . Here since the grain cart has a constant
forward speed, we only consider the case of IZ { 0 . It has been proved by (Chitsaz 2013) that if
a IZ -singular primitive contains a straight line segment, either the entire primitive is a straight
line segment, or

D (t ) r2E (t )
Z (t ) d r2sin( E (t )) (13)
S S 5S 7S
 d E (t ) d or d E (t ) d
6 6 6 6

in which D denotes the angle between the robot orientation and the line, d denotes the
distance between robot's center and the line. The path for the latter case is called a merging
curve.

Path Planning of Grain Cart

In our case, when the grain cart finishes unloading one combine, it should follow a path to the
second combine, and move parallel to it. This implies that the final path is supposed to end with
a singular primitive, denoted as S g . Considering the fact that a merging curve has the
constraints (13), we propose a combination of the path containing two regular primitives and two
IZ -singular primitives, as shown in Figure 4. The grain cart initially proceeds with a regular
primitive R1 . S1 is a singular primitive connecting to R1 . R2 and S g are the following regular
primitive and goal singular primitive, respectively.

Zhang, Bhattacharya 254

 Figure 4. The path consists of 2 singular primitives (red lines) and 2 regular primitives
(blue lines)

In order to minimize the travel time for the grain cart, we consider using straight lines instead of
merging curves for IZ -singular primitives. Furthermore, since qi and qg has the same T , it is
apparent that R1 and R2 should have the same length, and the same central angle J as well.
Therefore, with a given J , the slope of S1 can be computed. Since qi and qg are known, the
entire path can be derived. To minimize the travel time, we change the central angle
corresponding to the regular primitives, and take the trajectory with minimum time in all feasible
trajectories as the final path. The complete algorithm can be found in the following algorithm.

Trajectory computation using primitive-based approach:

Declare J to be the central angle of regular primitives, T f f to be the travel time of final path,
Pf to be the final path

For J 0 oS
Compute the path PJ starting from qi

If PJ reaches qg

If travel time of PJ  T f

Tf Travel time of PJ , Pf PJ
End if
End if
End for

DYNAMIC MOTION PLANNING

In this section, we consider the case when the target configuration is dynamic. To accommodate
this situation, we present a motion planning strategy that consists of two successive stages. In

Zhang, Bhattacharya 255

the first stage, we estimate a fixed goal configuration and perform the static path planning using
aforementioned approaches. In the second stage, we apply a PID control for the grain cart to
move synchronizing with the combine. Next, we elaborate both stages in detail.

Since the target configuration is mobile, we consider qg as a function of time,

i.e. qg (t ) ( xg (t ), yg (t ), T g (t ), E g (t )) . Figure 5 indicates the basic idea of the estimation on the
T

static goal configuration. In the figure, t0 denotes the starting time. Thus qg (t0 ) could be
considered as the goal configuration at the beginning. At this point, we attempt to find a 'T
satisfying the ideal case that the grain cart can successfully reach the desired goal
configuration qg (t0  'T ) . With this in mind, we approximate the 'T by using a lower bound on
the length of the path, which is v'T , as shown in Figure 5. Based on geometry, the
approximated 'T could be solved using the following equation.

( xg (t0 )  xi  vch 'T ) 2  ( yg (t0 )  yi ) 2 (v'T ) 2

d12 v 2  d 22 (v 2  vch2 )  d1vch (14)
'T
v 2  vch2

where d1 xg (t0 )  xi and d 2 yg (t0 )  yi .

Figure 5. Dynamic motion planning

We let qi and qg (t0  'T ) to be the initial and goal configuration in performing static path
planning. Due to the fact that v'T is the lower bound on the path length for a non-holonomic
vehicle, the grain cart will lag behind the combine when it reaches the goal configuration.
Therefore, in the second stage, we apply a PID feedback control for grain cart to catch up with
the combine. Figure 6 shows the block diagram for the presented control system. Position of the
combine is considered to be the reference input. PID control is applied to the control of the grain
cart's speed.

Zhang, Bhattacharya 256

 Figure 6. Block diagram for PID control system

SIMULATION
In this section, we present simulation results for the aforementioned techniques.

Figure 7 illustrates the paths obtained using numerical approach and primitive-based approach.
Because of the fact that the tractor-trailer model has four state variables, it is hard to visualize
the variations of all these variables. For this reason, in the simulation we only show the path of
( x, y ) , which represents the physical location of grain cart in the environment. A colored arrow
is added to show the final orientation. In this simulation, initial and goal configuration are set to
be qi (1,1, 0, 0)T and qg (4, 4, 0, 0)T , respectively. For numerical approach, Table 1 lists all
the refinement parameters in the computation of value function. In both approaches, the path
computation terminates when the state of the grain cart reaches a small range of the goal
configuration.

Table 1. Refinement parameters

Name Value
hx 0.2
hy 0.2
hT 0.251 (radium)
hE 0.251 (radium)
't 0.05
ò 0.005

Zhang, Bhattacharya 257

 Figure 7. Simulation results with given initial and final configurations

Simulation results show that both paths finally reach the goal configuration which validates the
proposed approaches. In the numerical approach, the path could be affected by the error of
using inappropriate refinement parameters. With proper refinement parameters, the path could
be more accurate, but it will lead to higher time consumption in the computation of value
function.

In the second simulation, we compare the performance of using two proposed approaches. We
xg  xi
keep yi yg  4 and plot the travel time with respect to the distance ratio , as shown in
y g  yi
Figure 8. The results show that with a given ratio, travel time of the two approaches are very
close. Since the numerical approach provides us the time-optimal solution, it can be seen that
the primitive-based approach also has a good performance. Note that in some cases, the
performance of primitive-based approach is better because of the effect of the error in numerical
approach.

Zhang, Bhattacharya 258

 Figure 8. Travel time of using two approaches

Figure 9 illustrates the last simulation, which is an implementation of using dynamic motion
planning. In this simulation, initial configuration of grain cart and initial position of combine are
set to be (1,5, 0, 0)T and (5,9) , respectively. The speed of combine vch 0.4 , whereas the
speed of grain cart v has the maximum 1. The simulation shows the paths of grain cart in both
stages, which demonstrate the feasibility of the proposed dynamic motion planning strategy.

Figure 9. Dynamic motion planning

CONCLUSION
In this paper, we addressed the problem of finding the time-optimal path for the grain cart to
navigate from one combine to the adjacent combine. Firstly, a numerical approach for
computing the path based on given static initial and final configuration was presented.

Zhang, Bhattacharya 259

According to the tractor-trailer model, we presented an update scheme for computing the value
function corresponding to the entire state space. Finally, the trajectory could be computed with
the help of this value function. Subsequently, we presented a primitive-based approach to
obtain the solution. The path was considered to be constructed by regular and singular
primitives that have local time optimality. Furthermore, we studied the case when target
configuration was dynamic, and presented a motion planning strategy adopting the results from
aforementioned approaches. Finally, simulation results were provided for these techniques. For
the future work, we will consider a case of multiple grain carts, and try to coordinate grain carts
so that the harvesting efficiency could be further improved.

REFERENCES
O. Ali and D. Van Oudheusden, “Logistics planning for agricultural vehicles,” in Industrial
Engineering and Engineering Management, 2009. IEEM 2009. IEEE International
Conference on, pp. 311--314, Dec 2009.

A. Astolfi, P. Bolzern, and A. Locatelli, “Path-tracking of a tractor-trailer vehicle along rectilinear

and circular paths: a lyapunov-based approach,” Robotics and Automation, IEEE
Transactions on, vol. 20, pp. 154--160, Feb 2004.

C. B. Basnet, L. R. Foulds, and J. M. Wilson, “Scheduling contractors' farm-to-farm crop

harvesting operations,” International Transactions in Operational Research, vol. 13, no.
1, pp. 1--15, 2006.

H. Chitsaz, “On time-optimal trajectories for a car-like robot with one trailer,” CoRR, pp. --1--1,
2013.

M. Chyba and S. Sekhavat, “Time optimal paths for a mobile robot with one trailer,” in Intelligent
Robots and Systems, 1999. IROS '99. Proceedings. 1999 IEEE/RSJ International
Conference on, vol. 3, pp. 1669--1674 vol.3, 1999.

A. Divelbiss and J. Wen, “Nonholonomic path planning with inequality,” in Robotics and
Automation, 1994. Proceedings., 1994 IEEE International Conference on, pp. 52--57
vol.1, May 1994.

A. Divelbiss and J. Wen, “Trajectory tracking control of a car-trailer system,” Control Systems
Technology, IEEE Transactions on, vol. 5, pp. 269--278, May 1997.

K. Ferentinos, K. Arvanitis, and N. Sigrimis, “Heuristic optimization methods for motion planning
of autonomous agricultural vehicles,” Journal of Optimization, vol. 23, no. 2, pp. 155--
170, 2002.

Y. Hao, B. Laxton, E. R. Benson, and S. K. Agrawal, “Differential flatness-based formation

following of a simulated autonomous small grain harvesting system,” Transactions of the
ASABE, vol. 47, no. 3, pp. 933--941, 2004.

M. L. Flufy and G. Stone, “Speed control of a combine harvester to maintain a specific level of
measured threshing grain loss,” Journal of Agricultural Engineering Research, vol. 28,
no. 6, pp. 537 -- 543, 1983.

Zhang, Bhattacharya 260

B. Fokkens and M. Puylaert, “A linear programming model for daily harvesting operations at the
large-scale grain farm of the ijsselmeerpolders development authority,” The Journal of
the Operational Research Society, vol.32, no. 7, pp. pp. 535--547, 1981.

L. Foulds and J. Wilson, “Scheduling operations for the harvesting of renewable resources,”
Journal of Food Engineering, vol. 70, no. 3, pp. 281 -- 292, 2005. Operational Research
and Food Logistics.

I. Hameed, D. Bochtis, and C. Sørensen, “An optimized field coverage planning approach for
navigation of agricultural robots in fields involving obstacle areas,” International Journal
of Advanced Robotic Systems, vol. 10, no. 231, pp. 1--9, 2013.

J. Isaacs, D. Klein, and J. Hespanha, “Algorithms for the traveling salesman problem with
neighborhoods involving a dubins vehicle,” in American Control Conference (ACC),
2011, pp. 1704--1709, June 2011.

Z. Liu, Q. Lu, P. Yang, and L. Chen, “Path planning for tractor-trailer mobile robot system based
on equivalent size,” in Intelligent Control and Automation, 2008. WCICA 2008. 7th World
Congress on, pp. 5744--5749, June 2008.

T. Makino, H. Yokoi, and Y. Kakazu, “Development of a motion planning system for an

agricultural mobile robot,” in SICE Annual, 1999. 38th Annual Conference Proceedings
of the, pp. 959--962, Aug 1999.

T. Oksanen and A. Visala, “Coverage path planning algorithms for agricultural field machines,”
Journal of Field Robotics, vol. 26, no. 8, pp. 651--668, 2009.

R. Takei, R. Tsai, H. Shen, and Y. Landa, “A practical path-planning algorithm for a simple car:
a hamilton-jacobi approach,” in American Control Conference (ACC), 2010, pp. 6175--
6180, June 2010.

X. Yu and J. Hung, “Optimal path planning for an autonomous robot-trailer system,” in IECON
2012 - 38th Annual Conference on IEEE Industrial Electronics Society, pp. 2762--2767,
Oct 2012.

Zhang, Bhattacharya 261