Automation in Construction 138 (2022) 104217

Contents lists available at ScienceDirect

Automation in Construction
journal homepage: www.elsevier.com/locate/autcon

IMU-based full-body pose estimation for construction machines using

kinematics modeling
Jingyuan Tang, Han Luo *, Weiwei Chen *, Peter Kok-Yiu Wong, Jack C.P. Cheng *
Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong, China

A R T I C L E I N F O A B S T R A C T

Keywords: Monitoring unsafe operations of construction machines is important for reducing accidents on sites. To automate
Inertial Measurement Unit (IMU) safety monitoring, this paper studies pose estimation of construction machines, which describes their motions in
Full-body pose estimation 3D space. Inertial Measurement Unit (IMU) is considered accurate, cost-saving and user-friendly for estimating
Kinematics model
the pose of machines. Nevertheless, the complexity and instability of construction sites hinder the application of
Construction safety
Machine operation
IMU to construction machines. Hence, this study first defines the information requirements of IMU-based full-
body pose estimation for construction machines. Then, an efficient method is developed to generate the tra­
jectories of a machine based on kinematics. Moreover, an optimal installation scheme for IMU sensors is
investigated systematically to optimize the location and number of IMUs to install. Overall, this study contributes
an IMU-based method to provide full-body poses of machines without environmental constraints on real con­
struction sites, and a theoretical basis of optimal IMU installation strategy for reliable motion-related analyses.

1. Introduction opinion and experience of the inspector. To improve the accuracy and
efficiency, automated monitoring of the operation safety of construction
The construction industry is recognized as one of the most hazardous machines is needed.
industries. According to the Occupational Safety and Health Statistics In the early stages, monitoring of operation safety was based on
from the Labour Department of Hong Kong [1], 33.4% of industrial object recognition, a technology that can provide categorical and loca­
accidents happened in the construction industry in 2018. The U.S. Bu­ tion information for construction machines [6–8]. However, this vague
reau of Labor Statistics (BLS) reported that 47.2% of all fatal work in­ information is not sufficient to replace the safety inspectors' eyes. For
juries happened in the construction industry in 2018 [2]. In the example, for municipal engineering construction, the machines need to
construction industry, the use of heavy construction machines is a major work in crowded downtown areas, and there is no obvious change in the
hazard factor on construction sites. Contact with moving or operational category and location of the machines. Nevertheless, great attention
machines was the third most common type of construction accident, as must be paid to avoid collision between moving components, pedes­
Hong Kong Labour Department reported in 2019 [3]. Any unsafe oper­ trians and vehicles. Obviously, object recognition is not enough to
ation of a heavy construction machine can result in serious accidents. retrospect the real construction site and help the safety inspectors to
Taking cranes as an example, between 2009 and 2013 in Hong Kong, make judgments. Therefore, researchers gradually realized that “pose” is
50% of all crane-related accidents were due to unsafe operations or the core of operation safety monitoring. Especially, the full-body pose is
failure to keep the machine steady [4]. Such accidents are causing in­ essential and necessary achieve the accurate monitoring in operation
juries and deaths, program delays, and huge financial losses. It is safety, which describes the complete motion of a construction machine
therefore important to monitor the operations of heavy machines on a including orientation of each independent component, relative motion
construction site to prevent accidents, as well as to improve operational of each pair of adjacent components, and system model of the machines.
safety and productivity. In the past, the operation safety monitoring of Fig. 1 shows the differences between the non-full-body pose and the full-
construction machines relied on inspectors visiting the construction sites body pose illustrated. Therefore, the full-body pose estimation of con­
or watching a video captured from surveillance cameras [5]. Such struction machines is expected to be introduced to the industry.
human supervision is a tedious and inefficient way and is subject to the In recent years, with surveillance cameras extensively installed on

* Corresponding authors.
E-mail address: [email protected] (J.C.P. Cheng).

https://doi.org/10.1016/j.autcon.2022.104217
Received 11 May 2021; Received in revised form 15 February 2022; Accepted 17 March 2022
Available online 6 April 2022
0926-5805/© 2022 Elsevier B.V. All rights reserved.
J. Tang et al. Automation in Construction 138 (2022) 104217

construction sites, studies have drawn attention to using computer constraints on real construction sites.
vision to monitor heavy machines by tracking their partial or full-body The rest of the paper is organized as follows. Section 2 reviews
poses [9,10]. However, this technology may not be able to keep up with relevant research on the pose estimation of construction machines using
the dynamism of a construction site, as its accuracy may be affected by, both visual and non-visual sensors. Section 3 describes the proposed
for example, insufficient lighting on a cloudy day, or targets occluded by information flow on the IMU-based full-body pose estimation of con­
other objects or clutter. 3D estimation, on the other hand, requires struction machines. In Section 4, an illustrative experiment is designed
special considerations of camera use, optical physics, site geometry and and conducted to validate the approach. Section 5 describes the optimal
layout, etc. To get around this problem, previous studies would be to IMU installation scheme based on experiments. Section 6 discusses the
non-invasively install non-visual sensors on the surface of machines or limitations and provides conclusions of this research.
set on site [11,12]. Among various approaches, the Inertial Measure­
ment Unit (IMU) with its low cost, user-friendliness, and reliable accu­ 2. Related works
racy has become a hot topic [13]. IMU sensors are usually of two types
— 6-axis IMU (an accelerometer and a gyroscope), 9-axis IMU (an This section reviews relevant research on using both visual and non-
accelerometer, a gyroscope, and a magnetometer) — and are able to visual sensors to estimate the pose of construction machines.
track the rotation of an object in 3D with good accuracy. The existing
information requirements designed for human posture estimation and 2.1. Pose estimation of construction machines based on visual sensors
gait analysis is inadequate in meeting the needs of the construction in­
dustry regarding to machine. Despite appearing to be suitable and Images collected by visual sensors provide information including
capable, there are still many challenges for pose estimation of con­ color and geometry, which is required by computer vision for the pose
struction machines using IMUs, such as the vibration of the machine, the estimation system. The robotic total stations and laser scanners [16]
interference due to the complex environment on sites, and the instablity provide 3D motion information for the pose estimation system for con­
of long-time work. In addition, reducing the computational load is also struction machines on site. Reliable data and precise results are highly
one of the requirements to improve the efficiency. Even though attempts significant characteristics of the pose estimation system for construction
have been made to employ IMUs, there is no comprehensive definition of machines based on precision devices, such as the robotic total station
the information flow for efficient pose estimation of construction ma­ and the laser scanner. However, due to the difficulty in multi-object
chines, which is needed for tracking the poses of machines on con­ identification, each estimated construction machine is required to be
struction sites [14,15]. equipped with a dedicated device. High device costs and complex site
This study therefore aims to define the information requirements on layouts make this method less feasible in practical application.
pose-related analysis of construction machines using IMU data. Based on Compared with data collection using precision devices, the use of a
the proposed information flow, the current full-body poses of con­ camera to capture image data is a cheaper and more convenient method.
struction machines can be estimated using a kinematics model and IMU Most of the recent related research used computer vision. Marker-based
data. In the study, we discuss the common motion modes of movable pose estimation, for example, uses optical cameras to monitor fiducial
components (e.g., lifting, droping, and rotating), which are generaliz­ markers, then estimates the spatial movements and orientations of the
able to different construction machines rather than specific types of markers. In previous studies, a fiducial marker would be attached onto a
machines. In this study, the movable components (e.g., bucket, arm, component of a machine (e.g., an arm, boom, or bucket of the exca­
boom, and cabin for excavators) refer to the components that can have vator), and then the motion status of the component would be estimated
independent motion as the machine is being operated. The approach by observing the orientation and location of the fiducial marker through
includes: (1) data collection and preprocessing, (2) orientation estima­ a camera [17,18]. Marker-based pose estimation helps to develop a low-
tion, (3) kinematics modeling, and (4) trajectory generation. Based on cost pose estimation system. However, marker-based sensors have
the developed approach, the study carries on exploring and discussing obvious shortcomings: besides durability, any form of marker occlusion
where and how many IMU sensors should be installed on a construction may result in distortion of the results, but in a dynamic site environment,
machine to enable the best IMU-based motion analyses. This study it is difficult to keep cameras and markers free from occlusion. There­
contributes an efficient information flow designed for pose-related fore, marker-based pose estimation lacks operability in construction
analysis of common construction machines using IMU data, as well as [19].
a theoretical basis and a reliable strategy for optimal IMU installation in Unmarked image processing is more flexible and therefore more
motion-related analyses. Using the given approach and installation feasible than marker-based methods for estimating the poses of con­
strategy, full-body and 3D pose information of construction machines struction equipment. Early on in the related research, edge detection
can be provided efficiently and stably without environmental based on conventional computer vision was common [20]. Yang et al.

Fig. 1. Examples of the full-body pose and the non-full-body pose.

2
J. Tang et al. Automation in Construction 138 (2022) 104217

[21] used video captured from a static surveillance camera to identify corrosion in the construction environment. Therefore, to ensure the
two categories of activities for tower cranes. Silhouette-based tracking accuracy, during the whole construction period, a sensor installed at a
algorithms were used to extract the edge of the jib on binary images and certain position on a machine needs to be replaced many times. When
to estimate the trajectory. However, due to the lack of RGB information, there are multiple machines, which have multiple components, they
segmentation results based solely on edge detection are unreliable. To need to be monitored during the long working period, the cumulative
improve their accuracy, Soltani et al. [10] presented an approach to cost caused by frequent replacement of these sensors will increase
estimate the pose of a partial excavator based on a 2D skeleton and significantly. Hence, these methods still lose out on the feasibility and
synthesis image. The K-means is introduced in the method to cluster the competitiveness in the construction industry due to the accumulated
RGB information. Then, the method extracted the object in the fore­ high costs of monitoring devices.
ground and estimated the orientation of each component of the exca­ Therefore, low cost, user-friendliness, reliability and accuracy form
vator through skeletonization. However, pose estimation of construction the criteria for selecting non-visual sensors as part of a system for esti­
machines based on conventional computer vision technology requires mating the poses of construction machines. With the rapid development
multiple detectors to be trained and numerous predefined features to be in MEMS inertial sensors, the sharp decline in the cost of IMUs allows
manually annotated. This process is not only tedious but is also likely to such devices to be adopted widely in the construction industry. In terms
result in inaccuracies. To address the problems of conventional com­ of user-friendliness, the lightweight and small size of the IMU also
puter vision technology, Luo et al. [9] proposed an end-to-end deep facilitate non-intrusive installation and old-machine transformation. In
learning method for machine full-body pose estimation. Using images terms of accuracy, IMU applications in human posture estimation and
collected from a surveillance camera, and labeling them with predefined gait analysis have proven that IMU-based posture estimation is both
keypoints, the authors trained three deep learning networks (i.e., accurate and stable. This method has been widely studied and applied in
Stacked Hourglass Network (HG), Cascaded Pyramid Network (CPN), fields like engineering and clinical rehabilitation [27,28], therefore IMU
and an ensemble model (HG-CPN) integrating Stacked Hourglass and sensors offer a potential solution to economical and accurate pose esti­
Cascaded Pyramid Network) to estimate the full-body pose of an exca­ mation of construction machines.
vator. The method showed good performance with an accuracy of over Researchers have made preliminary attempts to use IMU sensors to
90%. estimate the poses of construction machines: Talmaki et al. [29] pro­
Computer vision technology relying on camera images may be an posed a real-time damage prevention framework for excavation ma­
effective, low-cost and user-friendly way to estimate the poses of con­ chines based on hybrid virtuality. In this framework, they took the
struction machines, but it has obvious practical disadvantages, as fol­ rotations of moveable components as one of the inputs, which were
lows: First, in a dynamic, outdoor construction site, there may be provided by orientation sensors, including inertial components. There­
insufficient lighting (such as on a cloudy or rainy day), obstruction of fore, the possibility of using single inertial component—accelerometer
view, and frequent background changes and movements affect its ac­ to obtain these rotations was considered. Although the accelerometers
curacy. In addition, due to the limited range of the cameras, additional were eventually not involved in this work, the consideration prelimi­
special considerations must be given to camera layouts. narily showed the potential of applying IMU-based orientation estima­
tion in the construction industry. After that, the IMU-based orientation
2.2. Pose estimation of construction machines based on non-visual sensors sensing for independent components of construction machines was
further studied. For example, in the predictive control model of hy­
To address the shortcomings of visual sensors in pose estimation, draulic mini excavators developed by Bender et al. [30], IMU data were
studies have been performed on non-visual sensors installed non- used to estimate the orientation of independent components for the
invasively on the surface of construction machines or set on sites. excavator, and then provide the model input (cylinder strokes and slew
Many previous studies have used precision devices or high-precision angles). Unfortunately, although Kalman filter used in the paper has
location technology to obtain a more accurate spatial description of potentials of wide application, this work didn't provide complete pose
the motion state of construction machines. For example, high-density information based on the physical relationship between components.
light detection and ranging (LiDAR) devices can accurately estimate However, in automatic monitoring of operation safety, complete pose
machine poses by generating and processing point-cloud data, even in a information of construction machines is essential and necessary,
complex, unpredictable environments [12,22]. Zhang et al. [11] inves­ including orientation of each independent component, relative motion
tigated in detail the requirements and system settings for the ultra- of adjacent components, and system model of the machines. Therefore,
wideband (UWB) real-time location system (RTLS), and proposed a recently researchers put more effort on exploring the problem as how to
method to estimate crane poses using UWB RTLS, which performed well track the poses of construction machines using IMU. For example,
estimating the poses of crane booms. Companies such as Leica [23] and Péntek et al. [15] proposed both an offline and an online automatic
Trimble [24] also provided a commercial pose-estimation solution based parameter estimation method for IMU-based pose estimation of con­
on Global Positioning System (GPS). The reliable data collected from a struction vehicles, and tried to find a general pose estimation method for
high-precision GPS may suit the purpose of a construction site quite construction vehicles in control environment using the least square fit,
well. For example, Kim et al. [25,26] presented a method to estimate the the Kalman filter model and IMU data. However, the following limita­
position of the cutting edge of a bulldozer's blade using a sensor fusion tions make it unrealistic on construction sites and impossible to achieve
system. The above-mentioned system, consisting of a dual commercial satisfactory accuracy: (1) The proposed methods cannot deal with the
RTK GPS sensor and two motion sensors, can estimate the orientation inaccuracy caused by unmodeled errors (e.g., strong vibration); (2) The
and position of the blade with errors within 30 mm. high response latency caused by data post-processing cannot satisfy the
However, even if they manage to achieve excellent performance in requirements of operation safety monitoring. Lee et al. [14] presented a
pose estimation, the studies that use precision devices and high- method of excavator partial pose estimation (boom, arm, and bucket
precision location technologies such as LiDAR, UWB, and GPS are still movements) without processing, remodeling or transformation in the
not accepted by industry. In addition to the difficulty of device layout on existing system. Although it proposed the concept of IMU-based pose
the construction site and subsequent development, the high price of estimation for excavators, the study did not explore the full-body esti­
these kinds of devices has greatly increased the cost of safety monitoring mation method in detail and did not get convincing results.
for construction enterprises, and has become a fatal obstacle in the Generally, there is a lack of IMU-based full-body pose estimation
development of these technologies. Specifically, sensors which are framework that can be proved to be unified among different construc­
installed on the surface of construction machines directly or set on sites tion machines. In addition, these mentioned studies on IMU-based pose
need to be replaced periodically due to unavoidable collisions and estimation have a common problem: The lack of consideration of

3
J. Tang et al. Automation in Construction 138 (2022) 104217

machine attributes and site characteristics make the proposed methods transformed into sequences of quaternions via an orientation filter,
difficult to apply to real scenario and evaluate their performance: Spe­ which describes the rotational states of the machine components; this
cifically, the pose estimation of construction machines needs to consider step is called orientation estimation. Third, through constructing a ki­
the vibration of the machine, the interference due to the complex nematics model, the relationships between the rotational states of the
environment on site, and to ensure the stability of long-time work as far components are investigated, and keypoints that represent the spatial
as possible. In addition, to reduce the computational load is also one of positions of the components are defined. Fourth, based on the rotation
the requirements to improve the efficiency. Considering these research states of the components and the kinematics model, trajectories of
gaps, when IMU is applies to construction industry, it is necessary to defined relative angles are generated via a quaternion-based method.
define comprehensive information requirements of IMU-based full-body Fig. 3 shows the specific information requirements in the framework.
pose estimation for construction machines, which can satisfy the need of More details are given in the following sub-sections.
operation safety monitoring on construction sites.
3.1.1. Data collection and preprocessing
3. Methodology First, raw data are collected from the IMU sensors. Since the working
environment and the IMU devices are likely to introduce noise, pre­
As suggested in Section 2, the advantages of IMU sensors are its low processing the raw data is essential in eliminating errors for improved
cost, user-friendliness, reliability and accuracy. This study therefore accuracy in the subsequent stages.
develops an information flow based on IMU sensors to estimate the full-
body poses of construction machines. Fig. 2 shows the workflow of the 3.1.1.1. Data collection. Each inertial element (i.e., accelerometer, gy­
proposed approach, which consists of four steps: (1) data collection and roscope, and magnetometer) can be used for orientation estimation
preprocessing, (2) orientation estimation, (3) kinematics modeling, and separately. However, using any inertial element alone will produce
(4) trajectory generation. In addition, the study systematically discusses significant measurement errors, for example, local noise angles for ac­
and develops an optimal installation scheme (i.e., where and how many celerometers, drift away overtime for gyroscopes, and distortion for
such sensors should be installed on a machine) using the approach as the magnetometers. Hence, a fusion balance between multiple sorts of data
basis. More details on the study are given in the following sub-sections. is needed to achieve the better accuracy. Common IMU devices include
6-axis IMU and 9-axis IMU types. Considering that construction ma­
chines usually undertake long-time work, to ensure its stability, it is
3.1. Information flow of IMU-based full-body pose estimation for necessary to calibrate the gyroscope's drift error in the measurement
construction machines process using the magnetometer. Therefore, the external entity —9-axis
IMU— should be used to input three types of raw inertial data: accel­
Both the representation and processing of information are important eration data collected from the accelerometer of each sensor, angular
factors which affect the accuracy and the efficiency of pose estimation velocities collected from the sensor's gyroscope, and magnetic fluxes
for construction machines. To address specific problems, a framework of from its magnetometer. IMU sensors are attached to the surfaces of
information flow is designed. We explore what processes are needed to different movable components of a construction machine, to ensure that
overcome the targeted difficulties, such as the vibration of the machine there is at least one IMU sensor installed on each machine component. In
and the interference from the complex external environment, and to addition, the IMU sensors should be installed near the junction of each
ensure stability during long-time work. In addition, the representation of pair of adjacent components, which is the optimal location to install the
information participating in computation is also considered to reduce sensors.
the computational load and improve the efficiency. The focus is on the
generic framework of information flow (i.e., the information input, 3.1.1.2. Data preprocessing. Vibration in a construction environment
output, and processing), rather than evaluating any specific algorithms, and uncertainty of data collection devices introduce errors that will
i.e., our framework should be applicable to different existing algorithms. accumulate, affecting the quality of estimation. Hence, the raw data
As depicted in Fig. 2, the framework of information flow estimates from each inertial element must be preprocessed to reduce error.
the pose of a machine through the following four steps. First, raw data For accelerometers and gyroscopes, their systematical errors have
are collected from IMU sensors installed on various movable compo­ similar components, including scaling error, bias, and axis rotation. Eqs.
nents of the machine, and they are preprocessed (i.e., it is calibrated and (1) and (2) shows the systematical sensor error model.
undergoes noise reduction). Second, the preprocessed data are ( )
aO = T a K a aS + ba (1)
( )
ωO = T g K g ωS + bg (2)

where,
aO and ωO denote the ideal measurement accelerations and angular
velocities.
Ta and Tg denote the errors caused by axis rotation of accelerations
and angular velocities.
Ka and Kg denote the scaling of accelerations and angular velocities.
aS and ωS denote the real measurement accelerations and angular
velocities.
ba and bg denote the errors caused by the bias for accelerations and
angular velocities.
There are many ways to determine these systematical error param­
eters. For example, Tedaldi et al. [31] proposed a fast calibration
method of error parameters by the rotation of an accelerometer and a
gyroscope. This method has stable performance, fast processing speed,
and does not need the assistance of any external equipment. It has been
Fig. 2. Workflow of the proposed methodology. verified in testing of the low-cost IMU device (e.g., smart phone)

4
J. Tang et al. Automation in Construction 138 (2022) 104217

Fig. 3. Information Flow of IMU-based Full-body Pose Estimation for Construction Machines.

calibration experiment. In addition, the vibration of the construction on site (e.g., rebars and metal equipment). The hard‑iron effect causes
machine as it is being operated may also increase the noise on the ac­ the center of the sphere to deviate from (0, 0, 0), while the soft‑iron
celerations and angular velocities. Such noise comes from the vibration effect turns the sphere into an ellipsoid and produces rotation. These
of the engine, the bumpy ground, and the vibration caused by the mo­ distortions may even occur simultaneously and are corrected in two
tion. Therefore, accelerations and angular velocities need to be parts: (1) noise reduction, and (2) calibration.
smoothed to reduce the noise that the vibration brings. For example, in The following is how to corrects hard‑iron and soft‑iron distortions
signal processing, the moving average filter [32] is often used for noise and turn the magnetic-flux ellipsoid back into a sphere centered at the
reduction. The principle of the moving average filter is shown in Eq. (3). origin: To begin, some of the raw data points acquired by the magne­
After the noise reduction and the calibration, the preprocessed accel­ tometer may be noisy and are very likely to result in partial deviation
erations and angular velocities are the outputs. Fig. 4 shows acceler­ during the calibration process, and need to be found and removed.
ometer and gyroscope data before and after preprocessing. The left Clustering algorithms can be used to solve these problems. For example,
figure illustrates the raw accelerations and angular velocities without DBSCAN [33], a typical and effective algorithm, calculates the density of
any calibration. In the right figure, the raw data have been calibrated points near each data point. Points that lie within a high-density area are
systematical errors and smoothed. be fitted into the model, while points that fall inside a low-density area
are deemed noisy points and are removed. Fig. 5 juxtaposes magne­
x(n) + x(n − 1) + x(n − 2) + … + x(n − N + 1)
y(n) = (3) tometer data before and after noise reduction. After the noise reduction,
N
the ellipsoid formed by the magnetic flux data is prepared to be fitted.
where, Using ellipsoid fitting [34] in a 3D coordinate system, the geometric and
x denotes the input signal and y denotes the output signal. algebraic parameters of the ellipsoid are obtained to remove any har­
n denotes the calculation point currently. d‑iron and soft‑iron distortions. Afterwards, the raw data from the
N denotes the sampling size. magnetometer are adjusted by the parameters of the ellipsoid so that it
For the magnetometers, in the ideal case where there are no distor­ becomes as close to a sphere centered at the origin as possible, and the
tions from any ferromagnetic materials in the surroundings, magnetic preprocessed magnetic fluxes are the outputs. Fig. 6 shows the process of
flux data should form a sphere centered at (0,0,0) as the magnetometer ellipsoid fitting and adjustments.
rotates and scans 360 degrees. In the complex construction environ­
ment, however, the same data from the magnetometer may be affected 3.1.2. Orientation estimation
by hard‑iron and soft‑iron distortions caused by ferromagnetic materials The denoised data then proceeds to the orientation estimation step as
follows. The commonly used construction machine is composed of

Fig. 4. Examples of the accelerometer and gyroscope data before and after preprocessing.

5
J. Tang et al. Automation in Construction 138 (2022) 104217

Fig. 5. Examples of the magnetometer data before and after noise reduction.

Fig. 6. Examples of correction on both hard‑iron and soft‑iron effects via ellipsoid fitting and adjustment.

multiple movable components. Each movable component is indepen­ calculated in single chip at the same time. Therefore, to reduce the
dently operated for limited movement (e.g., lifting, droping, or computational load and improve the efficiency, it is very important to
rotating). Therefore, by describing the motion states of the independent select an efficient orientation representation.
movable components, a general pose estimation method can be pro­ There are three classic representations of rotation: (1) rotation
posed for most construction machines. Fig. 7 shows the examples of the matrices, (2) Euler angles, and (3) quaternions. Rotation matrices
movable components discussed in the study. Orientation estimation represent rotation by a matrix with 9 digits, which makes it inefficient in
means estimating the rotational state of a movable component of a pose estimation, especially inversion. Euler angles represent rotation by
construction machine. Considering the need of real-time updating of Pitch, Roll and Yaw, but the rotation order and singularity problem
operation safety monitoring, a large amount of data may need to be make it inconvenient to participate in the calculation directly.

Fig. 7. Examples of the movable components for construction machines.

6
J. Tang et al. Automation in Construction 138 (2022) 104217

Quaternions, on the other hand, are efficient and easy to calculate as element when performing a motion. For construction machines, their
they rely on only four digits (i.e., BAq = [q1 q2 q3 q4]), which can satisfy movable components can be abstracted as links. A joint, on the other
the needs of the construction industry [35]. Therefore, quaternions are hand, is the part that connects adjacent links. Only a limited range of
proposed as the representation of component orientation. The infor­ motions (i.e., translation and rotation) can occur at each joint. In con­
mation describing the orientations of components flows in quaternions struction machines, the connection relationship of any pair of adjacent
between processes is finally combined into the full-body pose informa­ components can be expressed by links and a joint. Fig. 8 shows examples
tion of construction machines. of the connection relationship of pairs of adjacent components for
In this process, the input preprocessed data are transformed into the construction machines.
quaternion-based orientation information of each component using a There are many ways to express the kinematics model. For example,
quaternion-based orientation filter. For example, the Madgwick filter use of the standard Denavit-Hatenberg (D–H) parameters [37] is a
[36] is a quaternion-based optimized gradient-descent orientation esti­ classic method to represent connections between links. Fig. 9 shows such
mation algorithm to calculate the rotations of the machine's components a representation. Accordingly, the description of the motion of a link is
using preprocessed data from the IMU sensors. Eq. (4) shows the basic determined by the relative positions and orientations of the axes of the
principle of the Madgwick Filter. The overall orientation estimation joints at both ends of the link. Therefore, the origin of each joint coor­
S S
E qest, t combines the estimations E q∇, t calculated from the accelerom­ dinate system is defined at the axes of the joints. The origins are the
eter and magnetometer at time t using the optimized gradient-descent keypoints that describe the spatial positions of the components of a
algorithm, and the orientation estimation ESqω, t from the gyroscope at machine.
time t by weight γt. After the orientation estimation, the quaternion Standard D–H parameters using standard D–H notations describing
sequences of component orientation are the outputs. the spatial pose of a set of links are used to describe the motion of the
links and to construct the kinematics equation. The four parameters are
S
q = γt SE q∇,t + (1 − γt )SE qω,t , 0 ≤ γt ≤ 1 (4)
E est,t divided into two groups: parameters related to the links (i.e., aj and αj),
which are used to determine the intrinsic characteristics of each link;
where,
and joint-related parameters (i.e., dj and θj), which are used to determine
E denotes the stationary object toward the north horizontally in the
the relative positions of adjacent links.
earth frame.
aj is the distance between the zj-1 axis and the zj axis along the xj axis
S denotes the state of the object after a motion in the sensor frame.
S (length of the link).
E q denotes the orientation change of state in the sensor frame
αj is the angle between the zj and zj-1 axes about the xj axis (torsional
relative to its state in the earth frame.
S angle of the link).
E qest, t denotes the estimate of orientation.
S dj is the distance from the origin of the j-1 coordinate system to the xj
E q∇, t denotes the orientation estimated by the accelerometer and
axis along the zj-1 axis (offset of the link).
magnetometer using the gradient-descent algorithm.
S
E qω, t denotes the orientation estimated by the gyroscope.
γ t denotes the weight between ESq∇, t and ESqω, t.

3.1.3. Kinematics modeling

Section 3.1.2 has laid out the principle behind estimating the
orientation of a movable machine component, and is only one inde­
pendent unit of a construction machine. Since each machine is
composed of multiple movable components, the estimated pose of a
machine is an ordered combination of the estimated orientations of all
these independent units. Hence, information on the connections be­
tween components are required to integrate the estimated orientation of
each component into a full-body pose of the whole machine. A robotic
kinematics model of the machine can make the scattered information of
mechanical structure, which is extracted from the external entities (i.e.,
construction machines), into a physical model for computing. In this
model, the connections of the components are expressed by explicit
mathematical relationship among the variables. This clear mathematical
relationship is one of the basic information items needed to obtain the Fig. 9. Definition of standard D–H parameters.
full-body pose of construction machines. (Note: The red and blue colors denote all things associated with links j-1 and j
The kinematics model consists of two elements: links and joints. A respectively. The numbers in the circles represent the order in which the
link is a rigid rod with a certain kinematic function, which is the smallest elementary transforms are applied [38].)

Fig. 8. Examples of the connection relationship of pairs of adjacent components for construction machines.

7
J. Tang et al. Automation in Construction 138 (2022) 104217

θj is the angle between the xj and xj-1 axes about the zj-1 axis (joint estimation. Fig. 10 illustrates an example of the cabin rotation angle
angle). based on an excavator.

3.1.4. Trajectory generation 3.1.4.2. Joint angle. In this part, we design a trajectory generation al­
Trajectory generation is the most important process in the full-body gorithm for the movable components of construction machine which can
pose estimation of construction machines. In this section, the pose of only lift and drop. It can combine the input quaternion sequences with
each pair of adjacent components in construction machines is directly the connections between pairs of adjacent components, and transform
described by the generated trajectories of the relative angles between them into the trajectories of joint angles, which describe the pose of
components. According to the kinematics model constructed in Section these components directly. Fig. 11 shows the joint angles of the common
3.1.3, with regard to rotational joints of common construction machines, construction machines concerned with this algorithm. To improve the
the distance dj between links j-1 and j along the xj axis is a fixed value, computational efficiency and the capabilities of input data processing,
and the relative motion between the two links can be described by the the designed algorithm still uses quaternions to represent rotations.
change in θj. Hence, it is important to find θj, which records the relative However, when describing the result, it is represented by quaternions
motions between the adjacent movable components. In this process, the and has no clear physical meaning nor is it intuitive to understand.
quaternion sequences which describes the rotation states of the com­ Hence, at the end of the algorithm, the quaternion-based result is
ponents achieved in the process of orientation estimation is combined transformed into the trajectory of the joint angles represented by the
with the connection information between components achieved in the Euler angles, which gives a clear, easily understood physical meaning to
process of kinematics modeling. Then, they are transformed into the the pose of the pair of adjacent components, and output.
information which directly describes the relative motion of any pair of Eq. (5) shows the basic principle for generating the trajectory of a
adjacent components, called relative angles. According to the relative joint angle: Suppose A and B are adjacent movable components. First,
angle obtained at each moment, a trajectory curve of this angle with the quaternion representations of A and B's estimated orientations are
respect of time can be drawn to describe the pose of two adjacent normalized for preparation. The quaternion sequence of B is then
components. When the pose of each pair of adjacent components in inverted. Next, the relative pose sequence is solved using multiplication:
construction machines is directly described by their generated trajec­
tories, the full-body pose estimation is completed. ξAB = ξEB ⊙ ξAE (5)
To propose a general pose estimation method for most construction
where,
machines, the different motions of movable components for common
ξBA denotes the relative rotation between A and B,
machines are taken into consideration, and divided into three types,
ξBE denotes the orientation change of state in the earth frame relative
namely, lifting, dropping, and rotating. For most construction machines,
to its state in the frame of sensor B,
their components can only lift and drop, such as the tilt for the truck
ξEA denotes the orientation change of state in the frame of sensor A
bodies, lifting and dropping for the bulldozer buckets, and lifting and
relative to its state in the earth frame.
dropping for the excavator booms. The pose of these components can be
Afterwards, the relative pose expressed by quaternions is converted
described by a two-dimensional angle on the same vertical plane, which
into a Euler angle representation. The joint angle between adjacent
is called joint angle. In addition, some movable components can rotate
movable components is calculated by the Yaw of the converted results
left and right, such as the rotation of the excavator cabins and the
represented by the Euler angle. The algorithm of the joint angle calcu­
rotation of the tower cranes. The pose of these components can be
lation is shown in Fig. 12. When the joint angle of each time slot is
described by the rotation angle on the horizontal plane, which is called
computed, the joint angle trajectory describing the pose of the pair of
cabin rotating angle. Hence, in this study, we define these two kinds of
adjacent components can be recorded.
relative angles for construction machines: (1) cabin rotating angle, and
(2) joint angle. We also generate two types of trajectories that are in line
with the modes of motions that the machine is capable of: (1) trajectory 3.2. Optimal IMU installation scheme
of the relative angle describing cabin rotation, and (2) trajectories of the
relative movements between pairs of adjacent components. Section 3.1 shows the information flow of the pose estimation for
construction machines. When using this approach to obtain the pose
3.1.4.1. Cabin rotating angle. For the movable components that can information in motion-related analysis, the installation scheme of the
only rotate left and right, their rotation angles on the horizontal plan are sensors is the primary problem. Hence, we propose a reliable strategy of
the angles rotating around the z axis in the world frame. Hence, for an optimal IMU installation for motion-related analysis, based on the
IMU sensor installed on the cabin, the angle of rotation is equal to the approach given in Section 3.1. In particular, it is necessary to consider
estimated Yaw of the cabin in terms of the Euler angles, transformed installation locations and the number of IMU sensors when designing
from the quaternion sequence obtained from the process of orientation and optimizing the installation strategy. Essentially, such a scheme
considers two questions: (1) Where should the IMU sensors be installed?
(2) What is the minimum number of IMU sensors to fully estimate the
pose of a construction machine?
To provide a general installation strategy, the investigation is carried
out at the level of the individual component, which is the basic units of
the machine motion. The strategy is generalizable to different machines
composed of multiple components. The investigation begins by exam­
ining how installing sensors at different locations influences the accu­
racy of the pose estimation by showing and comparing data collected
from different locations that pertain to the same machine component.
Suitable locations to install the IMU sensors are then selected consid­
ering accuracy and installability. The second part of the investigation
discusses the number of IMU sensors to install qualitatively based on the
Fig. 10. Example of the cabin rotation angle in an excavator. information involved in a single IMU and the motion modes of movable
(Note: θ1 denotes the Yaw of the cabin component orientation in terms of components for construction machines. The method of this investigation
Euler angles.) consists of six steps, illustrated as Fig. 13: (1) data collection and

8
J. Tang et al. Automation in Construction 138 (2022) 104217

Fig. 11. Examples of the joint angles for construction machines.

3.2.1. Statistical analysis

Since IMUs are installed on the same component, in an ideal case,
these trajectories computed from different IMUs should respectively
estimate the same motions of the component. However, in the real case,
the estimated results may be different due to the influence of the IMU
installation position. Hence, in this step, a statistical significance test is
needed to identify the difference between the estimated results from the
IMUs installed at different positions, so as to explore whether “instal­
lation location” has a significant impact on the accuracy of pose esti­
mation quantitatively.
The selection of statistical significance testing needs to be based on
the characteristics of the test samples. In this step, the test samples are
multiple trajectories of the relative angle from different IMUs, which are
time series of the Euler angles. The characteristics of the test samples
Fig. 12. Algorithm of the quaternion-based joint angle calculation. include multiple groups of samples from IMUs installed at different lo­
cations, unknown distribution of samples, and the independence of
samples. According to the above characteristics, the Kruskal-Wallis test
was selected as the statistical significance test method in this investi­
gation. The Kruskal-Wallis test [39] is a nonparametric method, which is
usually used for significance analysis of three or more groups of
consecutive independent samples. The Kruskal-Wallis test does not as­
sume that the data are normally distributed, which is its big advantage,
so it is suitable to evaluate the difference of time series.

3.2.2. Result discussion

The statistical analysis step indicates the influence of the IMU
installation locations on the accuracy of the pose estimation of con­
struction machines. In this step, the best installation scheme in motion-
related works, including the places and number of IMU sensors to install,
Fig. 13. Method of the investigation on optimal IMU installation scheme. are determined qualitatively. The discussion begins with the suitable
places to install the IMUs on the surfaces of construction machines,
considering the accuracy and installability. Afterwards, this step con­
preprocessing, (2) orientation estimation, (3) kinematics modeling, (4)
siders the information involved in a single IMU and the motion modes of
trajectory generation, (5) statistical analysis, and (6) result discussion.
movable components for construction machines in order to find the
The first four steps of the method share the proposed approach of the
minimum number of IMU sensors used in motion-related works.
full-body pose estimation. To explore the influence of the IMU instal­
lation position on the accuracy of the pose estimation, multiple 9-axis
4. Experiment 1: the IMU-based full-body pose estimation of
IMUs are installed at different locations (e.g., near the junction, and
construction machines
near the center of gravity) on the same movable machine component.
When the component moves independently, IMUs installed at different
4.1. Experiment setup
locations collect data simultaneously to describe the motion of the
component. Since the adjacent component remains stationary, the in­
To verify the IMU-based approach, the selection of the experimental
dependent motion of the concerned component can be described by the
object is the primary problem. In this study, each of the movable com­
relative angle of the adjacent component pair to which the component
ponents is regarded as the basic unit of a construction machine. The
belongs. According to the proposed approach in Section 3.1, after these
design of the approach depends on three motion modes of components:
four steps, the trajectories of the relative angle describing the pose of a
lifting, dropping, and rotating. However, not all machines use these
pair of adjacent components are the outputs. The technical details from
three motion modes. For example, as mentioned in Section 3.1.4, only
(1) data collection and preprocessing to (4) trajectory generation have
components that can lift and drop are involved in tracks and bulldozers,
already been introduced above and are not repeated here. This sub-
whereas the tower crane only involves the components that can be
section briefly indicates how to find the best IMU installation scheme
rotated. Therefore, to verify the approach in each motion mode,
based on these trajectories generated by different IMUs.
experimental machine containing all these three kinds of motion should

9
J. Tang et al. Automation in Construction 138 (2022) 104217

be selected. The excavator is a commonly used construction machine between the cabin and the boom, θ3 is the joint angle between the arm
and essentially, has a more comprehensive range of motion than other and the boom, θ4 is the joint angle between the bucket and the arm.
common construction machines. The excavator has four degrees of According to the estimated motion of the independent components
freedom, and its modes of motion include the cabin's rotation and the and the kinematics model, the experimental results, illustrated as
relative movements between adjacent components. At the component Table 2, show the trajectories of relative angles which describe the pose
level, the excavator is representative. The experimental results of pose of each pair of adjacent components. The accuracies of the estimated
estimation using excavators can represent the general performance of full-body pose are evaluated by comparing each trajectory of the relative
this approach applied to common construction machines. angles and the given ground truth. Root mean square error (RMSE) is a
To fully prove the working performance of the proposed approach, common metric to measure the difference between values and to
an experiment was carried out on a real construction site using a real represent the sample standard deviation of the difference between an
machine. Fig. 14 shows the equipment used in the study. Data acquisi­ estimated and an observed value. In this study, this difference reflects
tion was done using LPMS-B2 IMU sensors equipped with a Bluetooth how accurate the full-body pose estimation is. Table 3 shows the RMSEs
transmitting and receiving module, with sampling frequency of 100 Hz. for each trial.
The sensors were mounted onto the excavator, of type ZAXIS 60 made by From the figures of the trajectories of the relative angles, four distinct
HITACHI and consequently, any information within a 20-m range could cycles were observed, corresponding to four times of independent mo­
be collected and stored on a PC terminal. In addition, to avoid over tions repeated in each trial. Intuitively, the amplitudes of the estimated
saturation of the magnetometer, a plastic box was used to isolate the poses are consistent with the normal ranges of component motions in
IMU from the metal surface to prevent the IMU contacting the mental excavator operations. Some slight errors are found on the estimated
directly. pose, especially at the peak. These errors are stably distributed in every
In order to validate the pose of the construction machine, in this cycle in the form of the absolute value, which does not change with the
experiment, a depth camera (RealSense D435i) is placed on one side of amplitude of motion. However, in this paper, we only evaluate the ac­
the excavator to collect data as ground truths. Based on the depth and curacy of the results, and the specific reasons for the errors are beyond
RGB image data collected by the depth camera, the 3D coordinates of the scope of our work. From the perspective of RMSEs, based on the
keypoints, labeled manually, are obtained in each frame. Using vector proposed approach, the differences between the estimated results and
calculation, these 3D coordinates can be transformed into angles the ground truths for independent motions of components are in the
describing the pose of excavator directly. range of 1.09–2.77 degrees, and the average RMSE value is 1.79 degrees.
There is no significant difference among the trials. Referring to related
works on IMU performance evaluation [40,41], generally, the average
4.2. Experiment design and result discussion
measurement error caused by intrinsic characteristics of the IMU sensors
is within the range of [0.63–10.51] degrees of RMSE. To intuitively
Shown as Fig. 15, four sensors (i.e., IMU1, IMU2, IMU3, and IMU4)
clarify the possible negative impact of errors on the poses of actual
were attached to four movable components (i.e., cabin, boom, arm, and
machines, we apply the maximum errors shown in this experiment to the
bucket) to record the rotational state of each movable component in
arm of the excavator. For example, assuming the length of arm is 1.6 m,
space.
based on mentioned errors as 2.77 degrees, the maximum errors that
First, the accuracy of the proposed approach in motion estimation of
may occur at the end of the arm is about ±0.033 m. However, whether it
each component was evaluated. In the first case, each movable
has significant negative impact to normal operations needs to be dis­
component of the excavator was operated independently. The inde­
cussed with the consideration of real scenarios on sites: Considering the
pendent operations included the left and right rotation of the cabin, the
uncontrolled outdoor environment, the operations of construction ma­
lifting and dropping for the boom, arm, and bucket, respectively. To
chines can accept a normal range of errors and would not be disturbed.
ensure the stability of the experimental results, each kind of independent
For example, based on the observations in normal operations, it is found
motion was repeated four times in each trial.
that the end of the long arm normally vibrates due to the loosening of the
The data collected from IMUs required preprocessing and orientation
joint, especially in the case of idling. These vibrations were observed in
estimation to estimate the independent motions of the components.
the on-site experiment and were also captured by IMUs, as shown in
Then, according to the structure information of excavators, a kinematics
Table 2, especially when machine stopped suddenly. Based on the
model (Fig. 16) of the excavator was constructed to describe the spatial
experimental results, for the excavator used in this study, the average
connection between the components of the specific machine and to
vibration amplitudes of the end of the arm are about ±0.05 m, and they
define the keypoints. The kinematics model for excavators is expressed
are even larger with the increase of the arm length. In practice, operators
in standard D–H parameters notation, and the initial values are shown
are not concerned of these unmodeled vibrations and they do not affect
in the Table 1. The initial pose is described by the four angles listed in
the normal excavations. The maximum errors that may occur at the end
Table 2, where θ1 is the cabin rotating angle, θ2 is the joint angle

Fig. 14. Excavator and data acquisition sensor used in the study.

10
J. Tang et al. Automation in Construction 138 (2022) 104217

Fig. 15. IMU installation locations in pose estimation.

The estimated trajectories of the relative angles which describe the full-
body pose of the excavator are shown as Fig. 17, where θ1 is the cabin
rotating angle, θ2 is the joint angle between the cabin and the boom, θ3 is
the joint angle between arm and boom, and θ4 is the joint angle between
bucket and arm. Table 4 lists the figures of the estimated trajectories of
the relative angles and its ground truth, respectively, and the specific
RMSEs.
From the figures of the trajectories of relative angles, two clear cycles
were observed, corresponding to two cycles of continuous motion
repeated in this case. In addition, Fig. 17 shows the cooperative rela­
tionship between the successive motions of components, based on the
order of occurrence between trajectories. Similar to the first case, some
slight errors appear at the peaks and are distributed stably between
different cycles. In the trajectory of the joint angle between bucket and
arm, it is observed that, in the stable stages, the IMUs recorded obvious
jitter. This is because only one endpoint of the end-effector (i.e., bucket)
is fixed at the joint. When the cabin rotates or the boom and arm move,
the bucket obviously swings due to inertia. IMU is highly sensitive to
Fig. 16. Kinematics model of the excavator based on standard
these swings, which can be truly reflected in the pose estimation results.
D–H parameters.
From the perspective of RMSEs, using the proposed approach, the dif­
ferences between the estimated results and the ground truths for the
Table 1 continuous motion of components are in the range of 0.91–2.58 degrees.
Initial values of standard D–H notations for the excavator. There is no significant difference into relative angles. Based on the given
j Theta (rad) d (m) a (m) α (rad) Offsets Joint types
results on each pair of adjacent components, the average RMSE value of
the estimated continuous motions is 1.65 degrees. Referring to related
1 0 0 1.9 1.57 0 Rotational
works on IMU performance evaluation [40,41], the average errors have
2 0.35 0 3.5 0 0 Rotational
3 − 1.34 0 1.6 0 0 Rotational an acceptable range of measurement error for IMUs. Comparing the
4 − 0.70 0 0.8 0 0 Rotational estimation results of the first case, as shown in Fig. 18, the RMSEs of
these two cases have similar trends in the pose description of each pair of
adjacent components, and their average values are also similar. There­
of components are closed to or even lower than the vibrations usually fore, it is proven that the proposed approach can provide accurate pose
produced in normal operations. Hence, although the errors exist, it of each pair of adjacent components for stably estimating the full-body
doesn't have a significant negative impact on normal operations of pose of the construction machines, whether describing the independent
construction machines. Therefore, whether from the visual inspection of motion for a single component or continuous motions for multiple
the figures or from the quantitative investigation, IMU-based pose esti­ components. The total average errors for the full-body pose estimation
mation results have acceptable accuracy, under the condition of inde­ are 1.71 degrees. In order to intuitively show the error of the proposed
pendent motion. approach, the ratio of the average errors to the average motion ampli­
To further verify the practicability of the proposed approach in the tude is defined as an index to evaluate the performance of the approach.
working state of construction machines, in the second case, the exca­ According to the above experimental results, the average error accounts
vator was operated for digging and dumping on a construction site. for 9.73% of the average motion amplitude. In other words, according to
Multiple components of the excavator moved continuously in each the experimental results, this IMU-based approach can estimate the full-
working cycle, ensuring that every component and its motion modes body pose of construction machines at an accuracy level of 90.27%.
were involved in the experiment. To ensure the stability of the experi­ Unfortunately, because the application of IMU sensors for estimation the
mental results, the full working cycle of digging and dumping was poses of construction machines is still in its infancy, previous studies
repeated two times in this case. Each cycle started with a given initial cannot provide a complete IMU-based method that can compare with
pose, and the initial relative angles were updated as shown in Table 4.

11
J. Tang et al. Automation in Construction 138 (2022) 104217

Table 2
Experimental results for the independent motion estimation.
Trials Relative angles Results (deg)

results and even improve the accuracy of pose estimation for construc­
Table 3
tion machines.
RMSEs of the estimated relative angles in independent motions.
Afterwards, we discuss the practical implications and the application
Trial Types of angle Relative angles RMSEs potency of the IMU-based full-body pose estimation. The approach of
(deg)
tracking the motions of construction machine without environmental
1 Cabin rotating Cabin rotating angle 1.94 restrictions, can be potentially widely used in the operation safety
angle
monitoring in various scenes. For example, on a narrow and crowded
2 Joint angles Joint angle between the cabin and the 1.09
boom municipal engineering site, we often need to judge whether the moving
3 Joint angle between the arm and the 2.77 components of construction machines may collide with pedestrians and
boom vehicles. Hence, to replace the safety inspector's eyes, the proposed
4 Joint angle between the bucket and 1.36 approach is used to automatically monitor the movements of the ma­
the arm
chines and identify the hazards. Specifically, based on the kinematics
model, the outputs of the approach can be easily transformed into
the proposed approach on the accuracy. However, among the studies of various pose-related descriptions of a construction machine, such as the
imaged-based pose estimation of construction machines, some reference coordinates of the predefined keypoints and the position of the end-
accuracies representing state-of-art in recent years are found to evaluate effector. Using these pose-related descriptions, flexible applications on
the results in this study. Soltani et al. [42] proposed visual-based method operation safety monitoring can be developed, for example, delimiting a
to estimate the pose angles of excavators in 3D space with the average dynamic safety range based on the coordinates of the predefined key­
angle errors as 3.6 degrees. Mostafa et al. [43] mentioned that the points or tracking the positions of the end-effector to avoid collision with
average accuracy level of image-based pose estimation for construction obstacles.
machines is 85% in recent works. In conclusion, comparing with state-
of-art in this field, the proposed IMU-based method can achieve good

12
J. Tang et al. Automation in Construction 138 (2022) 104217

Table 4
Initial angles and RMSEs of the estimated relative angles in continuous motions.
Relative angles Initial angles (deg) Results (deg) RMSEs (deg)

Cabin rotating angle 0 1.57

Joint angle between the cabin and the boom 27 0.91

Joint angle between arm and boom 72 2.58

Joint angle between bucket and arm 139 1.53

5. Experiment 2: investigation on the optimal IMU installation Experiment 2 used a hydraulic excavator (i.e., HITACHI ZAXIS 60), a
scheme bulldozer made by HAIHONG, and IMU sensors (i.e., LPMS-B2 IMU
sensor). In order to take environment impact and feasibility of the
5.1. Experiment setup installation scheme into account, the experiment was carried out on a
real construction site.
The experiment aims to explore the optimal IMU installation scheme
for construction machines in motion analysis, and to further identify the 5.2. Experiment design and result discussion
practicability of the proposed approach. To quantitatively investigate
the suitable IMU installation location on a component, the experimental As illustrated in Fig. 19, four IMUs (i.e., IMU a, IMU b, IMU c, and
object was a single movable component of a construction machine. In IMU d) were installed at four different locations on the same boom of the
this experiment, although movable components of an excavator and a excavator to record the same motions of the component simultaneously.
bulldozer were selected as the experimental objects, there is no reason to The boom was operated three times in cycles of independent lifting and
limit the results to any specific machine. In other words, the experi­ lowering.
mental results can represent the general performance of IMU installed Using the method given in Section 3.2, the trajectories describing the
on any construction machine with such independent movable boom motions were obtained through IMUs installed at different loca­
components. tions. Each trajectory was treated as a separate group. Based on four

13
J. Tang et al. Automation in Construction 138 (2022) 104217

Table 5
Data and results of the statistical significance test for the excavator boom.
ID Locations Sample sizes Result

IMU a Near to the joint 1100 P = 0.1734 > 0.05

IMU b Near to the center of gravity 1100
IMU c Random position 1100
IMU d Near to the joint 1100

which is the statement we want to test and its opposite. In the Kruskal-
Wallis test, H0 is “the trajectories which are estimated by different IMUs
come from populations with the same distribution.”; H1 is “the trajec­
tories which are estimated by different IMUs come from populations
with different distributions.”. Afterwards, a significant level (P value) is
chosen to interpret the results. In order to balance the cost of any false
positive and false negative in the result of the significance test,
throughout this experiment, P < 0.05 was used as the significant level,
which is conventional in statistical analyses. This means that if the P
value obtained from the experiment is less than 0.05, H0 can be rejected;
if the P value is greater than or equal to 0.05, H0 cannot be rejected. In
Fig. 17. Trajectories of the changes of relative angles for describing the pose of
addition, the sample size needs to be emphasized. The Kruskal-Wallis
the excavator. test is distribution-free, i.e., it does not assume any particular data dis­
tribution, making it applicable to analyzing an arbitrary dataset, and has
no requirement of the minimum sample size for conducting this test.
Conventionally, samples less than five in each group are the general
accepted definition of “too small.” [39]. Therefore, 1100 or more sam­
pling points are quite a large sample size for the Kruskal-Wallis test.
According to the experiment result, the P value obtained from the
four groups of estimated trajectories is 0.1734, which is greater than the
conventional value 0.05. It is proven that although we tried four
different locations on thousands of sampling points, none of them really
changed the result of pose estimation. In other words, the factor of
“installation location” has no significant influence on the accuracy of
pose estimation, at least for the tested component.
In order to demonstrate whether this conclusion can be applied to
other components, repeated experiments were carried out on the arm of
the bulldozer. Similar to the boom of the excavator, three IMUs (i.e.,
IMU e, IMU f, and IMU g) were installed on the arm of the bulldozer, as
Fig. 18. Comparison on the RMSEs of estimated results from case 1 and case 2.
shown as Fig. 20, and we recorded the motions of lifting and lowering
simultaneously. Table 6 lists the result of the Kruskal-Wallis test and

Fig. 19. Installation positions of IMUs on the excavator boom.

groups of data, the Kruskal-Wallis test was used to verify whether the
factor of “installation location” has a significant impact on the accuracy
of motion estimation of the components. Table 5 lists the result of sta­
tistical significance test and specific details of the test data for the
excavator boom.
In analyzing the results of hypothesis testing, firstly we need to
determine the null hypothesis (H0) and the alternative hypothesis (H1),
Fig. 20. Installation positions of IMUs on the bulldozer arm.

14
J. Tang et al. Automation in Construction 138 (2022) 104217

Table 6
Data and results of the statistical significance test for the bulldozer arm.
ID Locations Sample sizes Results

IMU e Near to the joint 1500 P = 0.0757 > 0.05

IMU f Near to the center of gravity 1500
IMU g Random position 1500

specific details of the test data for the bulldozer arm.

According to the result, the P value obtained from the three groups of
estimated trajectories is 0.0757, which is greater than the conventional
value of 0.05. It means that the conclusion about the installation loca­
tions can be verified on random components. In other words, installing
IMU sensors in different locations has no significant impact on the ac­
curacy of the pose estimation for construction machines. Essentially, the
conclusion can be supported on a theoretical basis: as a rigid body, the
rotation is the same everywhere. Specifically, whether an IMU sensor is
installed near a joint, the center of gravity or any other locations on the
same component, it should be able to sense the same motion. This fact
never changes with the difference between components and machines.
In addition to the accuracy of pose estimation, installability is Fig. 21. Optimal IMU installation locations and number for the full-body pose
another important consideration when determining the best places to estimation of the excavator.
install IMU sensors. A location is deemed “installable” if mounting a
sensor there does not affect nor is affected by the operation of a machine. and can be reduced according to the structure and movement modes of
Actually, for construction machines, unsuitable IMU installation loca­ the construction machines. Based on above discussion, Fig. 22 illustrates
tions on the components often leads to sensors falling off, with data loss. the IMU installation locations and number for the full-body pose esti­
For example, for the bucket of an excavator or bulldozer, installing a mation of different construction machines. A truck has one movable
sensor anywhere away from its junction subjects the sensor to collisions components (i.e., body) as its cabin cannot rotate. Hence, only one IMU
with other objects during operation, causing it to fall off. Usually, to is mounted on its body to record the tilt relative to the ground. For a
ensure normal movement of the component pairs, their junction should bulldozer, it has two movable components (i.e., arm and blade) as its
be far away from the working part where the collision may occur. cabin cannot rotate. Two IMUs are mounted on its arm and blade,
Therefore, a place near the junction of component pairs is more reliable respectively, to collect the information required for estimation of the
and suitable for mounting a sensor. trajectories of the relative angles. Specially, to avoid the IMU falling off
Two important factors require attention when deciding on the due to collision, the sensor of the blade is installed on the back of the
number of sensors to install: (1) the information involved in a single blade and near the junction.
IMU; (2) the movement modes of the particular machine. According to
the proposed approach on IMU-based pose estimation, each IMU 6. Conclusions
installed on the component contains all the information describing the
motion of the component. It means that only one IMU needs to be This study defines the information flow of IMU-based pose estima­
mounted on each component to describe the motion of the component. tion for construction machines, which responds to the challenges, such
The pose estimation of a pair of adjacent components requires an IMU to as the vibration of the machine, interference due to external environ­
be mounted in both adjacent components. Hence, when installing an ment and instability of long-time works, when applying IMUs to con­
IMU on each component of the construction machine, the full-body pose struction machines. In the proposed information flow, a quaternion-
can be estimated. In other words, the number of IMUs using in the full- based trajectory generation method has been developed to describe
body pose estimation of construction machines should be equal to the the changes in relative angles of the components of a construction ma­
number of its movable components. Afterwards, considering the chine. The practicability and performance of the proposed approach has
movement modes of common construction machines, the conclusion can been verified through experiments on construction sites, the results of
be further optimized. Specifically, the minimum number of sensors is which show that the average RMSE between the estimated results and
found in the optimal scheme to achieve relatively complete full-body ground truth is 1.71 degrees, that is, the average accuracy is 90.27%.
pose estimation. The motion modes of the components of construction This approach efficiently and stably provides 3D pose information for
machines are limited, including the rotation of the cabin and the lifting construction machines without environmental constraints, which
and lowering of the arm or bucket. All rotating angles on the horizontal potentially facilitates monitoring of unsafe operations of heavy
plane come from the left and right rotation of the cabin and are recorded machines.
by IMU mounted on each component. Hence, the motion of the cabin can Based on the developed approach, we rigorously investigated a
be estimated using the horizontal rotation provided from sensors scheme for the optimal number and locations for installing IMU sensors
installed on other components. The IMU installed in the cabin can be to estimate the full-body pose of construction machines. Using statistical
removed in the optimal scheme. For example, the excavator has four significance testing, installing IMU sensors at different locations has no
movable components (i.e., cabin, boom, arm, and bucket). Based on the significant impact on the accuracy of the pose estimation for construc­
optimization of the IMU installation locations and number, as the IMU tion machines. Considering installability, we propose installing sensors
installed on the cabin can be removed, at least three IMU sensors should near a junction is more reliable and suitable. On the other hand, the
be installed to record the full-body pose of an excavator, as shown in number of IMU sensors to install should equal the number of movable
Fig. 21. components, which can be optimized according to the structure and
In conclusion, considering the accuracy of pose estimation and modes of motion of a specific construction machine.
installability of IMU sensors, it is recommended that the sensors be The limitation of this proposed method is that only relying on IMU
installed near junctions between adjacent movable components. The sensors makes the framework sensitive to unmodeled noise and devia­
number of installations is equal to the number of movable components, tion from the reference, which caused by mechanical offsets. For

15
J. Tang et al. Automation in Construction 138 (2022) 104217

Fig. 22. IMU installation locations and number for the full-body pose estimation of the truck and the bulldozer. The red points denote the recommended IMU
installation position. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

example, when the temperature of the IMU rises during execution, the [5] S. Xu, J. Wang, W. Shou, T. Ngo, A.-M. Sadick, X. Wang, Computer vision
techniques in construction: a critical review, Arch. Comput. Methods Eng. (2020)
performance of IMU inevitably decrease, including data loss and un­
1–15, https://doi.org/10.1007/s11831-020-09504-3.
controllable measurement errors. Due to the lack of feedback adjustment [6] H. Vogt, Efficient object identification with passive RFID tags, in: International
mechanism, these errors may accumulate in results. In the future, an Conference on Pervasive Computing, Springer, 2002, pp. 98–113, https://doi.org/
error feedback mechanism or a sensor fusion method can be introduced 10.1007/3–540-45866-2_9.
[7] M. Abderrahim, E. Garcia, R. Diez, C. Balaguer, A mechatronics security system for
to further improve the robustness based on the proposed method. In the construction site, Autom. Constr. 14 (4) (2005) 460–466, https://doi.org/
addition, though the installability and other factors have been consid­ 10.1016/j.autcon.2004.09.007.
ered qualitatively in the optimal IMU installation scheme, further [8] H. Cai, W. Rasdorf, Modeling road centerlines and predicting lengths in 3-D using
LIDAR point cloud and planimetric road centerline data, Comput. Aid. Civ.
research in this field would be of great help in exploring other quanti­ Infrastruct. Eng. 23 (3) (2008) 157–173, https://doi.org/10.1111/j.1467-
tative factors affecting the sensor installation strategy using optimiza­ 8667.2008.00518.x.
tion methods. In the future, for example, considering the relative [9] H. Luo, M. Wang, P.K.-Y. Wong, J.C. Cheng, Full body pose estimation of
construction equipment using computer vision and deep learning techniques,
locations between the data receivers and the IMUs, and the environment Autom. Constr. 110 (2020), 103016, https://doi.org/10.1016/j.
of construction sites, the objective functions can be built to minimize autcon.2019.103016.
information loss and find the optimal ranges of IMU installation. Com­ [10] M.M. Soltani, Z. Zhu, A. Hammad, Skeleton estimation of excavator by detecting its
parts, Autom. Constr. 82 (2017) 1–15, https://doi.org/10.1016/j.
bined with the optimal installation ranges and the proposed IMU autcon.2017.06.023.
installation scheme, this installation strategy can be further discussed in [11] C. Zhang, A. Hammad, S. Rodriguez, Crane pose estimation using UWB real-time
different scenarios. location system, J. Comput. Civ. Eng. 26 (5) (2012) 625–637, https://doi.org/
10.1061/(ASCE)CP.1943-5487.0000172.
[12] T.G. Phillips, P.R. McAree, An evidence-based approach to object pose estimation
from LiDAR measurements in challenging environments, J. Field Robot. 35 (6)
Declaration of Competing Interest (2018) 921–936, https://doi.org/10.1002/rob.21788.
[13] O.J. Woodman, An Introduction to Inertial Navigation, University of Cambridge,
The author(s) declare(s) that they have no conflict interests. Computer Laboratory, 2007, https://doi.org/10.48456/tr-696.
[14] S. Lee, M.-S. Kang, D.-S. Shin, C.-S. Han, Estimation with applications to dynamic
status of an excavator without renovation, in: Proceedings of the International
References Symposium on Automation and Robotics in Construction (ISARC), 2012, https://
doi.org/10.22260/ISARC2012/0093.
[15] Z. Péntek, T. Hiller, T. Liewald, B. Kuhlmann, A. Czmerk, IMU-based mounting
[1] Labour Department, Occupational Safety and Health Statistics. https://data.gov.
parameter estimation on construction vehicles, in: Proceedings of the 2017 DGON
hk/sc-data/dataset/hk-ld-aaidaao-aaid-oshstats, 2018.
Inertial Sensors and Systems (ISS), 2017, pp. 1–14, https://doi.org/10.1109/
[2] U.S. Bureau of Labor Statistics, Number and Rate of Fatal Work Injuries, by
InertialSensors.2017.8171504.
Industry Sector. https://www.bls.gov/charts/census-of-fatal-occupational-injurie
[16] A.H. Kashani, W.S. Owen, N. Himmelman, P.D. Lawrence, R.A. Hall, Laser scanner-
s/number-and-rate-of-fatal-work-injuries-by-industry.htm, 2018 (accessed: June
based end-effector tracking and joint variable extraction for heavy machinery, Int.
10 2020).
J. Robot. Res. 29 (10) (2010) 1338–1352, https://doi.org/10.1177/
[3] Legislative Council Panel on Manpower, Hong Kong's Occupational Safety
0278364909359316.
Performance in 2018 and the First Half of 2019. https://www.legco.gov.hk/yr19
[17] E. Olson, AprilTag: a robust and flexible visual fiducial system, in: Proceedings of
-20/english/panels/mp/papers/mp20191217cb2-371-3-e.pdf, 2019.
the 2011 IEEE International Conference on Robotics and Automation (ICRA), IEEE,
[4] Occupational Safety & Health Council, Survey on Occupational Safety and Health
2011, pp. 3400–3407, https://doi.org/10.1109/ICRA.2011.5979561.
of Mobile Crane Operators. http://www.oshc.org.hk/oshc_data/files/OSHInformat
ion/Research/OSH_of_Mobile_Crane_Operators_2017.pdf, 2017.

16
J. Tang et al. Automation in Construction 138 (2022) 104217

[18] E.R. Azar, C. Feng, V.R. Kamat, Feasibility of in-plane articulation monitoring of [31] D. Tedaldi, A. Pretto, E. Menegatti, A robust and easy to implement method for
excavator arm using planar marker tracking, J. Inf. Technol. Constr. 20 (15) (2015) IMU calibration without external equipments, in: Proceedings of the International
213–229. https://www.itcon.org/2015/15 (accessed: June 13 2020). Conference on Robotics and Automation (ISARC), 2013, https://doi.org/10.1109/
[19] K.M. Lundeen, S. Dong, N. Fredricks, M. Akula, J. Seo, V.R. Kamat, Optical marker- ICRA.2014.6907297.
based end effector pose estimation for articulated excavators, Autom. Constr. 65 [32] S.W. Smith, The Scientist & Engineer’s Guide to Digital Signal Processing,
(2016) 51–64, https://doi.org/10.1016/j.autcon.2016.02.003. California Technical Publishing, 1997, https://doi.org/10.5555/281875.
[20] I.J. Mulligan, A.K. Mackworth, P.D. Lawrence, A Model-Based Vision System for [33] M. Ester, H.-P. Kriegel, J. Sander, X. Xu, A density-based algorithm for discovering
Manipulator Position Sensing, University of British Columbia, Department of clusters in large spatial databases with noise, Proc. Knowl. Discov. Data Min. 96
Computer Science, 1989, https://doi.org/10.1109/TDSCEN.1989.68118. (34) (1996) 226–231, https://doi.org/10.5555/3001460.3001507.
[21] J. Yang, P. Vela, J. Teizer, Z. Shi, Vision-based tower crane tracking for [34] D. Turner, I. Anderson, J. Mason, M. Cox, An Algorithm for Fitting an Ellipsoid to
understanding construction activity, J. Comput. Civ. Eng. 28 (1) (2014) 103–112, Data, National Physical Laboratory, United Kingdom, 1999. http://citeseerx.ist.
https://doi.org/10.1061/(ASCE)CP.1943-5487.0000242. psu.edu/viewdoc/summary?doi=10.1.1.36.2773.
[22] T. Phillips, Determining and Verifying Object Pose from LiDAR Measurements to [35] W.R. Hamilton, Elements of quaternions, Nature (60) (1899) 387, https://doi.org/
Support the Perception Needs of an Autonomous Excavator, School of Mechanical 10.1038/060387a0.
and Mining Engineering, The University of Queensland, 2016, https://doi.org/ [36] S.O. Madgwick, A.J. Harrison, R. Vaidyanathan, Estimation of IMU and MARG
10.14264/uql.2016.787. orientation using a gradient descent algorithm, in: Proceedings of the IEEE
[23] Leica, Leica iCON iXE3 – 3D System. https://leica-geosystems.com/products/mach International Conference on Rehabilitation Robotics, 2011, pp. 1–7, https://doi.
ine-control-systems/excavator, 2019 (accessed: June 10 2020). org/10.1109/ICORR.2011.5975346.
[24] Trimble-Inc, Transforming the Way the World Works. https://www.trimble.com, [37] J.J. Craig, P. Hsu, S.S. Sastry, Adaptive control of mechanical manipulators, Int. J.
2018 (accessed: June 13 2020). Robot. Res. 6 (2) (1987) 16–28, https://doi.org/10.1177/027836498700600202.
[25] D.-I. Sun, S.H. Kim, Y.S. Lee, S.K. Lee, C.S. Han, Pose and position estimation of [38] P. Corke, Robotics, Vision and Control: Fundamental Algorithms in MATLAB®
dozer blade in 3-dimensional by integration of IMU with two RTK GPSS, in: Second, Completely Revised, Springer, 2017, https://doi.org/10.1007/978-3-319-
Proceedings of the International Symposium on Automation and Robotics in 54413-7.
Construction (ISARC), 2017, https://doi.org/10.22260/ISARC2017/0137. [39] J.H. Mcdonald, Kruskal–Wallis Test - Handbook of Biological Statistics.
[26] S.-H. Kim, et al., Development of bulldozer sensor system for estimating the http://www.biostathandbook.com/kruskalwallis.html, 2014.
position of blade cutting edge, Autom. Constr. 106 (2019), 102890, https://doi. [40] S. Sessa, M. Zecca, Z. Lin, L. Bartolomeo, H. Ishii, A. Takanishi, A methodology for
org/10.1016/j.autcon.2019.102890. the performance evaluation of inertial measurement units, J. Intell. Robot. Syst. 71
[27] I.H. Lopez-Nava, A. Muñoz-Meléndez, Wearable inertial sensors for human motion (2) (2013) 143–157, https://doi.org/10.1007/s10846-012-9772-8.
analysis: a review, IEEE Sensors J. 16 (22) (2016) 7821–7834, https://doi.org/ [41] Q. Mourcou, A. Fleury, C. Franco, F. Klopcic, N.J.S. Vuillerme, Performance
10.1109/JSEN.2016.2609392. evaluation of smartphone inertial sensors measurement for range of motion,
[28] P. Picerno, 25 years of lower limb joint kinematics by using inertial and magnetic Sensors 15 (9) (2015) 23168–23187, https://doi.org/10.3390/s150923168.
sensors: a review of methodological approaches, Gait Posture 51 (2017) 239–246, [42] M.M. Soltani, Z. Zhu, A. Hammad, Framework for location data fusion and pose
https://doi.org/10.1016/j.gaitpost.2016.11.008. estimation of excavators using stereo vision, J. Comput. Civ. Eng. 32 (6) (2018)
[29] S. Talmaki, V.R. Kamat, Real-time hybrid virtuality for prevention of excavation 04018045, https://doi.org/10.1061/(ASCE)CP.1943-5487.0000783.
related utility strikes, J. Comput. Civ. Eng. 28 (3) (2014) 04014001, https://doi. [43] K. Mostafa, T. Hegazy, Review of image-based analysis and applications in
org/10.1061/(ASCE)CP.1943-5487.0000269. construction, Autom. Constr. 122 (2021), 103516, https://doi.org/10.1016/j.
[30] F.A. Bender, S. Göltz, T. Bräunl, O. Sawodny, Modeling and offset-free model autcon.2020.103516.
predictive control of a hydraulic mini excavator, IEEE Trans. Autom. Sci. Eng. 14
(4) (2017) 1682–1694, https://doi.org/10.1109/TASE.2017.2700407.

17