Proceedings of the ASME 2023

International Mechanical Engineering Congress and Exposition

IMECE2023
October 29 - November 2, 2023, New Orleans, Louisiana

IMECE2023-112791

CONCEPTUAL ARCHITECTURE OF DIGITAL TWIN WITH HUMAN-IN-THE-LOOP -BASED

SMART MANUFACTURING

Duck Bong Kim Mahdi Sadeqi Bajestani

Department of Manufacturing and Engineering Department of Mechanical Engineering
Technology, Tennessee Technological University Tennessee Technological University
Cookeville, TN 38505, USA Cookeville, TN 38505, USA

Guodong Shao, Albert Jones Sang Do Noh

National Institute of Standards and Technology Department of Systems Management Engineering
100 Bureau Drive, MS 8260 Sungkyunkwan University, 2066, Seobu-ro,
Gaithersburg, MD 20899, USA Jangan-gu, Suwon-si, Gyeonggi-do, 16419, Korea

ABSTRACT 1. INTRODUCTION
This paper proposes a conceptual architecture of digital Manufacturing is the backbone of economic development in
twin with human-in-the-loop-based smart manufacturing (DH- the U.S. [1]. The COVID-19 pandemic crisis disrupted the
SM). Our proposed architecture integrates cyber-physical manufacturing industry in many countries, resulting in (1) major
systems with human spaces, where artificial intelligence and upheavals in their production networks, (2) substantial
human cognition are employed jointly to make informed reductions in new product demands, and (3) negative impacts in
decisions. This will enable real-time, collaborative decision- both their local and global supply chains [2,3]. Moreover, new
making between humans, software, and machines. For example, lessons for manufacturing to successfully tackle these pandemic
when evaluating a new product design, information about the impacts are still being learned [4]. One of those new lessons is
product’s physical features, manufacturing requirements, and the “non-contact manufacturing” paradigm [5], which is based
customer demands must be processed concurrently. Moreover, on remotely controlling a system, a process, and a part with
the DH-SM architecture enables the creation of an immersive minimal physical interactions.
environment that allows customers to be effectively involved in Before the pandemic, several strategic plans such as “Smart
the manufacturing process. The DH-SM architecture is well Manufacturing (SM)” in the USA and “Industry 4.0” in Germany
fitted to those relatively new manufacturing processes, such as [6-8] have helped advance manufacturing industries. SM brings
metal additive manufacturing, since they can benefit from using smart technologies such as smart sensors, high-performance
digital twins, data analytics, and artificial intelligence for computing, industrial internet of things (IIoT), artificial
monitoring and controlling those processes to support non- intelligence (AI), and data analytics to traditional production
contact manufacturing. The proposed DH-SM will enable processes and manufacturing systems. However, various
manufacturers to leverage the existing cyber-physical system manufacturing operations still remain manual, where humans
and extended reality technologies to generate immersive can perform them better than machines. To enable human
experiences for end users, operators, managers, and operators to better use SM technologies and also support the non-
stakeholders. A use case of wire + arc additive manufacturing is contact manufacturing concept, a new concept of Digital Twin
discussed to demonstrate the applicability of the proposed with Human-in-the-Loop-based Smart Manufacturing (DH-SM)
architecture. Relevant development and implementation is introduced.
challenges are also discussed. Metaverse enables the integration of a “virtual world” with
the “physical world” [9]. The resulting integration is based on an
Keywords: Digital Twin, Smart Manufacturing, Human-in- extended reality (XR) that combines augmented reality (AR),
the-loop, Collaborative Decision Making, Non-contact virtual reality (VR), and mixed reality (MR) technologies.
Manufacturing, Industrial Metaverse. Currently, the “virtual world” can be the “Digital Twin (DT)”,

1 © 2023 by ASME
which may be used for various purposes such as analyzing health formerly human-made decisions. Concurrent manufacturing
conditions of equipment for predictive maintenance, managing involves a systematic approach to simultaneously design the
the whole lifecycle of a physical asset, and improving decision- product and develop its manufacturing process. That approach,
making through engineering and numerical analysis [10]. In based on a Japanese idea called Kansei engineering, included
addition to the functionalities provided by DTs [11-14], the feelings, impressions, and emotions from stakeholders in
metaverse will add auditory, visual, and haptic realism to achieve concrete design parameters [15].
the embodiment. It will change the way humans interact with the SM includes transformative technologies for managing the
future virtual, physical, and pandemic worlds. Also, the interconnections among physical assets, their DTs, and related
metaverse has a great potential to positively change the computational capabilities. A CPS generally consists of a
manufacturing landscape by taking advantage of (1) immersive collection of DTs of physical assets, including (1) material
experiences, (2) freedom from a physical distance called inputs, (2) manufacturing processes, and (3) final products [16-
“telepresence,” and (3) interconnection. 18]. The idea is to embed data gained from advanced sensor
In this paper, we propose an architecture that supports DH- technologies into DTs of all three to improve process control and
SM. This architecture is intended to enable integrability, part quality. For example, in additive manufacturing (AM), DTs
interoperability, interactivity, and immersivity. The architecture can be used to model the variabilities that impact process
includes three modules: Cyber-Physical System (CPS), Avatar- repeatability, part reproducibility, and quality assurance [19].
User System (AUS), and Collaborative Decision-Making Engine These DTs can be comprised of “surrogate models” such as
(CDME). Collectively, these modules can support future human- physics-based, data-driven, and physics-informed, data-driven
in-the-loop research, demonstrations, and case studies in SM. models [12,14].
The proposed architecture can also support non-contact In this paper, we view the conceptual DH-SM architecture
manufacturing by allowing users to remotely access a shop floor as a framework that can help enhance SM to achieve convergent,
through immersive simulation to realize real-time monitoring collaborative, and non-contact manufacturing. It consists of
and control. multidisciplinary domains, such as advanced manufacturing
The remainder of this paper is organized as follows: Section capabilities, digital technologies, and cognitive engineering.
2 provides some background information about various This enables highly optimized processes/supply networks,
manufacturing paradigms. Section 3 introduces the conceptual customized products, and resilience for unexpected and
architecture of digital twin with human-in-the-loop -based smart manufacturing-unfriendly situations. The DH-SM concept and
manufacturing. Section 4 discusses a use case in wire + arc its information flow are depicted in Figure 2. This paper focuses
additive manufacturing (WAAM) to demonstrate the on AUS and CDME since the concept of CPS is well-established.
applicability of the proposed architecture. Section 5 discusses the The AUS supports better decision-making by creating an
relevant development and implementation challenges, and immersive and interactive user experience across the entire
Section 6 concludes the paper and discusses the future work. product life cycle. It comprises an enhanced user (EU) and
human digital twin (HDT). EU generates data, information, and
2. AUGMENTING SM WITH DH TECHNOLOGIES knowledge and provides them to the other modules for analysis,
decision-making, and control. The HDT, the digital replica of the
Manufacturing paradigms have been evolving for decades. EU, can analyze, accumulate, and synthesize the data and
Figure 1 shows the evolution, which includes traditional, knowledge acquired from the DT and EU modules to support
intelligent, concurrent, and smart manufacturing. In traditional collaborative decision-making. Each module is explained in
manufacturing, human workers use their senses to monitor, detail in Section 3.
operate, and update the process and inspect the final workpiece.
To reduce the roles of humans and speed up fabrication and
inspection, intelligent manufacturing was introduced to automate

2 © 2023 by ASME
FIGURE 1: MANUFACTURING FROM TRADITIONAL TO SMART MANUFACTURING AND THE PROPOSED PARADIGM OF DH-SM

example, given a customer’s product, its surface-appearance of

a product can be evaluated virtually using a Likert scale [21].
Immersive environments provide real-time interactive
immersion to the user(s). They can interact with the immersive
contents of a product via metaverse technologies (e.g., XR:
VR/AR/MR). The realistic interaction among those inputs can be
achieved via multi-modal senses in Sensory Realism. Based on
their results, users can generate their desired experiences, which
can be inputs to HDT for further analysis.
HDT consists of four components: Data Preparation,
Information Model, Immersive Analysis, and CraftsAvatar. In
the Data Preparation component, both quantifiable and
unquantifiable data from the EU can be stored and preprocessed.
The two prepared data types will be formalized and transferred
to other modules in the Information Model. Immersive Analysis
can analyze the different information models, and CraftsAvatar
is a digital replica of the enhanced user(s) that can perform
simulations, acquire data, perform data analysis, and semi-
independently make decisions for better performance.
Ultimately, CraftsAvatar can continuously evolve into a “Virtual
Master” in a domain-specific area. CDME includes Verification,
Validation, Uncertainty Quantification (VVUQ), and Multi-
FIGURE 2: THE CONCEPT AND INFORMATION FLOW OF
DIGITAL TWIN WITH HUMAN-IN-THE-LOOP -BASED SMART Criteria Decision Making (MCDM). In VVUQ, analytical
MANUFACTURING. models estimate the uncertainties of the process and the parts.
Those uncertainties can be due to the lack of knowledge
3. A CONCEPTUAL ARCHITECTURE OF DIGITAL (epistemic) or intrinsic randomness (aleatoric) [22]. MCDM
TWIN WITH HUMAN-IN-THE-LOOP- BASED includes scaling, normalization, weighting, and aggregation
SMART MANUFACTURING components for the final decision-making [23]. The following
Figure 3 shows the detailed modules and the information subsections will explain the information flows in DH-SM and its
flow in the proposed DH-SM architecture based on the previous issues.
studies [6,20]. Physical Entities comprise Process Parameters,
Experimental Configuration, Observable Manufacturing 3.1 Cyber-Physical System
Elements (OME), and Data Acquisition Devices. DTs include Cyber-Physical System includes two modules, Digital Twin
Data Preparation, Information Models, Modeling and and Physical Entities. The process parameters are first decided
Simulation, and Digital Twin Models. EU consists of User in the Physical Entities, and then the experimental configurations
Experience Plans, Immersive Environments, User(s), and are set, considering the available OMEs. Data acquisition
Sensory Realism. The user experience plans are based on the devices are then employed to obtain signatures to establish a
actual manufacturing requirements and case scenarios. For relationship among the process, structure, property, and

3 © 2023 by ASME
performance (PSPP) [24]. The signatures are divided into

FIGURE 3: THE MODULES AND INFORMATION FLOW OF DIGITAL TWIN WITH HUMAN-IN-THE-LOOP -BASED SMART
MANUFACTURING

process signatures and part signatures. The process signatures monitoring, and control should be simultaneously addressed in a
can be 1D, such as current and voltage; 2D, such as the data unified system. This governs that the system responsible for
extracted from a charge- coupled device (CCD) camera, high controlling the AM system must also communicate with the
dynamic range (HDR) camera, and high-speed camera; and 3D, software to run online diagnostics during manufacturing.
such as the data from the profilometer. 1D process signatures Through this approach, any detected failures could be corrected
are much easier to handle and store, while 2D and 3D or compensated by modifying or sending additional commands
signatures contain more information and can be used in to the system. The framework is constructed to allow quick
different data structures. adaptation to new manufacturing conditions and the
On the other hand, the acquired part signatures in the incorporation of multiple diagnostic tools.
architecture can be classified into three types. The first type is The other module in CPS is the Digital Twin, which aims to
the signatures based on the microstructure of the part, obtained create a digital replica of the physical entities and phenomena
through material characterization techniques such as optical through different modeling and simulation approaches, including
microscope image, scanning electron microscope (SEM), and but not limited to physics-based, data-driven, physics-informed
electron backscatter diffraction (EBSD). These signatures can data-driven, and surrogate models. Surrogate models are simpler
shed light on the anisotropic and heterogeneous behavior of AM versions that mimic the mechanisms of complex models. Their
parts. The second type is mechanical properties extracted by purpose of surrogates is to reduce the computation time. In the
tests such as tensile that can be used to obtain the stress-strain proposed architecture, surrogate models can be generated for
curve for the materials. Finally, the third type is signatures processes and parts. The design of experiment (DOE) is widely
based on the part geometry obtained using a coordinate- employed to create surrogate models due to its effectiveness and
measuring machine (CMM), which measures geometrical efficiency.
accuracy and surface roughness. The part signatures can be Nevertheless, (1) DOE cannot be used for real-time
used to validate the physics-based and data-driven models. monitoring and control, and (2) a considerable number of
The data acquisition devices can be (1) internal or built-in experiments have to be carried out, which requires significant
sensors such as welding power measurement and the position resources. To overcome the two limitations, machine learning
tracking systems for the robot and (2) external sensors such as (ML) surrogate models based on process signatures for real-time
pyrometers, HDR cameras, high-speed cameras, thermocouples,
and CCD cameras. In the proposed architecture, the process
4 © 2023 by ASME
monitoring and control have gained increasing attention due to required. To address these issues, new open-source software tools
low cost, less time consumption, and high accuracy. ML, and interface standards will be needed. Second, the
specifically the semi-supervised learning algorithm and implementations will require huge computational costs. For
generative adversarial network (GAN), has been extensively example, photorealistic visualization of an object by rendering
employed for anomaly detection as in AnoGAN [25] and MAD- tools is the result of complex interactions among light (e.g.,
GAN [26]. In addition, ensemble learning (EL) has been used spectrum), 3D models (e.g., texture), and viewing (e.g.,
to improve classification, prediction, and/or function direction) conditions. For this, an affordable high-end graphics
approximation [27]. EL systematically generates and combines process unit (GPU) should be available. Third, new types of user-
several models, such as classifiers or experts, to solve a friendly interfaces are required. For example, head-mounted
computer intelligence problem. Despite their advantages, ML- displays are reported to cause discomfort, pain, or visual fatigue
based surrogate models demand massive data, thus hindering after use [28].
their widespread application.
3.3 Collaborative Decision-Making Engine
3.2 Avatar-User System The resulting data will enable “collaborative decision-
The avatar user system includes two main modules: Human making” based on a multi-criteria optimization approach that can
Digital Twin and Enhanced User. In Human Digital Twin, the improve the performance of the process and the quality of the
Data Preparation component stores and preprocesses both parts. The analytical models should be verified, validated, and
quantifiable and unquantifiable data, while the Information uncertainty must be quantified for accurate analyses. Then,
Model component formalizes them. The Immersive Analysis different analytical models must be composed into a single,
component analyzes and synthesizes the data sets. CraftsAvatar aggregated, integrated analytical model. Figure 4 conceptually
has domain-specific data demonstrating how to perform a task shows the composability task using three different models (A:
and give advice based on the user’s skill levels or professional response surface model, B: artificial neural network, and C:
maturity. CraftsAvatar can access, analyze, and synthesize Kriging model). Then, the composed model needs to characterize
manufacturing domain-specific data, information, and its component’s uncertainties and their propagation in the
knowledge through the DT module. This will lead to domain- aggregated model. Propagation requires careful consideration in
specific wisdom and intuition after further analysis and synthesis scaling, normalizing, weighting, and aggregating methods. The
of information from DT and EH. decision will be transferred to the physical entities if a near-
There are several research and technical issues. First, the optimal solution is determined. If not, additional data must be
knowledge needed to digitize and formalize the data types is collected, and further analysis must be performed.
significantly lacking. For example, how to formalize human
knowledge and intuition should be investigated as part of the
Data Preparation and Information Model. Second, the concept
and the implementation of Immersive Analysis are still in the
initial stage. Third, detailed case studies should be performed to
demonstrate the effectiveness of a semi-autonomous
CraftsAvatar.
The second module in AUS, the Enhanced User module,
provides immersive interactions with other physical entities
based on the human’s visual, auditory, and touch senses (taste
and smell are excluded due to the current technological
limitations). Photorealistic visualization plays the most crucial
role in immersion since humans collect up to 80 % of their
surrounding information through vision. The auditory and tactile
senses greatly enhance the immersion by hearing (e.g., 3D
sound) and providing a touch feeling of an object (e.g., the
texture of a car shift knob and handlebar). This sensory realism
can be implemented utilizing industrial metaverse technologies
for remote virtual training, concurrent design, and remote
monitoring. Users can then generate the two types of data already
discussed: (1) the quantifiable (e.g., Likert scale of customers’
preference) and (2) the unquantifiable (e.g., description of
customers’ perception). FIGURE 4: DIFFERENT TYPES OF SURROGATE MODELS
Implementing this module, however, demands solutions to
several research and technical issues. First, as the number of data For this module, a robust approach is needed to compose
modalities increases, the user’s immersive experience will individual analytical models and propagate the uncertainties that
improve; but more complicated integration tasks will be arise from disparate manufacturing resources. To develop this

5 © 2023 by ASME
approach, four significant challenges must be overcome and/or acquires the arc current and voltage data delivered from the
managed: (1) high uncertainty, conflicting objectives, sensor [32]. An HDR camera is attached to the torch to capture
heterogeneous forms of data, multi-interests, and perspectives; weld pool and bead images along with the movement of the torch
(2) complex and dynamically evolving manufacturing processes [33]. This camera was optimized for arc welding with a dynamic
and environments; (3) inherent interoperability issues in range of 140 dB to capture high-quality video frames. Standard
MCDM; and (4) requirements of verification and validation in camera systems are inapplicable due to their low dynamic ranges
aggregated/composed model [29]. and lightning interferences in arc welding. A camera data
interface recorded the images and converted them into .jpg file
4. A USE CASE IN WIRE + ARC ADDITIVE formats.
MANUFACTURING
Metal AM has attracted much attention since many
complicated machinery components should be produced with
metal, and other conventional manufacturing processes are much
more time- and cost- intensive. Among various metal AM
processes, Directed Energy Deposition (DED) uses a focused
energy source where the material is melted and deposited by a
nozzle. One of the representative forms of DED is WAAM,
which uses metal wire as the feedstock and an arc as an energy
source. Since the material in this process is deposited through
metal wire, the amount of metal usage can be minimized. In
addition, it benefits from a high deposition rate and is suitable
for large and custom-made metal parts. To demonstrate a use
case, we use the in-house WAAM system [30].
Although WAAM has many benefits, most companies still
hesitate to adopt it due to some its drawbacks. The additive
process based on arc welding can raise problems such as
spatter, porosity, undercut, deformation, crack, and slag [31].
Moreover, the design space in the WAAM process is huge, and
numerous parameters, directly and indirectly, influence the
final part, therefore, choosing the near-optimal process
FIGURE 5: CONCEPTUAL ARCHITECTURE OF THE DH-SM
parameters becomes of prime importance. To ensure that optimal
FOR A WAAM CASE
parameters are selected, human cognition is kept in the loop
with AI in the collaborative decision-making engine. In
addition, in-situ process monitoring and control are ultimately
lacking in the current WAAM systems.
To address these issues, we instantiated the DH-SM
architecture for the WAAM problem. As shown in Figure 5, we
generated the system architecture based on Figure 2. The use
case consists of the physical entity, the digital twin, the human
digital twin, and the enhanced users. In the following subsection,
each of these components will be discussed.

4.1 Cyber-Physical System for the WAAM Case

In accordance with Section 3, the CPS comprises two main FIGURE 6: EXPERIMENTAL ENVIRONMENT INCLUDING
WIRE FEEDER, SHIELDING GAS, TIG POWER SOURCE, TIG
modules: Physical Entities and Digital Twin. The Physical TORCH, ROBOT, AND HDR CAMERA
Entities include a GTAW-based WAAM system, as shown in
Figure 6. It also consists of a robot arm that moves to the The other module in CPS is the Digital Twin. This module
coordinates designated by the controller. The tungsten inert gas paves the way for creating a digital twin of the physical entities
(TIG) torch is attached to the hand of the robot arm and supplied by applying 3D visualization, data analytics, physics-based
with energy from the energy source. It deposits a feeding models, data-driven models (machine learning), and surrogate
material provided by the wire feeder to generate weld beads on models. A DT of the WAAM process was built to improve
the substrate. Process parameters, including travel speed, Wire product quality and production efficiency. Real-time monitoring
Feed Rate (WFR), and current, are determined by the controller data was collected to consistently achieve the mapping and
input, wire feeder, and energy source, respectively. Current and interaction between data and models to form the real-time
voltage sensors measure the numerical values of the arc interaction between the physical and digital twins. Then the
characteristics in real time. A data interface monitors and

6 © 2023 by ASME
process is analyzed. Firstly, a DT with 3D visualization of the enhanced user module can be categorized into edge users and
WAAM process is implemented. Secondly, as shown in Figure cloud users. The edge EU benefits from AR technology through
7, residual stress modeling and Crystal Plasticity Finite Element XR glasses and suits to perform remote inspection and training.
Simulation Method (CPFEM) are employed to model the Using the XR technology, the users can access real-time data
physical phenomena and the microstructural evolution of the such as the process parameters, online video of the process
parts. Thirdly, data-driven approaches are employed for real- including high dynamic range images, thermal images and CCD
time process monitoring. Finally, surrogate models that aim to cameras, simulations, and models. In remote locations, the users
convert the computationally expensive models to a reduced- can have the same level of access to the system through XR and
order model are employed to enhance the DT of the process. VR. The cloud-enhanced users at the cloud layer can modify the
Figure 8 shows an example of real-time anomaly detection using process where needed, change the process parameters based on
machine learning. their cognition and the real-time data acquired for the sensors,
and perform a real-time inspection of the process and the parts.
In addition, stakeholders of an enterprise can also have direct
access to the physical entities and the process and investigate the
possible enhancements. Figure 9 demonstrates the edge and the
cloud-enhanced users. Both types of users can access to real-time
process parameters, computational analysis of the process, and
microstructural analysis of the parts. One of the significant
advantages of EU at the edge and the cloud layer is that it
enables easy training sessions for expert users to teach the
process to beginners remotely and in real time.

(a)

(a)

(b)
FIGURE 8: MACHINE LEARNING FOR REAL-TIME ANOMALY
DETECTION (a) NORMAL AND (b) ABNORMAL BEADS [32]
(b)
The Human Digital Twin module includes a craft avatar for
FIGURE 7: DT COMPONENTS FOR WAAM PARTS: (a) every individual, including operators, technicians, engineers,
RESIDUAL STRESS AND (b) CPFEM trainers, and trainees at the edge layers and managers and
stakeholders at the cloud layers. These craft avatars can be used
The data collection is realized by the sensors and data for different purposes, including but not limited to training, real-
acquisition devices. By combining the real-time process, process time monitoring, control, and maintenance. This becomes
design, and process simulation, the DT elements are constructed. advantageous, particularly in the case of complicated processes
Then, by combining the historical and real-time data, online like WAAM, where the window map is narrow, meaning that it
process monitoring is enabled by data analysis. In case of any is challenging to find the near-optimal process parameters. This
abnormality, the correction process parameters will be timely is also true for most metal AM processes due to low process
sent back to the physical entity. Thus, the quality prediction and repeatability and part reproducibility [19]. As shown in Figure
control of welding of ship group products are realized. 10, three types of interactions are foreseeable, human, avatar,
and human-avatar in the HDT module. Human interactions are
4.2 Avatar-User System for the WAAM Case already well-established and available in the industry; however,
In the AUS, the concept of Human-in-the-Loop is ultimately the other two types are still to be developed. These
realized by the human DT and enhanced user modules. An

7 © 2023 by ASME
communications have numerous applications, such as training, The architecture proposed in this paper, DH-SM, can be
where a beginner is taught fundamentals, and concepts of the used to support the integration of physical assets with humans
WAAM process or an apprentice is trained to become an and digital twins. It will help enable real-time, collaborative
operator. decision-making between humans, software, and machines. For
example, when evaluating a new product design, information
about the product’s physical features, manufacturing
requirements, and customer experience must be processed
concurrently [34]. Moreover, the DH-SM architecture can
support the creation of an immersive environment that allows
customers to be effectively involved in manufacturing.
However, due to current standards and technology
limitations, implementing the DH-SM architecture still has
some challenges. First, realizing the real-time bidirectional
information flow is demanding. For example, the 3D-rendered
object of a human in an immersive environment must take place
in real time, currently, however, implementing such a real-time
interaction is challenging because of the huge amount of data that
(a) must be collected and processed. In addition, since these
interactions occur in a wireless environment, the low data
transmission rate is another issue. Advanced wireless techniques
(e.g., 5G and 6G) should be investigated and developed. To
achieve this, the interfaces between the architecture modules
must be implemented. Second, the concept of “Plug and Play” is
difficult to achieve. Software or devices need to work perfectly
when first used or connected without the need for
reconfiguration or adjustment by the user. Interoperability
standards are required to support these functionalities and
integration. The interface specifications and communication
protocols are not yet well developed. Third, cybersecurity is an
ongoing issue that must be linked to those interoperability
standards because the abundance of personal data and immersive
(b) content is prone to cyber threats [35]. Fourth, many
manufacturers, especially Small and Medium-sized Enterprises
FIGURE 9: ENHANCED USERS AT (a) THE EDGE LAYER (b)
THE CLOUD LAYER (SMEs), lack the infrastructure needed to use cloud-based
standards such as open platform communication unified
architecture (OPC-UA).
To provide a sense of realism to users, new immersion
methods will also be needed. These methods will provide users
with a more accurate perception of real manufacturing activities
with comfort and intuition. For this to be realized, immersive
modeling techniques should be advanced. Common interactive
technologies include XR and human-computer interface.
Current issues with these technologies include (1) the
interactive devices are not lightweight and transparent enough,
(2) the cost of the devices is high, and (3) there are also
associated costs for learning and using these devices. In
addition, the user’s mental/physical health and socio-economic
impacts must also be considered. We believe the existing
sustainable and smart manufacturing guidelines can be
extended and updated for DH- SM.
FIGURE 10: INTERACTIONS BETWEEN THE HDTS AND EUS One of the main incentives for DH-SM is realizing the
concept of non-contact and remote manufacturing, which is one
of the fast-paced advancing technologies. Metaverse technology
5. DISCUSSION adds an immersive experience to the configuration layer of

8 © 2023 by ASME
cyber-physical systems. In manufacturing, the Industrial Iyengar, K.P. Robotics Applications in COVID
Metaverse’s purpose would be to speed up processes like repairs, -19: A Review. Journal of Industrial Integration
maintenance, starting new manufacturing lines, remote and Management. Vol. 05, No. 04, pp. 441-451,
monitoring, control, and new user/manager training through (2020). DOI:10.1142/S2424862220300033
simulation. [36]. More efforts will be needed to ensure the [5] Park, N. KISTEP 10 Emerging Technologies in the
seamless and secure communication and synchronization Untact Era. Center for International Cooperation Policy,
between digital twins, remote human users, and physical Korea Institute of S&T Evaluation and Planning. (2021).
systems for non-contact manufacturing. [6] Wang, B., Tao, F., Fang, X., Liu, C., Liu, Y., and Freiheit,
T. Smart Manufacturing and Intelligent Manufacturing:
6. CONCLUSION A Comparative Review. Journal of Engineering,
7(6):738-757(2021). DOI:10.1016/j.eng.2020.07.017
We believe that the DH-SM concept and the proposed [7] Nakagawa, E. Y., Antonino, P.O., Schnicke, F.,
architecture are particularly relevant today, especially for Capilla, R., Kuhn, T., Liggesmeyer, P. Industry 4.0
manufacturing processes that require human intervention. Since Reference Architectures: State of the Art and Future
these manufacturing processes include humans at every level, Trends, Computer & Industrial Engineering. 156 (2021)
taking advantage of the human’s perception, cognition, and 107241. DOI:10.1016/j.cie.2021.107241
intuition is essential. We believe the DH-SM architecture will [8] Zheng, P., Wang, H., Sang, Z., Zhong, R. Y., Liu, Y., Liu,
provide real industrial impacts by managing a user’s C., Mubarok, K., Yu, S. and Xu, X. Smart
involvement in the evolving, complex, and dynamically Manufacturing Systems for Industry 4.0: Conceptual
changing manufacturing environment. Accordingly, it will Framework, Scenarios, and Future Perspectives. Front.
change the manufacturing landscape and can guide future Mech. Eng. 13 (2018) 137-150. DOI:10.1007/s11465-
research directions for developing standards, reference 018-0499-5
architectures, technologies; researching necessary components; [9] Wang, Y., Su, Z., Zhang, N., Liu, D., Xing, R., Luan, T.
and implementing case studies. In further studies, we aim to H., Shen, X. A Survey on Metaverse: Fundamentals,
develop the presented conceptual framework with different Security, and Privacy. arXiv 2022.
experimental plans and physical entities to demonstrate the DOI:10.48550/arXiv.2203.02662
capabilities and efficacy of the proposed DH-SM architecture. [10] Liu, M., Fang, S., Dong, H., and Xu, C. Review of
Digital Twin about Concepts, Technologies,
DISCLAIMER and Industrial Applications, J. Manuf. Syst. 58 (2021)
346-361. DOI: 10.1016/j.jmsy.2020.06.017
Certain commercial software systems are identified in this
[11] Singh, M., Fuenmayor, E., Hinchy, E.P., Qiao, Y.,
paper to facilitate understanding. Such identification does not
Murray, N., and Devine, D. Digital Twin:
imply that these software systems are necessarily the best
Origin to Future. Appl. Syst. Innov. (2021),
available for the purpose. No approval or endorsement of any
4(2), 36. DOI:10.3390/asi4020036
commercial product by NIST is intended or implied. This
[12] Singh, M., Srivastava, R., Fuenmayor, E., Kuts, V., Qiao,
material is based upon work supported by the National Science
Y., Murray, N., and Devine, D. Applications of Digital
Foundation under Grant No. 2015693.
Twin Across Industries: A Review. Appl. Sci. 12 (11)
(2022) 5727. DOI:10.3390/app12115727
REFERENCES
[13] Botín-Sanabria, D.M., Mihaita, A., Peimbert-García, R.
[1] White House. Strategy for American Leadership in
E., Ramírez-Moreno, M.A., Ramírez-Mendoza, R.A.,
Advanced Manufacturing. A Report by
and Lozoya-Santos, J.D.J. Digital Twin Technology
the Subcommittee on Advanced Manufacturing
Challenges and Applications: A Comprehensive Review.
Committee on Technology of the National Science &
Remote Sens. 14 (6) (2022) 1335.
Technology Council.
https://www.manufacturing.gov/sites/default/files/2021- DOI:10.3390/rs14061335
06/Advanced-Manufacturing-Strategic-Plan-2018.pdf [14] Thelen, A., Zhang, X., Fink, O., Lu, Y., Ghosh, S.,
(2018). Youn, B.D., Todd, M.D., Mahadevan, S., Hu, C., and
[2] Kapoor, K., Bigdeli, A. Z., Dwivedi, Y.K., Raman, R. Hu, Z. Correction: A Comprehensive Review of Digital
How is COVID-19 Altering the Manufacturing Twin—Part 2: Roles of Uncertainty Quantification and
Landscape? A Literature Review of Optimization, a Battery Digital Twin, and Perspectives.
Imminent Challenges and Management Interventions. Struct Multidisc Optim, 66 (2023).
Ann Oper Res. (2021) Nov 17:1-33. DOI:10.1007/s00158-022-03476-7
DOI:10.1007/s10479-021- 04397-2. Epub ahead of [15] Olesen, J. Concurrent Development in Manufacturing,
print. PMID: 34803204; PMCID: PMC8596861. based on dispositional mechanisms. Technical
[3] Badkar, M., Greeley, B. New Data Shows Vast Scale of University of Denmark (DTU), page(s): 153. 1992.
US Economic Breakdown (2020). ISBN: 87-89867-12-2
https://www.ft.com/content/6361247d-a1df-4a50-b0aa- [16] Lee, J., Bagheri, B., and Kao, H. A Cyber-physical
59395801b3f3
[4] Javaid, M., Haleem, A., Vaish, A., Vaishya, R., and
9 © 2023 by ASME
 Systems Architecture for Industry 4.0-based [29] Tapia, G., King, W., Johnson, L., Arroyave, R.,
Manufacturing Systems. Manufacturing Letters.Volume3 Karaman, I., and Elwany, A. Uncertainty Propagation
(2015) pp. 18-23. DOI:10.1016/j.mfglet.2014.12.001 Analysis of Computational Models in Laser Powder
[17] Liu, C. and Jiang, P., A Cyber-physical System Bed Fusion Additive Manufacturing using Polynomial
Architecture in Shop Floor for Intelligent Chaos Expansions. J. Manuf. Sci. Eng. 140 (12):
Manufacturing. Procedia CIRP, Volume 56 (2016), pp. 121006, (2018). DOI:10.1115/1.4041179
372-377. DOI:10.1016/j.procir.2016.10.059 [30] Kim, Y.S., Yun, D., Han, J.H., Ahsan, M.R., Huang, E.,
[18] Monostori, L. Cyber-physical Production Systems: Jain, J., Ji, C., Kim, D.B., and Lee, S.Y. Bimetallic
Roots, Expectations and R&D Challenges. 17 (2014) Additively Manufactured Structure (BAMS) of Inconel
9-13. Procedia CIRP, Volume 17, 2014, Pages 9-13. 625 and Austenitic Stainless Steel: Effect of Heat-
DOI:10.1016/j.procir.2014.03.115 treatment on Microstructure and Mechanical Properties.
[19] Kim, D.K., Witherell, P., Lu, Y., and Feng, S. Toward a Int J Adv Manuf Technol. 121, pp.7539-7549 (2022).
Digital Thread and Data Package for Metals-additive DOI:10.1007/s00170-022-09723-x
Manufacturing. Smart Sustain Manuf Syst. [31] Islam, S., Seo, G., Ahsan, M.R., Villarraga-Gómez, H.,
(2017)1(1):75-99. DOI:10.1520/SSMS20160003. Lee, H., and Kim, D.B. Investigation of
[20] Griffor, E.R., Greer, C., Wollman, D.A., and Burns, Microstructures, Defects, and Mechanical Properties of
M.J. Framework for Cyber-physical Systems: Volume Titanium-zirconium-molybdenum Alloy Manufactured
1, Overview. NIST Special Publication 1500-201 by Wire arc Additive Manufacturing. International
(2017). DOI:10.6028/NIST.SP.1500-201 Journal of Refractory Metals and Hard Materials.
[21] Lee, Y., Park, H., Woo, W., Ryu, J., Kim, H.K., Baik, Volume 110 (2023) 106042.
S.W., Ko, K.H., Choi, H.K., Hwang, S., and Kim, D.B. DOI:10.1016/j.ijrmhm.2022.106042
Immersive Modeling System (IMMS) for Personal [32] Kim, E., Lee, D., Seo, G., Kim, D., and Shin, S.
Electronic Products Using a Multi-modal Interface. Development of a CNN-based Real-Time Monitoring
Computer-Aided Design, Volume 42, Issue 5, (2010), Algorithm for Additively Manufactured Molybdenum.
pp. 387-401. DOI:10.1016/j.cad.2009.11.003 Sensors and Actuators A: Physical. Volume 352, (2023)
[22] T.J. Sullivan, Measures of Information and 114205. DOI:10.1016/j.sna.2023.114205
Uncertainty. Introduction to Uncertainty Quantification [33] Cho, H., Shin, S., Seo, G., Kim, D.B., and Lee, D. Real-
(2015) pp. 75-90. ISBN: 978-3-319-2[35]3395-6 time Anomaly Detection using Convolutional Neural
[23] Pedrycz, W. Ekel, P., and Parreiras, R. Fuzzy Network in Wire Arc Additive Manufacturing:
Multicriteria Decision-making: Models, Methods and Molybdenum Material, J. Mater. Process. Technol. 302
Applications, John Wiley & Sons, 2011. ISBN: 978-0- (2022) 117495. DOI:10.1016/j.jmatprotec.2022.117495
470-68225-8 [34] Xu, L., Li, Z., Li, S., and Tang, F. A Decision Support
[24] Kim, D.B., Witherell, P., Lipman, R., and Feng, S.C. System for Product Design in Concurrent Engineering,
Streamlining the additive Decis. Support Syst. Volume 42, (2007) Issue 4, pp.
manufacturing digital spectrum: A systems 2029-2042. DOI:10.1016/j.dss.2004.11.007
approach. Additive Manufacturing, Volume 5, [35] Williams, I. and Yuan, X. Evaluating the Effectiveness
5 (2015), pp. 20-30. of Microsoft Threat Modeling Tool. InfoSec '15:
DOI:10.1016/j.addma.2014.10.004 Proceedings of the 2015 Information Security
[25] Schlegl, T., Seeböck, P., Waldstein, S.M., Langs, G., Curriculum Development Conference. (2015) pp. 1-6.
Schmidt-Erfurth, U. f-AnoGAN: Fast unsupervised DOI:10.1145/2885990.2885999
anomaly detectionwith generative [36] Lee, J. and Kundu, P. Integrated cyber-physical systems
adversarial networks, Med. Image Anal. 54 (2019), and industrial metaverse for remote manufacturing.
pp. 30-44. DOI:10.1016/j.media.2019.01.010 Manufacturing Letters, Volume 34 (2022) pp. 12-15.
[26] Li, D., Chen, D., Jin, B., Shi, L., Goh, J., and Ng, S. DOI:10.1016/j.mfglet.2022.08.012
MAD-GAN: Multivariate Anomaly Detection for Time
Series Data with Generative Adversarial Networks.
Artificial Neural Networks and Machine Learning –
ICANN 2019: Text and Time Series, (2019), Volume
11730, pp.ISBN : 978-3-030-30489-8
[27] Dong, X., Yu, Z., Cao, W., Shi, Y., and Ma, Q. A Survey
on Ensemble Learning. Front. Comput. Sci. 14, pp. 241–
258 (2020). DOI:10.1007/s11704-019-8208-z
[28] Cometti, C., Païzis, C., Casteleira, A., Pons, G., and
Babault, N. Effects of Mixed Reality Head-mounted
Glasses during 90 Minutes of Mental and Manual Tasks
on Cognitive and Physiological Functions. PeerJ.
Volume 6 (2018). DOI:10.7717/peerj.5847

10 © 2023 by ASME