Integrating Materials and Manufacturing Innovation

https://doi.org/10.1007/s40192-025-00392-8

TECHNICAL ARTICLE

Automatic Detection of Dendritic Microstructure Using Computer

Vision Deep Learning Models Trained with Phase Field Simulations
A. Viardin1 · K. Nöth1 · C. Pickmann1 · L. Sturz1

Received: 11 September 2024 / Accepted: 22 January 2025

© The Author(s) 2025

Abstract
In this study, a novel approach to analyze the dendritic microstructure in solidification processes is presented, using an
innovative method to prepare datasets for deep learning training with minimal human intervention through phase field
simulations. This simulation technique, known for its accurate description of dendritic morphologies, enables the creation
of comprehensive and precise microstructure datasets. By using advanced deep learning techniques, in particular Faster
R-CNN and Mask R-CNN methods, we have successfully automated the detection of dendritic growth in various scenarios.
In microgravity and terrestrial solidification experiments with optically transparent alloys, faster R-CNN was particularly
effective in identifying loosely connected dendrites in experimental images and showed superior performance over thresh-
olding methods, especially in detecting optically overlapping dendrites. For contiguous dendrites in directionally solidified
polycrystalline metal alloys, mask R-CNN proved to be extremely proficient due to its ability to accurately delineate closely
spaced dendrites. The use of phase field simulations to generate datasets played a crucial role in training and testing these
models. Our research highlights the significant potential of deep learning in describing complex microstructural patterns,
contributing to a deeper understanding of the solidification process and its effects on material properties, with the added
benefit of facilitating dataset generation through a highly accurate microstructure generation method. Furthermore, this
method can be applied to any type of microstructure and to different types of materials, as the phase field simulations can
accurately simulate many microstructural properties.

Keywords Microstructure · Phase field · Mask R-CNN · Faster R-CNN · Deeplearning · Computer vision

Introduction carbon steel. Similarly, Banerjee and Sparks [9] used CNN
to classify dendritic microstructures. Liotti et al. [10] devel-
The use of artificial intelligence (AI), in particular deep oped a supervised method using real-time X-ray images to
learning and convolutional neural networks (CNNs), for track crystal nucleation events during the solidification of
software-driven automatic identification, categorization, and aluminium alloys and achieved high accuracy with an auto-
analysis of material properties based on two-dimensional mated image analysis algorithm. Breumier et al. [11], Mar-
experimental images have recently gained considerable tinez-Ostormujof et al. [12] and Germain et al. [13] showed
interest [1, 2]. Agbozo and Jin [3] used mask R-CNN [4] that deep learning methods based on U-Net architecture [14]
to segment carbide particles, achieving over 90% accuracy, outperformed thresholding image analysis in tasks such as
while Fu et al. [5] used faster R-CNN [6] with transfer learn- graphite shape characterization and phase discrimination in
ing to detect dendrite cores in Ni-based superalloys, show- steels. For experimental optical microscopy images from
casing high precision. Mulewicz et al. [7] and Elbana et al. video sequences or computer tomography slices, thousands
[8] have demonstrated the effectiveness of CNNs for the of images need to be processed to extract pertinent physi-
classification microstructural features in steels and ultrahigh cal features from complex microstructures. This requires the
development of novel AI tools to overcome the limitations of
thresholding techniques for image processing. During solidi-
* A. Viardin fication processes [15, 16], a variety of microstructures are
[email protected] observed, including dendrites with columnar or equiaxed
1 morphologies. Equiaxed dendrites form under conditions
Access e.V., Intzestrasse 5, D‑52072 Aachen, Germany

Vol.:(0123456789)
 Integrating Materials and Manufacturing Innovation

close to isothermal cooling and their tips grow in crystallo- Cheloee Darabi [21], a hybrid deep learning model is pre-
graphic directions, while columnar dendrites predominantly sented, trained on phase field simulations and finite element
develop along a temperature gradient in an elongated shape. analysis of dual-phase steel microstructures, demonstrated
This study aims to automate the detection and characteriza- the ability to predict mechanical properties such as yield
tion of dendrites in materials using images from time series stress, ultimate stress, and fracture strain.
or slices from volume datasets and deep learning techniques.
However, object detection, segmentation, and identification
performance are highly dependent on the training data, Experimental Data
which can be very labour-intensive to create manually [10],
especially when an image contains hundreds of objects. Equiaxed Dendritic Growth in Experiment MEDI‑2
One goal of this work is to generate a significant number of
realistic microstructures for training deep learning models. Microgravity Conditions
Initially, this was applied to experimental images of static,
focused, growing, and eventually overlapping, or connected The binary organic alloy Neopentylglycol–20.0 wt.-% (D)
equiaxed dendrites (experiment "MEDI-2" [17, 18]). Camphor (NPG-DC) was chosen to study the equiaxed
A first study [19] by the authors summarizes the use of AI growth of dendrites with <100> crystallographic orienta-
tools for computer vision for this experiment. Here, we have tion. The phase diagram and relevant properties of this alloy
further developed our methods by improving and refining are detailed in references [18, 27, 28]. The corresponding
the detection parameters, training data and result analysis. MEDI-2 experiment was conducted aboard the TEXUS-55
In addition, this work will be extended to the analysis of sounding rocket mission in 2018, a scenario in which melt
3D computer tomography transversal sections of solidi- convection and dendrite sedimentation are virtually non-
fied Al-Cu alloys with densely interconnected columnar existent. The experimental conditions were carefully selected
dendrites. In the present work, phase field simulations are to demonstrate the formation of equiaxed dendritic structure
widely used to generate images to train deep learning archi- formation under diffusive conditions for both heat and mass
tectures to analyze experimental microstructures. Phase field transport during microgravity. The transparent dendrites
models are widely used for the simulation of microstructural were observed in-situ using two different optical systems,
development in processes such as solidification, phase trans- in order to visualize both global and microscopic aspects of
formations, and mechanical behaviour. For a comprehensive equiaxed solidification with the TEM 06-23 TEXUS experi-
overview, Tourret et al. [20] provide a detailed review of mental module. As a result of these observations, equiaxed
the theoretical foundations and applications of phase field dendrites with six arms growing perpendicular to each other
modelling for various transformation in material. Phase field were identified, indicating of a <100> crystallographic ori-
models are often used in machine learning to create surro- entation. Figure 1 shows a schematic representation of the
gate models for microstructure evolution. In recent work of TEM 06-23 TEXUS module and illustrates the experimental

Fig. 1  Schematic representation of the TEM 06-23 TEXUS module (left) and the experimental cell with optical concept (right)
Integrating Materials and Manufacturing Innovation

setup. This device was provided by Airbus Defence & Space. Terrestrial Gravity Conditions
The temperature variations within the solid or liquid alloy
material were recorded using three Ni-CrNi thermocouples, This solidification experiment, conducted under conditions
each with a diameter of 0.25 mm, positioned in the central akin to those of the MEDI-2 experiment on TEXUS-55, dif-
vertical plane of the experimental cell. fers in that it was conducted under standard gravity condi-
Figure 2 shows a series of 8-bit greyscale overview tions (1 g). Under these conditions, the dynamics of dendrite
images captured by the CCD macro camera. The field of growth are also influenced by additional phenomena such
view spans 13.6 mm in width and 10.9 mm in height and as convective flow and sedimentation, which cause the den-
covers the entire 3 mm depth of the sample. The overview drites to move. Figure 3 shows a series of overview images
camera worked at 10 frames per second and achieved a pixel taken with the CCD macro camera. The experiment enables
resolution of 10.625 𝜇m/pixel in the image plane. The pic- the observation and analysis of the influence of gravity on
tures also show the inserted thermocouples at the right side the solidification process, in particular the interplay between
of the frame. the growth of the dendrites and the convective currents and
At the beginning of the microgravity phase, the NPG-DC sedimentation of the dendrites in the melt. A fundamental
alloy is in a completely liquid state. Subsequently, under difference between the microgravity and terrestrial experi-
the chosen experimental parameters, a zone of equiaxed ments is the average size of the dendrites. While they grow
dendrites begins to expand upward in the experimental cell, from their nucleation position (Fig. 2), they settle in gravity
following a modest thermal gradient. In situ and real-time and are advected by convection (Fig. 3), which gives them
observation of this process allows the sequential nucleation less time to grow until they are captured by the mushy region
and growth of equiaxed dendrites that remain in their posi- extending from the ground. The nucleation density is also
tions in the microgravity environment, as depicted in Fig. 2. different, which can be attributed to the changed nucleation

Fig. 2  Overview images of the experiment cell showing multiple equiaxed dendritic growth during the microgravity period. Also shown are the
thermocouples inserted from the right side. The size of the experiment cell is 13.6 × 10.9 mm2 (width × height)
 Integrating Materials and Manufacturing Innovation

Fig. 3  Overview images of the experiment cell showing multiple equiaxed dendritic growth during the terrestrial experiment period. Also shown
are the thermocouples inserted from the right side. The size of the experiment cell is 13.6 × 10.9 mm2 (width × height)

conditions due to a changed temperature field caused by the X-ray computer tomography scan. Figure 4 shows four
convection. selected sections/images from the directional solidification.
In contrast to experiments with equiaxed dendrites (see pre-
Columnar Dendritic Growth in Directionally vious section), which are only loosely interconnected, in
Solidified Al‑Cu Alloy this scenario, all dendrites are strongly interconnected due
to the interdependent and full solidification of the macro-
This section refers to directional solidification experiments scopic grains. This interconnection leads to a pronounced
tests with an Al-Cu alloy containing 10 wt.-% copper. The microstructural character that differs significantly from the
process started with solidification of the sample at a higher structures observed in previous experiments.
pull-off speed of 10 mm/min over a distance of 50 mm,
which was kept in a constant high-temperature gradient
(G=10 K/mm). The solidification speed was then reduced Methods
to 2 mm/min for 20 mm. This variation in speed was used to
investigate how grain structures form and adapt to changes Phase Field Model to Generate Artificial Dendrites
in solidification speed, particularly with respect to dendrite
spacing and grain size. At the end of the experiment, the The phase field model [22, 23] is extensively used to study
sample was rapidly quenched at a high speed of 180 mm/ the kinetics of phase transformations in heterogeneous
min, a step that is crucial for the preservation of the struc- materials. It is based on the concept that phase interfaces
tures formed in the solid state. The focus of the analysis is can be represented by continuous fields instead of discrete
the sub-region comprising the end of directional solidifica- boundaries. This approach is particularly effective in simu-
tion at 2 mm/min. This area is represented by a series of lating dendritic growth, which is a common occurrence in
individual images obtained from the 3D volume dataset of materials. The model can reproduce the complex branching
Integrating Materials and Manufacturing Innovation

Fig. 4  Sections of computer tomography scans at different positions along the z-axis, i.e., the main growth direction. Diameter of the sample is
8 mm; black regions show pores. Dark grey regions correspond to dendrite cores, while bright grey regions correspond to interdendritic zones

patterns that occur during the solidification of alloys or the

[ n−1
]
∑
crystallization of different materials. The simulations were ΔG𝛼𝛽 = ΔS𝛼𝛽 ΔT − mi𝛼𝛽 Δci𝛼 (3)
performed using the multiphase field model implemented in i=1

the MICRESS software [24, 25].

In Eq. 1, M𝛼𝛽 is the mobility of the interface as a function of
Phase fields 𝜙𝛼( x⃗,t) are designated for each phase 𝛼 in
the interface orientation, described by the normal vector n⃗ .
space x⃗ , and time t. The phase field equation implemented ∗
𝜎𝛼𝛽 is the surface stiffness depending on anisotropy param-
in MICRESS software is given by:
eter 𝜖𝛼𝛽 , and K𝛼𝛽 is related to the local curvature of the inter-
face, 𝜂 is the interface width. The motion of the interface is
( )
𝜋
√
̇𝜙𝛼 =
∑
∗
( ) ( ) ( i )
M𝛼𝛽 n⃗ 𝜎𝛼𝛽 n⃗ K𝛼𝛽 + 𝜙𝛼 𝜙𝛽 ΔG𝛼𝛽 c , T
𝛽 𝜂 driven by the curvature contribution 𝜎𝛼𝛽 ∗
K𝛼𝛽 and by the ther-
(1) modynamic driving force ΔG𝛼𝛽 . This driving force is a func-
tion of temperature T and the local composition
(2)
2 2 2
( )
K𝛼𝛽 = 𝜙𝛽 ∇ 𝜙𝛼 − 𝜙𝛼 ∇ 𝜙𝛽 + (𝜋∕𝜂) 𝜙𝛼 − 𝜙𝛽
ci = (c1 , … , cn ) and couples the phase field equations to the
multiphase diffusion equations:
 Integrating Materials and Manufacturing Innovation

N N respectively. These simulations aim to replicate roughly the

ċ i = ∇
∑ ∑
𝜙𝛼 Dij𝛼 ∇cj𝛼 with ci defined by ci = 𝜙𝛼 ci𝛼 (4) two types of dendrites observed in experiments, each with
𝛼=1 𝛼=1 different arm morphologies. By capturing different dendritic
D𝛼 is the multicomponent diffusion coefficient matrix for
ij structures under different conditions, this approach provides
phase 𝛼 and the i th element. D𝛼 and ΔG𝛼𝛽 are calculated for
ij a comprehensive data set that is crucial for the effective
the given composition and temperature. Now if we consider training and testing of deep learning models.
only interaction between liquid and solid grains for binary
alloy, we can rewrite Eq. 3 like: Phase Field Simulations for Dendritic Growth in Al‑Cu Alloy

𝜎ls [ ] We selected an Al-3at.-%Cu alloy for our simulations. The

ΔGls =
Γls f
T − ml (cl − c0 ) , (5)
physical properties are detailed in Table 1. The growth is
initiated by circular seed in a square domain with isolated
and Eq. 4 like:
boundary conditions for the phase field and alloy concentra-
[ ]
ċ = ∇ 𝜙l Dl ∇cl + 𝜙s Ds ∇cs with cs = k ⋅ cl (6) tion. The domain is 1 mm2 , with a grid spacing of 0.5 𝜇m
and an interface thickness of 2 𝜇m. The simulations include
The different parameters nominal concentration c0 , melt- latent heat, initial isothermal conditions at 918 K, and a heat
ing temperature Tf , liquidus slope ml , diffusion in liquid flow of -100 Js−1 cm−3. Nucleation is randomized using a
Dl , diffusion in solid Ds , Gibbs-Thompson coefficient Γsl , nucleation density law and the simulations last 10 s. Dif-
interfacial energy 𝜎sl , partition coefficient k, and interfacial fusion in the solid is not considered. Figure 6a shows the
energy anisotropy 𝜖sl for two different alloys, SCN-Acetone results of a selected simulation, illustrating the dendrite mor-
and Al-Cu used in the paper can be found in Table 1. We phology evolution over time. We used the copper concentra-
highlight that the aim of the simulations is not to generate tion map at 10 s for training our detection model. Simula-
physically identical dendrites as in the experimental images, tions with varying initial nuclei distributions were run to
but dendrites having a similar morphology. generate different microstructural images. Each image was
converted to grayscale to match the experimental pictures
Phase Field Simulations for MEDI‑2 Experiments grey levels as closely as possible as we can see on Fig. 6b.

For our simulations, we selected a succinonitrile-1at.-% ace- Deep Learning Models

tone alloy. The physical properties of this alloy are detailed
in Table 1.We started the growth from a circular seed in the Faster R‑CNN for Loosely Connected Dendrite Detection
corner of a three-dimensional cubic domain. To optimize the and Analysis on MEDI‑2 Experiments
efficiency of the calculations, symmetric boundary condi-
tions were set for the phase field and the alloy concentration The goal is to use deep learning neural networks to identify
on each side of the domain. This approach allows the simu- and localize individual dendrites in experimental images
lation of only one eighth of a dendrite, taking advantage of as shown in Figs. 2 or 3. Various architectures exist for
the internal symmetry of the dendrite, which significantly this purpose, including U-Net [14], Mask R-CNN [4], and
reduces the calculation time. After the calculations, the Faster R-CNN [6]. We have chosen Faster R-CNN for this
entire dendrite is reconstructed by symmetry, as shown in study because it is very effective, and the preparation of the
Fig. 5a. The calculation domain length is 880 𝜇m, where the training data is relatively simple. For the gravity-influenced
grid is discretized into cubic finite volume elements with a experiments, the dendrite motion is tracked using an inter-
grid spacing of 1 𝜇m each. The interface thickness is set to section over union (IoU)-based algorithm (IoU)-based algo-
4 𝜇m in the simulations. Constant isothermal conditions are rithm (section Objects tracking), which utilizes the results of
maintained for the simulation of equiaxed growth. object detection with Faster R-CNN. Faster R-CNN stands
Figure 5d shows the results of a selected simulation illus- as a prominent method in object detection, evolving from
trating the temporal evolution of dendrite morphology with the R-CNN series developed by Girshick et al. [29]. It pro-
a distinct interfacial energy anisotropy. Variations of this gressed through fast R-CNN [30] and culminated in faster
parameter lead to dendrites with different appearances, as R-CNN [6]. Here, we focus on the key aspects of Faster
shown in Fig. 5b and c. In Fig. 5b, the dendrite arms appear R-CNN. This segment of the Faster R-CNN architecture is
smoother, while in Fig. 5c, has smaller tip radii and nee- referred to as the “backbone” network. The feature map is
dle-like tertiary arms, reflecting the influence of interfacial then used by a region proposal network (RPN) to gener-
energy anisotropies. Figure 5b and c corresponds to 𝜖 values ate region proposals using anchors, i.e. fixed- size reference
of 0.07–0.7%, and isothermal temperatures of 328 K (under- boxes placed evenly over the original image. These propos-
cooling of 1.13 K) and 328.67 K (undercooling of 0.46 K), als are then filtered by non-maximum suppression (NMS),
Integrating Materials and Manufacturing Innovation

Fig. 5  a Simulation results at t=1 s for 1/8 of a dendrite (iso-surface phase field method with interfacial energy anisotropy (𝜖sl ) of 0.7%
for a phase fraction of 0.5), with reconstruction of the entire dendrite and undercooling of 0.46 K at 8 s.; and d Simulated time evolution of
by symmetry.; b Artificial equiaxed dendrite generated via phase field the dendrite, reconstructed by symmetry. The final dendrite is identi-
method with interfacial energy anisotropy (𝜖sl ) of 0.07% and under- cal to (c)
cooling of 1.13 K at 8 s.; c Artificial equiaxed dendrite generated via

a method that selects the most relevant regions. The feature

map from the CNN is summarized together with the iden-
tified bounding boxes of the objects in regions of interest
Table 1  Thermophysical properties of the SCN-acetone [31] and (RoI) to form a new feature map. This feature map then
Al-Cu alloys [26] passes through fully connected layers for the prediction of
SCN-Acetone Al-Cu
the object range coordinates and class outputs that form the
head network of the faster R-CNN architecture. For our pur-
concentration (c0) [at.-%] 1 3 pose, we implemented faster R-CNN using the Detectron2
Melting temp. (Tf ) [K] 331.23 933.5 framework, an open-source AI platform [32]. Detectron2 is
Liquidus temp. (Tl ) [K] 329.13 915.5 a Facebook© library offering advanced algorithms for object
Liquidus slope (ml ) [K at.-%−1] –2.1 –6 detection and segmentation, often with pre-trained models.
Partition coefficient (k) 0.1 0.097 We employed a ResNet50 network with a feature pyramid
Diffusion coefficient in liquid ( Dl ) [m2 1.27⋅10−9 3 ⋅10−9 network (FPN) as the backbone.
s−1]
Diffusion coefficient in solid ( Ds) [m2 s−1] 3 ⋅10−13 0 Faster R‑CNN Datasets Generation and Parameters
Gibbs-Thompson coefficient ( Γsl ) [Km] 6.525 ⋅10−8 2.36 ⋅10−8
Interfacial energy (𝜎sl ) [ Jm ]
−2 6.525 ⋅10−2 1.6 ⋅10−2 The 3D simulation results from Fig. 5 provided various 2D
Anisotropy of interfacial energy (𝜖sl ) [%] 0.07/0.7 0.1 projections with different camera angles (achieved using
ParaView [33] scripting), as depicted in Fig. 7. These
 Integrating Materials and Manufacturing Innovation

Fig. 6  a Phase field results at different times of 0.5 s, 1.5 s, 2.5 s, 3.5 s, 4.5 s, and 10 s from top left to bottom right. The simulation domain has a
size of 0.2 × 0.2 mm2 b training image example. Each dendrite is identified by its mask (in blue), generated from phase field simulations

projections were then augmented through stretching, over- Table 2  Parameters for the faster R-CNN implemented with Detec-
lapping, resizing, rotating, and anchoring at random posi- tron2 [32]
tions. A bounding box was created for each dendrite and Image per batch 2
labelled as class "0" (for equiaxed dendrites). In addition, Base learning rate 0.003
like in Fig. 2 artificial black bars were added to simulate Warmup iterations 100
thermocouples. Figure 8a shows an example of a training Checkpoint period 10000
image with extended objects, and Fig. 8b shows the same Maximum iterations 130000
image with bounding boxes drawn in from the annotation Steps (30000,50000,100000)
files. For training, a total of 1000 images were generated as Gamma 0.5
shown in Fig. 8a, which were derived from the phase field Momentum 0.9
results. The test dataset included 100 images, like in Fig. 2, ROI heads/Batch size per image 128
from experimental data, with annotations. The parameters Detections per image 150
for the Faster R-CNN architecture are listed in Table 3. Evaluation period 1000
Training was performed on an NVIDIA RTX 6000 GPU
and lasted less than 48 h. The training data for Detectron2
are formatted in the “COCO” style (Common Objects in
Context) [36], which is used for annotations and categoriza- columnar dendrites in cross-sections look like equiaxed
tions, such as the definition of the bounding box or mask of dendrites, motivating the 2D phase field simulations with
an object and its class index (Table 2). equiaxed microstructure as training data.
We chose the Mask R-CNN method because it is diffi-
Mask R‑CNN for Contiguous Dendrites from Computer cult to delineate closely spaced dendrites using the faster
Tomography Sections R-CNN method, as parts of the dendrites often extend into
neighbouring bounding boxes. The environment we chose
This section focuses on the automatic detection of colum- was “mmdetection” from OpenMMLab ( [34]). While the
nar dendrites in CT cross-sections (see Fig. 9) with the basic implementation of mmdetection and Detectron2 are
aim of determining global and intragranular dendrite and similar, they differ in detail such as regularization techniques
grain spacing in polycrystalline Al-Cu alloys with colum- to avoid overfitting. Detectron2 uses group norm regulariza-
nar microstructure morphology using deep learning meth- tion and L1 regularization, while mmdetection uses batch
ods in a later step. Grain sizes and dendrite spacing are of normalization and L2 regularization. In addition, mmdetec-
crucial importance for influencing the mechanical material tion uses a stochastic gradient descent (SGD) optimizer with
properties. A significant difference to the MEDI-2 experi- an adaptive learning rate, in contrast to Detectron2 with a
ments is the stronger interlocking of the dendrites, as they fixed rate Adam optimizer.
are already in a completely solid state. We highlight that
Integrating Materials and Manufacturing Innovation

Fig. 7  Various projections of the simulated 3D dendrites from Fig. 5b and c

Fig. 8  Training images illustrating a augmented objects and b augmented objects with automatically generated bounding boxes

Mask R‑CNN Datasets Generation and Parameters Objects Tracking

The training dataset comprises 1000 images as shown in We used the well-known tracking algorithm SORT [35]
Fig. 6b, while the test dataset comprises 56 images with (Simple Online Real-time Tracking) which models and pre-
resolutions of 100 × 100 or 200 × 200 pixels. Each dendrite dicts the dynamics of objects from frame to frame. Within
from phase field simulations is assigned a grain number, the SORT algorithm, Intersection over Union (IoU) is an
which facilitates the creation of direct pixel masks, as shown important metric in computer vision, especially for object
in blue in Fig. 6b. The parameters for mm detection are detection and tracking. It quantifies the accuracy of an object
listed in Table 3, with the IoU threshold and mask threshold detector in a data set by measuring the overlap between the
values given as a list for a parametric study to evaluate the predicted bounding box and the actual object boundary
efficiency of the model. Figure 6b serves as a representative (ground truth). IoU is calculated as the area of the overlap
input for the mask R-CNN, similar to the microstructure divided by the area of the union between these two boxes.
observed in Fig. 4. Its value ranges from 0 (no overlap) to 1 (perfect overlap). In
 Integrating Materials and Manufacturing Innovation

Fig. 9  a - Image section from experiment, b - Image section (7.18 × 6 mm2) as input image for Mask R-CNN (correspond to the red square zone
in image (a)

object tracking, IoU is used to evaluate how well the tracking on the other hand, is a probability cut-off used to turn the
algorithm can track the movement of objects across multi- predicted mask (a probability map) into a binary mask by
ple frames in a video. A high IoU value indicates that the deciding which pixels belong to the object. Unlike IoU,
tracker is tracking the object accurately, while a low IoU which compares regions, the mask threshold operates at the
value indicates that the tracker may lose the object or track pixel level to finalize the segmentation.
its position and size inaccurately. A threshold value for IoU
is usually set to distinguish between true and false positives.
Tracking is used for the MEDI-2 experiment in gravity to Results and Discussion
track the movement of objects. Concerning faster R-CNN or
Mask R-CNN, IoU measures how much two regions overlap Results of Faster R‑CNN Detection
and is used to decide if a proposal matches an object (dur-
ing training) and to remove overlapping predictions (during Microgravity Experiment MEDI‑2
inference). In Mask R-CNN, IoU also helps select regions
for mask prediction. The mask threshold (in Mask R-CNN), Figure 10 shows the bounding box around the dendrites filled
with red colour in the left column, which was detected using
a simple threshold method for grey levels. In this method,
Table 3  Parameters for the Mask R-CNN implemented with Open- the threshold was manually selected to detect the perimeter
MMlab [34] (NMS=’Non-Maximum Suppression’) of solid dendrites. The subjective selection of the threshold
Number of boxes before NMS 2000/1000 may lead to a small error in the detected dendrite area. In the
Number of boxes in NMS 1000 right-hand column of Fig. 10, we have drawn the bounding
Number of boxes after NMS 1000 boxes around the dendrites recognized with faster R-CNN.
IoU threshold 0.7, 0.8, 0.9 The boundaries are represented by white boxes. The class
Threshold for filtering out boxes 0.05 probability is between 98 and 100% in all cases. The lower
NMS threshold 0.5 class probabilities are mainly associated with the smallest
Maximum number of detections per image 1000 dendrites. In general, the results are very promising. Faster
Mask threshold 0.1, 0.3, 0.5 R-CNN detects dendrites near the equiaxed dense front and
Epochs 30 overlapping dendrites much better than simple threshold-
based methods. Nevertheless, small remarks must be made:
Integrating Materials and Manufacturing Innovation

Fig. 10  Comparison of threshold-based detection, on the left, and bounding boxes are represented in white, false positive with green
detection with faster R-CNN, on the right, with selected image times: boxes in the right column). The size of the experiment cell is 13.6 ×
150 s, 250 s, 300 s, 380 s in the microgravity period (true positive 10.9 mm2 (width × height)
 Integrating Materials and Manufacturing Innovation

• There are errors with both methods: in the simple thresh- description of the number of dendrites in this experiment.
olding method, they are many false positive objects, e.g. The comparison with the manual count naturally shows that
at the bottom right of the experimental cell. With faster faster R-CNN provides a better approximate value. The man-
R-CNN, only one false positive object is detected in the ual count did not consider the deep and heavily sedimented
upper right image area where, the upper thermocouple dendrites at the bottom. The same observation can be made
is located. However, these detections in different frames for the average bounding box area in Fig. 11b. The better
had a class probability of less than 98 % and are shown detection of packed/overlapping dendrites with the faster
with green squares in Fig. 10. They are negligible com- R-CNN method leads to a higher average bounding box in
pared to the total detections. With a good choice of mini- the last phase of the experiments.
mum value of class probability, we can discriminate false
positive reliably.
• When dendrites are heavily packed in the lower part of Gravity Experiments for MEDI‑2
the experimental unit, it is difficult for both the simple
threshold method and the faster R-CNN algorithm to Figure 12 shows corresponding dendrites found with faster
detect them. R-CNN. It again shows 4 images in which some dendrites
• Whether or not thermocouples are introduced during in different images are linked in their motion by post-pro-
training has no significant effect on the dendrite detec- cessing and visualized with boxes in specific colours. The
tion results but does allow thermocouples to be detected. other dendrite bounding boxes are shown with white boxes.
Figure 13 shows a representation of all trajectories of all
For a more detailed analysis, the evolution of the number dendrites detected in Fig. 12. The dendrites settle in increas-
of detected dendrites and their area over time was evaluated ing positions from the bottom with increasing time. It can
using the results from the simple thresholding method, man- also be seen that the sedimentation layers can be clearly
ual counting and Faster R-CNN. These results are shown in recognized by superimposing the tracks on top of each other
Fig. 11. Artefacts, as described above, were removed man- (sequence: black, blue, green, red). The data could be ana-
ually for both the simple thresholding method and Faster lyzed in more detail for movement velocities, sedimentation
R-CNN. Regarding the evolution of the number of dendrites depending on the size of the dendrites or for the visualiza-
over time, it can be observed that in the early stages, i.e. tion of convective flows. This is not the subject of this work.
between 50 and 200 s, the number of dendrites detected by
the threshold-based method matches that detected by Faster Results of Mask R‑CNN Detection for CT Sections
R-CNN, without strong overlapping or deposition of den-
drites. However, at later time points, i.e. more than 220 s, Figure 14a shows the segmentation result for the example
faster R-CNN detects overlapping dendrites better than in Fig. 9b. Some dendrites, especially at the edges, remain
the threshold-based method and provides a more realistic unrecognizable (dark grey). For comparison, a manual

Fig. 11  Experimental results showing a the number of detected dendrites, and b the box area of detected dendrites, both as a function of experi-
ment duration
Integrating Materials and Manufacturing Innovation

Fig. 12  The results of the tracking algorithm are visualized by boxes of the same colour (red and green), each representing the same dendrite.
The boxes are obtained using the Faster R-CNN detection. The size of the experiment cell is 13.6 × 10.9 mm2

contouring/segmentation was attempted, but this is not an in the best configuration, with an average of about 90%. If
exact reference due to subjective criteria. Figure 14b shows the mask threshold is increased from 0.1 to 0.5, the number
the manual segmentation, and Fig. 14c overlays the manual of objects detected depends on the IoU threshold. At IoU
segmentation with the contours from Fig. 14a. Figure 15 = 0.7, the number decreases as partially matching objects
illustrates the results and challenges with two detailed exam- with less precise masks are filtered out when the thresh-
ples. The R-CNN mask occasionally subdivides more finely old becomes stricter. At IoU = 0.8, the number increases as
than manual segmentation, as can be seen in Fig. 15c and f, the higher mask threshold eliminates interfering or partial
especially in areas with bright lines indicating differences masks and isolates well-defined objects more effectively.
in material density that are likely due to segregation; some At IoU = 0.9, the number remains stable, as only objects
of these areas are marked with a red circle in Fig. 15a and c, with almost perfect matches are detected and the stricter
and the corresponding areas in Fig. 15c and f. mask threshold has only a minimal effect on these already
Deciding what constitutes a dendrite for segmentation in precise detections. At IoU = 0.9, an average of 91% of den-
this context is a challenge in general. Therefore, the origi- drites was detected (relative to the manual counting). Due
nal goal of determining dendrite distances was revised. A to the stability of results at IoU = 0.9 versus mask threshold
parametric study was performed to evaluate the efficiency variation, we consider this value as the best one for detec-
of the model according to IoU threshold and mask thresh- tion in our study. This behaviour illustrates the interaction
old and compare it with manual dendrite counting. The between IoU and mask thresholds, where a lower IoU allows
results can be seen in Fig. 16, with the manual count rep- more flexibility, but a higher IoU results in cleaner masks
resented by the black line at the top. Due to the exclusion to improve object validation. The problem of dendrites not
of dendrites at the edges in Mask R-CNN, the results are being recognized at the edges can be explained by the fact
always lower than with manual counting. Nevertheless, that some dendrites were cutoff when we cropped the area
Mask R-CNN detects between 82% in the worst and 95% of the image from the entire sample image. The integration
 Integrating Materials and Manufacturing Innovation

Fig. 13  Trajectories of all identified dendrites. The colour coding refers to the first appearance of the dendrite in the total of 2500 images
(’frames’), at a frame rate of 10 Hz

Fig. 14  a: Result of the segmentation with mask R-CNN b: manual segmentation c: overlay of the manual segmentation from Fig. 14b and the
contours of the segmented dendrites from Fig. 14a. Size of domain is 7.18 × 6 mm2

of deep learning methods, in particular Faster R-CNN and dynamics. For connected dendrites in polycrystalline Al-Cu
Mask R-CNN, has shown remarkable capabilities in the alloys, mask R-CNN proved effective in segmenting closely
automated detection and analysis of dendritic structures in intertwined structures, a task that is challenging for con-
solidification processes. In microgravity experiments, faster ventional methods. The generated phase field simulations
R-CNN outperformed threshold detection methods by effi- provided diverse and realistic training data that improved the
ciently identifying and tracking moving and overlapping accuracy and robustness of the models. This study not only
dendrites. Dendrites that move and overlap were efficiently demonstrates the applicability of deep learning in materials
identified and tracked, providing valuable insights into their science but also lays the foundation for future research in
Integrating Materials and Manufacturing Innovation

Fig. 15  a and d: Section from the original image, b and e: manual segmentation, c and f: segmentation with the AI algorithm, red circle indi-
cates zone of density difference due to segregation. Size of domain is 1.3 × 1.3 mm2

the field of automated microstructure analysis, which could and overlapping dendrites. For more connected dendrites
revolutionize the understanding and manipulation of mate- in directionally solidified columnar polycrystalline Al-Cu
rial properties for various applications. alloys, mask R-CNN proved effective in segmenting closely
intertwined structures in 2D transversal sections from com-
puter tomography, a task that is challenging for more con-
Conclusion ventional methods. The generated phase field simulations
provided diverse and realistic training data that improved the
The integration of deep learning methods, in particular faster accuracy and robustness of the models. This study not only
R-CNN and Mask R-CNN, has shown remarkable capabili- demonstrates the applicability of deep learning in identifi-
ties in the automated detection and analysis of dendritic cation of microstructure in materials science but also lays
structures in solidification processes. In microgravity and the foundation for future research in the field of automated
terrestrial experiments of transparent equiaxed dendrites, microstructure analysis, which could revolutionize the
faster R-CNN outperformed simple threshold detection understanding and manipulation of material properties for
methods by efficiently identifying and tracking moving various applications.
 Integrating Materials and Manufacturing Innovation

Fig. 16  Parametric study of variation of IoU threshold of 0.7, 0.8 and 0.9 in green, red and blue respectively vs mask threshold and comparison
with manual counting (black dashed line)

Editor’s Video Summary The online version of this article (https://​doi.​ otherwise in a credit line to the material. If material is not included in
org/​10.​1007/​s40192-​025-​00392-8) contains an Editor’s Video Sum- the article's Creative Commons licence and your intended use is not
mary, which is available to authorized users. permitted by statutory regulation or exceeds the permitted use, you will
need to obtain permission directly from the copyright holder. To view a
Acknowledgements This work was financially supported by BMWK copy of this licence, visit http://​creat​iveco​mmons.​org/​licen​ses/​by/4.​0/.
(Bundesministerium für Wirtschaft und Klimaschutz, project KICK-
G, contract number FKZ50WM2050). The authors want to thank Dr.
Markus Apel, Dr. Bernd Böttger and Dr. Janin Eiken for fruitful discus-
sion on phase field model. References
Funding Open Access funding enabled and organized by Projekt 1. Holm EA, Cohn R, Gao N, Kitahara AR, Matson TP, Lei B,
DEAL. Yarasi SR (2020) Overview: computer vision and machine
learning for microstructural characterization and analysis. Met-
Declarations all Mater Trans A 51(12):5985–5999. https://​doi.​org/​10.​1007/​
s11661-​020-​06008-4
Conflict of interest The author declares that there are no conflicts of 2. Alrfou K, Zhao T, Kordijazi A (2024) Deep learning methods
interest regarding the publication of this paper. The author has not re- for microstructural image analysis: the state-of-the-art and future
ceived any financial support or benefits from any commercial source perspectives. Integr Mater Manuf Innov 13:703–731. https://​doi.​
or third party and has no financial interests or connections, direct or org/​10.​1007/​s40192-​024-​00369-z
indirect, that could have influenced the research presented in this man- 3. Agbozo R, Jin W-Y (2020) Quantitative metallographic analysis
uscript. of GCr15 microstructure using mask R-CNN. JKSPE 37:361–369.
https://​doi.​org/​10.​7736/​JKSPE.​019.​144
Open Access This article is licensed under a Creative Commons Attri- 4. He K, Gkioxari G, Dollár P, Girshick RB (2017) Mask R-CNN,
bution 4.0 International License, which permits use, sharing, adapta- ArXiv. https://​doi.​org/​10.​48550/​arXiv.​1703.​06870
tion, distribution and reproduction in any medium or format, as long 5. Fu L, Yu H, Shah M, Simmons J, Wang S (2020) Crystallographic
as you give appropriate credit to the original author(s) and the source, symmetry for data augmentation in detecting dendrite cores. J
provide a link to the Creative Commons licence, and indicate if changes Electron Imaging 32:1–7. https://​doi.​org/​10.​1117/1.​JEI.​29.2.​
were made. The images or other third party material in this article are 023020
included in the article's Creative Commons licence, unless indicated
Integrating Materials and Manufacturing Innovation

6. Ren S, He K, Girshick RB, Sun J (2015) Faster R-CNN: towards 21. Cheloee Darabi A, Rastgordani S, Khoshbin M, Guski V,
real-time object detection with region proposal networks, ArXiv. Schmauder S (2023) Hybrid data-driven deep learning framework
https://​doi.​org/​10.​48550/​arXiv.​1506.​01497 for material mechanical properties prediction with the focus on
7. Mulewicz B, Korpala G, Kusiak J, Prahl U (2019) Autonomous dual-phase steel microstructures. Materials 16:447. https://​doi.​
interpretation of the microstructure of steels and special alloys. org/​10.​3390/​ma160​10447
Mater Sci Forum 949:24–31. https://​doi.​org/​10.​4028/​www.​scien​ 22. Zhuang S, Zhou GD, Huynh P, Areias T, Rabczuk T (2022) Phase
tific.​net/​MSF.​949.​24 field modeling and computer implementation: a review. Eng Fract
8. Elbana R, Mostafa R, Elkeran A (2020) Data processing for Mech 262:108234. https://​doi.​org/​10.​1016/j.​engfr​acmech.​2022.​
automatic classification of spheroidite microstructure using deep 108234
learning based on FCNNs. Int J Mech Mechatron Eng 20(3):18–31 23. Chen LQ (2002) Phase-field models for microstructure evolution.
9. Banerjee D, Sparks TD (2022) Comparing transfer learning to Annu Rev Mater Res 32:113–140. https://​doi.​org/​10.​1146/​annur​
feature optimization in microstructure classification. iScience ev.​matsci.​32.​112001.​132041
25(2):103774 24. "MICRESS 7," Available: http://www.micress.de
10. Liotti E, Arteta C, Zisserman A, Lui A, Lempitsky V, Grant PS 25. Böttger B, Apel M, Daniels B (2019) Systematic phase-field study
(2018) Crystal nucleation in metallic alloys using x-ray radiogra- on microstructure formation during brazing of mar-M247 with a
phy and machine learning. Sci Adv. 4(4):eaar4004 Si-based AMS4782 filler. Metall Mater Trans A 50:1732–1747.
11. Breumier S, Ostormujof T, Frincu B, Gey N, Couturier A, Louk- https://​doi.​org/​10.​1007/​s11661-​019-​05113-3
achenko N, Aba-Perea P-E, Germain L (2022) Leveraging EBSD 26. Boukellal AK, Rouby M, Debierre J-M (2021) Tip dynamics for
data by deep learning for bainite, ferrite and martensite segmenta- equiaxed Al-Cu dendrites in thin samples: phase-field study of
tion. Mater Charact 186:111805. https://​doi.​org/​10.​1016/j.​match​ thermodynamic effects. Comp Mat Sci 186:110051
ar.​2021.​111638 27. Sturz L, Hamacher M, Eiken J, Zimmermann G, "Multiple Equi-
12. Martinez Ostormujof T, Purohit PR, Breumier S, Gey N, Salib M, axed Dendrite Interaction Investigated on MASER-13," Proceed-
Germain L (2022) Deep learning for automated phase segmenta- ings of the 23rd ESA Symposium on European Rocket and Bal-
tion in EBSD maps: a case study in dual phase steel microstruc- loon Programmes and Related Research, ESA-SP 741, p. 618-623,
tures. Mater Charact 184:111638. https://​doi.​org/​10.​1016/j.​match​ (2017)
ar.​2021.​111638 28. Zimmermann G, Hamacher M, Sturz L, "Columnar-to-Equiaxed
13. Germain L, Sertucha J, Hazotte A, Lacaze J (2024) Classifica- Transition in the Transparent Alloy System NPG-DC Observed
tion of graphite particles in metallographic images of cast irons: in the Low-Gravity and Normal Gravity Experiment TRACE-3,
quantitative image analysis versus deep learning. Mater Charact In: Proceedings of the 23rd ESA Symposium on European Rocket
217:114333. https://​doi.​org/​10.​1016/j.​match​ar.​2024.​114333 and Balloon Programmes and Related Research, ESA-SP 741, p.
14. Ronneberger O, Fischer P, Brox T (2015) U-Net: Convolutional 630-637, (2017)
Networks for Biomedical Image Segmentation, ArXiv, 2015. 29. Girshick R, Donahue J, Darrell T, Malik J, "Rich Feature Hier-
https://​doi.​org/​10.​48550/​arXiv.​1505.​04597 archies for Accurate Object Detection and Semantic Segmenta-
15. Boettinger WJ, Warren JA, Beckermann C, Karma A (2002) tion, In: 2014 IEEE Conference on Computer Vision and Pattern
Phase-field simulation of solidification. Annu Rev Mater Res Recognition, p. 580-587, (2014). https://​doi.​org/​10.​1109/​CVPR.​
32:163–194. https://​doi.​org/​10.​1146/​annur​ev.​matsci.​32.​101901.​ 2014.​81
155803 30. Girshick RB (2015) Fast R-CNN, ArXiv. https://d​ oi.o​ rg/1​ 0.4​ 8550/​
16. Kurz W, Fisher DJ (1986) Fundamentals of Solidification. Trans arXiv.​1504.​08083
Tech Publications, Zurich 31. Viardin A, Zaloznik M, Souhar Y, Apel M, Combeau H (2017)
17. Zimmermann G, Hamacher M, Sturz L (2019) In-situ observation Mesoscopic modeling of spacing and grain selection in columnar
of equiaxed dendrites solidifying under reduced gravity condi- dendritic solidification: envelope versus phase-field model. Acta
tions during the Texus-55 mission - The Medi-2 experiment, In: Mater 122:386–399. https://​doi.​org/​10.​1016/j.​actam​at.​2016.​10.​
Proceedings of the 24rd ESA Symposium on European Rocket 004
and Balloon Programmes and Related Research, ESA-SP 742 32. Wu Y, Kirillov A, Massa F, Lo W-Y, Girshick R Detectron2.
p-250-257 https://​github.​com/​faceb​ookre​search/​detec​tron2
18. Sturz L, Hamacher M, Zimmermann G (2023) Effect of composi- 33. ParaView https://​www.​parav​iew.​org
tion on nucleation and growth of equiaxed dendrites in the trans- 34. OpenMMLab (2022) https://​github.​com/​open-​mmlab
parent alloy neopentylglycol-(d)camphor during solidification in 35. Bewley A (2023) SORT A simple online and realtime tracking
microgravity conditions. J Cryst Growth 620:127350. https://​doi.​ algorithm for 2D multiple object tracking in video sequences.
org/​10.​1016/j.​jcrys​gro.​2023.​127350 https://​github.​com/​abewl​ey/​sort
19. Viardin A, Noth K, Torabi Rad M, Sturz L (2022) Automatic 36. Lin T-Y, Maire M, Belongie SJ, Bourdev LD, Girshick RB, Hays
Detection of Equiaxed Dendrites Using Computer Vision Neural J, Perona P, Ramanan D, Dollar P, Zitnick CL (2014) Microsoft
Networks, ArXiv. https://​doi.​org/​10.​48550/​arXiv.​2207.​07428 COCO: common objects in context, ArXiv. https://​doi.​org/​10.​
20. Tourret D, Liu H, Llorca J (2022) Phase-field modeling of micro- 48550/​arXiv.​1405.​0312
structure evolution: recent applications, perspectives and chal-
lenges. Prog Mater Sci 123:100810. https://​doi.​org/​10.​1016/j.​ Publisher's Note Springer Nature remains neutral with regard to
pmats​ci.​2021.​100810 jurisdictional claims in published maps and institutional affiliations.