Acta Materialia 170 (2019) 118e131

Contents lists available at ScienceDirect

Acta Materialia
journal homepage: www.elsevier.com/locate/actamat

Full length article

Convolutional neural network-based method for real-time orientation

indexing of measured electron backscatter diffraction patterns
Yu-Feng Shen a, b, Reeju Pokharel a, *, Thomas J. Nizolek a, Anil Kumar a, Turab Lookman a
a
Los Alamos National Laboratory, Los Alamos, NM, 87544, USA
b
Carnegie Mellon University, Pittsburgh, PA, 15213, USA

a r t i c l e i n f o a b s t r a c t

Article history: Electron backscatter diffraction (EBSD) is the most commonly used technique for obtaining spatially
Received 13 September 2018 resolved microstructural information from polycrystalline materials. We have developed two convolu-
Received in revised form tional neural network approaches based on domain transform and transfer learning to reconstruct
13 March 2019
crystal orientations from electron backscatter diffraction patterns. Our models are robust to experi-
Accepted 20 March 2019
Available online 27 March 2019
mentally measured image noise and index orientations as fast as the highest EBSD scanning rates. We
demonstrate that the quaternion norm metric is a strong indicator for assessing the reliability of the
reconstructions in the absence of the ground truth. We demonstrate the applicability of the current
Keywords:
Microstructure reconstruction
methods on a tantalum sample.
Convolutional neural network © 2019 Acta Materialia Inc. Published by Elsevier Ltd. All rights reserved.
Electron backscatter diffraction

1. Introduction characterization of relatively large (0.1e10 mm) polycrystalline

samples [6,7]. EBSD provides orientation maps at sub-grain spatial
Spatial microstructural heterogeneities at the mesoscale resolution (~200 nm) and high crystal orientation resolution (<0.5
(~1e100 mm), including grain boundaries, phase boundaries, dis- deg). During an EBSD measurement, an electron beam is focused
locations, defect structures, and chemical segregation, directly onto a spot on the sample surface and a diffraction pattern is
control macroscopic material properties and behavior. Currently, recorded. Top of the line commercially available orientation imag-
the lack of understanding of the complex collective response of ing microscopes (OIM) can make one such measurement in less
mesoscale features to external stimuli poses fundamental chal- than 1 ms [8]. When optimizing for speed, the pattern quality de-
lenges in predicting materials’ microstructure evolution [1,2]. This grades, which results in reduced orientation indexing fidelity with
is critical for engineering new materials with tailored properties. conventional reconstruction methods [9]. In contrast, the state-of-
Experimental techniques have advanced significantly in the last the-art dictionary indexing (DI) method significantly improves the
two decades, enabling destructive and non-destructive character- orientation reconstruction accuracy from highly noisy EBSD pat-
ization of material microstructures and their evolution at high terns [9,10]. However, real-time orientation indexing using the
spatial and temporal resolutions [3e5]. With increasing data dictionary learning method is not yet viable, unable to match scan
collection rates and sizes of datasets per sample reaching tens of speeds which can reach as high as 1 ms/pattern, limiting the DI
TBs, there is considerable interest in exploring how efficiently approach to be an off-line method [11,12].
analysis methods based on machine learning can be applied to the Fast orientation indexing from noisy patterns is crucial for in-
reconstruction process. In this work, we show how convolution situ microstructure evolution [7,9] and other quantitative studies
neural networks (CNNs) can be used as “surrogate” models for that require statistically significant volume measurements at high
solving the inverse problem of reconstructing microstructures from spatial resolution [3]. In spite of progress towards fast data collec-
diffraction data. tion times and computation, there is a need for new methods to
Electron backscatter diffraction (EBSD) is the most extensively accelerate the orientation indexing from noisy EBSD patterns,
used technique for 2D and/or 3D spatially resolved crystallographic especially if we are to make progress towards real-time in-situ
studies and 3D microstructure characterization.
Recently, deep convolutional networks have been utilized for a
* Corresponding author. range of imaging techniques from automated image segmentation
E-mail address: [email protected] (R. Pokharel).

https://doi.org/10.1016/j.actamat.2019.03.026
1359-6454/© 2019 Acta Materialia Inc. Published by Elsevier Ltd. All rights reserved.
{ The specific crystallographic structure of the material causes the electrons to scatter in certain predictable directions. The angles and intensities of
these scattered electrons contain information about the crystal's orientation and microstructure.

Elastic scattering occurs at surface, but backscattered electrons are reflected back from some greater depth, thus they contain the above infomation.
 Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131 119

[13,14] to image reconstruction for magnetic resonance imaging high rates.

[15]. While CNNs have demonstrated immense success for prob-
lems related to image classification [16] and detection [17], only a 1.2. Outline
few studies have utilized CNNs to address regression problems
[18e20] for improving the performance of a forward model. The basic principle of EBSD (measurements and simulations) is
Liu et al. [11] demonstrated the first application of convolu- presented in Section 2. A brief background on CNNs is reviewed in
tional neural networks for EBSD orientation indexing from simu- Section 3, and their application to EBSD is presented in Section 4. In
lated diffraction patterns. Separate CNN models were trained to Section 5, we summarize our key findings related to the model's
predict individual components of the crystal orientation. Jha et al. sensitivity to image noise and experimental parameters, and
[12] extended this approach to predict the full crystal orientation discuss the computational cost of the proposed methods.
using a deep CNN network, employing mean disorientation in the
loss function to account for the non-linearity of the orientation 2. Electron backscatter diffraction
space. However, the implementation of complicated loss functions
led to relatively long training times. In both the studies, the models A diffraction pattern (intensity distribution) on a detector, X ¼
were only trained and tested on simulated EBSD patterns, in Xðx;yÞ, where ðx; yÞ is pixel location, can be expressed as a function
contrast to real experimental diffraction patterns; therefore, the of crystal orientation represented as a quaternion q ¼ ðq0 ;q1 ;q2 ;q3 Þ,
effects of measurement noise were not considered. In this work, of the form:
we demonstrate the efficacy of the neural network approach for
experimentally measured data with varying noise levels. We pre- X ¼ P ðfðqjM ; C ; G ÞÞ; (1)
sent two convolutional neural networks based EBSD reconstruc-
tion techniques that are applicable to experimental diffraction where M parameterizes the microscope, C denotes the crystal-
patterns and robust towards measurement noise. In particular, we lography of the sample, G represent the geometry of the setup, and
show that by utilizing a simulation-trained CNN to provide an P ðlÞ represents a random Poisson sample drawn from a Poisson
initial estimate for re-training a small part of the network or distribution with mean l. Like many other diffraction-based mea-
combining it with an additional network with a small amount of surement methods, reconstructing a crystallographic orientation
experimental data results in high prediction accuracy for from an electron backscatter diffraction (EBSD) pattern is achieved
measured data. by solving an inverse problem, i.e. finding the inverse function
which maps a given diffraction pattern to an orientation: gðXÞ ¼ q.
1.1. Summary of main results The most commonly used approach for orientation indexing
from measured EBSD patterns utilizes image processing techniques
We utilize the existing physics-based model to generate a large including Gaussian filtering, Hough transforms, butterfly convolu-
number of training sets that span the entire crystallographic tion, and peak detection for extracting Kikuchi bands [21,22].
orientation space and use these simulations to develop faster Crystal orientations are reconstructed by comparing the parame-
“surrogate” models based on convolutional neural networks for ters of the measured Kikuchi bands (i.e. angles between lines) with
predicting crystal orientations from simulated diffraction patterns. a pre-computed database. Recently, Chen et al. [23] proposed a
Since the simulation trained model cannot be directly used on dictionary indexing (DI) method for microstructure reconstruction
measured data, the novelty of this work is in the demonstration based on dictionary generation and pattern matching. The DI
that a simulation trained model can instead be used as an initial approach utilizes a physics-based forward model developed by
guess or estimate for re-training a small part of the network or Callahan et al. [24] for the dictionary generation, where the forward
combining it with an additional network that only requires a small model are tuned to the parameters of the experiment and the
number of expensive experimentally measured data (300 K crystal structure of the sample. Chen et al. simulated a dictionary of
simulated versus 1 K measured patterns). We also demonstrate diffraction patterns corresponding to a set of orientations that
that the re-training with a straight forward loss function is rela- uniformly covered the fundamental zone. In the DI approach, the
tively fast. The robustness of the CNN methods to the measure- reconstruction is performed by calculating inner products between
ment noise is demonstrated by collecting experimental data with an experimental pattern and every simulated pattern in the dic-
different exposure times and therefore varying noise levels in the tionary. The crystal orientation is then estimated using “k nearest
diffraction patterns. Because our techniques are robust to noise, neighbors”, by determining k largest inner products. The compu-
they have the potential to enable higher scan speeds, which are tational complexity of the dictionary method to index one pattern
currently limited, in part, by the fact that the signal to noise ratio is OðdnÞ, where d is the dictionary size and n is the number of pixels
drops as the scan speed is increased. The CNN based reconstruc- for each diffraction image, assuming k≪d [23]. Typically,
tion method takes as little as 1 ms for orientation indexing, n ¼ 60  60 and d  3  105 for cubic symmetry materials.
matching the highest practical scan rates, and thus enabling real-
time orientation indexing. We also demonstrate that a quaternion 2.1. Simulations
norm metric provides a good confidence measure for assessing the
quality of the reconstruction, without requiring ground truth The approach presented here also utilizes the forward model
{ novelty information. developed by Callahan et al. [24] to generate a large number of
To the best of our knowledge, this is the first demonstration of simulated diffraction data. We generated a “uniform grid” of 1:5+
the potential of neural network based methods for microstructure resolution on the cubic crystallographic orientation fundamental
reconstruction from experimentally measured EBSD diffraction zone (FZ), and simulated the diffraction pattern for each orientation
patterns with measurement noise. The CNN based approach using the EMsoft software package [25]. The geometrical parame-
demonstrated here is directly applicable to solving the inverse ters for the simulations were obtained from the known detector
problems faced by multiple experimental modalities such as and sample angles as well as the OIM analysis software fit pattern
transmission electron microscopy, tomography, high-energy X-ray center corresponding to the central scan point. Further minor
diffraction microscopy, and coherent diffractive imaging, enabling refinement of the pattern center parameters was performed using
mesoscale microstructure characterization and reconstruction at the EMsoftWorkbench software provided by the EMsoft package
120 Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131

[25]. In total, 333227 pairs of orientations and diffraction patterns standard deviation of approximately 0.03.
were generated. The same sample was also used to study the robustness of the
CNN model towards the presence of experimental noise. For this
purpose, a 1 mm  1 mm area of the Ta sample was scanned
2.2. Measurements
repeatedly with a 20 mm step size. The images were collected using
8  8 (60  60 pixels) binning. Different scans were recorded using
Experimental EBSD diffraction patterns were collected from an
various exposures ranging from 30 ms to 1 ms per point. The area
annealed tantalum sample with 35 mm average grain size. A
scanned was identical, and a final scan at the end of the experiment
1 mm  1 mm area was scanned with 2 mm step size using a FEI
was conducted again at 30 ms exposure to ensure that sample drift
Inspect scanning electron microscope equipped with an EDAX
was not significant during the experiment (i.e. the points scanned
Hikari Super camera. The electron beam accelerating voltage was
and orientations were similar across all datasets). The orientation
20 kV, the beam aperture diameter was 50 mm, and the nominal
maps for 30, 2, and 1 ms exposures are shown in Fig. 2(a) and B2(c),
working distance was 20 mm. A diffraction pattern was collected at
and 2(d), respectively. In Fig. 2(b), the misorientation map shows
each sample position, for a total of 501  501 ¼ 251001 diffraction
the difference in orientations between the initial and the final
patterns. The inverse pole figure (IPF) map obtained from the OIM
30 ms exposure time scans. The random distribution of low
analysis software is shown in Fig. 1(a).
misorientation angles between the initial and the final 30 ms scan
The original detector image was collected by binning at a 1  1
points suggests that sample drift was not an issue.
with 480  480 pixels; however, software binning (8  8) was
applied on the raw images to yield the 60  60 pixel images used
for the CNN modeling. It should be noted that the software binning 3. Convolutional neural network
results in a larger readout noise component compared to CCD
hardware binning (in this case approximately eight times larger Multilayer feed forward networks have proved to be universal
readout noise). However, for the high signal levels at the long function approximators to learn or estimate a mapping or function
exposure times considered here, the proportion of readout noise, between inputs (or features) and outputs (objectives) (e.g. y ¼ f(x))
and therefore the difference between hardware and software [26e28]. In particular, the convolutional neural network has
binning, is expected to be minimal. The measured (left) and become a key tool for the state-of-the-art methods in computer
simulated (right) diffraction patterns for the same orientation and vision [16]. Unlike deep neural networks where each neuron is
instrument settings are shown in Fig. 1(b). connected to all the neurons in the previous layer, the CNN limits
As an approximation, we also estimated the magnitude of the such spatial connectivity through a restricted region of activated
noise in the experimental patterns. Distinct grain orientations in neurons (receptive field) with shared weights [28]. Taking advan-
the experimentally measured sample were determined. In an ideal tage of such an architecture allows CNNs to have fewer degrees of
case for ordered samples, the diffraction patterns obtained from freedom than traditional multi-layer fully connected neural net-
orientations corresponding to the same grain should be identical works and make convolutional networks more efficient for solving
and any deviation from the average value of any given pixel in the problems related to high-dimensional images.
diffraction pattern would be due to random noise. Note that the
tantalum sample used in this study was well annealed; therefore, it 3.1. Pre-training the CNN using simulated data
is expected to have low intra-granular misorientations; therefore,
this noise calculation procedure (albeit unrepresentative of exper- We developed a CNN based regression model trained end-to-
imental noise) is reasonable to use in this case. An example of the end to approximate the mapping from EBSD diffraction patterns
2D intensity maps of the mean (left) and standard deviation (right) to crystallographic orientations. First, the simulated diffraction
for a given grain is shown in Fig. 1(c). We measured the standard patterns and the corresponding crystal orientations were used for
deviation of each pixel intensity associated with a given grain model training, validation, and testing. Note that the training sets
orientation and found that the normalized pixel intensities had a are used for updating the weights of neural network, while the

{ experimental Noise

Fig. 1. (a) Inverse pole figure map of the Ta sample. (b) Measured (left) and simulated (right) diffraction patterns, and (c) mean (left) and standard deviation (right) of the
experimental patterns associated with the same crystal orientation. The standard deviation of every pixel intensity is about 0.03, which is the estimated noise in the experimental
patterns.
 { Generated from Experimental patterns ( without transformation )
Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131 121

Fig. 2. EBSD orientation maps obtained from OIM analysis software of the same region acquired with different exposure times. (a) The first scan acquired with 30 ms exposure. (b)
Misorientation map showing difference in measured orientations for the first and the last scans, both acquired with 30 ms exposure times. The orientation maps acquired with (c)
2 ms and (d) 1 ms exposure times. Hough transform method produced lower fidelity orientation reconstructions as the noise levels in the diffraction patterns increased with
decreasing exposure times.

{ single EBSD measurement (producing one Kikuchi pattern) gives the orientation of the crystal lattice at that precise point. And this orientation can
be explained by Euler angles or quaternion points.

Fig. 3. Convolutional neural network architecture. The input is a diffraction pattern, which goes through several convolutional layers and then the fully connected layers. The output
is four real-valued numbers corresponding to quaternion components that represent a crystal orientation. Numbers in the parentheses show the dimensions of each step. The
convolutional operations are shown in red, max pooling operations are shown in black, full connections are shown in blue. (For interpretation of the references to color in this figure
legend, the reader is referred to the Web version of this article.)
122 Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131

validation set are used for evaluating the performance during the 1000 pairs for validation, and the remaining 332227 pairs for
training process. The test set is unseen by the neural network training. We also randomly generated a separate 1000 pairs of
during the training process and is used to evaluate model accuracy orientations and diffraction patterns from the cubic fundamental
and ensure that the model is not overfitting. All input images were zone, which were then used for testing.
normalized so that pixel intensity ranged between 0 and 1. Fig. 3 The results from 10 separate trainings are shown in Fig. 4.
shows the CNN architecture used. A detector image with 60  60 Fig. 4(a) shows the training history, where the shaded area is the
pixels with one channel for intensity was used as input to the CNN. standard deviation over 10 runs. The plot illustrates that both
The dimensions of the convolution filters and the fully connected validation and training errors converged after fewer than 40
layers are also shown in Fig. 3. The detector image passes through a epochs, which means that the neural network passed through all
convolutional layer which gets transformed into a feature map the training samples 40 times to update the network weights.
which then passes through a non-linear rectified linear unit Fig. 4(b) shows the ground truth and prediction of each quaternion
(ReLUðxÞ ¼ maxf0; xg) activation function [29]. The loss function component.
we used for training our neural network is the mean absolute error Fig. 4(c) shows the distribution of test error, where misorien-
(MAE) between each quaternion component of the prediction and tation between predicted orientations and ground truth is used as a
the ground truth. Pooling layers are also employed occasionally to metric for quantifying the “goodness” of the prediction. As we can
reduce the size of the feature map and the computational cost. In see, the peak is about 2+, and 97% of the predictions are below 10+
addition, pooling layers make the network less sensitive to local misorientation. Note that to employ a simple loss function, crys-
features, and help control overfitting to some degree. Finally the tallographic orientations were represented by quaternions instead
outputs of the convolutional and pooling layers go through the fully of Euler angles, as previously used in the work by Liu et al. and Jha
connected layers, which estimate the final output crystal orienta- et al. [11,12]. By predicting all four quaternion components, we are
tions corresponding to the input detector images. After passing allowing the quaternions to change in all four directions, while a
through several convolutional layers, and then the fully connected crystal orientation has only three rotational degrees of freedom.
layers, the output of the CNN is four real-valued numbers Since the training dataset has quaternions with unit norm, the
which represent quaternion components. A detailed description of training process in effect should penalize the outputs that are far
the convolutional neural network employed here is given in from unity, to some degree. This can be observed from the
Appendix A. reasonably good agreement between the predicted and the ground
We utilized a repeated random sub-sampling validation truth orientations, shown in Fig. 4(c). However, to ensure accurate
method, also known as Monte Carlo validation [30]. More specif- orientation prediction, one could impose explicit constraints that
ically, we trained the model several times and each time we enforces unit norm for output quaternions.
randomly split the dataset into training and validation sets, and the Fig. 4(d) shows a strong correlation between misorientation
errors were estimated by averaging over runs. In each split, we used angle and the deviation from unity of the quaternion norm.

Fig. 4. Results of the trained CNN model utilizing the simulated EBSD data. (a) Training history: Both training and validation errors converge to about 0.01 after 40 epochs. The
shadow area is the standard deviation of 10 runs. (b) Predicted quaternion components of the testing set, where most of predicted orientations (97%) compare well with the ground
truth, except for a few outliers. (c) The distribution of misorientation angle between the prediction and the ground truth. (d) The misorientation angle versus deviation from unity of
the quaternion norm.
{ GB-PFZs: These zones capture the unique influence of crystal symmetry on grain boundary character. They define a specific region in orientation space that
encompasses all possible "unique" grain boundary configurations for a particular crystal system.

Conventionally all the symmetric operators are applied to quaternions and the one having minimum disorientation is considered.
So, we can do the same by incorporating disorientation at all the transformation as loss function in CNN
Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131 123

However, the large deviation from unity is an obvious artifact of our respectively. For the training and validation sets (333227 data
CNN model. An interesting observation is that all the outliers had at points), the original model had 4307 (1.29%) outliers and the
least one quaternion component close to the cubic fundamental effective FZ approach had 191 (0.05%) outliers, where outliers are
zone boundaries (z± 0.383). An alternative approach to improve defined as quaternions with larger than 10+ misorientation angle
the prediction of crystal orientations at the boundary is discussed with the ground truth. Similar comparison for the test set using the
in the following sub-section. multiple CNN method is shown in Fig. 5(c) and the error versus
deviation of the predicted quaternion norm from unity is shown in
3.2. Outliers near the fundamental zone boundaries Fig. 5(d). For the test set (1000 data points), the original model had
22 outliers and the effective FZ approach had 1 outlier. The results in
Most of the outliers were found to have an extreme value for at Fig. 5 demonstrate significant improvement in orientation pre-
least one of the quaternion components, indicating that the outliers dictions from CNNs trained on multiple fundamental zones as
are close to the cubic fundamental zone (FZ) boundary. Conven- opposed to a single one. Preliminary analysis of the effective FZ
tionally, the minimum distance (rotation) between two crystallo- scheme suggests that three transformations might be adequate for
graphic orientations is given by disorientation, after accounting for improving the neural network prediction of the orientations falling
all symmetry operators for a given crystal structure. The CNN at the fundamental zone boundary. A detailed characterization of
model described above did not explicitly account for cubic sym- the optimal number of effective FZ for complete coverage of the
metry operators; therefore, the model showed poor prediction for boundary orientations is ongoing.
orientations close to FZ boundaries. In such a scenario, the model
performs poorly because physically equivalent crystal orientations 4. Application of CNNs to experiment
produce the same diffraction patterns while the output quaternions
can be very different. This means that if we use the quaternions in a Due to the inherent assumptions in the forward model, differ-
specific fundamental zone (FZ) as the ground truth and the MAE as ences between the simulated and experimental diffraction patterns
the loss function, the boundary points will not be dealt with are expected. Two approaches based on adaptive learning were
properly by the CNN model. utilized to apply the CNN initially trained on the simulated patterns
A rigorous way to deal with boundary points is to use the to experimentally measured EBSD data. 1) Domain transform: This
disorientation as a loss function, which considers all crystallo- method is based on training a separate model that maps the
graphic symmetry operators. Jha et al. [12] accounted for the near experimental patterns to the simulation patterns, so that the
boundary points in their CNN model by implementing an averaged initially trained CNN model could be utilized for orientation
disorientation angle-based loss function. Although this complex reconstruction. 2) Transfer learning: This method used a portion of
loss function can yield improved reconstructions near the FZ the experimental patterns for re-training the parameters of the
boundary, it significantly lengthens the reconstruction time. In our convolutional layers in the CNN model trained on simulated data,
attempt to deal with boundary orientations without compro- while fixing the parameters of the fully connected layers of the
mising the computational costs, we adopted a scheme to transform original network. These two methods are described in more detail
the boundary points in the original FZ in such a way that they are in the following sections. { In U-Net skip connection allows model to retain
important features ( since the image is
no longer at a boundary in the new orientation space referred to as compresses , very high probability to lose
the effective FZ. Three different effective FZs are generated based on 4.1. Domain transform using U-Netimportant features,. Hence model learns which
the three quaternion components (q1, q2, q3). Note that the effec- connection( transformation to skip)

tive FZ does not have a standard property of a fundamental zone Transformation of experimental patterns to simulated patterns
used in crystallography. In addition, the effective FZ is not just a was achieved through the U-Net architecture shown in Fig. 6(a).
new FZ transformed using one of the 24 cubic symmetry operators. The U-Net employs a stack of convolutional layers to encode the
In fact, one of the distinct characteristics of this effective FZ is that it high level information from the input patterns and then uses a
overlaps with the original FZ. To give an example of how one of stack of “deconvolutional” layers that decode the spatial informa-
these effective FZs is generated, we take a specific symmetry tion [31]. The loss of details during the down sampling in the
operator T ¼ [ [1,0,0], [0,0,1], [0,-1,0] ], out of the 24 cubic operators encoder is accounted for by the U-Net through “skip-layer” con-
and apply it to the unit quaternions with 0 < q1 < 0.38, where the nections between the encoder and the decoder. In our U-Net, the
range for q1 component in FZ is approximately 0.38 < q1 < 0.38, first convolutional layer had a 3  3 kernel size, 16 filters, and stride
leaving the rest of the quaternions unchanged. By doing this, we 1. It was followed by two convolutional layers with kernel size
change the range of q1 to approximately 0.76 < q1 < 0 in the 5  5, 32 filters, and stride 2, which effectively downsized the
effective FZ. This is repeated for q2 and q3 generating altogether feature maps by 2. A feature map refers to the spatial information
three effective FZs. For each of the translated points in a given regarding the distinct activation of neurons after applying a
effective FZ, a CNN model is trained. The original CNN performs convolution filter across the previous layer (image). After these
very well on almost all points, but does very poorly at the three convolutional layers, three deconvolutional layers were
boundaries. Each of the additional three CNNs is applied to points applied. The first two deconvolutional layers had kernel size 5  5,
which have been translated so that boundary points have been 32 filters, and stride 2, which expanded the feature maps by a factor
moved into the interior of the effective FZ, where the CNN pre- of 2. The final deconvolutional layer had kernel size 3  3, 1 filter,
dictions are accurate. When performing reconstructions, we run all and stride 1, which generated the output images. The “skip-layer”
four CNNs in parallel on the four sets of data and for each predicted connections concatenated the feature maps from the same level of
orientation keep only the solution whose norm has the smallest encoder and decoder layers, and then used it as the input for the
deviation from unity. A more detailed description of this approach next decoder level. We used mean absolute loss as the objective
is given in Appendix B. { minimum disorientation function, and L2 regularization for the weights to minimize the
To demonstrate the effectiveness of this approach, we trained risks of overfitting.
four CNNs on the same diffraction patterns with quaternion labels Since the simulated patterns were already very similar to the
transformed to different orientation spaces (one FZ and three corresponding experimental patterns obtained from measure-
effective FZs). Fig. 5(a) and (b) show the model prediction at the end ments shown in Fig. 1 (with low noise level), a small training set
of the training process for the single and multiple CNN methods, was adequate to train the U-Net. We randomly picked 1000 pairs of

{ quaternion (four parts: one real part and three imaginary parts) tells you both how much to rotate (real part) and which way to rotate
(imaginary parts) in a clear and concise manner : It represents crystallographic orientation.
q = w +xi +yj +zk, i, j, k are imaginary units
124 Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131

Fig. 5. The predicted quaternion components versus the ground truth for training dataset from (a) a single CNN model and (b) combination of four CNNs. The four models are
trained on four different datasets (same diffraction patterns but different corresponding quaternion components) , and for each diffraction pattern, corresponding quaternion with
the smallest deviation from the unit norm is chosen. Finally, all the predicted quaternions are transformed to the original fundamental zone. (c) The comparison between the
ground truth and the predicted quaternions from the multiple CNNs and (d) the misorientation angle versus deviation from unity of the quaternion norm for the test dataset.

experimental and simulation patterns for training, and another 100 the high level concepts, i.e., the inverse function. We therefore
pairs for testing. The training data and testing data were chosen retrained the convolution layers in the original CNN model using
from different regions of the sample, to ensure no overlap. Fig. 6(b) the experimental patterns, while keeping the weights of the fully
shows one example of an experimental pattern, simulation pattern, connected layers fixed. A similar idea has been applied to cross
U-Net transformed pattern, and the difference between the simu- modal learning [33].
lated and the U-Net transformed patterns (left to right). The lack of
any discernible features in the difference plot suggests that the
differences between the simulated and U-Net transformed images 5. Results and discussion
are very small, which means the U-Net transformed patterns are
appropriate to use as the input for the pre-trained CNN on simu- 5.1. Implementation on experimental data
lated patterns.
Fig. 7 shows the results of applying U-Net and re-trained CNN
{ simply fine tuning the existing methods for EBSD measurements of the 1 mm  1 mm tantalum
4.2. Transfer learning via re-training model on the experimental sample shown in Fig. 1. The orientations were reconstructed from
dataset diffraction patterns using the Hough transform method and were
The second method was inspired by re-training via “fine tuning” used as the ground truth. Fig. 7(a) and (d) display the misorienta-
as used in machine learning [32]. Transfer learning refers to the tion maps between the prediction and the ground truth. The red
idea of exploiting the learned information in one setting to be dots correspond to the sample location from which the diffraction
adapted for another similar setting. That is, given a CNN trained for patterns were taken for re-training the network. Although many
one purpose or dataset, only a few samples are needed to re-train it grains were not covered by the training set, most of the orientation
for another purpose, while keeping some of the weights fixed. In reconstructions for the unseen data were accurate within 10
our case, we had a CNN model that was trained on the simulated misorientation. Note that a low misorientation angle between the
data, and we assumed that the convolutional layers learned the low prediction and the ground truth indicates accurate prediction. High
level features of the images, and the fully connected layers learned misorientation angles are observed for the grain boundary pixels,
 Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131 125

Fig. 6. (a) U-Net architecture. The numbers in parentheses are the dimensions of feature maps, and the numbers on top of each layer corresponds to the number of filters. Dashed
arrows denote the “skip-layer” connections. The measured diffraction pattern is input to the network and the output is the transformed pattern, which resembles the simulated
pattern. (b) An example of transformed image. From left to right is the experiment pattern, simulated pattern, U-Net transformed pattern, and the difference between simulated and
U-Net transformed pattern.
{ transformed experimental pattern
UNet helps to represent experimental data similar to simulated data ( for better performance of pre-trained CNN), which preserving
important features of experimental data by skip connections
for which the Hough transform exhibits low fidelity re- 5.2. Robustness to pattern noise
constructions. Fig. 7(b) and (e) show the spatial distributions of the
deviation of the predicted quaternion norm from unity. The To study the robustness of model prediction to measurement
misorientation and quaternion norm plots indicate that a large noise, we used the same area scan of the Ta sample with different
misorientation corresponds to a quaternion norm that deviates exposure times, example orientation maps are shown in Fig. 2. The
away from the ideal value of 1. Therefore, the norm of the output orientations from diffraction patterns acquired with 30 ms expo-
quaternion provides a metric for quantifying the prediction error. sure were reconstructed using the Hough transform method and
This is very important for real applications, where ground truth used as the ground truth. In total, 51  51 ¼ 2601 pairs of orienta-
information is not available. The same training and testing sets tions and diffraction pairs were measured, where 2000 pairs were
were used for both domain transform and transfer learning used as a training set and the remaining 601 pairs were used for
methods. The testing error distributions are shown in Fig. 7(c) and testing. FusionNet [14] architecture was utilized for orientation
(f) from the U-Net and re-trained models, respectively. A slightly reconstruction. FusionNet, like U-Net, can learn the transformation
larger spread in misorientation distributions was observed for the from the real experimental patterns to the simulated patterns,
experimental images in comparison to the misorientation distri- while demonstrating higher accuracy, especially, for images with
butions predicted for simulation patterns, as seen in Fig. 4. Also, high noise levels.
during a large area scan, any changes in the pattern center can Fig. 8(a) shows the results of applying FusionNets to experi-
introduce further error in prediction. A correction for the orienta- mental diffraction patterns of the same orientation collected at
tion error induced by the pattern center shift across a large area various exposure times. The top row is the experimentally
EBSD scan can be applied using the method described in Singh et al. measured patterns and the bottom row corresponds to the
[34]. While we do not use this correction factor here, as we are FusionNet transformed patterns. The measured pattern quality
primarily interested in demonstrating that large reconstruction significantly degrades with decreasing exposure times. For shorter
errors can be reduced through re-training (shown in Fig. 7), ac- exposures, the reconstructed patterns are more deformed, but
counting for the pattern center shift as a function of scan position many features are retained in the transformed image. Fig. 8(b)
may further reduce the low misorientations shown in Fig. 7(f). shows the simulated pattern corresponding to the same orientation
 126 Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131

Fig. 7. (a), (d) The misorientation map between the predicted and the ground truth crystal orientations. The red points correspond to the positions from which measured diffraction
pattern and crystal orientation pairs were used for training the U-Net and the re-trained networks. (c), (f) Misorientation distribution calculated for the 100 measured patterns,
randomly chosen from the right most side of the sample (unseen during training), used as a testing set for the two models. The measured diffraction patterns from the remaining Ta
sample positions were used for orientation prediction. Low misorientation angle indicates good agreement. (b), (e) Spatial map for the deviation from unity of the quaternion norm.
Larger deviation indicates higher prediction error. (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

{ Robustness of model to pattern noise

{ experimental pattern

{ Reconstructed pattern
by FusionNet

Fig. 8. (a) EBSD detector patterns. The first row shows the measured diffraction images of the same orientation collected at various exposure times. The second row shows the
FusionNet transformed images that were used as the input for the CNN model for orientation indexing. (b) The simulated pattern corresponding to the same orientation in (a). (c)
The distributions of misorientations between the predicted and the ground truth crystal orientations of the 601 test dataset collected at various exposures. (d) Average misori-
entation angles for data points with less than 10+ misorientation versus the exposure times for CNN and Hough transform analyses. The error bar is the standard deviation obtained
from the misorientation distribution. Both methods exhibit a decrease in accuracy with increasing noise level.
 Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131 127

in Fig. 8(a). Misorientation distributions between the measured and pixels (pixel size is 71.4 mm). This was expected because detector
the predicted orientations for the entire scanned area are shown in translation and crystal rotation have approximately the same effect
Fig. 8(c). The results demonstrate that the CNN approach can, to a on the EBSD pattern. Therefore, similar to Hough transform and
degree, learn to index very low-quality diffraction images. dictionary methods, the proposed CNN based approach also re-
A comparison between the CNN and Hough transform recon- quires determination of the experimental setup parameters before
struction accuracy is shown in Fig. 8(d). Here, we show what per- reconstruction. Therefore, a single neural network will not provide
centage of points have less than 10+ misorientations with the accurate predictions for all experimental geometry parameters;
ground truth, as shown by the gray bars. Note that the CNN however, it might be possible to account for the change in instru-
consistently achieved a higher percentage of such points for all ment parameters via re-training the weights of the previously
exposure times. The difference is especially large for 1 ms exposure trained CNN model. { on this new dataset
times where the CNN approach resulted with  50% of all points
having < 10+ misorientations, while the Hough transform method 5.4. Computational cost
only had  15% of the points with < 10+ misorientation. Next, us-
ing only these < 10+ misorientation points, we calculated the Compared to a dictionary approach, one advantage of using
average misorientation and the standard deviation, as shown in neural networks is increased computational performance.
blue. For this choice of points, the Hough transform method has Reducing computation time is an important consideration in
lower average misorientation and lower variance. Overall, the CNN designing a network architecture. As we can see from Equation
approach is more robust to noise because a larger number of points (A.1), the computation time for each pixel in the convolutional layer
have overall lower predicted misorientations, while the Hough is proportional to the kernel size and the number of filters, which is
transform method is more accurate but only for a very small frac- related to the number of computations performed by the network.
tion of points, suggesting that only points with low noise were Thus, reducing the number of layers and keeping the kernel size
accurately reconstructed. and filter numbers managable for the convolutional layers will
Recent studies have also demonstrated CNNs’ robustness to increase the performance. However, a CNNs’ ability to approximate
noisy data [35,36]. This is not surprising because convolutional complex functions depends on the network size [41,42], and the
layers have been shown to be capable of learning the features from prediction power will decrease with decreasing network size.
the training data to recover a clean image from a noisy observation The network architectures of the CNN and U-Net used in this
[37e39]. Moreover, CNNs can learn high level features automati- work were optimized for both computational performance and
cally and are very robust to local noise, which makes them prediction accuracy, and we demonstrated that the models gener-
outperform traditional methods based on handcrafted features, alized well for both unseen simulated and experimental datasets.
which involve pre-determining characteristic regions or de- Using the network structure described here, the U-Net method
scriptors of an image, utilized in many computer vision tasks took about 17 min to reconstruct the 1  1 mm2 scanned region
[15e17,40]. (251001 data points) on the sample (i.e. 4 ms per pattern) on a
MacBook Pro with a 2.8 GHz Intel Core i7 processor, and the
5.3. Effects of uncertainty in experimental parameters retrained CNN took about 5 min (i.e. 1 ms per pattern). Further-
more, there are many neural network frameworks (e.g. tensorflow
Another challenge always encountered in experiments is the [43]) that can utilize GPUs to further speed up the reconstruction
uncertainty in experimental setup parameters. We conducted a without changing the code.
parametric study to investigate the effect of instrument setup un-
certainty on model prediction. For that purpose, we trained our 5.5. Combining CNN with the dictionary approach
model with simulated images generated from one set of detector
parameters, and tested it on the images that were generated using Both Liu et al. [11] and Jha et al. [12] reported the dictionary
different setup parameters. The results are shown in Fig. 9. In indexing approach to be slower than the CNN method. In this work,
Fig. 9(a), we only changed the distance between the detector and we have demonstrated that the CNN-based method is relatively fast
the sample, keeping everything else fixed. As long as the distance and can enable real-time reconstructions during the scanning
offset was smaller than 900 mm, the results were not strongly process. However, the CNN method is not as accurate as the dic-
influenced. Fig. 9(b) and (c) show the effects of X and Y translation, tionary approach, which can reconstruct noisy data with high ac-
which caused the error to shift towards the right suggesting poor curacy [10]. Further improvements may be made by combining the
prediction. The extent of this shifting was dependent on the choice two approaches: first apply CNN-based regression to obtain pre-
of the experimental setup, and in our case, it was about 4+ per 20 liminary predictions and a measure of confidence for each location

{ poor prediction for change in parameters i.e. X and Y

{ no effect translation

Fig. 9. The misorientation distributions when the training and testing patterns are generated under different detector parameter setups. (a) The sample to detector distance is
different for training and testing data, where the offset unit is mm. (b) The pattern center X coordinate on detector are different for training and testing data. (c) The pattern center Y
coordinate on detector are different for training and testing data. The offset units for X and Y are given in number of pixels (pixel size is 71.4 mm).
 128 Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131

given by the norm of the predicted quaternion. For locations with methods improved the accuracy of the experimental data recon-
high confidence reconstructions, run the pattern matching only in struction. This improvement can be understood by looking at the
the neighborhood (e.g. 10+) of the preliminary results in the dic- activation map of the last convolutional layer, which after max
tionary. For those with low confidence, run the full dictionary pooling is the input to the fully connected layer.
search. Fig. 10 shows one such example from the test dataset. Four
feature maps are shown, one for each of the four filters in the layer.
5.6. Regularization The first row is the simulation image and the filter outputs of the
original CNN. We can see that some features related to Kikuchi
Regularization is important for reducing the chance of over- bands have been learned. The second shows the results of inputing
fitting. We used L2 regularizations for our models without using experimental data into the CNN model which has been trained on
dropout [44]. Dropout is a common regularization technique used simulations only. Clearly the original CNN will respond differently
in most CNN classification problems. It randomly forces the neuron to experimental and simulation patterns. The difference was sig-
activations to be zero in training, but does nothing in testing. Pre- nificant enough that the original CNN was unable to accurately
vious studies have shown that dropout can improve the perfor- predict quaternions for experimental patterns.
mance of a particular CNN-based regression model. For example, The third row shows the filter outputs after a U-Net transformed
Kendall et. al. [45] used dropout in their model to infer camera image is fed into the original CNN; the activation maps look very
positions from a picture taken by that camera, and saw improved similar to those shown in the first row, which were generated from
predictions. We tried two dropout ratios (0:3; 0:5) in our CNN the simulated image. In the fourth row, we retrained the CNN so
models, but they resulted in lower fidelity predictions. However, that the weights of convolutional layers were adapted to the
without dropout, we obtained validation errors that were already experimental images. We can see that the activation maps are also
as low as the training error, as seen in Fig. 4(a). The reconstruction very similar to those in the first row, which supports our assump-
accuracy (mean error and variance of error) of our model for un- tion that differences between simulation and experimental pat-
seen data and for training data was similar, indicating that over- terns are low level features and they can be learned solely by
fitting was not taking place. convolutional layers.
The two approaches presented here for experimental pattern
5.7. Microstructure reconstruction indexing are simple and robust. Similar studies have being con-
ducted on transfer learning and domain transforms [46]. One such
Experimental EBSD data reconstruction was performed by example is a generative adversarial networks (GAN) [47] that is
employing two adaptive learning networks. One of the reasons for utilized to transform images from one domain to another [48,49].
not training another CNN model from scratch was due to the lack of Essentially, the generative networks are learning the metric for
labeled experimental patterns that spanned the entire orientation measuring similarity between images produced from different mo-
space (fundamental zone) for a given material crystal structure. We dalities, which is similar to our approach utilizing the U-Net and
observed that both the domain transform and transfer learning FusionNet to transform experimental patterns to simulated patterns.

{ simulated

{ There feature maps

are different

{ experimental

{ U-net transformed experimental

feature maps

{ Tf Feature map obtained

by re-training by TL

Fig. 10. Activation maps for the last convolutional layer. Case 1: simulated pattern as input to the original CNN. Case 2: experimental pattern as an input to the original CNN. Case 3:
U-Net transformed pattern as an input to the original CNN. Case 4: experimental pattern as input to the re-trained CNN.
 Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131 129

6. Conclusions

This work demonstrates the effectiveness of two CNN ap- X

s X
s X

proaches for microstructure reconstruction from measured EBSD hlði;jÞ;c ¼ wðm;nÞ;c;c0 al1
ðim;jnÞ;c þ bc;c
0 (A.1)
m¼s n¼s c0
patterns. Both approaches initially trained the CNN with a large
number of easily generated simulation data and then utilized a which is followed by the activation function
relatively small number of expensive experimentally measured
patterns to either re-train several convolutional layers or transform
experimental patterns into simulated patterns. Both methods were   n o
applied to a tantalum sample with a body centered cubic crystal alði;jÞ;c ¼ ReLU hlði;jÞ;c ¼ max 0; hlði;jÞ;c ;
structure. Noise in the patterns was one of the major contributors
that impacted the model prediction significantly. The proposed where hlði;jÞ;c and alði;jÞ;c are the feature map intensity and activation
CNNs demonstrated robustness to varying levels of noise in real value at ði; jÞ pixel coordinates, s is the kernel size and c is the
experimentally measured data (achieved by changing exposure channel index in layer l. The wðm;nÞ;c0 ;c are the convolutional kernel
times during data collection) and successfully reconstructed the
weights between channel c and channel c0 of neighboring layers
orientations while matching the highest EBSD scanning rates. The
and bc;c0 is the bias term.
proposed quaternion norm as a measure of confidence was
We used a 3  3 kernel for all convolutional layers with s ¼ 1,
demonstrated to be a strong indicator for assessing the reliability of
the range of m and n was m; n2f  1; 0; 1g, and zero padding was
the reconstructions. This will be crucial for determining the quality
employed. The filter numbers (i.e. number of channels) were 4; 8; 8
of reconstructions for real applications, when the ground truth is
in order. After these three convolutional layers, we applied a max
not available.
pooling layer with a pooling size of 2 and stride of 2, in which all the
The CNN based orientation indexing methods presented here, if
feature maps were divided into 2  2 non-overlapping units and
combined with the dictionary search method, has the potential to
every unit was assigned an intensity equal to the maximum pixel
accelerate the reconstruction and improve the reconstruction
intensity inside of that unit. The outputs of these layers were new
quality, beyond what is currently possible. Rigorous studies of
feature maps half the size of the input feature maps.
hyper-parameters and network architecture optimization along
Next, three convolutional layers and one max pooling layer
with implementation of physical loss functions can further reduce
were applied again, the number of filters for these three con-
the computation cost and minimize prediction outliers. The neural
volutional layers were 8; 8; 4 in order. After that, the feature maps
networks developed here is generalizable to a certain extent for
were flattened into a 1D vector and two fully connected layers
diffraction patterns recorded for the same material and with rela-
were applied with each one followed by a ReLUðxÞ activation
tively similar instrument parameters. In scenarios where exposure
function. 512 neurons were used in the fully connected layers. The
times are different, then the original CNN network does not need to
output layer was similar to the fully connected layer without
be retrained but the U-Net/Fusion-Net has to be retrained using a
applying the activation function. The four output neurons corre-
small subset of experimental images (e.g. 1000 patterns), which
sponded to the four quaternion components representing the
takes about 20 min in one GPU. On the other hand, if the detector
crystal orientation.
geometry is different, then the original CNN has to be retrained
When fitting a complex model to a small dataset, there is a high
(taking about 1 h using one GPU), while the original Fusion-Net/U-
chance of overfitting. An overfitted model can describe well the
Net can be used as is for transforming experimental images to
particular dataset that it is trained on, but fails with a comparable
simulated images. However, if the material and crystal structure are
level of prediction on unseen data. This lack of generalizability of
different, then both the CNN and the Fusion-Net/U-Net have to be
the trained model can be alleviated to some extent through regu-
retrained. Future work will explore, in detail, the generalizability of
larization. We used the mean absolute loss as the objective func-
these networks to differing instrument geometries for a given
tion, and L2 regularization for the weights of convolutional layers to
material system as well as different materials with same crystal
prevent overfitting. L2 regularization adds the sum of square of
structure. Finally, our study demonstrated the effectiveness of the
weights to the loss function, so that large weights are penalized,
CNN-based reconstruction method for EBSD measurements, and
effectively making the model “smoother”. The total loss function is
this general approach has the potential to be useful for many other
given by:
image-based microstructure reconstruction techniques.
Loss function in CNN
Acknowledgements b ðXÞ  qk1 þ lkWk2
L ¼ kg

This work was supported by the U.S. Department of Energy where g b ðXÞ represents the feed forward processing of our CNN
through the Los Alamos National Laboratory. Los Alamos National model and l is the penalty weight (in our model l ¼ 0:0001). The
Laboratory is operated by Triad National Security, LLC, for the Na- Adaptive Moment Estimation (Adam) algorithm [50], a variant of
tional Nuclear Security Administration of U.S. Department of En- stochastic gradient descent (SGD), was used as an optimizer to train
ergy (Contract No. 89233218CNA000001), This work was our model, finding the weights W and biases b that minimized the
supported by Los Alamos National Laboratory’s Momentum Labo- total loss on the training set consisting of ðXi ; qi Þ pairs. An
ratory Directed Research and Development (LDRD) Reserve Project approximate smooth function b g ðXi Þzqi was determined, which
#20180677ER and LDRD-ECR project #20190571ECR. well approximated the inverse function b g zg for orientation
reconstruction from diffraction pattern. Representing crystal ori-
entations as quaternions afforded a good way of estimating the
Appendix A. Convolutional neural network “confidence” metric for model predictions. Any deviation of the
norm of a quaternion from one indicated the inaccuracy in model
A convolutional layer can be written as: prediction.
130 Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131

Appendix B. Outliers near fundamental zone boundaries correspond to a unique orientation. Simply applying a cubic sym-
metry operator to every point in the original FZ was not adequate as
Let us first consider a simple one-dimensional case. Assuming all it would result in a new FZ that would not overlap with the original
crystal orientations are along the same axis and there is no crys- FZ. Instead, we choose a specific symmetry operator that translates
tallographic symmetry, the neural networks are trying to find the all the points from the original FZ to an adjacent FZ. Now if we apply
map from a diffraction pattern to a single angle, q. We can constrain this operator only to half of the points, then this half will translate
q 2 [0, 2p), which is effectively a “fundamental zone”. In this case, a to the neighboring FZ, while the remaining half will stay in the
neural network that simply uses a mean absolute error (MAE) as a original FZ. This new orientation space is what we call the effecti-
loss function will result in a poor prediction for the points close to ve FZ and it satisfies both the required conditions of the trans-
the fundamental zone boundaries. This is because the diffraction formed orientation space.
patterns from q ¼ 0:001 and q ¼ 1:999p are very similar, but during

Fig. 11. (a) Original and (b) transformed quaternions. The transformation was applied to all quaternions with q1 > 0.

the training process the neural network is forced to predict very Three effective FZ were generated based on q1 , q2 , and q3 values
different angle values for these diffraction patterns. Therefore, using in the original FZ. For the quaternions with q1 > 0, the trans-
MAE as a loss function, the neural network cannot be trained very formation matrix T ¼ ½½1; 0; 0; ½0; 0; 1; ½0; 1; 0 (one of the 24 cu-
well for points at the boundaries. One solution to this is to train a bic symmetry operators) was applied to their rotation matrix. The
separate neural network using translated data that map the original transformed quaternions were then used as the training label to
diffraction patterns to q 2 [p, 3p). [p, 3p) is a new effective FZ and it train a new CNN. In this effective FZ, the quaternions with
overlaps with [0, 2p), and the two do not have the same boundaries. q1  0:38 in the original FZ are no longer at the boundary, thus an
This new neural network performs poorly at angles near q ¼ p, but it improved CNN prediction was obtained for those quaternions close
performs well on angles near q ¼ 0. By combining these two neural to the boundary in the original FZ. This is schematically shown in
networks, we can predict every angle very well. Figure 11. Figure 11(a) shows the 2D projection of the points in the
Similarly, during the CNN training process for the EBSD data, the original FZ, the points shown in red are at the boundary and the
neural network performed poorly on boundary points, due to lack neural net would predict poorly for such points. Figure 11(b) shows
of knowledge about crystallographic symmetries. To address this the first effective FZ. The yellow dots are the new positions of the
problem, an alternative approach was developed that entailed transformed quaternions. The blue dots are the quaternions that
training three additional CNNs with transformed orientations, remain in the original FZ. The red dots are the new positions of the
while using the MAE loss function. The transformed orientation former outliers (boundary orientations). The original boundary
space needed to satisfy two conditions: 1) overlap the original FZ points and the corresponding transformed quaternion components
space and 2) every point in the orientation space should are shown in Table 1.

Table 1
Quaternions with transformation examples.

Original q ¼ ½q0 ; q1 ; q2 ; q3  Transformed q ¼ ½q0 ; q1 ; q2 ; q3 

0:8872043; 0:3644020;  0:0392778; 0:2802443 0:8850193;  0:3696770; 0:1703890; 0:2259362

0:9002409; 0:3707965; 0:2094063;  0:0906927 0:8987592;  0:3743737; 0:0839432;  0:2122021
0:8619023; 0:3281171;  0:3318592; 0:1983255 0:8414708;  0:3774431;  0:0944225; 0:3748973
0:8804877; 0:3594195; 0:2575050;  0:1710263 0:8767468;  0:3684508; 0:0611496;  0:3030175
 Y.-F. Shen et al. / Acta Materialia 170 (2019) 118e131 131

The other two effective FZs are also generated in a similar way, Graef, A.O. Hero, A dictionary approach to electron backscatter diffraction
indexing, Microsc. Microanal. 21 (2015) 739752.
where we apply two specific symmetry operators to half of the
[24] P.G. Callahan, M. De Graef, Dynamical electron backscatter diffraction pat-
original FZ, and move them to the neighboring FZs. For quaternions terns. part i: pattern simulations, Microsc. Microanal. 19 (2013) 1255e1265.
with component q2 > 0, we applied the transform [25] M. De Graef, Emsoft 3.0, 2015. Software available from: http://muri.materials.
T ¼ ½½0; 0; 1; ½0; 1; 0; ½1; 0; 0 and for q3 > 0, we applied T ¼ cmu.edu/?p¼858/.
[26] G. Cybenko, Approximation by superpositions of a sigmoidal function, Math.
½½0; 1; 0; ½  1; 0; 0; ½0; 0; 1. The outlier quaternions, usually at the Contr., Signals Syst. 2 (1989) 303e314.
boundary of the original FZ, is no longer at the boundaries in at least [27] K. Hornik, M. Stinchcombe, H. White, Multilayer feedforward networks are
one of these three effective FZ. universal approximators, Neural Network. 2 (1989) 359e366.
[28] Y. LeCun, Y. Bengio, et al., Convolutional networks for images, speech, and
time series, Handb Brain Theory Neural Netw. 3361 (1995) 1995.
References [29] X. Glorot, A. Bordes, Y. Bengio, Deep Sparse Rectifier Neural Networks, vol. 15,
2011, pp. 315e323.
[1] D.L. McDowell, A perspective on trends in multiscale plasticity, Int. J. Plast. 26 [30] W. Dubitzky, M. Granzow, D.P. Berrar, Fundamentals of Data Mining in Ge-
(2010) 1280e1309. nomics and Proteomics, Springer Science & Business Media, 2007.
[2] R. Suter, Multiscale measurements for materials modeling, Science 356 (2017) [31] M.D. Zeiler, D. Krishnan, G.W. Taylor, R. Fergus, Deconvolutional Networks,
704e705. 2010.
[3] A.J. Wilkinson, T.B. Britton, Strains, planes, and ebsd in materials science, [32] I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016. Book
Mater. Today 15 (2012) 366e376. available from: http://www.deeplearningbook.org/.
[4] A.E. Gleason, C.A. Bolme, E. Galtier, H.J. Lee, E. Granados, D.H. Dolan, [33] L. Castrejon, Y. Aytar, C. Vondrick, H. Pirsiavash, A. Torralba, Learning aligned
C.T. Seagle, T. Ao, S. Ali, A. Lazicki, D. Swift, P. Celliers, W.L. Mao, Compression cross-modal representations from weakly aligned data, in: Proceedings of the
freezing kinetics of water to ice vii, Phys. Rev. Lett. 119 (2017), 025701. IEEE Conference on Computer Vision and Pattern Recognition, pp.
[5] R. Pokharel, Overview of high-energy x-ray diffraction microscopy (hedm) for 2940e2949.
mesoscale material characterization in three-dimensions, in: Materials Dis- [34] S. Singh, F. Ram, M. De Graef, Application of forward models to crystal
covery and Design, Springer, 2018, pp. 167e201. orientation refinement, J. Appl. Crystallogr. 50 (2017) 1664e1676.
[6] R.A. Schwarzer, D.P. Field, B.L. Adams, M. Kumar, A.J. Schwartz, Present state of [35] T.S. Nazare , G.B.P. da Costa, W.A. Contato, M. Ponti, Deep convolutional neural
electron backscatter diffraction and prospective developments, in: Electron networks and noisy images, in: Iberoamerican Congress on Pattern Recogni-
Backscatter Diffraction in Materials Science, Springer, 2009, pp. 1e20. tion, Springer, 2017, pp. 416e424.
[7] S.I. Wright, M.M. Nowell, A review of in situ ebsd studies, in: Electron Back- [36] S. Diamond, V. Sitzmann, S. Boyd, G. Wetzstein, F. Heide, Dirty Pixels: Opti-
scatter Diffraction in Materials Science, Springer, 2009, pp. 329e337. mizing Image Classification Architectures for Raw Sensor Data, 2017. 1701.
[8] T. Britton, J. Jiang, Y. Guo, A. Vilalta-Clemente, D. Wallis, L. Hansen, 06487.
A. Winkelmann, A. Wilkinson, Tutorial: crystal orientations and ebsd or which [37] V. Jain, S. Seung, Natural image denoising with convolutional networks, in:
way is up? Mater. Char. 117 (2016) 113e126. Advances in Neural Information Processing Systems, 2009, pp. 769e776.
[9] S.I. Wright, M.M. Nowell, S.P. Lindeman, P.P. Camus, M. De Graef, M.A. Jackson, [38] K. Zhang, W. Zuo, Y. Chen, D. Meng, L. Zhang, Beyond a Gaussian denoiser:
Introduction and comparison of new ebsd post-processing methodologies, residual learning of deep cnn for image denoising, IEEE Trans. Image Process.
Ultramicroscopy 159 (2015) 81e94. 26 (2017) 3142e3155.
[10] S. Singh, Y. Guo, B. Winiarski, T.L. Burnett, P.J. Withers, M. De Graef, High [39] H.C. Burger, C.J. Schuler, S. Harmeling, Image denoising: can plain neural
resolution low kv ebsd of heavily deformed and nanocrystalline aluminium by networks compete with bm3d?, in: Computer Vision and Pattern Recognition
dictionary-based indexing, Sci. Rep. 8 (2018). (CVPR), 2012 IEEE Conference on IEEE, 2012, pp. 2392e2399.
[11] R. Liu, A. Agrawal, W. Liao, A. Choudhary, M.D. Graef, Materials discovery: [40] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in:
understanding polycrystals from large-scale electron patterns, in: 2016 IEEE Proceedings of the IEEE Conference on Computer Vision and Pattern Recog-
International Conference on Big Data (Big Data), 2016, pp. 2261e2269. nition, 2016, pp. 770e778.
[12] D. Jha, S. Singh, R. Al-Bahrani, W.-k. Liao, A. Choudhary, M. De Graef, [41] B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, S. Ganguli, Exponential ex-
A. Agrawal, Extracting grain orientations from ebsd patterns of polycrystalline pressivity in deep neural networks through transient chaos, in: Advances in
materials using convolutional neural networks, Microsc. Microanal. 24 (2018) Neural Information Processing Systems, 2016, pp. 3360e3368.
497502. [42] H. Mhaskar, Q. Liao, T. A. Poggio, When and Why Are Deep Networks Better
[13] A. Fakhry, T. Zeng, S. Ji, Residual deconvolutional networks for brain electron than Shallow Ones?, in: AAAI, pp. 2343e2349.
microscopy image segmentation, IEEE Trans. Med. Imaging 36 (2017) [43] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G.S. Corrado,
447e456. A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving,
[14] T.M. Quan, D.G.C. Hildebrand, W.-K. Jeong, Fusionnet: A Deep Fully Residual M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mane ,
Convolutional Neural Network for Image Segmentation in Connectomics, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner,
CoRR Abs/1612, 2016, 05360. I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Vie gas,
[15] B. Zhu, J.Z. Liu, S.F. Cauley, B.R. Rosen, M.S. Rosen, Image reconstruction by O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, X. Zheng, TensorFlow:
domain-transform manifold learning, Nature 555 (2018) 487. Large-Scale Machine Learning on Heterogeneous Systems, 2015. Software
[16] A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep available from: tensorflow.org.
convolutional neural networks, in: Advances in Neural Information Processing [44] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov,
Systems, 2012, pp. 1097e1105. Dropout: a simple way to prevent neural networks from overfitting, J. Mach.
[17] S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: towards real-time object Learn. Res. 15 (2014) 1929e1958.
detection with region proposal networks, in: Advances in Neural Information [45] A. Kendall, R. Cipolla, Modelling Uncertainty in Deep Learning for Camera
Processing Systems, 2015, pp. 91e99. Relocalization, 2015. 1509.05909.
[18] A. Kendall, M. Grimes, R. Cipolla, Posenet: a convolutional network for real- [46] S.J. Pan, Q. Yang, et al., A survey on transfer learning, IEEE Trans. Knowl. Data
time 6-dof camera relocalization, in: Proceedings of the IEEE International Eng. 22 (2010) 1345e1359.
Conference on Computer Vision, 2015, pp. 2938e2946. [47] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair,
[19] T. Shimobaba, T. Kakue, T. Ito, Convolutional neural network-based regression A. Courville, Y. Bengio, Generative adversarial nets, in: Advances in Neural
for depth prediction in digital holography, in: IEEE 27th International Sym- Information Processing Systems, 2014, pp. 2672e2680.
posium on Industrial Electronics (ISIE), IEEE, 2018, pp. 1323e1326. [48] P. Isola, J.-Y. Zhu, T. Zhou, A.A. Efros, Image-to-image translation with con-
[20] M.J. Cherukara, Y.S. Nashed, R.J. Harder, Real-time coherent diffraction ditional adversarial networks, in: Computer Vision and Pattern Recognition
inversion using deep generative networks, Sci. Rep. 8 (2018) 16520. (CVPR), 2017 IEEE Conference on, 2017.
[21] X. Tao, A. Eades, Errors, artifacts, and improvements in ebsd processing and [49] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A.A. Efros, T. Darrell,
mapping, Microsc. Microanal. 11 (2005) 79e87. Cycada: Cycle-Consistent Adversarial Domain Adaptation, 2017. 1711.03213.
[22] S.I. Wright, M.M. Nowell, Ebsd image quality mapping, Microsc. Microanal. 12 [50] D.P. Kingma, J. Ba, Adam, A Method for Stochastic Optimization, 2014. 1412.
(2006) 72e84. 6980.
[23] Y.H. Chen, S.U. Park, D. Wei, G. Newstadt, M.A. Jackson, J.P. Simmons, M. De