DDE-GAN: Integrating a Data-

Driven Design Evaluator Into

Chenxi Yuan
Department of Mechanical and Industrial
Generative Adversarial Networks
Engineering,
Northeastern University,
for Desirable and Diverse
Boston, MA 02115
Concept Generation

Downloaded from http://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/145/4/041407/6973255/md_145_4_041407.pdf by Yanshan University user on 16 June 2023

e-mail: [email protected]

Tucker Marion Generative adversarial networks (GANs) have shown remarkable success in various gener-
D’Amore-McKim School of Business, ative design tasks, from topology optimization to material design, and shape parametriza-
Department of Industrial Engineering, tion. However, most generative design approaches based on GANs lack evaluation
Northeastern University, mechanisms to ensure the generation of diverse samples. In addition, no GAN-based gen-
Boston, MA 02115 erative design model incorporates user sentiments in the loss function to generate samples
e-mail: [email protected] with high desirability from the aggregate perspectives of users. Motivated by these knowl-
edge gaps, this paper builds and validates a novel GAN-based generative design model with
Mohsen Moghaddam1 an offline design evaluation function to generate samples that are not only realistic but also
Department of Mechanical and diverse and desirable. A multimodal data-driven design evaluation (DDE) model is devel-
Industrial Engineering, oped to guide the generative process by automatically predicting user sentiments for the
Khoury College of Computer Sciences, generated samples based on large-scale user reviews of previous designs. This paper incor-
Northeastern University, porates DDE into the StyleGAN structure, a state-of-the-art GAN model, to enable data-
Boston, MA 02115 driven generative processes that are innovative and user-centered. The results of experi-
e-mail: [email protected] ments conducted on a large dataset of footwear products demonstrate the effectiveness of
the proposed DDE-GAN in generating high-quality, diverse, and desirable concepts.
[DOI: 10.1115/1.4056500]

Keywords: generative design, generative adversarial networks, design evaluation,

desirability, design diversity, user-centered design, artificial intelligence, data-driven
design, design automation

1 Introduction studying methods and tools to improve the effectiveness and effi-
ciency of creative tasks, such as concept development [4,15–17].
The generation of innovative, diverse, and user-centered design
Creativity is an essential and central part of the ideation process
concepts is an essential phase in the early stages of the product
[18]. In human-led design practices, ideation is often an iterative
development process and is known to have a significant impact
and exploratory process [19], where designers share, modify, and
on the quality and success of the design [1–4]. Creating a wide
use various stimuli to generate new ideas and concepts [20].
range of solutions that differ significantly from each other can
Humans approach this process through various cognitive processes,
benefit the ideation process of designers and therefore increase
which research has classified into types and has been shown to
the possibility of creating high-quality concepts [5–8]. Various
affect the effectiveness of ideation [21]. Over the past 25 years,
approaches in the literature focus on automatically developing
research on computers and artificial intelligence (AI) has increas-
diverse and innovative concepts. The argument is that a large set
ingly focused on how these systems can be used to enhance the cre-
of concepts promote creativity and logically allows the selection
ative ideation process [22,23]. With its ability to synthesize data and
of better ideas from the set [5,9]. However, it is difficult for design-
make predictions at great speed, the potential for AI to be a gener-
ers to manually generate a large set of samples with great diversity
ator of new and creative design ideas and concepts has garnered
and novelty because designers naturally tend to fixate on specific
substantial attention from both academia and industry [16,19].
design specifications [10–12]. Moreover, most existing design
The methods and frameworks used to apply AI and machine
problem-solving practices rely heavily on the designers’ experi-
ences and preferences. They lack advanced computing methods to learning in design and engineering are numerous. Deep learning
help navigate larger solution spaces by generating more diverse, and generative modeling have recently attracted researchers’ atten-
unexpected, and viable solutions [5,11,13,14]. tion for their potential impact. Recent advances in AI research have
Developing methods to assess and improve creativity has histor- made remarkable progress in the machine’s ability to generate
ically been challenging due to its intangible and subjective nature. design ideas [24]. AI can be an inspiration tool in the creative
Significant research in engineering design is currently focused on process and a generative tool to assist designers in developing
design concepts. AI-powered generative design tools can poten-
tially augment designers’ ability to create concepts faster and
1
Corresponding author. more efficiently due to their increased speed and efficiency. The
Contributed by the Design Theory and Methodology Committee of ASME for power of AI lies in the speed with which it can analyze large
publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received July 12,
2022; final manuscript received December 5, 2022; published online January 10, amounts of data and suggest design adjustments. The designer
2023. Assoc. Editor: Christopher Mccomb. can then choose and approve adjustments based on these data.

Journal of Mechanical Design Copyright © 2023 by ASME APRIL 2023, Vol. 145 / 041407-1
 An emerging research area on using AI to generate novel and design concept with descriptive phrases that can automatically
realistic design concepts is the use of generative adversarial net- convey a novel design concept remains a challenge. This work
works, or GANs [25]. A typical GAN architecture comprises two merely deploys the pretrained ResNet network of the DDE model
neural network architectures: a generator and a discriminator. The [23] to examine and evaluate the visual samples generated. The
generator neural network is trained to generate samples (e.g., DDE model, which excludes inputs from the product description,
images) almost identical to real samples. On the other hand, the dis- was incorporated into the architecture of the DDE-GAN model pre-
criminator neural network learns to differentiate between them. sented in Sec. 3. Future research should focus on building a multi-
GANs have made significant progress in synthesizing and gene- modal DDE-GAN model that couples images and descriptions for
rating “realistic” images as their central objective. Several success- automated generation and evaluation of design concepts.
ful GAN architectures have recently been proposed, mainly for The remainder of this paper is organized as follows. Section 2
synthesizing and generating facial images. Examples include provides a detailed overview of related work and topics in tradi-
CycleGAN [26], StyleGAN [27], PixelRNN [28], Text2Image [29], tional and GAN-based generative design. Section 3 provides the
and DiscoGAN [30]. These powerful image synthesis models can details of the proposed DDE-GAN model. Section 4 presents the

Downloaded from http://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/145/4/041407/6973255/md_145_4_041407.pdf by Yanshan University user on 16 June 2023

generate a large number of high-resolution images that are often dif- experimental results, analyses, and performance evaluation.
ficult to distinguish from authentic images without close inspection. Section 5 provides concluding remarks and directions for future
Nevertheless, the question remains on leveraging these models in research.
early-stage product design to generate realistic but also novel and
diverse concepts. Several technical limitations restrict the ability of
GANs to generate diverse and novel designs. These include 2 Related Work
network architectures, training issues, and a lack of reward mecha-
This section provides an introductory overview of GANs and
nisms to generate outputs that satisfy metrics other than realism,
their advantages and limitations for generative design, followed
such as diversity, novelty, or desirability. Taken together, these rep-
by a review of five main traditional generative design approaches
resent an impediment to design, where novelty and diversity are crit-
and their comparison with GANs.
ical factors in producing beneficial outcomes [31].
This paper presents a data-driven generative design model that
integrates a data-driven design evaluator (DDE) [23] into GANs, 2.1 Generative Adversarial Networks for Generative
called the DDE-GAN model, to improve the performance of Design: Advantages and Limitations. Deep generative modeling
GANs through large-scale user feedback on previous designs for is one of the most promising areas of modern AI studied within the
diverse and desirable generative design. The main contributions engineering design community to enhance diversity and perfor-
of this paper are as follows: mance. One way of design exploration is through generative
design, which involves programming that alters design geometry
(1) This paper empirically evaluates the potentials and limita-
parametrically and evaluates the performance of design output
tions of GANs for generative design. The observations
versus configurable constraints. The generative model is an archi-
point to the fact that state-of-the-art GAN models and archi-
tecture that, given a training dataset, can learn its probability distri-
tectures such as StyleGAN [27] are not capable of undertak-
bution and generate new samples with the same statistics as the
ing generative design tasks due to the lack of mechanisms to
training data. Among the generative models, GANs [25] offer
ensure diversity and desirability. Empirical evaluation of
excellent capabilities and success in generating realistic design
StyleGAN on a large-scale dataset of footwear products
images and continue to attract growing interest in the deep learning
reveals that although the model can generate realistic
community. GANs are generative models that involve a minimax
samples, the generated samples are remarkably similar to
game of two players between two models: a discriminative
authentic products in the training dataset. The results may
network D and a generative network G. The generator aims to
not benefit designers or promote their creativity, as the
learn a generative density function from the training data to
samples are neither novel nor aligned with user needs.
produce realistic samples. In contrast, the discriminator attempts
(2) This paper proposes a novel neural network architecture that
to discern whether an input sample is part of the original training
integrates a GAN-based model with a multimodal data-
set or a synthetic one generated by the generator in such a way as
driven design evaluation model, or the DDE model for
to distinguish fake samples from real ones. GANs have been
brevity, which was previously developed by the authors
applied to various domains such as computer vision [32,33],
[23]. The proposed DDE-GAN model tackles the challeng-
natural language processing [34], and semantic segmentation [35].
ing problem that existing GAN-based generative design solu-
Specifically, GANs have shown significant recent success in the
tions lack efficient mechanisms to guide the generator toward
field of computer vision in a variety of tasks such as image gener-
generating samples that are not only realistic but also diverse
ation [36], image-to-image translation [37], and image super-
and desirable (i.e., have high expected sentiment scores, both
resolution [38].
overall and attribute-level) by devising a novel DDE-GAN
GANs have been applied to the generation of engineering design,
model enhanced with DDE as a new loss function for auto-
such as the generation of 3D aircraft models in native format for
mated design evaluation. The DDE-GAN model can
complex simulation [39], numerous wheel design options optimized
predict user sentiments for each attribute of generated
for engineering performance [40], realistic samples from the distri-
samples and generates design concepts with high quality,
bution of paired fashion clothing and the provision of real samples
desirability, and diversity.
to pair with arbitrary fashion units for style recommendation [41],
(3) This paper conducts extensive experiments on a large
and new outfits with precise regions that conform to a description
dataset, scraped from a major online store for apparel and
of a language while maintaining the structure of the wearer’s
footwear, to demonstrate the effectiveness of the proposed
body [42]. Most of these models are usually built with quality to
DDE-GAN model in improving the diversity of generated
ensure high quality and usefulness; however, their intrinsic diver-
design samples, as well as their desirability based on pre-
sity is limited. The rationale behind the lack of diversity is that,
dicted user sentiments by comparing it with the StyleGAN
during the training process, the GAN generator is encouraged to
model [27] as a baseline. The model is applicable to any
generate samples close to the training data distribution to fool the
other domain, as long as both user data (e.g., reviews, com-
discriminator in a minimax game. GANs illustrate this proposition,
ments) and product data (e.g., images, technical descriptions)
as it prompts the generator G to map an arbitrary noise distribution
are available.
to realistic samples. On the contrary, the discriminator D tries to dis-
As the majority of cutting-edge generative models are built to tinguish the generated samples from the real ones, inevitably result-
create visual designs with great efficacy and success, creating a ing in limited diversity and creativity. However, due to the property

041407-2 / Vol. 145, APRIL 2023 Transactions of the ASME

that the generator attempts to learn to mimic the data, GANs are the designer in generating specific parts of a solution [60,61].
“emulative” [43,44] and have inspired researchers to investigate Swarm intelligence is inspired by natural phenomena in which
areas where diversity and creativity can be promoted in GANs [43]. flying or swimming animals move together in packs and allows
Extensive research has been conducted to enhance the diversity the system to interact locally with autonomous computational
of GAN-generated image styles [45,46]. The model can produce agents to achieve heterogeneous phenomena in generative pro-
diverse outputs by injecting noise vectors, such as the style variation cesses [62]. Despite these generative design methods’ significant
sampled from a normal distribution, into the generator and sampling progress and success, several critical knowledge gaps remain.
different style codes [46]. Some studies introduce modes as an addi- Most importantly, product forms in these quantitative design
tional input to transform conditional input into the target distribu- methods are typically expressed with a mathematical representation
tion [47]. The predetermined label is fed to the generator. It helps such as vectors, trees, graphs, and grammars, therefore, are limited
the model produce deterministic outputs that can map different by the tradeoff between flexibility and realism [63].
visual domains and styles, which has successfully generated Deep generative models have recently been proposed in the liter-
diverse outputs from a given source domain image. It is also ature to enable more effective and diverse concept generation as an

Downloaded from http://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/145/4/041407/6973255/md_145_4_041407.pdf by Yanshan University user on 16 June 2023

observed that the generators are most likely to generate samples alternative solution for generative design. Specifically, GANs [25]
from certain major modes/styles in the data but ignore the other have shown tremendous success in a variety of generative design
modes, for example, the modes that take a small count of distribu- tasks, such as topology optimization [40], material design [64],
tions. This problem is known as the “mode collapse” and is a and shape parametrization [39,63]. GANs are composed of a gener-
primary factor in the lack of diversity in GAN-generated samples. ator trained to generate new samples and a discriminator trained to
To address this problem, some researchers propose a regularization detect whether the generated samples are real. To better understand
term to maximize the distance between the generated outputs and how GANs work for generative design purposes, a brief comparison
the generated samples with latent codes injected [48]. DivAugGAN between GANs and the five conventional generative design
[49] further prevents mode collapse and improves the diversity of methods is conducted as follows.
generated images by using three randomly sampled latent codes
and two relative offsets. The model exerts a constraint on the gen- GANs versus cellular automata: In conventional cellular
erator to ensure that the changing scale of the generated samples is automata, generative rules are predefined, usually following
consistent with the various vectors injected into the latent space. much more basic transformations. GANs are composed of
Some researchers [50,51] believe that introducing a regularizer to many convolutional layers, and cellular automata can be rep-
GAN can address the model collapse problem and thus improve resented using a convolutional neural network with a
the diversity and quality of generated samples. If GANs are network-in-network architecture. Therefore, it is noticed
pushed too far from the data distribution for design generation, that a sufficiently complex neural network architecture,
the quality and realism of the generated samples will be negatively such as GAN, can be used to approximate each rule that
affected. Elgammal [44] proposes modifications to the GAN objec- fully comprises the cellular automata function. Moreover,
tive to allow it to generate creative art by maximizing the deviation the states of neurons in a neural network are continuous,
from established styles while minimizing the deviation from the art whereas cells in cellular automata have discrete states. In
distribution. Some researchers [32,52] suggest that only improving addition, neural networks are primarily concerned with the
diversity will cause GANs to deviate slightly from the original dis- output and not with the states of individual neurons,
tribution. With this motivation, this paper develops a new GAN whereas the output of cellular automata is a collection of
architecture that can guarantee high quality and also improve the its states.
diversity and desirability of the generated samples. GANs versus L-systems: L-systems is a programmable rewrit-
ing paradigm for producing samples. It is challenging to
predict the final rendering from the expression of the
2.2 Traditional Generative Design Methods Versus L-system alone since it is particularly sensitive to changes
Generative Adversarial Networks. Various generative design in expression. The deterministic L-system does not solve
methods have been developed to assist designers in the creative ide- the lack of variability for more realistic outputs [65].
ation process. Generative design is one of the design exploration However, GANs automatically discover and learn produc-
methods that can enable simultaneous exploration, validation, and tion rules by reading a large dataset. Beyond deterministic
comparison of thousands of design alternatives to support designers restrictions, GANs investigate alternative rules and relation-
and/or automate parts of the design process. There are five com- ships between characteristics. Because of the powerful
monly used generative design methods, including cellular automata processing power of GANs, they are smart enough to com-
[53], L-systems [54], shape grammars [55], genetic algorithms [56], prehensively learn the distribution of the training samples
and swarm intelligence [57]. As a popular generative strategy, cel- and reconstruct them. Consequently, GANs can guarantee
lular automata are characterized by the simplicity of its mechanisms the quality and realism of the results generated.
on the one hand and the potential complexity of its outcomes on the GANs versus shape grammars: Shape grammars allow for the
other. Cellular automata can modify the design specifications addition and subtraction of shapes that are eventually per-
according to predefined rules and produce unexpected design con- ceived as shape modifications. If the shape on the left side
cepts [58]. Cellular automata, as a popular generative strategy, are matches a shape on a drawing, then the rule can be
characterized by the simplicity of its mechanisms on the one hand applied, and the matching shape changes to match the right
and the potential complexity of its outcomes on the other. Cellular side of the rule. The generator and discriminator of a GAN
automata can modify design specifications according to predefined model are similar to the left- and right-hand sides of a
rules and produce unexpected design concepts [59]. Shape gram- shape grammar, respectively. The generator sample (equiva-
mars are geometry-based generative systems that describe how lent to the left side of a shape grammar) is validated as real by
complex shapes are built from simple entities and how a complex the discriminator (equivalent to the right side of a shape
shape can be decomposed into simpler subshapes. Unlike conven- grammar). The generating rule (latent representation
tional generative design methods that designers often communicate learned by GAN) can then be reinforced in the next iteration
initially, shape grammars involve designers more in making deci- of the training process, similar to shape grammars.
sions throughout the generative process stage [58]. Genetic algo- GANs versus genetic algorithms: Genetic algorithms are evo-
rithms, the most widely used method in generative design lutionary algorithms widely used to explore and optimize the
exploration, are applied as a generative and search procedure to generative design. The adversarial training procedure of
look for optimized design solutions and has the ability to modify GAN can be regarded as an evolutionary process. That is,
the sequence of the rules of design generation process to assist a discriminator acts as the environment (i.e., provides

Journal of Mechanical Design APRIL 2023, Vol. 145 / 041407-3

 adaptive loss functions), and a population of generators function based on the DDE model (Sec. 3.2). StyleGAN [70] is
evolves in response to the feedback from that environment. applied as a baseline in this work, and the novel loss function of
Genetic algorithms use a form of sampling to measure the the DDE-GAN model is improved over the loss function of Style-
relationship between a change in a parameter and a change GAN. The proposed DDE-GAN is described, followed by details
in the fitness (loss). In contrast, neural networks give a of the DDE model, previously developed by the authors [23],
means to directly calculate that relationship without sam- which is used as a newly added loss function in the developed
pling. Therefore, the speedup you experience when training model. The DDE model [23] accurately predicts the overall and
a neural network is the result of not needing to gather as attribute-level desirability of a new concept based on large-scale
many samples as the number of parameters you wish to tune. user sentiments and feedback on past designs. This work applies
GANs versus swarm intelligence: Swarm intelligence a well-trained DDE model as an augmented discriminator to
involves a collective study of how individuals act in their sur- promote user-centered image generation using the StyleGAN
rounding environment and interact with each other. It has model, to generate realistic, diverse, and desirable samples.
shown benefits in simplicity, ease of implementation, lack

Downloaded from http://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/145/4/041407/6973255/md_145_4_041407.pdf by Yanshan University user on 16 June 2023

of need for gradient information, and low parameter require-
ments [66]. Swarm intelligence is the approach that most 3.1 Generative Adversarial Network Formulation. GANs
closely resembles GANs out of the five methods. Although [25] can generate images from random noise and do not require
GANs are highly dependent on various parameters and the detailed information or labels from existing samples to start the gen-
backpropagation process to alter each layer to affect the erative process. The standard GAN structure consists of two neural
loss function, mode collapse is a frequent issue. To prevent networks: a generator G and a discriminator D. The generator G
mode collapse, swarm intelligence can be employed to takes random noise z ∼ P(z) sampled from a uniform or normal dis-
improve the generator’s performance in GAN and minimize tribution as input and maps the noise variable z ∼ P(z) to the data
iterations differently from conventional methods [67]. space x = G(z). The discriminator D distinguishes whether an
Despite the significant impact and progress made in the literature image is real or fake (i.e., made by the generator). The output
on GANs, existing work [52,68] is observed to lack sufficient eval- D(x) is the probability that the input x is real. If the input is a
uation mechanisms for desirability and diversity that would make fake image, D(x) would be zero. Through this process, the discrimi-
GANs suitable for generative conceptual design. The ability of a nator D is trained to maximize the probability of assigning the
model to generate concepts with iterative updating from evaluation correct label to both real samples and fake samples. Generator G
and feedback has the potential to lead to more creative and valuable is encouraged simultaneously to fit the true data distribution. The
design outcomes. The rationale is that the generative process must adversarial training processes that update the parameters of both
continually evaluate the generated samples concerning not only networks through backpropagation are formulated as the following
realism but also desirability and diversity; otherwise, the number learning objective:
of generated samples with lower desirability or diversity will con- min max E [log(D(x))] + E [log(1 − D(x̃))] (1)
tinue to grow without improvement, making it impossible for G D x∼Pr x̃∼Pg
designers to consider them meaningfully and accordingly. Some
studies in the literature have built GAN-based generative models where Pr is the distribution of the real image x and Pg is the model
with such evaluation processes [39,40,69]; however, their proposed distribution implicitly defined by x̃ ∼ G(z), z ∼ P(z). The genera-
evaluation tools are exclusively based on physics-based virtual tor’s input, z, is sampled from a simple noise distribution P, such
simulation environments that do not necessarily reflect user feed- as the uniform distribution or a spherical Gaussian distribution.
back. To bridge this gap, a user-guided evaluation DDE-GAN GAN models come in various forms, StyleGAN [27,70] being
model is proposed to enhance the generated design’s quality, diver- one of them. The StyleGAN extension to the GAN architecture pro-
sity, and desirability by incorporating synthetic user feedback from poses a major modification to the generator model, which uses (1) a
an evaluation process for its generated intermediate samples. mapping network to transfer latent space points to an intermediate
latent space, (2) an intermediate latent space to regulate style at
each point in the generator model, and (3) an addition of noise as
a source of variation at each point in the generator model. It features
3 Methodology a brand-new style-based generator architecture that creates high-
This section presents the architecture and formulation of the pro- resolution images with cutting-edge visual quality. In addition to
posed DDE-GAN model. A schematic of the model architecture is generating stunningly realistic high-quality images, the model
shown in Fig. 1. A brief overview of the GAN formulation and Sty- also provides control over the style of the generated image at
leGAN is presented first (Sec. 3.1), followed by the proposed loss various degrees of detail by adjusting the style vectors and noise.

Fig. 1 Architecture of the integrated automated design evaluator-generative adversarial network (DDE-GAN) model.
WGAN-GP: Wasserstein GAN + Gradient Penalty [70]; DDE: data-driven design evaluation [23].

041407-4 / Vol. 145, APRIL 2023 Transactions of the ASME

Moreover, most GAN-variants are generally sensitive to the where x̂ is a random sample, Px̂ is defined as samples along straight
problem domain. They perform exceptionally well in processing lines between pairs of points that come from the true data distribu-
generative tasks using large popular datasets such as human faces tion and the generator distribution, and λGP is a weighing factor.
and animals, where the novelty, diversity, or desirability of the gen-
erated samples are not important. However, their limitation
becomes evident in generative design tasks where the quality, diver- 3.2 Data-Driven Design Evaluator Loss. The preliminary
sity, and desirability of samples must be optimized simultaneously experiments conducted by the authors to generate images of foot-
[41]. Inspired by the success and popularity of StyleGAN, this wear products using StyleGAN revealed that although the model
paper develops a novel GAN model based on StyleGAN’s architec- is capable of generating realistic samples, the generated samples
ture, enhanced with the DDE model [23]. The loss function utilized are remarkably similar to the real products in the training dataset.
in StyleGAN is WGAN-GP [71], which is the most widely used loss These similarities can even be detected by simple visual inspections
function. WGAN-GP is constructed with the Wasserstein GAN [72] (see Fig. 2). With a sufficiently trained generator, even the discrimi-
formulation along with a gradient norm penalty to achieve Lipschitz nator would be unable to distinguish between the generated samples

Downloaded from http://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/145/4/041407/6973255/md_145_4_041407.pdf by Yanshan University user on 16 June 2023

continuity. The Wasserstein loss formulation applies the and the real ones. The second finding is that, although the generated
Wasserstein-l distance using a value function based on images are realistic, they may not benefit designers or promote their
Kantorovich–Rubinstein duality [73]. The loss function (1) is creativity as the samples are not necessarily novel or aligned with
then modified as follows: user needs. Although the model training procedure considers algo-
rithmic quality, it does not consider how users will receive and react
min max E [D(x)] − E [D(x̃)] (2) to these computer-generated designs. This paper argues that this
G D∈D x∼Pr x̃∼Pg
problem comes from the sole objective of existing generator-
where D is the set of 1-Lipschitz functions and Pg is the distribution discriminator architectures to maximize “realism.” That is, there
of the model implicitly defined by x̃ = G(z), z ∼ p(z). The Wasser- is an absence of a loss function that can incorporate other critical
stein loss is approximated given a set of k-Lipschitz, and the metrics in addition to realism, such as the alignment of the gener-
weights of the discriminator are clipped to some range. By adding ated samples with the perspectives and needs of users, which
the Wasserstein loss with gradient penalty, WGAN-GP enforces a could cause the discriminator to fail when updating the generator
soft restriction on the gradient norm of the discriminator’s output in terms of learning and producing features that maximize the use-
with respect to its input rather than clipping network weights as fulness of a design. To convey the measurement of the design per-
in Ref. [72], to guarantee the Lipschitz requirement. The objective formance score back to the generator for subsequent iteration
function of StyleGAN is then formulated as follows: improvements, the authors believe that new loss functions are

needed to force the discriminator to identify and locate other
2 metrics, such as novelty or desirability. This observation inspired
L = E [D(x̃)] − E [D(x)] + λGP E ∇x̂ D(x̂)2 −1 ] (3)
x̃∼Pg x∼Pr x̂∼Px̂ the authors to investigate when to incorporate the user-guided

Fig. 2 Examples of sneaker images generated using StyleGAN

Journal of Mechanical Design APRIL 2023, Vol. 145 / 041407-5

assessment mechanism into the discriminator, as described below, if scalable prediction of the new concepts’ overall desirability. By
the similarity between the produced and real images is effectively integrating the DDE loss into the StyleGAN’s discriminator, the
reduced. DDE-GAN model is created (Fig. 1). The DDE loss integrated
This paper applies DDE [23] as a user-centered design evaluation into the discriminator can measure the intermediate samples gener-
model to evaluate the generated samples with respect to the ated by the generator in each iteration and convey the loss back to
expected quality and performance of the generated designs. DDE the generator for a new set of parameters. The DDE loss evaluates
is a multimodal deep regression model that uses an attribute-level the results of each round in the iterative training process, which is
sentiment analyzer [74] to predict user-generated product ratings then used to backpropagate and optimize the generator and the dis-
based on online reviews. It was created to automate design evalua- criminator. The DDE-GAN architecture is expected to result in
tion and improve decision-making by domain experts. Based on better designs from the user’s point of view and simultaneously
extensive user evaluations of existing designs, the DDE model maintain excellent image quality.
offers designers a precise and scalable means to forecast new con- The objective function of the DDE-GAN model is therefore for-
cepts’ overall and attribute-level desirability. DDE is an end-to-end mulated as follows:

Downloaded from http://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/145/4/041407/6973255/md_145_4_041407.pdf by Yanshan University user on 16 June 2023

design assessment system that can interpret visuals, plain language, 
2 
and structured data. As shown in Fig. 3, the DDE system uses a L = E [D(x̃)] − E [D(x)] + λGP E ∇x̂ D(x̂)2 −1
ResNet-50 model [75] to evaluate and interpret images of a x̃∼Pg x∼Pr x̂∼Px̂

product. ResNet-50 can represent complex functionality and learn + λDE E [LDE (x̃)] (4)
x̃∼Pg
features at many different levels of abstraction to understand the
connections between orthographic representations of design con-
cepts (inputs) and user sentiment intensity values (outputs). The 1 N
2
bidirectional encoder representation from transformers (BERT) LDE (x) = f i (x) − ŷi (5)
N i=1
model, a different model in the DDE system, extracts and analyzes
product descriptions written in natural language [76]. The BERT where λDE is a constant that defines the loss weight, LDE is the DDE
model can determine the connection between a product’s technical loss function to evaluate the feature of the generated samples in the
description and the user’s emotional sentiment level. The DDE characteristic of performance, fi(x) is the prediction for all design x
system then integrates the various meaningful data collected from generated by the DDE model, and ŷi is the desired design evaluation
the Internet platform and models the relationships between score and is set as 1 for each attribute, indicating that the models are
images, text, and statistics. The DDE model synthesizes different trained to generate samples with the highest possible expectation.
modes of data using a novel fusion mechanism to develop a more The StyleGAN loss terms regulate the high quality of the produced
accurate context about the product and the associated user feedback pictures and the DDE loss guarantees that the produced samples
[23]. The DDE model was trained on a large-scale dataset that was have high user sentiment scores. Combining the two elements
scraped from a major online footwear store. In the dataset, each allows the proposed DDE-GAN model to simultaneously create
product has four types of information: six orthographic images, high-quality images and high user sentiment ratings. This new set
one numerical rating score, a list of textual product descriptions, of loss functions provides a more accurate and evaluation-guided
and real textual customer reviews from an e-commerce platform, generator and discriminator in DDE-GAN compared to previous
where images and feature descriptions are the inputs to the DDE work and can be easily tuned. Information on the constants and
model and the numerical rating score and sentiment intensity other implementation details is provided in Sec. 4.
values from customer reviews are the outputs. The dataset is consti-
tuted of a total number of 8706 images and 113,391 reviews for
1452 identified shoes. Numerical experiments on this large
dataset indicated promising performance by the DDE model with
0.001 MSE loss and over 99.1% accuracy. 4 Experiments and Results
The DDE model can accurately predict user sentiments for a new In this section, the dataset and implementation details of the pro-
design concept based only on its orthographic images and descrip- posed DDE-GAN model are first described, followed by the intro-
tions and provide numerical design performance values associated duction of metrics established to investigate the effectiveness of the
with each attribute of the generated concept. This paper builds a developed DDE-GAN model in generating realistic samples with
novel loss function based on the DDE model, called the DDE high desirability and diversity. The results of the experimental anal-
loss, into the GAN’s discriminator to enable an accurate and yses are presented next, comparing the outcomes generated by the

Fig. 3 The multimodal DDE model (adapted from Ref. [23])

041407-6 / Vol. 145, APRIL 2023 Transactions of the ASME

developed DDE-GAN model and the state-of-the-art StyleGAN as the distances between the mean embeddings of the features.
model as a baseline. Given two sets of data X and Y, the MMD is calculated as the dis-
tance between the feature means of X and Y. The expression is for-
mulated as follows:
4.1 Dataset and Implementation Details. To test and validate
the performance of StyleGAN in generating realistic and diverse MMD2 (P, Q) = EP [k(X, X)] − 2EP,Q [k(X, Y)] + EQ [k(Y, Y)] (7)
images, a large-scale dataset was scraped from a major online foot-
wear store to perform numerical experiments. The collected where k is the kernel function, P is the distribution over a set of
large-scale dataset contains a total of 7642 images with a size of input data X, and Q is the distribution over a set of generated data Y.
256 × 256 × 3. Several brands of footwear are included in the This paper uses two different kernel functions: a linear kernel and a
dataset to avoid mode collapse and increase the diversity of the polynomial kernel. The linear kernel is defined as
dataset, including Adidas, ASICS, Converse, Crocs, Champion,
FILA, PUMA, Lacoste, New Balance, Nike, and Reebok. k(x, y) = x⊤ y (8)

Downloaded from http://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/145/4/041407/6973255/md_145_4_041407.pdf by Yanshan University user on 16 June 2023

The DDE model is pretrained and serves as an offline network
and the polynomial kernel is defined as
added to the new StyleGAN loss. The implementation of the pre-
trained DDE model is discussed in this section. The experiments
d
k(x, y) = γx⊤ y + c0 (9)
were carried out with k fold, with k = 10, to randomly split the
dataset into train, validate, and test sets with a 7 : 1 : 2 ratio. All where x and y are the input vectors, d is the degree of the kernel, γ is
experimental results were conducted five times and reported as the weight, and c0 is a constant. In the experiments, the polynomial
mean ± std to alleviate the randomness effect. All neural networks kernel γ is set to 1, the kernel degree d is set to 0.5 and the coeffi-
were trained on PyTorch [77]. Adam [78] optimizer with β = (0.9, cient c0 is set to 0. Details of the results are discussed in the follow-
0.999) and the learning rate of =0.01 were used to train the ing sections.
model parameters for 50 epochs and save the model with the best
loss in the validation dataset. To avoid overfitting, a dropout layer
was added to the self-attention fusion model with a dropout rate
4.3 Results and Analyses. To test and validate the perfor-
of Pdrop = 0.1. The DDE model was trained over 40 epochs. The
mance of the proposed DDE-GAN model for design generation
training time cost per epoch was 5–7 min, which added up to 3–4
with improved desirability and diversity, a set of experiments was
h. All training and testing experiments were conducted on a
performed on a real dataset of footwear products with StyleGAN
single NVIDIA RTX 3090 GPU (24GB GRAM), an AMD Ryzen
as the baseline model. This section first presents the visual design
9 5950X CPU, and 64GB memory.
samples generated by the DDE-GAN models. The performance of
The weight of StyleGAN, λGP, was precisely calibrated by the
the proposed model is then compared with the baseline using the
original paper with the best performance achieved [27]. Therefore,
FID score, followed by an MMD analysis to examine the similarity
this paper follows the exact same value of 0.8192 as suggested in
between the generated images and the real images. Lastly, DDE is
the original StyleGAN paper. The weight of DDE loss, λDE, is
applied to test the images generated by the DDE-GAN and Style-
defined by binary search from 0.1 to 2 in general, and finally is
GAN models to evaluate their desirability prediction scores.
set as 0.5 to meet the tradeoff between high image quality (FID)
Visual results: As shown in Fig. 4, the DDE-GAN-generated
and predicted sentiment scores. Adam [78] was used as an optimizer
samples deliver the expected high quality and realism, which are
with a learning rate of 0.0025 to optimize the model and set β as
also observed in the StyleGAN-generated samples (Fig. 2). The
(0.9, 0.999), representing the coefficients used for computing
overall images are realistic, vibrant, clear, and have an aesthetic
running averages of gradient and its square. Beyond that, data aug-
understandable to the human mind. Although Fig. 4 reveals some
mentation methods such as random flip, rotation, scale, brightness,
differences, the uniqueness and diversity of the images are discov-
and contrast were applied to improve data diversity. The model was
ered in some samples. Some images contain features that might
trained 20,000 times for each experimental setting, and the average
sound novel or even strange. However, this paper defines these
performance statistics were reported.
characteristics as novelty and diversity. The authors noticed that
most of the generative model samples generated in the current
4.2 Evaluation Metrics. Frechet inception distance (FID) GAN-based design literature emphasize quality, while the images
[79] is used to assess the quality of the images created by a gener- are somehow similar to existing products. Yet, that may hinder
ative model. FID evaluates the statistics for both the target and innovation in the generative process, because a conventional
output images simultaneously. It compares the distribution of the GAN discriminator may easily label a “novel” sample that could
generated images with the distribution of real photos used to train potentially be an interesting sample from the design perspective
the generator. FID can also identify intraclass mode dropping and as “fake,” simply because it does not look like any real item
quantify the variety and quality of produced samples, making it a within that category and contains unknown features. This, in turn,
valuable tool for assessing the quality and diversity of synthetic would discourage the conventional GAN generator from generating
images. A lower FID score intuitively indicates a closer distribution more of these potentially novel samples. The DDE-GAN model
between the objectives and the results, which corresponds to a better proposed in this paper introduces an additional loss to encourage
performance of the generative model. The proposed DDE-GAN the generator to produce more novel and distinctive images. There-
model is compared with the state-of-the-art GAN architecture, Sty- fore, the authors define attributes such as “strange” or “never seen
leGAN, in terms of FID, using the following equation [79]: before” as one of the diversity criteria. Among the large size of
 2 
1/2 the generated samples, 16 distinct images are manually selected
FID = μr − μg  + Tr Σr + Σg − 2 Σr Σg (6) and presented in Fig. 5, as they are identified as designs with
novelty and diversity. It is clearly seen that these sneakers are far
where Tr refers to the trace linear algebra operation, (μr, Σr) and (μg, from “similar” to existing sneakers, compared to the other
Σg) refer to the mean and covariance matrices in the feature order of samples shown in Fig. 4. They look distinguishable with more
the embeddings obtained in real and generated images, respectively. novelty and diversity. To further validate the effects of the
Diversity assessment: To quantitatively measure whether the DDE-GAN model on novelty and diversity, a quantitative analysis
DDE-GAN model has guided the generator to synthesize new of FID and MMD is conducted next.
designs with greater variety, a kernel-based statistical test method Quality test: Table 1 shows the FID scores of the best samples for
called maximum mean discrepancy (MMD) is used to determine each generation when training models with the collected dataset.
the similarity between two distributions [80,81]. MMD is defined The FID scores are the mean values for ten different training
by the idea of representing the distances between the distributions results. As shown in Table 1, the StyleGAN model produces a

Journal of Mechanical Design APRIL 2023, Vol. 145 / 041407-7

 Downloaded from http://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/145/4/041407/6973255/md_145_4_041407.pdf by Yanshan University user on 16 June 2023
Fig. 4 Examples of novel designs generated by the proposed DDE-GAN model

Fig. 5 Examples of novel designs with high diversity observed

lower FID score than the DDE-GAN model. A lower FID score distance between the feature vectors calculated for the real and gen-
means that the model is more stable and correlates better with erated images. Lower scores indicate the two groups of images are
higher-quality images. However, the FID score of the DDE-GAN more similar, or have more similar statistics, with a perfect score
model and its standard deviation are close to StyleGAN with only being 0.0 indicating that the two groups of images are identical.
a small change (a 0.23 decrease), and it is empirically concluded Therefore, from the perspective of similarity, StyleGAN with
that an FID score below 10 is sufficient to demonstrate the effective- lower FID represents that the generated samples are more similar
ness of a generative model [82].2 In addition, the difference to real images compared with the DDE-GAN with a higher FID
between DDE-GAN (mean = 6.45) and StyleGAN (mean = 6.22) score. DDE-GAN with higher FID reveals that the generated
is verified with t-test, P = 0.0026. Therefore, the DDE-GAN samples are distinct from existing images, which is further validated
model performs well in achieving high-quality results. FID can next.
also be explained as a similarity metric, because it calculates the Diversity test: The primary rationale behind the proposed
DDE-GAN model is to promote the diversity of images generated
by GAN. The similarity between the produced samples and the orig-
2
https://nealjean.com/ml/frechet-inception-distance/ inal input is calculated using the MMD metric to estimate the

041407-8 / Vol. 145, APRIL 2023 Transactions of the ASME

Table 1 Comparison of the average FID score of the best 0.144) compared to DDE-GAN (mean = 0.163; t-test P = 4e − 14).
generators in StyleGAN and DDE-GAN Overall, DDE-GAN performs well in generating images with less
similarity to the original dataset and more diversity.
Algorithm FID score Desirability test: In addition to enhancing the diversity and
novelty of the generated images, another objective of this paper is
StyleGAN 6.22 ± 0.17
DDE-GAN 6.45 ± 0.21
to build a user-guided automated generative design model that
can produce designs that meet the desirability requirements. The
DDE model was trained on a large dataset of 1452 design images
labeled with user sentiments to learn and capture the relationship
between images and attribute performance. The DDE model
diversity of novel samples. A higher similarity value indicates that creates a collection of rating scores representing the performance
the generated samples contain less diversity, and vice versa. The of relative attributes when images of testing products are imported
MMD values are calculated based on the results of the proposed into the model. The number of products with an absolute value of

Downloaded from http://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/145/4/041407/6973255/md_145_4_041407.pdf by Yanshan University user on 16 June 2023

model and the baseline model, using linear and polynomial the desired values and the predicted values below a threshold was
kernels, as shown in Fig. 6. The proposed DDE-GAN model is counted using the prediction accuracy rate (PAR) metric. The per-
observed to produce higher MMD scores than the baseline Style- centage of the counted number to the overall testing number
GAN model, indicating a significantly lower similarity between serves as the accuracy metric. The well-trained DDE model was
the real training dataset and the samples generated by the verified to predict user sentiments for a new design concept based
DDE-GAN model. For the linear kernel, StyleGAN receives a only on its orthographic images and provides the numerical
mean of 124.77 and a standard deviation of 3.74, and DDE-GAN values of the design performance associated with each product attri-
obtains a lower mean of 110.15 with a lower standard deviation bute with a prediction accuracy of 76.54% [23]. To test whether the
of 5.02. The mean and variance of the polynomial kernel are new designs produced by DDE-GAN perform better than the
(0.145, 0.003) and (0.164, 0.002) for DDE-GAN and StyleGAN, designs created by StyleGAN, all 480 images were selected from
respectively. A statistical test was employed to examine the differ- the output of two models and tested using a well-trained DDE
ence between the performances of the two models. Results of model to predict their overall and attribute-level desirability based
assessments using linear kernel MMD show that the DDE-GAN on large-scale user reviews on existing products. The average
model (mean = 110.15; t-test P = 3e − 08) significantly outperforms numerical values of user sentiments in ten attributes and the
StyleGAN (mean = 124.77) in generating samples with high diver- overall performance of the designs are shown in Table 2 in which
sity. Likewise, StyleGAN, assessed using polynomial kernel MMD, the sentiment intensity of users ranges from [−1, 1], with −1 and
is shown to perform worse in generating diverse samples (mean = 1 representing extremely negative and extremely positive

Fig. 6 (a) The MMD linear kernel and (b) MMD polynomial kernel results for StyleGAN and
DDE-GAN

Table 2 Results of the DDE test [23] regarding “predicted sentiment values” on 480 randomly selected samples generated by
StyleGAN and DDE-GAN

Predicted sentiment value

Model

Attributes StyleGAN DDE-GAN Change (%) P-value

Traction 0.1652 ± 0.018 0.2064 ± 0.018 25 0.0035

Shape 0.2831 ± 0.016 0.3097 ± 0.024 9 0.0074
Heel 0.3736 ± 0.020 0.5142 ± 0.015 38 <0.0001
Cushion 0.1924 ± 0.019 0.3005 ± 0.031 56 <0.0001
Color 0.2783 ± 0.021 0.4179 ± 0.019 50 <0.0001
Fit 0.2350 ± 0.015 0.2168 ± 0.012 −8 0.0803
Impact absorption 0.2303 ± 0.027 0.3211 ± 0.016 39 <0.0001
Durability 0.2409 ± 0.039 0.2714 ± 0.034 13 0.0334
Permeability 0.1471 ± 0.020 0.1916 ±0.017 30 <0.0001
Stability 0.1892 ± 0.031 0.2073 ± 0.025 10 <0.0001
Overall 4.536 ± 0.0754 4.735 ± 0.0718 4 0.0002

Journal of Mechanical Design APRIL 2023, Vol. 145 / 041407-9

sentiment, respectively. DDE-GAN is observed to generate designs This work starts the journey of generative models integrated with
with higher expected user sentiment values for most individual attri- user data to build the foundation for data-driven, user-centered
butes and overall performance. In general, the predicted sentiment design. Potential AI-augmented design tools can range from user-
values of individual attributes of the samples created by centered design valuation, design generation, design selection, to
DDE-GAN obtained increases of 9–56% compared to StyleGAN, design recommendation. In future iterations, because DDE-GAN
except for the attribute “Fit.” To further explore the differences integrates user sentiment, the influence of extreme users on
between the two models, the predicted sentiment values of the design novelty will be explored. Integrating extreme user behaviors,
two models are analyzed by two-tailed independent samples needs, and sentiment has been shown to increase design creativity
t-tests, tested for significance at P < .05. As shown in Table 2, and novelty [83]. What’s more, the DDE-GAN model is developed
there is a significant improvement associated with the attributes to aggregate user feedback in the loss function to generate samples
“Traction,” “Shape,” “Heel,” “Cushion,” “Color,” “Impact absorp- with high desirability from the perspective of users, which is a lim-
tion,” “Permeability,” “Stability,” and the “Overall” rating of itation that the produced sample conveys most of the user feedback.
which P-values are much less than 0.05. However, the prediction Therefore, future work can build a model that can generate design

Downloaded from http://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/145/4/041407/6973255/md_145_4_041407.pdf by Yanshan University user on 16 June 2023

performance of the two models was not significantly different for concepts based on the individual user. Furthermore, another poten-
the attributes “Fit” and “Durability” (P-values 0.0803 and 0.0334, tial for enhanced user-guided GANs would be the ability to simul-
respectively). The potential reason is that these two features are taneously generate images and textual languages. The DDE model
not easy to be captured by generative models which is more was devised to extract visual and textual features and identify the
capable of learning the latent representation among visual features. dependency among various data types, such as image, text, and
This is an interesting and open issue for generative design tasks and structure data. This work only partially used the image evaluation
authors are inspired to tackle the problem in future work. The sta- tools in DDE to inspire the generator to create enhanced and
tistical test strongly verifies that the additional loss function in con- guided samples. Therefore, in the next step, the authors will con-
junction with the discriminator successfully improves the generator tinue to develop a multimodal GAN that can generate a visual
to learn features corresponding to user sentiments and design desir- image and natural language as a detailed description of design
ability. The evaluation results indicate that DDE-GAN-generated samples. Additionally, the DDE model will be efficiently used to
design samples will lead to greater user satisfaction compared to assess samples with images and text information for a more accurate
StyleGAN-generated samples that focus only on the “realism” of generative model. There is the potential to broaden model’s useful-
the generated samples. The proposed DDE-GAN model is opti- ness to other facets of the innovation process, including design,
mized to serve effectively as a user-centered generative design marketing, and product management. Moreover, future research
framework. will explore the increasing inclusion and reduction of bias in the
model, as this represents a significant issue in many AI applications
[84]. Lastly, future research should conduct semi-structured inter-
views and post-evaluation activities involving design experts to
5 Conclusions and Future Research Directions evaluate the results of generative design, both objectively and sub-
This paper takes a different approach to promote diversity and jectively. Further qualitative validation can evaluate how many
desirability in GAN-based generative design models. The lack of ideas designers can generate, the novelty of ideas in terms of how
these critical design metrics in the samples generated by existing innovative and unexpected they are, how well the design space is
GANs is caused by the limitation of adversarial training between explored (variety), and how feasible they are when comparing
the generator and the discriminator to generate only “realistic” them with design specifications (quality). The relevant attributes
samples. To address this problem, a multimodal data-driven design of the product in each dimension must be identified and evaluated
evaluation model, DDE, is introduced in the discriminator to according to the recommendations and considerations of Shah
encourage the generator to get creative and generate more “unfamil- [31] and Nelson [85].
iar” and potentially novel samples. Another problem this paper
addresses is devising a user-centered generative model that can gen-
erate real products with high usefulness and attractiveness from the Acknowledgment
user’s perspective. To bridge this gap, the DDE model is applied to
predict the performance of the generator samples in each iteration. This material is based upon work supported by the National
The predicted values are integrated with other loss functions and Science Foundation under the Engineering Design and System
transmitted to the models for backpropagation. The generator is Engineering (EDSE) Grant #2050052. Any opinions, findings, con-
updated and optimized for integrated DDE loss and finally is clusions, or recommendations expressed in this material are those of
enforced with the capability to generate well-performed designs. the author(s) and do not necessarily reflect the views of the National
To investigate the effectiveness of the developed DDE-GAN Science Foundation.
model in generating images with high quality, high diversity, and
desirability, the FID metric, the MMD, and the DDE testing tool
are deployed to conduct the DDE-GAN experiment analysis with Conflict of Interest
the baseline StyleGAN model. Visual output and quantitative anal- There are no conflicts of interest.
ysis validate the improvement of DDE-GAN. Specifically, the gen-
erated images contain novel features and characteristics from
human observation and further quantitative analysis. Average FID Data Availability Statement
scores confirmed the stability of the newly devised DDE-GAN
and stated the sufficient ability of DDE-GAN to generate high- The datasets generated and supporting the findings of this article
quality images. Lower MMD values again indicate that the DDE- are obtainable from the corresponding author upon reasonable
GAN enhances the generator’s ability to create more diverse request.
samples. The DDE offline model was applied to test the two sets
of novel images of DDE-GAN and StyleGAN, and DDE-GAN
has demonstrated the ability to design samples with improved desir- References
ability and popularity. The developed DDE-GAN model was suc- [1] Liu, Y.-C., Chakrabarti, A., and Bligh, T., 2003, “Towards an ‘ideal’ Approach
cessfully tested in a sneaker design case study, but is flexible for Concept Generation,” Des. Stud., 24(4), pp. 341–355.
[2] Yilmaz, S., Daly, S. R., Seifert, C. M., and Gonzalez, R., 2015, “How Do
enough to be readily expanded to other product categories and Designers Generate New Ideas? Design Heuristics Across Two Disciplines,”
can serve as an intelligent tool to produce photorealistic renderings Des. Sci., 1(1), p. e4.
of new concepts in other design applications. [3] Simon, H. A., 2019, The Sciences of the Artificial, MIT Press, Cambridge, MA.

041407-10 / Vol. 145, APRIL 2023 Transactions of the ASME

 [4] Cross, N., 2021, Engineering Design Methods: Strategies for Product Design, [36] Lee, M., Park, Y., Jo, H., Kim, K., Lee, S., and Lee, I., 2022, “Deep Generative
John Wiley & Sons, New York. Tread Pattern Design Framework for Efficient Conceptual Design,” ASME
[5] Osborn, A. F., 1953, Applied imagination, New York: Charles Scribner’s Sons, J. Mech. Des., 144(7), p. 071703.
New York. [37] Kazemi, H., Seepersad, C. C., and Alicia Kim, H., 2022, “Multiphysics Design
[6] Akin, Ö., and Lin, C., 1995, “Design Protocol Data and Novel Design Decisions,” Optimization Via Generative Adversarial Networks,” ASME J. Mech. Des.,
Des. Stud., 16(2), pp. 211–236. 144(12), p. 121702.
[7] Brophy, D. R., 2001, “Comparing the Attributes, Activities, and Performance of [38] Regenwetter, L., Nobari, A. H., and Ahmed, F., 2022, “Deep Generative
Divergent, Convergent, and Combination Thinkers,” Creativity Res. J., 13(3–4), Models in Engineering Design: A Review,” ASME J. Mech. Des., 144(7),
pp. 439–455. p. 071704.
[8] Atman, C. J., Chimka, J. R., Bursic, K. M., and Nachtmann, H. L., 1999, “A [39] Shu, D., Cunningham, J., Stump, G., Miller, S. W., Yukish, M. A., Simpson, T.
Comparison of Freshman and Senior Engineering Design Processes,” Des. W., and Tucker, C. S., 2020, “3D Design Using Generative Adversarial Networks
Stud., 20(2), pp. 131–152. and Physics-Based Validation,” ASME J. Mech. Des., 142(7), p. 071701.
[9] Christiaans, H., and Dorst, K. H., 1992, “Cognitive Models in Industrial [40] Oh, S., Jung, Y., Kim, S., Lee, I., and Kang, N., 2019, “Deep Generative Design:
Design Engineering: A Protocol Study,” Des. Theory Methodol., 42(1), pp. Integration of Topology Optimization and Generative Models,” ASME J. Mech.
131–140. Des., 141(11), p. 111405.
[10] Jansson, D. G., and Smith, S. M., 1991, “Design Fixation,” Des. Stud., 12(1), [41] Yuan, C., and Moghaddam, M., 2020, “Attribute-Aware Generative Design

Downloaded from http://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/145/4/041407/6973255/md_145_4_041407.pdf by Yanshan University user on 16 June 2023

pp. 3–11. With Generative Adversarial Networks,” IEEE Access, 8(1), pp. 190710–
[11] Purcell, A. T., and Gero, J. S., 1996, “Design and Other Types of Fixation,” Des. 190721.
Stud., 17(4), pp. 363–383. [42] Liang, X., Lin, L., Yang, W., Luo, P., Huang, J., and Yan, S., 2016, “Clothes
[12] Viswanathan, V. K., and Linsey, J. S., 2013, “Design Fixation and Its Mitigation: Co-Parsing Via Joint Image Segmentation and Labeling With Application to
A Study on the Role of Expertise,” ASME J. Mech. Des., 135(5), p. 051008. Clothing Retrieval,” IEEE Trans. Multimedia, 18(6), pp. 1175–1186.
[13] Beitz, W., Pahl, G., and Grote, K., 1996, “Engineering Design: A Systematic [43] Heyrani Nobari, A., Rashad, M. F., and Ahmed, F., 2021, “Creativegan: Editing
Approach,” MRS Bull., 71(1), pp. 1–617. Generative Adversarial Networks for Creative Design Synthesis,” International
[14] Kirton, M. J., 2004, Adaption-Innovation: In the Context of Diversity and Design Engineering Technical Conferences and Computers and Information
Change, Routledge, Abingdon, Oxfordshire. in Engineering Conference, Virtual, online, Aug. 17–20, Vol. 85383, American
[15] Sarkar, P., and Chakrabarti, A., 2011, “Assessing Design Creativity,” Des. Stud., Society of Mechanical Engineers, Paper No. V03AT03A002.
32(4), pp. 348–383. [44] Elgammal, A., Liu, B., Elhoseiny, M., and Mazzone, M., 2017, “CAN: Creative
[16] Crilly, N., 2019, “Creativity and Fixation in the Real World: A Literature Review Adversarial Networks Generating ‘Art’ by Learning About Styles and Deviating
of Case Study Research,” Des. Stud., 64(1), pp. 154–168. From Style Norms,” Eighth International Conference on Computational
[17] Buhl, A., Schmidt-Keilich, M., Muster, V., Blazejewski, S., Schrader, U., Creativity, ICCC 2017, Atlanta, GA, June 19–23.
Harrach, C., Schäfer, M., and Süßbauer, E., 2019, “Design Thinking for [45] Nobari, A. H., Chen, W., and Ahmed, F., 2022, “Range-Constrained
Sustainability: Why and How Design Thinking Can Foster Sustainability- Generative Adversarial Network: Design Synthesis Under Constraints Using
Oriented Innovation Development,” J. Cleaner Prod., 231(1), pp. 1248–1257. Conditional Generative Adversarial Networks,” ASME J. Mech. Des., 144(2),
[18] Cash, P., and Štorga, M., 2015, “Multifaceted Assessment of Ideation: Using p. 021708.
Networks to Link Ideation and Design Activity,” J. Eng. Des., 26(10–12), [46] Huang, X., Liu, M.-Y., Belongie, S., and Kautz, J., 2018, “Multimodal
pp. 391–415. Unsupervised Image-to-Image Translation,” Proceedings of the European
[19] Dorst, K., and Cross, N., 2001, “Creativity in the Design Process: Co-Evolution of Conference on Computer Vision (ECCV), Munich, Germany, Sept. 8–14, pp.
Problem–Solution,” Des. Stud., 22(5), pp. 425–437. 172–189.
[20] Sosa, R., 2019, “Accretion Theory of Ideation: Evaluation Regimes for Ideation [47] Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., and Choo, J., 2018,
Stages,” Des. Sci., 5(1), p. e23. “Stargan: Unified Generative Adversarial Networks for Multi-Domain
[21] Gonçalves, M., and Cash, P., 2021, “The Life Cycle of Creative Ideas: Towards a Image-to-Image Translation,” Proceedings of the IEEE Conference on
Dual-Process Theory of Ideation,” Des. Stud., 72(1), p. 100988. Computer Vision and Pattern Recognition, Salt Lake City, UT, June 18–23,
[22] Renner, G., and Ekárt, A., 2003, “Genetic Algorithms in Computer Aided pp. 8789–8797.
Design,” Comput.-Aided Des., 35(8), pp. 709–726. [48] Chen, Q., Wang, J., Pope, P., Chen, W., and Fuge, M., 2022, “Inverse Design of
[23] Yuan, C., Marion, T., and Moghaddam, M., 2022, “Leveraging End-User Data for Two-Dimensional Airfoils Using Conditional Generative Models and Surrogate
Enhanced Design Concept Evaluation: A Multimodal Deep Regression Model,” Log-Likelihoods,” ASME J. Mech. Des., 144(2), p. 021712.
ASME J. Mech. Des., 144(2), p. 021403. [49] Dingdong, Yang, and Honglak, Lee, 2018, “Diversity Augmented Conditional
[24] Vasconcelos, L. A., Cardoso, C. C., Sääksjärvi, M., Chen, C. -C., and Crilly, N., Generative Adversarial Network for Enhanced Multimodal Image-to-Image
2017, “Inspiration and Fixation: The Influences of Example Designs and System Translation,” International Conference on Learning Representations,
Properties in Idea Generation,” ASME J. Mech. Des., 139(3), p. 031101. Vancouver, Canada, Apr. 30–May 3.
[25] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, [50] Lin, J., Chen, Z., Xia, Y., Liu, S., Qin, T., and Luo, J., 2019, “Exploring Explicit
S., Courville, A., and Bengio, Y., 2014, “Generative Adversarial Networks,” Domain Supervision for Latent Space Disentanglement in Unpaired
Proceedings of the International Conference on Neural Information Processing Image-to-Image Translation,” IEEE Trans. Pattern Anal. Mach. Intell., 43(4),
Systems (NIPS 2014), Montreal, Canada, Dec. 8–13, pp. 2672–2680. pp. 1254–1266.
[26] Teng, L., Fu, Z., and Yao, Y., 2020, “Interactive Translation in Echocardiography [51] Che, T., Li, Y., Jacob, A. P., Bengio, Y., and Li, W., 2016, “Mode regularized
Training System With Enhanced Cycle-GAN,” IEEE Access, 8(1), pp. 106147– generative adversarial networks,” Fifth International Conference on Learning
106156. Representations, San Juan, Puerto Rico, May 2–4, p. 1.
[27] Karras, T., Laine, S., and Aila, T., 2019, “A Style-Based Generator Architecture [52] Chen, W., and Ahmed, F., 2021, “Mo-Padgan: Reparameterizing Engineering
for Generative Adversarial Networks,” Proceedings of the IEEE/CVF Conference Designs for Augmented Multi-Objective Optimization,” Appl. Soft Comput.,
on Computer Vision and Pattern Recognition, Long Beach, CA, June 15–20, pp. 113(1), p. 107909.
4401–4410. [53] Wolfram, S., 2002, A New Kind of Science, Vol. 5, Wolfram Media Champaign,
[28] Chen, L., Maddox, R. K., Duan, Z., and Xu, C., 2019, “Hierarchical Cross-Modal Champaign, IL.
Talking Face Generation With Dynamic Pixel-wise Loss,” Proceedings of the [54] Lindenmayer, A., 1968, “Mathematical Models for Cellular Interactions in
IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Development I. Filaments With One-Sided Inputs,” J. Theor. Biol., 18(3),
CA, June 15–20, pp. 7832–7841. pp. 280–299.
[29] Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., and Lee, H., 2016, [55] Stiny, G., 1980, “Introduction to Shape and Shape Grammars,” Environ. Plann. B:
“Generative Adversarial Text to Image Synthesis,” Proceedings of the 33rd Plann. Des., 7(3), pp. 343–351.
International Conference on International Conference on Machine Learning, [56] Holland, J. H., 1992, Adaptation in Natural and Artificial Systems: An
New York, NY, June 19– 24. Introductory Analysis With Applications to Biology, Control, and Artificial
[30] Kim, T., Cha, M., Kim, H., Lee, J. K., and Kim, J., 2017, “Learning to Discover Intelligence, MIT Press, Cambridge, MA.
Cross-Domain Relations With Generative Adversarial Networks,” International [57] Kennedy, J., 2006, Handbook of Nature-Inspired and Innovative Computing,
Conference on Machine Learning, Sydney, Australia, Aug. 6–11, PMLR, pp. Springer, New York, NY, pp. 187–219.
1857–1865. [58] Patel, N. M., Kang, B. -S., Renaud, J. E., and Tovar, A., 2009,
[31] Shah, J. J., Smith, S. M., and Vargas-Hernandez, N., 2003, “Metrics for “Crashworthiness Design Using Topology Optimization,” ASME J. Mech.
Measuring Ideation Effectiveness,” Des. Stud., 24(2), pp. 111–134. Des., 131(6), p. 061013.
[32] Chen, W., and Ahmed, F., 2021, “PaDGAN: Learning to Generate High-Quality [59] Hornby, G. S., Lipson, H., and Pollack, J. B., 2001, “Evolution of Generative
Novel Designs,” ASME J. Mech. Des., 143(3), p. 031703. Design Systems for Modular Physical Robots,” IEEE International Conference
[33] Xu, K., Zhang, G., Liu, S., Fan, Q., Sun, M., Chen, H., Chen, P.-Y., Wang, Y., on Robotics and Automation, Seoul, South Korea, May 21–26.
and Lin, X., 2020, “Adversarial T-shirt! Evading Person Detectors in a [60] Krish, S., 2011, “A Practical Generative Design Method,” Comput.-Aided Des.,
Physical World,” European Conference on Computer Vision, Glasgow, UK, 43(1), pp. 88–100.
Aug. 23–28, Springer, pp. 665–681. [61] Caldas, L. G., and Norford, L. K., 2002, “A Design Optimization Tool Based on a
[34] Cheong, H., and Shu, L., 2014, “Retrieving Causally Related Functions From Genetic Algorithm,” Autom. Constr., 11(2), pp. 173–184.
Natural-Language Text for Biomimetic Design,” ASME J. Mech. Des., 136(8), [62] Lobos, A., 2018, “Finding Balance in Generative Product Design,” Proceedings
p. 081008. of NordDesign 2018, Linköping, Sweden, Aug. 14–17.
[35] Behzadi, M. M., and Ilieş , H. T., 2022, “GANTL: Toward Practical and [63] Burnap, A., Liu, Y., Pan, Y., Lee, H., Gonzalez, R., and Papalambros, P. Y., 2016,
Real-Time Topology Optimization With Conditional Generative Adversarial “Estimating and Exploring the Product Form Design Space Using
Networks and Transfer Learning,” ASME J. Mech. Des., 144(2), p. 021711. Deep Generative Models,” International Design Engineering Technical

Journal of Mechanical Design APRIL 2023, Vol. 145 / 041407-11

 Conferences and Computers and Information in Engineering Conference, [75] He, K., Zhang, X., Ren, S., and Sun, J., 2016, “Deep Residual Learning for Image
Charlotte, NC, Aug. 21–24, Vol. 50107, American Society of Mechanical Recognition,” Proceedings of the IEEE Conference on Computer Vision and
Engineers, Paper No. V02AT03A013. Pattern Recognition, Las Vegas, NV, June 27–30, pp. 770–778.
[64] Yang, Z., Li, X., Catherine Brinson, L., Choudhary, A. N., Chen, W., and [76] Kenton, J. D. M. -W. C., and Toutanova, L. K., 2019, “Bert: Pre-training of Deep
Agrawal, A., 2018, “Microstructural Materials Design Via Deep Adversarial Bidirectional Transformers for Language Understanding,” Proceedings of
Learning Methodology,” ASME J. Mech. Des., 140(11), p. 111416. NAACL-HLT, Minneapolis, MN, June 2–7, pp. 4171–4186.
[65] Valencia-Rosado, L. O., Guzman-Zavaleta, Z. J., and Starostenko, O., 2022, “A [77] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T.,
Modular Generative Approach for Realistic River Deltas: When L-systems and Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito,
Cgans Meet,” IEEE Access, 10(1), pp. 5753–5767. Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and
[66] Huo, Q., Tang, G., and Zhang, F., 2019, “Particle Swarm Optimization Chintala, S., 2019, “Pytorch: An Imperative Style, High-Performance Deep
for Great Enhancement in Semi-supervised Retinal Vessel Segmentation Learning Library,” Advances in Neural Information Processing Systems
With Generative Adversarial Networks,” Machine Learning and Medical (NeurIPS 2019), Vancouver, Canada, Dec. 8–14, pp. 8026–8037.
Engineering for Cardiovascular Health and Intravascular Imaging and [78] Kingma, D. P., and Ba, J., 2015, “Adam: A Method for Stochastic Optimization,”
Computer Assisted Stenting, Shenzhen, China, Oct. 13, Springer, pp. 112– International Conference for Learning Representations, San Diego, CA, May 7–9,
120. p. 13.
[67] Iklima, Z., Adriansyah, A., and Hitimana, S., 2021, “Self-Collision Avoidance of [79] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S., 2017,

Downloaded from http://asmedigitalcollection.asme.org/mechanicaldesign/article-pdf/145/4/041407/6973255/md_145_4_041407.pdf by Yanshan University user on 16 June 2023

Arm Robot Using Generative Adversarial Network and Particles Swarm “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash
Optimization (GAN-PSO),” Sinergi, 25(2), pp. 141–152. Equilibrium,” Annual Conference on Neural Information Processing Systems
[68] Chen, W., and Ahmed, F., 2020, “PADGAN: A Generative Adversarial Network 2017, Long Beach, CA, Dec. 4–9.
for Performance Augmented Diverse Designs,” International Design Engineering [80] Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A., 2012,
Technical Conferences and Computers and Information in Engineering “A Kernel Two-Sample Test,” J. Mach. Learn. Res., 13(1), pp. 723–773.
Conference, Virtual, Online, Aug. 17–19, Vol. 84003, American Society of [81] Wilson, A. G., Hu, Z., Salakhutdinov, R., and Xing, E. P., 2016, “Deep Kernel
Mechanical Engineers, Paper No. V11AT11A010. Learning,” Artificial Intelligence and Statistics, Cadiz, Spain, May 9–11,
[69] Raina, A., McComb, C., and Cagan, J., 2019, “Learning to Design From Humans: PMLR, pp. 370–378.
Imitating Human Designers Through Deep Learning,” ASME J. Mech. Des., [82] Ravuri, S., Mohamed, S., Rosca, M., and Vinyals, O., 2018, “Learning Implicit
141(11), p. 111102. Generative Models With the Method of Learned Moments,” International
[70] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T., 2020, Conference on Machine Learning, Stockholm, Sweden, July 10–15, PMLR,
“Analyzing and Improving the Image Quality of Stylegan,” Proceedings of the pp. 4314–4323.
IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, [83] Raviselvam, S., Hölttä-Otto, K., and Wood, K. L., 2016, “User Extreme
WA, June 13–19, pp. 8110–8119. Conditions to Enhance Designer Empathy and Creativity: Applications Using
[71] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C., 2017, Visual Impairment,” International Design Engineering Technical Conferences
“Improved Training of Wasserstein GANs,” Adv. Neural Inf. Process. Syst., and Computers and Information in Engineering Conference, Charlotte, NC,
30(1), pp. 5769–5779. Aug. 21–24, Vol. 50190, American Society of Mechanical Engineers, Paper
[72] Arjovsky, M., Chintala, S., and Bottou, L., 2017, “Wasserstein Generative No. V007T06A005.
Adversarial Networks,” International Conference on Machine Learning, [84] Chong, L., Raina, A., Goucher-Lambert, K., Kotovsky, K., and Cagan, J., 2023,
Sydney, Australia, Aug. 6–11, PMLR, pp. 214–223. “The Evolution and Impact of Human Confidence in Artificial Intelligence and in
[73] Villani, C., 2016, Optimal Transport: Old and New, Grundlehren der mathema- Themselves on AI-assisted Decision-Making in Design,” ASME J. Mech. Des.,
tischen Wissenschaften, Springer, Berlin/Heidelberg. 145(3), p. 031401.
[74] Han, Y., and Moghaddam, M., 2021, “Analysis of Sentiment Expressions for [85] Nelson, B. A., Wilson, J. O., Rosen, D., and Yen, J., 2009, “Refined Metrics for
User-Centered Design,” Expert Syst. Appl., 171(1), p. 114604. Measuring Ideation Effectiveness,” Des. Stud., 30(6), pp. 737–743.

041407-12 / Vol. 145, APRIL 2023 Transactions of the ASME