Design Evaluation Methods

C h r i s t o p h e r S a l d a n a , P h . D.
Wo o d r u ff S c h o o l o f M e c h a n i ca l E n g i n e e r i n g
G e o r g i a I n s t i t u t e o f Te c h n o l o g y
Atlanta, Georgia USA
Learning Objectives

Understand the importance of evaluation in the design process

Identify criteria for evaluation
Utilize three levels of evaluation matrices for product
development through concept screening and scoring
Strategies for Visual/Descriptive Communication of Designs

2
Design Tools – Current Progress
Problem Understanding
House of Quality Specification Sheet Function Tree
Place Mass
on Target

Generate Transmit Hit Brake on

Power Power Target Target

Move Navigate
to to
Target Target

Concept Generation
Design Alternatives
Morphological Chart
Generate
Gravity Mouse Traps
Power

Transmit
Power
Car Hit by Trap Rip Cord Effect Ramp Catapult

Brake on
Target
Friction String Break Anchor Rubber Stopper Weighted Skid

Move to
Target
Rolling Sliding Projectile Launch

Navigate
to Target
Equal Size Wheels Larger Front Wheels

33
4
5
6
Example: Insulin Pen
Redesigning the Insulin Pen
Customer Wants
1. Dose metering accuracy
2. Portability
3. Durability
4. Ease of handling
5. Readability of settings
6. Ease of use
7. Ease of manufacture

47
Example: Insulin Pen
Redesigning the Insulin Pen

58
Example: Insulin Pen
Redesigning the Insulin Pen

69
Structured Evaluation
Benefits
• Customer-focused product, competitive design
• Better product-process coordination, faster product introduction
• Effective group decision-making
• Documentation of design process
Challenge
• Need to make informed decisions despite lack of information
• Selection requires estimation, analysis, prototyping
• Identify bad concepts versus picking optimal ones
10
Structured Evaluation

How do we evaluate each of these designs?

What criteria do we use for evaluation?

What is the best design?

11
Concept Selection Strategies

Multi-voting: team members vote

independently, work together to resolve
differences and/or average results
Strengths/Weaknesses: list strengths and
weaknesses of design concepts, use this to
evaluate based on specific opinions
Prototype and Test: build or simulate, use
empirical or simulated test data!

12
Concept Generation/Selection
Creating, screening, scoring alternatives

13
Stages and Types of Concept Selection
Concept screening
• First-level evaluation matrix
Concept scoring
• Second-level evaluation matrix
• Third-level evaluation matrix

14
First Level Evaluation (Pugh Matrix)
1. Identify the criteria for comparison.
2. Select the alternatives to be compared.
• Alternatives are developed during concept generation.
• All concepts should be compared at the same level of abstraction.
3. Generate scores.
• Use a design concept as datum, with all the other being compared to it
• Evaluate each alternative as better (+), same (S), or worse (-) relative to datum.
4. Compute the total score
• Sum the total number of (+)’s, (-)’s, (S)’s
• Compute overall score with +1 for (+)’s, -1 for (-)’s, 0 for (S)’s
5. Note: other variations on scoring in the first-level evaluation
• Optional scale: +3 if extremely better than datum +2, +1, 0, -1, -2, -3

15
First Level Evaluation (Pugh Matrix)

+ = better than datum; - = worse than datum; S = same as datum

16
First Level Evaluation Matrix (example)

+ = better than datum; - = worse than datum; S = same as datum 17

First-Level Evaluation Matrix
• Ranking depends on choice of datum
• Does not factor in how much better a specific alternative is
compared to others
• Some criteria may be more important
• Consider: (i) no datum, (ii) numerical rating, (iii) criteria weighting
• Second-level evaluation matrix – (i), (ii)
• Third-level evaluation matrix – (i), (ii) and (iii)

18
Second Level Evaluation Matrix Common Scale:
4 = very good (ideal)
3 = good
2 = adequate
1 = just tolerable
0 = unsatisfactory

Alternate Scale:
10 = ideal solution
9 = solution exceeds requirement
8 = very good solution
7 = good solution
6 = good solution with drawbacks
5 = satisfactory solution
4 = adequate solution
3 = tolerable solution
2 = weak solution
1 = very inadequate solution
0 = useless solution

19
Second Level Evaluation Matrix

20
Third Level Evaluation Matrix

21
Third Level Evaluation Matrix

22
Summary - Evaluation Matrix
Elements
• Designs rated relative to customer
requirements in HOQ
• Level 1: sum +/-/S relative to datum
• Level 2: numerical rating
• Level 3: weighted sum of num. rating

Describing this figure in text

• Which design performed best? Why?
• Which performed worst? Why?
• Use numerical information from figure
23
Concept Selection Pitfalls
• Not doing it
• Running with the first idea
• Forgetting the customer
• Evaluation matrix doesn't correspond to HOQ
• Letting an "experienced" designer make the choices
• Going by gut feel
• Letting a manager decide
• Not buying into the process as a team
• Ignoring cost

24
Design Tools – Complete Process
Problem Understanding
House of Quality Specification Sheet Function Tree
Place Mass
on Target

Generate Transmit Hit Brake on

Power Power Target Target

Move Navigate
to to
Target Target

Concept Generation Concept Selection

Design Alts. Final Design
Morphological Chart Evaluation Matrices
Generate
Gravity Mouse Traps
Power

Transmit
Power
Car Hit by Trap Rip Cord Effect Ramp Catapult

Brake on
Target
Friction String Break Anchor Rubber Stopper Weighted Skid

Move to
Target
Rolling Sliding Projectile Launch

Navigate
to Target
Equal Size Wheels Larger Front Wheels

25
26
Detailed Design - Communication
General organization
• Primary systems and subsystems
• Mechanisms, operation/sequencing,
construction and materials
• Performance relative to specifications
Clarity in written descriptions
• Be clear in describing design features. Match
words in the body to label text in figures.
• Avoid describing things that are not shown with
evidence or detail. Don’t rely on the reader’s
imagination.
26
226
27
Detailed Design - CAD
Important CAD elements:
Labels match text explanations
Mechatronics
Common COTS components (fasteners, etc.)
Detail views
Dimensions
Formatting (text, resolution, annotations)

27
24
Evaluation Tools Summary

Understand the importance of evaluation in the design process

Identify criteria for evaluation
Utilize three levels of evaluation matrices for product
development (First-level, Second-level, Third-level)
Strategies for Visual/Descriptive Communication of Designs

28