This preview shows page 1. Sign up to view the full content.
Unformatted text preview: ESD.77 – Multidisciplinary System Design Optimization Robust Design
main
effects twofactor interactions
nk Run a resolution III
on noise factors
Change
one factor Again, run a resolution III on
noise factors. If there is an
improvement, in transmitted
variance, retain the change k b
k n k
2 a c b
b a c a c If the response gets worse,
go back to the previous state B
k 1
2 C b
A a c Stop after you’ve changed
every factor once Dan Frey
Associate Professor of Mechanical Engineering and Engineering Systems Research Overview
Outreach
to K12 Concept
Design Adaptive Experimentation
and Robust Design
n Pr Complex
Systems 12 x1 x 2 0 12 ij 1 n
2 erf
0 C B D twofactor interactions
AB AC AD BC BD CD threefactor interactions
ABC ABD
ABCD ACD 1 x1
2 INT 2 x2 2
2 INT e
2 INT x12 1 ME ( n 2) 2 2
ME 2
INT (n 2) 1
2 2
INT 1
2 2 dx2dx1
2 Methodology
Validation main effects A x2 2 BCD fourfactor interactions Outline
• Introduction
– History
– Motivation • Recent research
– Adaptive experimentation
– Robust design “An experiment is simply a question put to nature …
The chief requirement is simplicity: only one question
should be asked at a time.”
Russell, E. J., 1926, “Field experiments: How they are made and what
they are,” Journal of the Ministry of Agriculture 32:9891001. “To call in the statistician after the
experiment is done may be no more
than asking him to perform a postmortem examination: he may be able
to say what the experiment died of.”
 Fisher, R. A., Indian Statistical Congress, Sankhya, 1938. Estimation of Factor Effects
(bc) Say the independent
i
l
f
experimental error of
observations
(a), (ab),
(a) (ab) et cetera is σε. (b) (ab) + B
We define the main effect
estimate Α to be (abc)  (1) (a)
A
+
1
A ≡ [(abc) + (ab) + (ac) + (a ) − (b) − (c) − (bc) − (1)]
4
The standard deviation of the estimate is 1
1
σA =
2σ ε
8σ ε =
4
2 (ac) (c) +
 C  A factor of two improvement in
efficiency as compared to
“single question methods” Fractional Factorial Experiments
“It will sometimes be advantageous
deliberately to sacrifice all possibility of
obtaining information on some points, these
being confidently believed to be unimportant
… These comparisons to be sacrificed will be
deliberately confounded with certain elements
of the soil heterogeneity… Some additional
care should, however, be taken…”
Fisher, R. A., 1926, “The Arrangement of Field Experiments,”
Journal of the Ministry of Agriculture of Great Britain, 33: 503513. Fractional Factorial Experiments +
B
+
 C  A + 2 3 1
III Fractional Factorial Experiments
Trial
1
2
3
4
5
6
7
8 A
1
1
1
1
+1
+1
+1
+1 B
1
1
+1
+1
1
1
+1
+1 C
1
1
+1
+1
+1
+1
1
1 D
1
+1
1
+1
1
+1
1
+1 E
1
+1
1
+1
+1
1
+1
1 F
1
+1
+1
1
1
+1
+1
1 G
1
+1
+1
1
+1
1
1
+1 FG=A
+1
+1
+1
+1
1
1
1
1 274 Design (aka “orthogonal array”)
Every factor is at each level an equal number of times (balance).
High replication numbers provide precision in effect estimation.
Resolution III. Robust Parameter Design
Robust Parameter Design … is a statistical /
engineering methodology that aims at reducing
the performance variation of a system (i.e. a
product or process) by choosing the setting of
its control factors to make it less sensitive to
noise variation. Wu, C. F. J. and M. Hamada, 2000, Experiments: Planning, Analysis, and
Parameter Design Optimization, John Wiley & Sons, NY. Cross (or Product) Arrays
2 Noise Factors
Control Factors 2 7 4
III 1
2
3
4
5
6
7
8 A
1
1
1
1
+1
+1
+1
+1 B
1
1
+1
+1
1
1
+1
+1 C
1
1
+1
+1
+1
+1
1
1 D
1
+1
1
+1
1
+1
1
+1 E
1
+1
1
+1
+1
1
+1
1 F
1
+1
+1
1
1
+1
+1
1 G
1
+1
+1
1
+1
1
1
+1 a
b
c 1
1
1 3 1
III 1 +1
+1 1
+1 +1 2 7 4
III Taguchi, G., 1976, System of Experimental Design. 2 +1
+1
1 3 1
III Step 1
Identify Project
and Team Step 2 Step 4 Summary:
• Determine control factor levels
• Calculate the DOF
• Determine if there are any interactions
• Select the appropriate orthogonal array Formulate
Engineered
System: Ideal
Function / Quality
Characteristic(s) Step 3
Formulate
Engineered
System:
Parameters Step 5 Sum
Step 1 Summary: 5
Step
• Form cross function team of experts. Determin
•
• Clearly define project objective.
• Determin
• Define Assign Noise
roles and responsibilities to team Surro
 memb
• Translate customerOuter
Comp
Factors to intent nontechnical terms
• Identify product quality issues.
 Treat
Array conditions and•describe th
• Isolate the boundary
Establish Step 2 Summary: 6
Step 6 Sum
Step
• Select a response function(s).
• Cross fun
• Select a signal parameter(s).
logistical a
Conduct
• Determine if problem is static or dynamic param
phase of th
and one or more responses. Dynamic has multip
Experiment and
• Identify F
signals.
• Determin
Collect Data
• Determine the S/N function. See section Step Step 3 Summary:
Step 7
• Select control factor(s).
• Rank control factors.
• Select noise factors(s).and
Analyze Data Select Optimal
Design Step 7 Sum
• Calculate
•
•
•
•
• • Interpret
•
•
• Step 4
Assign Control
Factors to Inner
Array Step 4 Summary:
Step 8 Sum
• DetermineStep factor levels.
control 8
• TBD  Ca
• Calculate the DOF.
• Determine if there are any interactions between
Predict and
• Select the appropriate Orthogonal Array. Confirm Majority View on “One at a Time”
One way of thinking of the great advances of the
science of experimentation in this century is as
the final demise of the “one factor at a time”
method, although it should be said that there are
still organizations which have never heard of
factorial experimentation and use up many man
hours wandering a crooked path. Logothetis, N., and Wynn, H.P., 1994, Quality Through Design: Experimental
Design, Offline Quality Control and Taguchi’s Contributions, Clarendon
Press, Oxford. Minority Views on “One at a Time”
“…the factorial design has certain deficiencies … It devotes
observations to exploring regions that may be of no
interest…These deficiencies … suggest that an efficient
design for the present purpose ought to be sequential;
that is, ought to adjust the experimental program at
each stage in light of the results of prior stages.”
Friedman, Milton, and L. J. Savage, 1947, “Planning Experiments
Seeking Maxima”, in Techniques of Statistical Analysis, pp. 365372. “Some scientists do their experimental work in single steps.
They hope to learn something from each run … they see
and react to data more rapidly …If he has in fact found out
a good deal by his methods, it must be true that the effects
are at least three or four times his average random error
per trial.” Cuthbert Daniel, 1973, “OneataTime Plans”, Journal of the
American Statistical Association, vol. 68, no. 342, pp. 353360. My Observations of Industry
• Farming equipent company has reliability problems
• Large blocks of robustness experiments had been
planned at outset of the design work
• More than 50% were not finished
• Reasons given
– Unforseen changes
– Resource pressure
– Satisficing “Well, in the third experiment, we
found a solution that met all our
needs, so we cancelled the rest
of the experiments and moved on
to other tasks…” More Observations of Industry
•
•
•
• Time for design (concept to market) is going down
Fewer physical experiments are being conducted
Greater reliance on computation / CAE
Poor answers in computer modeling are common
– Right model → Inaccurate answer
– Right model → No answer whatsoever
– Notso right model → Inaccurate answer
• Unmodeled effects
• Bugs in coding the model Human Subjects Experiment
• Hypothesis: Engineers using a flawed simulation
are more likely to detect the flaw while using
OFAT than while using a more complex design. • Method: Betweensubjects experiment with
human subjects (engineers) performing parameter
design with OFAT vs. designed experiment. Treatment: Design Space Sampling Method
Trial A B C D E F G 1        2 +       +      +     +      + • Adaptive OFAT  3
4
5 filled in as
required to adapt 6 + 7 8 – One factor changes in
each trial + Trial A B C D E F G 1        2 +  +   + + 3 + + +  +   4  +   + + + 5 +   + +  + 6   + + + +  7  + + +   + 8 + +  +  + • PlackettBurman L8  – Four factors change
between any two trials Using a 27 system avoids possible confounding factor of number of trials.
Increasing number of factors likely means increasing discrepancy in detection.
Larger effect sizes require fewer test subjects for given Type I and II errors. Parameter Design of Catapult to Hit Target
• Modeled on XPultTM
• Commonly used in
DOE demonstrations
• Extended to 27 system
by introducing
– Arm material
– Air relative humidity
– Air temperature Control Factors and Settings
Control Factor Nominal Setting Alternate Setting Relative Humidity 25% 75% Pullback 30 degrees 40 degrees Type of Ball Orange (Largeball TT) White (regulation TT) Arm Material Magnesium Aluminum Launch Angle 60 degrees 45 degrees Rubber Bands 3 2 Ambient
Temperature 72 F 32 F 50
45 80 35
30 60
Effect Size 25 Cumulative Percent 20 40 15
10 20 5 id
it y
em
pe
ra
tu
re al
l Am bi
en
tT e Hu
m of
B
at
iv
Re
l Ty
pe An
gl
e at
er
ia
l La
un
ch M Pu
llb
ac
k 0 Ar
m of
R ub
be
rB
an
ds 0 No
. • Arm material selected for its
moderate effect size
• Computer simulation equations
are “correct”, but intentional
mistake is that arm material
properties are reversed 100 40 • Control factor tied directly to
simulation mistake • Control factor ordering is not
random, to prevent variance due
to learning effect
• “Bad” control factor placed in 4th
column in both designs Results of Human Subjects Experiment • Pilot with N = 8
• Study with N = 55 (1 withdrawal)
• External validity high – 50 full time engineers and 5 engineering students
– experience ranged from 6 mo. to 40+ yr. • Outcome measured by subject debriefing at end Method Detected Not detected Detection Rate (95% CI) OFAT 14 13 (0.3195,0.7133) PBL8 1 26 (0.0009,0.1897) Adaptive OFAT Experimentation
Do an experiment
If there is an improvement,
retain the change Change one factor If the response gets worse, go
back to the previous state +
B +
 A +  C Stop after you’ve changed
every factor Frey, D. D., F. Engelhardt, and E. Greitzer, 2003, “A Role for One Factor at a Time
Experimentation in Parameter Design”, Research in Engineering Design 14(2): 6574. Empirical Evaluation of
Adaptive OFAT Experimentation
• Metaanalysis of 66 responses from
published, full factorial data sets
• When experimental error is <25% of the
combined factor effects OR interactions
are >25% of the combined factor
effects, adaptive OFAT provides more
improvement on average than fractional
factorial DOE.
Frey, D. D., F. Engelhardt, and E. Greitzer, 2003, “A Role for One Factor at a Time
Experimentation in Parameter Design”, Research in Engineering Design 14(2): 6574. Detailed Results
0.4 MS FE 0.1 MS FE OFAT/FF Interaction
Strength Gray if OFAT>FF 0
Mild
100/99
Moderate 96/90
Strong
86/67
Dominant 80/39 0.1
99/98
95/90
85/64
79/36 0.2
98/98
93/89
82/62
77/34 Strength of Experimental Error
0.3
0.4
0.5
0.6
0.7
96/96 94/94 89/92 86/88 81/86
90/88 86/86 83/84 80/81 76/81
79/63 77/63 72/64 71/63 67/61
75/37 72/37 70/35 69/35 64/34 0.8
77/82
72/77
64/58
63/31 0.9
73/79
69/74
62/55
61/35 1
69/75
64/70
56/50
59/35 A M th
Mathematical Model of Ad ti OFAT
ti l M d l f Adaptive
O0 = y (~1 , ~2 ,K ~n )
x x
x initial observation
observation with
first factor toggled
first factor set O1 = y (− ~1 , ~2 ,K ~n )
x x
x x1∗ = ~1sign{O0 − O1}
x for i = 2 K n repeat for all
tf
ll
remaining factors ( Oi = y x1∗ ,K x ∗−1 ,− ~i , ~i +1 ,K ~n
x x
x
i ) xi∗ = ~i sign{max(O0 , O1 ,KOi −1 ) − Oi }
x [ ∗
∗
process ends after n+1 observations with E y (x1∗ , x2 ,K xn ) Frey, D. D., and H. Wang, 2006, “Adaptive OneFactorataTime Experimentation
and Expected Value of Improvement”, Technometrics 48(3):41831. A Mathematical Model of a
Population of Engineering Systems
n −1 n n y ( x1 , x2 ,K xn ) = ∑ β i xi + ∑ ∑ β ij xi x j + ε k
i =1 system
response ( β i ~ Ν 0, σ ME
main effects ymax ≡ 2 ) i =1 j =i +1 ( β ij ~ Ν 0, σ INT 2 ) ( ε k ~ Ν 0,σ ε twofactor interactions 2 ) experimental
error the largest response within the space
g
p
p
of discrete, coded, twolevel factors xi ∈ {− 1,+1} Model adapted from Chipman, H., M. Hamada, and C. F. J. Wu, 2001, “A Bayesian Variable Selection
Approach for Analyzing Designed Experiments with Complex Aliasing”, Technometrics 39(4)372381. Probability of Exploiting an Effect
• The ith main effect is said to be “exploited” if
xi* 0
i
• The twofactor interaction between the ith and
jth factors is said to be “exploited” if
xx 0
ij i j • The probabilities and conditional probabilities
of exploiting effects provide insight into the
mechanisms by which a method provides
improvements The Expected Value of the Response
p
p
after the First Step [ [ *
∗
∗
E ( y ( x1 , ~2 , K , ~n )) = E β 1 x1 + ( n − 1) E β 1 j x1 ~ j
x
x
x [ Eβ x =
∗
1 1 π [ x
E β 1 j x ~n = 2
σ ME 2 2
2
σ ME + ( n − 1)σ INT ∗
1 1
+ σ ε2
2 2
σ INT 2 π 1
2 2
2
σ ME + ( n − 1)σ INT + σ ε2 1 main
effects [ ∗
E β1 j x1 ~n
x Legend 0.6 σ ε σ ME = 0.1
σ ε σ ME = 1 Theorem 1 Theorem 1
Simulation Theorem 1
Simulation × 0.8 σ ε σ ME = 10 + Simulation P [ E β 1 x1∗ twofactor interactions 0.4 0.2 0 n=7 0 0.25 0.5 σ INT σ ME 0.75 1 Probability f E l iti th Fi t Main Effect
P b bilit of Exploiting the First M i Eff t
1 1
Pr (β x > 0) = + sin −1
2 π σ ME *
1 1 σ ME + ( n − 1)σ INT
2 2 1 2
+ σε
2 1 Legend
0.9
09 If interactions are
small and error is
not too large,
OFAT will tend to
exploit main
effects σ ε σ ME = 0.1 Theorem 2 1
Theorem × Theorem2 1
Theorem
Simulation σ ε σ ME = 1 Simulation
Theorem 2
Theorem 1 + Simulation 0.8 σ ε σ ME = 10 0.7 0.6 0.5 0 σ INT σ ME 0.25 0.5 0.75 1 The Expected Value of the Response After
the Second Step [ [ [ *
*
*
*
* *
E ( y ( x1 , x2 , ~3 ,K, ~n )) = 2 E β1 x1 + 2(n − 2) E β1 j x1 + E β12 x1 x2
x
x ⎡
⎢
2
σ INT
2⎢
∗ ∗
E β 12 x1 x 2 =
π⎢ 2
σ ε2
2
⎢ σ ME + ( n − 1)σ INT +
2
⎣ [ ⎤
⎥
⎥
⎥
⎥
⎦ 1 main
effects [ [ [ ∗
E β x ~ j = E β 2 j x2 ~ j
x
x
∗
1j 1 Legend × 0.8 ∗
= E β 2 x2 Theorem 3 1
Theorem Simulation
Theorem 3 1
Theorem Simulation σ ε σ ME = 0.1
σ ε σ ME = 1 0.6 + P [ ∗
E β1 x1 twofactor interactions Theorem 3 1
Theorem Simulation σ ε σ ME = 10 0.4 [ ∗ ∗
E β 12 x1 x 2 0.2 0 0 0.25 0.5 σ INT σ ME 0.75 1 Probability of Exploiting the First Interaction
( ) Pr β12 x 1∗ x ∗ > 0 β12 > β ij >
2 ( σ INT ) 1 1
Pr β x x > 0 = + tan −1
2 π
∗ ∗
12 1 2 1
2 2
2
σ ME + (n − 2)σ INT + σ ε2 ∞ ∞
1 ⎛n⎞
⎜ ⎟
π ⎜ 2 ⎟ ∫ −∫2
⎝ ⎠0 x ⎡ ⎛ 1 x1
⎢erf ⎜
⎜
⎣ ⎝ 2 σ INT ⎞⎤
⎟⎥
⎟
⎠⎦ − x12 ⎛n⎞
⎜ ⎟ −1
⎜2⎟
⎝ ⎠ 2σ INT e 2 + − x2 2
1
⎞
⎛
2 ⎜ σ ME 2 + ( n − 2 )σ INT 2 + σ ε 2 ⎟
2
⎝
⎠ σ INT σ ME + ( n − 2)σ INT
2 2 dx2 dx1 1 2
+ σε
2 Legend
1 Theorem6 1
Theorem
× Simulation σ ε σ ME = 0.1 1 Legend σ ε σ ME = 1 Simulation 0.8 Theorem 6 1
Theorem σ ε σ ME = 0.1 Theorem 5 1
Theorem × 0.9 Theorem51
Theorem
Simulation Theorem
Theorem51 + Simulation Simulation 0.9 + Theorem 6
Theorem 1
Simulation σ ε σ ME = 1
σ ε σ ME = 10 0.8 σ ε σ ME = 10 0.7 0.7 0.6 0.6 0.5 0 0.25 0.5 σ INT σ ME 0.75 1 0.5 0 0.25 0.5 σ INT σ ME 0.75 1 And it Continues
twofactor interactions
nk *
*
*
E( y( x1 , x2 ,, xk , ~k 1 ,, ~n )) / ymax
x
x main
effects k
k n k
2
k 1
2 1 Legend
Eqn. 20 0.8 0 . 25 ME Simulation INT 0.5 ME 0.6 0.4 0.2 0 0 1 2 3 4 5 6 7 k Pr x xj ij i 0 Pr x x2 12 1 0 We can prove that the probability of exploiting interactions is sustained.
Further we can now prove exploitation probability is a function of j only
and increases monotonically. Final Outcome
1 Legend 1 Theorem 1
Eqn 21
Simulation *
*
*
E( y( x1 , x2 ,, xn )) / ymax 0.8 ME ME ME Simulation
Eqn 21
Theorem 1
Simulation Eqn. 20 1
Theorem
0.4 Theorem 1
Eqn.20
Simulation ME 0.1 ME 1 ME 10 Eqn.20 1
Theorem Simulation 0.2 0 10 0.6 Legend
Simulation 1 Eqn 21
Theorem 1 0.8 0.6 0.1 0.4 0.2 0 0.2 0.4 0.6 INT ME Adaptive OFAT 0.8 1 0 0 0.2 0.4 INT 0.6 0.8 ME Resolution III Design 1 Final Outcome
1 0.8 ME 1
0.6 0.4 0.2 0 0 ~0.25 0 0.2 0.4 INT Adaptive OFAT 0.6 0.8 ME Resolution III Design 1 Adaptive “One Factor at a Time” for
Robust Design
Run a resolution III
on noise factors
Change
one factor
b a c b
b a c a c Again, run a resolution III on
noise factors. If there is an
improvement, in transmitted
variance, retain the change If the response gets worse,
go back to the previous state B
C b
A a c Stop after you’ve changed
every factor once Frey, D. D., and N. Sudarsanam, 2007, “An Adaptive Onefactoratatime Method for Robust Parameter Design:
Comparison with Crossed Arrays via Case Studies,” accepted to ASME Journal of Mechanical Design. Sheet Metal Spinning Image by MIT OpenCourseWare. Smallerthebetter signal to noise ratio (dB) Results for Three Methods of Robust Design Applied to
the Sheet Metal Spinning Model
4
Maximum S/N 5
aOFAT x 2 3_1 Informed 6
7
8
9
10 2 6_3 3_1 x2 3_1 Largest noise
effect aOFAT x 2 Largest control
effect Random 11
12 Average S/N 0 0.5 1 1.5 2 2.5 3 2 Standard deviation of pure experimental error (mm )
Image by MIT OpenCourseWare. Paper Airplane
Results for Three Methods of Robust Design Applied to
the Paper Airplane Physical Experiment MIT Design of Experiments Exercise v2.0
B1 (up)
Parameter B: B2 (flat)
Stabilizer Flaps B3 (down) _
___
___ ____
#_
ent _____ ___
erim __
__
Exp ance _ _____
t
Dis e ____
Nam
C:
ter
ame gth
Par se Len
No Maximum signal to noise ratio
41 _ aOFAT x 23 1 Informed Parameter D:
Wing Angle
D1
D2
D3
D1
D2
D3 40 D1
D2
D3
Parameter A:
A1 Weight Position A2 A3 39
_ 3_1 aOFAT x 23 1 Random L9 x 2
38 Average signaltonoise ratio 0
Expt.
# Weight.
A Stabiliz.
B Nose
C Wing
D 1 A1 B1 C1 A1 B2 C2 A1 B3 C3 A2 B1 C2 A2 B2 C3 A2 B3 C1 A3 B1 C3 D2 8 A3 B2 C1 D3 9 A3 B3 C2 Standard deviation of pure experimental error (inches) D2 7 50 D1 6 40 D3 5 30 D3 4 20 D2 3 10 D1 2 Largest control factor effect Combined effects of noise 37 D1 Image by MIT OpenCourseWare. Results Across Four Case studies
Method used
_
Fractional array x k p
Fractional array X 2 kIII p
III _
aOFAT x 2k p
III
Informed Random Sheet metal Low ε 51% 75% 56% Spinning High ε 36% 57% 52% Low ε 99% 99% 98% High ε 98% 88% 87% Low ε 43% 81% 68% High ε 41% 68% 51% Low ε 94% 100% 100% High ε 88% 85% 85% Low ε 74% 91% 84% High ε 66% 70% 64% Low ε 43% to 99% 75% to 100% 56% to 100% High ε 36% to 88% 57% to 88% 51% to 87% Op amp Paper airplane Freight transport Mean of four cases Range of four cases Image by MIT OpenCourseWare. Frey, D. D., N. and Sudarsanam, 2006, “An Adaptive Onefactoratatime Method for
Robust Parameter Design: Comparison with Crossed Arrays via Case Studies,” accepted to
ASME Journal of Mechanical Design. Ensembles of aOFATs
100 100  Ensemble aOFATs (4)
 Fractional Factorial 272 90  Ensemble aOFATs (8)
 Fractional Factorial 271 90 80 80 70 70 60 60 50 50 Expected Value of Largest Control Factor = 16 Expected Value of Largest Control Factor = 16 40 40
0 5 10 15 20 25 Comparing an Ensemble of 4 aOFATs with a 272
Fractional Factorial array using the HPM 0 5 10 15 20 25 Comparing an Ensemble of 8 aOFATs with a 271
Fractional Factorial array using the HPM Conclusions
• A new model and theorems show that – Adaptive OFAT plans exploit twofactor
interactions especially when they are large
– Adaptive OFAT plans provide around 80%
of the benefits achievable via parameter
design • Adaptive OFAT can be “crossed” with
factorial designs which proves to be
highly effective Frey, D. D., and N. Sudarsanam, 2007, “An Adaptive Onefactoratatime Method for Robust Parameter Design:
Comparison with Crossed Arrays via Case Studies,” accepted to ASME Journal of Mechanical Design. MIT OpenCourseWare
http://ocw.mit.edu ESD.77 / 16.888 Multidisciplinary System Design Optimization
Spring 2010 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms. ...
View Full
Document
 Fall '03
 lDavidMiller

Click to edit the document details