Sensitivity Schedule
Tool/Project:
74.798: Analysis 2004-Nov-15 2004- NovPick topics by Oct. 22 Selection and approval Done by Nov. 19 Progress?
Fractional Factorial and Other Orthogonal Designs
1 2
Tool Topics Selected
Student Tool Topic due November 19 Andrew A manager for distributing, running and recording Daniel Maciej
results of simulations to be used in Sensitivity Analysis A manager for distributing, running and recording results of simulations to be used in Sensitivity Analysis A standalone application to analyze results from applying a Balanced Incomplete Block Design to estimate the variance contribution of one [or 2?] parameter at a time. Design of a SimLab-like all-inclusive application SimLaballor API for sensitivity analysis, possibly 4 implementing some features as proof of concept.
Tool Requirements
30 Marks
5 5 5 5 5 5 Ease of use Time saving (in comparison to alternative) Economical use of resources Accuracy, reliability of results Appropriate interfaces Functionality
Tyler
5
Software Tool Presentations
2004-Nov-26 2004- Nov5 minutes each present your tool: what's it good for? presentation not for marks
Final Project
Nov-26: Identify your target project NovDec-03: Hand in project description for 10 marks Decinput, model and output specifications
Dec-17: Hand in final project report DecMarks:
6: Input parameter distributions 6: Experimental design 6: Generation of samples 6: Execution of simulations from samples 6: Analysis of results
6 7
1
Process of Sensitivity Analysis
Elicit Parameter Distn Distribution Parameter Database Sample Description Generate Sample Exp Design Experiment Specification Design Exps Sample Set Model Run Sims Analyze Results SA Results
Fractional Factorial Analysis
Full Factorial Analysis Review
Regression analysis 2 levels, -1 and 1, to minimize number of runs Main effects and interactions Orthogonality of columns
Fractional factorial
Analysis Rules
Retain 2 levels, main effects, interactions, orthogonality Fewer runs, causing aliasing Hadamard matrices
Orthogonal arrays
8 9
Regression Analysis
If we can guess the form of the terms, fijk=aijkgijk, linear coefficients aijk can be ijk ijk ijk ijk estimated by least squares analysis.
f (x1, x 2 ,..., x n ) = f 0 + ai gi (x i ) +
i=1 ijk ijk 1i jkn n
All the Interaction Terms for n=4
In general, 2n terms requiring 2n simulations
0th order 1st order
f0 a1g1(x1) a2g2 (x2) a3g3 (x3) a4g4 (x4)
2nd order
a12g12(x1 ,x2) a13g13 (x1 ,x3) a14g14 (x1 ,x4) a23g23 (x2 ,x3) a24g24 (x2 ,x4) a34g34 (x3 ,x4)
a g (x , x )+
ij ij i j 1i jn i j k
a
g
(x , x , x )+ ...
3rd order
a123g123(x1 ,x2 ,x3) a124g124 (x1 ,x2 ,x4) a134g134 (x1 ,x3 ,x4) a234g234 (x2 ,x3 ,x4)
By performing an ANOVA, we can determine which terms are significant.
10
4th order
a1234g1234 (x1 ,x2 ,x3 ,x4)
11
Factorial Design on 2 Levels
Traditionally the 2 levels are called -1 and 1 (or and +) These levels are then mapped to the interval [0,1]: e.g., -1 goes to 0.05 and 1 goes to 0.95 (why not to 0 and 1?) These p-values are then mapped through the inverse cdf transform:
-1 goes to 0.05 goes to F-1(0.05) 1 goes to 0.95 goes to F-1(0.95)
12
Sample Dataset for Factorial Analysis Pattern? [Gray Codes]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 x1 + + + + + + + + x2 + + + + + + + + x3 + + + + + + + + x4 + + + + + + + + -
13
2
Analysis of Factorial Designs
Main effect of parameter xi:
Average outcome when xi is high average outcome when xi is low
Analysis of Factorial Designs
To calculate the main and interaction effects:
add up the simulation results with + coefficients subtract off the simulation results with coefficients divide result by n/2
Interaction effect of parameters xi, xj, :
Average outcome when product xi xj is high average outcome when xi xj is low
The table on the next page shows coefficients Note that with a 2-level design, it is not 2necessary to assume any functional forms other than simple products
14
15
Coefficients for 2-Level Factorial Analysis 21 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 x1 + + + + + + + + x2 + + + + + + + + x3 + + + + + + + + x4 + + + + + + + + x2 + + + + + + + + x 1x 2 + + + + + + + + x2x3x4 + + + + + + + + x1x2x3x4 + + + + + + + + - 16
Orthogonality of Coefficients
If you take the dot product of any two different vectors of coefficients, the result is 0 E.g.,
(coefficients of (coefficients x1x2) of x2x3x4) = (1 1 1 1 1 1 1 1 ) ( 1 1 1 1 1 1 1 1 ) (coefficients of x1x2 x2x3x4= x1x3x4) = 0
Therefore, effects of each combination can be determined independently
17
Fractional Factorial Designs
Take a small fraction of the original factorial design
reduces the number of runs required but no longer possible to estimate all effects
Fractional Factorial Designs
Which effects should be targeted?
n main effects, obviously ~n 2-factor interactions?
e.g., all the 2-factor interactions with a particular 2parameter, say x25 e.g., all the 2-factor interactions among a set of 2parameters with large main effects
Which effects should be targetted? targetted?
n main effects, obviously ~n2/2 2-factor interactions? for n = 100, requires 5000 runs
still too many!
18
These effects can be estimated with about 2n simulations
e.g., for n = 100, about 200 runs
19
3
Fractional Factorial Designs
What happens to the rest of the interaction effects?
with 2n runs, other interaction effects will have an influence if present interpreted to be caused by one of the targeted effects a source of error
x1 1 1 1 1 1 1 1 1
20
Sample FF Analysis
x2 1 1 1 1 1 1 1 1 0 x3 1 1 1 1 1 1 1 1 0 x4 1 1 1 1 1 1 1 1 0 7+3x1x2 x3 7+x1 (3 x4) 7+x2 x3(3x41) 7+3x (3 (3x 9 11 11 9 3 5 5 3 Main Effects 9 11 11 9 3 5 5 3 9 11 11 9 3 5 5 3
21
3
Sample FF Analysis
Note that in any simulation x1 x2 x3x4=1 This is a Resolution IV design
MainEffect(x1) 3 MainEffect(x MainEffect(x2) 0 MainEffect(x MainEffect(x3) 0 MainEffect(x MainEffect(x4) 0 MainEffect(x InteractionEffect(x1 x2) 0 InteractionEffect(x InteractionEffect(x1 x3) 0 InteractionEffect(x InteractionEffect(x1 x4) 1 InteractionEffect(x Overall average 7 InteractionEffect(x2 x3x4) InteractionEffect(x InteractionEffect(x1 x3x4) InteractionEffect(x InteractionEffect(x1 x2x4) InteractionEffect(x InteractionEffect(x1 x2x3) InteractionEffect(x InteractionEffect(x3 x4) InteractionEffect(x InteractionEffect(x2 x4) InteractionEffect(x InteractionEffect(x2 x3) InteractionEffect(x InteractionEffect(x1 x2 x3x4) InteractionEffect(x
22
Generating FF Designs: Hadamard Matrices
Hadamard matrix: a matrix H of 1's and matrix: 1's such that Ht H=nI, a diagonal matrix H=n e.g.
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
23
Generating FF Designs: Hadamard Matrices
The last 3 columns of this Hadamard Matrix make a Resolution III design (x2 x3x4=1). (x =1). e.g. 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
24
Generating FF Designs: Hadamard Matrices
Mirror the Hadamard Matrix to make a Resolution IV design having one more column and twice the runs (x1 x2 x3x4=1) Notice the ...