Analysis Statistical of Output from Terminating Simulations
Chapter 6
Last revision December 17, 2006
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Slide 1 of 23
What We'll Do ...
Talk about
Exam I Homework #7 Project #2
Presentations
Review comparing two scenarios Comparing many scenarios via the Arena Process Analyzer (PAN) Searching for an optimal scenario with OptQuest
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Slide 2 of 23
Overall Result for Exam I
ISyE 320 Mid-term Histogram 8 7 6 5 4 3 2 1 0 73 76 79 82 85 88 91 94 97 More
Slide 3 of 23
Average: 86.35 Median: 87.50 Max: 99
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Projected Letter Grades...
This is ONLY AN APPROXIMATION
Based on scores so far and weights on syllabus We may not disclose this information Actually this has been increased due to the exam grade adjustment!
10 8 6 4 2 0
A
AB
B
BC
C
D
F
Slide 4 of 23
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
What's Coming...
Four more homeworks
Due on April 2, April 9, April 16, April 23 Individual group advices going on Extra Awards Please email Nana by the end of this week if you have any preferences Count 7.5% of the class grade Evaluated by Peer Students!!! Extra Awards May 7
Project Part II (Due 4/30)
Project Presentation (April 21, 23, 28)
Exam II
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Slide 5 of 23
Compare Means via Output Analyzer
(cont'd.)
Output Analyzer is a separate application that operates on .dat files produced by Arena
Launch separately from Windows, not from Arena
To save output values (Expressions) of entries in Statistic data module (Type = Output) enter filename.dat in Output File column
Did for both Total Cost and Percent Rejected Will overwrite these file names next time
Either change the names here or out in the operating system before the next run
.dat files are binary ... can only be read by Output Analyzer
Slide 6 of 23
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Compare Means via Output Analyzer
(cont'd.)
Start Output Analyzer, open a new data group
Basically, a list of .dat files of current interest Can save data group for later use .dgr file extension Add button to select (Open) .dat files for the data group Add data files ... "A" and "B" for the two scenarios Select "Lumped" for Replications field Title, confidence level, accept Paired-t Test, do not Scale Display since two output performance measures have different units
Analyze > Compare Means menu option
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Slide 7 of 23
Compare Means via Output Analyzer
(cont'd.)
Results:
Confidence intervals on differences both miss 0
Conclude that there is a (statistically) significant difference on both output performance measures
Slide 8 of 23
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Evaluating Many Scenarios with the Process Analyzer (PAN) With (many) more than two scenarios to compare,
two problems are
Simple mechanics of making many parameter changes, making many runs, keeping track of many output files Statistical methods for drawing reliable, useful conclusions
Process Analyzer (PAN) addresses these PAN operates on program (.p) files produced when .doe file is run (or just checked) Start PAN from Arena (Tools > Process Analyzer) or via Windows PAN runs on its own, separate from Arena
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 9 of 23
PAN Scenarios
A scenario in PAN is a combination of:
A program (.p) file Set of input controls that you choose
Chosen from Variables and Resource capacities think ahead You fill in specific numerical values Chosen from automatic Arena outputs or your own Variables Values initially empty ... to be filled in after run(s)
Set of output responses that you choose
To create a new scenario in PAN, double-click where indicated, get Scenario Properties dialog
Specify Name, Tool Tip Text, .p file, controls, responses Values of controls initially as in the model, but you can change them in PAN this is the real utility of PAN Duplicate (right-click, Duplicate) scenarios, then edit for a new one
Slide 10 of 23
Think of a scenario as a row
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
PAN Projects and Runs
A project in PAN is a collection of scenarios
Program files can be the same .p file, or .p files from different model .doe files Controls, responses can be the same or differ across scenarios in a project usually will be mostly the same Think of a project as a collection of scenario rows a table Can save as a PAN (.pan extension) file
Select scenarios in project to run (maybe all) PAN runs selected models with specified controls PAN fills in output-response values in table
Equivalent to setting up, running them all "by hand" but much easier, faster, less error-prone
Slide 11 of 23
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Model 6-5 for PAN Experiments
Same as Model 6-4 but remove Output File entries in Statistic module
PAN will keep track of outputs itself, so this is faster Stick with 110 replications Name = Base Case Program File = Model 06-05.p (maybe with path) Resources > capacity of Trunk Line User Specified > New Tech 1, New Tech 2, New Tech 3, New Tech All, New Sales Total Cost, Percent Rejected
Start PAN, New project, double-click for scenario
Six controls all data type Integer
Could also do a designed experiment with PAN, for more efficient study of controls' effects, interactions
Slide 12 of 23
Responses both from User Specified
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Model 6-5 for PAN Experiments (cont'd.)
Experimental (non-base-case) scenarios
Suppose get you $1360 more per week for more resources
Must spend all $1360 on a single type of resource; could get
13 more trunk lines @ $98 each 4 more of any one of the single-product tech-support people @ $320 each 3 more of the all-product tech-support people @ $360 each 4 more sales people @ $340 each
Create six more PAN scenarios
Right-click, Duplicate Scenario(s), edit fields See the saved PAN file Experiment 06-05.pan
Slide 13 of 23
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Execute scenarios
Model 6-5 for PAN Experiments (cont'd.)
What to make of all this? Statistical meaningfulness?
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations Slide 14 of 23
Statistical Comparisons with PAN
Model 6-5 scenarios were made with 110 replications each
Better than one replication, but what about statistical validity of comparisons, selection of "the best"?
Select Total Cost column, Insert > Chart (or right-click on column, then Insert Chart)
or
Chart Type: Box and Whisker Next, Total Cost; Next defaults Next, Identify Best Scenarios
Smaller is Better, Error Tolerance = 0 (not the default) Show Best Scenarios; Finish
Repeat for Percent Rejected
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Slide 15 of 23
Statistical Comparisons with PAN (cont'd.)
Vertical boxes: 95% confidence intervals Red scenarios statistically significantly better than blues
Numerical values (including c.i. half widths) in chart right click on chart, Chart Options, Data So which scenario is "best"? Criteria disagree. Combine them somehow?
More precisely, red scenarios are 95% sure to contain the best one Narrow down red set more replications, or Error Tolerance > 0 More details in text
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Slide 16 of 23
Searching for an Optimal Scenario with OptQuest Scenarios considered via PAN are just a few of
many Seek input controls minimizing Total Cost while keeping Percent Rejected 5
Explore all possibilities add resources in any combination New rules:
26 number of trunk lines 50 Total number of new employees of all five types 15
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Slide 17 of 23
Searching for an Optimal Scenario with OptQuest Formulation Formulate as an optimization problem:
Objective function is a simulation-model output Minimize Total Cost Subject to 26 MR(Trunk Line) 50 0 New Sales + New Tech 1 + New Tech 2 + New Tech 3 + New Tech All 15 Percent Rejected 5 Constraint on another output
Constraints on the input control (decision) variables
Reasonable start best acceptable scenario so far
No PAN scenarios satisfied Percent Rejected 5, so start with more-resources case earlier (29 trunk lines, 3 new employees of each of five types)
Where to go from here? Explore all of feasible sixdimensional space exhaustively? No.
For this problem, choice (decision) variables are discrete, so can enumerate that there are 1,356,600 feasible scenarios with 110 replications per scenario, would take two months on 2.1GHz PC
Slide 18 of 23
Simulation with Arena, 4th ed. Chapter 6 Stat. Output Analysis Terminating Simulations
Searching for an Optimal Scenario with OptQuest Operation OptQuest searches intelligently for an optimum
Like PAN, OptQuest ...
runs as a separate application ... can be launched from Arena "takes over" the running of your model asks you to identify input controls, the output (just one) objective allows you to specify constraints on the input controls allows you to s...