This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: Output Analysis Output analysis is the statistical estimation of the performance measures of interest from a
simulation model. We begin by dividing the simulation models into two classes, which have different implications from the viewpoint of output analysis. Terminating simulation w A terminating simulation is one that runs for a certain duration
of time, and then, stops. This duration can be random. For example, a simulation that models
the operation of a grocery store from 8:00 am to 5:00 pm is a terminating simulation. The
simulation stops at time 5:00 pm. A simulation that models the operation of a production
facility starting from 6:00 am on Monday until the 100—th part is produced is also a terminating
simulation. Here, however, the termination time is random. Steady—state simulation — A steady—state simulation is one that runs continuously without
any stopping condition. In steady—state simulations, we are usually interested in the “long—run”
behavior of the model. For example, if the conditions in a production facility are envisioned
not to be ﬂuctuating over time, a simulation aimed at assessing the throughput is a steady—
state simulation. The simulation does not necessarily have a stopping condition and we are
interested in the long—run average throughput. When building a simulation model to examine
different architectures for a telecommunications network, one can assume that the demand
arrival rate remains constant over short periods of time (say, 1020 minutes) and data packets
arrive very frequently (say, millions per minute). In this case, we can View a simulation over
a few minutes as being essentially over an inﬁnite time horizon and we are interested in the long—run average congestion in the telecommunications network. 1. Output Analysis for Terminating Simulation The output of a terminating simulation can be analyzed by using the standard conﬁdence
interval methodology. L Let {Y(t) : t E [0,T]} be the trajectory of the simulation model. Assume that we are
interested in the performance measure 6=E{%/OTY(t)dt}. For example, Y(t) could be the length of a queue at time t, in which case 6 would be the
expected average queue length over the time interval [0, T]. We run the simulation model for R different replications, where each replication uses an
independent sequence of random numbers and starts from the same initial conditions. Let
{H.(t) : t E [0, T]} be the trajectory of replication 7“. From R simulation runs, we obtain R estimates of 9, each given by
_ 1 T
Y. = r [J W) dr 122 Then, we can estimate 6 by
a
_ ]_ _
Y = — 2 Yr.
R r=1
We also have R
_ 1 _ 1 l —
VGT(Y) = VG)?" (E Til K.) = E RVCLT'O/l) = E VGT(Y]'). In the expression above, we use the fact that E71, . . . , YR are i.i.d., because each simulation run
startsifrom the same initial conditions and is independent of the other simulation runs. Since
Vera/T) is unknown, we estimate it by si=ﬁ2(ﬁ—Y)2. Under fairly general assumptions,
Y — 6 sR/x/R approximately has t distribution with R — 1 degrees of freedom. Therefore, if to, {2, R_1 is the
(d/2)—th quantile of the t distribution with R — 1 degrees of freedom, then )7
w) Tlfru; 1P {ta/ZJZI S S —te:/2,R—I} W 1 — 05 which implies that _ s _ s
P{Y+io:/2R—1 —; S 9 S Y_toe/2,R~1 7%} W 1 w 04
Therefore, )7 3: tin/2,34 S—R gives an approximate 100(1 — 00% conﬁdence interval for 9. x/E Side note — The same methodology applies when (9 is deﬁned as an “observation—based” rather
than a “time—weighted” average. For example, let 6 be the expected average waiting time per
customer in a queueing system over the time interval [0, T]. In other words, if {Yb Y2, . . . ,Yn}
are the waiting times of the customers over the time interval [0,T], then 1 n
9—]E{E;Y,}. We run R simulation runs over the time interval [0, Tland let {Km Kg, . . . ,Ym} be the waiting
times of the customers in run 7". Then, deﬁning Y; = 7—1322; 3%, the same methodology
applies. 123 2. Output Analysis for SteadyState Simulation Let {Y(t) : t 2 0} be the trajectory of a steady—state simulation. We say that {Y(t) : t Z O}
has a well—deﬁned steadystate mean if
1 t
11111 — Y(s) ds = (9, t—loot 0 where 9 is a deterministic constant. In order words, the average value of {Y(t) : t Z 0} over
the interval [0, t] converges to a deterministic constant as the time interval grows. For example, let Y(t) be the length of the queue at'time t in a singleserver queueing system
with exponential interarrival and service times. Then, lim,;_,(,O % fat Y(s) 013 is the long—run
average queue length. It can be shown that limt_,00 % jot Y(s) ds converges to a deterministic
constant. Similarly, if Y; is the waiting time of the i—th customer, then lim,.,nc,oizzf:1 Y,
converges to a deterministic constant. In many textbooks on stochastic processes, virtually all of the discussion centers on the
analysis and computation of steady—state means. There are four main reasons for this: 1. If we choose to assess performance of a system over a ﬁnite time horizon, then the perfor—
mance is typically aﬁected by the choice of the initial conditions. For example, the throughput
of a production plant over a certain day depends on the initial inventory level. However, when
assessing the steady—state performance of a system, the choice of the initial conditions is
irrelevant. The steady—state behavior is independent of how we choose the initial conditions. 2. If we choose to assess performance of a system over a ﬁnite time horizon [0,15], then the
performance assessment is sensitive to the choice of 15. Thus, the modeler is to consider this choice carefully. On the other hand, in the steady—state analysis, no speciﬁcation of 15 needs
to be made. 3. The steady—state mean can provide an approximation for the performance of the system
over ﬁnite time intervals. For example, if Y(t) is the cost incurred at time t, then the total cost incurred over the time interval [0,15] is ca) : Iva) ds. We can approximate C(t) by
C(15) m 8t. 4. The steady—state performance measures are usually easier to compute analytically than the
transient performance measures. Of course, in the simulation world, terminating simulations
are easily handled. In fact, all our discussion thus far has centered on computation of ﬁnite horizon performance measures. Before proceeding to the details of steady—state output analysis, we should think through
the question of when steady—state analysis is more appropriate than finite—horizon analysis. It
is probably the case that steady—state analysis is used far too widely in the consideration of 124 stochastic systems. Consider the following application areas: 1. Finance — Steady—state analysis is almost never relevant here. Options have maturity dates,
investors apply discount rates to long—term investments. 2. Production — Steady—state analysis may often be inappropriate in this area. For example,
suppose that demand is known to follow a seasonal pattern or demand is highly nonstationary.
Steady—states typically do not exist in such settings. Furthermore, the “relaxation time” (time
to reach steady—state) in production settings can often be very long relative to the decision
horizon. However, there are certain situations, such as estimating the average throughput or work in process, Where steady—state analysis of a production facility makes sense. 3. Telecommunications — Usually communication networks operate on a millisecond time
scale, with a relaxation time on the order of seconds. Over such time scales, the arrival rates
to the system can be viewed as being constant, because time of day effects do not manifest
themselves over a few seconds. Thus, the steady—state is representative of the behavior of the system and steady~state analysis is very relevant in this situation. 3. Challenges of SteadyState Simulation There are two problems when using steady—state simulation. The ﬁrst one is related to the
fact that the simulation model is initialized by using arbitrary initial conditions and the
initial conditions can affect the relaxation time. The second one is related to the fact that the output obtained from a steady—state simulation model over nonoverlapping time intervals
can be correlated. Initial transient problem 7 When we are interested in the long—run performance of the
system, we usually initialize the simulation model by using arbitrary initial conditions. For
example, consider a simulation model of an assembly line that has been built for the purpose
of estimating the long—run average work in process. We would probably start this model with no parts in the line. This introduces a “bias” into the simulation. Namely, the initial segment of the simulation
is not representative of (in fact, can be far from) the steady—state behavior of the system. If
we run the simulation model for an inﬁnite period of time, then we know that the impact of
the initial conditions will disappear. However, given that we have to run the simulation model for a ﬁnite period of time, several questions arise: 1. How long does the initial transient period persist? 2. How can we identify the end of the initial transient period?
3. How can we mitigate the eﬁcect of the initial transient period? Consider a single—server queueing system with exponential interarrival and service times.
Assmne that the arrival rate is A and the service rate is a. As long as A < a, this system
has a well—deﬁned steady state. In particular, it can be shown that if Y(t) is the number of 125 customers in the system at time t, then 1 f,
h'm — Y(s) d5 = L,
“’00 t 0 1— p where p = A/u. Assume that A = 1/100 seconds—1 and a = 1/90 seconds—1, and thus, p = 0.9.
Then, 11mm“, % f; Y(s) ds = 9. We novvr build a simple simulation model in ProModel and run it for 100 replications to build a conﬁdence interval for the average number of customers
in the system. We run the simulation model for 1, 5, 10, 50J 100 and 500 hours. We obtain the following results: 95% conﬁdence interval for
limfﬂog % f; Y(s) d5
0.00, 0.00] : 0.00 3F 0.00 ,_ D—LMmLUIQHICD 1_'—' 5 10 ' 50 ' 100 ' 500 Therefore, if we use % f; Y(3) ds for a ﬁnite t in order to estimate the limit 11min“, % fut Y(s) ds,
then we have the problem that the low queue lengths at the beginning of the simulation
“drag down” the average. This is the bias introduced by the initial transient behavior of the simulation model. Autocorrelation problem — Assume that we are interested in the average time a customer
spends in the system in the long run. Letting Y1, Y2, . . . be the time spent in the system by 126 the successive customers, we know that 1 1“!
£20;ng converges to the average time a customer spends in the system. Since we cannot run a
simulation model for an inﬁnite amount of time, all we can do is to estimate the limit above
by % ZZZ, Y, for ﬁnite 17.. Unfortunately, we not cannot apply the standard conﬁdence interval
methodology here, because the times spent in the system by the successive customers (that
is, Yl, Yg, . . .) are not independent. If the current customer spends a long time in the system,
then it is likely that the next customer spends a long time in the system. The next two sections suggest solutions to the initial transient and autocorrelation problem. 4. Initial Transient Deletion and ReplicationDeletion Method for SteadyState
Simulation The bias introduced by the initial transient effects can be serious. Thus, many researchers
have tried to develop methods to remove this bias. The key idea is to run the model for a warm
up period during which no statistics are collected. Once the warm up period is completed, we start to collect statistics. This warm up period idea gives rise to the replication—deletion method for output analysis
of steady—state simulation. In this method, we run a simulation model for R independent
replications. Let {Y}(t) : t E [U,T]} be the trajectory of replication 7". To remove the initial
transient effects, the ﬁrst section of each of the R replications up to time point at is ignored (that is, [0, d] is the warm up period). Therefore, we use , 1 T
Kim/a; Y;(s)ds in order to estimate the performance measure 1 t
8 = lim —/ Y,.(s) d3.
15 c t—roo (Again, (9 does not depend on 7", because, probabilistically speaking, each replication is identi—
cal to the others.) Then, we can apply the approach of Section 1 in order to build a conﬁdence interval for H. In particular, the approximate 100(1 — 00% conﬁdence interval for I9 is given
7 3R
by Y IF ta/glR_1 E, where mm 17—er 3 —__1 for—i7)?
—R 'r‘; —R—1r:1 7 . Replication—deletion method is embedded in ProModel. By selecting Options under Sim—
ulation menu, we can set the number of replications to R, the run length to T — d, and the 127 warm up period to d. The following table replicates the results of the previous one. But in
this case, we use 20% of the run time for warm up. ' t 95% conﬁdence interval for
(hours) limtnoo— 1 )ds
“[ [0.00 0.]00=O.00).ZF000
[ 331 4.]35 ]=3..83:[:052 15 [576 802—689$113
[6.84 9.41]=8.13:]:1.29 [775 9.07]=8..41::066
[819 942]=880=F062 .8
D o—nwwemmwmm 1 5 ' 10 "T 50 ' 1‘00 ‘ 500 (The dashed lines replicate the previous graph for comparison purposes.) The replication—deletion method is quite popular because it is easily implemented and one
can use the standard conﬁdence interval methodology to get the error estimates. However,
every replication repeats the initial transient period, which is a waste of computational effort.
It would be nice if there were a method that did not duplicate this wasted effort. The method of batch means considered in the next section is one such method. 5. Method of Batch Means for Steady State Simulation Like the replication—deletion method, the method of batch means attempts to obtain inde
pendent observations, so that we can apply the standard conﬁdence interval methodology.
However, unlike the replication—deletion method, the method of batch means is applied to one
long simulation run (as opposed to R separate replications). Therefore, we go through the initial transient phase only once. Let {Y(t) : t E [0,T + 03]} be the trajectory of the simulation model and d be the warm
_ up period. We divide the time interval [11, T + (1] into K batches, each with length 13 = T/K. Then, the general procedure is as follows: 128 1. Compute the batch means as _ l d+£i
B,=—f Y(s)ds forz'=1,...,K.
sees—1) 2. Compute the point estimate of liming<3 % fut Y(s) ds as the mean of the batch means. That
is, set
1 K
Y = E :1 3,. 3. Estimate the variance of R; as 4. Then, an approximate conﬁdence interval for limtnoo % fut Y(s) tie is — .5
Y T tea/2,K—1 J—KE' The reason the methpd of batch means works is that if the batch length l? is large enough,
then the batch means 81,. . . ,BK are approximately i.i.d.. Therefore, Y is the mean of K
approximately i.i.d. random variables, and this justiﬁes the method described above. In this method, we should pay attention to choose 3 large enough so that the batch means
Bl, . . . , BK are approximately independent. To check the independency, we can estimate the one—step autocorrelation through 2:72:33 — YXBHI — ill (K4) a A good rule of thumb is to proceed with the method of batch means if this autocorrelation
estimate is less than 0.2. On the other hand, for this autocorrelation estimate to be unbiased,
K should be large (at least about 30 in general). Therefore, when T is ﬁxed, there is a conﬂict between choosing 1? large enough and K large enough. Since the method of batch means goes through the initial transient period only once, when
implemented well, the method of batch means is almost always preferable to the replication— deletion method. 129 ...
View
Full Document
 '08
 TOPALOGLU

Click to edit the document details