20111119_183434 - Output Analysis Output analysis is the...

Info icon This preview shows pages 1–8. Sign up to view the full content.

View Full Document Right Arrow Icon
Image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 2
Image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 4
Image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 6
Image of page 7

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 8
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Output Analysis Output analysis is the statistical estimation of the performance measures of interest from a simulation model. We begin by dividing the simulation models into two classes, which have different implications from the viewpoint of output analysis. Terminating simulation w A terminating simulation is one that runs for a certain duration of time, and then, stops. This duration can be random. For example, a simulation that models the operation of a grocery store from 8:00 am to 5:00 pm is a terminating simulation. The simulation stops at time 5:00 pm. A simulation that models the operation of a production facility starting from 6:00 am on Monday until the 100—th part is produced is also a terminating simulation. Here, however, the termination time is random. Steady—state simulation — A steady—state simulation is one that runs continuously without any stopping condition. In steady—state simulations, we are usually interested in the “long—run” behavior of the model. For example, if the conditions in a production facility are envisioned not to be fluctuating over time, a simulation aimed at assessing the throughput is a steady— state simulation. The simulation does not necessarily have a stopping condition and we are interested in the long—run average throughput. When building a simulation model to examine different architectures for a telecommunications network, one can assume that the demand arrival rate remains constant over short periods of time (say, 10-20 minutes) and data packets arrive very frequently (say, millions per minute). In this case, we can View a simulation over a few minutes as being essentially over an infinite time horizon and we are interested in the long—run average congestion in the telecommunications network. 1. Output Analysis for Terminating Simulation The output of a terminating simulation can be analyzed by using the standard confidence interval methodology. L Let {Y(t) : t E [0,T]} be the trajectory of the simulation model. Assume that we are interested in the performance measure 6=E{%/OTY(t)dt}.- For example, Y(t) could be the length of a queue at time t, in which case 6 would be the expected average queue length over the time interval [0, T]. We run the simulation model for R different replications, where each replication uses an independent sequence of random numbers and starts from the same initial conditions. Let {H.(t) : t E [0, T]} be the trajectory of replication 7“. From R simulation runs, we obtain R estimates of 9, each given by _ 1 T Y. = r [J W) dr- 122 Then, we can estimate 6 by a _ ]_ _ Y = — 2 Yr. R r=1 We also have R _ 1 _ 1 l — VGT(Y) = VG)?" (E Til K.) = E RVCLT'O/l) = E VGT(Y]'). In the expression above, we use the fact that E71, . . . , YR are i.i.d., because each simulation run startsifrom the same initial conditions and is independent of the other simulation runs. Since Vera/T) is unknown, we estimate it by si=fi2(fi—Y)2. Under fairly general assumptions, Y — 6 sR/x/R approximately has t distribution with R — 1 degrees of freedom. Therefore, if to, {2, R_1 is the (d/2)—th quantile of the t distribution with R — 1 degrees of freedom, then )7 w) Tlfru; 1P {ta/ZJZI S S —te:/2,R—I} W 1 — 05 which implies that _ s _ s P{Y+io:/2R—1 —; S 9 S Y_toe/2,R~1 7%} W 1 w 04 Therefore, )7 3|: tin/2,34 S—R gives an approximate 100(1 — 00% confidence interval for 9. x/E Side note — The same methodology applies when (9 is defined as an “observation—based” rather than a “time—weighted” average. For example, let 6 be the expected average waiting time per customer in a queueing system over the time interval [0, T]. In other words, if {Yb Y2, . . . ,Yn} are the waiting times of the customers over the time interval [0,T], then 1 n 9—]E{E;Y,-}. We run R simulation runs over the time interval [0, Tland let {Km Kg, . . . ,Ym} be the waiting times of the customers in run 7". Then, defining Y; = 7—1322; 3%, the same methodology applies. 123 2. Output Analysis for Steady-State Simulation Let {Y(t) : t 2 0} be the trajectory of a steady—state simulation. We say that {Y(t) : t Z O} has a well—defined steady-state mean if 1 t 11111 — Y(s) ds = (9, t—loot 0 where 9 is a deterministic constant. In order words, the average value of {Y(t) : t Z 0} over the interval [0, t] converges to a deterministic constant as the time interval grows. For example, let Y(t) be the length of the queue at'time t in a single-server queueing system with exponential interarrival and service times. Then, lim,;_,(,O % fat Y(s) 013 is the long—run average queue length. It can be shown that limt_,00 % jot Y(s) ds converges to a deterministic constant. Similarly, if Y; is the waiting time of the i—th customer, then lim,.,nc,oizzf:1 Y,- converges to a deterministic constant. In many textbooks on stochastic processes, virtually all of the discussion centers on the analysis and computation of steady—state means. There are four main reasons for this: 1. If we choose to assess performance of a system over a finite time horizon, then the perfor— mance is typically afiected by the choice of the initial conditions. For example, the throughput of a production plant over a certain day depends on the initial inventory level. However, when assessing the steady—state performance of a system, the choice of the initial conditions is irrelevant. The steady—state behavior is independent of how we choose the initial conditions. 2. If we choose to assess performance of a system over a finite time horizon [0,15], then the performance assessment is sensitive to the choice of 15. Thus, the modeler is to consider this choice carefully. On the other hand, in the steady—state analysis, no specification of 15 needs to be made. 3. The steady—state mean can provide an approximation for the performance of the system over finite time intervals. For example, if Y(t) is the cost incurred at time t, then the total cost incurred over the time interval [0,15] is ca) : Iva) ds. We can approximate C(t) by C(15) m 8t. 4. The steady—state performance measures are usually easier to compute analytically than the transient performance measures. Of course, in the simulation world, terminating simulations are easily handled. In fact, all our discussion thus far has centered on computation of finite- horizon performance measures. Before proceeding to the details of steady—state output analysis, we should think through the question of when steady—state analysis is more appropriate than finite—horizon analysis. It is probably the case that steady—state analysis is used far too widely in the consideration of 124 stochastic systems. Consider the following application areas: 1. Finance — Steady—state analysis is almost never relevant here. Options have maturity dates, investors apply discount rates to long—term investments. 2. Production — Steady—state analysis may often be inappropriate in this area. For example, suppose that demand is known to follow a seasonal pattern or demand is highly nonstationary. Steady—states typically do not exist in such settings. Furthermore, the “relaxation time” (time to reach steady—state) in production settings can often be very long relative to the decision horizon. However, there are certain situations, such as estimating the average throughput or work in process, Where steady—state analysis of a production facility makes sense. 3. Telecommunications — Usually communication networks operate on a millisecond time scale, with a relaxation time on the order of seconds. Over such time scales, the arrival rates to the system can be viewed as being constant, because time of day effects do not manifest themselves over a few seconds. Thus, the steady—state is representative of the behavior of the system and steady~state analysis is very relevant in this situation. 3. Challenges of Steady-State Simulation There are two problems when using steady—state simulation. The first one is related to the fact that the simulation model is initialized by using arbitrary initial conditions and the initial conditions can affect the relaxation time. The second one is related to the fact that the output obtained from a steady—state simulation model over nonoverlapping time intervals can be correlated. Initial transient problem 7 When we are interested in the long—run performance of the system, we usually initialize the simulation model by using arbitrary initial conditions. For example, consider a simulation model of an assembly line that has been built for the purpose of estimating the long—run average work in process. We would probably start this model with no parts in the line. This introduces a “bias” into the simulation. Namely, the initial segment of the simulation is not representative of (in fact, can be far from) the steady—state behavior of the system. If we run the simulation model for an infinite period of time, then we know that the impact of the initial conditions will disappear. However, given that we have to run the simulation model for a finite period of time, several questions arise: 1. How long does the initial transient period persist? 2. How can we identify the end of the initial transient period? 3. How can we mitigate the eficect of the initial transient period? Consider a single—server queueing system with exponential interarrival and service times. Assmne that the arrival rate is A and the service rate is a. As long as A < a, this system has a well—defined steady state. In particular, it can be shown that if Y(t) is the number of 125 customers in the system at time t, then 1 f, h'm — Y(s) d5 = L, “’00 t 0 1— p where p = A/u. Assume that A = 1/100 seconds—1 and a = 1/90 seconds—1, and thus, p = 0.9. Then, 11mm“, % f; Y(s) ds = 9. We novvr build a simple simulation model in ProModel and run it for 100 replications to build a confidence interval for the average number of customers in the system. We run the simulation model for 1, 5, 10, 50J 100 and 500 hours. We obtain the following results: 95% confidence interval for limfflog % f; Y(s) d5 0.00, 0.00] : 0.00 3F 0.00 ,_ D—LMm-LUIQHICD 1_'—' 5 10 ' 50 ' 100 ' 500 Therefore, if we use % f; Y(3) ds for a finite t in order to estimate the limit 11min“, % fut Y(s) ds, then we have the problem that the low queue lengths at the beginning of the simulation “drag down” the average. This is the bias introduced by the initial transient behavior of the simulation model. Autocorrelation problem — Assume that we are interested in the average time a customer spends in the system in the long run. Letting Y1, Y2, . . . be the time spent in the system by 126 the successive customers, we know that 1 1“!- £20;ng converges to the average time a customer spends in the system. Since we cannot run a simulation model for an infinite amount of time, all we can do is to estimate the limit above by % ZZZ, Y,- for finite 17.. Unfortunately, we not cannot apply the standard confidence interval methodology here, because the times spent in the system by the successive customers (that is, Yl, Yg, . . .) are not independent. If the current customer spends a long time in the system, then it is likely that the next customer spends a long time in the system. The next two sections suggest solutions to the initial transient and autocorrelation problem. 4. Initial Transient Deletion and Replication-Deletion Method for Steady-State Simulation The bias introduced by the initial transient effects can be serious. Thus, many researchers have tried to develop methods to remove this bias. The key idea is to run the model for a warm up period during which no statistics are collected. Once the warm up period is completed, we start to collect statistics. This warm up period idea gives rise to the replication—deletion method for output analysis of steady—state simulation. In this method, we run a simulation model for R independent replications. Let {Y}(t) : t E [U,T]} be the trajectory of replication 7". To remove the initial transient effects, the first section of each of the R replications up to time point at is ignored (that is, [0, d] is the warm up period). Therefore, we use , 1 T Kim/a; Y;(s)ds in order to estimate the performance measure 1 -t 8 = lim —/ Y,.(s) d3. 15 c t—roo (Again, (9 does not depend on 7", because, probabilistically speaking, each replication is identi— cal to the others.) Then, we can apply the approach of Section 1 in order to build a confidence interval for H. In particular, the approximate 100(1 — 00% confidence interval for I9 is given 7 3R by Y IF ta/glR_1 E, where mm 17—er 3 —__1 for—i7)? —R 'r‘; —R—1r:1 7- . Replication—deletion method is embedded in ProModel. By selecting Options under Sim— ulation menu, we can set the number of replications to R, the run length to T — d, and the 127 warm up period to d. The following table replicates the results of the previous one. But in this case, we use 20% of the run time for warm up. ' t 95% confidence interval for (hours) limtnoo— 1 )ds “[ [0.00 0.]00=O.00).ZF000 -[ 331 4.]35 ]=3..83:[:052 15 [576 802—689$113 [6.84 9.41]=8.13:]:1.29 [775 9.07]=8..41:|:066 [819 942]=880=F062 .8 D o—nwwemmwmm 1 5 ' 10 "T 50 ' 1‘00 ‘ 500 (The dashed lines replicate the previous graph for comparison purposes.) The replication—deletion method is quite popular because it is easily implemented and one can use the standard confidence interval methodology to get the error estimates. However, every replication repeats the initial transient period, which is a waste of computational effort. It would be nice if there were a method that did not duplicate this wasted effort. The method of batch means considered in the next section is one such method. 5. Method of Batch Means for Steady State Simulation Like the replication—deletion method, the method of batch means attempts to obtain inde- pendent observations, so that we can apply the standard confidence interval methodology. However, unlike the replication—deletion method, the method of batch means is applied to one long simulation run (as opposed to R separate replications). Therefore, we go through the initial transient phase only once. Let {Y(t) : t E [0,T + 03]} be the trajectory of the simulation model and d be the warm _ up period. We divide the time interval [11, T + (1] into K batches, each with length 13 = T/K. Then, the general procedure is as follows: 128 1. Compute the batch means as _ l d+£i B,=—f Y(s)ds forz'=1,...,K. sees—1) 2. Compute the point estimate of liming<3 % fut Y(s) ds as the mean of the batch means. That is, set 1 K Y = E :1 3,. 3. Estimate the variance of R; as 4. Then, an approximate confidence interval for limtnoo % fut Y(s) tie is — .5 Y T tea/2,K—1 J—KE' The reason the methpd of batch means works is that if the batch length l? is large enough, then the batch means 81,. . . ,BK are approximately i.i.d.. Therefore, Y is the mean of K approximately i.i.d. random variables, and this justifies the method described above. In this method, we should pay attention to choose 3 large enough so that the batch means Bl, . . . , BK are approximately independent. To check the independency, we can estimate the one—step autocorrelation through 2:72:33- — YXBHI — ill (K4) a A good rule of thumb is to proceed with the method of batch means if this autocorrelation estimate is less than 0.2. On the other hand, for this autocorrelation estimate to be unbiased, K should be large (at least about 30 in general). Therefore, when T is fixed, there is a conflict between choosing 1? large enough and K large enough. Since the method of batch means goes through the initial transient period only once, when implemented well, the method of batch means is almost always preferable to the replication— deletion method. 129 ...
View Full Document

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern