20111119_183509 - Comparison of Alternative Systems In...

Info iconThis preview shows pages 1–7. Sign up to view the full content.

View Full Document Right Arrow Icon
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 2
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 4
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 6
Background image of page 7
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Comparison of Alternative Systems In addition to estimating the performance of a particular system, simulation models can be used for comparing the performances of different systems. When comparing the performances of different systems, we have to make sure that the comparison is “fair” and statistically valid. In the next section, we address the question of fairness. The subsequent sections present statistically valid ways of comparing the performances of different systems. 1. Common Random Numbers Consider the multi—server queueing model with a nonstationary Poisson arrival process that we used to illustrate the subroutines in ProModel. We are interested in the expected value of the average time spent in the queue by a customer over the time period [0, 8] hours. In particular, our goal is to choose the number of servers (0) such that this duration of time is less than 2 minutes. We run the model with c 2 8 for one replication for 8 hours. The number of customer arrivals is 762. Next, we run the model with c = 9 for 8 hours. Now, the number of customer arrivals is 748. Is it fair to compare the average time spent in the queue when c = 8 and c = 9? The S-server system had to serve a higher number of customers over 8 hours. Therefore, the difference in the average time spent in the queue is not entirely due to the fact that the two systems differ in the number of servers, but also due to the fact that the 8-server system had to deal with a higher number of customers. Therefore, loosely speaking, the comparison between these two systems is not fair. In order to remedy this problem, we have to understand how random numbers work in ProModel. Remember that the random numbers we use in a simulation model are generated by using a linear congruential generator. A linear congruential generator is essentially a “wheel” of numbers that starts from a particular position and returns the successive numbers. ProModel also uses linear congruential generators. Each time ProModel needs a random number, it draws the next value from the wheel. In our multi—server queueing model, the interarrival times, the acceptance—rejection deci- sions and the service times are all generated by using a single sequence of random numbers. That is, a number in a particular position on the wheel can be used to generate an interarrival time and the number in the next position can be used to generate a service time. When we add an extra server, a random number that was previously used to generate an interarrival 130 time may now be used to generate a service time. This changes the sampled interarrival times and the acceptance—rejection decisions, and hence, the number of customers that arrive into the system over the time period [0, 8] hours is different. We need a mechanism to ensure that we get the same sequence of customer interarrival times, acceptance—rejection decisions and service times regardless of the number of servers. This can be accomplished through the use of random number streams. ProModel has a single wheel of random numbers. However, this wheel can be broken down into 100 separate sections, which we call “streams.” By specifying that the interarrival times, the acceptance rejection decisions and the service times are to be generated from different streams, we ensure that we always get the same interarrival times regardless of the number of servers. fine/mi my MILL/“spatial 1:160; S‘fflwl. a S‘wftc JTMrlLM Shim 3 “:3? */E {{ELJ‘JIIQMJ In order to use different streams, we can append the stream number when generating a sample from a probability distribution. For example, E(1.0, 5) generates samples of an exponentially distributed random variable with mean 1.0 by using the 5—th stream. UCO . 5 , O . 5 , 6) generates samples of a uniformly distributed random variable over [0, 1] by using the 6—th stream. When we run the model after assigning diiferent streams to the interarrival times, acceptance- rejection decisions and service times, we see that the number of arrivals is the same regardless of the number of servers. Any comparisons we make new will be “fairer.” This is an ex— ample of the use of a variance reduction technique called “common random numbers.” This technique can be quite beneficial when comparing alternative systems. 2. Significance of Statistical Significance This section shows the major issues involved in using simulation in order to compare two alternative systems. A bank is planning to install an automatic teller station and must choose between buying one ZippyTel machine or two KlunkyTel machines. One ZippyTel costs twice as much to purchase and maintain as two KlunlcyTels, so that the two options entail the same costs. The decision is based on the average time a customer has to wait in order to start getting service from the machines. Customers arrive one at a time according to a Poisson process with rate 1 customer per minute. The service times of a ZippyTel are exponentially distributed with mean 0.9 minutes. The service times of a KlunkyTel are exponentially distributed with mean 1.8 minutes. If two 131 buq: d.‘ Tilt)?" @ry Klvnlij W1, ’Ciunlty is ()6ch l KlunkyTels are installed, then the customers will form a single line. The performance measure of interest is the expected average delay in the queue of the first 100 customers. Let HZ and 9K be this performance measure for the ZippyTel and Klunky’I‘el configurations. An analyst in the bank decides to simulate 100 customer delays for each system and use the averages of the 100 delays to decide whether HZ or 6K is smaller. This is not a sound approach because for each replication, the estimates of 93 and 19K will be different. This problem can be remedied by carrying out multiple replications. But, how many replications are required? Assume that the analyst makes R replications and bases his decision on the averages of the average waiting times over R replications. The table below shows the probability of making the correct decision given that the analyst carries out a certain number of replications. No of Probability of replications correct decision 1 0.48 5 0.54 10 0.62 20 0.72 Note that with only 1 replication, the analyst is more likely to choose the wrong system than the right one. As the number of replications increases, the correct decision is made more often. But, how many simulation runs do we need in order to ascertain that the right system will be selected with a certain probability? In order to answer this question, we need a statistical method of comparison between two (or more) systems. 3. Confidence Intervals for the Performance Difi‘erence Between Two Systems Assume that we have two systems, system 1 and 2, that we want to compare with respect 1M0 a. certain performance measure. Let 61 and 6‘2 be the values of the performance measure in these two systems. We want to build a confidence interval for 91 — 62. We run each system for R replications. We assume that each replication is independent of the others. (However, replication 7" for system 1 and replication r for system 2 can be dependent.) Let K1 be an estimate of 01 obtained from replication 7*. Let I? be an estimate of 62 obtained from replication 7". For example, {1],}(15) : t E [0, T]} could be the length of a queue observed in system 1 over the simulation horizon during replication 7‘. In this case, we have 17,} = % I: Y} dt. Or, lez, . , could be the customer waiting times in system 1 over the simulation horizon during replication 7“. In this case, we have 17,} 2 all 3:, Then, the general procedure to build a confidence interval for (91 — 92 is as follows: 1. Let Dr be an estimate of 61 — 62 obtained from replication 1". That is, set 132 2. Compute a point estimate of 61 —— 92 as D = E D R T. Estimate the variance of D, as 3. Then, an approximate 100(1 — 00% confidence interval for 61 — 62 is 3R D T ta/2,R71 E: where try/2,34 is the or/2—th quantile of the 25 distribution with R — 1 degrees of freedom. A very important observation is that the procedure above only requires 1—31,...,DR to be independent of each other. Therefore, for a given '1", 17,1 and 17,2 can be dependent. In particular, we can use common random numbers in each replication r, so that the comparison between 1721 and 17,? is “fair.” The procedure above remains valid as long as the random numbers used in different replications are uncorrelated. In order to see the impact of using common random numbers, note that VW(DT):VM(Z1_K2): VMCQHJ + V64 (fig) '2 Cad ( YR} \fr-L) If system I and system 2 are run using uncorrelated random numbers in replication '1", then Vor( C(13) + \[9rfo1) On the other hand, if 051101312?) is positive, then Varfljr) is smaller than the case when Codi/2,1?) = 0. The role of common random numbers is_ to induce positive correlation between the output of two systems, and hence, reduce Var(D,.). In this way, the confidence intervals that we come up with by using the procedure above will be tighter. Var (13,.) = Example ~ The analyst in the bank simulates the waiting times of the first 100 customers for 15 replications. The table below gives the average delay in queue over 100 customers for different replications. The second column gives the results for the ZippyTel configuration. The third column gives the results for the KlunkyTel configuration obtained by using random numbers uncorrelated with the ones used in the ZippyTel replications. The fourth colunm gives the results for the KlunkyTel configuration obtained by using the same random numbers as the ones used in the Zippy’I‘el replications. (See the text description of the ProModel model at the end to see how one can ensure that the service times are correlated. The trick is to generate and store all necessary service times by using an inversion algorithm at the beginning of the simulation.) 133 KlunkyTel Indep. (Y3) KlunkyTel Rep. Zippflel 00m (173*) Dr = Z1 - Y? D: = 173 - Y?“ (7") (1713) 1 2 3 4 5 6 7 8 9 l—lHl—il—lHH CHI-hwwl—‘D Using the values in the column labeled Dr, we obtain 1') = —2.94, 83 = 6.60. We have £0,025,” 2 e214. Therefore, a 95% confidence interval for the difference in the expected average delay in queue is 6.60 —2.94 2F 2.14 — = [—6.59, 0.72]. x/fi This confidence interval contains 0. Hence, there is no statistically significant difference be- tween the performances of the two systems. Similarly, using the values in the column labeled Djf, we obtain 7 D* = 0.17, 5:23 = 0.41. Therefore, a 95% confidence interval for the difference in the expected average delay in queue is 0.41 0.17 2.14 — : —0.06,0.39. a m l J This confidence interval also includes 0, but it is much tighter than the previous one. Using the second confidence interval, we can conclude with 95% confidence that the difference between the expected average delays lies between —0.06 and 0.39. This is a much stronger statement than what we can say by using the first confidence interval. 4. A Modified Confidence Interval The method we describe in this section is useful when the two systems in consideration have been run for different numbers of replications, say R and Q. Of course, one can ignore |R— QI replications of one system to “equalize” the number of replications, but ignoring precious data 5 134 may not be a good idea. Let 57,} be an estimate of 91 obtained from replication 7‘, where r = 1, . . . , R. Let K? be an estimate of 92 obtained from replication 9‘", where r = l, . . . , Since we cannot make pair—wise comparisons, we now require that all replications of all systems are independent. Therefore, in all R —l- Q replications, we use uncorrelated random numbers. As a consequence of this requirement, we cannot use common random numbers. Then, the general procedure to build a confidence interval for (91 — 6'2 is as follows: 1. Compute a point estimate of 61 and (92 as ' R 171 = i 2:17;}, 172 = Z2, respectively. T=1 R _ 1 T=1 _ 1 1r=1 3. Since Vera—’1 — Y2) 2 Var(l?1)+ Varflfl) : % RVafiY’ ) + % Q Vera?) = grams?) + g M173). estimate the variance of 371 — 372 as (81;? + (8:3)2 4. Estimate the degrees of freedom by [(31—92 (52912 + R Q f _ _ [err/RF + [(322) /Q]2' 3—1 62—1 5. Then, an approximate 100(1 — oz)% confidence interval for 61 — 62 is 7 7 1 2 (S2 )2 Y1 _ Y2 ,5 (8R) Q A :F Cir/2,1" R + Q Unfortunately, for the procedure above to be statistically valid, many assumptions must be satisfied in our simulation model. Due to this reason and the fact that it cannot use common random numbers, the procedure above should not be used unless absolutely necessary. Instead, the procedure in Section 3 should be used. 135 5. Common Random Numbers for ZippyTel Configuration k§*++*i££§§+++*i**éii***ii**i*iiii*i***+i*****ii*****i**+k+ii**£é+**iié*+******* * i * Formatted Listing of Model: * * C:\huseyin\courses\ORIE5800\ORIESBOFallI]9\zippy_CRN.MOD * * a: *tii****iiii*£*iiiiifii***i****iiii**i*k**iiiikkikii*ti*ii£ifikki*n**iii***kiiii** Time Units: Minutes Distance Units: Feet Initialization Logic: REAL RN INT J = 1 INT STREAM = 1 WHILE ( J <: 200 ) DO I RN = U{0.5,0.5,STREAM) inter_ar:__times[ J ] = - LNtRN) J = J + 1 l J = 1 WHILE E J <: 110 ) DD [ RN = U(0.5,D.5,STREAM) service_timesl J ] = - 0.9 1* LN(RN] J = J + 1 } ****ii§*§t**itikii*+*ii**ii*+£ifiiii*i****i**iikii*+fi*i*i**ii£*&kii**&iii**ik**i* * Locations * itiik**titi£k}iiiiiiiiiiiiiikkiiiiiiiiii;iiiiiiii}*iiiikiiiiii}ii+*k}iii*ii+*ii* Name Cap Units Stats Rules Cost queue INF 1 Time Series Oldest, , zippy 1 1 Time Series Oldest, , .t*kkttkttkitttitttttt*titttt*ittt*i’tttt*kttkti’titfitkittttkitiifitttitfittkitktt’cti * Entities * kinktwéiiilnlnkktfldulizktalnfl-kitiinltititan-tittkiikttiiiiiktitiiitfiiwtikitwwikiiiiititiit Name Speed (fpm) Stats Cost customer 150 Time Series titti*tifiittktfi*fiitititatii'itfiatifivtiti#ittfi‘x*‘tirki‘tfi‘i’vkfinttiftfii**iit%fi**txifi**kfii{*‘k* it * Processing iiiiiiiiik*iiiiiiki*iiiitiiiiiriiiiiiiiiii*§*tiifiiiiiiifiitittifiiitkifiiiiiiiifi*ik Process Routing Entity Location Operation Blk Output Destination Rule Move Logic customer queue 1 customer zippyr FIRST 1 customer zippy timeiiniq'ueue : CLOCKIMIN) - arrivefitime WAIT service_times[currentjerfice] MIN curreut_service = cur:ent_se:vice + 1 nogserved = noiserved + 1 IF tno_served : 100 ) THEN STOP 1 customer EXIT FIRST 1 *i:J'QHka-k***k*‘k*?*****w*¢rit'kic‘kiré**>\'*ir**rti-iitkshfiirir't'lwkfii*‘kiiki*shki'ieiki'sti'titka'fiiririititt * Arrivals * 4'***ii**iinhhl-Irtirizk*ixh}*tikkiirtkkvni'kk*kiiktttiiktkkkitikkttifiiinkkgkxhkigiidnhkktnttir Entity Location Qty Each First Time Occurrences Frequency customer queue 1 INE‘ “a**un*9m*~ks*unmtt*itttmutanthuntiatihunt:urierttithnxtttiimxamittxte“Han * Attributes * u“we”unnuuuunu“in,”u++nu+fiu+i+inu“91H”“inauuuuuneu ID Type Classification arrive_time Rea1 Entity 1t*iit*k*t*tittt*fifiiiitki*iti**xi*t**itifi'fii»iiiiiic*icat*iiifi*iciifi'iirt‘k*ifiitt**ii*i**4* * Variables (global) * tiff-t*itittitti*iiki—i*****iki*+iiki++iiitii***k***i*fi*k**tiiik*A-ti-i**k**§ttk*ii- ID Type Initial value Stats n0_served Integer 0 None time_in_queue Real 0 Time Series currentiarr Integer 1 Time Series current_service Integer 1 Time Series *t-l-i-ki-iii-ki-i-ikiiiiiifiki-i-i-i-i-i-i*iii-i:££ii-i-9r*i-iiiikiivtA-A-islia-ati-A-i-néfi-ivti-i-niifiiwkfii*k**k * Arrays * ti*t#**tiiitfi*i*iiiii****ifit**ifiii***ifiiiifiik**i**iiki***##iii**§i*i*x*i*iii**k* ID Dimensions Type inter_arr_times 2 0 0 Real servicefitimes 1 10 Real interiarrit mes [currentiarr] MIN arrivefitime currentfiarr CLOCKflflIN] — currentfiarr + 1 136 ...
View Full Document

Page1 / 7

20111119_183509 - Comparison of Alternative Systems In...

This preview shows document pages 1 - 7. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online