This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: Comparison of Alternative Systems In addition to estimating the performance of a particular system, simulation models can be
used for comparing the performances of different systems. When comparing the performances
of different systems, we have to make sure that the comparison is “fair” and statistically
valid. In the next section, we address the question of fairness. The subsequent sections
present statistically valid ways of comparing the performances of different systems. 1. Common Random Numbers Consider the multi—server queueing model with a nonstationary Poisson arrival process that
we used to illustrate the subroutines in ProModel. We are interested in the expected value
of the average time spent in the queue by a customer over the time period [0, 8] hours. In
particular, our goal is to choose the number of servers (0) such that this duration of time is less than 2 minutes. We run the model with c 2 8 for one replication for 8 hours. The number of customer
arrivals is 762. Next, we run the model with c = 9 for 8 hours. Now, the number of customer
arrivals is 748. Is it fair to compare the average time spent in the queue when c = 8 and c = 9?
The Sserver system had to serve a higher number of customers over 8 hours. Therefore, the
difference in the average time spent in the queue is not entirely due to the fact that the two
systems differ in the number of servers, but also due to the fact that the 8server system had to deal with a higher number of customers. Therefore, loosely speaking, the comparison
between these two systems is not fair. In order to remedy this problem, we have to understand how random numbers work in ProModel. Remember that the random numbers we use in a simulation model are generated by
using a linear congruential generator. A linear congruential generator is essentially a “wheel”
of numbers that starts from a particular position and returns the successive numbers. ProModel also uses linear congruential generators. Each time ProModel needs a random
number, it draws the next value from the wheel. In our multi—server queueing model, the interarrival times, the acceptance—rejection deci
sions and the service times are all generated by using a single sequence of random numbers.
That is, a number in a particular position on the wheel can be used to generate an interarrival
time and the number in the next position can be used to generate a service time. When we
add an extra server, a random number that was previously used to generate an interarrival 130 time may now be used to generate a service time. This changes the sampled interarrival times
and the acceptance—rejection decisions, and hence, the number of customers that arrive into
the system over the time period [0, 8] hours is different. We need a mechanism to ensure that we get the same sequence of customer interarrival
times, acceptance—rejection decisions and service times regardless of the number of servers.
This can be accomplished through the use of random number streams. ProModel has a single wheel of random numbers. However, this wheel can be broken down into 100 separate sections, which we call “streams.” By specifying that the interarrival times,
the acceptance rejection decisions and the service times are to be generated from different
streams, we ensure that we always get the same interarrival times regardless of the number of servers. fine/mi my MILL/“spatial 1:160; S‘fﬂwl. a S‘wftc JTMrlLM Shim 3 “:3? */E {{ELJ‘JIIQMJ In order to use different streams, we can append the stream number when generating a sample
from a probability distribution. For example, E(1.0, 5) generates samples of an exponentially
distributed random variable with mean 1.0 by using the 5—th stream. UCO . 5 , O . 5 , 6) generates
samples of a uniformly distributed random variable over [0, 1] by using the 6—th stream. When we run the model after assigning diiferent streams to the interarrival times, acceptance rejection decisions and service times, we see that the number of arrivals is the same regardless
of the number of servers. Any comparisons we make new will be “fairer.” This is an ex—
ample of the use of a variance reduction technique called “common random numbers.” This technique can be quite beneﬁcial when comparing alternative systems. 2. Signiﬁcance of Statistical Signiﬁcance This section shows the major issues involved in using simulation in order to compare two
alternative systems. A bank is planning to install an automatic teller station and must choose between buying
one ZippyTel machine or two KlunkyTel machines. One ZippyTel costs twice as much to
purchase and maintain as two KlunlcyTels, so that the two options entail the same costs. The
decision is based on the average time a customer has to wait in order to start getting service from the machines. Customers arrive one at a time according to a Poisson process with rate 1 customer per
minute. The service times of a ZippyTel are exponentially distributed with mean 0.9 minutes.
The service times of a KlunkyTel are exponentially distributed with mean 1.8 minutes. If two 131 buq: d.‘
Tilt)?" @ry
Klvnlij W1, ’Ciunlty is ()6ch l KlunkyTels are installed, then the customers will form a single line. The performance measure of interest is the expected average delay in the queue of the ﬁrst
100 customers. Let HZ and 9K be this performance measure for the ZippyTel and Klunky’I‘el conﬁgurations. An analyst in the bank decides to simulate 100 customer delays for each system
and use the averages of the 100 delays to decide whether HZ or 6K is smaller. This is not a sound approach because for each replication, the estimates of 93 and 19K will be different. This problem can be remedied by carrying out multiple replications. But, how many
replications are required? Assume that the analyst makes R replications and bases his decision
on the averages of the average waiting times over R replications. The table below shows the
probability of making the correct decision given that the analyst carries out a certain number of replications. No of Probability of
replications correct decision 1 0.48
5 0.54
10 0.62
20 0.72 Note that with only 1 replication, the analyst is more likely to choose the wrong system than
the right one. As the number of replications increases, the correct decision is made more often.
But, how many simulation runs do we need in order to ascertain that the right system will
be selected with a certain probability? In order to answer this question, we need a statistical method of comparison between two (or more) systems. 3. Conﬁdence Intervals for the Performance Diﬁ‘erence Between Two Systems Assume that we have two systems, system 1 and 2, that we want to compare with respect 1M0
a. certain performance measure. Let 61 and 6‘2 be the values of the performance measure in these two systems. We want to build a conﬁdence interval for 91 — 62. We run each system for R replications. We assume that each replication is independent
of the others. (However, replication 7" for system 1 and replication r for system 2 can be
dependent.) Let K1 be an estimate of 01 obtained from replication 7*. Let I? be an estimate
of 62 obtained from replication 7". For example, {1],}(15) : t E [0, T]} could be the length of a
queue observed in system 1 over the simulation horizon during replication 7‘. In this case, we
have 17,} = % I: Y} dt. Or, lez, . , could be the customer waiting times in system
1 over the simulation horizon during replication 7“. In this case, we have 17,} 2 all 3:, Then, the general procedure to build a conﬁdence interval for (91 — 92 is as follows: 1. Let Dr be an estimate of 61 — 62 obtained from replication 1". That is, set 132 2. Compute a point estimate of 61 —— 92 as
D = E D
R T. Estimate the variance of D, as 3. Then, an approximate 100(1 — 00% conﬁdence interval for 61 — 62 is 3R D T ta/2,R71 E: where try/2,34 is the or/2—th quantile of the 25 distribution with R — 1 degrees of freedom. A very important observation is that the procedure above only requires 1—31,...,DR to
be independent of each other. Therefore, for a given '1", 17,1 and 17,2 can be dependent. In
particular, we can use common random numbers in each replication r, so that the comparison
between 1721 and 17,? is “fair.” The procedure above remains valid as long as the random numbers used in different replications are uncorrelated. In order to see the impact of using common random numbers, note that VW(DT):VM(Z1_K2): VMCQHJ + V64 (ﬁg) '2 Cad ( YR} \frL) If system I and system 2 are run using uncorrelated random numbers in replication '1", then
Vor( C(13) + \[9rfo1) On the other hand, if 051101312?) is positive, then Varﬂjr) is smaller than the case when
Codi/2,1?) = 0. The role of common random numbers is_ to induce positive correlation between the output of two systems, and hence, reduce Var(D,.). In this way, the conﬁdence
intervals that we come up with by using the procedure above will be tighter. Var (13,.) = Example ~ The analyst in the bank simulates the waiting times of the ﬁrst 100 customers
for 15 replications. The table below gives the average delay in queue over 100 customers for
different replications. The second column gives the results for the ZippyTel configuration.
The third column gives the results for the KlunkyTel conﬁguration obtained by using random
numbers uncorrelated with the ones used in the ZippyTel replications. The fourth colunm
gives the results for the KlunkyTel conﬁguration obtained by using the same random numbers
as the ones used in the Zippy’I‘el replications. (See the text description of the ProModel model
at the end to see how one can ensure that the service times are correlated. The trick is to
generate and store all necessary service times by using an inversion algorithm at the beginning of the simulation.) 133 KlunkyTel
Indep. (Y3) KlunkyTel Rep. Zippﬂel
00m (173*) Dr = Z1  Y? D: = 173  Y?“ (7") (1713) 1
2
3
4
5
6
7
8
9 l—lHl—il—lHH
CHIhwwl—‘D Using the values in the column labeled Dr, we obtain
1') = —2.94, 83 = 6.60. We have £0,025,” 2 e214. Therefore, a 95% conﬁdence interval for the difference in the
expected average delay in queue is 6.60
—2.94 2F 2.14 — = [—6.59, 0.72]. x/ﬁ This conﬁdence interval contains 0. Hence, there is no statistically signiﬁcant difference be
tween the performances of the two systems. Similarly, using the values in the column labeled Djf, we obtain 7
D* = 0.17, 5:23 = 0.41. Therefore, a 95% conﬁdence interval for the difference in the expected average delay in queue is
0.41 0.17 2.14 — : —0.06,0.39.
a m l J This conﬁdence interval also includes 0, but it is much tighter than the previous one. Using the
second conﬁdence interval, we can conclude with 95% conﬁdence that the difference between
the expected average delays lies between —0.06 and 0.39. This is a much stronger statement
than what we can say by using the ﬁrst conﬁdence interval. 4. A Modiﬁed Conﬁdence Interval The method we describe in this section is useful when the two systems in consideration have been run for different numbers of replications, say R and Q. Of course, one can ignore R— QI
replications of one system to “equalize” the number of replications, but ignoring precious data 5 134 may not be a good idea. Let 57,} be an estimate of 91 obtained from replication 7‘, where r = 1, . . . , R. Let K? be
an estimate of 92 obtained from replication 9‘", where r = l, . . . , Since we cannot make
pair—wise comparisons, we now require that all replications of all systems are independent.
Therefore, in all R —l Q replications, we use uncorrelated random numbers. As a consequence of this requirement, we cannot use common random numbers. Then, the general procedure
to build a conﬁdence interval for (91 — 6'2 is as follows: 1. Compute a point estimate of 61 and (92 as ' R
171 = i 2:17;}, 172 = Z2, respectively. T=1 R _ 1 T=1 _ 1 1r=1
3. Since
Vera—’1 — Y2) 2 Var(l?1)+ Varﬂﬂ) : % RVaﬁY’ ) + % Q Vera?)
= grams?) + g M173).
estimate the variance of 371 — 372 as (81;? + (8:3)2 4. Estimate the degrees of freedom by [(31—92 (52912
+ R Q f _ _ [err/RF + [(322) /Q]2'
3—1 62—1 5. Then, an approximate 100(1 — oz)% conﬁdence interval for 61 — 62 is 7 7 1 2 (S2 )2
Y1 _ Y2 ,5 (8R) Q A
:F Cir/2,1" R + Q Unfortunately, for the procedure above to be statistically valid, many assumptions must be
satisﬁed in our simulation model. Due to this reason and the fact that it cannot use common
random numbers, the procedure above should not be used unless absolutely necessary. Instead, the procedure in Section 3 should be used. 135 5. Common Random Numbers for ZippyTel Configuration k§*++*i££§§+++*i**éii***ii**i*iiii*i***+i*****ii*****i**+k+ii**£é+**iié*+******* * i
* Formatted Listing of Model: *
* C:\huseyin\courses\ORIE5800\ORIESBOFallI]9\zippy_CRN.MOD *
* a:
*tii****iiii*£*iiiiiﬁi***i****iiii**i*k**iiiikkikii*ti*ii£iﬁkki*n**iii***kiiii** Time Units: Minutes Distance Units: Feet Initialization Logic: REAL RN INT J = 1 INT STREAM = 1 WHILE ( J <: 200 ) DO I
RN = U{0.5,0.5,STREAM)
inter_ar:__times[ J ] =  LNtRN)
J = J + 1 l J = 1 WHILE E J <: 110 ) DD [
RN = U(0.5,D.5,STREAM) service_timesl J ] =  0.9 1* LN(RN]
J = J + 1
}
****ii§*§t**itikii*+*ii**ii*+£iﬁiii*i****i**iikii*+ﬁ*i*i**ii£*&kii**&iii**ik**i*
* Locations *
itiik**titi£k}iiiiiiiiiiiiiikkiiiiiiiiii;iiiiiiii}*iiiikiiiiii}ii+*k}iii*ii+*ii*
Name Cap Units Stats Rules Cost
queue INF 1 Time Series Oldest, ,
zippy 1 1 Time Series Oldest, ,
.t*kkttkttkitttitttttt*titttt*ittt*i’tttt*kttkti’titﬁtkittttkitiiﬁtttitﬁttkitktt’cti
* Entities *
kinktwéiiilnlnkktﬂdulizktalnﬂkitiinltititantittkiikttiiiiiktitiiitﬁiwtikitwwikiiiiititiit
Name Speed (fpm) Stats Cost
customer 150 Time Series titti*tiﬁittktﬁ*ﬁitititatii'itﬁatiﬁvtiti#ittﬁ‘x*‘tirki‘tﬁ‘i’vkﬁnttiftﬁi**iit%ﬁ**txiﬁ**kﬁi{*‘k*
it * Processing iiiiiiiiik*iiiiiiki*iiiitiiiiiriiiiiiiiiii*§*tiiﬁiiiiiiﬁitittiﬁiitkiﬁiiiiiiiﬁ*ik
Process Routing Entity Location Operation Blk Output Destination Rule Move Logic customer queue 1 customer zippyr FIRST 1 customer zippy timeiiniq'ueue : CLOCKIMIN)  arriveﬁtime WAIT service_times[currentjerﬁce] MIN
curreut_service = cur:ent_se:vice + 1 nogserved = noiserved + 1 IF tno_served : 100 ) THEN STOP 1 customer EXIT FIRST 1 *i:J'QHkak***k*‘k*?*****w*¢rit'kic‘kiré**>\'*ir**rtiiitkshﬁirir't'lwkﬁi*‘kiiki*shki'ieiki'sti'titka'ﬁiririititt * Arrivals *
4'***ii**iinhhlIrtirizk*ixh}*tikkiirtkkvni'kk*kiiktttiiktkkkitikkttiﬁiinkkgkxhkigiidnhkktnttir Entity Location Qty Each First Time Occurrences Frequency customer queue 1 INE‘ “a**un*9m*~ks*unmtt*itttmutanthuntiatihunt:urierttithnxtttiimxamittxte“Han * Attributes *
u“we”unnuuuunu“in,”u++nu+ﬁu+i+inu“91H”“inauuuuuneu ID Type Classification arrive_time Rea1 Entity 1t*iit*k*t*tittt*ﬁﬁiiitki*iti**xi*t**itiﬁ'ﬁi»iiiiiic*icat*iiiﬁ*iciiﬁ'iirt‘k*iﬁitt**ii*i**4* * Variables (global) * tifft*itittitti*iiki—i*****iki*+iiki++iiitii***k***i*fi*k**tiiik*Atii**k**§ttk*ii ID Type Initial value Stats n0_served Integer 0 None time_in_queue Real 0 Time Series
currentiarr Integer 1 Time Series
current_service Integer 1 Time Series *tlikiiiikiiikiiiiiiﬁkiiiiiii*iiii:££iii9r*iiiiikiivtAAisliaatiAinéﬁivtiiniiﬁiwkﬁi*k**k * Arrays *
ti*t#**tiiitﬁ*i*iiiii****iﬁt**iﬁii***iﬁiiiﬁik**i**iiki***##iii**§i*i*x*i*iii**k*
ID Dimensions Type
inter_arr_times 2 0 0 Real
serviceﬁtimes 1 10 Real interiarrit mes [currentiarr] MIN arriveﬁtime
currentﬁarr CLOCKﬂﬂIN]
— currentﬁarr + 1 136 ...
View
Full Document
 '08
 TOPALOGLU

Click to edit the document details