Coursehero >> Alabama >> W. Alabama >> CS 856

Course Hero has millions of student submitted documents similar to the one below including study guides, homework solutions, papers, exam answer keys and textbook solutions.

Versus Open Closed: A Cautionary Tale Bianca Schroeder Carnegie Mellon University Pittsburgh, PA 15213 bianca@cs.cmu.edu Adam Wierman Carnegie Mellon University Pittsburgh, PA 15213 acw@cs.cmu.edu Mor Harchol-Balter Carnegie Mellon University Pittsburgh, PA 15213 harchol@cs.cmu.edu Abstract Workload generators may be classi ed as based on a closed system model, where new job arrivals are only triggered by job completions (followed by think time), or an open system model, where new jobs arrive independently of job completions. In general, system designers pay little attention to whether a workload generator is closed or open. Using a combination of implementation and simulation experiments, we illustrate that there is a vast difference in behavior between open and closed models in realworld settings. We synthesize these differences into eight simple guiding principles, which serve three purposes. First, the principles specify how scheduling policies are impacted by closed and open models, and explain the differences in user level performance. Second, the principles motivate the use of partly open system models, whose behavior we show to lie between that of closed and open models. Finally, the principles provide guidelines to system designers for determining which system model is most appropriate for a given workload. Table 1 surveys the system models used in a variety of web related workload generators used by systems researchers today. The table is by no means complete; however it illustrates the wide range of workload generators and benchmarks available. Most of these generators/benchmarks assume a closed system model, although a reasonable fraction assume an open one. For many of these workload generators, it was quite dif cult to gure out which system model was being assumed the builders often do not seem to view this as an important factor worth mentioning in the documentation. Thus the choice of a system model (closed versus open) is often not really a researcher s choice, but rather is dictated by the availability of the workload generator. Even when a user makes a conscious choice to use a closed model, it is not always clear how to parameterize the closed system (e.g. how to set the think time and the multiprogramming level MPL) and what effect these parameters will have. In this paper, we show that closed and open system models yield signi cantly different results, even when both models are run with the same load and service demands. Not only is the measured response time different under the two system models, but the two systems respond fundamentally differently to varying parameters and to resource allocation (scheduling) policies. We obtain our results primarily via real-world implementations. Although the very simplest models of open and closed systems can be compared analytically, analysis alone is insuf cient to capture the effect of many of the complexities of modern computer systems, especially size based scheduling and realistic job size distributions. Real-world implementations are also needed to capture the magnitude of the differences between closed and open systems in practice. The case studies we consider are described in Section 4. These include web servers receiving static HTTP requests; the back-end database in e-commerce applications; and an auctioning web site. In performing these case studies, we needed to develop 1 Introduction Every systems researcher is well aware of the importance of setting up one s experiment so that the system being modeled is accurately represented. Representing a system accurately involves many things, including accurately representing the bottleneck resource behavior, the scheduling of requests at that bottleneck, and workload parameters such as the distribution of service request demands, popularity distributions, locality distributions, and correlations between requests. However, one factor that researchers typically pay little attention to is whether the job arrivals obey a closed or an open system model. In a closed system model, new job arrivals are only triggered by job completions (followed by think time), as in Figure 1(a). By contrast in an open system model, new jobs arrive independently of job completions, as in Figure 1(b). Think Send Receive Send Think With probability p submit again New Arrivals Queue Server Queue Server With probability (1 p) leave system New Arrivals Queue Server (a) Closed system (b) Open system (c) Partly-open system Figure 1: Illustrations of the closed, open, and partly-open system models. a exible suite of workload generators, simulators, and trace analysis tools that can be used under closed, open, and other system models. The details of this suite are provided in Section 4. Our simulation and implementation experiments lead us to identify eight principles, summarizing the observed differences between open and closed system models, many of which are not obvious. These principles may be categorized by their area of impact. The rst set of principles (see Sections 4 and 5.1) describe the difference in mean response time under open and closed system models and how various parameters affect these differences. We nd, for example, that for a xed load, the mean response time for an open system model can exceed that for a closed system model by an order of magnitude or more. Even under a high MPL, the closed system model still behaves closed with respect to mean response time, and there is still a significant difference between mean response times in closed and open systems even for an MPL of 1000. With respect to service demands (job sizes), while their variability has a huge impact on response times in open systems, it has much less of an effect in closed models. The impact of these principles is that a system designer needs to beware of taking results that were discovered under one system model and applying them to a second system model. For example, if the workload generator being used creates a closed system model, whereas the real world application is closer to an open system model, then the results obtained using the workload generator will be far from those witnessed in practice. The second set of principles (see Section 5.2) deal with the impact of scheduling on improving system performance. Scheduling is a common mechanism for improving mean response time without purchasing additional resources. While Processor-Sharing scheduling (PS) and First-Come-First-Served (FCFS) are most commonly used in computer systems, many system designs give preference to short jobs (requests with small service demands), applying policies like Non-Preemptive-ShortestJob-First (SJF) or Preemptive-Shortest-Job-First (PSJF) to disk scheduling [51] and web server scheduling [19, 33, 15]. When system designers seek to evaluate a new scheduling policy, they often try it out using a workload generator and simulation test-bed. Our work will show that, again, one must be very careful that one is correctly modeling the application as closed or open, since the impact of scheduling turns out to be very different under open and closed models. For example, our principles show that favoring short jobs is highly effective in improving mean response time in open systems, while this is far less true under a closed system model. We nd that closed system models only bene t from scheduling under a narrow range of parameters, when load is moderate and the MPL is very high. The message for system designers is that understanding whether the workload is better modeled with an open or closed system is essential in determining the effectiveness of scheduling. The third set of principles (see Section 6) deal with partly-open systems. We observe that while workload generators and benchmarks typically assume either an open system model or a closed system model, neither of these is entirely realistic. Many applications are best represented using an in-between system model, which we call the partly-open model. Our principles specify those parameter settings for which the partly-open model behaves more like a closed model or more like an open model with respect to response time. We also nd that, counter to intuition, parameters like think time have almost no impact on the performance of a partlyopen model. The principles describing the behavior of partly-open system models are important because realworld applications often t best into partly-open models, and the performance of these models is not well understood. In particular, the effect of system parameters and scheduling on performance in the partly open system points which our principles address are not known. Our results motivate the importance of designing versatile workload generators that are able to support open, closed, and partly open system models. We create such versatile workload generators for several common systems, including web servers and database systems, and use these throughout our studies. The third set of principles also provides system de- Type of benchmark Model-based web workload generator Playback mechanisms for HTTP request streams Proxy server benchmarks Database benchmark for e-commerce workloads Auction web site benchmark Online bulletin board benchmark Database benchmark for online transaction processing (OLTP) Model-based packet level web traf c generators Mail server benchmark Java Client/Server benchmark Web authentication and authorization Network le servers Streaming media service Name Surge [10], WaspClient [31], Geist [22], WebStone [47], WebBench [49], MS Web Capacity Analysis Tool [27] SPECWeb96 [43], WAGON [23] MS Web Application Stress Tool [28], Webjamma [2], Hammerhead [39], Deluge [38], Siege [17] httperf [30], Sclient [9] Wisconsin Proxy Benchmark [5], Web Polygraph [35], Inktomi Climate Lab [18] TPC-W [46] RUBiS[7] RUBBoS[7] TPC-C [45] IPB (Internet Protocol Benchmark) [24], GenSyn [20] WebTraf [16], trafgen [14] NS traf c generator [52] SPECmail2001 [42] SPECJ2EE [41] AuthMark [29] NetBench [48] SFS97 R1 (3.0) [40] MediSyn [44] System model Closed Open Closed Open Closed Closed Closed Closed Closed Closed Open Open Open Closed Closed Open Open Table 1: A summary table of the system models underlying standard web related workload generators. signers with guidelines for how to choose a system model when they are forced to pick a workload generator that is either purely closed or purely open, as are almost all workload generators (see Section 7). We consider ten different workloads and use our principles to determine for each workload which system model is most appropriate for that workload: closed, open, or partly-open. To the best of our knowledge, no such guide exists for systems researchers. Yet given the tremendous impact of the system model on performance, as described above, it is critical that one take care to make this decision carefully. Nthink + Nsystem = N . The response time, T , in a closed system is de ned to be the time from when a request is submitted until it is received. In the case where the system is a single server (e.g. a web server), the server load, denoted by , is de ned as the fraction of time that the server is busy, and is the product of the mean throughput X and the mean service demand (processing requirement) E[S]. Figure 1(b) depicts an open system con guration. In an open system model there is a stream of arriving users with average arrival rate . Each user is assumed to submit one job to the system, wait to receive the response, and then leave. The number of users queued or running at the system at any time may range from zero to in nity. The differentiating feature of an open system is that a request completion does not trigger a new request: a new request is only triggered by a new user arrival. As before, response time, T , is de ned as the time from when a request is submitted until it is completed. The server load is de ned as the fraction of time that the server is busy. Here load, , is the product of the mean arrival rate of requests, , and the mean service demand E[S]. Neither the open system model nor the closed system model is entirely realistic. Consider for example a web site. On the one hand, a user is apt to make more than one request to a web site, and the user will typically wait for the output of the rst request before making the next. In these ways a closed system model makes sense. On 2 Closed, open, and partly-open systems In this section, we de ne how requests are generated under closed, open, and partly-open system models. Figure 1(a) depicts a closed system con guration. In a closed system model, it is assumed that there is some xed number of users, who use the system forever. This number of users is typically called the multiprogramming level (MPL) and denoted by N . Each of these N users repeats these 2 steps, inde nitely: (a) submit a job, (b) receive the response and then think for some amount of time. In a closed system, a new request is only triggered by the completion of a previous request. At all times there are some number of users, Nthink , who are thinking, and some number of users Nsystem , who are either running or queued to run in the system, where the other hand, the number of users at the site varies over time; there is no sense of a xed number of users N . The point is that users visit to the web site, behave as if they are in a closed system for a short while, and then leave the system. Motivated by the example of a web site, we study a more realistic alternative to the open and closed system con gurations: the partly-open system shown in Figure 1(c). Under the partly-open model, users arrive according to some outside arrival process as in an open system. However, every time a user completes a request at the system, with probability p the user stays and makes a followup request (possibly after some think time), and with probability 1 p the user simply leaves the system. Thus the expected number of requests that a user makes to the system in a visit is Geometrically distributed with mean 1/(1 p). We refer to the collection of requests a user makes during a visit to the system as a session and we de ne the length of a session to be the number of requests in the session/visit. The server load is the fraction of time that the server is busy equalling the product of the average outside arrival rate , the mean number of requests per visit E[R], and the mean service demand E[S]. For a given load, when p is small, the partly-open model is more similar to an open model. For large p, the partly-open model resembles a closed model. 3 Comparison methodology In this section we discuss the relevant parameters and metrics for both the open and the closed system models and discuss how we set parameters in order to compare open and closed system models. Throughout the paper we choose the service demand distribution to be the same for the open and the closed system. In the case studies the service demand distribution is either taken from a trace or determined by the benchmark used in the experiments. In the model-based simulation experiments later in the paper, we use hyperexponential service demands, in order to capture the highly variable service distributions in web applications. Throughout, we measure the variability in the service demand distribution using the square coef cient of variation, C 2 . The think time in the closed system, Z, follows an exponential distribution, and the arrival process in the open system is either a Poisson arrival process with average rate , or provided by traces.1 The results for all simulations and experiments are presented in terms of mean response times and the system load . While we do not explicitly report numbers for another important metric, 1 Note that we choose a Poisson arrival process (i.e. exponential inter-arrival times) and exponential think times in order to allow the open and closed systems to be as parallel as possible. This setting underestimates the differences between the systems when more bursty arrival processes are used. mean throughput, the interested reader can directly infer those numbers by interpreting load as a simple scaling of throughput. In an open system, the mean throughput is simply equal to = /E[S], which is the same as throughput in a closed system. In order to fairly compare the open and closed systems, we will hold the system load for the two systems equal, and study the effect of open versus closed system models on mean response time. The load in the open system is speci ed by , since = E[S]. Fixing the load of a closed system is more complex, since the load is affected by many parameters including the MPL, the think time, the service demand variability, and the scheduling policy. The fact that system load is in uenced by many more system parameters in a closed system than in an open system is a surprising difference between the two systems. Throughout, we will achieve a desired system load by adjusting the think time of the closed system (see Figure 7(a)). The scheduling policies we study in this work span the range of behaviors of policies that are used in computer systems today. FCFS (First-Come-First-Served) Jobs are processed in the same order as they arrive. PS (Processor-Sharing) The server is shared evenly among all jobs in the system. PESJF (Preemptive-Expected-Shortest-Job-First) The job with the smallest expected duration (size) is given preemptive priority. SRPT (Shortest-Remaining-Processing-Time-First): At every moment the request with the smallest remaining processing requirement is given priority. PELJF (Preemptive-Expected-Longest-Job-First) The job with the longest expected size is given preemptive priority. PELJF is an example of a policy that performs badly and is included to understand the full range of possible response times. 4 Real-world case studies In this section, we compare the behavior of four different applications under closed, open, and partly open system models. The applications include (a) a web server delivering static content in a LAN environment, (b) the database back-end at an e-commerce web site, (c) the application server at an auctioning web site, and (d) a web server delivering static content in a WAN environment. These applications vary in many respects, including the bottleneck resource, the workload properties (e.g. job size variability), network effects, and the types of scheduling policies considered. We study applications (a), (b), and (d) through full implementation in a real testbed, while our study of application (c) relies on tracebased simulation. As part of the case studies, we develop a set of workload generators, simulators, and trace analysis tools that facilitate experimentation with all three system models: open, closed, and partly-open. For implementationbased case studies we extend the existing workload generator (which is based on only one system model) to enable all three system models. For the case studies based on trace-driven simulation, we implement a versatile simulator that models open, closed, and partly-open systems and takes traces as input. We also develop tools for analyzing web traces (in Common Log le Format or Squid log format) to extract the data needed to parameterize workload generators and simulators. Sections 4.1 4.4 provide the details of the case studies. The main results are shown in Figures 2 and 4. For each case study we rst explain the tools developed for experimenting in open, closed, and partly-open models. We then then describe the relevant scheduling policies and their implementation, and nally discuss the results. The discussions at the end of the case studies are meant only to highlight the key points; we will discuss the differences between open, closed, and partly-open systems and the impact of these differences in much more detail in Sections 5 and 6. 4.1 Static web content Our rst case study is an Apache web server running on Linux and serving static content, i.e. requests of the form Get me a le, in a LAN environment. Our experimental setup involves six machines connected by a 10/100 Ethernet switch. Each machine has an Intel Pentium III 700 MHz processor and 256 MB RAM, and runs Linux. One of the machines is designated as the server and runs Apache. The others generate web requests based on a web trace. Workload generation: In this case study we generate static web workloads based on a trace. Below we rst describe our workload generator which generates web requests following an open, closed, or partly-open model. We then describe the tool for analyzing web traces that produces input les needed by the workload generator. Finally we brie y describe the actual trace that we are using in our work. Our workload generator is built on top of the Sclient[9] workload generator. The Sclient workload generator uses a simple open system model, whereby a new request for le y is made exactly every x msec. Sclient is designed as a single process that manages all connections using the select system call. After each call to select, Sclient checks whether the current x msec interval has passed and if so initiates a new request. We generalize Sclient in several ways. For the open system, we change Sclient to make requests based on arrival times and lenames speci ed in an input le. The entries in the input le are of the form < ti , fi >, where ti is a time and fi is a le name. For the closed system, the input le only speci es the names of the les to be requested. To implement closed system arrivals in Sclient, we have Sclient maintain a list with the times when the next requests are to be made. Entries to the list are generated during runtime as follows: Whenever a request completes, an exponentially distributed think time Z is added to the current time tcurr and the result Z + tcurr is inserted into the list of arrival times. In the case of the partly-open system, each entry in the input le now de nes a session, rather than an individual request. An entry in the input le takes the form < ti , fi1 , . . . , fin > where ti speci es the arrival time of the session and < fi1 , . . . , fin > is the list of les to be requested during the session. As before, a list with arrival times is maintained according to which requests are made. The list is initialized with the session arrival times ti from the input le. To generate the arrivals within a session, we use the same method as described for the closed system above: after request fij 1 completes we arrange the arrival of request fij by adding an entry containing the arrival time Z + tcurr to the list, where tcurr is the current time and Z is an exponentially distributed think time. All the input les for the workload generator are created based on a web trace. We modify the Webalizer tool [12] to parse a web trace and then extract the information needed to create the input les for the open, closed, and partly-open system experiments. In the case of the open system, we simply output the arrival times together with the names of the requested les. In the case of the closed system, we only extract the sequence of le names. Creating the input le for the partly-open system is slightly more involved since it requires identifying the sessions in a trace. A common approach for identifying sessions (and the one taken by Webalizer) is to group all successive requests by the same client (i.e. same IP address) into one session, unless the time between two requests exceeds some timeout threshold in which case a new session is started. In our experiments, we use the timeout parameter to specify the desired average session length. The trace we use consists of one day from the 1998 World Soccer Cup, obtained from the Internet Traf c Archive [21]. Virtually all requests in this trace are static. Number of Req. 4.5 106 Mean size 5KB Variability (C 2 ) 96 Min size 41 bytes Max size 2MB Scheduling: Standard scheduling of static requests in a web server is best modeled by processor sharing (PS). However, recent research suggests favoring requests for small les can improve mean response times at web Closed System 300 300 Open System 250 200 150 100 50 0 0 0.2 0.4 Load 0.6 0.8 1 Partly-open System 300 250 200 150 100 50 0 0 Mean response time (msec) Mean response time (msec) Mean response time (msec) PS SRPT PS SRPT 250 200 150 100 50 0 0 5 PS SRPT 0.2 0.4 Load 0.6 0.8 1 10 15 Mean number of visits 20 (a) Static web LAN 10 8 6 4 2 0 0 8 6 4 2 0 0 Mean Response Time (sec) Mean Response Time (sec) Mean Response Time (sec) PELJF PS PESJF 10 PELJF PS PESJF 10 8 6 4 2 0 0 PELJF PS PESJF 0.2 0.4 Load 0.6 0.8 1 0.2 0.4 Load 0.6 0.8 1 5 10 15 Mean number of visits 20 (b) E-commerce site 20 15 15 Mean Response Time (sec) Mean Response Time (sec) Mean Response Time (sec) FCFS PS PSJF 20 FCFS PS PSJF 20 15 FCFS PS PSJF 10 10 10 5 5 5 0 0 0.2 0.4 Load 0.6 0.8 1 0 0 0.2 0.4 Load 0.6 0.8 1 0 0 5 10 15 Mean number of visits 20 (c) Auctioning site Figure 2: Results for real-world case studies. Each row shows the results for a real-world workload and each column shows the results for one of the system models. In all experiments with the closed system model the MPL is 50. The partly-open system is run at xed load 0.9. servers [19]. In this section we therefore consider both PS and SRPT policies. We have modi ed the Linux kernel and the Apache Web server to implement SRPT scheduling at the server. For static HTTP requests, the network (access link out of the server) is typically the bottleneck resource. Thus, our solution schedules the bandwidth on this access link by controlling the order in which the server s socket buffers are drained. Traditionally, the socket buffers are drained in Round-Robin fashion (similar to PS); we instead give priority to sockets corresponding to connections where the remaining data to be transferred is small. Figure 3 shows the ow of data in Linux after our modi cations. There are multiple priority queues and queue i may only Socket 1 TCP proc. IP proc. port for the user/kernel Netlink Socket, QOS and Fair Queuing, and the Prio Pseudoscheduler and by using the tc[6] user space tool. We also modify Apache to use setsockopt calls to update the priority of the socket as the remaining size of the transfer decreases. For details on our implementation see [19]. Synopsis of results: Figure 2(a) shows results from the the static web implementation under closed, open, and partly open workloads in a LAN environment. Upon rst glance, it is immediately clear that the closed system response times are vastly different from the open response times. In fact, the response times in the two systems are orders of magnitude different under PS given a common system load. Furthermore, SRPT provides little improvement in the closed system, while providing dramatic improvement in the open system. The third column of Figure 2(a) shows the results for the partly-open system. Notice that when the mean number of requests is small, the partly-open system behaves very much like the open system. However, as the mean number of requests grows, the partly-open system behaves more like a closed system. Thus, the impact of scheduling (e.g. SRPT over PS) is highly dependent on the number of requests in the partly-open system. 1st Priority Queue feed first! 2nd Priority Queue feed second. TCP proc. IP proc. Ethernet Card Network Wire Socket 2 TCP proc. IP proc. Socket 3 Figure 3: Flow of data in Linux with SRPT-like scheduling (only 2 priority levels shown). drain if queues 0 to i 1 are empty. The implementation is enabled by building the Linux kernel with sup- 4.2 E-commerce site Our second case study considers the database back-end server of an e-commerce site, e.g. an online bookstore. We use a PostgreSQL[32] database server running on a 2.4-GHz Pentium 4 with 3GB RAM, running Linux 2.4, with a buffer pool of 2GB. The machine is equipped with two 120GB IDE drives, one used for the database log and the other for the data. The workload is generated by four client machines having similar speci cations to the database server connected via a network switch. Workload generation: The workload for the ecommerce case study is based on the TPC-W [46] benchmark, which aims to model an online bookstore such as Amazon.com. We build on the TPC-W kit provided by the Pharm project [13]. The kit models a closed system (in accordance with TPC-W guidelines) by creating one process for each client in the closed system. We extend the kit to also support an open system with Poisson arrivals, and a partly-open system. We do so by creating a master process that signals a client whenever it is time to make a new request in the open system or to start a new session in the partly-open system. The master process repeats the following steps in a loop: it sleeps for an exponential interarrival time, signals a client, and draws the next inter-arrival time. The clients block waiting for a signal from the master process. In the case of the open system, after receiving the signal, the clients make one request before they go back to blocking for the next signal. In the case of the partly-open system, after receiving a signal, the clients generate a session by executing the following steps in a loop: (1) make one request; (2) ip a coin to decide whether to begin blocking for a signal from the master process or to generate an exponential think time and sleep for that time. TPC-W consists of 16 different transaction types including the ShoppingCart transaction, the Payment transaction, and others. Statistics of our con guration are as shown: Database size 3GB Mean size 101 ms Variability (C 2 ) 4 Min size 2 ms Max size 5s To implement the priorities needed for achieving PESJF and PELJF, we modify our PostgreSQL server as follows. We use the sched setscheduler() system call to set the scheduling class of a PostgreSQL process working on a high priority transaction to SCHED RR, which marks a process as a Linux realtime process. We leave the scheduling class of a low priority process at the standard SCHED OTHER. Realtime processing in Linux always has absolute, preemptive priority over standard processes. Synopsis of results: Figure 2(b) shows results from the e-commerce implementation described above. Again, the difference in response times between the open and closed systems is immediately apparent the response times of the two systems differ by orders of magnitude. Interestingly, because the variability of the service demands is much smaller in this workload than in the static web workload, the impact of scheduling in the open system is much smaller. This also can be observed in the plot for the partly open system: even when the number of requests is small, there is little difference between the response times of the different scheduling policies. 4.3 Auctioning web site Our third case study investigates an auctioning web site. This case study uses simulation based on a trace from one of the top-ten U.S. online auction sites. Workload generation: For simulation-based case studies we implement a simulator that supports open, closed, and partly-open arrival processes which are either created based on a trace or are generated from probability distributions. For a trace-based arrival process the simulator expects the same input les as the workload generator described in Section 4.1. If no trace for the arrival process is available the simulator alternatively offers (1) open system arrivals following a Poisson process; (2) closed system arrivals with exponential think times; (3) partly-open arrivals with session arrivals following a Poisson process and think times within the sessions being exponentially distributed. The service demands can either be speci ed through a trace or one of several probability distributions, including hyper-exponential distributions and more general distributions. For our case study involving an auctioning web site we use the simulator and a trace containing the service demands obtained from one of the top ten online auctioning sites in the US. No data on the request arrival process is available. The characteristics of the service demands recorded in the trace are summarized below: Number of jobs 300000 Mean size 0.09s Variability (C 2 ) 9.19 Min size 0.01s Max size 50s Scheduling: The bottleneck resource in our setup is the CPU, as observed in [25]. The default scheduling policy is therefore best described as PS, in accordance with Linux CPU scheduling. Note that in this application, exact service demands are not known, so SRPT cannot be implemented. Thus, we experiment with PESJF and PELJF policies where the expected service demand of a transaction is based on its type. The Bestseller transaction, which makes up 10% of all requests, has on average the largest service demand. Thus, we study 2-priority PESJF and PELJF policies where the Bestseller transactions are expected to be long and all other transactions are expected to be short. Scheduling: The policy used in a web site serving dynamic content, such as an auctioning web site, is best Closed System Mean Response Time (msec) 1000 Open System Mean Response Time (msec) 1500 PS SRPT 1000 1500 PS SRPT 500 500 0 0.4 0.6 Load 0.8 1 0 0.4 0.6 Load 0.8 1 (a) Static web Good WAN conditions Mean Response Time (msec) Mean Response Time (msec) 1500 PS SRPT 1500 PS SRPT 1000 1000 500 500 0 0.4 0.6 Load 0.8 1 0 0.4 0.6 Load 0.8 1 (b) Static web Poor WAN conditions Figure 4: Effect of WAN conditions in the static web case study. The top row shows results for good WAN conditions (average RTT=50ms, loss rate=1%) and the bottom row shows results for poor WAN conditions (average RTT=100ms, loss rate=4%). In both cases the closed system has an MPL of 200. Note that, due to network effects, the closed system cannot achieve a load of 1, even when think time is zero. Under the settings we consider here, the max achievable load is 0.98. modeled by PS. To study the effect of scheduling in this environment we additionally simulate FCFS and PSJF. Synopsis of results: Figure 2(c) shows results from the auctioning trace-based case study described above. The plots here illustrate the same properties that we observed in the case of the static web implementation. In fact, the difference between the open and closed response times is extreme, especially under FCFS. As a result, there is more than a factor of ten improvement of PSJF over FCFS (for > 0.7), whereas there is little difference in the closed system. This effect can also be observed in the partly-open system, where for a small number of requests per session the response times are comparable to those in the open system and for a large number of requests per session the response times are comparable to those in the closed system. The actual convergence rate depends on the variability of the service demands (C 2 ). In particular, the ecommerce case study (low C 2 ) converges quickly, while the static web and case auctioning studies (higher C 2 ) converge more slowly. module for the Linux kernel that can drop or delay incoming and outgoing TCP packets (similarly to Dummynet [34] for FreeBSD). More precisely, we change the ip rcv() and the ip output() functions in the Linux TCP-IP stack to intercept in- and out-going packets to create losses and delays. In order to delay packets, we use the add timer() facility to schedule the transmission of delayed packets. We recompile the kernel with HZ=1000 to get a ner-grained millisecond timer resolution. In order to drop packets, we use an independent, uniform random loss model which can be con gured to a speci ed probability, as in Dummynet. Synopsis of results: Figures 4 compares the response times of the closed and the open systems under (a) relatively good WAN conditions (50ms RTT and 1% loss rate) and under (b) poor WAN conditions (100ms RTT and 4% loss rate). Note that results for the partly-open system are not shown due to space constraints; however the results parallel what is shown in the closed and open systems. We nd that under WAN conditions the differences between the open and closed systems are smaller (proportionally) than in a LAN (Figure 2 (a)), however, they are still signi cant for high server loads (load > 0.8). The reason that the differences are smaller in WAN conditions is that response times include network overheads (network delays and losses) in addition to delays at the server. These overheads affect the response times in the open and closed system in the same way, causing the proportional differences between open and closed systems to shrink. For similar reasons, scheduling has less of an effect when WAN effects are strong, even in the case of an open system. SRPT improves signi cantly over PS only for high loads, and even then the improvement is smaller than in a LAN. 5 Open versus closed systems We have just seen the dramatic impact of the system model in real-world case studies. We will now develop principles that help explain both the differences between the open and closed system and the impact of these differences with respect to scheduling. In addition to the case studies that we have already discussed, we will also use model-based simulations in order to provide more control over parameters, such as job size variability, that are xed in the case studies. 4.4 Study of WAN effects To study the effect of network conditions, we return to the case of static web requests (Section 4.1), but this time we include the emulation of network losses and delays in the experiments. Workload generation: The setup and workload generation is identical to the case study of static web requests (Section 4.1), except that we add functionality for emulating WAN effects as follows. We implement a separate 5.1 FCFS Our study of the simple case of FCFS scheduling will illustrate three principles that we will exploit when studying more complex policies. Principle (i): For a given load, mean response times are signi cantly lower in closed systems than in open systems. 10 4 2000 1500 10 3 Mean Response Time (msec) Mean response time (sec) Mean response time (sec) MPL 10 MPL 100 MPL 1000 open MPL 10 MPL 100 MPL 1000 open 10 5 Open MPL 1000 MPL 100 MPL 10 10 4 1000 10 2 10 3 500 0 0.2 0.4 Load 0.6 0.8 1 0 0 10 20 30 2 Job Size Variability (C ) 40 50 10 2 0.2 0.4 0.6 Load 0.8 1 (a) Resp. time vs. load (b) Resp. time vs. C 2 (c) Auctioning site Figure 5: Open versus closed under FCFS. Model and trace-based simulation results showing mean response time as a function of load and service demand variability under FCFS scheduling. (a) and (b) use model based simulation, while (c) uses trace-based simulation. In all cases, the solid line represents an open system and the dashed lines represent closed systems with different MPLs. The load is adjusted via the think time in the closed system, and via the arrival rate in the open system. In the model-based simulations, E[S] = 10. In (a) we x C 2 = 8 and in (b) we x = 0.9. Principle (i) is maybe the most noticeable performance issue differentiating open and closed systems in our case studies (Figure 2). We bring further attention to this principle in Figure 5 due to its importance for the vast literature on capacity planning, which typically relies on closed models, and hence may underestimate the resources needed when an open model is more appropriate. For xed high loads, the response time under the closed system is orders of magnitude lower than those for the open system. While Schatte [36, 37] has proven that, under FCFS, the open system will always serve as an upper bound for the response time of the closed system, the magnitude of the difference in practical settings has not previously been studied. Intuitively, this difference in mean response time between open and closed systems is a consequence of the xed MPL, N , in closed systems, which limits the queue length seen in closed systems to N even under very high load. By contrast, no such limit exists for an open system. Principle (ii): As the MPL grows, closed systems become open, but convergence is slow for practical purposes. Principle (ii) is illustrated by Figure 5. We see that as the MPL, N , increases from 10 to 100 to 1000, the curves for the closed system approach the curves for the open system. Schatte [36, 37] proves formally that as N grows to in nity, a closed FCFS queue converges to an open M/GI/1/FCFS queue. What is interesting however, is how slowly this convergence takes place. When the service demand has high variability (C 2 ), a closed system with an MPL of 1000 still has much lower response times then the corresponding open system. Even when the job service demands are lightly variable, an MPL of 500 is required for the closed system to achieve response times comparable to the corresponding open system. Further, the differences are more dramatic in the case-study results than in the model-based simulations. This principle impacts the choice of whether an open or closed system model is appropriate. One might think that an open system is a reasonable approximation for a closed system with a high MPL; however, though this can be true in some cases, the closed and open system models may still behave signi cantly differently if the service demands are highly variable. Principle (iii): While variability has a large effect in open systems, the effect is much smaller in closed systems. This principle is dif cult to see in the case-study gures (Figure 2) since each trace has a xed variability. However, it can be observed by comparing the magnitude of disparity between the e-commerce site results (low variability) and the others (high variability). Using simulations, we can study this effect directly. Figure 5(b) compares open and closed systems under a xed load = 0.9, as a function of the service demand variability C 2 . For an open system, we see that C 2 directly affects mean response time. This is to be expected since high C 2 , under FCFS service, results in short jobs being stuck behind long jobs, increasing mean response time. In contrast, for the closed system with MPL 10, C 2 has comparatively little effect on mean response time. This is counterintuitive, but can be explained by observing that for lower MPL there are fewer short jobs stuck behind long jobs in a closed system, since the number of jobs in the system (Nsystem ) is bounded. As MPL is increased, C 2 can have more of an effect, since Nsystem can be higher. It is important to point out that by holding the load constant in Figure 5(b), we are actually performing a conservative comparison of open and closed systems. If we didn t hold the load xed as we changed C 2 , increasing C 2 would result in a slight drop in the load of the closed system as shown in Figure 7(b). This slight drop in load, would cause a drop in response times for closed systems, whereas there is no such effect in open systems. 5.2 The impact of scheduling The value of scheduling in open systems is understood and cannot be overstated. In open systems, there are or- Closed System 1000 Open System 1000 Closed System 1500 Open system 1500 Mean response time (sec) Mean response time (sec) 800 Mean response time (sec) 800 1000 Mean response time (sec) PELJF FCFS PS PESJF PELJF FCFS PS PESJF PELJF FCFS PS PESJF PELJF FCFS PS PESJF 1000 600 600 400 400 500 500 200 200 0 0 0.2 0.4 Load 0.6 0.8 1 0 0.2 0.4 0.6 Load 0.8 1 0 10 20 30 40 2 Job Size Variability (C ) 50 0 10 20 30 40 2 Job Size Variability (C ) 50 (a) Response time vs. load (b) Response time vs. variability Figure 6: Model-based simulation results illustrating the different effects of scheduling in closed and open systems. In the closed system the MPL is 100, and in both systems the service demand distribution has mean 10. For the two gures in (a) C 2 was xed at 8 and in the two gures in (b) the load was xed at 0.9. 1 0.8 MPL 1000 MPL 100 MPL 10 1 0.8 Load 0.6 0.6 0.4 0.4 MPL 1000 MPL 100 MPL 10 10 20 30 Job size variability (C2) 40 50 0.2 0.2 0 0 0.5 1 1.5 Think time 2 x 10 4 0 0 (a) Think time vs. load (b) Variability vs. load Figure 7: Model-based simulation results illustrating how the service demand variability, the MPL, and the think time can affect the system load in a closed system. These plots use FCFS scheduling, however results are parallel under other scheduling policies. der of magnitude differences between the performance of scheduling policies because scheduling can prevent small jobs from queueing behind large jobs. In contrast, scheduling in closed systems is not well understood. Principle (iv): While open systems bene t signi cantly from scheduling with respect to response time, closed systems improve much less. Principle (v): Scheduling only signi cantly improves response time in closed systems under very speci c parameter settings: moderate load (think times) and high MPL. Figure 2 illustrates the fundamentally different behavior of mean response time in the open and closed systems in realistic settings. In Figure 6, we further study this difference as a function of (a) load and (b) variability using simulations. Under the open system, as load increases, the disparity between the response times of the scheduling policies grows, eventually differing by orders of magnitude. In contrast, at both high and low loads in the closed system, the scheduling policies all perform similarly; only at moderate loads is there a signi cant difference between the policies and even here the differences are only a factor of 2.5. Another interesting point is that, whereas for FCFS the mean response time of an open system bounded that in the corresponding closed system from above, this does not hold for other policies such as PESJF, where the open system can result in lower re- sponse times than the closed system. We can build intuition for the limited effects of scheduling in closed systems by rst considering a closed feedback loop with no think time. In such a system, surprisingly, the scheduling done at the queue is inconsequential all work conserving scheduling policies perform equivalently. To see why, note that in a closed system Little s Law states that N = XE[T ], where N is the constant MPL across policies. We will now explain why X is constant across all work conserving scheduling policies (when think time is 0), and hence it will follow that E[T ] is also constant across scheduling policies. X is the long-run average rate of completions. Since a new job is only created when a job completes, over a long period of time, all work conserving scheduling policies will complete the same set of jobs plus or minus the initial set N . As time goes to in nity, the initial set N becomes unimportant; hence X is constant. This argument does not hold for open systems because for open systems Little s Law states that E[N ] = E[T ], and E[N ] is not constant across scheduling policies. Under closed systems with think time, we now allow a varying number of jobs in the queue, and thus there is some difference between scheduling policies. However, as think time grows, load becomes small and so scheduling has less effect. A very subtle effect, not yet mentioned, is that in a closed system the scheduling policy actually affects the throughput, and hence the load. Good policies, like PESJF, increase throughput, and hence load, slightly (less than 10%). Had we captured this effect (rather than holding the load xed), the scheduling policies in the closed system would have appeared even closer, resulting in even starker differences between the closed and open systems. The impact of Principles (iv) and (v) is clear. For closed systems, scheduling provides small improvement across all loads, but can only result in substantial improvement when load (think time) is moderate. In contrast, scheduling always provides substantial improve- Load Response Time vs. Requests per session 120 Response Time vs. Think time Mean response time (msec) Mean response time (sec) PELJF FCFS PS PESJF 70 60 50 40 30 20 10 PELJF FCFS PS PESJF 90 80 70 60 50 40 30 20 10 10 1 Mean response time (sec) Mean response time (sec) 100 80 60 40 20 0 PELJF FCFS PS PESJF 500 PS SRPT 400 300 200 100 5 10 15 Mean number of Visits per session 20 0 5 10 15 Mean number of visits per session 20 0 10 Think Time 2 10 3 0 0 10 10 10 Think Time (sec) 1 2 10 3 (a) = 0.6 (b) = 0.9 (c) Model-based simulation (d) Static web impl. Figure 8: Model and implementation-based results for the partly-open system. (a) and (b) are model-based simulations showing mean response time as a function of the expected number of requests per session. (c) and (d) show the mean response time as a function of the think time, for a xed load. In (a)-(c), E[S] = 10 and C 2 = 8. In (c) and (d), we x = 0.6 and p = 0.75, which yields and average of 4 requests per session. ments for open systems. Principle (vi): Scheduling can limit the effect of variability in both open and closed systems. For both the open and closed systems, better scheduling (PS and PESJF) helps combat the effect of increasing variability, as seen in Figure 6. The improvement; however, is less dramatic for closed systems due to Principle (iii) in Section 5.1, which tells us that variability has less of an effect on closed systems in general. we studied, and across a wide range of parameters, the point where the separation between the performance of scheduling policies becomes small is, as a rule-of-thumb, around 10 requests per session. Note however that this point can range anywhere between 5 and 20 requests per session as C 2 ranges from 4 to 49 respectively. We will demonstrate in Section 7 how to use this rule-of-thumb as a guideline for determining whether a purely open or purely closed workload generator is most suitable, or whether a partly-open generator is necessary. Principle (viii): In a partly-open system, think time has little effect on mean response time. Figure 8 illustrates Principle (viii). We nd that the think time in the partly-open system does not affect the mean response time or load of the system under any of these policies. This observation holds across all partly-open systems we have investigated (regardless of the number of requests per session), including the casestudies described in Section 4. Principle (viii) may seem surprising at rst, but for PS and FCFS scheduling it can be shown formally under product-form workload assumptions. Intuitively, we can observe that changing the think time in the partlyopen system has no effect on the load because the same amount of work must be processed. To change the load, we must adjust either the number of requests per session or the arrival rate. The only effect of think time is to add small correlations into the arrival stream. 6 Partly-open systems In this section, we discuss a partly-open model that (a) serves as a more realistic system model for many applications; and (b) helps illustrate when a purely open or closed system is a good approximation of user behavior. We focus on the effects of the mean number of requests per session and the think time because the other parameters, e.g. load and job size variability, have similar effects to those observed in Sections 5.1 and 5.2. Throughout the section, we x the load of the partly-open system by adjusting the arrival rate, . Note that, in contrast to the closed model, adjusting the think time of the partly-open model has no impact on the load. Principle (vii): A partly-open system behaves similarly to an open system when the expected number of requests per session is small ( 5 as a rule-of-thumb) and similarly to a closed system when the expected number of requests per session is large ( 10 as a rule-of-thumb). Principle (vii) is illustrated clearly in the case study results shown in Figure 2 and in the simulation results shown in Figure 8(a). When the mean number of requests per session is 1 we have a signi cant separation between the response time under the scheduling policies, as in open systems. However, when the mean number of requests per session is large, we have comparatively little separation between the response times of the scheduling policies; as in closed systems. Figures 2 and 8(a) are just a few examples of the range of con gurations 7 Choosing a system model The previous sections brought to light vast differences in system performance depending on whether the workload generator follows an open or closed system model. A direct consequence is that the accuracy of performance evaluation depends on accurately modeling the underlying system as either open, closed, or partly-open. A safe way out would be to choose a partly-open system model, since it both matches the typical user behavior in many applications and generalizes the open and 1 2 3 4 5 6 7 8 9 10 Type of site Large corporate web site CMU web server [3] Online department store Science institute (USGS[1]) Online gaming site [50] Financial service provider Supercomputing web site [4] Kasparov-DeepBlue match Site seeing slashdot effect Soccer world cup [21] Date Feb 01 Nov 01 June 00 Nov 02 May 04 Aug 00 May 04 May 97 Feb 99 Jul 98 Total #Req. 1609799 90570 891366 107078 45778 275786 82566 580068 194968 4606052 2 x 10 5 18 Number of requests per session Site 3 Site 6 Site 10 16 14 12 10 8 6 4 2 0 0 500 1000 1500 2000 Timeout Site 3 Site 6 Site 10 Number of sessions 1.5 1 0.5 0 0 500 1000 1500 2000 Timeout 2500 3000 2500 3000 (a) Number of sessions vs (b) Number of requests vs Timeout length Timeout length Figure 9: Choosing a system model. Statistics for 3 representative web traces (sites 3, 6, and 10) illustrating (a) the number of user sessions as a function of the timeout threshold and (b) the expected number of requests per session as a function of the timeout threshold. The vertical line on each plot corresponds to a timeout of 1800s. From these plots we can conclude that an open model is appropriate for site 6, a closed model is appropriate for site 10, and neither an open or a closed is appropriate for site 3. Table 2: A summary table of the studied web traces. closed system models depending on the parameters it can behave more like an open or more like a closed system. However, as Table 1 illustrates, available workload generators often support only either closed or open system models. This motivates a fundamental questions for workload modeling: Given a particular workload, is a purely open or purely closed system model more accurate for the workload? When is a partly-open system model necessary? In the remainder of this section we illustrate how our eight principles might be used to answer this question for various web workloads. Our basic method is as follows. For a given system we follow these steps: 1. Collect traces from the system. 2. Construct a partly-open model for the system, since the partly-open model is the most general and accurate. In particular, obtain the relevant parameters for the partly-open model. 3. For the partly-open model, decide whether an open or a closed model is appropriate, or if the partlyopen model is necessary. Table 2 summarizes the traces we collected as part of Step 1. Our trace collection spans many different types of sites, including busy commercial sites, sites of major sporting events, sites of research institutes, and an online gaming site. We next model each site as a partly-open system. According to Principles (vii) and (viii) the most relevant parameter of a partly-open system model is the number of requests issued in a user session. Other parameters such as the think time between successive requests in a session are of lesser importance. Determining the average number of requests per user session for a web site requires identifying user sessions in the corresponding web trace. While there is no 100% accurate way to do this, we employ some common estimation techniques [8, 26]. First, each source IP address in a trace is taken to represent a different user. Second, session boundaries are determined by a period of inactivity by the user, i.e. a period of time during which no requests from the corresponding IP address are received. Typically, this is accomplished by ending a session whenever there is a period of inactivity larger than timeout threshold . In some cases, web sites themselves enforce such a threshold; however, more typically must be estimated. We consider two different ways of estimating . The rst one is to use a defacto standard value for , which is 1800s (30 min) [26]. The second method is to estimate from the traces themselves by studying the derivative of how affects the total number of sessions in the trace. We illustrate this latter method for a few representative traces in Figure 9(a). Notice that as the threshold increases from 1-100s the number of sessions decreases quickly; whereas from 1000s on, the decrease is much smaller. Furthermore, Figure 9(b) shows that with respect to the number of requests, stabilization is also reached at > 1000s. Hence we adopt = 1800s in what follows. The mean number of requests per session when = 1800s is summarized below for all traces: Site Requests per session Site Requests per session 1 2.4 6 1.4 2 1.8 7 6.0 3 5.4 8 2.4 4 3.6 9 1.2 5 12.9 10 11.6 The table indicates that the average number of requests for web sessions varies largely depending on the site, ranging from less than 2 requests per session to almost 13. Interestingly, even for similar types of web sites the number of requests can vary considerably. For example sites 8 and 10 are both web sites of sporting events (a chess tournament and a soccer tournament), but the number of requests per session is quite low (2.4) in one case, while quite high (11.6) in the other. Similarly, sites 2, 4, and 7 are all web sites of scienti c institutes but the number of requests per sessions varies from 1.8 to 6. Using the rule of thumb in principle (vii), we can con- clude that neither the open nor the closed system model accurately represents all the sites. For sites 1, 2, 4, 6, 8, and 9 an open system model is accurate; whereas a closed system model is accurate for the sites 5 and 10. Further, it is not clear whether an open or closed model is appropriate for sites 3 and 7. The impact of choosing between open and closed system models correctly is demonstrated by site 10, the world cup dataset. This is the same dataset used in the static web case study, where we saw large differences depending on whether we modeled the workload using an open or a closed system. We have just concluded that a closed model is most appropriate for this workload, thus the magnitude of differences between the open and closed results in Figure 2 illustrates the impact of the choice of a system model. (a) the magnitude of the difference in response times between closed and open systems can be very large, even under moderate load; (b) the convergence of closed to open as MPL grows is slow, especially when service demand variability (C 2 ) is high; and (c) scheduling is far more bene cial in open systems than in closed ones. We also compare the partly-open model with the open and closed models. We illustrate the strong effect of the number of requests per session and C 2 on the behavior of the partly-open model, and the surprisingly weak effect of think time. These principles underscore the importance of choosing the appropriate system model. For example, in capacity planning for an open system, choosing a workload generator based on a closed model can greatly underestimate response times and underestimate the bene ts of scheduling. All of this is particularly relevant in the context of web applications, where the arrival process at a web site is best modeled by a partly-open system. Yet, most web workload generators are either strictly open or strictly closed. Our ndings provide guidelines for choosing whether an open or closed model is the better approximation based on characteristics of the workload. A high number of simultaneous users (more than 1000) suggests an open model, but a high number of requests per session (more than 10) suggests a closed model. Both these cutoffs are affected by service demand variability: highly variable demands requires larger cutoffs. Contrary to popular belief, it turns out that think times are irrelevant to the choice of an open or closed model since they only affect the load. Once it has been determined whether a closed or open model is a better approximation, that in turn provides a guideline for the effectiveness of scheduling. Understanding the appropriate system model is essential to understanding the impact of scheduling. Scheduling is most effective in open systems, but can have moderate impact in closed systems when both the load is moderate (roughly 0.7-0.85) and C 2 is high. In conclusion, while much emphasis has been placed in research on accurately representing workload parameters such as service demand distribution, think time, locality, etc, we have illustrated that similar attention needs to be placed on accurately representing the system itself as either closed, open, or partly-open. We have taken a rst step toward this end by providing guidelines for choosing a system model and by creating tools and workload generators versatile enough to support all three system models. We hope that this work will encourage others to design workload generators that allow exibility in the underlying system model. 8 Prior Work Work explicitly comparing open and closed system models is primarily limited to FCFS queues. Bondi and Whitt [11] study a general network of FCFS queues and conclude that the effect of service variability, though dominant in open systems, is almost inconsequential in closed systems (provided the MPL is not too large). We corroborate this principle and illustrate the magnitude of its impact in real-world systems. Schatte [36, 37] studies a single FCFS queue in a closed loop with think time. In this model, Schatte proves that, as the MPL grows to in nity, the closed system converges monotonically to an open system. This result provides a fundamental understanding of the effect of the MPL parameter; however the rate of this convergence, which is important when choosing between open and closed system models, is not understood. We evaluate the rate of convergence in realworld systems. Though these theoretical results provide useful intuition about the differences between open and closed systems, theoretical results alone cannot evaluate the effects of factors such as trace driven job service demand distributions, correlations, implementation overheads, and size-based scheduling policies. Hence, simulation and implementation-based studies such as the current paper are needed. 9 Conclusion This paper provides eight simple principles that function to explain the differences in behavior of closed, open, and partly-open systems and validates these principles via trace-based simulation and real-world implementation. The more intuitive of these principles point out that response times under closed systems are typically lower than in the corresponding open system with equal load, and that as MPL increases, closed systems approach open ones. Less obviously, our principles point out that: 10 Acknowledgements We would like to thank Arun Iyengar, Erich Nahum, Paul Dantzig, Luis von Ahn, and Chad Vizino for providing access to the logs we used in Section 4.3 and Section 7. References [1] The U.S. Geological Survey. http://www.usgs.gov. [2] Webjamma world wide web traf c analysis tools. http://research.cs.vt.edu/chitra/webjamma.html. [3] Carnegie Mellon University, School of Computer Science. http://www.cs.cmu.edu/, 2005. [4] The Pittsburgh Supercomputing Center (PSC). http://www.psc.edu/, 2005. [5] J. Almeida and P. Cao. Wisconsin proxy benchmark 1.0. http://www.cs.wisc.edu/cao/wpb1.0.html, 1998. [6] W. Almesberger. Linux network traf c control implementation overview. http://diffserv.sourceforge.net, 1999. [7] C. Amza, E. Cecchet, A. Chanda, A. Cox, S. Elnikety, R. Gil, J. Marguerite, K. Rajamani, and W. Zwaenepoel. Speci cation and implementation of dynamic web site benchmarks. In Workshop on Workload Characterization, 2002. [8] M. Arlitt. Characterizing web user sessions. SIGMETRICS Perform. Eval. Rev., 28(2):50 63, 2000. [9] G. Banga and P. Druschel. Measuring the capacity of a web server under realistic loads. World Wide Web, 2(1-2):69 83, 1999. [10] P. Barford and M. Crovella. The surge traf c generator: Generating representative web workloads for network and server performance evaluation. In In Proc. of the ACM SIGMETRICS, 1998. [11] A. B. Bondi and W. Whitt. The in uence of service-time variability in a closed network of queues. Perf. Eval., 6:219 234, 1986. [12] Bradford L. Barrett. The Webalizer log le analysis program. http://www.mrunix.net/webalizer, 2005. [13] T. Cain, M. Martin, T. Heil, E. Weglarz, and T. Bezenek. Java TPC-W implementation. http://www.ece.wisc.edu/ pharm/tpcw.shtml, 2000. [14] R. Chinchilla, J. Hoag, D. Koonce, H. Kruse, S. Ostermann, and Y. Wang. The trafgen traf c generator. In Proc. of Int. Conf. on Telecommunication Sys., Mod. and Anal. (ICTSM10), 2002. [15] M. Crovella, R. Frangioso, and M. Harchol-Balter. Connection scheduling in web servers. In USENIX Symposium on Internet Technologies and Systems, October 1999. [16] A. Feldmann, A. C. Gilbert, P. Huang, and W. Willinger. Dynamics of IP traf c: A study of the role of variability and the impact of control. In Proc. ACM Sigcomm, 1999. [17] J. Fulmer. Siege. http://joedog.org/siege. [18] S. Gigandet, A. Sudarsanam, and A. Aggarwal. The inktomi climate lab: an integrated environment for analyzing and simulating customer network traf c. In Proc. ACM SIGCOMM Workshop on Int. Meas., pages 183 187, 2001. [19] M. Harchol-Balter, B. Schroeder, M. Agrawal, and N. Bansal. Size-based scheduling to improve web performance. ACM Transactions on Computer Systems, 21(2), May 2003. [20] P. E. Heegaard. Gensyn - generator of synthetic internet traf c. http://www.item.ntnu.no/ poulh/GenSyn/gensyn.html. [21] ITA. The Internet traf c archives. Available at http://town.hall.org/Archives/pub/ITA/, 2002. [22] K. Kant, V. Tewari, and R. Iyer. GEIST: Generator of ecommerce and internet server traf c. In Proc. of Int. Symposium on Performance Analysis of Systems and Software, 2001. [23] Z. Liu, N. Niclausse, and C. Jalpa-Villanueva. Traf c model and performance evaluation of web servers. Performance Evaluation, 46(2-3):77 100, 2001. [24] B. A. Mah, P. E. Sholander, L. Martinez, and L. Tolendino. Ipb: An internet protocol benchmark using simulated traf c. In MASCOTS 1998, pages 77 84, 1998. [25] D. T. McWherter, B. Schroeder, A. Ailamaki, and M. HarcholBalter. Priority mechanisms for OLTP and transactional web applications. In Int. Conf. on Data Engineering, 2004. [26] D. Menasce and V. Almeida. Scaling for E-Buisness: technologies, models, performance, and capacity planning. Prentice Hall, 2000. [27] Microsoft IIS 6.0 Resource Kit Tools. Microsoft Web Capacity Analysis Tool (WCAT) version 5.2. [28] Microsoft TechNet. Ms web application stress tool (WAST). [29] Mindcraft. The AuthMark benchmark. http://www.mindcraft.com/authmark/. [30] D. Mosberger and T. Jin. httperf: A tool for measuring web server performance. Performance Evaluation Review, 26(3), 1998. [31] E. Nahum, M. Rosu, S. Seshan, and J. Almeida. The effects of wide-area conditions on WWW server performance. In Proc of ACM SIGMETRICS, pages 257 267, 2001. [32] PostgreSQL. http://www.postgresql.org. [33] M. Rakwat and A. Kshemkayani. SWIFT: Scheduling in web servers for fast response time. In Symp. on Net. Comp. and App., April 2003. [34] L. Rizzo. Dummynet: a simple approach to the evaluation of network protocols. ACM Comp. Comm. Review, 27(1), 1997. [35] A. Rousskov and D. Wessels. High performance benchmarking with web polygraph. Software - Practice and Experience, 1:1 10, 2003. [36] P. Schatte. On conditional busy periods in n/M/GI/1 queues. Math. Operationsforsh. u. Statist. ser. Optimization, 14, 1983. [37] P. Schatte. The M/GI/1 queue as limit of closed queueing systems. Math. Operationsforsh. u. Statist. ser. Optimization, 15:161 165, 1984. [38] sourceforge.net. Deluge - a web site stress test tool. http://deluge.sourceforge.net/. [39] sourceforge.net. Hammerhead 2 - web testing tool. http://hammerhead.sourceforge.net/. [40] Standard Performance Evaluation Corporation. SFS97 R1 (3.0). http://www.specbench.org/osg/web99/. [41] Standard Performance Evaluation Corporation. SPECJ2EE. http://www.specbench.org/osg/web99/. [42] Standard Performance Evaluation Corporation. SPECmail2001. http://www.specbench.org/osg/web99/. [43] Standard Performance Evaluation Corporation. SPECweb99. http://www.specbench.org/osg/web99/. [44] W. Tang, Y. Fu, L. Cherkasova, and A. Vahdat. Medisyn: A synthetic streaming media service workload generator. In Proceedings of 13th NOSSDAV, 2003. [45] Transaction Processing Performance Council. TPC benchmark C. Number Revision 5.1.0, December 2002. [46] Transaction Processing Performance Council. TPC benchmark W (web commerce). Number Revision 1.8, February 2002. [47] G. Trent and M. Sake. WebStone: The rst generation in HTTP server benchmarking. http://www.mindcraft.com/webstone/paper.html. [48] VeriTest. Netbench 7.0.3. http://www.etestinglabs.com/benchmarks/netbench/. [49] VeriTest. Webbench 5.0. http://www.etestinglabs.com/benchmarks/webbench/. [50] L. von Ahn and L. Dabbish. Labeling images with a computer game. In CHI 04: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 319 326, 2004. [51] B. L. Worthington, G. R. Ganger, and Y. N. Patt. Scheduling algorithms for modern disk drives. In Proc. of Sigmetrics, 1994. [52] M. Yuksel, B. Sikdar, K. S. Vastola, and B. Szymanski. Workload generation for ns simulations of wide area networks and the internet. In Proc. of Comm. Net. and Dist. Sys. Mod. and Sim., 2000.

Find millions of documents here - Study Guides, Homework Solutions, Papers, Exam Answer Keys and more. Course Hero has millions of course related materials that will enable you to learn better, faster and get an A in all your courses.
Below is a small sample set of documents: