Fun The of using TCP for an MMORPG
Carsten Griwodz1,2 , Pl Halvorsen1,2
1
IFI, University of Oslo, Norway
2
Simula Research Laboratory, Norway
{griff, paalh}@ifi.uio.no
ABSTRACT
Massive multi-player online games have become a popular, fast growing, multi-million industry with a very high user mass supporting hundreds or thousands of concurrent players. In many cases, these games are centralized and every player communicates with the central server through a timecritical unicast event stream. Funcom's Anarchy Online is one of these; it is based on TCP. We find that its kind of traffic has some interesting properties that inspire changes to protocol or architecture. In these game streams, TCP does not back off, using TCP does not have to be slower than using UDP, and almost only repeated timeouts ruin the game experience. Improving the latter in the sender implementation does not impose any remarkable penalty on the network. Alternatively, a proxy architecture for multiplexing could save about 40% resources at the server, allow congestion control to work and also reduce the lag of the game.
Keywords
MMORPGs, thin streams, TCP performance
1.
INTRODUCTION
Large improvements in computer technology have enabled highly interactive distributed applications such as distributed virtual environments, massive multi-player online games (MMOGs) and computer supported cooperative work. These applications often include several types of media ranging from text to continuous media and may have stringent requirements with respect to the quality of the client data playout. In the MiSMoSS project, we aim for better system support for such applications by trying to make more efficient use of the available resources and increase the perceived quality at the user. In particular, we look at MMOGs due to the mix of different media, the stringent latency requirements, the dynamic client groups and the fact that it has become a popular, fast growing, multi-million industry with
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NOSSDAV '06 Newport, Rhode Island USA Copyright 2006 ACM 1-59593-285-2/06/0005 ...$5.00.
a very high user mass. Today, MMOGs are increasing in size and complexity, supporting hundreds or thousands of concurrent players [13], and they typically include a mixture of game situations from role-playing games, first person shooter games and real-time strategy games. Players in the game move around and interact with other players, seemingly as if they were located next to each other. Frequently, many players interact with each other and the same object in the game world; these are then said to share an areaof-interest (AoI) or to be within each others AoI. In such a case, it is important that the users get the same information more or less at the same time to have a consistent view of the game world. Today, most MMOGs apply a central server (or a central server cluster) approach for collecting and processing game events generated by players, and point-to-point communication for the distribution of game state updates. With respect to delivering events to players in time, the game traffic must compete with all other traffic in the network, and in case of errors, the servers usually use standard protocol implementations available in the operating system kernels. In this paper, we focus on the challenge of delivering realtime data in MMOGs. As a starting point, we examine the traffic of a particular game, Funcom's popular massive multi-player online role playing game (MMORPG) Anarchy Online [10], which uses an unmodified default TCP variation to deliver data to the players. A trace from one of Anarchy Online's game regions shows that the event streams are very thin (i.e., less than 4 small packets per second), and some users experience huge delays due to retransmissions. We therefore look at the various TCP mechanisms in Linux to see if the experienced game play can be improved. Our experimental results show that, in such thin game streams, 1) TCP does not back off, i.e., each stream may be thin, but the application may have hundreds or thousands concurrent streams which together may give congestion without reducing the sending rate, 2) using TCP does not have to be slower than using UDP as the send buffer is usually empty and an event may be sent immediately, and 3) almost only repeated timeouts ruin the game experience because the number of in-flight packets is so small that fast retransmissions due to multiple duplicate acknowledgments is an exceptional occurance. Furthermore, improving the latter in the sender implementation does not impose any remarkable penalty on the network. Alternatively, a proxy architecture for multiplexing could save about 40% resources at the server, allow congestion control to work and also reduce the lag of the game.
100
max RTT max application delay avg RTT 250ms mark number of packets per RTT
1.2
180 160
1 140 connection index 0 20 40 60 80 100 120 140 connection RTTs sorted by packets/RTT 160 180 0.8 120 100 80 60 40 0.2 20
time in seconds
10
0.6
1
0.4
0.1 0 20 40 60 80 100 120 140 connections sorted by max values 160 180
0
0 0 20 40 60 80 100 connection index 120 140 160 180
(a) RTT versus maximum application delay
(b) Packets per RTT with standard deviation
(c) Matrix of connections that probably share a path
Figure 1: Anarchy Online packet trace analysis The rest of this paper is organized as follows. In section 2, we briefly describe some of the main observations from the Anarchy Online game trace. Section 3 presents some related work, and experiments and results using the various Linux implementations (and enhancements) are presented in section 4. Finally, we summarize the paper in section 5. tions have a high probability of shared paths. It visualizes the results of a Wilcoxon test [11] of connection pairs that checks whether their RTT values stem from the same RTT base value set. A dot in the figure shows that two connections with high probability share a path. With respect to losses, the original trace shows a loss probability of slightly less than 1 % but contains several instances of 6 retransmissions. This implies that it is not the loss rate itself that is unacceptable, but the occasional huge delays when multiple retransmissions are needed. Moreover, we have not found any correlation between losses of the various connections in the trace, and we would like to conclude that losses across the connections in the trace are uncorrelated. This would imply that losses are not due to server-sided bottlenecks. We use this as a working assumption, however, the thinness of the streams may actually hide loss correlation. In summary, the lag that is experienced by players with lossy connections can be huge, and the thinness of the streams implies that retransmissions occur more or less only due to timeouts, since less than one packet is sent per RTT. Simultaneously, the congestion window grows very slowly even in slow start since several writes will be merged into a single packet, and repeated losses can keep the congestion window size close to 1. Thus, the main challenge for this kind of traffic is not the bandwidth, but the delay due to retransmissions that is typical for this kind of distributed multimedia application today. Therefore, it is important to note and deal with the most important observations, i.e., the connections are so thin that 1) they hardly ever trigger fast retransmissions but mainly retransmit due to timeout and 2) TCP's congestion control does not apply, i.e., the TCP stream does not back off. We therefore consider it reasonable to look at the TCP stack for an alleviation. Additionally, there are several connections that probably share several links in the path to the server, and we should use this observation to conserve network resources.
2.
ANALYSIS OF ANARCHY ONLINE GAME TRAFFIC
To look at an example, we have analyzed a one hour packet trace1 , obtained using tcpdump, containing all packets from one of a few hundred game regions in Funcom's popular MMORPG Anarchy Online. Each machine in the centralized server cluster manages one or more regions, i.e., no region spans several machines. Furthermore, Anarchy Online keeps point-to-point, default (New Reno) TCP connections open, from one or more Linux servers to every client. We found approximately 175 distinct connections in this packet dump, and knowing that the servers are located in the US, one can assume from the observed minimum latencies that there are players concurrently located in the US, Europe and Asia. Some of our findings are depicted in figure 1. From figure 1(a), we see that the average RTT is somewhat above 250 ms with variations up to one second, i.e., these RTTs make the game playable [7]. However, looking at the maximum application delay (time for receiving a successful acknowledgment) which may include several retransmissions as a result of packet loss, we have extreme worst case delays of up 67 (!) seconds. Obviously, we cannot distinguish between lost packets and lost acknowledgments in this serversided trace, but we can see the potential for several secondlong delays in delivering packets to the application on the client side. Furthermore, figure 1(b) shows that, on average, less than one packet is sent per RTT. Combined with the measured RTTs, we see that the number of packets per second is low, below 4 packets per second. Considering that each packet is very small (about 120 bytes on average, see table 1), this demonstrates how thin the individual streams are. Finally, figure 1(c) shows that groups of these connecWe have mostly looked at server to client communication as the client to server traffic is very smooth, and we do not have any control over the client TCP in order to make changes.
1
3. RELATED WORK
The body of work that has analyzed game traffic has grown considerably in the recent past. For example, Claypool [7] has investigated how latency affects the perceived quality for real-time strategy games where the results show that some latency is tolerable. Moreover, Feng et al. [5, 9] provide a comprehensive analysis of Counter-Strike traffic
and investigate traces of several games concerning the predictability of game workloads. There, the conclusion is that game traffic varies strongly with time and with the attractiveness of the individual game. Chen et al. [6] investigate the traffic of MMORPGs. They find that streams are in general very thin, but that they are also bursty and show a correlation of inter-arrival times on the minute scale within individual streams. Furthermore, fitting multi-player game traffic to probability distributions is described by Borella [1]. Little work has been done on providing network-related mechanisms specifically for games, but Busse et.al. [3] present an admission control algorithm for a wireless network. They show its suitability for a simple game whose bandwidth usage approximately follows a Poisson distribution. However, we do not know any papers that analyze how underlying mechanisms affect the game performance and make proposals for improving the support for our kind of network traffic in the general Internet. In Anarchy Online, TCP is used for communication, and we assume that this will remain the appropriate choice for MMORPGs for a while, due to the current state of firewall technology. We will therefore look more closely at TCP. Many variations of TCP exist to improve the throughput of individual streams without penalizing other TCP streams under various conditions. The most basic TCP variant that is included in the Linux kernel is New Reno. It performs the usual slow start mechanism, halves the congestion window threshold and reduces the congestion window to 1 on timeout. If it receives duplicate acknowledgments it also performs fast retransmit and a variation of fast recovery that improves on TCP Reno. In Linux, it can be combined with selective acknowledgments (SACK). SACK was developed independently from New Reno. It resolves the ambiguity in senders' interpretation of duplicate ACKs by listing additional ranges of arrived bytes behind a gap. Also available are duplicate SACKs (DSACKs) that allow the receiver to inform that an ACK does not belong to new but to duplicate data. Forward acknowledgment (FACK) [12] is a SACK extension that keeps track of the amount of data that has been received and triggers a fast retransmit when the receiver indicates in a SACK block that it has a gap that is at least three segments long. In addition to these variations, Linux offers a variety of TCP variations meant for higher throughput over connections with a high bandwidth-delay product. Binary increase congestion control (BIC) [14] tries to find the fair share of the bandwidth faster than Reno by performing a binary search for the number of packets per RTT that can be sent without packet loss. However, this happens only between two thresholds for the congestion window size. Below the lower threshold, normal congestion window development applies. Above the higher threshold, additive increase is applied. Vegas [2] uses RTT measurements to detect whether queue lengths increase, and stops to increase transmission speed when they do. This leads to very stable operation if only Vegas is used. The Linux implementation does not modify retransmissions in any way, but within one RTT, it can reaffirm the appropriate transmission speed after the connection has been temporarily idle. Westwood [4] avoids Vegas' problem of loosing bandwidth in competition with New Reno connections. It uses the flows of ACKs instead of RTTs to estimate the bandwidth. This algorithm does not modify basic loss recovery either.
There are, however, TCP variations not included in Linux that make such changes. A variation is fixed-RTO [8] that is meant to improve TCP performance over wireless networks. It does this, but on the general Internet it would obviously be considerably more aggressive in recovering after a congestion event than the other TCP variations.
4. EXPERIMENTS
Figure 1 shows that the basic average RTT between the Anarchy Online servers and the clients is rather high. But sporadically, clients experience really huge lag (delay in the application). This might be devastating to the perceived game experience. It is easily confirmed that the problem is due to the thinness of the streams. Few packets sent per RTT prevent fast retransmission, the congestion window does hardly grow, and packet loss is only detected by timeout, which additionally reduces the congestion window to 1 MSS. In addition, the sender reacts to repeated losses with exponential back-off. To see whether the existing TCP variants in Linux could improve the situation, we replayed the packet trace through emulated lossy networks. FACK, for example, does not require triple duplicate ACKs to trigger retransmission and could reduce average lag, although probably not maximum. Additionally, we have looked at the possible gain of sending all the data in one single connection to save network resources. The idea is that the packets are redistributed to the clients by a proxy server.
4.1 Test Setup
We performed the tests using two small local networks and three Pentium4 machines. The server application regenerates and sends all the original packets in the trace to the client (retransmissions present in the trace are not sent). To simulate delay, jitter and loss, we used netem (Linux 2.6.15 kernel) to add 100 ms and 300 ms delay with and without 10 % jitter. Due to instabilities of netem, we used a separate packet dropper implemented on a network processor (IXP2400) to drop 1 % and 5 % of the packets, respectively. The properties apply both ways in the network, i.e., since we have as many lost acknowledgments as lost packets, the client will have received about half of the retransmitted packets early, improving the actual situation from the perspective of the gamer.
4.2 Replaying the Game using Different Linux TCP Variants
To see if there is any difference in how the various Linux TCP versions perform with respect to retransmission delay when low-rate, real-time event streams are sent to clients, we tested the existing Linux variants, i.e., New Reno (plain, with SACK, DSACK, FACK, and with DSACK and FACK), Westwood, BIC and Vegas. Additionally, we ran the tests with and without LOWLAT and FRTO turned on, but these options did not change anything. Figure 22 shows the average time between the first transmission of a packet from a server and the retransmission that is finally successfully acknowledged. Each of the subfigures shows the average time for the 8 TCP settings in The results in figure 2 and 3 show the 100 ms delay only. However, the 300 ms experiments show identical results only with higher delays according to a higher RTT.
2
Times for first retransmission RTT=100ms 900 800 2500 700 600 Time in ms 500 400 300 200 500 100 0 reno var+1p sack var+1p dsack var+1p fack var+1p dsack fack var+1p westwood var+1p bic var+1p vegas var+1p reno var+5p sack var+5p dsack var+5p fack var+5p dsack fack var+5p westwood var+5p bic var+5p vegas var+5p reno 1p sack 1p dsack 1p fack 1p dsack fack 1p westwood 1p bic 1p vegas 1p reno 5p sack 5p dsack 5p fack 5p dsack fack 5p westwood 5p bic 5p vegas 5p 0 Time in ms 2000 1500 1000 3000
Times for second retransmission RTT=100ms
reno var+1p sack var+1p dsack var+1p fack var+1p dsack fack var+1p westwood var+1p bic var+1p vegas var+1p
(a) Successful retransmission
(b) 1st Successful 2nd retransmissions
Figure 2: Average retransmission delay, simplex streams, 100ms delay the 2.6.11 kernel in various situation. The figures are arranged in four blocks according to network properties, i.e., 1 % loss/no delay variation, 1 % loss/10 % delay variation, 5 % loss/no delay variation and 5 % loss/10 % delay variation. Looking at the results, there are only small variations between the different options available in Linux. The first retransmissions are delayed by more or less the same amount of time3 (about 400 ms, figure 2(a)). Since nearly all retransmissions are triggered by timeouts, the second (and later) retransmission have delays according the TCP backoff mechanism, i.e., about 1200 ms as shown in figure 2(b) for the second retransmission. Obviously, changing the TCP version itself has no or very little effect in this scenario. Therefore, to see if we can improve the gaming quality by other means, we have experimented with a modified server architecture introducing proxies and a modified retransmission timeout (RTO) calculation at the sender in section 4.3 and 4.4, respectively. ers comprise a huge part of the number of transmitted bytes. As table 1 shows (for FACK), a lot of resources can be saved at the server. In our scenario, where about 175 individual thin streams are multiplexed into a single connection with an average data rate of 0.122 Mbps, we have reduced the amount of data and the number of packets by approximately 40 %. While this may seem irrelevant for the individual connection, an operator of a successful MMORPG has to dimension the access network for tens or even hundreds of thousands of concurrent connections. Besides this, we see also the hoped-for improvement of per-client latency. We used the settings from section 4.2 to emulate multiplexed streams (figure 3). Comparing the two alternatives, we see that the average latency experienced over multiplexed connections is lower if the packet loss rate is low. In this approach, fast retransmissions are typical. This implies that the multiplexed stream is subject to congestion control, and an MMORPG that uses this approach would have to be able to scale the multiplexed stream to adapt to the available bandwidth. It is also important to note that the reduction in retransmission times is less pronounced when the latency is higher. The high-speed variations do not perform well in the thin streams scenario. We found this particularly surprising for Westwood+, which is meant for efficient transfer over wireless connections. However, the implementation does never touch the smoothed RTT which is used for the RTO calculation; Westwood+ is meant to work well in the wireless scenario by not simply halving the congestion window in case of loss. If throughput is not the issue, it provides no advantage. Similarly, Vegas affects only congestion control. According to figure 3, the most promising of all TCP settings in our scenario are the more recent extensions to New Reno, DSACK and/or FACK. For FACK, figure 4 shows a direct comparison of average latencies between direct and multiplexed connections. It demonstrates that the average latency is lower for the multiplexed connections, but it shows also that the advantage is lost when the packet loss rate is very high. In the following section, we are therefore looking at a the potential of a protocol variation.
4.3 Modified Architecture
To see the effect on delay and server resources, we compared individual connections to every client with a single connection carrying the multiplexed streams for all clients to an imaginary proxy which performs operations on behalf of the server and splits the single multiplexed TCP connection to each individual client. The test depicted in figure 1 has shown that quite a few clients probably share a path, so the proxy approach makes sense in that manner. A separate consideration is the increase in end-to-end latency that is naturally introduced by a proxy architecture. Minimal and average end-to-end latency would degrade, and the increase in bandwidth share that comes with TCP connection splitting is irrelevant in our scenario. However, we would benefit from improved loss recovery times. Several interesting observations can be made. First, since packets are small (far below 100 bytes), the appended head3 The timeout calculation in Linux increases the average delay with increasing RTT variance. However, with a maximum 10 % variance, this variation is too small to have a visible impact.
reno var+5p sack var+5p dsack var+5p fack var+5p dsack fack var+5p westwood var+5p bic var+5p vegas var+5p
reno 1p sack 1p dsack 1p fack 1p dsack fack 1p westwood 1p bic 1p vegas 1p
reno 5p sack 5p dsack 5p fack 5p dsack fack 5p westwood 5p bic 5p vegas 5p
total time simplex multiplex 1855 s 1804 s
total #bytes 44974424 bytes 27582922 bytes
total #packets 372711 113149
average packet size 120 bytes 243 bytes
bandwidth requirement 0.194 Mbps 0.122 Mbps
Table 1: Packet statistics (New Reno with FACK, 100 ms delay, 1% loss)
Times for first retransmission RTT=100ms 800 700 600 Time in ms Time in ms 500 400 300 200 100 0 reno var+1p sack var+1p dsack var+1p fack var+1p dsack fack var+1p westwood var+1p bic var+1p vegas var+1p reno var+5p sack var+5p dsack var+5p fack var+5p dsack fack var+5p westwood var+5p bic var+5p vegas var+5p reno 1p sack 1p dsack 1p fack 1p dsack fack 1p westwood 1p bic 1p vegas 1p reno 5p sack 5p dsack 5p fack 5p dsack fack 5p westwood 5p bic 5p vegas 5p 1800 1600 1400 1200 1000 800 600 400 200 0 reno var+1p sack var+1p dsack var+1p fack var+1p dsack fack var+1p westwood var+1p bic var+1p vegas var+1p reno var+5p sack var+5p dsack var+5p fack var+5p dsack fack var+5p westwood var+5p bic var+5p vegas var+5p reno 1p sack 1p dsack 1p fack 1p dsack fack 1p westwood 1p bic 1p vegas 1p reno 5p sack 5p dsack 5p fack 5p dsack fack 5p westwood 5p bic 5p vegas 5p Times for second retransmission RTT=100ms
(a) Successful 1st retransmission
(b) Successful 2nd retransmissions
Figure 3: Average retransmission delay, a multiplexed stream, 100 ms delay
Times for FACK 2500 simplex multiplex
2000 Time in ms
1500
1000
smoothed RTT (7/8 srtt + 1/8 new rtt), rttvar is the variance of the RTT (3/4 medium rtt deviation + 1/4 new rtt) and n is the number of retransmissions. In our modified scheme, we used a new RTO calculation if less than 3 packets are "in-flight" (packets_out < 3), i.e., when we assume a thin stream that will not cause congestion. This RTO calculation is implemented as
min([(srtt 3)+min((srtt 4), rttvar)], T CP _RT O_M AX).
500
0 100ms var+1p 1retr 100ms var+5p 1retr 100ms var+1p 2retr 100ms var+5p 2retr 300ms var+1p 1retr 300ms var+5p 1retr 300ms var+1p 2retr 300ms var+5p 2retr 100ms 1p 1retr 100ms 5p 1retr 100ms 1p 2retr 100ms 5p 2retr 300ms 1p 1retr 300ms 5p 1retr 300ms 1p 2retr 300ms 5p 2retr
Figure 4: Average retransmission delay, comparing FACK results
4.4 Modified Kernel
To see whether we can gain anything from making the TCP retransmission scheme slightly more aggressive to better support thin real-time event streams without violating the basic congestion mechanisms, we modified a 2.6.15 kernel. The default standard RTO calculation and the traditional back-off mechanism is implemented as (using Linux notation4 )
min([(srtt 3) + rttvar] n, T CP _RT O_M AX)
where T CP _RT O_M AX is 2 minutes (default), srtt is the
4 Other implementations may use other formulas like RT O = rtt + 4 mdev which for example is used in BSD.
Thus, as long as the traditionally calculated timeout is not shorter, this scheme uses the smoothed RTT only for the RTO calculation and does not apply back-off. Note, however, that our experimental modification is not meant to provide a perfect formula for thin streams. For example, the approach is not well suited for very high loss rates or correlated losses, and aggressive piggy-backing on other packets should also be considered. However, this is subject for further research, and we wanted here only to see if such a scheme could be used without wasting too many resources in additional retransmissions. To see how the modified RTO scheme influences the retransmission delay, we reran a selection of the tests using New Reno and the basic extensions. The average retransmission delay with standard deviation for FACK is shown in figure 5 (the other results are similar). The retransmission latency can never be higher than in the original scheme, but the improvement increases considerably with the number of retransmissions that are necessary, i.e., the lag experienced by the player due to repeated packet loss is reduced. On the other hand, the approach ignores rttvar and tends to choose a more aggressive retransmission timer setting. This may increase the number of unnecessary retransmissions due to early timeouts. In table 2, we present the num-
Standard and modified timeout calculation, 100ms delay, average retransmission time 5000 old new 5000
Standard and modified timeout calculation, 300ms delay, average retransmission time old new
4000
4000
Time in ms
2000
Time in ms
3000
3000
2000
1000
1000
0 var+1p 1retr var+5p 1retr var+1p 2retr var+5p 2retr var+1p 3retr var+5p 3retr 1p 1retr 5p 1retr 1p 2retr 5p 2retr 1p 3retr 5p 3retr
0 var+1p 1retr var+5p 1retr var+1p 2retr var+5p 2retr var+1p 3retr var+5p 3retr 1p 1retr 5p 1retr 1p 2retr 5p 2retr 1p 3retr 5p 3retr
(a) 100 ms path delay, successful 1., 2. and 3. retransmissions
(b) 300 ms path delay, successful 1., 2. and 3. retransmissions
Figure 5: Comparing standard and modified RTO, New Reno with FACK
delay variance 0% 10% 0% 10% 0% 10% 0% 10% 0% 10% 0% 10% 0% 10% 0% 10% total #packets 184230 184213 182687 182855 182480 182820 176573 175851 184164 184145 179416 179927 177401 182320 161321 159906 max #retrans. 3 3 3 3 4 4 5 4 3 3 3 3 4 6 4 4 1st retrans self incl. 3400 3298 2737 2717 15187 15253 11729 12081 3369 3396 2709 2659 14654 15158 11527 11413 1128 1030 1545 1601 5345 5304 7069 7209 1273 1284 1530 1544 4461 4642 5355 5295 2nd retrans total self incl. 65 69 32 43 1236 1153 855 945 58 65 43 46 1433 1352 1119 1111 53 60 22 30 1055 945 629 705 49 51 32 35 1185 1135 846 838 12 9 10 13 181 208 226 240 9 14 11 11 248 217 273 273 3rd retrans total self incl. 2 3 0 0 127 115 54 93 2 1 4 0 143 116 93 98 2 2 0 0 109 96 45 65 2 1 2 0 121 101 76 78 0 1 0 0 18 19 9 28 0 0 0 0 22 15 17 20
loss Standard 1% 1% 1% 1% 5% 5% 5% 5% Modified 1% 1% 1% 1% 5% 5% 5% 5%
delay 100ms 100ms 300ms 300ms 100ms 100ms 300ms 300ms 100ms 100ms 300ms 300ms 100ms 100ms 300ms 300ms
total 4528 4319 4282 4318 20532 20557 18798 19290 4642 4680 4239 4203 19115 19800 ...