TCP Modeling Latency
Neal Cardwell, Stefan Savage, Thomas Anderson cardwell, savage, tom @cs.washington.edu Department of Computer Science and Engineering University of Washington Seattle, WA 98195 USA
Abstract Several analytic models describe the steady-state throughput of bulk transfer TCP ows as a function of round trip time and packet loss rate. These models describe ows based on the assumption that they are long enough to sustain many packet losses. However, most TCP transfers across todays Internet are short enough to see few, if any, losses and consequently their performance is dominated by startup effects such as connection establishment and slow start. This paper extends the steadystate model proposed in [34] in order to capture these startup effects. The extended model characterizes the expected value and distribution of TCP connection establishment and data transfer latency as a function of transfer size, round trip time, and packet loss rate. Using simulations, controlled measurements of TCP transfers, and live Web measurements we show that, unlike earlier steady-state models for TCP performance, our extended model describes connection establishment and data transfer latency under a range of packet loss conditions, including no loss.
I. I NTRODUCTION Many of todays popular Internet applications, including the World-Wide Web, e-mail, le transfer, Usenet news, and remote login, use TCP as a transport protocol. As a consequence, TCP controls a large fraction of ows, packets, and bytes that travel over wide-area Internet paths [41, 10]. Recently researchers have proposed a number of analytic models to characterize TCP performance in terms of round-trip delay and packet loss rate [12, 24, 18, 27, 22, 25, 21, 34, 30, 33, 40, 9]. Beyond achieving a better understanding of the sensitivity of TCP performance to network parameters, these models have helped inform the design of active queue management schemes [13, 32] and TCP-friendly multicast protocols [6, 42]. The analytic models proposed to date can be split into two broad classes: models for steady-state bulk transfer throughput, and models for short ows that suffer no packet loss. The majority of models t in the rst class; they focus on characterizing bulk transfer throughput. While these models are very successful at predicting steady-state throughput [5, 38], many recent studies have noted that the majority of TCP ows traveling over the wide-area Internet are very short, with mean sizes around 10KB and median sizes less than 10KB [11, 4, 23, 16, 41, 10]. Because these ows are so short, they often spend their entire lifetime in TCPs slow start mode, without suffering a single loss. Since the steady-state models assume ows suffer at least one loss, they are undened for this common case. The second class of models focuses on these short ows that suffer no packet losses [18, 24, 35]. However, these models do not consider delayed acknowledgments, sender or receiver buffering limitations, alternative initial congestion windows, or losses during connection establishment, each of which can have
a dramatic performance impact. This paper proposes a new model for TCP performance that integrates the results from both classes of models. In particular, we extend the steady-state results from [34] by deriving new models for two aspects that can dominate TCP latency: the connection establishment three-way handshake and slow start. Using simulation, controlled measurements, and live Web traces we show that our new slow start model works well for TCP ows of any length that suffer no packet losses, and the model from [34] often works well for ows of any length that do suffer packet losses. Thus our combined approach, which integrates these two models, is appropriate for predicting the performance of both short and long ows under varying packet loss rate conditions. In addition, we suggest a technique for estimating the distribution of of data transfer latencies. The rest of this paper is organized as follows. Section II describes the new model and relates it to the models from which it is descended. Section III compares the connection establishment model with simulations and Section IV compares the data transfer model to simulations, TCP measurements, and HTTP traces. Finally, Section V summarizes our conclusions. II. T HE M ODEL A. Assumptions The extended model we develop here has exactly the same assumptions about the endpoints and network as the steady state model presented in [34]. The following section describes these assumptions in detail, including a few assumptions not stated explicitly in [34], since these details can have a large impact on the latency of short TCP ows. Throughout our presentation of this model we use the same terminology and notation as [34]. A.1 Assumptions about Endpoints First, we assume that the sender is using a congestion control algorithm from the TCP Reno family; we refer readers to [37, 2, 20] for details about TCP and Reno-style congestion control. While we describe the model in terms of the simpler and more common TCP Reno algorithm, it should apply just as well to newer TCP implementations using NewReno, SACK, or FACK [14, 28, 26]. Previous measurements suggest the model should be even more faithful to these more sophisticated algorithms, as they are more resilient to bursts of packet losses [27, 5]. Since we are focusing on TCP performance rather than general client-server performance, we do not model sender or re-
ceiver delays due to scheduling or buffering limitations. Instead, we assume that for the duration of the data transfer, the sender sends full-sized segments (packets) as fast as its congestion window allows, and the receiver advertises a consistent ow control window. Similarly, we do not account for effects from the Nagle algorithm or silly window syndrome avoidance, as these can be minimized by prudent engineering practice [17, 29]. We assume that the receiver has a typical delayed acknowledgment implementation, whereby it sends an acknowledgment (ACK) for every data segments, or whenever its delayed ACK heartbeat timer expires, whichever happens rst. Although Linux 2.2, for example, uses a more adaptive implementation of delayed ACKs, for very short ows this technique can be modeled well with , and for longer ows the effect of this approach is only to shave off at most one or two round trips. A.2 Assumptions about the Network We model TCP behavior in terms of rounds, where a round starts when the sender begins the transmission of a window of packets and ends when the sender receives an acknowledgment for one or more of these packets. We assume that the time to send all the packets in a window is smaller than the duration of a round and that the duration of a round is independent of the window size. Note that with TCP Reno congestion control this can only be true when the ow is not fully utilizing the path bandwidth. We assume that losses in one round are independent of the losses in any other round, while losses in the same round are correlated, in the sense that any time a packet is lost, all further packets in that round are also lost. These assumptions are idealizations of observations of the packet loss dynamics of paths using FIFO drop-tail queuing [7, 36, 44] and may not hold for links using RED queuing [15] or paths where packet loss is largely due to link errors rather than congestion. We assume that the probability of packet loss is independent of window size; again, this can only hold for ows that are not fully utilizing a link. When modeling data transfer, we assume that packet loss happens only in the direction from sender to receiver. This assumption is acceptable because low levels of ACK loss have only a small effect with large windows, and network paths are often much more congested in the direction of data ow than the direction of ACK ow [41, 39]. Because packet loss has a far more drastic result during connection establishment, we model packet loss in both directions when considering connection establishment. A.3 Assumptions about the Transfer Though we share the assumptions of [34] about the endpoints and network, we relax several key assumptions about the data transfer. Namely, we allow for transfers short enough to suffer a few packet losses, or zero losses, and thus to be dominated by connection establishment delay and the initial slow start phase.
i Failures Passive Opener
SY Nx
Active Opener
1
j Failures
x+
Connection Effectively Established
SY
N
y
C ,A
K
# ! $"
) ' 0 (&%
AC
Ky
+1
Fig. 1. TCP connection establishment example.
B. Model Overview Our model describes two aspects of TCP performance. First, we derive expressions for the expected value and distribution of time required for the connection establishment handshake that begins a TCP connection. Second, we derive an expression for the expected latency to transfer a given amount of data and then describe our methodology for extrapolating the distribution of this transfer latency. Separating connection establishment latency from data transfer latency allows us to apply the model to applications that establish a single TCP connection and use it for several independent data transfers. C. Connection Establishment Every successful TCP connection begins with a three-way handshake in which the endpoints exchange initial sequence numbers. Figure 1 shows an example. The initiating host, typically the client, performs an active open by sending a SYN segment with its initial sequence number, . The server performs a passive open; when it receives a SYN segment it replies with a SYN segment of its own, containing its initial sequence number, , as well as an ACK for the active openers initial sequence number. When the active opener receives this SYN/ACK packet, it knows that the connection has been successfully established. It conrms this by sending an ACK of the passive openers initial sequence number. At each stage of this process, if either party does not receive the ACK that it is expecting within some SYN timeout, , initially three seconds [8], it retransmits its SYN and waits twice as long for a response. To model this process in the presence of packet loss in either direction, we dene as the forward packet loss rate along the path from passive opener to active opener (forward since this is usually the primary direction of data ow) and as the reverse packet loss rate. Let be the average round trip delay between the two hosts. Our model of the three-way handshake consists of the following stages. First, the active opener transmits its SYN times unsuccessfully, until the -th SYN arrives successfully at the passive opener. Next the passive opener will ignore further
SYNs from the active opener while it repeatedly retransmits its SYN/ACK until it receives a response. In general it will send its SYN/ACK times unsuccessfully until nally the -th SYN/ACK arrives successfully at the active opener. For the purposes of the model, we consider the connection to be established at this point, since, in most application protocols, immediately after sending the ACK , the active opener sends a data segment to the passive opener that contains a redundant ACK . Let be the probability of having a three-way handfailures transmitting shake episode consisting of exactly SYNs, followed by one successful SYN, followed by exactly failures transmitting SYN/ACKs, followed by one successful SYN/ACK. Then
data transfer. From this we can deduce the time spent in slow start, the nal congestion window in slow start, and thus the expected cost of loss recovery, if any. Then we use the steady-state throughput from [33] to approximate the cost of sending the remaining data, if any. Finally, we add any extra cost from delayed ACKs. We discuss each of these aspects in turn. D.1 Initial Slow Start We assume that the transfer is either the rst transfer of a connection, or a later transfer on a connection that has experienced no losses yet. Under these circumstances, TCP begins in slow start mode, where it quickly increases its congestion window, , until it detects a packet loss. , the expected latency for the initial slow start phase, depends on the structure of the slow start episode. There are two grows conimportant cases. In the rst case, the senders tinuously until it detects a packet loss. In the second case, the senders is eventually bounded by a maximum window, , imposed by sender or receiver buffer limitations. To determine which case is appropriate, we need to calculate , the number of data segments we expect the sender to send before losing a segment. From this we can deduce , the window we would expect TCP to achieve at the end of slow start, were there no maximum window constraint. If , is simply then the window limitation has no effect, and the time for a sender to send in the exponential growth then mode of slow start. On the other hand, if is the time for a sender to slow start up to and then send the remaining data segments at a rate of segments per round. , the number of data segments we First we calculate expect to send in the initial slow start phase before a loss occurs (not including the lost segment). Let be the data segment loss , we expect to be able to send all segments in rate. If slow start, so . On the other hand, if , and we assume that the loss rate is independent of sender behavior, then
of segments, then If a sender starts with an initial , the amount of data sent by the end of slow start round
'
'
' q # k & # k & 1y ) | `D ( (z )# k |z{y D # k | z {y | z {y
t '
v w '
I~# k }
| z {y
e t
x
As dened here, a data transfer begins when an application places data in its send buffer and ends when TCP receives an acknowledgment for the last byte in the buffer. We assume that during this transfer the sending application places data in the send buffer quickly enough that the sending TCP can send as fast as its window allows. We decompose the data transfer latency, , for data segments into four aspects: the initial slow start phase, the resulting packet loss (if any), the transfer of any remaining data, and the added delay from delayed acknowledgments. We begin by calculating the amount of data we expect to send in the initial slow start phase before encountering a packet loss or nishing the
D. Data Transfer
Next we deduce the time spent in slow start. During slow start, as always, each round the sender sends as many data segallows. Since the receiver sends one ACK for ments as its every -th data segment that it receives, each round the sender will get approximately ACKs. Because the sender is in slow start, for each ACK it receives, it increases its by one segment. Thus, if we use to denote the senders congestion window at the beginning of round and, following [1], use to denote the rate of exponential growth of during slow start , we have:
( ! 0 ( ! 0 ( 0 ( uD qfpi s! 2fpEf) o
This model assumes the TCP implementation complies with the TCP specication [37]. It does not model non-compliant implementations, such as current versions of Linux 2.2, that achieve slightly better performance by responding to retransmitted SYN segments with retransmitted SYN/ACK segments.
k ! 0 ( D W r& ! ! 0 ( S X& o sC)utsmp& sqCp T XR Q I o PG F
R
(4)
#
eh
h
f
h f #
Most TCP implementations abort connection establishment attempts after 4-6 failures. For loss rates low enough that most handshakes succeed before TCP gives up, it can be shown that (4) is a good approximation for the expected handshake time:
lVV j nmik i
h b f
(3)
h
f
The probability that , the overall latency for a three-way handshake episode, is seconds or less is:
' ' 'g fh f e h f g h d' f
'
h
(2)
h X f
f
'
$
WV XB U R Q YXB R Q ECCA D WV U T S D B B I G PH# F IT PSG 3 F
R
V B D U 0 3 U c# V ba0 3 U `D V ba0 # B ( B (
' ' h f7
)
h f
yiwbv u)9rsq p x t % Q
R
The latency,
, for this process is
2'
h
f
(1)
'
'
) 0 ' %
5 21)'65 4)& % 21)'$" ! 0 ( & 3 ! ! 0 ( & #% !
D B B U `ECCA D B B U `ECCA
he g f iY"d7 E
'
d7
) % 7
e
@9 8
# !
) &%
(5)
(6)
, can be closely approximated by a geometric series as
(7) (8)
D.2 The First Loss For some TCP ows, the initial slow start phase ends with the detection of a packet loss. Since slow start ends with a packet loss if and only if a ow has at least one loss, the probability of this occurrence is:
Solving for , the number of slow start rounds to transfer segments of data, we arrive at:
(9)
(11)
(12)
segments during this phase. From (9), this will take
(13)
rounds. During the second phase, the ow sends the remaining packets per round, which will take data at a rate of
(14)
when when
(15)
BCCbwp9l iV pc ! x f) B A& V j i 0 ( D j i& lV j i x & V 9l D m9 {iV pc ! jF 1iV
v
V h iV i B
rounds. Combining (12), (13), and (14) for the case where when , and using (9) for the simpler case where , the time to send data segments in slow start is approximately
The expected cost of a fast recovery period depends on the number of packets lost, the fast recovery strategy of the senders TCP implementation, and whether the receiving TCP implementation is returning selective acknowledgment (SACK) options. In the best case, where there is a single loss or the sender can use SACK information, fast recovery often takes only a single ; in the worst case, NewReno will require one for each lost packet. Our experience indicates that which of these possibilities actually occurs is usually not important to the models predictions, so, as in the model of [34], for the sake of simplicity, we assume that fast recovery always takes a single . Combining these results, the expected cost for any RTOs or fast recovery that happens at the end of the initial slow start phase is:
) %
g! U {uC{D D ! U D ! P( s ! f D e ! d ( D ! ! D
so we can now determine whether we expect to be constrained by during slow start. If then slow start proceeds in two phases. First, grows up to ; from (10) the ow will send
where is the average duration of the rst timeout in a sequence of one or more successive timeouts, and is given by:
(19)
T
!qf( 0 B !
l j PDS i
) f %
) f %
w
w
Given typical parameters of and , equation (10) implies that . Put another way, to reach any congestion window, , a ow needs to send approximately . Interestingly, this implies that to reach full utilization for a bandwidth-delay product like 1.5Mbps 70ms 13KBytes, a TCP ow will need to transfer 39KBytes, a quantity larger than most wide-area TCP ows transfer. From this it is easy to see why many Internet ows spend most of their lifetimes in slow start, as observed in [3]. From (5) and (10) we can calculate the window size we would expect to have at the end of slow start, if we were not constrained by :
" #
) 1 %
w
) 1 %
w
(10)
The probability that a sender will detect a loss via triple duplicate ACKs is simply . Although was derived under the assumption that the sender is in congestion avoidance mode and has an unbounded amount of data to send, our experience has shown that it is equally applicable to slow start and senders with a limited amount of data. This is largely because is quite insensitive to the rate of growth of , and senders with a limited amount of data are already at high risk for RTOs because they will usually have small windows. In practice, we suspect that the fast recovery strategy used by the sender has a far greater impact; senders using SACK, FACK, or NewReno should be able to achieve the behavior predicted , while Reno senders will have difculty achieving by this performance when they encounter multiple losses in a single round [26, 27]. The expected cost of an RTO is also derived in [34]:
(18)
) f %
w
sqCpafp U sqCpEf) ! 0 ( 0 ( ! 0 ( 0 ( p G sqCpaCp4sqCpuPp '( ! 0 ( 0 ( ! 0 ( D (
U
y
c !
z
, the window TCP From (7) and (9) it follows that achieves after sending segments in unconstrained slow start, is
There are two ways that TCP detects losses: retransmission timeouts (RTOs) and triple duplicate ACKs. [34] gives a derivation of the probability that a sender in congestion avoidance will detect a packet loss with an RTO, as a function of packet loss rate and window size. They denote this probability by :
x
(17)
! 0 ( 0 o s21)EC(
V iV
v
I G # {& I
V 4 2Y 83U 0
V W j ubl V i V YU t I 482307 (uD iPG sqi8o gId cdfA rp h h f e F & B B lV T R P SQH IG V ji `W 0 ik i F I lVV j G V D ( W R DP U H F C & B B b`D XSQIGEDdCCA T
g g
'
I D ba29k ( 0 I z I PG {& (`D bE29k & I ( 0 z
z
a b T R SP H
I
D&&& D bb`C{& I z h X f
D ( 0 V j ba9Hl iV k i
) p% I ( ( 0 `D ba9 # k z
( uD
V k I k 0 iV
I
)
z
4 2 30
!
0
( 0 E2
I
x
4 2 0 31
z
&( '% $
z
4 2 0 837
6
( 0 a ( a0 # D d{& I z ( z
I k 'g e h f ' ' '
) )%
lV j niV i
& I D I z z
'
5
I
k V 44iV
5
g h ' e h '
#
9@@
@
@@
@B
A
lVV j s{iB i
k
(16)
'
e t f f
"
(20)
D.3 Transferring the Remainder In order to approximate the time spent sending any data remaining after slow start and loss recovery, we estimate the amount of remaining data and apply the steady-state model from [34]. The amount of data left after slow start and any following loss recovery is approximately
(21)
Another source of error derives from the fact that (22) does not model slow start after retransmission timeouts (RTOs). For loss rates above 1%, the error this introduces should be small, since for these loss rates congestion avoidance has throughput that is similar to that of slow start after RTOs. For lower loss rates, RTOs should be uncommon. However, when they occur, their long delays may overwhelm the details of growth. Finally, RTO durations vary widely. Using an average RTO to model the duration of a specic short TCP ow will introduce signicant error. D.4 Delayed Acknowledgments Delayed acknowledgments comprise the nal component of TCP latency that we consider in our model. There are a number of circumstances under which delayed acknowledgments can cause relatively large delays for short transfers. The most comof 1 mon delay occurs when the sender sends an initial MSS. In this case the receiver waits in vain for a second segment, until nally its delayed ACK timer res and it sends an ACK. In BSD-derived implementations this delay is uniformly distributed between 0ms and 200ms, while in Windows 95 and Windows NT 4.0 this delay is distributed uniformly between 100ms and 200ms. Delayed ACKs may also lead to large delays when the sender sends a small segment and then the Nagle algorithm prevents it from sending further segments [17, 29], or when the sender sends segments that are not full-sized, and the receiver implementation waits for two full-sized segments before sending an ACK. For our simulations measurements, and when senders use an initial of 1 MSS we model the expected cost of the rst delayed ACK, which we denote , as the expected delay between the reception of a single segment and the delayed ACK for that segment 100ms for BSD-derived stacks, and 150ms for Windows. D.5 Combining the Results To model the expected time for data transfer, we use the sum of the expected delays for each of the components, including (15), (20), (24), and the delayed ACK cost:
if
(22)
otherwise
(23)
(24)
Using a model for steady-state throughput to characterize the cost of transferring the remaining data introduces several errors. First, when the sender detects a loss in the initial slow start phase, its will often be much larger than the steady-state average . Combining (10) and the analysis of [27], the sender will have to detect roughly loss indications to bring from its value at the end of slow start, , to its steady state value, . For loss rates of 5% and higher, the sender exits slow start at nearly the steady-state window value, so the error in our approach should be small. For loss rates of 0.1% and below, it can take three or more loss indications corresponding to megabytes of data to reach steady state, so our approach will often overestimate the latency of such transfers.
E. Modeling Distributions The model as given in (25) is a prediction of latency given the particular parameters experienced by a particular ow. In our experience, for a set of transfers of the same size over the same high bandwidth-delay path, the most important determinants of overall latency are the number of losses, the average timeout duration, and the cost of delayed ACKs. As a result, to approximate the distribution of latency for a set of ows, one can consider the range of possible loss rates and delayed ACK costs, and estimate the likelihood of each scenario, along with the latency expected with that scenario. In section IV-A.2 we apply this method to simulations using a Bernoulli loss model and delayed ACK costs uniformly distributed between 0 and 200ms.
R j D l h Co B i sl B Esl iV i B Esl iV B i $CB i j i D V h j i D V j l j
R
2
2
Using these results for the expected throughput, we approximate the expected time to send the remaining data, , as
hQ
f
where is given in (17), is given in (19), and is the expected congestion window at the time of loss events when in congestion avoidance, also from [33]:
'
This is only an approximation, because the actual amount of data remaining will also depend on where in the window the loss occurs, how many segments are lost, the size of the congestion , and the recovery algorithm. window at the time of loss, However, since the model seems accurate in most cases even with this simplication, for the sake of simplicity we use Equation (21). segments left to send, we approxiWhen there are mate the time to transfer this data using the steady-state throughput model from [33]. This model gives throughput, which we , as a function of loss rate, , will denote round trip time, , average RTO, , and maximum window : constraint,
'
PH I&
'
) %
# h f e
"v
4 2 0 31
V 4t } } T R ( " ) &' %& SP # R t T SP H r 2}
w
FA r A 9 7 D B ( E @86 D B ( C
4 2 0 31
f D U
0 1 !
)ff{c ! A l k ul B i B B B A j i j 2 2 T
) ( '&'
V j l iV k i 0 ul k i k j f D 2! ff ) ! 0 ( 5 D D U s !
g ' ) ' 1 % t t r
) %
V !i~t r H r }tI } % " &$ # H r } } V
v ) % "
'
T R SP
T SP R
V
# 1h f e
2
G
} }
V
T SP H R
2'
3
r b
b
' 2'
) f %
@@ @ @@ @B
9@@ @@ @@ A w
`A
(25)
1e+06 900000 800000 Bandwidth (bytes/sec) 700000 600000 500000 400000 300000 200000 100000 0 0.0001
Cumulative Fraction
Proposed (2 KB) Proposed (64 KB) Proposed (256 KB) Proposed (1024 KB) [PFTK98] [MSMO97]
1
0.8
0.6
0.4
Simulated CDF Modeled CDF Simulated mean Modeled mean Approximate Model
0.2
0 0.001 0.01 0.1 Frequency of Loss Indications (p) 1 0 5 10 15 Time (sec) 20 25
12 10 8 Time (sec) 6 4 2 0 0 0.05 0.1 0.15 0.2 0.25 0.3 Forward loss rate 0.35 0.4 0.45 Simulated Modeled Approximate Model
F. Comparison with Earlier Models Our proposed model, (25), is a generalization of two previous approaches. In the case where there are no packet losses, (25) reduces to (15), a model for the time to send segments in slow start mode. This special case corresponds closely to the simpler models derived in [18, 24]. In the case where is very large, the total time given by (25) is dominated by (24), the time to transfer data after the rst loss. In this case, the behavior corresponds very closely with the underlying throughput model, (22), from [34]. Figure 2 explores the relationship between our proposed model and the models of [27] and [34]. It gives the throughput predicted by the proposed model, (25), for each of several transfer sizes, as well as the steady-state throughputs predicted by the expression , from [27], and (22), from [34]. As mentioned earlier, when there is at least one expected loss, our proposed model agrees closely with [34], which has been shown to work well for ows that suffer even a few losses [5]. On the other hand, when there are no losses, the proposed model predicts that short ows suffer because they do not have time to reach a steady-state , whereas long ows will do well because their grows beyond its steady-state value. III. V ERIFYING
THE
Fig. 4. Expected connection establishment latency from ns simulations (1000 trials at each loss rate), the model (3), and the approximate model (4).
imate model, comparing them against 1000 ns trials in scenarand . The full model ts ios with well across these simulations, but the approximate model diverges sharply as approaches 0.5, where its assumption of unbounded wait times fails. IV. V ERIFYING A. Simulations A.1 Flows Without Loss For simulated TCP ows that do not suffer packet loss, the expression (15) describes TCP behavior very closely, as Figure 5 shows. This gure depicts the simulated and modeled latency for 2,376 TCP transfers simulated in ns. In each simulation, a FullTCP sender transfers data over a 1Gbit link to a FullTCP receiver. The link buffers were provisioned to prevent packet loss. The trials consisted of the cross-product of segments, (with and withTHE
C ONNECTION E STABLISHMENT M ODEL
DATA T RANSFER M ODEL
$ ! & !
x
$ " ! %# 0"
2
Figure 3 compares the mean and distribution of connection establishment times given by (3) and the mean given by the approximate model (4) to the mean and distribution for 1000 ns [31] simulations with ms and Bernoulli packet losses with and . These simulations used the FullTCP implementation, modeled closely after the 4.4BSD TCP implementation. Both the model and the approximation t well. Results are similar for other scenarios with both and well below 0.5. Figure 4 summarizes the performance of the full and approx-
f
5 !
!
#
g
g #
# #
U % !
Fig. 2. Comparing the throughput predicted by the steady state models to the throughput predicted by our proposed model for varying transfer sizes. Here bytes, ms, segment, , sec, MBytes.
( B T
Fig. 3. Distribution and mean of connection establishment times from the model , (3), the approximate model (4), and 1000 ns simulations with .
q (
v ) v ) E % fc"
( I
# #
z
(
B B uCCA 2'
"
G
#
(
d 4 ( 3 c0 2
160 Modeled Transfer Time (RTT) 140 120 100 80 60 40 20 0 0 20 40 60 80 100 120 Simulated Transfer Time (RTT) 140 160 Modeled Time Simulated Time Cumulative Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.5 1 1.5 2 2.5 3 3.5 4 Time (sec) Simulated CDF Simulated (Mean) [PFTK98] Proposed Proposed CDF
Fig. 5. The simulated latency for 2,376 TCP data transfers that experienced no packet loss, compared with the modeled latency from (25) (or, equivalently, (15)).
600000 500000 Bandwidth (bytes/sec) 400000 300000 200000 100000 0 1e-05
Fig. 7. The distribution and mean of latencies from the experiment described in Figure 6.
2.5e+06 Simulated Proposed [PFTK98] [MSMO97]
2e+06 Bandwidth (bytes/sec)
Simulated Proposed [PFTK98] [MSMO97]
1.5e+06
1e+06
500000
0.0001
0.001
0.01
0.1
1
0 1e-05
0.0001 0.001 0.01 0.1 Frequency of Loss Indications (p)
1
1
out delayed ACKs), segments, bytes, segments, and ms. The model agrees quite closely with the simulations; the average error is 0.69 s, and the average relative error is 22%. The three outliers at 37, 72, and 143 s correspond to trials with a window of just 8 segments, where throughput was hurt because the 200ms delayed ACK timer of the recipient was mis-aligned with the 256ms ACK-clocking employed by the sender. A.2 Flows Suffering Losses Figures 6 and 7 illustrate how well [34] and (25) match the performance of ows that suffer moderate-to-high levels of loss. Figure 6 shows a scatter plot depicting the bandwidth and loss rate experienced by each of the 100 simulated FullTCP ows, with the model predictions overlaid. Each ow transferred
0.9 0.8 Cumulative Fraction 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 1 2 3 4 Time (sec) 5 6 Simulated CDF Simulated (Mean) [PFTK98] Proposed Proposed CDF
Fig. 9. The distribution and mean of latencies from the experiment described in Figure 8.
B
d
% ! I 5 ! ( (
Fig. 6. Scatter plot of simulated performance with model predictions overbytes, of laid. These were 64 KByte transfers with 4MBytes, segment, , ms, ms, and .
Fig. 8. Scatter plot of simulated performance with model predictions overlaid. These were 1 MByte transfers with bytes, of 4MBytes, segment, , ms, ms, and .
4 2 30
Frequency of Loss Indications (p)
T
B B ( tCCA ( d c( `
z
4 2 0 3%
(
T
$ % $ $ 0 ! 0 % " # ! ' !
B
B B ( tCCA ( d( cYE
`
! $ ! # " ! ! "
%! I (
z
5!
1
1 Measured Proposed (Slow Start) (15) Proposed (Full) (25) [PFTK98]
0.9 0.8 Cumulative Fraction 0.7 0.6 0.5 0.4 0.3 0.2 Proposed Model
0.8
Transfer Time (sec)
0.6
0.4
0.2
0.1 0 -1
0 20000 40000 60000 80000 Data Transferred (Bytes) 100000 120000
0
-0.5
0 Error (RTT)
0.5
1
Fig. 10. Measured and modeled latencies for 403 transfers from the University of Washington to the UC-Davis.
Fig. 11. The error, modeled measured , between the proposed model, (25), and the HTTP measurements, for all 33,208 ows (97%) that suffered no packet losses. Note that this is -normalized error, so the model is within 1 of the actual time for 85% of ows.
1 0.9 0.8 Cumulative Fraction 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 -1 -0.5 0 Relative Error 0.5 1 Proposed Model [PFTK98] [MSMO97]
64KBytes over a path with synthetically-generated Bernoulli and . The losses with an average loss rate of proposed model, (25) ts the trials that experience no packet loss, while both [34] and (25) provide a reasonable t to those trials that do experience loss. Figure 7 shows the distribution of latencies for these trials. Both [34] and (25) capture the average latency, and the modeled distribution, derived using the technique described in Section II-E, provides a reasonable characterization of the distribution of latencies. There is considerable variance in the latency in this case, with 25% of ows completing in half the average time, and 25% of ows taking half again as long as the average time. In our experience, this technique yields a good approximation to the latency distribution whenever there are enough packet losses for [34] to provide a good t for the average latency. Figures 8 and 9 provide the corresponding view of long transfers (1 MByte) over paths with low loss rates. The proposed model, (25), captures the average latency as well as the latency experienced by the half of ows that see no loss. However, neither [34] nor (25) predicts the performance seen by ows that experience a single loss, as these ows enter congestion avoidfar larger than the steady-state value. We have ance with a preliminary results characterizing the dynamics of as it converges to the steady-state value after slow start. It should be possible to capture aspects of the behavior of these long ows that suffer only a few losses by using an approach along these lines. B. Controlled Internet Measurements In order to examine how well our proposed model, (25), ts TCP behavior in the Internet, we performed a number of TCP transfers from a Linux 2.0.35 sender at the University of Washington to other Internet sites. Figure 10 shows an example. It depicts the latency of 403 transfers of varying sizes to the University of California at Davis, together with the predictions from (25) and [34], using the average packet loss rate across all trials. Since the average loss rate was only 0.02%, and as a
Fig. 12. The relative error between the models and the HTTP measurements, for all 357 ows (1%) that suffered triple duplicate ACKs but no RTOs.
consequence most ows suffered no losses, (25) ts quite well, whereas [34] does not. C. Live HTTP Measurements In order to undestand how well the various TCP models describe typical TCP data transfers, we compared them to a set of HTTP traces. These traces consist of client-side packet-level traces of 34,318 TCP ows transporting single HTTP GET operations made from wget web clients at three well-connected U.S. universities the University of Washington-Seattle, the University of California-Berkeley, and Duke University to 50 web servers spread throughout the US. Twenty-ve of the servers were chosen from the webs most popular sites, as determined by the analysis of web proxy logs [19]. The remaining 25 servers were chosen at random from Yahoo!s database of web sites [43]. Each HTTP GET operation fetched the index.html object from the given site; the average size of
B B CCA B B CCA
0
B B CCA
'
#
!
...