(Circuit Outline
CTCP TCP) v1.0
Helali Bhuiyan and Malathi Veeraraghavan {helali,mv5g}@virginia.edu
Purpose of CTCP Requirements to run CTCP code CTCP Components CTCP Operation CTCP patch details CTCP Usage Usage of CTCP across an Internet path
July 06, 2006 2
July 06, 2006
1
Introduction
What is CTCP?
It is a transport protocol for dedicated end to end circuits. Implemented by disabling TCP congestion control.
Purpose of CTCP
IP routers NIC I Host1 NIC II
Internet Internet
IP routers NIC I Host2
Why CTCP?
Bandwidth is reserved for an end to end circuit in a call setup phase. Therefore there is no contention for resources (no congestion) in the data plane. CTCP maintains a constant sending rate matched with the reserved bandwidth of the circuit.
GbE
CHEETAH CHEETAH Network Network
GbE
NIC II
Circuit (dynamically
What does it do?
Converts a TCP socket into a CTCP socket disabling congestion control algorithm.
What it does not do
Setup a circuit between end hosts.
Sycamore SN16000 setup and Sycamore SN16000 SONET released) MCNC, NC Atlanta, GA CTCP was developed for use in the CHEETAH (http://cheetah.cs.virginia.edu/) network. Each host in CHEETAH is equipped with two NIC cards.
The primary NIC is connected to the internet; it is used to carry control-plane messages. The secondary NIC (data plane) is connected to the CHEETAH network.
TCP connections set up between two NIC IIs that are connected via a circuit are configured to operate in CTCP mode
July 06, 2006 3 July 06, 2006 4
1
Requirements for CTCP
Linux Kernel v2.6.11 patched with the corresponding web100 patch (http://web100.org/) Plans are underway to upgrade CTCP to 2.6.16.
CTCP Components
Kernel Space
CTCP patch that modifies regular TCP inside the kernel disabling the congestion control algorithm.
User Space
ctcpWad This is a user-space application which runs as a daemon. It communicates with the kernel via web100. Tester A users space application to submit a request for a CTCP connection configuration.
July 06, 2006
5
July 06, 2006
6
CTCP patch
Modifies both TCP kernel code and Web100 code Details of the CTCP patch will be provided later Usage: Compile kernel with this patch
ctcpWad
This application runs as a daemon on both sending and receiving hosts. Listens on a UNIX socket for messages carrying the following parameters:
Source IP address Destination IP address Sending Rate
ctcpWad polls the web100 code in a loop awaiting the creation of TCP connections between the specified source IP address and destination IP address. Upon detection of a new TCP connection between the specified source and destination IP addresses, it sets parameters related to the sending rate.
July 06, 2006 8
July 06, 2006
7
2
ctcpWad
This application has two parts:
Front End (FE) Back End (BE)
ctcpWad
When ctcpWad receives a CTCP connection configuration request from tester, it does two things
Tell its BE to look for a TCP connection that matches the connection specification (source and destination IP addresses) received from the tester. Sends the connection specification to the remote host's ctcpWad's FE.
FE forks BE FE creates three sockets
A Unix socket to listen for a CTCP connection configuration request from an external application like tester. A Unix socket to communicate with the BE. An Inet socket to communicate with the ctcpWad on the remote end of the CTCP connection.
Then it starts listening on these sockets
July 06, 2006 9 July 06, 2006 10
ctcpWad
The remote FE then tells its BE to look for the same tcp connection. Both the BEs check periodically in /proc/web100 directory for the appearance of any TCP connection that matches the specification. If a matching TCP connection exists:
BE does the following
Disables the congestion control algorithm for that socket by setting a new web100 parameter called WAD_Cheetah Extracts the rtt (round trip time) measurement (in milliseconds) from web100 for the specified source-destination pair Calculates required bdp (bandwidth-delay product) in packets to maintain the sending rate (in Mbps).
bdp = (cktRate * rtt * 1000)/(8 * PKT_SIZE) + 1) Set computed bdp in a second new web100 parameter, WAD_Bdp.
Tester
This is a simple application that tells the ctcpWad which TCP socket to configure for operation as a CTCP socket. Passes source IP address and destination IP address and sending rate (in Mbps) to ctcpWad.
All TCP connections between this source and destination addresses will be configured as CTCP sockets.
July 06, 2006 12
Congestion window (cwnd) variable is ignored and WAD_Bdp is used to maintain the sending rate. On the receiver side, advertised window (rwnd) is clamped to the WAD_Bdp value.
July 06, 2006 11
3
Outline
Purpose of CTCP Requirements to run CTCP code CTCP Components CTCP Operation CTCP Patch details CTCP Usage Usage of CTCP across an Internet path
July 06, 2006 13
CTCP Operation
Sender Receiver
web100 lib /proc
patch Socket API patch Socket API patch
web100 lib /proc
patch
Linux v2.6.11
TCP web100 Instr. IP DLL
NIC II NIC I
TCP Internet Internet IP DLL
NIC I NIC II
web100 Instr.
Linux v2.6.11
July 06, 2006
cheetah cheetah
14
Sender
Setup a circuit
Receiver
Sender
ctcpWad FE
Initiate ctcpWad
Receiver
ctcpWad FE
BE
BE
web100 lib /proc
patch Socket API patch Socket API patch
web100 lib /proc
patch
web100 lib /proc
patch Socket API patch Socket API patch
web100 lib /proc
patch
Linux v2.6.11
TCP web100 Instr. IP DLL
NIC II NIC I
TCP Internet Internet IP DLL
NIC I NIC II
web100 Instr.
Linux v2.6.11
Linux v2.6.11
TCP web100 Instr. IP DLL
NIC II NIC I
TCP Internet Internet IP DLL
NIC I NIC II
web100 Instr.
Linux v2.6.11
July 06, 2006
cheetah cheetah
15
July 06, 2006
cheetah cheetah
16
4
Sender
{Source & Destination IP addresses, Send rate}
ctcpWad FE
Run tester with connection specification
Receiver
ctcpWad FE
Sender
ctcpWad FE
Sender FE sends connection specification to remote host
Receiver
ctcpWad FE
tester
BE
BE
BE
BE
web100 lib /proc
patch Socket API patch Socket API patch
web100 lib /proc
patch
web100 lib /proc
patch Socket API patch Socket API patch
web100 lib /proc
patch
Linux v2.6.11
TCP web100 Instr. IP DLL
NIC II NIC I
TCP Internet Internet IP DLL
NIC I NIC II
web100 Instr.
Linux v2.6.11
Linux web100 v2.6.11
TCP Instr. IP DLL
NIC II NIC I
TCP Internet Internet IP DLL
NIC I NIC II
web100 Instr.
Linux v2.6.11
July 06, 2006
cheetah cheetah
17
July 06, 2006
cheetah cheetah
18
Sender
ctcpWad FE
BE
FE provides BE connection specification for which to monitor TCP socket creations
Receiver
ctcpWad FE
Sender
ctcpWad FE
User App
Run user application
User App
Receiver
ctcpWad FE
BE
BE
BE
web100 lib /proc
patch Socket API patch Socket API patch
web100 lib /proc
patch
web100 lib /proc
patch Socket API patch Socket API patch
web100 lib /proc
patch
Linux v2.6.11
TCP web100 Instr. IP DLL
NIC II NIC I
TCP Internet Internet IP DLL
NIC I NIC II
web100 Instr.
Linux v2.6.11
Linux v2.6.11
TCP web100 Instr. IP DLL
NIC II NIC I
TCP Internet Internet IP DLL
NIC I NIC II
web100 Instr.
Linux v2.6.11
July 06, 2006
cheetah cheetah
19
July 06, 2006
cheetah cheetah
20
5
Sender
ctcpWad FE
User App
User application creates a TCP socket
User App
Receiver
ctcpWad FE
Sender
ctcpWad FE
User App
BE finds the TCP socket and configures it into a CTCP connection
Receiver
ctcpWad FE
User App
BE
BE
BE
BE
web100 lib /proc
patch Socket API patch Socket API patch
web100 lib /proc
patch
web100 lib /proc
patch Socket API patch Socket API patch
web100 lib /proc
patch
Linux v2.6.11
TCP web100 Instr. IP DLL
NIC II NIC I
TCP Internet Internet IP DLL
NIC I NIC II
web100 Instr.
Linux v2.6.11
Linux v2.6.11
TCP web100 Instr. IP DLL
NIC II NIC I
TCP Internet Internet IP DLL
NIC I NIC II
web100 Instr.
Linux v2.6.11
July 06, 2006
cheetah cheetah
21
July 06, 2006
cheetah cheetah
22
Outline
Purpose of CTCP Requirements to run CTCP code CTCP Components CTCP Operation CTCP Patch details CTCP Usage Usage of CTCP across an Internet path
July 06, 2006 23
CTCP Patch for web100
ctcpWad uses web100 to disable the congestion control algorithm from being executed on a particular TCP connection. Two new writable variables are added to the web100 to modify a TCP connection's behavior
WAD_Bdp Denotes the sending rate. WAD_Cheetah A boolean variable to identify whether this a TCP socket or a CTCP 24 July 06, 2006 socket.
6
CTCP Patch for Kernel
Modifications are made to Linux 2.6.11 TCP to disable the congestion control algorithm.
tcp.h, tcp_input.c and tcp_output.c files are modified for this purpose.
Outline
Purpose of CTCP Requirements to run CTCP code CTCP Components CTCP Operation CTCP Patch details CTCP Usage Usage of CTCP across an Internet path
July 06, 2006 26
If WAD_Cheetah == 1
Before sending each packet, sender bypasses the comparison of the current in_flight_packets with cwnd value Instead it compares the WAD_Bdp value with current in_flight_packets New packets are allowed to be sent if
in_flight_packets <= WAD_Bdp
Hence slow start and other congestion avoidance algorithms are disabled and a constant sending rate is maintained.
July 06, 2006
25
CTCP Usage: Overview
Ensure that Linux kernel v-2.6.11 modified by web100 and CTCP patch is running. Run ctcpWad on both end hosts as a daemon. Start tester on either host Start application that creates a TCP connection
July 06, 2006 27
CTCP Usage: ctcpWad
Run ctcpWad on both end hosts as a daemon.
ctcpWad <Control_Plane_IP> & Control_Plane_IP: IP address of the primary NIC - see Slide 4
For example, if control-plane IP addresses of the sender and receiver are 10.0.0.1 and 10.0.0.2, respectively, run ctcpWad as follows
Sending Host: ctcpWad 10.0.0.1 & Receiving Host: ctcpWad 10.0.0.2 &
July 06, 2006
28
7
CTCP Usage: tester
Tester is invoked as
tester <Data_Plane_IP> <Remote_Data_Plane_IP> <sending_rate(in Mbps)> <Remote_Control_Plane_IP> Data_Plane_IP: IP address of the seco...