Scalable L14-15 Interconnection Networks
1
Scalable, High Performance Network
At Core of Parallel Computer Architecture Requirements and trade-offs at many levels
Elegant mathematical structure Deep relationships to algorithm structure
Managing many traffic flows Electrical / Optical link properties
Scalable Interconnection Network
Little consensus
interactions across levels Performance metrics?
Cost metrics? Workload?
M CA P M CA
network interface
P
=> need holistic understanding
2
1
Requirements from Above
Communication-to-computation ratio
=> bandwidth that must be sustained for given computational rate traffic localized or dispersed? bursty or uniform?
Programming Model
protocol granularity of transfer
degree of overlap (slackness)
=> job of a parallel machine network is to transfer information from source node to dest. node in support of network transactions that realize the programming model
3
Goals
Latency as small as possible As many concurrent transfers as possible
operation bandwidth data bandwidth
Cost as low as possible
4
2
Outline
Introduction Basic concepts, definitions, performance perspective Organizational structure Topologies Routing and switch design
5
Basic Definitions
Network interface Links
bundle of wires or fibers that carries a signal connects fixed number of input channels to fixed number of output channels
Switches
6
3
Links and Channels
...ABC123 => Receiver
...QR67 =>
Transmitter
transmitter converts stream of digital symbols into signal that is driven down the link receiver converts it back
tran/rcv share physical protocol
trans + link + rcv form Channel for digital info flow between switches link-level protocol segments stream of symbols into larger units: packets or messages (framing) node-level protocol embeds commands for dest communication assist within packet
7
Formalism
network is a graph V = {switches and nodes} connected by communication channels C V V Channel has width w and signaling rate f = 1/
channel bandwidth b = wf phit (physical unit) data transferred per cycle flit - basic unit of flow-control
Number of input (output) channels is switch degree Sequence of switches and links followed by a message is a route Think streets and intersections
8
4
What characterizes a network?
Topology
(what)
physical interconnection structure of the network graph direct: node connected to every switch indirect: nodes connected to specific subset of switches
Routing Algorithm
(which)
restricts the set of paths that msgs may follow many algorithms with different properties
gridlock avoidance?
Switching Strategy
(how)
how data in a msg traverses a route circuit switching vs. packet switching
Flow Control Mechanism
(when)
when a msg or portions of it traverse a route what happens when traffic is encountered?
9
What determines performance
Interplay of all of these aspects of the design
10
5
Topological Properties
Routing Distance - number of links on route Diameter - maximum routing distance Average Distance A network is partitioned by a set of links if their removal disconnects the graph
11
Typical Packet Format
Routing and Control Header Error Code Trailer Data Payload
digital symbol
Sequence of symbols transmitted over a channel
Two basic mechanisms for abstraction
encapsulation fragmentation
12
6
Communication Perf: Latency
Time(n)s-d = overhead + routing delay + channel occupancy + contention delay occupancy = (n + ne) / b Routing delay? Contention?
13
Store&Forward vs Cut-Through Routing
Store & For ward R outing S o urc e 3 2 1 0 3 2 1 3 2 3 0 1 0 2 1 0 3 2 1 0 3 2 1 3 2 3 0 1 0 2 1 0 3 2 1 0 Tim e 3 2 1 0 De s t 3 2 1 0 3 2 1 3 2 3 0 1 2 3 0 1 2 3 0 1 0 2 1 0 3 2 1 0 C ut-T hrough R outing Dest
h(n/b + ) vs what if message is fragmented? wormhole vs virtual cut-through
n/b + h
14
7
Contention
Two packets trying to use the same link at same time
limited buffering drop?
Most parallel mach. networks block in place
link-level flow control tree saturation
Closed system - offered load depends on delivered
15
Bandwidth
What affects local bandwidth?
packet density routing delay contention
b x n/(n + ne) b x n / (n + ne + w)
endpoints within the network
Aggregate bandwidth
bisection bandwidth
sum of bandwidth of smallest set of links that partition the network
total bandwidth of all the channels: Cb suppose N hosts issue packet every M cycles with ave dist
C/N channels available per node link utilization = MC/Nh < 1
each msg occupies h channels for = n/w cycles each
16
8
Saturation
80
0.8 0.7 0.6 0.5 0.4
60
La te n c y
50 40
Saturation
30 20 10 0 0 0.2 0.4 0.6 0.8 1
D e liv e re d B a n d w id th
70
Saturation
0.3 0.2 0.1 0 0 0.2 0.4 0.6 0.8 1 1.2
D e liv e re d B a n d w id th
O ffe re d B a n d w id th
17
Outline
Introduction Basic concepts, definitions, performance perspective Organizational structure Topologies Routing and switch design
18
9
Organizational Structure
Processors
datapath + control logic control logic determined by examining register transfers in the datapath links
Networks
switches network interfaces
19
Link Design/Engineering Space
Cable of one or more wires/fibers with connectors at the ends attached to switches or interfaces
Narrow: - control, data and timing multiplexed on wire Synchronous: - source & dest on same clock
Short: - single logical value at a time
Long: - stream of logical values at a time
Asynchronous: - source encodes clock in signal
Wide: - control, data and timing on separate wires
20
10
Example: Cray MPPs
T3D: Short, Wide, Synchronous
(300 MB/s)
24 bits: 16 data, 4 control, 4 reverse direction flow control
single 150 MHz clock (including processor) flit = phit = 16 bits
two control bits identify flit type (idle and framing)
no-info, routing tag, packet, end-of-packet
T3E: long, wide, asynchronous
(500 MB/s)
14 bits, 375 MHz, LVDS flit = 5 phits = 70 bits
64 bits data + 6 control
switches operate at 75 MHz framed into 1-word and 8-word read/write request packets
Cost = f(length, width) ?
21
Switches
Receiver Input Ports Input Buffer Output Buffer Transmiter Output Ports
Cross-bar
Control Routing, Scheduling
22
11
Switch Components
Output ports
transmitter (typically drives clock and data)
Input ports
synchronizer aligns data signal with local clock domain essentially FIFO buffer
Crossbar
connects each input to any output degree limited by area or pinout
Buffering Control logic
complexity depends on routing logic and scheduling algorithm determine output port for each incoming packet arbitrate among inputs directed at same output
23
Outline
Introduction Basic concepts, definitions, performance perspective Organizational structure Topologies Routing and switch design
24
12
Interconnection Topologies
Class networks scaling with N Logical Properties:
distance, degree length, width
Physcial properties
Fully connected network
diameter = 1
degree = N cost?
bus => O(N), but BW is O(1) crossbar => O(N2) for BW O(N)
- actually worse
VLSI technology determines switch degree
25
Linear Arrays and Rings
Linear Array Torus
Torus arranged to use short wires
Linear Array
Diameter? Average Distance?
Bisection bandwidth? Route A -> B given by relative address R = B-A
Torus? Examples: FDDI, SCI, FiberChannel Arbitrated Loop, KSR1
26
13
Multidimensional Meshes and Tori
2D Grid
3D Cube
d-dimensional array
n = kd-1 X ...X kO nodes described by d-vector of coordinates (id-1, ..., iO)
d-dimensional k-ary mesh: N = kd
k = dN described by d-vector of radix k coordinate
d-dimensional k-ary torus (or k-ary d-cube)?
27
Properties
Routing
relative distance: R = (b d-1 - a d-1, ... , b0 - a0 ) traverse ri = b i - a i hops in each dimension dimension-order routing
Average Distance
Wire Length?
d x 2k/3 for mesh dk/2 for cube
Degree? Bisection bandwidth?
Partitioning?
k
d-1
bidirectional links Short wires
28
Physical layout?
2D in O(N) space higher dimension?
14
Real World 2D mesh
1824 node Paragon: 16 x 114 array
29
Embeddings in two dimensions
6x3x2
Embed multiple logical dimension in one physical dimension using long wires
30
15
Trees
Diameter and avg. distance are logarithmic
k-ary tree, height d = logk N address specified d-vector of radix k coordinates describing path down from root
Fixed degree Route up to common ancestor and down
R = B xor A let i be position of most significant 1 in R, route up i+1 levels down in direction given by low i+1 bits of B
H-tree space is O(N) with O(N) long wires Bisection BW?
31
Fat-Trees
Fat Tree
Fatter links (really more of them) as you go up, so bisection BW scales with N
32
16
Butterflies
4 3 2 1 0
16 node butterfly
0 1 0 1 0 1 0 1
0
1
building block
Tree with lots of roots! N log N (actually N/2 x logN) Exactly one route from any source to any dest R = A xor B, at level i use straight edge if ri=0, otherwise cross edge Bisection N/2 vs N (d-1)/d
33
k-ary d-cubes vs d-ary k-flies
Degree d N switches Diminishing BW per node Requires locality vs vs vs N log N switches constant little benefit to locality
Can you route all permutations?
34
17
Benes network and Fat Tree
16-node Benes Network (Unidirectional)
16-node 2-ary Fat-Tree (Bidirectional)
Back-to-back butterfly can route all permutations
off line
35
What if you just pick a random mid point?
Hypercubes
Also called binary n-cubes. # of nodes = N = 2n O(logN) hops Good bisection BW Complexity
out degree is n = logN correct dimensions in order
with random comm. 2 ports per processor
0-D
1-D
2-D
3-D
4-D
5-D !
36
18
Relationship of Butterflies to Hypercubes
Wiring is isomorphic Except that Butterfly always takes log n steps
37
Properties of Some Topologies
Topology 1D Array 1D Ring 2D Mesh 2D Torus Degree Diameter 2 2 4 4 N-1 N/2 2 (N1/2 - 1) N1/2 nk/2 Ave Dist N/3 N/4 2/3 N1/2 1/2 N1/2 Bisection 1 2 N1/2 2N1/2 nk/4 n/2 63 (21) 32 (16) 15 (7.5) @n=3 N/2 10 (5) D (D ave) @ P=1024 huge
k-ary n-cube 2n Hypercube
nk/4 n
n =log N
All have some bad permutations
many popular permutations are very bad for meshes (transpose) ramdomness in wiring or routing makes it hard to find a bad one!
38
19
Real Machines
Wide links, smaller routing delay Tremendous variation
39
How Many Dimensions in Network?
n = 2 or n = 3
Short wires, easy to build Many hops, low bisection bandwidth Requires traffic locality
n >= 4
Harder to build, more wires, longer average length Fewer hops, better bisection bandwidth Can handle non-local traffic
k-ary d-cubes provide a consistent framework for comparison
N = kd scale dimension (d) or nodes per dimension (k)
assume cut-through
40
20
Traditional Scaling: Latency(P)
250 140 120 100 80 60 40 20 0 0 5000 10000 0 0 2000 4000 6000 8000 10000 Machine Size (N) d=2 d=3 d=4 k=2 n/w Ave Late nc y T(n=140) Ave Late nc y T(n=40) 200
150
100
50
Ma c hine S iz e (N)
Assumes equal channel width
independent of node count or dimension dominated by average distance
41
Average Distance
100 90 80 70 Ave Dis tanc e 60 50 40 30 20 10 0 0 5 10 15 20 25 256 1024 16384 1048576
Avg. distance = d (k-1)/2
Dim e nsion
but, equal channel width is not equal cost! Higher dimension => more channels
42
21
In the 3-D world
For n nodes, bisection area is O(n2/3 )
For large n, bisection bandwidth is limited to O(n2/3 )
Dally, IEEE TPDS, [Dal90a] For fixed bisection bandwidth, low-dimensional k-ary n-cubes are better (otherwise is higher better) i.e., a few short fat wires are better than many long thin wires What about many long fat wires?
43
Equal cost in k-ary n-cubes
Equal number of nodes? Equal number of pins/wires? Equal bisection bandwidth? Equal area? Equal wire length? What do we know? switch degree: d diameter = d(k-1) total links = Nd pins per node = 2wd bisection = kd-1 = N/k links in each directions 2Nw/k wires cross the middle
44
22
Latency(d) for P with Equal Width
250 A v e ra g e La te n c y = = ) 0 , (n 2 4 256 1024 200 16384 1048576 150
total links(N) = Nd
45
Latency with Equal Pin Count
300 256 node s 250 A v e La te n c y T(n = 4 0 B ) 1024 node s A v e La te n c y T(n = 1 4 0 B ) 16 k node s 200 1M nodes 250 300
150
100
50
0 0 5 10 15 20 25
Baseline d=2, has w = 32 (128 wires per node) fix 2dw pins => w(d) = 64/d distance up with d, but channel time down
46
100
50
0 0 5 10 15 20 25
Dim e n s io n
200
150
100
256 node s 1024 node s 16 k node s 1M nodes
50
0 0 5 10 15 20 25
D im e n s io n ( d )
D im e n s io n ( d )
23
Latency with Equal Bisection Width
1000 900 800 Ave Late nc y T(n=40) 700 600 500 400 300 200 100 0 0 5 10 15 20 25 256 no de s 1024 n ode s 16 k node s 1M n ode s
N-node hypercube has N bisection links 2d torus has 2N 1/2 Fixed bisection => w(d) = N 1/d / 2 = k/2 1 M nodes, d=2 has w=512!
Dime nsion (d)
47
Larger Routing Delay (w/ equal pin)
1000 900 800 Ave Late ncy T(n= 140 B) 700 600 500 400 300 200 100 0 0 5 10 15 20 25 256 node s 1024 node s 16 k node s 1M node s
Dim e nsion (d)
Dallys conclusions strongly influenced by assumption of small routing delay
48
24
Latency under Contention
300
250
n40,d2,k32 200 Late nc y n40,d3,k10 n16,d2,k32 150 n16,d3,k10 n8,d2,k32 n8,d3,k10 100 n4,d2,k32 n4,d3,k10
50
0 0 0.2 0.4 0.6 0.8 1
Channe l Utiliz a tio n
Optimal packet size? Channel utilization?
49
Saturation
250
200
150 Late nc y
n/w=40 n/w=16 n/w=8 n/w = 4
100
50
0 0 0.2 0.4 0.6 0.8 1 Ave Channe l Utilization
Fatter links shorten queuing delays
50
25
Phits per cycle
350
300
250
Late nc y
200
150
100
50
n8, d3, k10 n8, d2, k32
0 0 0.05 0.1 0.15 0.2 0.25 Flits pe r c yc le per proc e s s or
Higher degree network has larger available bandwidth
cost?
51
Topology Summary
Rich set of topological alternatives with deep relationships Design point depends heavily on cost model
nodes, pins, area, ... Wire length or wire delay metrics favor small dimension
Long (pipelined) links increase optimal dimension
Need a consistent framework and analysis to separate opinion from design Optimal point changes with technology
52
26
Outline
Introduction Basic concepts, definitions, performance perspective Organizational structure Topologies Routing and switch design
53
Routing and Switch Design
Routing Switch Design Flow Control Case Studies
54
27
Routing
Recall: routing algorithm determines
which of the possible paths are used as routes
how the route is determined R: N x N -> C, which at each switch maps the destination node nd to the next channel on the route
Issues:
Routing mechanism
arithmetic source-based port select table driven general computation
Properties of the routes Deadlock feee
55
Routing Mechanism
need to select output port for each input packet
in a few cycles ex: x, y routing in a grid
Simple arithmetic in regular topologies
west (-x) x < 0 east (+x) x > 0 x = 0, y < 0 south (-y) x = 0, y > 0 north (+y) x = 0, y = 0 processor
Reduce relative address of each dimension in order
Dimension-order routing in k-ary d-cubes e-cube routing in n-cube
56
28
Routing Mechanism (cont)
P3 P2 P1 P0
Source-based
message header carries series of port selects used and stripped en route CRC? Packet Format?
CS-2, Myrinet, MIT Artic message header carried index for next port at next switch
Table-driven
o = R[i] o, I = R[i ]
57
table also gives index for following hop
ATM, HPPI
Properties of Routing Algorithms
Deterministic
route determined by (source, dest), not intermediate state (i.e. traffic) route influenced by traffic along the way only selects shortest paths no traffic pattern can lead to a situation where no packets mover forward
Adaptive
Minimal
Deadlock free
58
29
Deadlock Freedom
How can it arise?
necessary conditions:
shared resource incrementally allocated non-preemptible
think of a channel as a shared that is acquired incrementally
resource
source buffer then dest. buffer channels along a route
How do you avoid it?
constrain how channel resources are allocated ex: dimension order
59
How do you prove that a routing algorithm is deadlock free
Proof Technique
Resources are logically associated with channels Messages introduce dependences between resources as they move forward Need to articulate possible dependences between channels Show that there are no cycles in Channel Dependence Graph
find a numbering of channel resources such that every legal route follows a monotonic sequence
=> no traffic pattern can lead to deadlock Network need not be acyclic, on channel dependence graph
60
30
Example: k-ary 2D array
Theorem: x,y routing is deadlock free Numbering
+x channel (i,y) -> (i+1,y) gets i
similarly for -x with 0 as most positive edge +y channel (x,j) -> (x,j+1) gets N+j
similary for -y channels
1 00 18 10 17 20 16 30 19 31 32 33
61
Any routing sequence: x direction, turn, y direction is increasing
2 01 2 17 11 18 21 22 23 1 12 02 0 13 3 03
Channel Dependence Graph
1 2 2 1 18 17 1 2 17 18 1 2 16 19 1 2 16 19 2 1 17 18 2 1 16 19 3 0 2 1 17 18 3 0 16 19 18 17 3 0 17 18 3 0 18 17
1 00 18 10 17 20 16 30 19 31 18 21 2 17 11 01
2 02 1 12
3 03 0 13
18 17
22
23
32
33
62
31
More examples
Why is the obvious routing on X deadlock free?
butterfly?
tree? fat tree?
Any assumptions about routing mechanism? amount of buffering? What about wormhole routing on a ring?
2 3 4 5 6 1 0 7
63
Deadlock free wormhole networks?
Basic dimension-order routing doesnt work for k-ary d-cubes
only for k-ary d-arrays (bi-directional) provide multiple virtual channels to break dependence cycle good for BW too!
Input Ports Output Ports
Idea: add channels!
Cross-Bar
Dont need to add links, or xbar, only buffer resources
64
This adds nodes the the CDG, remove edges?
32
Breaking deadlock with virtual channels
Packet switches from lo to hi channel
65
Up*-Down* routing
Given any bidirectional network Construct a spanning tree Number of the nodes increasing from leaves to roots UP increase node numbers Any Source -> Dest by UP*-DOWN* route
up edges, single turn, down edges
Performance?
Some numberings and routes much better than others interacts with topology in strange ways
66
33
Turn Restrictions in X,Y
+Y
-X
+X
-Y
XY routing forbids 4 of 8 turns and leaves no room for adaptive routing Can you allow more turns and still be deadlock free
67
...