Memory Distributed and Cluster Computers
Alistair Rendell
(Adapted From Parallel Programming, Techniques and Applications using Networked Workstations and Parallel Computers, B. Wilkinson and M. Allen, Prentice Hall)
1.1
Message-Passing Multicomputer
Interconnection network Messages Processor
Local memory
Computers
All communication by messages sent over the interconnection network More scalable than shared memory parallel computers (The boundary between distributed and shared memory systems is not clear cut)
1.2
Interconnection Networks
Key characteristics
latency, bandwidth, bisection bandwidth, diameter, cost
Static
exhaustive interconnections 2- and 3-dimensional meshes hypercubes
Dynamic
Crossbar Trees Multistage interconnection networks
1.3
Two-dimensional array (mesh)
Links Computer/ processor
Also torus and 3-dimensional - used in some large high performance systems.
1.4
Three-dimensional hypercube
110 100 111
101
010 000
011 001
1.5
Four-dimensional hypercube
0110 0100 0101 0111 1100 1110 1101 1111
0010 0000 0001
0011 1000
1010 1001
1011
Hypercubes popular in 1980s - not now
1.6
Crossbar switch
Memories
Processors
Switches
1.7
Tree
Root Links Switch element
Processors
Whats a FAT tree?
1.8
Multistage Interconnection Network
Example: Omega network
2 2 switch elements (straight-through or crossover connections) 000 001 010 011 Inputs 100 101 110 111 000 001 010 011 Outputs 100 101 110 111
1.9
Communication Methods
Switching options (no direct link)
Circuit switching (Intel IPSC-2) Packet switching (everything else)
Store-and-forward
message entirely received by intervening nodes
Wormhole routing
Message divided into flits that are pipelined through the communication network
Care to avoid possible deadlock and/or livelock scenarios
1.10
Networked Computers as a Computing Platform
A network of computers became a very attractive alternative to expensive supercomputers parallel and computer systems for high-performance computing in early 1990s. Several early projects. Notable: Berkeley NOW (network of workstations) project. NASA Beowulf project. (Will look at this one later)
1.11
Key Drivers
Amazing increase in performance of commercial off the shelf PCs at very low cost Availability of commodity high performance networking The GPL software revolution Advantages
The latest processors can easily be incorporated into the system as they become available. Existing software can be used or modified.
1.12
Software Tools for Clusters
Based upon Message Passing Parallel Programming: Parallel Virtual Machine (PVM) - developed in late 1980s. Became very popular. Message-Passing Interface (MPI) - standard defined in 1990s. Both provide a set of user-level libraries for message passing. Use with regular programming languages (C, C++, ...).
1.13
Beowulf Clusters*
A group of interconnected commodity computers achieving high performance with low cost. Typically using commodity interconnects high speed Ethernet, and Linux OS. * Beowulf comes from name given by NASA Goddard Space Flight Center cluster p...