AND DISTRIBUTED HIGH-PERFORMANCE COMPUTING
HPC ARCHITECTURES
Paul Coddington Department of Computer Science University of Adelaide Room 1052 http://www.dhpc.adelaide.edu.au/ [e-mail] July - October 2000
HPC ARCHITECTURES
DHPC
HPC Architectures
Aims for High-Performance Computing
Users of high-performance computing generally have two dierent, but related, goals: Maximize performance measured by MIPS, MFLOPS, SPECmarks, LINPACK and other benchmarks. Minimize turnaround time to complete specic application problem, or Maximize the problem size that can be solved in a given amount of time. Necessary for large-scale problems that could not be done otherwise. These are true supercomputers, cost tens of $M, O(1000) times faster than desktop systems. Maximize price-performance Best Flops per dollar. Ideal for medium-sized problems (and budgets). Avoid spending big bucks on machines that are virtually obsolete in 5 years. These high-performance computers can be any size desktop, deskside, room-size.
[KAH/PDC]
DHPC, July-October 2000, 2
DHPC
HPC Architectures
Architectures for High-Performance Computing
Two ways of maximizing performance: 1. Reduce the time per instruction (clock cycle) 2. Increase the number of useful instructions executed per clock cycle Moores Law performance of processors doubles roughly every 18 months, from a combination of both these methods. Performance gains in early computers came almost entirely from method 1, but this becomes increasingly dicult, and will eventually reach physical constraints (speed of light, power/heat contraints, etc). More recently, much of the performance increase in processors has come from method 2, by using pipelining and parallel execution of instructions. Performance gains in supercomputers have also improved due to an additional important factor: 1. Reduce the time per instruction 2. Pipelining and parallel execution of instructions 3. Multiple processors tackling the same program concurrently
[KAH/PDC]
DHPC, July-October 2000, 3
DHPC
HPC Architectures
Single Processor Parallelism
Arithmetic operations +, , , / require many low-level instructions to be executed. Vector or pipelined architectures create pipelines that can execute multiple instructions concurrently, producing one or more arithmetic operations per clock cycle. Some processors have multiple pipelines, e.g. one add and one multiply. Vector processors may have even more, e.g. two adds and two multiplies. Typical commodity processors have clock rates of hundreds of MHz and can execute 1 or 2 Flops each cycle, giving peak performance of order 1 GFlops. Loading a data element from memory can take several clock cycles. Not such a problem for early computers (when Flops also took many cycles), but on modern processors this can be a major bottleneck. To achieve peak performance, need memory heirarchy lots of registers, large rapid-access cache memory, low latency high bandwidth access to main memory. Can be big dierence between peak performance and actual performance hard to keep pipelines full and minimize memory access overhead, especially for apps with irregular data structures and algorithms. Need good compilers to optimise pipelining and memory access.
[KAH/PDC] DHPC, July-October 2000, 4
DHPC
HPC Architectures
Multi-Processor Parallelism
Want to utilize multiple processors to speed up program run-time, by dividing up the entire computation among the processors. Usually achieved by splitting up a large data set among processors. Each processor does computation on its section of the data, concurrently (in parallel) with all the other processors. Introduces an extra level in the memory hierarchy processors may need to access data stored in the memory of another processor, or in large banks of memory shared between processors. Key performance issues that need to be addressed by software, hardware and compilers are: single processor performance memory access times (particularly to non-local or shared memory) I/O and disk access Also need to balance computational load over processors, which is addressed by parallel languages, compilers, algorithms.
[KAH/PDC]
DHPC, July-October 2000, 5
DHPC
HPC Architectures
Taxonomy of Architectures
Flynns Taxonomy provides a simple, but very broad, classication of architectures for high-performance computers: Single Instruction, Single Data (SISD) A single processor with a single instruction stream, operating sequentially on a single data stream. Single Instruction, Multiple Data (SIMD) A single instruction stream is broadcast to every processor, all processors execute the same instructions in lock-step on their own local data stream. Multiple Instruction, Multiple Data (MIMD) Each processor can independently execute its own instruction stream on its own local data stream. SISD machines are the traditional single-processor, sequential computers also known as Von Neumann architecture, as opposed to non-Von parallel computers. SIMD machines are synchronous, with more ne-grained parallelism they run a large number parallel processes, one for each data element in a parallel vector or array. MIMD machines are asynchronous, with more coarse-grained parallelism they run a smaller number of parallel processes, one for each processor, operating on the large chunks of data local to each processor.
[KAH/PDC] DHPC, July-October 2000, 6
DHPC
HPC Architectures
Distributed Memory
Data set is distributed among processors, each processor accesses only its own data from local memory, if data from another section of memory (i.e. another processor) is required, it is obtained by passing a message containing the data between the processors. One or more host processors provide links to the outside world (disk and le access, compilers, Ethernet, Unix, etc). For SIMD machines, a host (or control) processor may broadcast the instructions to all processors. Much larger overhead (latency) for accessing non-local data, but can scale to large numbers (thousands) of processors for many applications.
Communication Network
Host Processor
P1
P2
. . .
Pn
M1
M2
Mn
P = processors, M = local memory
[KAH/PDC]
DHPC, July-October 2000, 7
DHPC
HPC Architectures
Shared Memory
Each processor has access to all the memory, through a shared memory bus and/or communication network. Requires locks and semaphores to avoid more than one processor accessing or updating the same section of memory at the same time. Lower overhead for accessing non-local data, but dicult to scale to large numbers of processors, usually used for small numbers (order 100 or less) of processors. Heirarchical shared memory attempts to provide scalability by using an additional message-passing approach between SMP clusters that emulates true shared memory, but with non-uniform access times.
Shared Global Memory
Communication Network
P1
P2
. . .
Pn
[KAH/PDC]
DHPC, July-October 2000, 8
DHPC
HPC Architectures
Main HPC Architectures
SISD mainframes, workstations, PCs SIMD Shared Memory Vector machines, Cray and imitators (NEC, Hitachi, Fujitsu, etc) MIMD Shared Memory Encore, Alliant, Sequent, KSR, Tera, Silicon Graphics, Sun, DEC/Compaq, HP SIMD Distributed Memory ICL/AMT/CPP DAP, TMC CM-2, Maspar MIMD Distributed Memory nCUBE, Intel, Transputers, TMC CM-5, plus more recent PC and workstation clusters (IBM SP2, DEC Alpha, Sun) connected with various networking/switching technologies Note that modern sequential machines (workstations and PCs) are not purely SISD modern processors use many concepts from vector and parallel architectures (pipelining, parallel execution of instructions, prefetching of data, etc) in order to achieve one or more arithmetic operations per clock cycle. Many concepts from yesterdays supercomputers are used in todays PCs.
[KAH/PDC]
DHPC, July-October 2000, 9
DHPC
HPC Architectures
Issues for Distributed Memory Architectures
Latency and Bandwidth for accessing distributed memory is the main performance issue. Eciency in parallel processing is usually related to ratio of time for calculation vs time for communication the higher the ratio, the better the performance. Processors are rapidly increasing in speed (Moores Law), but it is hard to obtain similar increases in speed for accessing memory, so memory access has become a bottleneck. Caching and memory heirarchies are used, but it is dicult to develop compilers to utilize these eectively. Problem is even more severe when access to distributed memory is needed, since there is an extra level in the memory heirarchy, with latency and bandwidth that can be orders of magnitude slower than local memory access. Scalability to more processors is a key issue. Access times to distant processors should not be very much slower than access to nearby processors, since non-local and collective (all-to-all) communication is important for many programs. This can be a problem for large parallel computers (hundreds or thousands of processors). Many dierent approaches to network topology and switching have been tried in attempting to alleviate this problem.
[KAH/PDC] DHPC, July-October 2000, 10
DHPC
HPC Architectures
Distributed Memory Access
Latency is the overhead in setting up a connection between processors for passing data. This is the most crucial problem for all parallel computers obtaining good performance over a range of applications depends critically on low latency for accessing remote data. Current processors can perform hundreds of Flops per microsecond, whereas typical latencies can be 1 - 100 microseconds. Bandwidth is the amount of data per unit time that can be passed between processors. This needs to be large enough to support ecient passing of large amounts of data between processors, as well as collective communications, and I/O for large data sets. Scalability is how well latency and bandwidth scale with the addition of more processors. This is usually only a problem for supercomputers with hundreds or thousands of processors. Smaller congurations of tens of processors can usually be eciently connected. Many dierent kinds of network topologies have been used.
[KAH/PDC]
DHPC, July-October 2000, 11
DHPC
HPC Architectures
Network Architectures
Network Topology is how the processors are connected. 2D or 3D mesh is simple and ideal for programs with mostly nearest-neighbour communication. General communications can be good if fast routers are used. Hypercube is an attempt to minimize number of hops between any two processors. Doesnt scale well too many wires for large dimensions required for large numbers of processors. Switched connections have all processors directly connected to one or more high-speed switches, which introduce an overhead but can be quite fast. Many other more complex or hierarchical topologies are also used. Network topology should be transparent to the user. Portable languages such as MPI and HPF interface to point-to-point and collective communications at a high level; details of the implementation for a specic network topology are left to the compiler.
[KAH/PDC]
DHPC, July-October 2000, 12
DHPC
HPC Architectures
Vector Architectures
Key ideas: vector registers; pipelines; fast I/O channels; fast Physical memory - no cache; banks memory of and disk systems; multiple instructions possible at once; multi head (processor) systems use shared memory to communicate; can fake other communications paradigms using SHMem Put and Get; lots of custom compiler technology; lots of automatic and user optimisations (compiler directives); physical engineering - liquid cooling, minimal wire lengths; custom silicon of GaAs... The vector ideas have been incorporated into many other machines and processors.
[KAH/PDC]
DHPC, July-October 2000, 13
DHPC
HPC Architectures
Vector Machines
Historically some of the major vector supercomputers include: Cray 1, Cray XMP, Cray YMP, Cray 2, C90, J90, T90 (Cray Research Inc.) Cray 3, Cray 4. (Cray Computer Corp.) CDC Cyber 205, ETA. Fujitsu VP Hitachi S810 NEC SX2, SX3, SX5 Early vector machines were very successful due to: large amounts of fast memory to allow large problem sizes very fast processors (relative to scalar processors of the time) straightforward porting of code using compiler directives good compilers Commodity processors are now providing better price/performance, so future of vector systems is unclear. May see vector processors on commodity chips for image processing etc.
[KAH/PDC] DHPC, July-October 2000, 14
DHPC
HPC Architectures
Processors for SIMD Architectures
Thousands of cheap, simple processors, or hundreds of more expensive, high-end processors? Lots of puny processors Early SIMD machines (DAP, CM-1) used thousands of feeble, single-bit processors. This approach is good for applications with mostly logic operations, bit manipulation, integer arithmetic, e.g. image processing, neural networks, data processing. Not good for oating point arithmetic. With this model, can put many processors on a single chip, then connect many such chips to produce high-speed but very compact machines (e.g. image processing boards in smart missiles). Not so many heavyweight processors Later SIMD machines (CM-5, MP-2) used more powerful 32-bit processors, often with vector oating point units. This approach is better for applications that are oating point intensive. With this model, a SIMD machine can be viewed as a large, distributed memory version of a vector machine (Cray), but more like a parallel array (2D grid) model than a vector (1D pipeline) model.
[KAH/PDC] DHPC, July-October 2000, 15
DHPC
HPC Architectures
Programming for SIMD Architectures
SIMD model can only eciently support regular, synchronous applications, which are perhaps only half of all HPC applications. Thus SIMD computers are usually used as special-purpose machines, e.g. for image processing, regular grid problems, neural networks, data mining, data processing. Key issue is whether SIMD machines are more cost-eective than MIMD machines for the applications they can handle. Currently only true for small number of image processing and data processing applications, since commodity processors have greatly increased in power. Is this limited segment of a limited HPC market enough to keep the remaining companies (Maspar and CPP) commercially viable, when under increasing pressure from e.g. SGI, IBM, Sun, Compaq? SIMD machines are easier to program (using high-level data parallel languages) than MIMD machines (using lower-level message-passing languages), but the possible applications are restricted. Most modern general-purpose parallel computers are MIMD, but provide compilers for data parallel languages that emulate a SIMD model on a MIMD message-passing machine.
[KAH/PDC] DHPC, July-October 2000, 16
DHPC
HPC Architectures
SIMD Distributed Memory Machines
ICL/AMT/CPP DAP 1K to 4K proprietary bit-serial processing elements (PEs). 64 PEs per chip. Later models have optional oating point unit (FPU). Connected as 2D grid, with fast row/column data highways. TMC CM-1 and CM-2 16K to 64K proprietary bit-serial PEs, multiple processors per chip. Connected as hypercube. 32 PEs can share a vector FPU in the CM-2. Maspar MP-1 and MP-2 1K to 16K proprietary PEs, 4-bit for MP-1 (16 to a chip), 32-bit for MP-2. 2D grid, with connections to 8 nearest neighbors. SIMD machines were originally targeted at specic problems like image processing and AI (CM-1 could initially only be programmed in parallel Lisp!). Attempted to become more general purpose by adding FPUs, but could not compete with MIMD machines. Now reverting to special-purpose architecture. Some interest in building SIMD-like supercomputers using programmable logic chips (FPGAs). Many processors now have SIMD extensions (G4 Velocity Engine, Pentium SSE, Athlon 3D Now!, graphics chips).
[KAH/PDC] DHPC, July-October 2000, 17
DHPC
HPC Architectures
Processors for MIMD Architectures
Early MIMD machines (e.g. Caltechs Cosmic Cube) used cheap o-the-shelf processors, with purpose-built inter-processor communications hardware. Motivation for building early parallel computers was that many cheap microprocessors could give similar performance to an expensive Cray vector supercomputer. Later machines (e.g. nCUBE, transputers) used proprietary processors, with on-chip communications hardware. These could not compete with the rapid increase in performance of mass-produced processors for workstations and PCs, e.g. year-old 16-processor nCUBE or transputer machine typically had same performance as a new single-processor workstation! Current MIMD machines use o-the-shelf processors (as before), usually RISC processors used in state-of-the-art high-performance workstations (IBM RS-6000, SGI MIPS, DEC Alpha, Sun UltraSPARC, HP PA-RISC). Processors for PCs are now of comparable performance, and the rst machine to reach 1 Teraop (1 trillion oating point operations per second), the 9200-processor ASCI Red from Intel, uses Pentium Pro processors. These machines still need special communications hardware, and expensive high-speed networks and switches.
[KAH/PDC] DHPC, July-October 2000, 18
DHPC
HPC Architectures
Massively Parallel Processors
The rst MIMD parallel computers were tightly coupled machines, with racks of motherboards containing processors plus memory plus communications hardware. Each board might contain multiple nodes (processors plus memory). Local memory is usually quite small (256 Kbytes for early machines) and has to hold a copy of the operating system and the user program, as well as local data. Could not use full-featured OS like Unix, so used small proprietary OS (often with undesirable features). Often called massively parallel processors (MPPs) since their components are cheap and tightly coupled, enabling them to scale to large numbers of processors. Examples are nCUBE, Intel Paragon, TMC CM-5, Meiko, Cray T3E. Dominated HPC parallel processing market during 80s and early 90s.
[KAH/PDC]
DHPC, July-October 2000, 19...