{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

DiplomaThesis Improving performance of selected MPI Collective comm operation on infiniband

DiplomaThesis Improving performance of selected MPI Collective comm operation on infiniband

Info iconThis preview shows pages 1–6. Sign up to view the full content.

View Full Document Right Arrow Icon
Faculty of Computer Science Diploma Thesis Improving the Performance of Selected MPI Collective Communication Operations on InfiniBand Networks Carsten Viertel Chemnitz, April 30, 2007 Supervisor: Prof. Dr.-Ing. W. Rehm Advisor: Dipl.-Inf. Torsten Hoefler
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Contents List of Figures iii 1 Introduction 1 2 InfiniBand 2 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.2 Components and Topology . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.3 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.3.1 Queuing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.3.2 Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.3.3 Types of Service . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.3.4 Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.4 Verbs API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 3 Open MPI 6 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.2 Component Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.2.1 Point-to-Point Communication Components . . . . . . . . . . . . . 7 3.2.2 Collective Component . . . . . . . . . . . . . . . . . . . . . . . . 7 3.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 4 Model 10 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.2 Overview of Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.2.1 PRAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.2.2 BSP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.2.3 LogP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4.2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4.3 LogP Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 4.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 4.3.2 Extended Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4.3.3 Measuring LogGP Parameters . . . . . . . . . . . . . . . . . . . . 15 i
Background image of page 2
C ONTENTS 5 Algorithms 19 5.1 MPI_Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5.1.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5.1.2 Linear Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5.1.3 Hierarchical Algorithms . . . . . . . . . . . . . . . . . . . . . . . 21 5.1.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.2 MPI_Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 5.2.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 5.2.2 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 5.2.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 5.3 MPI_Allgather . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 5.3.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 5.3.2 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 5.3.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 6 Implementation Details 36 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 6.2 Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 6.3 Memory Registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 6.4 Rendezvous Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 6.5 Eager Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 6.6 Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 6.7 Implemented Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 7 Benchmark Results 42 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 7.2 MPI_Scatter and MPI_Gather . . . . . . . . . . . . . . . . . . . . . . . . 42 7.3 MPI_Allgather . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 8 Conclusion and Future Work 47 Bibliography 48 A Testing Environments 52 A.1 OSCAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 A.2 CHiC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 B Glossary 53 C Theses 55 D Acknowledgements 56 ii
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
List of Figures 2.1 InfiniBand Channel Adapters and Fabric . . . . . . . . . . . . . . . . . . . 2 2.2 InfiniBand Consumers and Queues . . . . . . . . . . . . . . . . . . . . . . 3 3.1 Open MPI Modular Component Architecture . . . . . . . . . . . . . . . . 6 3.2 Open MPI Point-to-point Component Frameworks . . . . . . . . . . . . . . 7 3.3 Open MPI Collective Component Life Cycle . . . . . . . . . . . . . . . . . 8 4.1 Sending a Message in the LogP Model . . . . . . . . . . . . . . . . . . . . 12 4.2 Sending and Receiving on Multiple Nodes in the LogP Model . . . . . . . 13 4.3 Sending and Receiving on Multiple Nodes in the LogGP Model . . . . . . 14 4.4 Gap Parameters and Resulting Function . . . . . . . . . . . . . . . . . . . 18 4.5 PRTT(1,0,s) and LogGP prediction . . . . . . . . . . . . . . . . . . . . . . 18 5.1 Data Distribution before and after Scatter on 6 Processes . . . . . . . . . . 19 5.2 Scatter using a Recursive Splitting Algorithm on 8 Processes . . . . . . . . 21 5.3 Binomial Tree of Order 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 5.4 Tree of Order i > 2 and with a per step Fanout of k > 1 . . . . . . . . . . . 24 5.5 N-Way Binomial Tree Scatter with a Step Fanout of 2 on 9 Processes . . . . 24 5.6 Schematic of the Scatter Algorithm on 9 Nodes using a per step Fanout of 2 26 5.7 Scatter LogfP Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.8 Scatter LogfP expected Communication Times . . . . . . . . . . . . . . . 28 5.9 Data Distribution before and after Gather on 6 Processes . . . . . . . . . . 29 5.10 Data Distribution before and after Allgather on 6 Processes . . . . . . . . . 30 5.11 Allgather using a Ring Algorithm on 5 Processes . . . . . . . . . . . . . . 31 5.12 Allgather using a Modified Ring Algorithm on 5 Processes . . . . . . . . . 32 5.13 Allgather using a Neighbor Exchange Algorithm on 6 Processes . . . . . . 32 5.14 Allgather using a Recursive Doubling Algorithm on 8 Processes . . . . . . 33 5.15 Allgather using the Bruck Algorithm on 6 Processes . . . . . . . . . . . . . 34 6.1 Usage of the Ranks in MPI_COMM_WORLD to access Queue Pairs . . . . 37 6.2 Rendezvous Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 6.3 Eager Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 7.1 Gather Performance Scaling with Communicator Size . . . . . . . . . . . . 42 7.2 Gather Performance Scaling with Buffer Size . . . . . . . . . . . . . . . . 43 iii
Background image of page 4
L
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 6
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}