Archer Charles Project Plan CSCI 8101 --------------------For my project, I plan to implement an in memory key-value storage architecture for HPC (High Performance Computing)systems. As background, IBM has or supports many applications that make calculations and have to store the computations to disk. For example, the application GAMESS, a quantum chemistry code, computes many integrals. However, the integrals do not t into memory and must be stored to disk. Also, the computed integrals may be used by several nodes in a distributed computer system. DHT's and consistent hashing may provide a faster way to implement these integral calculations. BlueGene, like many HPC environments, has very fast internal networks, and relatively slow disk access. As a result, storing computations to disk becomes prohibitively expensive. Storing data in memory and allowing access to the networks through the fast networks may be desirable. Also, storing the data in multiple memory locations is not desirable because of the limited memory per node (512 or 1024 MB per node) makes the redundant storage of data counter productive. An ideal structure and storage model seems to be consistent hashing. The consistent hashing can be used to efciently implement a distributed hash table on BlueGene. I believe that implementing the ideas from Chord and replicating their results would be very benecial for understanding how scalable distributed applications can be built. I hope to build a software architecture that looks like this:
I plan to use the following schedule starting on Oct 20th, and ending Dec 11th:
Task
1) Implement Chord Like Lookup 2) Implement 3) DHT Write Function Signature Lookup Simulation 4) Workload distribution application 5) Performance and Scalability Measurements 6) Write up and Paper Evaluation
E!ort
3w 1w 1w 1w 1w 2w
Each week (9 weeks total), should contain approximately 10 hours of work dedicated to the project, for a total of 80-90 hours. An explanation of each of the tasks: 1) Implement chord-like lookup: Using the Chord paper, I will implement a chord like consistent hashing mechanism. This mechanism will be implemented on top of MPI for the transport. I've considered the use of sockets or some other transport, but I haven't seen any work that uses DHT on top of MPI, and I think that many applications could take advantage of it. If this code is written properly, it should still be portable to other platforms that support MPI 2) I will implement a distributed hash table on top of the Chord-like library.
3) As a test application, I would like to write a simple numerical integration routine, that stores the results of the integration after the rst computation. I would like to measure the timings vs a static partition (which should be the fastest), vs random function signature lookup. 4)I would like to write a simple workload management application. Perhaps an application that is traditionally written using a client/server model (such as a movie encoding application). The work units could be distributed using the DHT. 5) I would like to duplicate the performance and scalability numbers given in the Chord paper, but at Blue Gene scales (thousands of nodes) 6) Put it all together into a coherent study of the Chord application in HPC environments.