Programs Writing that Run EveryWare on the Computational Grid
Rich Wolski, John Brevik, Graziano Obertelli, Neil Spring, and Alan Su
A BSTRACT The Computational Grid [12] has been proposed for the implementation of high-performance applications using widely dispersed computational resources. The goal of a Computational Grid is to aggregate ensembles of shared, heterogeneous, and distributed resources (potentially controlled by separate organizations) to provide computational power to an application program. In this paper, we provide a toolkit for the development of globally deployable Grid applications. The toolkit, called EveryWare, enables an application to draw computational power transparently from the Grid. It consists of a portable set of processes and libraries that can be incorporated into an application so that a wide variety of dynamically changing distributed infrastructures and resources can be used together to achieve supercomputer-like performance. We provide our experiences gained while building the EveryWare toolkit prototype and an its use in implementing a large-scale Grid application. Keywords: Computational Grid, EveryWare, Ramsey Number search, grid infrastructure, ubiquitous computing, distributed supercomputer I. I NTRODUCTION Increasingly, the high-performance computing community is blending parallel and distributed computing technologies to meet its performance needs. A new architecture, known as The Computational Grid [12], has recently been proposed which frames the software infrastructure required to implement high-performance applications using widely dispersed computational resources. The goal of a Computational Grid is to aggregate ensembles of shared, heterogeneous, and distributed resources, potentially controlled by separate organizations, to provide computational
Supported by the National Partnership for Advanced Computational Infrastructure (NPACI), NSF grant ASC-9701333, Advanced Research Projects Agency/ITO under contract #N66001-97-C-8531. University of California, Santa Barbara [e-mail] Holycross University [e-mail] University of Calfornia, San Diego [e-mail] University of Washington [e-mail] University of California, San Diego [e-mail]
power to an application program. Applications should be able to draw compute cycles, network bandwidth, and storage capacity seamlessly from the Grid in a way analogous to the way in which household appliances draw electrical power from a power utility. The framers of the Computational Grid paradigm identify four qualitative criteria for the concept to be realized. According to [12] (page 18), a Computational Grid must deliver consistent, dependable, pervasive, and inexpensive cycles to the end user. In this paper, we outline ve quantitative requirements which, if met, fulll the qualitative criteria from [12]. We also describe EveryWare a toolkit for constructing Computational Grid programs and evaluate quantitatively how well an example EveryWare program fullls the Computational Grid vision. Our evaluation is based on ve quantitative metrics: 1. Execution Rate: measures the sustained computational performance of the entire application. Although not mentioned explicitly as a criterion, the Grid must be able to deliver efcient execution performance which we measure in terms of sustained execution rate. 2. Adaptivity: measures the difference between the performance variability exhibited by the underlying resources and the performance variability exhibited by the application itself. If program execution is stable, independent of uctuations in resource performance (i.e. the program adapts to varying performance conditions successfully) we suggest that the program is able to sustain consistent execution. 3. Robustness: measures the overall duration of continuous program execution in the presence of resource failures. A program that can continue to execute effectively in the presence of unpredictable resource failure is a dependable program. 4. Ubiquity: measures the the degree of heterogeneity a program can exploit in terms of the number of different resource types used by the application. If a program can execute using any and all available resources (both software and hardware) it is a pervasive program. 5. Expense: measures the cost of the resources necessary to implement the infrastructure. This metric maps directly to the the expense criterion described in [12].
We will capitalize the word Grid when referring to Computational Grid throughout this paper.
2
Therefore, a program that achieves a high execution rate, which is able to adapt to rapidly changing performance conditions, which is robust to resource failures, which can execute ubiquitously and which requires little added expense over a single-machine program possess all of the qualities described in [12] that a Grid program must possess. EveryWare is a software toolkit consisting of of three separate components: a portable lingua franca that is designed to allow processes using different infrastructures and operating systems to communicate, a set of performance forecasting libraries that enable an application to make short-term resource and application performance predictions in near-real time, and a distributed state exchange service that allows application components to manage and synchronize program state in a dynamic environment. The goal is to allow a user to write Grid programs that combine the best features of different Grid infrastructures such as Globus [11], Legion [19], Condor [36], or NetSolve [6] as well as the native functionality provided by Java [18], Windows NT [27], and Unix to the performance advantage of the application. EveryWare is implemented as a highly portable set of libraries and processes that can glue different locally available infrastructures together so that a program may draw upon these resources seamlessly. If sophisticated systems such as Globus, Legion, or Condor are available, the EveryWare program must be able to use the features provided by those systems effectively. If only basic operating system functionality is present, however, an EveryWare program should be able to extract what ever functionality it can, realizing that these resources may be less effective than those supporting better infrastructure. The ability to execute ubiquitously with respect to all of the resources accessible by the user is key to meeting the pervasiveness criterion. By leveraging the most performance-efcient infrastructure that is present on those resources, an EveryWare application can ensure the best possible execution performance and the greatest degree of robustness possible. Designed to be quickly and easily portable, EveryWare is intended to be the thinnest middleware layer capable of unifying heterogeneous resources with various software infrastructures to accomplish a computational task. In a Grid environment with several incompatible software infrastructure choices, it has been challenging to build a distributed application running everywhere, until EveryWare. We have implemented a prototype toolkit to test the efcacy of the EveryWare approach. In an experiment entered as a contestant in the High-Performance Computing Challenge [22] at SC98, we were able to use this prototype to leverage Globus, Legion, Condor, and NetSolve
Grid computing infrastructures, the Java language and execution environment, native Windows NT, and native Unix systems simultaneously in a single, globally distributed application. The application, a program that searches for Ramsey Number counter-examples, does not perform an exhaustive search, but instead uses search heuristics such as simulated annealing to negotiate the enormous search space. Effectively implementing this approach requires careful dynamic scheduling to avoid substantial communicati...
ev-results