Ian Foster
Tom Garritano
University of Chicago and Argonne National Laboratory
Each day brings another magazine or newspaper story about Grid computing. That new generation of collaborative middleware addresses the great challenges of science and engineering research, from particle colliders generating petabytes of high-energy physics data to seismological studies that combine the latest computer modeling with remote access to massive "shake tables" for simulating earthquakes.
Just as the World Wide Web was initially the sole province of researchers but soon migrated to industry, many observers predict that Grids are on the verge of moving beyond science and engineering, influencing business and mainstream computing users. Thanks to the increasing affordability and speed of desktop computers and commodity networks that can aggregate to deliver supercomputer-level performance, the mainstream use of Grids technology is within reach.
The Grid Research, Integration, Deployment and Support Center (GRIDS, http://www.grids-center.org) was formed as part of the National Science Foundation Middleware Initiative (NMI) (see Resources), with the goal of creating a stable middleware infrastructure to permit seamless resource sharing across virtual organizations. NMI and GRIDS grew out of workshops and white papers that identified the need for production-quality software based on open-source and open-standard approaches.
The GRIDS Center Software Suite was first issued in mid-2002 via NMI Release One (NMI-R1). It includes the Globus Toolkit, Condor-G, the Network Weather Service and KX.509. NMI-R2 in October 2002 adds GSI-OpenSSH, GPT (Grid Packaging Tools), and GridConfig. These components were chosen for their combined benefits - they work together for maximum utility in which the whole is greater than the sum of its parts (see NMRI2).
Future GRIDS Center releases will follow regularly in April and October of each year. NSF funding of NMI extends to September 2004, with the strong possibility of additional years. Users can rely upon the GRIDS software distribution for timeliness, quality and a high degree of interoperability to make the most of on-line resources. GRIDS project organizers hope it will become the preferred source for major e-science projects such as GriPhyN, NEESgrid, TeraGrid, and others.
GRIDS Center software suite components
As part of NMI, GRIDS develops and seeks standard components and mechanisms for:
- Authentication, authorization, policy
- Resource discovery and directory
- Remote access to computers, data, instruments
GRIDS also promotes integration of these components with end-user tools (conferencing, data analysis, data sharing, distributed computing, etc.), with campus infrastructures, and with commercial technologies. The center's goals are to help define, develop, deploy, and support integrated software supporting 21st-century science and engineering applications. The result will be a national infrastructure that can be used by application communities to explore full-scale, meaningful Grid applications. (See Grid Primerfor application examples and links to project web sites.)
At the core of GRIDS is open-source software aimed at the national research, education, and scientific communities. For NMI-R2, GRIDS software supports Red Hat Linux 7.2 on IA32 or IA64 and Solaris 8.0 on 32-bit Sparc, with other platforms to be added in subsequent releases. Packaged GRIDS software includes:
| Globus Toolkit | The de facto standard for Grid computing, this open-source software provides a modular set of services that simplifies collaboration across dynamic, multi-institutional virtual organizations. It includes tools for authentication, scheduling, file transfer and resource description. |
| Condor-G | Whereas high performance computing (HPC) is often measured in operations per second, Condor-G emphasizes high throughput computing (HTC) to deliver processing capacity over longer periods of time -- days, weeks, months and beyond. NMI-R1 includes an enhanced version of the core Condor software that is optimized to work with the Globus Toolkit for managing Grid jobs. |
Network Weather Service (NWS) |
This distributed system periodically monitors and dynamically forecasts the performance that various network and computational resources can deliver over a given time interval, using a distributed set of performance sensors for instantaneous readings. |
GSI-OpenSSH |
This modified version of OpenSSH adds support for Grid Security Infrastructure (GSI) authentication. GSI-OpenSSH provides a single sign-on remote login capability for the Grid. |
Grid Packaging Tools (GPT) |
This collection of packaging tools is built around an XML-based packaging data format that provides a straight forward way to define complex dependency and compatibility relationships between software packages. GPT was used to create all of the Grids Center Software Suite bundles and is a pre-requisite for installing them. |
GridConfig |
These tools manage the configuration of GRIDS software components. It provides an easy way to generate and regenerate configuration files in native formats, and to ensure configuration consistency. |
Also packaged with the GRIDS software is a tool from the other NMI project team "EDIT" (Enterprise and Desktop Integration Technologies): KX.509 and KCA. This tool provides a bridge between Kerberos and PKI infrastructure. It is included in NMI-R1 to enable the PKI-based security infrastructure of the Globus Toolkit to integrate with Kerberos-based authentication implemented at university campuses.
How the GRIDS Components Interoperate
Note that this document is a high-level overview. For installation and user guides, see the NSF Middleware documentation. Although each component in the GRIDS Center Software Suite may operate on other operating systems, Solaris 8.0 on SPARC and RedHat 7.2 or 7.3 on IA32 are the officially tested and supported platforms for NMI-R1. Installation of the software is simplified due to bundling with GPT, which lets users choose whether to install components collectively or individually.
The Globus Toolkit comes from the Globus Project, which is led by Argonne National Laboratory and the University of Southern California's Information Sciences Institute. This software is the foundation on which users may build Grids and Grid-based applications. It has three key functions:
- Resource Management to allocate shared computers, storage, sensors, instruments, networks and software
- Information Services to characterize these shared resources, and
- Data Management to let users access and organize information generated by the resources.
Three primary tools provide these functions: the Globus Resource Allocation Manager GRAM), the Monitoring and Discovery Service (MDS), and GridFTP. All components use the Grid Security Infrastructure (GSI) security protocol, which is based on X.509 certificate authentication.
Condor-G is from the University of Wisconsin at Madison. It manages both a queue of jobs and a set of resources from one or more sites where those jobs can execute. Condor-G communicates with these resources and transfers files to and from these resources using Globus mechanisms. In particular, it uses the GRAM protocol for job submission, and runs a local server for file transfers.
Condor-G lets the user submit many jobs at once and monitor them with a convenient interface. It notifies the users when jobs complete or fail, and it maintains Globus Toolkit credentials that may expire while a job is running. Condor-G is also fault-tolerant -- if a machine crashes, the user can still perform all of these functions after the system reboots.
For example, every user machine in a campus Grid would use Condor-G for submitting jobs to the Globus Toolkit. The GRAM gatekeeper component would authenticate users and hand their jobs to the job manager, which in turn would interface with the campus's preferred local scheduler.
The Network Weather Service (NWS) is from the University of California at Santa Barbara. It is a distributed, generalized system for producing short-term performance forecasts based on historical performance measurements. NWS has four functions:
- A Name Server directory used to bind process and data names with low-level contact information (such as TCP/IP port numbers or address pairs)
- A Memory Server for persistent storage of data
- A Sensor to gather performance measurements from a specified resource
- A Forecaster to predict the deliverable performance of a resource during a specified time
Each component runs in the background, listening on a dedicated port, which the user may specify if desired. The user would install one Name Server, one or more Memory Servers (which may reside on different machines), and a Sensor running on each machine whose resources would be monitored. These functions complement the Globus Toolkit's and Condor-G's functions of defining and managing an array of Grid resources.
The GRIDS Center software suite includes KX.509, from the University of Michigan. It is a client program that lets sites easily convert Kerberos certificates to the X.509 format used by Grids. The certificate and private key generated by KX.509 are stored in the same cache alongside the user's Kerberos credentials. That lets the system use an automated mechanism for removing unused Kerberos credentials - if one is in place on-campus - to also automatically remove the X.509 credentials. Netscape and Internet Explorer will accept these credentials for secure https web activity.
New in NMI-R2
New to the GRIDS suite since October 2002, GSI-OpenSSH is a modified version of OpenSSH that adds support for Grid Security Infrastructure (GSI) authentication. Developed by the National Center for Supercomputing Applications (NCSA), GSI-OpenSSH can be used to log into remote systems and transfer files between systems without entering a password. Instead, all operations can be authenticated using a valid GSI credential. GSI-OpenSSH can forward GSI credentials to the remote system on login, so the user is not required to manually create a new GSI proxy credential on that system. In this way, GSI-OpenSSH provides a single sign-on remote login capability for the Grid.
NCSA's GPT, which was used behind the scenes to package all GRIDS software in the original NMI release, is now an official component in the latest NMI-R2. It provides a straightforward way to define complex dependency and compatibility relationships between packages. The tools let developers easily define the packaging data for inclusion in their source code distribution. Binary packages can be automatically generated from the data. Packages defined by GPT are compatible with other packages and can easily be converted. GPT permits collections of packages to be built and/or installed, providing a package manager for systems that lack one.
And finally, GridConfig from the San Diego Supercomputer Center (SDSC) is included as a "preview." It is used to configure and finetune Grid technologies. It provides an easy way to generate and regenerate configuration files in native formats, and to ensure consistency within and among applications. For NMI-R2, the use of GridConfig is optional, although it will be a standard feature of subsequent releases.
GRIDS Center Software Suite components were chosen by the NMI leadership for their collective value in creating and managing computational Grids that facilitate use of powerful on-line resources. For technical assistance, send mail to nmi-support@nsf-middlware.org. For more information, send mail to contact@grids-center.org and visit http://www.grids-center.org.
In addition to support from NSF, the GRIDS software developers wish to acknowledge contributions from the U.S. Department of Energy, the Defense Advanced Research Projects Agency and NASA.
Resource:
Grid Research Integration and Support Center
Thanks for this great site!
Thanks for this great site! I've found a lot of interesting information about Grid futures here! My page...