High-Performance Data Grid Toolkit

A SciDAC Program Project

Constructing the distributed computing machinery to support SciDAC goals requires efficient, high-performance, reliable, secure, and policy-aware management of large-scale data movement. An effective solution requires an integrated approach to the development of "Data Grid" infrastructure that recognizes and exploits commonalities in two independent dimensions: among different levels of the "service stack" so that network, middleware, and application-specific services are developed in a coordinated fashion; and across different applications, in order to develop infrastructure that is truly general purpose.

The goal is to develop a set of essential network and middleware services required to support large-scale, data-intensive Collaboratory applications.

The objectives of this project are

• Develop new protocols taking advantage of unique properties of data grids
• Develop innovative techniques for co-reservation of computer, network, and storage resources with guarantees for each stage
• Develop market brokering services to make efficient decisions in the face of constraints, when multiple resources are available to fulfill a request
• Investigate variants of two-phase I/O strategies used in parallel I/O systems for data transfer optimization
• Develop intelligent, adaptive recovery and performance strategies based on knowledge of end-to-end routes and guarantees

For more info, see:

• Project website: High-Performance Data Grid Toolkit
• 2-Page Summary, March 2004 (PDF)
• 2-Page Summary, March 2003 (PDF)
• 4-Slide Presentation, March 2003 (PowerPoint Presentation)
• Overview: SciDAC Data Grid Middleware (PDF 178KB)
• Proposal: SciDAC Data Grid Middleware Proposal