Data Grid Services: The Storage Resource Broker. Andrew A. Chien CSE 225, Spring 2004 May 26, Administrivia

Transcription

1 Data Grid Services: The Storage Resource Broker Andrew A. Chien CSE 225, Spring 2004 May 26, 2004 Administrivia This week:» 5/28 meet ½ hour early (430pm) Project Reports Due, 6/10, to Andrew s Office Project Presentations, 6/11, 2-4pm, 4301 APM CSE225 Spring 2004

2 Last Time Reputation Systems Inform, Incent, Select Most useful when most Interactions with Strangers E-bay works well because» Buyers DO give feedback, average ratings are largely positive, changing ID s doesn t give advantage Not a monolithic technology large Design space Relation to Grids?» Open system for establishing/manage pairwise trust» Doesn t require a trusted oracle in control» Defaults can tie policy to reputation (effectively) Supports CPU Resource Allocation Less clear how to do Data/Storage Access Way of building Pairwise Trust CSE225 Spring 2004 Readings for Today Storage Resource Broker» Arcot Rajasekar, Michael Wan and Reagan Moore. My & - Components of a Data Grid, The 11th International Symposium on High Performance Distributed Computing (HPDC-11) Edinburgh, Scotland, July 24-26, 2002 CSE225 Spring 2004

3 Storage Resource Broker Sheau-Yen Chen Charles Cowart Arun Jagatheesan George Kremenek Reagan Moore Roman Olschanowsky Arcot Rajasekar Mike Wan Bing Zhu Introduction Features Case Studies Agenda

4 Introduction Problems with Data Handling Environment Problems with Metadata Handling and MCAT as a Solution Features of the System Motivation Scientific data is different from webdata Metadata is as important as the data itself Static links (hyperlinks) are deficient Needs dynamic linking through a higher layer Metadata as a means to do this linking Collection-owned data Deal with highly distributed data Virtualization of data access becomes important

5 Problems with Data Handling Large Datasets; Large Number of Datasets; Scaling Distributed, Heterogeneous Storage Collaboration, Access Control, Authentication, Security Replication, Coherency Fault Tolerance and Load Distribution Scheduling, Caching & Data Placements Data Migration over Time & Space Data/Collection Curation Uniform Name Space Problems with Metadata Types of Metadata Relational to XML to unstructured Standardized to User-defined Metadata Large Number of Attributes; Large Size; Scaling Federation - integration over space Evolution - integration over time Evolution - integration over contexts Discovery and Search Presentation user friendly

6 Data Grid Requirements Seamless access to data and information stored at local and remote sites Virtualization of data, collection and meta information Handle Dataset Scaling size & number Integrate Data Collections & Associated Metadata Handle Multiplicity of Platforms, Resource & Data Types Handle Seamless Authentication Handle Access Control as a Solution The Storage Resource Broker is a middleware It virtualizes resource access It mediates access to distributed heterogeneous resources It uses a MetaCATalog to facilitate the brokering It integrates data and metadata Application MCAT Server HRM DB2, Oracle, Illustra, ObjectStore HPSS, ADSM, UniTree UNIX, NTFS, HTTP, FTP Distributed Storage Resources (database systems, archival storage systems, file systems, ftp, http, )

7 SDSC Storage Resource Broker & Meta-data Catalog Application Resource, User MCAT User Defined Dublin Core Application Meta-data C, C++, Linux I/O Archives HPSS, ADSM, UniTree, DMF Unix Shell HRM Java, NT Browsers Prolog Python Web File Systems Databases Unix, NT, DB2, Oracle, Mac OSX Sybase Third-party copy Remote Proxies DataCutter Single SignOn Authentication Secure Password, GSI or SEA 1 Application Session Established Master (Host,port) 2 4 (port) Identification & Initialization Server spawned agents 3 CA 3 MCAT

8 Federated Operation Logical Name Or Attribute Condition Read Application Peer-to-peer Brokering Parallel Data Access server /6 server 4 agent 2 5 agent 1.Logical-to-Physical mapping 2. Identification of Replicas 3.Access & Audit Control R1 MCAT Data Access R2 Server(s) Spawning Concepts Abstraction of User Space Single sign-on Multiple authentication schemes certificates, (secure) passwords, tickets, group permissions, roles Virtualization of Resources Resource Location, Type & Access transparency Logical Resource Definitions - bundling Abstraction of Data and Collections Virtual Collections: Persistent Identifier and Global Name Space Replication & Segmentation Data Discovery system & application metadata User-defined Metadata Structural & Descriptive Attribute-based Access (path names become

9 Replication Core Functionality Synchronous Replication Replication via Logical Resource definition integrated into open/create & write function Can choose: k out of n Associate replication with containers/collections Consistency Asynchronous Replication - Offline srbobjreplicate API, Sreplicate command, GUI Out of Band Replication - outside Registering of Replicas using srbregisterreplica API Containers Physical Grouping of Objects Similar to tar but has significant differences Multiple Uses: To take advantage of resource characteristics To aid access patterns Move data sets together Tie together logically different files Automatic Archiving/Caching Chaining of Containers Sharing of metadata Containers for Collections

10 Containers (Contd.) 1.Create Container 2.Write File 1 3. Write File 2 4.Sync Container 5.Read File 1 6.Write File 3 7.Sync & Purge Access Control Access Control Datasets Collections Resources Multi-level Access Read, Annotate,Write, Curate, Own Access Control for Users and Groups Ticket-Based Access Control Audit Access

11 Parallel I/O Client-directed parallel I/O Thread-safe client Server-directed parallel I/O Server plans and decides number of threads to use Control and data transfer sockets Client listens on the control socket and spawns threads to handle data transfer Similar to HPSS Works seamlessly with HPSS Mover MCAT: Metadata Catalog Stores metadata about Data sets, Collections, Users, Resources, Proxy Methods Maintains replica information for data & containers Provides Collection abstraction for data Provides Global User name space & authentication Provides Authorization through ACL & tickets Maintains Audit trail on data & collections Maintains metadata for methods and resources Provides Resource Transparency - logical resources Implemented as a relational database Oracle or DB2 or Sybase

12 Meta Data Management Insert, Update and Delete of Metadata Access Control for Metadata Querying across system-level, user-defined metadata and annotations Query under collections & across collections Browsing on user-defined metadata Metadata supported for legacy files & directories Extract Metadata (using proxy operations) as a Commodity DataGrid Federated Server Architecture Uniform Access Interface Thread-safe Client Programmatic API (C, C++, Java, Perl-through-C, Python) GUI (Java for Unix, Windows Browser [ NT, Me, 98, 2000 ]) Web Support (CGI-Scripts, Portals, WSDL/SOAP) My a web-based client Command Line Interface (Unix, Linux, DOS) Metadata Catalog Oracle, DB2, Sybase,SQLServer Handles authentication, access control, auditing, Provides transparencies, replication, collection, container support, User-defined Metadata Multi-Platform Support Unix, Linux, Windows, MacOSX (from Cray to Desktop) HPSS, ADSM, UniTree,, UnixFS, NTFS,, Oracle, DB2, Integration of Data Handling & Digital Library Functionalities

13 Projects Digital Libraries UCB, Umich, UCSB, Stanford,CDL NSF NSDL - UCAR / DLESE NASA Information Power Grid DOE ASCI Data Visualization Corridor Astronomy National Virtual Observatory 2MASS Project (2 Micron All Sky Survey) Particle Physics Particle Physics Data Grid (DOE) GriPhyN SLAC Synchrotron Data Repository Medicine Digital Embryo (NLM) Earth Systems Sciences ESIPS LTER Persistent Archives NARA LOC Neuro Science & Molecular Science TeleScience/NCMIR, Brain Images SLAC, AfCS, DR Space DR DR DL Client Client DL DL Client Client Client DR DR MC Client DL DR DR - Data Repository DL - Dig Library MC - Meta Catalog

14 Discussion What kind of support/capability does provide for a user? What problem does it solve?» Performance oriented, large data manipulation, caching, coherence» Uniform access heterogeneity, local/remote How does this compare/differ from the traditional support for shared data?» Better metadata and access control than shared filesystem» Database system: support for external data, scaling, not much support for distributed federation What important problems does this solve for communities and their data?» Recognition and support for a richer set of roles than read/write CSE225 Spring 2004 Summary Requirements:» Seamless access to data and information stored at local and remote sites» Integrate Data Collections & Associated Metadata» Manage Access Control Provides» Abstraction of User Space : Single sign-on; flexible authentication» Resource Virtualization: Location, Type & Access transparency, Logical Resource Definitions» Abstraction of Data and Collections: Virtual Collections: Persistent Identifier and Global Name Space» Data Discovery system & application metadata» Uniform Access Methods» my provides a nice user interface CSE225 Spring 2004