The ARC Information System

Transcription

1 The ARC Information System Overview of a GLUE2 compliant production system Florido Paganelli, Lund University EGI Community Forum 2012, Munich, March 2012

2 Outline Existing solutions ARC key concepts and technologies Technical insight Integration challenges Summary 2

3 Information systems existing strategies pull LCG Clients BDII top pull EGIIS Unicore Registry pull BDII site ARC LIB reg reg pull BDII resource Local/Resource glite ARIS Local/Resource ARC CIP Local/Resource Unicore Courtesy of Laurence Field 3

4 GRID information ecosystems Stack Service Level Registry Information Data Model Model Global Cache Transport Federated Model glite BDII (resource) GOC DB (Not EMI) GLUE 1.3 LDIF BDII (Top) Pull Kind of ARC ARIS EGIIS NorduGrid Schema LDIF No Pull Not really Unicore CIP Unicore Registry GLUE 2.0 XML No Pull Not really EMI ERIS EMIR GLUE 2.0 BDII Pull LDIF Yes Courtesy of Laurence Field 4

5 The ARC Information system key concepts Information must be as fresh as possible. The main source is therefore the local or resource level Index must be as lightweight as possible: EGIIS contains just URLs. No caching of information: it only makes sense when fresh Being able to speak all the possible information system dialects As a consequence, clients accessing the resources need to discovery and query resources on their own. 5

6 Technologies A collection of Perl scripts, the infoproviders, collecting information LDAP: Web Services: A proprietary backend consisting of a schema and the EGIIS index A dynamically populated OpenLDAP server and a collection of ldap Berkeley Database update scripts A proprietary Web Services Container ARCHED, serves info via WSRF based on XML performance only depends on slapd and optimization of perl scripts 6

7 ARC Information System Architecture Enhanced Grid Information Indexing Service ARC Resource Information System (local information provider) 7

8 ARC IS: extended to speak other dialects EMI Registry 8

9 A deeper look: ARIS components and dialects WSRF XBES (A-REX BES) arched WS XML Starts EMIES LDAP Configuration and Startup Script A-REX Infoproviders Configuration Stand-alone Infoproviders LDAP SLAPD BDII LDIF 9

10 A deeper look: ARC registration and caching RESTful arched WS XML Starts A-REX Infoproviders LDAP Configuration and Startup Script LDAP Stand-alone Configuration Registration Scripts Stand-alone Infoproviders SLAPD BDII LDIF ARC Computing Element (resource level) 10

11 Spoken dialects models and renderings NorduGrid Glue1.2/1.3 GLUE2 XML GLUE2 LDIF NorduGrid Glue1.2/1.3 GLUE2 11

12 Spoken dialects models and renderings NorduGrid Glue1.2/1.3 GLUE2 GLUE2 Most Complete! Includes all info the other schema have! XML GLUE2 LDIF NorduGrid Glue1.2/1.3 GLUE2 12

13 Local information tree: LDAP o=glue (GLUE2) All the information is collected and generated on the cluster by the infoproviders. Infosys supports all the three schema alltogether. backward compatibility is assured. Each schema a separate LDAP db: no disruption if one of the schema has problems Slapd configuration is automatically created when the infosys starts The trees are dynamically updated o=grid Mds-vo-name=local,o=grid (NorduGrid) Mds-vo-name=resource,o=grid (Glue 1.2/1.3) 13

14 Local information tree: LDAP GLUE2 o=glue (GLUE2) AdminDomain GLUE2GroupID=grid GLUE2GroupID=resource Service (ARIS) Endpoint(s) ComputingService (A-REX) ComputingEndpoint(s) AccessPolicy(ies) ComputingManager GLUE2GroupID= ComputingActivities ComputingShare MappingPolicy(ies) ComputingActivity(ies) GLUE2GroupID= ApplicationEnvironments ApplicationEnvironment(s) ApplicationHandle(s) GLUE2GroupID= ExecutionEnvironments Benchmark Location ExecutionEnvironment(s) Benchmark Can be attached to AdminDomain or Service Contact 14

15 Local information tree: XML GLUE2 <Domain> AdminDomain Service (ARIS) Endpoint(s) AccessPolicy(ies) ComputingService (A-REX) ComputingEndpoint(s) ComputingManager ComputingShare MappingPolicy(ies) <ComputingActivities> ComputingActivity(ies) <ApplicationEnvironments> ApplicationEnvironment(s) <ExecutionEnvironments> Benchmark Location ExecutionEnvironment(s) Can be attached to AdminDomain or Service Contact ApplicationHandle(s) Benchmark 15

16 Integration challenges: caching ARC resource information in Top-bdii o=glue (GLUE2) Service (ARIS) Endpoint(s) ARC CE AdminDomain GLUE2GroupID=grid GLUE2GroupID=resource ComputingService (A-REX) ComputingEndpoint(s) ComputingShare ComputingManager o=glue (GLUE2) GLUE2GroupID= ComputingActivities AccessPolicy(ies) GLUE2GroupID=resource MappingPolicy(ies) GLUE2GroupID=grid ComputingActivity(ies) Service (BDII) GLUE2GroupID= ApplicationEnvironments Endpoint(s) AdminDomain1 GLUE2GroupID= ComputingService ExecutionEnvironments Benchmark Location Can be attached to AdminDomain or Service ApplicationEnvironment(s) Contact ApplicationHandle(s) Service ExecutionEnvironment(s) TOP-BDII Location Can be attached to AdminDomain or Service AdminDomain2 Benchmark Contact ComputingService (glite). 16

17 Summary ARC enhanced information system has a layer structure with clean separation of functionalities: local/resource level (ARIS) index level (EGIIS and third party: Top-BDII, EMIR); ARIS is a powerful local information service: speaks multiple dialects (NorduGrid, Glue1, GLUE2) Creates LDIF and XML renderings ARC is the only middleware that renders both Offers information via both LDAP and Web Services Information consumers: are easy to write as they would be based on well known standard technologies; get from ARIS very accurate information. Aiming at a full GLUE2 integration with Top-BDII, EMIR 17

18 Thanks! 18

19 Backup slides 19

20 Resource Information Consumers LDAP: Can simply be queried using ldapsearches ldapsearch x h piff.hep.lu.se p 2135 b 'GLUE2GroupID=resource,o=glue' Web Services: Communication encrypted (certificates) WSRF GLUE2: https using arcwsrf (SOAP) arcwsrf BES, ARC XBES, EMI-ES: Any client capable of speaking these protocols 20

21 LDAP vs XML o=glue (GLUE2) AdminDomain GLUE2GroupID=grid GLUE2GroupID=resource Service (ARIS) Endpoint(s) ComputingService (A-REX) ComputingEndpoint(s) ComputingManager GLUE2GroupID= ComputingActivities AccessPolicy(ies) ComputingShare MappingPolicy(ies) ComputingActivity(ies) GLUE2GroupID= ApplicationEnvironments ApplicationEnvironment(s) GLUE2GroupID= ExecutionEnvironments Benchmark Location ExecutionEnvironment(s) Can be attached to AdminDomain or Service Contact ApplicationHandle(s) Benchmark 21