The Role and Functions of European Grid Infrastructure Luděk Matyska Masaryk University and CESNET Czech Republic (Ludek.Matyska@cesnet.cz) EGI_DS Project Director
What is a Grid? A distributed system that connects primary computers and storage owned by different actors through high speed network allowing shared use of all these resources Without a single administrative authority In a coordinated way, with clear guarantees No volunteering Eventually SLA (Service Level Agreements) Croatian Grid Day 26th October 2009 2
How users are organized? Virtual Organizations (VOs) Users with same (similar) goal True organization, with a manager and membership rules Negotiation with resource owners Resources made available to VOs, not individual users VO takes part of the responsibility for their members VO also takes care of quality of resources Croatian Grid Day 26th October 2009 3
How to build a Grid? Infrastructure for collaboration Must exists an interest in collaboration Defining the basic functions and responsibilities Setting up VOs Putting all this together Croatian Grid Day 26th October 2009 4
EGEE Production Grid Infrastructure Baltic Grid TERAGRI D OSG DEISA EUMedGrid See-Grid EUChinaGrid NAREGI EUIndiaGrid EUAsiaGrid EELA cca 300 sites in 50 countries 300 kjobs/day around 80 000 CPU cores 15 PB storage > 250 Virtual Organizations Croatian Grid Day 26th October 2009 5
National Evolution Sustainable European Grid Global Routine Usage Testbeds Utility Service Croatian Grid Day 26th October 2009 6
European Grid Initiative Vision Join efforts currently fragmented in different Grid infrastructure projects To create a shared (common) European Grid Infrastructure That connects all actors users and resource providers and their institutions In a sustainable way Fully described in the EGI Blueprint A deliverable from EGI DS project, endorsed by EGI Policy Board on 20 th January 2009 Croatian Grid Day 26th October 2009 7
EGI building blocks National Grid Initiatives (NGIs) EGI.eu as a coordinating body With the explicit involvement of International organizations (CERN, EMBO, ) Users And with support of National funding agencies European Commission Croatian Grid Day 26th October 2009 8
National Grid Initiatives National coordinating institutions legal bodies that take care of national Grid infrastructure To guarantee the coordination of efforts within a country Different organizational models Sustainable business models that guarantee long term financial stability As do have NRENs currently Croatian Grid Day 26th October 2009 9
EGI.eu The coordinating body at the European level Medium sized organization, owned by NGIs Proposed to have around 50 FTEs Guarantees the coherent setup and operations of the European Grid infrastructure Will be located in Amsterdam Selection process started in Autumn 2008 Decision taken by NGI representatives on March 2 nd Statutes currently under preparation Expected real startup late winter 2010 Croatian Grid Day 26th October 2009 10
Basic Principles EGI does not create NGIs, the process is (and must be) exactly the opposite NGIs are building blocks NGI setup and operation is left on each individual country Different approaches expected EGI framework defines the international aspects of national grids interoperation Task that will be necessary regardless of the name and organizational setup The unified framework should lead to the more efficient, less expensive operations than independent bilateral and multilateral agreements All steps and proposals consulted with the NGI representatives and received huge endorsement Croatian Grid Day 26th October 2009 11
EGI benefits For international collaborations (e.g. ESFRI projects or HEP) Provides compute and storage infrastructure at the pan-european (global) scale Provides unified access to resources already available (owned by the community) but fragmented without EGI Increases efficiency of use of the resources Increases robustness For small or fragmented communities Provides means and tools to interconnect their own resources The standard access problems known to each new (international) collaboration already solved within EGI Ability to quickly provide access to data (with access control), compute and eventually other resources (e.g. instruments) Could offer additional resources through scavenging Unused resources from other communities For Funding Agencies Increase of efficiency through resource sharing Increase in robustness of provided infrastructure Croatian Grid Day 26th October 2009 12
EGI.eu and NGI Tasks NGI NGI NGI NGI EGI.eu EGI.eu global tasks NGI international tasks NGI local tasks Croatian Grid Day 26th October 2009 13
EGI.eu Global Tasks Operations and security services Those spanning the whole infrastructure Drawing on man power provided by NGIs Middleware services No development, but verification and testing Also steering further mw development to suits best EGI requirements User community services Help users to organize at the international level Provide coordination of user support and training services External Liaison services Relationship with non-european activities/infrastructures Standardization liaison Policy Croatian Grid Day 26th October 2009 14
EGI.eu Operations and Security Primary coordination and monitoring EGI operations portal EGI configuration repositories EGI monitoring data repositories EGI accounting repositories Coordination of EGI resource allocation and brokerage EGI catch-all core services Highest level of user support EGI central ticket handling system Highest level of security coordination and reaction EGI security policies and coordination EGI global incident response coordination Middleware deployment and rollout Interoperations with non-eu grids Croatian Grid Day 26th October 2009 15
EGI.eu Middleware Services UMD Universal Middleware Distribution Best interoperable components from different middleware stacks Initialized with the components from the glite, ARC, UNICORE Production level quality EGI.eu responsibility during the selection process EGI.eu responsible for: Quality and conformance criteria for UMD components A repository of certified UMD components Tools to enable and support the processes Croatian Grid Day 26th October 2009 16
EGI.eu User Community Services Services for new and small communities Front desk operation Both at the international level Coordination of Specialized Support Centers (SSC) Coordination of training effort Repository of training materials Event organization Grid planning and technical coordination Croatian Grid Day 26th October 2009 17
EGI.eu External Liaison Services Dissemination Central EGI Web maintenance Policy and external liaisons Standardization bodies liaison Croatian Grid Day 26th October 2009 18
NGI International Tasks Tasks that must be run by each NGI to create a truly international EGI Cover the responsibilities stemming from international collaboration Reflect same or similar tasks at the national level Different size of NGI teams responsible for these tasks Reflects NGI size and also its involvement in the international collaboration Croatian Grid Day 26th October 2009 19
NGI International Tasks II Corresponds to the EGI.eu coordination tasks Operations and security services Middleware services User community services Nationalization of these services In fact interface between national and international service Based on man power used to run national grid infrastructure Croatian Grid Day 26th October 2009 20
EGI Management/Governance Members NGI 1, NGI 2, NGI 3, NGI n Associate Members e.g. EIROforum members, Non-voting Representatives extra-eu NGIs, Chair of UFSC, User Forum Steering Committee (UFSC) User Forum (UF) EGI.eu EGI Council EGI Director Advisory Committees e.g. Middleware Coordination Board (MCB) UCO User Coordination CTO Middleware Maintenance CAO Admin & PR COO Operations User Community Services Middleware Unit Administration & PR Unit Operations Unit EGI Croatian Grid Day 26th October 2009 21
Middleware Middleware is the glue of a Grid Software components providing general tasks for the whole infrastructure Workload and data management, authentication and authorization services, monitoring, Current situation similarly fragmented as with the Grid infrastructures Agreement on the Unified Middleware Distribution Croatian Grid Day 26th October 2009 22
Unified Middleware Distribution Production level middleware components Specifically tailored for EGI Coming from different sources The middleware consortia VOs and their middleware developers Eventually commercial middleware adoption Process strictly in hands of EGI Inclusion of components, conformance testing and certification Croatian Grid Day 26th October 2009 23
Transition towards EGI A path from current fragmented environment into the shared EGI A concern for large infrastructures (e.g. EGEE) No interruption of services ( happy users and resource providers) Staffing of the EGI.eu Move/outsource personnel from other sites (esp. CERN) Outsourcing part of activities to NGIs (global tasks) Croatian Grid Day 26th October 2009 24
Funding National and Institutional funding for research infrastructures NGI Grids EC funding International coordination EGI.eu global tasks International activities NGI International tasks Innovation Start up phase National co-funding of all these activities Croatian Grid Day 26th October 2009 25
EGI Financial Reference Model NGI International tasks Operations User services Middleware maintenance EGI.org Costs EGI Effort (FTEs) 225 110 70 51 Total EGI costs (M /year) 20.25 9.90 6.30 4.59 41 Income NGI membership fees 1,0 1 (M /year) EC Contribution (M /year) 10.125 4.95 3.15 1.80 20 NGI, Middleware consortia and EIRO/ESFRI co-funding Contribution (M /year) 10.125 4.95 3.15 1.80 20 Croatian Grid Day 26th October 2009 26
EU Call Opened 30 th July 2009 Focus on DCI (Distributed Computing Infrastructure) EGI explicitly mentioned in the first priority However, not enough money in one pot to follow directly the EGI Blueprint model Several closely related projects prepared to implement the EGI vision as defined in the EGI Blueprint Croatian Grid Day 26th October 2009 27
EGI InSPIRE The major EGI project Focus on operation of EGI Grid Global and International tasks Middleware unit to deal with middleware offers Also to guarantee first level support Specific support of Heavy User Communities (HUC) included HEP, Bio, Does not deal with users in general nor with middleware development and extended support Croatian Grid Day 26th October 2009 28
ROSCOE Robust Science Communities for EGI Project to implement the well known SSC (Science Support Centers) Provides tailored innovation for selected scientific areas HEP, Life Sciences, Computational Chemistry and Material Science Technology, Grid Observatory, Complexity Science, Also training and dissemination Several other SSC-related projects under preparation Croatian Grid Day 26th October 2009 29
EMI European Middleware Initiative Middleware developers from glite, ARC, and UNICORE Selection of components from all three stacks Further development, harmonization, and also maintenance Pawing the road to the Unified Middleware Distribution (UMD) New areas like virtualization support also considered Again, other middleware related tasks under preparation Croatian Grid Day 26th October 2009 30
Schedule and Milestones 13 th June: Final MoU published 30 th June: Draft EGI.eu Conventions and Statutes available 1 st July: Deadline for MoU signature (first round) 9 th July: Interim EGI Council convened in Amsterdam Endorsement of steps already taken Endorsement of Editorial Board Endorsement of the interim EGI.eu director EGI.eu Convention and Statutes discussion Setup of the Steering Committee 30 th July: EC Call open 24 th September: 2 nd EGI Council meeting EGI Council chair elected Per Oster Decisions about the EGI project 1 st October: Financial contributions to EGI Collaboration due 6 th October: EGI DS 2 nd review 23 rd October: 3 rd EGI Council meeting Vice chair appointed 4 year project endorsed, other related decisions taken Presentation and discussion on new version of EGI.eu statutes Steering committee authority precised 24 th November: EC Call closed 3-4 th December; EGI DS Workshop (From past to the future) 4 th EGI Council meeting Final EGI.eu statutes? February 2010: EGI.eu inauguration Winter 2010: Negotiation phase for EGI projects 1 st May 2010: EGI projects launched Croatian Grid Day 26th October 2009 31
Summary EGI represents the future of European Grid Infrastructure It is also very important part of the ERA einfrastructure (together with networks and supercomputers) EGI will provide a general compute and data storage platform for scientific community in Europe International collaboration through international VOs Environment where users can meet and work together Huge compute resources Huge data resources Appropriate security models to reflect different users needs Based on national activities and initiatives The primary guarantee for sustainability Also guarantees that national interests are not neglected VOs as means for users to organize and influence EGI Making sure users are strong enough with a voice EC support complements the national funding Sustainability based on national commitment to grid infrastructure EGI project preparation in very good shape A common European undertaking Croatian Grid Day 26th October 2009 32
http://www.eu-egi.eu contact@eu-egi.eu Croatian Grid Day 26th October 2009 33