FP6 Phosphorus and FP7 Federica projects

Transcription

1 FP6 Phosphorus and FP7 Federica projects Network resources for grid applications and Europe-wide experimental open infrastructure Joan Antoni García Espín Network Engineer i2cat Foundation, Barcelona (Catalonia, Spain)

2 Outline The Phosphorus project (FP6) Network resource abstraction and allocation for Grid apps The Federica project (FP7) Open network infrastructure virtualisation The IaaS framework and Federica Beyond the Articulated Private Networks

3 PHOSPHORUS project overview What: 6 th FP project in the area Research networking test-beds 5.1 M EC contribution, 6.9 M budget 20 partners, 814 Person Months When: 1 st October th March 2009 (30 months) More: Project Vision and Mission The project addresses some of the key technical challenges in enabling on-demand e2e network services across multiple, heterogenous domains Phosphorus has demonstrated solutions and functionalities across a test-bed involving European NRENs, GÉANT2, Cross Border Dark Fibre and GLIF

4 Members of the PHOSPHORUS consortium NRENs & RON: CESNET Poznan Supercomputing and Networking Center SURFnet MCNC Manufacturers: SMEs: ADVA Optical Networking Hitachi Europe Ltd. NORTEL NextWorks HPC centers, Universities and Research Institutions: Communication Research Centre Fraunhofer Gesellschaft Fundació I2CAT (with UPC as Third party) Forschungszentrum Jülich Interdisciplinair instituut voor BreedBand Technologies Research Academic Computer Technology Institute Research and Education Society in Information Technology SARA Computing and Networking Services University of Bonn University of Amsterdam University of Essex University of Wales Swansea / University of Leeds

5 PHOSPHORUS Key Features Integration between application middleware and transport networks, based on three planes: Service plane: Middleware extensions and APIs to expose network and Grid resources and make reservations with those resources Security mechanisms (AAA) for network domains participating in a global network infrastructure, allowing both network resource owners and applications to have a stake in the decision to allocate specific network resources Network Resource Provisioning plane: Adaptation of existing Network Resource Provisioning Systems (NRPS) Implementation of interfaces between different NRPS to allow multi domain interoperability with Phosphorus resource reservation system Control plane: Enhancements of the GMPLS Control Plane (Grid-GMPLS or G²MPLS) to provide optical network resources as first-class Grid resource Interworking of GMPLS-controlled network domains with NRPS-based domains, i.e. interoperability between G 2 MPLS and Argia/UCLP, DRAC and ARGON

6 PHOSPHORUS Architecture

7 Harmony system: NRPS and NSP interfaces (WP1) Network Service Plane GRID APLICATIONS & MIDDLEWARE Request Handler Reservation Handler EWI Network Resource Provisioning Systems ARGON DRAC UCLP bw NBI SBI Transport Network NRPS Broker Scheduler Path Computer SBI GMPLS DB Transport Network Type of advance reservations bw TOPOLOGICAL CONFIGURATION NBI EI EI PHASE 2 OTHER PROJECTS G 2 MPLS Transport Network NBI: North-Bound Interface SBI: South-Bound Interface EWI: East-West Interface EI: External Interface t t (a) fixed reservation (b) deferrable reservation (c) malleable reservation bw NBI: It receives the reservation requests from the Grid Middleware. It is used to indicate to the NSP which are the resources under control (NRPSs, endpoints links). EWI: It is in charge of the communication between NRPSs. t SBI: EI: It handles the communication between the NRPSs and the lower layers (GMPLS or transport layer). It provides intero-perability between the NSP and the G 2 MPLS CP or other projects. Key points: Ability to create point-to-point connections using resources from several domains AAA mechanism for global authentication Advance reservations: users and Grid applications can program fixed, deferrable or malleable resource reservations with one or more connections

8 The NRPS systems Network Resource Provisioning Systems (NRPSs) ARGON DRAC UCLP (ARGIA) The Allocation and Reservation in Grid-enabled Optic Networks system was developed to manage resources of advanced network equipment as it is present in the German VIOLA testbed. The advance reservation service of ARGON is able to operate on the GMPLS as well as on the MPLS level. It guarantees a certain QoS for applications for the requested time interval. This feature enables a Meta-Scheduling Service to seamlessly integrate the network resources into a Grid environment. The Dynamic Resource Allocation Controller system was developed by NORTEL and it is a commercial-grade network abstraction and mediation middleware platform, acting as an agent for network clients (users, applications, compute resource managers) to negotiate and reserve appropriate network resources on their behalf. DRAC uses client's QoS requirements and predefined policies to negotiate end-to-end connectivity across heterogeneous in support of just-intime or scheduled computing workflows. The User Controlled LightPaths is provided by CRC, Inocybe and i2cat. It provides a network virtualization framework upon which communities of users can build their own services or applications. Articulated Private Networks (APNs) are presented as the first services. An APN can be considered as a next generation Virtual Private Network where a user can create a complex, multi-domain topology by binding together network resources, time slices, switching nodes and virtual/real routing services.

9 Current prototype: the Harmony Architecture Key points: Distributed (P2P) or hierarchical architecture for the Network Service Plane Harmony Service Interface common for the adaptation layer and the network service plane The Network Service Plane is composed by independent entities (Inter Domain Brokers) The distinct IDBs flood the information of each domain they control The new P2P architecture is being tested over the new virtual testbed

10 Harmony system overview 10

11 Harmony global architecture and NSP detail Middleware (WP3) Provided by the partners Implemented in Phosphorus Network Service Plane (NSP) zoom NRPS Adapter NRPS Adapter NRPS Adapter Thin NRPS Administrator Topology client Middleware MSS User Application ARGON (NRPS) DRAC (NRPS) G²MPLS G-Lambda GÉANT2 (JRA3) Reservation-WS NSP UCLP (NRPS) GMPLS drive Topology-WS Reservation-WS WS-Modules Notification-WS Req handler Network Service Plane Authn Topology Validation Path comp Resv setup Resv operation NRPS manager Java Database DB MySQL Key Points Ability to create point-to-point connections using resources from several domains AAA mechanism for global authentication Advance reservations: users and Grid applications can program fixed or malleable resource reservations with one or more connections Reservation-WS Reservation-WS Reservation-WS ARGON Adapter UCLP Adapter DRAC Adapter ARGON UCLP DRAC ARGON ARGON UCLP UCLP DRAC DRAC Domain Domain Domain Domain Domain Domain Reservation-WS GMPLS thin GMPLS GMPLS Domain Domain

12 The NRPS Manager (as part of the NSP) Module inside the NSP in charge of direct communications with NRPSs, by means of their endpoints It coordinates the call from the NSP to the NRPSs adapters and returns the replies The requests to the NRPSs adapters are launched at the same time to let the NRPSs work in parallel in order to shorten the request processing time. NRPS Controllers NRPS 1 Controller NRPS 2 Controller NRPS n Controller A B C D E F G H NRPS 1 NRPS 2 NRPS n A B C E D F G H

13 Existing NRPS comparison ARGON DRAC UCLP Network virtualization Advanced reservations AAA Restoration / Protection Partly (under further development) Partly (under further development) Point to point connections Point to multipoint connections Interface Type WS WS WS

14 NRPS Layer system architecture The project does not cover developments on internal architectures of the NRPSs systems but development of interfaces for NRPSs interoperability through the NSP Interfaces define common data types and common operations for NRPSs Specification of suitable NRPS Adapters for the interoperability layer. They act as a wrapper translator NRPS adapters provide translation of common operations to specific NRPS The NSP is the controller of the underlying NRPS for interoperability, since each NRPS uses its own specific interface operations and own specific data types: Reservation interface: reservation of resources in e2e path provisioning Topology interface: updating NSP with topology information The NRPS adapter enables communication between the NSP and NRPS Interoperability is achieved by web service implementation of abstract service interface descriptions, defined within WSDL

15 NRPS & NSP interoperability (high level view) Legend: East / West interfaces Reservation service interface Topology service interface Grid Middleware Request for end-to-end resource provisioning Current NRPS adapter functionalities Topology information Topology Service NRPS address information Network Service Plane NRPS Manager Reservation service client Database Path Computer Reservation WS: Availability Request Reservation Request Cancel Reservation Status Request Retrieve Features Retrieve Endpoints Reservation service Adapter Topology client Reservation service Adapter Resource reservations Topology client Reservation service Adapter Topology client ARGON driver UCLP driver DRAC driver ARGON domain UCLP domain DRAC domain Interoperability layer Topology WS: Add domain Delete domain Edit domain Retrieve domain Add Endpoints Delete Endpoint Edit Endpoints Retrieve Endpoints Add Link Delete Link Edit Link Retrieve Link

16 The NRPS Adapters WS Interface Common NRPS Adapter Driver Common NRPS Common AdapterNRPS Driver Common NRPS Common Adapter NRPS Driver Common NRPS Common AdapterNRPS Driver Specific NRPS Driver Adapter Specific NRPS Driver Adapter Specific NRPS Driver Adapter Specific NRPS Driver Adapter NRPS ARGON DRAC UCLP Middleware NSP NRPS Adapter UCLP NRPS Adapter ARGON NRPS Adapter DRAC NRPS UCLP NRPS ARGON THIN NRPS NRPS DRAC GMPLS Driver UCLP controlled network ARGON controlled network GMPLS controlled network DRAC controlled network

17 Thin NRPS for GMPLS CP Thin NRPS: is a network resource provisioning system for domains with a GMPLS control plane. It is a NRPS with restricted functionality Provides a reservation web service to reserve, create and delete network connections via the GMPLS driver Provides advance reservation services (checking end points availability and possible overlapping reservations) Provides notifications receiver interface Acts as a client of the Topology manager WS of the NSP Acts as a client of the GMPLS driver Domain registering Handles reservation request from NSP

18 The GMPLS Driver GMPLS driver: an interface between NRPS and the GMPLS CP. It is a general WS to create, delete and monitor paths for different GMPLS implementations, provides a WEB interface for testing the WEB service, Internal data base containing topology, path and status information, modules for accessing vendor specific GMPLS control planes (e.g. Alcatel-Lucent or Nortel and G(²)MPLS, dummy interface to GMPLS GMPLS driver services: Path creation Path termination Path monitoring Path discovery Endpoint discovery Registration service Path delete notification EndPOint update notification

19 The Network Service Plane (NSP) Phase 1 : Centralized Architecture (flat or hierarchical) IDB (top) IDB NRPS NRPS... NRPS NRPS... NRPS TN TN... TN TN... TN Phase 2 : Distributed-heriarchical Architecture (peering multi-level) IDB IDB IDB NRPS NRPS... NRPS NRPS NRPS... NRPS NRPS NRPS... IDB NRPS NRPS...

20 Enhancements to the GMPLS Control Plane for Grid Network Services (GNS) Extensions to the GMPLS CP for automatic and single step setup of Grid & network resources Grid GMPLS (G 2 MPLS) main research tracks: seamless coexistence with NRPS (UCLP, DRAC and ARGON) & Grid MW Grid aware network reference points (G.O UNI, G.E NNI, G.I NNI) CBR algorithms for recovery and TE Integration with AAA system G.O UNI G 2 G 2 G 2 G 2 Grid site A Expected innovation in the field of co allocation of Grid and network resources, because of faster dynamics for service setup adoption of well established procedures for traffic engineering, resiliency and crankback uniform interface for the Grid user to trigger Grid & network transactions NRPS G.I NNI G.E NNI G 2 MPLS G.O UNI Grid site B Grid site C

21 Demonstration scenario (real test-bed)

22 Demonstrator Client (create reservation) 22

23 Administrative Web (create reservation)

24 Administrative Web (reservation list and status)

25 G 2 MPLS roadmap In a short term Open source GMPLS Control Plane prototype for: Optical LSP setup/teardown via UNI interface compatible with OIF UNI 2.0 LSP protection/restoration and crankback Flexible adaptation to the management interfaces exported by the underlying Transport Plane (e.g. TL1, SNMP) Multi domain operations through OIF ENNI compliant interface Basic modules for routing and signalling in single control domain delivered on Q Recovery strategies and inter domain expected by Q In a longer term (but within project lifetime) Grid GMPLS enhancements to the Control Plane prototype, in terms of Discovery and advertisement of Grid capabilities and resources of the participating Grid sites Service setup and maintenance Coordination, co allocation and configuration of grid and network resources associated with a Grid job Recovery of the installed network services and possible escalation of procedures Advance reservations of Grid and network resources Service monitoring: retrieving of the status of a Grid job and the related network connections

26 Strategy to integrate WP1-WP2 developments MIDDLEWARE or DEMONSTRATOR CLIENT TOPOLOGY CLIENT Connect A to B Network Service Plane OIF signalling (propagate network info, NO Grid info) NRPSs ADAPTER ADAPTER ADAPTER ARGON DRAC UCLP ADAPTER THIN NRPS AdvResv capable OUNI-C ADAPTER OUNI WRAPPER OUNI-C OUNI-N GMPLS OUNI-N E-NNI G 2 MPLS AdvResv capable Grid resource A Transport Network Transport Network Transport Network Transport Network Grid resource B

27 AAA Authorisation Service Architecture Operates as integral part of the general Network Resource Provisioning Service (NRPS) Can be called from NRPS domain controller, or Can drive reservation/provisioning process Incorporates 3 basic generic AAA AuthZ sequences Push, pull, and agent Implements 3 basic AAA AuthZ operational models for complex multidomain network resource provisioning: Polling, Relaying and Agent Supports different and multiple policy decision mechanisms Multiple policy combination and/or mapping Creates and supports dynamic user and resource associations Supports different AuthZ frameworks interoperation Supports different policy enforcement models Using AuthZ tickets and tokens at Service and Control planes Integration of token based enforcement mechanism inside GMPLS control plane using RFC2750 policy data object

28 AAA AuthZ components and basic operational models Agent A R P User Client AAA PDP PEP NRPS NE AAA PDP PEP NRPS NE AAA PDP PEP NRPS NE AAA Dest Host Appli cation Service (AAA) plane Control plane Network plane TVS Token Validation Service DRAM Dynamic Resource Allocation and Mngnt PDP Policy Decision Point PEP Policy Enforcement Point Polling sequence (P) when the user client polls all individual network domains to make a reservation. Relay (R) or hop by hop reservation when the user client contacts only the local network domain/provider and each consecutive domain provides path to the next domain. Agent (A) sequence when the User delegates network provisioning negotiation to the Agent that will do all necessary negotiations with all involved domains

29 Middleware and Applications goals Main goals Adapt and extend Grid middleware to present PHOSPHORUS services to applications Coordinated reservation and allocation of compute, storage, and network resources Integration of MetaScheduling Service (MSS) into UNICORE 6 Allow semantic annotation of resources to improve resource selection Interface with NPRS, Control Plane and AAA activities Adapt and extend network based applications to evaluate and demonstrate the PHOSPHORUS developments in the test bed Make use of network / grid resource reservation services Prepare for deployment in test bed, support tests

30 Middleware and Applications Integration of network reservation services into existing Grid middleware services for user driven or application driven set up of execution environments with dedicated capabilities & performance Compute nodes, storage systems, visualization devices Network resources with defined QoS Integration of applications WISDOM: Wide in silicio docking on Malaria KoDaVis: collaborative, distributed visualization of huge data sets TOPS: Streaming of ultra high resolution data sets over lambda networks DDSS: Distributed Data Storage System INCA: Intelligent Network Caching Architecture Provide application access to PHOSPHORUS services and showcase their benefit via applications

31 Supporting Studies Job Job routing routing & scheduling algorithms Network & resource management Middleware and Applications activity Control Plane activity NRPS activity Job demand models QoS resource scheduling Grid job routing algorithms Physical layer constraints Advance reservations Simulation environment Control Control plane plane design design Architectural issues Integration strategies Recommendations Optical network Advanced control plane Network service plane

32 Supporting Studies results Research oriented results Job demand models Grid job routing algorithms Include physical impairments in routing decision Multi domain job routing QoS aware resource scheduling Support for advance reservations Simulation environment Java, no dependencies, discrete event Modeling network and Grid resources Dynamic OCS path set up and tear down Flexible job models (based on Markov states) GUI to define network topology and traffic models Code available on svn: Mail for login details

33 Test bed & Demonstration activities Objectives: Requirements analysis and design of the test bed Construction of the test bed and configuration of all related software components, middleware and applications Tests of project s developments Demonstration of project s results Real scientific application and real life scenarios used to verify project developments Architecture: Several local test bed supporting different technologies (optical, Lambdas, Ethernet, SDH switching), GRID & HPC resources and applications Lightpaths to interconnect local test beds Current status: Most resources in local test beds available for applications, almost all interconnection in place and NPRSes installed Verification of the test bed Installation of middleware and applications

34 PHOSPHORUS Current test bed data links

35 PHOSPHORUS Multi-domain testbed

36 PHOSPHORUS Dissemination and Contribution to standards Disseminate information concerning the technical developments to NRENs Related projects: Géant2 JRA3 AutoBAHN, NOBEL, e-photon ONE+, RINGRID, BELIEF, Argia/UCLP, G-Lambda, DRAGON/OSCARS, FEDERICA, Coordinate direct contributions to standards (mainly OGF NSI and Glif GNI) Build a collaborative framework for participation to test bed activities from within and external to EU

37 FEDERICA project e-infrastructure What: 7 th FP project in the area Capacities - Research Infrastructures 3.7 M EC contribution, 5.2 M budget 20 partners, 461 Person Months When: 1 st January June 2010 (30 months) Virtualization infrastructure, a Network Factory to provide slices to researchers in Future Internet, where a slice is a mix of network circuits and computing elements. Built using resources (Gb Ethernet circuits) from GÉANT2 and NRENs as contributions to the project. Open to interconnect other Infrastructures Connected to Internet (through NRENs)

38 FEDERICA - Goals Summary In scope Act as a forum and support for researchers/projects on Future Internet. Support of experimental activities to validate theoretical concepts, scenarios, architectures, control and management solutions. Users have full control of their slice Provide on European scale network and system agnostic e-infrastructure to be deployed in phases. Provide its operation, maintenance and ondemand configuration Validate and gather experimental information for the next generation of research networking also through basic tool validation Dissemination and cooperation between NRENs and researchers community Contribution to standards in form of requirements and experience Out of scope Internal extended research, e.g. advanced optical technology Development and support of Grid applications Offer raw computing power Offer transit capacity

39 FEDERICA What and how Scope Create an e-infrastructure for (future) Internet research made of network and computing systems, providing virtualized networks/facilities for end-users, allowing disruptive emulations. Research in multi-(virtual)-domain control, management and monitoring, including user oriented control (IaaS philosophy). Pave the way/create experience for GN3 How Employ a mesh of initially up to 1 Gbps MPLS & GigE circuits from NRENs and GEANT2 (using the GN2+ service) Install virtualization computing nodes (capable of hosting e.g. open source routers) and open API routers and switches in selected FEDERICA PoPs Develop a tool-bench for managing virtual e2e facilities and the infrastructure itself

40 FEDERICA e-infrastructure

41 FEDERICA Main characteristics Suited to core network research, where a network is a (stack of) virtual networks as mix of circuits and computing elements. A slice is as transparent and agnostic from specific technologies as possible. Interdomain communication developments (monitoring, services, topology) also between real and virtual networks Full user control on its slice, including disruptive experimentsopen to host any application/service Open to host researchers hardware and to external connections Internal research activities on control, management and monitoring of virtual infrastructures Basic monitoring data is provided to the user Phase 1 Start with manual configuration of slices (e.g. L2 networks, IP routed networks) and proceed developing automated tools

42 The way to network virtualization. What is IaaS? Virtualization consists of representing a physical device/substrate as a Software entity (P2V) Initially started with PC virtualization (VMware, Virtual Iron, VirtualPC, VirtualBox, and others) Amazon and BlueLock pioneer the IaaS service by renting hardware using proprietary solutions IaaS is equivalent of SaaS for hardware devices Users pay to use shared infrastructures Monthly fees or Pay per use Long term exchanges compared to on-demand services Users control/own the (virtual) infrastructure Don t buy an optical switch, just get some ports of it for a certain period of time!

43 UCLP, Argia and the IaaS Framework Two UCLP research programs were put in place by CANARIE to provide a virtualization solution for optical networks, starting in 2001 UCLP initial goal was to provide e2e to end paths across domains (Cisco was a sponsor of the program) UCLPv2 goals were to create reusable and configurable network blocks UCLPv2 concepts are evolving into many different Physical to Virtual (P2V) products and R&D projects that are built on the IaaS Framework: Argia -> Product for Optical Networks Ether -> R&D for Ethernet and MPLS Networks MANTICORE -> R&D for (logical) IP Networks GRIM -> R&D for Instruments and Sensors RMC GRIM ETHER MANTICORE CHRONOS

44 IaaS framework: Open Source for building IaaS solutions A generalized architecture to the outcome of years of research under the UCLP Research programs funded by CANARIE A software resource architecture to expose the physical devices as infrastructure resources A set of capabilities that can be used to quickly provide functionalities like permissions/security, reservation, lifetime management Libraries and tools to manage persistence or communication with the hardware devices (IaaS Engine -> Driver Architecture) A basic RMC GUI extensible with plugins The enabling technology for upcoming products and R&D initiatives (Phosphorus, FEDERICA, )

45 IaaS Framework Resource Architecture Presentation (SOAP WS) Presentation (REST WS) Presentation (Web GUI) Capability Capability Capability Capability Resource Representations / Service Interfaces (Java) Persistent Information Transient Information Business Logic Persistance Layer IaaS Engine (Driver Architecture) Application Container Security Framework Data Sources DB LDAP File System Physical Devices

46 User Roles Physical Network Administrator (Infrastructure Provider): Owner of the physical infrastructure (e.g. optical switches and links) decides which parts of the infrastructure can be accessed by what user (e.g. User A can access device X, ports X.1, X.2, or ALL users can access device Y). Makes sure the infrastructure is in a good shape (e.g. if a card stops working he must substitute it) APN (or Virtual Network) Administrator (Infrastructure Integrator): Asks for resources to physical network administrators or other APN Administrators (e.g. Get device X from network A, port X.1, and also device Y from network B, ) Integrates all the resources under a single control/management domain (its own domain) Provides services to the End User (e.g. deploys a circuit reservation service) Can handoff the control of the resources it has to other APN Admins. End User: Typical end user, requests a network service (such as bandwidth on demand, circuit reservations, )

47 Resource Trading (I): Direct Export User A Provider 2 Resource List Resource List Resource List Resource List Provider 1 User C User B

48 Resource Trading (II): Brokering sites 48

49 IaaS Framework, Products and Research Projects Architecture Unless specified otherwise the development is being performed in partnership by i2cat, CRC and Inocybe Technologies. WS WS WS WS WS Framework (Open Source) Resource Broker Resources End User Services Connections Resources Argia (Product) (Optical Networks) Resource List and PN Resources Physical Network Resources VLAN Resources Resource List Resources Ether (Product) (Ethernet Networks) IP Network Resources MANTICORE (IP Research Project) Resource Management Centre and User Web Portal. Instrument Resources RCP Device Partition Resources Ethernet Port Resources GRIM (Virtual Instruments Research) WEB (Development) Support Services Device Controller Resources Optical Switch Resources Ethernet Switch Resources Router Resources GRIM Resources TDM Timeslot Resources WDM Resources User Workspace Resources GUI DB Resources...

50 Thanks to Sergi Figuerola and Eduard Grasa from i2cat and both Phosphorus and Federica consortiums Joan Antoni García Espín I2CAT Foundation, Internet and Digital Innovation in Catalonia Phone: (+34) Fax: (+34) Thank you for your attention