ESnet4: Networking for the Future of DOE Science

Transcription

1 ESnet4: Networking for the Future of DOE Science International ICFA Workshop on Digital Divide Issues for Global e-science October 25, 2007 Joe Metzger Senior Engineer Energy Sciences Network Lawrence Berkeley National Laboratory Networking for the Future of Science 1

2 DOE s Office of Science: Enabling Large-Scale Science The Office of Science (SC) is the single largest supporter of basic research in the physical sciences in the United States, providing more than 40 percent of total funding for the Nation s research programs in high-energy physics, nuclear physics, and fusion energy sciences. ( SC funds 25,000 PhDs and PostDocs A primary mission of SC s National Labs is to build and operate very large scientific instruments - particle accelerators, synchrotron light sources, very large supercomputers - that generate massive amounts of data and involve very large, distributed collaborations ESnet - the Energy Sciences Network - is an SC program whose primary mission is to enable the large-scale science of the Office of Science that depends on: Sharing of massive amounts of data Supporting thousands of collaborators world-wide Distributed data processing Distributed data management Distributed simulation, visualization, and computational steering Collaboration with the US and International Research and Education community In addition to the National Labs, ESnet serves much of the rest of DOE, including NNSA about 75, ,000 users in total 2

3 Lawrence Berkeley National Laboratory Pacific Northwest National Laboratory Office of Science US Community Drives ESnet Design for Domestic Connectivity Idaho National Laboratory Ames Laboratory Argonne National Laboratory Fermi National Accelerator Laboratory Brookhaven National Laboratory Stanford Linear Accelerator Center Princeton Plasma Physics Laboratory Lawrence Livermore National Laboratory General Atomics Institutions supported by SC Sandia National Major User Facilities DOE Specific-Mission Laboratories DOE Program-Dedicated Laboratories DOE Multiprogram Laboratories Laboratories Los Alamos National Laboratory Thomas Jefferson National Accelerator Facility Oak Ridge National Laboratory National Renewable Energy Laboratory

4 ESnet as a Large-Scale Science Data Transport Network A few hundred of the 25,000,000 hosts that the Labs connect to every month account for 50% of all ESnet traffic (which is all science data - primarily high energy physics, nuclear physics, and climate at this point) = the R&E source or destination of ESnet s top 100 sites (all R&E) (the DOE Lab destination or source of each flow is not shown)

5 What ESnet Is A large-scale IP network built on a national circuit infrastructure with high-speed connections to all major US and international research and education (R&E) networks An organization of 30 professionals structured for the service An operating entity with an FY06 budget of $26.6M A tier 1 ISP (direct peerings will all major networks) The primary DOE network provider Provides production Internet service to all of the major DOE Labs* and most other DOE sites Based on DOE Lab populations, it is estimated that between 50, ,000 users depend on ESnet for global Internet access additionally, each year more than 18,000 non-doe researchers from universities, other government agencies, and private industry use Office of Science facilities * PNNL supplements its ESnet service with commercial service 5

6 Japan (SINet) Australia (AARNet) Canada (CA*net4 Taiwan (TANet2) Singaren KAREN/REANNZ ODN Japan Telecom America NLR-Packetnet Abilene/I2 CA*net4 MREN France StarTap GLORIAD Taiwan (TANet2, (Russia, China) ASCC) Korea (Kreonet2 SINet (Japan) Russia (BINP) CERN (USLHCnet DOE+CERN funded) GÉANT - France, Germany, Italy, UK, etc JGI LBNL NERSC SLAC SNV SDN AU AU SEA SDSC general Internet2 NASA Ames PacWave NLR PacificWave NLR PNNL IARC LIGO LNOC LLNL SNLL SNV PAIX-PA Equinix MAE-W KAREN / REANNZ Internet2 SINGAREN ODN Japan Telecom America YUCCA MT GA AMPATH (S. America) CLARA CUDI INEEL BECHTEL-NV ALB NREL UNM Denver LANL SNLA DOE-ALB Allied Signal ELP PANTEX Starlight USLHCNet Internet2 NLR FNAL ANL AMES ARM CHI-SL Equinix KCP CHI OSTI NOAA NETL OSC GTN NNSA ORAU ORNL ATL Lab DC Offices DC SRS NYC NSF/IRNC funded MAN LAN Internet2 PPPL JLAB MIT BNL MAE-E Equinix MAXGPoP general AMPATH (S. America) CLARA ESnet Provides Global High-Speed Internet Connectivity for DOE Facilities and Collaborators (2007) ~45 end sites: Office Of Science Sponsored Other Sponsored R&E commercial peering points Specific R&E network peers network s ESnet core hubs Other R&E peering points high-speed peering points with Internet2 IP Internet2 SNV International (high speed) 10 Gb/s SDN core 10G/s IP core MAN rings ( 10 G/s) Lab supplied links OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less

7 ESnet s Place in U. S. and International Science ESnet, Internet2/Abilene, and National Lambda Rail (NLR) provide most of the nation s transit networking for basic science Abilene provides national transit networking for most of the US universities by interconnecting the regional networks (mostly via the GigaPoPs) ESnet provides national transit networking and ISP service for the DOE Labs NLR provides various science-specific and network R&D circuits GÉANT plays a role in Europe similar to Abilene and ESnet in the US it interconnects the European National Research and Education Networks (NRENs), to which the European R&E sites connect A GÉANT operated, NSF funded link currently carries all non-lhc ESnet traffic to Europe, and this is a significant fraction of all ESnet traffic 7

8 Operating Science Mission Critical Infrastructure ESnet is a visible and critical piece of DOE science infrastructure if ESnet fails,10s of thousands of DOE and University users know it within minutes if not seconds Requires high reliability and high operational security in the systems that are integral to the operation and management of the network Secure and redundant mail and Web systems are central to the operation and security of ESnet trouble tickets are by engineering communication by engineering database interfaces are via Web Secure network access to Hub routers Backup secure telephone modem access to Hub equipment 24x7 help desk and 24x7 on-call network engineer trouble@es.net (end-to-end problem resolution) 8

9 A Changing Science Environment is the Key Driver of the Next Generation ESnet Large-scale collaborative science big facilities, massive data, thousands of collaborators is now a significant aspect of the Office of Science ( SC ) program SC science community is almost equally split between Labs and universities SC facilities have users worldwide Very large international (non-us) facilities (e.g. LHC and ITER) and international collaborators are now a key element of SC science Distributed systems for data analysis, simulations, instrument operation, etc., are essential and are now common (in fact dominate data analysis that now generates 50% of all ESnet traffic) 9

10 Planning and Building ESnet4 Requirements are primary drivers for ESnet science focused Sources of Requirements 1. Office of Science (SC) Program Managers The Program Offices Requirements Workshops 2007» Basic Energy Science» Biological & Environmental Research 2008» Fusion Energy Science» Nuclear Physics 2009» High Energy Physics» Advance Scientific Computing Research 2. Direct gathering through interaction with science users of the network Example case studies (from 2005/2006 update) Magnetic Fusion Large Hadron Collider (LHC) Climate Modeling Spallation Neutron Source 3. Observation of the network 10

11 Science Networking Requirements Aggregation Summary Science Drivers Science Areas / Facilities End2End Reliability Connectivity Today End2End Band width 5 years End2End Band width Traffic Characteristics Network Services Magnetic Fusion Energy % (Impossible without full redundancy) DOE sites US Universities Industry 200+ Mbps 1 Gbps Bulk data Remote control Guaranteed bandwidth Guaranteed QoS Deadline scheduling NERSC and ACLF - DOE sites US Universities International Other ASCR supercomputers 10 Gbps 20 to 40 Gbps Bulk data Remote control Remote file system sharing Guaranteed bandwidth Guaranteed QoS Deadline Scheduling PKI / Grid NLCF - DOE sites US Universities Industry Backbone Band width parity Backbone band width parity Bulk data Remote file system sharing International Nuclear Physics (RHIC) - DOE sites US Universities International 12 Gbps 70 Gbps Bulk data Guaranteed bandwidth PKI / Grid Spallation Neutron Source High (24x7 operation) DOE sites 640 Mbps 2 Gbps Bulk data

12 Science Network Requirements Aggregation Summary Science Drivers Science Areas / Facilities End2End Reliability Connectivity Today End2End Band width 5 years End2End Band width Traffic Characteristics Network Services Advanced Light Source - DOE sites US Universities Industry 1 TB/day 300 Mbps 5 TB/day 1.5 Gbps Bulk data Remote control Guaranteed bandwidth PKI / Grid Bioinformatics - DOE sites US Universities 625 Mbps 12.5 Gbps in two years 250 Gbps Bulk data Remote control Point-tomultipoint Guaranteed bandwidth High-speed multicast Chemistry / Combustion - DOE sites US Universities Industry - 10s of Gigabits per second Bulk data Guaranteed bandwidth PKI / Grid Climate Science - DOE sites US Universities International - 5 PB per year 5 Gbps Bulk data Remote control Guaranteed bandwidth PKI / Grid Immediate Requirements and Drivers High Energy Physics (LHC) % (Less than 4 hrs/year) US Tier1 (FNAL, BNL) US Tier2 (Universities) International (Europe, Canada) 10 Gbps 60 to 80 Gbps (30-40 Gbps per US Tier1) Bulk data Coupled data analysis processes Guaranteed bandwidth Traffic isolation PKI / Grid

13 Are These Estimates Realistic? YES! Megabytes/sec of data traffic FNAL outbound CMS traffic for 4 months, to Sept. 1, 2007 Max= 8.9 Gb/s (1064 MBy/s of data), Average = 4.1 Gb/s (493 MBy/s of data) Destinations: Gigabits/sec of network traffic 13

14 Large-Scale Data Analysis Systems (Typified by the LHC) have Several Characteristics that Result in Requirements for the Network and its Services The systems are data intensive and high-performance, typically moving terabytes a day for months at a time The system are high duty-cycle, operating most of the day for months at a time in order to meet the requirements for data movement The systems are widely distributed typically spread over continental or inter-continental distances Such systems depend on network performance and availability, but these characteristics cannot be taken for granted, even in well run networks, when the multi-domain network path is considered The applications must be able to get guarantees from the network that there is adequate bandwidth to accomplish the task at hand The applications must be able to get information from the network that allows graceful failure and auto-recovery and adaptation to unexpected network conditions that are short of outright failure These requirements are generally true for systems with widely distributed components to be reliable and consistent in performing the sustained, complex tasks of large-scale science This slide drawn from [ICFA SCIC]

15 Enabling Large-Scale Science These requirements are generally true for systems with widely distributed components to be reliable and consistent in performing the sustained, complex tasks of large-scale science Networks must provide communication capability that is service-oriented: configurable, schedulable, predictable, reliable, and informative and the network and its services must be scalable 15

16 3. Observed Evolution of Historical ESnet Traffic Patterns Terabytes / month ESnet total traffic passed 2 Petabytes/mo about mid-april, 2007 top 100 sites to site workflows 0 Jan, 00 Jan, 01 Jan, 02 Jan, 03 Jan, 04 Jan, 05 Jan, 06 site to site workflow data not available Jan, 07 ESnet Monthly Accepted Traffic, January, 2000 June, 2007 ESnet is currently transporting more than1 petabyte (1000 terabytes) per month More than 50% of the traffic is now generated by the top 100 sites large-scale science dominates all ESnet traffic

17 ESnet Traffic has Increased by 10X Every 47 Months, on Average, Since Nov., TBy/mo. Apr., PBy/mo Jul., TBy/mo. Terabytes / month Aug., MBy/mo. Oct., TBy/mo. 57 months Jan, 90 Jan, 91 Jan, 92 Jan, 93 Jan, 94 Jan, 95 Jan, 96 Jan, 97 Jan, 98 Jan, 99 Jan, 00 Jan, 01 Jan, 02 Jan, 03 Jan, 04 Jan, 05 Jan, 06 Jan, months 53 months 38 months Log Plot of ESnet Monthly Accepted Traffic, January, 1990 June, 2007

18 What is the High-Level View of all ESnet Traffic ESnet Inter-Sector Traffic Summary, May % 4 % Commercial DOE sites ~10% 41 % 16 % ESnet 54 % 17 % Research and Education Inter-Lab traffic 20 % 38 % International (almost entirely R&E sites) Traffic notes more than 90% of all traffic is Office of Science Traffic coming into ESnet = Green Traffic leaving ESnet = Blue Traffic between ESnet sites 18

19 Requirements from Network Utilization Observation In 4 years, we can expect a 10x increase in traffic over current levels without the addition of production LHC traffic Nominal average load on busiest backbone links is ~1.5 Gbps today In 4 years that figure will be ~15 Gbps based on current trends Measurements of this type are science-agnostic It doesn t matter who the users are, the traffic load is increasing exponentially Predictions based on this sort of forward projection tend to be conservative estimates of future requirements because they cannot predict new uses 19

20 Requirements from Traffic Flow Observations Most of ESnet science traffic has a source or sink outside of ESnet Drives requirement for high-bandwidth peering Reliability and bandwidth requirements demand that peering be redundant Multiple 10 Gbps peerings today, must be able to add more bandwidth flexibly and cost-effectively Bandwidth and service guarantees must traverse R&E peerings Collaboration with other R&E networks on a common framework is critical Seamless fabric Large-scale science is now the dominant user of the network Satisfying the demands of large-scale science traffic into the future will require a purpose-built, scalable architecture Traffic patterns are different than commodity Internet 20

21 Summary of All Requirements To-Date Requirements from SC Programs: 1A) Provide consulting on system / application network tuning Requirements from science case studies: 2A) Build the ESnet core up to 100 Gb/s within 5 years 2B) Deploy network to accommodate LHC collaborator footprint 2C) Implement network to provide for LHC data path loadings 2D) Provide the network as a service-oriented capability Requirements from observing traffic growth and change trends in the network: 3A) Provide 15 Gb/s core within four years and 150 Gb/s core within eight years 3B) Provide a rich diversity and high bandwidth for R&E peerings 3C) Economically accommodate a very large volume of circuit-like traffic 21

22 Estimated Aggregate Link Loadings, Portland Seattle Boise Existing site supplied circuits unlabeled links are 10 Gb/s 13 9 Boston Sunnyvale Salt Lake City Denver KC Chicago Indianapolis Clev. Pitts. NYC Philadelphia Wash DC 2.5 LA San Diego (1) Albuq. Tulsa Nashville Atlanta OC48 Raleigh El Paso Jacksonville ESnet IP switch/router hubs ESnet IP switch only hubs ESnet SDN switch hubs Houston Layer 1 optical nodes at eventual ESnet Points of Presence Layer 1 optical nodes not currently in ESnet plans Lab site 2.5 Committed bandwidth, Gb/s Baton Rouge 2.5 ESnet IP core (1λ) ESnet Science Data Network core ESnet SDN core, NLR links Lab supplied link LHC related link MAN link International IP Connections 22

23 Estimated Aggregate Link Loadings, Portland Seattle (>1 λ) unlabeled links are 10 Gb/s labeled links are in Gb/s Sunnyvale Boise Salt Lake City (16) 50 4λ Denver 5λ KC Chicago Indianapolis 50 Clev Pitts. Boston 50 NYC Philadelphia 50 Wash. DC 10 LA San Diego ESnet IP switch/router hubs ESnet IP switch only hubs ESnet SDN switch hubs El Paso Albuq. Tulsa Houston Layer 1 optical nodes at eventual ESnet Points of Presence Layer 1 optical nodes not currently in ESnet plans Lab site Nashville Baton Rouge 30 Atlanta 40 OC Committed bandwidth, Gb/s 40 link capacity, Gb/s Raleigh Jacksonville 20 ESnet IP core (1λ) ESnet Science Data Network core ESnet SDN core, NLR links (existing) Lab supplied link LHC related link MAN link International IP Connections 23

24 ESnet4 - The Response to the Requirements I) A new network architecture and implementation strategy Provide two networks: IP and circuit-oriented Science Data Network Reduces cost of handling high bandwidth data flows Highly capable routers are not necessary when every packet goes to the same place Use lower cost (factor of 5x) switches to relatively route the packets Rich and diverse network topology for flexible management and high reliability Dual connectivity at every level for all large-scale science sources and sinks A partnership with the US research and education community to build a shared, large-scale, R&E managed optical infrastructure a scalable approach to adding bandwidth to the network dynamic allocation and management of optical circuits II) Development and deployment of a virtual circuit service Develop the service cooperatively with the networks that are intermediate between DOE Labs and major collaborators to ensure and-to-end interoperability III) Develop and deploy service-oriented, user accessable network monitoring systems IV) Provide consulting on system / application network performance tuning 24

25 Next Generation ESnet: I) Architecture and Configuration Main architectural elements and the rationale for each element 1) A High-reliability IP core (e.g. the current ESnet core) to address General science requirements Lab operational requirements Backup for the SDN core Vehicle for science services Full service IP routers 2) Metropolitan Area Network (MAN) rings to provide Dual site connectivity for reliability Much higher site-to-core bandwidth Support for both production IP and circuit-based traffic Multiply connecting the SDN and IP cores 2a) Loops off of the backbone rings to provide For dual site connections where MANs are not practical 3) A Science Data Network (SDN) core for Provisioned, guaranteed bandwidth circuits to support large, high-speed science data flows Very high total bandwidth Multiply connecting MAN rings for protection against hub failure Alternate path for production IP traffic Less expensive router/switches Initial configuration targeted at LHC, which is also the first step to the general configuration that will address all SC requirements Can meet other unknown bandwidth requirements by adding lambdas 25

26 ESnet Target Architecture: IP WAN+Science Data Network WAN+Metro Area Rings international connections international connections international connections 1625 miles / 2545 km international connections Sunnyvale Seattle LA San Diego IP core hubs SDN hubs Primary DOE Labs Possible hubs Denver Metropolitan Area Rings international connections Loop off Backbone SDN Core IP Core Albuquerque 2700 miles / 4300 km Chicago Atlanta Cleveland New York Washington DC international connections Gbps circuits Production IP Wide Area Network Science Data Network WAN Metropolitan Area Networks or backbone loops for Lab access International connections 26

27 ESnet Metropolitan Area Network Ring Architecture for High Reliability Sites US LHCnet switch SDN core west US LHCnet switch SDN core switch SDN core switch SDN core east ESnet production IP core hub IP core west T320 IP core router IP core east ESnet SDN core hub ESnet IP core hub Large Science Site MAN site switch MAN fiber ring: 2-4 x 10 Gbps channels provisioned initially, with expansion capacity to ESnet MAN switch Independent port card supporting multiple 10 Gb/s line interfaces ESnet managed λ / circuit services ESnet switch ESnet production IP service ESnet managed virtual circuit services tunneled through the IP backbone Site SDN circuits to site systems Site router MAN site switch Site LAN Virtual Circuit to Site Site edge router Virtual Circuits to Site Site gateway router 27

28 ESnet4 Internet2 has partnered with Level 3 Communications Co. and Infinera Corp. for a dedicated optical fiber infrastructure with a national footprint and a rich topology - the Internet2 Network The fiber will be provisioned with Infinera Dense Wave Division Multiplexing equipment that uses an advanced, integrated opticalelectrical design Level 3 will maintain the fiber and the DWDM equipment The DWDM equipment will initially be provisioned to provide10 optical circuits (lambdas - λs) across the entire fiber footprint (40/80 λs is max.) ESnet has partnered with Internet2 to: Share the optical infrastructure Develop new circuit-oriented network services Explore mechanisms that could be used for the ESnet Network Operations Center (NOC) and the Internet2/Indiana University NOC to back each other up for disaster recovery purposes 28

29 ESnet4 ESnet is building its next generation IP network and its new circuit-oriented Science Data Network primarily on the Internet2 circuits (λs) that are dedicated to ESnet, together with a few National Lambda Rail and other circuits ESnet will provision and operate its own routing and switching hardware that is installed in various commercial telecom hubs around the country, as it has done for the past 20 years ESnet s peering relationships with the commercial Internet, various US research and education networks, and numerous international networks will continue and evolve as they have for the past 20 years 29

30 ESnet4 ESnet4 will also involve an expansion of the multi- 10Gb/s Metropolitan Area Rings in the San Francisco Bay Area, Chicago, Long Island, Newport News (VA/Washington, DC area), and Atlanta provide multiple, independent connections for ESnet sites to the ESnet core network expandable Several 10Gb/s links provided by the Labs that will be used to establish multiple, independent connections to the ESnet core currently PNNL and ORNL 30

31 Internet2 and ESnet Optical Node ESnet metro-area networks support devices: measurement out-of-band access monitoring security SDN core switch ESnet IP core M320 Network Testbed Implemented as a Optical Overlay dynamically allocated and routed waves (future) T640 grooming Cienadevice CoreDirector optical interface to R&E Regional Nets Internet2 support devices: measurement out-of-band access monitoring. various equipment and experimental control plane management systems BMM BMM fiber west fiber east Infinera DTN Internet2/Level3 National Optical Infrastructure fiber north/south 31

32 ESnet 3 Backbone as of January 1, Seattle Sunnyvale Chicago New York City Washington DC San Diego Albuquerque Atlanta El Paso Future ESnet Hub ESnet Hub 10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) MAN rings ( 10 G/s) Lab supplied links

33 ESnet 4 Backbone Target September 30, 2007 Seattle Boise Clev. Boston New York City Sunnyvale Denver Chicago Washington DC Los Angeles San Diego Albuquerque Kansas City Nashville Atlanta Future ESnet Hub ESnet Hub El Paso Houston 10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings ( 10 G/s) Lab supplied links 33

34 ESnet4 Metro Area Rings, 2007 Configurations Seattle West Chicago MAN 600 W. Chicago Long Island MAN USLHCNet (28) Portland (8) Starlight USLHCNet 32 AoA, NYC BNL Sunnyvale (29) (23) LA San Diego (24) Boise Boston (9) (7) Chicago (11) Clev. (10) FNAL ANL NYC Ames (32) (13) (25) Denver Salt KC Philadelphia (15) (26) Lake Pitts. San Francisco City Wash DC (21) (0) (22) (30) Bay Area MAN Raleigh JGI Tulsa Nashville Albuq. OC48 (1(3)) (3) LBNL (4) (1) Atlanta Newport News - Elite SLAC (19) NERSC (20) (2) Jacksonville Wash., El Paso DC (17) (6) MATP ESnet IP switch/router hubs LLNL Atlanta MAN ORNL ESnet IP switch only hubs SNLL Nashville ESnet SDN switch hubs All circuits are 10Gb/s. 56 Marietta Wash., DC (SOX) Layer 1 optical nodes at eventual ESnet Points of Presence Layer 1 optical nodes not currently in ESnet plans Lab site (16) Houston (12) Indianapolis 180 Peachtree (14) (27) ESnet IP core JLab ESnet Science Data ELITE Network core ESnet SDN core, NLR links (existing) Lab supplied ODU link LHC related link MAN link International IP Connections 34

35 ESnet4 IP + SDN, 2008 Configuration (Estimated) Seattle Portland (? λ) Sunnyvale 1λ 1λ LA UCSD San Diego 1λ 2λ 2λ Boise Salt Lake City (24) 1λ 1λ 2λ 1λ Denver Albuq. 2λ Tulsa KC 1λ StarLight 2λ Indianapolis 2λ Nashville Chicago 2λ 2λ Atlanta Clev. 1λ 2λ OC48 2λ Pitts. Boston 2λ NYC Philadelphia 2λ Wash. DC Raleigh ESnet IP router hubs El Paso ESnet IP internal switch hubs ESnet SDN OSCARS/MPLS switch hubs ESnet SDN internal switch hubs 1λ Houston Layer 1 optical nodes at eventual ESnet Points of Presence Layer 1 optical nodes not currently in ESnet plans Lab site Corrected Map Baton Rouge 1λ Jacksonville ESnet IP network (Internet2 circuits) ESnet Science Data Network (Internet2) ESnet SDN (NLR circuits) Lab supplied link LHC related link MAN link International IP Connections Status indefinite / not installed 35

36 Estimated ESnet Configuration (Some of the circuits may be allocated dynamically from shared a pool.) Seattle (28) Portland (8) (? λ) Sunnyvale (29) (23) 2λ 2λ 3λ (32) 3λ Boise (7) Salt Lake City (16) 3λ 2λ Denver (0) (15) 3λ (22) KC Boston Chicago (9) 3λ (11) Clev. 3λ (10) NYC (13) 3λ (25) Philadelphia 2λ 3λ (26) (21) Wash. DC 3λ (12) 2λ Indianapolis (14) (27) Pitts. (30) San Diego LA 2λ (24) 2λ 2λ Tulsa Albuq. (1) 1λ (19) (20) El Paso (17) 2λ ESnet IP switch/router hubs ESnet IP switch only hubs ESnet SDN switch hubs Houston Layer 1 optical nodes at eventual ESnet Points of Presence Layer 1 optical nodes not currently in ESnet plans Lab site (5) Nashville Baton Rouge 2λ Atlanta (3) 2λ (6) (20) OC48 3λ (2) (4) Raleigh Jacksonville ESnet IP core ESnet Science Data Network core ESnet SDN core, NLR links (existing) Lab supplied link LHC related link MAN link International IP Connections Internet2 circuit number

37 ESnet4 IP + SDN, 2011 Configuration Seattle (28) Portland (8) (>1 λ) Sunnyvale (29) (23) 4λ 4λ 5λ (32) Boise (7) Salt Lake City (16) 5λ 4λ Denver (0) (15) 5λ (22) KC Boston Chicago (9) 5λ (11) Clev. 5λ (10) NYC (13) 5λ (25) Philadelphia 4λ 5λ (26) (21) Wash. DC 5λ (12) 3λ Indianapolis (14) (27) Pitts. (30) San Diego LA 4λ (24) 4λ 4λ Tulsa Albuq. (1) 3λ (19) (20) El Paso (17) 4λ ESnet IP switch/router hubs ESnet IP switch only hubs ESnet SDN switch hubs Houston Layer 1 optical nodes at eventual ESnet Points of Presence Layer 1 optical nodes not currently in ESnet plans Lab site (5) Nashville Baton Rouge 3λ Atlanta (3) 4λ (6) (20) OC48 5λ (2) (4) Raleigh Jacksonville ESnet IP core (1λ) ESnet Science Data Network core ESnet SDN core, NLR links (existing) Lab supplied link LHC related link MAN link International IP Connections Internet2 circuit number 37

38 1625 miles / 2545 km Asia-Pacific Australia Seattle Sunnyvale Australia LA San Diego Boise Canada (CANARIE) South America (AMPATH) ESnet4 Planed Configuration Gbps in , Gbps in Canada Denver Asia Pacific Science Data Network Core IP Core El Paso Kansas City Albuquerque (CANARIE) GLORIAD (Russia and China) Tulsa Houston Asia- Pacific Chicago Atlanta Cleveland CERN (30 Gbps) Jacksonville CERN (30 Gbps) Europe (GÉANT) Boston New York Washington DC South America (AMPATH) IP core hubs SDN (switch) hubs Primary DOE Labs High speed cross-connects with Ineternet2/Abilene Possible hubs Core network fiber path is ~ 14,000 miles / 24,000 km 2700 miles / 4300 km Production IP core (10Gbps) SDN core ( Gbps) MANs (20-60 Gbps) or backbone loops for site access International connections

39 New Network Service: Virtual Circuits Guaranteed bandwidth service User specified bandwidth - requested and managed in a Web Services framework Traffic isolation and traffic engineering Provides for high-performance, non-standard transport mechanisms that cannot co-exist with commodity TCP-based transport Enables the engineering of explicit paths to meet specific requirements e.g. bypass congested links, using lower bandwidth, lower latency paths End-to-end (cross-domain) connections between Labs and collaborating institutions Secure connections Provides end-to-end connections between Labs and collaborator institutions The circuits are secure to the edges of the network (the site boundary) because they are managed by the control plane of the network which is isolated from the general traffic Reduced cost of handling high bandwidth data flows Highly capable routers are not necessary when every packet goes to the same place Use lower cost (factor of 5x) switches to relatively route the packets 39

40 Virtual Circuit Service Functional Requirements Support user/application VC reservation requests Source and destination of the VC Bandwidth, latency, start time, and duration of the VC Traffic characteristics (e.g. flow specs) to identify traffic designated for the VC Manage allocations of scarce, shared resources Authentication to prevent unauthorized access to this service Authorization to enforce policy on reservation/provisioning Gathering of usage data for accounting Provide virtual circuit setup and teardown mechanisms and security Widely adopted and standard protocols (such as MPLS and GMPLS) are well understood within a single domain Cross domain interoperability is the subject of ongoing, collaborative development secure and-to-end connection setup is provided by the network control plane accommodate heterogeneous circuit abstraction (e.g.. MPLS, GMPLS, VLANs, VCAT/LCAS) Enable the claiming of reservations Traffic destined for the VC must be differentiated from regular traffic Enforce usage limits Per VC admission control polices usage, which in turn facilitates guaranteed bandwidth Consistent per-hop QoS throughout the network for transport predictability 40

41 Environment of Science is Inherently Multi-Domain End points will be at independent institutions campuses or research institutes - that are served by ESnet, Abilene, GÉANT, and their regional networks Complex inter-domain issues typical circuit will involve five or more domains - of necessity this involves collaboration with other networks For example, a connection between FNAL and DESY involves five domains, traverses four countries, and crosses seven time zones FNAL (AS3152) [US] GEANT (AS20965) [Europe] DESY (AS1754) [Germany] ESnet (AS293) [US] DFN (AS680) [Germany] 41

42 OSCARS [6] Inter-domain virtual circuit reservation involves brokers in each domain that present a uniform view of the different domain circuit management mechanisms (e.g. OSCARS in ESnet) (Diagram by Tom Lehman, ISI) 42

43 ESnet Virtual Circuit Service Roadmap Dedicated virtual circuits Dynamic virtual circuit allocation Generalized MPLS (GMPLS) Initial production service Full production service Interoperability between GMPLS circuits, VLANs, and MPLS circuits (layer 1-3) Interoperability between VLANs and MPLS circuits (layer 2 & 3) Dynamic provisioning of Multi-Protocol Label Switching (MPLS) circuits in IP nets (layer 3) and in VLANs for Ethernets (layer 2) 43

44 Monitoring as a Service-Oriented Communications Service perfsonar is a community effort to define network management data exchange protocols, and standardized measurement data gathering and archiving Path performance monitoring is an example of a perfsonar application provide users/applications with the end-to-end, multi-domain traffic and bandwidth availability provide real-time performance such as path utilization and/or packet drop Multi-domain path performance monitoring tools are in development based on perfsonar protocols and infrastructure One example Traceroute Visualizer [TrViz] has been deployed in about 10 R&E networks in the US and Europe that have deployed at least some of the required perfsonar measurement archives to support the tool

45 Federated Trust Services Support for Large-Scale Collaboration Remote, multi-institutional, identity authentication is critical for distributed, collaborative science in order to permit sharing widely distributed computing and data resources, and other Grid services Public Key Infrastructure (PKI) is used to formalize the existing web of trust within science collaborations and to extend that trust into cyber space The function, form, and policy of the ESnet trust services are driven entirely by the requirements of the science community and by direct input from the science community International scope trust agreements that encompass many organizations are crucial for large-scale collaborations ESnet has lead in negotiating and managing the cross-site, crossorganization, and international trust relationships to provide policies that are tailored for collaborative science This service, together with the associated ESnet PKI service, is the basis of the routine sharing of HEP Grid-based computing resources between US and Europe 45

46 ESnet Conferencing Service (ECS) A highly successful ESnet Science Service that provides audio, video, and data teleconferencing service to support human collaboration of DOE science Seamless voice, video, and data teleconferencing is important for geographically dispersed scientific collaborators Provides the central scheduling essential for global collaborations ESnet serves more than a thousand DOE researchers and collaborators worldwide H.323 (IP) videoconferences (4000 port hours per month and rising) audio conferencing (2500 port hours per month) (constant) data conferencing (150 port hours per month) Web-based, automated registration and scheduling for all of these services Very cost effective (saves the Labs a lot of money) 46

47 Summary ESnet is currently satisfying its mission by enabling SC science that is dependant on networking and distributed, large-scale collaboration: The performance of ESnet over the past year has been excellent, with only minimal unscheduled down time. The reliability of the core infrastructure is excellent. Availability for users is also excellent - DOE 2005 annual review of LBL ESnet has put considerable effort into gathering requirements from the DOE science community, and has a forward-looking plan and expertise to meet the five-year SC requirements A Lehman review of ESnet (Feb, 2006) has strongly endorsed the plan presented here 47

48 References 1. High Performance Network Planning Workshop, August Science Case Studies Update, 2006 (contact 3. DOE Science Networking Roadmap Meeting, June DOE Workshop on Ultra High-Speed Transport Protocols and Network Provisioning for Large-Scale Science Applications, April Science Case for Large Scale Simulation, June Workshop on the Road Map for the Revitalization of High End Computing, June (public report) 7. ASCR Strategic Planning Workshop, July Planning Workshops-Office of Science Data-Management Strategy, March & May For more information contact Chin Guok Also see - [LHC/CMS] [ICFA SCIC] Networking for High Energy Physics. International Committee for Future Accelerators (ICFA), Standing Committee on Inter-Regional Connectivity (SCIC), Professor Harvey Newman, Caltech, Chairperson. - [E2EMON] Geant2 E2E Monitoring System developed and operated by JRA4/WI3, with implementation done at DFN [TrViz] ESnet PerfSONAR Traceroute Visualizer 48