THOUGHTS ON SDN IN DATA INTENSIVE SCIENCE APPLICATIONS Artur Barczyk/Caltech Internet2 Technology Exchange Indianapolis, October 30 th, 2014 October 29, 2014 Artur.Barczyk@cern.ch 1
HEP context - for this talk LHC experiments now moving 100+ PB per year Main driver of R&E network bandwidth utilization over the past decade LHC restart in Spring 2015, expect traffic to grow ATLAS data transfer rates, monthly average 10 k Transfer Throughput 20 0 9-0 1-0 1 0 0 :0 0 t o 20 14-0 2-0 8 0 0 :0 0 UT C LHCONE VRF October 29, 2014 Artur.Barczyk@cern.ch 2 T hroughput (MB/s) 7.5k 5k 2.5k 0 k 20 0 9-0 1 20 0 9-0 2 20 0 9-0 3 20 0 9-0 4 Destinations 20 0 9-0 5 20 0 9-0 6 20 0 9-0 7 20 0 9-0 8 20 0 9-0 9 20 0 9-10 20 0 9-11 20 0 9-12 20 10-0 1 20 10-0 2 20 10-0 3 20 10-0 4 20 10-0 5 20 10-0 6 20 10-0 7 20 10-0 8 20 10-0 9 20 10-10 20 10-11 20 10-12 20 11-0 1 20 11-0 2 20 11-0 3 20 11-0 4 20 11-0 5 20 11-0 6 20 11-0 7 20 11-0 8 20 11-0 9 20 11-10 20 11-11 20 11-12 20 12-0 1 2012-0 2 2012-0 3 20 12-0 4 2012-0 5 2012-0 6 2012-0 7 2012-0 8 2012-0 9 20 12-10 20 12-11 20 12-12 20 13-0 1 2013-0 2 2013-0 3 20 13-0 4 2013-0 5 2013-0 6 2013-0 7 2013-0 8 2013-0 9 20 13-10 20 13-11 20 13-12 20 14-0 1 CA CERN DE ES FR IT ND NL RU TW UK US
New data movement and access patterns Change in Computing Models towards more flexibility (source/destination pairings) more dynamic (popularity based caching) remote access Different flow characteristics and patterns Data production Data processing ; cached vs remote access One constant: Will remain massively distributed, WAN performance will be key to success! Tier-0 CAF Tier-1 Tier-1 Tier-1 T1 Tier-1 T1 Tier-1 T1 Tier-1 T1 Tier-1 T2 T2 T2 T2 T2 October 29, 2014 Artur.Barczyk@cern.ch 3
New capability through SDN example Leveraging provider diversity in LHCONE: many NRENs, providing multiple paths with multiple Standard network forwards all packets for a given subnet on the same path Multipath configurations not trivial Especially when coupled with multiple administrative domains SDN approach allows to build a flexible yet robust and deterministic forwarding scheme Similarities with SDX (general) concept Slides from LHCONE workshop February 2013 October 29, 2014 Artur.Barczyk@cern.ch 4
OpenFlow + Dynamic Circuits Provision capacity between OpenFlow switches based on real-time requirements Approach in the OLiMPS project: flow management in OpenFlow controller triggers circuit requests to OSCARS controller I.e. create a topology optimizing the load distribution in the network Future: couple with data transfer application October 29, 2014 Artur.Barczyk@cern.ch 5
Another SDN Value Proposition Fact: Not all flows/users/application_requirements are the same A (the?) key value proposition of SDN in R&D networks: it allows to provide not only differentiated, but tailored services; customized to the particular need of a research community Can be done with minimal amount of effort (once the system is built) HEP: Example of a vision could be to differentiate between data production and transfer flows ( elephants ) and analysis data flows in remote access scenarios ( mice ), e.g. old model : not the same sites (Tier1/2/3 definition!) new model : roles defined on temporal varying requirements not the same capacity and access latency requirements not the same resiliency requirements October 29, 2014 Artur.Barczyk@cern.ch 6
How can HEP profit from SDN? At the computing sites: Network Virtualization Scalability, flexibility, robustness, security (think ScienceDMZ here) In the WAN flexible services But also through A programmatic interface (TBD) between application and the network Tight integration; direct feedback loop -> reactive system Increased predictability; reduced distribution tails Dynamic workflow optimization Increased efficiency October 29, 2014 Artur.Barczyk@cern.ch 7
SDN and Bandwidth on Demand BoD aka Dynamic Circuit Networks, aka Lightpaths, aka Is a form of Software Defined Networking Circuits are configured by a controller Typically implemented as a (central) Domain Controller A proof-of-concept experiment is being built by LHCONE in collaboration with GLIF AutoGOLE efforts Interconnect a small number of sites initially Interface into CMS and ATLAS data movement and workflow management But really, it will be a first step towards a tight integration between network and applications through broader SDN concepts October 29, 2014 Artur.Barczyk@cern.ch 8
Automated GOLE Fabric October 29, 2014 Artur.Barczyk@cern.ch 9
Conclusions For data intensive science applications, SDN has the potential to enable Better application performance More determinism in workflows Resource optimization In order to harness the full potential of SDN, we need a good application-network interface multi-domain capability support in the R&E networks end-to-end coordinated effort between the service providers and the scientists developing their applications October 29, 2014 Artur.Barczyk@cern.ch 10
THANK YOU! Artur.Barczyk@cern.ch October 29, 2014 Artur.Barczyk@cern.ch 11