SNAG: SDN-managed Network Architecture for GridFTP Transfers

Transcription

1 SNAG: SDN-managed Network Architecture for GridFTP Transfers Deepak Nadig Anantha, Zhe Zhang, Byrav Ramamurthy, Brian Bockelman, Garhan Attebury and David Swanson Dept. of Computer Science & Engineering, University of Nebraska-Lincoln 3rd Intl. Workshop on Innovating the Network for Data-Intensive Science (INDIS), SC 16, Salt Lake City, Utah. 13 th November 2016 This material is based upon work supported by the National Science Foundation under Grant No

2 OUTLINE Introduction SNAG Approach Integration Architecture Implementation Results Conclusions and Future Work 2

3 INTRODUCTION Software Defined Networks (SDN) are transforming research and education (R&E) networks We are interested in: Classifying data flows from scientific projects A policy-driven approach to network management & security 3

4 SCIENTIFIC UNL Holland Computing Center UNL Supports well-known scientific projects such as: Compact Muon Solenoid (CMS) Laser Interferometer Gravitational-Wave Observatory (LIGO) Large datasets with projects consuming significant storage and networking resources Our Work: Ability to apply policies to these projects at the experiment level Ability to differentiate CMS traffic over LIGO 4

5 Globus GridFTP Protocol for cluster and grid environments Enables large-volume data transfers GridFTP maximizes data transfer throughput Creates multiple TCP streams per transfer Overcomes TCP limitations for high-latency, high-bandwidth WANs CMS and LIGO both use GridFTP for data transfers Cost: Fairness 5

6 PROBLEM Associate policies based on application layer properties with network layer flow-level information. GridFTP breaks TCP fairness Need to differentiate high-priority transfers GridFTP control channel is encrypted Traffic cannot be classified by sniffing control channel 6

7 Solution: SNAG Our SDN-managed Network Architecture for GridFTP Transfers SNAG Architecture: Integrates SDN capabilities with GridFTP Provides network monitoring and management capabilities Differentiates GridFTP transfers: To/from various sources, and By owners 7

8 SNAG ARCHITECTURE MODULE REST REST Southbound API 8

9 REST REST SNAG COMPONENTS Southbound API Three main components namely: i. GridFTP Servers, Globus XIO Module and GridFTP-HDFS plugin ii. SDN controller (ONOS), SNAG App and GridFTP App iii. Monitoring System (InfluxDB + Grafana) Communication between components using RESTful APIs L3, L4 info (src/dst IPs, port-pairs) Application layer info (file transfers, direction etc.) 9

10 GridFTP XIO Callout + SNAG Client 5 1 Data Path Control Path 26 GridFTP Server FLOW_MOD 4 3 REST POST GridFTP XIO Callout: SRC_IP DST_IP, DST_PORT TRANSFER_STATUS FILENAME Monitor 4 REST POST ONOS SDN Controller 10

11 11 NETWORK TOPOLOGY

12 RESULTS (1) Classified users based on four types of traffic: a) CMS PhEDEx - CMS production data movement Maps user initiated transfers to the PhEDEx data placement system Consists of large physics datasets (.root files) to/from sites b) CMS Analysis - Represents analysis transfers associated with users' jobs (typically mapped to an individual user) c) CMSProd - transfers associated with CMS production workflows d) LCG Admin - transfers associated with SAM (Site Availability Monitoring) 12

13 RESULTS (2) Classification of CMS Data Transfers by Users Each measurement shows the #users at every 15 min intervals Data normalized over 65 users 13

14 RESULTS (3) Classification of #Transfer Streams per User type Represents the number of active TCP streams by user type 14

15 CONCLUSIONS SNAG builds network layer views based on application layer properties Resulting monitoring views cannot be achieved through the traditional approaches SNAG accounts for resource usage and provides insights into opportunistic sharing (such as LIGO) 15

16 FUTURE WORK Transition from passive monitoring to active network management Proactively change network flows based on monitoring information Traffic prioritization and QoS for CMS/LIGO Transfers Optimize capacity and increase network utilization Integration with other data flows such as XROOTD Insights into access patterns (Site-level info) 16

17 Thank You Deepak Nadig Anantha 17