CC- NIE/AL2S High Performance Transfers Jim Pepin CTO Clemson University. Broadening the Reach Workshop, Raleigh, NC 09/04/14 09/05/14

Transcription

1 CC- NIE/AL2S High Performance Transfers Jim Pepin CTO Clemson University Broadening the Reach Workshop, Raleigh, NC 09/04/14 09/05/14

2 CC- NIE Next GeneraEon campus network 20 Buildings IntegraEon award Support funds to jumpstart uses. Campus DMZ in place along with 100Gb to ALS2 (InnovaEon PlaTorm) before award Prototype for 10Gb to desktop in future Bridge to our campus network refresh 40+ Gb/s future for building level access SDN as for muleple uses Aggregate for our Condo Cluster, build slices

3

4 100Gb/SDN/40Gb ConnecEon mid 2013 Lessons Learned Clight agreements with Charter helped Be sure to use right opecs (we did but be sure) Length limitaeons on 100Gb LR4s How to build paths in ALS2 world New skill for network staff New "ACL" tab in the AL2S tool (OESS) greatly simplifies path- building Brocade has intereseng SDN/normal issues Packet loss thing in ALS2 also happened in local box. FEC needed.

5 100Gb/SDN/40Gb 40Gb ports on Brocade were late Loaner 10Gb but Force10 40Gb spli`ng to 10 has SDN issues. (just a bother) We are teseng several flavors of switches Some are l2 SDN some both. Sorta confusing Pricing models for next gen campus solueons Brocade is center of Condo cluster as well and DMZ/home for campus deployment. Good and Bad. MTU size parameter between different vendors is a mess.

6 InteresEng Use Cases Joint projects with USC USC Digital Repository Mirror Visual History foundaeon Samqfs Digital repository EMC Atmos 4x10Gb/s EMC Isilon Nx10Gb/s Two Arista switches aggregate to 2x10Gb/s ports (today) Fun and games with tcp window sizes Warner Brothers Archive mirror in progress (new project)

7 InteresEng Use Cases Disaster recovery USC/Clemson has mutual DR agreement In place for last 3-4 years on 2x1Gb/s path over NLR, migraeon to 100Gb/s is done. 3 racks on each site Swap space in each others STK tape library

8 Lets do something real First light Build direct conneceon between DMZ Work with I2 and Cenic required Lots of fussing around to get light to Cenic in LA and I2 in Atlanta Had to then do staec roueng at each and have network/ systems folks speak both locally and remotely USC/Clemson have close relaeonship so it worked.» ACI- REF» MulEple proposals together» Personal history» USC OrangeFS customer

9 Doing it First light tests Perfsonar 9.8Gb/s Packet loss seen (had to fuss a lot to get to 9.8) Found out about AL2S issues later Eventually discovered FEC needs to be on even on Brocades like ours with only 1x100Gb/s NutTCP 10x10Gb/s hosts got to 80ish Gb/s (during the packet loss Eme also on shared Cenic path. OrangeFS throughput 12 server FS at USC 12 Clients Clemson 40ish peak Gb/s with muleple streams per client 75ms ping Emes. Cenic path via NorCal and then back through LA on I2 then southern path. (intereseng story)

10 As the network turns Later on Found out about FEC perfsonar was able to sustain 9.8Gb/s without fussing Earlier work with perfsonar Stanford/USC/Clemson (then others) Stanford not ALS2 yet so good test case for normal 10Gb/s campus (actually more but who is couneng). Never got above 4 Gb/s if you lek the near Stanford area. (guess what they use about 5Gb/s normal use so this makes sense) Build a matrix of speeds between friendly (condo of condo sites) Hard to do so far. Many sites are sell 1Gb/s on their perfsonar nodes

11 Now that it works right More later on Decided redo the first light work but have 4 booths at SC13 with 10Gb/s each (USC, Clemson, I2 and Omnibond) plus 6 hosts at USC on their DMZ at 10Gb/s. Connect to 12 node OrangeFS parallel file system connected at 10Gb/s directly on Clemson DMZ. Very intereseng mule- body problem. Booth setup folks from 4 places, OrangeFS, Networking at USC/Clemson. Then I2/Cenic How being on SCInet comminee helps.

12 Now that it works right More later on Eventually things got setup The shared 100Gb/s factor in scheduling Demos Shared Cenic path between Caltech and USC. Full ALS2 paths. Had to schedule 4 booths for same block of Eme. Southern AL2S path was down for a day.» USC- Clemson path migrated Results Took 3 days to get to best ~40Gb/s max on FS, 30Gb/s steady state 98Gb/s nuttcp Talk about permutaeons of Scinet/USC clients

13 SC13 Test Period

14 SC13 Test Period OrangeFS

15 It s normal to use now More later on Added AL2S to OARnet NIH Utah Wisconsin Vanderbuilt Workflow to NIH for genomics transfers. Low speed over old I2 conneceons (low 100s of Mb/s max)

16 NCBI Performance

17 NCBI Performance InteresEng Tidbits Disk subsystems maner DTN System calls maner for disk I/O String/byte/direct I/O Packet loss deadly to performance 50Mb/s becomes max (do some math around tcp retry games). Firewalls and other packet disrupeon devices are deadly and silent killers. WorkstaEon with 10Gb net is great but disk is not InvesEgaEng limits on single thread disk streaming transfers.

18 What Else SDN in OpenStack environment Enterprise and HPC ACI- REF Hawaii,USC,Utah,Wisconsin,Harvard and Clemson. Final contract details in progress ACI- REF focus, SDN team. All connect to AL2S 100Gb/s network Most have CC- NIE awards Condo Cluster schedule SDN paths via PBSpro. Large data users on campus. Layer 2 or 3 SDN solueons (or both?) How do we integrate into an exiseng network? NSFcloud award- CloudLab with UTAH(lead), Wisconsin and others