PIRE ExoGENI ENVRI preparation for Big Data science

Transcription

1 System and Network Engineering MSc Research project PIRE ExoGENI ENVRI preparation for Big Data science Stavros Konstantaras, Ioannis Grafis February 5, 2014

2 Background Big Data science Huge amount of data Many sources Data Movement (DM) is very important Described by 5V s (Volume, Velocity, Variety, Variability and Value) Software Defined Networking (SDN) Separate control plane from data plane Single entity controls the network Forwarding intelligence relies on programmers 2

3 Research questions The main research question is the following: To what degree can the performance of the data movement protocols be optimized by using Software Defined Networking technology? The main research question includes the following subquestions: What network level problems exist which limit the performance of the data movement protocols? How can SDN eliminate these problems? 3

4 Outline Theory part Problem analysis Solution profiles Experimental part Prototyping HIDE (Hybrid Intelligent Data Enhancer) Scenarios and Results Conclusion 4

5 Data Movement Application problems Application Positives Negatives Network limits -Open source -High scalability -Difficult to deploy -Network speed limit: -High reliability (13 Gbps for TCP GridFTP -Option to resume transfers version) (Globus) that are stopped because of -Decrease window size failures for every loss packet and resend the packet bbftp (NASA) FDT (CERN) -Open source -High scalability -High reliability -Multi-stream TCP -Easy to deploy -Resume file transfer session -Open source -Runs on all major platforms (Java application) -Multi-stream TCP -Resume file transfer session -Transfer only files, not directories -Little industry adoption -Little documentation -Little industry adoption -Little documentation -Network speed limit (4.5 Gbps) -Application is not aware for the topology and the path that data flows -Most of times the speed of transferring data is limited due to network traffic 5

6 Performance problem? Does the application perform well? Do nothing Is it a network problem? Is it a corrupted/ broken link? Examine / Improve the Application Fix the link Is it a busy link? Proceed to the Decision tree to select a QoS solution Examine the other entities of the network 6

7 Available technologies Traffic monitoring Deep Packet Inspection (DPI) Inspect client/server interfaces Inspect flow counters Flow management Port level Socket level (IP address and TCP port) Network Controllability Commands to the controller (API) Commands to the switches 7

8 Do nothing Can not provide Application level solution Replace the Application Extend the source code without network input Full Access level Can we modify the source code? Can we monitor the traffic? Extend the source code Traffic monitoring Inspect client/server interfaces Can we use the Data Application for that? DPI Do we have access to the Application? How can we improve the Application s performance? Some Access level Use of an API to communicate with Build a separate component to solve Traffic monitoring at Interfaces Traffic monitoring Flow counters Can we use the network for that? Can we control network to boost performance? Do we have access to Controller? How can we make use of the Controller? Full Access level Extend the source code Flow management sockets Decision tree ports Can not provide network level solution. We need to grand some access rights Can not use SDN to solve the problem Some Access level Use an API to communicate Network control Commands to switches Flow management Excluded options Parallel options Network control Do nothing Commands to Controller Traffic monito ring Ports Sockets Inspect flow counters 8

9 Can we use the Data Application for that? How can we improve the Application s performance? Can we use the network for that? Can we control network to boost performance? Do we have access to Controller? How can we make use of the Controller? Full Access level Extend the source code Decision tree Can not provide network level solution. We need to grand some access rights Can not use SDN to solve the problem Some Access level Use an API to communicate Flow management Excluded options Parallel options Do nothing Traffic monito ring Ports Sockets Traffic monitoring Flow management Network control Network control Inspect flow counters DPI Flow counters sockets ports Commands to switches Commands to Controller 9

10 Can not provide Application level solution Can we use the Data Application for that? Do we have access to the Application? How can we improve the Application s performance? Can we use the network for that? Decision tree Excluded options Parallel options Do nothing Replace the Application Extend the source code without network input Full Access level Can we modify the source code? Can we monitor the traffic? Extend the source code Traffic monitoring Some Access level Use of an API to communicate with Build a separate component to solve Traffic monitoring at Interfaces Inspect client/server interfaces 10

11 Solution development profiles Requirements Application level Programmer Network Programmer (API) Network Programmer (full) Hybrid Programming Develop at Application level Develop at Network level Make use of SDN Technology Access to the Application SOME Access to the Controller SOME SOME Network topology knowledge Network status knowledge SOME Traffic monitor using DPI Traffic monitor on flow level Traffic monitor at Interfaces Flow management Network controllability SOME 11

12 How can we improve the Application s performance? Solution tracks Application approach Our approach Do nothing Can not provide Application level solution Replace the Application Extend the source code without network input Full Access level Can we modify the source code? Can we monitor the traffic? Extend the source code Traffic monitoring Inspect client/server interfaces Can we use the Data Application for that? DPI Do we have access to the Application? Some Access level Use of an API to communicate with Build a separate component to solve Traffic monitoring at Interfaces Traffic monitoring Flow counters Can we use the network for that? Can we control network to boost performance? Do we have access to Controller? How can we make use of the Controller? Full Access level Extend the source code Flow management sockets ports Can not provide network level solution. We need to grand some access rights Can not use SDN to solve the problem Some Access level Use an API to communicate Network control Commands to switches Network approach (Full) Network approach (API) Flow management Do nothing Network control Commands to Controller Traffic monito ring Ports Sockets Inspect flow counters 12

13 Controller-Application relationship Controller Dependent Our approach Network level Network level (API) Application level Application Independent Application Dependent Controller Independent 13

14 HIDE component Floodlight OpenFlow Controller HIDE server2 Iperf FDT server1 SW2 FDT SW1 SW3 SW4 Iperf client2 Path1 Path2 1 Gbps links 100Mbps links 10Mbps links client1

15 15

16 COMPONENT FDT HIDE overhead 6s 5s 5s 5s t0 t1 t2 t3 t4 new connection 1 st FDT output 2 nd FDT output 3 rd FDT output 4 th FDT output time 6s 2s 8s 5s t0 t1 t2 t2 t3 t4 new connection discover QoS problem Ignored output confirm that problem solved time Δt = FDT + HIDE = 16s send commands to change path 16

17 Scenarios Scenario 1 Transferring files via Path1 with and without interfering traffic for getting reference points Scenario 2 Transferring files via Path1 with interfering traffic and component enabled Scenario 3 Interfering traffic change path every 30s in order to stress HIDE for longer period 17

18 Time in seconds Scenario results FDT performance on transfering different files FDT ideal Scenario 1 HIDE disable Scenario 2 HIDE enable Scenario 2 HIDE disabled Scenario 2 HIDE enabled , , , , File sizes in Megabytes 18

19 Time in seconds Total transfer time Total time for transfering three different files Scenario 1 Scenario 2 HIDE disabled Scenario 2 HIDE enabled Scenario 3 HIDE disabled Scenario 3 HIDE enabled 125 MB GB GB

20 Speed in Mbps Representative sample of Scenario Component disabled Time in seconds Component enabled File size 1.25Gb Data points every 5s 20

21 Discussion Adequate level of abstraction and portability Using SDN to enhance data movement Intelligence based on real time input Lower bound of reaction time depended on FDT server Topology knowledge should be requested from controller 21

22 Conclusion Data Movement Applications can gain benefits from SDN Through the mentioned degrees of solving the QoS problem we touched one and it was successful ExoGENI is well designed environment to deploy topologies and perform experiments 22

23 Future work Reduce reaction time (highly depended on FDT) Improve intelligence (get topology knowledge from controller) Investigate a prediction algorithm (avoid network overload) 23

24 Thank you 24