Extending an InfiniBand Fabric around the world

Transcription

1 Extending an InfiniBand Fabric around the world Yves Poppe A*STAR Computational Resource Centre Singapore APAN 39 Cloud WG Fukuoka, Kyushu, Japan, March 1-6 th 2015

2 Please, maximize my effective throughput Connect HPC resources at Fusionopolis with the storage and genomics pipeline in the Biopolis Matrix Building Pg 2

3 Pg 3

4 Tests started with Mellanox METRO-X early 2013 and were followed up with trials using the Obsidian Strategics Longbow C400. Today the sites are connected with 2x40gbps connections running native InfiniBand and reaching approx. 98.4% of maximum theoretical possible throughput. Pg 4

5 The big picture: NSCC Singapore s National SuperComputer Centre Funding (MTI) and co-funding (A*STAR, NUS, NTU) approved in November 2014 NSCC Calls for new 1-3+ PetaFLOP Supercomputer Recurrent investment every 3 to 5 years Pooling up and high tier compute resources at A*STAR and IHLs Co-investment from primary stakeholders Science, Technology and Research Network (STAR-N) High bandwidth network to connect distributed compute resources Provides high speed access to users, both public and private, anywhere Supports transfer of large data sets both locally and internationally Builds local and international connectivity Pg 5

6 NSCC and STAR-N timetables NSCC Tender opens on 20th of January 2015 Tender closes on 14 th of April 2015 Facility opens to users: October 2015 STAR-N Singapore infininet connecting A*STAR, NUS and NTU International access point to infininet at Global Switch Partner negotiations with Europe, Japan and the USA in collaboration with Singaren. Timetables to coincide with NSCC readinesss. Major intercontinental demonstration planned at SC15 in November 2015 in Austin TX. Pg 6

7 The Ultimate InfiniBand Jailbreak HPC s and Infiniband were suffocating within the Data Center walls. Range extenders like the Mellanox MetroX gave Infiniband and consequently HPC s and data centres themselves more breathing room and ways to expand on metro level. Obsidian Strategics took the final step: It took Data Centre walls away completely. InfiniBand connections can cross continents and circle the globe. The ultimate step: BGFC makes InfiniBand routeable and opens the possibility to permeate the globe giving rise to an InfiniCortex. Internet gave us classrooms without walls. InfiniCortex will give us supercomputing without walls Pg 7

8 Pg 8 proved itself in a spectacular way

9 A*STAR s vision: InfiniCortex a Supercomputer of Supercomputers Professor Tan Tin Wee and Dr. Marek Michalewicz proposed to demonstrate something totally new, never done before, Very High speed transcontinental transmission of native Long Distance Infiniband between High Performance Computing (HPC) centres continents apart and have them operate as one, tackling the biggest computational challenges and opening a possible avenue to exascale supercomputing where the most vexing problem is power and heat generation. This is not cloud computing, this is not Grid Pg 9

10 The four elements that made this possible Very high speed transmission as made possible by ACA100 Asia connects America at 100gbps, challenge issued by Yves Poppe, then at Tata Communications, at APAN 37 in Bandung, Indonesia. InfiniBand over trans-pacific distances Made possible with Obsidian Strategics InfiniBand range extenders. Galaxy of Supercomputers Supercomputer interconnect topology and graph theory work by Y.Deng, M.Michalewicz and L.Orlowski. InfiniBand subnetting using the BGFC protocol and the new Obsidian Crossbow InfiniBand router. Application layer File transfer optimization based on the development of Dsync+ for simple file transfers all the way to complex work flows with ADIOS (Adaptable I/O System) developed by Dr. Scott Klasky and his team at Oak Ridge National Laboratories. Pg 10

11 InfiniBand range extender and router Longbow Device Crossbow Device Developed by Obsidian Strategics based in Edmonton, Canada Crossbow plus Longbows give rise to Galaxy and open the door to an InfiniNet Pg 11

12 Pg 12

13 Galaxy of Supercomputers Supercomputers located at different geolocations connected into a Super-Graph or Nodes of Super-Network Supercomputers may have arbitrary interconnect topologies Galaxy is based on a topology with small diameter and lowest possible link number. In terms of graph representation it is an embedding of graphs representing Supercomputers topologies into a graph representing the Galaxy topology. Pg 13

14 Pg 14

15 Making the vision a reality Testing within Singapore completed using dark fibre between two A*CRC sites and also with the National University of Singapore using SingAREN s new SLIX over 80km. Convincing results led us to deploy two 40gbps InfiniBand connections between our Biopolis and Fusionopolis sites. InfiniBand over Ethernet testing with Tokyo Institute of Technology s Tsubame-KFC successfully completed using SingAREN, APAN and JGN-X. InfiniBand over IP testing completed with the NCI (National Computational Infrastructure) at the Australian National Unversity in Canberra using existing SingAREN, APAN and AARnet infrastructure. 10gbps dedicated link between Singapore and the USA for layer 2 native InfiniBand testing with ORNL and others starting end October. Rather spectacular results of the 100gbps trial and demos between Singapore and New Orleans at SC14 Pg 15

16 Pg 16 SC14: New Orleans Nov 16th-20 th 2014

17 Galaxy14 Network Topology SC14 100gbps linking A*STAR in Singapore to the A*STAR booth at SC14 in New Orleans via Singaren, the Tata Communications transpacific cables TGN-IA and TGN-P to Seattle, Century Link to New Orleans and Scinet on the SC14 conference grounds. Pg 17

18 Proving the point Pg 18 Extract from the presentation prepared by Jakub Chrzeszczyk and Andrew Howard, NCI, Australia

19 Pg 19

20 Pg 20

21 Pg 21

22 Pg 22

23 The power of Collaboration Pg 23

24 Taking into consideration the 20 to 30 fold increase in effective throughput, the infiniband links correspond to what 2 to 3 terabit per second links between A*STAR, NUS and NTU could carry using traditional FTP and 10 to 15Tbps between Biopolis and Fusionopolis. Pg 24

25 Growing a global infininet Japan USA Australia Géant Pioneer Janet... cesnet Renater Poznan Warsaw. Wales. Prague. Reims Lille. Pg 25

26 Pg 26 Demonstration of intercontinental 10gbps infiniband : Singapore-USA-Poland A*STAR CRC Singaren A*STAR 10gbps ESnet Géant Pionier Poznan PSNC

27 A*STAR CRC Singaren A*STAR nx10gbps internet2 ANA200 nx10gbps FCCN Géant Reims Renater Pionier Lille Prague CESNET Warsaw Poznan Pg 27 Evolving configuration certainly not final

28 SC15 Austin Texas Nov th 2015 Circumnavigate the globe with the Singapore Europe 10gbps connection and the Singapore and Japan-- US West Coast 100gbps connections in place USA Europe Japan Singapore Pg 28

29 Pg 29 Next steps

30 Pg 30

31 Long Distance InfiniBand: a game changer for R&E networking Over the past year the long distance infiniband approach to global collaborative HPC was presented to the global NREN community including TERENA, APAN, GLIF, Géant, ASREN and internet2. Long distance infiniband was successfully tested within Singapore, with Japan, Australia and the USA, in collaboration with our NREN partners, culminating in a 100gbps transpacific demonstration at SC14 which produced spectacular results. Close collaboration with the HPC community provides the NREN s a unique opportunity to regain the lead in innovation and to clearly differentiate themselves from commercial network service offerings. Pg 31

32 Long Distance InfiniBand: a game changer for HPC In Singapore, A*STAR CRC in collaboration with Singaren, is deploying an infininet to cater to the needs of the new NSCC. High speed infiniband connectivity to Europe, Japan, Australia and the USA is required to connect with participating overseas HPC partners. The HPC community globally is faced with an unabated exponential growth of data generated. Current NREN internetworking capacity is already insufficient just to meet the anticipated needs of genomics data interchange. The next frontier is exascale computing, where we demonstrated the viability of a distributed and collaborative approach which has the added benefits of solving the huge power requirements, data replication and disaster recovery. Global HPC and global R&E networking can only live and thrive in symbiosis Pg 32

33 Thank You Creativity requires the courage to let go of certainties. Erich Fromm Pg 33