Supercomputer Networking via IPv6 Xing Li, Congxiao Bao, Wusheng Zhang 2018-08-08
Outline Sunway TaihuLight Supercomputer CERNET and CERNET2 Challenges Solution Performance Q&A Remarks 2
National Supercomputing Center in Wuxi (NSCC-Wuxi) http://www.nsccwx.cn 125 PetaFlops No.1 among the TOP500 List Published at ISC High Performance, June 20, 2016. 3
Sunway TaihuLight: System Overview The first China homegrown many-core processor: SW26010 260 cores per chip divided into four core groups (CGs) 3 Tflops The first system in the world that provides over 100 Pflops performance with over 10 million cores theoretical peak 125 Pflops, 2.5 times improvement over before LINPACK performance 93 Pflops, 3 times improvement over before High energy-efficiency of the overall system 6.05 Gflops/Watt, 3 to 6 times improvement over before Co-operated by Jiangsu Gov., Wuxi Gov., andtsinghua University 4
10M-core system 163,840 processes 65 threads racks chips core-groups cores total number of cores 1,024x 40 x 260-core Chip Rack System 5
Wind Resource Estimation (National Climate Center, located at Beijing) 4020 simulation cases Data generation rate: ~1.5TB/hour Need stable and high bandwidth network connection 80m_wind(m/s) T2(K) 6
Dark Matter Particle Explorer (WuKong) Purple Mountain Observatory Chinese Academy of Sciences, located at Nanjing Observation data need to be transfered from Nanjing to Wuxi. Processing result feed back to Nanjing. ~19 hundred million particles per year. 7
CERNET and CERNET2 IPv4 IPv6 100G 100G TaihuLight connected to CERNET/CERNET2 via 100G 8
Challenges High performance Single flow TCP BGP multi-homing Both to academic and commodity networks University firewall Performance bottleneck Traffic based charging Distinguish the traffic Security VPN IPv4-only applications Upgrade IPv6 takes time BGP optimization MPLS SDN? Try something differently 9
Solution Infrastructure Wuxi supercomputer center: dual stack CERNET2: 100G IPv6 Campus network: dual stack Building block Stateless IPv4/IPv6 translation technology 10
Stateless translation (RFC7915) 11
Address mapping (RFC6052) 12
Translation example 13
Double translation (4aaS) IPv4 prefix1 IPv6 prefix1 IPv6 prefix2 IPv4 prefix2 IPv4 IVI core IPv6 IVI edge IPv4 IPv6 IPv4 IPv4 IPv6 IPv4 Decomposition: User IPv4 prefix Network IPv6 prefix 14
Features The network can by fine-toned using IPv6 prefixes More flexible and controllable than IPv4 BGP More cost effective than MPLS The network O&M is very simple End to end address transparency, no encapsulation is required No need to upgrade the application at this stage The application can still be IPv4 The different charging model can be applied via difference IPv6 prefix 20Gbyte/user/month global Internet access Unlimited limited domain (Wuxi) IPv6 prefix 15
Topology IVI-B IPv4 RFC1918 Wuxi TUNET IPv4 IVI-A CERNET2 IPv6 IVI-B IPv4 RFC1918 IPv4 IPv6 IPv4 IPv4 IPv6 IPv4 IPv6 100G 100G 10G 16
Performance testing max= 9.42362G max= 3.04592G max= 3.1011G max= 9.41128G max= 7.69666G TCP single flow 17
Q&A Why not use IPv4 VPN? Performance, not transparent Why not use IPv6-only? Need to upgrade the applications Why not use IPv4 over IPv6? Network O&M need decapsulation Why not use stateful NAT64? Cannot support bidirectional initiated communication Why not use 464xlat? Cannot support bidirectional initiated communication Why not use stateless single translation? Need to upgrade the applications 18
Traffic control 2001:da8:ff3a:c8fb:300:: -> 202.38.101.26 70M (blue) IPv6 null route 202.38.101.26 -> 2001:da8:ff3a:cbfb:200:: (w/ rate-limit) 202.38.101.26 -> 2001:da8:ff3a:cbfb:300:: (w/o rate-limit) Beginning of ratelimit End of ratelimit 19
Next steps Link more end sites 20+ International sites Scheduling mechanism (central control) More supercomputer centers Wuxi: Sunway TaihuLight(1) Guangzhou Tianjin Shenzhen Jinan Changsha Qingdao etc 20
International collaboration Sunway TaihuLight Supercomputer is an open facility operating by Tsinghua University Team It can provide services for the international partners Contact: xing@cernet.edu.cn congxiao@cernet.edu.cn zws@tsinghua.edu.cn 21