CERNET2 Updates. Zhonghui Li CERNET / Tsinghua University 29 Mar 2018

Transcription

1 CERNET2 Updates Zhonghui Li CERNET / Tsinghua University 29 Mar 2018

2 Outline CERNET2 100G Upgrade IPv6 Memcache DDoS Attack to CNGI-6IX

3 Outline CERNET2 100G Upgrade IPv6 Memcache DDoS Attack to CNGI-6IX

4 CERNET2 Overview Native IPv6 national backbone Since PoPs in 20 cities 10G/2.5G backbone link ~600 University, R&E institute 5M users

5 CERNET2 Backbone Traffic

6 100G Upgrade Plan of CERNET2 Backbone Objective Before upgrade After upgrade Protocol Pure IPv6 Pure IPv6 PoP Province/city Backbone link 10G/2.5G 100G/10G Total bandwidth 127.5G 2950G IPv6 user 5M 10M Role Experimental Productive

7 DWDM Optical Transmission Network 沈阳 哈尔滨 呼和浩特 北京 天津 大连 长春 太原 石家庄 银川兰州 西宁 西安 郑州 济南 青岛 成都 昆明 桂林 重庆 贵阳 南宁 长沙 武汉 广州 合肥 南昌 厦门 南京杭州 福州 上海 100G 10G 海口 深圳

8 New Backbone Topology 郑州西安合肥广州沈阳天津重庆济南厦门大连长春长沙南京哈尔滨北京武汉上海杭州成都兰州深圳北邮北航呼和浩特太原银川西宁乌鲁木齐拉萨青岛石家庄南昌福州贵阳昆明桂林南宁海口北大复旦同济 100G 10G 10G 一类节点二类节点 CNGI-6IX

9 IVI Deployment (IPv6 Transition) IPv4 Internet CERNET IPv4 Transition Transition Transition Transition IPv6 Internet 6IX CERNET2 IPv6 Pure IPv6 Campus Network Dual Stack Campus Network

10 SAVI Deployment (IPv6 Cyber Security) Source Address Validation Improvement Verifiable User Auth Unification ID 40bit Time Stamp Encr Last 64 bit of IPv6 Address Source Address Authentication Address Ex-Header Network Entity Identification Privacy Decr ID Management/Audit Retrieve TP-Layer Payload IPv6 Packet Retrievable

11 SAVI Deployment (IPv6 Cyber Security) Deployment in CERNET2 campus network

12 Technical consideration on customer network s connection to CERNET and CERNET2 Consideration Separated IPv4 & IPv6 backbone CERNET (AS4538): Pure IPv4 (after the 100G upgrade of CERNET2) CERNET2 (AS23910): Pure IPv6 Most of customer networks prefer single connection to CERNET & CERNET2 PoP (same location) Difference in routing design between CERNET and CERNET2 CERNET: PoP router and access router in same domain (AS4538) CERNET2: PoP router and access router in different domain All PoP routers belong to AS23910 Access router in each node has its own AS number (AS AS24372) Experimental role of CERNET2

13 Option 1 (Dual-stack Access Router) Pro: Easy to implement (only IPv4/v6 static route is needed), and IPv4 & IPv6 access is transparent to customer networks Hierarchical & Robust (L3 access via access router) Easy for monitoring and management (via L3 access router) Con: Higher CAP-EX (access router) Routing complexity (individual AS for each node) Less flexible (routing, multihoming)

14 Option 2 (Local Exchange Point) Pro: Lower CAP-EX (local switch) Simplicity (L2 connection) More flexible (routing, multihoming, private peering) Con: Complicate implementation (both for NOC and customer networks), and IPv4 & IPv6 access is not transparent to customer networks Nonhierarchical & Vulnerable (L2 access) Difficult to monitor, manage and control (Less function on L2 switch)

15 Our choice Based on current situation for most of CERNET & CERNET2 customer networks, we choose Option 1 (Dual-stack Access Router) as preferred implementation scheme for CERNET2 Upgrade Project To simplify the operation of CERNET2, the individual AS number for edge/access router in each PoP might be withdrawn gradually Option 2 might be considered for some nodes in the future, when The customer network border routers become more powerful (full/part routing table, security, QoS etc.) Most of customer networks in the node have their own AS numbers Most of customer networks are more familiar with BGP Increasing demand on multi-homing from customer networks Increasing demand on private peering among customer networks

16 Progress of 100G Upgrade Delivery and activation of 41 PoP core routers (Huawei NE40e X8A) Done 100G x 28 Almost done except for 3 intracity links Beijing (THU) BUAA Shanghai (SJTU) Fudan Univ. Shanghai (SJTU) Tongji Univ. 10G x 16 Done 40Urumuqi 34XiNing 20LanZhou A1 NODE 24BUAA 32Hohhot 33YinChuan 5XiAn 8ChengDu 35KunMing 36GuiYang 100GE DWDM 100GE Fiber 23BUPT 22PKU 41Lhasa 31TaiYuan 17ChongQing 38GuiLin 37NanNing CNGI-6IX BJ 1BeiJing 27ShiJiaZhuang 12ZhengZhou 2WuHan 14ChangSha 4GuangZhou 6ShenYang 9TianJin 11HeFei 28NanChang 13ChangChun 10JiNan 18DaLian 7NanJing 30FuZhou 19Harbin 29Tsingtao 25FuDan 3ShangHai 26TongJi 15HangZhou 16XiaMen B NODE A1 8 main nodes 10GE DWDM 10GE L2VPN 39HaiKou 21ShenZhen

17 Next Step Complete the implementation of new pure IPv6 backbone of CERNET2 Implemented and running in parallel with current CERNET2 IPv6 backbone Migrate customer traffic to new 100G CERNET2 backbone Migrate CERNET2 edge/access routers to the new backbone Tendering for new CERNET2 edge/access router for each PoP 100G/10G uplink to core router 100G capable for particular customer, e.g. National Supercomputing Centers Sufficient 10G/1G ports for CERNET/CERNET2 customer access link New tech & service under consideration and evaluation SRv6 EVPN DDoS mitigation etc.

18 Outline CERNET2 100G Upgrade IPv6 Memcache DDoS Attack to CNGI-6IX

19 IPv6 Memcache DDoS Attack to CNGI-6IX The first large-scale IPv6 DDoS attack to CERNET/CNGI-6IX infrastructure Start time of DDoS attack 23:20 BJT on 1 st Mar 2018 Attack target IPv6 address on CNGI-6IX BJ-LA 10G link Source of attack traffic One of IPv6 commodity ISP peering in LA Impact Continuous and serious traffic congestion on BJ-LA 10G link

20 Action taken Applied firewall filter for UDP port on LA router interface connected with that ISP Requested the ISP to block the DDoS attack traffic inside its backbone

21 How did the attack traffic go through CNGI-6IX backbone

22 How did the attack traffic go through CNGI-6IX backbone

23 How did the attack traffic go through CNGI-6IX backbone

24 How did the attack traffic go through CNGI-6IX backbone

25 How did the attack traffic go through CNGI-6IX backbone Add firewall filter for UDP 11211

26 Lessons and thinking Necessity of online real-time IPv6 flow analysis system with abnormal traffic detection and alerting IPv6 DDoS mitigation measures Traffic scrubbing: high cost and not work for this case Attack target is infrastructure not customer network BGP Flowspec: cooperation from external peer Is it the time for us to pay more attention to IPv6 network security?

27 Thanks!