The Automatic On-boarding System Used in China Mobile's NFV/SDN Network Fu Qiao, China Mobile, fuqiao@chinamobile.com Wang Xu, China Mobile, wangxuyjy@chinamobile.com Liang Qi, Huawei, liangqi1@huawei.com Gao Litao, Windriver, Litao.Gao@windriver.com
Index Best Practice: Telecom Integrated Cloud(TIC) in China Mobile Automatic On-Boarding system: reducing TIC integration from weeks to minutes Demo
NFV(Network Function Virtualization) NFV Network Function Virtualization has been widely experimented and deployed around the world. Operators are continuously moving traditional Network services to the NFV Cloud, in order to reduce its cost and accelerate service launching. Through the years, NFV has developed from concept and framework to field trail and real deployment. Cloud vepc vims vcpe vcdn vbras
China Mobile s Trip to NFV/SDN China Mobile announced its future network as NovoNet Based on NFV/SDN, NovoNet is a new network with shared virtual resources and agile network structure. NovoNet provides network users with scalable resources and open API to utilize China Mobile began its vims trail in 4 provinces since 2015, NB-IoT trail in 6 provinces in 2016 China Mobile began its NovoNet Experiment network since 2016. This is a grand new network built for experiments and testing of NFV/SDN. The network currently includes 4 provinces. Multiple services, including NB-IoT, vbras, scpe, E-BoD are tested in this network 2015.1 China Mobile Open NFV Testlab 2015.7 Open NFV lab membership announcement 2016.12 Launching NovoNet Experiment Network 2014 2015 2016 2017 2018 2014.9 Founding member for OPNFV 2015.7 NovoNet white paper 2016.1 Open NFV lab membership announcement 2017.1 Founding member of Open-O & ONAP 2017.3 NovoNet Experiment Network Phase 1 2017.9 NovoNet Experiment Network Phase 2 4
China Mobile Future Network Architecture China Mobile structures its NFV Cloud into TICs (Telecom Integrated Cloud) TIC is a standard unit, with unified hardware and software infrastructure. Multiple VNF services are deployed in TICs TICs are categorized into Core TIC and Edge TIC, in which core TIC mainly supports control plane services, and Edge TIC is for forwarding plane services ONAP Core TIC Core TIC Core TIC 50~100 SDN-Ctrl Edge TIC Edge TIC 1000~3000 Edge TIC Edge TIC Edge TIC AP 1,000,000 enodeb Mobile Access AP Residential User PTN Enterprise User
TIC: Standard Unit for NFV Deployment TIC is a standard unit for NFV Deployment Limited number of design template, including NFVI, VIM, and NFVO Unified Hardware model Standard network design Standard Network Strict network plane partition, including Management net, service net, and storage net. SG design OSS VNFs Hypervisor NFVO VNFM OpenStack Unified Hardware Unified hardware design with limited number of models suitable for different category of services. Hareware SDN-Ctrl Design Template 4 TIC design templates to carry VNF services of different kind, including: Template for Control plane services, stressing on compute capability Template for Data plane services, stressing on data forwarding capability Template for Edge services, stressing on low cost and light weight Template for Storage services, stressing on storage capability Key Features for TIC: Standard unit, Template-based, Easy to copy 6
OpenStack SDN-Ctrl CSCF ATS OMU USCDB OMU PCRF OMU UGW CG MSE RIAD Distributed Storage SBC BSF AP 7 Zoom in to each TIC TIC is constructed with custom hardware and virtual software platform 1.7 TIC 整体视图 OpenStack is chosen to use as VIM NFVO/VNFM/EMS are deployed in Core TIC on VMs SDN is used within each TIC for network deployment TIC Manage EOR FW Service/Storage EOR /SDN GW Manage TOR Service TOR Storage TOR PNF(not controlled by SDN) CP HA DP HA Storage HA DMZ NFVO+ EMS......... VNFM PNF Hypervisor Hardware
China Mobile NovoNet Experiment Network Since 2016, China Mobile is building its Novonet Experiment Network in 4 Provinces. 7 DC sites are included in the network, 14 TICs are constructed VNFs include vepc for NB-IoT, vbras, vcpe, and E-BoD 9 virtual infrastructure vendors, 5 VNF vendors, 3 orchestrator vendors, 4 SDN vendors, are included in the network In Phase 1, most of the integration and testing work are done manually, which cost us more than 9 months to construct the 14 TICs. In phase 2, we plan to utilize the automatic system so as to significantly reduce complexity and man s labor Beijing Shanghai Zhejiang Guangdong
TIC Integration Workflow TIC Integration Workflow Manual Operation: 1 weeks 3 weeks 1 weeks 1 weeks Service Testing Planning OpenStack Deployment Service Deployment Environment Configuration NFVI Testing Automatic system 30 Minutes 60 Minutes 30 Minutes 10 Minut 9
Automatic On-boarding system: Weeks to Minutes Automatic On-boarding system is crucial for future network integration and large scale deployment The integration of TIC includes components from multiple vendors with complicated process. Manual integration leads to huge workload, and is almost impossible to repeat. Different from traditional network made of bunch of hardware, future software infrastructure on-boarding of TIC requires functional, performance and availability testing before launching An automatic integration and on-boarding system provide standard process and tools for integration and on-boarding testing for multiple vendors. Making the whole TIC easy to copy and re-integrated Multiple Vendors Impossible for manual work Software integration Standard on-boarding test Rapid construction 01 02 03 04 05 NFV Host os installation TIC infrastructure deployment VNF deployment On Boarding Test Hand over NFV 10
System overview Provide agile selection of automatic on-boarding tools, including integration, testing and configuration Provide pipeline mechanism for work flow management Easy to use for integrators from End Users Integrator UI Workflow Management Test Result Test Task management Integration management Failure Detection & Root cause analysis Hardware Configuration Developer UI Automatic Testing Automatic Integration Automatic Configuration Code development Git Gerrit Service testing infrastruc ture VNF Functest Testing Suite Infrastructure testing Test cases API hardware Tool sets Yardstick MANO Dovetail Ixia Spirent Configuration translator Deployment tools fuel Triple-O Daisy Ansible Compass Puppet HW configuratio n before deployment (MAC IP VLAN, etc.) Configuratio n after deployment (NUMA huge page DPDK, etc..) Work flow management CI/CD Jenkins Physical resource Bare Metal VM resource VM Container resource Docker Testing resource 11
System overview Compatible to multiple installers. Present generic configuration files to End Users to reduce operation difficulties Integrator UI Workflow Management Test Result Test Task management Integration management Failure Detection & Root cause analysis Hardware Configuration Developer UI Automatic Testing Automatic Integration Automatic Configuration Code development Git Gerrit Service testing infrastruc ture VNF Functest Testing Suite Infrastructure testing Test cases API hardware Tool sets Yardstick MANO Dovetail Ixia Spirent Configuration translator Deployment tools fuel Triple-O Daisy Ansible Compass Puppet HW configuratio n before deployment (MAC IP VLAN, etc.) Configuratio n after deployment (NUMA huge page DPDK, etc..) Work flow management CI/CD Jenkins Physical resource Bare Metal VM resource VM Container resource Docker Testing resource 12
System overview Utilizing testing framework from both open-source (OPNFV/OpenStack) and commercial products Integrator UI Workflow Management Test Result Test Task management Integration management Failure Detection & Root cause analysis Hardware Configuration Developer UI Automatic Testing Automatic Integration Automatic Configuration Code development Git Gerrit Service testing infrastruc ture VNF Functest Testing Suite Infrastructure testing Test cases API hardware Tool sets Yardstick MANO Dovetail Ixia Spirent Configuration translator Deployment tools fuel Triple-O Daisy Ansible Compass Puppet HW configuratio n before deployment (MAC IP VLAN, etc.) Configuratio n after deployment (NUMA huge page DPDK, etc..) Work flow management CI/CD Jenkins Physical resource Bare Metal VM resource VM Container resource Docker Testing resource 13
Lab overview Continuously downloading updated codes from OPNFV More than 3 TICs are included into the automatic system for integration and on-boarding testing Finish development of more than 150 test cases on infrastructure and services code@github update code@opnfv Internet Lab Sync code@lab Jenkins master (web ui) @115.3:8080 Result Shell Script Yardstick/Functest Jenkins slave 1 Jenkins slave 2 IXIA IXload Jenkins slave 3 Spirent itest TIC 1 TIC 2 TIC 3 @115.222 @35.96 @ 14
Auto Testing NFVI testing Based on China Mobile NFVI testing specification Includes functional, performance, and availability testing of more than 200 test cases Utilizing testing framework provided by OPNFV yardstick Service testing Based on China Mobile service testing specification Includes functional, performance, and availability testing Utilizing Ixia automation tools Jenkins master Jenkins slave (Jumphost) Yardstick (docker) Ixia-Ixload (Windows VM) SUT vims vepc CentOS Hardware Server Open-O OpenStack Physical testing tools Ixia-Ixchasis vepc test cases 15
Yardstick Architecture Overview Yardstick framework is utilized for functional testing. This framework provides quite rich and easy to use Restful API, contexts, and scenarios. Support Restful API,can be easily integrated Support Plug-in, can integrate other tools Support system level configuration Support test case control Support test result distribution and visualization Test case decomposed with framework
Auto testing - Architecture Jenkins Shell 1. Git code update 2. Clean Env. 3. Select test cases 4. Execute Docker container Ansible/Yardstick test cases ssh NFVI Openstack API VM1 controller VM2 compute1 compute2 Jenkins job configuration to define test Use docker container as test Env. Verify the installation of NFVI Ssh via controller Host aggregate compute3 17
Auto integration - Architecture First time to use Compass4nfv to build commercial Openstack platform. Components Jenkins & Registry tools Compass4nfv (for Jump Server OS provision) Installers (WRS Titanium, RHEL Director, etc.) Features Support multiple Installers Unified network and env configuration Jenkins + Ansible driven 18
Next Steps Auto testing: Finish development of NFVI performance testing and availability testing cases Integrate more service test cases, including vepc, vims, vcpe, and etc. Improve test result presentation Auto integration: Integrate more installers Provide generic configuration file and translator 19
THANK YOU!