Reliability Testing In OPNFV. Hao Pang San Francisco 11/09/2015

Transcription

1 Reliability Testing In OPNFV Hao Pang San Francisco 11/09/2015

2 Agenda Overview of reliability Reliability research in OPNFV Introduction of reliability testing Reliability testing framework Test case demo Work Plan and Related links 2

3 Overview of Reliability Reliability is the ability of a system to perform and maintain its functions in routine circumstances, as well as hostile or unexpected circumstances. High reliability is often defined as a system availability of %, which meets the carrier-level requirement and protects carriers from business and reputation loss due to system breakdown. No Single Point of Failure Overload Control High Reliability Automatic Recovery Fault Isolation 3

4 Reliability is Critical Important ore and more business services rely on the Internet and cloud. A short outage can cause a huge economic loss. When the natural disaster happening, the telecom reliability supports the nation security and life emergency. 4

5 Reliability in NFV and OPNFV Standard Title Avail ability Doctor NFV-REL 001 NFV-REL 002 Resiliency Requirements Report on Scalable Architectures for Reliability anagement ulti- Site Pin Point NFV-REL 003 E2E reliability models Escalat or Predict ion NFV-REL 004 Active monitoring and failure detection NFV-REL 005 Quality Accountability Framework OPNFV Platform 5

6 The Goal of Reliability Testing Customer Requires Reliability Testing Goal Device had better never have fault If the fault occur, Do not affect the main services If the main services affected, locate and recover the fault as soon as possible Test the product ability of not having fault Test the product ability of fault recovery Test the product ability of fault detection, location and tolerance. Availability Numbers of 9 Downtime In a year(minute) Apply Product 99.9% Computer / Server 99.99% 4 50 Enterprise-class Device % 5 5 Common carrier -class Device % Higher carrier-class Device 6

7 Type of Reliability Testing Known Fault Unknown Scenario Simulate fault directly Fault Injection Simulate fault directly Trigger fault By External trigger Scenario Fault Injection Known Scenario Predict it by odeling Trigger it by the Stress Scenario 7 Reliability Prediction Unknown Fault Stability Testing

8 etrics for Service Availability General etrics Failure detection time: the time interval between the failure happens and the failure been detected. Service recovery time: the time interval between the failure happens and the service finish its recovery. the time interval from the occurrence of an abnormal event (e.g. failure, manual interruption of service, etc.) until recovery of the service. Fault repair Time: the time interval between the failure detected and faulty entity is repaired. Service failover time: the time from the moment of detecting the failure of the instance providing the service until the service is provided again by a new instance. Carrier etrics Network access success rate: the proportion of the end-user can access the network when requested. Call Drop Rate: the proportion of mobiles which having successfully accessed suffer an abnormal release, caused by loss of the radio link. 8

9 Scope of Reliability Testing (1) Item Layer Failure ode Priority Hardware Reliability testing Controller node Compute node Storage node Networking Control node server failure emory failure / emory condition not ok CPU failure / CPU condition not ok anagement network failure Storage network failure Compute node server failure emory failure / emory condition not ok CPU failure / CPU condition not ok anagement network failure Storage network failure Service network failure Storage node server failure Hard disk failure HW failure of physical switch/router Network Interface failure H H 9

10 10 Scope of Reliability Testing (2) Item Layer Failure ode Priority Software Reliability testing System reliability testing HOST OS failure H VI important process failure H Controller node VI important service failure H anagement network failure OVS Storage network failure OVS HOST OS failure H VI important service failure VI important process failure Compute node anagement network failure OVS Storage network failure OVS Service network failure-ovs Hypervisor(KV libvirt Qemu)failure H V failure H V Gest OS failure V service failure L V process failure L Robustness test Various Faults random injection testing Stability testing Long time testing with load L

11 Architecture of Reliability Testing Scheduler Traffic Flow VNF VNF Attacker onitor Reporter Infrastructure Analyst 11

12 Reliability Testing Framework: Yardstick Goal: a Test framework for verifying the infrastructure compliance when running VNF applications. Scope: Generic test cases to verify the NFVI in perspective of a VNF. Test stimuli to enable infrastructure testing such as upgrade and recovery. Covered various aspects: performance, reliability, security and so on. Design: Yardstick testing framework is flexible to support various contexts of SUT, and easy to develop additional plugin for fault injection, system monitor, and result evaluation. It is also convenient to integrate other existing test frameworks or test tools. 12

13 Reliability Testing Workflow cases.yaml output.json 1.input 5.output TaskCommand Output(Process) Yardstick Context Runner Scenarios BContext HeatContext 2.deploy 6.undeploy Attacker onitor 4.collect data System under test(sut) Nova Heat Neutron 3.fault Inject V V Host 13 Note: when some fault injection methods break the SUT(e.g. kill one controller), a option step recovering the broken SUT might be executed before the scenario is over.

14 Case Sample: OpenStack Service HA 8 test result 4 setup attacker Yardstick 2 setup monitor Attacker 7 Recover the broken 5 fault injection nova-api nova-api 1 Check nova-api onitor 3 monitor service 6 get the SLA Controller Node #1 Controller Node #2 Controller Node #3 14 Controllers HA

15 15 Test Case Configure and SLA

16 Test Case Demo Show 1. The nova-api processes are running on Controller Node #3. 2. The nova service is running normally. Before the Fault Injection 16

17 Test Case Demo Show 3. The nova-api processes have been killed. 4. Nova service is still working normally. 5. ServiceHA test case ran by using Yardstick and recorded its test result. 17 After the Fault Injection

18 Work Plan TC Name Testing Purpose Priority Status OpenStack controller service failure Controller Node abnormally shutdown anagement network timeout V abnormally down Verify the services running on the controller is high available. Verify the controllers cluster deployment is high available. Verify the controllers cluster deployment is high available. Verify the V running on the compute is high available. H Done Doing TODO TODO 18

19 Related Links Wiki of Yardstick project: Requirement of HA project: HA Test cases in Yardstick: 19

20 Copyright 2015 Huawei Technologies Co., Ltd. All Rights Reserved. The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time without notice.