Joint Agile Delivery Phase 3 Frank Massoudian Edward Pershwitz 2017 TM Forum 1
Joint Agile Delivery Phase 3 Project Lead Champions Participants 2017 TM Forum 2
Catalyst Timeline Oct 10-13, 2016 Stockholm Phase 1 Phase 2 Phase 3 May 2016 July 2016 Oct 2016 Dec 2016 May 2017 Dec 2017 2017 TM Forum 3
JAD Service Validation & Assurance Introduction Problem Statement & Proposed Solution Accomplishments to Date Phase 1 & 2 Test Model, Language & API s Contribution to TM Forum 2017 TM Forum 4
Multiparty Collaboration NFV standards still maturing Solutions coming from multiple parties (including the Service Provider) Separation of software from hardware poses additional requirements and certification challenges No single party can test/certify or assure their contribution in isolation Open Collaboration with CSP acting as gatekeeper Collaboration is a solution that is only possible if processes are standardized and the parties utilize a platform to enable joint activities 2017 TM Forum 5
Joint Agile Delivery Platform A standardized & platform-based approach to developing and delivering world-class software that capitalizes on cross-organizational (cross-value fabric) synergies to dramatically improve TTM, quality, and cost. Joint Requirements & Solution Test Strategy On A Unified Collaboration Platform Continuous Integration At The Network Service Level Standardized Test Cases & Test Language Joint Solution (Service) Verification Environment Continuous Rapid Delivery At The Solution (Service) Level Real-time Customer Feedback Throughout The Development & Deployment Cycle Service Provider Suppliers Integrator End Users Open APIs 2017 TM Forum 6
Accomplishments to Date Phase 1 & 2 7 2017 TM Forum 7
Requirements Management Digital Service Providers (VNF) Suppliers Integrator End Users/Customers 2017 TM Forum 8
Workflow Management Workflow-driven Processes With Continuous Project Tracking on a Collaboration Platform 2017 TM Forum 9
JAD Service Validation Principles Pooling of resources Identify the interface points and the key interactions Open TM Forum Test APIs Strategic reuse Test Plans, Test Cases, Test Execution Platforms One single source of truth Common Test Repository One common language Standardized Test Metamodel and language (JADL) Accessibility and visibility for all the stakeholders involved Collaboration Platform Traceability across the entire lifecycle 2017 TM Forum 10
VNF Certification JAD Platform VNF Update Change Request VNF Supplier VNF Supplier CSP Integrator VNF SW Test Cases Service Assurance Change Request Review VNF Acceptance Test Plan VNF Package New Version Test Plans Change Request Perform Impact Analysis Acceptance, Regression, Functional, Performance, Interoperability, Service Certification Minor Release Major Release Integrator Perform Regression Testing Perform Acceptance Testing Perform Service- Level Testing Certified VNF Package JAD Test Execution Platform Failed OK JAD Test Execution Platform Failed OK Services Update Service-Level Test Plans CCM Test Management Agile Planning & Tracking Change Request VNF Supplier
VNF Certification JAD Platform VNF Update Change Request VNF Supplier VNF Supplier CSP Integrator VNF SW Test Cases Service Assurance Change Request Review VNF Acceptance Test Plan VNF Package New Version Test Plans Change Request Perform Impact Analysis Acceptance, Regression, Functional, Performance, Interoperability, Service Certification Minor Release Major Release Integrator Perform Regression Testing Perform Acceptance Testing Perform Service- Level Testing Certified VNF Package JAD Test Execution Platform Failed OK JAD Test Execution Platform Failed OK Services Update Service-Level Test Plans CCM Test Management Agile Planning & Tracking Change Request VNF Supplier
Service Assurance Capabilities Adaptive Healer Rules Engine Service Topology Events/ Alarm Mgmt Monitoring Layer Common Mediation Layer Service Orchestrator Network Configuration JAD Ecosystem TOSCA NetConf YANG SNMP/ CLI REST / JSON Based Open API Lumos Test Management Common mediation for interfacing with different partner capabilities / VNFs via appropriate protocols Open architecture to plug and play capabilities from different partners and legacy capabilities with operator Improving predictive assurance capabilities - Knowledge Based Automation via ML and AI to predict degradation scenarios and trigger Service reconfiguration (closed loop ) Change request for VNF supplier (wide loop) 2017 TM Forum 13
JAD Test Model & Language (JADL) 1 2017 TM Forum 14
JADL Introduction JADL encompasses a standardized Test Model, a Domain-Specific Test Language, and an integrated Test Automation Platform: Test are written in an intuitive, easy to use and maintain Test Case DSL Tests follow a standardized test case model that include reusable artifacts: script, data, abstract resources, environment, etc. standardization enables JAD; reuse leverages assets; systematic approach improves quality Test execution environment is created dynamically using Dynamic Resource Management to allow multiple providers contribute resources, improve resource utilization and reduce test execution time Test are executed on a Test Execution Platform that ties Test Case elements together and provides intended interactions with the Test Infrastructure unified solution across providers reduces overhead 2017 TM Forum 15
General Test Domain Concepts System Under Test (SUT): a piece of software, a device, a network, etc. Test Case Resources: abstract and concrete Test Execution Flow: controlled and uncontrolled state transitions High-Level Functions: a repeatable sequence of controlled state transitions Test case data: resource-specific and non-resource-specific Test Case Segments: executable Higher-Level Functions Test Case Data (Symbol Lookup) blocks of the test case 2017 TM Forum 16
Test Case Elements The test case has a test script and uses test data, high-level function libraries and segment ordering data The test script has resource declaration and execution flow definition parts The resource declaration section aggregates a set of abstract resources that realize their respective APIs The execution flow can call the abstract resource APIs; it can also invoke higher-level functions which can invoke other higher-level functions and call abstract resource APIs 2017 TM Forum 17
Test Environment Abstract test environments aggregate abstract resources which are mapped to concrete resources Three major types of concrete resources: SUT elements, test tools (used to interact with the SUT) and reservable data The glue between the abstract and the concrete environments is provided by the Physical/Concrete Environment Resource Model (PERM) The PERM models the concrete resource space as a set of reservable entities and provides a number of abstractions (e.g. pools) that can be referenced from the abstract environment definition The PERM captures domain knowledge about the concrete resource space and considerably simplifies abstract environment definition 2017 TM Forum 18
Test Scenarios Test suites aggregate test cases that verify functional requirements. Test cases in a test suite are executed once Success and failure are considered on individual test case basis. Call models aggregate test cases that verify non-functional requirements (performance, robustness, etc.) Test cases execute repeatedly, each at its own rate. Together, they emulate real network conditions where multiple activities happen at the same time. Success and failure are considered on the entire call model basis and a certain number of individual test case failures is typically expected 2017 TM Forum 19
JADL Components Projectional Modeler Execution Engine 1 GPL 1 Representation Pool gsm_mobile: orig01,..., orig05, term01,..., term05 10 aliases: orig01 -> gsmth:mobile101... orig05 -> gsmth:mobile105 term01 -> gsmth:mobile106 Conceptual Model Execution Engine 2 GPL 2 Representation Pool gsm_mobile: orig01,..., orig05, term01,..., term05 10 aliases: orig01 -> gsmth:mobile101... orig05 -> gsmth:mobile105 term01 -> gsmth:mobile106 IDE Concrete Syntax Test Execution Platform Test Execution Platform Grammar Parser Constraints Optimizations Resource Manager Provisioning Manager Concrete Environment 2017 TM Forum 20
Text-based Test Case Example 2017 TM Forum 21
Test Case Creation and Execution Test Execution Platform (TEP) is an additional software layer between the test written in a DSL and its target platform. Provides required interactions Template 10 aliases: Pool orig: orig01, orig02,..., orig05..., term05 with the rest of the tools infrastructure Controls all aspects of the test case execution on the target platform Test Case Framework allows for: Template generated by the framework Test Script in DSL 10 aliases: orig01 -> gsmth:mobile101... orig05 -> gsmth:mobile105 term01 -> gsmth:mobile106... term05 -> gsmth:mobile110 Pool orig: orig01, orig02,..., orig05 Pool term: term01, term02,..., term05 Pool gsm_mobile: orig01,..., orig05, term01,..., 10 aliases: orig01 -> gsmth:mobile101... orig05 -> gsmth:mobile105 term01 -> gsmth:mobile106... term05 -> gsmth:mobile110 Pool orig: orig01, orig02,..., orig05 Pool term: term01, term02,..., term05 Pool gsm_mobile: Test Case Framework Test Flow Pool orig: orig01, orig02,..., orig05 Custom code 10 aliases: Pool orig: orig01, orig02,..., orig05 Generated code Pool gsm_mobile: orig01,..., orig05, term01,..., term05 10 aliases: orig01 -> gsmth:mobile101... orig05 -> gsmth:mobile105 term01 -> gsmth:mobile106 10 aliases: orig01 -> gsmth:mobile101... orig05 -> gsmth:mobile105 term01 -> gsmth:mobile106... term05 -> gsmth:mobile110 Pool orig: orig01, orig02,..., orig05 Pool term: term01, term02,..., term05 Framework code Abstract Resources Test Execution Platform Segment Ordering Data removing the technicalities of the test case interactions with the infrastructure from the body of the test case. enforcing that these interactions happen as intended by design Higher-Level Functions Test Case Data (Symbol Lookup) Resource Manager Concrete Environment Provisioning Manager 2017 TM Forum 22
Test Management API Shared On SWAGGERhub 2017 TM Forum 23
Test Case Management API 2017 TM Forum 24
Test Environment Management API 2017 TM Forum 25
Test Data Management API 2017 TM Forum 26
Test Result Management API 2017 TM Forum 27
Contributions To Frameworx 2 2017 TM Forum 28
Contribute the newly defined API s to Frameworx JAD Test Management APIs JAD Test Execution APIs Evaluate JAD catalyst and "JAD framework" to assess the feasibility to overlay Frameworx artifacts for: Guidebook: DevOps framework for the Telco industry Contribute the followings to integrate and align with SID and ZOOM projects: JAD Test Metamodel Testing DSL Document the catalyst learnings in IG1137A (Validation) in Release 17 and IG1137B (Assurance) in Release 17.5 2017 TM Forum 29