Zigbee Test Tool brings major benefits to IoT device manufacturers white paper Authors: Henk Veldhuis, Global Director Smart Lighting/Smart Home TÜV Rheinland and Musa Unmehopa, Vice Chairman of the Zigbee Alliance Board of Directors and Senior Director, Standards & Regulations, Philips Lighting Highlights Improved quality via test automation and testing of hardware to simulate conditions and use cases Easy validation due to diagnostics and monitoring capabilities and clear test reports Reduced cost and time to market and de-risked approbation process through enabling in-house pre-testing by the device developer Increased consistency through repeatable test cases and reproducible test conditions Introduction The Zigbee Alliance offers a Software Test Tool as part of the Zigbee Certification Program to assist with Zigbee device development, improve the testing and verification processes, and to reduce integral cost and time to market. The use of the Zigbee Test Tool leads to increased productivity due to the ability to manually test and automate the testing processes, increased quality of software components through extensive test case coverage, and reduced time to market by allowing in-house pre-testing by the device developer during the development stage of the Zigbee product. The Zigbee Test Tool is an important part of the total Zigbee solution for a wide-range of Internet of Things (IoT) applications. The Zigbee Test Tool (ZTT) The Software Test Tool consists of a test engine (Windows-based application) that uses a wide collection of test scripts and a radio USB-dongle (supported hardware listed by the
alliance). These components are configured to analyze and test a Zigbee device implementation by executing it under an extensive range of varying circumstances and conditions. The process validates compliance and conformance with the Zigbee specifications, confirms its behavior and outputs, and generates associated test reports. Figure 1: Screen shot of the Zigbee Test Tool Specific test scripts are used to handle different test data and execute various test scenarios. The ability to setup test pre-conditions allows for test cases which are otherwise hard to simulate (e.g. network overload) as well as for negative testing (e.g. receiving malformed commands, encountering non-conformant scenarios) to ensure that the implementation can gracefully handle invalid attributes and variables or unexpected user interactions. Testing the implementation s failover routines for error conditions is typically hard to achieve using golden units or reference implementations. The generated test reports provide a quick verdict and a coherent result summary, which gives the manufacturer immediate and clear insight into the Zigbee performance of the device and enables implementers and testers with diagnostics and monitoring capabilities to identify which test has passed, which has failed, and why. The Test Tool analyzes the Page 2
response of the implementation under test and compares it to the scripted expected behavior, which is much less error-prone than analyzing results manually. Figure 2: Zigbee Test Tool Furthermore, the Zigbee Test Tool allows for the automation of tests, calling test routines with supplied parameters and pre-conditions. Test cases executed using the ZTT are highly controllable in terms of a precise sequence of steps and exact timing of commands and events. The inventory of available test scripts facilitates subsequent test runs to be repeated under the same set of conditions and input parameter values. This ensures a consistent test result irrespective of which Zigbee-authorized third-party accredited test house is contracted to perform the testing process. Another major benefit of having the Zigbee Test Tool available to members is a consistency of performance and de-risking of the approbation process. Zigbee device developers can pre-test their implementation in-house before submitting their device to an authorized third-party test house. The developer gets immediate feedback and can reliably predict whether certification testing will be successful or not. Eliminating the need for re-testing saves money and time by shortening the development cycles and improving the ratio of first-time testing success. Page 3
Moreover, Zigbee device developers can easily extend the functionality of the Test Tool by creating and adding their own test cases through dedicated and specialized test scripts. This allows developers to add robustness in response to specific market requirements, or include testing of value-added differentiators. The availability of Zigbee Certified devices will increase choice and flexibility for developers and users, and deliver the confidence that products and services will all work together. The Zigbee Test Tool allows member companies to take full advantage of the professional services offered by the Zigbee Alliance and go to market quickly and with confidence. The Zigbee Certified Program The Zigbee Certified Program was put in place more than 10 years ago to provide Zigbee alliance members and the market in general with a defined, repeatable and independent set of methods by which a product s implementation of Zigbee alliance standards could be validated to be compliant with the standards it purported to support. The Zigbee Certified Program provides two major levels of certification: Zigbee Compliant Platform o This certification level validates a member s implementation of the IEEE 802.15.4 PHY/MAC and the applicable Zigbee alliance network specification (e.g. Zigbee PRO). Zigbee Certified Product o Required to be built on top of a Zigbee Compliant Platform, this certification level validates that the Zigbee application level standards (i.e. Zigbee 3.0) implemented in the product conform to the stated requirements. o Application-level interoperability is validated as part of this program. Page 4
The Zigbee Certified Program follows international certification guidelines (ISO Guide 67 Type 1b), and substantially complies with ISO Guide 65 General requirements for bodies operating product certification systems. The program maintains distinction between testing and certification as: Testing is performed by independent test service providers Certification is performed only by the Zigbee Alliance The Zigbee Alliance currently has five Authorized Test Service Providers with worldwide locations to ensure local availability for members: Element Materials (Europe) China Electronics Standardization Institute (China) National Technical Systems (North America) TÜV Rheinland (North America, Europe, Asia) Underwriter Laboratories (North America, Europe) All Alliance test service providers are ISO 17025 registered and undergo a rigorous process that validates both their knowledge of Zigbee technology as well as their capability to undertake alliance-defined testing processes before being designated as authorized to provide testing services to alliance members. These providers also undergo annual audits to ensure that their services and capabilities are continuing to meet Zigbee alliance requirements. The Zigbee Test Tool is an important component of the Zigbee certification process. All Zigbee alliance Authorized Test Service Providers have been outfitted and trained with the use of the Zigbee Test Tool. They are required to utilize the tool along with the validated and approved Zigbee Alliance test scripts when testing a Zigbee product. The output of the Page 5
tool is part of the report they issue to the member company and submit to the Alliance as part of the product s certification application. Usage of the same tool set across our multiple test service providers ensures repeatability of results across all tested products, improves process quality, and enhances member value. While the Zigbee Certification Program validates compliance, the software test harness can optionally be used by manufacturers to test interoperability. The Zigbee Test Tool provides detailed information to track and trace over-the-air Zigbee messages which may be helpful in finding root causes if something does not go as expected. Where can I get the Zigbee Test Tool? Alliance members can gain access to the Zigbee Alliance suite of Test Tools via the member workspace. Installers, usage guides, test script source and access to support resources can all be found in the member workspace. While the tools are available to ALL Zigbee Alliance members, some members (based on their level of membership) may be eligible for discounts on annual licenses. For additional information on the Zigbee Alliance test suite, please contact certification@zigbee.org. Closing words The Zigbee Test Tool is a software test tool which offers a lot of benefits including improved quality via test automation and testing of hard-to-simulate conditions and use cases, easy validation due to diagnostics and monitoring capabilities and clear test reports, reduced time to market, a de-risked approbation process through in-house pre-testing, and increased consistency through repeatable test cases and reproducible test conditions. Together with the robust Zigbee Certification Program, supported by five reputable Authorized Test Service Providers with global coverage, this Zigbee Test Tool by the Zigbee Alliance brings major benefits to IoT device manufacturers. Page 6