TEST AUTOMATION FOR THE IOT: IOT-TESTWARE. Axel Rennoch, IoTbarcamp, Munich, October 5th, 2017

Transcription

1 TEST AUTOMATION FOR THE IOT: IOT-TESTWARE Axel Rennoch, IoTbarcamp, Munich, October 5th, 2017

2 BERLIN CENTER FOR DIGITAL TRANSFORMATION 2

3 AGENDA IoT test objects, goals, and configuration IoT test automation IoT test language TTCN-3 IoT-Testware project IoT-T testlab IoT-Quality Engineering outlook 3

4 INTRODUCTION Where are we?

5 REFERENCE MODEL (ONE OF MANY) 7

6 TEST OBJECTS IoT devices, Mikrocontroller (MCU), Gateways (Bosch XDK, IoT starterkits) IoT platforms RIOT, relayr, Thread, mbed service layer (onem2m, FiWare) IoT protocols Constrained Application Protocol (CoAP) MQ Telemetry Transport (MQTT) onem2m LPWAN LoRa IoT challenges: complexity, asynchronism, long operation phase, resource constraints 8

7 INTEGRATION OF SEVERAL TESTING APPROACHES Protocol Testing Software Testing IoT Testing Test Automation System Testing Security Testing 9

8 TEST AUTOMATION Less resources needed (time and money) Avoid human mistakes due to manually testing During test development and execution Speed-up of regression tests and product time-to-market 10

9 MULTIPLE TEST CONFIGURATION (SAMPLES) 11

10 TESTWARE Toolset (selection of available means) Protocol tester/monitor Test devices GUI tester Web services tester (Eclipse Titan, Wireshark) (RFID kit, Bluetooth test device) (Selenium, SikuliX, Chrome headless) (soapui) Public Testsuites (in preparation) Application of a standardized notation Abstract and platform-independent 12

11 IOT TEST LANGUAGE What do we use?

12 CHALLENGE TEST AUTOMATION TTCN-3 is the Testing and Test Control Notation Internationally standardized testing language for formally defining test scenarios. Designed purely for testing testcase Hello_Bob () { p.send( How do you do? ); alt { []p.receive( Fine! ); {setverdict( pass )}; [else] {setverdict( inconc )} //Bob asleep! } } 14

13 TESTING AND TEST CONTROL NOTATION (TTCN-3) Distributed testing Automatic execution (TTCN-3 -> Java/C++) and logging Import and use of external data types (ASN.1, IDL, XML, JSON) API / Communication Tester SUT testcase Hello_Bob () { p.send( How do you do? ); alt { []p.receive( Fine! ); {setverdict( pass )}; [else] {setverdict( inconc )} //Bob asleep! } } TTCN-3 API / Network System 15

14 DESIGN PRINCIPLES OF TTCN-3 One test technology for different tests Distributed, platform-independent testing Integrated graphical test development, documentation and analysis Adaptable, open test environment Areas of Testing Conformance and functional testing Interoperability and integration testing Real-time, performance, load and stress testing Security testing Regression testing Used for system and product qualification and certification (e.g. handset certification) 16

15 TTCN-3 HISTORY TTCN (1992) published as ISO standard Tree and Tabular Combined Notation used for protocol tests: GSM, N-ISDN, B-ISDN TTCN-2/2++ (1997) enhancements by ETSI MTS module concept, concurrency used for conformance tests 17

16 TTCN-3 HISTORY (CONT.) TTCN-3 (2000) Further development by ETSI MTS Testing and Test Control Notation Standardised test specifications: SIP, SCTP, M3UA, IPv6 HiperLan, HiperAccess, WiMAX 3GPP LTE, 3GPP NB-IoT OMA TETRA MOST, AUTOSAR EUROCONTROL onem2m 18

17 IMPLEMENTATION Test Specification Test System TTCN-3 TE Test Engine + TCI Adaptors TRI Adaptors Systems Under Systems Test Under Systems Test Under Systems Test Under Test Communication / Invocation Automated Test Execution and Reporting 19

18 TTCN-3 FOR IOT Test technology address various protocols and interfaces Control of real and/or virtual devices (special hardware-interfaces, simulators) Elevator Earlier days Today CD Changer Tester TTCN-3 Test er Head Unit MOST Bus Amplifier / Tuner Tester TTCN-3 MQTT Gateway (SUT) (SUT) Test cases: loudness, playcdtitle Speaker Tester Test cases: remote control (SUT) PLC (SUT) 20

19 CONTRIBUTION TO IOT TESTING What else?

20 THE CONTEXT 22

21 THE ECLIPSE PROJECT Supplement to running Eclipse projects Paho, OM2M, Titan New project at Eclipse Foundation: TTCN-3 test suites for CoAP, MQTT, OPC-UA, LoRa? Assured licenses for users Currently in cooperation with relayr GmbH, Ericsson, LAAS/CNRS, itemis AG, Spirent Communications, Easy Global Market, Iskratel/Sintesio, 23

22 IOT-TESTWARE 24

23 TEST CONFIGURATIONS CoAP MQTT 25

24 TESTSUITE STRUCTURE CoAP MQTT 26

25 TEST PURPOSES TESTZIEL-KATALOG CoAP MQTT 27

26 TTCN-3 REPOSITORY 28 Fraunhofer FOKUS

27 THE TEST EXECUTION TOOL 29

28 SUMMARY AND OUTLOOK What comes next?

29 APPROACH Advanced + mature testing technology: (open source) community: Standardized basis (for certification): 35

30 IOT QUALITY ENGINEERING ASQF/GTB Certified Professional for IoT Foundation Level New syllabus for 3-day lectures QUALITY ENGINEERING FOR THE INTERNET OF THINKS (IOT-QE) 1) Motivation/challenges 2) Constructive QE Quality characteristics 3) Constructive QE IoT architecture 4) Constructive QE Processes and methods 5) Analytic QE (incl. testing) 6) Lifecycle 36

31 IOT QUALITY CALENDAR FOR BERLIN STV17/INTUITEST workshops: UCAAT conference: Industrial IoT Forum: ASQF Quality Day: Thank you for your attention! 37

32 CONTACTS Fraunhofer FOKUS Business Unit Quality Engineering (SQC) Kaiserin-Augusta-Allee Berlin, Germany Axel Rennoch phone