SBLIM: Test Suite. Test Suite Specification and Implementation Details. Revision 4 Last updated June 13, 2003

Transcription

1 SBLIM: Test Suite Test Suite Specification and Implementation Details Revision 4 Last updated June 13, 2003 Document Owner: Heidi Neumann LTC Systems Management heidineu@de.ibm.com Distribution: SBLIM Open Source Project http.//oss.software.ibm.com/developerworks/projects/sblim/ Master Copy is located on PC of document owner

2 1 Document Control Change History Reviewers List Contributing Authors Introduction Purpose of this document Test Approach / Requirements Interface Test Consistence Test Specification Test Problem Areas SBLIM Test Suite Prerequisites Architecture File Structure User Interfaces Script : run.sh Script : interface.pl Script : consistence.pl Interface Test Input File Class specific Syntax Header Syntax Paragraph Syntax Example Association specific Syntax Header Syntax Paragraph Syntax Example Report File Class specific Syntax Association specific Syntax Instance Interface enuminstancenames() or enuminstances() getinstance() setinstance() createinstance() deleteinstance() execquery() Association Interface associatornames() & associators() referencenames() & references() Algorithm to Handle the Input Parameter role, resultrole and resultclass associatornames() referencenames() Report File Method Interface Property Interface Indication Interface Common Interface Tests Check for expected Exception Consistence Test Capabilities Input File Class specific Syntax

3 Header Syntax Paragraph Syntax on Class Level Paragraph Syntax on Property Level Paragraph Syntax on Instance Level Example Association specific Syntax Header Syntax Paragraph Syntax on Reference Level Example Report File Class Specific Syntax Association Specific Syntax Class Specific Test Cases Class Level Property Level Instance Level Association Specific Test Cases Association Level Specification Test

4 1 Document Control 1.1 Change History Revision Date Description 1 04/04/2003 Initial Draft Version 2 04/25/ reorganization of input and report file descriptions for interface test (moved from chapter 3 to chapter 4) - Chapter 5 : Consistence Test written 3 05/15/ Chapter 5 : additional return codes on property level defined - Chapter 5 : input / report file descriptions and return codes for association test defined 4 06/13/ Chapter 6 : comment added to be future task - Review of complete document in preparation to the announcement 1.2 Reviewers List Name Robert Kieninger Adrian Schuur Viktor Mihajlovski Role & Responsibility 1.3 Contributing Authors Name address Description Robert Kieninger kieningr@de.ibm.com Input for Requirements and Purpose of the test suite 4

5 2 Introduction 2.1 Purpose of this document Performing hand-operated tests of available CIM Instrumentation can become a nightmare, if tests for different operating systems, Linux distributions, releases and hardware platforms are required. In order to guarantee that one CIM Instrumentation is functioning correct within different environments, a test suite is being developed that offers the user a well defined set of test types to define Instrumentation dependent test scenarios and the possibility to perform automated function verification tests against the installed CIM Instrumentation. The SBLIM Test Suite is subject of this document. The reader should be familiar with CIM, CIM terms and should have some deeper knowledge about provider implementations. 2.2 Test Approach / Requirements The test suite acts as a CIM Client and uses a generic command line interface (SBLIM WBEMCLI package) that can be parameterized to perform the CIMOM calls. The command line interface WBEMCLI can communicate with different CIMOM Implementations, like OpenCimom and Pegasus. The test suite itself is script based and interprets the return values of the WBEMCLI command. The test suite has to be easy extendable regarding the test of new classes / providers. The test suite covers the following three types of tests for CIM Instrumentation. One that generically verifies that all required provider interfaces are properly implemented, one that checks if the returned values are really meaningful and one that checks if the returned values follow the specification. All tests have to be as much self checking as possible Interface Test The most important and supposing easiest to implement test scenario is a general functionality test of the interface types supported by the provider. Known types are the Instance, Association, Property, Method and Indication (or Event) Interfaces. Each interface type supports a defined set of operations. The first iteration of the test suite will be able to execute each operation of the Instance and Association Interfaces and interpret the returned results. A requirement against the design of the test suite is the possibility to easily support the other interface types. A modularized approach will be the choice here. The user is able to define on operations level which tests to perform. Therefore he needs knowledge about the interface types and their special operations. This allows the user to specify the scope of the test scenario. The user should be able to specify a certain Exception he expects. Some implementations require, that an exception like NOT_SUPPORTED is thrown by the provider. This makes sense, if an operation is callable, but has no functionality. For example, the createinstance() call of the OperatingSystem instance provider does not make sense. Supporting this operation would mean, that an OS should install another OS over itself! To offer the client a self-describing return code, the decision was made, to return NOT_SUPPORTED. Returning a status of OK would mean that the operation was successfully performed. This is wrong, because there is no functionality implemented for this operation. 5

6 2.2.2 Consistence Test The consistence test distinguished between class and association level. The class level contains a test between the numbers of reported instances against the system state. The number of valid instances is collected by a script or only one single command. Described on a file system example, the provider should return not more and not less instances than file systems are known to the operating system. So it can be checked if the provider returns duplicates or does not report some of the instances. On property level two types are important to check the correctness of the returned property values. One can check the observance of threshold values. The other one is checking returned property values against system values. The last test step checks the consistence of the property values within each single instance. The association level contains a test between the number of reported target instances against the association definition and the system state. A further test checks the correctness of the instances reported by the association provider against the instances, reported by the instance provider of the target instance. The system values are collected by test scripts. To automate the execution of these test steps the system test scripts follow certain syntax Specification Test The specification test makes use of the meta-information about the model. Within this test type, it is possible to figure out, if the provider implements the class definition in the required manner. For example, the definition of a class marks a certain property as required. But the provider does not return a value for this property. This is a violation of the CIM specification. 2.3 Problem Areas Due to the dynamic nature of the returned data it will be hard, if not impossible, to create tests that check for semantically correct results (chapter Consistence Test). However, because of the layered architecture it should be possible to create a dummy version of the Resource Encapsulation Library, that will allow the providers to generate reproducible results. They can be used for automated validation of correct data processing within the providers. CIMOM CMPI CMPI Provider CMPI Provider CMPI Provider Resource Encapsulation Library (Data Gathering) Instrumentation Package R R R R R Figure 1 : Architecture of CIM Instrumentation Packages 6

7 3 SBLIM Test Suite 3.1 Prerequisites To guarantee a correct functioning of the SBLIM Test Suite, the user has to make sure, that the following prerequisites are satisfied. the CIMOM is configured and functioning correct the CIMOM is running the model to test is completely loaded into the CIMOM repository the installation process of the CIM Instrumentation (provider) was successfully completed the CIMOM is capable of calling the provider 3.2 Architecture The entry point to the test suite is class level. That means each test type (see 2.2 Test Approach / Requirements ) is performed to one single class name. To automate the test for a collection of classes a small shell script can be written, which calls the test suite entry point with the respective class name. The used programming languages are Perl and shell scripting. For each class and test type an input file must be written by the user. The semantic of this file follows a well defined structure, which is described in the chapters 4.1 Input File and 5.2 Input File. For each performed test of a certain class a report file is created. These statistical data describe the status of each test step ( ok or failed and a description ), the time spend in system and user mode (describes the resource consumption of the test process), a summary of the succeeded and failed test steps and the overall times spend in system and user mode File Structure File Name run.sh cimom.pm interface.pl instance.pm associator.pm shell script perl module perl script perl module perl module Functionality contains the collection of test types to perform (interface, constistence, specification) Contains the access information to the specific CIMOM -> edit to fit your system parses the input file and selects the corresponding interface type implementation of the instance interface tests implementation of the association interface tests 7

8 consistence.pl consistence.pm system/<platform>/ createkeyfiles.sh perl script perl module shell script parses the input file and selects the corresponding level to test (class or association) implementation of the class and association level tests creates two files : ComputerSystem.keys and OperatingSystem.keys; the files offer the CreationClassName and Name of the classes for reference by other classes, e.g. Process cim directory contains the input files for the interface tests system/<platform> directory contains the input files for the consistence tests stat directory contains all report files after test execution; for each executed test one report file Each type of test (Interface, Consistence and Specification) is divided into two major parts, where the first part is mandatory and the second part a recommendation to automate the execution of tests for more than one class / association. The first part is writing the necessary definition files, later called input files to define the classes / associations to test. The input files have to follow certain syntax. The syntax is described in the corresponding chapter. In addition it can be necessary to write shell script for the system data gathering. The second part is writing a small shell script to automate the execution of the test types. The script run.sh can be used as entry point User Interfaces Script : run.sh The script run.sh is the entry point for a user, who wants to use the whole functionality of the test suite. The script executes first the interface test and second the consistence test for the given class. Usage : First the variable SYSTEM_PATH has to be edited to fit to the actual system. The variables HOST and TEST_DIR should be initialized correctly, but they can be change if necessary../run.sh <class name> Script : interface.pl The perl script interface.pl executes the interface operations against the provider as defined by the input file. In addition the parameter -assocparams defines, that the algorithm which handles the validation of the input parameter role, resultrole and resultclass of the association operations has to be tested. Usage :./interface.pl <class name>./interface.pl <class name> -assocparams Script : consistence.pl The perl script consistence.pl executes the consistence test and compares the values returned by the provider against the system data as defined by the input file. 8

9 If the consistence test is started for the first time on a certain machine, the script system/<platform>/createkeyfiles.sh is executed to create the platform specific key files. If this step fails, the complete test will fail. Under circumstances it can be necessary to enhance the functionality of this test script. It depends on the classes to test. Usage :./consistence.pl <class name> <system/<platform>> 9

10 4 Interface Test 4.1 Input File The input file defines the class / association to test. For each class / association one input file has to exist in the directory cim. The name of the file has to be <class name>.cim. The syntax of the input files distinguishes between two parts. The header contains overall information of the class / association itself. The header is followed by paragraphs, describing the interface type and operation to test. It is not necessary to insert all operations of a certain interface. By adding a certain paragraph to the input file, the decision was made to test this interface operation. By leaving a certain paragraph out, the decision was made to not test this interface operation Class specific Syntax Header Syntax The key words class and objectpath are mandatory. The class qualifier defines the name of the class. The objectpath qualifier defines a full qualified object path of a sample instance of this class. The object path must not contain a newline! The object path is necessary to perform operations like createinstance() and deleteinstance(). If the intention is only to test a certain returned Exception, the sample instance can be a dummy instance and does not need to exist on the test system. The qualifier volatile should only be used for classes, which represent volatile data, for example the class CIM_Process. The intention of this qualifier is to find a mechanism how to handle the dynamical nature of some data. It is very likely that each time you call especially the getinstance() operation of a provider for volatile data, the returned answer can vary between ok and NOT_FOUND. To avoid in principle failing the test has to be aware of this difference. ****************************************************************************** class : <class name> objectpath : <object path of a sample instance> [ volatile ] Paragraph Syntax Each paragraph defines the interface type Instance and the operation to perform. Supported key words for operations are enuminstancenames, enuminstances, get, set, create, delete and execquery. The qualifier Expected Exception is optional and needs only be defined, if a certain Exception is expected to be thrown (as described in Interface Test ). Instance enuminstancenames 10

11 Instance delete [Expected Exception : <insert expected exception type>] Instance execquery Example ***************************************************************************** class : Linux_ComputerSystem objectpath : Linux_ComputerSystem.CreationClassName=Linux_ComputerSystem,Name\ =testsystem.ibm.com Instance enuminstancenames Instance create Expected Exception : NOT_SUPPORTED Association specific Syntax An association is defined in mof syntax as followed : [ Association ] class Linux_RunningOS : CIMRunningOS { [Override Antecedent ] Linux_OperatingSystem REF Antecedent; [Override Dependent ] Linux_ComputerSystem REF Antecedent; } Header Syntax The key words association and objectpath are mandatory. The association qualifier defines the name of the association. The objectpath qualifier defines a full qualified object path of a sample instance of this association. The object path must not contain a newline! The object path is necessary to perform instance operations like createinstance() and deleteinstance(). If the intention is only to test a certain returned Exception, the sample instance can be a dummy instance and does not need to exist on the test system. The qualifiers sourcerole, targetrole, sourceclass and targetclass are mandatory and define the association specific reference properties. The sourcerole specifies the name of the source reference. It is recommended to use the name of the first reference in the association, for example Antecedent. The targetrole specifies the name of the target reference. It is recommended to use the name of the second reference in the association, for example Dependent. The sourceclass specifies the class name of the source reference, for example Linux_OperatingSystem. If the provider supports a CIM_<class name> and returns instances with class names of other, derived classes, then list first the CIM_<class name> followed by all the derived class names. The targetclass specifies the class name of the target reference, for example Linux_ComputerSystem. If the provider supports a CIM_<class name> and returns instances with class names of other, derived classes, then list first the CIM_<class name> followed by all the derived class names. 11

12 ****************************************************************************** association : <association name> objectpath : < object path of a sample instance> sourcerole : <role of the source reference> targetrole : <role of the target reference> sourceclass : <class name(s) of the source reference> targetclass : <class name(s) of the target reference> Paragraph Syntax Each paragraph defines the interface type Association and the operation to perform. Supported operations are associators, associatornames, references and referencenames. Association associators Association referencenames Example ***************************************************************************** association : Linux_RunningOS objectpath : Linux_RunningOS.Antecedent= Linux_OperatingSystem.CSCreationClass Name=Linux_ComputerSystem,CSName=testsystem.ibm.com,CreationClassName=Linux_Op eratingsystem,name=testsystem.ibm.com,dependent= Linux_ComputerSystem.Creatio nclassname=linux_computersystem,name=testsystem.ibm.com sourcerole : Antecedent targetrole : Dependent sourceclass : Linux_OperatingSystem targetclass : Linux_ConputerSystem Instance enuminstancenames Instance create Expected Exception : NOT_SUPPORTED Association associators Association associatornames Association references Association referencenames 12

13 4.2 Report File The report file is generated per tested class. For each executed interface test one report file exists in the directory stat. The name of the file is <class name>.stat. The syntax of the report files is similar to the syntax of the input files. The report file contains some additional information. Each interface type has its own chapter. If an input file defines tests on instance and association operations this test results in a report file, containing one chapter for the results of the instance interface test and one chapter containing the results of the association interface tests. Each chapter header specifies the start date of the test. The paragraph of the single operations depends on the returned execution status. Each report file contains a summary to report an overall statistic of the tests. It contains the number of succeeded and failed tests and the times the whole test case spend in user and system mode Class specific Syntax Possible return codes of the operations are defined in chapter 4.3 Instance Interface. If the test returned with ok, a statistic of the time spent in system and user mode is reported. In addition if an enumeration was tested the number of returned instances is reported. Example : ****************************************************************************** class : Linux_ComputerSystem Fri Apr 4 8:43: Instance enuminstancenames Status : ok RC : 1 Number of returned Instances : 1 user time : 0.01 child user time : 0 ut : 0.01 system time : 0 child system time : 0 st : 0 Instance create Status : failed unexpected exception occurred - <wbemcli output> RC : 2 ****************************************************************************** Linux_ComputerSystem Number of Tests where Status ok : 1 Number of Tests where Status failed : 1 user time : 0.02 child user time : 0 ut : 0.02 system time : 0 child system time : 0.01 st : Association specific Syntax Possible return codes of the operations are defined in chapter 4.4 Association Interface. The traversal of an association can be performed in two directions. The test distinguishes between them. Therefore the 13

14 status of both directions is reported separately. If both tests returned with ok a statistic of the time spend in system and user mode is reported. Example : ****************************************************************************** class : Linux_RunningOS Fri Apr 4 8:44: Instance enuminstancenames Status : ok RC : 1 Number of returned Instances : 1 user time : 0.01 child user time : 0 ut : 0.01 system time : 0 child system time : 0 st : 0 Instance create Status : ok operation returned with NOT_SUPPORTED RC : 1 ****************************************************************************** association : Linux_RunningOS Fri Apr 4 8:45: Association associators RC : 1.1 Status : Linux_OperatingSystem to Linux_ComputerSystem : ok Number of returned Instances : 1 Status : Linux_ComputerSystem to Linux_OperatingSystem : ok Number of returned Instances : 1 user time : 0.03 child user time : 0 ut : 0.03 system time : 0 child system time : 0.01 st : 0.01 Association associatornames RC : 1.1 Status : Linux_OperatingSystem to Linux_ComputerSystem : ok Number of returned Instances : 1 Status : Linux_ComputerSystem to Linux_OperatingSystem : ok Number of returned Instances : 1 user time : 0.05 child user time : 0 ut : 0.05 system time : 0 child system time : 0.02 st : 0.02 Association references RC : 1.1 Status : Linux_OperatingSystem to Linux_ComputerSystem : failed <description> Association referencenames RC : 1.1 Status : Linux_OperatingSystem to Linux_ComputerSystem : failed <description> 14

15 ****************************************************************************** Linux_RunningOS Number of Tests where Status ok : 4 Number of Tests where Status failed : 2 user time : 0.08 child user time : 0.03 ut : 0.11 system time : 0.03 child system time : 0.02 st : Instance Interface The CIM Instance Provider Interface supports the following operations : enuminstances() enuminstancenames() getinstance() setinstance() createinstance() deleteinstance() execquery() The perl functions to test the instance operations are implemented in the perl module instance.pm. The module exports the functions enumerate, get, create, delete, execquery and check_returnedexception enuminstancenames() or enuminstances() The perl function enumerate is responsible for testing the instance provider operations enuminstancenames() and enuminstances(). The perl function sends an enuminstancenames() or enuminstances() request to the CIMOM and validates the returned values. Test Return Code Return Code Description check if an exception occurred 2 failed - exception occurred - <wbemcli output> check if the returned instance(s) is/are of failed - returned instance(s) is/are of wrong 3 the requested class type class type 1 ok getinstance() The perl function get is responsible for testing the instance provider operation getinstance(). The perl function sends an enuminstancenames() request to the CIMOM to get the object paths of all available instances. For each object path an getinstance() request is send to the CIMOM and the returned value is validated. 15

16 Test Return Code Return Code Description check if enuminstancenames() returned 2 failed - enumeration - exception occurred - with an exception <wbemcli output> check if the by enuminstancenames() returned instance(s) are of the requested 3 failed - enumeration - returned instance(s) is/are of wrong class type class type check if an exception occurred 12 failed - exception occurred - <wbemcli output> check if the returned instance is of the requested class type 13 failed - returned instance is of wrong class type - <wbemcli output> check if a volatile class type returned with another exception than NOT_FOUND 14 warning - volatile class type returned with exception - <wbemcli output> 1 ok setinstance() Currently not supported createinstance() The perl function create is responsible for testing the instance provider operation createinstance(). The perl function sends an createinstance() request to the CIMOM and validates the returned value. If successful, an getinstance() operation of the new instance is executed to test if the instance was really created. Test Return Code Return Code Description check if an exception occurred 2 failed - exception occurred - <wbemcli output> check if instance was created 3 failed - instance was not created check if in getinstance an exception occurred 12 failed - exception occurred - <wbemcli output> 1 ok deleteinstance() The perl function delete is responsible for testing the instance provider operation deleteinstance. The perl function sends an deleteinstance() request to the CIMOM and validates the return value. If successful, an getinstance() operation of the old instance is executed to test if the instance was really deleted. Test Return Code Return Code Description check if an exception occurred 2 failed - exception occurred - <wbemcli output> check if instance was deleted 3 failed - instance was not deleted check if in getinstance an exception occurred 12 failed - exception occurred - <wbemcli output> 1 ok 16

17 4.3.6 execquery() Currently not supported. 4.4 Association Interface The CIM Association Provider Interface supports the following operations : associators() associatornames() references() referencenames() The perl functions to test the association operations are implemented in the perl module associator.pm. The module exports the functions associators, references and check_role_resultrole_resultclass_params associatornames() & associators() The perl function associators is responsible for testing the association provider operations associatorname s() and associators(). For each reference class name (source and target reference) the perl function associators sends an enuminstancenames() request to the CIMOM to get the object paths of all available instances. For each object path an associatornames() or associators() request is send to the CIMOM and the returned values are checked. If the qualifier sourceclass or targetclass define a list of class names, only the first entry with the CIM_<class name> is taken to perform this test.! The parameter assocclass is set to the class name of the association. If this parameter is not supported by the provider, the complete test will fail! Test check if in enuminstancenames() an exception occurred check if an exception occurred 12 check if the returned instance is of the expected target class type Return Return Code Description Code failed - enumeration <source class name> - 2 exception occurred - <wbemcli output> failed - (associators associatornames) - exception occurred - <wbemcli output> 13 failed - returned instance(s) is/are of wrong class type <target class name> 1 ok referencenames() & references() The perl function references is responsible for testing the association provider operations referencenames() and references(). 17

18 For each reference class name (source and target reference) the perl function references sends an enuminstancenames() request to the CIMOM to get the object paths of all available instances. For each object path an referencenames() or references() request is send to the CIMOM and the returned values are checked. If the qualifier sourceclass or targetclass define a list of class names, only the first entry with the CIM_<class name> is taken to perform this test.! The parameter resultclass is set to the class name of the association. If this parameter is not supported by the provider, the complete test will fail! Test Return Code Return Code Description check if in enuminstancenames() an failed - enumeration <source class name> - 2 exception occurred exception occurred - <wbemcli output> check if an exception occurred 12 failed - (references referencenames) - exception occurred - <wbemcli output> check if the returned instance is of the expected target class type 13 failed - returned instance(s) is/are of wrong class type <target class name> 1 ok Algorithm to Handle the Input Parameter role, resultrole and resultclass The perl function check_role_resultrole_resultclass_params is responsible for testing the algorithm, which validates the parameters role, resultrole and resultclass. associators() supports role, resultrole and resultclass associatornames() supports role, resultrole and resultclass references() supports role and resultclass referencenames() supports role and resultclass First a clarification to the meaning / handling of the input parameters assocclass and resultclass within the association operations needs to be done. The associators() / associatornames() calls return instance(s) / object path(s) of the referenced target class. The references() / referencenames() calls return instance(s) / object path(s) of the association class itself. The parameter assocclass defines the name of the association class the requester is interested to. The parameter resultclass defines the class name of the returned instance(s) / object path(s). So for referencenames() / references() the parameters assocclass and resultclass have the same meaning. Both define the name of the association class. This behavior is visible in the Client request. The Client request only supports setting the resultclass parameter for the references() / referencenames() calls. The CIMOM takes the value of the resultclass parameter and submits it as assocclass parameter to the provider. So don t be confused, when using the association provider interface. As described in the two chapters before, the support of the parameter assocclass is mandatory for the execution of the tests. That s why an additional test for this parameter is not performed. For references() / referencenames() only the role parameter is checked. The test requires, that at least referencenames() and references() on the one hand and associatornames() and associators() on the other hand use the same algorithm to validate the input parameter. Ideally all four operations would use the same algorithm. This requirement is based on the implementation of the perl function check_role_resultrole_resultclass_params. Only the referencenames() and associatornames() calls are used for the execution of the test. 18

19 The perl function check_role_resultrole_resultclass_params sends an enuminstancenames() request of the sourceclass to the CIMOM to get the object paths of all available instances. The described tests are only performed, if the source is not a CIM_<class name>. If the input file defines a list of class names starting with the common CIM_<class name>, the CIM_<class name> is jumped over and the enumeration is performed for each of the derived class names. For each object path returned by the enumeration, the following described tests against the associatornames() and referencenames() are executed. To test the algorithm the other way around, the perl function check_role_resultrole_resultclass_params switches the content of the sourceclass and the targetclass. This means the targetclass becomes the sourceclass and the former sourceclass becomes the targetclass. Then the test is performed again associatornames() For each object path returned by the enumeration, the per l function check_resultclass is called. The function tests the validation of the input parameter resultclass. If this test was successful, the perl function check_role_resultrole, to test the validation of the input parameter role and resultrole is called. The order is mandatory and based on the implementation of check_role_resultrole. The resultclass parameter in check_role_resultrole is set for each call. For completeness the perl functions check_resultclass and check_role_resultrole set the resultclass parameter to each possible targetclass. This is important, if a list of class names for a certain reference is supported by the association provider. Test check if in enuminstancenames() an exception occurred check if in associatornames() an exception occurred Return Code 2 12 Return Code Description failed - enumeration <source class name> - exception occurred - <wbemcli output > failed - exception occurred - <wbemcli output > set the value of the resultclass parameter to the target class name - check if instance(s) of the target class is/are returned set the value of the resultclass parameter to the source class name - check if no instances are returned set the value of the role parameter to the source role - check if instance(s) of the target class is/are returned set the value of the role parameter to the target role - check if no instances are returned set the value of the resultrole parameter to the target role - check if instance(s) of the target class is/are returned 17 failed - resultclass parameter seems to be ignored (1) 1 ok 18 failed - resultclass parameter seems to be ignored (2) 1 ok - no instance(s) returned 13 failed - role parameter seems to be ignored (1) 1 ok 14 failed - role parameter seems to be ignored (2) 1 ok - no instance(s) returned 15 failed - resultrole parameter seems to be ignored (1) 19

20 1 ok set the value of the resultrole parameter to the source role - check if no instances are returned 16 failed - resultrole parameter seems to be ignored (2) 1 ok - no instance(s) returned referencenames() For each object path returned by the enumeration, the perl function check_role_resultrole is called. The function tests the validation of the input parameter role. Test check if in enuminstancenames() an exception occurred check if in referencenames() an exception occurred Return Code 2 12 Return Code Description failed - enumeration <source class name> - exception occurred - <wbemcli output > failed - exception occurred - <wbemcli output > set the value of the role parameter to the source role - check if the source class plays the specified role in the association instance set the value of the role parameter to the target role - check if no instances are returned 13 failed - role parameter seems to be ignored (1) 1 ok 14 failed - role parameter seems to be ignored (2) 1 ok - no instance(s) returned Report File The report file is generated per tested association and only if the test of the algorithm was requested explicitly by calling./interface.pl <class name> -assocparams. For each tested association one report file is created in the directory stat. The name of the file is params_<class name>.stat. The file header specifies the name of the association and the start date of the test. Each interface type (associatornames() and referencenames())has its own chapter. The chapter contains information about the object path of the source class, the name of the target class and the tested input parameter. Reported are the status description of the test and the number of returned instances. Each report file ends with a summary to report an overall statistic of the tests. It contains the number of succeeded and failed tests and the times the whole test case spend in user and system mode. The report file may become very large but each chapter is self-describing and should be easy to read. Example : ****************************************************************************** class : Linux_RunningOS test algorithm to validate the input parameter role, resultrole and resultclass Fri Apr 4 8:44:

21 associatornames sourceclass : testsystem.ibm.com:5988/root/cimv2:linux_operatingsystem.cscreat ionclassname=linux_computersystem,csname=testsystem.ibm.com,creationclassname= Linux_OperatingSystem,Name= testsystem.ibm.com targetclass : Linux_ComputerSystem test content : resultclass is set to Linux_ComputerSystem Status : ok Number of returned Instances : 1 test content : resultclass is set to Linux_OperatingSystem Status : ok no instance(s) returned referencenames sourceclass : testsystem.ibm.com:5988/root/cimv2:linux_operatingsystem.cscreat ionclassname=linux_computersystem,csname=testsystem.ibm.com,creationclassname= Linux_OperatingSystem,Name= testsystem.ibm.com targetclass : Linux_ComputerSystem test content : role is set to Antecedent Status : ok Number of returned Instances : 1 test content : role is set to Dependent Status : ok no instance(s) returned associatornames sourceclass : testsystem.ibm.com:5988/root/cimv2:linux_operatingsystem.cscreat ionclassname=linux_computersystem,csname=testsystem.ibm.com,creationclassname= Linux_OperatingSystem,Name= testsystem.ibm.com targetclass : Linux_ComputerSystem test content : resultrole is set to Dependent resultclass is set to Linux_ComputerSystem Status : ok Number of returned Instances : 1 test content : resultrole is set to Antecedent resultclass is set to Linux_ComputerSystem Status : ok no instance(s) returned associatornames sourceclass : testsystem.ibm.com:5988/root/cimv2:linux_computersystem.creation ClassName=Linux_ComputerSystem,Name=testsystem.ibm.com targetclass : Linux_OperatingSystem test content : resultclass is set to Linux_OperatingSystem Status : ok Number of returned Instances : 1 21

22 test content : resultclass is set to Linux_ComputerSystem Status : ok no instance(s) returned ****************************************************************************** Linux_RunningOS Number of Tests where Status ok : 16 Number of Tests where Status failed : 0 user time : 0.01 child user time : 0.13 ut : 0.14 system time : 0.02 child system time : 0.08 st : Method Interface An automated test for Method Provider is currently not implemented, but planed. 4.6 Property Interface An automated test for Property Provider is currently not implemented, but planed. 4.7 Indication Interface An automated test for Indication Provider is currently not implemented, but planed. 4.8 Common Interface Tests Check for expected Exception The perl function check_returnedexception can be used for each supported instance and association operation. Intention is to test if a certain exception, like NOT_SUPPORTED is thrown by the provider. If the input file contains the qualifier Expected Exception for a certain operation, the perl function check_returnedexception is called and the returned value is checked for the expected exception. The perl function sends the appropriate request to the CIMOM and validates the returned value to be the expected exception. Test Return Code check if an exception occurred 8 check if an unexpected exception occurred 9 1 Return Code Description failed - <expected exception> expected check behavior of the provider by hand! failed - unexpected exception occurred - <wbemcli output> ok - operation returned with <expected exception> 22

23 5 Consistence Test 5.1 Capabilities The consistence test offers different capabilities for two different levels. It is distinguished between class specific and association specific test cases. On class level the following cases can be tested for consistence : test the number of provider reported instances against the number reported by the system itself o The number of available file system instances is a subset of the entries in the /etc/fstab and /etc/mtab (only supported fs types and without duplicates). test the property values reported by the provider against the values reported by the system itself o The provider reported Name for ComputerSystem is the hostname. The property has to contain the same value, as it is given by the hostname call. test the observance of threshold values for property values o The reported value has to be <, > or = the user defined value test the property values of a certain instance, to be consistent to each other o The processor speed of ID 0 is 930 MHz and of ID MHz. The provider has to report within the instance for ID 0 a CurrentClockSpeed of 930 MHz and nothing else. On association level the following cases can be tested for consistence : test the number of reported target instances against the association definition (system values) o The association RunningOS between ComputerSystem and OperatingSystem is one-to-one. The provider has to report one instance for both directions. test if the association provider supported associators() call reports instance(s), that is / are equivalent to the corresponding instance(s), reported by the getinstance() call against the class provider o The association provider reported a ComputerSystem instance. This instance has to be equal to the ComputerSystem instance reported by the getinstance() call against the class provider of ComputerSystem. 5.2 Input File The input file defines the classes to test. Each class / association, where the consistence test should be performed, has to have its input file in the directory system/<platform>. The name of the file has to be <class name>.system. Otherwise the consistence test is not executed for this class / association. 23

24 5.2.1 Class specific Syntax Header Syntax The key word class is mandatory and defines the name of the class. The qualifier volatile should only be used for classes, which represent volatile data, for example the class CIM_Process. The intention of this qualifier is to find a mechanism how to handle the dynamical nature of some data. It is very likely that each time you call especially the getinstance() operation of a provider for volatile data, the returned answer can vary between ok and NOT_FOUND. To avoid in principle failing the test has to be aware of this difference. ****************************************************************************** class : <class name> [ volatile ] Paragraph Syntax on Class Level Each line defines the name of the class, the compare operation to execute and the number of reported instances. Supported key words for the compare operations are : -eq ok if instance count is equal to the expected number -lt ok if instance count is less than the expected number -gt ok if instance count is greater than the expected number Supported syntax and key words for the number of reported instances are : -cmd <command to execute> command has to return a number <value> The execution of the command ( -cmd ) has to return only one value. A list of values can not be handled! <class name> : (compare operation) (expected number) <class name> : (compare operation) (expected number) Paragraph Syntax on Property Level Each line defines the name of the property, the compare operation to execute and the system value. The values given for this test case are checked at each reported instance, independent of the system object it represents. This is important to know, because checking of instance specific values is not possible with this test scenario. The next chapter describes how to check value consistence within a certain instance. Supported key words for the compare operations are : -eq ok if property value is equal to system value -lt ok if property value is less than system value -gt ok if property value is greater than system value -grep ok if property value contains a certain regular expression -set ok if property value contains a value -empty ok if property value does not contain a value 24

25 Supported syntax and key words for the system values are : -cmd <command to execute> -file <filename>[<line index>] <value> The execution of the command ( -cmd ) has to return only one value. A list of values can not be handled! If the enumeration of the class returns more than one instance, the given system value has to fulfill the compare operation (-eq, -lt ) for each instance! Otherwise the test will fail by default. The tester has to be aware of this behavior when writing the input files for the consistence test. <property name> : (compare operation) (system value) <property name> : (compare operation) (system value) Paragraph Syntax on Instance Level This test case checks the data consistence within a certain instance. It is difficult to realize and needs some special attention. Adding the corresponding paragraph to the input file is only the first of two steps. The following two lines have to be placed in the <class name>.system file. The first line defines the test type Instance Level and a script to execute. The second line describes the order for the mapping between the values in the generated system data file and the CIM properties. The compare operation is always equal. Instance Level : <full qualified script name> Property Order : <unique key> <prop1> <prop2> <propn> The second step is writing a script called <class name>.sh. The script must be located in the system/<platform> directory. The script collects the system values on instance level and generates a file <class name>.instance in the system/<platform> directory. The script has to support the option -rm to delete the generated <class name>.instance file. The content of the generated file has to be organized as following : a paragraph contains the system values of one special instance the order of the values has to follow the property order, defined in the input file of the class delimiter between paragraphs is an empty line Example ***************************************************************************** class : Linux_ComputerSystem Linux_ComputerSystem : -eq 1 CreationClassName : -eq -file ComputerSysten.keys[0] Name : -eq -file ComputerSysten.keys[1] EnabledStatus : -eq 2 Instance Level : Linux_ComputerSystem.pl Property Order : Name PrimaryOwnerName PrimaryOwnerContact 25

26 5.2.2 Association specific Syntax Header Syntax The key word association is mandatory and defines the name of the association. ****************************************************************************** association : <class name> Paragraph Syntax on Reference Level Each line defines the name of the source class, the name of the target class, the compare operation to execute and the number of reported instances. In addition of testing the number of returned instances reported by the association provider via the associators() call, these instances are compared to the instance(s) reported by the corresponding instance provider of the target class. Syntax to define the target class name is : -target <target class name> Supported key words for the compare operations are : -eq ok if instance count is equal to the expected number -lt ok if instance count is less than the expected number -gt ok if instance count is greater than the expected number Supported syntax and key words for the number of reported instances are : -cmd <command to execute> command has to return a number <value> The execution of the command ( -cmd ) has to return only one value. A list of values can not be handled! <source class name> : (target) (compare operation) (expected number) <source class name> : (target) (compare operation) (expected number) Example ***************************************************************************** association : Linux_RunningOS Linux_ComputerSystem : -target Linux_OperatingSystem eq 1 Linux_OperatingSystem : -target Linux_ComputerSystem eq Report File The report file is generated per tested class. The consistence test creates one report file in the directory stat. The name of the file is <class name>.system. 26

27 5.3.1 Class Specific Syntax The syntax of the report files is similar to the syntax of the input files. The report file contains some additional information. Each level has its own chapter. Each chapter header specifies the name of the class, the start date of the test and the tested level. Each report file contains a summary to report the overall statistic of the tests. It contains the number of succeeded and failed tests. ****************************************************************************** class : Linux_ComputerSystem Fri Apr 4 8:43: Class Level Status : ok counted instances 1 equal 1 equal ****************************************************************************** class : Linux_ComputerSystem Fri Apr 4 8:43: Property Level CreationClassName Status : ok property value was : Linux_ComputerSystem Name Status : ok property value was : testsystem.ibm.com EnabledStatus Status : ok property value was : 2 equal equal equal ****************************************************************************** Linux_ComputerSystem Number of Tests where Status ok : 4 Number of Tests where Status failed : Association Specific Syntax The syntax of the report files is similar to the syntax of the input files. The report file contains some additional information. The header of the report file specifies the class name of the association and the start date of the test. Each source to target line of the input file has its own chapter. Each header specifies the source class name and the target class name. Each paragraph contains the full qualified object path of the source instance, the return codes and status descriptions for the test to check the number of returned references and the test to compared the referenced instances with the instances reported by the getinstance() call. The report file ends with a summary to report the overall statistic of the tests. It contains the number of succeeded and failed tests. 27