An Empirical Evaluation of the PLUSS Toolkit

Transcription

1 An Empirical Evaluation of the PLU Toolkit agnus Eriksson, Henrik orast, Jürgen Börstler and Kjell Borg UINF IN UEÅ UNIVERITY Department of Computing cience E UEÅ WEDEN

2 An Empirical Evaluation of the PLU Toolkit agnus Eriksson 1, Henrik orast 2, Jürgen Börstler 3 and Kjell Borg 1 1 BAE ystems Hägglunds AB E Örnsköldsvik weden {agnus.eriksson, Kjell.Borg}@baesystems.se 2 yntell AB PO Box E tockholm weden morast@syntell.se 3 Umeå University Dept. of Computing cience E Umeå weden jubo@cs.umu.se ABTRACT The PLU approach (Product Line Use case modeling for ystems and oftware engineering) is a domain modeling method tailored towards the development of long lived software intensive systems, for example defense systems. The goal of the PLU approach is to provide means to maintain a common and complete use case model for a whole family of systems, and to provide a good overview of all variants within such a family. In this paper, we describe how the commercial requirements management tool Telelogic DOOR and the UL modeling tool IB-Rational Rose can be extended and used for managing system family models in accordance with the PLU approach. We refer to these extensions as the PLU toolkit. An empirical study is presented where the PLU toolkit is applied and evaluated in the target domain. Results indicate that the toolkit allow developers to work effectively. ome minor shortcomings of the toolkit were however identified and we therefore propose improvements to increase its efficiency. 1 INTRODUCTION oftware intense defense systems, for example vehicles, are developed in short series. They are always customized for specific customer needs and they are expected to have a very long life span, often 30 years or longer. For a business to be competitive in a market like this it is important to achieve high levels of reuse and effective maintenance. An interesting approach to address such issues is known as software product line development. The basic idea of this approach is to use domain knowledge to identify common parts within a family of products and to separate them from the differences between the products. The commonalties are then used to create a product platform that can be used as a common baseline for all products within the product family. For embedded software we believe it is important that product line concepts such as domain modeling are also introduced into the systems engineering process. This due to the fact that embedded software requirements are for the most part not posed by customers or end users, but by systems engineering and the systems architecture. Due to earlier positive single system experiences with use cases, we are interested in identifying a use case driven product line approach that can be applied by both our systems- and software engineering teams. To address issues in existing product line use case modeling approaches we have developed a domain modeling approach that utilizes features models [11], use cases [10] and use case realizations [13]. We refer to this approach as the PLU (Product Line Use case modeling for ystems and oftware engineering) approach [5]. The goal of the PLU approach is to provide means to maintain a common and complete use case model for a whole family of systems. Furthermore, the approach must be suitable for both systems- and software engineering, and provide a good overview of all variants within such a family model.

3 For software product line development to become efficient in every day practice, adequate tool support is an important factor. Unfortunately, tools for software product line development are today almost nonexistent. There are some experimental tools and very few commercial tools available. Furthermore, existing tools are not providing the functionality needed to support the PLU approach in a satisfying manner. The working group for Practices and Issues Related to Product Line Tool upport of the Fourth Product Line Practice Workshop identified a number of risks associated with using tools designed for single system development in a software product line environment [2]. The following list summarizes some of those risks: 1. There is a high risk that resources may be expended unnecessarily because of lack of adequate tool support. 2. There is a high risk that artifacts become inconsistent because tools do not enable linkage or traceability between the different types of artifacts. 3. There is a high risk that tools become difficult to maintain because of the need for local modifications to enable product line development. Even tough we recognize and agree with these risks, we have chosen to use our existing single system CAE tool environment to support the PLU approach. The two main arguments for this decision were: 1. Using the existing CAE tool environment will minimize the need for training of personnel. 2. Using commercially available tools will minimize the amount of tool code that need to be maintained. The decision has forced us to implement a number of small extensions to the tool set; we will for the reminder of this paper refer to these extensions as the PLU toolkit. Our strategy is however to keep these extensions as small as possible. The main tools used to support the PLU approach, is the requirements management tool Telelogic DOOR [18] and the UL modeling tool IB-Rational Rose [9]. Both of these tools are widely used and accepted in industry. The main contribution of this paper is an empirical evaluation of how well these tools, when using with the PLU Toolkit extensions, can be utilized to manage system family use case models according to the PLU approach. The remainder of the paper is organized as follows: ection 2 provides a short introduction to the PLU approach. ection 3 describes how we use DOOR and Rose to manage our system family models and which extensions we have implemented to better support the approach. In section 4, we present an empirical industrial study in which the PLU toolkit is evaluated in the target domain. In section 5, we summarize the paper, draw conclusions and present some future work in the area. 2 THE PLU APPROACH DOAIN ODELING WITH FEATURE, UE CAE AND UE CAE REALIZATION The concept of use cases was first introduced by Ivar Jacobson in the early 1990s and has today become a widely accepted and used requirement modeling technique, both in the software- and the systems engineering communities. Use case modeling is a functional decomposition technique that provides a semi formal framework for structuring requirements. A good use case should address one complete and well-defined goal that the user wants to accomplish by interacting with the system [1]. As we described in [5], it must be possible to manage variability among family members to be able to maintain a common use case model for a whole system family. Our approach for

4 managing variability in PLU is based on the work by Griss et al. on FeatuREB [8]. In FeatuREB a feature model is added to the 4+1 view model adopted by Jacobson et al. in REB [10]. The feature model in FeatuREB takes center stage and provides a high-level view of the domain architecture and the reusable assets in the product family. Kang et al. first proposed using feature models in 1990 as part of Feature Oriented Domain Analysis (FODA) [11]. In feature models, system features are organized into trees of AND and OR nodes that represent the commonalities and variations within a family of related systems. General features are located at the top of the tree and more refined features are located below. Originally, FODA described andatory, Optional and Alternative features that may have requires and excludes relations to other features. andatory features are available in all systems built within a family. Optional features represent variability within a family that may or may not be included in products. Alternative features represent an exactly-one-outof-many selection that has to be made among a set of features. A requires relationship indicates that a feature depends on some other feature to make sense in a system. An excludes relationship between two features indicates that both features can not be included in the same system. Like Griss et al. we argue that feature models are better suited for domain modeling than for example UL use case diagrams and that a feature model therefore should be used as the high level view of the product family. However, in PLU the primary purpose of the feature model is not to take center stage, but rather to be a tool for visualizing variants in our abstract product family use case model. We use the feature model as a tool for structuring and instantiating our abstract family models into concrete product use case models for each system built within the family. We also maintain use case realizations [13] and change cases [3] as part of the PLU system family model. We utilize use case realizations to trace variant use case behavior to the system design, and change cases to mark proposed however not yet accepted functionality in the domain 2.1 PLU odeling Notations PLU Use Case odeling As described in [4,5] and shown in Figure 1 and Figure 2, we have chosen to adopt an extended version of the RUP-E Flow of Events notation [16] for describing use case scenarios and use case realizations. We have chosen natural language descriptions since the PLU approach must be applicable for both systems and software engineering. This increases the number and diversity of stakeholders interested in the models and thereby makes for example UL unsuitable for the purpose. Our natural language descriptions can however be supplemented with UL diagrams to improve understandability as needed. As shown in Figure 1, the extension provides means to specify variants of use case scenarios in a very compact way. As shown in step 3c and step 5b of Figure 1 the PLU notation also provides means to specify parameters within use case descriptions. Like annion et al. [14], we distinguish between local parameters and global parameters. In PLU, the scope of a local parameter is the use case in which it resides and the scope of a global parameter is the whole domain model. Like annion et al. we use the symbols $ respectively to denote local and global parameters.

5 andatory step Exactly one to be selected for a mandatory step At least one to be selected for a mandatory step Optional step At least one to be selected for an optional step Exactly one to be selected for an optional step tep Actor Action 1 The Actor a 3b 3c (4) (5)a (5)b (5)c (6) (6) (6) Blackbox ystem Response The ystem Blackbox Budgeted Req. It $PARA_2 Figure 1: The PLU notation for describing variants in use case scenarios tep Actor Action The Actor The use case ends when Blackbox ystem Response The ystem Blackbox Budgeted Req. It shall Whitebox Action DesignElement_1 DesignElement_2 DesignElement_3 Whitebox Budgeted Req. It shall Figure 2: An Example Use Case Realization in the PLU Notation In PLU, change cases are modeled in UL use case diagrams utilizing the UL stereotype mechanism [15]. As shown in Figure 3, we stereotype use case as <<change case>> and the association relation as <<impact>> to visualize the needed entities. <<include>> Crew ember View Video <<extend>> elect Video ource <<impact>> <<change case>> View Picture-in-Picture Video Figure 3: An example Change case in UL

6 2.1.2 PLU Feature odeling Original FODA has no defined mechanism to specify the relation at-least-one-out-of-many [6]. We address this, according to our experience, important shortcoming by defining a new feature type called ultiple Adaptor feature in PLU. This feature type is similar to FODA s alternative features, but instead of representing the exactly-one-out-of-many relationship, it captures the missing relationship. Its name follows the naming scheme proposed by annion et al. for the equivalent relation in their work on reusable requirements [14]. We have also chosen to rename alternative features to ingle Adaptor features following the same naming scheme. The feature modeling notation used in PLU is based on the FODA notation but it has been slightly modified to better suit our modeling needs as shown in Figure 4. As in the original notation a filled black circle represents a mandatory feature and a non-filled circle represents an optional feature. ingle and multiple adaptor features are represented by the letters and, respectively, surrounded by a circle. ingle and multiple adaptor features are also colorcoded in the PLU notation to ease identification in large feature graphs. Domain a <<requires>> b aa ab ac ba bb bc aaa aab aac aba <<excludes>> abb bba bbb bca bcb bcc andatory Optional ingle Adaptor ultiple Adaptor Requires Excludes Figure 4: A Feature model example in the PLU notation In the original FODA notation an arc connects sets of alternative features. Due to limitations in our tool support we had to remove this arc from the PLU notation. Instead we imposed a modeling constraint saying that only one set of single adaptor and multiple adaptor features respectively may exist under the same parent feature. This was necessary to avoid ambiguity in models regarding which set of single or multiple adaptor features a certain feature belongs to. If several sets are needed under the same parent, each set must be grouped under their own parent feature. 2.2 Relating Features and Use Cases Gomaa [7] proposed to model each feature as a use case package. We have extended this idea saying that possibly a whole set of features compose a use case package. This have the advantage of enabling us to also visualize variants within use cases specifications using the feature model. As shown in the example in Figure 5, use cases as well as use case scenarios can be mapped to features of any type in the feature model to capture required variants among the members

7 of a system family. As shown in the example in Figure 6, there is also a predefined set of feature constructs that should be mapped against each type of variant in the use case scenario notation described in section This means that we are actually maintaining redundant information in our notation. We do however consider this to be a minor problem since an automatic consistency check between the models could easily be implemented and we believe the extra information considerably improves the readability of the notation. As shown in the example in Figure 6, also use case parameters are mapped to the feature model. A local parameter is mapped to a set of features on an appropriate level below the feature holding the use case owning the local parameter in the feature tree. A global parameter is mapped to a set of features on an appropriate level above the feature holding the highest level use case referencing the global parameter. Optional Optional DomainFeature: Feature: ab ab a b <<extend>> aa ab ba <<change case>> bb bc <<impact>> aaa aab aac aba abb bba bbb bca bcb bcc Use Use Case: Case: <X> <X> Introduction Introduction <ome info...> <ome info...> ain uccess cenario ain uccess cenario.. Alternative cenarios Alternative cenarios <cenario Name 1> <cenario Name 1>.. <cenario Name 2> <cenario Name 2>.. Exceptional cenarios Exceptional cenarios <cenario Name> <cenario Name>.. Figure 5: An Example of the Relationship between Features, Use Cases and Use Case cenarios tep Actor Action 1 The Actor a 3b 3c (4) (5)a (5)b (5)c (6) (6) (6) Blackbox ystem Response The ystem Blackbox Budgeted Req. It $PARA_2 Figure 6: An Example of the Relationship between Features Variant cenario teps in the PLU notation

8 A meta-model for integration of features, use cases and use case realizations is presented Figure 7. It describes how use cases, scenarios and scenario steps are included by feature selections. It also shows how the included use case scenario steps prescribes a certain set of design elements via use case realizations. Variant use case behavior is thereby traced to the system design. Parameter 0..1 instantiate Global Parameter Local Parameter Use Case include 1 1 Use Case Realization 1.. require Design Element cenario include 0..1 realize Feature tep include 0..1 include composed-of 1.. ystem 2.. exclude refine ystem Family 1 1 Domain odel Figure 7: The PLU eta-model in UL [5] 2.3 Product Instantiation in the PLU Approach ince we are maintaining a common use case model for the whole system family in PLU, product instantiation is basically done by adding any new requirements to the model and then using the feature model to choose among its variants. The set of included features directly corresponds to a specific set of included use cases for the product. A product use case model is then generated from the domain model by applying a filter, sorting out features not included in the current system. This will result in three types of generated reports: A Use Case odel urvey including all use cases for the current product, Use Case pecifications for all use cases in the survey, and Use Case Realizations for all use cases in the survey. 3 PLU TOOL UPPORT We use Telelogic DOOR to manage our system family use case models according to the PLU meta-model and we use IB-Rational Rose for drawing feature graphs and UL diagrams. We then generate appropriate reports from our domain model as Word documents that are publish to the development teams in our oftware Configuration anagement (C) system as shown in Figure 8.

9 Feature odel, Use Case pecifications, Change Case pecifications and Use Case Realizations DOOR Feature Graph and UL Diagrams Rose Repository of Published Reports Word Reports oftware C ystem 3.1 Telelogic DOOR Figure 8: Overview of CAE Tool Environment Telelogic DOOR is an object database intended for requirements management. DOOR could be described as a document oriented database since all objects are contained in modules that very much appear like documents to the user. Objects may be arranged in a hierarchic structure in modules. Typically, node objects are used as headings and leaf objects are used for data items such as requirements. Like in a document, both headings and text are usually displayed in a single column in DOOR. This column is referred to as the main column. It is possible to define attributes on both module and object level in DOOR. An object attribute holds a value for each object in a module. odule attributes can be used for storing additional data not related to specific objects in modules, for example module status. An attribute definition specifies what kind of information an attributes can store. It is for example possible to define multi-valued enumerated attributes that may have any number of items set in an enumeration. DOOR has a link concept that can be used for defining relationships between any two objects within the database. Links form the basis for traceability in DOOR. Rules can be set up for controlling what kinds of relationships are allowed, based on which modules objects belong to. Object attributes can be displayed in columns within a module in the DOOR graphical user interface. Combining this possibility with DOOR support for filtering on properties of objects, a user may define views suiting different needs. These views are used when working with data and also for reporting and exporting information to other tools. Views may be saved in a module for later use. 3.2 Feature odeling in DOOR Feature models are managed in specific DOOR modules. The basic idea is to use the standard DOOR object hierarchy to capture the refine relation used for building the feature tree structure. Each feature becomes a heading and leaf objects are used for capturing different kinds of information regarding each feature. A number of attributes are defined in a feature model module:

10 A Characteristics attribute is used for classifying each object in the module as: General Information, a Feature, a Feature description, a Feature graph or a Use Case Diagram. General information objects are typically used for introductory information in the domain model, such as use case actor descriptions etc. A Feature Type attribute is used for classifying features in the feature hierarchy. It can take the values mandatory, optional, single adaptor or multiple adaptor. A Require attribute and an Exclude attribute is used to capture the requires and excludes relation between features. A reference holding the identity of the required or excluded feature objects are used for this. These references are then shown as the names of the referenced features in the user interface using DOOR dynamic columns [19]. A Use Case Package attribute is used when filtering out a use case model survey from the domain model. As mentioned in section 2.2, a sub-tree of the feature model typically corresponds to a use case package for a specific system within a system family. This attribute is used for marking such sub trees in the model. A Product Instances attribute is used to mark features as included or not by systems within a system family. This information is managed with a multi-valued enumeration attribute that can assume the names of all systems in the family as shown in the rightmost column of Figure 10. This information is then used for filtering out product specific use case models for the different systems in the family. Equivalent information can however also be captured as incoming links from higher level specifications to feature objects. Which method to use, depends on the requirements for traceability to higher level specifications in the specific project. Figure 9: The DOOR Domain odel View (note that project specific info. is blurred)

11 3.3 Feature Graphs in Rose To be able to draw feature graphs, we utilized the UL stereotype mechanism [15] as follows: Icons for all PLU feature types were created. These icons were then associated with UL Classes that had been stereotyped to the different feature types in Rose. The UL Dependency relation was stereotyped as require. The UL Association relation was stereotyped as excludes. Furthermore, standard UL classes are used for visualizing domain names and standard UL association relations are used for visualizing the refinement relation in Rose. We draw these feature graphs in use case diagrams, and we have therefore also added the required feature modeling elements to the use case diagram toolbar in Rose as shown in Figure 9. Figure 10: The Rose Feature odeling View (note that project specific info. is blurred) 3.4 Use Case pecifications and Use Case Realizations in DOOR Each use case is managed as a separate module in the DOOR database. Introductory information is captured in a traditional DOOR document structure with headings and text in the DOOR main column. To capture use case scenarios and use case realizations in DOOR, we use one DOOR object to describe each scenario step. To be able to do this, object attributes for the tep, Blackbox ystem Response, Blackbox Budgeted Req., Whitebox Action and Whitebox Budgeted Req. columns must be defined in DOOR use

12 case modules. For the Actor Action column, the DOOR main column is used. The relationship between blackbox and whitebox scenario steps (realizations) is managed by making whitebox step objects children to their corresponding blackbox step object using the standard DOOR object hierarchy. This means that use case specifications and the use case realisation are actually different views of the same DOOR module. 3.5 Relating Features and Use Cases in DOOR We relate features and use cases in accordance with PLU meta-model shown in Figure 7, as follows in DOOR: A feature can be related to zero or more complete use cases. We manage this relation in DOOR by inserting a UL use case diagram showing those use cases as leaf objects in a feature model module under a feature object. A feature can be related to zero or more use case scenarios. We manage this relation in DOOR by creating an outgoing link from a heading object naming a scenario in a use case module, to a feature object in a feature model module using the DOOR link concept. A feature can be related to zero or more scenario steps. We manage this relation in DOOR by creating an outgoing link from a scenario step object in a use case module to a feature object in a feature model module, using the DOOR link concept. As shown in Figure 6 we do not relate mandatory scenario steps to the feature model. The same is also true for mandatory use case scenarios. This is not needed since the primary purpose of the feature model is not to provide a total view of a system family, but rather to visualize variants in the family use case model. 3.6 DOOR Extensions for the PLU Approach DOOR extension Language (DXL) [19] is an integrated part of the DOOR product. DXL is a scripting language based on the C/C++ notation that can be used for customizing DOOR according to special needs. DXL supports all kinds of manipulation of the DOOR database, from simple reading and modifying data to advanced database administrative tasks. DXL also support creation of user interfaces. Interactive, dynamic dialogue boxes can be created and it is possible to add new menus in the standard DOOR user interface. This provides a possibility to create integrated customized DOOR tools. DXL can also be used for controlling other Windows applications that provide automation interfaces. This possibility is utilised for example in the DOOR standard export to Word and Excel. The following sections briefly describe some DXL tools that were developed as part of the PLU toolkit. In total, approximately lines of low complexity DXL code was developed to extend DOOR to better support the PLU approach Check Feature odel The purpose of the Check Feature odel tool is to help the user with a basic consistency check of feature model modules. The tool verifies that a number of rules regarding feature model modules are fulfilled; any violations of the rules are presented to the user in the graphical user interface. The tool verifies that the following set of conditions hold: Only heading objects are marked as features. All features objects have a defined feature type.

13 No require relations exist within a set of single adaptor features. Only feature objects have require relations. Only feature objects have exclude relations. All exclude relations are bi-directional. No contradictions exist regarding exclude relations and mandatory features. No exclude relations exist between ancestors and descendant in the feature hierarchy. No contradictions exist regarding exclude relations and require relations Check Configuration The purpose of the Check Configuration tool is to help the user verify that a specific product instance of the domain model does not contradict the rules of the feature model. Any violations of the rules are presented to the user in the graphical user interface. The tool verifies that the following set of conditions hold: All mandatory features with included parent features are included in the configuration. All ancestors to included features in the feature hierarchy are included in the configuration. The configuration does not violate exclude relations. The configuration does not violate require relations. The configuration does not violate the rules of single adaptor features. The configuration does not violate the rules of multiple adaptor features Illustrate Feature odel Hierarchy The purpose of the Illustrate Feature odel Hierarchy tool is to illustrate a portion of the feature model hierarchy in the DOOR graphical user interface. From a chosen feature object in a feature model module, a feature tree with all ancestors of the selected feature is drawn in a dialog box. The tool provides possibilities to collapse and expand the nodes of the tree. The tool also provides the user with an option to insert the illustration as a picture in a new DOOR object below the chosen feature object Create Use Case odule The purpose of the Create Use Case odule tool is to provide the possibility to automatically create use case modules with the needed structure, attributes and views predefined. The tool accomplishes this by copying and renaming a predefined use case template module with the required properties. This tool helps to align the way different projects manage use cases in DOOR and it reduces the modeling effort needed Create Product Use Case odel urvey The purpose of the Create Product Use Case odel urvey tool is to provide the possibility to filter out a use case model survey for a specific product instance from a feature model module. The tool accomplishes this by removing, from the selected view, those objects that does not fulfill each of the following conditions:

14 If an object is marked as General Information, it must be either a descendant of an included Feature object or marked as included by the specified product instance itself. If the object is marked as a Feature, it must also be marked as included by the specified product instance and be marked as being a Use Case Package. The name of these features then becomes headings in the use case model hierarchy in the survey. If the object is marked as a Use Case Diagram, it must be a descendant of a Feature object marked as included by the specified product instance. The resulting view can then be exported into a Use Case odel urvey Report for the selected product instance using the standard DOOR Word export Export Use Case The purpose of the Export Use Case tool is to provide the possibility to export use case modules from DOOR to Word as use case specifications and use case realizations using the PLU notation. The Export Use Case tool is based on the standard DOOR Word export which is also written in DXL. The basic idea of the tool is to export blackbox and whitebox scenarios as specially formatted tables and all other information as ordinary headings and text. To be able to recognize scenarios within use case modules, the tool assumes that each scenario step have a step identifier as prescribed by the PLU notation Import Use Case from Word The purpose of the Import Use Case from Word tool is to provide the possibility to import use cases written in Word to DOOR and format these use case according to the DOOR use case template. This tool utilizes the DOOR standard Word import and then performs some post processing to structure the module contents according to the company standard. 4 EPIRICAL EVALUATION OF THE PLU TOOLKIT This empirical tool evaluation was designed as a blocked subject-project study [17]. The objective of the study was to evaluate how well the existing CAE tool environment, extended with the PLU toolkit, supported the PLU approach. A number of response variables relevant for measuring the performance of the tool environment were identified during the study design. One important input to the identification of these measures was the risks related to using tools designed for single system development discussed in section tudy Context The study was preformed with the wedish defense contractor BAE ystems Hägglunds AB, which is a leading manufacturer of combat vehicles, all terrain vehicles and a supplier of various turret systems. BAE ystem Hägglunds AB develops software according to a tailored version of the IB-Rational Unified Process (RUP) [13]. RUP is a use case driven software development process, widely used in industry. The study included data collection from two different projects including subjects form both

15 systems- and software engineering teams. The first project regarded development of a turret system and the second project regarded development of a vehicle system. Both systems were considered to be of typical size and complexity for the organization. All subjects had previous experience and/or formal training in using the CAE tools utilized in the study. 4.2 ethod The study involved collecting four different types of data: The first type of data was collected by examining documentation [17]. The resulting modeling artifacts were inspected by the research team to identify possible common modeling errors that could be related to poor tool support. The research team also examined inspection records for the resulting artifacts to identify possible common errors that were captured during inspections and reviews. The second type of data was collected by participant observation [17]. The research team participated in a number of modeling sessions to get first hand information regarding possible problems encountered by the teams using the tools. The research team also attended a brainstorming session where problems with existing tools and needed improvements were discussed by the developers. The third type of data was collected through questionnaires [12]. The main topics of the questionnaires were experiences working with the tools and descriptions of possible extensions to the tools created by developers. Questionnaires were designed to have both specific and open ended questions to also elicit unexpected types of information. Questionnaires were also used for eliciting information regarding subject attitudes towards the modeling notations used and information regarding the amount of experience of each subject using the tools. This information was later taken into account during data analysis. The final type of data was collected trough interviews [17]. emi-structured interviews were performed with team members and main topics of these interviews were related to how the risks mentioned in section 1 applied to the way single system tools are used in the PLU approach. The different types of data collected were first analyzed individually to find patterns and trends in the responses. The different types of data were then analyzed all together and conclusions were drawn. 4.3 Threats to Validity To minimize threats to the study s construct validity, data was collected with several different methods that also allowed unexpected types of information to be elicited. To minimize threats to the study s internal validity, the case study project was staffed using the organizations normal staff-allocation procedures. Furthermore, subject attitudes regarding notations and their level of experience using the tools were taken into account during data analysis to minimize the effect of these possibly confounding factors [12]. To minimize threats to the study s external validity, the study was conducted in the target domain of extremely long-lived software intense systems and projects were selected to be of typical size and complexity for the organization [12]. To minimize threats to the study s conclusion validity, results were triangulated by collecting data with four different methods from several different sources. Furthermore, results were discussed with the teams to assure that their opinions were represented correctly [19].

16 4.4 Results Documentation Examination Document examination reviled that four types of modeling errors existed in the resulting documentation that could have been automatically detected. Instead these errors were now being captured during inspections, which is more costly. The following list summarizes these errors: tyle violations regarding how use case scenarios should begin and end. Whitebox Budgeted Requirements not derived from Blackbox Budgeted Requirements. Use case scenario step numbering errors. Variant use case scenario steps not related to features in the feature model Participant observation Participant observation reviled that shortcomings in the graphical user interfaces of both DOOR and Rose were sources of irritation for all teams involved in the evaluation. DOOR lack of shortcut keys, inability to remember previous selections in dialog boxes and unpredictable behavior regarding screen focus are examples of such shortcomings. Rose appeared unpredictable in its behavior regarding its auto-layout functions and also regarding the way diagrams were redrawn when new modeling elements were added to them. Besides from irritation, these problems also caused extra time to be spent on workarounds. One example of such a workaround was a Visual Basic application written by one subject that automatically filled out the DOOR Word export dialog with the correct information whenever it appeared on the screen. Another example was the need to manually move objects around in Rose diagrams to force redrawing of objects to remove spurious drawing artifacts. Participant observation also reviled that synchronization between DOOR and Rose was a fairly time consuming and to some extent error prone activity. The synchronization consisted of mainly two tasks: drawing and updating a feature graph in Rose according to the underlying feature model maintained in DOOR, and inserting UL use case diagrams drawn in Rose as pictures in a feature model module maintained in DOOR. The DXL tool Illustrate Feature odel Hierarchy was meant to relieve the teams of the first of these tasks. Participant observation did however show that subjects preferred to draw feature graphs in Rose instead of using the tool. The reason was that the DXL tool did not implement any algorithm to minimize the width of the tree, but simply put all features on a specific level on same height in the graph. This resulted in very wide graphs that could not easily be printed on paper. A problem pointed out during the tool support brainstorming meeting was that the DOOR Word use case export was not good looking and that it required manual fixing before complying with the company standard outline. This problem had however already been address by one of the subjects who created a VBA script in his Word use case export template that automatically formatted the DOOR output according to the company standard. This template was later also distributed to the other subjects. Another shortcoming of the tool environment pointed out during the brainstorming meeting was the inability to export whole set of use cases based on certain criteria. The teams had experienced a need to be able export for example all use case in the model, use cases that had been changed since the last export and a manually selected subset of a use case model. A DOOR DXL extension with this

17 functionality was later implemented and distributed to the subjects. One team also expressed a need for stronger means to keep track of the status of each use case specification during the brainstorming meeting. The team had also already implemented a simple DXL tool addressing this shortcoming. The tool extracted status information, for example if the use case was ready for review or approved, from the use case model and then exported this information to Excel. Excel then presented the information graphically using a macro defined by the team. Participant observation also revealed that one subject had developed a method to automatically extract and insert UL sequence diagrams created in Rose in their use case realizations exported from DOOR. This feature was implemented by adding textual address tags to DOOR use case modules were they wanted diagrams to appear. These use case modules were then exported using a Word template including a VBA script which started Rose and used the Rose API to extract and replace the textual tags with the required pictures Questionnaires Also questionnaires indicated that shortcomings in the graphical user interfaces of DOOR and Rose caused irritation among the subjects. A need for better integration between the tools were also expressed in questionnaires, subjects considered it to be much work to first draw diagrams in Rose and then manually insert them in DOOR. Questionnaires did however also show that subjects considered the tools to have a number of positive characteristics. They considered the tools to be easy to use and very flexible, and that this allowed for good organization of the models. They also considered it to be positive that the existing CAE tool environment was utilized, since it made it easier to get started and reduced the need for training. ubjects also considered Rose to work well for drawing feature graphs. As general summary of the questionnaires, subjects felt that the tool environment worked well when having access to the PLU toolkit. On the question what they would like to see in the ideal tool, they did however come up with the following suggested improvements: A WYIWYG function that enable printing of nice looking specifications in accordance with the PLU notation directly from DOOR, without first having to export to Word. The possibility to draw UL diagrams directly in DOOR. A stronger model checking function that verifies that rules of the model are not violated Interviews Interviews showed that subjects were satisfied with the overall performance of the tools. An initial resource investment had been required to adjust the tools to match the process. However at the time of the study, subjects felt that the tool environment enabled them to work effectively. Interviews also showed that subjects considered it to be a risk that the tools were not integrated enough and that this might be a cause for artifacts to become inconsistent. ubject also felt that it might be confusing to people that models are versioned in DOOR and that the C system is only used for publishing reports. ubjects did however feel that these issues should not cause any major problems, as long as clear instructions on how to use the tools are

18 available. Furthermore, interviews showed that subjects did not consider the PLU toolkit to pose a major maintenance problem since the extensions are of relatively small size and low complexity. One subject did however see a risk that the organization might be afraid of moving to newer and better releases of the tools, because of the risk for problems with the extensions. ubjects furthermore considered it to be important to have DXL knowledge inhouse for future maintenance issues. 5 UARY AND CONCLUION We have described how commercially available CAE tools, traditionally used in single system development, can be extended and utilized for managing system family models in accordance with the PLU approach. The described tools were applied and evaluated in an empirical study in the target domain. The study showed that subjects considered the tools to allow them to work effectively and that the local extensions should not pose a major maintenance problem. Problems with the graphical user interfaces of both DOOR and Rose were however sources of irritation among the subjects. ince the study indicated that the PLU toolkit allowed subjects to work effectively, we will focus our future work on improving the existing environment rather than replacing it. To avoid our tool extensions becoming a major maintenance issue, we do however want to continue keeping them as small as possible. Based on the study results, we have identified the following three prioritized improvements to the toolkit: A Use case model checker that notifies developers of the different types of common modeling errors that were identified during document examination. An improved version of the Illustrate Feature odel Hierarchy tool that implements an algorithm to minimize the width of feature graphs. This would likely lead to more widespread use of the tool among the developers. This would in turn remove some of the effort needed for manual synchronization between DOOR and Rose. An improved version of the Create Product Use Case odel urvey tool that automatically extracts UL use case diagrams from Rose. This would remove much of the effort needed for synchronization between DOOR and Rose. Another way to address this problem would be to evaluate if the commercial product DOOR Analyst [18] would resolve this issue. The DOOR Analyst product enables drawing of UL diagrams directly in DOOR 6 ACKNOWLEDGEENT The authors would like to thank all the people at BAE ystems Hägglunds AB who contributed to this work. REFERENCE [1] Adolph., Bramble P., Cockburn A., Pols A.: Patterns for effective Use Cases, Addison-Wesley (2003) [2] Bass L., Clements P., Donohoe P., cgregor J. Nortrop L.: Fourth Product Line Practice Workshop Report, Technical Report CU/EI-2000-TR-002, oftware Engineering Institute, Carnegie ellon University, Pittsburgh, PA (2000) [3] Ecklund E., Delcambre L., Freiling.: Change Cases - Use Cases that Identify Future

19 Requirements, Proceedings of OOPLA 96, an Jose, Ca, October 6-10, (1996) [4] Eriksson., Börstler J., Borg K.: arrying Features and Use Case for Product Line Requirements odeling of Embedded ystems, Proceedings of the Fourth Conference on oftware Engineering Research and Practice in weden ERP 04, Institute of Technology, UniTryck, Linköping University, weden (2004) [5] Eriksson., Börstler J., Borg K.: The PLU Approach Domain odeling with Features, Use Cases and Use Case Realizations, To appear in the proceedings of the Ninth International oftware Product Line Conference, (2005) [6] Fey D., Fajta R., Boros A.: Feature odeling: A eta-model to enhance Usability and Usefulness, Proceedings of the International Conference on oftware Product Lines, Lecture Notes in Computer cience, Vol. 2371, pringer-verlag, (2002) [7] Gooma H. Designing oftware Product Lines with UL From Use Cases to Pattern- Based oftware Architectures, Addisson-Wesley (2004) [8] Griss., Favaro J., d Alessandro.: Integrating Feature odeling with the REB, Proceedings of the Fifth International Conference on oftware Reuse, Vancouver, BC, Canada, (1998) [9] IB-Rational oftware, Available at: (2005) [10] Jacobson I., Griss., Jonsson P.: oftware Reuse Architecture, Process and Organization for Business success, Addison-Wesley (1997) [11] Kang K. Cohen., Hess J., Novak W., Peterson A.: Feature Oriented Domain Analysis (FODA) Feasibility tudy, Technical Report CU/EI-90-TR-021, oftware Engineering Institute, Carnegie ellon University, Pittsburgh, PA (1990) [12] Kitchenham B., Pickard L., Pfleeger.: Case tudies for ethod and Tool Evaluation, IEEE oftware, Vol. 12 Nr. 45 (1995) [13] Kruchten P.: The Rational Unified Process - An Introduction, econd Edition, Addison- Wesley (2000) [14] annion., Lewis O., Kaindl H., ontroni G., Wheadon J.: Representing Requirements on Generic oftware in an Application Family odel, Proceedings of the International Conference on oftware Reuse ICR-6 (2000) [15] Object anagement Group: Unified odeling Language Version 2.0, Available at: (2005) [16] Rational oftware: The Rational Unified Process for ystems Engineering Whitepaper, Ver. 1.1, Available at: (2003) [17] eaman C.: Qualitative ethods in Empirical tudies of oftware Engineering, IEEE Transactions on oftware Engineering, July/August (1999) [18] Telelogic AB, Available at: (2005) [19] Telelogic AB, DXL Reference anual, DOOR 7.1 anual creation date: (3 ay 2004)