Rational ClearQuest Release Report

Transcription

1 Rational ClearQuest Release Report Dated: 25 July, 2013 IBM Corporation 2013

2 Trademarks IBM, the IBM logo, and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at The listed trademarks of the following companies require attribution: Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. ITIL is a registered trademark and a registered community trademark of The Minister for the Cabinet Office, and is registered in the U.S. Patent and Trademark Office. UNIX is a registered trademark of The Open Group in the United States and other countries. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and other countries. Other product and service names might be trademarks of IBM or other companies. Dated 25-July-2013 Printed 25 July 2013 Page 2 of 20

3 Contents 1 Introduction Rational ClearQuest Version Feature Business Value Enhanced usability Multirecord Update Create Query from Selection Visible feedback on in-progress data modifications New Save and Close button Actions that users do not have permission to perform can be hidden by the schema designer Improved full text search capabilities by deploying Apache Solr ClearQuest Web Security Enhancements Improved smartcard authentication support for the integration between ClearCase and ClearQuest Secure notification with the new SMTP Relay transport in the Plus package Timely security updates ClearQuest Web integrations enhancements Prefilling OSLC 2.0 linked records New OSLC link types Support for integrations with IBM Rational Lifecycle Adapters Schema designer enhancements ratlperl and CQPerl are upgraded Hide actions that users do not have permission to perform Configure immediate or queued notification Testing and Methodology Metrics APAR and Defect Fixes Testing Strategy Functional Verification Testing Coverage Systems Verification Testing Coverage In-House Deployment Testing Beta Testing Key Process Improvement Areas ClearQuest Web Performance, Scalability and Availability ClearQuest Web Performance ClearQuest Web Scalability ClearQuest Web Availability Customer Feedback The Design Partner Program Beta Program Future Feedback Dated 25-July-2013 Printed 25 July 2013 Page 3 of 20

4 1 Introduction This document provides a high-level overview of the content of the IBM Rational ClearQuest release, and highlights the value of this release to IBM Rational customers. Rational ClearQuest Version focuses on the following areas: Business Value: Improving features and addressing Requests For Enhancement (RFEs) to benefit our customers. Testing Methodology and Metrics: Testing the code so that it is designed to meet feature requirements and does not have regressions in the performance of previous releases. System Performance: Providing that the Rational ClearQuest release is scalable, reliable and meets performance requirements. Customer Feedback: Working with customers to understand and incorporate their feedback into Rational ClearQuest to better meet business needs. Disclaimer Any data contained herein, including performance-related data, were ascertained in a controlled environment. The results obtained in other operating environments may vary significantly. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Any test results are specific to the product software, test configuration, workload, and environment that were used. Product performance in other environments or conditions may be different compared to the data reported in this document. Statements regarding IBM s future direction and intent are subject to change or withdrawal without notice and represent goals and objectives only. Dated 25-July-2013 Printed 25 July 2013 Page 4 of 20

5 2 Rational ClearQuest Version Feature Business Value To succeed, a product must provide business value to customers. The following aspects of enhancing business value were the focus of this release: The following sections detail the features included in the release, 2.1 Enhanced usability ClearQuest includes the following usability and productivity enhancements for ClearQuest Web users: New implementation of the multirecord-update feature The multirecord-update feature in ClearQuest Web is redesigned and enhanced to improve the efficiency of large-scale changes. New Create Query from Selection feature The Create Query from Selection feature lets you create a new query by selecting a subset of records in a result set. Visible feedback on in-progress database modifications Prevent data loss with better cues and notifications of inadvertent tab or browser closure New Save and Close button Streamlined interaction with hiding of actions that the user is unable to perform Actions that users do not have permission to perform can be hidden by the schema designer Dated 25-July-2013 Printed 25 July 2013 Page 5 of 20

6 Improved Full Text Search capabilities by deploying Apache Solr version Increased search flexibility for analyzing, tokenizing and streaming data by deploying Solr version in your full text search installation Multirecord Update The multirecord-update feature in ClearQuest Web is redesigned and enhanced to improve the efficiency of large-scale changes. Notable additions include: o o o o Support for recording state changes Ability to run a multirecord update on more than the 100 records that display on a result set page Ability to create a query by selecting records from a result set with the Create Query from Selection command Implementation of a new user interface Tune in to the IBM Rational Support YouTube channel ( to view videos of the multirecord update feature. For more information on this feature, see the Modifying multiple records using multirecord update ( c%2ftopics%2fc_modifying_multi_recs.htm) section of the IBM Rational ClearQuest Information Center Create Query from Selection The Create Query from Selection feature lets you create a new query by selecting a subset of records in a result set. The new query presentation has same display fields and sort order as the original query. You can use this new feature to select records for a multirecord update operation in ClearQuest Web Visible feedback on in-progress data modifications ClearQuest Web includes better visual cues and notifications of inadvertent tab or browser closure. o o The tab label displays in red font with a preceding asterisk when you have in-progress modifications that are not saved. A warning dialog box displays when you have in-progress modifications and you attempt to close a tab or browser window, use the back or forward browser buttons, refresh the page, or navigate to a different URL New Save and Close button A new Save and Close button allows users to save and close a record in a single action. Alternatively users can click on the Save button which saves the record but leaves it open for further updates. Dated 25-July-2013 Printed 25 July 2013 Page 6 of 20

7 2.1.5 Actions that users do not have permission to perform can be hidden by the schema designer The schema designer can hide actions that the user cannot perform to streamline user productivity. For more information, see the Schema designer enhancements section (see Hide actions that users do not have permission to perform) Improved full text search capabilities by deploying Apache Solr ClearQuest includes version of Apache Solr. This version of Solr offers support for 30 languages. Additionally this version increases flexibility for analyzing, tokenizing and streaming of data. For example, you can search using a wildcard at the beginning of the string, or you can filter out HTML encoding within your search. For more information, see the Upgrading Solr to the latest version supported by ClearQuest section of the IBM Rational ClearQuest Information Center. ( oc%2ftopics%2ft_fts_scn_solr_upgr_latest_ver.htm). For more information on Apache Solr, see the following links: Solr wiki Analyers, Tokenizers, and Token Filters ( Apache Solr ClearQuest Web Security Enhancements ClearQuest provides the following security enhancements to ensure process and data integrity Improved smartcard authentication support for the integration between ClearCase and ClearQuest The integration between ClearCase and ClearQuest with smart card authentication is supported by the Clear- Case Change Management Integration (CMI). ClearCase communicates with ClearQuest through the ClearQuest Web OSLC interface. For more information, see Technote , Configure the base ClearCase-ClearQuest integration to use the Change Management Integration ( Secure notification with the new SMTP Relay transport in the Plus package The IBM Rational ClearQuest Relay service implements a private relay service to forward requests to a server that is registered for Simple Mail Transfer Protocol (SMTP) server access. For more information, see Mail relay ( oc%2ftopics%2fsch_pkgs%2fc_emp_mail_relay.htm Dated 25-July-2013 Printed 25 July 2013 Page 7 of 20

8 2.2.3 Timely security updates IBM provides timely security vulnerability updates on the IBM Support home page for Rational ClearQuest ( and through security flashes and alerts ClearQuest Web integrations enhancements The addition of OSLC 2.0 consumer support in ClearQuest enables the following integrations enhancements: Prefilling OSLC 2.0 linked records OSLC 2.0 consumer support in ClearQuest Web enables configuring ClearQuest to prefill field values when opening the creation dialogs for associated OSLC 2.0 projects. When a creation dialog displays, values for designated fields in the new record are initialized by using content from configured fields of the initiating ClearQuest record. For more information on prefilling OSLC 2.0 linked records, see Using ClearQuest to prefill OSLC 2.0 linked records ( oc%2ftopics%2fc_config_prefill_oslc_records.htm ) in the IBM Rational ClearQuest Information Center New OSLC link types New link types are provided as defined in the Open Services for Lifecycle Collaboration Change Management Specification Version 2.0. The new link types enable you to define more descriptive relationships between OSLC friends. For more information, see Link types ( doc%2ftopics%2fc_clm_int_oslc_link_types_sel.htm) in the IBM Rational ClearQuest Information Center Support for integrations with IBM Rational Lifecycle Adapters ClearQuest includes support for the OSLC-based integrations between IBM Rational ClearQuest and HP ALM, and between Rational ClearQuest and Atlassian JIRA, through the IBM Rational Lifecycle Integration Adapters. For more information, see Use ClearQuest integrations for Lifecycle Integration Adapters Versions 1.0 and 1.1. ( doc%2ftopics%2fc_clm_int_use.htm) in the IBM Rational ClearQuest Information Center 2.4 Schema designer enhancements ClearQuest includes the following enhancements for the schema designer: Dated 25-July-2013 Printed 25 July 2013 Page 8 of 20

9 2.4.1 ratlperl and CQPerl are upgraded ratlperl and CQPerl are upgraded to version For information about the upgrade, see Technote , What are the potential Perl hook/script upgrade issues with IBM Rational ClearQuest 8.0.1? ( Hide actions that users do not have permission to perform The HIDE_ACTIONS schema property enables the schema designer to hide actions that the user cannot perform based on a hook. Hiding actions can reduce the choices presented to the user, making the interface simpler. For more information, see Hiding actions that a user cannot perform ( oc%2ftopics%2fc_hide_action.htm) in the IBM Rational ClearQuest Information Center Configure immediate or queued notification With the Plus package, schema designers can configure the ClearQuest Relay server to deliver as soon as a ClearQuest record is committed, or queued. For more information, see Mail Relay ( oc%2ftopics%2fsch_pkgs%2fc_emp_mail_relay.htm) in the IBM Rational ClearQuest Information Center. Dated 25-July-2013 Printed 25 July 2013 Page 9 of 20

10 3 Testing and Methodology Metrics To ensure the reliability of Rational ClearQuest, tests were run to validate as much of the product s functionality as possible, while providing comprehensive client and server platform coverage. 3.1 APAR and Defect Fixes The release also included many more defect fixes that were the result of internally found defects or externally reported APARs from customers. Please refer to charts below for metrics around quality goals for this release. The IBM Corporate Instruction definition of reliability is that the software must execute with no program errors and produce correct results in all client usage scenarios. One element of reliability commitments is to focus on the improvement of the quality of the product in terms of APARs, which are defects that have been reported by a customer. Defect resolution percentages: The quality goals for the release are to fix 100% DS1 & DS2 defects (current release & internal backlog), and 95% all DS1 DS4s found in current release (Non-APAR). Both goals were met or exceeded. The severity distribution for the internally found defect fixes in this release is shown below, with DS1 being most severe and DS4 being least severe. Figure 1: Defect resolution percentages 3.2 Testing Strategy As part of our ongoing process improvements in agile development practices, testing was conducted within each iteration, throughout the release targeting the changes delivered in that iteration. In order to facilitate agile best practices, the entire development team uses ClearQuest integrated with Rational Team Concert (RTC) and Rational Quality Manager (RQM) to support agile planning, defect tracking, test planning, test tracking and overall status reporting via dashboards. Dated 25-July-2013 Printed 25 July 2013 Page 10 of 20

11 The testing strategy for Rational ClearQuest includes Functional testing, System testing, In-House Deployment, Performance testing and Beta testing. Functional tests examine each feature area or component and validate that the user can perform the expected actions. System tests examine typical customer usage, for example, deploying Rational ClearQuest and Rational ClearQuest MultiSite into a large enterprise environments as well as integrations with other products. Rational ClearQuest and Rational ClearQuest MultiSite are used in the development process. Before Rational ClearQuest is installed on internal servers or at clients, testing occurs to ensure that there have been no regressions since the last release and that all new features are qualified. This is followed by In-House Deployment testing, where the product is used in production by the entire ClearCase and ClearQuest development and support organizations. Performance testing is covered throughout the release targeting specific performance goals. Beta testing was conducted throughout the release including three open beta programs available to all existing or potential customers worldwide Functional Verification Testing Coverage As part of adoption of agile development practices, Functional Verification Testing (FVT) planning of the release begins with Epic and User Stories. These stories define requirements and new capabilities for the release that include identifying acceptance criteria for new functionality. Acceptance criteria include the set of tests that must pass in order to declare that the feature is ready to be released. Additionally FVT includes overall regression testing that must be conducted during the release to validate that overall functionality continues to function as expected, and that no regressions were introduced. FVT attempts to cover as much as of the product functionality as possible via both automation and manual testing. New tests are developed to cover new or changed functionality during iterations. The chart below shows the ClearQuest test coverage by Functional Area. ClearQuest FVT Coverage by Area CQWeb/CMServer 26% 24% Admin/Designer CQ Clients(non-w eb) FullTextSearch Help 1% 2% 2% 14% 7% 2% 6% 11% 1% 4% Hide Action Upgrade/Install MRU Integrations/OSLC Perl Reporting Core Figure 2: FVT coverage by area Dated 25-July-2013 Printed 25 July 2013 Page 11 of 20

12 3.2.2 Systems Verification Testing Coverage System Verification Testing (SVT) encompasses a variety of testing arenas. Complex customer-like schemas are tested before and after upgrading ClearQuest from a prior release to this version. SVT also covers integration testing between Rational ClearQuest and other applications. Particular focus was on testing integrations between ClearQuest and the Collaborative Lifecycle Management tools: Rational Team Concert, Rational Quality Manager, Rational Requirements Manager as well as integrations between Rational ClearCase and Rational ClearQuest. In addition, testing focused on the integration between Rational ClearQuest and the Rational Lifecycle Integration Adapters with HP ALM and Atlassian JIRA In-House Deployment Testing The testing strategy also includes In-House Deployment in which tested development versions of the ClearQuest product are installed on ClearQuest client systems as well as production servers that are widely used by the development, testing and support teams. The ClearQuest application is a complex multi-sited schema with integrations to many Rational products including Rational Team Concert (RTC) and Rational Quality Manager (RQM) as well integrations to other business systems within IBM. Features such as ClearQuest data visualisation using RTC dashboards were tested in our production environment. ClearQuest Web was deployed on servers with WebSphere Application Server 8. Additionally, Full Text Search was deployed with WebSphere Application Server 8 using the Full Text Search Administration tool and Solr version Beta Testing At IBM Rational, we are committed to providing high-quality products through a regular program of thorough product testing both inside and outside of the company. Internal controls in product development, quality assurance, and technical support ensure the technical quality of the product. But it is the Beta testing at client sites that ensures products can withstand the challenges of real-world processing. The Rational Beta Program works with product teams to choose the beta program that best suites their goals. Beta programs allow you to evaluate and provide feedback on IBM products before the products' general availability. All Beta programs require that you accept the terms of use of the beta product. These programs are typically confidential. There are two types of beta programs that were run in the release described below. Managed beta: This evaluation requires a high level of commitment from the customers to provide feedback through periodic status calls and surveys. Customers might be asked to provide a quote or become a reference for the product. Several customers participated in the managed beta programs for ClearQuest for the Release. Open beta: These programs are not confidential and are open to anyone. Many customers participated in the open beta program for this release. This can range from hundreds on some products to thousands on other products. All defects reported and survey results are evaluated for inclusion in the current product and if not included may be deferred to a future release. ClearQuest had 3 beta programs during the Release. Some key statistics are shown below. Dated 25-July-2013 Printed 25 July 2013 Page 12 of 20

13 # of PRs Entered 34 # of Defects Entered 26 # of Defects Resolved 24 % Defects Resolved 92.30% # of RFEs Entered 5 # of RFEs Fixed 5 % RFEs Fixed % Figure 3: Beta statistics *PRs are Problem Reports entered into the Problem Reporting Tool. *Defects and RFEs (Request For Enhancement) are actual Work Items that were created Key Process Improvement Areas Test Process Improvement Testing was conducted within each iteration that included a new or changed feature or capability. In this way, defects were discovered and fixed while the code was being developed. Automated build-triggered tests were run on a wide cross section of supported operating system platforms and database versions with extensive coverage of ClearQuest core components. Junit Testing Junit tests were developed and enhanced to cover ClearQuest Web Server capabilities including OSLC-based integrations. Automation In addition to unit and functional test coverage for the ClearQuest core and ClearQuest Web Server components, test automation was expanded to cover the Graphical User Interfaces (GUI) for ClearQuest Web, the ClearQuest Designer (Eclipse based) and ClearQuest client (Eclipse based). A subset of these tests written with Rational Functional Tester (RFT) was executed daily via the build-triggered system. More comprehensive suites of GUI-based automated tests were run on a wider variety of configurations throughout the release. Dated 25-July-2013 Printed 25 July 2013 Page 13 of 20

14 4 ClearQuest Web Performance, Scalability and Availability In terms of system performance, the following areas were the focus for IBM Rational ClearQuest Version 8.0.1: Performance: The ability of a system to respond acceptably to user requests Scalability: The ability of a system to respond to increased load and still respond with acceptable performance Availability: The ability of a system to respond for long durations with consistently acceptable performance Disclaimer for Performance Data: Any Performance data contained herein was determined in a controlled environment. The results obtained in other operating environments may vary significantly. Actual results may vary. Users of this document should verify the applicable data for their specific environment. These test results are specific to the product software, test configuration, workload, and environment that were used. Product performance in other environments or conditions may be different compared to the data reported in this document. 4.1 ClearQuest Web Performance ClearQuest continues the trend of improved single user performance. The following graph (Figure 6) compares the response times between Rational ClearQuest Web version and Rational ClearQuest using WebSphere version The test schema was the ALM schema populated with approximately 510,000 records. Most measured transactions are faster in the ClearQuest Web version compared to the ClearQuest Web version release. The average response time improvement for these measured transactions averages 2.5%. This is on top of the approximately 20% performance improvement in measured transactions achieved during the 8.0 release. Figure 4: Rational ClearQuest Web Single-user Benchmark, vs Dated 25-July-2013 Printed 25 July 2013 Page 14 of 20

15 Figure 5: Rational ClearQuest Web 200-user Benchmark, vs ClearQuest Web Scalability IBM Rational ClearQuest Web version scales very well. Scalability testing for Rational ClearQuest Web version measured single and multi-user response time. The graph below (Figure 6) compares multi-user performance (75, 150 and 200 users) on ClearQuest Web using WebSphere version Dated 25-July-2013 Printed 25 July 2013 Page 15 of 20

16 Figure 6: IBM Rational ClearQuest Web scalability comparison ClearQuest Web transactions scale well as user load increases. Comparing with 75 users, there was only a 4% performance degradation for 150 users and an 8% performance degradation for 200 users. In Figures 7a and 7b below, multi-record update (MRU) performance test results are shown. In this testing 100 records are updated by 50, 100, 150 and 200 simultaneous users on a single server. A variety of reading and writing actions are tested. The data is split between two charts to be able to clearly display the transaction information. Note: the scale of milliseconds differs between the 2 charts. Dated 25-July-2013 Printed 25 July 2013 Page 16 of 20

17 Figure 7a: IBM Rational ClearQuest Web Multirecord-Update Stress Test Figure 7b: IBM Rational ClearQuest Web Multirecord-Update Stress Test (continued) Dated 25-July-2013 Printed 25 July 2013 Page 17 of 20

18 Response Time (ms) 4.3 ClearQuest Web Availability Throughout the development cycle, the availability of IBM Rational ClearQuest was continuously tested via the use of 3-day workloads. These workloads were designed to keep the system running at very high loads during which metrics were taken at regular intervals to validate the server s availability. The Rational ClearQuest availability scenarios tested the ClearQuest Web Server running on RedHat Enterprise Linux version 6 Update 3, x86 64-bit. The test environment included WebSphere Application Server The following goals of the ClearQuest availability scenarios were as follows: Assess the availability of the single ClearQuest Web server under conditions that simulate the maximum number of supported users (200 users), a reasonable user activity rate (averaging 15 transactions-perhour-per-user), and long run duration (3 days) Measure and record performance characteristics under load and stress to ensure no major degradation during a 3-day time period System under load remained stable and responsive for extended period (3 days) with no unplanned restarts System under load remained stable and responsive under normal and stress level workloads without availability failures or data corruption The ClearQuest Web Server availability tests completed with no performance issues found. This scenario had 200 active users with each user performing 15 transactions per hour. The ClearQuest repository was populated with 510,000 records. 7 of the transactions included the new multirecord-update feature denoted with (MRU) ClearQuest Web Reliability for 3 Days, users, averaging 15 transactions-per-hour-per-user Single CQ Web + WAS Instance, ALM Schema SSL enabled, server-side data Day1 Day2 Day CQ Home Login Create new request Save new request Find request Generate task workload (2) Open task Modify task (initiate) Modify task (commit) Find task Generate activity workload (4) Open activity Save new activity Modify activity (initiate) Modify activity (commit) Activate activity Complete activity Run saved query Load from query result ReRunqry and OpenTemplateTask(MRU) template task(mru) CommitBatchUpdate_Modify(MRU)_eachRecord template task(mru) CommitBatchUpdate_Activate(MRU)_eachRecord template task(mru) CommitBatchUpdate_Reopen(MRU)_eachRecord Logout Figure 8: IBM Rational ClearQuest Web availability test Dated 25-July-2013 Printed 25 July 2013 Page 18 of 20

19 5 Customer Feedback The Rational ClearQuest version release development cycle involved a high level of customer interaction to create a usable and consumable product. Customers influenced the product through the IBM Rational Design Partner Program (DPP), Beta Program, and additional customer feedback though the Rational Software Innovate Conference, Voice of the Customer Event (VoiCE), and reports from customers to the IBM Rational Support system. 5.1 The Design Partner Program The Design Partner Program (DPP) helps IBM acquire continuous customer feedback that can be leveraged early in the product development cycle, and used to influence the product strategy, direction, and features. Customers from various industries such as Healthcare, Electronics, and Financial Services participate in the program and share their ideas. The goal of continued collaboration with Design Partners is to deliver the right features and improve consumability, while increasing customer satisfaction and loyalty. The DPP is also interested in vetting product strategy, new cutting-edge development, and ensuring our future roadmaps align with the overall business needs of our customers. As a result of the Design Partner Program, IBM is able to create products that customers are excited about and committed to, because these customers exerted considerable influence on the development plans. Select design partners participated in the Beta Program for Rational ClearQuest version 8.0.1, some deploying on large distributed test environments, and others in small and medium test environments. Design Partners participated in activities such as feature presentations and feedback discussions, individual customer interviews, surveys, and usability testing on Rational ClearQuest version Beta Program Rational conducted a three phase Beta program for this release. Over 40 customers from around the world downloaded and evaluated early versions of the code. Interactions with the managed beta customers included: Weekly meetings with customers and engineers to educate customers on the features of the product and to gather feedback on the customers experience with the beta code. Quality Surveys to gather feedback on specific strengths and weaknesses of the beta. Engineering was able to make adjustments to the final release based on this customer feedback. Customers of the Rational ClearQuest beta took particular interest in the new multirecord-update feature, the hide actions feature and other web usability enhancements. The Rational Beta Program provides customers with the following benefits: Early access to beta code Free education Free support Early development of skills Direct input to Rational Development and Product management to influence the development of the software Early ability to assess how the new release can benefit their organization Dated 25-July-2013 Printed 25 July 2013 Page 19 of 20

20 The Beta Program provides Rational with the following benefits: Validation of product acceptability (functionality, usability) with key customers to provide input for this and future releases Validation of General Availability (GA) readiness and support capability Direct customer input to development, marketing, and sales References and testimonials from early users of the product Ability to provide selected IBM customers and Business Partners with the opportunity to gain early experience with the product Results from the Beta Surveys gave the release an overall score of 8.8 (on a scale of 1-10) with 1 meaning do not ship the concept behind this offering is flawed and potentially needs a redesign, and 10 meaning ship this offering is excellent. 5.3 Future Feedback Customers interested in learning more about the Rational Design Partner Program, future Beta Evaluation programs, usability testing or Rational Software Innovate Conferences should contact their local sales representative. Dated 25-July-2013 Printed 25 July 2013 Page 20 of 20