The Unity v4 Voting System and Unity Voting System are comprised of previously VSTL tested components and state certified components.

Size: px

Start display at page:

Download "The Unity v4 Voting System and Unity Voting System are comprised of previously VSTL tested components and state certified components."

Shonda Hodges
5 years ago
Views:

2 1 Introduction The purpose of this Test Report is to document the procedures that Pro V&V, Inc. followed to perform certification testing of the ES&S Unity v4 Voting System to the requirements set forth in the Federal Election Commission (FEC) 2002 Voting Systems Standards (VSS). The Unity v4 Voting System and Unity Voting System are comprised of previously VSTL tested components and state certified components. 1.1 References The documents listed below were utilized in the development of this Test Report: ES&S Unity v4 Testing Project Scope ES&S Document FLU4003v4_C_D_0100_SysOvr ES&S System Overview, Version 1.0 dated October 28, 2013 ES&S Document FLU4003v4_DOC_D_0100_ChangeNotes ES&S Change Notes, Version 1.0 dated October 28, 2013 Election Assistance Commission Testing and Certification Program Manual, Version 2.0 Election Assistance Commission Voting System Test Laboratory Program Manual, Version 2.0 Federal Election Commission (FEC) 2002 Voting Systems Standards (VSS) National Voluntary Laboratory Accreditation Program NIST Handbook 150, 2006 Edition, NVLAP Procedures and General Requirements (NIST Handbook 150), dated February 2006 National Voluntary Laboratory Accreditation Program NIST Handbook , 2008 Edition, Voting System Testing (NIST Handbook ), dated May 2008 Pro V&V, Inc. Quality Assurance Manual, Revision 7.0 United States 107 th Congress Help America Vote Act (HAVA) of 2002 (Public Law ), dated October P a g e

3 1.2 Terms and Abbreviations The terms and abbreviations applicable to the development of this Test Report are listed below: ADA Americans with Disabilities Act 1990 BMD Ballot Marking Device CM Configuration Management COTS Commercial Off-The-Shelf DRE Direct Recording Electronic EAC Election Assistance Commission EMS Election Management System ES&S Election Systems & Software FCA Functional Configuration Audit HAVA Help America Vote Act PCA Physical Configuration Audit TDP Technical Data Package QA Quality Assurance UPS Uninterruptible Power Supply VSTL Voting System Test Laboratory 2002 VSS Federal Election Commission (FEC) 2002 Voting System Standards 1.3 Background The Unity 4.0 System was tested at SysTest (a VSTL) for conformance to the 2002 Voting System Standards (VSS) and a test report was provided to ES&S on February 9, Unity 4.0 became Unity v1 which was tested and certified by the State of Florida on May 7, The subsequent release, Unity v3, which is the baseline for Unity v4, was tested and approved by the State of Florida on August 15, Unity v4 was tested and certified by the State of Florida on April 10, Unity v4 includes enhancements to its legacy systems (Unity /Unity ). This testing campaign includes the DS850 central tabulator from the EAC Certified Unity system and a bridging function. 3 P a g e

12 Figure 2.2 Polling Place Configuration 2.2 Testing Configuration The testing event utilized one setup of the Unity v4 System and its components as configured for normal use. The following is a breakdown of the Unity v4 System components and configurations for the test setup: Standard Testing Platform: The central office setup consists of a standalone or bridged EMS configuration accompanied by either a DS850 or M650. The precinct setup consists of ivotronic (DRE) and/or M100s and DS200s with plastic and metal ballot boxes. The ivotronic units can be set up as accessible voting stations or non-accessible voting stations. 2.3 Test Support Equipment/Materials All test support equipment and materials required to facilitate testing were supplied by ES&S. 12 P a g e

14 System Functionality Description ES&S Ballot Image Manager (ESSIM) Version Number ES&S System Functionality Description DS200 FL Unity v4 Document ID: FLU4003v4_SFD00_DS200.docm Software Functionality Description Hardware Programming Manager (HPM) Version Number System Functionality Description ivotronic Image Manager (ivim) Version Number System Functionality Description ivotronic Firmware Version System Functionality Description Model 650 Firmware Version Number Hardware Version 1.1 and 1.2 Unity System Functionality Description Document Revision 1.0 U3414_C_D_0200_SFD 03 System Hardware Specifications DS200 Engineering Hardware Specification, Document Revision 1.0, Hardware Revision 1.2 Document ID: DS200HW_M_SPC_0312_HWSpec.docm DS200 Engineering Hardware Specification, Document Revision 2.0, Hardware Revision 1.3 Document ID: DS200HW_M_SPC_0313_HWSpec.docm DS850 Engineering Hardware Specification, Document Revision 1.2 Hardware Revision: 1.0 Document ID: DS850HW_M_SPC_0310_HWSpec System Hardware Specification ivotronic Hardware Version P a g e

15 System Hardware Specification Model 100 Hardware Version System Hardware Specification Model 650 Hardware Version Software Design and Specification Ballot on Demand (BOD) Software Design and Specification Version Data Acquisition Manager Software Design Specifications Software Version Software Design and Specification Election Data Manager (EDM) Version Number Software Design and Specifications Election Reporting Manager (ERM) Version Number ESSCRYPT Functional Specification Version ESSDECPT Functional Specification Version Software Design and Specification ES&S Ballot Image Manager (ESSIM) Version Number Coding Standards Revision: 1.1 System Development Program Revision: 1.3 ES&S Software Design Specifications DS200 FL Unity v4 Document ID: FLU4003v4_SDS00_DS200.docm, Rev P a g e

16 Software Design and Specifications Hardware Programming Manager (HPM) Version Number Software Design and Design ivotronic Image Manager (ivim) Version Number Software Design and Specification ivotronic Version Number Software Specifications Model 100 Precinct Tabulator Version Software Design and Specification Model 650 Version Number Ballot Image Processing Specification Document Revision: 1.3 ES&S Software Design Specifications DS850 Unity Revision 1.0 ES&S Software Design Specifications Election Data Manager (EDM) Unity v Revision 1.0 U3414_SDS00_EDM01_CountyModel U3414_SDS00_EDM01_CountyModel U3414_SDS00_EDM02_ElectionModel U3414_SDS00_EDM02_ElectionModel ES&S Software Design Specifications Election Reporting Manager (ERM) Unity v Revision 1.0 ES&S Software Design Specifications Election Reporting Manager (ERM) Appendices Unity v Revision P a g e

17 ES&S Software Design and Specification ES&S Ballot Image Manager (ESSIM) Unity v Revision 1.0 ES&S Software Design and Specification Hardware Programming Manager (HPM) Unity v Revision 1.0 ES&S Software Design and Specification Hardware Programming Manager (HPM) Appendices Unity v Revision 1.0 ES&S Software Design Specifications LogMonitor Unity v Revision System Security Specification Jurisdiction Security Procedures Election Systems and Software Version ES&S System Security Specification Version Release Unity Voting System Security Specification Revision 1.1 Document ID: U3414_CM_D_SecuritySpec Unity Hardening Procedures for the Election System Revision 1.3 Document ID: U3414_CM_P_HardeningProcedures Unity Security Script Description Revision 1.2 Document ID: U3414_CM_P_SecScriptDesc 07 System Operations Procedures 17 P a g e

18 ES&S Audit Manager System Procedures Operations Version Release ES&S Ballot On Demand Printer Setup and Printing Procedures Version Release ES&S Data Acquisition Manager System Operations Procedures Version Release ES&S Election Data Manager System Operations Procedures Version Release ES&S Election Reporting Manager System Operations Procedures Version Release ES&S Image Manager System Operations Procedures Version Release DS200 Operator s Guide Document Version 2.0 Firmware Version 1.5 May 14, 2014 DS200 Operator s Guide Document Version 2.0 Firmware Version 1.5 October 18, 2013 ES&S Hardware Programming Manager System Operations Procedures Version Release ES&S ivotronic Image Manager System Operations Procedures Version Release P a g e

19 ES&S ivotronic System Operations Procedures Hardware Version 1.1 Firmware Version ES&S M100 Operator s Guide Version Release ES&S Model 650 System Operations Procedures Firmware Version Hardware Version 1.1 Election Reporting Manager Unity Bridge Guide Document Version 1.0 Software Version 7.9 Ballot on Demand Printer Setup & Printing Procedures Document Version: 1.1 Software Version: 7.7 DS850 Operator Guide Document Version 1.0 Firmware Version 2.9 Election Data Manager Operator s Guide Document Version 1.1 Software Version 7.8 Election Reporting Manager User s Guide Document Version 1.1 Software Version 7.9 ES&S Image Manager User s Guide Document Version 1.1 Software Version 7.8 Hardware Programming Manager User s Guide Document Version 1.0 Software Version 5.9 ES&S Log Monitor User s Guide Document Version 1.0 Software Version P a g e

20 08 System Maintenance Manuals DS200 Maintenance Guide Document Version 3.0 Firmware Version 1.5 ES&S ivotronic System Maintenance Manual Hardware Revision 1.1 Firmware Version ES&S M100 Maintenance Manual Version Release Hardware Version 1.3 ES&S Model 650 System Maintenance Manual Firmware Version Hardware Version 1.1 DS850 Maintenance Guide Document Version 1.1 Firmware Version Personnel Deployment Personnel Deployment and Training Program, Document Revision 3.0 Document ID: ESSSYS_T_D_0900_TrainingProgram.docx Ballot on Demand Training Manual Version Number Data Acquisition Manager Training Manual Version Number DAM / ERM Election Day Training Manual Version Number 7.7.x DS200 Precinct Ballot Scanner Election Day Training Manual Version Number 1.4.x 20 P a g e

21 DS200 Precinct Ballot Scanner Pre-Election Day Training Manual Version Number 1.4.x Election Data Manager Training Manual Version Number Election Reporting Manager Pre-Election Day Training Manual Version Number 7.7.x ESSIM Training Manual Version Number ivotronic ADA Unit Control Panel Definition HANDOUT A - ivo ADA Unit Descriptions Handout A - PEB_M100 Printing Handout A - PEB_M100 Printing Handout A-Setting Date_Time_DST Handout A-Setting Date_Time_DST Handout B - Wireless Modem Handout B - Wireless Modem Handout C - Changing the M100 Paper Handout C - Changing the M100 Paper Hardware Program Manager Training Manual Version Number ivim Training Manual Version Number Model 100 Election Day Checklist Version Number 5.x Model 100 Pre-Election Day Checklist Version Number 5.x Model 650 Election Day Checklist Version Number 2.x 21 P a g e

22 Model 650 Pre-Election Day Checklist Version Number 2.x ivotronic Election Day Checklist Version Number ivotronic Pre-Election Day Checklist Version Number DS200 Precinct Ballot Scanner Election Day Training Manual Version Number 1.5.x DS200 Precinct Ballot Scanner Pre-Election Day Training Manual Version Number 1.5.x 10 Configuration Management Plan ES&S Standards and Procedures Configuration Management Program Revision 2.1 ES&S Standards and Procedures ES&S Technical Documentation Program Revision Quality Management Program Manufacturing Quality Assurance Program Revision: 1.2 ES&S Standards and Procedures Software Quality Assurance Program Revision System Change Notes Florida Unity version 4 System Change Notes Revision P a g e

23 Florida Unity version 4 System Change Notes Cumulative from Florida Unity Revision 1.0 Unity System Change Notes Revision Attachments Ballot Production Guide for Unity Revision Test Process and Results The following sections outline the test process that was followed to evaluate the Unity v4 Voting System under the scope defined in Section General Information All testing was conducted under the guidance of Pro V&V by personnel verified by Pro V&V to be qualified to perform the testing. The examination was performed at the Pro V&V, Inc. test facility located in Huntsville, AL. 3.2 Test Cases/Procedures Test cases/procedures were developed to evaluate the system being tested against the stated requirements. Prior to execution of the required test cases/procedures, the system under test was subjected to testing initialization to establish the baseline for testing and ensure that the testing candidate matched the expected testing candidate and that all equipment and supplies were present. The following tasks were completed during the testing initialization: Ensured proper setup of equipment. Checked network connections, power cords, keys, etc. Checked version numbers of (system) software and firmware on all components. Verified the presence of only the documented COTS. Ensured removable media is clean. 23 P a g e

24 Ensured batteries are fully charged. Inspected supplies and test decks. Recorded protective counter on all tabulators. Reviewed physical security measures of all equipment. Recorded basic observations of the testing setup and review. Recorded serial numbers of equipment. Retained proof of version numbers. 3.3 Test Results The procedures that were utilized during the test engagement and the results obtained are summarized in the following paragraphs. During the evaluation, the test team made observations of general system behavior. TDP Review - This review is conducted only for stated functionality review and verification. This review does not address consistency or completeness of documents. Results of the review of each document are entered on the TDP Review Checklist and are reported to the customer for disposition of any discrepancies. This process is ongoing until all discrepancies are resolved. Any revised documents during the TDP review process are compared with the previous document revision to determine changes made, and the document is re-reviewed to determine whether the discrepancies have been resolved. Summary Findings: A Limited TDP Review was performed to ensure that information concerning each component of the Unity v4 System was included. Trusted Build To perform the trusted build, ES&S-submitted source code, COTS, and Third Party software products were inspected and combined to create the executable code. Additionally, during the performance of the trusted build, the build documentation was reviewed. Summary Findings: During execution of the Trusted Build, the source code submitted by ES&S and reviewed by PRO V&V was successfully built using the submitted COTS and third party software products, and the reviewed build documentation Functional Configuration Audit (FCA) During this area of testing, the specific functionality of the added components (ivotronic) and modified functionality (bridging functionality to the DS850) under evaluation that is claimed by the manufacturer was targeted to ensure the product 24 P a g e

25 functions as documented. This testing used both positive and negative test data to test the robustness of the system. Summary Findings: All FCA testing passed. Accuracy An accuracy test was performed to ensure that the added components (ivotronic) and modified functionality (bridging functionality to the DS850) could process ballot positions within the allowable target error rate. This test was designed to test the ability of the system to capture, record, store, consolidate, and report specific voter selections and absences of a selection. Summary Findings: To perform the Accuracy Test, sufficient ballots were voted on both ivotronics and the DS850 with all results successfully bridged together in the EMS. A breakdown of the test parameters is provided below: Ivo: Ballot size Min # of Ballots Needed # Ballot Positions per Ballot Actual # Auto Voted Actual # Hand Voted # of Machines in Test Total (>1,549,703) 19 inch * 390*** 3** 1,988,292 *The L&A test consisting of 2,278 ballots was executed twice ** 130 ballots were voted on each of the three ivo s (Each a different model) *** ~10% of the minimum needed 3855 ballots (3855 ballots = 1,549,710 positions) DS850: Ballot size Min # of Ballots Needed # Ballot Positions per Ballot Actual # Auto Voted Actual # Hand Voted # of Machines in Test Total (>1,549,703) 19 inch ,608,800 During execution of the test procedure, it was verified that the system successfully completed the accuracy test with all actual results obtained during test execution matching the expected results. System Integration This system level testing address the integration of the added hardware and software. This testing focuses on the compatibility of the voting system software components and subsystems with one another and with other components of the voting system. During test performance, the system is configured as would be for normal field use. 25 P a g e

26 Summary Findings: GEN1, GEN2, GEN3, PRIM1, & PRIM3 were all successfully run on both the ivotronics and DS850. All results were combined back in the EMS and included other (regression testing) results from the M100, DS200, and M650. Regression Testing Regression testing was performed on all system components to verify that all functional and/or firmware modifications made during the test campaign did not adversely affect the system and its operation. Summary Findings: Regression Testing was performed to verify that any functional testing discrepancies discovered during the test case design process for the Functional Configuration Audit were addressed by ES&S. During execution of the test procedure, it was verified that the Unity v4 Voting System successfully completed the functional regression test with all actual results obtained during test execution matching the expected results. Physical Configuration Audit (PCA) A PCA was performed to compare the voting system components submitted for testing to the manufacturer s technical documentation. The PCA was conducted in two phases: Initial and Final. The Initial PCA was conducted in order to baseline the system prior to test campaign commencement. The Final PCA was conducted in order to verify the final software and hardware configurations. Summary Findings: During execution of the test procedure, the components of the system were documented by component name, model, serial number, major component, and any other relevant information needed to identify the component. For COTS equipment, every effort was made to verify that the COTS equipment had not been modified for use. Additionally, each technical document submitted in the TDP was recorded by document name, description, document number, revision number, and date of release. At the conclusion of the test campaign, test personnel verified that any changes made to the software, hardware, or documentation during the test process were fully and properly documented Security During the execution of a security penetration evaluation, the system was inspected to verify that various controls and measure were in place in order to meet the objectives of the security standards which include: protection of the critical elements of the voting system; establishing and maintaining controls to minimize errors; protection from intentional manipulation, fraud and malicious mischief; identifying fraudulent or erroneous changes to the voting system; and protecting the secrecy in the voting process. 26 P a g e

27 Summary Findings: During the security penetration evaluation, test personnel first verified that the manufacturer s TDP contained documented access and physical controls and then, following the manufacturer s documented procedures, configured the voting system for use and functionally verified that the documented controls were in place and were adequate to meet the stated requirements. Information which was not present in the TDP was presented to ES&S for resolution. ES&S then submitted updated documentation which was reviewed to ensure that the required information was present. During execution of the security penetration evaluation procedures, it was verified that Unity v4 Voting System successfully completed the security test with all actual results obtained during test execution matching the expected results Usability This area of testing focuses on the usability of the system being tested. Usability is defined generally as a measure of the effectiveness, efficiency, and satisfaction achieved by a specified set of users with a given product in the performance of specified tasks. In the context of voting, the primary user is the voter, the product is the voting system, and the task is the correct recording of the voter ballot selections. Additional requirements for task performance are independence and privacy: the voter should normally be able to complete the voting task without assistance from others, and the voter selections should be private. Lack of independence or privacy may adversely affect effectiveness (e.g., by possibly inhibiting the voter's free choice) and efficiency (e.g., by slowing down the process). Summary Findings: These were all VSS should, but nothing was noted during testing. Accessibility This area of testing evaluates the requirements for accessibility. These requirements are intended to address HAVA 301 (a) (3) (B). Ideally, every voter would be able to vote independently and privately. As a practical matter, there may be some number of voters whose disabilities are so severe that they will need personal assistance. Nonetheless, these requirements are meant to make the voting system independently accessible to as many voters as possible. Summary Findings: All accessibility requirements were reviewed and passed; however, the ivotronics units under test did not support the sip-n-puff device. Pro V&V was only supplied with a 3-Key and 4-Key ivotronic. According to documentation, the sip-n-puff only functions with the 6-key ivotronic. 3.4 Conditions of Satisfaction The voting system was evaluated against the 2002 VSS requirements. Throughout the test campaign, as tests were executed, resultant data was inspected and technical documentation reviews were performed to ensure that each applicable requirement was met; therefore, fulfilling the conditions of satisfaction. 27 P a g e

28 4 Conclusions Based on the results obtained during the test campaign, Pro V&V determines the Unity v4 Voting System, as presented for evaluation, meets the requirements for voting systems of the 2002 VSS. Additionally, Pro V&V determines that the Unity bridge process functions as described in ES&S' documentation and did not introduce any errors when tested in conjunction with the Unity v4 system. 28 P a g e

State Certification Roundtable

United States Election Assistance Commission State Certification Roundtable - September 23, 2011 1 Overview Conformance Testing Voting Standards Out of Scope Items 2005 VVSG Program Operations 2 Conformance