Task 3 Report Test and Evaluation Master Plan (TEMP) Prepared by and Prepared for U.S. Department of Transportation Federal Highway Administration Contract No. November 21, 2008
TABLE OF CONTENTS Page PURPOSE... 1 BACKGROUND... 1 TEST PLAN... 1 Overview... 1 FIELD SUBSYSTEM TESTS... 2 Solar Power System Test... 2 24vDC Power System Test... 2 Border Crossing Tag Read Test... 2 Tag Reading Reliability Test... 2 System Latency Test... 2 Data Logger Test... 3 Wireless Signal Strength Test... 3 Static IP Address Test... 3 Detection Station Communication Accessibility Test... 3 Detection Station Auto-Power Cycle Test... 3 RFID Reader Shutdown Test... 4 CENTRAL SUBSYSTEM TESTS... 4 Wireless Data Transfer Test... 4 Remote Data Retrieval Test... 4 Data Retrieval Application Test... 4 KEY PERFORMANCE PARAMETERS... 5 Tag Read Time Stamp Accuracy... 5 Accuracy of Crossing Time between Entry-Exit Readings... 5 SYSTEM EVALUATION PLAN... 5 APPENDIX A: BORDER CROSSING TIME MEASUREMENT SYSTEM TEST AND EVALUATION MASTER PLAN SUMMARY MATRIX... A-1 Test and Evaluation Master Plan (TEMP) i November 21, 2008
BORDER CROSSING TIME MEASUREMENT SYSTEM TEST AND EVALUATION MASTER PLAN (TEMP) Purpose This document is the master plan for system level tests to verify the proper operation of the Border Crossing Time Measurement System (BCTMS). The tests are intended to establish the high-level working nature of the system and not to test individual system components for specification compliance. A summary of the parameters as well as the capability thresholds of each test developed during this task can be found in Appendix A of this report. References to the Texas Transportation Institute (TTI) and its El Paso office are made to indicate the configuration intended for RFID implementation at the Bridge of the Americas (BOTA), but in all other regards this TEMP is intended to be applicable for BCTMS implementation of at any land port of entry. The TEMP is a deliverable and represents sub-task 3.1.4 of the Project Plan. Background The BCTMS is an automated system to measure border crossing times for commercial vehicles (trucks) at an international port of entry. The system utilizes passive radio frequency identification (RFID) tag readers placed at strategic locations within or near the international border complex. RFID readers acquire a unique identifier from trucks as they pass each tag reader/detection station in the system. The station time-stamps each tag ID and transmits it to a central location for processing. Each data sample will have a reader location, a truck identifier, and a time when the truck passed the location. Real-time processing can calculate average crossing times for vehicles as well as other metrics. Test Plan Overview The BCTMS can be organized into three subsystems: the Field Subsystem, the Central Subsystem, and the User Subsystem. The Field Subsystem is responsible for gathering tag ID data and forwarding the data to the Central Subsystem. The Central Subsystem receives the data, processes it and produces measures such as average crossing time. The User Subsystem provides a mechanism to access the processed border crossing time information from the Central Subsystem. The evaluation plan consists of comparing border crossing times measured by the RFID and geographic positioning system (GPS) data logger deployed inside the trucks with Texas Department of Public Safety (DPS) tags. This document will discuss the Field and Central Subsystems testing plan and a system evaluation plan. Test and Evaluation Master Plan (TEMP) 1 November 21, 2008
Field Subsystem Tests Solar Power System Test The solar power system charge controller includes a display which shows the inbound amperage from the solar panel array (two panels), the voltage on the battery bank, and the amperage which is being drawn by the load (all the equipment at the site). The solar power system test will verify whether each of those three readings is appropriate given the conditions at test time. If the solar panels are being strongly illuminated by the sun, there should be significant electric current flowing from them. The voltage on the battery bank should be between 11.5vDC and 14vDC under normal conditions (unless the battery bank has been depleted). The load current consumption should be less than 3.5 amps based on the detailed design system power requirement estimate. 24vDC Power System Test This step will test the voltage at the output of the 24vDC converter to ensure a proper voltage is applied to the RFID reader and programmable relay with both units connected and drawing power. Border Crossing Tag Read Test The RFID reader must be tested to show that it can read the border crossing tags. DPS provided a group of sample tags for this project. Researchers will affix at least two tags to the inside of the windshield of a test vehicle in the manner prescribed by DPS. The tags can be temporarily applied to the windshield using simple cellophane tape. Windshield tags require glass in front of the tag for proper operation; holding a tag without the front glass in the RFID coverage zone may not be sufficient for the reader to interrogate the tag properly or at all. The test vehicle will be driven through the field site at the expected speed of the target trucks and researchers will verify that the tags were read properly. This test will verify that each RFID reader is emitting RF energy, receiving reflected RF from the sample tags, and is configured properly to decode the tag identifier (tag protocol setting on the RFID unit). Tag read data can be collected using Windows HyperTerminal on a laptop computer connected directly (via RS-232) to the RFID unit. Tag Reading Reliability Test The RFID reader must be able to read at least 85% of the readable tags on trucks in lanes within the RFID unit s coverage zone. A researcher will stand near the detection station and visually count the number of trucks passing with an identifiable tag. The researcher will allow at least 50 trucks with visually identifiable tags to pass and will recover the number of tags the RFID reader found. Tag read data can be collected using Windows HyperTerminal on a laptop computer connected directly (via RS-232) to the RFID unit. System Latency Test There are many separate components that make up the field subsystem. This test is designed to show that all of these components, once turned on, properly function together. Once the system is turned on, it should take no longer than 5 minutes for the entire field subsystem to be fully functional. Test and Evaluation Master Plan (TEMP) 2 November 21, 2008
Data Logger Test The detection site contains a data logger to time-stamp incoming tag reads from the RFID unit and locally stores a copy of each record. The data logger should be configured to operate in pass-through mode. In pass-through mode, all data coming from the RFID reader are timestamped (creating a data record), a copy of the record is stored locally on a flash memory card and the record is passed through to the wireless router COM port 1 for transport to the Central Subsystem. Researchers will test the data logger operation by connecting a laptop computer running a terminal program (Windows HyperTerminal) to the port labeled Configuration Port. The Data Port should be connected to the RFID reader. Researchers will clear any files or create a new file on the data logger to hold the upcoming test data. With the test file configured, researchers will generate some RFID tag reads by putting a few sample tags in the tag reader s view. The data logger data indicator should flash when data arrive and tag data should appear on the terminal program (demonstrating pass-through). Researchers will review the data on the terminal program to verify that a proper date and time has been added to the tag ID data. They will retrieve the test file (by sending the proper commands to the logger from the connected laptop) and check whether the same data on the terminal program were also logged. Wireless Signal Strength Test From the router web interface, researchers will check the system status to ensure the cellular wireless signal strength is sufficient to maintain a continuous connection with the respective provider, which typically has signal strength of at least -90dBm. Static IP Address Test Each detection station must have a static (non-changing) Internet Protocol (IP) address so the station can be found on the wireless network. Researchers will cycle the power and check the wireless router IP when it completes its reboot operation. They will access the IP information from a locally connected laptop using the router s Ethernet Interface. They will perform this test three (3) times with ten (10) minutes between power cycles to ensure the IP does not change with a power cycle. Detection Station Communication Accessibility Test Researchers will test the wireless router to ensure it can be accessed over the cellular wireless infrastructure. Connecting a computer and reaching the Internet from the router is not sufficient to demonstrate that the router can be accessed from the Internet. The router has an internal webpage that can be accessed from anywhere on the Internet by a user with the proper credentials. Being able to retrieve the top webpage is a sufficient test to show accessibility via the wireless network. Detection Station Auto-Power Cycle Test The detection station has the ability to sense the loss of communication to the wireless network. The auto rebooter (iboot) routinely pings a known server or servers on the Internet to insure a communication path out of the detection station. If ping responses are not received, the rebooter assumes the wireless network connection has been lost and cycles the power on the wireless router. Researchers will test the local auto rebooter by removing its Ethernet connection (pulling Test and Evaluation Master Plan (TEMP) 3 November 21, 2008
the network cable) from the wireless router. Without a network connection, the rebooter will not be able to contact any outside servers. After the configured amount of server inaccessible time, the unit should cycle the power on the wireless router. RFID Reader Shutdown Test The RFID reader consumes a large majority of the power at the detection station and should be turned off during the hours when the border crossing is closed. A programmable relay is used to connect and disconnect power to the RFID reader. Researchers will load a test program into the programmable relay which can cycle the power on the RFID unit (trip the relay) for testing. The program should provide power for a period of five (5) minutes then turn power off for one (1) minute. Researchers will test the RFID s ability to fully power up and automatically go into tag read mode and read sample tags. Central Subsystem Tests Wireless Data Transfer Test The wireless router moves data from the output of the data logger (in pass-through mode) to a software application located at a central location. To test the data link, researchers will connect a laptop to the COM port 1 on the router backplane. They will use a remote (i.e., somewhere on the Internet, not from an Ethernet port on the router) terminal program (such as Windows HyperTerminal) to connect to the router via the wireless network. When a connection is made, researchers will type characters on the local laptop and verify via cell phone that the same characters were received by the remote terminal program. They will type characters on the remote terminal program and verify the characters are received by the local laptop. Remote Data Retrieval Test The data logger can be accessed via the wireless network and Internet to retrieve data if a long term communication outage has occurred. Researchers will test the ability to retrieve data from the logger remotely by opening a connection to the router COM port 1 using an Internet hosted terminal program. Researchers will send the proper commands to the data logger and verify that stored data are returned to the network terminal application. Data Retrieval Application Test As described in the Final Design Document, RFID readers will transmit the data through a wireless data modem to a central computer at the TTI El Paso office. A core application in the central computer will retrieve data from data loggers at a pre-determined time interval. The application will be tested for resiliency to make sure the application does not fail completely in the event one or more RFID reader fails. A separate application will be used to read matching tag reads from the Raw Tag Archive table within a pre-determined time interval. The algorithm to match tag identification will be thoroughly tested using the live and simulated data to make sure the application correctly matches identical tags. Test and Evaluation Master Plan (TEMP) 4 November 21, 2008
Key Performance Parameters Building on the Field and Central Subsystem tests, two key performance parameters have been developed by the project team in order to ensure the system accurately collects border crossing times for U.S.-bound truck traffic. These key performance parameters are: Tag Read Time Stamp Accuracy A manual observation of a truck will be recorded as the truck passes a given detection station. The time recorded by the observing researcher will be based on a time-keeping system that uses GPS technology for accurate measure. The time of the system read will be compared with the manually observed time to measure the time stamp accuracy of the system. These two times must be within one (1) minute of each other 95% of the time in order to ensure the accuracy of the system. Accuracy of Crossing Time between Entry-Exit Readings As discussed above, the project team will distribute GPS data loggers to a small sample of trucks. A side-by-side comparison of crossing times measured by the RFID system and the GPS data loggers will take place for every truck equipped with both types of equipment during the evaluation phase of the project. The difference between the crossing times collected by RFID and GPS equipment should be within three (3) minutes of each other 85% of the time to guarantee the RFID system s accuracy. System Evaluation Plan The system evaluation plan consists of comparing crossing times of individual truck trips as measured by RFID system and GPS data loggers deployed inside the trucks alongside TTIprovided tags. GPS data loggers can log geographic position of trucks including time stamps at 1-second, 5- second, 20-second, 30-second, 1-minute, 5-minute, or 15-minute frequency. During the data collection, GPS loggers will be set to record data every five (5) seconds to reduce number of data stored in the device. Each GPS logger has 8MB storage, so it has enough capacity to hold several days of data collection. The data logger starts collecting position data once the device is powered on. To reduce the amount of excess data logged by the device and subsequent data processing time, the truck driver will be expected to connect the GPS to the power outlet 5-10 minutes before the truck leaves the Customs and Border Protection (CBP) compound (to provide enough time to acquire GPS satellites) and disconnect the device from the power outlet after leaving the DPS facility during every trip. TTI will distribute not less than 10 GPS data loggers to trucks and also instruct drivers to operate (power on and off) GPS data loggers while driving trucks to enter the U.S. through the BOTA port of entry. Test and Evaluation Master Plan (TEMP) 5 November 21, 2008
Appendix A: Border Crossing Time Measurement System Test and Evaluation Master Plan Summary Matrix Operational Capability Solar Power System Test 24 Volt DC Power System Test Border Crossing Tag Read Test Tag Read Reliability Test at Typical Vehicle Speed System Latency Data Logger Test Parameter Field Subsystem Tests Measures amperage from the solar panel array (two panels), the voltage on the battery bank, and the amperage which is being drawn by the load Determines whether proper voltage is applied to the RFID reader and programmable relay with both units connected and drawing power Verifies that each RFID reader is emitting RF energy, receiving reflected RF from the sample tags, and is configured properly to decode the tag identifier (tag protocol setting on RFID unit) Compares the number of tags read by the tag reader with the number of tags manually verified for the same vehicles Determines the time required for the system to function after being turned on Verifies that the data logger contained in the detection site is configured to operate in passthrough mode to time-stamp incoming tag reads from the RFID unit and locally store a copy of each record. In pass-through mode, all data coming from the RFID reader are time- stamped (creating a data record), a copy of the record is stored locally on a flash memory card and the record is passed through to the wireless router COM port 1 for transport to the Central Subsystem. Capability Threshold Voltage on the battery bank between 11.5vDC and 14vDC, load current consumption less than 3.5 amps Voltage present at the output of the 24vDC converter Two tags, affixed to the windshield of a test vehicle moving at the expected speed of large trucks speeds, are read from data collected using Windows HyperTerminal on a laptop computer connected directly (via RS-232) to the RFID unit. 85% of tags in sample size of 50 trucks are read from data collected System functional in no more than five (5) minutes Researchers will: (1) test the data logger operation by connecting a laptop computer running a terminal program (Windows HyperTerminal) to the port labeled Configuration Port. The Data Port should be connected to the RFID reader; (2) clear any files or create a new file on the data logger to hold the upcoming test data; (3) with the test file configured, generate some RFID tag reads by putting a few sample tags in the tag reader s view. The data logger data indicator should flash when data arrives and tag data should appear on the terminal program (demonstrating pass through); (4) review the data on the terminal program to verify that a proper date and time has been added to the tag ID data; and (5) retrieve the test file (by sending the proper commands to the logger from the connected laptop) and check whether the same data on the terminal program was also logged. Test and Evaluation Master Plan (TEMP) A-1 November 12, 2008
Operational Capability Wireless Signal Strength Test Static IP Address Test Detection Station Communication Accessibility Test Detection Station Auto-Power Cycle Test RFID Reader Shutdown Test Parameter Field Subsystem Tests (Continued) Verifies that the cellular wireless signal strength is sufficient to maintain a continuous connection with the respective provider, typically a signal strength of at least -90dBm. Demonstrates that a detection station has a static (non-changing) Internet Protocol (IP) address so the station can be found on the wireless network. Verifies that the wireless router can be accessed over the cellular wireless infrastructure, since connecting a computer and reaching the Internet from the router is not sufficient to demonstrate that the router can be accessed from the Internet. Verifies that the detection station has the ability to sense the loss of communication to the wireless network. The auto rebooter (iboot) routinely pings a known server or servers on the Internet to insure a communication path out of the detection station. If ping responses are not received, the rebooter assumes the wireless network connection has been lost and cycles the power on the wireless router. Verifies that power can be cycled on the RFID unit in order to trip a programmable relay used to connect and disconnect power to the RFID reader. Capability Threshold Researchers will check the system status from the router web interface. Researchers will cycle the power and check the wireless router IP when it completes its reboot operation. They will access the IP information from a locally connected laptop using the router s Ethernet Interface. They will perform this test three (3) times with ten (10) minutes between power cycles to ensure the IP does not change with a power cycle. Retrieve the top webpage to show accessibility of the router s internal webpage via the wireless network from anywhere on the Internet by a user with the proper credentials. Researchers will test the local Auto rebooter by removing its Ethernet connection (pulling the network cable) from the wireless router. Without a network connection, the rebooter will not be able to contact any outside servers. Within the configured server inaccessible time, the unit should cycle the power on the wireless router. Researchers will ensure that the program provides power for a period of five (5) minutes then they will turn power off for one (1) minute. They will then test the RFID reader s ability to fully power up and automatically go into tag read mode and read sample tags. Test and Evaluation Master Plan (TEMP) A-2 November 12, 2008
Operational Capability Wireless Data Transfer Test Remote Data Retrieval Test Data Retrieval Application Test Operational Capability Tag Read Time Stamp Accuracy Accuracy of Crossing Time Between Entry-Exit Readings Parameter Central Subsystem Tests Verifies that each wireless router moves data from the output of the data logger (in pass-through mode) to a software application located at a central location, using a remote terminal program to connect to the router via the wireless network Verifies that the data logger can be accessed via the wireless network and Internet to retrieve data if a long term communication outage has occurred, by testing the ability to retrieve data from the logger remotely by opening a connection to the router COM port 1 using an Internet hosted terminal program. Verifies that RFID readers can transmit the data through a wireless data modem to a central computer inside the TTI El Paso office, where a core application in the central computer will retrieve data from data loggers at a pre-determined time interval. Parameter Compares RFID read time to manually recorded time of passage Compares crossing time obtained by automated RFID system to crossing time derived from GPS installed in trucks. Key Performance Parameters Capability Threshold When a connection is made, researchers will type characters on the local laptop and verify via cell phone that the same characters were received by the remote terminal program. They will type characters on the remote terminal program and verify the characters are received by the local laptop. Researchers will send the proper commands to the data logger and verify that stored data are returned to the network terminal application. Researchers will test the application for resiliency to make sure the application does not fail completely in the event one or more RFID reader fails. Capability Threshold Differences in times are within 1 (one) minute of each other 95% of the time Difference between crossing times obtained by RFID and GPS, for which the truck will have both technologies, is within three (3) minutes for 85% of total trucks equipped with GPS and RFID tag. Test and Evaluation Master Plan (TEMP) A-3 November 12, 2008