Homework 9: Software Design Considerations Team Code Name: Mind Readers Group No. 2 Team Member Completing This Homework: Richard Schuman E-mail Address of Team Member: _rschuman_ @ purdue.edu Evaluation: SEC DESCRIPTION MAX SCORE 1.0 Introduction 5 2.0 Software Design Considerations 30 3.0 Software Design Narrative 30 4.0 Summary 5 5.0 List of References 10 App A Flowchart/Pseudo-code for Main Program 10 App B Hierarchical Block Diagram of Code Organization 10 TOTAL 100 Comments:
Introduction The Mind Reader is a mobile brain-to-computer interface. The main function of the project is to navigate through an interface overlaid on a live video feed by tracking eye movements via an EOG, and reading concentration levels from an EEG. The main interface program will be running on a Beagleboard-xM computer, which will be obtaining a live stream from a USB webcam, and the receiving the navigation data from a separate microcontroller. The microcontroller will be used mainly to control information flow by interfacing with a multiple off-chip Analog-to-Digital Converters, an Artificial Neural Network on an FPGA, and then packing the data to be sent to the Beagleboard. The ADCs will be converting signals from the two separate EOG sensors placed near the eyes. The EEG signal will be read in through a UART connection found on both the microcontroller and dongle, then passed through an FFT function on the microcontroller. 2.0 Software Design Considerations There are four main devices guiding the total design of the software package. These are the Beagleboard-xM, the DLP FPGA, the DSPIC microcontroller, and the TI ADC. Each of these devices have limitations that need to be considered when developing the software to perform their individual tasks. The Beagleboard uses an ARM Cortex A8 processor that runs around 1 GHz and contains 512 MB of RAM [7]. As a result, the main interface program needs be optimized to take full advantage of what little memory is available. For the live stream video capture, the frames will have to be drawn to the screen at a minimum rate of 30 Hz in one of two ways. The first consideration is using the GtK GUI windows provided by the OpenCV libraries. Unfortunately, this library requires the use of window manager such as the Ubuntu Desktop GUI, which severely cuts back on the amount of RAM and processing time available. The second option currently being researched is drawing the frames directly to the screen from the bare terminal, in order to fully utilize the RAM. The USB 2.0 webcam will be transferring frames at speeds near 35 MB/s, so the frame size needs to be limited to 640x480 in order to achieve the minimum refresh rate. The single board computer will also be obtaining the EOG and EEG data from the microcontroller through an SPI interface. The SPI interface can run at a max speed of 48 MHz and contains a 16-bit shift register [6]. However, the SPI on the microcontroller is limited to a speed of around 1 MHz which will bottleneck the total packet transfer speed [5]. -1-
Since 2 bytes of each packet will be sent a time, the way the data is shifted in will need to be considered as well. Currently, the ARM Cortex processor is configured as a little endian addressing mode, where the processor reads data from a low address to a high address [7]. An example of this is if the 16-bit memory location contains the value 03E8, the processor will read it as E803. As a result, extra processing may be necessary to place the bytes in the correct order. The DSPIC microcontroller is the main communication component of the project. In order to communicate with the other devices, the microcontroller will be using 3 SPI modules to communicate with the two external ADC modules and the Beagleboard. It will also utilize a UART connection in order to receive the data off the EEG dongle. Many things needed to be considered when designing the software for microcontroller. For the ADC modules, the SPI on the microcontroller will be communicating with a small SPI module on the ADC. Since the ADC has a 1 MHz transfer limit, the registers must be set to either match or go below this [4]. To accomplish this, the SPI will be running off of an external 7.37 MHz oscillator and the internal register configuration will set the SPRE and PPRE bits to 6 and 2 in order to divide the clock by 8, yielding a 0.92 MHz clocking signal [5]. While the shift register limit is 16-bits for the Beagleboard and 4 bytes for the ADC, the microcontroller contains an 8 word FIFO for each available SPI block, so shift memory won t be an issue [5]. Our The ADC also has an active low DRDY pin which is toggled low when the converted data is ready [4]. The microcontroller will run the SPI using an interrupt based on this pin. When communicating with the Beagleboard, the microcontroller will act as an SPI slave, and simply transfer the shift the data accumulated in the FIFO 16-bits at a time. The Beagleboard will be configured to read the data in the FIFO at a rate slightly faster than its being entered so as not to overflow the FIFO buffer. The micro will also be talking with the EEG via an interrupt based UART connection. In order to effectively talk with it, the UART settings need to be adjusted to a baud rate of 115200, 1 stop bit, no parity, and 8 data bits [3]. Total memory usage is estimated to be 5kb for the program and 2kb for the data out of the 512 KB Flash and 52 KB RAM on the microcontroller [5]. 3.0 Software Design Narrative Upon power up, the microcontroller will immediately set up the on-chip peripheral registers. Two of the SPI modules will be set as masters for the external ADC chips and the clock will be assigned to the ~1 MHz rate. The third SPI will be set as a slave to the Beagleboard with a clock -2-
rate of ~1 MHz as well and then the register will be cleared. The UART will be set to the 115200 baud rate and bit configuration setup. Afterwards, the micro will send the configuration bits to the ADCs. Following Appendix G chart, the microcontroller will send out the 8-bit instruction code to tell the ADC that it wants to configure it. This will then be followed by a 32-bit write cycle following the same chart to set up the ADC modules with the correct sampling rate and conversion configurations. The ADCs will then be set into a continuous read mode where the data will be placed into the SPI register for the micro to read instead of needing the micro to request it [4]. The next boot step for the micro is to initiate the handshake with the EEG dongle. Following the chart on Appendix C, the microcontroller will send an AutoConnect byte (0xC2) to the dongle and expect a (0xAAAA04D1) back from the set [3]. If a different set of bytes are received as outlined in the chart, the micro will need to send a Disconnect byte (0xC1) and start over [3]. Once the connection has been established, the micro will poll the SPI slave register from the Beagle to see if it is active. If the register is still cleared, then the Beagle is still loading. Once the superfluous handshake data has been shifted in, the micro will enable its interrupt generation and begin its data operations on the EOG and EEG data. While the micro is performing its initialization setup, the Beagle should be loading its kernel. Once the kernel is loaded and ready, it will start the main program automatically. At the start of the program, the Beagle will access and load all of the pre-determined GUI components such as the icons into RAM. Using the OpenCV libraries, it will also establish a connection with the USB camera and prepare the master SPI module on the extension header [1]. Once the initialization is complete, the Beagle will send a 0x01 to the micro to signal that it is on. After delaying a second or two, the Beagle will read off the micro SPI to see if new data has been placed inside. If so, then the connection has been established, otherwise it ll continue to send out the 0x01 until it is ready. At this point and time, the initialization has been planned but not yet written or tested for the micro or Beagle. Once the setup is complete, the micro and the Beagle will be running multiple independent tasks. The micro will mainly be waiting for an interrupt from either the ADCs or the EEG. For the ADC, the interrupt will be triggered when the DRDY signal is set low. Using the chart on Appendix H, the micro will read off the digital EOG values and convert them to a 10-bit output on the GPIO lines to the FPGA. After a yet-to-be-determined delay based on the critical -3-
path of the ANN, the micro will then read in the 4 bit GPIO output and create a 16-bit EOG packet to be placed in the Beagle SPI FIFO. The packet will be a 0x01 followed by a 3-bit code determining the eye direction or error code. (See Appendix I for more details). Another interrupt will be generated from the EEG dongle UART. Following the chart on Appendix D, the micro will look for a 0xAAAAAA. If so, it will read in the length and data bytes then run a checksum check on it. If an error occurs, then the packet will be dropped and the process will restart. Otherwise, the data will be parsed according to the chart found on Appendix E. [3] If an error occurs, the packet will be dropped. Afterwards, the gathered data will run through 3 functions in the FFT library to do a Fourier Transform on the data. Once the data is transformed, the data will be compiled into a 16-bit EEG concentration packet and a raw EEG data packet. For the raw EEG packet, the MSB will be set to one and then other 15 bits will be the raw data. For the concentration packet, the first byte will contain a (0x02) signifying it s a Concentration packet and the last 6 bits will be the concentration level off the dongle. Both packets will be shifted into the Beagle SPI FIFO for reading later. On the Beagleboard, once the handshake with the micro is complete, the program will spawn 4 separate threads to perform the needed tasks. The first thread is the live-stream overlay thread. Using the OpenCV library code, a single frame from the webcam will be extracted and placed as the background in the frame buffer [1]. The thread will then obtain a lock on the EEG data values, the concentration levels, and the EOG eye coordinates. It will then use this data to compute the overlay graphics. The raw EEG values will be plotted on a mini graph on the bottom of the screen. The eye coordinates will be used to highlight an application icon from the main menu. The EEG concentration levels will then be used to update the concentration gauge on the selected application. These graphics will then be overlaid on the live frame in the buffer. The lock on the data will then be lifted. The buffer afterwards will be drawn as a single picture to the video glasses. A 17 ms delay will then be implemented to give other threads substantial processing time. The communication thread will read in 16 bits from the SPI register shifted in from the micro. The beagle will then decode the data using the by examining certain bits in the first byte to determine what packet it is and where the desired data is. The thread will then obtain a lock variables associated with the type of packet that it is. Once the data has been updated, the lock will be lifted and another small delay around 1 ms will be implemented to allow other threads to -4-
have access to the core. The application thread will activate when an app has been fully selected as determined by the communication thread. The app will then read utilize the necessary data from the EEG and EOG as its respective function needs. Every app will have an option to return to the home screen in which the data will be used to select other applications. For the main program code, only a live stream frame with facial recognition has been fully written and tested. No progress has yet to be made on the apps or the microcontroller. For error detection, the Beagle will send a handshake packet if it reads a 0x00 from the SPI register and display warnings to the user if an error code arrives. There is also the fourth SPI watchdog thread which follows the chart found on Appendix F. The thread will read data from SPI and check to see if it is the same value as the previous one found. If so, a counter will be accumulated until a 3 sec threshold is reached at which point an error will be set prompting a reset of the micro. The microcontroller will handle errors for onboard debugging by placing yetto-be-determined error codes within the proper packets. So far, the only error codes that have been determined are within the EOG packet since EEG data from the UART is dropped during an error. The 3 bit error code for EOG is as follows. If a 101 is read, then no signal was received from the horizontal EOG, a 110 is no signal from the vertical EOG, and a 111 is no signal from either. 4.0 Summary The software for the Mind Reader needs to consider many factors during development. The maximum clock for the ADC and the minimum clock scalars on the microcontroller put a large limit on the SPI speed. The Beagleboard s limited RAM and CPU speed also are a substantial hurdle to overcome. Overall, the code flow is broken down into as simple and independent tasks as possible to fully utilize the core s power and limited space. The microcontroller will gather its data through interrupts while the Beagleboard will read and use the data during pre-defined, optimized intervals. Code for errors and debugging has also been considered and planned in order help speed the final design along to completion. -5-
5.0 List of References [1] OpenCV, OpenCV Documentation, October 24 2012 [Online]. Available: http://docs.opencv.org/ [2] NeuroSky, "MindSet Communications Protocol", June 28, 2010. [Online]. Available: http://wearcam.org/ece516/mindset_communications_protocol.pdf [3] NeuroSky, "MindWave Dongle Communication Protocol", May 24, 2011. [Online]. Available: http://developer.neurosky.com/docs/lib/exe/fetch.php?media=app_notes:mindwave_rf_exte rnal.pdf [4] Texas Instruments, ADS1210, September 2005. [Online] Available: http://www.ti.com/lit/ds/symlink/ads1210.pdf [5] Microchip, DSPIC33EP512MU10, November 2011 [Online] Available: https://engineering.purdue.edu/477grp2/nb/matt/dspic33ep512mu10%20datasheet.pdf [6] Brian s Life. SPI Bus on the Beagleboard-xM, February 2012 [Online] Available: http://www.brianhensley.net/2012/02/python-controlling-spi-bus-on.html [7] ARM. ARM Cortex A8, May 2010 [Online] Available: http://infocenter.arm.com/help/topic/com.arm.doc.ddi0344k/ddi0344k_cortex_a8_r3p2_tr m.pdf -6-
Appendix A: Flowchart/Pseudo-code for Main Program -7-
Appendix B: Hierarchical Block Diagram of Code Organization -8-
Appendix C: EEG Connection Flowchart -9-
Appendix D: EEG Data Packet Receive Flowchart -10-
Appendix E: Data Packet Parsing Flowchart -11-
Appendix F: Beagleboard-Micro Watchdog Flowchart -12-
Appendix G: External ADC SPI Write Flowchart -13-
Appendix H: External ADC SPI Read Flowchart -14-
Appendix I: SPI Data Packet Table Packet/ Bit 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 EOG 0 0 0 0 0 0 0 1 0 0 0 0 0 EOG EOG EOG EEG Raw EEG Concen. 1 Raw Raw Raw Raw Raw Raw Raw Raw Raw Raw Raw Raw Raw Raw Raw 0 0 0 0 0 0 1 0 0 EEG Conc EEG Conc EEG Conc EEG Conc EEG Conc EEG Conc EEG Conc Raw 15 bit signed int EEG Conc. 7 bit unsigned int EOG CMD 000 - Nothing 001 Go up 010 Go down 011 Go left 100 Go Right 101 Lost hor. signal 110 Lost ver. signal 111 Lost both signal -15-