WiiArtist ECE532 Design Project: Group Report

Transcription

1 WiiArtist ECE532 Design Project: Group Report Armia Nazeer Jie Hao Wu Jordan Varley April 9, 2012

2 Contents Overview... 1 Project Motivation, Description & Goals... 1 Architecture Overview... 3 Outcome... 5 Project Schedule... 6 Module Descriptions... 8 IR Glove... 8 WiiMote... 9 PC... 9 Microblaze Processor... 9 XPS_uartlite XPS_intc XPS_timer Video_to_RAM RAM_to_video Colour_detect Video_draw XPS_tft On-screen menu Design Tree Description Tips and Tricks References... 1

3 Overview Project Motivation, Description & Goals With touch screens becoming quite common on a number of consumer electronic devices (mobile phones, computers, vehicles, etc.) one particular limitations of these interfaces is their small size. Larger, more immersive solutions have been successful in implementation, such as the Microsoft Surface however their cost still makes them prohibitive to the bulk of consumers. By utilizing much more cost effective hardware, the design team found motivation to create a gesture-based drawing system, dubbed WiiArtist. The system will allow a user situated in front of the display projector (or monitor) to draw on-screen using their hands, without touching the screen. Four infrared (IR) LEDs, each attached to a finger will allow the WiiMote (from Nintendo) to track the location of the fingers, allowing the user to draw on screen by moving their hands. A physical palette will be provided to allow the user to select their drawing colours, as a traditional artist would. For this purpose, a standard video camera is provided to aid with colour detection alongside the WiiMote. Through an on-screen menu, the user will also be able to select artist tools such as brushes, erasers, etc. to use while creating their drawing. A high-level system diagram is provided below in Figure 1. WiiMote for IR (finger) tracking Video Connection Video Camera for Colour Detection BT Link FPGA Board Laptop Colour & Tool Selection IR LEDs for WiiMote to Track on User Fingers WiiArtist Drawing Area Projector Video Connecton Stationary Palette To select colours Figure 1. High Level System Diagram for the WiiArtist Design. 1

4 Figure 2. Detailed system Data Flow Diagram 2

5 Project motivation also exists from the technical side due to the system dependency on a mixture of digital and analog hardware, software that runs on both microcontrollers and PCs and a visual user interface that requires real-time image processing. Architecture Overview Refer to the image below for the WiiArtist system architecture. The majority of system components will be interconnected via the Processor Local Bus (PLB) that can interface between custom modules and the MicroBlaze microprocessor. A number of off the shelf modules will be sourced to assist with the system design, including the following: XPS_UART_LITE: used to interface the external PC with the MicroBlaze to transmit IR coordinates to the system Ram_to_Video: used to retrieve a frame from memory for use by a hardware module GPIO: used to allow access to external switches for palette colour selection XPS_TFT: used to output a video stream to the projector BRAM: used for local memory to aid the MicroBlaze and store coordinate values System components external to the XUP2VP board will operate as described in section 3 and will be sourced by the project team. Three hardware modules will be created to: Detect the colour that the user is selecting from the physical pallete o Inputs: Video Frames from VGA Camera o Outputs: Colour selected Draw video frames according to the user movement and colour/tool selection: o Inputs: Colour selected, Tool selected, Coordinates Outputs: Video frames for the output video stream on the projector Video_to_Ram: used to take a frame and store it in memory for buffering purposes Custom software will be implemented on the MicroBlaze to enable the following features: Coordinate filtering from the WiiMote Gesture detection based on the WiiMote coordinates Generation and Selection of Menu-based items 3

6 Figure 3. Detailed System Block Diagram. 4

7 Outcome Upon completion of the WiiArtist project, all of the features set out in the project proposal were delivered, with the exception of the on-screen tool selector and one of the tools to be selected. The on-screen tool selector posed a significant challenge to the project, particularly the dynamic operation we were targeting where the menu would slide up to appear and slide downwards when the menu was to go away. The successful implementation of this was not achieved due to the introduction of image corruption in the implementations the group had tried. Additionally, the paint can tool was not successfully implemented in both a reliable and quick method. An implementation that was either reliable or quick was achievable; however both were unsuitable for use in the WiiArtist. Nonetheless, our WiiArtist system still supports: o Drawing on-screen digitally by tracking up to four independent IR points in front of an IR camera o Detecting a gesture to both pull-up the on-screen palette placeholder and subsequently remove the menu o Detecting a gesture to change between tools from the on-screen palette placeholder o Use a physical palette to allow for drawing colour selection Further improvements to the system could be complete by addressing the following areas: o Adding additional filtering on the IR data transmission over the serial port to ensure a more continuous stream of data is transmitted and that the data is not periodically corrupted o Adding support for more than 8 colours to be detected and drawn on the WiiArtist system o Implementing a successful dynamic menu to improve the usability of the WiiArtist system The WiiArtist system also serves as a good foundation for future projects where more emphasis can be placed on the graphics used by the system. Future projects could focus on more complex artist operations such as: o Particular shapes such as circles and rectangles to improve the tools available to the user o More complex colour operations such as gradients and image textures to make the WiiArtist much more life-like o A PC could be taken out of the project if the Wiimote could directly interface with the FPGA If the project were to be undertaken again, additional emphasis could have been placed up-front to further define the implementation of the modules at the outset of the project. The WiiArtist team did define module interfaces during the design proposal stage, however treated each module as a black-box and did not define the implementation details. This led to a handful of discussions that were rather quick and informal, to 5

8 determine what the implementation details of each module would be for the upcoming milestone and by attacking this upfront from a system perspective, the WiiArtist development could have been streamlined more. Project Schedule The following table outlines the targeted and actual achievements for each milestone, in addition to briefly discussing the discrepancies that exist for some milestones. Table 1. Description of Milestone Targets and Discrepancies for the WiiArtist System. Milestone Target Achievement Actual Achievement 1 [Feb 8] o Demonstrate video input and output module capabilities o Demonstrate a video frame builder for video buffering o All targets met, with a bug in the frame builder that moved the pixel to the left with each successive write to memory 2 [Feb 15] o Demonstrate Coordinates from WiiMote Through UART to Microblaze Drawing Coordinates on Video Output (Cross-hairs at coordinate location) 3 [Feb 29] o Demonstrate Gesture Engine: Swipe Detection o Demonstrate On-Screen Pallete: Static Images for Menu o Demonstrate Video Draw: Eight colours at an input coordinate location(s) 4 [Mar 7] o Demonstrate Gesture Engine: Grab Detection o Demonstrate On-Screen Palette: Dynamic Menu Operation o Demonstrate Video Draw: Drawing Capabilities with Line Thickness and Pencil/Brush Tools o All targets met, with a bug in the UART sending corrupted numerical values periodically o All targets met except On-Screen Pallete, which had a purple box replace the actual menu image to show functionality of the menu o All targets met except On-Screen Pallete, which had the menu image sent into memory via a TCL script but wasn t replacing the purple box acting as a placeholder for the menu 6

9 5 [Mar 14] o Integrate Gesture Engine and On-Screen Palette Functionality o Demonstrate Video Draw: Image Erase Features and Paint Can Tool 6 [Mar 21] o System Integration for Basic WiiArtist (No Physical Palette) o Demonstrate Video Input to Colour Detection Module 7 [Mar 28] o Demonstrate Colour Identification and Selection from Stationary physical palette 8 [Apr 4] o Final Integration and Debugging for Demo Day o Both the On-Screen Palette and the paint can tool were not implemented successfully this week. The paintcan tool was able to work either reliably and slow, or quickly with some image corruption. A balanced implementation wasn t found for this milestone. The menu s dynamic operation was not successfully implemented due to on-screen image corruption when introducing the menu dynamics. The gesture engine was successfully integrated into the system, however. o All targets met, with exception of continued delays in integrating the menu s dynamic operation. A successful implementation was still hindered by the on-screen image corruption. o All targets met. o All targets met. 7

10 Module Descriptions The following sections describe in detail, each of the modules used in the WiiArtist system. IR Glove Origin: Group Description: IR Glove houses the IR LEDs for the WiiMote to track. It consists of a light winter glove, 3 IR LEDs and a dual-aa battery holder to power the setup. The LEDs are biased using a series 80-ohm resistor, to prevent the LEDs from overheating while allowing enough current to flow, permitting the LEDs to be bright enough for the WiiMote to track. The figure below followed by the brief schematic detail the IR glove used by the WiiArtist system. Figure 4. IR Glove used in the WiiArtist system. Finger 1 Finger 2 Finger 3 AA [1.5V] 80Ω 80Ω 80Ω AA [1.5V] Figure 5. Schematic of IR Glove used in the WiiArtist system. 8

11 WiiMote Origin: Group Description: The WiiMote is used to track the IR LEDs on the IR Glove and provide a set of coordinates describing the location of up to four independent IR points. The WiiMote communicates with the WiiArtist system using a Bluetooth link to a PC, described below. The hardware used in the setup is a standard WiiMote from the Nintendo Wii gaming system, with the Wiimote library sourced from an open source project developed by Johnny Chung Lee [1]. PC Origin: Group Description: A PC is used to read the IR coordinates from the WiiMote via Bluetooth, the coordinates are sent to the FPGA through UART. A small C# program runs alongside the WiiMote software to transmit the coordinates reported by the WiiMote software through the UART to the MicroBlaze system. Each coordinate is represented by 4 bytes (32 bits). Using 2 bytes to represent the X and 2 bytes to represent the Y coordinate. The coordinates are transmitted with the following protocol: 1) * - indicate the beginning of the sequence 2) X Indicates the next two bytes will be used for the X coordinate 3) Y Indicates the next two bytes will be used to represent the Y coordinate Microblaze Processor Origin: Xilinx XPS IP Core Library Version: 7.10.d Description: The Microblaze used in the WiiArtist system is used as a 32-bit soft processor and does the following tasks: Functions as the control unit for the system Receives input from the UART and extract the X and Y coordinate from it Sends appropriate commands to the hardware modules through the software accessible registers based on the inputs coming from the UART A detailed description of the functions involved with the microblaze: 1. Image Drawing/plotting: Using the selected tool, the drawing takes place on the coordinate that is being sent through UART (Details see xps_uartlite section). However, because not every single movement is captured on the PC due to the speed of Wiimote, we would see discontinuous dots. The Brehenham s line algorithm[2], the dots are connected if the distance between them is small. Otherwise, we plot them separately using the selected tool (see menu Selection for a list of tools). 2. Gesture engine: Determines if a gesture was triggered based on the coordinates. A number of gestures can be detected including swipe up, down, left and right as well as point and hold. These gestures allow the user to interact with the WiiArtist system and control the appearance of the menu, select colours from the physical palette and select tools from the on-screen menu to be drawn with. (See Gesture Engine below for more details) 9

12 3. Menu Selection: The menu provides a visual way for the user to select from the different tools and tool widths that the user can draw on screen with. Tools include pencil-points, slanted lines, squares, diamonds and vertical lines. 4. Colour Selection: Through use of the physical palette, the user is able to select from 8 different colours to draw with. The microblaze tracks the coordinates and enables the colour_detect module when appropriate to find the colour selected from the physical palette. XPS_uartlite Origin: Xilinx XPS IP Core Library Version: 1.00.a Description: The XPS Universal Asynchronous Receiver Transmitter Lite interface was used to receive serial data (the stream of IR coordinates) coming from the PC. XPS_intc Origin: Xilinx Version: Description: The LogiCORE IP XPS Interrupt Controller Interrupt is triggered by the XPS Timer to keep track of time. Since hand movement do not move as quickly as the clock speed. It relies on a number of flags that are incremented every 1 second to determine if a gesture was triggered. The following variables are incremented to determine if a gesture such as swipe or hold was executed: Unsigned short up[4]; Unsigned short down[4]; Unsigned short left[4]; Unsigned short right[4]; Unsigned short stationary[4]; Gesture Engine Origin: Group Description: The gesture engine is used to determine what actions we can take on the system. The gestures it supports include hold and point, up-down sequence, swipe left, swipe right. The hold and point gesture is used to bring up a menu. Swipe left, Swipe right: Select tools on the menu Up-down sequence: Close menu To determine the gesture, coordinates sent from the PC is compared with the previous value to determine a trend. Depending on the changes in the X and Y coordinates, we assign a direction to it. Once we have up[i] + down[i] + left[i] + right[i] > 30 (this number 10

13 is being tweaked around for sensitivity), a directional gesture is recognized using the direction that has the highest value. XPS_timer Origin: Xilinx Version: Description: The XPS Timer/Counter Timer feature is reset every second to trigger the timer interrupt, to keep track of system time and aid with gesture detection. Video_to_RAM Origin: Group Version: 1.00.a Description: This Custom IP was implemented using the PLB Burst Master IP Generator in XPS. This implements read and write capabilities in 16-word bursts to transfer video frames to/from the video_draw module. This module handles the low-level PLB bus transactions used to transfer data to memory via the MPMC so frames can be delivered to the xps_tft module and presents a higher-level FSM to allow the later modules to easily read and write to the PLB Bus. RAM_to_video Origin: ECE532 Demo System Version: 1.00.a Description: This IP was adapted from the ECE532 demo system to control the video decoder card and produce a video frame in memory to be used by the colour_detect module. A small modification was made to this IP to choose the correct video input from the VDEC1 card, as we were using an S-Video capable camera. Colour_detect Origin: Group Version: 1.00.a Description: This Custom IP was developed to take a frame co-ordinate and return the 3-bit colour corresponding to that location in the frame. This module is used for the physical palette feature where a colour to be drawn with is selected from the physical palette, by making a gesture. With the gesture detected, the MicroBlaze will wait a small duration of time before the colour_detect is enabled, to determine which colour the user had selected from the palette. Because the WiiArtist system supports 8 colours, only a 3-bit response is provided. Through user-acessible registers the MicroBlaze system is able to provide X and Y coordinates, instruct the module to start and then observe the result. The table below describes the register-level access available to the Microblaze. 11

14 Table 2. Software-accessible register map for the color_detect module. Register Address Register Name Register Description BASE_ADDR + 0x0 X-Coordinate Input to color_detect describing the X-coordinate to find the colour at Bits 22:31: X-coordinate Bits 0:21: Unused BASE_ADDR + 0x4 Y-Coordinate Input to color_detect describing the Y-coordinate to find the colour at Bits 23:31: Y-Coordinate Bits 0:22: Unused BASE_ADDR + 0x8 HW Control Control register to enable the hardware module Bit 31: Enable Bits 0:30: Unused BASE_ADDR + 0xC Color Result Output of colour_detect describing 3-bit colour found at X,Y coordinate Bits 29:31: 3-bit colour 0x0 Black 0x3 Yellow 0x6 Cyan 0x1 Red 0x4 Blue 0x7 White 0x2 Green 0x5 Purple Bits 0:28: Unused To determine the colour at the specified (X,Y) location, the hardware will sample a window of pixels around the (X,Y) location measuring 8 rows vertically and 16 pixels horizontally. This window is depicted in the figure below. The colour result from each pixel is separated into a blue, red, and green result in accordance with the XPS_tft method of describing the colour for each pixel. Each red, blue and green colour is averaged across the window of all pixels to come up with a resultant bit that is high if enough pixels had sufficient quantity of each colour. This implementation was sufficient for the WiiArtist system with the 3-bit colour support used throughout the system. Y-3 Sample Window Y Y+3 X-7 X X+7 (X,Y) Pixel Location Figure 6. Windowing used to determine colour at X,Y location. 12

15 Video_draw Origin: Group Version: 1.00.a Description: This Custom IP is used to implement the drawing capabilities of the WiiArtist system including: drawing with support for 8 different colours drawing with 6 different tools including pencil point (single pixel), slanted lines, vertical lines, square and diamonds erasing the drawing on the screen quickly, to start over The figure below illustrates the different tools the user can draw with. The width of these tools is user-selectable (between 1 and 31 pixels in 2 pixel steps) via the on-screen menu, allowing a variety of drawings to be created. Additionally, the 8 colours used by the WiiArtist system are displayed in the second figure. Figure 7. Tools Available to the User, selectable from the on-screen menu. Figure 8. Eight colours used by the WiiArtist system. As this is the main module that produces output frames for the xps_tft module, the following system functions are also implemented: Preserving the drawings already made on the screen to change only the present pixels being drawn Implementing menu dynamics for the on-screen menu Initializing the screen output upon boot-up with blue colour 13

16 The module is accessible from the Microblaze using software-accessible registers that are described in the following table. Table 1. Software-accessible register map used for the video_draw module. Register Address Register Name Register Description BASE_ADDR + 0x0 X-Coordinate Input to color_detect describing the X-coordinate to be drawn at Bits 22:31: X-coordinate Bits 0:21: Unused BASE_ADDR + 0x4 Y-Coordinate Input to color_detect describing the Y-coordinate to be drawn at Bits 23:31: Y-Coordinate Bits 0:22: Unused BASE_ADDR + 0x8 HW Control Control register to enable the video_draw module Bits 29:31: Debugging Bit 28: Enable Menu Bit 27: Disable Menu Bit 26: Debugging Bits 22:25: Tool Selection 0x0 Dot 0x3 Vertical Line 0x1 Slant Left 0x4 Square 0x2 Slant Right 0x5 Diamond Bits 18:21: Tool Width (Width = 1 + 2*Tool_Width) Bits 3:17: Unused Bits 0:2: Drawing Colour 0x0 Black 0x3 Yellow 0x6 Cyan 0x1 Red 0x4 Blue 0x7 White 0x2 Green 0x5 Purple 14

17 XPS_tft Origin: Xilinx Version: Description: XPS module to translate video frames in memory into the VGA signals used to drive the monitor. On-screen menu Origin: Group Version: 1.00.a Description: This module allows the developers to design an on-screen menu as a *BMP (Bitmap) file using a graphics editing program (ex. Paint or Photosop), parse and load the menu BMP file into memory. When the menu is activated, Video_Draw module will take the stored menu data and write it to the bottom one third of the screen. This allows for configurable and graphically appealing menus. The following image illustrates a graphical menu on the bottom 1/3 of the screen and drawing on the top 2/3 rd of the screen. Figure 9. On-Screen menu Illustration This menu block required the successful integration of the following blocks: 1) *BMP (Bitmap) image parser to determine RGB values for each pixel on menu in the xps_tft color format. This was a custom C-Program written by the group. 15

18 BMP files must be pre-processed using the following Linux command before being passed to the C parser. >convert -resize 640x160 menu.bmp menu.txt Where, 640 horizontal image resolution 160 vertical image resolution (bottom 1/3 of screen) menu.bmp bitmap image of menu (input to Linux command) menu.txt RGB values of each pixel in the file The C parser takes menu.txt output and converts it into the appropriate XMD commands to write the image data to corresponding address. 2) Means of storing RGB pixel values into SDRAM upon initialization. A XMD tcl script was written to write to SDRAM the appropriate menu RGB data. Writing to large portion of SDRAM using XMD tcl script is time consuming. A more optimal method (ex. Ethernet port or Compact Flash Card) can be implemented in the future to allow for fast demonstration of the system. 3) Modifications to the Video_Draw module mentioned above to read menu RGB data stored in DRAM upon initialization and write it to the bottom 1/3 of the screen upon menu activation. Source code for the custom C and Verilog used for this module can be found in the zipped design tree. Design Tree Description WiiArtist root directory System.xmp XPS Project file System.mss System Hardware Specification Files System.mhs System Software Specification Files Microblaze_0 Folder containing the component libraries including BRAM, UART and XMDStub files (source and executable) Data System Constraints File System.ucf Drivers Drivers for custom pcores Video_draw_v1_00_a Colour_detect_v1_00_a Etc Option files for bitgen and downloading Pcores Custom hardware modules Video_draw_v1_00_a Memory_accessor_v1_00_a Colour_detect_v1_00_a Script_tcl TCL scripts used to load images into memory Code Software code Mfs_files Program that is used to convert image into raw data to be sent to the SDRAM Main.c The main software control loop Wii C# code for the wiimote 16

19 Tips and Tricks Some additional tips the team can offer are the following: 1. Set up the infrastructure for your project early: directory structures, SVN repositories and the tools to iron out any issues early on 2. Use an SVN repository or some other version control scheme. 3. Check your hardware on the development board periodically success in simulation doesn t mean working on hardware on the board 4. Write scripts, scripts can help with testing significantly 17

20 References [1]Johnny Chung Lee (2011). Johnny Chung Lee Projects Wii. [Online] Accessed: January, Available: [2]Brensenham s line algorithm. Brensenham s Line Algorithm Wikipedia. [Online]Accessed: March, Available: