Key Phrase Detection Using Compact CNN Accelerator IP Reference Design

Transcription

1 Key Phrase Detection Using Compact CNN Accelerator IP FPGA-RD Version 1.2 October 2018

2 Contents 1. Introduction Related Documentation Software Requirements Hardware Requirements Directory Structure Key Phrase Detection Overview Block Diagram Top Level Blocks Top Level Module mdp_audio_top.v Compact CNN Accelerator Engine lsc_ml_ice40_bnn.ipx SPI Loader spi_loader2_spram.v I2S Master is2m_in.v Audio Buffer ice40_audio_buffer.v Filter Bank Storage ice40_audio_fb_storage.v Audio to Fingerprint ice40_audio_fb_cu.v, dnn_ice40_fc_eu.v, and dnn_ice40_mul.v Clock Generator ice40_audio_clkgen.v Comparator/RGB Driver Instantiated in top module Generating the Firmware File... 7 Technical Support Assistance... 9 Revision History... 9 Figures Figure 5.1. Key Phrase Detection Directory Structure... 3 Figure 6.1. Key Phrase Detection Block Diagram... 4 Figure 6.2. How Data/Address is fed into Compact CNN Accelerator for Key Phrase Detection... 6 Figure 7.1 SensAI Project Settings Part Figure 7.2 SensAI Project Settings Part Figure 7.3 Fractional Bit Change FPGA-RD

3 1. Introduction The Lattice Key Phrase Detection using Compact CNN Accelerator document describes how to implement a Key Phrase Detection design on our ice40 UltraPlus FPGA. This design utilizes the Lattice Radiant Compact CNN Accelerator IP core which is optimized for Binary Neural Network implementations. 2. Related Documentation In addition to using this guide to help you get started developing Compact CNN solutions on your device, you can refer to other applicable documents that may contain more detailed information that is beyond the scope of this guide. The following documents can be obtained on the Lattice website: Lattice SensAI Neural Network Compiler Software User Guide (FPGA-UG-02052) - This document explains how to create the firmware file which contains the command sequence as well as weights that go into the Compact CNN Accelerator IP Core. Compact CNN Accelerator IP Core User Guide (FPGA-IPUG-02038) - This document provides additional detail about the Compact CNN IP Core contained in this design. Lattice Radiant Software User Guide - This reference design is designed in Lattice Radiant software, and this user guide explains everything you need to know about Lattice Radiant. 3. Software Requirements The following software requirements are required in order to design and use the Key Phrase : Radiant Software 1.0 or greater - This is used to build the reference design. Radiant Programmer tool - This is used to program the bitstream to the External SPI Flash on the board. Compact CNN Accelerator IP Core - This is used to create the Compact CNN Accelerator Engine. Lattice SensAI Neural Network Compiler Software This is used to generate the firmware file from the TensorFlow/Caffe outputs. Note: If you re using Radiant Software 1.0 with Service Pack 1, optimize the reference design with Area by selecting the Radiant Software tab. Navigate to Project>Active Strategy>Synplify Pro Settings, and tick the Area checkbox. 4. Hardware Requirements The reference design has been targeted to support the Mobile Development Platform board but can be modified for use in other HW as needed. 5. Directory Structure Figure 5.1 shows the directory structure when unzipping the Implementing Key Phrase Detection Using Compact CNN Accelerator IP Source Code. The figure explains what files are contained in each folder. Figure 5.1. Key Phrase Detection Directory Structure FPGA-RD

4 6. Key Phrase Detection Overview 6.1. Block Diagram The purpose of this is to show an implementation of a machine learning design with our Compact CNN Soft IP Engine. The Key Phrase Detection Design will be first described with a block diagram of the design, followed by a description for each top block module. Figure 6.1 provides a top level diagram of the Key Phrase design. While working on this design, please note that different PAR iterations can give different results. It is better to try more than one PAR iteration for best results. ice40 UltraPlus External Flash SPI Loader (spi_loader2_spram) Compact CNN Accelerator Engine (lsc_ml_ice40_bnn.ipx) Result (3 Classes) Comparator & Window Filter LED Filter Bank Storage (ice40_audio_fb_storage) Audio to Fingerprint Audio Samples External Microphone I2S Master (i2sm_in) Audio Buffer (ice40_audio_buffer) Figure 6.1. Key Phrase Detection Block Diagram The block diagram is separated into white and gray. White blocks are the main top level module implemented on the ice40 UltraPlus, while gray blocks are external parts. There are two main inputs that go into the Compact CNN Accelerator Engine, which then outputs three values that get evaluated in order to determine whether the key phrase is detected or not Top Level Blocks This section discusses the details of each main top level modules that makes up the Key Phrase Detection Reference Design. Each top module includes which Verilog (.V) or IP Catalog (.IPX) file(s) it contains. These files can all be found in the source directory of the zip file Top Level Module mdp_audio_top.v This is the top level module that contains all the blocks and connections found in Figure FPGA-RD

5 Compact CNN Accelerator Engine lsc_ml_ice40_bnn.ipx This block contains the Compact CNN Accelerator Engine. In this design, the engine is configured to Machine Learning Type: BNN, Memory Type: Single SPRAM and BNN Blob to +1/0. To understand what this means, please refer to the Compact CNN Accelerator IP Core User Guide (FPGA-IPUG-02038). This IPX file takes the binary file from the External SPI Flash and the Fingerprint from the External Microphone to output three values: Silence, Key Phrase, and No Key Phrase SPI Loader spi_loader2_spram.v This module reads the external flash for two different files: the command sequence file and the filter bank storage file. These two files are then outputted into their respective modules. The binary command sequence file is outputted to the Accelerator IP Core, while the filter bank storage file is sent to the filter bank module. This module is programmed to read the filter bank storage file at address 20 h20000, and the firmware file at address 20 h This firmware file is generated by the Lattice Neural Network complier tool. The input to the tool are the.caffemodel and.proto file. These files are located in the NN_Compiler_Files directory for you to generate your own firmware file I2S Master is2m_in.v This block communicates with the I2S microphone on the MDP board. It receives the audio serial data from the External Microphone and transports the data to the audio buffer module. This block also helps calculate whether the audio data level is high enough to drive the output LED light Audio Buffer ice40_audio_buffer.v This module stores the audio that is inputted from the I2S master above in two SPRAM blocks. When instructed, it will then transmit the stored audio data into the Audio to Fingerprint section Filter Bank Storage ice40_audio_fb_storage.v In addition to the firmware file, this design requires a filter bank storage file that is also read from the external SPI Flash. A filter bank is needed since the method we are using to analyze the audio is by converting audio data in a spectrogram-like picture. Creating a traditional spectrogram using FFT is computationally expensive, while using a filter bank is a lot more computationally cheaper Audio to Fingerprint ice40_audio_fb_cu.v, dnn_ice40_fc_eu.v, and dnn_ice40_mul.v This section consists of three modules. The audio to fingerprint is implemented in the top level module. It takes the stored audio data from the audio buffer along with the filter bank values from the Filter bank storage, and creates two 32 x 32 images which then goes into the Accelerator IP. How these images are inputted into the Accelerator with their corresponding address is shown below in Figure 6.2. The reason for having two images instead of one is to have a much more accurate performance. FPGA-RD

6 Image 1 Start Address Image 2 Start Address 1024 Image 1 End Address Image 2 End Address 2047 Figure 6.2. How Data/Address is fed into Compact CNN Accelerator for Key Phrase Detection Clock Generator ice40_audio_clkgen.v This module is not included in Figure 6.1, but is still relevant. This module controls all the clocks in the design. Whenever a module is not being used, this module will stop transferring the clock signal to that module in order to save power. For example, if there is an extended amount of silence detected from the microphone, this module will cut off clock transmission to all blocks except the I2S Master. Once audio is detected again, then it will transmit the clock again Comparator/RGB Driver Instantiated in top module The comparator is a simple algorithm that takes the difference of computed key phrase and no key phrase. That value is then compared with a threshold value. If the difference is greater than threshold value, the LED stays or turns on. Otherwise, the LED stays off or turns off. There is a RGB driver implemented into the design to help with debugging. In the design, there are two thresholds that can be changed. Each threshold is compared to the difference between Key Phrase and No Key Phrase. If the certain threshold is met, either Red or Blue will turn on. Green LED turns on when noise is detected. This can help debug the thresholds for the main LED. 6 FPGA-RD

7 7. Generating the Firmware File To generate the Key Phrase Firmware file: 1. Using the files located in the NN_Compiler_Files directory, create a new project in SensAI and apply the following settings as shown in Figure Click Next. Framework Caffe Device Ultra Plus Class BNN Network File scmddet.proto Model File scmddet.caffemodel Image/Video/Audio Data 0d2bcf9d_nohash_1.wav Note: Change the file type when searching for this file. Figure 7.1 SensAI Project Settings Part 1 3. In the second section, apply the Neural Network engine settings as shown in Figure 7.2: On-Chip Memory Block Size Mean Value 0 Scale Value 1.0 Quantization Mode for BNN 0/1 4. Click OK. FPGA-RD

8 Figure 7.2 SensAI Project Settings Part 2 5. Note that this reference design requires fractional bit changes to fine tune the accuracy of the design. After analyzing the network, edit the fractional bit for each layer. After changing the fractional bit, save the project file and reanalyze the design. Figure 7.3 shows the change of the fractional bit. 6. Click Compile to generate the Key Phrase firmware file. Figure 7.3 Fractional Bit Change 8 FPGA-RD

9 Technical Support Assistance Submit a technical support case through Revision History Revision 1.2, October 2018 Section Software Requirements Key Phrase Detection Reference Design Overview Change Summary Added note in this section. Updated Figure 6.1. Key Phrase Detection Block Diagram in Block Diagram. Revision 1.1, September 2018 Section Change Summary All Renamed the document from Key Phrase Detection Using BNN Accelerator IP to Key Phrase Detection Using Compact CNN Accelerator IP. Changed all instances from BNN Accelerator to Compact CNN Accelerator. Software Requirements Key Phrase Detection Reference Design Overview Generating the Firmware File Revision History Added another bullet point for Lattice SensAI Neural Network Compiler Software. Updated Figure 6.1. Key Phrase Detection Block Diagram. Added this section in the document. Updated revision history table to new template. Revision 1.0, May 2018 Section All Change Summary Initial release. FPGA-RD

10 7 th Floor, 111 SW 5 th Avenue Portland, OR 97204, USA T