BlueCoin, the Robotic Ear

Transcription

1 BlueCoin, the Robotic Ear Audio and Sensor Platforms R&D AST Central Labs

2 Voice Communication a key driver of innovation since 1800 s 2

3 IoT evolution of Voice Automation: the IoT voice assistant 3 How can I help you? to Smart Mobiles to Anything Connectable From professional PC applications to Home / Office Terminals

4 MEMS microphones and Audio Quality at system level 4 Voice Terminal Audio capture & render Signal processing Low power Constained geometry Voice & data Gateway Seamless connectivity Cloud Natural Language Processing Dialogue Management Services Play Music Control Lighting, heating, News, sport, traffic, weather, Answer questions, create to-do lists, shopping lists, Place orders online, use other online services: taxi, pizza,

5 Bluecoin

6 BlueCoin: the Robotic Ear 6 Augmented hearing plus range, environmental and motion sensing Motion, Activity and Balance Acoustic Beamforming Voice over Bluetooth Low Energy Embedded Processing Sound Localization

7 BlueCoin: the Robotic Ear 7 Features Advanced audio processing Sound Source Localization Beamforming Wide band audio over BLE (BlueVoice) Advanced Audio and ToF User Interaction Sensor fusion Inertial, environmental, acoustic, optical (Time of Flight) Complete development kit SW Compatible with STM32 ODE CoinStation + small Cradle BlueCoin + battery 25mm

8 STEVAL-BCNKT01V1 - BlueCoin Starter Kit 8 Programming development kit Standalone mode bundle Standalone mode bundle CoinStation Programming cable BlueCoin Cradle Plastic Box Programming development kit LiPo Battery

9 STEVAL-BCNCS01V1 - BlueCoin 9

10 STEVAL-BCNCS01V1 - BlueCoin 10 LSM6DSM 6-axis inertial module LSM303AGR ecompass LPS22HB Barometer 4x MP34DT04-C1 MEMS Microphone Expansion connectors 2x buttons STBC03J Power management STM32F446 MCU BlueNRG-MS Bluetooth Low Energy 25 mm SMD Antenna BALF-NRG-01D3 Balun Filter 8 LEDs

11 2.048 MHz PDM BlueCoin Architecture 11 Vin ( V) STBC03J Battery charger With 3V LDO VBat Vsys VDD 3V 4x MP34DT04-C1 MEMS microphones Integrated antenna BALF-NRG-01D3 Integrated balun 16MHz LSM6DSM acc+gyro LSM303AGR acc+mag LPS22HB barometer I2C 400 KHz STM32F446 Cortex-M4F 180MHz SPI 4-wire 5 MHz BlueNRG-MS Bluetooth MHz NRST SWD SPI I2S UART USB I2C PDM SDIO

12 BlueCoin Power Supply 12 STEVAL_BCNCS01V1 VIN [4.5V-5.5V] VDD [3V] STBC03J LDO SYS Bat 150mA max VDD 3V STM32F446 Sensors BlueNRG-MS VSYS VIN or VBat Battery Connector VBat

13 STEVAL-BCNST01V1 - CoinStation 8 Stereo Audio DAC USB connector Expansion connector 36x36mm VL53L0X Time-of-Flight sensors LDK120M 2.8V LDO 3.5mm Jack SWD Programming interface

14 VL53L0X FlightSense TM 14 FlightSense Principle CoinStation Distance measurement Field of View Time-of-Flight: VL53L0X Ranging up to 2 meters 25 degrees Emitter Sensor Photon Target Accuracy +/-3 % (*) Programmable modes 3 modes in API (High-speed, long distance, high accuracy) distance Small all-in-one module 2.4 x 4.4 x 1 mm Measured distance = Photon travel time /2 x Speed of light Time-of-Flight Low power consumption HW stdby <5uA Ranging: 20mW (average at 10Hz with 33ms ranging sequence)

15 STEVAL-BCNCR01V1 ucradle 15 Support for standalone operation TOP VIEW BOTTOM VIEW BlueCoin Connectors Micro SD Card Battery Connector ST1S12XX 3.3V DCDC converter Micro USB Plastic Case

16 BlueCoin Integrated Development Platform 16 Core System Coin Station Cradle

17 Entry Level to Advanced Audio in 3 steps 17 Integrated Development Platforms Advanced Free-Licensed Premium libraries Audio Over BLE Beamforming Source Localization Acoustic Echo-Cancellation MEMS microphones on Function Packs

18 BlueCoin compatible software 18 STSW-BCNKT01 Embedded software examples for BlueCoin firmware package, including sensor data streaming via USB and BLE, data logging on SD card, gesture recognition, audio acquisition and playback FP-SNS-ALLMEMS1 (former BLUEMICROSYSTEM2) IoT node with BLE connectivity, digital microphone, environmental and motion sensors, motion and audio middleware libraries FP-AUD-SMARTMIC1 (former SMARTACOUSTIC1) Smart audio IN-OUT software expansion for STM32Cube FP-AUD-BVLINK (former BLUEVOICELINK1) Software Free Download from in June 2017 Bluetooth Low Energy and microphones software expansion for STM32Cube

19 MEMS microphone array Spatial Audio Processing 19 Source Localization osxacousticsl Freely licensed FW Libraries for STM32 Beamforming osxacousticbf Beamforming: osxacousticbf Spatial Filter Outputs the Audio that comes from a given direction - Voice Activity Detection - Statistical moments - Noise estimation -... Adaptively cancels audio signals coming from other directions Sound Localization: osxacousticsl Estimates the Direction of Arrival of the Main sound source Independent from beamforming May control the beam direction

20 Beamforming osxacousticec Acoustic Echo Cancellation 20 Removes echo of playback audio in speech capture application Single Microphone application STM32 is connected to both the microphone and the loudspeaker Known Audio Source e. g. music / voice AEC (estimates room reverberation) Reverberant Room The Open.AUDIO AEC library is an optimized STM32 port based on the Open Source project Speex:

21 Putting together SW libraries SmartAcoustic1 Example project in source code built on STM32Cube software technology Includes acoustic Beam Forming, Echo Cancellation, and Source Localization. Immediate test and performance evaluation 4-MEMS microphones array Source Localization User-selectable angle resolution User-selectable activation treshold Based on 4 MEMS microphones 360 localization range 21 reference audio Beamforming Acoustic Echo Cancellation User-selectable neam direction User-selectable beamforming algorithm Based on 4 MEMS microphones GUI highlights the chosen microphone couple Based on a single MEMS microphone Reference audio is stored on STM32 FLASH Uses Audio OUT to play back audio while streaming cleaned speech on USB

22 FP-AUD-SMARTMIC1 - Beamforming 22 Beamforming = spatial filter Record only sounds arriving from the selected direction Omnidirectional mic Beamforming output

23 FP-AUD-SMARTMIC1 - BeamToSource 23 Select the desired Beamforming Switch on the Source Localization Beam to Source can understand the source direction and select the related mics

24 Time-Of-Flight Gain Control 24 With Gain Control Example of Automatic Gain Control: Volume changes automatically according to the distance of the speaker measured by the Time of Flight sensor Without Gain Control

25 Acoustic Echo Cancellation 25 AEC Output Omnidirectional mic

26 BlueVoice: audio over BlueTooth 4 26 The ST HW and SW solution for Ultra low-power Voice streaming over Bluetooth Low-Energy Bluetooth Low Energy MEMS Microphone Microcontroller BlueNRG RF connectivity STM32 Signal Processing and Application Firmware & Software BLUEVOICELINK BLE and microphone reference application based on STM32Cube OSXBLUEVOICE BLUEMICROSYSTEM BlueMS, BlueST-SDK Voice-over-BLE vendor-specific profile library for STM32 and BlueNRG Voice and Sensor data over a BLE link to an Android - ios Smartphone App Bluetooth Smart and Sensors Technology Application for Android and ios

27 BlueVoice Mapping over Standard BlueVoice Vendor Specific Profile Bluetooth 4.0 protocol stack 27 Central Unit (Master) Client Audio processing GAP configuration GATT configuration Peripheral Unit (Slave) Server Application: BlueVoice Profile (vendor specific) Generic Attribute Profile (GATT) Attribute Protocol Generic Access Profile (GAP) Security Manager Logical Link Control and Adaptation Protocol Server Client Host-Controller Interface Audio is exported as a Service Service Characteristic Service Characteristic Link Layer Direct Test Mode Physical Layer Characteristic Descriptor Characteristic Descriptor Bluetooth Low Energy Stack

28 ST BlueMS & BlueVoice library BlueVoice Service Integration in BlueMicrosystem environment 28 ST BlueMS available on Google Play and App Store Cloud-based ASR Service BlueVoiceLink Peripheral osxbluevoice khz ADPCM

29 BlueVoice Architecture: 29 Audio processing and transmission Server - TX Audio Acquisition PDM PDM to PCM conversion PCM Audio Compression Raw Data symmetrical architecture for bi-directional communication Audio Decompression Serial Audio Out USB, I2S, Client - RX

30 BlueCoin Support for BlueVoice 30 BlueCoin Terminal Mobile Device Cloud-based Services «Natural Language» Platform Signals Comm Interface

31 Samantha VUI Conversational Interface dev with Android and BlueCoin 31 CoinStation+BlueCoin 2x VL53L0X Asymmetric BlueVoice communication Your question transcription Your Question Voice input Beamforming 8 khz ADPCM 24 khz OPUS Your answer Cloud-based ASR Service Gesture Recognition To start communication Your Answer Voice output Computational Knowledge Engine