Aurora C++ SDK & Proof of Concept. Final Report

Transcription

1 Aurora C++ SDK & Proof of Concept Final Report CS 130 Spring 2018 Discussion 1A Team Name: Team Yaacov James Dickie ( ) Seth Eisner ( ) Benjamin Lawson ( ) Jaron Mink ( ) Yaacov Tarko ( ) Nicandro Vergara ( )

2 1 Table of Contents 1. Motivation and Application Overview 3 2. Improvements Since Midterm Report 3 3. Features UML Use Case Diagram Use Case Descriptions 4 4. SDK Design UML Class Diagram Application Architecture General Application Architecture API Class Backend Class Config Class Text Class Speech Class Interpret Class AudioFile Class External Libraries C++ Requests PortAudio JSON for Modern C Libsndfile Google Test 8 5. User Interface Documentation 9 6. Testing Unit Tests Example Programs Proof of Concept GitHub Links Individual Contributions 15 James Dickie 15 Seth Eisner 15 Benjamin Lawson 16

3 2 Jaron Mink 16 Yaacov Tarko 16 Nicandro Vergara 16

4 3 1. Motivation and Application Overview The Aurora C++ SDK is a software library that helps facilitate the use of Aurora API services in C++ applications. This application abstracts away the intricacies of recording audio, playing audio, and making API requests to Aurora servers through a custom library intended for use by developers wishing to receive these services. It's main use is for internet-of-things (IoT) devices and thus performance and dependencies are of utmost importance in this SDK. Additionally, we created a small proof of concept program to help explain the general workflow of the SDK and to demonstrate how these services can be utilized by a simple C++ program. The C++ port of the Aurora API software development kit provides a high-level interface for Aurora s text-to-speech, speech-to-text, and language interpretation services. C++ application developers, our users, are able to have their applications utilize the SDK in order to receive resources from Aurora's API, located on their remote servers. This API is already directly accessible for C++ developers via an HTTP API however, it it is far more accessible to developers if an SDK abstracts the HTTP requests away and provides idiomatic classes and functions for the concepts of speech, text, and interpreting. The C++ SDK should make it easy to integrate Aurora s services with embedded systems and IoT devices, which have low memory and power requirements. 2. Improvements Since Midterm Report Since the midterm report, we have implemented all of the audio functionality needed in the SDK. The SDK now supports recording audio, playing audio, speech-to-text, and text-to-speech. Furthermore, our AudioFile class now supports writing and reading WAV files, and trimming or padding silence in audio. Additionally, we wrote many more unit tests and some example programs that exercise the functionality of our new classes and check for corner cases (where exceptions are thrown). Finally, we developed the proof-of-concept trivia game requested by our client that uses our API to implement a purely voice-based trivia game. In the game, the computer announces trivia questions, and the player speaks their answers. The computer then announces the final score. There has not been a need to make any significant design changes since the midterm report.

5 4 3. Features 3.1 UML Use Case Diagram Figure 3.1 Use Case Diagram showing the five use cases 3.2 Use Case Descriptions 1. Developer wishes to convert text to speech by invoking the speech class Example: A developer would like an IoT device to ask a question by converting the question to speech 2. Developer creates a scenario where a user of their program talks to device using either listen() or continuouslylisten() in order to create a speech object Example: A developer would like to record speech to either later manipulate or convert into text 3. Developer creates text from audio by either utilizing the Speech class or by using listenandtranscribe() Example: A developer would like to convert user speech into text so they can determine the interpretation

6 5 4. The device interprets a set of text to determine the user s intent and any entities that may have been detected in the utterance. Example: A developer is creating a voice activated lamp. They can implement the IoT to turn off or on based on the intents of turn off or turn on with the entities of light or lamp while ignoring things such as turn up the music 5. The developer manipulates or plays the audio data by using the AudioFile class and its methods Example: A developer would like to remove all silence from a recording and then save it as a wav file. 4. SDK Design 4.1 UML Class Diagram Figure 4.1: UML Class Diagram showing the class structure of the C++ sdk

7 6 4.2 Application Architecture General Application Architecture Figure 4.2: The design of the SDK This SDK will be used to communicate with the Aurora API Endpoint servers on behalf of a C++ application. Using this SDK provides a generous amount of abstraction, allowing the application to believe it is calling methods on local variables when it is in fact communicating with a database via an HTTP API. In Figure 4.2, there is a C++ application, such as our proof of concept trivia game, that is developed using the SDK. The SDK allows the developer to easily connect to the Aurora Endpoint server to facilitate development. In our proof of concept, the user will communicate with the application via speech. The application then uses the SDK to communicate with the server in order to learn how to respond to the user API Class The API class provides the backend methods that the Speech, Interpret, and Text classes use to invoke calls with the Aurora server. gettts() queries the Aurora TTS server with text and returns an audio file getsst() queries the Aurora SST server with an audio file and returns text getinterpret() queries the API with provided Text object and returns a list of intents and entities Backend Class The Backend class provides an abstraction of HTTP requests, with the call() method, that the API class uses to make requests to the Aurora API. The Text class is a container for a string object. It can be created by using a string or by transcribing audio. Interpret() uses the API class in order to create an interpret object based on the text Speech() uses the API class in order to create a speech object based on the text gettext() returns the text held as a string

8 Config Class The Config class is a container for API credentials that the user can set. The Backend class sends these credentials with each HTTP request Text Class The Text class is a container for a string object. It can be created by using a string or by transcribing audio. Interpret() uses the API class in order to create an interpret object based on the text Speech() uses the API class in order to create a speech object based on the text gettext() returns the text held as a string Speech Class The Speech class is an object created using an AudioFile, an API response to text, or by using the listen() method Text() uses the API in order to create a text version of the audio in the speech object Interpret Class The Interpret class holds interpretation of a text object. It has no methods outside of the constructor, but has two members Intent - a string that holds the intent of the text based on pre-determined values on the Aurora dashboard Entities - an unordered map of all entities mentioned in the text object. It is a key-value listing according to values on the Aurora dashboard AudioFile Class The AudioFile class has the purpose of playing, recording, and manipulating WAV data. WriteToFile() writes WAV data to the specified location. Pad(), PadLeft(), PadRight(), and TrimSilence() all adjust the data within the WAV Stop() and Play() allow the user to start and stop playing the underlying audio recording NewRecordingStream() records new audio based on given parameters of length

9 8 Figure 4.3: Class Interaction Workflow Class Workflow Diagram Above shows the general interactions between the user and the C++ SDK and within the SDK itself. 4.3 External Libraries A large portion of design considerations we had to make were decisions on choosing C/C++ libraries to utilize. A majority of project development time has been spent configuring CMake to build these libraries, reading library documentation, and then integrating these libraries with our SDK C++ Requests C++ Requests is a C++ wrapper around the popular C networking library libcurl. It abstracts the process of performing HTTP requests, which is needed in the SDK to make GET and POST requests to the Aurora API server.

10 PortAudio PortAudio is a cross-platform library for processing audio data. It enables recording and playing back audio data on most modern computers. It is needed in the SDK for the speech-to-text and text-to-speech functionality JSON for Modern C++ Unlike many other modern languages, the C++ standard library does not support parsing or creating JSON data structures. JSON for Modern C++ enables us to convert the response string from the Aurora API server into a C++ Map object, which is a practical format that can be given to the user Libsndfile Libsndfile is a popular C library for reading and writing audio data into common audio file formats. The Aurora API handles all audio in the WAV file format, so in order to avoid parsing and forming WAV file headers and bodies ourselves, we are using this library to abstract that process Google Test Google Test is an extremely popular library for creating unit tests for C++ code. It supports mocking classes so that we can insert stub method implementations and return dummy values and avoid making actual network requests. It also provides many types of assert statements and a framework for naming, setting up, executing, and tearing down groups of tests. 5. User Interface Since this project is an SDK, the users are developers and the user interface is the collection of high-level classes and public methods for interacting with the Aurora SDK. The front facing classes that users will be utilizing are Text, Speech, AudioFile, Interpret, and Config. The methods and members of these classes are described in the design section. Additionally, users can use the free functions listen(), listencontinuously(), and listenandtranscribe() to utilize the speech-to-text functionality of the SDK. There are several classes in our design that are not intended for users to use or even know about, and so they are not considered part of the user interface. They are implementation details. These classes include the API, WAV, and Backend classes.

11 Documentation Documentation is a critical part of the user s interaction with the SDK, and it is highly valued by our client. Most of our code is commented with Doxygen formatted comments, and we have a build target that generates Doxygen documentation files from our source files. The user interface of our SDK s Doxygen documentation looks like figure 5.1 and figure 5.2. Figure 5.1 : The list of classes and their descriptions.

12 11 Figure 5.2 : The documentation page of our Text class. It includes a detailed description of the interpret() method. Furthermore, our code repository s README file has many code examples illustrating how to use core features of our API.

13 12 Figure 5.3 : README code example documentation showing usage of Speech class. 6. Testing Both unit tests and example programs were used to detect bugs in our SDK and increase our confidence in the correctness of our software product. 6.1 Unit Tests The Google Test framework was chosen to automate our unit tests. These tests check each of the major classes in our design diagram. Most tests fall into one of two categories: those which attempt to ensure that the unit under test produces correct output when provided with valid input, and those which attempt to ensure that the program correctly detects and handles failures caused by invalid input or invalid server responses. After building the project with make, all tests can be run automatically with the make test command, as shown in figure 6.1.

14 13 Figure 6.1: Running all test groups and getting a results summary. Figure 6.2: Google Test output showing passing API class tests for interpretation, exceptions, and speech-to-text.

15 14 Every class used in our project is covered by unit tests. Unit tests cover nearly every substantial class method as well as most of our free functions. In order to test our Backend class, we created a MockBackend subclass which produces realistic output without communicating with the Aurora server. We test robustness in multiple scenarios in which the backend would be instantiated incorrectly, such as faulty credentials, bad call methods, and empty queries. The MockBackend subclass is also used to test other classes which depend on the Backend class s functionality, without requiring a connection to Aurora s servers. All classes are tested for an encompassing range of fault scenarios, including errors returned from the server, incorrect typing, empty or missing data, and missing or incorrect credentials. Additionally, all classes are tested using normal input data which is expected to result in successful execution, and the outputs of that class s functions are then compared with the expected outputs. After any bug was found, a test case was added which fails when the bug is present and passes after it is fixed. As a single example, the WAV class s trimsilent function is intended to delete segments below a volume threshold, except for a buffer of a specified length, from the beginning and end of a wav file. Tests include: Trimming segments below the maximum threshold and leaving no buffer, and ensuring the entire file is erased. Trimming segments below an intermediate threshold and leaving no buffer, and ensuring some but not all of the file is erased. Trimming segments below an intermediate threshold and attempting to leave a buffer larger than the entire file, and ensuring that none of the file is erased. This test case was added after a bug was located which caused the program to loop infinitely when the buffer was larger than the file. Trimming segments below a realistic threshold, and leaving a buffer of realistic size (similar to the way our program will actually use the function), and ensuring that the correct portion, and only the correct portion, of the file is deleted. 6.2 Example Programs In addition to unit tests, several example programs have been created to test and demonstrate the proper usage of our SDK s functionality. One example program takes English phrases (or other text strings) as input, posts them to the Aurora server, and retrieves and prints a JSON object describing the semantic meaning of the string. Another example program, our Proof of Concept game, is a broader test of our SDK s ability to listen to audio from a user s device microphone, convert its audio output to text using the Aurora server, convert stored text to audio using Aurora s server, and play that stored text over the user s microphone.

16 15 7. Proof of Concept The Proof of Concept is a trivia game designed to showcase the Text, Speech, and AudioFile functions. First the program converts a predefined list of questions to audio format using the Text class s speech() method. The program then plays this audio to the player, by utilizing the AudioFile class, and uses ListenAndTranscribe() to convert their speech to a Text object. The program then verifies the answer and plays audio indicating whether the user gave a correct or incorrect response. After all questions have been responded to, the program gives the user their total score. 8. GitHub Links Aurora API SDK GitHub: Aurora API SDK Tests Folder: Aurora API SDK Example Folder: Proof of Concept GitHub: 9. Individual Contributions James Dickie Wrote part of SRS Wrote most of midterm report Created midterm report diagrams Created midterm presentation Developed and tested Proof of Concept program Wrote most of final report Created final presentation Seth Eisner Started proof of concept Wrote midterm/final report introduction section Helped with midterm presentation Helped with final presentation

17 16 Benjamin Lawson Wrote large portion of SRS Configured CMake build systems for Aurora SDK & proof of concept Configured Doxygen documentation generation Implemented Backend, Speech, Text, AudioFile, and Config classes Wrote unit tests for API, Text, Backend, Speech, AudioFile classes Wrote code example documentation (README) Wrote External Libraries, User Interface, and Improvements sections of midterm & final report Midterm/Final presentation Documentation slide Recorded final presentation demo video Jaron Mink Implemented API class Implemented listenandtranscribe() function (speech-to-text) within Speech Class Implemented silence detection and trimming in record() function and rewrote to conform to client standards Formatted SRS in LaTex Wrote part of SRS Made SRS diagrams Developed midterm presentation demo program Wrote examples Created class workflow diagram for final presentation and final report Refined Final Report Yaacov Tarko Implemented WAV class to generate, read, and write.wav audio files Designed and implemented test cases for the WAV class Implemented utilities required for the WAV class. These functions were required to determine whether a WAV file is silent, calculate the root mean square of an array, and convert between little-endian byte arrays and unsigned integers of any available size. Assisted with configuration of CMake for automatic build management Wrote the Test section of the final report

18 17 Nicandro Vergara Formatted some of the midterm report Proofread midterm report with minor fixes Midterm & final presentation formatting Midterm presentation speaker notes Implemented CMake configuration bug fix for Windows systems Added testing and wrote requirements slides for final presentation