User s Guide: An Unsupervised Noise Classifier Smartphone App for Hearing Improvement Studies

Transcription

1 User s Guide: An Unsupervised Noise Classifier Smartphone App for Hearing Improvement Studies N. ALAMDARI AND N. KEHTARNAVAZ UNIVERSITY OF TEXAS AT DALLAS MARCH 2018 This work was supported by the National Institute of the Deafness and Other Communication Disorders (NIDCD) of the National Institutes of Health (NIH) under the award number 1R01DC The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

2 Table of Contents Introduction... 3 Devices used for testing this app...3 Folder Description...3 Part ios... 5 Section 1: ios GUI Title View Settings View Button View Status View Settings Title View General Settings View SVDD Settings View Audio Settings View...9 Section 2: Code Flow GUI (View) Classes (Controller) Supporting Files Native Codes Part Android Section 1: Android GUI Title View Settings View Status View Button View Section 2: Code Flow Project Organization Native Codes References

3 Introduction This user s guide describes how to run and use the code of an unsupervised noise classifier app discussed in [1]. The algorithm implemented in this app is discussed in detail in [2, 3]. The main advantage of this app is that it does not require any offline training to classify noise environments that are of interest to a particular user. This user s guide consists of two parts. The first part covers the ios version of the app and the second part its Android version. Each part has two sections. The first section describes the app GUI and the second section the app code flow. Devices used for testing this app iphone7 and iphone8 as ios platforms Google Pixel and Pixel 2 as Android platforms Folder Description The codes for the ios and Android versions of the app are arranged into ios and Android folders (see Fig 1): OFC_iOS This folder includes the ios Xcode project. To open the project, double click on the OFC_iOS.xcodeproj inside the folder. OFC_Android This folder includes the Android Studio project. To open the project, open Android Studio, click on Open an existing Android Studio Project and navigate to the project OFC_Android. 3

4 Fig 1 Folders of the unsupervised noise classifier app codes implementing the algorithm named online frame based clustering (OFC). 4

5 Part 1 ios 5

6 Section 1: ios GUI This section covers the ios GUI of the app consisting of the main page and also the settings page. The GUI layer, written in Swift [4], is used to handle the entire user interface for the app by implementing several view controllers with navigation between them. The main page is the root view for navigation, and is the view that opens when the app is launched and it consists of the following four views (see Fig. 2a): Title View Settings View Button View Status View The settings page contains the following views (see Fig. 2b): Settings Title View General Settings View SVDD Settings View Audio Settings View 6

7 (a) (b) Fig. 2: App GUI ios version 1.1 Title View This view displays the app s title. 1.2 Settings View This view controls the app settings via: Settings page Audio Level in db SPL (sound pressure level) 1.3 Button View This view allows the user to start and stop the app. 7

8 1.4 Status View This view provides real-time feedback on: Currently detected cluster How many clusters detected so far 1.5 Settings Title View This view displays the app s title. 1.6 General Settings View This view controls the app settings as listed below: Hybrid classification - By enabling this switch, the app uses previously detected and saved clusters for classification of future noise frames. Saving classification Data - By enabling this switch, all the information regarding current detected clusters are saved for future use. Saving Extracted Features - By enabling this switch, the noise signal features are saved. Sampling Frequency - This field reports the sampling frequency of the audio i/o. #Decisions in Chunk - This field sets how many feature frames to have in the chunk for evaluating and creating a new cluster. Total Number of Clusters - This field sets the upper limit on the number of clusters to create. The sampling frequency is kept fixed at 48 khz in order to have the lowest audio latency imposed by the i/o hardware of smartphones. 1.7 SVDD Settings View The two parameters (named fraction rejection and sigma) of the SVDD (support vector data description) module of the unsupervised noise classifier app need to be tuned for realistic noise environments that are of interest to a particular user. This view allows the user to adjust these two parameters of the SVDD module (the default values of these parameters are set to 2.0 and 0.01, respectively, based on the noise environments examined in [1]): Fraction Rejection - This field sets the fraction rejection parameter for rejecting outlier data when creating new cluster. Sigma - This field sets the sigma parameter for the width of clustering spheres. 8

9 1.8 Audio Settings View This view provides the settings for capturing audio, computation of results, and GUI display rate: Window Size in milliseconds - This field reports the frame size in milliseconds. Overlap Size in milliseconds - This field sets the overlap size in milliseconds. By setting the overlap size to 50%, the frame size gets set to double the size of the overlap size. Display Update Rate in seconds - This field defines the display or update rate of the GUI, i.e. how often the classification outcome is displayed. Decision Rate in seconds - This field reports the decision rate of the classification outcome depending on over how many frames a majority decision is taken. 9

10 Section 2: Code Flow This section states the app code flow. The app OFC_iOS.xcodeproj can be run from the codes that are listed in Fig. 3. The connections between Swift, Objective-C and native C/C++ codes are illustrated in Fig. 4. Fig 3. App codes listing - ios version. 10

11 App GUI (Swift) Audio Settings Controller (Swift) App Audio and Audio Callbacks (Objective-C) Audio Processing Algorithms (C) Audio Settings (C) File I/O (Swift) Fig. 4: App code flow ios version The code includes four parts: GUI (View) Classes (Controller) Supporting Files Native Codes 2.1 GUI (View) This part involves the app GUI containing the following components: Main.storyboard - This component provides the layout of the app GUI. MainViewController - This component provides the main view GUI elements and actions of the GUI View. AdvancedSettingsViewController - This component enables the additional settings. 2.2 Classes (Controller) This part provides the controllers to pass data from the GUI to the native C/C++ code and to update the status from the native code to appear in the GUI. The components are: AudioSettingsController - AudioSettingsController provides audio settings adjustments at the Swift level or layer. This component acts as a bridge between the view and the native code. 11

12 IosAudioController - This component controls the audio i/o setup for processing incoming audio frames by calling the native code. In other words, the app s audio settings are handled by this component. 2.3 Supporting Files This folder includes supporting files including ic_launcher-web.png, which is the app s launcher image at different resolutions. It also includes the dlib library [6], named libdlib.a, which is used by the SVDD module. 2.4 Native Codes This part includes the classification module written in C. It is divided into the following components: Main native codes folder o audioprocessing - This is the main native code called by the Objective-C code IosAudioContoller. This code initializes all the settings and then computes the FFT of incoming audio frames, extracts features using subband features, then feeds the extracted features into the clustering module. o OFC_iOS-Bridging-Header.h - This component make objective-c and C codes accessible in the swift layer. o Timer.h - This component is for computing processing time. Clustering native codes folder This folder consists of: o EuclideanDistance - This component computes the Euclidean distance. o FrameProcessing.h - This component analyzes a current frame based on existing clusters, and if necessary creates a new cluster. o massevaluation.h - This component evaluates the chunk to see whether a new cluster needs to be created. o Sdtafx.h - This component contains the required headers and structure definitions for clustering C code. o Utilities.h - This component contains basic arithmetic functions such as mean, max, median, etc. o UnsupervisedClassification.h - This component is the main C code of the unsupervised classification which includes the headers of EuclideanDistance, FrameProcessing, massevaluation, and sdtafx. 12

13 Subband features native codes folder This folder uses the functions of SubbandFeatures.h to compute the subband features for each frame. FFTTransform native codes folder This folder consists of o FIRFilter.h - This component is for down-sampling the incoming frame to 16 khz. o filtercoefficients - This component contains coefficient needs in FIRFilter.h. o Transforms.h - This component computes the FFT of the incoming audio frames. o SPLBuffer.h - This component is used to compute sound pressure level (SPL). 13

14 Part 2 Android 14

15 Section 1: Android GUI This section covers the GUI of the Android version of the app. It consists of the following four views (see Fig. 5): Title View Settings View Status View Button View Fig. 5: App GUI Android version 1.1 Title View The view displays the app s title. 15

16 1.2 Settings View This view controls the app settings as noted below: Decision Rate in seconds (sec) - This field is used to set the decision rate of the classification outcome. Depending on the overlap size, it defines how many frames are taken for majority voting decision. The overlap size is set to 12.5ms. Therefore the frame size (also called window size) gets set to double the size of the overlap size. Sigma - This field sets the sigma parameter for the width of clustering spheres. Fraction Rejection - This field sets the fraction rejection parameter for rejecting outlier data when creating a new cluster. #Decisions in Chunk - This field sets how many feature frames to have in the chunk for evaluating and creating a new cluster. Hybrid classification - By enabling this switch, the app uses previously detected and saved clusters for classification of future noise frames. Saving classification Data - By enabling this switch, all the information regarding current detected clusters are saved for future use. Saving Extracted Features - By enabling this switch, the noise signal features are saved. The sampling frequency is kept fixed at 48 khz in order to have the lowest audio latency imposed by the i/o hardware of smartphones. The two parameters (named fraction rejection and sigma) of the SVDD (support vector data description) module of the unsupervised noise classifier app need to be tuned for realistic noise environments that are of interest to a particular user. This view allows the user to adjust these two parameters of the SVDD module (the default values of these parameters are set to 2.0 and 0.01, respectively, based on the noise environments considered in [1]). 1.3 Status View The status view provides real-time feedback on: Current frame belonging to which cluster Total or how many clusters created so far SPL (sound pressure level) in db 1.4 Button View This view allows the user to start and stop the app. 16

17 Section 2: Code Flow To be able to run the Android version of the app, the Superpowered SDK library [5] needs to be linked to it. In addition, in order to perform the SVDD classification, the dlib library [6] needs to be downloaded. The dlib_v19.7 can be downloaded from the Dlib4Android git repository [7] and then in gradle/local.properties, the path for both superpowered.dir and dlib.dir need to be changed based on the location they are saved. 2.1 Project Organization After the project is opened in Android Studio, the project organization appears as shown in Fig. 6. The app/src/main consists of the folders as shown in Fig. 6. Fig. 6: App codes listing Android version Java - This folder contains the MainActivity.java file which handles all the operations 17

18 of the app and allows the user to link the GUI to the native code. JNI - This folder includes the native codes to start the audio i/o and also the audio processing files to perform noise classification. 2.2 Native Codes JNI folder contains the code of the audio processing for the unsupervised noise classification which is divided into the following subfolders and headers: Frequency Domain - This component is the C++ file that acts as a bridge between the native code and the java GUI. It creates an audio i/o interface with the GUI settings. Timer - This component is for computing processing time of each frame, which includes time for extracting features and performing classification. Speech Processing - This component is the entry point of the native codes of the app. It initializes all the settings for the modules and then processes incoming audio signal frames to recognize the noise or cluster label of a current frame. It involves the following subfolders: o FFTTransform - This subfolder includes transform.h and FIRFilter.h to compute the FFT of the incoming audio frames and down-sampling the input audio frames, respectively. o Clustering - This subfolder includes the native codes of the unsupervised noise classification. This subfolder includes the following components: EuclideanDistance - This component computes the Euclidean distance. FrameProcessing.h - This component analyzes a current frame based on existing clusters, and if necessary creates a new cluster. massevaluation.h - This component evaluates the chunk to see whether a new cluster needs to be created. Sdtafx.h - This component contains the required headers and structure definitions for clustering C code. Utilities.h - This component contains basic arithmetic functions such as mean, max, median, etc. UnsupervisedClassification.h This component is the main C code of the unsupervised classification which includes the headers of EuclideanDistance, FrameProcessing, massevaluation, and sdtafx. o SuubandFeatures This subfolder includes the native codes to compute the subband features. 18

19 References [1] N. Alamdari, F. Saki, A. Sehgal, and N. Kehtarnavaz, An unsupervised noise classification smartphone app for hearing improvement devices, Proceedings of IEEE Signal Processing in Medicine and Biology Symposium, Philadelphia, Dec [2] F. Saki and N. Kehtarnavaz, On-line frame-based clustering with unknown number of clusters, Pattern Recognition Journal, vol. 57, pp , September [3] F. Saki and N. Kehtarnavaz, Real-time unsupervised classification of environmental noise signals, IEEE Trans. on Audio, Speech, and Language Processing, vol. 25, pp , August [4] [5] [6] [7] 19