A Robust Sign Language Recognition System with Sparsely Labeled Instances Using Wi-Fi Signals

Transcription

1 A Robust Sign Language Recognition System with Sparsey Labeed Instances Using Wi-Fi Signas Jiacheng Shang, Jie Wu Center for Networked Computing Dept. of Computer and Info. Sciences Tempe University

2 Motivation Wi-Fi signas are avaiabe amost everywhere. Wi-Fi signas can monitor surrounding activities using Channe State Information (CSI).

3 Motivation Sign anguage (SL) mainy uses manua communication to convey meaning.

4 Motivation Automatic SL Recognition is sti in its infancy. Currenty, a commercia transation services are human-based, and therefore, expensive.

5 Motivation Automatic SL Recognition is sti in its infancy. Currenty, a commercia transation services are human-based, and therefore, expensive.

6 Motivation Automatic SL Recognition is sti in its infancy. Currenty, a commercia transation services are human-based, and therefore, expensive. American Language Services offers interpreters starting at $125 per hour and a two-hour minimum is required

7 Probem Statement Sign anguage recognition using Wi-Fi signas Uses commercia Wi-Fi devices (routers and aptops) to recognize sign anguage. Strengths Can work in the dark Avoids breaching user privacy No need to wear sensors Low cost

8 Limitations of Existing Systems Limitations of existing systems: modes are trained based on a arge dataset Large training datasets are usuay hard and expensive to get in practice. Many works have the potentia requirement that abe distributions in the training dataset and the testing dataset shoud be the same. Our approach: reduce the size of the training dataset by everaging the knowedge in the unabeed dataset and others training datasets

9 Limitations of Existing Systems Why are current modes trained using a arge dataset?

10 Limitations of Existing Systems Why are current modes trained using a arge dataset? Accuracy: 79%

11 Sign Language Recognition Pipeine

12 Signa Preprocessing Subcarrier Seection Different subcarriers have different sensitivities to different human activities 30 subcarriers TX RX

13 Signa Preprocessing Noise remova Smoothing: removes outiers Low-pass fiter: removes high frequency noise The average ampitude and the average median absoute deviation are chosen as the features.

14 Leverage knowedge in unabeed datasets Labeed instances are often very time consuming and expensive to obtain. The new user may ony be abe to abe some instances whie most instances stay unabeed. Knowedge in unabeed instances can be used to improve the recognition s performance. Co-training is an efficient semi-supervised earning paradigm

15 Training Leverage knowedge in unabeed dataset Cassification mode 1 Cassification mode 2 Predicting Labeed dataset Act as training data Extracted knowedge Unabeed dataset Extracted knowedge: those unabeed instances that are predicted as the same abe by both (of two) cassification modes

16 Reuse others training datasets The abiity to recognize and appy knowedge obtained in previous tasks Why Reuse? Buid every mode from scratch? Time consuming and expensive Reuse knowedge extracted from existing tasks and datasets More practica How can we decide what data shoud be transferred to the new user?

17 Reuse others training dataset Transfer agorithm: find those usefu instances from existing abeed source domain data Features vaue discretization on each dimension with a grid size of τ. A source domain instance is transferred to target domain iff there is a target domain instance with the same abe in the same grid. Target Domain data that is abeed as 1 Target Domain data that is abeed as -1 Source domain data that is abeed as 1 Source domain data that is abeed as -1

18 Reuse others training dataset Transfer agorithm: find those usefu instances from existing abeed source domain data Features vaue discretization on each dimension with a grid size of τ. A source domain instance is transferred to target domain iff there is a target domain instance with the same abe in the same grid Target Domain data that is abeed as 1 Target Domain data that is abeed as -1 Source domain data that is abeed as 1 Source domain data that is abeed as -1

19 Reuse others training dataset Source domain data Auxiiary set finding agorithm Predictive modes (SVM) Target domain data A few abeed data coected from the new user Target domain data Unabeed testing data

20 Evauation Commercia hardware with no modifications Transmitter: TP-Link TL-WR1043ND Wi-Fi router Receiver: Lenovo X100e aptop with Inte 5300 NIC Downoading a arge fie from an FTP server within the same oca network area

21 Evauation Resuts Mean accuracies vs. different soutions Two proposed soutions can achieve better accuracies with sparsey abeed training data.

22 Evauation Resuts Mean accuracies vs. different users Our approaches can achieve a mean prediction accuracy of about 87% for a participants.

23 Evauation Resuts Accuracies vs. different τ There is no inear reationship between the accuracy and τ. Τ is determined based on the distribution and density of the data.

24 Evauation Resuts Accuracies vs. different iterations We set the number of iterations to 5 in our system.

25 Concusion CSI measurements contain fine-grained movement information and can be used to recognize sign anguage. Propose a sign anguage recognition system that can achieve a good performance with sparsey abeed data. Leveraging the knowedge in an unabeed dataset. Reusing others training datasets. Experimenta resuts show that our system can achieve a mean prediction accuracy of about 87%.

26 Thanks! Q & A