Kinsight: Localizing and Tracking Household Objects using Depth-Camera Sensors

Transcription

1 Kinsight: Localizing and Tracking Household Objects using Depth-Camera Sensors Shahriar Nirjon and John Stankovic Department of Computer Science University of Virginia

2 Household Object Localization Problem Assigning 3D location to household objects and updating the assignment when objects are moved.

3 Household Object Localization Problem Assigning 3D location to household objects and updating the assignment when objects are moved. Obj. Time 3D Location

4 Household Object Localization Problem Assigning 3D location to household objects and updating the assignment when objects are moved. Obj. Time 3D Location 12:30 pm (6, 4, 2)

5 Household Object Localization Problem Assigning 3D location to household objects and updating the assignment when objects are moved. Obj. Time 3D Location 12:35 pm (6, 4, 2)

6 Household Object Localization Problem Assigning 3D location to household objects and updating the assignment when objects are moved. Obj. Time 3D Location 12:35 pm (6, 4, 2) (5, 0, 1)

7 Example Usage: Activity Recognition Accurately recognizing household activities using the spatiotemporal information of object usage.

8 Example Usage: Activity Recognition Accurately recognizing household activities using the spatiotemporal information of object usage. Objects Location Inferred Activity Kitchen

9 Example Usage: Activity Recognition Accurately recognizing household activities using the spatiotemporal information of object usage. Objects Location Inferred Activity Kitchen Preparing Coffee

10 Example Usage: Activity Recognition Accurately recognizing household activities using the spatiotemporal information of object usage. Objects Location Inferred Activity Kitchen Preparing Coffee Living Room

11 Example Usage: Activity Recognition Accurately recognizing household activities using the spatiotemporal information of object usage. Objects Location Inferred Activity Kitchen Preparing Coffee Living Room Entertainment

12 Properties of Our Approach

13 Properties of Our Approach (1) Non-intrusive sensing using depth-camera sensors.

14 Properties of Our Approach (1) Non-intrusive sensing using depth-camera sensors. No need to attach sensors to objects

15 Properties of Our Approach (1) Non-intrusive sensing using depth-camera sensors. Kinect No need to attach sensors to objects Color Image and Depth Image

16 Properties of Our Approach (2) Indirect tracking from human-object interactions.

17 Properties of Our Approach (2) Indirect tracking from human-object interactions. Indirect Tracking Tracking Skeleton Detecting Interaction

18 Properties of Our Approach (3) Computations are distributed in phases.

19 Properties of Our Approach (3) Computations are distributed in phases. Motion Analysis Sampling Detect Object Object Loading Filtering Recognize Object Check Locations Store Samples Update Location Event Phase Real-Time Phase Post Process Phase

20 Outline of the rest of the talk What we did. Results we achieved.

21 What we did Study on household objects locations. Depth-sweep algorithm for extracting objects from images. Context Oriented Object Recognition algorithm for object recognition. A prototype system Kinsight.

22 Study on household objects locations A week-long study in 2 homes. RFID tags on 60 objects, activities are video recorded.

23 Study Result (1) More than 95% of the objects stay in at most 3 locations. Importance: Improves object recognition accuracy.

24 Study Result (2) The number of different objects used in different activities vs. the size of the time-window: Importance: Limit on the processing capability of object classifier.

25 Study Result (3) The mean idle time (time during which there is no human-object interaction) in different activities: Importance: Upper Limit on the post-processing phase.

26 Example: Object Extraction (Depth-Sweep)

27 Example: Object Extraction Step 0: Obtain color and depth image.

28 Example: Object Extraction Step 0: Obtain color and depth image.

29 Example: Object Extraction Step 1: Use depth to remove unwanted pixels.

30 Example: Object Extraction Step 1: Use depth to remove unwanted pixels.

31 Example: Object Extraction Step 2: Use skeleton to locate interaction window.

32 Example: Object Extraction Step 2: Use skeleton to locate interaction window.

33 Example: Object Extraction Step 2: Use skeleton to locate interaction window.

34 Example: Object Extraction Step 3: Run image segmentation on the window.

35 Example: Object Recognition (1) Using only appearance (color, shape and size):

36 Example: Object Recognition (1) Using only appearance (color, shape and size): Obj 1: Obj 2: Object Database

37 Example: Object Recognition (1) Using only appearance (color, shape and size): Obj 1: Obj 2: Unknown Object Object Database

38 Example: Object Recognition (1) Using only appearance (color, shape and size): Obj 1: Obj 2: Unknown Object Object Database

39 Example: Object Recognition (1) Using only appearance (color, shape and size): Obj 1: Obj 2: Unknown Object Object Database

40 Example: Object Recognition (2) Using only appearance (color, shape and size): Obj 1: Obj 2: Unknown Object Object Database

41 Example: Object Recognition (2) Using only appearance (color, shape and size): Obj 1: Obj 2: Unknown Object Object Database

42 Example: Object Recognition (2) Using only appearance (color, shape and size): Obj 1: Obj 2: Unknown Object Object Database

43 Example: Object Recognition (3) Using activity and location history with appearance: Obj 1: Obj 2: Unknown Object Object Database

44 Example: Object Recognition (3) Using activity and location history with appearance: Obj 1: Obj 2: Unknown Object Being used together with At the location Object Database

45 Example: Object Recognition (3) Using activity and location history with appearance: Obj 1: Obj 2: Unknown Object Being used together with At the location Object Database

46 Example: Object Recognition (3) Using activity and location history with appearance: Obj 1: Obj 2: Unknown Object Being used together with At the location Object Database

47 Results we achieved Effect of Distance, Speed, Sampling Rate, Color, Size and Multiple Views. Performance during 6 activities in two homes.

48 Evaluation Metric Localization error: Distance between the actual location and the computed location where our system thinks the object is.

49 Evaluation Metric Localization error: Distance between the actual location and the computed location where our system thinks the object is. Error originates from two sources: Measurement Error: Localization error when the object is recognized correctly. Recognition Error: Localization error when the object is not recognized correctly.

50 Effect of Distance (a) Without Context (b) Kinsight Result: When we apply context, the recognition error becomes 8-10 times smaller, the measurement error becomes mostly constant with distance.

51 Effect of Speed (a) Without Context (b) Kinsight Result: (a) Without context, as the speed grows beyond 50 cm/sec, classification errors lead to higher localization error, and (b) the recognition errors are 5-11 times reduced when we use context.

52 Effect of Sampling Rate (a) Without Context (b) Kinsight Result: Both portions of the error are reduced when the sampling rate is as high as 10 samples/sec. The same error margin is achieved in (b) at a lower frequency of 4 samples/sec when we use context.

53 Effect of Color (a) Without Context (b) Kinsight Result: The errors are due to some of the copies that were too close(within 13 cm) to each other to be distinguishable by Kinsight and the transparent objects were not properly visible by the camera.

54 Effect of Size (a) Without Context (b) Kinsight Result: For tiny objects (8x4x3 cm 3 ), misclassification is reduced from 74% to 46% when we use context. But when they are more than 2.5 m away, we do not get enough pixel information from them.

55 Using Multiple Cameras Master Cam L-shaped Area Slave Cam Result: Using this two node system, 98% of the objects are correctly recognized yielding to a localization error of 11.7 cm.

56 Performance during different activities 2 households. 6 activities min long experiments. 25 objects (on average).

57 Results Best results are obtained during cleaning, coffee making, and entertainment as they involve a small number of (3-5) easy to distinguish items. 1. Cleaning 2. Coffee making 3. Cooking 4. Eating 5. Entertainment 6. Study

58 Results Best results are obtained during cleaning, coffee making, and entertainment as they involve a small number of (3-5) easy to distinguish items. Larger errors are observed in cooking, eating and studying, as these activities involve a large number of small items. 1. Cleaning 2. Coffee making 3. Cooking 4. Eating 5. Entertainment 6. Study

59 Conclusion We describe Kinsight, that uses depth-camera sensor network to localize household objects. We use 4 types of information: Color information Depth information Activity context Location history The localization error is only 13 cm on average.

60 Thank You ( 谢谢 )