Human Robot Communication. Paul Fitzpatrick

Transcription

1 Human Robot Communication Paul Fitzpatrick

2 Human Robot Communication Motivation for communication Human-readable actions Reading human actions Conclusions

3 Motivation What is communication for? Transferring information Coordinating behavior What is it built from? Commonality Perception of action Protocols

4 Communication protocols Computer computer protocols TCP/IP, HTTP, FTP, SMTP,

5 Communication protocols Human human protocols Initiating conversation, turn-taking, interrupting, directing attention, Human computer protocols Shell interaction, drag-and-drop, dialog boxes,

6 Communication protocols Human human protocols Initiating conversation, turn-taking, interrupting, directing attention, Human robot protocols Human computer protocols Shell interaction, drag-and-drop, dialog boxes,

7 Requirements on robot ENGAGED ACQUIRED Pointing (53,92,12) Fixating (47,98,37) Saying /o ver[200] \/there[325] Human-oriented perception Person detection, tracking Pose estimation Identity recognition Expression classification Speech/prosody recognition Objects of human interest Human-readable action Clear locus of attention Express engagement Express confusion, surprise Speech/prosody generation

8 Example: attention protocol Expressing attention Influencing other s attention Reading other s attention

9 Foveate gaze Motivation for communication Human-readable actions Reading human actions Conclusions

10 Human gaze reflects attention (Taken from C. Graham, Vision and Visual Perception )

11 Types of eye movement Ballistic saccade to new target Right eye Left eye Vergence angle Smooth pursuit and vergence co-operate to track object (Based on Kandel & Schwartz, Principles of Neural Science )

12 Engineering gaze Kismet

13 Collaborative effort Cynthia Breazeal Brian Scassellati And others Will describe components I m responsible for

14 Engineering gaze

15 Engineering gaze Cyclopean camera Stereo pair

16 Tip-toeing around 3D Field of view New field of view Object of interest Wide View camera Narrow view camera Rotate camera

17 Example

18 Influences on attention

19 Influences on attention Built in biases

20 Influences on attention Built in biases Behavioral state

21 Influences on attention Built in biases Behavioral state Persistence slipped recovered

22 Directing attention

23 Head pose estimation Motivation for communication Human-readable actions Reading human actions Conclusions

24 Head pose estimation (rigid) Yaw * Pitch * Roll * Translation in X, Y, Z * Nomenclature varies

25 Head pose literature Horprasert, Yacoob, Davis 97 McKenna, Gong 98 Wang, Brandstein 98 Basu, Essa, Pentland 96 Harville, Darrell, et al 99

26 Head pose: Anthropometrics Horprasert, Yacoob, Davis McKenna, Gong Wang, Brandstein Basu, Essa, Pentland Harville, Darrell, et al

27 Head pose: Eigenpose Horprasert, Yacoob, Davis McKenna, Gong Wang, Brandstein Basu, Essa, Pentland Harville, Darrell, et al

28 Head pose: Contours Horprasert, Yacoob, Davis McKenna, Gong Wang, Brandstein Basu, Essa, Pentland Harville, Darrell, et al

29 Head pose: mesh model Horprasert, Yacoob, Davis McKenna, Gong Wang, Brandstein Basu, Essa, Pentland Harville, Darrell, et al

30 Head pose: Integration Horprasert, Yacoob, Davis McKenna, Gong Wang, Brandstein Basu, Essa, Pentland Harville, Darrell, et al

31 My approach Integrate changes in pose (after Harville et al) Use mesh model (after Basu et al) Need automatic initialization Head detection, tracking, segmentation Reference orientation Head shape parameters Initialization drives design

32 Head tracking, segmentation Segment by color histogram, grouped motion Match against ellipse model (M. Pilu et al)

33 Mutual gaze as reference point

34 Mutual gaze as reference point

35 Tracking pose changes Choose coordinates to suit tracking 4 of 6 degrees of freedom measurable from monocular image Independent of shape parameters X translation Y translation Translation in depth In-plane rotation

36 Remaining coordinates 2 degrees of freedom remaining Choose as surface coordinate on head Specify where image plane is tangent to head Isolates effect of errors in parameters Tangent region shifts when head rotates in depth

37 Surface coordinates Establish surface coordinate system with mesh

38 Initializing a surface mesh

39 Example

40 Typical results Ground truth due to Sclaroff et al.

41 Merits No need for any manual initialization Capable of running for long periods Tracking accuracy is insensitive to model User independent Real-time

42 Problems Greater accuracy possible with manual initialization Deals poorly with certain classes of head movement (e.g. 360 rotation) Can t initialize without occasional mutual regard

43 Motivation for communication Human-readable actions Reading human actions Conclusions

44 Other protocols

45 Other protocols Protocol for negotiating interpersonal distance Person backs off Person draws closer Too close withdrawal response Comfortable interaction distance Too far calling behavior Beyond sensor range

46 Other protocols Protocol for negotiating interpersonal distance Protocol for controlling the presentation of objects Comfortable interaction speed Too fast, Too close threat response Too fast irritation response

47 Other protocols Protocol for negotiating interpersonal distance Protocol for controlling the presentation of objects Protocol for conversational turn-taking Protocol for introducing vocabulary Protocol for communicating processes Protocols make good modules

48 Cameras Eye, neck, jaw motors QNX Ear, eyebrow, eyelid, lip motors sockets, CORBA Motor ctrl Tracker Attent. system Dist. to target Eye finder Motion filter dual-port RAM Face Control L Percept & Motor Speakers NT speech synthesis affect recognition CORBA Skin filter Recog. pose Color filter Track pose audio speech comms Track head Emotion Drives & Behavior Microphone Linux Speech Linux recognition Speech recognition CORBA

49 Wide Camera Wide Camera 2 Left Foveal Camera Right Foveal Camera Wide Frame Grabber Left Frame Grabber Left Frame Grabber Right Frame Grabber Skin Detector Color Detector Motion Detector Face Detector Distance to target Eye finder Foveal Disparity W W W W Attention Tracked target Salient target Wide Tracker Behaviors Motivations Fixed Action Pattern Affective Postural Shifts w/ gaze comp. Ballistic movement q, dq, d 2 p p q p q, dq, d 2 f f q f Saccade w/ neck VOR comp. q, dq, d 2 s s q s Q, Q, Q Arbitor... Locus of attention Smooth Pursuit & Vergence w/ neck comp. q, dq, d 2 v v q v Disparity Eye-Head-Neck Control Motion Control Daemon Eye-Neck Motors

50 Other protocols What about robot robot protocol? Basically computer computer But physical states may be hard to model Borrow human robot protocol for these

51 Current, future work Protocols for reference Know how to point to an object How to point to an attribute? Or an action? Until a better answer comes along: Communicate task/game that depends on attribute/action Pull out number of classes, positive and negative examples for supervised learning

52 FIN