Final Grant Report Grant/Contract Number: NNJ04HI11A

Transcription

1 Final Grant Report Grant/Contract Number: NNJ04HI11A Augmented Reality, Path Planning, and Physically-based Modeling for Remote Robotic Operations Abhilash Pandya (PI), Keshav Chintamani, R. Darin Ellis, Chin-An Tan, Alex Cao Wayne State University Kenneth Baker/ ER4, Chiun-Hong Chien / ER4, James C. Maida/ SF3, Barbara Woolford (COTR) NASA Johnson Space Center apandya@ece.eng.wayne.edu June/2008

2 Intro This is an interactive presentation with movies linked from an external website. When you see this symbol, click on the movie link next to it. This presentation was given to the Robotics and Human Factors community at Johnson Space Center, Astronaut Nancy Currie and to the head of ISS robotics- Dr. Shakeel Razvi. 2

3 Related Robotics Projects This clip shows our lab s related robotics projects in: Medical Robotics Military Robotics Space Robotics 3 Video: Related Robotics Projects

4 Previous NASA grant work in AR for Medical Application Video Shows AR models overlaid on a phantom skull Trajectory information overlaid Danger zones blinking Coordinate axis displayed Video: Link to movie 4

5 Background & Significance The astronaut controls the robot by manipulating the end-effecter with joysticks Visual feedback is provided by a camera mounted on the end-effecter and other peripheral cameras The astronaut depends on the video views to manipulate the robot This can be error-prone due to oblique views Difficult to simulate space dynamics on earth 5

6 Main Research Questions Can Augmented Reality (AR) cues improve user performance? Can optimal path planning help? Can we train with virtual AR robots /environments? Can dynamic environments be used for training? Are there better ways to control /select camera views? 6

7 Outline Camera control issues (pan/tilt/zoom) AR navigation cues Path planning Dynamic simulation for training Conclusions Future work 7

8 Preview The Video Shows: Video: A virtual robot registered to an actual robot. The virtual robot can be detached from the actual robot and manipulated Once the final position is reached, the actual robot can be triggered to follow the actual robots movements. The research which follows detailed aspects of this work. Link to Movie 8

9 Tele-robotic Test Bed (Hardware) This is the test bed setup at Wayne State University. Top, remote user console with joysticks and camera views. Bottom, robotic test bed with cameras 9

10 Tele-robotic Test Bed (Software) User Input 10

11 Where We Left Off Last Our software for axis augmentation has been implemented on the Dexterous Manipulator Trainer at NASA/JSC The joysticks were color-coded AR coordinate frames are displayed at the end-effector in multiple camera views 11

12 Camera control issues (pan/tilt/zoom) AR navigation cues Path planning Dynamic simulation for training Conclusions Future work CAMERA CONTROL ISSUES (PAN/TILT/ZOOM) 12

13 Camera Model Extrinsic Camera Parameters: Change with pan/tilt (6 Parameters) Translation component Rotation component Intrinsic Camera Parameters: Change with zoom (9 Parameters) Focal length Principal point Distortion 13 Camera Control Issues

14 Changing Camera Extrinsic: Pan/Tilt Allow AR with pan / tilt of camera Allow camera to track end-effecter or other objects Optimize camera views 14 Camera Control Issues

15 Camera Movement with AR Video clip shows The ability to pan and tilt the camera and maintain the AR scene. The ability to auto-track the end effector of the robot. Video: Link to Movie 15 Camera Control Issues

16 Zoom: Changing the intrinsic Camera Parameters Zoom-level vs. Parameter 18-zoom levels Fits for 9 intrinsic parameters 16 Camera Control Issues

17 Updating Zoom Parameters Video Shows The ability to zoom a camera within the AR scene. The effects of intrinsic parameter updates. Video: Link to Movie 17 Camera Control Issues

18 Camera control issues (pan/tilt/zoom) Augmented reality navigation cues Path planning Dynamic simulation for training Conclusions Future work AUGMENTED REALITY NAVIGATION CUES 18

19 Robot Navigation using Augmented Coordinates EE-Hand controller misalignments Hinder performance Errors in direction inputs Proposed Method Simple hand controller to EE axes mapping using AR Cues for ORU/receptacle alignment 19 Navigation Cues

20 Methods Each HC axes: equipped with color labels Indicate direction (+/-) Translation Rotation End-effector axes: Coordinate system with axes in RGB Additional orientation cues 3D cones for direction and depth Worksite axes Stationary coordinates in RGB Variation in graphics to prevent confusion 20 Navigation Cues

21 Navigation using Augmented Coordinates Video Shows The use of augmented coordinate system for navigation The user has color coded joysticks corresponding to the AR cue. She manipulates the end effector and aligns the coordinate frames for insertion. Video: Link to Movie 21 Navigation Cues

22 Results Path Deviation (p<0.007) 22 Navigation Cues

23 Reversal Errors Path Distance (cm) Results 150 NO AR Experiment 1 Within Subject Experienced users (12) Conditions NO AR AR Experiment 2 Between groups Experienced users (6 per group) Conditions NO AR AR Within Subject Path Distance (p<0.005) AR NO AR AR 0 Reversal Errors (p<0.003) 23 Navigation Cues

24 Extensions to Augmented Waypoints Waypoints can be used to break the task into subtasks Waypoint (WP) Location along a collision free path Orientation/position directed towards goal Going from WP i to WP i+1 1 rotation 1 translation 24 Navigation Cues

25 Augmented Waypoints Video Shows A system that places a waypoint coordinate system for the user Once the user achieves the necessary tolerance, the next waypoint is displayed. Until the payload is inserted. Video: Link to Movie 25 Navigation Cues

26 Operator Trajectories AR without Waypoints Trial 1 Trial 2 Trial 3 AR with Waypoints 26 Navigation Cues

27 Limitations Waypoints were pre-defined Placing waypoints manually is a cumbersome process Tolerances led to deviations with waypoint progression Displays get cluttered Visual collision checks only Real-time path generation is required 27 Navigation Cues

28 Camera control issues (pan/tilt/zoom) Augmented reality navigation cues Path planning Dynamic simulation for training Conclusions Future work PATH PLANNING 28

29 Augmented Collision-free GPS Paths Navigation support in complex cluttered environments Operator can preview Collision free paths Robot configurations required to reach goal Assist mission planning with subtasks Uncluttered AR cues to help user guide the robot along the path 29 Path Planning

30 Collision-free Paths from Probabilistic Searches Video shows Probabilistic Roadmap (PRM) to find Paths Collision free Kinematically comfortable Assumptions Environment geometry is known Immobile obstacles Can be extended to moving obstacles Video: Link to Movie 30 Path Planning

31 Path Planning Video Video Shows Path generated in multiple views Virtual robot arm being moved along planned path Video: Link to Movie 31 Path Planning

32 Path Planning/AR :Applications Training Navigating in cluttered environments Preview of optimal robot motions Minimum distance, angles, energy Smart guidance cues for EE navigation On-orbit Intelligently help operator guide robot through tasks & subtasks Optimal payload manipulation 32 Path Planning

33 Modes of Robot Control Manual Provide on-demand cues for navigation Supervisory Allow robot to proceed along planned trajectory in user-controlled increments Preview robot movements with virtual robot Autonomous Provide visual monitoring cues for safe operations 33 Path Planning

34 Manual Navigation Camera 1 Camera 2 34 Path Planning

35 Supervisory Navigation Camera 1 Camera 2 35 Path Planning

36 Autonomous Provide information on robots current and future objectives Display proximity to obstacles Display error modes and robot constraints 36 Path Planning

37 Path Planning Plans Human Robot Interaction Evaluation Performance Situational awareness Usability testing Training Study impact on training curves Proficiency Advanced Development Intelligent path guidance Real-time path generation Dynamic environments 37 Path Planning

38 Camera control issues (pan/tilt/zoom) Augmented reality navigation cues Path planning Dynamic simulation for training Conclusions Future work DYNAMIC SIMULATION FOR TRAINING 38

39 Rationale for Dynamic AR Simulation Tele-operated arms: Very effective in space (SSRMS, SPDM) Pre-mission training: Dexterous Manipulator Trainer (DMT) Operational environment: Microgravity Training environment: Earth Methods of training: Orbital Replacement Units (ORU) (simulated) Space shuttle mock up environment Helium simulates weightlessness 39 Dynamic Simulation

40 Terrestrial Test Beds? On-orbit Problem/Solution Test bed currently are limited E.g. DMT cannot interact with larger modules Expensive equipment (robots, ORUs, mockups) Cumbersome setup procedures Safety measures AR: solution: Physically-based virtual environment embedded in live video stream Provide detailed virtual equipment Provide dynamic simulation of onorbit No safety concerns over virtual robot movement Dynamic simulation in AR 40 Dynamic Simulation

41 Simulation: Satellite Payload Rendezvous/Collision (g 0.001m/s 2 ) 41 Dynamic Simulation

42 Virtual Robot Interacts with Virtual Object Video Show the interaction of a virtual satellite with a virtual robot. Inertial properties are modeled. Link to Movie Link to Movie 42 Dynamic Simulation

43 Actual Robot Interacts with Virtual Satellite Video Shows An actual robot interacts with the virtual satellite The inertial properties of the satellite are modeled. Video: Link to Movie 43 Dynamic Simulation

44 Dynamic Simulation: Issues Physical AR simulation maybe an alternative to current training methods. End-user training proficiency evaluation needed Object occlusion First order simulations inaccuracies Fidelity of robot/environment models Haptic capability: Will it help? 44 Dynamic Simulation

45 Summary Original Grant Aims Hybrid AR system prototype development HF analysis for the DMT Continuous zoom / pan-tilt camera calibration Achievements Developed fullyfunctioning testbed with software compatibility with the DMT HF testing on WSU testbed (due to budget cuts) Demonstration of continuous zoom/pan-tilt camera calibration 45

46 Beyond Grant Aims Developed path planning techniques for operator guidance Dynamic simulations in AR Demonstrated alternative advanced methods for camera control 46

47 Future Work Implement advanced concepts on the DMT (NASA test bed) Models of the DMT Kinematics of DMT Perform HF evaluations on the DMT Access to expert users Access to DMT Study training effects of physically-based AR Extend concepts to Unmanned Ground Vehicles. Study the occlusion problem For additional information and related references, contact 47

48 Acknowledgements WSU Dr. Greg Auner Dr. Cheng-Zhong Xu Aditya Nawab Dr. Robert Erlandson David Sant Students Thomas Overgaard Anuj Awasthi Vukasin Denic Kelly Foster Robert Xu NASA/JSC Kristian Mueller Andrew Cheang John Pace Charles Bowen Nancy Currie Shakeel Razvi Alberto Magh