(12) (10) Patent N0.: US 6,720,949 B1 Pryor et al. (45) Date of Patent: *Apr. 13, 2004

Transcription

1 United States Patent US B1 (12) (10) Patent N0.: US 6,720,949 B1 Pryor et al. (45) Date of Patent: *Apr. 13, 2004 (54) MAN MACHINE INTERFACES AND 5,521,616 A * 5/1996 Capper et al /156 APPLICATIONS 5,566,283 A * 10/1996 Modegiet a /426 5,616,078 A * 4/1997 Oh /156 (76) Inventors: Timothy ifcumseh. R. Pryor, Omano 416 ~ (CA) Old Tecumseh NSN 5,913,727 2 A Z: * 6/1999 Ahydoot gberg 0 amae... a. ~ ~ ~. ~ ~ ~ ~.. ~~ 345/158 eter Smlth 1935 Orchardvlew A * 10/1999 Maeda et al. 707/1041 Dr-,AI1I1Arb0r,MI(U$) ,982,352 A * 11/1999 Pryor /158 6,049,327 A * 4/2000 Walker et al /158 (*) Notlcel Thls Pawnt Issued on a Contlnued PYOS- 6,084,979 A * 7/2000 Kanade et al /425 eclltion application?led under 37 CFR 6,097,369 A * 8/2000 Wambach / (d), and is subject to the twenty year 6,198,487 B1 * 3/2001 Fortenbery et al /420 patent term provisions of 35 U.S.C. 154(a)(2). * cited by examiner Subject to any disclaimer, the term of this lisiiginssifrdjsgllsted under 35 Primary Examiner_Kent Chang (74) Attorney, Agent, or Firm Larson & Taylor PLC (57) ABSTRACT (21) Appl' NO" 09/ Affordable methods and apparatus are disclosed for input (22) Filed: Aug. 21, 1998 ting position, attitude (orientation) or other object charac teristic data to computers for the purpose of Computer Aided Related US. Application Data Design, Painting, Medicine, Teaching, Gaming, Toys, (60) Provisional application No. 60/056,639,?led on Aug. 22, Simulations, Aids to the disabled, and internet or other 1997, and Provisiona1 application NO- 60/059561:?led on experiences. Preferred embodiments of the invention utilize Sep' ' electro-optical sensors, and particularly TV Cameras, pro (51) Int. Cl G09G 5/08 viding optically inputted data from specialized datum s on (52) 345/158; 345/156; 345/419 objects and/or natural features of objects. Objects can be (58) Field of Search /425, 156, both Static and in motion, from Which individual datum 345/157, 158, 428, 418, 419; 463/30, 31, 32, 37 positions and movements can be derived, also With respect to other objects both?xed and moving. Real-time photo grammetry is preferably used to determine relationships of (56) References Cited portions of one or more datums With respect to a plurality of US. PATENT DOCUMENTS cameras or a single camera processed by a conventional PC. 5,325,472 A * 6/1994 Horiuchi et al / Claims, 33 Drawing Sheets

2 U.S. Patent Apr. 13, 2004 Sheet 1 0f 33 US 6,720,949 B1 FIG. 1a

3 U.S. Patent Apr. 13, 2004 Sheet 2 0f 33 US 6,720,949 B \ [ /v 52 Ag / K40 ) / 51\A 50 SEC / FIG. 1b

4 U.S. Patent Apr. 13, 2004 Sheet 3 0f 33 US 6,720,949 B / 81

5 U.S. Patent Apr. 13, 2004 Sheet 4 0f 33 US 6,720,949 B1 A B l D c V i E > F e > H Y ' > J <. V K L IQ, OTHER INPUTS OPTIONAL SUCHASVOICE FIG. 1d

6 U.S. Patent Apr. 13, 2004 Sheet 5 0f 33 US 6,720,949 B1 MAPPING ONE RGB COMPONENT OF A COLOR HERE, THE RED COMPONENT IS USED. THE SAME PROCESS CAN BE USED FOR THE GREEN AND BLUE COMPONENTS AS WELL. Ar - IS THE RED COMPONENT OF THE AQUA COLOR Or - IS THE RED COMPONENT OF THE ORANGE COLOR Pr - IS THE RED COMPONENT OF THE PIXEL COLOR Cr - IS THE RED COMPONENT OF THE COLOR ADJUSTED TO BE BETWEEN AAND O Cr = (Pr - Ar) / (Or - Ar) Cr = (Pr - Ar) / (Or - Ar) FIG. 1e

7 U.S. Patent Apr. 13, 2004 Sheet 6 0f 33 US 6,720,949 B1 ( H nf jy E \ K I 220 FIG. 2a

8 U.S. Patent Apr. 13, 2004 Sheet 7 0f 33 US 6,720,949 B1 FIG. 2b

9 U.S. Patent Apr. 13, 2004 Sheet 8 0f 33 US 6,720,949 B1 4/4645 FIG. 3a

10 U.S. Patent Apr. 13, 2004 Sheet 9 0f 33 US 6,720,949 B FIG. 3b

11 U.S. Patent Apr. 13, 2004 Sheet 10 0f 33 US 6,720,949 B FIG. 4

12 U.S. Patent A r Sheet 11 0f 33 US 6,720,949 B1

13 U.S. Patent Apr. 13, 2004 Sheet 12 0f 33 US 6,720,949 B PUPPETS LIPS

14 U.S. Patent Apr. 13, 2004 Sheet 13 0f 33 US 6,720,949 B I A / v $605 FIG. 6

15 U.S. Patent Apr. 13, 2004 Sheet 14 0f 33 US 6,720,949 B1 A step I I CAPTURE PIXEL FIELD OF EVERY CAMERA FOR NEW VIDEO FRAME. DISTINGUISH EVERY TARGET IN EACH CAMERA FIELD FROM THE REST OF THE IMAGE. STORE THE PIXELS THAT MAKE UP EACH TARGET STEP2 I I CAPTURE INTERRUPT MEMBER" POSITIONS I STEP 3 I DETERMINE THE ID OF THE WORK TOOL FROM THE UNIQUE TARGET AND SUBTARGET, COLOR. SHAPE, AND ORIENTATION INFORMATION. THIS IS DONE BY FINDING THE BEST MATCH OF THIS TARGET INFORMATION TO BE STORED IN A TABLE. EACH PATTERN STORED IN THE TABLE HAS A CORRESPONDING ID NUMBER. STEP4 I DETERMINE X,Y,Z, AND ORIENTATION ANGLES AI, A2, A3 OF TARGETS USING PHOTOGRAPHIC OR LOOKUP TABLE PATTERN MATCHING TECHNIQUES. STEP5 ' I COMPUTE THE LOCATION AND ORIENTATION OF THE WORK TOOL'S ACTION TIP" BY APPLYING AN OFFSET VECTOR TO THE TARGET POSITION AND ANOTHER VECTOR CALCULATION IS USED TO MODIFY THE ORIENTATION. THIS INFORMATION IS ALSO CONVERTED TO THE PROPER APPLICATION COORDINATES. STORE THIS INFORMATION. STEP6 I I CHECK TO SEE IF ANY INTERRUPT BUTTON HAS BEEN CHANGED. IF NO, REPEAT FIRST STEPS, ELSE GO ON. I STEP TA I DISTINGUISH UNIQUE COMBINATIONS OF INTERRUPT BUTTONS AND TARGET PATH INFORMATION (LOCATION, ORIENTATION AND TIMING PATTERNS] I. ONE UNIQUE BUTTON/PATH COMBINATION DETERMINES PATH LOCATION, ORIENTATION AND TIMING INFORMATION (REFERRED TO AS PATHDATA). 2. ALL OTHER COMBINATIONS IDENTIFY INPUT PARAMETERS OR OBJECTS OR FUNCTIONS OR PROGRAM INSTRUCTIONS (REFERRED TO AS OBJECTDATAI. step TB I IF THIS IS PATHDATA, CONVERT IT TO THE APPROPRIATE TASK PATH INFORMATION AND USE THE EXTRA TARGET INFORMATION BEYOND THAT NEEDED FOR THE PATH TO DEFINE CONTROL PARAMETERS. I STEP TO I IF THIS IS OBJECTDATA, FIND THE BEST MATCH OFATARGET PATH OR SET OF TARGET PATHS INCLUDING THE SEVEN DIMENSIONAL DATA (IOOATION, ORIENTATION AND TIME) OF EACH TARGETAND THEIR CORRESPONDING WORK TOOLIDS. ASSOCIATEAUNIQUE NUMBER (REFER TO AsAOIIANTI TO THE ABSOLUTE OR RELATIVE PATH INFORMATION. STEP 8 I= INPUT THE QUANTS TO THE CONVERSION TABLE AND OUTPUT CORRESPONDING PROGRAM MENU SELECTIONS. MACRO INITIATIONS, OBJECT SELECTIONS. FUNCTION CALLS, PARAMETER SELECTIONS. ETC. IN OTHER WORDS, THE CONVERSION TABLE OUTPUT CAN BE USED TO DEFINE PROGRAM COMMANDS, OR REDEFINE THE OBJECT THAT IS ALIASED BY THE TARGETED TOOL OR ITS CORRESPONDING FUNCTIONS OR PARAMETERS. A USER DEFINED CONVERSION TABLE ALLOWS USERS FLEXIBILITY SUCH AS TO DEFINE THE PATH PATTERN THAT HEISHE FINDS MOST COMFORTABLE. STEP 9 I I APPLY CONVERSION TABLE OUTPUT TOA COMPUTER PROORAII EITHER DIRECTLY OR WA LINK SUCH AS INTERNET ' FIG. 7

16 U.S. Patent Apr. 13, 2004 Sheet 15 0f 33 US 6,720,949 B1 FIG. 8a

17 U.S. Patent Apr. 13, 2004 Sheet 16 0f 33 US 6,720,949 B < g 847 FIG. 8b

18 U.S. Patent Apr. 13, 2004 Sheet 17 0f 33 US 6,720,949 B1 FIG. 8c

19 U.S. Patent Apr. 13, 2004 Sheet 18 0f 33 US 6,720,949 B1 PREREGISTER THE LOCATION AND ORIENTATION OF A SET OF OBJECTS WITH FREEFORM ATTACHMENTS T OBJECTS REMAINING? ATTACH OBJECTTOOBJECTHOLDER J ATTACHEOAPPRORIATETARGETTOBAsEBOARO SO THAT THE TARGET IS WITHIN THE TARGET ) ATTACH OBJECT HOLDER To BASEBOARD J VOLUME WITH AN ORIENTATION ROUGHLY 4' FACING THE LENS PLANE OF THE CAMERA ROTATE AND TRANSLATE OBJECT HOLDER To DESIRED POSITION L ATTACH APPROPRIATE TARGET TO OBJECT HOLDER SO THAT THE TARGET IS WITHIN THE TARGET vollii/ie WITH AN ORIENTATION gag? FACING THE LENS PLANE OF THE I ENTER THE OBJECT III, THE TARGET IO, AND I I ENTER THE BASEBOARD IO, THE TARGET ID. ANO THE ID OF THE TARGET ATTACHMENT LOCATION ON THE BASEBOARD I BTART THE CAI/IERA SYSTEM, ACQUIRE THE TARGET, CONIPIITE THE ORIENTATION AND LOCATION OF THE BAsEBOARO, UPDATE THE COMPUTER OATABAsE I THE ID OF THE ATTACHMENT LOCATION OF THE OBJECT HOLOER WHERE THE TARGET IS R REMOVE TARGET ATTACHED I START THE CAMERA SYSTEM, ACOUIRE THE TARGET, COMPUTE THE ORIENTATION AND LOCATION OF THE OBJECT, UPDATE THE COMPUTER DATABASE I ) REMOVE TARGET J I ADD TO A DATABASE DEFINING THIS OBJECT INCLUDING A) THE X,Y OF THE OBJECT CENTER. B) THE OBJECT PERIMETER PROJECTED ONTO THE BASEBOARD, C) THE GEOMETRIC DEFINITION OF THE OBJECT, D) MECHANICAL CHARACTERISTICS SUCH AS HOW A STEERING WHEEL WILL TURN, OR A SCALPEL WILL CUT INTO FLESH TILL IT HITS BONE, E) ASSOCIATED DATA SUCH AS QUICK LOOK UP OF DRUGS THAT CAN BE USED TO COMBAT A LEAKING COLON, F) ASSOCIATED COMPUTER PROGRAMS SUCH AS THOSE THAT COULD BE USED TO GUIDE A ROBOT CONTROLLED LAPROSCOPIC J FIG. 8d

20 U.S. Patent Apr. 13, 2004 Sheet 19 0f 33 US 6,720,949 B1 PREREGISTER THE LOCATION AND ORIENTATION OF A SET OF OBJECTS WITH FIXED ATTACHMENTS ATTACH OBJECT TO OBJECT HOLDER THAT ATTACH APPRORIATE TARGET TO BASEBDARD APPROPRIATELY ORIENTS AND RAISES OBJECT SO THAT THE TARGET IS WITHIN THE TARGET L volume WITH AN ORIENTATION ROUGHLY FACING THE LENS PLANE OF THE CAMERA ATTACH OBJECT HOLDER TO BASEBOARD AT APPROPRIATE LOCATION I ENTER THE OBJECT ID THE OBJECT HOLDER ID, AND THE LOCATION AND ORIENTATION OF THE ATTACHMENT TO THE BASEBOARD I THE ASSOCIATED COMPUTER PROGRAM WILL ADD TO A DATABASE DEFINING THIS OBJECT INCLUDING A) THE X,Y OF THE OBJECT CENTER, B) THE OBJECT PERIMETER PROJECTED ONTO THE BASEBOARD, C THE GEOMETRIC DEFINITION OF THE OBJECAT, D) MECHANICAL CHARACTERISTICS SUCH AS HOW A STEERING WHEEL WILL TURN, ORA SCALPEL WILL CUT INTO FLESH TILL IT HITS BONE, E) ASSOCIATED DATA SUCH AS QUICK LOOK UP OF DRUGS THAT CAN BE USED TO COMBAT A LEAKING COLON, F) ASSOCIATED COMPUTER PROGRAMS SUCH AS THOSE THAT COULD BE USED TO GUIDE A ROBOT CONTROLLED LAPROSCOPIC HEART SURGERY. l V l ENTER THE BASEBOARD ID, THE TARGET ID, AND THE D OF THE TARGET ATTACHMENT LOCAT'ON ON THE BASEBOARD l START THE CAMERASYSTEMACOUIRE THE TARGET COMPUTE THE OR'ENTAT'ON AND LOCAT'ON OF THE BASEBOARD- UPDATE THE COMPUTER DATABASE I FIG. 8e REMOVE TARGET

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64