(73) Assignee: MOTION GAMES, LLC, Austin,

Transcription

1 US A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2012/ A1 Pryor (43) Pub. Date: Feb. 16, 2012 (54) INTERACTIVE VIDEO BASED GAMES (60) Provisional application No. 60/059,561,?led on Sep. USING OBJECTS SENSED BY TV CAMERAS 19, 1997, provisional application No. 60/056,639,?led on Aug. 22, (75) Inventor: Timothy R. Pryor, Sylvania, OH (Us) Publication Classi?cation (73) Assignee: MOTION GAMES, LLC, Austin, (51) Int. Cl. A63F 13/00 (200601) TX (Us) A63F 13/06 ( ) (21) Appl. No.: 13/267, US. Cl /32' 463/37 ( ) (57) ABSTRACT (22) Filed: Oct. 6, 2011 A method and apparatus for 1nteract1ve TV camera based.. games in Which position or orientation of points on a player or Related U's' Apphcatlon Data of an object held by a player are determined and used to (63) Continuation of application No, 12/941,304,?led on control a video display. Both single camera and stereo camera Nov. 8, 2010, now Pat. No. 8,068,095, Which is a pair based embodiments are disclosed, preferably using ste continuation of application No. 11/186,898,?led on reo photogrammetry Where multi-degree of freedom infor Jul. 22, 2005, now Pat. No. 7,843,429, Which is a mation is desired. Large video displays, preferably life-size continuation of application No. 09/138,339,?led on may be used Where utmost realism of the game experience is Aug. 21, 1998, now abandoned. desired

2 Patent Application Publication Feb. 16, 2012 Sheet 1 0f 39 US 2012/ A1 FIG. 1a

3 Patent Application Publication Feb. 16, 2012 Sheet 2 0f 39 US 2012/ A [ \ [UN / / / / f I 50 5; FIG. 1b

4 Patent Application Publication Feb. 16, 2012 Sheet 3 0f 39 US 2012/ A1 61 /

5 Patent Application Publication Feb. 16, 2012 Sheet 4 0f 39 US 2012/ A1 I A D = 0 _'" - -l i V B V: E = F G _...> H = J <.....F K L M OTHER lnputs OPTIONAL SUCH AS VOICE FIG. 1d

6 Patent Application Publication Feb. 16, 2012 Sheet 5 0f 39 US 2012/ A1 Cr = (Pr -Ar) / (Or - Ar) Cr = (Pr - Ar) / (Or -Ar) FIG. 1e

7 Patent Application Publication 1 6., S h t 6 f US 2012/ A1 0.. F 2 w 0 9 O 2/ FIG. 2a

8 Patent Application Publication Feb. 16, 2012 Sheet 7 0f 39 US 2012/ A1 FIG. 2b

9 Patent Application Publication Feb. 16, 2012 Sheet 8 0f 39 US 2012/ A1 [ID Q 5 5% GIG /.Q 361 FIG. 3a

10 Patent Application Publication Feb. 16, 2012 Sheet 9 0f 39 US 2012/ A1 FIG. 3b

11 Patent Application Publication Feb. 16, 2012 Sheet 10 0f 39 US 2012/ A1 FIG; 4

12

13 Patent Application Publication Feb. 16, 2012 Sheet 12 0f 39 US 2012/ A PUPPETS LIPS

14 Patent Application Publication Feb. 16, 2012 Sheet 13 0f 39 US 2012/ A O W / J FIG. 6

15 Patent Application Publication Feb. 16, 2012 Sheet 14 0f 39 US 2012/ A1 STEPI I CAPTURE PIXEL FIELD OF EVERY CAMERA FOR NEW VIDEO FRAME. DISTINGUISH EVERY TARGET IN EACH CAMERA FIELD FROM THE REST OF THE IMAGE. STORE THE PIXELS THAT MAKE UP EACH TARGET. STEP2 I CAPTURE INTERRUPT MEMBER" POSITIONS STEP3 I DETERMINE THE ID OF THE WORK TOOL FROM THE UNIQUE TARGET AND SUBTARGET, COLOR, SHAPE, AND ORIENTATION INFORMATION. THIS IS DONE BY FINDING THE BEST MATCH OF THIS TARGET INFORMATION TO BE STORED IN A TABLE. EACH PATTERN STORED IN THE TABLE HAS A CORRESPONDING ID NUMBER. STEP4 I DETERMINE X,Y,Z, AND ORIENTATION ANGLES AI, A2, AS OF TARGETS USING PHOTOGRAPHIC OR LOOKUP TABLE PATTERN MATCHING TECHNIQUES. STEP5 I COMPUTE THE LOCATION AND ORIENTATION OF THE WORK TOOL S ACTION TIP" BY APPLYING AN OFFSET VECTOR TO THE TARGET POSITION AND ANOTHER VECTOR CALCULATION IS USED TO MODIFY THE ORIENTATION. THIS INFORMATION IS ALSO CONVERTED TO THE PROPER APPLICATION COORDINATES. STORE THIS INFORMATION. STEPS I - A I-I CHECK TO SEE IF ANY INTERRUPT BUTTON HAS BEEN CHANGED. IF NO, REPEAT FIRST STEPS, ELSE GO ON. STEP 7A I DISTINGUISH UNIQUE COMBINATIONS OF INTERRUPT BUTTONS AND TARGET PATH INFORMATION (LOCATION, ORIENTATION AND TIMING PATTERNS) 1. ONE UNIQUE BUTTON/PATH COMBINATION DETERMINES PATH LOCATION, ORIENTATION AND TIMING INFORMATION (REFERRED TO AS PATHDATAIL 2. ALL OTHER COMBINATIONS IDENTIFY INPUT PARAMETERS OR OBJECTS OR FUNCTIONS OR PROGRAM INSTRUCTIONS (REFERRED TO AS OBJECTDATA). STEP7B I IFTHIS IS PATHDATA, CONYERT IT TO THE APPROPRIATE TASK PATH INFORMATION AND USE THE EXTRA TARGET INFORMATION BEYOND THAT NEEDED FOR THE PATH TO DEFINE CONTROL PARAMETERS. STEPTC I I IF THIS IS OBIECTDATA, FIND THE BEST MATCH OFATARGET PATH DR SET DFTARCET PATHS INCLUDING THE SEYEN DIMENSIONAL DATA (LOCATION, ORIENTATION AND TIME) OF EACH TARGET AND THEIR CORRESPONDING WORK TOOL IDS. ASSOCIATE AUNIOUE NUMBER (REFER TO AS AQUANTI TO THE ABSOLUTE OR RELATIYE PATH INFORMATION. ' I STEP8 I= INPUT THE OUANTS TO THE CONYERSION TABLE AND OUTPUT CORRESPONDING PROGRAM MENU SELECTIONS, MACRO INITIATIONS, OBJECT SELECTIONS, FUNCTION CALLS, PARAMETER SELECTIONSETC. IN OTHER WORDS, THE CONYERSION TABLE OUTPUT CAN BE USED TO DEFINE PROGRAM COMMANDS, OR REDEFINE THE OBJECT THAT IS ALIASED BY THE TARGETED TOOL OR ITS CORRESPONDING FUNCTIONS OR PARAMETERS. A USER DEFINED CONVERSION TABLE ALLOWS USERS FLEXIBILITY SUCH AS TO DEFINE THE PATH PATTERN THAT HEIsHE FINDS MOST COMFORTABLE. STEP 9 I APPLY CONVERSION TABLE OUTPUT TO ACOMPUTER PROGRAM EITHER DIRECTLY OR YIA LINK SUCH AS INTERNET] I FIG. 7

16 Patent Application Publication Feb. 16, 2012 Sheet 15 0f 39 US 2012/ A1

17 Patent Application Publication Feb. 16, 2012 Sheet 16 0f 39 US 2012/ A

18 Patent Application Publication Feb. 16, 2012 Sheet 17 0f 39 US 2012/ A1 FIG. 80

19 Patent Application Publication Feb. 16, 2012 Sheet 18 0f 39 US 2012/ A1 PREREGISTER THE LOCATION AND ORIENTATION OF A SET OF OBJECTS WITH FREEFORM ATTACHMENTS OBJECTS REMAINING? N0 I ATTACH OBJECT TO OBJECT HOLDER I ATTACHED APPRORIATE TARGET To BASEBOARD I SO THAT THE TARGET IS WITHIN THETARGET volume WITH AN ORIENTATION ROUGHLY I AITACH QBIECT HOiDER To BASEBOARD I FACING THE LENS PLANE OF THE CAMERA I ROTATE AND TRANSLATE OBJECT HOLDER TO DESIRED POSITION I ATTACH APPROPRIATE TARGET TO OBJECT HOLDER SO THAT THE TARGET IS WITHIN THE TARGET volume WITH AN ORIENTATION ggaggg FACING THE LENS PLANE OF THE V I ENTER THE BASEBOARD ID, THE TARGET ID, AND THE ID OF THE TARGET ATTACHMENT LOCATION ON THE BASEBOARD I START THE CAMERA SYSTEM, ACOUIRE THE TARGET, COMPUTE THE ORIENTATION AND LOCATION OF THE BASEBOARD, UPDATE THE COMPUTER DATABASE I I ENTER THE OBJECT ID, THE TARGETID, AND THE ID OF THE ATTACHMENT LOCATION OF THE OBJECT HOLDER WHERE THE TARGET IS I REMOVE TARGET ATTACHED I START THE CAMERA SYSTEM, ACQUIRE THE TARGET, COMPUTE THE ORIENTATION AND LOCATION OF THE OBJECT, UPDATE THE COMPUTER DATABASE I REMOvE TARGET I ADD TO A DATABASE DEFINING THIS OBJECT INCLUDING A) THE X,Y OF THE OBJECT CENTER. B) THE OBJECT PERIMETER PROJECTED ONTO THE BASEBOARDC) THE GEOMETRIC DEFINITION OF THE OBJECT, D) MECHANICAL CHARACTERISTICS SUCH AS HOWA STEERING WHEEL WILL TURN, OR A SCALPEL WILL CUT INTO FLESH TILL IT HITS BONE, E) ASSOCIATED DATA SUCH AS QUICK LOOK UP OF DRUGS THAT CAN BE USED TO COMBAT ALEAKING COLON, F) ASSOCIATED COMPUTER PROGRAMS SUCH AS THOSE THAT COULD BE USED TO GUIDE A ROBOT CONTROLLED LAPROSCOPIC FIG. 8d

20 Patent Application Publication Feb. 16, 2012 Sheet 19 0f 39 US 2012/ A1 OBJECTS REMAINING? NO ATTACH OBJECT TO OBJECT HOLDER THAT APPROPRIATELY ORIENTS AND RAISES OBJECT I ATTACH OBJECT HOLDER TO BASEBOARD AT APPROPRIATE LOCATION I ENTER THE OBJECT ID, THE OBJECT HOLDER ID, AND THE LOCATION AND ORIENTATION OF THE ATTACHMENT TO THE BASEBOARD I THE ASSOCIATED COMPUTER PROGRAM WILL ADD TOA DATABASE DEFINING THIS OBJECT INCLUDING A) THE X,Y OF THE OBJECT CENTER, B) THE OBJECT PERIMETER PROJECTED ONTO THE BASEBOARD, C) THE GEOMETRIC DEFINITION OF THE OBJECT, D) MECHANICAL CHARACTERISTICS SUCH AS HOW A STEERING WHEEL WILL TURN, OR A SCALPEL WILL CUT INTO FLESH TILL IT HITS BONE, E) ASSOCIATED DATA SUCH AS QUICK LOOK UP OF DRUGS THAT CAN BE USED TO COMBATA LEAKING COLON, F) ASSOCIATED COMPUTER PROGRAMS SUCH AS THOSE THAT COULD BE USED TO GUIDE A ROBOT CONTROLLED LAPROSCOPIC HEART SURGERY. II ATTACH APPRORIATE TARGET TO BASEBOARD SO THAT THE TARGET IS WITHIN THE TARGET VOLUME WITH AN ORIENTATION ROUGHLY FACING THE LENS PLANE OF THE CAMERA I ENTER THE BASEBOARD ID, THE TARGET ID, AND THE ID OF THE TARGET ATTACHMENT LOCATION ON THE BASEBOARD I START THE CAMERA SYSTEM, ACOUIRE THE TARGET, COMPUTE THE ORIENTATION AND LOCATION OF THE BASEBOARD, UPDATE THE COMPUTER DATABASE I REMOVE TARGET FIG. 8e

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78