Yuichiro Fujimoto, Tomohisa Yamada, Takafumi Taketomi, Goshiro Yamamoto, Jun Miyazaki, Hirokazu Kato Nara Institute of Science and Technology, Japan
Designing for product development For developing sales products, it is important to make prototypes and improve product s design. if developers or designers want to change the color or texture of prototypes, they have to remake the prototype. very high cost 918 spyder porsche, http://www.porsche-design.com/international/jp/ 2
Projection-based AR for design support This system can change the appearance of the target object without the need of remaking it every time Multiple people can observe the target object simultaneously. J. C. Verlinden, A. d. Smit, A. W. J. Peeters, and M. H. van Gelderen. Development of a flexible augmented prototyping system. In WSCG, 2003. 3
Limitation of current system for design support Factors for design Appearance Shape Only the appearance can be changed easily Useful if the shape of the object is able to be changed too Method This is possible if deformable materials can be used to make prototypes. 4
Goal and overview of completed system Geometrically-correct projection onto a object which can be reshaped by hand Several projector-camera systems, axes of which are corresponded with each other Target object made of deformable materials (Project with University of South Australia) 5
Factor 1. Recognize the local positions on the target object Required factors to achieve geometrically-correct projection Need to take correspondences Factor 2. Appropriate projection by several projector-camera systems Target object Projected image Method to achieve it: Place (retro-reflective) markers which can be detected by an IR-camera on the target object 6
Method to recognize the local positions on the object We develop a deformable marker and its recognition Projector + IR-camera Projected texture Deformable marker using retro-reflective materials 7
Overview of current system For a first step, one projector-camera system is used Instead of using a projector-camera system axes of which were corresponded, gray code patterns were used to take correspondences between a projector and a camera Projector Camera (switchable RGB-to-IR function) Target marker 8
Required features for new marker Marker 1. Can be recognized partially, because the whole marker can not always be observed 2. Can be recognized even if the marker is bent (by hands) flat Occlusion by Obstacles reshape Partially observed bent 9
Related works: Pattern marker Pattern marker which can be recognized as long as a 4 4 grid region is observed Reference pattern 1. Can be recognized partially when the whole marker can not always be observed 2. Can be recognized even if the marker is bent I. Szentandrasi, M. Zacharias, J. Havel, A. Herout, M. Dubska, and R. Kajan. Uniform marker fields: Camera localization by orientable de bruijn tori. In Proceedings of the 2012 11th IEEE International Symposium on Mixed and Augmented Reality, p. 2. Institute of Electrical and Electronics Engineers, 2012. 10
Overview of proposed deformable marker The shape of this marker can be easily changed by hand Metal mesh Back side 11
ID for each location on the marker Marker has dot patterns (retro-reflective material) Using an IR-camera, these patterns are used as the ID of each location in the marker 12
Approach for marker recognition For 2. Can be recognized even if marker is bent Need a new recognition algorithm Plane surface Curved surface It is difficult to acquire the whole shape information using the whole region at one times Divide into small regions, recognize smaller patterns and then merge all 13
Flow of marker recognition processes 1.Detect points 2.Divide regions in a hierarchical way 3.Create grids in each segment Problem to be solved (meaning of recognition) Calculate corresponding points on the reference pattern Reference pattern 4.Matching between detected grid points and points on the reference pattern 5. Merge all grids and reject errors...... Captured image (project a texture onto the marker) 14
Each step of recognition (1/4) 1. Detect points 2. Divide regions in a hierarchical way Small segment Large segment 15
Each step of recognition(2/4) 3. Create grids in each segment 16
Each step of recognition(3/4) 4. Matching between detected grid points and points on the reference pattern A part of detected marker Reference pattern 17
Each step of recognition(4/4) 5. Merge all grids and reject errors 1 2 3 4 5 2526 27 28 29 4950 51 52 53 7475 76 77 78 9899100101102 3 4 5 6 7 6 7 8 9 10 2728 29 30 31 3031 32 3334 + 5152 53 54 55 + 5455 56 5758 +... 7677 78 79 80 100101102103104 7980 81 8283 103104 105 106 107 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 2526 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 4950 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99100101102103104105 106107108109110111112113114115116117118119120 121 122123124125126127128129130131132133134135136137138139140141142143144145 146147148149150151152153154155156157158159160161162163164165166167168169 170171172173174175176177178179180181182183184185186187188189190191192193 194195196197198199200201202203204205206207208209210211212213214215216217 218220221222223224225226227228229230232233234235236237238239240241242245 246247248249250251252253254255256257258259260261262263264265266267268269 270271272273274275276277278279280281282283284285286287288289290291294295 296297298299300301302303304305306307308309310311312313314315316317318 319 320321322323324325326327328329330331332333334335336373338339340341342 343 344345346347348349350351352353354355356357358359360361362363364365366 367 368369370371372373374375376377378379380381382383384385386387388389390 391 Point ID in the reference pattern 18
Projection results 19
Another projection results Projected image 20
Accuracy of projection Project grid points (red points) and measure the distance between each projected point to each corresponding retro-reflective point on the marker on the marker. (a) (b) Shape of marker (a) (b) Mean error (mm) 1.8 1.6 21
Problems and next steps Marker is partially recognized only. Reason 1. Retro-reflective points are not observed on the surface, which normal is oblique toward the light axis of the camera difficult to observe Reason 2. Current marker recognition algorithm is too depend on heuristics How do you decide the size of segment and hierarchal steps? How do you detect detect threshold of voting number to reject error matching between detected points and reference points? 22
Goal Summary and future work Geometrically-correct projection onto the material which can be reshaped by hand for design support Method development of a deformable marker and its recognition method Achievement Can project texture images onto the correctly-recognized part of the deformable marker Next step Improving the accuracy of the marker recognition developing how to project by several projector-cameras simultaneously 23