Journal of Advanced Mechanical Design, Systems, and Manufacturing

Transcription

1 Note on Tangible Interaction Using Paper Models for AR-Based Design Evaluation* Hyungjun PARK**, Sang-Jin PARK** and Ho-Kyun JUNG** ** Department of Industrial Engineering, Chosun University 375 Seosuk-Dong, Dong-Gu, Gwangju, Republic of Korea Abstract Using paper models is very attractive for tangible user interaction in an augmented reality (AR) environment. In this paper, we propose an approach to tangible interaction using paper models, which is available at very low cost without hardwire connections, and accessible without restriction on location. For tangible interaction, we facilitate product- and ring-types of paper models which are all fabricated by paper crafting. The approach has been tested and applied in virtual design evaluation of digital handheld products. We found that the approach is accurate enough to be applied to the design evaluation process and tangible enough to provide a pseudo feeling of manipulating virtual products with human hands. Key words: Tangible Interaction, Paper Models, Augmented Reality, Design Evaluation 1. Introduction *Received 19 Apr., 2013 (No ) [DOI: /jamdsm.7.827] Copyright 2013 by JSME Virtual reality (VR) systems combine software and hardware tools (e.g., head mounted displays, data gloves and haptic devices) to provide realistic display of objects in a simulated environment and to offer various interaction and evaluation means (1). However, it is not easy to acquire immersive visualization and tangible interaction with low cost VR devices. Tangible AR interaction, which has gained great attention from both academia and industry, is obtained by combining a tangible user interface (TUI) with an AR system. It aims at removing the gap between the interaction with a natural environment and the interaction with a computer system. In tangible AR interaction, each virtual object is registered to a physical object and the user interacts with virtual objects by manipulating the corresponding physical objects in an AR environment, which makes user interaction tangible and intuitive (2). Soft mockups (i.e., physical objects made of soft material) or rapid prototypes (RP) can be combined with simple devices used as basic input interfaces (3)-(10). Such physical objects used in tangible AR systems are cheaper than hardware devices used in VR systems. However, they are still not easy to acquire from the view of end users who usually have to use the tangible AR systems in remote locations. When hardwired or wireless devices are incorporated into the physical objects, the construction cost of the tangible AR systems goes higher and their accessibility to the users becomes more restricted. However, using paper models as physical objects in tangible AR systems is very attractive since paper models are very cheap and highly accessible to users. In this paper, we address how to make good use of paper models to realize tangible interaction with sense of touch (i.e., a pseudo feeling of touching buttons with the fingertip) during AR-based design evaluation of digital handheld products. 827

2 2. Related Work In tangible AR interaction, physical objects are used as tangible objects in an AR environment to make user interaction tangible and intuitive. There has recently been a proliferation of works on such tangible AR interaction. Some notable ones are mentioned below. Verlinden et al. applied the concept of augmented prototyping which projects digital images on physical objects manufactured by a rapid prototyping technique to perform the design review of handheld products like a voice recorder (11). Lee and Park proposed augmented foam which AR techniques are combined with physical blue foams (4). The concept of integrating hardware and software in AR environments has been presented (3),(4). Basically, it augments a virtual display onto the soft mockup of a product by incorporating various devices (e.g., micro switches or buttons, magnetic sensors, data gloves) support direct and tangible interfaces which are hardwired or wireless. Radio frequency identification (RFID) technology, which was originally devised to track deliveries of goods, has been applied for user identification and interaction in many applications (5),(6),(7). Depth-sensing cameras have started to be applied for 3D user interaction. Klompmaker et al. properly combined RFID technology with depth-sensing cameras to enable personalized authenticated tangible interactions on a tabletop (6). Kanai et al. suggested the use of a glove-type wearable RFID R/W device and a physical mockup with small RFID tags attached to its faces for testing the usability of information appliances in an AR environment (7). Aoyama and Kimishima proposed an AR-based system for evaluating designability and operability of digital appliances in which a video see-through HMD, a data glove, and a physical object with a marker are combined with magnetic sensors to provide the pseudo feeling of touching product (8). Murakami and Fujii used a transparent RP mockup with a small camera embedded inside in order to process the images captured from the internal camera and detect user operations of the product (9). Takahashi and Kawashima proposed a system for viewing a digital product while grasping a tangible mockup and creating user inputs by fingertip touch (10). To support fingertip touch and tracking, they combined a touch sensor attached to a fingertip with a small maker attached to the fingernail. Physical mockups with hardwired connection can provide direct and accurate interfaces, but significant efforts are usually required to implement and build them. Moreover, it is not easy to make them available and accessible to many people who are located at different places. Park et al. proposed an approach to AR-based user interaction which uses simple physical objects to provide tangible interaction without any hardwired connections in a simple and cheap AR environment (12). In their approach, the user creates input events by touching specified regions of the product-type object with the pointer-type object, and the virtual product reacts to the events by rendering its visual and auditory contents on the output devices. However, the user often encounters hand occlusion in which his or her hands are occluded by augmented virtual objects. Moreover, with the pointer-type object, the user does not get a natural feeling of touching or pressing buttons with his or her fingertip. In subsequent works (13),(14), Park and Moon coined the ideas of resolving hand occlusion and replacing a finger fixture by the pointer-type object for achieving better visual immersion and more tangible interaction with sense of touch. 3. Proposed Approach As shown in Fig. 1, we use two types (product-type and ring-type) of paper models as tangible objects for interaction between the user and the product in a computer vision-based AR environment. In the figure, a game phone is used as a digital handheld product. The product-type object is used to acquire the position and orientation of the product, and the ring-type object is used to recognize the position of a fingertip. 828

3 (a) Paper models with AR markers (b) Augmented virtual objects Fig. 1 Paper models used as tangible objects As shown in Fig. 2, physical components used to build up the AR environment include paper models, a calibrated camera, an LCD monitor, and stereo speakers. The AR-based user interaction using paper models can be applied in virtual design evaluation of digital handheld products. The virtual design evaluation can enhance sense of touch or grasp, and facilitate the visual appearance and the functional simulation of the products. Fig. 3 shows the overall process of the virtual design evaluation, which is similar to the one described in Park et al. (12). The AR engine performs marker recognition and 3D object tracking, and controls the design evaluation process. The visualization of virtual objects in the AR environment is accomplished by overlaying the rendered image of the objects (free of hand occlusion) on the real world image in real time. The hand occlusion solver presented by Park and Moon can be employed to improve the visualization without hand occlusion during user interaction (14). For functional simulation, the functional behavior of the product should be captured into the form of a functional behavior model in advance (12). Fig. 2 Virtual design evaluation using the proposed tangible interaction Fig. 3 Overall process of virtual design evaluation 829

4 When the user wearing the ring-type object in his or her index finger creates input events by touching specified regions (buttons or sliders) of the product-type object with the fingertip, the AR engine recognizes the events based on the spatial relations between the tangible objects. Then, the finite state machine (FSM) makes the virtual product react to the events by referring to the functional behavior model. That is, the FSM dynamically controls the states of the virtual product according to the events by conducting state-specific tasks (i.e., rendering its visual and auditory contents on the output devices). 3.1 Fabrication of Tangible Objects The tangible objects are all fabricated by paper crafting and AR markers are attached to their specific faces. In order to make the paper model of a three dimensional (3D) object, we approximate the geometric model of the 3D object by a developable polygonal mesh and generate a cut-out sheet model by unfolding the developable polygonal mesh onto a two dimensional (2D) plane. 2D patterns or symbols of representing product components (i.e., buttons, sliders, LCD panels) can be included in the cut-out sheet model. Also, glue tabs are added onto boundary edges of the sheet model. Geometric simplification is usually required to construct a developable polygonal mesh from the 3D geometric model of an object of interest. This simplification causes the geometrical difference between the object and its paper model. As the degree of geometric simplification increases, the effect of using tangible objects decreases. In this work, we perform developable polygonal mesh approximation manually not only to maintain the overall shape of the product, but also to make the paper crafting easy. We use a CAD tool (e.g., Rhino3D) for such approximation. After generating an initial polygonal mesh representing the overall shape of the 3D triangular mesh, we cut out unnecessary pieces from it by progressively applying a Boolean subtraction operation between two polygonal meshes. During the iteration, all the polygonal meshes are generated by linear extrusion of planar polygon loops which are obtained from the outlines or cross-sectional profiles of the 3D triangular mesh. As such linear extrusion always produces developable polygonal meshes, a final polygonal mesh is developable. We stop the iteration when a current polygonal mesh represents the 3D triangular mesh decently. Using a paper crafting tool (e.g., PepaKura Designer), we generate a cut-out sheet model with glue tabs from the developable polygonal mesh. Then, we import the sheet model into the CAD tool and refine it by including 2D patterns, symbols, or glue tabs in the sheet model. For the ring-type object shown in Fig. 1, we designed its shape and dimensions to satisfy that it is defined as a developable polygonal mesh itself for easy paper crafting and allows the user to feel sense of touch with the fingertip of his or her index finger wearing it. The shape parameters of the ring-style object are shown in Fig. 4. Considering AR maker 2 size ( 24 24mm ) and the measurement of university students index fingers, we determined the shape parameters as follows: l 2 13mm, l3 28mm, l4 20mm, ( l 5 14mm, l6 16mm) for students with fingers of normal thickness, and ( l 5 16mm, l6 18mm) for students with relatively thick fingers. Using the shape parameters in parametric 3D modeling, we generated the 3D geometric model of the ring-style object, which is a developable polygonal mesh itself. Fig. 4 Shape parameters of the ring-type object 830

5 Fig. 5 shows a cut-out sheet obtained from the geometric model of the product-type object for the game phone. Fig. 6 shows a cut-out sheet obtained from the polygonal model of the ring-type object. Each paper model is built by printing its unfolded sheet on a thick blue paper, cutting out the sheet with scissors and folding back with glue. Finally, AR markers are pasted on the specified regions of the paper model since each tangible object has at least one AR marker used to augment the image of the real world with its rendered image. Blue paper is used to make paper models create good contrast to skin color, which helps good and robust results for resolving hand occlusion (14). The paper models in Fig. 1 are those made by using the cut-out sheets in Figs. 5 and 6. Fig. 5 Unfolded sheet for the product-type object of the game phone Fig. 6 Unfolded sheet for the ring-type object 3.2 User Interaction Mechanism We basically follow the user interaction mechanism described in Park et al. (12). For digital handheld products having buttons or sliders, we assume that input events are created either by pushing buttons or by moving sliders. We consider that an input event occurs if the following conditions are satisfied: (i) The distance from the fingertip to the button (or slider) is the shortest among the distances from the fingertip to the other buttons and sliders, and (ii) the distance is kept smaller than a tolerance during a specified time period. For distance computation between two points defined in different coordinate frames, we have to transform the points into a reference coordinate frame. Using the camera calibration information, we can acquire coordinate transformations between the camera and the AR markers associated with their tangible objects. As seen in Fig. 7, the distance d ( p1, p 2) between the fingertip location p 1 in the coordinate frame O 1 X 1 Y 1 Z 1 and a point p 2 (i.e., the center of a button) in the coordinate frame O 2 X 2 Y 2 Z 2 is defined as the distance between two points which are transformed into the camera coordinate frame. 831

6 Fig. 7 Distance between the fingertip and the button 3.3 Determination of Fingertip Location To determine the fingertip location, we need to know the length ( l 1 ) of the first link of the index finger in Fig. 8. Given the length, the fingertip location Q with respect to the local coordinate frame of the AR marker can be determined using simple computation based on the dimensions of the ring-type object as follows. Q O ( l l l ) Y l Z O, Y (0,1,0), and (0,0,1) where m (0,0,0) m 1 1 m m 2 3 m Z m. Fig. 8 Computation of the reference point For accurate tangible interaction using paper models, it is necessary to make the fingertip location the same with respect to the AR marker. It is good to use the ring-type object which fits tightly on the user s index finger, which reduces the bending movement of the index finger. It is also important to estimate or measure the length l 1 as exactly as possible. In this work, we determined the length l 1 for each individual user as follows. We roughly measure it with a rule. Then, we make a small red sphere drawn at the reference location Q in the augmented view, and we use a simple and easy GUI to modify the length interactively around the measured one such that the small red sphere (i.e., the reference location) is rightly on the user s fingertip. 4. Implementation and Application We implemented the proposed tangible interaction approach by using C and C++ languages on a windows-based desktop personal computer (Intel Core i GHz processor, 4GB SDRAM and ATI Radeon HD 4870 X2 2GB graphic card), and integrated it into virtual design evaluation of digital handheld products as shown in Fig. 2. For 3D model generation, CAD software called Rhino3D version 3.0 and reverse engineering software called RapidForm version 2004 were used. Software called RapidPlus version 8.0 was used to construct the functional behaviour model of each product and its FMS module, which are stored in C source code. We used ARToolKit for camera calibration, marker recognition, and 3D object augmentation. The AR engine based on ARToolKit includes modules for marker tracking, 3D object augmentation and visualization, HMI event handling, sound play, LCD image display. The LCD image display module is used to create state-specific 832

7 images which appear on the LCD display of each product. For the visualization module, we used OpenGL and GLUT as graphics libraries. For the sound play module, we used Direct Show for MP3 decoding. To develop the hand occlusion solver, we wrote our own source code by combining OpenCV with OpenGL and ARToolKit. To assess the accuracy of the proposed interaction approach, we conducted button selection experiments with a subject group consisting of 20 university students. As shown in Fig. 9, each subject was asked to complete a set of button selection tasks. Buttons are square and their side size decreases from 20, 16, 12, 8, 6, 5, 4, 3 mm. Given a sequence of 4 random numbers in each selection task, the user has to touch the center regions of the given numbers with his index finger holding the ring-type object. When using the ring-type object, every subject is allowed to see the small red sphere whose center is at the reference location Q in the augmented view. Different sounds are given to differentiate between right and wrong selections of each number. In this work, a button is selected if the reference point is closer to the button than the other buttons and the distance is kept smaller than half the side size of the button for 0.5 second. Each task is not completed until the user selects the 4 numbers correctly. After each selection task, we recorded the time required and the number of wrong selections. Fig. 10 shows the results of their assessment. Although the size of buttons of digital handheld products is very diverse, it is mostly not less than an adult s little fingernail whose size is usually bigger than 5x5 mm 2. For example, small keypad buttons of cellular or smart phones are not less than a 5x5 square, but users often make mistakes in typing tasks using such small keypad buttons. Based on this observation and by analyzing the experimental results, we found that the time and the error frequency are tolerable if the button size is not less than 5 mm, which means that the proposed user interaction can be applied to a wide variety of digital handheld products. (a) Real image (b) AR image Fig. 9 Button selection test for 12 mm buttons (a) Time required for each button selection task (b) Number of mistakes for each button selection task Fig. 10 Accuracy of tangible AR interaction As shown in Figs. 2 and 11, using the proposed tangible interaction approach, users were asked to perform the design evaluation of several digital handheld products such as MP3 players, game phones, and portable multimedia players. Most of them commented that 833

8 they could experience the functional behaviour and the visual appearance of the product easily and vividly, and that it is intuitive and tangible enough to provide a feeling like manipulating products with human hands. (a) Game phone (b) Portable multimedia player Fig. 11 Application of AR-based tangible interaction 5. Concluding Remarks Using paper models as physical objects for tangible interaction in an AR environment is very attractive. Paper models are much cheaper and thereby more accessible to users than soft mockups or rapid prototypes. Paper models can guarantee good rigidity if they are made of thick and sturdy paper. If they are made to reflect the geometric shape of their virtual objects properly, they are enough to provide users with the tangibility of interaction with descent accuracy. In this work, we took an initial step toward exploiting the use of paper models as tangible objects which can realize AR-based user interaction with sense of touch available at very low cost without hardwire connections, and accessible without restriction on location. We presented how to make and use the two types (product-type and pointer-type) of paper models as AR-based tangible objects to perform click operations with them in the AR environment. The proposed approach has been tested and applied in AR-based design evaluation of digital handheld products. From users feedbacks, we found that the approach is accurate enough to be applied to virtual design evaluation of various digital handheld products, and tangible enough to provide a pseudo feeling of manipulating products with human hands. We plan to continue our research in the following aspects. First, we will devise a way of decreasing the size of the pointer-type object in order to provide a wider view of virtual products in an AR environment. For this, it is required to efficiently decrease the size of AR markers while keeping their accuracy of recognition and tracking. Second, it is meaningful to establish an automatic process for fabricating a paper model of a virtual object of interest, even though it was enough to take a manual process in this work. The automatic process requires two key tasks: the automatic construction of a developable polygonal mesh from a non-manifold geometric model of arbitrary shape and the automatic unfolding of the developable polygonal mesh onto a plane (15),(16). Although there have been notable works on these two topics, it is necessary to modify the works to fit them to our specific problem. For example, in most previous works on developable surface model generation, geometric simplification is done mainly based on the approximation error between a given geometric model and its developable surface model. In this work, however, we have to consider 834

9 geometric simplification which depends not only on approximation error, but also on geometric characteristics such as symmetry and parallelism. Acknowledgements This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (NRF ). References (1) Burdea, G.C. and Coiffet, P., Virtual reality technology (2003), John Wiley & Sons. (2) Billinghurt, M., Kato, H. and Poupyrev. I., 2001, Collaboration with tangible augmented reality interfaces, Proceedings of International Conference on Human-Computer Interaction (HCI), pp (3) Nam, T.J., 2005, Sketch-based rapid prototyping platform for hardware-software integrated interactive products, Proceedings of Conference on Human Factors in Computing Systems (CHI), pp (4) Lee, W. and Park, J., 2006, Augmented foam: touchable and graspable augmented reality for product design simulation, Bulletin of Japanese Society for the Design Science, Vol., 52, pp (5) Bravo, J., Hervas, R., Chavira, G., Nava, S.W., and Villarreal, V., 2008, From implicit to touching interaction: RFID and NFC approaches, Proceedings of Conference on Human System Interactions, pp (6) Klompmaker. F., Fischer, H., and Jung. H., 2012, Authenticated tangible interaction using RFID and depth-sensing cameras Supporting collaboration on interactive tabletops, Proceedings of the Fifth International Conference on Advances in Computer-Human Interactions, pp (7) Kanai, S., Horiuchi, S., Kikuta, Y., Yokoyama, A., and Shiroma, Y., 2007, An integrated environment for testing and assessing the usability of information appliances using digital and physical mock-ups, Lecture Notes in Computer Science, Vol. 4563, pp (8) H. Aoyama, H. and Kimishima, Y., 2008, Mixed reality system for evaluating designability and operability of information appliances, International Journal of Interactive Design and, Vol. 3, No. 3, pp (9) Murakami, T. and Fujii, K., 2009, Internal video analysis for product usability evaluation at early stage of design, International Journal on Interactive Design and, Vol. 3, No. 3, pp (10) H. Takahashi, H. and Kawashima, T., 2010, Touch-sensitive augmented reality system for development of handheld information appliances, International Journal on Interactive Design and, Vol. 4, No. 1, pp (11) Verlinden, J., Van den Esker, W., Wind, L., and Horvath, I., 2004, Qualitative comparison of virtual and augmented prototyping of handheld products, Proceedings of International Design Conference (DESIGN), pp (12) Park, H., Moon, H.C. and Lee, J.Y., 2009, Tangible augmented prototyping of digital handheld products, Computers in Industry, Vol. 60, No. 2, pp (13) Park, H. and Moon, H.C., 2010, Note on augmented reality-based tangible interaction using a finger fixture, Proceedings of Asian Conference on Digital Design and Engineering (ACDDE), pp (14) Moon, H.C. and Park, H., 2011, Resolving hand region occlusion in tangible augmented reality environments, Transactions of the Society of CAD/CAM Engineers, Vol. 16, No. 4, pp (15) Massarwi, F., Gotsman, C., and Elber, G., 2008, Paper-craft from 3D polygonal models using generalized cylinders, Computer Aided Geometric Design, Vol. 25, No. 8, pp (16) Straub, R. and Prautzsch, H., 2005, Creating optimized cut-out sheets for paper models from meshes, Proceedings of SIAM Conference on Geometric Design and Computing. 835