Geometric Correction of Projection Using Structured Light and Sensors Embedded Board

Transcription

1 2014 by IFSA Publishing, S. L. Geometric Correction of Projection Using Structured Light and Sensors Embedded Board 1 LAN Jianliang, 2 DING Youdong, 3 PENG Taile 1 School of Computer Engineering & Science, Shanghai University, Shanghai, , China 2 School of Film and TV Arts & Technology, Shanghai University, Shanghai , China 3 School of Communication & Information Engineering, Shanghai University, Shanghai, , China 1 lanjianliang@126.com Received: 3 June 2014 /Accepted: 27 June 2014 /Published: 30 June 2014 Abstract: Computer vision based projector-camera calibration approaches usually involve with complex geometry transformation and trivial operations, mostly inappropriate for scenarios that under poor illumination or interfering background. An automatic geometric correction method for projector using projection board embedded with optic sensors is proposed, aiming at simplifying the calibration operations and improving the robustness of geometric correction. By capturing the Gray-coded structured light frames, the correspondence between the points of interest on the projection board and the pixels on image plane could be obtained; furthermore, homograph matrixes are exploited to transform the images to be projected. A special one-to-many hardware design paradigm is adopted so that multiple projection boards are able to connect to one renderer node simultaneously. To further enhance the projection precision against each projection board, a self-adaptive illumination density threshold determination method is used. Copyright 2014 IFSA Publishing, S. L. Keywords: Geometric correction, Projector calibration, Structured light, Optic sensor, Embedded devices. 1. Introduction Comparing with CRT and LCD display technology, modern projectors are capable of projecting images onto various surfaces, playing an irreplaceable role in many displaying applications. Due to rapid development of the projector manufacturing technology, cheap and portable projectors are quite common nowadays, and what s more, projectors are becoming more and more important in many fields like interactive display, intelligent space and ubiquitous computing scenarios or other applications. In contrast with camera calibration, there are some special aspects on projector calibration. First, projectors can only project 2D images into 3D space, which is an irreversible process, that is to say, a projector could not be used to take photos like camera; the way to obtain the correspondences between 3D points and 2D image pixels is different from that in camera calibration. Second, most projector calibration approaches use inverse-camera model to simplify the projector calibration procedure [1, 8, 9], assuming the projector s principal point is close to the image center, while some projectors use an off-axis projection, so that the inversed-camera model may not suitable for some projectors or some special situations [2]. Using a camera to help calibrating projector is quite common in vision based approaches. A simplest projector-camera (P-C) system consists of a projector, a camera and target projecting surface. In a P-C system, camera will be calibrated first, and then be used to capture the 3D points on the surface illuminated by the projector. After that, the correspondence between 3D points and 2D image 244

2 pixels could be obtained to reconstruct the 3D-2D geometric relationship. The required manipulations are tedious for a skilled person, not to mention for an untrained terminal user. Structured light based approaches are proposed recent years to simplify the procedure by simultaneously calibrating the projector and camera in P-C system [3-5]. Audet and Okutomi [6] propose a method that uses fiducial markers to derive pre-warped patterns, and therefore complete the calibration procedure within several minutes. While the problem is, vision based algorithms could be severely affected by some factors such as poor illumination or mussy background, thus these algorithms have limitations in practical use. In those applications like interactive projection, complex surfaces projection, advertising exhibition and so on, what interests us is how to make the projected images fit with 3D surfaces, and how to make the whole procedure fast, robust and low-cost. In this sense, geometric correction is one of the most important aspects needed to be considered in projector calibration. Lee and Dietz [7] exploit the light sensors to capture the structured light and acquire the sensors location relative to projection plane and then pre-warp the image to accurately fit the target projection surfaces. This method does not require full geometric calibration and makes the whole procedure more robust. In this paper, we will focus on finding a fast, robust, automatic and low-cost geometric correction method for projectors. We leverage structured light and projection boards embedded with micro-control units and sensors. Like Lee and Dietz s method, no additional camera is needed in our implementation, and what s more, multiple target projection boards could be simultaneously calibrated within few seconds, and little manipulations are involved during the whole process. The structured light patterns illumination density threshold for each sensor is dynamically determined; therefore the position calculation precision is enhanced. 2. System Architecture Our implementation is composed of projector, renderer and projection board embedded with sensors (see Fig. 1). The projector is a usual un-calibrated commercial projector. The projection board serves as two roles, the auxiliary device for projector calibration or the target surfaces to be projected image on. This means that the projection board could help the geometric correction of projector at first, and after that, being the target projection surface. The renderer usually runs on a computer or SOC (system on chip) which provides enough computing ability for decoding the position information from the sensors and rendering the images or video Gray-Code Structured Light As an effective method for surface reconstruction, structured light method has been massively studied and enhanced in the last decades. In our implementation, structured light will be exploited not to reconstruct the target surface but to help geometry correction for projector. Fig. 1. System architecture. We use one dimension black and white stripshape Gray-code structured light to encode the projection plane s 2D coordinates, therefore two series of structured light frames are needed to cover the whole target surface, one for horizontal coordinates X, another for vertical coordinates Y. It is known that the Hamming Distance of successive two integers Gray-code is always 1, which means Graycode has the characteristic of remaining local similarities. So that, when the adjacent areas of the target surface illuminated by Gray-code structured light frames, they will receive similar light patterns, and these patterns could be decoded to similar position coordinates instead of dramatically varying ones. The structured light frames are projected in a certain order that frames indicating large-scale positions followed by frames indicating small-scale positions (Fig. 2), and for those adjacent surface regions, only the least significant bits of the projected patterns may be different. In a word, Gray-code structured light has a certain degree of fault tolerance when used to identify different regions of target projection surfaces. Fig. 2. Gray-code structured light frames. 245

3 2.2. MCU and Wireless Transport Module The devices used in our implementation include micro-control unit (MCU), wireless transport module (WTM), optic sensors and wired optic fibers. The logic relationships could be depicted as Fig. 3. MCU is an AVR ATMega328P chip running at 20 MHz, with multiple (at least 6) built-in channels of analogto-digital converter (ADC), therefore multiple independent channels of light signals could be simultaneously processed. We adopt nrf24l01 as the WTM, in favor of its ability of read/write multiple pipes of data concurrently. And what s more, nrf24l01 could be extended by cascading them in the logical layers, thus massive sensors are able to join in the network, which is a very important characteristic in our implementation. The communication between the projection board and the renderer is done through two kinds of WTMs; one kind contains one and only WTM at the renderer side as the server node, and the other kind on the board may be many of WTMs, as the client nodes. Fig. 3. Logic relationship among MCU, sensors, WTM and renderer Projection Board How to precisely segment the Gray-code structured light strips and extract the edge position is the key to vision-based algorithms, while this is usually rather complicated because of the difficulty of locating strips and identifying those vague boundaries. We leverage optic sensors to overcome those difficulties. Optic sensors are sensitive to lightness changing which is considered as analog signals to be digitized. When the sensors are wired with optic fiber whose diameter is only 1 millimeter, light signals are transmitted from the optic fiber tip which is embedded in the projection board, to the sensors themselves, and then to the MCU attached on the board. The tiny optic fiber tips are buried in the board therefore imperceptible to human eyes, and they won t affect the quality of images projected onto the board. The projection boards used in our implementation could be of any shapes or materials, like rectangle acrylic board or square wooden board. The optic fiber tips are buried in the four corners of the board which considered as four points of interest (POI). As mentioned before, we use black and white structured light to illuminate the projection board, thus the optic fiber tips on the corners will receive signals corresponding to the location in the 2D projection plane. MCU attached on the board will translate the location signals received by optic sensors to projection plane coordinates, and these coordinates will be sent to renderer via wireless transport module. The program running on MCU is working under a so-called passive mode, that is to say, MCU responds only when received valid instructions from the renderer. There are two types of instructions for now, illumination detection (ID) and coordinates detection (CD). The former one intends to determine the proper threshold of digitalizing the analog signal of illumination intensity. For each light sensor, the illumination intensity results from the superimposition of varying structured light patterns and the ambient light, therefore it s necessary to calibrate the illumination threshold for decoding right bits of structured light patterns. For instance, when the MCU received an ID instruction, it will sample the illumination intensities from i-th sensor with respect to all-white frame and all-black frame, namely IW and IB, and do an average to get the threshold When the MCU received a CD instruction, it will begin the procedure of detecting coordinates. For the i-th sensor, the illumination intensities sampled from the structure light frame patterns, indicated as, will be compared with to determine the final bit that compose the graycode which is to be decoded to the sensor s corresponding coordinates on the source projection image plane. In our implementation, we have 4 sensors embedded in the projection board s corners and 8 Gray-code structure light frames for each dimension, so that the above procedure could be informally described as Eq. 1 ~ Eq. 3 below. 0 i 3), (1) 1, if 0, if 0 7,0 3), 1, if 0, if 0 7,0 3), (2) (3) t 0 7,0, (4) The symbol in Eq. 3 is bitwise OR operator, and is bitwise left-shift operator. For a given i th sensor, when B i is determined, the sensor s corresponding coordinates in the projection image could be decoded through the following Gray-code decoding algorithm in the form of pseudo code. 246

4 0) 1) 1 2) If M 0 do Steps 3 ~ 5, otherwise go to Step 6 3) ^ 4) 1 5) Go to Step 2 6) Return C i as i th sensor s corresponding 2D coordinate on the projection image plane. In the above algorithm, >> is bitwise right-shift operator, and ^ is exclusive OR operator. When all the (0 i 3) are calculated, these coordinates will be sent to the render by the WTM. It s possible that there are more than one WTM attached to the projection board acting as client nodes, especially the case that two or more projection boards exist under the projection area. In this case, each client WTM should receive instructions from the same local device address while send feedbacks to different remote pipe number when communicating with the server WTM; the server WTM issues ID or CD instructions in the broadcast way to one unique device address and receives feedback results coming from client WTMs through different receiving pipes. Fig. 4. Projection boards embedded with MCU and sensors. (a) Two boards with different shapes and materials. (b) MCU: AVR ATMega328P; WTM: nrf24l01. (c) Optic fiber tips are buried in the corners of the board. (d) The installation angles of fiber tips should be as perpendicular to the board as possible Renderer and Projector In practical terms, we would not like to impose too much strict spatial constraints upon the relationship between the projector and projection surfaces, because it is a time consuming and laborious process to adjust the projector s position and orientation. In this case, the renderer is responsible for the geometric correction job. The renderer runs at the platform with enough computing ability, such as a modern PC or SOC. The development libraries mainly include OpenCV, OpenGL and Qt framework. OpenCV handles the structured light frames generation and applying transform matrixes to the projection images; OpenGL and Qt framework are in charge of displaying the rendered results to projector. Just as the name implies, the renderer renders the source images to make the projected images fit with the target projection surface and enjoyable to human eyes. To achieve this goal, the correspondence between 3D points and 2D image points should be estimated first. In our implementation, the projection boards could be considered as planes, and for each board there are 4 points of interest (POI). For the i-th POI, the 3D coordinate is denoted as,,,1, and the mapped 2D image point is denoted as C i =[x, y, 1] T, then the homograph could be expressed as:, (5) The parameter s is an arbitrary scale factor, and H is a homograph matrix. In the last section of this paper, all from the sensors have been gained. Without loss of generality, the world coordinate system could be centered on the known-size board, thus the Z of could be set to 0, thus could be rationalized to a new form Q i =[X, Y, 1], and X and Y could also be simply assumed to some value related to the board s dimension. For instance, if the projection board is a square board embedded with 4 sensors, then Q 0, Q 1, Q 2 and Q 3 could be assumed to [0, 0, 1] T, [0, 1, 1] T, [1, 1, 1] T and [1, 0, 1] T respectively. Mathematically, given four (Q i, C i ) pairs, the homograph matrix H could be solved except the scale factor s which controls the scale of the projected image on the plane. While in our implementation, the scale factor s need not to be estimated, because what want to do is pre-warping an 2D image which has an approximate proportion in shape against the projection board, to fit in the polygon defined by 0 3 ; in other words, we can use the 2D image s width (w) and height (h) to assume the 0 3 as 0,0,1, 0,, 1,,,1 and, 0,1. The homograph matrix with the scale factor s could be easily solved using Eq. 5 below., (6) The renderer will apply the sh transform matrix to the images to be projected towards the specified board, then send the rendered frame to the projector, thus get the undistorted projection on that board. This method is valid for the scenario that there are two or more projection boards in the projection area. Each projection board has the corresponding (sh) k, thus multiple board projection is also possible for the renderer. 247

5 3. Experiment Results The experiment is designed to verify the feasibility and effectiveness of our design work and algorithms. We use one projector (BenQ DLP MP525P XGA: 1024x768), two projection boards embedded with MCU, WTM and optic sensors, one PC running with the renderer. The two projection boards are randomly posed to the projector, and assuring the optic fiber tips could be illuminated by the projector. Experiments are carried out with good lightness condition (daytime) and bad lightness condition (nighttime) several times, and the two boards placements are also changed every time. We use 8-frame of gray-coded structure light patterns, therefore the maximum inaccuracy of location is ±4 pixels, and the projection quality is acceptable for human eyes. The whole time from issuing the ID and CD instructions to projecting images onto the target boards are merely about 3~4 seconds, which is much faster than the approaches that vision based and do not require any additional human operations. What s more, the board s tilt angle could be rather big (up to ±60 ) relative to the projector s optic axis, which means the boards could be posed with quite few spatial constraints. The experiment result could be seen in Fig. 5. There are two different-size boards placed at different positions and orientations. The whole geometric correction procedure is automatically carried out. The final projection quality is satisfying. When the board s tilt angle relative to the projector s optic axis getting too large, like larger than 70, the projection begins to be unstable; the images could not be projected fitting with the board s surface. The reason is the lightness signal received by the sensors will decay when tilt angle getting too large, causing wrong coordinates obtained by the renderer. This problem cannot be solved completely, but could be alleviated to some extent. Lee and Dietz [7] suggest one possible method, which is, using a light diffuser on the optic fiber tips to help bounce light into the fiber even at large tilt angles. 4. Conclusion A novel approach of geometric correction of projection with the help of structured light and optic sensors has been implemented and further studied in this paper. This approach does not require additional cameras to get the 3D-2D correspondence but rely on the optic sensors embedded in the board, and the Gray-code structured light frames. Since it is not vision based, the ambient illumination and the background condition won t be the interference problem of POI locating. We also adopt the one-tomany paradigm for wireless communication between the sensors and the renderer, thus make it possible that multiple projection boards could be located simultaneously, without costing additional time. The locating precision of projection boards or optic sensors is important to the whole process of geometric correction, and it mainly depends on many factors like the amount of structured light frames, the fabrication technology of sensors and optic fibers, the location decoding algorithms, et al. All these could be improved to achieve better locating precision which make our approach feasible and extendable. In our implementation, the optic fibers are buried in the projection boards, but it does not mean that we have to drill holes on the projection surfaces in actual use. The whole MCU, WTM and sensors could be well integrated in a small mobile module; therefore it could be attached at the designated locations of the target surface like a sticker. When the locating procedure is done, this module could be removed therefore the surfaces will remain intact. Fig. 5. (a) Two boards placed at different positions and orientations. (b) Detecting illumination thresholds when MCU received ID instructions. (c) Projecting horizontal Gray-code structured light frames towards the boards. (d) Projecting vertical Gray-code structured light frames towards the boards. (e) The pre-warped projection images after applying homograph matrixes. (f) Projecting images onto the boards without distortion. References [1]. Martynov I., Kamarainen J. K., Lensu L., Projector Calibration by Inverse Camera Calibration, in Proceedings of the 17 th Scandinavian Conference on Image Analysis, Vol. 5, 2011, pp [2]. Raskar R., Beardsley P., A Self-Correcting Projector, Computer Vision and Pattern Recognition (CVPR), Vol. 2, 2001, pp

6 [3]. Yamazaki S., Mochimaru M. Kanade T., Simultaneous Self-Calibration of A Projector and A Camera Using Structeured Light, IEEE in Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2011, pp [4]. Gao W., Wang L., and Hu Z. Y., Flexible Calibration of a Portable Structured Light System through Surface Plane, Acta Automatica Sinica, Vol. 34, Issue 11, 2008, pp [5]. Lu J, Song C. Y., Structured Light System calibration based on Gray Code Combined with Line-Shift, Journal of Optoelectronics Laser, Vol. 23, Issue 6, 2012, pp [6]. Audet S., Okutomi M., A User-Friendly Method to Geometrically Calibrate Projector-Camera Systems, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2009, pp [7]. Lee J. C., Dietz P. H, Mayes D. et. al., Automatic Projector Calibration with Embedded Light Sensors, in Proceedings of the 17 th Annual ACM Symposium on User Interface Software and Technology (UIST), 2004, pp [8]. Fernandez S., Salvi J. Planar-based Camera-Projector Calibration, in Proceedings of the International Symposium on Image and Signal Processing and Analysis (ISPA), 2011, pp [9]. Hurtos T., Falcao G., and Massich J., Plane-based calibration of a projector camera system, VIBOT master, Vol. 1, 2008, pp Copyright, International Frequency Sensor Association (IFSA) Publishing, S. L. All rights reserved. ( 249