CAMERA-BASED CALIBRATION OF MULTIPLE LASER PROJECTORS FOR COLLABORATIVE PROJECTION

Transcription

1 CAMERA-BASED CALIBRATION OF MULTIPLE LASER PROJECTORS FOR COLLABORATIVE PROJECTION MENENDEZ Francisco J., HALABI Osama, FUJIMOTO Tadahiro, CHIBA Norishige Department of Computer Science Graduate School of Engineering Iwate Universit [ohalabi fujimoto ABSTRACT In this paper we propose a method to automaticall calibrate and combine multiple laser projectors for collaborative projection. One of the main caveats of toda s laser projectors is the relationship between the speed in which the galvanometers can move and the complexit of the data we are tring to displa. As image complexit grows, we are forced to increase the scanning speed in order to avoid flickering. However, the inertia effect of the rotating mirrors, which manifests as curved deformations in sharp angles, becomes more evident as we force the speeds up. To improve the wa these projectors behave, we propose combining two or more projectors to displa one single image. In this wa, the amount of lines that can be drawn without flickering would increase, without forcing the galvanometers to move at greater speeds or jumping greater angles. Furthermore, to obtain an accurate calibration, we use a digital camera to capture the drawing area of each singular projector and produce the necessar transformations to allow for a full overlap of these areas. 1. INTRODUCTION Using laser light to construct shapes and images is hardl new, however, it has not been until recentl that laser displa technolog has become relativel cheaper and more available. The growing availabilit of this technolog opens a new door for Computer Graphics researchers, as laser projectors have their ver own characteristics that separate them from raster displa and even other vector-based devices. Some of the particular problems associated with laser projection deal mainl with the limitations imposed b the hardware itself. To understand these problems we must first understand how a laser projector works. A laser projection sstem is basicall comprised of three basic parts: 1) A computer or some other electronic source that generates signals that control the projector. 2) A hardware interface that serves as a communication bridge between the computer and the projector. Depending on the technolog, this interface ma exist in the computer as a connected board, inside the projector, as part of its hardware; or as a third, separated component (e.g.: a network-based interface). 3) The projector itself, that usuall consists of one laser beam (three beams in the case of RGB projectors), and a couple of high-speed scanners (mirrors attached to their corresponding galvanometers), that rotate according to the signals sent b the computer, in order to make the laser beam move and displa the desired images. Some of the problems derive from the phsical nature of the method used to move the beam. Limitations in rotation speed lead to problems related with image complexit and flickering. The weight of the mirrors and its inherent rotational inertia causes deformations when drawing sharp angles. Some other problems are directl related with direct ee exposure to the beam. Safet measure should be taken regarding where to place the projector and selecting the appropriate laser power for each occasion. In this paper we aim to solve two of these problems. To keep the beams awa from the audience, we must sometimes place the projector in a position that does not guarantee a correct projection. With the help of a camera, we will show how to overcome this problem. Later on, we will extend this method to combine multiple projectors to increase the refresh rate, and reduce flickering. 1.1 Previous Work There is some previous work regarding projector calibration [BRO05] [RAS98], studing the alignment of various raster projectors, in order to increase the displa area. While our objective is to create a single, collaborative projection area (instead of increasing the size), some basic concepts are similar (such as the use of grids and cameras to calibrate). Also, in our case, the correct alignment of the projection areas is critical, since even the smallest differences can become quite evident. Furthermore, an attempt to reduce flickering was proposed b Abderim et al. [ABD07], b removing unnecessar

2 jumps (blanking lines) as much as possible. In this paper we will not consider the blanking lines problem, but a combination of both methods would theoreticall allow creating more complex imager without the undesired flickering effect that usuall follows. 2. SINGLE PROJECTOR CALIBRATION In a previous work [MEN07] we discussed the need for a simple and efficient wa to displa images regardless of the location of the projector. This was because of the concerns that rose regarding the possibilit of direct ee exposure to the laser beams. As we will explain later, we first need to calibrate each of the involved projectors individuall using the this method, before we can proceed to the multiple-projector calibration. This method uses a camera, placed where the audience would be, and automaticall captures the deformations caused b the projection angle and the irregularities in projection surface, processes the data and creates a calibration grid used to interpolate the coordinates before displaing Capture and image processing To detect the deformations caused b the angle between the projector and the projection surface, as well as its irregularities, we project a pre-defined grid on the surface and then calculate the difference between what we project and what we obtain. In order to recognize the projected grid from the surface, we must first take a picture of the surface without anthing projected on it. Since we want this process to be full automatic, we take advantage of a common feature of most late digital cameras called PTP (Picture Transfer Protocol) [ISO05]. This protocol is what allows cameras to directl send images to inkjet printers to produce prints without the need of a computer. Surprisingl, this protocol also allows for more than just data transfer, but to control man of the camera functions, including taking snapshots. (a) Figure 1: (a) Ideal grid and (b) resulting projection The next step is to detect the grid from the images. To do so, we calculate the difference between the first and second images, using equation (1), where diffimg(x,) will store the color difference of the pixels from imgbase(x,), base image and, imggrid(x,) projected grid, at the x- coordinate, thus removing the background and leaving onl the dots from the projected grid. diffimg( x, ) = max( imgbase( x, ) imggrid ( x, ), 0) (1) This difference image might have some noise product of changes in luminosit or undesired beam reflections. To eliminate these noises, we binarize the difference image, changing the threshold according to the number of dots found, versus the number of dots expected. In other words, if we find more dots than we expect, it is reasonable to suppose those extra dots are caused b noise and then we raise the threshold to filter that noise. This is shown in equation (2), where diffbin(x,) will hold the binar pixels and we use the threshold θ to filter noise. A result of this binarization is shown in figure 2 1 if diffimg( x, ) > θ diffbin( x, ) 0 otherwise (b) (2) Once we have placed the camera where our audience would be, it should not be moved until the calibration process is complete. Using PTP, the computer communicates with the camera and commands it to take the necessar images at the appropriate times, thus eliminating an possible errors caused b pressing the shutter button manuall. After we have taken the first picture of our projection screen, we proceed to take a second picture with a dot grid pattern projected. For calibration purposes, we need this grid to be of an odd number of dots (e.g.: 3x3, 5x5, 7x7, etc). The size of the grid will be determined b the maximum and minimum laser coordinates, and the distance between the projection surface and the projector. An example of this grid can be found in figure 1. Figure 2: Binar difference image. In this step we have isolated the grid from the background Next, we estimate the coordinates, in pixels, of the detected grid points from the binarized image. We accomplish this b choosing the center of each block of pixels that represent one grid dot and store it in memor for further processing.

3 2.2 Grid detection Now we have a set of uncorrelated x- coordinates, in pixel units, representing the dots. However, before we can use this data, we must restore the detected dots into their original grid arrangement. To accomplish this, we first find the corner points, that is, the four points that have the highest and lowest x- values. Taking these four points, we take a first guess at the arrangement, b placing a grid which corners match exactl these four points. The rest of the detected points are matched to the remaining grid points b using a weight function, shown in (3), where pt holds the x- coordinate in pixels of the point we want to match, grid x, holds the x- coordinate in pixels of the possible grid point we are evaluating; alwas preferring that grid point that returns the highest weight value. grid pt if grid has been matched x, x, wmatch( pt, grid ) = x, 0 otherwise w( pt, grid ) = wmatch( pt, grid ) + wmatch( pt, grid ) x, x+ 1, x 1, + wmatch( pt, grid ) + wmatch( pt, grid ) + 2 grid pt x, x, + 1 x, 1 In plain words, this means that for each grid point we evaluate, we will not onl prefer the closest one point, but also consider those grid points that have alread been matched. In this wa, if two points are equidistant from the grid point being considered, the grid point will prefer that one that has the closest matched neighbors. We repeat this process until all grid points have been assigned a point detected from the binar image. Now this grid holds all the detected points in the correct arrangement (see figure 3) However, since the data was taken from an image, these points are expressed in pixel units instead of the original laser projector units. In order to obtain a grid we can actuall use, we need to transform these pixel coordinates into laser coordinates. (3) numbers in pixels. However, since the center of projection for a laser projector is alwas (0, 0), we must translate the grid so the center points are aligned, as shown in equation (4), where optima lcenterx,center is the center, in laser coordinates of the ideal grid we first projected (usuall 0,0), and the observed centerx,center is the center of the observed grid, in pixel units. translate = optimal observed (4) centerx, center centerx, center The correlation between laser units and pixels (pixel-to-laser scale) is estimated b an iterating binar search algorithm. We tr to minimize the equation shown in (5), where optimal holds the ideal grid in laser units and observed is the visuall obtained grid in pixel units. Appling different scale values, we approximate the pixel grid to be as close as possible to the original maximum and minimum values of the projected gird, until we get to the maximum level of iterations or an acceptable error value is reached. The error is estimated as the absolute distance between the corners of both grids (originall projected and visuall obtained). distance( scale) = optimal scale ( observed + translate) 0,0 0,0 + optimal scale ( observed + translate) 0, size 0, size + optimal scale ( observed + translate) size,0 size,0 + optimal scale ( observed + translate) size, size size, size After appling the translation and scaling to the observed grid, we obtain the final calibration grid as the difference between the ideal grid and observed deformation, as shown in equation (6). We can consider this final, scaled grid as the visual deformation caused b the projection angle and surface irregularities, expressed in laser units. However, this is not the correction grid we need. This calibration grid can be thought of an inverted grid that counters the aforementioned deformations, and can be obtained as the difference between the visuall obtained deformation grid and the original projected (not-deformed) grid. ( ) calibrated = 2 optimal observed + translate scale (6) x, x, x, (5) 2.3. Calibration Now that we have a calibration grid we can correct the coordinates of our image and overcome the observed deformations. Figure 3: The observed points arranged in a grid fashion The first step is to correlate the middle point of the grid with the center of the laser projection zone. Namel, in the detected grid, the center point will be a given pair of x- The calibration process starts b taking each vector in the original image and finding which cells it intersects from the initial, unchanged gird. The vector is then broken down into smaller parts, each part full contained in its cell. Each part is then transformed, using bilinear interpolation, so it matches the deformations in the calibration grid. These new set of vectors is what we finall displa with the laser projector.

4 An example of a single-projector calibration can be seen in figure 4. considered an extension of the calibration for a single projector. First, for each projector involved we perform the single calibration method as previousl explained. The next step is a matter of fine-aligning the projection areas and resizing the areas accordingl. (a) (b) To align the projection areas we take the middle point of the grids in pixels, as we observed them in (4), and find the average middle point. Since this middle point is in pixels, we multipl b the pixel-to-laser scale value, for each projector and move the center of the projection area for each projector to the common averaged center b appling a pre-translation to the entire grid, as shown in (7), where translate i is the translate value obtained in (4), for the projector i, and scale i is the scale value obtained from minimizing (5). pretranslate translatei center = nprojectors i = center scale i (7) The scaling problem is solved in a similar wa. We find the average pixel-to-laser scale value of each projector and then appl a pre-scale to the calibration grid prior to make the necessar corrections. (c) Figure 4: (a) Test pattern prior to calibration, (b) compensation due to calibration and (c) final, corrected image. 3. MULTIPLE PROJECTORS CALIBRATION Since our goal is to combine the projection areas for each single projector to work as a single unit, initiall we manuall aim the projectors to be as overlapped as possible, regardless of the projection deformations that ma arouse from this. An example of this setting can be seen in figure 5 To perform the final calibration, we simpl use the new centered, resized grid, instead of the one obtained with the single-calibration method. 4. RESULTS For our experiments we used two ILDA compliant laser projectors. Two interface boards were used. A Pangolin QM2000 internal PCI interface and a QM2000.NET, network based interface. In order to perform the communication, we used the SDK provided b the manufacturer. While, for best results, we should place both projectors to have their projection zones as overlapped as possible, for our experiment we separated the projectors on purpose. Figure 6 shows each projection grid for both projectors, prior calibration. We can see that both size and perspective differ from each other Once we have calibrated each projector individuall, we performed the necessar calculations to overlap the two regions. Projector 1 Projector 2 Audience (digital camera) Figure 5: Calibration grids for both projectors The process for calibrating multiple projectors can be An example of the dual-calibration in action can be seen on figure 7. In this example, we took our universit logo, transformed it into vectors and split it into two parts with similar amount of vectors. In the first image, we see the two parts being projected prior to the calibration takes place. We can see that there is an evident gap between the two parts. After calibration, however, we can see that the original image is restored and correctl displaed To test the effectiveness of our method, we created frames

5 with increasing complexit and displaed them with both methods (single and dual projection), and changed the scan rate until a noticeable flickering effect appeared. To measure the flickering effect, we set up a camera to take snapshots with a shutter speed of 1/30 th of a second. If the picture taken showed the entire drawing as it was intended, the scan rate was adequate and we can assume that no flicker appears. However, if the picture was incomplete, this meant that the projector failed to refresh at an appropriate speed. The results are shown in figure 8 (a) Figure 6: Calibration grids for both projectors We can see from this result that using two projectors instead of one, allows us to draw more points per frame using almost exactl half the scan-rate. In other words, b using the same regular scan-rate than a single projector, we can achieve images that are twice as complex. 5. CONCLUSIONS We have shown a method that automaticall corrects the deformations caused b projection angles and surface irregularities for single projector. In this wa, we can safel project onto a surface without risking ee injuries on the viewers. We also extended this method b combining two projectors to full overlap, creating a single projection area. This new area can help reduce the flickering effect for complex images with large numbers of vectors, while keeping the same scan-rate. Figure 7: (a) Universit logo before and (b) after calibration e t a R n a c S (b) Single Projector Dual Projector Points Figure 8: Points per frame versus required scan rate to avoid flickering

6 6. FUTURE WORK While we have shown a consistent method for combining multiple laser projectors for collaborative projection, the problem of obtaining an adequate break-down of the image and assigning the responsibilities to each projector according to their capabilities remains to be done. This would be helpful, for example, in a heterogeneous environment, where faster projectors can compensate for slower ones. Another interesting point of stud is to extend this method to handle not onl still frames, but animations as well. In this case, the speed of the projectors must be taken into account to avoid the slower projector lagging behind. Issues such as snchronization and frame-rate will become areas of research. ACKNOWLEDGEMENTS This work was partiall supported b the Japanese Ministr of Education, Science, Sports and Culture, Grant-in-Aid for Explorator Research REFERENCES [ABD07] An Efficient Drawing Algorithm for Laser Graphics, Purkhat ABDERYIM, Osama HALABI and Norishige CHIBA, p9, Tohoku-Section Joint Convention of Institutes of Electrical and Information Engineers 2007 [BRO05] Camera-Based Calibration Techniques for Seamless Multi-Projector Displas. BROWN Michael, MAJUMDER Aditi, YANG Ruigang, IEEE Transactions on Visualization and Computer Graphics, March/April 2005 (Vol. 11, No. 2) pp [ISO05] Picture transfer protocol (PTP) for digital still photograph devices. ISO Standard 15740:2005 [MEN07] Camera-Based Calibration for Laser Projectors, Francisco MENENDEZ, Osama HALABI and Norishige CHIBA, p.3, Tohoku-Section Joint Convention of Institutes of Electrical and Information Engineers 2007 [RAS98] Seamless Projection Overlaps Using Image Warping and Intensit Blending, RASKAR R., WELCH G., and FUCHS H, Fourth International Conference on Virtual Sstems and Multimedia, (Gifu, Japan), Nov. 1998