Perspective Correction Implementation for Embedded (Marker-Based) Augmented Reality

Transcription

1 Perspective Correction Implementation for Embedded (Marker-Based) Augmented Reality Bernardo F. Reis, João Marcelo X. N. Teixeira, Veronica Teichrieb, Judith Kelner Universidade Federal de Pernambuco, Centro de Informática, Grupo de Pesquisa em Realidade Virtual e Multimídia {bfrs, jmxnt, vt, jk}@cin.ufpe.br Abstract Most of augmented reality applications need mobility and processing power to efficiently provide immersion to the end user. These requirements can be achieved using embedded devices such as FPGAs to execute the application. In order to work properly, idmarker-based applications require removing the perspective distortion of the input images, so that it s possible to have the correct reading of the id. Several approaches can be taken to fix the perspective distortion in an image, as 4 points mapping and the use of parallel and perpendicular lines. This work presents an FPGA implementation of a module that uses 4 points mapping to calculate the corresponding homography between images, in order to solve the problem of perspective distortion correction in an embedded AR system. 1. Introduction Augmented reality (AR) applications play their most significant role as systems that can be easily taken and used anywhere. However, just as any other graphic application, they require some processing power so that a realistic experience can be provided to the user. An embedded architecture developed on an FPGA (Field Programmable Gate Array) would supply both mobility and processing power to the AR solution, as it is a tiny chip and has the efficiency of hardware level execution. Embedding the whole AR pipeline would spread the use of this technology, since it would enable AR applications to run on compact devices. Marker-based AR systems use markers to keep track of the position in which the digital objects will be rendered. Fiducial marker systems consist in patterns that should be uniquely identified in the environment by some detection algorithm. One of these systems is the BCH marker system, based on the Bose- Chaudhuri-Hocquenghem code [1], which allows up to 4096 markers and is used in the ARToolkit Plus library. This type of marker consists of a black square border and an interior region filled with a 6x6 grid of black and white cells. These cells represent a 36-bit code. Such id-based marker systems are widely used in AR applications, where the reference of each marker is important. The detection algorithm of an id-based system such as the one described above basically identifies the markers on the image frame and reads the code drawn upon it. However, if the marker s plane is not aligned to the screen s plane, it s needed to find a mapping between the two planes, known as homography. The homography is used to make possible the correct reading of the id drawn on the marker, which is then checked using a specific matching algorithm. In order to do so, there are some algorithms already developed to extract such perspective distortion. One of these approaches uses parallel and perpendicular lines, which is described in [2]. It identifies a pair of parallel lines that intersect each other in the perspective image, uses the line that passes through the intersection points to remove the projective component of the distortion, finds a pair of perpendicular lines and uses it to remove the affine component. The approach chosen to be used for the perspective distortion correction module presented in this paper requires 4 non-collinear points to create the mapping between the two planes. These points generate a set of 8 equations. To solve these equations an LU decomposition algorithm may be applied, due to its simplicity, as described in [3]. This algorithm writes the corresponding matrix of the set of equations as a product of a lower and upper triangular matrix. The equations are then solved using forward and backward substitution. This work focuses on the development and validation of an embedded perspective correction module, based on the LU decomposition algorithm, thus enabling the identification of id-based markers.

2 This module is part of ARCam, a framework for the development of embedded AR solutions. ARCam aims on facilitating the creation of new applications using the available hardware infrastructure. With this framework, it will be possible to create different types of solutions, for example, smart cameras programmed to perform equipment inspection [4][5][6]. This work also presents an optimized module, which doesn t require to perform the LU decomposition itself. In sequence, section 2 presents related work on embedded AR systems, section 3 explains id-based marker detection and section 4 describes perspective distortion modeling, as well as how to correct it. After that, section 5 explains the LU decomposition algorithm used and section 6 presents the FPGA implementation of perspective distortion correction using LU decomposition. Section 7 introduces the optimizations done to the previous developed module and section 8 presents its implementation. Finally, some conclusions are discussed in section Related work Most of the AR applications developed so far has been done targeting PCs or workstations. However, there are some AR applications already developed targeting embedded platforms, such as [7] and [8]. Piekarski et al. [7] propose a low powered AR application using FPGAs, applying colored balls as markers. Since the system scans the image for a spot of a specific color, it s very hard to distinguish a marker from a random object of the same color, producing high false positive rates. Also, there is no way to differentiate two markers on the scene. Real world image is acquired and processed using an FPGA. The coordinates of the markers are calculated and sent to a computer that renders the virtual image. The virtual image and the real world image are put together by a video combiner and then the output is displayed using a HMD (Head Mounted Display). Toledo et al. [8] implement an AR application for visually impaired people on an FPGA. No markers are used as the system only needs to superimpose contour information of the real world image. A Cellular Neural Network is used to extract the contour information. There are some other Computer Vision applications implemented on FPGAs, including [9], that proposes an FPGA architecture for static background subtraction in real time, which is very similar to any AR architecture. As for AR and Computer Vision applications, its hardware implementation is clearly feasible. Solving linear systems using embedded devices is also possible, as shown in [10], where sparse matrices are factorized in parallel. In order to do so, Wang et al. [10] implemented a highly parallel Block-Diagonal- Bordered algorithm for LU decomposition. To simplify the design and reduce the implementation time, there were used six Altera Nios configurable Intellectual Property processors for computation and control. They also compare their parallel solution with a sequential one, based on Nios and a hardware floating-point unit (FPU), achieving significant speedup. 3. Basic concepts Before explaining the implementation of the perspective distortion correction module, some necessary background is provided. First, the AR pipeline and the perspective distortion are explained. Then the LU decomposition algorithm and the optimizations applied are presented Marker detection In order to compose an AR scene, it's necessary to take several steps since the input image from the camera. These steps, detailed in Figure 1, compose the AR pipeline. Apart from the step shown with dashed contour, every other step in the pipeline has already been developed in ARCam, what leads to an almost complete AR embedded framework. The step of marker detection is one of the procedures required to develop AR applications. Markers create a link between physical and digital worlds. Once a marker is found in a frame, it is possible to extract its position and other properties to render some digital object upon it. Marker s characteristics depend on the marker system it belongs to. There are several marker systems already developed. The application requirements for pose estimation and tracking, like accuracy of the position detection, how quick is the detection and diversity of markers are some of the characteristics of marker systems that should be taken into account during an AR project. Newer marker systems even enable detection over marker partial occlusion and are robust to lightning conditions, which normally affect the detection rate. Figure 2 shows some examples of markers.

3 Figure 1. AR pipeline 3.2. Perspective distortion As shown in [2], perspective distortion can be modeled by the following projective transformation, as written in equation (1), where p and p are 3-vectors representing a point and H is a homogeneous nonsingular 3x3 matrix. (1) Figure 2. Examples of marker systems The procedure that detects a planar, square shaped and id-based marker, as the one chosen in this work, consists in scanning the image looking for square shapes, correcting the perspective distortion of the image, reading and identifying the id drawn on the marker. This work tackles the perspective distortion correction issue in an embedded solution. The square shape of the markers has an important role in its detection, as its corners provide the 4 points that will be used on the mapping between reference s and marker s plane. To get the coordinates of the corners, a specific algorithm to detect rectangles is applied to the input image. This algorithm scans the image for contours based on the contrast of the marker s black border. Then it checks if each contour is a rectangle, analyzing the angle between the edges and the number of edges. If the contour is finally accepted as a rectangle, the coordinates of the vertices are used to correct the perspective distortion of the marker. With 4 point correspondences, it is possible to extract 8 equations to generate matrix H, as written in equation (2). (2) Assuming that 1 and since we only have 8 equations, it is possible to mount the following matrix presented in equation (3) using the previous 8 equations: (3) It is enough to solve H up to an insignificant multiplicative factor, as only the ratio of the elements matters. In order to eliminate this multiplicative factor, it is imposed that the norm of the matrix be 1. To solve this set of linear equations, it is possible to apply several algorithms. This work proposes using LU decomposition along with forward and backward

4 substitution. As H is found, it is possible to estimate image s marker from the reference marker and then to read its code LU decomposition The LU decomposition algorithm is used to factorize a matrix into a lower and an upper triangular matrix, as shown in equation (4). Once these triangular matrices are found, it becomes extremely easy to perform some tasks as to solve a set of linear equations, find the inverse matrix and calculate the determinant Optimizations Although the LU decomposition algorithm runs fast enough for being used in AR systems, there were some optimizations that could be done to improve its performance even more. Observing that the reference points are always the same, as the marker plane is always mapped to the screen plane, a lot of operations could be removed from the solution. Placing the reference marker s vertices as shown in Figure 3 enables a significant simplification in matrix H calculation. (4),,,,, ,,,,,,,,, ,,,,,,,,, , 0,,,.,,, There is more than one way to perform the LU decomposition algorithm. This work implements LU decomposition based on Crout s method, following the description in [3]. Crout s method splits the matrix into a lower and an unit upper triangular matrix. It requires pivoting in order to ensure stability, so implicit pivoting was implemented. The implicit pivoting basically interchanges the rows in order to have, for each column, the row with the highest pivot. The procedure of decomposing the matrix using Crout s method can be formulated as the following assignments (equations 5 and 6), for each 1, and 1, :,,,,, (5) 1,,, (6), Once the matrix is finally decomposed into the two triangular matrices, it s ready to be used for solving the linear system. In order to solve a set of linear equations such as (7), it is needed to solve first (8) for y using forward substitution and then (9) for x by back substitution. (7) (8) (9) Figure 3. Reference marker coordinates after optimizations. After applying the simplifications to the matrix, it is no longer necessary to execute the LU decomposition algorithm, since the linear system has become simple enough to be solved with only a few operations. In the end, two different modules were implemented. The first one, explained in section 4, solved the linear system using LU decomposition, while the second one the enhanced version, presented in section 4.1 simply do some mathematics operations to find matrix H. 4. Implementation The FPGA implementation of the module that performs the perspective distortion correction based on the LU decomposition algorithm was done targeting a Stratix II EP2S60F1020C4 DSP development board from Altera, running at 25MHz. It was performed using the programming language for hardware design VHDL (Very-High-Speed Integrated Circuits Hardware Description Language). The entire process was implemented in two components, which are executed in sequence. The first one creates and splits the matrix H of the linear system into two triangular matrices and the second solves the linear system forward and backwards as explained in

5 section 3.3. The first component presents a heavy overhead caused by control flow operations, as it has to iterate over the whole matrix twice. The components share two M4K memory blocks that store the matrix. The board features a 100MHz oscillator, but, due to design constraints, the clock frequency is limited to 25MHz. In order to speed up execution, fixed-point arithmetic was adopted. 16 bits were used to represent the integer part and other 16 bits to represent the fractional part. Precision loss is not critical, but coordinates values needed to be shifted right by 5 bits due to overflow problems. The first component used four 32-bit multipliers and one 32-bit divider. The second component used one 32-bit multiplier, one 32-bit divider and one 48-bit square root calculator. The multipliers and dividers were supplied as Altera s MegaCore functions, while the square root calculator was implemented by the authors. The multipliers use one 36x36 digital signal processor (DSP) each and the divider is pipelined with a latency of 10 clock cycles. Both components contain a finite state machine (FSM) that controls the data flow and models the implementation. First component s FSM has 158 states while second component s FSM has 47 states. When it isn t possible to split the matrix into two triangular matrices, i.e. the case of the matrix being singular, an error flag is set, warning other modules about the problem, and the procedure stops. Running at 25MHz, the total execution time is 63us, allowing almost LU decompositions per second. This way, it is possible to execute it more than 500 times per frame at 30 fps Enhanced version This optimized version of the perspective correction module was implemented targeting the same FPGA development board as the one used for the first implementation. Several simplifications were done to the whole system, including the reduction of operations executed and the removal of the memory block that allocated the matrix. However, the clock frequency remained the same due to a bottleneck in the division module. The solution of the linear system became so simple that only 23 additions/subtractions, 9 multiplications and 2 divisions were done to solve it. There was still need for applying the constraint which determines that the norm of the matrix has to be 1. Therefore, another 8 additions, 9 multiplications, 1 division and 1 squareroot operation were applied. As fewer operations were required, more bits could be used to perform them, achieving more precise results. 48-bit multipliers and divider were used, instead of the 32-bit versions utilized for the module that performed the LU decomposition. They were also supplied as Altera s MegaCore functions. Despite the enhanced version is the result of many optimizations, some improvements were also implemented to increment its functions capabilities, which increased the size of the module. Therefore, the amount of logic elements demanded for synthesizing it also increased. Table 1 presents the board occupation of the two implementations. Table 1. Logic elements consumption. ALUT ALM DSP blocks Memory bits LU Enhanced The main optimization is pertinent to the execution time. The reason is that, since the decomposition of the matrix is no longer needed, all the iterating over the matrix and the overhead due to its control flow were removed. The operations required to calculate the transformation matrix on the optimized perspective correction module are significantly simpler than performing the LU decomposition. Great speedup was achieved, as shown in Table 2. The maximum frequency indicated in Table 2 was provided by the Altera s synthesis tool Quartus II 5.1 and the clock cycles and execution times were acquired using Altera s logic analyzer Signal Tap II via JTAG (Joint Test Action Group) interface. Table 2. Execution time. Maximum frequency Clock cycles Execution time (25MHz) LU 34,66 MHz ,08 us Enhanced 25,42 MHz 114 4,54 us 5. Conclusion This work presents an FPGA implementation of a perspective distortion correction module for embedded marker-based AR systems. It solves a system of linear equations to calculate the homography between two images, as it is needed to detect markers. Besides the use within AR systems, part of the module implemented in this work can also be integrated into any other project that needs to solve a system of linear equations or find the inverse of a matrix. Minor changes may be necessary, because

6 some adaptations have been done to the input and output ports. Due to algorithm complexity, there was some instability in execution, which demanded a lot of time to be fixed. This instability also motivated the implementation of the enhanced module, which was created based on some optimizations that could be done to the linear system. The comparison between both modules showed that the obtained speedup was significant. The conclusion of the perspective distortion correction module takes ARCam one step closer to its first release. Having every step in the AR pipeline implemented, in a near future it will be possible to use the provided modules fully integrated for developing embedded AR applications. 6. Acknowledgements The authors want to thank MCT and CNPq, for financially supporting this research (process /2004-7). 7. References [1] R.C. Bose, and D.K. Ray-Chaudhuri, On a Class of Error Correcting Binary Group Codes, Information and Control, Elsevier, March 1960, pp [2] R. Hartley, and A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, Cambridge, UK, [3] W.H. Press, B.P. Flannery, S.A. Teukolsky, W.T. Vetterling, Numerical Recipes in C: The Art of Scientific Computing, Cambridge University Press, Cambridge, UK, [4] G. Guimarães, J.P. Lima, J.M. Teixeira, G. Silva, V. Teichrieb, J. Kelner. FPGA infrastructure for the development of augmented reality applications, in Proceedings of Symposium on Integrated Circuits and Systems Design, Rio de Janeiro, 2007, pp [5] J.P. Lima, G. Guimarães, G. Dias, J.M. Teixeira, E. Xavier, V. Teichrieb, J. Kelner. ARCam: an FPGA-based augmented reality framework, in Proceedings of Symposium on Virtual and Augmented Reality, Petrópolis. Porto Alegre, 2007, pp [6] V. Teichrieb, J.M. Teixeira, J.P. Lima, J. Kelner. Markerless augmented reality based cameras using systemon-chip technology, in Proceedings of Interservice/Industry Training, Simulation & Education Conference, Orlando, EUA, 2007, pp [7] W. Piekarski, R. Smith, G. Wigley, B. Thomas, D. Kearney, Mobile Hand Tracking Using FPGAs for Low Powered Augmented Reality, in Proceedings of International Symposium on Wearable Computers, Washington, DC, USA, 2004, pp [8] E.J. Toledo, J.J. Martinez, E.J. Garrigos, J.M. Ferrandez, FPGA Implementation of an Augmented Reality Application for Visually Impaired People, in Proceedings of International Conference on Field Programmable Logic and Applications, Tampere, Finland, August 2005, pp [9] J. Oliveira, A. Printes, R.C.S. Freire, E. Melcher, I.S.S. Silva, FPGA Architecture for Static Background Subtraction in Real Time, in Proceedings of Symposium on Integrated Circuits and Systems Design, Ouro Preto, MG, Brazil, 2006, pp [10] X. Wang, S.G. Ziavras, Parallel Direct Solution of Linear Equations on FPGA-Based Machines, in Proceedings of International Symposium on Parallel and Distributed Processing, Nice, France, 2003, pp. 113.