JOHN FRITSCHE & HANS WERNER CS534 SPRING11 Page [1]

Transcription

1 e JOHN FRITSCHE & HANS WERNER CS534 SPRING11 Page [1]

2 FACE-IT Face Authoring Compiler Engine - Interactive Tool - { An online application to construct a new face automatically from existing images with no image processing experience needed } it.info [ In this paper ] 1) Introduction a) Abstract b) Motivation for project c) Formal statement of the problem 2) Methods and Algorithms: a) Harvesting the images b) Defining a coordinate system of a face c) Image processing d) Inserting the face part into the base image e) Blending of the face parts to the base image f) Compiling the output image g) Masking h) Database Structure 3) Experimental results a) Good Results b) Bad Results 4) Conclusions a) Closing remarks b) Related work c) Future Work d) References JOHN FRITSCHE & HANS WERNER CS534 SPRING11 Page [2]

3 [ 1 Introduction ] [ Abstract ] In this paper, we present an image based technique of creating a novel face from a library of face parts. Our implementation uses cutting edge web technology to provide a simple user interface as a means of compiling a new face. The user selects from a set facial features broken down into eyes, noses, and mouths, placing them in any combination onto a base image. Our system uses a library of facial features, and re-renders a new image each time the user selects a different feature. This allows the interface to provide a real-time preview of results, and save the experimental faces at different stages if desired. Similar facial image manipulation mechanisms currently available differ from our system by keeping the main features of a subject's face together. What sets our idea apart is the freedom to separate the main human facial areas into a unitary feature, and combine them in any fashion. The new combination of parts is stitched together automatically entirely by the compiling engine; with no 3D information required or user assistance needed, other than the selection of a part. While algorithms exist that perform face replacement in photographs, such as the algorithm presented in the research paper "Face Swapping: Automatically Replacing Faces in Photographs," we are unaware of any algorithms that construct feature by feature a new face from existing photographs. We have developed a system that can quickly produce images of human faces comprised of portions of multiple different photographs. Five distinct facial features are used to create a composite image: nose, mouth, left eye, right eye, and base facial structure These components are blended together to create what is hopefully a somewhat believable human face. [ Motivation ] The applications for such an algorithm range from entertainment to more practical uses. As an entertainment device, swapping facial features between two photos can produce a huge variety of amusing results. On the practical side, our algorithm could be adapted to construct semi-realistic photos of a person that could be used as a type of 'police sketch' of a criminal. With a large enough set of faces and facial features, it would theoretically be possible to recreate a an image of a face that is accurate enough to be used to identify someone either in person or by using facial recognition algorithms to search the internet for similar photos. [ Problem Statement ] We strive to develop a web interface that allows a user to quickly and easily create a recognizable human face composed of portions of different photographs. Some of the main problem areas include mismatches between images in terms of skin tone, facial feature size, gender, facial orientation and hair/facial hairstyles. JOHN FRITSCHE & HANS WERNER CS534 SPRING11 Page [3]

4 [ 2 Methods & Algorithms ] [ Image Processing ] We chose to replace the Matlab image processing toolbox with the GD image processing toolbox packaged with PHP, in order to take advantage of the speed of localizing the image processing on the same server, as opposed to a server running Matlab. GD is an open source code library for the dynamic creation of images by programmers. GD is written in C, and "wrappers" are available for Perl, PHP and other languages. GD creates PNG, JPEG and GIF images, among other formats. GD is commonly used to generate charts, graphics, thumbnails, and most anything else, on the fly. While not restricted to use on the web, the most common applications of GD involve website development. [ Harvesting the images ] During the initial testing phase of our application we chose to search and download free head on face shots of celebrities and general faces that can be segmented into individual parts. Using Photoshop we manually cropped each base image into its individual parts and stored them a database by parts. JOHN FRITSCHE & HANS WERNER CS534 SPRING11 Page [4]

5 [ Adjusting image angles ] An important consideration when harvesting facial features and defining feature coordinates is the angle of the face in the source image. If the subject s head is tilted slightly and their eyes aren t level, small differences can lead to problematic results as seen in the figure to the right. For this reason, it s important to rotate the source image to ensure that the eyes are relatively level before harvesting the facial features. [ Defining a coordinate system ] In order to allow for the process to be entirely automatic we developed a system of predefined 2D coordinates of the where the locations of each feature might need to be placed. We started by extracting the x,y coordinates of each part for each image, and storing it in the database with the base image. The coordinate system is then processed on the fly and calculated based on the images selected. [ Creating a compiled face ] Using the web interface, the user chooses the base image and the facial features they wish to add to the base. Every time a selection is made, a jquery post is made to the PHP script that compiles the image, passing along the ID of each of the selected features. When the user is satisfied with their compiled face the resulting output file is moved to a results folder and the filename saved in the results table. No interim results are saved, or ever reach the database. JOHN FRITSCHE & HANS WERNER CS534 SPRING11 Page [5]

6 [ Overview of blending algorithm ] 1. collect images a. retrieve filenames of base image and facial features from database b. create jpeg images using these filenames 2. retrieve feature coordinates from database for the selected base image 3. masking a. retrieve mask images from database b. apply these masks to the facial feature images 4. blending a. using the coordinates retrieved in step 2, copy the masked facial features onto the base image b. note for best results, add the nose first and allow the eyes and mouth to overlap it [ Masking ] First, the alpha mask is resized to fit the image that it will be applied to. Then the image is scanned pixel by pixel, and at each location the alpha value of the output is calculated based on the value of the alpha mask at that location. The alpha mask is a black and white image in which white indicates fully opaque, black represents transparent and grey values are somewhere in between. Having an alpha mask that fades to black on the outside edges allows us to hide the seams of the facial features when we copy them onto the base image. Some interesting issues arose while trying to determine the best way to define the alpha masks to use. It became apparent that it would be difficult to create a single set of masks that could correctly handle every facial feature from every source image since there is so much variation between human faces. For example, an alpha mask that would accommodate Justin Bieber s JOHN FRITSCHE & HANS WERNER CS534 SPRING11 Page [6]

7 nose might not be large enough to fit the Fonz s nose, which is significantly larger. We then attempted to create multiple masks for each facial feature so that each source image s features could be cropped using the bestfitting mask. The problem with this approach is that a small cropped feature might not fully cover the feature of the base image. For example, Steve Buscemi s tight smile is much smaller than Beyonce s toothy grin, and if Buscemi s mouth is cropped using a smaller mask then his mouth will fail to cover up Beyonce s mouth. For this reason, we settled on a single set of masks that best fits all of the facial features, since this gives the best possibility that a cropped feature will fully cover the feature in the base image. However, it is impossible to define masks that work for every facial feature. The figure on the left for an example: while the mask captures the correct portion of Beyonce s eye, it captures too much of Justin Bieber s face and some of his hair is included in the cropped region. [ Database Structure and Image Library ] We chose to drive our image library with MySQL, and relational database standards. We take advantage of these protocols to power the randomization of any face part to any face base, by exploiting the many to many relationships of a typical MySQL database setup. The image library is broken down into core parts that represent the many ways a user can interact with the system. The first way is selecting a base image to work with. We provide a database table of bases which holds only the file name of the base, and the number of features this base can take on (which will allow for future face parts to be added to the system at a later time). The second mode of interaction is the swapping of the facial features which is powered by several database tables and relationships. The coordinate system is structured as series of 3 tables (a) face_locations - a fixed table of codes that represent the position on a face. (b) base_coordinates the x and y coordinates of where the specific location code lies on the given face, and (c) parts - the library of face parts, which has the same location code that designates each image as an eye, nose, or other part accordingly. The integers 1 9 have been assigned to each location of a face and make up sections of a human face starting from the left eye as 1, and working clockwise around the face. Finally the actual face is dynamically compiled live on the front end by associating all the tables together using php and jquery to access and save to the database. The preview face image is manipulated by the only in front end web application, and only the final result is stored into the database. JOHN FRITSCHE & HANS WERNER CS534 SPRING11 Page [7]

8 [ 3 Experimental Results ] The images created by our algorithm vary greatly in terms of believability and resemblance to an actual human face. Pairing a base image with facial features that match the skin tone, facial orientation and gender of the base image will often result in a somewhat plausible human face. However, a gender mismatch is almost always immediately recognizable for most feature-base combinations. In addition, if the skin tone does not match somewhat closely, it will be very obvious that the image is stitched together using parts from multiple faces. Slight difference is facial orientation will also show up as glaring signals that the face is not real. Another problem is that the hair and facial hair might not always match between the base and the cropped features. For More Results visit JOHN FRITSCHE & HANS WERNER CS534 SPRING11 Page [8]

9 [ 4 Conclusions ] [ Closing remarks ] While our system is limited in the number of facial feature combinations that will produce a truly believable human face, we ve shown that it is possible to approximate a face composed of parts of multiple photographs by using relatively simple blending techniques. Our algorithm has the potential to be put to practical uses in the future, but in its current form it can be used to create a variety of highly entertaining facial compositions. [ Related Work ] Our project is closely related to the system proposed in the paper "Face Swapping: Automatically Replacing Faces in Photographs." The major difference between their approach and ours is that they attempt to replace an entire face in a photograph, while we seek to achieve a finer grain of control by allowing each facial feature to be replaced independently. While their approach requires that candidate images for face matches are very similar in orientation, our system is slightly more forgiving of differences in orientation. This is because our algorithm preserves as much of the base photograph as possible, effectively tricking the human eye into seeing facial features with mismatching orientation as being in line with the orientation of the base image. [ Future Work ] We intend to continue to improve upon the functionality of the website as an entertainment device, perhaps providing easy methods for users to share their generated images with friends on social networking sites. We also intend to add more source images for users to select from and possibly a method to allow users to upload their own images and extract the facial features from the images. Our system could potentially be adapted to automatically extract features using feature detection algorithms, which would make it possible to collect photos from online photo repositories. A number of improvements could be made to the blending algorithm itself. The biggest improvement would be to implement skin tone blending to reduce the contrast between different facial features. Another potential improvement would be to generate the alpha masks by processing the facial feature and attempting to fit the custom fit a mask specifically for each feature. JOHN FRITSCHE & HANS WERNER CS534 SPRING11 Page [9]

10 [ References ] "Face Swapping: Automatically Replacing Faces in Photographs," D. Bitouk, N. Kumar, S. Dhillon, P. N. Belhumeur, and S. K. Nayar, ACM Trans. on Graphics (also Proc. of ACM SIGGRAPH), PHP GD Image Processing toolbox JOHN FRITSCHE & HANS WERNER CS534 SPRING11 Page [10]