The Legend of Sleepy Hollow Program

Transcription

1 The Legend of Sleepy Hollow Program By: Harrison Evans, Jack Meuwissen, and Jacob Wurtz ABSTRACT The Legend of Sleepy Hollow Program is a project which allows the user to take any photo of a person and make them have a pumpkin as a head. Not only is this application fun and entertaining, it is also very versatile with the object you are making the head out of, just in case you aren t a fan of pumpkin (but I mean come on who isn t?). 1. INTRODUCTION Most people are working hard just about every day. When we aren t working hard, we like to relax a bit or do something that brings us joy so that we stay sane. One way to kick back and have fun is to use our Legend of Sleepy Hollow Program. Whether you have a friend you want to poke fun at, a boss who gets on you nerve, or the desire to see what you d look like with an egg for a head, you can use this program to get a couple laughs. Post a picture on Facebook of your friend with a football for a head and see the likes go up as people laugh out loud and comment how hilarious they look. Make a picture of your boss with a cone for a head and print it out and hang it on your dart board, let off some steam. Anyway you do it, it will provide you with good laughs. 2. RELATED WORKS You can find many examples of similar work online, annoying orange is a great example although somewhat different. Annoying orange just take the eyes and mouth of someone while ours takes the full face. Also a common example of the blending algorithm we use (which will be talked about in detail later) is people putting faces on the Mona Lisa Smile. Not sure why everybody loves seeing themselves as Mona Lisa, probably for the same reason they would use our program, just for the fun of it. Our program does take it a little further though. Instead of just blending the face to the object. We use a mixed gradient algorithm so that the face gets some of the texture of the object the face is being applied to. This way it will make it look a tiny bit more realistic. Evans, Meuwissen, and Wurtz 1

2 3. Methods There are multiple different parts to this work of art. You obviously can t just take a picture of a person and a picture of a pumpkin and expect them to return as a person with a pumpkin head. First things first, you need to select the face. Getting the Face From the get go, the user will be prompted to trace (as best they can) the face of the person they want to apply the program to. This is done by using a function called imfreehand(). We set that to a variable and then are able to use.createmask() on it to, well, create a mask. This mask is a binary image which has 1s on the inside of the trace and 0s elsewhere. Even if the user doesn t have that steady of a hand and makes a couple of mistakes tracing, the blending algorithm should take care of most of the artifacts that could come about. The mask is then used in combination with the original image to map the position of the face. The mask uses the pixels with the value of 1 to find the face and can then apply it to the image of the pumpkin (or whatever object you decide to use). Evans, Meuwissen, and Wurtz 2

3 Getting the Pumpkin After the face has been traced. The next thing the user will have to do is outline the pumpkin. This is done a little differently than how we found the mask of the face. Instead of tracing, the user will click to connect lines about the object. Making sure that the whole object is inside the lines (for best results) a binary image of the pumpkin (or object) is created. In both the face and the pumpkin scenario, we use the binary image to get the center of the image. Finding the min and max x- and y-coordinate for which the pixel values are 1. Then subtracting min from max and dividing that value by 2. Then subtracting THAT value from both the x and y max. Scaling of the Pumpkin and Face Unless we got extremely lucky, most of the time the face and pumpkin would turn out to be different sizes. To accommodate for this, depending on which item was bigger (the face or the pumpkin), we would need to either scale down the pumpkin (if the pumpkin was bigger) or scale both the face and pumpkin (if the face ended up being larger). The algorithm had a tendency to take a very long time to run on sufficiently high resolutions, which resulted in us having to downsample images containing more than ten thousand pixels. Applying the Face to the Pumpkin (Poisson Blending) Evans, Meuwissen, and Wurtz 3

4 The act of applying the face to the pumpkin we accomplished using a Poisson Blending algorithm, first described by Perez, et al. in their Poisson Image Editing. We decided that the act of recreating the Perez algorithm in Matlab was akin to reinventing the wheel and would not be a productive use of our time. To this end, we identified and acquired a working, open-source Poisson algorithm from MathWorks Matlab Central code repository. The code we acquired was specified to a specific instance of blending, which required us to generalize the code to work with image masks of awkward and undefined sizes, as well as images of varying sizes. We also modified almost all variables in the code, vectorized the code, and made use of logical indexing to improve performance. The end result is code that operates along the same lines as that we originally obtained, but is highly modified and adjusted to work more efficiently in MatLab s unique framework. The blending function works by first determining the location of our facial mask on the target pumpkin by using the various statistics about size and center points that we obtained previously. Upon properly aligning the mask, the algorithm begins the blending operation by first pasting the pixels from the source image into the target image, and then iterating over all of these pixels and averaging with their neighboring pixels, starting from the outer pixels and moving inward until all pixels have been blended to the specified threshold or the maximum number of iterations has been reached. Limits are placed on the number of iterations in order to maintain a realistic operating time for the algorithm. Mixed Gradients Throughout our project, we had aimed to mix the gradients of both the source face and the target in order to get a face blended onto the pumpkin in such a way that it maintained the pumpkin texture in addition to the facial features. However, during the process we found that our blending of the gradients often lead to discoloration of the face in relation to the surrounding pumpkin. The act of pulling the gradient from the three separate channels, even as we followed the algorithm for mixing gradients as provided by Perez et al., added significant amounts of noise to the laplacian image devised for blending, which resulted usually in a blueish tinge being applied to the blended face when placed upon the pumpkin. Given sufficient time and information, we probably could have fixed this in order to get the desired effect, but we were unable to fully implement it before the deadline and decided to drop the feature. Applying the Pumpkin Head back to the Original Image To get the final product, we need now put the pumpkin head back onto the original image so the face of the pumpkin head is right after the face if the original image. Doing this is very similar to getting the face onto the pumpkin just without the blending. Evans, Meuwissen, and Wurtz 4

5 4. Experimental Results Through our experimentation with the methods we used, we got many different results. From using others codes to our own implementations, the results were interesting. Failures: Final Image with a bad pumpkin mask Including a stem disturbs the centering algorithm Evans, Meuwissen, and Wurtz 5

6 Final Image with a bad face mask Image with improper mask dimensions 5. Conclusion and Ideas for Improvement Now that we have created a person with the head of a pumpkin, people days will get a lot more entertaining. While we ve accomplished giving people enjoyment and are satisfied with our results there are still many ways in which to make this application better or more advanced. Here are a couple things that can be improved upon or added: - Multiple Faces: Our program is just set to take the face of a single person. Whether you re input image is a portrait or a group picture, you will still get a pumpkin headed version of the person whose face the user traced. An addition we could include is having the user trace multiple faces and input just as many objects as traced faces and then allow them to select which object to have the face blended on to. - Better and Automatic Face and Pumpkin Detection: Originally we wanted to use an auto detect feature which detected the faces of an image and we would use the location the function found to create the mask. There were a couple downsides to this: the location of the face that was returned was a block and went outside the area of the face. This would cause problems with the blending method since there is normally some sort of contrast between the face and whatever is beyond it. Also this face detect method would sometimes return something that wasn t a face so we would be putting a pumpkin on something that isn t a face. We now know of better face detection methods (from the Duckbill program presentation) but we don t have the time to implement it before the deadline. It d be something we could look into after we finish Evans, Meuwissen, and Wurtz 6

7 everything else up. We d like to apply this automatic detection to the pumpkin as well, which would in turn bring up the quality of the masks that we receive from these, removing the variable of human error. - Feathering Pumpkin Edges: We would have liked to implement a proper feathering system for the edges of the pumpkin when it was placed upon the head. That way, minor inconsistencies in the mask could be ignored and blended into the background to make a cleaner-looking image. - Size of Face vs Size of Pumpkin: There currently exists an error that occurs and is based around the size of the face image in relation to the size of the pumpkin image. If the image selected for the face is dominated primarily by the face and has no other defining characteristics, the placement of the pumpkin upon the head will not work as designed and has very strange results. Alternatively, if the center of the face is regarded as being a higher value than that of the pumpkin it can cause the same effect. This can result in the program crashing with an out-of-bounds error. - Support for Higher Resolutions: As was mentioned previously, we currently need to downsample images in order to allow our code to operate in a reasonable time-frame. Perhaps by adjusting our algorithm, we would be able to process larger images in a more reasonable amount of time. 6. Code Statistics The majority of the code is our own, but we did acquire a large amount of open-source blending code from two different sources, which we then incorporated into our own. The files that hold the greatest amount of this code are the PoissonJacobi.m and calc_mask_in_targ_image.m. Calc_mask_in_targ_image.m is almost completely unmodified, while PoissonJacobi.m has been modified to suit our needs. In addition to this, Blender.m is a very heavily modified version of the code provided by M. Tanaka. All other code was written by our team and the entirety of the code is around 350 lines, being split between six different files. Harrison Evans took charge of creating facial masks, selectface.m and assisted in the modification of the main.m file. Jack Meuwissen was in charge of modifying the blending code and incorporating it with the remainder of the project. His primary files include PoissonJacobi.m, and blender.m while he also had a significant hand in the modification of the main.m file. Jacob Wurtz was in charge of creating pumpkin masks (selectpumpkin.m) and also wrote the majority of the main.m file and the entirety of the stats.m file. References Evans, Meuwissen, and Wurtz 7

8 P. Prez, M. Gangnet, and A. Blake, Poisson image editing, ACM Transactions on Graphics (TOG), Vol. 22, No.3, pp , Y. Simson, Poisson Image Edition, For code contribution M. Tanaka, Poisson Image Editing, For code contribution Evans, Meuwissen, and Wurtz 8