Predicting 3D Geometric shapes of objects from a Single Image

Transcription

1 Predicting3DGeometricshapesofobjectsfromaSingleImage TamerAhmedDeif::SavilSrivastava CS229FinalReport Introduction Automaticallyreconstructingasolid3Dmodelfromasingleimageisanopen computer vision problem. Fully automatic solutions such as Make3d [1] and PhotoPopup[2]areonlyabletomake2.5Dmodelsfromimages.Ontheotherhand, [3]areabletousemanualintervention(namely,drawingedgesoftheobject)and symmetrytoreconstructthe3dshape.thisworkattemptstousemachinelearning to learn the solid 3D structure of an object in an image. It assumes objects can be brokenintobuildingblocksofcuboids,pyramids,etc.wewereonlyabletomanage toinvestigateforthecaseofcuboidal onlyobjects. Ourapproachisasfollows.Weposethelearningofedgesoftheobjectinthe imageasalinearregressionproblem.weusetheseedgestopredictwhatkindof 3D cuboidal shape could have given the object its particular 2D projection in the image. This is defined as an energy optimization problem, which is solved by gradientdescent. DataSet: OurdatasourceisGoogle3DWarehouseSketchupmodels.Thereareseveral motivationsforthatchoice,amongstwhichare: Thefreeandabundantavailabilityofdata. Themodelsrepresentactualbuildingsandtheyareratedbytheirquality,so realisticmodelscanbechosen. Themodelsallowfutureextensionoftheprojectformorecomplex,aswellas non cuboidbodies. Modelshadtexture,whichcanbeusedtogivethe3Dmodelamorerealistic look. GoogleSketchupallowstomanipulatethosemodelsandlookatthemfrom differentangles,aswellas,takesnapshotsfromdifferentviewangles,which translatesinto2dinputimagesforthealgorithm. In order to prepare our Data set, we downloaded 50 models of different buildings fromgoogle3dwarehouse.wealsowrotearubyscriptthattakes6differentsnap shotsofthemodelfrom6differentviewingangles.intotalwehad300,different2d images(eachofsize558x404pixels).sketchupallowsitsuserstoremovetexture andshadowsfromthemodel.soweexploitedthesefeaturestotakeanadditional snapshotateachofthe6positionswithonlytheedgesofthemodel(referfig.1). Theseextraimagesrepresentthegroundtruthdatafortheedgedetection learning algorithm.

2 Fig1:Showsthetexturedimageofthe3Dmodel,theedgemapthatwegeneratefromsketchuptouseas groundtruthdata,andtheedgemapwedetectafterlinearregression MethodandResults: Part1:SignificantEdgeDetection Wedesignedfeaturesandperformedlinearregressiontodeterminethegeometrical shapeedgesoftheobjectintheimage.inparticular,foranypixelwewanttoclassify itaseitheredgeornon edge. Weselectivelysamplewhichpixelstotrainon,sinceconsideringallwouldresultin alargefeaturematrixwithwhichmatlabwouldrunoutofmemory.currently,we are sampling 90% edge pixels and 2% of non edge pixels. Our final sample has a ratioofedge:non edgepixelsof3:1. Last,theresultofthelinearregressiongivesuspredictionsonwhetherthepixelis edgeornotasarealnumberintherange[0,1].foreachimage,weadaptivelyselect whatthresholdtouse,forclassifyingasedgeornon edge,bychoosingthethreshold thatminimizestheerror. The following table summarizes the average overall error for training, cross validationandtesting: Features/Error% Training CrossValidation Testing Y,Cb,Cr Y,Cb,Cr,Hough Y,Cb,Cr,Hough, Percentage

3 FeaturesandResults We consider 101 features for any given pixel. First, we consider a 9x9 pixel neighborhoodpatchinycbcrcolorspace.thisistheabletocapturethedifference in Luminance, and blue/red color space. For Luminance, we consider a 9x9 patch andforthecolorcomponentsa3x3patch.thissmallerpatchwasconsideredsince MATLABwouldrunintomemoryissueswhiletrainingonthisdata.Thetrainingand testerrorforthisisshowninfigure2. Fig2:Falseandpositiveclassificationserrorfortraining(totheleft)andtesting(tothe right)datawhenusingy,cb,crfeatures Wethenconsideredaddingtwomorefeatures,whichhadamoreexplicit relationshipwithedges. 1)ConsiderwhetherthepixelliesonaHoughline.Forthis,aHoughtransformwas doneoneachinputimageandthe8mostlikely to belineswereselected.pixels lyingtheselinesweretaggedwiththisfeature. 2)Percentageofedgepixelsinthe9x9neighborhoodpatch. TheresultsfromthiscanbeseeninFig.3andFig.4. Fig3:FalseandpositiveclassificationserrorfortrainingandtestingdatawhenusingY,Cb,Crand Houghlinesfeatures

4 Fig4:FalseandpositiveclassificationserrorfortrainingandtestingdatawhenusingY,Cb,Cr,Hough linesandedge pixelspercentageastheinputfeatures We note that we are most interested in lowering the false positive rate. The false negativeratecanbetoleratedtobehighbecause(1)thegroundtruthlabelshave thickedges(2)losingafewoftheedgepixelsdoesnotaffecttheenergyfunctionin Part2asmuchashavingnoisypixels. MethodPart2:CuboidFitting Inthisstep,wepredictwhat3Dcuboidcouldhavebeenprojectedontotheimageto give the object shape(as determined by the edges predicted in Method Part 1). A parameterized 3D cuboid is projected onto the image plane with correct perspective, and we calculate an energy function based on this projection. We generatethisenergyfunctionbyblurringtheedgeimagegeneratedinpart1such the maximum energy lies on the actual edge pixels. Then we perform gradient descentonthefollowingfunction: The problem of optimizing this function suffers from a lot of local minima. Hence whatweinitializetheparametersourcubetoiscrucialtosuccess.tomitigatethis problem, we run gradient descent initially on 50 different sets of initialization parameters for 5 iterations. We then pick the 10 best models, and perform more iterations. We repeat this funnel selection method till we get the most optimal model from our 50 initializations. As seen in Fig. 5 below, where the initialization wasdoneautomatically,theoptimizationfallsintolocalminima.

5 Fig5:Fittingstuckatalocalminimum However,insomecases,theoptimizationisabletosnaptheinitializedpredicted modelontothepredictededges(fig.6). Fig6:Wheninitializedclosetotherightpositionofthebuildingitsnapsinandfitsaccurately We also tried adding an extra energy term; proportional to the number of edge pixels the cuboid was on. However, in our tests this resulted in less optimal solutions. LimitationsandFutureWork TheCuboidFittingstepfailsforanumberofreasons: 1)Theenergyfunctionhasalotoflocalminima. A) Currently, when we project the cube onto the image, we draw all the edges of the cube. However, by calculating which side of the cuboidisvisibleandonlyprojectingtheedgesofthosesides,wecould haveabetterenergyfunction. B)Oncewehaveafinalresult,thenonecantryperturbingitandsee whetheritsnapsbacktowhereitwasoriginally.

6 2) While the algorithm can predict edges from the input well, it has a false positive rate that can results in quite a few non edges being classified as edges. This noise affects the energy function and even when the cube is initializedsuchthatitshouldreasonablysnapintoplace. Furthermore, in the edge learning phase, the artificial (Sketchup Model) images that we test on have only one object of prominence in the image. For real images, there may be more peripheral objects whose edges also get detected. One would havetodesignauiwhereausercanindicateregion Of Interest(ROI)andconsider edgesonlywithinthisroi. Acknowledgments WethankAshutoshSaxena,ChristianTheobaltandSiddharthBatrafortheir guidanceandhelpfuldiscussionswhileplanningtheproject. References 1.ASaxena,MSun,AYNg Learning3DStructurefromasinglestillimage. ComputerVision,2007.ICCV2007.IEEE11thInternational, DHoiem,AAEfros,MHebert "AutomaticPhotoPop up"proceedingsofacm SIGGRAPH2005, PEDebevec,CJTaylor,JMalik ModellingandRenderingArchitecturefrom Photographs:AhybridgeometryandimaProceedingsofthe23rdannualconference oncomputer,1996