Level Set Models for Computer Graphics David E. Breen Department of Computer Science Drexel University Ross T. Whitaker School of Computing University of Utah Ken Museth Department of Science and Technology Linkoeping University, Sweden Outline Level Set Model Introduction APPLICATIONS 3D Volumetric Metamorphosis Volume Dataset Segmentation Surface Editing What is a Level Set Model? What is a Level Set Model? A deformable implicit model, φ(x,t) = 0 A deformable implicit model, φ(x,t) = 0 Sampled representation of dimension n+1 Images (2D) represent curves Volumes (3D) represent surfaces What is a Level Set Model? What is a Level Set Model? A deformable implicit model, φ(x,t) = 0 Sampled representation of dimension n+1 Images (2D) represent curves Volumes (3D) represent surfaces Change level set by modifying samples A deformable implicit model, φ(x,t) = 0 Sampled representation of dimension n+1 Images (2D) represent curves Volumes (3D) represent surfaces Change level set by modifying samples Change sample values by solving a PDE 1
This is a Level Set Model! of a curve! Changing the image moves the curve Red curve is defined by φ(x,y) = 127 What is a Level Set Model? What is a Level Set Model? φ(x) is not defined by a specific equation φ(x) is not defined by a specific equation φ(x) is represented by a regular 3D sampling Signed distance volume dataset What is a Level Set Model? What is a Level Set Model? φ(x) is not defined by a specific equation φ(x) is represented by a regular 3D sampling Signed distance volume dataset Level Set model is deformed by evolving the Level Set equation on the sampling Osher & Sethian 1988 r φ( r )/ t = φ ( ) F( r,...) X X X φ(x) is not defined by a specific equation φ(x) is represented by a regular 3D sampling Signed distance volume dataset Level Set model is deformed by evolving the Level Set equation on the sampling Osher & Sethian 1988 r φ( X r )/ t = φ ( X ) F( X r,...) Connects changes in sample values to changes of the level set curve/surface 2
Level Set Speed Term Advantages of LS Models LS model deformation is controlled by F( ) F(X, ) defines the speed of the LS surface in direction of the surface normal N at each point X on the surface F( ) is defined by a user for each application in order to achieve a specific goal We have defined many different speed terms for our segmentation applications Always produce closed, non-selfintersecting (simple) surfaces Deforming Mesh May Self-Intersect No Self-Intersection with Level Set Deformations Important for GCAD, CAM and Analysis Advantages of LS Models Easily change topological genus Holes may close or open Separate pieces come together/split apart Ideal for complex deformable models of unknown genus Figurine automatically manufactured from a level set model 3
Mug-to-Chain Morph Advantages of LS Models Concise interface for control F( ) Advantages of LS Models Advantages of LS Models Concise interface for control F( ) Free of mesh connectivity and quality issues Concise interface for control F( ) Free of mesh connectivity and quality issues No need to reparameterize during deformation Disadvantages of LS Models No inherent parameterization Problem for texture mapping & some deformations Computationally expensive (?) High memory requirements (?) Narrow Band methods are O(surface area)! Adalsteinsson & Sethian 1995, Whitaker 1998, Peng et al. 1999 Cannot control genus (?) Han et al. 2001 Cannot represent fine or sharp features (?) Losasso et al. 2004, Nielsen & Museth 2006 Using Level Set Models in Computer Graphics Involved in three level set modeling projects 4
Using Level Set Models in Computer Graphics Using Level Set Models in Computer Graphics Involved in three level set modeling projects 3D Volumetric Metamorphosis Make one solid model become another Involved in three level set modeling projects 3D Volumetric Metamorphosis Make one solid model become another Volume Dataset Segmentation Extract complex structures from volumes Using Level Set Models in Computer Graphics Involved in three level set modeling projects 3D Volumetric Metamorphosis Make one solid model become another Volume Dataset Segmentation Extract complex structures from volumes Surface Editing Cut, paste, blend, smooth arbitrary 3D objects 3D Volumetric Metamorphosis Level Set Morphing Movements Level Set Morphing Movements Each point on surface moves in the direction of local normal. Step-size proportional to signed distance to target γ B Each point on surface moves in the direction of local normal. Step-size proportional to signed distance to target γ B ( ) "# r X "t = $# ( X r r )% B X ( ) 5
Level Set Morphing Movements Level Set Morphing Movements Each point on surface moves in the direction of local normal. Step-size proportional to signed distance to target γ B ( ) "# r X = $# r X r ( )% B X ( ) "t Regions inside expand Regions outside contract Guaranteed convergence Each point on surface moves in the direction of local normal. Step-size proportional to signed distance to target γ B ( ) "# r X = $# r X r ( )% B X ( ) "t Regions inside expand Regions outside contract Guaranteed convergence Not moving points! Different Alignments Produce Different Morphs Morphing Challenge How to morph between these three models? Polygonal Mesh CSG Model MRI Scan Morphing Between Different Types of Models Combining A variety of scan conversion algorithms A flexible metamorphosis (morphing) technique based on level set models Produces A technique for morphing between different types of geometric models 6
LS Segmentation Framework Volume Dataset Segmentation Extract closed, smooth models from volumes LS Segmentation Framework LS Segmentation Framework Extract closed, smooth models from volumes Rough estimate Initialization Initial conditions Extract closed, smooth models from volumes Rough estimate + Fine tuning (sub-voxel) Initialization + Level Set Deformation Initial conditions + Evolve Level Set PDE Frog Embryo Example Interactively fit sphere/toroid to input data Frog Embryo Example Interactively fit sphere/toroid to input data Scan convert models Initialization 7
Frog Embryo Example Results - MRI Frog Embryo Interactively fit sphere/toroid to input data Scan convert models Initialization Level Set Deformation Attracted to maximum gradient magnitude Curvature-based smoothing Mouse Embryo Segmentation Ventricles Results - Diffusion Tensor Scalar fields derived from a diffusion tensor MRI scan (isotropic & anisotropic regions) Liver Results - Diffusion Tensor Results - Diffusion Tensor Brain Ventricles (Isotropic Diffusion) White Matter (Anisotropic Diffusion) 8
Same Dataset - Different Models Exploded View Different Dataset Same Parameters Segmentation from Multiple Non-uniform Datasets Input Several scans of the same object Different orientations Different, non-uniform resolutions Isosurface Initialization After Level Set Deformation Segmentation from Multiple Non-uniform Datasets Segmentation from Multiple Non-uniform Datasets Input Several scans of the same object Different orientations Different, non-uniform resolutions Unable to extract satisfactory model from any one dataset Input Several scans of the same object Different orientations Different, non-uniform resolutions Unable to extract satisfactory model from any one dataset Approach Drive level set segmentation with a moving least-squares polynomial approximation based on all datasets 9
Segmentation Results CT MRI Surface Editing Laser Scan Problem Statement Problem Statement? +? Level Set Surface Editing Level Set Surface Editing Perform surface editing operations within a level set framework to gain LS benefits Perform surface editing operations within a level set framework to gain LS benefits Guaranteed to produce closed, non-self-intersecting objects Easily handle change of genus No mesh-connectivity/triangle-quality issues Single framework for editing different types of complex models Unified interface to surface deformation 10
Level Set Surface Editing LS Surface Editing Pipeline Perform surface editing operations within a level set framework to gain LS benefits Guaranteed to produce closed, non-self-intersecting objects Easily handle change of genus No mesh-connectivity/triangle-quality issues Single framework for editing different types of complex models Unified interface to surface deformation Many objects are volumetric anyway! Input data Input data LS Surface Editing Pipeline LS Surface Editing Pipeline Loops back! Input data Conversions Input data Conversions Input data Conversions Operators LS Surface Editing Pipeline Operator Speed Function Building Blocks Loops back! Input data Conversions Operators Viewing 11
Operator Speed Function Building Blocks D q (d) Cut-off function based on a distance d to geometric object q Regionally constrains level set deformation and computation Operator Speed Function Building Blocks D q (d) Cut-off function based on a distance d to geometric object q Regionally constrains level set deformation and computation G(γ) Function of geometric measure γ e.g. curvature 2 Operator Speed Function Building Blocks D q (d) Cut-off function based on a distance d to geometric object q Regionally constrains level set deformation and computation G(γ) Function of geometric measure γ e.g. curvature 2 C(γ) Filters geometric measure γ Controls local geometric properties Automatic Blending Position Position red object relative to gray object Automatic Blending Automatic Blending Position Paste Paste objects together to make a single object (CSG union operation) Position Paste Find intersection curve (voxels close to both objects) 12
Automatic Blending Blending Speed Function Position Paste Blend Blending only occurs near intersection curve after a paste operation Blending Speed Function Blending Speed Function D q (d) Goes to zero at a user-specified distance from the intersection curve D q (d) Goes to zero at a user-specified distance from the intersection curve G(γ) Mean curvature Blending Speed Function Blending Speed Function D q (d) Goes to zero at a user-specified distance from the intersection curve G(γ) Mean curvature C(γ) Goes to zero as curvature approaches a user-specified value D q (d) Goes to zero at a user-specified distance from the intersection curve G(γ) Mean curvature C(γ) Goes to zero as curvature approaches a user-specified value Motion constrained to only move outwards 13
Spikes in eyes (reconstruction error) Localized Smoothing Spikes in eyes (reconstruction error) Localized Smoothing Position superellipsoids Spikes in eyes (reconstruction error) Localized Smoothing Smoothing Speed Function Position superellipsoids Region is smoothed Smoothing Speed Function Smoothing Speed Function D q (d) Goes to zero as X approaches surface of superellipsoid D q (d) Goes to zero as X approaches surface of superellipsoid G(γ) Mean curvature 14
Smoothing Speed Function Smoothing Speed Function D q (d) Goes to zero as X approaches surface of superellipsoid G(γ) Mean curvature C(γ) Goes to zero as curvature approaches a user-specified value D q (d) Goes to zero as X approaches surface of superellipsoid G(γ) Mean curvature C(γ) Goes to zero as curvature approaches a user-specified value Motion constrained to only move inwards Fixing a Teapot Model Fixing a Teapot Model Model errors Model errors Position superellipsoid Fixing a Teapot Model Smoothing Speed Function Model errors Position superellipsoid Region is smoothed 15
Smoothing Speed Function Smoothing Speed Function D q (d) Goes to zero as X approaches surface of superellipsoid D q (d) Goes to zero as X approaches surface of superellipsoid G(γ) Mean curvature Smoothing Speed Function Smoothing Speed Function D q (d) Goes to zero as X approaches surface of superellipsoid G(γ) Mean curvature C(γ) Goes to zero as curvature approaches a user-specified value D q (d) Goes to zero as X approaches surface of superellipsoid G(γ) Mean curvature C(γ) Goes to zero as curvature approaches a user-specified value Motion constrained to only move outwards Topology Simplification Embossing a Teapot Model Surface attracted to point set Convert image into 3D points 16
Point Embossing Embossing Speed Function r P i U Unit vector from X to attraction point P γ N U measure of angle F(X, N, φ, ) = D q (d)g(p)c(γ) r U i r X i v N i Embossing Speed Function Embossing Speed Function F(X, N, φ, ) = D q (d)g(p)c(γ) U Unit vector from X to attraction point P γ N U measure of angle between surface normal and direction to attraction point F(X, N, φ, ) = D q (d)g(p)c(γ) U Unit vector from X to attraction point P γ N U measure of angle between surface normal and direction to attraction point D q (d) Goes to zero as X approaches surface of superellipsoid Embossing Speed Function Embossing Speed Function F(X, N, φ, ) = D q (d)g(p)c(γ) U Unit vector from X to attraction point P γ N U measure of angle between surface normal and direction to attraction point D q (d) Goes to zero as X approaches surface of superellipsoid G(P) φ(p) (Deformation stops at attraction point) F(X, N, φ, ) = D q (d)g(p)c(γ) U Unit vector from X to attraction point P γ N U measure of angle between surface normal and direction to attraction point D q (d) Goes to zero as X approaches surface of superellipsoid G(P) φ(p) (Deformation stops at attraction point) C(γ) Non-zero for positive values of γ Motion constrained to only move in one direction 17
Point Set Embossing Global Smoothing with a Morphological Opening Opening Erosion followed by a Dilation Global Smoothing with a Morphological Opening Opening Erosion followed by a Dilation Erosion Offset surface inwards by σ Dilation Offset surface outwards by σ Global Smoothing with a Morphological Opening Opening Erosion followed by a Dilation Erosion Offset surface inwards by σ Dilation Offset surface outwards by σ Quickly calculate distance field from level set model with Sethian s Fast Marching Method Faster than solving LS equation over complete surface Global Smoothing with a Morphological Opening Global Smoothing with a Morphological Opening Laser scan reconstruction with spiky errors Offset surface inwards 18
Global Smoothing with a Morphological Opening How to Repair a Greek Bust? Offset surface outwards to create smoothed surface Copy Right Chin Reflect and Position on Left Paste onto Face Local Smoothing 19
Copy a Nose Paste on New Nose Local Smoothing Different Hair Style? Sharpen Hair How to make a doubleheaded, winged dragon? 20
Load in Dragon Model Enclose the Head Cut Off the Head Make Two Copies of Head Paste and Blend Heads Together Load in Body 21
Paste and Blend Together Heads and Body Enclose and Copy the Wing Load in Double-headed Dragon Paste and Blend the Wing Position a Mirror-copy of Wing Load in a Supertoroid 22
Paste Supertoroid to Body Blend Along Intersection Curve Interactive Cut, Paste & Blend Two-Headed Winged Dragon Charm Closed Surface Ideal for Automated Manufacturing Editing Operators Summary Model produced by Utah s Geometric Design and Computation Group Design LS surface editing framework Single representation and numerous tools Definition of new LS surface editing operators Automatic blending along intersection curves Spatially constrained smoothing and embossing User control over local surface properties User control of surface movement direction 23
Level Set Model Summary Acknowledgments Level set models may be utilized in a wide variety of computer graphics applications 3D metamorphosis Volume segmentation Surface editing Have some shortcomings, being addressed Offer unique and powerful capabilities and outstanding benefits Thanks to Al Barr, Sean Mauch, Leonid Zhukov, Jason Wood (Leeds), Jia Mao, Neha Soni, Erica Stanley, David Johnson (Utah), Mathieu Desbrun (USC), Khrysaundt Koenig, and Katrine Museth Supported by NSF, NIH, ONR, NLM, Caltech Datasets provided by CIT Biological Imaging Center & Multires Modeling Group, Utah s SCI Institute & Geometric Design and Computation Group, Stanford CG Lab, U. of Edinburgh, Microsoft, Cyberware, City of Hope Hospital Related Papers D.E. Breen and R.T. Whitaker, A Level-Set Approach for the Metamorphosis of Solid Models, IEEE Transactions on Visualization and Computer Graphics, Vol. 7, No. 2, pp. 173-192, April-June 2001 D.E. Breen, S. Mauch, R.T. Whitaker and J. Mao, 3D Metamorphosis Between Different Types of Geometric Models, Proc. Eurographics, pp. 36-48, 2001 R.T. Whitaker, D.E. Breen, K. Museth and N. Soni, Segmentation of Biological Volume Datasets Using a Level-Set Framework, Volume Graphics 2001, Springer, pp. 249-263, 2001 K. Museth, D. Breen, L. Zhukov and R. Whitaker, ``Level Set Segmentation From Multiple Nonuniform Volume Datasets,'' Proc. IEEE Visualization, pp. 179-186, 2002 L. Zhukov, K. Museth, D.E. Breen, R.T. Whitaker, A.H. Barr, Level Set Segmentation and Modeling of DT-MRI Human Brain Data, Journal of Electronic Imaging, Vol. 12, No. 1, pp. 125-133, January 2003 K. Museth, D.E. Breen, R.T. Whitaker and A.H. Barr, Level Set Surface Editing Operators, Proc. SIGGRAPH, pp. 330-338, 2002 D.E. Breen, S. Mauch and R.T. Whitaker, 3D Scan Conversion of CSG Models into Distance, Closest-Point and Colour Volumes, Volume Graphics, Springer, pp. 135-158, 2000 24