Kernel Density Estimation

Transcription

1 Kernel Density Estimation An Introduction Justus H. Piater, Université de Liège

2 Overview 1. Densities and their Estimation 2. Basic Estimators for Univariate KDE 3. Remarks 4. Methods for Particular Domains 5. Case Study: Pose Estimation 6. Conclusion 2 / 29

3 Densities and their Estimation

4 Probability Density Functions We can assign probabilities to subsets of a continuous sample space by integrating probability density functions (PDFs) over them: A PDF is nonnegative, but can exceed unity, and. Densities and their Estimation 4 / 29

5 Parametric Estimation of Densities If we have a parametric, generative model of the PDF, we can estimate its parameters from samples. Example 1. Univariate Normal (Gaussian) Distribution Densities and their Estimation 5 / 29

6 Parametric Estimation of Densities Example 2. Mixture of Univariate Normal Distributions with mixture proportions the prior probabilities that a given observation was produced by mixture component, and : Expectation-Maximization, gradient descent over the parameters, Densities and their Estimation 6 / 29

7 Nonparametric Estimation of Densities Without a generative model, all we can use to estimate is our available sample. Uses: Faithfully model arbitrary distributions from finite samples Intuitive presentation and exploration Identify multimodality Resample Densities and their Estimation 7 / 29

8 Histogram Bin the data into intervals, for : of width Issues: Sensitivity to,, and (for multivariate data) the coordinate directions Step function Generalized histogram: Densities and their Estimation 8 / 29

9 Naive Estimator From the definition of a PDF: Thus, we can further generalize our histogram: How many observations fall into a fixed-with box centered at? Issue: Step function Densities and their Estimation 9 / 29

10 Basic Estimators for Univariate KDE

11 Kernel Estimator Replace the weight function by a kernel function with estimator,. Then, by analogy with the naive with bandwidth or smoothing parameter. Properties: If is nonnegative, then it is a PDF, and so is. inherits all continuity and differentiability properties from. The fixed bandwidth may oversmooth in some regions and undersmooth in others. Basic Estimators for Univariate KDE 11 / 29

12 Nearest Neighbor Estimator Complementary to the naive estimator, what is the smallest box containing a given number of observations? Define. Then, the th nearest neighbor density estimate is defined by Properties: Bandwidth depends on the density of observations around the query value. Derivative full of discontinuities Basic Estimators for Univariate KDE 12 / 29

13 Nearest Neighbor Estimator (Continued) This can be generalized by introducing a kernel function: Basic Estimators for Univariate KDE 13 / 29

14 Variable Kernel Estimator The bandwidth depends on the densities around the observations. Properties: Same as for the basic kernel density estimator. Basic Estimators for Univariate KDE 14 / 29

15 Remarks

16 Estimators for Multivariate Densities All discussed methods extend straightforwardly. Remarks 16 / 29

17 Sampling from a Kernel Density Estimate Basically: 1. Choose an observation at random. 2. Sample from the kernel centered at that observation. In general, methods exist for sampling from arbitrary PDFs (inverse transform sampling, rejection sampling, ). For many specific kernel functions, more efficient, specialized methods exist (normal distributions: Box- Muller transform, ). Remarks 17 / 29

18 Methods for Particular Domains

19 Bounded Domains Outside of the boundaries, set to zero: Observations near the boundaries contribute less to, and is underestimated near the boundaries. Take logs: e.g., for positive data, if is the density estimated from the logarithm of the data, then There will be a spike at the singularity. Methods for Particular Domains 19 / 29

20 Bounded Domains (Continued) Reflect the data: e.g., for positive data, if is the density estimated from this augmented data set, then Methods for Particular Domains 20 / 29

21 Circular Domains For example, replicate angular data twice, shifted by (or even more copies, if necessary). Then, apply a conventional estimator. A similar trick applies to toroidal domains. Methods for Particular Domains 21 / 29

22 Spherical Domains For example, represent data as unit vectors use a von Mises Fisher kernel:, and where is a concentration parameter. [Mardia and Jupp 1999] The normalizing constant is messy to compute explicitly and can in practice be determined by numerical integration. For axial domains, use a Dimroth-Watson kernel (inner product squared): : antipodal : uniform : girdle Methods for Particular Domains 22 / 29

23 Case Study: Pose Estimation Renaud Detry [Detry et al. 2009]

24 Local Appearance + Spatial Relations X 6 traf#c sign # 4,6 # 5,6 brid ge triangul ar X 4 X pattern 5 fram e # 1,4 # 2,4 # 3,5 X 1 X 2 X 3 # 1 # 2 # 3 Y 1 Y 2 Y 3 Vertex (random variable): distribution of positions Edge (potential): distribution of spatial relations Object Model (Markov network): structure; potentials, Case Study: Pose Estimation 24 / 29

25 A Kernel for Pose Densities Pose Assuming that positions and orientations are independent, use an kernel that factors into a location kernel (normal) an orientation kernel (Dimroth-Watson), with denoting quaternion representations. Case Study: Pose Estimation 25 / 29

26 Uses of KDE Sampling: from weighted-particle representations for forming message products for stochastic integration of products of densities (convolutions) Finding maximum: for determining the maximum-likelihood pose to find the best gripper pose for grasping Case Study: Pose Estimation 26 / 29

27 Uses of KDE (Continued) To sample in : 1. Choose an observation at random. 2. Sample from the position part of the kernel centered at that observation. 3. Sample from the orientation part of the kernel centered at that observation [Wood 1994]. Case Study: Pose Estimation 27 / 29

28 Conclusion

29 References J. Copas, M. Fryer, Density estimation and suicide risks in psychiatric treatment. Journal of the Royal Statistical Society (Series A) 143, pp , R. Detry, N. Pugeault, J. Piater, A Probabilistic Framework for 3D Visual Object Representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(10), pp , K. Mardia, P. Jupp, Directional Statistics, Wiley B. Silverman, Density Estimation for Statistics and Data Analysis, Chapman & Hall/CRC A. Wood, Simulation of the von Mises Fisher distribution. Commun. Statist. Simula. 23(1), pp , Conclusion 29 / 29