Density Estimation

Proseminar Data Mining

Tim Pfeifle
Fakultät für Informatik
Technische Universität München
Email: [email protected]

Abstract— Due to the rising importance of Big Data in many

different industries, from automotive to retail, the amount of
collected data is growing exponentially. Although our computa-
tional abilities to process this data are increasing at a similiar
rate, we often face the difficulty of comprehending the general
structure of the data. With these enormous amounts of data
has come an increasing demand for tools that can grasp and
summarize the structure in this data. The density of the data gives
a good overview of the data and its graphical representation is
fairly easily comprehensible even for non-mathematicans. In this
paper we’d like to give a short introduction as to what density
estimation is and into some common estimators.
Index Terms— Density Estimation, Histograms, Naive Estima- Fig. 1. Kernel density estimates constructed from observations of the height
tor, Kernel Density Estimator, Nearest Neighbour Estimator of a steel surface (skewed density) [3]

I. I NTRODUCTION
A. What is (probability) density estimation?
To understand what density estimation is we should first
recapitulate what a probability density function (pdf) is: Given
a random variable X we can specify the probability density as
a function f whose values are the relative likelihoods, that the
value of the random variable X would equal a given random
sample. So if we’d like to know the probability, that a sample
falls into an interval from a to b we would calculate the area
under the graph of the density function f given by the formula
1. Z b Fig. 2. Density estimate constructed from the observations of the direction
of turtle data. [4]
P (a < X < b) = f (x)dx (1)
a
This function is continuous, nonnegative and the integral
each of the turtles was observed to swim when released [2].
over X integrates to one. With density estimation we try to
The graph has two maxima, one global maxima at 60 and one
estimate this unknown probability density function [1] from
local at about 260 which means that most of the turtles swam
the observed data points X1 , ..., Xn . We call this estimated
into the 60 direction and a small group preferred the opposite
function in the following fˆ.
direction. This multimodality is easily comprehensible from
B. Why do we need (probability) density estimation? the density estimate.
One of the most common uses of pdfs is in the basic While we saw that the plotted density estimate of a given
investigation of the properties of observed data, like skewness dataset is a good starting point for an initial analysis it’s
and multimodality. Figure 1 shows the probability of different also very useful for presenting the results to a client. For
heights of a steel surface from a collection of observations. this application density estimates often are a good fit because
We see that the highest density is at around 35µm and that of their comprehensibility to non-mathematicians. Apart from
the probability that the height of the steel surface is in the the graphical output, density estimates are also often used as
range from 20µm to 40 µm is quite high. If we calculate the intermediate products for other algorithms and applications
integral over this range we get the exact probability. We can like for example classification [5] or discriminant analysis [6].
also see that the distribution of the height is skewed, so that
the tail on the left side is longer than the tail on the right side. C. Intuitive approaches to pdf
In the example of Figure 2 we see the density estimate of So how do we get an estimate of the unknown density of
a turtle dataset. This dataset contains the directions in which our samples X1 to Xn ? Well lets start with the simplest way
 50 0.1

8 · 10−2
40

Density estimate
Age [years]

6 · 10−2
30
4 · 10−2

20
2 · 10−2

0 100 200 300 400 500 0

0 10 20 30 40 50 60 70
xi Age [years]
Fig. 3. Scatterplot: each person xi of our ”Afghanistan Population” [7]
Fig. 4. Population in Afghanistan with the small bin-width of 2 years
dataset represents one point

contains a comparison of the presented methods and a prospect

to present data: the table. It’s very easy to compare single
of density estimators.
entries of data. But with increasing numbers of features and
datasets it gets increasingly complicated to summarize and II. H ISTOGRAM
detect the underlying structure of the data. One way to deal So a more advanced form of a density estimator is the
with this problem is to use traditional statistic metrics like for histogram. A histogram is simply a function that counts the
example the mean, also known as the average. The mean has number of observations that fall into each of the disjoint
the disadvantage of being affected by outliers, entries being categories, often referred as bins. This gives us the visual
too high or too low compared to the rest of the data. [3] So information of the relative frequency of our observation, which
if we only look at the average, outliers in our input data, like is the essence of a density function. [3] In our population
measurement errors, can obfuscate our average extremely. example these bins could be the age in steps of 5 (0 to 5, 5 to
So another statistic metric which is also often used, is the 10, ...). Generally the intervals are defined as [xo + mh, x0 +
median, which is defined as the value separating the lower (m + 1)/h) with an origin x0 and the bin width h. Given n
half of the data from the higher half of the data. To give observations, we can count the samples falling in each of these
some examples, we’ll use a dataset of the Demographic and intervals/bins and end up with function (2).
Health Surveys in Afghanistan [7] from 2015 throughout this
1 [# observations in same bin as x]
paper. We’ll focus on the feature ”age” of the samples. With fˆ(x) = ∗ (2)
this example we get a mean of 31 years and a median of 30 n [width of bin]
years. While this gives a broader picture of the data, than by Using this function on the dataset of the population of
skimming through the table row by row, we’re still missing out Afghanistan and a bin-width of 10 years we get the density
on a lot of the information of the data. We can still only make estimate of Fig. 6.
wild guesses about the density of the dataset. So for example The histogram always requires the two parameters bin-
whether there are more 15-20 year olds or 20-25 year olds. width and the starting position of the first bin. Somewhat
So a good idea might be to plot a graph with each sample of surprisingly the starting position has an impact on the graph
the population as a point. If we plot these bivariate datapoints of the histogram, but often is simply set to 0. If we plot
with their age as their Y-value and their index as their X-Value the population example from fig. 6 again with bin-width 10,
we get the scatterplot of Figure I-C. but set the starting position x0 to 5 instead of 0 we get a
This first graphical representation of the data gives a first considerably different histogram. If we have a small bias the
impression about the density of the data, but it is still very choice of the starting position has no big impact. But in this
difficult to estimate what percentage of people are 15 to 20 example because of our large bin-width we have a lot of bias
years old. As we’ve seen the table, the mean and median and and therefore the starting position has a considerable impact.
the scatterplot, weren’t a good way to estimate the density of For sufficiently large sample sizes this isn’t a problem and can
the data. So in the next chapter we’ll introduce the probably be neglected, but for small sample sizes it can be significant.
most used density estimator, the Histogram. While it’s a useful The bin-width is also called the smoothing parameter. In
first approach for a density estimator, we’re going to show why Figure 6 a large value for the bin-width results in a very
it isn’t always a good fit and therefore talk in Chapter 3 about smooth output function, while the output with a small value for
the Main nonparametric density estimators. The last chapter the bin-width has many more bumps which don’t get smoothed
 if derivatives of the estimate are required. This is often the case
0.4 when the estimate is not the result, but merely an intermediate
component.
The real density function is continuous, which means that
Density estimate

0.3 our histogram estimator must differ quite a lot from the
true density function. This difference between the expected
value of our estimator and the true density function of the
0.2 data is called Bias. In the case of the histogram one easy
way to decrease our bias is to increase our number of bins.
But if maximize our number of bins we end up with a
0.1 scatterplot as in figure I-C. As we discussed in the Introduction
a scatterplot is difficult to read, because of its enourmous
variance. Therefore we are forced into a difficult tradeoff:
0 If we decrease bias, we increase variance and vice versa.
5 15 25 35 45 55 65
This tradeoff is characteristic for the density estimation. An-
Age [years] other problem is that the values of adjacent bins can vary
extremely simply if there are fluctuations in the sample. If
Fig. 5. Starting position 5, ”Population in Afghanistan” 2015 with the large
bin-width of 10 years we increase the amount of bins we can fix this problem,
but then we create the problem of oversmoothing our density
and thereby loosing details in our distribution. One way to
solve this are adaptive methods which we’ll introduce with
0.4 the nearest neighbour method.

A. Multivariate Histograms
Density estimate

0.3 Apart from the discussed univariate (depending on one

variable) histograms we’re often require multivariate density
estimates. Multivariate visual representations are needed to
0.2 visualize the correlation of multiple different variables. For
example in our Dataset of the Afghan population if we’re
interested not only in the age, but also in the weight of the
0.1
population. Since all multivariate methods are generalizations
of the univariate case we can apply most of our knowledge
0 from before. In the multivariate case with samples x ∈ Rd of
0 10 20 30 40 50 60 70 dimension d, the bins are rectangles of size h1 ∗ h2 ∗ ... ∗ hd .
Age [years] [Figure II-A] We use rectangles and not cubes, because we
don’t want to make the assumption that the provided data
Fig. 6. Population in Afghanistan 2015 with the large bin-width of 10 years is properly scaled or normalized. The number of bins grows
exponentially with the number of dimensions. So in higher
dimensions we need a very large sample-set or else most of our
out [Figure 4]. So how should we choose the smoothing bins would be empty, as we already see in the 2-dimensional
parameter? It’s quite clear that if we have only few samples case of figure II-A. This problem is also called the curse
and base our density estimate simply on the samples, without of dimensionality. [8] Since many important applications of
smoothing this input data, the resulting function will likely density estimation are with multivariate samples this is a huge
have many more bumps than the real underlying density. But drawback of histograms.
if we smooth to much we might loose bumps that in fact are
present in the underlying density. This problem of choosing III. M AIN NONPARAMETRIC APPROACHES
the perfect smoothing parameter got quite extensive coverage In this chapter we’ll talk about alternatives to the histogram
from B.W. Silverman [1], D.W. Scott [3] and many other. In and we’ll focus on the nonparametric approaches. But what
most cases Sturges’ number-of-bins rule k = 1 + log2 n gives exactly is the difference between parametric and nonparamet-
good results. [3]. With the number-of-bins we can calculate ric approaches?
the smoothing parameter/bin-width h by dividing the sample 1) Parametric Density Estimation: The approach of para-
range into k-bins. metric density estimation assumes, that the data is from a
Apart from selecting good parameters h and x0 the his- known parametric family of distributions, like for example the
togram has one major drawback: Because of its rectangular normal distribution with parameters mean µ and variance σ 2 .
nature [Figure 6] it has discontinuities, regardless of the Using this approach we’d estimate these two parameters and
underlying data. These discontinuities cause extreme difficulty then substitute these estimates into the formula for the normal
 and thereby end up with a mathematical expression for our
prosa equation 4.
n
1 X x − Xi
fˆ(x) = ∗ w( ) (5)
nh i=1 h
(
1
if |x| < 1
w(x) = 2 (6)
0 otherwise
One big problem of the naive estimator is its discontinuity
at the points Xi h.
B. Kernel Density Estimator
Similar to the naive estimator the kernel density estimate
centers a function at each data point and then summs them
to get a density estimate. The Kernel Density Estimate can be
considered as a sum of ’bumps’ placed at the observations. The
kernel function K determines the shape of the bumps while the
Fig. 7. Multivariate histogram with x ∈ R2 [3] window width h determines their width. [1]. Figure 8 shows
this with a small sample size of seven.
As h tends to zero, we get a similar output to a scatterplot
density. [1] This works good, if the unknown density really is a and as h becomes large all detail, spurious or otherwise,
normal distribution, but otherwise get an inaccurate estimate. If is obscured. [1], similar to the smoothing parameter of the
for example our underlying density is skewed as in 1 using the histogram.
parametric family of normal distributions would discard this
skewness and we’d loose this information. With parametric
density estimation we could try many different parametric
families of distributions untill we find a good match. In this
paper we’ll only focus on the non-parametric methods, by
making less rigid assumptions about the underlying density of
the observed data. We’ll allow the data to ”speak for itself”,
more than if we would assume that the density of f was a
member of a given parametric family. [1]
A. Naive Estimator
Fig. 8. Figure depicting how the individual kernels get summed, Window
Many complex estimators are based on the naive estimator, width 0.4
so lets start our journey here. If the variable X has the density
f than following the definition of the density we can write it The kernels depicted as ”bumps” in Figure 8 are a positive
as in 3. P (...) means the probability that x falls into this bin. function, integration to unity (7).
Z
1 K(x) > 0 and K(x)dx = 1 (7)
f (x) = lim ∗ P (x − h < X < x + h) (3)
2h
h→0
As there are many different functions satisfying this condi-
We can estimate P (x − h < X < x + h) simply by the
tion, there was quite some research done, to find an ”optimal
proportion of our sample falling into the interval (x−h, x+h).
kernel”. [3] Some popular choices of kernels are depicted in
If we do this and chose a small value for h we get the naive
figure 9.
estimator fˆ as in 4
While an optimal kernel brings only moderate improve-
ments and because the kernel density estimate takes over all
1
fˆ(x) = ∗ [#of X1 , ..., Xn falling in (x − h, x + h)] properties of its kernel, a popular choice is to choose a smooth,
2hn clearly unimodal and symmetric about the origin kernel. [1]
(4)
We thereby center our bins on every sample, in contrast So a popular choice for the kernel function, which we’ll also
to the previous histogram where our bins are independent of use in the following, is the gaussian function N (µ, σ 2 ).
our data. This means that our values x are chosen to be only While Figure 8 shows the graphical representation of the
locations of our samples. Formalizing the equation 9 we get kernel estimator the mathematical definition can be expressed
the equation 5 where w is a weight function defined by 6. We as in (8).
n
sum over all samples looking for samples falling into the bin 1 X x − xi
fˆ(x) = ∗ K( ) (8)
centered at the sample x and weighting it with the factor 21 nh i=1 h
 Fig. 9. Left: Gaussian Kernel, Right: Epanechnikov Kernel

where K is the kernel and h is the bandwidth. From the

definition of the kernel estimator we see that if our kernel is
non-negative and integrates to one that fˆ will be a probability
density. It will also inherit all the continuity and differentiabil-
ity properties of the kernel. So if we use the gaussian kernel,
fˆ will be smooth and will also have derivatives of all orders
[1]. From the definition of the kernel estimator we see that
the KDE is a naive estimator with a special weight function,
now called the kernel. Similar to the naive estimator h is the Fig. 10. Kernel estimates, Top: Window width 20 with spurious noise in the
smoothing parameter. tails, Bottom: Window width of 60 with smoothed tails but lost details in the
main part [1]
But how should we choose the smoothing/bandwidth pa-
rameter? The bandwidth is often set, by minimizing the mean
integrated squared error (MISE). like the previous estimators. The distance d(x, y) between two
1) Measures of discrepancy fˆ to f: To measure the dif- samples is defined as usual as |x − y|. To get the distance to
ference between our estimate fˆ and the density f we could the k-th nearest sample, we sort for each sample the distances
calculate the difference at each point, also known as the mean to the other samples ascending. So the distance dk (t) is the
error. But to always get a positive error-value, even if the distance from the sample t to the k-th nearest sample. With
difference is negative and to affirm the difference we instead this introduction we can define the nearest neighbor estimate
use the following measures: as in 9
• at a single point x: M SEx (fˆ) = E{fˆ(x) − f (x)} k
2
fˆ(t) = (9)
ˆ ˆ ˆ 2 2 ∗ n ∗ dk (t)
R
• global of f : M ISE(f ) = E {f (x) − f (x)} dx

The biggest challenge for Kernel Estimators are varying data Because the distance dk (t) in the tails is larger than in the
densities. With the fixed window-width of the kernel estimator dense part of the distribution, the estimate is there smaller and
we often have spurious noise in the tails of the estimate like thereby smooths adaptive to the underlying density.
in Fig. 10a. If we smooth out the noise in the tails, we also While we see from the definition 9 that fˆ is continuous
smooth out the detail in the main part of the distribution as we its derivate at the positions of dk is discontinuous. A major
see in Fig. 10b. But often we’d like to preserve the variance difference to the KDE is that the nearest neighbor estimate
in regions with high density and smooth the data in regions doesn’t integrate to one, because the tails of fˆ decrease very
with a very low density. slowly. So if we’re only interested in a smaller part of the data
There are different adaptive methods to deal with this the NNE is fine, but if we’re interested in the entire dataset
problem. One way method is to use small bandwidths in we’d be better of with a different estimator.
regions with high density and large bandwidths in regions with
low density. [3]. This approach is also known as the adaptive IV. C ONCLUSION AND PROSPECT
kernel estimator. In the next chapter we’ll talk about a different Nonparametric density estimation has the huge advantage
approach to deal with this smoothing problem, called the (kth) that we can make less rigid assumptions about the underlying
nearest neighbor estimator. data. But we still have to choose a density estimator. The
Nevertheless the kernel density estimator is apart from the histogram is the most common pick for visualising results
histogram probably the most commonly used estimator and for a client or getting a first impression of the data. If a
certainly the most studied mathematically [1]. more advanced estimator is needed, for example if we need
derivatives of our estimate the kernel density estimator is often
C. Nearest Neighbor Estimator (NNE) a good fit.
The nearest neighbor estimator is proportional to the dis- The k-th nearest neighbour estimator is not that common
tance to the k-th nearest sample, instead of being based on for density estimation, but in cases with spurious noise in the
the number of samples falling into a bin with fixed width tails, where the KDE has problems finding a good smoothing
parameter, it still can be a good fit. The idea of the k-th nearest
neighbour on the other hand is very common for classification
problems. [2] The MISE and the bias and variance tradeoff
are also very useful in other disciplines like for example in
signal processing.
R EFERENCES
[1] B. Silverman, Density Estimation for Statistics and Data
Analysis, ser. Chapman & Hall/CRC Monographs on Statistics &
Applied Probability. Taylor & Francis, 1986. [Online]. Available:
https://books.google.de/books?id=e-xsrjsL7WkC
[2] E. Fix and J. L. Hodges Jr, “Discriminatory analysis-nonparametric
discrimination: consistency properties,” DTIC Document, Tech. Rep.,
1951.
[3] D. Scott, Multivariate Density Estimation: Theory, Practice, and
Visualization, ser. Wiley Series in Probability and Statistics. Wiley, 2009.
[Online]. Available: https://books.google.de/books?id=wdc8Xme FfkC
[4] B. Silverman, “Choosing the window width when estimating a density,”
Biometrika, vol. 65, no. 1, pp. 1–11, 1978, cited By 76.
[5] M. Kobos and J. Mandziuk, “Classification based on combination of
kernel density estimators,” Artificial Neural Networks - ICANN 2009, pp.
125–134, 2009.
[6] L. W. Xipeng Qiu, “Nearest neighbor discriminant analysis,” International
Journal of Pattern Recognition and Artificial Intelligence, 2006.
[7] “Afghanistan: Standard demographic and health surveys, 2015,” 2015.
[8] B. R., “Adaptive control processes,” Princeton University Press, Prince-
ton, NJ, 1961.