Scribd
Upload
18 views

ZhuoLiu SVclustering

Support vector clustering is an unsupervised machine learning technique that maps data points into a Hilbert space and finds a minimal enclosing sphere to determine clusters. It addresses limitations of K-means by allowing clusters of arbitrary shapes and densities. The algorithm uses a Gaussian kernel to map points into a higher dimensional space where a minimal enclosing sphere is found. Points inside the sphere are clustered together, while those outside are outliers. The technique was shown to outperform other clustering methods on benchmark datasets like iris data.

Uploaded by

josephashwinkurian
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
18 views

ZhuoLiu SVclustering

Support vector clustering is an unsupervised machine learning technique that maps data points into a Hilbert space and finds a minimal enclosing sphere to determine clusters. It addresses limitations of K-means by allowing clusters of arbitrary shapes and densities. The algorithm uses a Gaussian kernel to map points into a higher dimensional space where a minimal enclosing sphere is found. Points inside the sphere are clustered together, while those outside are outliers. The technique was shown to outperform other clustering methods on benchmark datasets like iris data.

Uploaded by

josephashwinkurian
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 28

Support Vector

Clustering
A SA BE N- H U R , D AVI D H O R N, H AVA T. SI E GE L M A N N,
VL A D I M I R VA P NI K

Zhuo Liu
Clustering
• Grouping a set of objects which are similar
• Similarity: distance, density, statistical distribution
• Unsupervised learning
Limitation of K-means: Differing Density

Original Data K-means (3 Clusters)


Limitation of K-means: Non-globular Shapes

Original Data K-means (2 Clusters)


Support Vector Clustering
• Data points are mapped by Gaussian kernel (NOT polynomial kernel
or linear kernel) to a Hilbert space
• Find minimal enclosing sphere in Hilbert space
• Map back the sphere back to data space, cluster forms
• Procedure to find this sphere is called the support vector domain
description (SVDD)
• SVDD is mainly used for outlier detection or novelty detection
• SVC is a unsupervised learning method
Support Vector Domain Description (SVDD)
• is a data set of N points
• Φ is a nonlinear transformation from to a Hilbert space
• Task:
minimize , with constraint
Support Vector Domain Description (SVDD)
• Lagrangian:

where , are Lagrange multipliers, is a constant, is the penalty term.


Support Vector Domain Description (SVDD)
Take partial derivatives and set them to be zeroes:

And KKT complementarity conditions of Fletcher (1987) result in:


Support Vector Domain Description (SVDD)
• If , then , then , then , so point lies outside the sphere, it is called a
bounded support vector or BSV.
• If and , then , it is inside the sphere.
• If and , then , it lies on the surface of the sphere. Such a point will be
referred to as a support vector or SV.
• Note that when no BSVs exist.
Support Vector

Bounded Support
Vector

Inner Point
Support Vector Domain Description (SVDD)
• Wolfe dual form:

with constraints:
• Now, we can introduce kernel function such that

• How does different kernel work?


Polynomial Kernel
𝑑
𝐾 ( 𝑥 𝑖 , 𝑥 𝑗 ) =(𝑥 𝑖 .𝑥 𝑗 +1)
Gaussian Kernel
2
𝐾 ( 𝑥𝑖 ,𝑥 𝑗 )=𝑒𝑥𝑝(−(𝑥 𝑖−𝑥 𝑗 )¿¿2/𝑠 )¿
Cluster Assignment
• Generating adjacency matrix
• has component with value either 0 or 1
• 0: line segment between and cross out the sphere
1: line segment between and is always in the sphere
• Clustering based on graph-based model
[ ] [ ]
1 1 1 0 0 0 2 −1 −1 0 0 0 Second Smallest Eigenvalue
1 1 1 0 0 0 −1 2 −1 0 0 0 for Laplacian:
1 1 1 0 0 0 −1 −1 2 0 0 0
𝐴= 𝐿𝑎𝑝𝑙𝑎𝑐𝑖𝑎𝑛=
0 0 0 1 1 1 0 0 0 2 −1 −1 So there are two clusters.
0 0 0 1 1 1 0 0 0 −1 2 −1
0 0 0 1 1 1 0 0 0 −1 −1 2
Example

2
𝐾 ( 𝑥𝑖 ,𝑥 𝑗 )=exp(−𝑞‖𝑥𝑖 −𝑥 𝑗‖ )
Example with BSVs
• In real data, clusters are usually not as well separated as in previous
example, so we need to allow some BSVs.
• BSVs are assigned to the cluster that they are closest to.
• An important parameter - upper bound on the fraction of BSVs:

where is number of points, is the coefficient for penalty term.


• Asymptotically (for large ), the fraction of outliers tends to .
Example with BSVs
Clusters with Overlapping Density Functions
Experiment on Iris Data
• There are three types of flowers, represented by 50 instances each
• First two principal components space:
1. q = 6 p = 0.6
2. the third cluster split into two
3. When these two clusters are considered together, the result is 2 misclassifications
• First three principal component space:
1. q = 7.0 p = 0.70
2. four misclassifications
• First four principal component space:
1. q = 9.0 p = 0.75
2. 14 misclassifications
• # of SVs: 18 in 2D, 23 in 3D, 34 in 4D
• Reason for improvement in 2d and 3d: PCA reduces noise
Experiment on Iris Data
Compare with Other Non-Parametric
Clustering Algorithms
• The information theoretic approach of Tishby and Slonim (2001) : 5
misclassifications.
• The SPC algorithm of Blatt et al. (1997), when applied to the dataset
in the original data-space: 15 misclassifications.
• SVC: 2 misclassification in first two PCs space, 4 misclassification in
first three PCs space.
Principle to Choose Parameter
• Starting from a small value of q and increasing it. Initial value can be chosen
as:

which will result in a single cluster, so no outliers are needed, hence choose .
• Criteria : a low number of SVs guarantees smooth boundaries.
• If the number of SVs is excessive, or a number of singleton clusters form, one
should increase to allow SVs to turn into BSVs, and smooth cluster
boundaries emerge.
• In other words, we need to systematically increase q and p along a direction
that guarantees a minimal number of SVs.
Complexity
• SMO algorithm of Platt (1999) to solve the quadratic programming
problem – very efficient
• Labeling part:
• If # of SVs is O(1), labeling part:
• Memory usage: O(1).
• In overall, SVC is useful even for very large datasets
Conclusion
• SVC has no explicit bias of either the number, or the shape of clusters
• SVC is a unsupervised clustering algorithm
• Two parameters:
q: when it increases, clusters begin to split
p: soft margin constant that controls the number of outliers
• A unique advantage: cluster boundaries can be of arbitrary shape,
whereas other algorithms are most often limited to hyper-ellipsoids
References
A. Ben-Hur, A. Elisseeff, and I. Guyon. A stability based method for discovering structure in clustered data. in Pacific Symposium on
Biocomputing, 2002.

A. Ben-Hur, D. Horn, H.T. Siegelmann, and V. Vapnik. A support vector clustering method. in International Conference on Pattern
Recognition, 2000.

A. Ben-Hur, D. Horn, H.T. Siegelmann, and V. Vapnik. A support vector clustering method. in Advances in Neural Information
Processing Systems 13: Proceedings of the 2000 Conference, Todd K. Leen, Thomas G. Dietterich and Volker Tresp eds., 2001.

C.L. Blake and C.J. Merz. Uci repository of machine learning databases, 1998.

Marcelo Blatt, Shai Wiseman, and Eytan Domany. Data clustering using a model granular magnet. Neural Computation, 9(8):1805–
1842, 1997.

R.O. Duda, P.E. Hart, and D.G. Stork. Pattern Classification. John Wiley & Sons, New York, 2001. R.A. Fisher. The use of multiple
measurments in taxonomic problems. Annals of Eugenics, 7:179–188, 1936.

R. Fletcher. Practical Methods of Optimization. Wiley-Interscience, Chichester, 1987.

K. Fukunaga. Introduction to Statistical Pattern Recognition. Academic Press, San Diego, CA, 1990.

A.K. Jain and R.C. Dubes. Algorithms for clustering data. Prentice Hall, Englewood Cliffs, NJ, 1988.

H. Lipson and H.T. Siegelmann. Clustering irregular shapes using high-order neurons. Neural Computation, 12:2331–2353,
2000.
References
J. MacQueen. Some methods for classification and analysis of multivariate observations. In Proc. 5th Berkeley
Symposium on Mathematical Statistics and Probability, Vol. 1, 1965.
G.W. Milligan and M.C. Cooper. An examination of procedures for determining the number of clusters in a data set.
Psychometrika, 50:159–179, 1985.
J. Platt. Fast training of support vector machines using sequential minimal optimization. In Advances in Kernel
Methods — Support Vector Learning, B. Sch¨olkopf, C. J. C. Burges, and A. J. Smola, editors, 1999.
B.D. Ripley. Pattern recognition and neural networks. Cambridge University Press, Cambridge, 1996.
S.J. Roberts. Non-parametric unsupervised cluster analysis. Pattern Recognition, 30(2): 261–272, 1997.
B. Sch¨olkopf, R.C. Williamson, A.J. Smola, J. Shawe-Taylor, and J. Platt. Support vector method for novelty
detection. in Advances in Neural Information Processing Systems 12: Proceedings of the 1999 Conference, Sara A.
Solla, Todd K. Leen and Klaus-Robert Muller eds., 2000.
Bernhard Sch¨olkopf, John C. Platt, John Shawe-Taylor, , Alex J. Smola, and Robert C. Williamson. Estimating the
support of a high-dimensional distribution. Neural Computation, 13:1443–1471, 2001.
R. Shamir and R. Sharan. Algorithmic approaches to clustering gene expression data. In T. Jiang, T. Smith, Y. Xu, and
M.Q. Zhang, editors, Current Topics in Computational Biology, 2000.
D.M.J. Tax and R.P.W. Duin. Support vector domain description. Pattern Recognition Letters, 20:1991–1999, 1999.
N. Tishby and N. Slonim. Data clustering by Markovian relaxation and the information bottleneck method. in
Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference, Todd K. Leen, Thomas
G. Dietterich and Volker Tresp eds., 2001.
V. Vapnik. The Nature of Statistical Learning Theory. Springer, New York, 1995.
Thanks!

You might also like