Nonlinear component analysis as a kernel eigenvalue problem
2 likes4,278 views
This presentation summarizes paper #7 titled "Nonlinear component analysis as a kernel eigenvalue problem" by Scholkopf, Smola, and Muller. It introduces Kernel Principal Component Analysis (KPCA) as an extension of PCA that maps data into a higher dimensional feature space. The presentation discusses how KPCA frames PCA as a kernel eigenvalue problem and computes principal components in this new feature space. It provides the mathematical formulation and algorithm for KPCA. The presentation also discusses applications, advantages, disadvantages, and experiments comparing KPCA to other dimensionality reduction techniques.
![Datasets
Experiment Name Created by Representation
y x2 C y= x2
● Simple 1+2 = 3 Uniform distribution C noise sd 0.1
- Unlabelled
example1 Dist [-1, 1]
- 2 Dimensions
Three clusters
1+2 = 3
Three Gaussians - Unlabelled
● Simple
sd = 0.1 - 2 Dimensions
example2 Dist [1,1] x [0.5, 1]
Kernels
A circle and square
The eleven gaussians - Unlabelled
● De-noising
Dist [-1, 1] with zero mean - 10 Dimensions
● USPS Hand written digit
Character -Labelled
Recognition -256 Dimensions
-9298 Digits](https://image.slidesharecdn.com/pdf-111210062918-phpapp01/85/Nonlinear-component-analysis-as-a-kernel-eigenvalue-problem-24-320.jpg)
![Simple Example
● Recreated example ● Nonlinear PCA paper ex
Dataset: The USPS Handwritten digits Dataset: 1+ 2 =3 The uniform dist with sd 0.2
Training set: 3000
Classifier: The polynomial Kernel 1 - 4
Classifier: The SVM dot product Kernel 1 -7
PC: 32 – 2048 x2 PC: 1 – 3
The
eigenvector
3D 1 -3 of
highest
by a eigenvalue
Kernel
Do
PCA
Kernel Polynomial 1 -4
Accurate
2D The function y = x2 + B
Clustering
of Non with noise B of sd= 0.2
linear from uniform distribution
features [-1, 1]](https://image.slidesharecdn.com/pdf-111210062918-phpapp01/85/Nonlinear-component-analysis-as-a-kernel-eigenvalue-problem-28-320.jpg)
![References
[1] J.T. Kwok and I.W. Tsang, “The Pre-Image Problem in Kernel Methods,”
IEEE Trans. Neural Networks, vol. 15, no. 6, pp. 1517-1525, 2004.
[2] Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of
data with neural networks. Science, 313, 504-507.
[3] Sebastian Mika , Gunnar Rätsch , Jason Weston , Bernhard Schölkopf ,
Klaus-Robert Müller
[4] Trevor Hastie; Werner Stuetzle, “Principal Curves,” Journal of the American
Statistical Association, Vol. 84, No. 406. (Jun. 1989), pp. 502-516.
[5] G. Moser, "Analisi delle componenti principali", Tecniche di trasformazione
di spazi vettoriali per analisi statistica multi-dimensionale.
[6] I.T. Jolliffe, "Principal component analysis", Spriger-Verlag, 2002.
[7] Wikipedia, "Kernel Principal Component Analysis", 2011.
[8] A. Ghodsi, "Data visualization", 2006.
[9] B. Scholkopf, S. Mika, A. Smola, G. Ratsch, and K.R. Muller, "Kernel PCA
pattern reconstruction via approximate pre-images". In Proceedings of the 8th
International Conference on Artificial Neural Networks, pages 147 - 152, 1998.](https://image.slidesharecdn.com/pdf-111210062918-phpapp01/85/Nonlinear-component-analysis-as-a-kernel-eigenvalue-problem-41-320.jpg)
![References
[10] J.T.Kwok, I.W.Tsang, "The pre-image problem in kernel methods",
Proceedings of the Twentieth International Conference on Machine Learning
(ICML-2003), 2003.
● K-R, Müller, S, Mika, G, Rätsch, K,Tsuda, and B, Schölkopf “An
Introduction to Kernel-Based Learning Algorithms” IEEE
TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 2, MARCH
2001
● S, Mika, B, Schölkopf, A, Smola Klaus-Robert M¨uller, M,Scholz, G, Rätsch
“Kernel PCA and De-Noising in Feature Spaces”](https://image.slidesharecdn.com/pdf-111210062918-phpapp01/85/Nonlinear-component-analysis-as-a-kernel-eigenvalue-problem-42-320.jpg)
Nonlinear component analysis as a kernel eigenvalue problem
- 1. Presentation of paper #7: Nonlinear component analysis as a kernel eigenvalue problem Scholkopf, Smola, Muller Neural Computation 10, 1299-1319, MIT Press (1998) Group C: M. Filannino, G. Rates, U. Sandouk COMP61021: Modelling and Visualization of high-dimensional data
- 2. Introduction ● Kernel Principal Component Analysis (KPCA) ○ KPCA is an extension of Principal Component Analysis ○ It computes PCA into a new feature space dimension ○ Useful for feature extraction, dimensionality reduction
- 3. Introduction ● Kernel Principal Component Analysis (KPCA) ○ KPCA is an extension of Principal Component Analysis ○ It computes PCA into a new feature space ○ Useful for feature extraction, dimensionality reduction
- 4. Motivation: possible solutions Principal Curves Trevor Hastie; Werner Stuetzle, “Principal Curves,” Journal of the American Statistical Association, Vol. 84, No. 406. (Jun. 1989), pp. 502-516. ● Optimization (including the quality of data approximation) ● Natural geometric meaning ● Natural projection http://pisuerga.inf.ubu.es/cgosorio/Visualization/imgs/review3_html_m20a05243.png
- 5. Motivation: possible solutions Autoencoders Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504--507. ● Feed forward neural network ● Approximate the identity function http://www.nlpca.de/fig_NLPCA_bottleneck_autoassociative_autoencoder_neural_network.png
- 6. Motivation: some new problems ● Low input dimensions ● Problem dependant ● Hard optimization problems
- 7. Motivation: kernel trick KPCA captures the overall variance of patterns
- 11. Motivation: kernel trick Video
- 12. Principle Data Features "We are not interested in PCs in the input space, we are interested in PCs of features that are nonlinearly related to the original ones"
- 13. Principle Data "We are not interested in PCs New features in the input space, we are interested in PCs of features that are nonlinearly related to the original ones"
- 14. Principle Given a data set of N centered observations in a d-dimensional space ● PCA diagonalizes the covariance matrix: ● It is necessary to solve the following system of equations: ● We can define the same computation in another dot product space F:
- 15. Principle Given a data set of N centered observations in a high-dimensional space ● Covariance matrix in new space: ● Again, it is necessary to solve the following system of equations: ● This means that:
- 16. Principle ● Combining the last tree equations, we obtain: ● we define a new function ● and a new N x N matrix: ● our equation becomes:
- 17. Principle ● let λ1 ≤ λ2 ≤ ... ≤ λN denote the eigenvalues of K, and α1, ..., αN the corresponding eigenvectors, with λp being the first nonzero eigenvalue then we require they are normalized in F: ● Encoding a data point y means computing:
- 18. Algorithm ● Centralization For a given data set, subtracting the mean for all the observation to achieve the centralized data in RN. ● Finding principal components Compute the matrix using kernel function, find eigenvectors and eigenvalues ● Encoding training/testing data where x is a vector that encodes the training data. This can be done since we calculated eigenvalues and eigenvectors.
- 19. Algorithm ● Reconstructing training data The operation cannot be done because eigenvectors do not have a pre-images in the original dimension. ● Reconstructing test data point The operation cannot be done because eigenvectors do not have a pre-images in the original dimension.
- 20. Disadvantages ● Centering in original space does not mean centering in F, we need to adjust the K matrix as follows: ● KPCA is now a parametric technique: ○ choice of a proper kernel function ■ Gaussian, sigmoid, polynomial ○ Mercer's theorem ■ k(x,y) must be continue, simmetric, and semi-defined positive (xTAx ≥ 0) ■ it guarantees that there are non-zero eigenvalues ● Data reconstruction is not possible, unless using approximation formula:
- 21. Advantages ● Time complexity ○ we will return to this point later ● Handle non linearly separable problems ● Extraction of more principal components than PCA ○ Feature extraction vs. dimensionality reduction
- 22. Experiments ● Applications ● Data Sets ● Methods compared ● Assessment ● Experiments ● Results
- 23. Applications ● Clustering ○ Density Estimation ■ ex High correlation between features ○ De-noising ■ ex Lighting removing from bright images ○ Compression ■ ex Image compression ● Classification ○ ex categorisations
- 24. Datasets Experiment Name Created by Representation y x2 C y= x2 ● Simple 1+2 = 3 Uniform distribution C noise sd 0.1 - Unlabelled example1 Dist [-1, 1] - 2 Dimensions Three clusters 1+2 = 3 Three Gaussians - Unlabelled ● Simple sd = 0.1 - 2 Dimensions example2 Dist [1,1] x [0.5, 1] Kernels A circle and square The eleven gaussians - Unlabelled ● De-noising Dist [-1, 1] with zero mean - 10 Dimensions ● USPS Hand written digit Character -Labelled Recognition -256 Dimensions -9298 Digits
- 25. Experiments 1 Simple Example 1 experiment Dataset : 1+ 2 = 3 The uniform dist sd = 0.2 Kernel: Polynomial 1 – 4 2 USPS Character Recognition Parameters Dataset: USPS Kernel PCA Kernel Polynomial 1 7 Components 32 2048 (x x2) Methods Five layer Neural Networks Kernel SVM PCA SVM Neural Networks and SVM The best parameters for the task 3 De- noising Parameters Dataset: De-noising 11 gaussians sd = 0.1 The best parameters for the task Methods Kernel Autoencoders Principal Curves Kernel PCA Linear PCA 4 Kernels Parameters The best parameters for the task Radial Basis Function Sigmoid
- 26. Methods These are the methods we used in the experiments Dimensionality reduction Classification ● Supervised Unsupervised Linear PCA Linear Neural Networks Kernel PCA ● SVM Kernel Autoencoders Linear Non ● Kernel LDA Principal Curves Face Recognition
- 27. Assessment ● 1 Accuracy Classification: Exact Classification Clustering: Comparable to other clusters ● ● 2 Time Complexity ● The time to compute ● ● 3 Storage Complexity ● The storage of the data ● ● 4 Interpretability ● How easy it is to understand
- 28. Simple Example ● Recreated example ● Nonlinear PCA paper ex Dataset: The USPS Handwritten digits Dataset: 1+ 2 =3 The uniform dist with sd 0.2 Training set: 3000 Classifier: The polynomial Kernel 1 - 4 Classifier: The SVM dot product Kernel 1 -7 PC: 32 – 2048 x2 PC: 1 – 3 The eigenvector 3D 1 -3 of highest by a eigenvalue Kernel Do PCA Kernel Polynomial 1 -4 Accurate 2D The function y = x2 + B Clustering of Non with noise B of sd= 0.2 linear from uniform distribution features [-1, 1]
- 29. Character recognition Dataset: The USPS Handwritten digits Training set: 3000 Classifier: The SVM dot product Kernel 1 -7 PC: 32 – 2048 (x x2) ● The performance is better for Linear Classifier trained on non linear components than linear components ● The performance is improved from linear as the number of component is increased Fig The result of the Character Recognition experiment ( )
- 30. De-noising Dataset: The De-noising eleven gaussians Training set: 100 Classifier: The Gaussian Kernel sd parameter PC: 2 The de-noising on non linear feature of the distribution Fig The result of the denoising experiment ( )
- 31. Kernels The choice of Kernel regulates the accuracy of the algorithm and is dependent on the application. The Mercer Kernels Gram Matrix are Experiments Radial Basis Function Dataset Three gaussian sd 0.1 Classifier y exp x y 0.1 Kernel 1 4 PC 1 8 Sigmoid Dataset Three Gaussian sd 0.1 Classifier Kernel PC 1 3
- 32. Results -The PC 1-2 separate the 3 clusters RBF - The PC of 3 -5 half the clusters PC 1 PC 2 PC 3 PC 4 -The PC of 6-8 split them orthogonally PC 5 PC 6 PC 7 PC8 The clusters are split to 12 places. Sigmoid -The PC 1 -2 separates the 3 clusters - The PC 3 half the 3 clusters -The same no of PC’s to separate PC 1 PC2 clusters. PC3 - The Sigmoid needs < PC to half.
- 33. Results Experiment 1 Experiment 2 Experiment 3 Experiment 4 1 Accuracy Kernel Polynomial 4 Polynomial 4 Gaussian 0.2 Sigmoid Components 8 Split to 12 512 2 3 split to 6 Accuracy 4.4 2 Time 3 Space 4 Interpretability Very Good Very Good Complicated Very good
- 34. Discussions: KDA Kernel Fisher Discriminant (KDA) Sebastian Mika , Gunnar Rätsch , Jason Weston , Bernhard Schölkopf , Klaus-Robert Müller ● Best discriminant projection http://lh3.ggpht.com/_qIDcOEX659I/S14l1wmtv6I/AAAAAAAAAxE/3G9kOsTt0VM/s1600-h/kda62.png
- 35. Discussions Doing PCA in F rather in Rd ● The first k principal components carry more variance than any other k directions ● The mean squared error observed by the first k principles is minimal ● The principal components are uncorrelated
- 36. Discussions Going into a higher dimensionality for a lower dimensionality ● Pick the right high dimensionality space Need of a proper kernel ● What kernel to use? ○ Gaussian, sigmoidal, polynomial ● Problem dependent
- 37. Discussions Time Complexity ● Alot of features (alot of dimensions). ● KPCA works! ○ Subspace of F (only the observed x's) ○ No dot product calculation ● Computational complexity is hardly changed by the fact that we need to evaluate kernel function rather than just dot products ○ (if the kernel is easy to compute) ○ e.g. Polynomial Kernels Payback: using linear classifier.
- 38. Discussions Pre-image reconstruction maybe impossible Approximation can be done in F Need explicite ϕ ● Regression learning problem ● Non-linear optimization problem ● Algebric Solution (rarely)
- 39. Discussions Interpretablity ● Cross-Features Features ○ Dependent on the kernel ● Reduced Space Features ○ Preserves the highest variance among data in F.
- 40. Conclusions Applications ● Feature Extraction (Classification) ● Clustering ● Denoising ● Novelty detection ● Dimensionality Reduction (Compression)
- 41. References [1] J.T. Kwok and I.W. Tsang, “The Pre-Image Problem in Kernel Methods,” IEEE Trans. Neural Networks, vol. 15, no. 6, pp. 1517-1525, 2004. [2] Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504-507. [3] Sebastian Mika , Gunnar Rätsch , Jason Weston , Bernhard Schölkopf , Klaus-Robert Müller [4] Trevor Hastie; Werner Stuetzle, “Principal Curves,” Journal of the American Statistical Association, Vol. 84, No. 406. (Jun. 1989), pp. 502-516. [5] G. Moser, "Analisi delle componenti principali", Tecniche di trasformazione di spazi vettoriali per analisi statistica multi-dimensionale. [6] I.T. Jolliffe, "Principal component analysis", Spriger-Verlag, 2002. [7] Wikipedia, "Kernel Principal Component Analysis", 2011. [8] A. Ghodsi, "Data visualization", 2006. [9] B. Scholkopf, S. Mika, A. Smola, G. Ratsch, and K.R. Muller, "Kernel PCA pattern reconstruction via approximate pre-images". In Proceedings of the 8th International Conference on Artificial Neural Networks, pages 147 - 152, 1998.
- 42. References [10] J.T.Kwok, I.W.Tsang, "The pre-image problem in kernel methods", Proceedings of the Twentieth International Conference on Machine Learning (ICML-2003), 2003. ● K-R, Müller, S, Mika, G, Rätsch, K,Tsuda, and B, Schölkopf “An Introduction to Kernel-Based Learning Algorithms” IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 2, MARCH 2001 ● S, Mika, B, Schölkopf, A, Smola Klaus-Robert M¨uller, M,Scholz, G, Rätsch “Kernel PCA and De-Noising in Feature Spaces”
- 43. Thank you