Machine Learning Srihari

Kernel Methods

Sargur Srihari
[email protected]

1
Machine Learning Srihari

Topics in Kernel Methods

1. Linear Models vs Memory-based models

2. Stored Sample Methods
3. Kernel Functions
• Dual Representations
• Constructing Kernels
4. Extension to Symbolic Inputs
5. Fisher Kernel

2
Machine Learning Srihari

Linear Models vs Memory-based models

• Linear parametric models for regression and

classification have the form y(x,w)
• During learning we either get a maximum likelihood
estimate of w or a posterior distribution of w
• Training data is then discarded
• Prediction based only on vector w
• This is true of Neural networks as well
• Memory-based methods keep the training
samples (or a subset) and use them during the
prediction phase
3
Machine Learning Srihari

Memory-Based Methods
• Training data points are used in prediction
• Examples of such methods
• Parzen probability density model
• Linear combination of kernel functions centered on each training data point

• Nearest neighbor classification

• These are memory-based methods
• They require a metric to be defined
• Fast to train, slow to predict

4
Machine Learning Srihari

Kernel Functions
• Linear models can be re-cast
• into equivalent dual where predictions are based on
kernel functions evaluated at training points
• Kernel function is given by
k (x,x ) = ϕ(x)T ϕ(x )
• where ϕ(x) is a fixed nonlinear mapping (basis function)
• Kernel is a symmetric function of its arguments
k (x,x ) = k (x ,x)
• Kernel can be interpreted as similarity of x and x
• Simplest is identity mapping in feature space ϕ(x) = x
• In which case k (x,x ) = xTx
• Called Linear Kernel 5
Machine Learning Srihari

Kernel Trick
• Formulated as inner product allows extending
well-known algorithms
• by using the kernel trick
• Basic idea of kernel trick
• If an input vector x appears only in the form of
scalar products then we can replace scalar
products with some other choice of kernel
• Used widely
• in support vector machines
• in developing non-linear variant of PCA
• In kernel Fisher discriminant 6
Machine Learning Srihari

Other Forms of Kernel Functions

• Function of difference between arguments
k (x,x ) = k (x-x )
• Called stationary kernel since invariant to
translation
• Radial basis functions
k (x,x ) = k (||x-x ||)
• Depends on magnitude of distance between arguments
• Note that the kernel function is scalar while x is M-dimensional
For these to be valid kernel
functions they should be
shown to have the property
k (x,x ) = φ (x)T φ(x )
7
Machine Learning Srihari

Dual Representation

• Linear models for regression/classification can

be reformulated in terms of dual representation
• In which kernel function arises naturally
• Plays important role in SVMs
• Consider linear regression model
• Parameters from minimizing sum-of-squares
N
1 2 λ
J(w) = ∑{wT φ (x n ) − t n } + wT w
2 n=1 2
ϕ is the set of M
where w = (w0 ,.., w M-1 )T , φ = ( φ0 ,..φ M-1 )T basis functions
we have N samples {x1 ,..x N } or feature vector
λ is the regularization coefficient

• Minimum obtained by setting gradient of J(w) wrt w equal to zero

Machine Learning Srihari

Solution for w: linear combination of φ (xn)

• Equating derivative J(w) wrt w to zero we get
1 N
w = − ∑{wT φ (x n ) − t n }φ (x n )
λ n=1
N
= ∑ an φ (x n )
n=1

= ΦT a
• Solution for w: linear combination of φ (xn) whose
coefficients
€
are functions of w where
• Φ is design matrix whose nth row is given by φ (xn)T
% φ 0 (x1 ) . . φ M −1 (x1 ) (
' *
' . . *
Φ = 'φ 0 (x n ) . . φ M −1 (x n )* is a N × M matrix
' . . *
'&φ (x ) . . φ M −1 (x N )*)
0 N

• Vector a=(a1,..,aN)T has the definition

€
1 T
an = −
λ
{
w φ(x n )−tn }
Machine Learning Srihari

Transformation from w to a

• Thus we have w=Φ a T

• Instead of working with parameter vector w we

can reformulate least squares algorithm in
terms of parameter vector a
• giving rise to dual representation
1 T
an = −
λ
{w φ(x n )−tn }
• Although the definition of a still includes w
It can be eliminated by the use of the kernel function

10
Machine Learning Srihari

Gram Matrix and Kernel Function

• Gram matrix K=ΦΦT is N x N matrix

• with elements
Knm= ϕ (xn)Tϕ (xm)=k(xn,xm)
• where kernel function k (x,x ) = ϕ (x)T ϕ (x )
⎡ k(x ,x ) . . k(x 1,x N ) ⎤⎥
⎢ 1 1 Gram Matrix Definition:
⎢ ⎥ Given N vectors, it is the
K = ⎢⎢ . ⎥
⎥
⎢ . ⎥ matrix of all inner products
⎢ k(x ,x ) k(x N ,x N 1 ) ⎥⎥
⎢⎣ N 1 ⎦

• Notes:
• Φ is NxM and K is NxN Note: N M times M N
• K is a matrix of similarities of pairs of samples (thus it is symmetric) 11
Machine Learning Srihari
Error Function in Terms of Gram Matrix
• Error Function is
1 N
λ
J(w) = ∑ {w φ(x )−t } + w w
2
T T
n n
2 n=1 2
• Substituting w = ΦTa into J(w) gives
1 1 λ
J(w) = aT ΦΦT ΦΦTa −aT ΦΦT t + tT t + aT ΦΦTa
2 2 2
where t = (t1,..,tN)T

• Error function is written in terms of Gram matrix as

1 1 λ
J(a) = aT KKa −aT Kt + tT t + aT Ka
2 2 T 2
• Solving for a by combining w=Φ a and an = −
1 T
{w φ (x n ) − tn }
λ
a =(K +λIN)-1t
Solution for a can be expressed as a linear€ combination of elements of ϕ(x)
whose coefficients are entirely in terms of kernel k(x,x ) from which we can
recover original formulation in terms of parameters w
Machine Learning Srihari
Prediction Function

• Prediction for new input x

• We can write a =(K +λIN)-1t by combining w =ΦTa and
1 T
an = −
λ
{w φ (x n ) − tn }
• Substituting back into linear regression model,
€
y(x) = wT φ(x)
= aT Φφ(x)
= k(x)T (K + λI N )−1 t where k(x) has elements kn (x) = k(x n ,x)

• Prediction is a linear combination of the target

values from the training set.
Machine Learning Srihari

Advantage of Dual Representation

• Solution for a is entirely in terms of k(x,x )
• From a we can recover w using w = Φta
• In parametric formulation, solution is wML = (ΦT Φ)−1 ΦT t

• We invert M x M matrix, since Φ is N x M

• In dual formulation, solution is a =(K +λIN)-1t
• We are inverting an N x N matrix
• An apparent disadvantage, since N >>M
• Advantage of dual: work with kernel k(x,x ), so:
• avoid working with a feature vector ϕ(x) and
• problems associated with very high or infinite
dimensionality of x 14
Machine Learning Srihari

Constructing Kernels
• To exploit kernel substitution need valid kernels
• Two methods:
• (1) from ϕ (x) to k(x,x’)
• (2) from k(x,x’) to ϕ (x)
• First Method
• Choose ϕ (x) and use it to find corresponding kernel
k(x,x ') = φ(x)T φ(x ')
M
=∑ φi (x)φi (x ')
i=1

• where ϕi(x) are basis functions

• One-dimensional input space is shown next
15
Machine Learning Srihari

Kernel Functions from basis functions

One-dimensional input space

Polynomials Gaussian Logistic Sigmoid

Basis
Functions
ϕi(x)

Kernel
Functions
k(x,x ) = ϕ (x)Tϕ (x’)
Red cross is x

16
Machine Learning Srihari

Method 2: Direct Construction of Kernels

• Kernel has to correspond to a scalar product in
some (perhaps infinite dimensional) space
• Consider kernel function k(x,z) = (xTz)2
k(x,z) = (x T z)2 = (x1z1 + x 2z 2 )2
• In 2-D space = x12z12 + 2x1z1x 2z 2 + x 22z 22
= (x12, 2x1x 2,x 22 )(z12, 2z1z 2,z 22 )T
= φ(x)T φ(z)
• Feature mapping takes the form φ(x) = (x12, 2x1x 2, x 22 )
• Comprises of all second order terms with a specific weighting
• Inner product needs computing 6 feature values, 9 multiplications
• Kernel function k(x,z) has 2 multiplies and a squaring
• For(xTz+c)2 we get constant, linear, second order terms
• For (xTz+c)M we get all powers of x (monomials)

17
Machine Learning Srihari

Testing whether a function is a valid kernel

• Without constructing ϕ(x), Necessary and
sufficient condition for k(x,x ) to be a kernel is
• Gram matrix K, with elements k(xn,xm) is positive
{xn}semi-definite for all possible choices of the set
• Positive semi-definite is not the same thing as a matrix whose elements are non-
negative
zT Kz ≥ 0 for non - zero vectors z with real entries
• It means
i.e., ∑ ∑ K nm zn zm ≥ 0 for any real numbers zn , zm
n m

• Mercer s theorem: any continuous, symmetric, positive semi-definite kernel

function k(x, y) can be expressed as a dot product in a high-dimensional
€
space

• New kernels can be constructed from simpler

kernels as building blocks 18
Machine Learning Srihari

Techniques for Constructing Kernels

• Given valid kernels k1(x,x ) and k2(x,x ) the following new
kernels will be valid
1. k(x,x ) =ck1(x,x )
Where
2. k(x,x )=f(x)k1(x,x )f(x ) f (.) is any function

3. k(x,x )=q(k1(x,x )) q(.) is a polynomial with non-negative coefficients

4. k(x,x )=exp(k1(x,x ))
5. k(x,x )=k1(x,x )+k2(x,x )
6. k(x,x )=k1(x,x )k2(x,x )
7. k(x,x )=k3(f(x).f(x )) f(x) is a function from x to RM
k3 is a valid kernel in RM
8. k(x,x )=xTAx A is a symmetric positive semidefinite matrix

9. k(x,x )=ka(xa,xb )+kb(xb,xb ) xa and xb are variables with x=(xa, xb)

10. k(x,x )=ka(xa,xa )kb(xb,xb ) ka and kb are valid kernel functions
Machine Learning Srihari

Kernels for specific applications

• Requirements for k(x,x )

• It is symmetric
• Its Gram matrix is positive semidefinite
• It expresses the appropriate similarity between x
and x for the intended application

20
Machine Learning Srihari
Gaussian Kernel
• Commonly used kernel is
k(x,x ) = exp (-||x-x ||2/2σ2)
• It is seen as a valid kernel by expanding square
||x-x ||2 = xTx + (x )Tx -2xTx
• To give
k(x,x ) = exp (-xTx/2σ2) exp (-xTx /σ2) exp (-(x )Tx /2σ2)
• From kernel construction rules 2 and 4
• together with validity of linear kernel k(x,x )=xTx
• Can be extended to non-Euclidean distances
k(x,x ) = exp {(-1/2σ2)[κ(x,x )+κ (x ,x )-2κ (x,x )]}
21
Machine Learning Srihari

Use of a kernel method

3-D space ϕ(x)

2-D space x

Infinite dimensional space ϕ(x)

Induced by a Gaussian kernel

Oil slick 2D- image

that looks like a feature space x 22
Machine Learning Srihari

Extension of Kernels to Symbolic Inputs

• Important contribution of kernel viewpoint:
• Inputs are symbolic, not vectors of real numbers
• Kernels defined for graphs, sets, strings, text documents
• If A1 and A2 are two subsets of objects
• A simple kernel is k(A1,A2 ) = 2|A ∩A |
1 2

• where |A | indicates no of elements in A

• A valid kernel since it can be shown to correspond to an
inner product in a feature space A={1,2,3,4,5}
A1={2,3,4,5}, A2={1,2,4,5}, A1∩A2={2,4,5}
Hence k(A1,A2)=8

Example feature vectors:

ϕ(A1)=[2 2 2 2] and ϕ(A2) =[1 1 1 1]
such that
ϕ(A1)Tϕ(A2)=8
Machine Learning Srihari

Kernels for Complex Objects

• ML methods studied so far require input
represented as a fixed-size feature vector xiε RD
• For certain objects it is not clear how best to
represent them as feature vectors, E.g.,
• Text or protein sequence of variable length
• Molecular structure with complex 3-D geometry
• Evolutionary tree which has variable size and shape
• Solutions
• Define generative model, whose parameters are
features. Plug these features into standard models
• Measure similarity between objects which does not
require preprocessing into feature vector format 24
Machine Learning Srihari

Kernels for Comparing Documents

• In IR and document classification, need a way
of comparing documents xi and xi’
• Bag of words representation: xij is the no of times
word j occurs in document i xi=[xi1,..,xiD]

• Cosine similarity k(x ,x ) = x iT x i '

i i'
|| x i ||2|| x i ' ||2

• It measures cosine of angle between xi and xi’

• Since xi is a count vector (and hence non-negative) the
cosine similarity is between 0 and 1
• Where 0 means the vectors are orthogonal and therefore have no
words in common

25
Machine Learning Srihari

Disadvantage of Cosine Similarity

• It does not work well for two reasons
1. Stop words
• If xi has any word in common with xi’ it is deemed similar,
even though some words such as the, and occur commonly
in many documents and are not discriminative
2. High frequency words in a document
• If a discriminatory word occurs many times in a document,
the similarity is artificially boosted
• Even though word usage would be bursty, i.e., once a word is used
in a document it is very likely to be used again

26
Machine Learning Srihari

TF-IDF
• Cosine similarity performance can be improved
with some preprocessing
• Use feature vector called TF-IDF representation
• Term frequency-Inverse document frequency
tf(x ij ) ! log(1 + x ij )
• TF is a log-transform of the count
N
• IDF is defined as idf(j) ! log
1 + ∑ I(x > 0)
N
i=1 ij
xij is the no of times word j occurs in document i

• Where N is the total no of documents

• Denominator counts how many documents contain term j
• Finally we define tf-idf(xi) ! [tf(x ij )×idf(j)]Vj =1

• We use this inside cosine similarity measure

φ(x i )T φ(x i ' )
k(x i ,x i ' ) =
|| φ(x i ) ||2|| φ(x i ' ) ||2

• Where ϕ (x)=tf-idf(x) 27
Machine Learning Srihari

String Kernels
• Real power of kernels: inputs are structured
• Compare two variable-length strings
• Strings x, and x’ of length D, D’ defined over A
• E.g., two amino acid sequences defined over 20-letter
alphabet A={A,R,N,D,C,E,Q,G,H,I,L,K,M,F,P,S,T,W,Y,V}
• x is the sequence of length 110
IPTSALVKETLALLSTHRTLLIANETLRIPVPVHKN……VNQFLDYLQEFLGVMNTEWI

• x’ is a string of length 153

• PHRRDLCSRSIWLARKIRSDLTALTESYVKHQGLWSELTEAERLQENLQAYRTFHVLLA……..

• The strings have the substring LQE in common

• We can define the similarity of two strings to be the no of
substrings they have in common
• Definition given next
28
Machine Learning Srihari

Mercer Kernel
• If s is a substring of x we can write x=usv for
some (possibly empty) strings u,s and v
• ϕs(x) is no of times substring s appears in x
• Define kernel between two strings x and x’ as
k(x,x ') = ∑ ws φs (x)φs (x ')
s∈A*

• Where ws≥0 and A* is the set of all strings from alphabet A

(known as Kleene * operator)
• Can be computed in O(|x|+|x’|) time
• Cases of interest:
• ws=0 for |s|>1 we get bag-of-characters kernel– defines
ϕ(x) as no of times each character in A occurs in x
• If we require s to be bordered by white space we get bag-
29
of- words kernel
Machine Learning Srihari

Combining Discriminative and

Generative Models
• Generative models deal naturally with missing
data and with HMM of varying length
• Discriminative models such as SVM have better
performance
• Can use a generative model to define a kernel
and use kernel in discriminative approach

30
Machine Learning Srihari

Kernels based on Generative Models

• For a model p(x) we define a kernel by

k (x,x ) = p(x) p(x )
• A valid kernel since an inner product in 1-dimensional
feature space defined by the mapping p(x)
• Two inputs x and x are similar if they have
high probabilities

31
Machine Learning Srihari

Kernel Functions based on Mixture Densities

• Extension to sums/products of distributions
k(x,x ') = ∑ p(x | i)p(x ' | i)p(i)
i

• where p(i) are positive weighting coefficients

• A valid kernel based on two rules of kernel construction:
k(x,x ) =ck1(x,x ) and k(x,x )=k1(x,x )+k2(x,x )
• Two inputs x and x will give a large value of k , and
hence appear similar, if they have a significant
probability under a range of different components
• Taking the limit to infinite sum
k(x,x ') = ∫ p(x | z)p(x ' | z)p(z)dz

• where z is a continuous latent variable

32
Machine Learning Srihari

Kernels for Sequences

• Data consists of ordered sequences of length L
X={x1,..,xL}
• Generative model for sequences is HMM
• Hidden states Z={z1,..,zL}
• Kernel Function for measuring similarity of
sequences X and X is
k(X, X ') = ∑ p(X | Z)p(X ' | Z ')p(Z)
Z

• Both observed sequences are generated by same

hidden sequence Z

33
Machine Learning Srihari

Fisher Kernel
• Alternative technique for generative models
• In document retrieval, protein sequences
• Consider parametric generative model p(x|θ)
where θ denotes vector of parameters
• Goal: find kernel that measures similarity of two
vectors x and x induced by the generative model
• Define Fisher score as gradient wrt θ
g(θ,x) = ∇θ ln p(x | θ ) A vector of same dimensionality as θ

• Fisher Kernel is k(x, x ') = g(θ, x)T F−1g(θ, x’) Fisher score is more
generally the gradient
€ F = E x [ g(θ,x)g(θ,x)T ] of the log-likelihood

where F is the Fisher information matrix 34

Machine Learning Srihari

Fisher Information Matrix

• Presence of Fisher information matrix causes
kernel to be invariant under non-linear
parametrization of the density model θ àψ(θ)
• In practice, infeasible to evaluate Fisher Information
Matrix. Instead use the approximation
N
1
F ≈ ∑ g(θ,x )g(θ,x )
n n
T
k(x,x') = g(θ ,x) F g(θ,x’)
T −1
N n=1

• This is the covariance matrix of the Fisher scores

• So the Fisher kernel corresponds to whitening of the
€ €
Fisher scores k(x,x') = g(θ,x)T g(θ,x')
• More simply omit F and use non-invariant kernel
35
€
Machine Learning
Sigmoidal Kernel Srihari

• A link between SVMs and neural network

k (x,x ) = tanh (axTx + b)
• Its Gram matrix is not positive semidefinite
• But used in practice because it gives SVMs a
superficial resemblance to neural networks
• Bayesian neural network with an appropriate
prior reduces to a Gaussian process
• Provides a deeper link between neural networks
and kernel methods

36