![5
Feature Vector Representation
X=[x1, x2, … , xn],
each xj a real number
xj may be an object
measurement
xj may be count of
object parts
Example: object rep.
[#holes, #strokes,
moments, …]](https://image.slidesharecdn.com/patternrecognition-241019083058-412c4761/85/Pattern-Recognition-Basic-Lecture-Notes-5-320.jpg)
More Related Content
Similar to Pattern Recognition- Basic Lecture Notes (20)
Recently uploaded (20)
Pattern Recognition- Basic Lecture Notes
- 1. 1 Pattern Recognition Pattern recognition is: 1. The name of the journal of the Pattern Recognition Society. 2. A research area in which patterns in data are found, recognized, discovered, …whatever. 3. A catchall phrase that includes • classification • clustering • data mining • ….
- 2. 2 Two Schools of Thought 1. Statistical Pattern Recognition The data is reduced to vectors of numbers and statistical techniques are used for the tasks to be performed. 2. Structural Pattern Recognition The data is converted to a discrete structure (such as a grammar or a graph) and the techniques are related to computer science subjects (such as parsing and graph matching).
- 3. 3 In this course 1. How should objects to be classified be represented? 2. What algorithms can be used for recognition (or matching)? 3. How should learning (training) be done?
- 4. 4 Classification in Statistical PR • A class is a set of objects having some important properties in common • A feature extractor is a program that inputs the data (image) and extracts features that can be used in classification. • A classifier is a program that inputs the feature vector and assigns it to one of a set of designated classes or to the “reject” class. With what kinds of classes do you work?
- 5. 5 Feature Vector Representation X=[x1, x2, … , xn], each xj a real number xj may be an object measurement xj may be count of object parts Example: object rep. [#holes, #strokes, moments, …]
- 6. 6 Possible features for char rec.
- 7. 7 Some Terminology Classes: set of m known categories of objects (a) might have a known description for each (b) might have a set of samples for each Reject Class: a generic class for objects not in any of the designated known classes Classifier: Assigns object to a class based on features
- 8. 8 Discriminant functions Functions f(x, K) perform some computation on feature vector x Knowledge K from training or programming is used Final stage determines class
- 9. 9 Classification using nearest class mean Compute the Euclidean distance between feature vector X and the mean of each class. Choose closest class, if close enough (reject otherwise)
- 10. 10 Nearest mean might yield poor results with complex structure Class 2 has two modes; where is its mean? But if modes are detected, two subclass mean vectors can be used
- 11. 11 Scaling coordinates by std dev
- 12. 12 Nearest Neighbor Classification • Keep all the training samples in some efficient look-up structure. • Find the nearest neighbor of the feature vector to be classified and assign the class of the neighbor. • Can be extended to K nearest neighbors.
- 13. 13 Receiver Operating Curve ROC Plots correct detection rate versus false alarm rate Generally, false alarms go up with attempts to detect higher percentages of known objects
- 14. 14 Confusion matrix shows empirical performance
- 16. 16 Classifiers often used in CV • Decision Tree Classifiers • Artificial Neural Net Classifiers • Bayesian Classifiers and Bayesian Networks (Graphical Models) • Support Vector Machines
- 17. 17 Decision Trees #holes moment of inertia #strokes #strokes best axis direction #strokes - / 1 x w 0 A 8 B 0 1 2 < t t 2 4 0 1 0 60 90 0 1
- 18. 18 Decision Tree Characteristics 1. Training How do you construct one from training data? Entropy-based Methods 2. Strengths Easy to Understand 3. Weaknesses Overtraining
- 19. 19 Entropy-Based Automatic Decision Tree Construction Node 1 What feature should be used? What values? Training Set S x1=(f11,f12,…f1m) x2=(f21,f22, f2m) . . xn=(fn1,f22, f2m) Quinlan suggested information gain in his ID3 system and later the gain ratio, both based on entropy.
- 20. 20 Entropy Given a set of training vectors S, if there are c classes, Entropy(S) = -pi log (pi) Where pi is the proportion of category i examples in S. i=1 c 2 If all examples belong to the same category, the entropy is 0. If the examples are equally mixed (1/c examples of each class), the entropy is a maximum at 1.0. e.g. for c=2, -.5 log .5 - .5 log .5 = -.5(-1) -.5(-1) = 1 2 2
- 21. 21 Information Gain The information gain of an attribute A is the expected reduction in entropy caused by partitioning on this attribute. Gain(S,A) = Entropy(S) - ----- Entropy(Sv) v Values(A) |Sv| |S| where Sv is the subset of S for which attribute A has value v. Choose the attribute A that gives the maximum information gain.
- 22. 22 Information Gain (cont) Attribute A v1 vk v2 Set S Set S repeat recursively Information gain has the disadvantage that it prefers attributes with large number of values that split the data into small, pure subsets. S={sS | value(A)=v1}
- 23. 23 Gain Ratio Gain ratio is an alternative metric from Quinlan’s 1986 paper and used in the popular C4.5 package (free!). GainRatio(S,A) = ------------------ Gain(S,a) SplitInfo(S,A) SplitInfo(S,A) = - ----- log ------ |Si| |S| |Si| |S| where Si is the subset of S in which attribute A has its ith value. 2 i=1 ni SplitInfo measures the amount of information provided by an attribute that is not specific to the category.
- 24. 24 Information Content Note: A related method of decision tree construction using a measure called Information Content is given in the text, with full numeric example of its use.
- 25. 25 Artificial Neural Nets Artificial Neural Nets (ANNs) are networks of artificial neuron nodes, each of which computes a simple function. An ANN has an input layer, an output layer, and “hidden” layers of nodes. . . . . . . Inputs Outputs
- 26. 26 Node Functions a1 a2 aj an output output = g ( aj * w(j,i) ) Function g is commonly a step function, sign function, or sigmoid function (see text). neuron i w(1,i) w(j,i)
- 27. 27 Neural Net Learning That’s beyond the scope of this text; only simple feed-forward learning is covered. The most common method is called back propagation. We’ve been using a free package called NevProp. What do you use?
- 28. 28 Support Vector Machines (SVM) Support vector machines are learning algorithms that try to find a hyperplane that separates the differently classified data the most. They are based on two key ideas: • Maximum margin hyperplanes • A kernel ‘trick’.
- 29. 29 Maximal Margin 0 0 0 0 1 1 1 1 Margin Hyperplane Find the hyperplane with maximal margin for all the points. This originates an optimization problem Which has a unique solution (convex problem).
- 30. 30 Non-separable data 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 1 1 1 1 0 0 What can be done if data cannot be separated with a hyperplane?
- 31. 31 The kernel trick The SVM algorithm implicitly maps the original data to a feature space of possibly infinite dimension in which data (which is not separable in the original space) becomes separable in the feature space. 0 0 0 0 0 1 1 1 Original space Rk 0 0 0 0 0 1 1 1 Feature space Rn 1 1 Kernel trick
- 32. 32 Our Current Application • Sal Ruiz is using support vector machines in his work on 3D object recognition. • He is training classifiers on data representing deformations of a 3D model of a class of objects. • The classifiers are starting to learn what kinds of surface patches are related to key parts of the model (ie. A snowman’s face)