03 Data Mining Techniques
Download as pptx, pdf3 likes2,580 views
This document provides an overview of data mining techniques discussed in Chapter 3, including parametric and nonparametric models, statistical perspectives on point estimation and error measurement, Bayes' theorem, decision trees, neural networks, genetic algorithms, and similarity measures. Nonparametric techniques like neural networks, decision trees, and genetic algorithms are particularly suitable for data mining applications involving large, dynamically changing datasets.
03 Data Mining Techniques
- 1. Chapter 3 Data Mining Techniques
- 2. 3.1 Introduction • Parametric models describe the relationship between input and output through the use of algebraic equations what are some parameters are not specified. These unspecified parameters are determined by providing input examples. • Nonparametric techniques are more appropriate for data mining applications. A non-parametric model is one that is data-driven. Recent techniques are able to learn dynamically as data are added to the input. This dynamic Learning process allows the model to be created continuously. The more data, the better the model. • Nonparametric techniques are particularly suitable to the database applications with large amounts of dynamically changing data. Nonparametric techniques include neural networks, decision trees, and genetic algorithms.
- 3. 3.2 Statistical Perspective. Point Estimation • The bias of an estimator is the difference between the expected value of the estimator and the actual value. Let 𝐸 Θ denote the expected value 𝐵𝑖𝑎𝑠 = 𝐸 Θ − Θ = 𝐸 Θ − Θ • One measure of the effectiveness of an estimate is the mean squared error (MSE), which is the expected value of difference between the estimates and the actual value: 𝐸 Θ − Θ 2 • The root mean square error (RMSE) is found by taking the square root of the MSE. • The root mean square (RMS) may also be used to estimate error or as another statistic to describe a distribution. Unlike mean, it does indicate the magnitude of the values. 𝑅𝑀𝑆 = 𝑗=1 𝑛 𝑥𝑗 2 𝑛 • At popular estimating technique is the jackknife estimate. With this approach, the estimate of a parameter, Θ, is obtained by omitting one value from the set of observed values. Given set of jackknife estimates, Θ 𝑖 we can obtain an overall estimate Θ . = 𝑖=1 𝑛 Θ(𝑖) 𝑛 • When we determine a range of values, within which the true parameter value should fall. This range is called a confidence interval.
- 4. 3.2.2 Estimation and Summarization Models • Maximum likelihood estimate (MLE) technique for point estimation. The approach obtains parameter estimates that maximize the probability that that sample data 𝑋 = 𝑥𝑖, … , 𝑥 𝑛 occur for the specific model 𝑓 𝑥𝑖 Θ . The likelihood function is thus defined as 𝐿 Θ 𝑥𝑖, … , 𝑥 𝑛 = 𝑖=1 𝑛 𝑓(𝑥𝑖|Θ) . The value Θ that maximizes 𝐿 is the estimate chosen. This can be found by taking the derivative with respect to Θ. • The expectation maximization (EM) algorithm can solve the estimation problem with incomplete data. The EM algorithm finds an MLE for a parameter (such as a mean) using a two step process: estimation and maximization. These steps are applied iteratively until successive parameter estimates converge. Such iterative estimates must satisfy 𝜕𝑙𝑛𝐿(Θ|𝑋) 𝜕𝜃𝑖 = 0 • Models based on summarization provide an abstraction and the summarization of the data as a whole. Well-known statistical concepts such as mean, variance, standard deviation, median, mode are simple models of the underlying population. Fitting population into a specific frequency distribution provides an even better model of the data. • Visualization techniques help to display the structure of the data graphically (histograms, box plots, scatter diagrams).
- 5. 3.2.3 Bayes Theorem • Bayes rule is a technique to estimate the likelihood of a property given the set of data as evidence or input. Suppose that either hypothesis ℎ1 or hypothesis ℎ2 must occur and 𝑥𝑖 is an observable event, the Bayes rule states 𝑃 ℎ1 𝑥𝑖 = 𝑃 𝑥𝑖|ℎ1 𝑃 ℎ1 𝑃 𝑥𝑖|ℎ1 𝑃 ℎ1 + 𝑃 𝑥𝑖|ℎ2 𝑃 ℎ2 • 𝑃 ℎ1 𝑥𝑖 is called the posterior probability, while 𝑃 ℎ1 is the prior probability associated with hypothesis ℎ1. 𝑃 𝑥𝑖 is the probability of the occurrence of data value 𝑥𝑖 and 𝑃 𝑥𝑖|ℎ1 is the conditional probability that, given a hypothesis the tuple satisfies it. Bayes rule allows to assign probabilities 𝑃 ℎ𝑗 𝑥𝑖 of hypotheses given a data value 𝑃 ℎ1 𝑥𝑖 = 𝑃 𝑥𝑖|ℎ1 𝑃 ℎ1 𝑃 𝑥𝑖 • Hypothesis testing helps to determine if a set of observed variable values is statistically significant (differs from the expected case). This approach explains the observed data by testing a hypothesis against it. A hypothesis is first made, then the observed values are compared based on this hypothesis to those of the expected case. Assuming that 𝑂 represents the observed data and 𝐸 is the expected values based on hypothesis, the chi-squared statistic, 𝜒2 , is defined as: 𝜒2 = 𝑂 − 𝐸 2 𝐸
- 6. 3.2.5 Correlations and Regression • Linear regression assumes that a linear relationship exists between the input and the output data. The common formula for a linear relationship is: 𝑦 = 𝑐0 + 𝑐1 𝑥1 + ⋯ + 𝑐 𝑛 𝑥 𝑛 • There are: 𝑛 input variables, which are called predictors or regressors; one output variable being predicted (called a response); 𝑛+1 constants, which are chosen to match model by the input sample. This is called multiple linear regression because there is more than one predictor. • Both bivariate regression and correlation can be used to evaluate the strength of a relationship between two variables. • One standard formula to measure linear correlation is the correlation coefficient 𝑟 ∈ −1,1 . Here negative correlation indicates that one variable increases while the other decreases: 𝑟 = (𝑥𝑖 − 𝑋)(𝑦𝑖 − 𝑌) (𝑥𝑖 − 𝑋)2 (𝑦𝑖 − 𝑌)2 • When two data variables have a strong correlation, they are similar. Thus, the correlation coefficient can be used to define similarity for clustering or classification.
- 7. 3.3 Similarity Measures Those tuples, that answer the query should be more like each other than those that do not answer it. Each IR query provides the class definition in the form of the IR query itself. So classification problem then becomes one of determining similarity between each tuple and the query 𝑂 𝑛 rather than 𝑂 𝑛2 problem. Common similarity measures used: • Dice 𝑠𝑖𝑚 𝑡𝑖, 𝑡𝑗 = 2 ℎ=1 𝑘 𝑡 𝑖ℎ 𝑡 𝑗ℎ ℎ=1 𝑘 𝑡 𝑖ℎ 2 + ℎ=1 𝑘 𝑡 𝑗ℎ 2 relates the overlap to the average size of the two sets together • Jaccard 𝑠𝑖𝑚 𝑡𝑖, 𝑡𝑗 = ℎ=1 𝑘 𝑡 𝑖ℎ 𝑡 𝑗ℎ ℎ=1 𝑘 𝑡 𝑖ℎ 2 + ℎ=1 𝑘 𝑡 𝑗ℎ 2 − ℎ=1 𝑘 𝑡 𝑖ℎ 𝑡 𝑗ℎ measures overlap of two sets as related to the whole set caused by their union • Cosine 𝑠𝑖𝑚 𝑡𝑖, 𝑡𝑗 = ℎ=1 𝑘 𝑡 𝑖ℎ 𝑡 𝑗ℎ ℎ=1 𝑘 𝑡 𝑖ℎ 2 ℎ=1 𝑘 𝑡 𝑗ℎ 2 relates the overlap to the geometric average of the two sets • Overlap 𝑠𝑖𝑚 𝑡𝑖, 𝑡𝑗 = ℎ=1 𝑘 𝑡 𝑖ℎ 𝑡 𝑗ℎ min ℎ=1 𝑘 𝑡 𝑖ℎ 2 , ℎ=1 𝑘 𝑡 𝑗ℎ 2 determines the degree to which two sets overlap Distance or dissimilarity measure are often used instead of similarity measures. These measure how unlike items are. • Euclidean 𝑑𝑖𝑠 𝑡𝑖, 𝑡𝑗 = ℎ=1 𝑘 (𝑡𝑖ℎ − 𝑡𝑗ℎ)2 • Manhattan 𝑑𝑖𝑠 𝑡𝑖, 𝑡𝑗 = ℎ=1 𝑘 (𝑡𝑖ℎ−𝑡𝑗ℎ) Since most similarity measures assume numeric (and often discrete) values, they may be difficult to use for general data types. A mapping from the attribute domain to a subset of integers may be used and some approach to determining the difference is needed.
- 8. 3.4 Decision Trees A decision tree (DT) is a predictive modeling technique used in classification, clustering, and prediction. A computational DT model consists of three steps: • A decision tree • An algorithm to create the tree • An algorithm that applies the tree to data and solves the problem under consideration (complexity depends on the product of the number of levels and the maximum branching factor). Most decision tree techniques differ in how the tree is created. An algorithm examines data from a training sample with known classification values in order to build the tree, or it could be constructed by a domain expert.
- 9. 3.5 Neural Networks • The NN can be viewed as directed graph 𝐹 = 𝑉, 𝐴 consisting of vertices and arcs. All the vertices are partitioned into source(input), sink (output), and internal (hidden) nodes; every arch 𝑖, 𝑗 is labeled with a numeric value 𝑤𝑖𝑗; every node 𝑖 is labeled with a function 𝑓𝑖. The NN as an information processing system consists of a directed graph and various algorithms that access the graph. • NN usually works only with numeric data. • Artificial NN can be classified based on the type of connectivity and learning into feed-forward or feedback, with supervised or unsupervised learning. • Unlike decision trees, after a tuple is processed, the NN may be changed to improve future performance. • NN have a long training time and thus are not appropriate for real-world applications. NN can be used in massively parallel systems.
- 10. Activation Functions The output of each node 𝑖 in the NN is based on the definition of an activation function 𝑓𝑖, associated with it. An activation 𝑓𝑖is applied to the input values 𝑥1𝑖, ⋯ , 𝑥 𝑘𝑖 and weights 𝑤1𝑖, ⋯ , 𝑤 𝑘𝑖 . The inputs are usually combined in a sum of products form 𝑆 = 𝑤ℎ𝑖 𝑥ℎ𝑖 . The following are alternative definitions for activation function 𝑓𝑖 𝑆 at node 𝑖: • Linear: 𝑓𝑖 𝑆 = 𝑐𝑆 • Threshold or step: 𝑓𝑖 𝑆 = 1 𝑖𝑓 𝑆 > 𝑇 0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 • Sigmoid: 𝑓𝑖 𝑆 = 1 (1+𝑒−𝑐𝑆) . This function possesses a simple derivative 𝜕𝑓 𝑖 𝜕𝑆 = 𝑓𝑖 1 − 𝑓𝑖 • Hyperbolic tangent: 𝑓𝑖 𝑆 = (1−𝑒−𝑆) (1+𝑒−𝑐𝑆) • Gaussian: 𝑓𝑖 𝑆 = 𝑒 −𝑆2 𝑣
- 12. 3.6 Genetic Algorithms • Initially, a population of individuals 𝑃 is created. They typically are generated randomly. From this population, a new population 𝑃/ of the same size is created. The algorithm repeatedly selects individuals from whom to create new ones. These parents (𝑖1, 𝑖2), are then used to produce offspring or children (𝑜1, 𝑜2) using a crossover process. Then mutants may be generated. The process continues until the new population satisfies the termination condition. • A fitness function 𝑓 is used to determine the best individuals in a population. This is then used in selection process to chose parents to keep. Given an objective by which the population can be measured, the fitness function indicates how well the goodness objective is being met by an individual. • The simplest selections process is to select individuals based on their fitness. Here 𝑝𝐼 𝑖 is the probability of selecting individual 𝐼𝑖. This type of selection is called roulette wheel selection. 𝑝𝐼 𝑖 = 𝑓(𝐼𝑖) 𝐼 𝑖∈𝑃 𝑓(𝐼𝑗) • A genetic algorithm (GA) is computational model consisting of five parts: 1) starting set, 2) crossover technique, 3) mutation algorithm, 4) fitness function, 5) GA algorithm.
- 13. References: Dunham, Margaret H. “Data Mining: Introductory and Advanced Topics”. Pearson Education, Inc., 2003.