DataMining and Knowledge Discovery in DB.ppt
- 1. Data Mining and Knowledge Discovery in Databases
- 2. Outline • What is Data Mining and KDD? • Characteristics • Applications • Methods • Packages & Close Relatives
- 3. What is Data Mining & KDD? • “The process of identifying hidden patterns and relationships within data” or • “Data mining helps end users extract useful business information from large databases”
- 4. What’s the Appeal? • Hidden nuggets of valuable information buried deep within a mountain of otherwise unremarkable data • Pervasive data • Seek competitive advantage
- 6. Process: Knowledge Discovery In Databases database database data warehouse cleaning & integration modify data selection modify data selection data mining collect and transform discovered patterns data mining engines, models evaluation & presentation user interface and expert knowledge domain modify methods, parameters
- 7. Context • Where you stand on Data Mining depends on where you sit: • Business User • Researcher • Computer Scientist
- 8. Data Mining Might Mean… • Statistics • Visualization • Artificial intelligence • Machine learning • Database technology • Neural networks • Pattern recognition • Knowledge-based systems • Knowledge acquisition • Information retrieval • High performance computing • And so on...
- 9. What’s needed? • Suitable data • Computing power • Data mining software • Skilled operator who knows both the nature of the data and the software tools • Reason, theory, or hunch
- 10. Typical Applications of Data Mining & KDD • Marketing • Market Basket Analysis • Customer Relationship Management • New Product Development
- 11. Typical Applications of Data Mining & KDD • Financial Services • Credit Approval • Fraud Detection • Marketing
- 12. Typical Applications of Data Mining & KDD • Health Care • Epidemiological Analysis - incidence and prevalence of disease in large populations and detection of the source and cause of epidemics of infectious disease • Knowledge for funding • Policy, programs
- 13. Two Basic Approaches • Supervised • A dependent or target variable • Unsupervised • “Pure Data Mining” • Fewer assumptions • Typically used for clustering techniques
- 14. Automation • The ability to aim a tool at some data and push a button • Some methods of KDD/Data mining are more suitable for automation than others
- 15. Seven Basic Methods: 1. Decision Trees 2. (Artificial) Neural Networks 3. Cluster/Nearest Neighbour 4. Genetic Algorithms/Evolutionary Computing 5. Bayesian Networks 6. Statistics 7. Hybrids
- 16. • Graphical representations of relationships with data • Excel at Classification & Prediction Models Decision Trees
- 17. Sample of a Decision Tree gender female male <65 >=65 married? age yes no good health? yes no - + urban? yes no pet owner? yes no + - - + pet owner? yes no - +
- 18. Decision Trees • Strengths • Easily understood and interpreted • Represent complexity in a compact form • Handle non-linear data well • Relatively well suited to automation. • Weaknesses • Large trees with large numbers of variables become difficult to understand • Missing data must be appropriately managed in construction and use of the models
- 19. Neural Networks • Derived from Artificial Intelligence Research • Modelled on the Human Neuron
- 20. Neural Networks Age Gender Income Prediction Hidden Layer Input Variables 0.6 0.3 0.1 0.5 0.7 0.8 0.4 Weights Weights 0.3 0.2
- 21. Neural Networks • Strengths • Accuracy of prediction • Robust performance with a wide variety of data types • Weaknesses • Prone to overfitting • Poor clarity of model
- 22. Clustering/Nearest Neighbour • Aim to assign “like” records to a group • Groups assigned according to some target variable or criteria • Nearest neighbour used for prediction
- 23. Clustering/Nearest Neighbour • Applications: • Text processing: search engines • Image processing: radiology/image processing • Fraud detection: outliers
- 24. Clustering/ Nearest Neighbour • Strengths • Easily understood and interpreted • Easily implemented in basic situations • Weaknesses • complex data not well suited to automation (much preprocessing required)
- 25. Genetic Algorithms/ Evolutionary Computing • Grounded in Darwin – applied using mathematics • Require • a way to represent a solution to a problem • a way to test the “fitness” of the solution • Solutions are mathematically “mutated” • Fittest solutions survive • Convergence
- 26. Genetic Algorithms/ Evolutionary Computing • Strengths • Suited to novel problems that are poorly understood • Suitable where data is dirty or missing • May be useful where other methods cannot be applied • Weaknesses • Not easily automated • Require creativity in their application
- 27. Bayesian Networks • Based on Bayes’ rule: • P(a|b) = P(b|a) * P(a) / P(b) • Can construct networks of linked events, each with prior probabilities
- 28. Bayesian Network Example J.R. Shot Bobby shot him Just a dream sequence Mistress shot him Wife shot him Suicid e J. R. Treated for Depressio n Bobby publicly threatened Producer s desperat e for ratings Big fight between wife, mistress
- 29. Bayesian Networks • Strengths • Clarity of the resulting models • Good precision in predicting • Easily adapt to new probabilities • Weaknesses • Time consuming to construct and maintain • Poor at predicting rare events
- 30. Statistics • With an outcome or dependent variable: • Correlations • ANOVA • Regression • Used by themselves or to confirm findings of another method
- 31. Statistics • Strengths • “Gold Standard” – valid and trusted in scientific circles • Weaknesses • Limits findings to those techniques that are applied and their associated limitations (normality, linearity, and so on)
- 32. Hybrids • Techniques used in combination • Example: use of a genetic algorithm to identify target variables for inclusion in a neural network model
- 33. Recap • Data Mining is the core activity or method within a process of Knowledge Discovery in Databases • Done in order to find useful information in large amounts of data not possible using “conventional” approaches • Variety of methods • Knowledge of data domain, methods, as well as creativity
- 34. Data Mining Packages • Major vendors of database/data management products (IBM, SPSS, Oracle PeopleSoft, SAS, and so on) • Added as a component of turnkey packages • May incorporate several methods (SAS Enterprise Miner) • Single method (TreeAge Software Inc.: a dedicated decision tree product)
- 35. How to implement? • Do it yourself (you know the data domain) • Put a team together (domain and method specialists) • Hire a consultant (who knows both your domain and the tools) • Vertical markets in data mining
- 36. Close Relatives of Data Mining • On-Line Analytical Processing (OLAP) • Pivot tables in spreadsheets • General statistical packages • Intelligent Data Analysis – comprises the use of data mining methods in the analysis of “small” datasets
Editor's Notes
- #3: Definitions: “The process of identifying hidden patterns and relationships within data” or “Data mining helps end users extract useful business information from large databases”
- #5: Humans aren’t particularly well suited to finding patterns in data Computers, on the other hand…
- #6: Distinguish the core data mining component from the overall KDD process. I’ll accept the common usage for this lecture and use the term data mining to mean both the core methods and the overall process.
- #7: Importance of context: A business user will be interested in efficiency and results, validity may not be as important. A researcher clearly will be interested in a different type of results, and validity will be important. A computer scientist may be interested in introducing new algorithms or computational approaches and achieving improved results or more efficient processing.
- #8: Data Mining is not a precisely defined area – embodies many related academic and applied specialties.
- #9: The first three things are clear. Human component is critical: knowledge of the business/data domain and knowledge of the tools are both important – do you train your domain people to know data mining? - do you train data mining people to know the domain? – do you build a team? - do you bring in a consultant who may know a lot of both in a specific area (financial services for example) – do you build “intelligent agents” to guide users? Some data mining may clearly have a stated purpose: Identify the three most notable demographic characteristics among our customers that spend over $X – others are more open ended – look for patterns in radio-telescope signals coming from deep space. Whatever the case, data mining requires some rationale for committing the time and resources to do it.
- #10: Marketing – why? – large amounts of transactional data on purchases, WalMart for example – market basket analyses – who buys what with what – influence on retail decisions – also Customer Relationship Management (CRM) – churn – important in business-to-business environment
- #11: Financial services – again, large numbers of computerized records – data mining used to profile characteristics of poor or excellent credit risks, or exception analyses for fraud detection
- #12: HealthCare – may have large amounts of data: hospital admissions/discharges, or billing information for physician procedures epidemiological (definition: study of patterns of disease) studies using population health data, for a province or entire nation – impact on health intervention, funding, training, policy, etc.
- #17: Start at the top with all records, nodes use field values to go “left or right” – branch to further nodes, or to end points (leafs) that define higher or lower likelihood of the outcome or dependent variable Supervised Data Mining Asymetric models – note that a variable can be used in more than one place – so although fairly clear to interpret if model is small, can quickly become difficult to interpret Constructing these: both automated and interactive semi-automated – choose which variables to use, and for continous variables where cutpoints will be Branches can be restricted to 2 or not Typically divide a dataset into training and validation subsets.
- #19: Biological neuron has multiple inputs (dendrites), resulting in an output This is modelled in its simplest form with multiple inputs (X’s), each with weights (W’s), resulting in an output Y
- #20: This is a hypoethical model that might be used to predict health status. This model is supervised (target variable: good health or poor health) Input variables at the bottom Feedforward as inputs to a hidden layer of nodes Weights are set and adjusted during training; convergence Variety of variations and approaches using feedback architectures Algorithms to optimize computing time which can be extensive: multiple training passes
- #28: Events are linked by probabilities. By estimating or quantifying probabilities among linked events, can estimate most likely cause. Updated probabilities alter the most likely cause – these models can “learn” from new information.
- #31: But is it really “data mining”?
- #34: Database standards allow any data mining package to access raw data Turnkey packages: for example hospital information management system, enterprise packages
- #35: Vertical markets: development of data mining packages for one specific kind of customer, bank fraud, for example
- #36: Cognos Powerplay, for example, that slice/dice across multiple dimensions – very popular with sales and financial analysis in support of business decisions. The term data mining tends to be over used.