Difference between data warehouse and data mining

Editor's Notes

• #3: Data Warehousing- a repository of information, or archive information, gathered from multiple sources stored under a unified schema. OLAP – on-line analytical processing - a method of analysis of data based on multi-dimensional databases. Data Mining- analyses data, discovers rules and patterns from the data. Both OLAP and Data Mining are end-user tools for data analysis
• #4: DBMS in industry are pervasive throughout industry. Designed to handle high transaction throughput, where transactions typically make small changes to the operational data, for day-to-day running of the organisation. Can range in size: small databases being mbs where large databases can require terabytes or even petabytes. Decision makers require access to all the organisation’s data to provide comprehensive analysis of the organisation, its business, its requirements and its trends. This needs access to both current and past data. A data warehouse is different to an OLTP system in that it fits the definition given on this slide. Subject oriented: organised around the major subjects of the enterprise (e.g. customers, products, sales) rather than the major application areas (e.g. customer invoicing, stock control, product sales) Integrated: coming together of the source data from different enterprise-wide applications systems. Often inconsistent, e.g. different formats Time-variant: data in the warehouse is only accurate at some point in time or over some time interval. Data represents a series of snapshots Non-volatile: data is not updated in real time, refreshed from operational systems on a regular basis. New data is always added, rather than replaced.
• #5: These are the benefits. Organisations normally must commit a huge amount of investment and resources in developing the data warehouse, but the potential returns on that investment due to increasing productivity and the competitive advantage that gives can be very large, the IDC quote is 90% of companies had >40% return over three years( normal ROI would be 8-15%). The competitive advantage is gained by the access to data that was previously unavailable, and increases productivity as the data is integrated from previously incompatible systems. Competitive advantage is gained by giving access to this data by management decision makers for forecasting trends etc.
• #6: This shows the typical architecture of a data warehouse. It shows: Data sources - can vary from mainframes to departmental databases to external data. There are lots of different sources and different data types. Load manager (or frontend) performs the extraction and loading of data into the warehouse. Warehouse manager performs all the operations associated with the management of the data in the warehouse. Operations include - ensuring consistency of data, indexes and views, denormalising, aggregating data, backing up and archiving. Query manager (backend) manages the user queries. Complexity depends on flexibility of end-user access tools. Can include directing queries to tables, scheduling execution of queries, generating query profiles to assist warehouse manager in managing indexes and views. Detailed data: this is all the detailed data in the schema. Normally stored offline and aggregated into next level of data. Lightly/highly summarised data: this is the aggregated data generated by the warehouse manager. This is subject to change on an on-going basis depending on the types of queries. Purpose is to speed up queries. Meta-data: description of data in warehouse. Changes according to structure of data in warehouse.
• #7: There are numerous types of data in a data warehouse. Detailed data is the actual data which has been pulled in from the various sources. Summarised data tends to create various views of the detailed data, to answer specific queries. It needs to be summarised because there is such a large amount of data. Because these views can change, there also needs to be meta-data. Archive/backup – the data warehouse will always grow, so some of the older data can be archived, in a way that it can still be included in queries if required.
• #8: This diagram shows the main flows of data in the warehouse. The next slide explains each of the five flows.
• #9: This explains the processes associated with each of the information flows. Inflow – cleans dirty data, restructures data to suit new requirements, ensure source is consistent with data already in the warehouse Upflow – summarise data into more convenient views, pack data into more useful formats, distributes data to increase availability/accessibility Downflow – transfer data of limited value restore following crash Outflow – 2 activities: 1. accessing – satisfy individual end user requests for data from tools; 2. delivering – delivery of information to end users Metaflow - - process which moves metadata responding to changing needs, i.e. updating metadata accordingly
• #10: These are the typical problems with a data warehouse, most arise from the problems of integrating the source data(see pages 1050-1052, Connolly & Begg) 80% of development time is spent on data loading Problems with source systems, e.g. nulls allow incomplete data, needs to be fixed OLTP systems may not store data needed – so may need to alter the OLTP systems Users become aware of capabilities – need better tools Homogenization can lessen value of data – similarties v. differences in data Disk space, large no. of indexes Data accessible to all users Reorganisation of business processes – change to DW Can take 3 years to build – data marts support only one department so may be quicker Need to integrate all tools to ensure benefits the organisation
• #11: The types of queries we need to be able to perform are different to those in an OLTP system as they are more factual, analytical and temporal. An example is given - try doing this in a relational system. So normal modelling techniques (E-R model) are not suitable as the relationships between the data can sometimes be too complex, therefore we use dimensionality modelling: a logical design technique that aims to present the data in a standard, intuitive form that allows for high-performance access.
• #12: Our dimensional model is based on the E-R model but with some restrictions, to support the types of queries required. A model with these restrictions is called a Star Schema. The use of surrogate keys aids performance in joins.
• #13: A star schema contains two types of tables as defined. An example is given on the next couple of slides. Fact tables contain factual data. The FK can be classed as either unintelligent (an unique identifier represented by the actual data), or intelligent (a surrogate FK) Fact tables are unlikely to change – as facts normally occur in the past Dimensional tables contain reference data – i.e. the data which supports the fact. Star schemas given increased performance as dimension tables are denormalised, thus minimising the number of joins.
• #14: This is an E-R model taken from Connolly and Begg for the Dream Home database. Notice that it contains complex relationships between the various objects, which would make it different to answer the types of queries required. So we redesign using a star schema as given on the next slide...
• #15: This is the Star Schema version. Now we have one table in the centre which contains all the links to the dimension tables, which contains the data. The fact table is just like a M:N relationship in a relational database. Note that the can be more than one fact table in a star schema.
• #16: Snowflake: Star schema is denormalised Snowflake schemas contain no denormalised data, the data is normalised Starflake A combination of normalised and denormalised data. This is the most appropriate schema to use.
• #17: There are 2 tools commonly used for data analysis. These are OLAP and data mining. OLAP tools are query centric – database schemas are array oriented and multi-dimensional in nature, e.g. market analysis. So OLAP tools work on the concept of multi-dimensional (i.e. >2 dimensions) data – to support complex analytical applications. So there is a new type of database – multi-dimensional DB – as there is a need to retrieve large numbers of records from very large data sets and summarise this data on they fly. For example, the dimensions of a database could be property type, city and time. Typical operations which can be performed are: Consolidation – aggregate data, roll up, e.g branch offices -> city, city -> country Drill down – the reverse of consolidation, displays detailed data Slicing and dicing, aka pivoting – look at data from different viewpoints, normally along a time axis.
• #18: Codd defined a number of rules for OLAP systems Must be intuitively analytical and easy to use Transparent to users – users are familiar with particular front end tools All data sources must be accessible (network, hierarchical, relational, etc.) As number of dimensions increased, the performance must remain consistent Must operate efficiently in a client-server architecture There must be no bias towards any one dimension There may be instances where a large number of nulls are stored – this must not have any averse impact on accuracy and speed of access Must support concurrent users Must be able to support the typical operations on the last slide, e.g. performing roll-up within/across dimensions Slicing and dicing, drill down and consolidation must be intuitive, e.g. via a point and click interface, or drag and drop operations on a data cube Must be able to retrieve any view of the dta No. of dimensions should be unlimited
• #19: Types of tools are categorised according to the architecture of the underlying database. There are 3 main categories. MOLAP – specialised data structures and MDDBMSs to organise, navigate and analyse data Data typically aggregated to enhance performance Use array technology and sparse data management ROLAP – this is the fastest growing technology. Works by providing multi-dimensional views of 2D data. SQL is enhanced to increase performance and support complex operations on multi dimensions MQE – this is the newest technology. Data can be delivered either directly from the RDB or from a MOLAP/ROLAP server in the form of a datacube. The datacube is stored and analysed locally – therefore they are simple to install, and each user can build a custom data cube.
• #23: There are various end-user tools which can be used with data warehouses, and data mining is one of those sets of tools which is used for analysing data within a database to find hidden/unexpected information within a database.
• #24: These are some examples of typical applications
• #26: Although there are four operations which can be used independently, many applications work well when several or a combination of operations are used. There are specific techniques used with each operation.
• #27: Predictive modelling uses observations to form a model of the important characteristics of some phenomenon. Can be used to analyse an existing database to determine some essential characteristics about the data set. Two main techniques: Classification: used to establish a specific predetermined class for each record in a database from a finite set of possible class values, e.g. if a customer has rented for > 2 years and > 25 years old then they are most likely to buy property. Can use neural/tree induction. Value prediction: used to estimate a continuous numeric value that is associated with a database record, uses statistical techniques, e.g. linear/non-linear regression. An example of classification is given on the next slide.
• #29: The second technique is database segmentation. Aims to cluster records so that they share a number of properties, i.e. homogenous. Uses unsupervised learning to discover sub-populations in the database. Two types – demographic and neural clustering. An example is given on the next slide of a scatterplot.
• #31: Link analysis establishes association between records. An example is given. Various techniques which look for associations/patterns/similar time sequences: Association – items which imply the presence of other item in same event Sequential – presence of 1 set of item implies presence of another in a period of time (e.g. long term customer buying behaviour) Similar time sequence – discovery of link between 2 sets of data that are time dependent, e.g. buying property -> buy household goods within 2 months.
• #32: Deviation detections identifies records where a value is out of the ordinary. Can be done either statistically (e.g. linear regression) or by visualisation (e.g. graphically), as in the example on the next slide. Good for fraud detection.
• #34: So to finish off on warehousing, if we look at the requirements for a data mining tool and then compare this to what we get from a data warehouse, then we can see that the ideal data source for data mining is a data warehouse. DW data is clean and consistent which is a prerequisite for Data mining Multiple sources allow to discover as many inter-relationships as possoble Query capabilities allow for selection of relevant subsets of records and fields Go back to data source – provides a way for data mining results to allow further investigation of uncovered patterns.