DATA MINING

What we study in this chapter?

1.1 Why Data Mining ?

1.1.1 Moving toward the Information Age
1.1.2 Data Mining as the Evolution of Information Technology

1.2 What Is Data Mining?

1.3 What Kinds of Data Can Be Mined?

1.3.1 Database Data
1.3.2 Data Warehouses
1.3.3 Transactional Data
1.3.4 Other Kinds of Data
 1.4 What Kinds of Patterns Can Be Mined?
1.4.1 Class/Concept Description: Characterization and Discrimination
1.4.2 Mining Frequent Patterns, Associations, and Correlations
1.4.3 Classification and Regression for Predictive Analysis
1.4.4 Cluster Analysis
1.4.5 Outlier Analysis

1.4.6 Are All Patterns Interesting?

3.2 Data Cleaning

3.2.1 Missing Values
3.2.2 Noisy Data
3.2.3 Data Cleaning as a Process
Data Mining
1.1 - Why Data Mining ..?

We live in a world where vast amounts of data are collected daily.

Analyzing such data is an important need.

We further looks at how data mining can meet this need by providing
tools to discover knowledge from data.

Also, we observe how data mining can be viewed as a result of the

natural evolution of information technology.
1.1.1 - Moving toward the Information Age

We are living in the information age is a popular saying; however, we

are actually living in the data age.

Terabytes or petabytes of data pour into our computer networks, the

World Wide Web (WWW), and various data storage devices every day
from business, society, science and engineering, medicine, and almost
every other aspect of daily life.

This explosive growth of available data volume is a result of the

computerization of our society and the fast development of powerful
data collection and storage tools.
 For example, large stores, such as Big Bazar, handle hundreds of
millions of transactions per week at thousands of branches around the
world.

Scientific and engineering practices generate high orders of petabytes

of data in a continuous manner, from remote sensing, process
measuring, scientific experiments, system performance, engineering
observations, and environment surveillance.

To know how data will be produced, we have two activities to get the
awareness of data collection.

1. What you did in holidays..?

2. How your friends enjoyed Holidays..?

 As per the previous tasks we have some of the big data collection
details and they are

Telecommunication networks carry tens of petabytes of data traffic

every day.

The medical and health industry generates tremendous amounts of

data from medical records, patient monitoring, and medical imaging.

Billions of Web searches supported by search engines process tens of

petabytes of data daily.

Communities and social media have become increasingly important

data sources, producing digital pictures and videos, blogs, Web
communities, and various kinds of social networks.

The list of sources that generate huge amounts of data is endless.

 This explosively growing, widely available, and gigantic body of data
makes our time truly the data age.

Powerful and versatile tools are badly needed to automatically

uncover valuable information from the tremendous amounts of
data and to transform such data into organized knowledge.

This necessity has led to the birth of Data Mining.

The field is young, dynamic, and promising. Data mining has and will
continue to make great strides in our journey from the data age toward
the coming information age.
Example: Data mining turns a large collection of data into knowledge.

A search engine (e.g., Google) receives hundreds of millions of queries

every day.

Each query can be viewed as a transaction where the user describes

her or his information need.

What novel and useful knowledge can a search engine learn from such
a huge collection of queries collected from users over time?

Let us make this point clear

 If i type F in Google search what you can see ..?

It is because almost all people search Facebook simultaneously.

If large people search this, automatically more data is collecting and it

shows first.
1.1.2 - Data Mining as the Evolution of Information Technology

Data mining can be viewed as a result of the natural evolution of

Information Technology.

The database and data management industry evolved in the

development of several critical functionalities.

The functionalities are data collection and database creation, data

management (including data storage and retrieval and database
transaction processing), and advanced data analysis (involving data
warehousing and data mining).
 The early development of data collection and database creation
mechanisms served as a prerequisite for the later development of
effective mechanisms for data storage and retrieval, as well as query
and transaction processing.

Now we will talk according to the figure one by one.

According to figure, in 1960s, database and information technology has

evolved systematically from primitive file processing systems to
sophisticated and powerful database systems.
 The research and development in database systems started in 1970s.

In addition, users gained convenient and flexible data access through

query languages, user interfaces, query optimization, and transaction
management.

Efficient methods for online transaction processing (OLTP), where a

query is viewed as a read-only transaction become a major tool for
accessing data, retrieval, and management of large amounts of data.
 These systems incorporate new and powerful data models such as extended-
relational, object-oriented, object-relational, and deductive models.

Application-oriented database systems have flourished, including spatial,

temporal, multimedia, active, stream and sensor, scientific and engineering
databases, knowledge bases, and office information bases.

Issues related to the distribution, diversification, and sharing of data have

been studied extensively.
 One emerging data repository architecture is the Data Warehouse.

This is a repository of multiple heterogeneous data sources organized

under a unified schema at a single site to facilitate management
decision making.

Data warehouse technology includes data cleaning, data integration,

and online analytical processing (OLAP)that is, analysis techniques
with functionalities such as summarization, consolidation, and
aggregation, as well as the ability to view information from different
angles.
 Huge volumes of data have been accumulated beyond databases and
data warehouses.

During the 1990s, the World Wide Web and web-based databases (e.g.,
XML databases) began to appear.

Internet-based global information bases, such as the WWW and

various kinds of interconnected, heterogeneous databases, have
emerged and play a vital role in the information industry.

The effective and efficient analysis of data from such different forms of
data by integration of information retrieval, data mining, and
information network analysis technologies is a challenging task.
 To know the importance of data warehouse, we have a good example for
this.

In Google we always browse for Facebook, YouTube, Movies, Twitter and

so on.

But have you ever browsed these things in Google.?

How many soldiers were killed in Kargil war?
Important persons born in 1977
Guess who is this ?
 Have you typed your name in Google and have you seen what images
will get?

Abhilash Thulsidas --------

Sister Vela --------------------

Sister Arul --------

Don Mannul --------

 In summary, the abundance of data, coupled with the need for
powerful data analysis tools, has been described as a data rich but
information poor situation.

The fast-growing, tremendous amount of data, collected and stored in

large and numerous data repositories, has far exceeded our human
ability for comprehension without powerful tools.

The widening gap between data and information calls for the systematic
development of data mining tools that can turn data tombs into
golden nuggets of knowledge.
1.2 What Is Data Mining?

Data Mining can be defined in many different ways.

Analogously, data mining should have been more appropriately named

knowledge mining from data, which is unfortunately somewhat long.

Nevertheless, mining is a vivid term characterizing the process that

finds a small set of precious nuggets from a great deal of raw material.
 In addition, many other terms have a similar meaning to data mining
for example, knowledge mining from data, knowledge extraction,
data/pattern analysis, data archaeology, and data dredging.

Many people treat data mining as a synonym for another popularly

used term, Knowledge Discovery from Data, or KDD.

The knowledge discovery process is shown in Figure as an iterative

sequence of the following steps:
1. Data Cleaning - to remove noise and inconsistent data.

2. Data Integration - where multiple data sources may be combined.

3. Data Selection - where data relevant to the analysis task are retrieved
from the database.

4. Data Transformation - where data are transformed and consolidated

into forms appropriate for mining by performing summary or aggregation
operations.

5. Data Mining - an essential process where intelligent methods are

applied to extract data patterns.

6. Pattern Evaluation - to identify the truly interesting patterns

representing knowledge based on interestingness measures.

7. Knowledge Presentation - where visualization and knowledge

representation techniques are used to present mined knowledge to users.
 Therefore, Data Mining is the process of discovering interesting
patterns and knowledge from large amounts of data.

The data sources can include databases, data warehouses, the Web,
other information repositories, or data that are streamed into the
system dynamically.
1.3 - What Kinds of Data Can Be Mined?

As a general technology, data mining can be applied to any kind of data

as long as the data are meaningful for a target application.

The most basic forms of data for mining applications are Database
Data, Data Warehouse Data and Transactional Data.
1.3.1 - Database Data

A database system, also called a database management system (DBMS),

consists of a collection of interrelated data, known as a database, and a
set of software programs to manage and access the data.

The software programs provide mechanisms for defining database, data

storage, data specifying and managing, distributed data access, and for
ensuring consistency and security of the information.

A relational database is a collection of tables, each of which is assigned

a unique name.

Each table consists of a set of attributes (columns or fields) and usually

stores a large set of tuples (records or rows).

Each tuple in a relational table represents an object identified by a unique

key and described by a set of attribute values.

A semantic data model, such as an entity-relationship (ER) data model,

is often constructed for relational databases.
Example: A relational database for AllElectronics

The fictitious AllElectronics store is used to illustrate concepts

throughout this chapter.

The company is described by the following relation tables: customer,

item, employee and branch.

The headers of the tables described here are shown in Figure.

 The relation customer consists of a set of attributes describing the customer
information, including a unique customer identity number (cust ID),
customer name, address, age, occupation, annual income, credit
information, and category.

Similarly, each of the relations item, employee, and branch consists of a set
of attributes describing the properties of these entities.

Tables can also be used to represent the relationships between or among

multiple entities.

In our example, these include purchases (customer purchases items,

creating a sales transaction handled by an employee), items sold (lists items
sold in a given transaction), and works at (employee works at a branch of
AllElectronics).
 Relational data can be accessed by database queries written in a
relational query language.

A given query is transformed into a set of relational operations, such as

join, selection, and projection, and is then optimized for efficient
processing.

A query allows retrieval of specified subsets of the data.

Suppose that your job is to analyze the AllElectronics data. Through

the use of relational queries, you can ask things like Show me a list
of all items that were sold in the last quarter.

we can also use some aggregate functions such as sum, avg

(average), count, max (maximum), and min (minimum) and we may
ask quires like as fallows.

Show me the total sales of the last month, grouped by branch, or

How many sales transactions occurred in the month of December? or
Which salesperson had the highest sales?
 Relation database quires can be do in any format of examples. For
example,

All students are requested to sit in ONE group.

Now you have only one group, lets make separate from different relations.

Here who are all from Kerala ?

Who knows both Kannada and Malayalam languages ?

Which person name end with letter a ..?

Who will be always late for class ..?

Who has a confidence that he/she will fail atleast one subject in this
semester ?

Now look at the group..

 When mining relational databases, we can go further by searching for
trends or data patterns.

Relational databases are one of the most commonly available and

richest information repositories, and thus they are a major data form
in the study of data mining.
1.3.2 Data Warehouses

Suppose that AllElectronics is a successful international company with

branches around the world.

Each branch has its own set of databases.

The president of AllElectronics has asked you to provide an analysis of

the companys sales per item type per branch for the third quarter.

This is a difficult task, particularly since the relevant data are spread
out over several databases physically located at numerous sites.

For this the solution is, if AllElectronics had a Data Warehouse, this
task would be easy.
 In the previous slide we said that if AllElectronics company asks for the
third quarter report it will be difficult.

To know how it will be difficult, we have an example.

Can you say how many students are there in Depaul College ..?

Can you say how may students have taken BCA, B.Com, BBA & BA ..?

Now all two above queries are of Depaul College

Next question is the big problem .

How many students are studying only BCA in India .. ?

 A Data Warehouse is a repository of information collected from
multiple sources, stored under a unified schema, and usually residing
at a single site.

Data warehouses are constructed via a process of data cleaning, data

integration, data transformation, data loading, and periodic data
refreshing.

Figure shows the typical framework for construction and use of a data
warehouse for AllElectronics.
 To facilitate decision making, the data in a data warehouse are
organized around major subjects (e.g., customer, item, supplier, and
activity).

The data are stored to provide information from a historical perspective,

such as in the past 6 to 12 months, and are typically summarized.

For example, rather than storing the details of each sales transaction,
the data warehouse may store a summary of the transactions per item
type for each store or, summarized to a higher level, for each sales
region.

A Data Warehouse is usually modeled by a multidimensional data

structure, called a data cube, in which each dimension corresponds
to an attribute or a set of attributes in the schema, and each cell
stores the value of some aggregate measure such as count or sum.

A data cube provides a multidimensional view of data and allows the

precomputation and fast access of summarized data.
 Example : A data cube for AllElectronics - A data cube for
summarized sales data of AllElectronics is presented in Figure (a).
 The cube has three dimensions: address (with city values Chicago, New
York, Toronto, Vancouver), time (with quarter values Q1, Q2, Q3, Q4),
and item (with item type values home entertainment, computer, phone,
security).
 The aggregate value stored in each cell of the cube is sales amount (in
thousands).

For example, the total sales for the first quarter, Q1, for the items
related to security systems in Vancouver is $400 as stored in cell
(Vancouver, Q1, security).
 By providing multidimensional data views and the precomputation of
summarized data, data warehouse systems can provide inherent
support for OLAP.

Online Analytical Processing operations make use of background

knowledge regarding the domain of the data being studied to allow the
presentation of data at different levels of abstraction.

Such operations accommodate different user viewpoints.

Examples of OLAP operations include drill-down and roll-up, which

allow the user to view the data at differing degrees of summarization,
as illustrated in Figure 1.7(b).
 Again I am repeating the same point, Examples of OLAP operations
include drill-down and roll-up, which allow the user to view the data at
differing degrees of summarization, as illustrated in Figure 1.7(b).

Here I told summarization it means compress.

Do you have any idea how we can compress the above diagram details ?
 For instance, we can drill down on sales data summarized by quarter to
see data summarized by month.

Similarly, we can roll up on sales data summarized by city to view data

summarized by country.
1.3.3 - Transactional Data

In general, each record in a transactional database captures a

transaction, such as a customers purchase, a flight booking, or a
users clicks on a web page.

A transaction typically includes a unique transaction identity number

(trans ID) and a list of the items making up the transaction, such as
the items purchased in the transaction.

A transactional database may have additional tables, which contain

other information related to the transactions, such as item description,
information about the salesperson or the branch, and so on.
Example 1.4 :A transactional database for AllElectronics.

Transactions can be stored in a table, with one record per transaction.

A fragment of a transactional database for AllElectronics is shown in

Figure 1.8.
 As an analyst of AllElectronics, you may ask, Which items sold well
together?

This kind of market basket data analysis would enable you to bundle
groups of items together as a strategy for boosting sales.

If we have market basket data analysis type we have many advantages

like

Example 1, given the knowledge that printers are commonly purchased

together with computers, you could offer certain printers at a step
discount (or even for free) to customers buying selected computers, in
the hopes of selling more computers (which are often more expensive
than printers).

Example 2 : Nowadays if people purchase phones they will an offer to

purchase Memory Card etc.
1.3.4 - Other Kinds of Data

Besides Relational Database Data, Data Warehouse Data, and

Transaction Data, there are many other kinds of data that have
versatile forms and structures and rather different semantic meanings.

Such kinds of data can be seen in many applications:

Time-related or Sequence Data (e.g., historical records, stock

exchange data, and time-series and biological sequence data)

Data Streams (e.g., video surveillance and sensor data, which are
continuously transmitted)

Spatial Data (e.g., maps)

 Engineering Design Data (e.g., the design of buildings, system
components, or integrated circuits)

Hypertext And Multimedia Data (including text, image, video, and

audio data)

Graph And Networked Data (e.g., social and information networks)

Web (a huge, widely distributed information repository made available

by the Internet)

These applications bring about new challenges like,

How to handle data carrying special structures (e.g., sequences, trees,

graphs, and networks)

Specific semantics (such as ordering, image, audio and video contents,

and connectivity), and how to mine patterns that carry rich structures
and semantics.
1.4 - What Kinds of Patterns Can Be Mined?

There are a number of Data Mining Functionalities.

Data mining functionalities are used to specify the kinds of patterns to

be found in data mining tasks.

In general, such tasks can be classified into two categories:

Descriptive and Predictive.

Descriptive mining tasks characterize properties of the data in a target

data set.

Predictive mining tasks perform induction on the current data in order

to make predictions.
1.4.1-Class/Concept Description: Characterization and Discrimination

Data entries can be associated with classes or concepts.

For example, in the AllElectronics store, classes of items for sale include
computers and printers, and concepts of customers include bigSpenders
and budgetSpenders.

It can be useful to describe individual classes and concepts in

summarized, concise, and yet precise terms.

Such descriptions of a class or a concept are called Class/Concept

Descriptions.
 These descriptions can be derived using

(1) Data Characterization, by summarizing the data of the class under

study (often called the target class) in general terms.

(2) Data Discrimination, by comparison of the target class with one or a

set of comparative classes (often called the contrasting classes)

(3) both data characterization and discrimination.

() The output of data characterization can be presented in various forms

like pie charts, bar charts, curves, multidimensional data cubes, and
multidimensional tables etc.
Seminar Time

1.4.2 Mining Frequent Patterns, Associations, and Correlations

1.4.3 Classification and Regression for Predictive Analysis

1.4.4 Cluster Analysis

We have a separate chapter on this we will study later.

1.4.5 Outlier Analysis

A data set may contain objects that do not comply with the general
behavior or model of the data.

These data objects are outliers.

Many data mining methods discard outliers as noise or exceptions.

The analysis of outlier data is referred to as outlier analysis or

anomaly mining.
1.4.6 - Are All Patterns Interesting?

A data mining system has the potential to generate thousands or even

millions of patterns, or rules.

You may ask, Are all of the patterns interesting?

Typically, the answer is Noonly a small fraction of the patterns

potentially generated would actually be of interest to a given user.

This raises some serious questions for data mining.

You may wonder, What makes a pattern interesting? Can a data

mining system generate all of the interesting patterns? Or, Can the
system generate only the interesting ones?
 To answer the first question, a pattern is interesting if it is

(1) Easily understood by humans.

(2) Valid on new or test data with some degree of certainty.

(3) Potentially useful.

(4) Novel (new and original, not like anything seenbefore) .

()An interesting pattern represents knowledge.

3.2 Data Cleaning

Real-world data tend to be incomplete, noisy, and inconsistent.

Data cleaning (or data cleansing) routines attempt to fill in missing

values, smooth out noise while identifying outliers, and correct
inconsistencies in the data.
3.2.1 Missing Values

Imagine that you need to analyze AllElectronics sales and customer

data.

You note that many tuples have no recorded value for several attributes
such as customer income.

How can you go about filling in the missing values for this attribute?

Lets look at the following methods.

1. Ignore the tuple:

. This is usually done when the class label is missing.

. This method is not very effective, unless the tuple contains several
attributes with missing values.

. It is especially poor when the percentage of missing values per

attribute varies considerably.

. By ignoring the tuple, we do not make use of the remaining attributes

values in the tuple.
2. Fill in the missing value manually:

In general, this approach is time consuming and may not be feasible

given a large data set with many missing values.

3. Use a global constant to fill in the missing value:

Decide on a new global constant value, like unknown, N/A orminus

infinity, that will be used to fill all the missing values.

This technique is used because sometimes it just doesnt make sense to

try and predict the missing value.

For example, lets look at the students enrollment database. If we have

a tuple of Pan Card number, number of students doesnt have Pan Card
so in this case we can enter N/A data into this tuple.
 Use a measure of central tendency for the attribute (e.g., the mean or
median) to
fill in the missing value: Chapter 2 discussed measures of central tendency,
which
indicate the middle value of a data distribution. For normal (symmetric) data
distributions,
the mean can be used, while skewed data distribution should employ
the median (Section 2.2). For example, suppose that the data distribution
regarding
the income of AllElectronics customers is symmetric and that the mean income is
$56,000. Use this value to replace the missing value for income.
5. Use the attribute mean or median for all samples belonging to the
same class as
the given tuple: For example, if classifying customers according to credit risk, we
may replace the missing value with the mean income value for customers in the
same
credit risk category as that of the given tuple. If the data distribution for a given
class
is skewed, the median value is a better choice.
6. Use the most probable value to fill in the missing value: This may be
determined
with regression, inference-based tools using a Bayesian formalism, or decision tree
3.2.2 Noisy Data

What is noise? Noise is a random error or variance in a measured

variable.

How can we smooth out the data to remove the noise?

Lets look at the following data smoothing techniques.

 Binning:

Binning methods smooth a sorted data value by consulting its

neighborhood, that is, the values around it.

The sorted values are distributed into a number of buckets, or bins.

Because binning methods consult the neighborhood of values, they

perform local smoothing.
 Figure illustrates some binning techniques.
 In this example, the data for price are first sorted and then partitioned
into equal-frequency bins of size 3 (i.e., each bin contains three values).

In smoothing by bin means, each value in a bin is replaced by the

mean value of the bin.

For example, the mean of the values 4, 8, and 15 in Bin 1 is 9.

Therefore, each original value in this bin is replaced by the value 9.

 In smoothing by bin boundaries, the minimum and maximum values
in a given bin are identified as the bin boundaries.

Each bin value is then replaced by the closest boundary value.

In general, the larger the width, the greater the effect of the smoothing.
 Regression:

Data smoothing can also be done by regression, a technique that

conforms data values to a function.

Linear regression involves finding the best line to fit two attributes (or
variables) so that one attribute can be used to predict the other.

Multiple linear regression is an extension of linear regression, where

more than two attributes are involved and the data are fit to a
multidimensional surface.
 Outlier analysis:

Outliers may be detected by clustering, for example, where similar

values are organized into groups, or clusters.

Intuitively, values that fall outside of the set of clusters may be

considered outliers
3.2.3 Data Cleaning as a Process

Missing values, noise, and inconsistencies contribute to inaccurate

data.

So far, we have looked at techniques for handling missing data and for
smoothing data.

But data cleaning is a big job. What about data cleaning as a process?
How exactly does one proceed in tackling this task? Are there any tools
out there to help?
 The first step in data cleaning as a process is Discrepancy Detection.
(finding variations in data).

Discrepancies can be caused by several factors, including poorly

designed data entry forms that have many optional fields, human error
in data entry, data integration (e.g., where a given attribute can have
different names in different databases) and data decay (e.g., outdated
addresses).

Other sources of discrepancies include errors in instrumentation

devices that record data and system errors.

Errors can also occur when the data are (inadequately) used for
purposes other than originally intended.
 So, how can we proceed with discrepancy detection? As a starting
point, use any knowledge you may already have regarding properties of
the data.

Such knowledge or data about data is referred to as Metadata.

This is where we can make use of the knowledge we gained about our
data.

Here we will keep on questioning about data.

In this step, you may write your own scripts and/or use some of the
tools. From this, you may find noise, outliers, and unusual values that
need investigation.

Further explanation about this topic will be done by Sr. Arul Devika