Data Mining:

Concepts and Techniques

— Chapter 2 —

December 7, 2021 Data Mining: Concepts and Techniques 1

 Chapter 2: Data Preprocessing

 Why preprocess the data?

 Descriptive data summarization
 Data cleaning
 Data integration and transformation
 Data reduction
 Discretization and concept hierarchy generation
 Summary
December 7, 2021 Data Mining: Concepts and Techniques 2
 Why Data Preprocessing?
 Data in the real world is dirty
 incomplete: lacking attribute values, lacking

certain attributes of interest, or containing

only aggregate data
 e.g., occupation=“ ”
 noisy: containing errors or outliers
 e.g., Salary=“-10”
 inconsistent: containing discrepancies in codes
or names
 e.g., Age=“42” Birthday=“03/07/1997”
 e.g., Was rating “1,2,3”, now rating “A, B, C”
 e.g., discrepancy between duplicate records
December 7, 2021 Data Mining: Concepts and Techniques 3
 Why Is Data Dirty?
 Incomplete data may come from
 “Not applicable” data value when collected
 Different considerations between the time when the data was
collected and when it is analyzed.
 Human/hardware/software problems
 Noisy data (incorrect values) may come from
 Faulty data collection instruments
 Human or computer error at data entry
 Errors in data transmission
 Inconsistent data may come from
 Different data sources
 Functional dependency violation (e.g., modify some linked data)
 Duplicate records also need data cleaning
December 7, 2021 Data Mining: Concepts and Techniques 4
 Why Is Data Preprocessing Important?

 No quality data, no quality mining results!

 Quality decisions must be based on quality data
 e.g., duplicate or missing data may cause incorrect or even
misleading statistics.
 Data warehouse needs consistent integration of quality
data
 Data extraction, cleaning, and transformation comprises
the majority of the work of building a data warehouse

December 7, 2021 Data Mining: Concepts and Techniques 5

 Multi-Dimensional Measure of Data Quality

 A well-accepted multidimensional view:

 Accuracy

 Completeness

 Consistency

 Timeliness

 Believability

 Value added

 Interpretability

 Accessibility

 Broad categories:
 Intrinsic, contextual, representational, and accessibility

December 7, 2021 Data Mining: Concepts and Techniques 6

 Major Tasks in Data Preprocessing
 Data cleaning
 Fill in missing values, smooth noisy data, identify or remove
outliers, and resolve inconsistencies
 Data integration
 Integration of multiple databases, data cubes, or files
 Data transformation
 Normalization and aggregation
 Data reduction
 Obtains reduced representation in volume but produces the same
or similar analytical results
 Data discretization
 Part of data reduction but with particular importance, especially
for numerical data

December 7, 2021 Data Mining: Concepts and Techniques 7

 Forms of Data Preprocessing

December 7, 2021 Data Mining: Concepts and Techniques 8

 Chapter 2: Data Preprocessing

 Why preprocess the data?

 Descriptive data summarization
 Data cleaning
 Data integration and transformation
 Data reduction
 Discretization and concept hierarchy generation
 Summary
December 7, 2021 Data Mining: Concepts and Techniques 9
 Mining Data Descriptive Characteristics

 Motivation
 To better understand the data: central tendency, variation
and spread
 Data dispersion characteristics
 median, max, min, quantiles, outliers, variance, etc.
 Numerical dimensions correspond to sorted intervals
 Data dispersion: analyzed with multiple granularities of
precision
 Boxplot or quantile analysis on sorted intervals
 Dispersion analysis on computed measures
 Folding measures into numerical dimensions
 Boxplot or quantile analysis on the transformed cube
December 7, 2021 Data Mining: Concepts and Techniques 10
 Measuring the Central Tendency


1 n x
 Mean (algebraic measure) (sample vs. population): x   xi
n i 1 N
n
 Weighted arithmetic mean: w x i i
x i 1
 Trimmed mean: chopping extreme values n

w
i 1
i

 Median: A holistic measure

 Middle value if odd number of values, or average of the middle two
values otherwise
 Estimated by interpolation (for grouped data): n / 2  ( f )l
median  L1  ( )c
 Mode f median
 Value that occurs most frequently in the data
 Unimodal, bimodal, trimodal
 Empirical formula: mean  mode  3  (mean  median)
December 7, 2021 Data Mining: Concepts and Techniques 11
 Symmetric vs. Skewed Data

 Median, mean and mode of symmetric,

positively and negatively skewed data

December 7, 2021 Data Mining: Concepts and Techniques 12

 Measuring the Dispersion of Data
 Quartiles, outliers and boxplots
 Quartiles: Q1 (25th percentile), Q3 (75th percentile)
 Inter-quartile range: IQR = Q3 – Q1
 Five number summary: min, Q1, M, Q3, max
 Boxplot: ends of the box are the quartiles, median is marked, whiskers, and
plot outlier individually
 Outlier: usually, a value higher/lower than 1.5 x IQR
 Variance and standard deviation (sample: s, population: σ)
 Variance: (algebraic, scalable computation)
1 n 1 n 2 1 n 1 n
1 n
s 
2

n  1 i 1
( xi  x ) 
2
[ xi  ( xi ) 2 ]
n  1 i 1 n i 1
 2

N

i 1
( xi  
2
) 
N
 xi   2
i 1
2

 Standard deviation s (or σ) is the square root of variance s2 (or σ2)

December 7, 2021 Data Mining: Concepts and Techniques 13

 Properties of Normal Distribution Curve
 The normal (distribution) curve
 From μ–σ to μ+σ: contains about 68% of the

measurements (μ: mean, σ: standard deviation)

 From μ–2σ to μ+2σ: contains about 95% of it
 From μ–3σ to μ+3σ: contains about 99.7% of it

December 7, 2021 Data Mining: Concepts and Techniques 14

 Boxplot Analysis

 Five-number summary of a distribution:

Minimum, Q1, M, Q3, Maximum
 Boxplot
 Data is represented with a box
 The ends of the box are at the first and third
quartiles, i.e., the height of the box is IQR
 The median is marked by a line within the box
 Whiskers: two lines outside the box extend to
Minimum and Maximum

December 7, 2021 Data Mining: Concepts and Techniques 15

Visualization of Data Dispersion: Boxplot Analysis

December 7, 2021 Data Mining: Concepts and Techniques 16

 Histogram Analysis

 Graph displays of basic statistical class descriptions

 Frequency histograms

 A univariate graphical method

 Consists of a set of rectangles that reflect the counts or
frequencies of the classes present in the given data

December 7, 2021 Data Mining: Concepts and Techniques 17

 Quantile Plot
 Displays all of the data (allowing the user to assess both
the overall behavior and unusual occurrences)

December 7, 2021 Data Mining: Concepts and Techniques 18

 Quantile-Quantile (Q-Q) Plot
 Graphs the quantiles of one univariate distribution against
the corresponding quantiles of another
 Allows the user to view whether there is a shift in going
from one distribution to another

December 7, 2021 Data Mining: Concepts and Techniques 19

 Scatter plot
 Provides a first look at bivariate data to see clusters of
points, outliers, etc
 Each pair of values is treated as a pair of coordinates and
plotted as points in the plane

December 7, 2021 Data Mining: Concepts and Techniques 20

 Loess Curve
 Adds a smooth curve to a scatter plot in order to
provide better perception of the pattern of dependence
 Loess curve is fitted by setting two parameters: a
smoothing parameter, and the degree of the
polynomials that are fitted by the regression

December 7, 2021 Data Mining: Concepts and Techniques 21

 Positively and Negatively Correlated Data

December 7, 2021 Data Mining: Concepts and Techniques 22

 Not Correlated Data

December 7, 2021 Data Mining: Concepts and Techniques 23

 Chapter 2: Data Preprocessing

 Why preprocess the data?

 Descriptive data summarization
 Data cleaning
 Data integration and transformation
 Data reduction
 Discretization and concept hierarchy generation
 Summary
December 7, 2021 Data Mining: Concepts and Techniques 24
 Data Cleaning
 Importance
 “Data cleaning is one of the three biggest problems

in data warehousing”—Ralph Kimball

 “Data cleaning is the number one problem in data

warehousing”—DCI survey
 Data cleaning tasks
 Fill in missing values
 Identify outliers and smooth out noisy data
 Correct inconsistent data
 Resolve redundancy caused by data integration
December 7, 2021 Data Mining: Concepts and Techniques 25
 Missing Data

 Data is not always available

 E.g., many tuples have no recorded value for several
attributes, such as customer income in sales data
 Missing data may be due to
 equipment malfunction
 inconsistent with other recorded data and thus deleted
 data not entered due to misunderstanding
 certain data may not be considered important at the time of
entry
 not register history or changes of the data
 Missing data may need to be inferred.

December 7, 2021 Data Mining: Concepts and Techniques 26

 How to Handle Missing Data?
 Ignore the tuple: usually done when class label is missing (assuming
the tasks in classification—not effective when the percentage of
missing values per attribute varies considerably.
 Fill in the missing value manually: tedious + infeasible?
 Fill in it automatically with
 a global constant : e.g., “unknown”, a new class?!
 the attribute mean
 the attribute mean for all samples belonging to the same class:
smarter
 the most probable value: inference-based such as Bayesian formula
or decision tree

December 7, 2021 Data Mining: Concepts and Techniques 27

 Noisy Data
 Noise: random error or variance in a measured variable
 Incorrect attribute values may due to
 faulty data collection instruments

 data entry problems

 data transmission problems

 technology limitation

 inconsistency in naming convention

 Other data problems which requires data cleaning

 duplicate records

 incomplete data

 inconsistent data

December 7, 2021 Data Mining: Concepts and Techniques 28

 How to Handle Noisy Data?
 Binning
 first sort data and partition into (equal-frequency) bins

 then one can smooth by bin means, smooth by bin

median, smooth by bin boundaries, etc.

 Regression
 smooth by fitting the data into regression functions

 Clustering
 detect and remove outliers

 Combined computer and human inspection

 detect suspicious values and check by human (e.g.,

deal with possible outliers)

December 7, 2021 Data Mining: Concepts and Techniques 29

 Data Cleaning as a Process
 Data discrepancy detection
 Use metadata (e.g., domain, range, dependency, distribution)

 Check field overloading

 Check uniqueness rule, consecutive rule and null rule

 Use commercial tools

 Data scrubbing: use simple domain knowledge (e.g., postal

code, spell-check) to detect errors and make corrections

 Data auditing: by analyzing data to discover rules and

relationship to detect violators (e.g., correlation and clustering

to find outliers)
 Data migration and integration
 Data migration tools: allow transformations to be specified

 ETL (Extraction/Transformation/Loading) tools: allow users to

specify transformations through a graphical user interface

 Integration of the two processes
 Iterative and interactive (e.g., Potter’s Wheels)

December 7, 2021 Data Mining: Concepts and Techniques 30

 Chapter 2: Data Preprocessing

 Why preprocess the data?

 Data cleaning
 Data integration and transformation
 Data reduction
 Discretization and concept hierarchy generation
 Summary

December 7, 2021 Data Mining: Concepts and Techniques 31

 Data Integration
 Data integration:
 Combines data from multiple sources into a coherent store
 Schema integration: e.g., A.cust-id  B.cust-#
 Integrate metadata from different sources
 Entity identification problem:
 Identify real world entities from multiple data sources, e.g., Bill Clinton =
William Clinton
 Detecting and resolving data value conflicts
 For the same real world entity, attribute values from different sources are
different
 Possible reasons: different representations, different scales, e.g., metric vs.
British units

Correlation analysis & chi-square test

December 7, 2021 Data Mining: Concepts and Techniques 32

 Data Transformation

 Smoothing: remove noise from data

 Aggregation: summarization, data cube construction
 Generalization: concept hierarchy climbing
 Normalization: scaled to fall within a small, specified
range
 min-max normalization
 z-score normalization
 normalization by decimal scaling
 Attribute/feature construction
 New attributes constructed from the given ones

December 7, 2021 Data Mining: Concepts and Techniques 33

 Chapter 2: Data Preprocessing

 Why preprocess the data?

 Data cleaning
 Data integration and transformation
 Data reduction
 Discretization and concept hierarchy generation
 Summary

December 7, 2021 Data Mining: Concepts and Techniques 34

 Data Reduction Strategies

 Why data reduction?

 A database/data warehouse may store terabytes of data

 Complex data analysis/mining may take a very long time to run

on the complete data set

 Data reduction
 Obtain a reduced representation of the data set that is much

smaller in volume but yet produce the same (or almost the
same) analytical results
 Data reduction strategies
 Data cube aggregation:

 Dimensionality reduction — e.g., remove unimportant attributes

 Data Compression

 Numerosity reduction — e.g., fit data into models

 Discretization and concept hierarchy generation

December 7, 2021 Data Mining: Concepts and Techniques 35

 Data Compression
 String compression
 There are extensive theories and well-tuned algorithms

 Typically lossless

 But only limited manipulation is possible without

expansion
 Audio/video compression
 Typically lossy compression, with progressive

refinement
 Sometimes small fragments of signal can be

reconstructed without reconstructing the whole

 Time sequence is not audio
 Typically short and vary slowly with time

December 7, 2021 Data Mining: Concepts and Techniques 36

 Data Compression

Original Data Compressed

Data
lossless

ss y
lo
Original Data
Approximated

December 7, 2021 Data Mining: Concepts and Techniques 37

 Dimensionality Reduction:
Wavelet Transformation
Haar2 Daubechie4
 Discrete wavelet transform (DWT): linear signal
processing, multi-resolutional analysis
 Compressed approximation: store only a small fraction of
the strongest of the wavelet coefficients
 Similar to discrete Fourier transform (DFT), but better
lossy compression, localized in space.

December 7, 2021 Data Mining: Concepts and Techniques 38

 DWT for Image Compression
 Image

Low Pass High Pass

Low Pass High Pass

Low Pass High Pass

December 7, 2021 Data Mining: Concepts and Techniques 39

 Dimensionality Reduction: Principal
Component Analysis (PCA)
 Given N data vectors from n-dimensions, find k ≤
n orthogonal vectors (principal components) that
can be best used to represent data
 Works for numeric data only
 Used when the number of dimensions is large

December 7, 2021 Data Mining: Concepts and Techniques 40

 Principal Component Analysis

X2

Y1
Y2

X1

December 7, 2021 Data Mining: Concepts and Techniques 41

 Data Reduction Method (3): Clustering

 Partition data set into clusters based on similarity, and store cluster
representation (e.g., centroid and diameter) only
 Can be very effective if data is clustered but not if data is “smeared”
 Can have hierarchical clustering and be stored in multi-dimensional
index tree structures
 There are many choices of clustering definitions and clustering
algorithms
 Cluster analysis will be studied in depth in Chapter 7

December 7, 2021 Data Mining: Concepts and Techniques 42

 Data Reduction Method (4): Sampling
 Sampling: obtaining a small sample s to represent the
whole data set N
 Allow a mining algorithm to run in complexity that is
potentially sub-linear to the size of the data
 Choose a representative subset of the data
 Simple random sampling may have very poor

performance in the presence of skew

 Develop adaptive sampling methods
 Stratified sampling:

 Approximate the percentage of each class (or

subpopulation of interest) in the overall database

 Used in conjunction with skewed data

 Note: Sampling may not reduce database I/Os (page at a

time)
December 7, 2021 Data Mining: Concepts and Techniques 43
 Sampling: with or without Replacement

W O R
SRS le random
i m p h ou t
( s e wi t
l
samp ment)
pl a c e
re

SRSW
R

Raw Data
December 7, 2021 Data Mining: Concepts and Techniques 44
 Sampling: Cluster or Stratified Sampling

Raw Data Cluster/Stratified Sample

December 7, 2021 Data Mining: Concepts and Techniques 45

 Chapter 2: Data Preprocessing

 Why preprocess the data?

 Data cleaning
 Data integration and transformation
 Data reduction
 Discretization and concept hierarchy generation
 Summary

December 7, 2021 Data Mining: Concepts and Techniques 46

 Discretization

 Three types of attributes:

 Nominal — values from an unordered set, e.g., color, profession
 Ordinal — values from an ordered set, e.g., military or academic
rank
 Continuous — real numbers, e.g., integer or real numbers
 Discretization:
 Divide the range of a continuous attribute into intervals
 Some classification algorithms only accept categorical attributes.
 Reduce data size by discretization
 Prepare for further analysis

December 7, 2021 Data Mining: Concepts and Techniques 47

 Discretization and Concept Hierarchy
 Discretization
 Reduce the number of values for a given continuous attribute by
dividing the range of the attribute into intervals
 Interval labels can then be used to replace actual data values
 Supervised vs. unsupervised
 Split (top-down) vs. merge (bottom-up)
 Discretization can be performed recursively on an attribute
 Concept hierarchy formation
 Recursively reduce the data by collecting and replacing low level
concepts (such as numeric values for age) by higher level concepts
(such as young, middle-aged, or senior)

December 7, 2021 Data Mining: Concepts and Techniques 48

 Discretization and Concept Hierarchy
Generation for Numeric Data
 Typical methods: All the methods can be applied recursively
 Binning (covered above)
 Top-down split, unsupervised,
 Histogram analysis (covered above)
 Top-down split, unsupervised
 Clustering analysis (covered above)
 Either top-down split or bottom-up merge, unsupervised
 Entropy-based discretization: supervised, top-down split
 Interval merging by 2 Analysis: unsupervised, bottom-up merge
 Segmentation by natural partitioning: top-down split, unsupervised

December 7, 2021 Data Mining: Concepts and Techniques 49

 Summary
 Data preparation or preprocessing is a big issue for both
data warehousing and data mining
 Discriptive data summarization is need for quality data
preprocessing
 Data preparation includes
 Data cleaning and data integration
 Data reduction and feature selection
 Discretization
 A lot a methods have been developed but data
preprocessing still an active area of research
December 7, 2021 Data Mining: Concepts and Techniques 50
December 7, 2021 Data Mining: Concepts and Techniques 51