Scribd
Upload
9 views

Lecture Notes

Uploaded by

kyaligonzaerick
Copyright
© © All Rights Reserved
Available Formats
Download as DOC, PDF, TXT or read online on Scribd
9 views

Lecture Notes

Uploaded by

kyaligonzaerick
Copyright
© © All Rights Reserved
Available Formats
Download as DOC, PDF, TXT or read online on Scribd
You are on page 1/ 37

Exploratory Data Analysis

and Descriptive Statistics


Today

• What is descriptive statistics and


exploratory data analysis?

• Basic numerical summaries of data

• Basic graphical summaries of data


“Central Dogma” of Statistics

Probability
Population
Descriptive
Statistics

Sample

Inferential Statistics
EDA
Before making inferences from data it is
essential to examine all your variables.

Why?

To listen to the data:


- to catch mistakes
- to see patterns in the data
- to find violations of statistical assumptions
- to generate hypotheses
…and because if you don’t, you will have trouble later
Types of Data

Categorical Quantitative

binary nominal ordinal discrete continuous

2 categories
more categories
order matters
numerical
uninterrupted
Dimensionality of Data Sets

• Univariate: Measurement made on one variable


per subject

• Bivariate: Measurement made on two variables


per subject

• Multivariate: Measurement made on many


variables per subject
Numerical Summaries of Data

• Central Tendency measures. They are


computed to give a “center” around which the
measurements in the data are distributed.

• Variation or Variability measures. They


describe “data spread” or how far away the
measurements are from the center.
Location: Mean

1. The Mean

To calculate the average x of a set of observations, add


their value and divide by the number of observations:
Location: Median
• Median – the exact middle value

• Calculation:
- If there are an odd number of observations, find the middle value

- If there are an even number of observations, find the


middle two values and average them

• Example
Some data:
Age of participants: 17 19 21 22 23 23 23 38

Median = (22+23)/2 = 22.5


Which Location Measure Is Best?

• Mean is best for symmetric distributions without outliers

• Median is useful for skewed distributions or data


with outliers

012345678910 012345678910

Mean = 3 Mean = 4

Median = 3 Median = 3
Scale: Variance

• Average of squared deviations of values


from the mean
Why Squared Deviations?

• Adding deviations will yield a sum of ?


• Absolute values do not have nice
mathematical properties
• Squares eliminate the negatives

• Result:
– Increasing contribution to the variance as
you go farther from the mean.
Scale: Standard Deviation
• Variance is somewhat arbitrary

• What does it mean to have a variance of


10.8? Or 2.2? Or 1459.092? Or 0.000001?

• Nothing. But if you could “standardize” that


value, you could talk about any variance (i.e.
deviation) in equivalent terms

• Standard deviations are simply the square root


of the variance
Scale: Standard Deviation

1. Score (in the units that are meaningful)


2. Mean
3. Each score’s deviation from the mean
4. Square that deviation
5. Sum all the squared deviations (Sum of Squares)
6. Divide by n-1
7. Square root – now the value is in the units we started with!!!
Scale: Quartiles and IQR
IQR
25% 25% 25% 25%

Q1 Q2 Q3

• The first quartile, Q1, is the value for which 25% of


the observations are smaller and 75% are larger

• Q2 is the same as the median (50% are smaller,


50% are larger)

• Only 25% of the observations are greater than the


third quartile
Percentiles (aka Quantiles)
th
In general the n percentile is a value such that n% of
the observations fall at or below or it

n%

th
Q1 = 25 percentile
th
Median = 50 percentile
th
Q2 = 75 percentile
Graphical Summaries of Data

A (Good) Picture Is
Worth A 1,000 Words
Univariate Data: Histograms
and Bar Plots
• What’s the difference between a histogram and bar plot?
Bar plot
• Used for categorical variables to show frequency or
proportion in each category.
• Translate the data from frequency tables into a
pictorial representation…

Histogram
• Used to visualize distribution (shape, center, range,
variation) of continuous variables
• “Bin size” important
Effect of Bin Size on Histogram
• Simulated 1000 N(0,1) and 500 N(1,1)

Frequency
Frequency
Frequency More on Histograms
• What’s the difference between a frequency
histogram and a density histogram?
More on Histograms
• What’s the difference between a frequency
histogram and a density histogram?
Frequency Histogram Density Histogram
Box Plots
100.0
maximum

66.7 Q
3

IQR
Years

median

Q1
33.3

minimum

0.0
AGE
Variables
Bivariate Data

Variable 1 Variable 2 Display


Categorical Categorical Crosstabs
Stacked Box Plot

Categorical Continuous Boxplot

nuous Continuous Scatterplot Stacked


Box Plot
Multivariate Data
Clustering
• Organize units into clusters
• Descriptive, not inferential
• Many approaches
• “Clusters” always produced

Data Reduction Approaches (PCA)


• Reduce n-dimensional dataset into much smaller number
• Finds a new (smaller) set of variables that retains
most of the information in the total sample
• Effective way to visualize multivariate data
How to Make a Bad Graph
The aim of good data graphics:
Display data accurately and clearly

Some rules for displaying data badly:


– Display as little information as possible
– Obscure what you do show (with chart junk)
– Use pseudo-3d and color gratuitously
– Make a pie chart (preferably in color and 3d)
– Use a poorly chosen scale

From Karl Broman: http://www.biostat.wisc.edu/~kbroman/


Example 1
Example 2
Example 3
Example 4
Example 5
R Tutorial

• Calculating descriptive statistics in R

• Useful R commands for working with


multivariate data (apply and its derivatives)

• Creating graphs for different types of


data (histograms, boxplots, scatterplots)

• Basic clustering and PCA analysis

You might also like