Descriptive Statistics for a sample data like mean, median,

variance and correlation

In the mind of a computer, a data set is any collection of data. It can be anything from an
array to a complete database.

Example of an array:

[99,86,87,88,111,86,103,87,94,78,77,85,86]

Example of a database:

Carname Color Age Speed AutoPass

BMW red 5 99 Y
Volvo black 7 86 Y
VW gray 8 87 N
VW white 7 88 Y
Ford white 2 111 Y
VW white 17 86 Y
Tesla red 2 103 Y
BMW black 9 87 Y
Volvo gray 4 94 N
Ford white 11 78 N
Toyota gray 12 77 N
VW white 9 85 N
Toyota blue 6 86 Y

By looking at the array, we can guess that the average value is probably around 80 or 90, and
we are also able to determine the highest value and the lowest value, but what else can we
do?

And by looking at the database we can see that the most popular color is white, and the oldest
car is 17 years, but what if we could predict if a car had an AutoPass, just by looking at the
other values?

That is what Data mining is for! Analyzing data and predicting the outcome!

In Data mining it is common to work with very large data sets. In this tutorial we will try to
make it as easy as possible to understand the different concepts of data mining, and we will
work with small easy-to-understand data sets.

Page 1
Data Types

To analyze data, it is important to know what type of data we are dealing with.

We can split the data types into three main categories:

 Numerical
 Categorical
 Ordinal

Numerical data are numbers, and can be split into two numerical categories:

 Discrete Data
- numbers that are limited to integers. Example: The number of cars passing by.
 Continuous Data
- numbers that are of infinite value. Example: The price of an item, or the size of an
item

Categorical data are values that cannot be measured up against each other. Example: a color
value, or any yes/no values.

Ordinal data are like categorical data, but can be measured up against each other. Example:
school grades where A is better than B and so on.

By knowing the data type of your data source, you will be able to know what technique to use
when analyzing them.

Data mining - Mean Median Mode

What can we learn from looking at a group of numbers?

In Data mining (and in mathematics) there are often three values that interests us:

 Mean - The average value

 Median - The mid point value
 Mode - The most common value

Example: We have registered the speed of 13 cars:

speed = [99,86,87,88,111,86,103,87,94,78,77,85,86]

What is the average, the middle, or the most common speed value?

Page 2
Mean

The mean value is the average value.

To calculate the mean, find the sum of all values, and divide the sum by the number of
values:

(99+86+87+88+111+86+103+87+94+78+77+85+86) / 13 = 89.77

Example

Use the NumPy mean() method to find the average speed:

import numpy

speed = [99,86,87,88,111,86,103,87,94,78,77,85,86]

x = numpy.mean(speed)

print(x)

Median

The median value is the value in the middle, after you have sorted all the values:

77, 78, 85, 86, 86, 86, 87, 87, 88, 94, 99, 103, 111

It is important that the numbers are sorted before you can find the median.

The NumPy module has a method for this:

Example

Use the NumPy median() method to find the middle value:

import numpy

speed = [99,86,87,88,111,86,103,87,94,78,77,85,86]

x = numpy.median(speed)

print(x)

If there are two numbers in the middle, divide the sum of those numbers by two.

77, 78, 85, 86, 86, 86, 87, 87, 94, 98, 99, 103

(86 + 87) / 2 = 86.5

Page 3
Example

Using the NumPy module:

import numpy

speed = [99,86,87,88,86,103,87,94,78,77,85,86]

x = numpy.median(speed)

print(x)

Mode

The Mode value is the value that appears the most number of times:

99, 86, 87, 88, 111, 86, 103, 87, 94, 78, 77, 85, 86 = 86

Example

Use the SciPy mode() method to find the number that appears the most:

from scipy import stats

speed = [99,86,87,88,111,86,103,87,94,78,77,85,86]

x = stats.mode(speed)

print(x)

Data mining - Standard Deviation

Standard deviation is a number that describes how spread out the values are.

A low standard deviation means that most of the numbers are close to the mean (average)
value.

A high standard deviation means that the values are spread out over a wider range.

Example: This time we have registered the speed of 7 cars:

speed = [86,87,88,86,87,85,86]

The standard deviation is:

0.9

Page 4
Meaning that most of the values are within the range of 0.9 from the mean value, which is
86.4.

Let us do the same with a selection of numbers with a wider range:

speed = [32,111,138,28,59,77,97]

The standard deviation is:

37.85

Meaning that most of the values are within the range of 37.85 from the mean value, which is
77.4.

As you can see, a higher standard deviation indicates that the values are spread out over a
wider range.

The NumPy module has a method to calculate the standard deviation:

Example

Use the NumPy std() method to find the standard deviation:

import numpy

speed = [86,87,88,86,87,85,86]

x = numpy.std(speed)

print(x)

Example
import numpy

speed = [32,111,138,28,59,77,97]

x = numpy.std(speed)

print(x)

Variance

Variance is another number that indicates how spread out the values are.

In fact, if you take the square root of the variance, you get the standard deviation!

Page 5
Or the other way around, if you multiply the standard deviation by itself, you get the
variance!

To calculate the variance you have to do as follows:

1. Find the mean:

(32+111+138+28+59+77+97) / 7 = 77.4

2. For each value: find the difference from the mean:

32 - 77.4 = -45.4
111 - 77.4 = 33.6
138 - 77.4 = 60.6
28 - 77.4 = -49.4
59 - 77.4 = -18.4
77 - 77.4 = - 0.4
97 - 77.4 = 19.6

3. For each difference: find the square value:

(-45.4)2 = 2061.16
(33.6)2 = 1128.96
(60.6)2 = 3672.36
(-49.4)2 = 2440.36
(-18.4)2 = 338.56
(- 0.4)2 = 0.16
(19.6)2 = 384.16

4. The variance is the average number of these squared differences:

(2061.16+1128.96+3672.36+2440.36+338.56+0.16+384.16) / 7 = 1432.2

Luckily, NumPy has a method to calculate the variance:

Example

Use the NumPy var() method to find the variance:

import numpy

speed = [32,111,138,28,59,77,97]

x = numpy.var(speed)

print(x)

Page 6
Standard Deviation

As we have learned, the formula to find the standard deviation is the square root of the
variance:

√1432.25 = 37.85

Or, as in the example from before, use the NumPy to calculate the standard deviation:

Example

Use the NumPy std() method to find the standard deviation:

import numpy

speed = [32,111,138,28,59,77,97]

x = numpy.std(speed)

print(x)

Symbols

Standard Deviation is often represented by the symbol Sigma: σ

Variance is often represented by the symbol Sigma Square: σ2

Data mining - Percentiles

Percentiles are used in statistics to give you a number that describes the value that a given
percent of the values are lower than.

Example: Let's say we have an array of the ages of all the people that lives in a street.

ages = [5,31,43,48,50,41,7,11,15,39,80,82,32,2,8,6,25,36,27,61,31]

What is the 75 percentile? The answer is 43, meaning that 75% of the people are 43 or
younger.

The NumPy module has a method for finding the specified percentile:

Example

Use the NumPy percentile() method to find the percentiles:

import numpy

Page 7
ages = [5,31,43,48,50,41,7,11,15,39,80,82,32,2,8,6,25,36,27,61,31]

x = numpy.percentile(ages, 75)

print(x)

Example

What is the age that 90% of the people are younger than?

import numpy

ages = [5,31,43,48,50,41,7,11,15,39,80,82,32,2,8,6,25,36,27,61,31]

x = numpy.percentile(ages, 90)

print(x)

Correlation

Correlation is the statistical measure that defines to which extent two variables are linearly
related to each other

import numpy as np

x = np.array([1,3,5,7,8,9, 10, 15])

y = np.array([10, 20, 30, 40, 50, 60, 70, 80])

print(np.corrcoef(x, y))

Try

1. Create an array of 6 zeros

2. Create an array of 6 ones
3. Create an array of integers from 1 to 99
4. Create an array of 5 random integers between 0 and 9
5. Create an array of all the odd integers ranging from 1 to 99
6. Create a 2X2 matrix filled with values from 1 to 4
7. Create a 4X3 matrix filled with values from 9 to 17
8. Find mean value of a randomly generated array of size 30
9. Find variance value of a randomly generated array of size 25
10. Find correlation coefficient value of a randomly generated two array x and y of size 5

Page 8
Generate random numbers using Numpy
import numpy as np
np.random.rand(5)

np.random.rand(3, 3)

np.random.randint(1, 1000, size=5)

np.random.normal(0, 1, 10)

import pandas as pa

df=pa.read_csv('/content/sales_data_ex1.csv')

df.shape

df.columns

df.info()

df.count()

df.head(3)

df.tail(3)

df.describe()

Page 9
df1 = pa.DataFrame({'num_legs': [2, 4, 8, 0],

'num_wings': [2, 0, 0, 0],

'num_specimen_seen': [10, 2, 1, 8]},

index=['falcon', 'dog', 'spider', 'fish'])

df1

p.random.seed(1)

df2 = pd.DataFrame({"C" : np.random.randint(low=1, high=100,

size=10),

"D" : np.random.normal(0.0, 1.0,

size=10)

})

df2

from sklearn.datasets import load_iris

data = load_iris()

df = pd.DataFrame(data=data.data,columns=data.feature_names)

data.target_names

Page 10