Course Introduction

Prof. Sourav Saha

 Expectations from the Course?
• Learn tools
• Learn R, Python, etc
• Become programmers
• Become analysts
• Become Data Scientists
 Tools you will expect to encounter

Tool name & Version Functionality & Usability

• Microsoft Excel 2016 • Spreadsheet – formulae based
• SPSS 23.0 • Statistical Tool – GUI based
• R version 3.4
• Programming & Analytics Tool –
• Python version 2.7 command based
• SAS 18
• Enterprise Analytics Tool – Menu based
• Tableau
• Power BI
• Visualization Tool – GUI based
 Assessment
• Class Problems & Assignments • 20%
• Mid-Term (system) • 20%
• Group Project • 20%
• End-Term Examination • 40%
 Computer Program?

Some mysterious
processing
Output

Input
Pick your choice
 What is a Program
• Data Structures + Algorithms
• Data Structure = A Container stores Data
• Algorithm = Logic + Control
 Data Types & Data Structures
• Applications/programs read data, store data temporarily, process it and
finally output results.
• What is data? Numbers, Characters, etc.

Instructions / Logic / Algorithms / Output Data

Input Data
Programs / Applications / Deduction
 Data Types & Data Structures
• Data is classified into data types. e.g. char, float, integer, etc.
• A data type is (i) a domain of allowed values and (ii) a set of
operations on these values.
• System signals an error if wrong operation is performed on data
of a certain type. For example, char x,y,z; z = x*y is
not allowed.
 Data Types & Data Structures
• Examples
Data Type Domain Operations
boolean 0,1 and, or, =, etc.
char ASCII =, <>, <, etc.
integer -maxint to +, -, =, ==, <>,
+maxint <, etc.
 Data Types & Data Structures
• int i,j; i, j can take only integer values and only integer operations can
be carried out on i, j.
• Built-in types: defined within the language e.g. int,float, etc.
• User-defined types: defined and implemented by the user e.g. using
typedef or class.
 Data Types & Data Structures
• Simple Data types: also known as Atomic data types  have no
component parts. E.g. int, char, float, etc.

21 3.14 ‘a’
 Data Types & Data Structures
• Compound Data or Structured Data types: can be broken into component
parts. E.g. an object, array, set, file, etc. Example: a student object.

Name A H M A D
Age 20
Branch C S C
A Component part
 Data Types & Data Structures
• A data structure is a data type whose values
• (i) can be decomposed into a set of component elements each of which is
either simple (atomic) or another data structure
• (ii) include a structure involving the component parts.
 More Data Structure
Possible Structures: Set, Linear, Tree, Graph.

LINEAR
SET

TREE

GRAPH

15
 Functions of Data Structures
• Add
• Index
• Key
• Position
• Priority
• Retrieve
• Modify
• Delete
Which Data Structure or Algorithm is better?
• Must Meet Requirement
• High Performance
• Low System footprint
• Easy to implement
Which Package to Use?
 Agenda
 Present an Overview of what packages or solutions are available in the
market for data analysis
 Understanding as to what is popular today and what are the trends for
tomorrow
 Overview of some individual software packages
 Assess their demand and few features
 Some Definitions
• SPSS: Statistical Package for the Social Sciences (IBM SPSS Statistics these
days)
• SAS: Statistical Application Systems (Just SAS these days)
• Minitab
• Excel
• SPSS, SAS, and Minitab are statistical packages while Excel is a spreadsheet
Available Options for Statistical Analysis

Proprietary Free Software

 Excel
 R
 SPSS

 MINITAB  Python
 SAS
 Weka
 Eviews
 Gretl
 Stata
 What people are using

R (Blue) & SAS (Orange) R (Blue) & Python (Orange)

Scholarly Articles / Research?
 No. of Jobs

Python & SQL are in most demand

Microsoft Excel
 MS Excel
COST PRO
 Individual License for Microsoft Office  Nearly ubiquitous and is often pre-installed on
Professional $350 new computers
 Microsoft Office University Student License: $99  User friendly
 Volume Discounts available for large  Very good for basic descriptive statistics, charts
organizations and universities and plots
 Free Starter Version available on some new PCs CON
 Costs money
 Not sufficient for anything beyond the most
basic statistical analysis
MINITAB
 Minitab

Con Pro
 Costs Money at $1,395 per single  Easy to learn and use
user
 Often taught in schools in
 Unsuitable for very complicated introductory statistics courses
statistical computation and analysis
 Widely used in engineering for
 Not often used in academic process improvement
research
SPSS
 SPSS
COST PRO
 From $1000 to $12000 per license  One of the most widely used statistical
depending on license type. packages in academia and industry
 More powerful then Minitab that is also
CON easy to learn and use
 Very expensive  Has a command line interface in addition
to menu driven user interface
 Not adequate for modeling and cutting
edge statistical analysis
 Complicated – too many options
SAS
 SAS
COST PRO
 Complicated pricing model  Widely accepted as the leader in statistical
analysis and modeling
 $8,500 first year license fee
 Widely used in the industry and academia
CON
 Very flexible and very powerful
 Very very expensive
 Not user friendly
 Steep learning curve
 Relatively poor graphics capabilities
R
 R
PRO COST
 Widely used and accepted in industry  Free / Open Source
and academia
CON
 Very powerful and flexible
 Not user friendly
 Very large user base
 Requires steep learning curve
 Lots of books and manuals
 Several User Interface Shells available
 R brownies
Functionality:
• R provides a wide variety of statistical and graphical techniques, including linear and nonlinear modeling, classical statistical tests, time-
series analysis, classification, clustering, and others.
• R is easily extensible through functions and the R community is noted for its active contributions in terms of packages.
• Many of R's standard functions are written in R itself, which makes it easy for users to follow the algorithmic choices made.
Packages:
• R is highly extensible through the use of user-submitted packages for specific functions or specific features
• R has stronger object-oriented programming facilities than most statistical computing languages.
• Extending R is also eased by its permissive lexical scoping rules.
Graphics:
• Another strength of R is static graphics, which can produce publication-quality graphs, including mathematical symbols.
• Dynamic and interactive graphics are available through additional packages
Installing R
Screenshots
 How Easy to Use?
• Excel very easy to use!
• SPSS and Minitab are relatively easy to use
• SAS a bit more difficult to use but made easier via Enterprise Guide (EG)
• R & Python are mainly command based and requires active learning
• In general most software are easy to use if you learn how to use them! Getting
Started Guides are available for R, SPSS, Minitab, Excel and SAS.
 Descriptive (Summary) Statistics Options

SPSS Minitab

SAS EG

Excel
 Advanced Statistics: Model Building

Excel

SPSS Minitab
SAS

Fertility: Average number of kids.

Infant mortality: deaths per 1000 live births
 30.0% Bar Charts
25.0%

20.0%

Percent
15.0% SPSS SAS

10.0%

5.0%

0.0%
CONDO RANCH SPLIT TWOSTORY
style

Chart of Style
30 Percents

25 30
25
20
20
Percent

15 15 Percents
10
10
5
0
5

RY
T
LI
ND

NC

O
SP
0

ST
CO

RA
CONDO RANCH SPLIT TWOSTORY

O
Style

TW
Percent within all data.

Minitab Excel
Package Assessment
 Usage In Industry Vs Academia
• SPSS and Minitab heavily used in Academia, used in Industry but not a lot
• SAS not heavily used in Academia, heavily used in Industry (most clinical
trials use SAS)
• Excel heavily used in both Academia and Industry
• Both R & Python usages picking up in Academia & Industry fast
 Operating Systems
• R runs on Windows, Macintosh & Linux
• SPSS runs on Windows, Macintosh, and Unix
• Python runs on Linux, Macintosh, Windows and Unix
• Minitab runs mainly on Windows and Macintosh
• SAS runs on Windows, Macintosh, Unix, Linux
• Excel runs mainly on Windows and Macintosh
 Pivot Tables
Very good for displaying information online. Can be very interactive
• Minitab: static, not interactive
• Excel: interactive
• SPSS: interactive
• SAS: interactive
• R: ??
• Python: Absent
 Text Analytics
• SPSS
• SPSS Modeler (different from IBM SPSS Statistics)
• SAS
• SAS Enterprise Miner (different SAS Base and Enterprise Guide)
• R
• With external packages
• Python
• With external libraries
• Minitab
• Not Available
• Excel
• With extension
 Statistical Modelling
A statistical model is a class of mathematical model, which embodies a set of assumptions
concerning the generation of some sample data, and similar data from a larger population.
A statistical model represents, often in considerably idealized form, the data-generating
process”
• There are three purposes for a statistical model:
• Predictions
• Extraction of information
• Description of stochastic structures

R, SAS, SPSS and Minitab are good for statistical modelling

ANOVA

Post- Latin
Product (Software) One-Way Two-Way MANOVA GLM hoc Squares
Tests Analysis

Minitab Yes Yes Yes Yes Yes Yes

R Yes Yes Yes Yes Yes
SAS Yes Yes Yes Yes Yes Yes
SPSS Yes Yes Yes Yes Yes Yes
Excel Yes Add on Add on Add on
Regression

Product (Software) OLS WLS 2SLS NLLS Logistic GLM LAD Stepwise

Minitab Yes Yes No Yes Yes Yes No Yes

R Yes Yes Yes Yes Yes Yes Yes Yes

SAS
Yes Yes Yes Yes Yes Yes Yes Yes
Excel Yes
SPSS Yes Yes Yes Yes Yes Yes No Yes

Ordinary Least Square (OLS); Weighted Least Square (WLS); Two Stage Least Square 2SLS;
Non-Linear Least Square NLLS); General Linear Model (GLM); Least Absolute Deviation regression (LAD)
Time Series Analysis

Cointegration Multivariat
Product ARIMA GARCH Unit root test VAR
test e GARCH

Minitab Yes No No No No
R Yes Yes Yes Yes Yes

SAS Yes Yes Yes Yes Yes Yes

Excel No No No No No No
SPSS Yes Yes No No No No
 Big Data Analytics
• Python
• SAS Enterprise Miner
• IBM SPSS Modeller
• R
 Summary
• DOE: Minitab or SAS
• Power / Sample size calculation: Minitab or SAS
• Best way to store your data: Notepad
• Automatic Update of Output: Minitab or Excel
• SAS very popular in industry
• Pivot table: SAS, SPSS or Excel
• Modelling: SAS, SPSS or Minitab
• Summary statistics: SAS, SPSS or Minitab
 References

https://sites.google.com/site/r4statistics/popularity
http://en.freestatistics.info/
http://lib.stat.cmu.edu/
http://www.comfsm.fm/~dleeling/statistics/notes000.html
Questions