BIG DATA

ANALYTICS

1
 UNIT-01
INTRODUCTION TO BIG DATA

1. INTRODUCTION TO BIGDATA PLATFORM

Introduction Data

Data and Information

 Data are plain facts.
 The word "data" is plural for "datum."
 Data is nothing but facts and statistics stored or free flowing over a network, generally it's raw
and unprocessed.
 When data are processed, organized, structured or presented in a given context so as to make
them useful, they are called Information.
 It is not enough to have data (such as statistics on the economy).
 Data themselves are fairly useless, but when these data are interpreted and processed to
determine its true meaning, they becomes useful and can be called Information.

For example: When you visit any website, they might store you IP address, that is data, in return
they might add a cookie in your browser, marking you that you visited the website, that is data,
your name, it's data, your age, it's data.

• What is Data?
– The quantities, characters, or symbols on which operations are performed by a
computer,
– which may be stored and transmitted in the form of electrical signals and
– recorded on magnetic, optical, or mechanical recording media.

• 3 Actions on Data
– Capture
– Transform
2
 – Store
BigData

• Big Data may well be the Next Big Thing in the IT world.

• Big data burst upon the scene in the first decade of the 21st century.

• The first organizations to embrace it were online and startup firms.

• Firms like Google, eBay, LinkedIn, and Facebook were built around big data from the
beginning.
• Like many new information technologies,
• big data can bring about dramatic cost reductions,
• substantial improvements in the time required to perform a computing task, or

• new product and service offerings.

• Walmart handles more than 1 million customer transactions every hour.
• Facebook handles 40 billion photos from its user base.
• Decoding the human genome originally took 10years to process; now it can be achieved in one
week.

• What is Big Data?

– Big Data is also data but with a huge size.
– Big Data is a term used to describe a collection of data that is huge in size and yet
growing exponentially with time.
– In short such data is so large and complex that none of the traditional data
management tools are able to store it or process it efficiently.

No single definition; here is from Wikipedia:

• Big data is the term for

– a collection of data sets so large and complex that it becomes difficult to process using
on-hand database management tools or traditional data processing applications.

3
Examples of Bigdata

• Following are some the examples of Big Data-

– The New York Stock Exchange generates about one terabyte of new trade data per
day.
– Other examples of Big Data generation includes
• stock exchanges,

• social media sites,

• jet engines, etc.

Types Of Big Data

• BigData could be found in three forms:

1. Structured
2. Unstructured
3. Semi-structured

What is Structured Data?

• Any data that can be stored, accessed and processed in the form of fixed format is
termed as a 'structured' data.
• Developed techniques for working with such kind of data (where the format is well
known in advance) and also deriving value out of it.
• Foreseeing issues of today :
– when a size of such data grows to a huge extent, typical sizes are being in the rage of
multiple zetta bytes.
• Do you know?
• 1021 bytes equal to 1 zettabyte or one billion terabytes forms a zettabyte.
– That is why the name Big Data is given and imagine the challenges involved in its
storage and processing?

4
 • Do you know?
– Data stored in a relational database management system is one example of a
'structured' data.

• An 'Employee' table in a database is an example of Structured Data:

Employee_ID Employee_Name Gender Department Salary_In_lacs

2365 Rajesh Kulkarni Male Finance 650000

3398 Pratibha Joshi Female Admin 650000

7465 Shushil Roy Male Admin 500000

7500 Shubhojit Das Male Finance 500000

7699 Priya Sane Female Finance 550000

Unstructured Data
• Any data with unknown form or the structure is classified as unstructured data.
• In addition to the size being huge,

– un-structured data poses multiple challenges in terms of its processing for

deriving value out of it.
– A typical example of unstructured data is
• a heterogeneous data source containing a combination of simple text files,
images, videos etc.
• Now day organizations have wealth of data available with them but unfortunately,

– they don't know how to derive value out of it since this data is in its raw form or
unstructured format.

• Example of Unstructured data

– The output returned by 'Google Search'

5
Semi-structured Data

• Semi-structured data can contain both the forms of data.

• Semi-structured data as a structured in form

– but it is actually not defined with e.g. a table definition in relational DBMS.
• Example of semi-structured data is

– a data represented in an XML file.

• Personal data stored in an XML file.

<rec>
<name>Prashant Rao</name>
<sex>Male</sex>
<age>35</age>
</rec>
<rec>
<name>Seema R.</name>
<sex>Female</sex>
<age>41</age>

</rec>
<rec>
<name>Satish Mane</name>
<sex>Male</sex>
<age>29</age>
</rec>
<rec>
<name>Subrato Roy</name>
<sex>Male</sex>
<age>26</age>
</rec>
6
<rec>
<name>Jeremiah J.</name>
<sex>Male</sex>
<age>35</age></rec>

Characteristics of BD OR 3Vs of Big Data

• Three Characteristics of Big Data V3s:

1) Volume
 Data quantity
2) Velocity
 Data Speed
3) Variety
 Data Types

Growth of Big Data

7
Storing Big Data

• Analyzing your data characteristics

– Selecting data sources for analysis
– Eliminating redundant data
– Establishing the role of NoSQL
• Overview of Big Data stores
– Data models: key value, graph, document, column-family
– Hadoop Distributed File System (HDFS)

– Hbase
– Hive
Processing Big Data
• Integrating disparate data stores
– Mapping data to the programming framework
– Connecting and extracting data from storage
– Transforming data for processing
– Subdividing data in preparation for Hadoop MapReduce
• Employing Hadoop MapReduce

– Creating the components of Hadoop MapReduce jobs

– Distributing data processing across server farms
– Executing Hadoop MapReduce jobs
– Monitoring the progress of job flows

Why Big Data?

• Growth of Big Data is needed

– Increase of storage capacities

– Increase of processing power
– Availability of data(different data types)
8
 – Every day we create 2.5 quintillion bytes of data; 90% of the data in the world today
has been created in the last two years alone
 Huge storage need in Real Time Applications
– FB generates 10TB daily
– Twitter generates 7TB of data Daily
– IBM claims 90% of today’s stored data was generated in just the last two years.
How Is Big Data Different?
1) Automatically generated by a machine (e.g. Sensor embedded in an engine)
2) Typically an entirely new source of data(e.g. Use of the internet)
3) Not designed to be friendly(e.g. Text streams)
4) May not have much values
– Need to focus on the important part
Big Data sources

• Users

• Application

• Systems
• Sensors

Moved to

• Large and growing files (Big data files)

Risks of Big Data

• Will be so overwhelmed
– Need the right people and solve the right problems
• Costs escalate too fast

– Isn’t necessary to capture 100%

• Many sources of big data is privacy
9
 – self-regulation
– Legal regulation
Leading Technology Vendors

Example Vendors Commonality

IBM – Netezza • MPP architectures

EMC – Greenplum • Commodity Hardware

Oracle – Exadata • RDBMS based

• Full SQL compliance

Basics of Bigdata Platform

 Big Data platform is IT solution which combines several Big Data tools and utilities into one
packaged solution for managing and analyzing Big Data.

 Big data platform is a type of IT solution that combines the features and capabilities of several
big data application and utilities within a single solution.
 It is an enterprise class IT platform that enables organization in developing, deploying,
operating and managing a big data infrastructure /environment.

What is Big Data Platform?

 Big Data Platform is integrated IT solution for Big Data management which combines several
software system, software tools and hardware to provide easy to use tools system to enterprises.
 It is a single one-stop solution for all Big Data needs of an enterprise irrespective of size and
data volume. Big Data Platform is enterprise class IT solution for developing, deploying and
managing Big Data.
 There are several Open source and commercial Big Data Platform in the market with varied
features which can be used in Big Data environment.
10
  Big data platform is a type of IT solution that combines the features and capabilities of several
big data application and utilities within a single solution.
 It is an enterprise class ITplatformthat enables organization in developing, deploying,
operating and managing abig datainfrastructure /environment.
 Big data platform generally consists of big data storage, servers, database, big data
management, business intelligence and other big data management utilities
 It also supports custom development, querying and integration with other systems.
 The primary benefit behind a big data platform is to reduce the complexity of multiple
vendors/ solutions into a one cohesive solution.
 Big data platform are also delivered through cloud where the provider provides an all
inclusive big data solutions and services.

1.1.2.2 Features of Big Data Platform

Here are most important features of any good Big Data Analytics Platform:

a) Big Data platform should be able to accommodate new platforms and tool based on the
business requirement. Because business needs can change due to new technologies or due to
change in business process.

b) It should support linear scale-out

c) It should have capability for rapid deployment

d) It should support variety of data format

e) Platform should provide data analysis and reporting tools

f) It should provide real-time data analysis software

g) It should have tools for searching the data through large data sets

Big data is a term for data sets that are so large or complex that traditional data processing applications are
inadequate.

Challenges include
11
  Analysis,
 Capture,
 Data Curation,
 Search,
 Sharing,
 Storage,
 Transfer,
 Visualization,
 Querying,
 Updating
Information Privacy.
 The term often refers simply to the use of predictive analytics or certain other
advancedmethods to extract value from data, and seldom to a particular size of data set.

 ACCURACY in big data may lead to more confident decision making, and better decisions
can result in greater operational efficiency, cost reduction and reduced risk.
 Big data usually includes data sets with sizes beyond the ability of commonly used
 software tools to capture, curate, manage, and process data within a tolerable elapsed
 time. Big data "size" is a constantly moving target.
 Big data requires a set of techniques and technologies with new forms of integration to
 reveal insights from datasets that are diverse, complex, and of a massive scale

1.1.2.3 List of BigData Platforms

a) Hadoop
b) Cloudera
c) Amazon Web Services
d) Hortonworks
e) MapR
f) IBM Open Platform
g) Microsoft HDInsight
12
 h) Intel Distribution for Apache Hadoop
i) Datastax Enterprise Analytics
j) Teradata Enterprise Access for Hadoop
k) Pivotal HD

a) Hadoop

What is Hadoop?

 Hadoop is open-source, Java based programming framework and server software which is used
to save and analyze data with the help of 100s or even 1000s of commodity servers in a
clustered environment.
 Hadoop is designed to storage and process large datasets extremely fast and in fault
tolerant way.
 Hadoop uses HDFS (Hadoop File System) for storing data on cluster of commodity computers.
If any server goes down it know how to replicate the data and there is no loss of data even in
hardware failure.
 Hadoop is Apache sponsored project and it consists of many software packages which runs
on the top of the Apache Hadoop system.
 Top Hadoop based Commercial Big Data Analytics Platform
 Hadoop provides set of tools and software for making the backbone of the Big Data
analytics system.
 Hadoop ecosystem provides necessary tools and software for handling and analyzing Big
Data.
 On the top of the Hadoop system many applications can be developed and plugged-in to
provide ideal solution for Big Data needs.

b) Cloudera

 Cloudra is one of the first commercial Hadoop based Big Data Analytics Platform
offering Big Data solution.
 Its product range includes Cloudera Analytic DB, Cloudera Operational DB, Cloudera Data
Science & Engineering and Cloudera Essentials.
13
  All these products are based on the Apache Hadoop and provides real-time processing and
analytics of massive data sets.
Website: https://www.cloudera.com
c) Amazon Web Services
 Amazon is offering Hadoop environment in cloud as part of its Amazon Web Services
package.
 AWS Hadoop solution is hosted solution which runs on Amazon’s Elastic Cloud
Compute and Simple Storage Service (S3).
 Enterprises can use the Amazon AWS to run their Big Data processing analytics in the cloud
environment.
 Amazon EMR allows companies to setup and easily scale Apache Hadoop, Spark,
HBase, Presto, Hive, and other Big Data Frameworks using its cloud hosting
environment.

Website: https://aws.amazon.com/emr/

d) Hortonworks
 Hortonworks is using 100% open-source software without any propriety software.
Hortonworks were the one who first integrated support for Apache HCatalog.
 The Hortonworks is a Big Data company based in California.
 This company is developing and supports application for Apache Hadoop.

Hortonworks Hadoop distribution is 100% open source and its enterprise ready with following features:

 Centralized management and configuration of clusters

 Security and data governance are built in feature of the system

 Centralized security administration across the system

Website: https://hortonworks.com/

14
e) MapR
 MapR is another Big Data platform which us using the Unix file system for handling data.
 It is not using HDFS and this system is easy to learn anyone familiar with the Unix
system.
 This solution integrates Hadoop, Spark, and Apache Drill with a real-time data
processing feature.
 Website: https://mapr.com

f) IBM Open Platform

 IBM also offers Big Data Platform which is based on the Hadoop eco-system software.
 IBM well knows company in software and data computing.

It uses the latest Hadoop software and provides following features (IBM Open Platform Features):

 Based on 100% Open source software

 Native support for rolling Hadoop upgrades

 Support for long running applications within YEARN.

 Support for heterogeneous storage which includes HDFS for in-memory and SSD in
addition to HDD

 Native support for Spark, developers can use Java, Python and Scala to written program

 Platform includes Ambari, which is a best tool for provisioning, managing & monitoring
Apache Hadoop clusters

 IBM Open Platform includes all the software of Hadoop ecosystem e.g. HDFS, YARN,
MapReduce, Ambari, Hbase, Hive, Oozie, Parquet, Parquet Format, Pig, Snappy, Solr, Spark,
Sqoop, Zookeeper, Open JDK, Knox, Slider

 Developer can download the trial Docker Image or Native installer for testing and
learning the system

15
  Application is well supported by IBM technology team

Website: https://www.ibm.com/analytics/us/en/technology/hadoop/

g) Microsoft HDInsight
 The Microsoft HDInsight is also based on the Hadoop distribution and it’s a commercial Big
Data platform from Microsoft.
 Microsoft is software giant which is into development of windows operating system for
Desktop users and Server users.
 This is the big Hadoop distribution offering which runs on the Windows and Azure
environment.
 It offer customized, optimized open source Hadoop based analytics clusters which uses Spark,
Hive, MapReduce, HBase, Strom, Kafka and R Server which runs on the Hadoop system on
windows/Azure environment.

Website: https://azure.microsoft.com/en-in/services/hdinsight/

h) Distribution for Apache Hadoop

 Intel also offers its package distribution of Hadoop software which includes company’s Graph
builder and Analytics toolkit.
 This distribution can be purchased with various channel partners and come with support and
yearly subscription.

Website: http://www.intel.com/content/www/us/en/software/intel-distribution-for-apache-
hadoop-software-solutions.html

i) Datastax Enterprise Analytics

 Datastax Enterprise Analytics is another play in the Big Data Analytics platform which offers
its own distribution which is based on Apache Cassandra database management system which
runs on the top of Apache Hadoop installation.
16
  It also included propriety system with a dashboard which is used for security management,
searching data, dashboard for viewing various details and visualization engine.
 It can handle analysis of 10 million data points every second, so it’s a powerful system.

Features:
 It provides powerful indexing, search, analytics and graph functionality into the Big Data
system
 It supports advanced indexing and searching features

 It comes with powerful integrated analytics system

 It provides multi-model support into the platform. It supports key-value, tabular,

JSON/Document and graph data formats. Powerful search features enables the users to get
required data in real-time

Website: http://www.datastax.com/

j) Teradata Enterprise Access for Hadoop

 Teradata Enterprise Access for Hadoop is another player into Big Data Platform and it offers
package Hadoop distribution which again based on Hortonworks distribution.
 Teradata Enterprise Access for Hadoop offers Hardware and software in its Big Data
solution which can be used by enterprise to process its data sets.

Company offers:
 Teradata

 Teradata Aster and

 Hadoop

as part of its package solution.

Website: http://www.teradata.com

17
k) Pivotal HD

Pivotal HD offers is another Hadoop distribution with includes includes database tools Greenplum
and analytics platform Gemfire.

Features:

 It can be installed on-premise and in public clouds

 This system is based on the open source software

 It supports data evolution within the 3 years subscription period.

Indian railways, BMW, China Citic Bank and many other big players are using this distribution of Big
Data Platform

Website: https://pivotal.io/

Open Source Big Data Platform

There are various open-source Big Data Platform which can be used for Big Data handling and data
analytics in real-time environment.

Both small and Big Enterprise can use these tools for managing their enterprise data for getting best value
from their enterprise data.
i) Apache Hadoop

 Apache Hadoop is Big Data platform and software package which is Apache sponsored
project.
 Under Apache Hadoop project various other software is being developed which runs on the top
of Hadoop system to provide enterprise grade data management and analytics solutions to
enterprise.
 Apache Hadoop is open-source, distributed file system which provides data processing and
analysis engine for analyzing large set of data.
 Hadoop can run on Windows, Linux and OS X operating systems, but it is mostly used on
Ubunut and other Linux variants.

18
ii) MapReduce

 The MapReduce engine was originally written by Google and this is the system which enables
the developers to write program which can run in parallel on 100 or even 1000s of computer
nodes to process vast data sets.
 After processing all the job on the different nodes it comes the results and return it to the
program which executed the MapReduce job.
 This software is platform independent and runs on the top of Hadoop ecosystem. It can
process tremendous data at very high speed in Big Data environment.

iii) GridGain

 GridGain is another software system for parallel processing of data just like MapRedue.
GridGain is an alternative of Apache MapReduce.

 GridGain is used for the processing of in-memory data and its is based on Apache Iginte
framework.
 GridGain is compatable with the Hadoop HDFS and runs on the top of Hadoop
ecosystem.
 Then enterprise version of GridGain can be purchased from official website of
GridGain. While free version can be downloaded from GitHub repository.

Website: https://www.gridgain.com/

iv) HPCC Systems

 HPCC Systems stands for "high performance computing cluster” and this system is
developed by LexisNexis Risk Solutions.
 According to the company this software is much faster than Hadoop and can be used in the
cloud environment.
 HPCC Systems is developed in C++ and compiled into binary code for distribution.
 HPCC Systems is open-source, massive parallel processing system which is installed in cluster
19
 to process data in real-time.
 It requires Linux operating system and runs on the commodity servers connected with high-
speed network.
 It is scalable from one node to 1000s of nodes to provide performance and scalability.
 Website: https://hpccsystems.com/

v) Apache Storm

 Apache Storm is a software for real-time computing and distributed processing.

 Its free and open-source software developed at Apache Software foundation. It’s a real- time,
parallel processing engine.
 Apache Storm is highly scalable, fault-tolerant which supports almost all the
programming language.

vi) Apache Strom can be used in:

 Realtime analytics

 Online machine learning

 Continuous computation

 Distributed RPC

 ETL

 And all other places where real-time processing is required.

Apache Strom is used by Yahoo, Twitter, Spotify, Yelp, Flipboard and many other data giants.

Website: http://storm.apache.org/

20
vii) Apache Spark

 Apache Spark is software that runs on the top of Hadoop and provides API for real-time, in-
memory processing and analysis of large set of stored in the HDFS.
 It stores the data into memory for faster processing.
 Apache Spark runs program 100 times faster in-memory and 10 times faster on disk as
compared to the MapRedue.
 Apache Spark is here to faster the processing and analysis of big data sets in Big Data
environment.
 Apache Spark is being adopted very fast by the business to analyze their data set to get real
value of their data.
 Website: http://spark.apache.org/

viii) SAMOA

 SAMOA stands for Scalable Advanced Massive Online Analysis,

 It’s a system for mining the Big Data streams.
 SAMOA is open-source software distributed at GitHub, which can be used as distributed
machine learning framework also.
 Website: https://github.com/yahoo/samoa

Thus, the Big Data industry is growing very fast in 2017 and companies are fast moving their
data to Big Data Platform. There is huge requirement of Big Data in the job market; many
companies are providing training and certifications in Big Data technologies.

*********************

21
 UNIT-II

CHALLENGES OF CONVENTIONAL SYSTEMS

Introduction to Conventional Systems

What is Conventional System?

Conventional Systems.
 The system consists of one or more zones each having either manually operated call points
or automatic detection devices, or a combination of both.
 Big data is huge amount of data which is beyond the processing capacity ofconventional
data base systems to manage and analyze the data in a specific time interval.

Difference between conventional computing and intelligent computing

 The conventional computing functions logically with a set of rules and calculations while
the neural computing can function via images, pictures, and concepts.
 Conventional computing is often unable to manage the variability of data obtained in the real
world.
 On the other hand, neural computing, like our own brains, is well suited to situations that have
no clear algorithmic solutions and are able to manage noisy imprecise data. This allows them to
excel in those areas that conventional computing often finds difficult.

Comparison of Big Data with Conventional Data

Big Data Conventional Data

Huge data sets Data set size in control.

22
Unstructured data such as text, video, and Normally structured data such as numbers
audio. and categories, but it can take other forms as
well.
Hard-to-perform queries and analysis Relatively easy-to-perform queries and
analysis.
Needs a new methodology for analysis. Data analysis can be achieved by using
conventional methods.
Need tools such as Hadoop, Hive, Tools such as SQL, SAS, R, and Excel
Hbase, Pig, Sqoop, and so on. alone may be sufficient.

The aggregated or sampled or filtered data. Raw transactional data.

Used for reporting, basic analysis, and Used for reporting, advanced analysis, and
text mining. Advanced analytics is only in a predictive modeling .
starting stage in big data.
Big data analysis needs both programming Analytical skills are sufficient for conventional
skills (such as Java) and analytical skills to data; advanced analysis tools don’t require
perform analysis. expert programing skills.
Petabytes/exabytes of data. Millions/billions of accounts.

Billions/trillions of transactions. Megabytes/gigabytes of data.

Thousands/millions of accounts. Millions of transactions

Generated by big financial institutions, Facebook, Generated by small enterprises and small banks.
Google, Amazon, eBay,
Walmart, and so on.

2.2 List of challenges of Conventional Systems

The following list of challenges has been dominating in the case Conventional systems in real time
scenarios:

23
1) Uncertainty of Data Management Landscape
2) The Big Data Talent Gap
3) The talent gap that exists in the industry Getting data into the big data platform
4) Need for synchronization across data sources
5) Getting important insights through the use of Big data analytics

1) Uncertainty of Data Management Landscape:

 Because big data is continuously expanding, there are new companies and technologies that
are being developed everyday.

 A big challenge for companies is to find out which technology works bests for them
without the introduction of new risks and problems.
2) The Big Data Talent Gap:
 While Big Data is a growing field, there are very few experts available in this field.
 This is because Big data is a complex field and people who understand the complexity and
intricate nature of this field are far few and between.

3) The talent gap that exists in the industry Getting data into the big data platform:
 Data is increasing every single day. This means that companies have to tackle limitless
amount of data on a regular basis.
 The scale and variety of data that is available today can overwhelm any data practitioner and
that is why it is important to make data accessibility simple and convenient for brand mangers
and owners.

4) Need for synchronization across data sources:

 As data sets become more diverse, there is a need to incorporate them into an analytical
platform.
 If this is ignored, it can create gaps and lead to wrong insights and messages.

5) Getting important insights through the use of Big data analytics:

24
  It is important that companies gain proper insights from big data analytics and it is
important that the correct department has access to this information.
 A major challenge in the big data analytics is bridging this gap in an effective fashion.

Other Three challenges of Conventional systems

Three Challenges That big data face.

1. Data
2. Process
3. Management

1. Data Challenges

Volume

1. The volume of data, especially machine-generated data, is exploding,

2. how fast that data is growing every year, withnew sources of data that are emerging.
3. For example, in the year 2000, 800,000petabytes (PB) of data were stored in the world, and it is
expected to reach 35 zetta bytes (ZB) by2020 (according to IBM).

Social media plays a key role: Twitter generates 7+ terabytes (TB) of data every day. Facebook, 10 TB.
• Mobile devices play a key role as well, as there were estimated 6 billion mobile phones in 2011.
• The challenge is how to deal with the size of Big Data.

Variety, Combining Multiple Data Sets

• More than 80% of today’s information is unstructured and it is typically too big to manage
effectively.
• Today, companies are looking to leverage a lot more•data from a wider variety of sources both inside
and outside the organization.
• Things like documents, contracts, machine data, sensor data, social media, health records, emails,
etc. The list is endless really.

25
Variety•A lot of this data is unstructured, or has a complex structure that’s hard to represent in rows and
columns.
2. Processing

 More than 80% of today’s information isunstructured and it is typically too big to
manage effectively.

 Today, companies are looking to leverage a lot more data from a wider variety of
sources both inside and outside the organization.

 Things like documents, contracts, machine data, sensor data, social media, health
records, emails, etc. The list is endless really.

3. Management

 A lot of this data is unstructured, or has acomplex structure that’s hard to represent in rows
and columns.
Big Data Challenges

– The challenges include capture, duration, storage, search, sharing, transfer,

– analysis, and visualization.

• Big Data is trend to larger data sets

• due to the additional information derivable from analysis of a single large set of related data,

– as compared to separate smaller sets with the same total amount of data, allowing
correlations to be found to

• "spot business trends, determine quality of research, prevent diseases, link legal
citations, combat crime, and determine real-time roadway traffic conditions.”

Challenges of Big Data

26
The following are the five most important challenges of the Big Data

a) Meeting the need for speed

In today’s hypercompetitive business environment, companies not only have to find and analyze the
relevant data they need, they must find it quickly.

b) Visualization helps organizations perform analyses and make decisions much more
rapidly, but the challenge is going through the sheer volumes of data and accessing the level
of detail needed, all at a high speed.

c) The challenge only grows as the degree of granularity increases. One possible solution is
hardware. Some vendors are using increased memory and powerful parallel
processing to crunch large volumes of data extremely quickly

d) Understanding the data

 It takes a lot of understanding to get data in the RIGHT SHAPE so that you can use
 visualization as part of data analysis.

d) Addressing data quality

 Even if you can find and analyze data quickly and put it in the proper context for the
 audience that will be consuming the information, the value of data for DECISION-
MAKING PURPOSES will be jeopardized if the data is not accurate or timely.
This is a challenge with any data analysis.

e) Displaying meaningful results

 Plotting points on a graph for analysis becomes difficult when dealing with extremely
 large amounts of information or a variety of categories of information.
 For example, imagine you have 10 billion rows of retail SKU data that you’re trying to
 compare. The user trying to view 10 billion plots on the screen will have a hard time
 seeing so many data points.
 . By grouping the data together, or “binning,” you can more effectively visualize the data.
27
f) Dealing with outliers
 The graphical representations of data made possible by visualization can communicate
 trends and outliers much faster than tables containing numbers and text.
 Users can easily spot issues that need attention simply by glancing at a chart. Outliers
typically represent about 1 to 5 percent of data, but when you’re working with massive
amounts of data, viewing 1 to 5 percent of the data is rather difficult
 We can also bin the results to both view the distribution of data and see the outliers.
 While outliers may not be representative of the data, they may also reveal previously
 unseen and potentially valuable insights.
 Visual analytics enables organizations to take raw data and present it in a meaningful way
that generates the most value. However, when used with big data, visualization is bound to
lead to some challenges.

INTELLIGENT DATA ANALYSIS

INTRODUCTION TO INTELLIGENT DATA ANALYSIS (IDA)

Intelligent Data Analysis (IDA) is one of the hot issues in the field of artificial
intelligence and information.

What is Intelligent Data Analysis (IDA)? IDA is

… an interdisciplinary study concerned with the effective analysis of data;

… used for extracting useful information from large quantities of online data; extracting desirable
knowledge or interesting patterns from existing databases;

28
  the distillation of information that has been collected, classified, organized, integrated,
abstracted and value-added;

 at a level of abstraction higher than the data, and information on which it is based and can be
used to deduce new information and new knowledge;

 usually in the context of human expertise used in solving problems.

 the distillation of information that has been collected, classified, organized, integrated,
abstracted and value-added;

 at a level of abstraction higher than the data, and information on which it is based and can be
used to deduce new information and new knowledge;

 usually in the context of human expertise used in solving problems.

Goal:

Goal of Intelligent data analysis is to extract useful knowledge, the process demands a combination of
extraction, analysis, conversion, classification, organization, reasoning, and so on.

29
1,3,2 Uses / Benefits of IDA

Intelligent Data Analysis provides a forum for the examination of issues related to the research and
applications of Artificial Intelligence techniques in data analysis across a variety of disciplines and the
techniques include (but are not limited to):

The benefit areas are:

 Data Visualization
 Data pre-processing (fusion, editing, transformation, filtering, sampling)
 Data Engineering
 Database mining techniques, tools and applications
 Use of domain knowledge in data analysis
 Big Data applications
 Evolutionary algorithms
 Machine Learning(ML)
 Neural nets
 Fuzzy logic
 Statistical pattern recognition
 Knowledge Filtering and
 Post-processing

Intelligent Data Analysis (IDA)

30
Why IDA?

 Decision making is asking for information and knowledge

 Data processing can give them

 Multidimensionality of problems is looking for methods for adequate and deep data
processing and analysis

 Epidemiological study (1970-1990)

31
  Sample of examinees died from cardiovascular diseases during the period

 Question: Did they know they were ill?

1 – they were healthy

2 – they were ill (drug treatment, positive clinical and laboratory findings)

1.3.4 Intelligent Data Analysis

Knowledge Acquisition

 The process of eliciting, analyzing, transforming, classifying, organizing and integrating

knowledge and representing that knowledge in a form that can be used in a computer system.

Knowledge in a domain can be expressed as a number of rules

A Rule :

A formal way of specifying a recommendation, directive, or strategy, expressed as "IF premise

THEN conclusion" or "IF condition THEN action".

How to discover rules hidden in the data?

1.3.4 Intelligent Data Examples:

Example of IDA

 Epidemiological study (1970-1990)

 Sample of examinees died from cardiovascular diseases during the period

Question: Did they know they were ill?

1 – they were healthy

2 – they were ill (drug treatment, positive clinical and laboratory findings)

32
Illustration of IDA by using See5

 application.names - lists the classes to which cases may belong and the attributes
used to describe each case.

 Attributes are of two types: discrete attributes have a value drawn from a set of
possibilities, and continuous attributes have numeric values.

 application.data - provides information on the training cases from which See5 will extract
patterns.

 The entry for each case consists of one or more lines that give the values for all
attributes.

 application.data - provides information on the training cases from which See5 will extract
patterns.

 The entry for each case consists of one or more lines that give the values for all
attributes.

 application.test - provides information on the test cases (used for evaluation of results).

 The entry for each case consists of one or more lines that give the values for all
attributes.

Goal 1.1 :
application.names – example
gender:M,F activity:1,2,3
age: continuous smoking:
No, Yes
…

33
Goal:1,2 :
application.data – example
M,1,59,Yes,0,0,0,0,119,73,103,86,247,87,15979,?,?,?,1,73,2.5
M,1,66,Yes,0,0,0,0,132,81,183,239,?,783,14403,27221,19153,23187,1,73,2.6
M,1,61,No,0,0,0,0,130,79,148,86,209,115,21719,12324,10593,11458,1,74,2.5
… …

Result:
Results – example

Rule 1: (cover 26)

gender = M
SBP > 111
oil_fat > 2.9
-> class 1 [0.929]

Rule 1: (cover 26)

gender = M
SBP > 111
oil_fat > 2.9
-> class 1 [0.929]

Rule 4: (cover 14)

smoking = Yes SBP
> 131
glucose > 93
glucose <= 118
oil_fat <= 2.9
-> class 2 [0.938]
34
Rule 15: (cover 2)
SBP <= 111
oil_fat > 2.9

-> class 2 [0.750]

Evaluation on training data (199

cases):
(a) (b) <-classified as
---- ----
107 3 (a): class 1
17 72 (b): class 2

Results on (training set):

Sensitivity=0.97
Specificity=0.81

Sensitivity=0.97
Specificity=0.81

Sensitivity=0.98
Specificity=0.90

Evaluation of IDA results

 Absolute & relative accuracy

 Sensitivity & specificity
 False positive & false negative
 Error rate
35
  Reliability of rules
 Etc

NATURE OF DATA

INTRODUCTION

Data

 Data is a set of values of qualitative or quantitative variables; restated, pieces of

data are individual pieces of information.
 Data is measured, collected and reported, and analyzed, whereupon it can be visualized using
graphs or images.

Properties of Data
For examining the properties of data, reference to the various definitions of data.
Reference to these definitions reveals that following are the properties of data:
a) Amenability of use
b) Clarity
c) Accuracy
d) Essence
e) Aggregation
f) Compression
g) Refinement

.Amenability of use: From the dictionary meaning of data it is learnt that data are facts used in deciding
something. In short, data are meant to be used as a base for arriving at definitive conclusions.

a) Clarity: Data are a crystallized presentation. Without clarity, the meaning desired to be
communicated will remain hidden.
b) Accuracy: Data should be real, complete and accurate. Accuracy is thus, an essential

36
 property of data.

c) Essence: A large quantities of data are collected and they have to be Compressed and
refined. Data so refined can present the essence or derived qualitative value, of the matter.

d) Aggregation: Aggregation is cumulating or adding up.

e) Compression: Large amounts of data are always compressed to make them more meaningful.
Compress data to a manageable size.Graphs and charts are some examples of compressed
data.
f) Refinement: Data require processing or refinement. When refined, they are capable of
leading to conclusions or even generalizations. Conclusions can be drawn only when data are
processed or refined.

TYPES OF DATA

 In order to understand the nature of data it is necessary to categorize them into various types.
 Different categorizations of data are possible.
 The first such categorization may be on the basis of disciplines, e.g., Sciences, Social
Sciences, etc. in which they are generated.
 Within each of these fields, there may be several ways in which data can be categorized into types.

There are four types of data:

 Nominal

 Ordinal

 Interval

 Ratio

Each offers a unique set of characteristics, which impacts the type of analysis that can be performed.

37
The distinction between the four types of scales center on three different characteristics:

1. The order of responses – whether it matters or not

2. The distance between observations – whether it matters or is interpretable

3. The presence or inclusion of a true zero

Nominal Scales

Nominal scales measure categories and have the following characteristics:

 Order: The order of the responses or observations does not matter.

 Distance: Nominal scales do not hold distance. The distance between a 1 and a 2 is not the
same as a 2 and 3.

 True Zero: There is no true or real zero. In a nominal scale, zero is uninterruptable.

Appropriate statistics for nominal scales: mode, count, frequencies

Displays: histograms or bar charts

Ordinal Scales

At the risk of providing a tautological definition, ordinal scales measure, well, order. So, our

38
characteristics for ordinal scales are:

 Order: The order of the responses or observations matters.

 Distance: Ordinal scales do not hold distance. The distance between first and second is
unknown as is the distance between first and third along with all observations.

 True Zero: There is no true or real zero. An item, observation, or category cannot finish zero.

Appropriate statistics for ordinal scales: count, frequencies, mode

Displays: histograms or bar charts

Interval Scales

Interval scales provide insight into the variability of the observations or data.

Classic interval scales are Likert scales (e.g., 1 - strongly agree and 9 - strongly disagree) and Semantic
Differential scales (e.g., 1 - dark and 9 - light).

In an interval scale, users could respond to “I enjoy opening links to thwebsite from a company email”
with a response ranging on a scale of values.

The characteristics of interval scales are:

 Order: The order of the responses or observations does matter.

 Distance: Interval scales do offer distance. That is, the distance from 1 to 2 appears the same as
4 to 5. Also, six is twice as much as three and two is half of four. Hence, we can perform
arithmetic operations on the data.

 True Zero: There is no zero with interval scales. However, data can be rescaled in a manner
that contains zero. An interval scales measure from 1 to 9 remains the same as 11 to 19
because we added 10 to all values. Similarly, a 1 to 9 interval scale is the same a -4 to 4 scale
because we subtracted 5 from all values. Although the new scale contains zero, zero remains
uninterruptable because it only appears in the scale from the transformation.

Appropriate statistics for interval scales: count, frequencies, mode, median, mean, standard deviation
(and variance), skewness, and kurtosis.
39
Displays: histograms or bar charts, line charts, and scatter plots.

Ratio Scales

Ratio scales appear as nominal scales with a true zero. They

have the following characteristics:

 Order: The order of the responses or observations matters.

 Distance: Ratio scales do do have an interpretable distance.

 True Zero: There is a true zero.

Income is a classic example of a ratio scale:

 Order is established. We would all prefer $100 to $1!

 Zero dollars means we have no income (or, in accounting terms, our revenue exactly equals
our expenses!)

 Distance is interpretable, in that $20 appears as twice $10 and $50 is half of a $100. For
the web analyst, the statistics for ratio scales are the same as for interval scales.
Appropriate statistics for ratio scales: count, frequencies, mode, median, mean, standard deviation (and
variance), skewness, and kurtosis.

Displays: histograms or bar charts, line charts, and scatter plots.

The table below summarizes the characteristics of all four types of scales.

Nominal Ordinal Interval Ratio

Order Matters No Yes Yes Yes

Distance Is No No Yes Yes

Interpretable

Zero Exists No No No Yes

40
DATA CONVERSION

 We can convert or transform our data from ratio to interval to ordinal to nominal. However,
we cannot convert or transform our data from nominal to ordinal to interval to ratio.

 Scaled data can be measured in exact amounts.

For example, 60 degrees , 12.5 feet, 80 Miles per hour

 Scaled data can be measured with equal intervals.

For example, Between 0 and 1 is 1 inch, Between 13 and 14 is also 1 inch
 .Ordinal or ranked data provides comparative Amounts
Example:
1st Place 2nd Place 3rd Place

 Not equal intervals

1st 2nd Place 3rd
Place Place

19.6 feet 18.2 feet 12.4 feet

DATA SELECTION

Another Example that handle the question as :

What is the average driving speed of teenagers on the freeway?

a) Scaled
b) Ordinal

Scaled – Speed:- Speed can be measured in exact amounts withequal intervals.

Example :
41
60 degrees 12.5 feet 80 Miles per hour

 Ordinal or ranked data provides comparative amounts.

For example, 1st Place 2nd Place 3rd Place

 Percentiles provide comparative amounts.

In this case, 93% of all hospital have lower patient satisfaction scores than Eastridge hospital. 31% have
lower satisfaction scores than Westridge Hospital.

Thus the nature of data and its value have great influence on data insight in it.

***********************

42
 UNIT-III

ANALYTIC PROCESS AND TOOLS

• There are 6 analytic processes:

ses:

1. Deployment

2. Business Understanding

3. Data Exploration

4. Data Preparation

5. Data Modeling

6. Data Evaluation

43
Step 1: Deployment

• Here we need to:

– plan the deployment and monitoring and maintenance,

– we need to produce a final report and review the project.

– In this phase,

• we deploy the results of the analysis.

• This is also known as reviewing the project.

Step 2: Business Understanding

• Business Understanding

– The very first step consists of business understanding.

– Whenever any requirement occurs, firstly we need to determine the business

objective,

– assess the situation,

– determine data mining goals and then

– produce the project plan as per the requirement.

• Business objectives are defined in this phase.

Step 3: Data Exploration

• The second step consists of Data understanding.

– For the further process, we need to gather initial data, describe and explore the data
44
 and verify data quality to ensure it contains the data we require.

– Data collected from the various sources is described in terms of its application and
the need for the project in this phase.

– This is also known as data exploration.

• This is necessary to verify the quality of data collected.

Step 4: Data Preparation

• From the data collected in the last step,

– we need to select data as per the need, clean it, construct it to get useful
information and

– then integrate it all.

• Finally, we need to format the data to get the appropriate data.

• Data is selected, cleaned, and integrated into the format finalized for the analysis in this phase.

Step 5: Data Modeling

• we need to
– select a modeling technique, generate test design, build a model and assess the model
built.
• The data model is build to

– analyze relationships between various selected objects in the data,

– test cases are built for assessing the model and model is tested and implemented on the
data in this phase.
• Where processing is hosted?
– Distributed Servers / Cloud (e.g. Amazon EC2)
• Where data is stored?

45
 – Distributed Storage (e.g. Amazon S3)
• What is the programming model?
– Distributed Processing (e.g. MapReduce)

• How data is stored & indexed?

– High-performance schema-free databases (e.g. MongoDB)
• What operations are performed on data?
– Analytic / Semantic Processing
• Big data tools for HPC and supercomputing
– MPI
• Big data tools on clouds
– MapReduce model
– Iterative MapReduce model
– DAG model
– Graph model
– Collective model
• Other BDA tools

– SaS
– R
– Hadoop

Thus the BDA tools are used through out the BDA applications development.

46
ANALYSIS AND REPORTING

INTRODUCTION TO ANALYSIS AND REPORTING

What is Analysis?
• The process of exploring data and reports
– in order to extract meaningful insights,
– which can be used to better understand and improve business performance.

• What is Reporting ?
• Reporting is

– “the process of organizing data

– into informational summaries

– in order to monitor how different areas of a business are performing.”

COMPARING ANALYSIS WITH REPORTING

• Reporting is “the process of organizing data into informational summaries in order to

monitor how different areas of a business are performing.”
• Measuring core metrics and presenting them — whether in an email, a slidedeck, or
online dashboard — falls under this category.
• Analytics is “the process of exploring data and reports in order to extract meaningful
insights, which can be used to better understand and improve business performance.”
• Reporting helps companies to monitor their online business and be alerted to when data falls
outside of expected ranges.
• Good reporting
• should raise questions about the business from its end users.
• The goal of analysis is
• to answer questions by interpreting the data at a deeper level and providing
actionable recommendations.

47
 • A firm may be focused on the general area of analytics (strategy, implementation,
reporting, etc.)
– but not necessarily on the specific aspect of analysis.

• It’s almost like some organizations run out of gas after the initial set-up-related
activities and don’t make it to the analysis stage

A reporting activity deliberately proposes Analysis activity.

CONTRAST BETWEEN ANALYSIS AND REPORTING

The basis differences between Analysis and Reporting are as follows:

Analysis Reporting
Provides what is needed Provides what is asked for
Is typically customized Is Typically standardized
Involves a person Does not involve a person
Is extremely flexible Is fairly Inflexible

• Reporting translates raw data into information.

• Analysis transforms data and information into insights.

• reporting shows you what is happening

• while analysis focuses on explaining why it is happening and what you can do about it.

 Reports are like Robots n monitor and alter you and where as analysis is like parents - c an

48
 figure out what is going on (hungry, dirty diaper, no pacifier, , teething, tired, ear infection,
etc).
 Reporting and analysis can go hand-in-hand:
 Reporting provides no limited context about what is happening in the data. Context is critical
to good analysis.
 Reporting translate a raw data into information
 Reporting usually raises a question – What is happening ?
 Analysis transforms the data into insights - Why is it happening ? What you can do about
it?

Thus, Analysis and Reporting is synonym to each other with respect their need and utilizing in the
needy context.

MODERN ANALYTIC TOOLS

Introduction to Modern Analytic Tools

• Modern Analytic Tools: Current Analytic tools concentrate on three classes:

a) Batch processing tools

b) Stream Processing tools and
c) Interactive Analysis tools.

a) Big Data Tools Based on Batch Processing:

Batch processing system :-
• Batch Processing System involves
– collecting a series of processing jobs and carrying them out periodically as a
group (or batch) of jobs.
• It allows a large volume of jobs to be processed at the same time.

• An organization can schedule batch processing for a time when there is little
activity on their computer systems, for example overnight or at weekends.
49
• One of the most famous and powerful batch process-based Big Data tools is
Apache Hadoop.
 It provides infrastructures and platforms for other specific Big Data
applications.

50
b) Stream Processing tools
• Stream processing – Envisioning (predicting) the life in data as and when it
transpires.
• The key strength of stream processing is that it can provide insights faster, often
within milliseconds to seconds.
 It helps understanding the hidden patterns in millions of data records in real
time.
 It translates into processing of data from single or multiple sources
 in real or near-real time applying the desired business logic and emitting the
processed information to the sink.
• Stream processing serves

 multiple

 resolves in today’s business arena.

Real time data streaming tools are:
a) Storm
• Storm is a stream processing engine without batch support,
• a true real-time processing framework,
• taking in a stream as an entire ‘event’ instead of series of small batches.

• Apache Storm is a distributed real-time computation system.

• It’s applications are designed as directed acyclic graphs.

b) Apache flink
• Apache flink is

– an open source platform

– which is a streaming data flow engine that provides communication fault
tolerance and
– data distribution computation over data stream .

51
 – flink is a top level project of Apache flink is scalable data analytics framework that is
fully compatible to hadoop .
– flink can execute both stream processing and batch processing easily.
– flink was designed as an alternative to map-reduce.

c) Kinesis
– Kinesis as an out of the box streaming data tool.
– Kinesis comprises of shards which Kafka calls partitions.

– For organizations that take advantage of real-time or near real-time access to large
stores of data,
– Amazon Kinesis is great.
– Kinesis Streams solves a variety of streaming data problems.
– One common use is the real-time aggregation of data which is followed by
loading the aggregate data into a data warehouse.
– Data is put into Kinesis streams.
– This ensures durability and elasticity.

c) Interactive Analysis -Big Data Tools

• The interactive analysis presents

 the data in an interactive environment,

 allowing users to undertake their own analysis of information.
• Users are directly connected to

 the computer and hence can interact with it in real time.

• The data can be :

 reviewed,
 compared and
 analyzed
• in tabular or graphic format or both at the same time.

52
IA -Big Data Tools -

a) Google’s Dremel is the google proposed an interactive analysis system in 2010. And named
named Dremel.
– which is scalable for processing nested data.
– Dremel provides
• a very fast SQL like interface to the data by using a different technique than
MapReduce.
• Dremel has a very different architecture:

 compared with well-known Apache Hadoop, and

 acts as a successful complement of Map/Reduce-based computations.

• Dremel has capability to:

 run aggregation queries over trillion-row tables in seconds

 by means of:

• combining multi-level execution trees and

• columnar data layout.

b) Apache drill
• Apache drill is:

 Drill is an Apache open-source SQL query engine for Big Data exploration.
 It is similar to Google’s Dremel.
• For Drill, there is:

 more flexibility to support

• a various different query languages,
• data formats and
• data sources.

53
 • Drill is designed from the ground up to:

 support high-performance analysis on the semi-structured and

 rapidly evolving data coming from modern Big Data applications.

• Drill provides plug-and-play integration with existing Apache Hive and Apache
HBase deployments.

Categories of Modern Analytic Tools

a) Big data tools for HPC and supercomputing

– MPI
b) Big Data Tools for HPC and Supercomputing
• MPI(Message Passing Interface, 1992)

– Provide standardized function interfaces for communication between

parallel processes.
• Collective communication operations
– Broadcast, Scatter, Gather, Reduce, Allgather, Allreduce, Reduce-
scPopular implementations
– atter.
– MPICH (2001)
– OpenMPI (2004)
c) Big data tools on clouds
i. MapReduce model
ii. Iterative MapReduce model

iii. DAG model

iv. Graph model

v. Collective model

a) MapReduce Model
Jeffrey Dean et al. MapReduce: Simplified Data Processing on Large Clusters. OSDI
2004.

54
 b) Apache Hadoop (2005)
Apache Hadoop YARN: Yet Another Resource Negotiator, SOCC 2013.
Key Features of MapReduce Model
Designed for clouds
Large clusters of commodity machines
Designed for big data
Support from local disks based distributed file system (GFS / HDFS) Disk
based intermediate data transfer in Shuffling
MapReduce programming model
Computation pattern: Map tasks and Reduce tasks Data
abstraction: KeyValue pairs

Iterative MapReduce Model

• Twister:
A runtime for iterative MapReduce.
Have simple collectives: Boradcasting and aggregation.

• HaLoop
• An efficient Data Processing on Large clusters

• Have features:

– Loop-Aware Task Scheduling

– Caching and Indexing for Loop-Invariant Data on local disk.

Resilient Distributed Datasets(RDD):

• A Fault-Tolerant Abstraction for In-Memory Cluster Computing

– RDD operations
• MapReduce-like parallel operations
– DAG of execution stages and pipelined transformations
– Simple collectives: broadcasting and aggregation
55
DAG (Directed Acyclic Graph) Model
– A Distributed Data-Parallel Programs from Sequential Building Blocks,
• Apache Spark

– Cluster Computing with Working Sets

Graph Model

• Graph Processing with BSP model

• Pregel (2010)
– A System for Large-Scale Graph Processing. SIGMOD 2010.
– Apache Hama (2010)
• Apache Giraph (2012)
– Scaling Apache Giraph to a trillion edges

Pregel & Apache Giraph

• Computation Model
– Superstep as iteration
– Vertex state machine:
Active and Inactive, vote to halt
– Message passing between vertices
– Combiners
– Aggregators
– Topology mutation
• Master/worker model

• Graph partition: hashing

• Fault tolerance: checkpointing and confined recovery

56
GraphLab (2010)
• GraphLab: A New Parallel Framework for Machine Learning. UAI 2010.
• Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud.
• Data graph

• Update functions and the scope

• PowerGraph (2012)
– PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs.
– Gather, apply, Scatter (GAS) model
• GraphX (2013)
– A Resilient Distributed Graph System on Spark. GRADES
Collective Model
• Harp (2013)

– A Hadoop Plugin (on Hadoop 1.2.1 and Hadoop 2.2.0)

– Hierarchical data abstraction on arrays, key-values and graphs for easy
programming expressiveness.
– Collective communication model to support various communication operations on the
data abstractions.
– Caching with buffer management for memory allocation required from
computation and communication
– BSP style parallelism
– Fault tolerance with check-pointing.

Other major Tools

a) AWS
b) BigData
c) Cassandra
d) Data Warehousing
e) DevOps
f) HBase
57
g) Hive
h) MongoDB
i) NiFi
j) Tableau
k) Talend
l) ZooKeeper

Thus the modern analytical tools play an important role in the modern data world.

**********

58
STATISTICAL CONEPTS: SAMPLING DISTRIBUTIONS

Fundamental Statistics

 Statistics is a very broad subject, with applications in a vast number of different fields.
 In generally one can say that statistics is the methodology for collecting, analyzing,
interpreting and drawing conclusions from information.
 Putting it in other words, statistics is the methodology which scientists and mathematicians
have developed for interpreting and drawing conclusions from collected data.
 Everything that deals even remotely with the collection, processing, interpretation and
presentation of data belongs to the domain of statistics, and so does the detailed planning of that
precedes all these activities.
Data and Statistics

• Data consists of
– information coming from observations, counts, measurements, or responses.

Statistics is :
-science of collecting, organizing, analyzing, and interpreting data in order to make decisions.

A population is:
- the collection of all outcomes, responses, measurement, or counts that are of
interest.

A sample is:
- a subset of a population.

59
 • All data can be classified as either,
– 1) Categorical or
– 2) Quantitative.

• there exist deferent ways to summaries each type of data;

– for example an average measure of eye colour is nonsensical.

What is statistics?

Definition 1.1 (Statistics). Statistics consists of a body of methods for collecting and analyzing data.
(Agresti & Finlay, 1997)

• Statistics is the science of data.

• A bunch of numbers looking for a fight.

• It involves:

– collecting,
– classifying,
– summarizing,
– organizing,
– analyzing, and
– interpreting numerical information.

 Statistics is much more than just the tabulation of numbers and the graphical
presentation of these tabulated numbers.
 Statistics is the science of gaining information from numerical and categorical data.

 Statistical methods can be used to find answers to the questions like:

• What kind and how much data need to be collected?
• How should we organize and summarize the data?
• How can we analyse the data and draw conclusions from it?

60
 • How can we assess the strength of the conclusions and evaluate their
uncertainty?

 Categorical data (or qualitative data) results from descriptions, e.g. the blood type of
person, marital status or religious affiliation.

Why Statistics ?

 Statistics is the science of dealing with uncertain phenomenon and events.

 Statistics in practice is applied successfully to study the effectiveness of medical

treatments, the reaction of consumers to television advertising, the attitudes of young people
toward sex and marriage, and much more.
 It’s safe to say that nowadays statistics is used in every field of science.
 A statistic is a piece of data from a portion of a population.
 It’s the opposite of a parameter.

 A parameter is data from a census.

 A census surveys everyone.
 If you have a bit of information, it’s a statistic.
 If you look at part of a data set, it’s a statistic.

 If you know something about 10% of people, that’s a statistic too.

 Parameters are all the information.

 And all the information is rarely known.
 That’s why we need stats!
Statistics provides methods for:
1. Design: Planning and carrying out research studies.
2. Description: Summarizing and exploring data.
3. Inference: Making predictions and generalizing about phenomena represented by the data.

Example 1.1 (Statistics in practice).

61
Consider the following problems:

 Agricultural problem: Is new grain seed or fertilizer more productive?

 Medical problem: What is the right amount of dosage of drug to treatment?
 Political science: How accurate are the gallups and opinion polls?
 Eeconomics: What will be the unemployment rate next year?
 Technical problem: How to improve quality of product?

Statistical Concepts

Basic statistical operations

• Mean A measure of central tendency for Quantitative data i.e. the long term average value.
• Median A measure of central tendency for Quantitative data i.e. the half-way point.
• Mode The most frequently occurring (discrete), or where the probability density

function peaks (contin- ious).

• Minimum The smallest value.
• Maximum The largest value.
• Inter quartile range Can be thought or as the middle 50 of the (Quantitative) data, used
as a measure of spread.
• Variance - Used as a measure of spread, may be thought of as the moment of inertia.
• Standard deviation - A measure of spread, the square root of the variance.

Data shape

Continuous data can be;

• Symmetrical Perfectly symmetrical about the mean, i.e. Normally distributed data.
• Skewed Right/Positively skewed i.e. long tail to the right of the peak, Left/Negitively
skewed i.e. long tail to the left of the peak .
• Multimodal Multiple peaks in the data.
In addition when dealing with continuous data the shape of the data effects which summary statistic
are most appropriate to use i.e heavily skewed data will pull the mean toward the tail of the data.
62
 (A) Symmetrical (B) Rigt Skewed (c) Left Skewed
FIGURE 1. Displaying different data shapes with median, mean, density respectively.
Population and Sample

 Population and sample are two basic concepts of statistics.

 Population can be characterized as the set of individual persons or objects in which an
investigator is primarily interested during his or her research problem.
 Sometimes wanted measurements for all individuals in the population are obtained, but often
only a set of individuals of that population are observed; such a set of individuals constitutes a
sample.
 This gives us the following definitions of population and sample.
Definition Population: Population is the collection of all individuals or items under consideration in
a statistical study. (Weiss, 1999)

Definition Sample: Sample is that part of the population from which information is collected. (Weiss,
1999)

A population is:
- the collection of all outcomes, responses, measurement, or counts that are of
interest.

A sample is:
- a subset of a population.

Example of Populations & Samples

63
Question: In a recent survey,
• 250 college students at Union College were asked:
– if they smoked cigarettes regularly.
• Answer:
– 35 of the students said yes.
• Identify the population and the sample.

• Responses of all students at Union College (population)

• Responses of students in survey (sample)

Parameters & Statistics

A parameter is
-a numerical description of a population characteristic.

A statistic is:
- a numerical description of a sample characteristic.

Statistical process

Statistics ( in action have) Two Processes

1) Describing sets of data
2) Drawing conclusions
(making estimates, decisions, predictions, etc. about sets of data based on sampling)

What is a Statistic used for?

• Statistics is
64
 – a way to understand the data that is collected about us and the world.
• For example,
– every time you send a package through the mail, that package is tracked in a huge
database.
• All of that data is meaningless without a way to interpret it,

– which is where statistics comes in.

• Statistics is:
– about data and variables.

• It’s also about:

– analyzing that data and
– producing some meaningful information about that data.

Other Statistical Definitions

Variable: -
• Any characteristic of an individual or entity.
• A variable can take different values for different individuals.

Variables can be categorical or quantitative.

- Per S. S. Stevens…

• Variables are influenced by their nature of values that it holds.

• Nominal - Categorical variables with no inherent order or ranking sequence such as names
or classes (e.g., gender). Value may be a numerical, but without numerical value (e.g., I, II,
III). The only operation that can be applied to Nominal variables is enumeration.
• Ordinal - Variables with an inherent rank or order, e.g. mild, moderate, severe. Can be
compared for equality, or greater or less, but not how much greater or less.
• Interval - Values of the variable are ordered as in Ordinal, and additionally, differences
between values are meaningful, however, the scale is not absolutely anchored. Calendar dates
and temperatures on the Fahrenheit scale are examples. Addition and subtraction, but not
multiplication and division are meaningful operations.

65
 • Ratio - Variables with all properties of Interval plus an absolute, non-arbitrary zero point,
e.g. age, weight, temperature (Kelvin). Addition, subtraction, multiplication, and division are
all meaningful operations.

Why study statistics?

1. Data are everywhere.
2. Statistical techniques are used to make many decisions that affect our lives.
3. No matter what your career, you will make professional decisions that involve data.
4. An understanding of statistical methods will help you make these decisions efectively.

Applications of statistical concepts in the business world

• Finance – correlation and regression, index numbers, time series analysis

• Marketing – hypothesis testing, chi-square tests, nonparametric statistics

• Personel – hypothesis testing, chi-square tests, nonparametric tests

• Operating management – hypothesis testing, estimation, analysis of variance, time series

analysis
Application Areas

• Economics
– Forecasting
– Demographics

• Sports
– Individual & Team Performance

• Engineering
– Construction
– Materials
• Business
66
 – Consumer Preferences
– Financial Trends

Statistics Vs Statistical analysis

• Statistics :- The science of
– collectiong,
– organizing,
– presenting,
– analyzing, and
– interpreting data
• to assist in making more effective decisions.

• Statistical analysis: – used to

– manipulate summarize, and

– investigate data,
• so that useful decision-making information results.

Fundamental Elements of Statistics

1. Experimental unit
• Object upon which we collect data

2. Population
• All items of interest

3. Variable
• Characteristic of an individual
experimental unit
4. Sample
• Subset of the units of a population

67
 • P in Population & Parameter
arameter
• S in Sample & Statistic
tatistic

5. Statistical Inference
• Estimate or prediction or generalization about a population based on information
contained in a sample
6. Measure of Reliability
• Statement (usually qualified) about the degree of uncertainty associated with a
statistical inference

Types or Branches of Statistics

The study of statistics has two major branches: descriptive statistics and inferential statistics.

• Descriptive statistics: –
– Methods of organizing, summarizing, and presenting data in an informative way.
– Involves: Collecting Data
 Presenting Data
 Characterizing Data
 Purpose

68
  Describe Data

• Inferential statistics: –
– The methods used to determine something about a population on the basis of a
sample:

– Population –The entire set of individuals or objects of interest or the measurements

obtained from all individuals or objects of interest
– Sample – A portion, or part, of the population of interest

Descriptive and Inferential Statistics

• Example:

• In a recent study, volunteers who had less than 6 hours of sleep were four times more likely
to answer incorrectly on a science test than were participants who had at least 8 hours of
sleep.
• Decide which part is the descriptive statistic and what conclusion might be drawn using
inferential statistics?
Answer:-
The statement “four times more likely to answer incorrectly” is a descriptive statistic.
An inference drawn from the sample is that

all individuals sleeping less than 6 hours are more likely to answer science question
incorrectly than individuals who sleep at least 8 hours.

69
Inferential Statistics & Its’ Techniques

• Inference is the process of drawing conclusions or making decisions about a

population based on sample results Involves

• Estimation
• Hypothesis
Testing
Purpose
• Make decisions about population characteristics

• Estimation:-
– e.g., Estimate the population mean weight using the sample mean weight
• Hypothesis testing:-
– e.g., Test the claim that the population mean weight is 70 kg

Four Elements of Descriptive Statistical Problems

1. The population or sample of interest
2. One or more variables (characteristics of the population or sample units) that are to be
investigated
3. Tables, graphs, or numerical summary tools
70
 4. Identification of patterns in the data
Five Elements of Inferential Statistical Problems
1. The population of interest
2. One or more variables (characteristics of the population units) that are to be investigated
3. The sample of population units
4. The inference about the population based on information contained in the sample
5. A measure of reliability for the inference

SAMPLING DISTRIBUTION

Introduction to Sampling

• What is sample?
• A sample is “a smaller (but hopefully representative) collection of units from a
population used to determine truths about that population” (Field, 2005)

Why sample?
– Resources (time, money) and workload
– Gives results with known accuracy that can be calculated mathematically
• The sampling frame is the list from which the potential respondents are drawn
– Registrar’s office
– Class rosters
– Must assess sampling frame errors
Major Types of Samples

• Probability (Random) Samples

• Simple random sample

– Systematic random sample

– Stratified random sample
– Multistage sample
– Multiphase sample
71
 – Cluster sample
• Non-Probability Samples
– Convenience sample
– Purposive sample
– Quota
Specific Types of Samples

1. Stratified Samples
2. Cluster Samples
3. Systematic Samples
4. Convenience Samples

1. Stratified Samples
A stratified sample has:
i. members from each segment of a population.
ii. This ensures that each segment from the population is represented.

2. Cluster Samples

• A cluster sample has

• all members from randomly selected segments of a population.

• This is used when the population falls into naturally occurring subgroups.
72
3. Systematic Samples

• A systematic sample is
• a sample in which each member of the population is assigned a number.

• A starting number is

• randomly selected and sample members are selected at regular intervals.

4. Convenience Samples
A convenience sample consists only of available members of the population.

Example:
You are doing a study to determine the number of years of education each teacher at your
college has.
Identify the sampling technique used if you select the samples listed.

73
1.8.5.2 Sampling Distribution

What is Sampling Distribution?

• The way our means would be distributed
– if we (1) collected a sample,

– recorded the mean and

– threw it back, and
– (2) collected another,
– recorded the mean and
– threw it back, and
– did this again and again….
- ad nauseam!

74
• From Vogt:
A theoretical frequency distribution of the scores for or values of a
statistic, such as a mean.
– Any statistic that can be computed for a sample has a sampling distribution.

• A sampling distribution is:

– the distribution of statistics

– that would be produced

– in repeated random sampling (with replacement) from the same population.

Glimpses of Sampling Distribution

• Sampling distributions is
– all possible values of a statistic and
– their probabilities of occurring for a sample of a particular size.
• Sampling distributions are used to
– calculate the probability that sample statistics
– could have occurred by chance and
– thus to decide whether something that is true of a sample statistic is
• also likely to be true of a population parameter.

A Positive move of Sampling Distribution

• We are moving from descriptive statistics to inferential statistics.
• Inferential statistics allow the researcher:

– to come to conclusions about a population

– on the basis of descriptive statistics about a sample.

75
 Examples of Sampling Distribution

1) Your sample says that

a candidate gets support from 47%.
2) Inferential statistics allow you to say that

– (a) the candidate gets support from 47% of the population

– (b) with a margin of error of +/- 4%.
– This means that the support in the population is
likely somewhere between 43% and 51%.

Error on Sampling Distribution

• Margin of error is taken directly from a sampling distribution.

• It looks like this:

A Real Time Scenario on Sampling Distribution

Let’s create a sampling distribution of means…

a) Take a sample of size 1,500 from the US.
76
b) Record the mean income.
c) Our census said the mean is $30K.

 Let’s create a sampling distribution of means…

 Take another sample of size 1,500 from the US. Record the mean income. Our census said
the mean is $30K.
 Take another sample of size 1,500 from the US. Record the mean income. Our census said
the mean is $30K.
 Take another sample of size 1,500 from the US. Record the mean income. Our census said
the mean is $30K.
 ….
 Say that the standard deviation of this distribution is $10K.
 Think back to the empirical rule. What are the odds you would get a sample mean that is
more than $20K off.

Knowing the likely variability

riability of the sample means from repeated sampling gives us a context
within which to judge how much we can trust the number we got from our sample.
For example, if the variability is low, , we can trust our number more than if the
variability is high, .

77
• The first sampling distribution above, a, has a lower standard error. Now
a definition!
The standard deviation of a normal sampling distribution is called the standard error.
• Statisticians have found that
– the standard error of a sampling distribution is :
• quite directly affected by
• the number of cases in the sample(s), and

• the variability of the population distribution.

Statisticians have found that the standard error of a sampling distribution is quite
directly affected by the number of cases in the sample(s), and the variability of the
population distribution.
Population Variability:
For example, Americans’ incomes are quite widely distributed, from $0 to Bill Gates’.
Americans’ car values are less widely distributed, from about $50 to about $50K.
The standard error of the latter’s sampling distribution will be a lot less variable.

78
 Population Variability:

The standard error of income’s sampling distribution will be a lot higher than car price’s.

Probability Distributions

• A Note:
– Not all theoretical probability distributions are Normal. One example of many is the
binomial distribution.
• The binomial distribution gives
– the discrete probability distribution of obtaining exactly n successes out of N trials
• where the result of each trial is true with known probability of success and

79
 false with the inverse probability
probability.
• The binomial distribution has
– a formula and
– changes shape with each probability of success and number of trials.

• However, in this class the normal probability distribution is the most useful!

Permutation Tests
 In classical hypothesis testing,
 we start with assumptions about the underlying distribution and
 then derive the sampling distribution of the test statistic under H0.
 In Permutation testing,
 the initial assumptions
mptions are not needed (except exchangeability), and
 the sampling distribution of the test statistic under H0 is computed by using
permutations of the data.

Permutation Tests (example)

• The Permutation test is a technique that bases inference on “experiments”

ments” within the
observed dataset.
• Consider the following example:
• In a medical experiment, rats are randomly assigned to a treatment (Tx) or control (C) group.
• The outcome Xi is measured in the ith rat.
• Under H0,
80
 • the outcome does not depend on whether a rat carries the label Tx or C.

• Under H1,

• the outcome tends to different, say larger for rats labeled Tx.
• A test statistic T measures
• the difference in observed outcomes for the two groups.

• T may be the difference in the two group means (or medians), denoted as t for the
observed data.
• Under H0,

• the individual labels of Tx and C are unimportant, since they have no impact on the
outcome. Since they are unimportant, the label can be randomly shuffled among the
rats without changing the joint null distribution of the data.
• Shuffling the data creates a:

• “new” dataset.
• It has the same rats, but with the group labels changed so as to appear as there were
different group assignments.
• Let t be the value of the test statistic from the original dataset.
• Let t1 be the value of the test statistic computed from a one dataset with permuted labels.
• Consider all M possible permutations of the labels, obtaining the test statistics, t 1, …,
tM .
• Under H0, t1, …, tM are all generated from the same underlying distribution that
generated t.
• Thus, t can be compared to the permuted data test statistics, t 1, …, tM , to test the
hypothesis and obtain a p-value or to construct confidence limits for the statistic.

• Survival times
• Treated mice 94, 38, 23, 197, 99, 16, 141

• Mean: 86.8

81
 • Untreated mice 52, 10, 40, 104, 51, 27, 146, 30, 46

• Mean: 56.2
(Efron & Tibshirani)
 Calculate the difference between the means of the two observed samples – it’s 30.6 days in
favor of the treated mice.
 Consider the two samples combined (16 observations) as the relevant universe to
resample from.
 Draw 7 hypothetical observations and designate them "Treatment"; draw 9 hypothetical
observations and designate them "Control".
 Compute and record the difference between the means of the two samples.
 Repeat steps 3 and 4 perhaps 1000 times.
 Determine how often the resampled difference exceeds the observed difference of 30.6
 If the group means are truly equal,
 then shifting the group labels will not have a big impact the sum of the two groups (or mean
with equal sample sizes).
 Some group sums will be larger than in the original data set and some will be smaller.

Permutation Test Example 1

• 16!/(16-7)!= 57657600
• Dataset is too large to enumerate all permutations, a large number of random
permutations are selected.
• When permutations are enumerated, this is an exact permutation test.

1. Bootstrap
• The bootstrap is
– a widely applicable tool that
– can be used to quantify the uncertainty associated with a given estimator or
statistical learning approach, including those for which it is difficult to obtain a

82
 measure of variability.
• The bootstrap generates:

– distinct data sets by repeatedly sampling observations from the original data set.
– These generated data sets can be used to estimate variability in lieu of sampling
independent data sets from the full population.

• 1969 Simon publishes the bootstrap as an example in Basic Research Methods in Social
Science (the earlier pigfood example)
• 1979 Efron names and publishes first paper on the bootstrap

• Coincides with advent of personal computer.

• The sampling employed by the bootstrap involves randomly selecting n observations with
replacement,
– which means some observations can be selected multiple times while other
observations are not included at all.
• This process is repeated B times to yield B bootstrap data sets,
– Z∗1,Z∗2,…,Z∗B,which can be used to estimate other quantities such as standard error.

How Bootstrap Works?

Bootstrapping is a method for estimating the sampling distribution of an estimator by resampling with
83
replacement from the original sample.

 The bootstrap procedure is a means of estimating the statistical accuracy . . . from the data in
a single sample.
 Bootstrapping is used to mimic the process of selecting many samples when the
population is too small to do otherwise
 The samples are generated from the data in the original sample by copying it many
number of times (Monte Carlo Simulation)

 Samples can then selected at random and descriptive statistics calculated or regressions run for
each sample
 The results generated from the bootstrap samples can be treated as if it they were the result
of actual sampling from the original population

Characteristics of Bootstrapping

84
Bootstrapping Example

When Bootstrapping should be used?

Bootstrapping is especially useful in situations when no analytic formula for the sampling distribution
is available.
• Traditional forecasting methods, like exponential smoothing, work well when demand is
constant – patterns easily recognized by software
• In contrast, when demand is irregular, patterns may be difficult to recognize.
• Therefore, when faced with irregular demand, bootstrapping may be used to provide more
accurate forecasts, making some important assumptions…

Assumptions and Methodology

• Bootstrapping makes no assumption regarding the population
• No normality of error terms

• No equal variance
• Allows for accurate forecasts of intermittent demand
• If the sample is a good approximation of the population, the sampling distribution may be
85
 estimated by generating a large number of new samples
• For small data sets, taking a small representative sample of the data and replicating it will
yield superior results

Applications and Uses

a) Criminology:- Statistical significance testing is important in criminology and criminal justice.
b) Actuarial Practice:- Process of developing an actuarial model begins with the creation of
probability distributions of input variables. Input variables are generally asset-side generated
cash flows (financial) or cash flows generated from the liabilities side (underwriting)
c) Classifications Used by Ecologists:- Ecologists often use cluster analysis as a tool in the
classification and mapping of entities such as communities or landscapes
d) Human Nutrition:- Inverse regression used to estimate vitamin B-6 requirement of young
women & Standard statistical methods were used to estimate the mean vitamin B-6 requirement.
e) Outsourcing:- Agilent Technologies determined it was time to transfer manufacturing of its 3070
in-circuit test systems from Colorado to Singapore & Major concern was the change in
environmental test conditions (dry vs humid).

Bootstrap Types
a) Parametric Bootstrap
b) Non-parametric Bootstrap

a) Parametric Bootstrap
• Re-sampling makes no assumptions about the population distribution.

• The bootstrap covered thus far is a nonparametric bootstrap.

• If we have information about the population distr., this can be used in resampling.
• In this case, when we draw randomly from the sample we can use population distr.
• For example, if we know that the population distr. is normal then estimate its parameters using
the sample mean and variance.
• Then approximate the population distr. with the sample distr. and use it to draw new
samples.
86
 • As expected, if the assumption about population distribution is correct then the
parametric bootstrap will perform better than the nonparametric bootstrap.
• If not correct, then the nonparametric bootstrap will perform better.

Bootstrap Example
• A new pigfood ration is tested on twelve pigs, with six-week weight gains as follows:

• 496 544 464 416 512 560 608 544 480 466 512 496

• Mean: 508 ounces (establish a confidence interval)

Draw simulated samples from a hypothetical universe that embodies all we know about the universe that
this sample came from – our sample, replicated an infinite number of times.
The Bootstrap process steps
1. Put the observed weight gains in a hat
2. Sample 12 with replacement
3. Record the mean
4. Repeat steps 2-3, say, 1000 times
5. Record the 5th and 95th percentiles (for a 90% confidence interval)
Bootstrapped sample means

b) Nonparametric bootstrap
 nonparametric bootstrap method which relies on the empirical distribution function.
 The idea of the nonparametric bootstrap is to simulate data from the empirical cdf Fn.
87
  Since the bootstrap samples are generated from Fn, this method is called the
nonparametric bootstrap.
 Here Fn is a discrete probability distribution that gives probability 1/n to each
observed value x1, · · · , xn.
 A sample of size n from Fn is thus a sample of size n drawn with replacement from the
collection x1, · · · , xn.
 The standard deviation of ˆθ is then estimated by
 sθˆ = SQRT( 1 B X B i=1 (θ ∗ i − ¯θ ∗)^ 2 )
 where θ ∗ 1 , . . . , θ∗ B are produced from B sample of size n from the collection x1, ·
· · , xn.

Example of Bootstrap (Nonparametric)

• Have test scores (out of 100) for two consecutive years for each of 60 subjects.

• Want to obtain the correlation between the test scores and the variance of the correlation
estimate.

2. Jackknife Method
• Jackknife method was introduced by Quenouille (1949)
– to estimate the bias of an estimator.
• The method is later shown to be useful in reducing the bias as well as in estimating the
variance of an estimator.

• Let ˆθn be an estimator of θ based on n i.i.d. random vectors X1, . . . , Xn, i.e., ˆθn = fn(X1,
. . . , Xn), for some function fn. Let

• A statistical method for estimating and removing bias* and for deriving robust estimates of
standard errors and confidence intervals.
• Created by systematically dropping out subsets of data one at a time and assessing the
resulting variation.

88
 • θn,−i = fn−1(X1, . . . , Xi−1, Xi+1, . . . , Xn) be the corresponding recomputed statistic based
on all but the i-th observation. The jackknife estimator of bias E( ˆθn) − θ is given by
• biasJ = (n − 1) n Xn i=1 ¡ ˆθn,−i − ˆθn ¢ .

• Jackknife estimator θJ of θ is given by:

– θJ = ˆθn − biasJ = 1 n Xn i=1 ¡ nˆθn − (n − 1)ˆθn,−i ¢ .
(2)
• Such a bias corrected estimator hopefully reduces the over all bias.
• The summands above θn,i = ˆθn − (n − 1)ˆθn,−i , i = 1, . . . , n are called pseudo-values.

A comparison of the Bootstrap & Jackknife

• Bootstrap
– Yields slightly different results when repeated on the same data (when estimating the
standard error)
– Not bound to theoretical distributions
• Jackknife
– Less general technique
– Explores sample variation differently
– Yields the same result each time
– Similar data requirements
3. Cross validation
• Cross-validation is
– a technique used to protect against overfitting in a predictive model,
– particularly in a case where the amount of data may be limited.

• In cross-validation, you make a fixed number of folds (or partitions) of the data, run the
analysis on each fold, and then average the overall error estimate.

• Cross validation is a re-sampling method that

– can be used to estimate a given statistical methods test error or to determine the
appropriate amount of flexibility.
89
 – Model assessment is the process of evaluating a model’s performance.
– Model selection is the process of selecting the appropriate level of flexibility for a
model.
– Bootstrap is used in a number of contexts,
– but most commonly it is used to provide a measure of accuracy of a
given statistical learning method or parameter estimate.

Need of Cross validation

• Use the entire data set when training a learner.

• Some of the data is removed before training begins.

• Then when training is done, the data that was removed can be used to test the
performance of the learned model on ``new'' data.
• This is the basic idea for a whole class of model evaluation methods called cross
validation.
Bootstrap vs. Cross-Validation
• Bootstrap
– Requires a small of data
– More complex technique – time consuming

• Cross-Validation
– Not a resampling technique
– Requires large amounts of data
– Extremely useful in data mining and artificial intelligence

Cross Validation Methods

1. holdout method
2. K-fold cross validation
3. Leave-one-out cross validation

90
1. holdout method

• The holdout method is the simplest kind of cross validation.

• The data set is separated into

– two sets, called the training set and the testing set.
• The function approximator fits a function using the training set only.
• Then the function approximator is asked to
– predict the output values for the data in the testing set (it has never seen these
output values before).
• The errors it makes are accumulated as before to

– give the mean absolute test set error, which is used to evaluate the model.
• The advantage of this method is that

– it is usually preferable to the residual method and takes no longer to compute.

– However, its evaluation can have a high variance.
• The evaluation may depend heavily on
– which data points end up in the training set and which end up in the test set, and
– thus the evaluation may be significantly different depending on how the division is
made.
2. K-fold cross validation

• K-fold cross validation is one way to improve over the holdout method.
– The data set is divided into k subsets, and the holdout method is repeated k times.
• Each time, one of the k subsets is used as

– the test set and

– the other k-1 subsets are put together to form a training set.
– Then the average error across all k trials is computed.
91
• The advantage of this method is that

– it matters less how the data gets divided.

• Every data point gets to be in a test set exactly once, and gets to be in a training set k- 1
times.
• The variance of the resulting estimate is reduced as k is increased.

• The disadvantage of this method is that the training algorithm has to be rerun from
scratch k times, which means it takes k times as much computation to make an evaluation.
• A variant of this method is to randomly divide the data into a test and training set
kdifferent times.
• The advantage of doing this is that

– you can independently choose how large each test set is and how many trials you
average over.

Leave-one-out cross validation

• Leave-one-out cross validation is K-fold cross validation taken to its :

– logical extreme, with K equal to N, the number of data points in the set.
– That means that N separate times, the function approximator is trained on all the data
except for one point and a prediction is made for that point.
– As before the average error is computed and used to evaluate the model.

– The evaluation given by leave-one-out cross validation error (LOO-XVE) is good,

but at first pass it seems very expensive to compute.
– Fortunately, locally weighted learners can make LOO predictions just as easily as they
make regular predictions.
– That means computing the LOO-XVE takes no more time than computing the
residual error and it is a much better way to evaluate models.

92
STATISTICAL INFERENCE

Introduction to Statistical Inference

Inference : - Use a random sample to learn something about a larger population

• Two ways to make inference

1) Estimation of parameters
-Point Estimation (X
X or p)
-Intervals Estimation
2) Hypothesis Testing
Statistical Inference

The process of making guesses about the truth from a sample.

• Statistical inference is
– the process through which inferences
– about a population are made based on certain statistics calculated from a sample of
data drawn from that population.
– What is the practical example of statistical inference?

• Use a statistical approach to make an inference about the distribution of a

sample of data we collect.
• What distribution(s) are the data from?

Suppose that they are from the normal distribution.

• What normal distribution(s) are the data from?

93
Use a statistical approach to make an inference about the distribution of a sample of data we
collect.
The population or macroscopic phenomenon is always unknown itself, because some, but not all,
of the data of our interest can be taken.

Types of statistical inference

There are Two most common types of Statistical Inference and they are:
1. Confidence intervals and
2. Tests of significance.

94
 1. Confidence Intervals

• Range of values that m is expected to lie within

• 95% confidence interval
– .95 probability that m will fall within range
– probability is the level of confidence
• e.g., .75 (uncommon), or .99 or .999
• Which level of confidence to use?

– Cost vs. benefits judgement ~

Finding Confidence Intervals
• Method depends on whether s is known

• If s known

Meaning of Confidence Interval

• 95% confident that m lies between lower & upper limit
– NOT absolutely certain
– .95 probability
• If computed C.I. 100 times

– using same methods

– m within range about 95 times
• Never know m for certain

– 95% confident within interval ~

Example

95
 • Compute 95% C.I.

• IQ scores
– s = 15
• Sample: 114, 118, 122, 126
– SXi = 480, X = 120, sX = 7.5
– 120 ± 1.96(7.5)
– 120 + 14.7
– 105.3 < m < 134.7

• We are 95% confident that population means lies between 105.3 and 134.7 ~

Changing the Level of Confidence

• We want to be 99% confident

– using same data

– z for area = .005
– zCV..01 = 2.57
• 120 ± 2.57(7.5)
– 100.7 < m < 139.3
• Wider than 95% confidence interval

– wider interval ---> more confident ~

96
Student’s t Distribution
• if s = s, then t and z identical
– if s ¹ s, then t wider
• Accuracy of s as point-estimate
– depends on sample size
– larger n ---> more accurate
• n > 120
– s»s
– t and z distributions almost identical ~

Degrees of Freedom
• Width of t depends on n

• Degrees of Freedom
– related to sample size
– larger sample ---> better estimate
– n - 1 to compute s ~

Critical Values of t
• Table A.2: “Critical Values of t”

• df = n - 1
• level of significance for two-tailed test
– a
– area in both tails for critical value
• level of confidence for CI ~
– 1-a ~

97
Confidence Intervals: s unknown
• Same as known but use t

– Use sample standard error of mean

– df = n-1

4 factors that affect CI width

• Would like to be narrow as possible
– usually reflects less uncertainty

• Narrower CI by...

1. Increasing n
– decreases standard error
2. Decreasing s or s
– little control over this ~
3. s known
– use z distribution, critical values
4. Decreasing level of confidence
– increases uncertainty that m lies within interval
– costs / benefits ~
2. Test of Significance ( Hypothesis testing)

• A statistical method that uses:

– sample data to evaluate a hypothesis about a population parameter.

98
 • It is intended to help researchers differentiate between real and random patterns in the data.

What is a Hypothesis?
• A hypothesis is an assumption about the population parameter.
– A parameter is a Population mean or proportion
– The parameter must be identified before analysis.

Hypothesis Testing
• Is also called significance testing
• Tests a claim about a parameter using evidence (data in a sample
• The technique is introduced by considering a one-sample z test

• The procedure is broken into four steps

• Each element of the procedure must be understood

Hypothesis Testing Steps

A. Null and alternative hypotheses
B. Test statistic
C. P-value and interpretation
D. Significance level (optional)

A. The Null Hypothesis, H0

• States the Assumption (numerical) to be tested

• e.g. The average # TV sets in US homes is at least 3 (H0: m ³ 3)

• Begin with the assumption that the null hypothesis is TRUE.

(Similar to the notion of innocent until proven guilty)
• The Null Hypothesis may or may not be rejected.

99
The Alternative Hypothesis, H1
• Is the opposite of the null hypothesis e.g. The average # TV sets in US homes is
less than 3 (H1: m < 3)
• Challenges the Status Quo
• Never contains the ‘=‘ sign

• The Alternative Hypothesis may or may not be accepted

Identify the Problem

Steps:
– State the Null Hypothesis (H0: m ³ 3)
– State its opposite, the Alternative Hypothesis (H1: m < 3)
• Hypotheses are mutually exclusive & exhaustive

• Sometimes it is easier to form the alternative hypothesis first.

Hypothesis Testing Process

Reason for Rejecting H0

100
Sampling Distribution :

Level of Significance, a
• Defines Unlikely Values of Sample Statistic if Null Hypothesis Is True

– Called Rejection Region of Sampling

Distribution
• Designated a (alpha)
– Typical values are 0.01, 0.05, 0.10
• Selected by the Researcher at the Start
• Provides the Critical Value(s) of the Test

Errors in Making Decisions

• Type I Error
– Reject True Null Hypothesis
– Has Serious Consequences
– Probability of Type I Error Is a
• Called Level of Significance

• Type II Error
– Do Not Reject False Null Hypothesis
– Probability of Type II Error Is b (Beta)
101
Hypothesis Testing: Steps

Test the Assumption that the true mean SBP of participants is 120 mmHg. State
H0 H0 : m = 120
State H1 H1 : m  120
Choose a a = 0.05
Choose n n = 100
Choose Test: Z, t, X2 Test (or p Value)

One sample-mean Test

• Assumptions
– Population is normally distributed

• t test statistic

sample mean  null value x  0

t 
standard error s
n

102
Example 2:
• In a survey of diabetics in a large city, it was found that 100 out of 400 have
diabetic foot. Can we conclude that 20 percent of diabetics in the sampled
population have diabetic foot.
• Test at the a =0.05 significance level.

Solution

Critical Value: 1.96

103
Decision:
We have sufficient evidence to reject the Ho value of 20%
We conclude that in the population of diabetic the proportion who have diabetic foot does not equal
0.20
****************

104
PREDICTION ERROR

Introduction to Prediction Error

 A prediction error is the failure of some expected event to occur.

 Errors are an inescapable element of predictive analytics that should also be quantified and
presented along with any model, often in the form of a confidence interval that indicates how
accurate its predictions are expected to be.
 A prediction error is the failure of some expected event to occur.
 When predictions fail, humans can use metacognitive functions, examining prior
predictions and failures.
 For example, whether there are correlations and trends, such as consistently being unable
to foresee outcomes accurately in particular situations.
 Applying that type of knowledge can inform decisions and improve the quality of future
predictions.

Error in Predictive Analysis

 Errors are an inescapable element of predictive analytics that should also be quantified and
presented along with any model, often in the form of a confidence interval that indicates how
accurate its predictions are expected to be.
 Analysis of prediction errors from similar or previous models can help determine
confidence intervals.
Predictions always contain errors

 Predictive analytics has many applications, the above mentioned examples are just the tip of
the iceberg.
 Many of them will add value, but it remains important to stress that the outcome of a
prediction model will always contain an error. Decision makers need to know how big that
error is.
 To illustrate, in using historic data to predict the future you assume that the future will have
105
 the same dynamics as the past, an assumption which history has proven to be dangerous.
 In artificial intelligence (AI), the analysis of prediction errors can help guide machine
learning (ML), similarly to the way it does for human learning.

 In reinforcement learning, for example, an agent might use the goal of minimizing error
feedback as a way to improve.
 Prediction errors, in that case, might be assigned a negative value and predicted outcomes
a positive value, in which case the AI would be programmed to attempt to maximize its
score.
 That approach to ML, sometimes known as error-driven learning, seeks to stimulate
learning by approximating the human drive for mastery.

Prediction Error in statistics

Standard Error of the Estimate

 The standard error of the estimate is a measure of the accuracy of predictions.

 Recall that the regression line is the line that minimizes the sum of squared deviations of
prediction (also called the sum of squares error).

 The standard error of the estimate is closely related to this quantity and is defined below:

where σest is the standard error of the estimate, Y is an actual score, Y' is a predicted score, and N is the
number of pairs of scores.

106
  In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an
estimator (of a procedure for estimating an unobserved quantity) measures
the average of the squares of the errors—that is, the average squared difference between the
estimated values and what is estimated.
 MSE is a risk function, corresponding to the expected value of the squared error loss.

 The fact that MSE is almost always strictly positive (and not zero) is because
of randomness or because the estimator does not account for information that could produce a
more accurate estimate.
Mean squared prediction error

 In statistics the mean squared prediction error or mean squared error of

the predictions of a smoothing or curve fitting procedure is the expected value of the squared
difference between the fitted values implied by the predictive function and the values of the
(unobservable) function g.
 The MSE is a measure of the quality of an estimator—it is always non-negative, and values
closer to zero are better.
 Root-Mean-Square error or Root-Mean-Square Deviation (RMSE or RMSD)
 In an analogy to standard deviation, taking the square root of MSE yields the root-mean- square
error or root-mean-square deviation (RMSE or RMSD), which has the same units as the
quantity .being estimated; for an unbiased estimator, the RMSE is the square root of the
variance, known as the standard error.
 The RMSD represents the square root of the second sample moment of the differences
between predicted values and observed values or the quadratic mean of these differences.
 These deviations are called residuals when the calculations are performed over the data sample
that was used for estimation and are called errors (or prediction errors) when computed out-
of-sample.
 The RMSD serves to aggregate the magnitudes of the errors in predictions for various times
into a single measure of predictive power.
 RMSD is a measure of accuracy, to compare forecasting errors of different models for a
particular dataset and not between datasets, as it is scale-dependent.

107
  RMSD is always non-negative, and a value of 0 (almost never achieved in practice) would
indicate a perfect fit to the data.
 In general, a lower RMSD is better than a higher one. However, comparisons across different
types of data would be invalid because the measure is dependent on the scale of the numbers
used.
 RMSD is the square root of the average of squared errors.
 The effect of each error on RMSD is proportional to the size of the squared error; thus larger
errors have a disproportionately large effect on RMSD.

 Consequently, RMSD is sensitive to outliers.

What is prediction error in regression?

 Regressions differing in accuracy of prediction.
 The standard error of the estimate is a measure of the accuracy of predictions.
 Recall that the regression line is the line that minimizes the sum of squared deviations of
prediction (also called the sum of squares error).

Thus, the prediction error influence the analytics functionalities and its applications
areas.

******************

108