Unit-1

NOTES
Faculty – Ms Priyanka (Assistant Professor)
Introduction to Data Analytics
Different Sources of Data for Data Analysis

Data collection is the process of acquiring, collecting, extracting, and storing the
voluminous amount of data which may be in the structured or unstructured form like text,
video, audio, XML files, records, or other image files used in later stages of data
analysis. In the process of big data analysis, “Data collection” is the initial step before starting
to analyze the patterns or useful information in data. The data which is to be analyzed must
be collected from different valid sources.

Data Growth over the years

The data which is collected is known as raw data which is not useful now but on cleaning the

impure and utilizing that data for further analysis forms information, the information obtained
is known as “knowledge”. Knowledge has many meanings like business knowledge
or sales of enterprise products, disease treatment, etc. The main goal of data
collection is to collect information-rich data.

Data collection starts with asking some questions such as what type of data is to be collected
and what is the source of collection. Most of the data collected are of two types known as
“qualitative data “which is a group of non-numerical data such as words, sentences mostly
focus on behavior and actions of the group and another one is “quantitative data” which is in
numerical forms and can be calculated using different scientific tools and sampling data.

The actual data is then further divided mainly into two types known as:

•Primary data

•Secondary data

1. Primary Data:

The data which is Raw, original, and extracted directly from the official sources is known as
primary data. This type of data is collected directly by performing techniques
such as questionnaires, interviews, and surveys. The data collected must be according to the
demand and requirements of the target audience on which analysis is performed otherwise it
would be a burden in the data processing.
Few methods of collecting primary data:

a. Interview method:

The data collected during this process is through interviewing the target audience by a person
called interviewer and the person who answers the interview is known as the interviewee.
Some basic business or product related questions are asked and noted down in the form of
notes, audio, or video and this data is stored for processing. These can be both structured and
unstructured like personal interviews or formal interviews through telephone, face to face,
email, etc.

b. Survey method:

The survey method is the process of research where a list of relevant questions is asked and
answers are noted down in the form of text, audio, or video. The survey method can be
obtained in both online and offline mode like through website forms and email. Then that
survey answers are stored for analyzing data. Examples are online surveys or surveys through
social media polls.

c. Observation method:

The observation method is a method of data collection in which the researcher keenly
observes the behavior and practices of the target audience using some data collecting tool
and stores the observed data in the form of text, audio, video, or any raw formats. In this
method, the data is collected directly by posting a few question on the participants.
For example, observing a group of customers and their behavior towards the products.
The data obtained will be sent for processing.

d. Experimental method:
The experimental method is the process of collecting data through performing experiments,

research, and investigation. The most frequently used experiment methods are CRD, RBD,

LSD, FD.

•CRD- Completely Randomized design is a simple experimental design used in data

analytics which are based on randomization and replication. It is mostly used for comparing

the experiments.

•RBD- Randomized Block Design is an experimental design in which the experiment is

divided into small units called blocks. Random experiments are performed on each of the

blocks and results are drawn using a technique known as analysis of variance (ANOVA).

RBD was originated from the agriculture sector.

•LSD – Latin Square Design is an experimental design that is similar to CRD and RBD

blocks but contains rows and columns. It is an arrangement of NxN squares with an equal

number of rows and columns which contain letters that occurs only once in a row. Hence the

differences can be easily found with fewer errors in the experiment. Sudoku puzzle is an

example of a Latin square design.

•FD- Factorial design is an experimental design where each experiment has two factors each

with possible values and on performing trail other combinational factors are derived.

2. Secondary data:
Secondary data is the data which has already been collected and reused again for some valid

purpose. This type of data is previously recorded from primary data and it has two types of
sources named internal source and external source.

Internal source:

These types of data can easily be found within the organization such as market record, a sales
record, transactions, customer data, accounting resources, etc. The cost and time
consumption is less in obtaining internal sources.

External source:

The data which can’t be found at internal organizations and can be gained through external
third-party resources is external source data. The cost and time consumption is more because
this contains a huge amount of data. Examples of external sources are Government
publications, news publications, Registrar General of India, planning commission,
international labor bureau, syndicate services, and other non-governmental publications.

Other sources:

•Sensors data: With the advancement of IoT devices, the sensors of these devices collect

data which can be used for sensor data analytics to track the performance and usage of

products.

•Satellites data: Satellites collect a lot of images and data in terabytes on daily basis through

surveillance cameras which can be used to collect useful information.

•Web traffic: Due to fast and cheap internet facilities many formats of data which is uploaded
by users on different platforms can be predicted and collected with their permission for data
analysis. The search engines also provide their data through keywords and queries searched
mostly.

Classification of data (structured, semi-structured, unstructured)

Big Data includes huge volume, high velocity, and extensible variety of data. These are 3 types:

Structured data, Semi-structured data, and Unstructured data.

1. Structured data –
Structured data is data whose elements are addressable for effective analysis. It has been

organized into a formatted repository that is typically a database. It concerns all data which

can be stored in database SQL in a table with rows and columns. They have relational keys

and can easily be mapped into pre-designed fields. Today, those data are most processed in

the development and simplest way to manage information. Example: Relational data.

Examples Of Structured Data

An 'Employee' table in a database is an example of Structured Data

Semi-Structured data –
Semi-structured data is information that does not reside in a relational database but that have

some organizational properties that make it easier to analyze. With some process, you can

store them in the relation database (it could be very hard for some kind of semi-structured

data), but Semi-structured exist to ease space. Example: XML data.

Examples Of Semi-Structured Data

Personal data stored in an XML file-

3. Unstructured data –
Unstructured data is a data which is not organized in a predefined manner or does not have a

predefined data model, thus it is not a good fit for a mainstream relational database. So for

Unstructured data, there are alternative platforms for storing and managing, it is increasingly

prevalent in IT systems and is used by organizations in a variety of business intelligence and

analytics applications. Example: Word, PDF, Text, Media logs.

Examples Of Un-structured Data

Differences between Structured, Semi-structured and Unstructured data:

Differences between Structured, Semi-structured and Unstructured data

Characteristics of Data
The seven characteristics that define data quality are:

1. Accuracy and Precision

2. Legitimacy and Validity

3. Reliability and Consistency

4. Timeliness and Relevance

5. Completeness and Comprehensiveness

6. Availability and Accessibility

7. Granularity and Uniqueness

Accuracy and Precision: This characteristic refers to the exactness of the data. It cannot have
any erroneous elements and must convey the correct message without being misleading. This
accuracy and precision have a component that relates to its intended use. Without
understanding how the data will be consumed, ensuring accuracy and precision could
be off-target or more costly than necessary. For example, accuracy in healthcare might
be more important than in another industry (which is to say, inaccurate data in
healthcare could have more serious consequences) and, therefore, justifiably worth
higher levels of investment.

Legitimacy and Validity: Requirements governing data set the boundaries of this
characteristic. For example, on surveys, items such as gender, ethnicity, and nationality are
typically limited to a set of options and open answers are not permitted. Any answers other
than these would not be considered valid or legitimate based on the survey’s requirement.
This is the case for most data and must be carefully considered when determining its quality.
The people in each department in an organization understand what data is valid or not to
them, so the requirements must be leveraged when evaluating data quality.

Reliability and Consistency: Many systems in today’s environments use and/or collect the
same source data. Regardless of what source collected the data or where it resides, it cannot
contradict a value residing in a different source or collected by a different system. There must
be a stable and steady mechanism that collects and stores the data without contradiction or
unwarranted variance.

Timeliness and Relevance: There must be a valid reason to collect the data to justify the effort
required, which also means it has to be collected at the right moment in time. Data collected
too soon or too late could misrepresent a situation and drive inaccurate decisions.
Completeness and Comprehensiveness: Incomplete data is as dangerous as inaccurate data.
Gaps in data collection led to a partial view of the overall picture to be displayed. Without a
complete picture of how operations are running, uninformed actions will occur. It’s important
to understand the complete set of requirements that constitute a comprehensive set of data
to determine whether or not the requirements are being fulfilled.

Availability and Accessibility: This characteristic can be tricky at times due to legal and
regulatory constraints. Regardless of the challenge, though, individuals need the right level of
access to the data in order to perform their jobs. This presumes that the data exists and is
available for access to be granted.

Granularity and Uniqueness: The level of detail at which data is collected is important,
because confusion and inaccurate decisions can otherwise occur. Aggregated, summarized
and manipulated collections of data could offer a different meaning than the data implied at
a lower level. An appropriate level of granularity must be defined to provide sufficient
uniqueness and distinctive properties to become visible. This is a requirement for operations
to function effectively.

Introduction to Big Data platform

Big Data is a collection of data that is huge in volume, yet growing exponentially with time. It
is a data with so large size and complexity that none of traditional data management tools can
store it or process it efficiently. Big data is also a data but with huge size.

Examples Of Big Data

1. Stock Exchange: The New York Stock Exchange generates about one terabyte of new trade
data per day.

2. Social media: The statistic shows that 500+terabytes of new data get ingested into the
databases of social media site Facebook, every day. This data is mainly generated in terms of
photo and video uploads, message exchanges, putting comments etc.

3. Jet Engine: A single Jet engine can generate 10+terabytes of data in 30 minutes of flight
time. With many thousand flights per day, generation of data reaches up to many Petabytes.

Characteristics Of Big Data

Big data can be described by the following characteristics:

•Volume

•Variety

•Velocity

•Variability
(i) Volume – The name Big Data itself is related to a size which is enormous. Size of data plays
a very crucial role in determining value out of data. Also, whether a particular data can actually
be considered as a Big Data or not, is dependent upon the volume of data. Hence, 'Volume' is
one characteristic which needs to be considered while dealing with Big Data.

(ii) Variety – The next aspect of Big Data is its variety.

Variety refers to heterogeneous sources and the nature of data, both structured and
unstructured. During earlier days, spreadsheets and databases were the only sources of data
considered by most of the applications. Nowadays, data in the form of emails, photos, videos,
monitoring devices, PDFs, audio, etc. are also being considered in the analysis applications.
This variety of unstructured data poses certain issues for storage, mining and analyzing data.

(iii) Velocity – The term 'velocity' refers to the speed of generation of data. How fast the data
is

generated and processed to meet the demands, determines real potential in the data.

Big Data Velocity deals with the speed at which data flows in from sources like business
processes, application logs, networks, and social media sites, sensors, Mobile devices, etc. The
flow of data is massive and continuous.

(iv) Variability – This refers to the inconsistency which can be shown by the data at times, thus

hampering the process of being able to handle and manage the data effectively.

The Need for Data Analytics

Data analytics is essential in today’s digital world, enabling organizations to transform raw
data into meaningful insights. It helps in decision-making, improving efficiency, and gaining a
competitive edge. Below are key reasons why data analytics is crucial:

1. Better Decision-Making

Explanation: Businesses use data analytics to make informed decisions rather than relying on
intuition. It helps in risk assessment and predicting future trends.

A flowchart showing the decision-making process:

Raw Data ➡ Data Processing ➡ Insights Generation ➡ Data-Driven Decision-Making

2. Increased Efficiency & Productivity

Explanation: By analyzing data, companies can optimize operations, reduce costs, and
improve resource allocation.

A process efficiency model:

Current Process ➡ Data Collection ➡ Analysis ➡ Optimization Strategies ➡ Improved

Productivity

3. Customer Insights & Personalization

Explanation: Businesses use analytics to understand customer behavior, preferences, and

trends, leading to personalized experiences.

A customer behaviour analysis model:

Customer Data ➡ Segmentation ➡ Predictive Analytics ➡ Personalized

Recommendations

4. Fraud Detection & Risk Management

Explanation: Banks and financial institutions use analytics to detect anomalies and prevent
fraud.
A fraud detection process:

Transaction Data ➡ Pattern Recognition ➡ Anomaly Detection ➡ Fraud Prevention

5. Competitive Advantage

Explanation: Businesses leverage data analytics to stay ahead of competitors by identifying

market trends and customer needs.

A competitive intelligence cycle:

Market Data ➡ Competitor Analysis ➡ Trend Prediction ➡ Strategic Decisions

 Evolution of Analytic Scalability
Analytic scalability has evolved significantly over time, driven by increasing data volumes,
technological advancements, and the need for real-time insights. Below is an overview of its
evolution:

1. Traditional Analytics (Pre-2000s)

Characteristics:

• Small-scale, structured data (e.g., relational databases).

• Manual data processing using SQL queries and spreadsheets.

• Limited scalability due to hardware constraints.

Small Data ➡ Relational Databases ➡ Manual Processing

2. Big Data Era (2000s - 2010s)

Characteristics:

• Explosion of unstructured and semi-structured data (social media, IoT, web logs).

• Technologies like Hadoop and NoSQL databases enabled large-scale data storage.

• Batch processing for scalability (e.g., MapReduce).

Large Data ➡ Hadoop/NoSQL ➡ Batch Processing

3. Real-Time Analytics (2010s - Present)

Characteristics:

• Need for real-time insights led to technologies like Apache Spark and Kafka.

• Cloud-based solutions improved scalability (AWS, Azure, GCP).

• Streaming analytics and AI-powered decision-making.

Streaming Data ➡ Cloud & AI ➡ Real-Time Processing

4. AI-Driven & Edge Analytics (Future)

Characteristics:

• AI and machine learning automate data processing at scale.

• Edge computing reduces latency by processing data closer to the source.

• Scalable, decentralized analytics for IoT and smart devices.

AI & Edge Devices ➡ Decentralized Processing ➡ Scalable & Real-Time Analytics

Analytic Process and Tools

1. Analytic Process

The analytic process involves a series of steps to extract insights from raw data. Below is a
structured approach:

Step 1: Data Collection

What Happens?

• Data is gathered from multiple sources (databases, APIs, IoT devices, social media,
etc.).

• Can be structured (databases), semi-structured (JSON, XML), or unstructured (text,

images).

Tools: SQL, Apache Kafka, Google BigQuery, AWS Data Pipeline

Data Sources ➡ Data Extraction Tools ➡ Raw Data Storage

Step 2: Data Cleaning & Preparation

What Happens?

• Handling missing values, duplicates, and inconsistencies.

• Transforming data into a usable format.

Tools: Pandas (Python), OpenRefine, Talend, Trifacta

Raw Data ➡ Cleaning & Transformation ➡ Structured Data

Step 3: Data Exploration & Analysis

What Happens?

• Identifying patterns, trends, and relationships in the data.

• Exploratory Data Analysis (EDA) using statistical methods and visualization.

Tools: Python (Pandas, NumPy, Matplotlib, Seaborn), R, Excel, Power BI

Structured Data ➡ Exploratory Analysis ➡ Initial Insights

Step 4: Data Modeling & Machine Learning

What Happens?

• Applying algorithms to find patterns and make predictions.

• Training and validating machine learning models.

Tools: Scikit-learn, TensorFlow, PyTorch, SAS, RapidMiner

Training Data ➡ Model Selection ➡ Predictions & Insights

Step 5: Data Visualization & Reporting

What Happens?

• Converting insights into dashboards, charts, and reports for decision-making.

• Helps stakeholders understand findings easily.

Tools: Tableau, Power BI, Google Data Studio, D3.js

Predictions & Insights ➡ Dashboard & Reports ➡ Business Decisions

Step 6: Decision-Making & Optimization

 What Happens?

• Using insights for strategic decisions.

• Optimizing processes, predicting trends, and automating actions.

Tools: AI-based decision engines, Business Intelligence tools, Automated Systems

Insights ➡ Strategic Decision-Making ➡ Continuous Improvement

Analysis vs. Reporting

Analysis and reporting are two key components of data-driven decision-making, but they
serve different purposes.

1. What is Reporting?

Definition: Reporting is the process of organizing and presenting data in a structured

format to communicate what has happened in the past.

Purpose:

• Provides historical data in a readable format.

• Answers "What happened?"

• Focuses on facts, summaries, and trends.

Examples:

• Monthly sales reports.

• Website traffic dashboards.

• Financial statements.

Raw Data ➡ Summarized Reports ➡ Stakeholder Review

Tools: Tableau, Power BI, Google Data Studio, Excel

2. What is Analysis?

Definition: Analysis involves examining data to uncover patterns, relationships, and

insights that help in decision-making.
 Purpose:

• Answers "Why did it happen?" and "What will happen?"

• Uses statistical methods, machine learning, and predictive models.

• Helps in making proactive business decisions.

Examples:

• Analyzing why sales dropped in a region.

• Predicting customer churn using machine learning.

• Identifying factors affecting customer engagement.

Data Exploration ➡ Patterns & Insights ➡ Business Strategies

Tools: Python (Pandas, Scikit-learn), R, SQL, SAS, IBM SPSS

3. Key Differences: Reporting vs. Analysis

Feature Reporting Analysis

Purpose Describes past events Explains causes, predicts future

Question Why did it happen? & What will

What happened?
Answered happen?

Data aggregation &

Methodology Statistical, predictive modeling
visualization

Tools Tableau, Excel, Power BI Python, R, Machine Learning tools

Outcome Summarized data Actionable insights

4. Analogy

Think of a car dashboard:

• Reporting is like the speedometer and fuel gauge – it tells you what’s happening.

• Analysis is like a mechanic diagnosing why the car is not running efficiently and
predicting maintenance needs.
 Modern Data Analytics Tools
Modern data analytics tools help organizations collect, process, analyze, and visualize data
efficiently. These tools can be categorized based on their primary functions:

1. Data Collection & Integration Tools

These tools gather and integrate data from multiple sources.

Popular Tools:

• Apache Kafka – Real-time data streaming and event processing.

• Talend – ETL (Extract, Transform, Load) tool for data integration.

• Google BigQuery – Cloud-based data warehouse for large-scale data processing.

• AWS Glue – Serverless data integration service for data lakes.

Data Sources (APIs, Databases, IoT, Social Media) ➡ Data Collection Tools ➡ Centralized
Storage

2. Data Storage & Management Tools

These tools store structured and unstructured data efficiently.

Popular Tools:

• SQL Databases (MySQL, PostgreSQL, Microsoft SQL Server) – For structured data.

• NoSQL Databases (MongoDB, Cassandra, Firebase) – For semi-

structured/unstructured data.

• Apache Hadoop – Distributed storage for handling big data.

• Amazon S3 / Google Cloud Storage – Cloud-based storage solutions.

Raw Data ➡ Data Storage Tools ➡ Secure & Scalable Storage

3. Data Processing & Analytics Tools

These tools process and analyze data to generate insights.

Popular Tools:
 • Apache Spark – Big data processing and machine learning.

• Hadoop MapReduce – Batch processing for large datasets.

• Google Dataflow – Real-time and batch data processing.

• Python (Pandas, NumPy, Scikit-learn) – Widely used for statistical and machine
learning analysis.

Stored Data ➡ Processing Tools ➡ Structured & Analyzed Data

4. Data Visualization & Business Intelligence (BI) Tools

These tools present insights in an understandable way through dashboards and reports.

Popular Tools:

• Tableau – Interactive visual analytics.

• Microsoft Power BI – Business intelligence tool for real-time reporting.

• Google Data Studio – Free tool for data visualization.

• D3.js – JavaScript library for custom data visualizations.

Analyzed Data ➡ Visualization Tools ➡ Charts, Graphs & Reports

5. Machine Learning & AI Tools

These tools apply advanced analytics, predictive modeling, and AI-driven decision-making.

Popular Tools:

• TensorFlow / PyTorch – Deep learning frameworks.

• Google Vertex AI / AWS SageMaker – AI model training and deployment.

• IBM Watson – AI-powered analytics for business insights.

• RapidMiner – No-code machine learning and predictive analytics.

Cleaned Data ➡ ML & AI Tools ➡ Predictions & Automation

6. Cloud-Based Analytics Platforms

Cloud platforms provide end-to-end analytics solutions, including storage, processing, and
visualization.

Popular Platforms:

• Google Cloud Platform (GCP) – Includes BigQuery, Dataflow, AI/ML tools.

• Amazon Web Services (AWS) – Offers Redshift, Glue, SageMaker, QuickSight.

• Microsoft Azure – Azure Synapse Analytics, AI & ML services.

Cloud Data Pipelines ➡ Scalable & On-Demand Analytics

Choosing the Right Tool

The right tool depends on your needs:

For data collection → Apache Kafka, Talend
For storage → SQL/NoSQL databases, Hadoop
For analysis → Python, Spark, R
For visualization → Tableau, Power BI
For AI & ML → TensorFlow, Google Vertex AI

Applications of Data Analytics

1. Business and Marketing Analytics

a) Customer Behavior Analysis

• Analyzes customer preferences and purchasing habits.

• Helps businesses personalize marketing strategies.

• Example: E-commerce platforms use recommendation engines (Amazon, Flipkart).

b) Market Basket Analysis

• Identifies relationships between products purchased together.

• Used in cross-selling and upselling strategies.

• Example: Supermarkets place related items together (Diapers & Baby Wipes).

c) Customer Segmentation

• Groups customers based on demographics, interests, and behaviors.

 • Helps in targeted advertising and customized offers.

• Example: Netflix recommends content based on user profiles.

d) Sentiment Analysis

• Uses NLP to analyze customer opinions from social media, reviews, and surveys.

• Helps companies improve customer experience.

• Example: Twitter sentiment analysis for brand reputation management.

2. Healthcare and Medicine

a) Disease Prediction and Diagnosis

• Uses machine learning to detect diseases based on medical records and imaging data.

• Example: AI-powered radiology scans for early cancer detection.

b) Drug Discovery and Development

• Analyzes molecular structures to predict the effectiveness of new drugs.

• Reduces time and cost of drug trials.

• Example: AI in COVID-19 vaccine development.

c) Electronic Health Records (EHR) Analytics

• Helps hospitals manage patient data and predict patient deterioration.

• Example: Predicting hospital readmission rates.

d) Fraud Detection in Healthcare Claims

• Identifies fraudulent insurance claims using anomaly detection.

• Example: Identifying false claims in medical billing.

3. Finance and Banking

a) Fraud Detection

• Uses machine learning to identify suspicious transactions in real time.

• Example: Credit card fraud detection in banks.

b) Risk Management

• Predicts loan defaults and credit risks based on customer history.

 • Example: Credit score calculation by financial institutions.

c) Algorithmic Trading

• Uses predictive analytics to automate stock market trading.

• Example: High-frequency trading by hedge funds.

d) Customer Lifetime Value Prediction

• Estimates how valuable a customer will be over time.

• Example: Banks offering premium services to high-value customers.

4. Manufacturing and Supply Chain Analytics

a) Predictive Maintenance

• Predicts machine failures before they happen using sensor data.

• Example: IoT-based monitoring of factory equipment.

b) Inventory Optimization

• Reduces excess stock and prevents shortages using demand forecasting.

• Example: Amazon’s inventory prediction system.

c) Quality Control and Defect Detection

• Uses computer vision to detect defects in manufacturing lines.

• Example: Automated defect detection in semiconductor production.

d) Logistics and Route Optimization

• Uses analytics to determine the fastest and most cost-effective delivery routes.

• Example: FedEx and UPS optimizing delivery networks.

5. Government and Public Sector

a) Smart Cities and Urban Planning

• Uses real-time data to optimize traffic, waste management, and energy consumption.

• Example: Traffic flow optimization in Singapore.

b) Crime Prediction and Prevention

• Analyzes past crime patterns to allocate police resources effectively.

 • Example: Predictive policing in the US.

c) Tax Fraud Detection

• Identifies fraudulent tax filings and financial irregularities.

• Example: IRS detecting tax evasion.

d) Disaster Management

• Uses data analytics to predict and mitigate the impact of natural disasters.

• Example: Predicting flood-prone areas using satellite data.

6. Retail and E-Commerce

a) Personalized Recommendations

• Uses collaborative filtering to suggest products based on past purchases.

• Example: Amazon’s recommendation engine.

b) Dynamic Pricing

• Adjusts product prices in real-time based on demand and competitor prices.

• Example: Airline ticket pricing systems.

c) Customer Churn Prediction

• Identifies customers likely to stop using a service.

• Example: Telecom companies predicting subscriber cancellations.

d) Demand Forecasting

• Predicts future product demand based on historical data.

• Example: Seasonal sales forecasting in retail.

7. Sports and Gaming

a) Performance Analytics

• Tracks player performance using motion sensors and game statistics.

• Example: Cricket teams using AI for player improvement.

b) Injury Prediction and Prevention

• Uses biomechanics data to prevent injuries in athletes.

 • Example: Wearable sensors predicting stress injuries.

c) Game Strategy Optimization

• Analyses opponent strategies to suggest better gameplay.

• Example: AI-powered coaching in football.

d) E-Sports and Gaming Analytics

• Tracks in-game behaviours and player engagement.

• Example: Improving in-game experience in multiplayer online games.

8. Education and Learning Analytics

a) Personalized Learning

• Adapts coursework based on student performance and engagement.

• Example: AI-driven adaptive learning platforms like Coursera.

b) Dropout Prediction

• Identifies students at risk of dropping out based on performance metrics.

• Example: Universities using predictive analytics for student retention.

c) Skill Gap Analysis

• Helps companies assess workforce skills and training needs.

• Example: HR analytics in corporate training.

d) Exam and Assessment Analytics

• Uses AI to grade essays and assess knowledge gaps.

• Example: Automated grading systems.

9. Social Media and Entertainment

a) Social Media Analytics

• Measures engagement, sentiment, and trends in social media content.

• Example: Instagram and Facebook’s ad targeting algorithms.

b) Fake News Detection

• Uses NLP to detect misinformation and deepfakes.

 • Example: AI-powered fact-checking tools.

c) Video and Audio Analytics

• Analyzes user preferences for content recommendations.

• Example: YouTube’s video recommendation system.

d) Audience Engagement Analysis

• Determines which content generates the most user interaction.

• Example: Netflix optimizing content production.

10. Cybersecurity and Network Analytics

a) Threat Detection

• Identifies abnormal network activity that may indicate cyberattacks.

• Example: AI-driven Intrusion Detection Systems (IDS).

b) Phishing Attack Prevention

• Detects fraudulent emails and websites.

• Example: Google’s AI-powered spam filters.

c) Network Traffic Analysis

• Monitors real-time network traffic for anomalies.

• Example: Preventing DDoS attacks.

d) Identity Verification

• Uses biometric data for secure authentication.

• Example: AI-powered facial recognition in banking.

Data Analytics Lifecycle

1. Introduction: Need for Data Analytics Lifecycle
The Data Analytics Lifecycle is a structured approach to executing analytics projects
efficiently. The need for this lifecycle arises due to:

• Growing Data Complexity: Handling vast volumes of structured and unstructured

data.
 • Business Decision-Making: Ensuring data-driven strategies in organizations.

• Improved Predictive Accuracy: Refining models for better forecasts.

• Scalability: Making analytics solutions robust and reusable.

• Efficiency: Reducing time and resources spent on analytics projects.

A well-defined lifecycle helps in systematically processing data, deriving meaningful insights,

and ensuring successful implementation of data-driven solutions.

2. Key Roles in Successful Analytics Projects

A successful data analytics project requires collaboration among various roles:

a) Business Analyst

• Understands business requirements.

• Bridges the gap between technical teams and stakeholders.

• Defines key performance indicators (KPIs) and success metrics.

b) Data Engineer

• Manages data collection, storage, and pipeline development.

• Ensures data quality, transformation, and accessibility.

c) Data Scientist

• Develops and applies statistical models and machine learning techniques.

• Works on predictive analytics, clustering, and pattern recognition.

d) Machine Learning Engineer

• Deploys machine learning models into production environments.

• Optimizes algorithms for scalability and performance.

e) Data Analyst

• Performs exploratory data analysis (EDA) and visualization.

• Creates dashboards and reports for insights.

f) Domain Expert

• Provides industry-specific knowledge to validate data interpretations.

• Ensures that analytical results align with real-world scenarios.

g) Project Manager

• Oversees the project timeline, scope, and deliverables.

• Ensures coordination between different teams.

Having the right combination of these roles ensures an effective and successful analytics
implementation.

3. Phases of the Data Analytics Lifecycle

The Data Analytics Lifecycle consists of six phases:

Phase 1: Discovery

Objective:

• Understand business problems, project objectives, and data requirements.

• Identify potential sources of data.

Key Tasks:

• Define project scope and success criteria.

• Identify stakeholders and required resources.

• Assess data availability and limitations.

Deliverables:

• Project charter.

• Initial hypotheses or research questions.

• Preliminary data inventory.

Phase 2: Data Preparation

Objective:

• Collect, clean, and preprocess raw data to make it suitable for analysis.

Key Tasks:

• Data extraction from various sources (databases, APIs, logs).

• Data cleaning (handling missing values, duplicates, inconsistencies).

• Feature engineering (creating new variables, transformations).

Deliverables:

• Cleaned and structured dataset.

• Data dictionary or metadata documentation.

• Exploratory Data Analysis (EDA) reports.

Phase 3: Model Planning

Objective:

• Select appropriate models and analytical techniques.

• Define an experimental setup for testing models.

Key Tasks:

• Identify statistical or machine learning methods to be used.

• Perform feature selection and transformation.

• Split data into training and testing sets.

• Decide on model evaluation metrics (e.g., accuracy, RMSE, precision-recall).

Deliverables:

• Model selection framework.

• Data partitions for model training and validation.

• Hypotheses about feature relationships.

Phase 4: Model Building

Objective:

• Develop, train, and evaluate models based on selected methodologies.

Key Tasks:

• Implement machine learning algorithms (e.g., regression, decision trees, neural

networks).

• Tune hyperparameters for optimal performance.

• Perform cross-validation to avoid overfitting.

• Compare multiple models and select the best-performing one.

Deliverables:

• Trained model(s) with performance metrics.

• Model evaluation reports.

• Code/scripts for reproducibility.

Phase 5: Communicating Results

Objective:

• Interpret and present findings to stakeholders in an understandable format.

Key Tasks:

• Create visualizations, reports, and dashboards.

• Translate model insights into business recommendations.

• Address limitations and potential risks.

Deliverables:

• Summary report with key insights.

• Visualizations (graphs, charts, dashboards).

• Recommendations for business strategy.

Phase 6: Operationalization

Objective:

• Deploy the model into production and monitor its performance.

Key Tasks:

• Implement the model into an operational environment (e.g., cloud services, APIs).

• Automate model predictions and integrate with business applications.

• Monitor model performance and retrain if necessary.

• Ensure compliance with ethical and regulatory standards.

Deliverables:

• Deployed model with real-time or batch processing capabilities.

• Model monitoring and maintenance plan.

 • Final project documentation and lessons learned.

4. Summary
The Data Analytics Lifecycle provides a structured framework to develop data-driven
solutions effectively. Each phase plays a critical role in ensuring the accuracy, relevance, and
scalability of analytics projects. Successful execution depends on:

1. A clear problem definition in the Discovery phase.

2. Proper data preparation to avoid biases and errors.

3. Selection of appropriate modeling techniques in Model Planning and Building.

4. Effective communication of results to stakeholders.

5. Seamless deployment and maintenance of models for long-term impact.

By following this lifecycle, organizations can extract valuable insights from data and make
informed decisions.