Data

dataset

Data Scientist- person who uses data to solve the problems, uncover patterns and gain insights.

Data scientist may leverage knowledge from statistics, computer programming, data bases, machine
learning and mathematical models.

A day in the life of data scientist:

Understand the organisational issues

acquire and gather the data: hypothesisi created

Check the data and analysed using visualisation tools: Hypothesis testing using stats

Insights communicated to the stakeholders

Importance of data scientist: to make better decisions by getting better insights

History of innovation cycles:

Creative destruction plays a key role in entrepreneurship and economic development.

Coined by economist Joseph Schumpeter in 1942, the theory of “creative destruction” suggests that
business cycles operate under long waves of innovation. Specifically, as markets are disrupted, key
clusters of industries have outsized effects on the economy.

The image illustrates the history of innovation cycles as identified by economist Joseph Schumpeter,
who introduced the concept of "creative destruction" in 1942. This theory connects technological
and economic advancements to long-wave business cycles spanning approximately 250 years. Each
wave of innovation is characterized by the emergence and maturation of specific technologies or
industries, transforming economies and societies.

Breakdown of Innovation Cycles:

1. First Wave (1785 - 1845):

 Core Technologies: Water power, textiles, iron

 Key Innovations: The Industrial Revolution marked the emergence of the first factories, such
as the cotton mill in Britain. The widespread use of water power and advancements in textile
machinery revolutionized production processes.

 Duration: 60 years

2. Second Wave (1845 - 1900):

 Core Technologies: Steam power, rail, steel

 Key Innovations: The expansion of rail networks and the industrialization of steel production
transformed urban growth and connectivity. Steam power fueled significant industrial and
logistical progress.

 Duration: 55 years

3. Third Wave (1900 - 1950):

 Core Technologies: Electricity, chemicals, internal combustion engine

 Key Innovations: The introduction of the assembly line (e.g., Henry Ford’s Model T) brought
revolutionary changes to manufacturing. Chemical industries and the internal combustion
engine spurred advancements in mobility and urbanization.

 Duration: 50 years

4. Fourth Wave (1950 - 1990):

 Core Technologies: Petrochemicals, electronics, aviation

 Key Innovations: The rise of aviation enabled global connectivity, while petrochemicals and
electronic innovations supported industrial and consumer growth. This era saw mass
production and the development of global supply chains.

 Duration: 40 years

5. Fifth Wave (1990 - 2020):

 Core Technologies: Digital networks, software, new media

  Key Innovations: The internet revolutionized communication and commerce. By 2016, over
3.4 billion people globally were using the internet, leading to the emergence of software-
driven industries, social media, and e-commerce.

 Duration: 30 years

6. Sixth Wave (2020 - Present):

 Core Technologies: AI, IoT (Internet of Things), robotics, clean tech

 Key Innovations: This wave addresses challenges like climate change by emphasizing
sustainability and clean technologies. Innovations in AI, robotics, and IoT reshape industries,
driving efficiency and creating intelligent systems.

 Duration: Ongoing (estimated at 25 years or more)

Observations:

1. Shortening Cycles: The duration of innovation waves has progressively decreased, reflecting
the accelerating pace of technological advancement.

2. Interconnectedness: Later waves build upon previous innovations, enhancing and

diversifying their applications (e.g., electronics evolving into digital networks).

3. Sustainability Focus: The sixth wave prioritizes solving global challenges, such as climate
change, highlighting a shift towards sustainable and eco-friendly technologies.

This framework underscores how technological innovation drives economic growth and societal
transformation across generations.

MARKET POWER

To the economist Schumpeter, technological innovations boosted economic growth and improved
living standards.

However, these disruptors can also have a tendency to lead to monopolies. Especially during a cycle’s
upswing, the strongest players realize wide margins, establish moats, and fend off rivals. Typically,
these cycles begin when the innovations become of general use.

Of course, this can be seen today—never has the world been so closely connected. Information is
more centralized than it has ever been, with Big Tech dominating global search traffic, social
networks, and advertising.

Like the Big Tech behemoths of today, the rail industry had the power to control prices and push out
competitors during the 19th century. At the peak, listed shares of rail companies on the New York
Stock Exchange made up 60% of total stock market capitalization.

WAVES OF CHANGE

As cycle longevity continues to shorten, the fifth wave may have a few years left under its belt.

The sixth wave, marked by artificial intelligence and digitization across information of things (IoT),
robotics, and drones, will likely paint an entirely new picture. Namely, the automation of
systems, predictive analytics, and data processing could make an impact. In turn, physical goods and
services will likely be digitized. The time to complete tasks could shift from hours to even seconds.
At the same time, clean tech could come to the forefront. At the heart of each technological
innovation is solving complex problems, and climate concerns are becoming increasingly pressing.
Lower costs in solar PV and wind are also predicating efficiency advantages.

The growth in real-time analytics is being driven by several key factors that address modern business
challenges and opportunities:

1. Customer Growth & Retention

 Businesses need to provide personalized, real-time experiences to attract and retain

customers.

 For example, companies like Spotify use real-time data to recommend songs based on user
preferences, increasing customer satisfaction and loyalty.

 A streaming service like Netflix uses real-time analytics to recommend shows or movies to
users based on their viewing history. By analyzing user behavior in real time, Netflix ensures
customers find content they love, encouraging them to stay subscribed and recommend the
platform to others.

2. Innovation

 Real-time analytics enables rapid experimentation and innovation by providing instant

feedback on new products, services, or strategies.

 For example, Tesla uses real-time analytics from vehicle sensors to improve features and
develop new autonomous driving capabilities.

 A retail company like Amazon launches a new product category. Using real-time analytics,
they monitor customer feedback, sales data, and browsing patterns. This helps Amazon
identify whether the new product is successful and make rapid adjustments (e.g., improving
recommendations or offering discounts).

3. Business Optimization
  Companies use real-time data to optimize operations, minimize costs, and enhance
productivity.

 For instance, Uber dynamically adjusts pricing based on real-time demand and supply data,
maximizing efficiency and revenue.

 An airline company like Delta uses real-time analytics to optimize operations. By analyzing
flight data, weather conditions, and passenger traffic, they adjust routes, manage fuel
consumption, and allocate resources efficiently, reducing operational costs while maintaining
high service quality

4. Global 24/7 Operations

 Businesses operating globally must ensure constant availability and performance.

 For example, Amazon Web Services (AWS) uses real-time monitoring to prevent system
downtimes and maintain uninterrupted service across global regions.

These drivers are fueled by advancements in data collection technologies, the growing importance of
customer experience, and the need to remain competitive in rapidly changing markets.

AMAZON

1. Customer Growth & Retention

 Amazon Example: Amazon uses real-time analytics to personalize the customer experience.
For instance, when a customer browses products, Amazon's recommendation engine
suggests similar or complementary items based on real-time browsing and purchase
behavior.

o Result: This keeps existing customers engaged and attracts new ones by offering
tailored recommendations and promotions.

2. Innovation

 Amazon Example: Real-time analytics helps Amazon experiment with new features like
Amazon Go (checkout-free stores). By analyzing real-time data from cameras and sensors,
they quickly determine if the concept works and refine it based on customer behavior.

o Result: This innovation helps Amazon lead the market with cutting-edge services and
products.

3. Business Optimization

 Amazon Example: Amazon's supply chain relies on real-time analytics to optimize inventory
levels, reduce costs, and ensure timely delivery. For instance, it uses predictive analytics to
restock warehouses before items run out.

o Result: Reduced operational costs and efficient resource usage while ensuring
customer satisfaction with faster deliveries.
4. Global 24/7 Operations

 Amazon Example: With operations spanning the globe, Amazon uses real-time analytics to
monitor its servers, warehouses, and delivery networks. This ensures seamless shopping
experiences for customers worldwide.

o Example: During peak events like Prime Day, real-time data ensures uninterrupted
service, even with high traffic volumes.

o Result: Amazon avoids downtime, ensuring that customers can shop anytime
without disruption.

By leveraging real-time analytics across these drivers, Amazon has set a benchmark for customer-
centric innovation, operational excellence, and global service reliability.

This chart highlights the factors driving interest in big data analysis across organizations in 2014 and
2015. It shows the percentage of respondents identifying specific data sources or challenges as
motivators for using big data analytics. Here's a breakdown of the insights:

1. Finding Correlations Across Disparate Data Sources (e.g., clickstreams, geospatial, transactions):

 48% in 2015 (up from 43% in 2014)

 Explanation: Organizations increasingly focus on integrating multiple types of data to

uncover valuable insights, such as customer behavior patterns or operational inefficiencies.
2. Predicting Customer Behavior:

 46% in 2015 (slightly up from 44% in 2014)

 Explanation: Companies prioritize understanding customer preferences and actions to

personalize marketing and improve customer retention strategies.

3. Predicting Product or Service Sales:

 40% in 2015 (up from 36% in 2014)

 Explanation: Big data helps forecast demand, optimize inventory, and align production with
market trends.

4. Predicting Fraud or Financial Risk:

 32% in 2015 (up from 27% in 2014)

 Explanation: Organizations are leveraging analytics to detect fraudulent activities or mitigate

financial risks in real-time.

5. Analyzing Social Network Comments for Consumer Sentiment:

 29% in 2015 (up from 24% in 2014)

 Explanation: Businesses analyze social media to gauge public opinion, understand brand
perception, and tailor marketing strategies.

6. Analyzing High-Scale Machine Data from Sensors, Web Logs, etc.:

 28% in 2015 (up from 23% in 2014)

 Explanation: Industries like manufacturing and IoT use sensor data to monitor operations
and predict maintenance needs.

7. Identifying Computer Security Risks:

 29% in 2015 (up from 25% in 2014)

 Explanation: Real-time analytics is critical for detecting and addressing cybersecurity threats.

8. Analyzing Web Clickstreams:

 24% in 2015 (slightly down from 26% in 2014)

  Explanation: Web clickstream analysis remains a focus for optimizing website usability and
tracking customer journeys.

9. Other Factors:

 Minimal growth (1% in 2015, 3% in 2014)

 Indicates niche or organization-specific applications of big data.

10. Lack of Interest in Big Data Analytics:

 Declined from 14% in 2014 to 12% in 2015

 Explanation: Organizations increasingly recognize the value of big data analytics, reflecting
growing adoption rates.

Overall Insight:

The chart reveals a consistent upward trend in the adoption of big data analytics for predictive
modeling, customer behavior insights, and operational efficiencies, driven by an increasing ability to
integrate and analyze diverse data sources.

Data science and data analytics are closely related fields but differ in scope and focus. Here's a
comparison of their applications:

Applications of Data Science

Data science involves using complex algorithms, machine learning models, and statistical methods to
extract insights from large datasets and make predictions.

1. Predictive Analytics

o Example: Forecasting customer demand or stock prices using machine learning

models.

o Use Case: E-commerce companies predicting product trends.

2. Recommendation Systems

o Example: Netflix or Amazon suggesting movies or products based on user behavior.

o Use Case: Improving customer experience and engagement.

3. Fraud Detection

o Example: Using anomaly detection algorithms to spot fraudulent transactions.

o Use Case: Banking and financial services.

4. Natural Language Processing (NLP)

 o Example: Chatbots, sentiment analysis, and language translation.

o Use Case: Customer support or analyzing social media sentiment.

5. Autonomous Systems

o Example: Training models for self-driving cars to interpret surroundings.

o Use Case: Automotive and robotics industries.

6. Healthcare Diagnostics

o Example: Predicting diseases using patient data (e.g., cancer detection through
image recognition).

o Use Case: Hospitals and pharmaceutical companies.

7. Image and Video Analysis

o Example: Facial recognition systems or medical imaging analysis.

o Use Case: Security, healthcare, and media.

Applications of Data Analytics

Data analytics focuses on extracting actionable insights from data, typically answering specific
questions about past performance.

1. Business Intelligence and Reporting

o Example: Dashboards showing sales trends over time.

o Use Case: Retail companies tracking sales performance.

2. Customer Behavior Analysis

o Example: Analyzing customer purchase patterns to optimize marketing campaigns.

o Use Case: CRM and targeted marketing.

3. Operational Efficiency

o Example: Identifying bottlenecks in supply chains to reduce costs.

o Use Case: Logistics and manufacturing.

4. Market Basket Analysis

o Example: Finding products frequently purchased together (e.g., bread and butter).

o Use Case: Retailers designing promotions.

5. Web and Social Media Analytics

o Example: Analyzing website traffic and engagement to improve user experience.

o Use Case: E-commerce and digital marketing.

 6. Financial Analysis

o Example: Examining cash flow or budget allocation for better resource management.

o Use Case: Corporate finance and accounting.

7. Trend Identification

o Example: Monitoring seasonal trends in product sales.

o Use Case: Fashion and consumer goods industries.

Key Difference

 Data Science: Predictive and prescriptive analytics using machine learning and AI for future-
focused insights.

 Data Analytics: Descriptive and diagnostic analytics focused on understanding past and
present data to drive decision-making.

Both fields are vital in modern businesses, often overlapping, with data science providing advanced
tools for problems that require deeper exploration or prediction, while data analytics helps with day-
to-day decision-making.
The dashboard shown is an example of how business analytics can be applied to analyze and
visualize car sales data. Let's break down the key sections and relate them to the applications of
business analytics:

1. Car Sales Analysis - Color and Gender (Bar Chart)

 What it shows:

o Car sales are broken down by car color and further segmented by gender (male and
female buyers).

 Application of Business Analytics:

o Customer Segmentation: Helps identify which colors are more popular among men
versus women. For instance, "Green" appears to be the most popular color among
both genders, but with higher sales for males.

o Marketing Strategy: Companies can design gender-specific marketing campaigns for

car colors (e.g., promoting blue cars to men and turquoise cars to women).

o Product Development: Insights can guide manufacturers on which color preferences

to prioritize in different regions.
2. Car Sales by Country (Map)

 What it shows:

o A geographical representation of car sales, where darker shades indicate higher

sales. For example, China (CN) has the highest sales, followed by Russia (RU) and
Brazil (BR).

 Application of Business Analytics:

o Market Penetration Analysis: Identifies regions with high and low sales to focus
sales efforts on underperforming regions.

o Regional Strategy: Companies can allocate resources, like advertising budgets or

dealerships, to countries with higher demand.

o Supply Chain Optimization: Insights from this map can optimize inventory and
shipping based on regional sales trends.

3. Car Color and Gender Distribution (Pie Chart)

 What it shows:

o The percentage contribution of each car color to overall sales, segmented by gender.
Green has the highest share (19.3%), followed by Red (8.1%).

 Application of Business Analytics:

o Product Mix Optimization: Helps businesses decide the proportion of each color to
produce based on demand.

o Customer Insights: Businesses can identify preferences among demographics and

tailor offers (e.g., discounts on less popular colors to clear inventory).

4. Car Sales by Day (Line Chart)

 What it shows:

o Daily car sales trends, segmented by currency (EUR and USD). Peaks and dips
indicate variations in daily sales performance.

 Application of Business Analytics:

o Trend Analysis: Identifies high and low sales days. For example, sales peaked sharply
on the 9th.

o Demand Forecasting: Businesses can predict future sales trends and prepare
inventory accordingly.

o Campaign Timing: Companies can align marketing campaigns with periods of high
customer activity.
Conclusion (Business Analytics Application)

 Descriptive Analytics: The dashboard summarizes historical data to show trends, patterns,
and regional performance.

 Diagnostic Analytics: Helps answer "why" certain trends occur (e.g., green cars are more
popular).

 Predictive Analytics: Can guide decisions about future production, marketing, and sales
strategies by predicting demand.

 Prescriptive Analytics: Suggests actions, such as increasing inventory for green cars in China
or targeting discounts on red cars.

This dashboard demonstrates how business analytics transforms raw sales data into meaningful
insights that support better decision-making across marketing, sales, supply chain, and product
development.

BIG DATA

Large volume of data – both structured and unstructured – that inundates a business on a day-to-day
basis. Big data refers to the incredible amount of structured and unstructured information that
humans and machines generate—petabytes every day, according to PwC. It’s the social posts we
mine for customer sentiment, sensor data showing the status of machinery, financial transactions
that move money at hyperspeed. It’s also too massive, too diverse, and comes at us way too fast for
old-school data processing tools and practices to stand a chance.

It’s also much too valuable to leave unanalyzed. Big data infers the ability to extract insights from this
broad collection of data to help an organization become more efficient, innovate faster, earn more
money, and just all around win.

Luckily, advancements in analytics and machine learning technology and tools make big data analysis
accessible for every company.

Big data refers to extremely large and complex data sets that cannot be easily managed or analyzed
with traditional data processing tools, particularly spreadsheets. Big data includes structured data,
like an inventory database or list of financial transactions; unstructured data, such as social posts
or videos; and mixed data sets, like those used to train large language models for AI. These data
sets might include anything from the works of Shakespeare to a company’s budget spreadsheets for
the last 10 years.

Big data has only gotten bigger as recent technological breakthroughs have significantly reduced the
cost of storage and compute, making it easier and less expensive to store more data than ever
before. With that increased volume, companies can make more accurate and precise business
decisions with their data. But achieving full value from big data isn’t only about analyzing it—which is
a whole other benefit. It’s an entire discovery process that requires insightful analysts, business
users, and executives who ask the right questions, recognize patterns, make informed assumptions,
and predict behavior.

Early 2000s, industry analyst Doug Laney articulated the now-mainstream definition of big data as
the three V’s
VOLUME:  This is about the amount of data.

 Big data involves handling a huge quantity of information, often unorganized or incomplete.

 For example, think of all the tweets on X (formerly Twitter), clicks on websites, or signals from
sensors in machines. Some organizations handle data as small as a few terabytes (thousands of
gigabytes), while others deal with massive amounts—hundreds of petabytes (millions of gigabytes).

VELOCITY:  This refers to how fast data is coming in.

 Data flows in quickly, and organizations often need to process it just as fast—sometimes in real
time.

 For instance, think about notifications you get instantly on your phone or the quick response of a
smart home device. This is all about acting on data as it arrives.

VARIETY:  This means different types of data.

 In the past, data was structured, like numbers in tables. Now, we have unstructured data—like
videos, voice recordings, photos, or social media posts—that need special tools to make sense of
them.

Two other V’s:

1. Veracity:

o This is about how accurate or trustworthy the data is.

o Data can be messy or incomplete, and its quality affects how useful it is. Good data
helps businesses make better decisions, while bad data can lead to mistakes.

2. Value:

o Data becomes valuable only when we extract useful insights from it.

o For example, businesses can use data to improve their processes, understand
customers better, or create new products. The real challenge is to find meaningful
insights in the sea of information.

In short, big data is about managing large, fast-moving, and diverse data to make it accurate and
useful for solving real-world problems or driving better business decisions.

Big Data Benefits

 Better insights. When organizations have more data, they’re able to derive better insights. In
some cases, the broader range confirms gut instincts against a more diverse set of
circumstances. In other cases, a larger pool of data uncovers previously hidden connections
and expands potentially missed perspectives. All of this allows organizations to have a more
comprehensive understanding into the how and why of things, particularly when automation
allows for faster, easier processing of big data.

 Decision-making. With better insights, organizations can make data-driven decisions with
more reliable projections and predictions. When big data combines with automation and
analytics, that opens an entire range of possibilities, including more up-to-date market
trends, social media analysis, and patterns that inform risk management.
  Personalized customer experiences. Big data allows organizations to build customer profiles
through a combination of customer sales data, industry demographic data, and related data
such as social media activity and marketing campaign engagement. Before automation and
analytics, this type of personalization was impossible due to its sheer scope; with big data,
this level of granularity improves engagement and enhances the customer experience.

 Improved operational efficiency. Every department generates data, even when teams don’t
really think about it. That means that every department can benefit from data on an
operational level for tasks such as detecting process anomalies, identifying patterns for
maintenance and resource use, and highlighting hidden drivers of human error. Whether
technical problems or staff performance issues, big data produces insights about how an
organization operates—and how it can improve.

Big data use case:

IT infra optimisation

Aadhar project

Advertisement analysis

Weather forecasting

Predictive analysis

Healthcare analysis

Customer churn analysis

Natural resource exploration

Big Data Examples

• The New York Stock Exchange (NYSE) produces one terabyte of new trade data every day.

• Each day 500 million tweets are sent.

• Amazon, in order to recommend products, on average, handles more than 15 million+

customer clickstreams per day.

• Walmart an American Multinational Retail Corporation handle about 1 million+ customer

transactions per hour.

• 65 billion+ messages are sent on Whatsapp every day.

Big Data Examples

• A single Jet engine generates more than 10 terabytes of data in-flight time of 30 minutes.

• On average, everyday 294 billion+ emails are sent.

• Modern cars have close to 100 sensors for monitoring tire pressure, fuel level, etc. , thus
generating a lot of sensor data.

• Facebook stores and analyzes more than 30 Petabytes of data generated by the users each
day.

• YouTube users upload about 48 hours of video every minute of the day.
 BUSINESS ANALYTICS LIFECYCLE

The lifecycle of business analytics follows a systematic approach to deriving insights and
making data-driven decisions. It consists of several stages, which can vary slightly depending
on methodologies, but the core steps typically include the following:

1. Identifying the Business Problem

 Objective: Clearly define the problem or opportunity.

 Activities:

o Understand business goals and key performance indicators (KPIs).

o Collaborate with stakeholders to frame the problem.

 Example: A retail company wants to reduce customer churn.

2. Data Collection

 Objective: Gather relevant data from various sources.

 Activities:

o Identify internal and external data sources (e.g., databases, web logs, customer
surveys).

o Extract and consolidate data.

 Example: Collect transaction data, customer demographics, and feedback.

3. Data Preparation

 Objective: Clean, transform, and prepare data for analysis.

 Activities:

o Handle missing, duplicate, or inconsistent data.

o Format data for compatibility (e.g., normalize, encode categorical variables).

 Example: Remove null entries, correct typos, and create a single dataset for analysis.

4. Data Exploration and Visualization

 Objective: Understand data patterns and relationships.

 Activities:

o Perform exploratory data analysis (EDA) to identify trends and anomalies.

o Use visualization tools like Power BI, Tableau, or Python libraries (e.g., Matplotlib).
 Example: Create charts to observe seasonality in sales trends.

5. Model Building

 Objective: Apply statistical methods or machine learning algorithms to analyze data.

 Activities:

o Choose the right model based on the business problem (e.g., regression,
classification, clustering).

o Train and test the model on the prepared data.

 Example: Use a predictive model to forecast customer churn probability.

6. Model Evaluation

 Objective: Validate the performance of the model.

 Activities:

o Use evaluation metrics like accuracy, precision, recall, or RMSE (Root Mean Square
Error).

o Refine the model if necessary by tuning parameters or testing additional features.

 Example: Test how well the model predicts churn based on historical data.

7. Deployment and Decision-Making

 Objective: Implement the model's results into business processes.

 Activities:

o Deploy predictive models into production systems.

o Provide actionable insights to decision-makers.

 Example: Use churn predictions to target at-risk customers with retention strategies.

8. Monitoring and Optimization

 Objective: Continuously assess model performance and business outcomes.

 Activities:

o Monitor KPIs to ensure goals are met.

o Update models and analytics processes as data or business conditions change.

 Example: Re-train the churn prediction model with new customer behavior data.
 Iterative Nature

Business analytics is an iterative process. Once the results are applied, new problems or
opportunities may arise, and the lifecycle restarts, refining the approach with more data and
better models.

Tools Used Across Lifecycle:

 Data Collection: SQL, APIs, ETL tools (e.g., Talend, Informatica).

 Data Preparation: Excel, Python (Pandas), R.

 Visualization: Tableau, Power BI, Matplotlib, Seaborn.

 Modeling: Python (Scikit-learn, TensorFlow), R, SAS.

 Deployment: AWS, Azure, Google Cloud.

This structured lifecycle ensures that analytics efforts are aligned with business goals and
deliver measurable value.