unit 5_Real time Data Analysis vsp.pptx

UNIT 5:
REAL TIME DATA ANALYSIS
DR.V.S.PRAKASH | V BCA 1

TIME SERIES
Time series analysis is a crucial part of data analysis, and Python provides several tools and libraries for working
with time series data.

Date and Time Data Types and Tools:
Datetime Module:
Python's standard library includes the datetime module, which provides classes for working with dates and
times. You can use the datetime class to represent both date and time information.
from datetime import datetime
now = datetime.now() # Current date and time
print(now)
Date and Time Objects:
The datetime module includes date and time classes that allow you to work with just the date or time portion.

String to Datetime Conversion:
You can convert a string representing a date and time to a datetime object using the strptime method.
date_str = "2023-10-24"
datetime_obj = datetime.strptime(date_str, "%Y-%m-%d")
Datetime to String Conversion:
To convert a datetime object back to a string, you can use the strftime method.
formatted_date = datetime_obj.strftime("%Y-%m-%d")

TIME SERIES BASICS:
Time series data consists of data points collected or recorded at successive, equally spaced time intervals.
Common examples of time series data include stock prices, temperature measurements, and website traffic.
Here are some fundamental concepts and tools for working with time series data in Python:
Pandas: The Pandas library is a powerful tool for working with time series data. It provides data structures like
DataFrame and Series that are ideal for organizing and analyzing time series data.
import pandas as pd
time_series_data = pd.Series([10, 20, 30, 40], index=pd.date_range(start='2023-10-01', periods=4, freq='D')

Time Resampling:
You can resample time series data to change the frequency of data points (e.g., from daily to monthly) using
Pandas' resample method.
Pandas dataframe.resample() function is primarily used for time series data.
A time series is a series of data points indexed (or listed or graphed) in time order. Most commonly, a time
series is a sequence taken at successive equally spaced points in time. It is a Convenience method for
frequency conversion and resampling of time series.
monthly_data = time_series_data.resample('M').mean()
Plotting Time Series Data:
Libraries like Matplotlib and Seaborn can be used to visualize time series data.
import matplotlib.pyplot as plt
time_series_data.plot()
plt.xlabel("Date")
plt.ylabel("Value")
plt.show()

Time Series Analysis:
You can perform various time series analysis tasks, including trend analysis, seasonality detection, and
forecasting, using tools like Statsmodels and Scikit-learn.
from statsmodels.tsa.seasonal import seasonal_decompose
decomposition = seasonal_decompose(time_series_data)
trend = decomposition.trend
seasonal = decomposition.seasonal
Time series decomposition involves thinking of a series as a combination of level, trend, seasonality, and noise
noise components.
• Level: The average value in the series.
• Trend: The increasing or decreasing value in the series.
• Seasonality: The repeating short-term cycle in the series.
• Noise: The random variation in the series.

INDEXING AND SELECTION:
Selecting by Index:
You can access specific elements in a time series using index labels.
import pandas as pd
time_series_data = pd.Series([10, 20, 30, 40], index=pd.date_range(start='2023-10-01', periods=4, freq='D'))
selected_data = time_series_data['2023-10-01']
Selecting by Slicing: Use slicing to select a range of data.
selected_range = time_series_data['2023-10-01':'2023-10-03']
Subsetting by Conditions:
You can subset time series data based on conditions using boolean indexing.
subset = time_series_data[time_series_data > 20]

Date Ranges and Frequencies:
Date Ranges: You can create date ranges using Pandas' date_range function. This is useful for generating date
index for time series data.
date_range = pd.date_range(start='2023-10-01', end='2023-10-10', freq='D')
Frequencies: You can specify various frequencies when creating date ranges. Common frequencies include 'D'
(day), 'H' (hour), 'M' (month end), and more.
hourly_range = pd.date_range(start='2023-10-01', periods=24, freq='H')
monthly_range = pd.date_range(start='2023-01-01', end='2023-12-31', freq='M')
Shifting Data:
Shifting is a common operation when working with time series data, often used for calculating differences or
creating lag features.
Shift Data: You can shift the data forward or backward using the shift method.
shifted_data = time_series_data.shift(periods=1) # Shift data one period forward
Calculating Differences: To compute the difference between consecutive values, you can subtract the shifted
series.

SHIFTING DATA:
Shifting data involves moving time series data forward or backward in time. This is useful for various time series
analysis tasks.
Shift Data: You can shift data using Pandas' shift method.
The simplest call should have an argument periods (It defaults to 1) and it represents the number of shifts for the
desired axis. And by default, it is shifting values vertically along the axis 0 . NaN will be filled for missing values
introduced as a result of the shifting.
import pandas as pd
# Shifting data one period forward
shifted_data = time_series_data.shift(periods=1)
# Shifting data two periods backward
shifted_data = time_series_data.shift(periods=-2)
Calculating Differences: Shifting is often used to calculate the differences between consecutive values.
# Calculate the difference between consecutive values

Generating Date Ranges and Frequencies:
Pandas provides powerful tools for generating date ranges with different frequencies.
Date Ranges: Use the date_range function to create date ranges.
date_range = pd.date_range(start='2023-01-01', end='2023-12-31', freq='D') # Daily frequency
Frequencies: You can specify various frequencies such as 'D' (day), 'H' (hour), 'M' (month end), 'Y' (year end),
and more when creating date ranges.
hourly_range = pd.date_range(start='2023-01-01', periods=24, freq='H')
monthly_range = pd.date_range(start='2023-01-01', end='2023-12-31', freq='M')

Time Zone Handling:
Pandas can handle time zones and convert between them.
Setting Time Zone:
time_series_data = time_series_data.tz_localize('UTC') # Set time zone to UTC
Converting Time Zones:
time_series_data = time_series_data.tz_convert('US/Eastern') # Convert to US Eastern Time
Quarterly Period Frequencies:
Quarterly periods can be generated with the "Q" frequency code.
quarterly_range = pd.period_range(start='2023Q1', end='2023Q4', freq='Q')

TIME SERIES ANALYSIS
Time series analysis often involves various data manipulation tasks, including plotting, data munging,
splicing data from multiple sources, decile and quartile analysis, and more. Let's explore these concepts and
some sample applications in the context of time series analysis:
Time Series Plotting:
Plotting is crucial for visualizing time series data to identify patterns and trends.
import matplotlib.pyplot as plt
# Plot time series data
time_series_data.plot()
plt.xlabel("Date")
plt.ylabel("Value")
plt.title("Time Series Plot")
plt.show()

Data Munging:
Data munging involves cleaning, transforming, and preparing data for analysis. In time series analysis, this might
include handling missing values, resampling, or dealing with outliers.
# Handling missing values
time_series_data = time_series_data.fillna(method='ffill')
# Resampling to a different frequency
resampled_data = time_series_data.resample('W').mean()
Splicing Together Data Sources:
In some cases, you may need to combine time series data from multiple sources.
import pandas as pd
# Concatenating data from multiple sources
combined_data = pd.concat([data_source1, data_source2])
Decile and Quartile Analysis:
Decile and quartile analysis helps you understand the distribution of data.
# Calculate quartiles
quartiles = time_series_data.quantile([0.25, 0.5, 0.75])
# Calculate deciles
deciles = time_series_data.quantile([i/10 for i in range(1, 10)])

Sample Applications:
Stock Market Analysis: Analyzing stock price time series data for trends and predicting future stock prices.
Temperature Forecasting: Analyzing historical temperature data to forecast future weather conditions.
Demand Forecasting: Analyzing sales data to forecast future product demand.
Future Contract Rolling:
In financial time series analysis, rolling futures contracts is crucial to avoid jumps in time series data when
contracts expire.
# Rolling futures contracts in a DataFrame
rolled_data = contract_rolling_function(time_series_data, window=10)
Rolling Correlation and Linear Regression:
Rolling correlation and regression are used to understand how the relationship between two time series
changes over time.
# Calculate rolling correlation between two time series
rolling_corr = time_series_data1.rolling(window=30).corr(time_series_data2)
# Calculate rolling linear regression between two time series
rolling_regression = time_series_data1.rolling(window=30).apply(lambda x: your_lin

Data Munging:
Data munging is a more general term that encompasses various data preparation tasks, including
cleaning, structuring, and organizing data.
It often involves dealing with missing data, handling outliers, and addressing issues like data
format inconsistencies.
Data munging can also include tasks such as data loading, data extraction, and basic data
exploration.
It is a broader term that doesn't specify a particular methodology or approach.
Data Wrangling:
Data wrangling is a subset of data munging that specifically refers to the process of cleaning,
transforming, and structuring data for analysis.
Data wrangling typically involves tasks like filtering, aggregating, joining, and reshaping data to
create a dataset that is ready for analysis or machine learning.
It is often associated with data preparation in the context of data analysis and is more focused on
making data suitable for specific analytical tasks.

unit 5_Real time Data Analysis vsp.pptx

More Related Content

Similar to unit 5_Real time Data Analysis vsp.pptx (20)

More from prakashvs7 (14)

Recently uploaded (20)

unit 5_Real time Data Analysis vsp.pptx