UNIT V ANALYSIS OF VARIANCE AND PREDICTIVE ANALYTICS

The F-test is a statistical test used to compare two or more population variances or to assess
the goodness of fit in models. It is commonly used in analysis of variance (ANOVA),
regression analysis, and to test hypotheses about variances.
Common Uses of the F-test:
1. ANOVA (Analysis of Variance):
o The F-test is used in ANOVA to determine if there are any significant
differences between the means of three or more groups.
o The null hypothesis typically assumes that all group means are equal, while
the alternative hypothesis suggests that at least one group mean differs from
the others.
2. Testing Equality of Variances:
o The F-test can be used to test if two populations have the same variance.
o The null hypothesis typically states that the variances are equal, while the
alternative hypothesis suggests that the variances are not equal.
3. Regression Analysis:
o In multiple regression, the F-test is used to test if the regression model as a
whole is a good fit for the data.
o The null hypothesis assumes that all regression coefficients are equal to zero
(i.e., the model has no explanatory power).

ANOVA
ANOVA (Analysis of Variance) is a statistical method used to test differences between the
means of three or more groups. It helps determine whether there are any statistically
significant differences between the means of the groups being compared.
Key Concepts of ANOVA:
Types of ANOVA:
1. One-Way ANOVA:
o Used when comparing the means of three or more independent groups based
on one factor (independent variable).
o Example: Comparing the test scores of students from three different teaching
methods.
2. Two-Way ANOVA:
o Used when comparing the means of groups based on two factors (independent
variables).
o It can also assess the interaction effect between the two factors.
o Example: Comparing the test scores of students based on both teaching
method and gender.
3. Repeated Measures ANOVA:
o Used when the same subjects are tested under different conditions or at
different times.
o Example: Measuring the effect of a drug on the same group of patients at
multiple time points.
Assumptions of ANOVA:
 Independence: The samples or groups should be independent of each other.
 Normality: The data in each group should be approximately normally distributed.
 Homogeneity of Variances: The variances across the groups should be
approximately equal (this is known as homoscedasticity).
ANOVA Steps:
1. Calculate Group Means:
o Compute the mean for each group.
2. Calculate Overall Mean:
o Compute the overall mean (grand mean) of all the data combined.
 3. Calculate the Sum of Squares:
o Total Sum of Squares (SST): Measures the total variation in the data.
o Between-Group Sum of Squares (SSB): Measures the variation due to the
differences between the group means and the overall mean.
o Within-Group Sum of Squares (SSW): Measures the variation within each
group (i.e., how individual observations vary from their group mean).

1. Make a Decision:
o Compare the calculated F-statistic to the critical value from the F-distribution
table at the desired significance level (usually 0.05).
o If the calculated F-statistic is greater than the critical value, reject the null
hypothesis (indicating that there is a significant difference between the group
means).
Example of One-Way ANOVA:
Imagine we have three groups of people who were given different diets, and we want to test if
their weight loss differs. The groups are:
 Group 1 (Diet A)
 Group 2 (Diet B)
 Group 3 (Diet C)
We would:
1. Calculate the mean weight loss for each group.
2. Compute the overall (grand) mean of weight loss.
3. Calculate the sums of squares (SST, SSB, SSW).
4. Compute the F-statistic.
5. Compare the F-statistic with the critical value from the F-distribution table to decide
if the differences are significant.
Interpretation:
  If the F-statistic is large, it suggests that the between-group variability is large relative
to the within-group variability, indicating that at least one group mean is different.
 If the F-statistic is small, it suggests that the group means are not significantly
different.
Two-factor experiments
Two-factor experiments involve testing two independent variables (factors) simultaneously to
understand how they individually and interactively affect a dependent variable (response).
These types of experiments are especially useful when you want to assess not just the
individual effect of each factor, but also whether there is an interaction effect between the two
factors.
In a two-factor experiment, you have:
 Two independent variables (factors): These could be categorical or continuous. For
example, in a study on plant growth, factors could be "soil type" and "fertilizer type."
 Levels of the factors: Each factor will have different levels. For example, "soil type"
could have two levels (e.g., sandy, loamy), and "fertilizer type" could have three
levels (e.g., organic, chemical, none).
 Response variable (dependent variable): This is the outcome or measurement you're
interested in, such as plant height or crop yield.
Example of a Two-Factor Experiment:
Imagine you want to study the effects of two factors on the growth of plants:
1. Factor 1: Type of fertilizer (with 2 levels: Organic, Synthetic)
2. Factor 2: Amount of water (with 3 levels: Low, Medium, High)
You would test the different combinations of these two factors:
 Organic Fertilizer + Low Water
 Organic Fertilizer + Medium Water
 Organic Fertilizer + High Water
 Synthetic Fertilizer + Low Water
 Synthetic Fertilizer + Medium Water
 Synthetic Fertilizer + High Water
Key Concepts:
1. Main Effects: These represent the individual effects of each factor (independent
variable) on the dependent variable.
o Main Effect of Factor 1 (Fertilizer): Does the type of fertilizer (organic vs.
synthetic) affect plant growth?
 o Main Effect of Factor 2 (Water): Does the amount of water (low, medium,
high) affect plant growth?
2. Interaction Effect: This is the combined effect of the two factors on the dependent
variable. The interaction effect assesses whether the effect of one factor depends on
the level of the other factor.
o For example, the effect of fertilizer on plant growth might differ depending on
the amount of water. If plants with organic fertilizer grow well under high
water but poorly under low water, there is an interaction between the two
factors.
Types of Two-Factor Designs:
1. Two-Factor Design with Replication:
o In this design, each combination of the two factors is repeated multiple times
(replications) to reduce the impact of random variation. This helps provide
more reliable results.
2. Two-Factor Design without Replication:
o Each combination of the factors is tested only once. This design can be less
reliable because the results could be influenced by uncontrolled variables or
randomness.
Statistical Analysis of Two-Factor Experiments:
In a two-factor experiment, you typically perform a two-way analysis of variance (ANOVA).
This allows you to assess:
 Main effects of the two factors: How each factor (independently) affects the
dependent variable.
 Interaction effect: Whether the effect of one factor depends on the level of the other
factor.
Steps in Two-Way ANOVA:
1. Hypotheses:
o Null Hypothesis (H ): No effect from either factor or their interaction. (i.e.,
Factor 1 has no effect, Factor 2 has no effect, and there is no interaction
effect).
o Alternative Hypothesis (H ): At least one of the effects (main effects or
interaction) is significant.
2. Two-Way ANOVA Table: This table typically contains:
o Sum of Squares (SS): The variation attributable to each factor and the
interaction term.
o Degrees of Freedom (df): The number of levels minus one for each factor and
the interaction term.
 o Mean Squares (MS): Sum of Squares divided by their respective degrees of
freedom.
o F-statistics: The ratio of the Mean Square for each effect divided by the Mean
Square for error (within-group variation).
3. Decision Rule:
o Compare the F-statistic for each effect (Factor 1, Factor 2, and Interaction)
with the critical value from the F-distribution.
o If the F-statistic is larger than the critical value, reject the null hypothesis for
that effect.
Example of Two-Way ANOVA Analysis:
Let’s continue with the plant growth example:
 Factor 1 (Fertilizer): Organic vs. Synthetic
 Factor 2 (Water): Low, Medium, High
The ANOVA table might look something like this (hypothetical data):

Sum of Squares Degrees of Mean Square F- p-

Source of Variation
(SS) Freedom (df) (MS) statistic value

Factor 1 (Fertilizer) 150 1 150 5.3 0.03

Factor 2 (Water) 200 2 100 3.6 0.05

Interaction (Fertilizer *
50 2 25 1.2 0.30
Water)

Error (Residual) 300 12 25 - -

Interpreting the Results:

 Factor 1 (Fertilizer): p-value = 0.03, which is less than 0.05, so we reject the null
hypothesis and conclude that fertilizer type affects plant growth.
 Factor 2 (Water): p-value = 0.05, which is exactly at the significance level, so we
might conclude that the amount of water does have an effect, but it is marginally
significant.
 Interaction: p-value = 0.30, which is greater than 0.05, so we fail to reject the null
hypothesis for the interaction term. This suggests there is no significant interaction
between fertilizer and water on plant growth.
Visualizing Two-Factor Results:
To better understand the results, a two-way interaction plot is often helpful. It shows how the
levels of one factor affect the dependent variable at different levels of the other factor.
Advantages of Two-Factor Experiments:
  Efficiency: You can investigate two factors simultaneously.
 Interaction Effects: You can detect interaction effects between factors, which might be
missed if factors are tested separately.
Three f-tests
The F-test is used to compare variances or to test the overall significance in statistical
models, such as ANOVA or regression analysis. There are three primary contexts in which F-
tests are commonly applied:
1. F-test for Comparing Two Variances (One-Tailed Test)
 Purpose: To test whether two populations have the same variance.
 Scenario: You want to compare the variability of two different groups, for example,
the variability of test scores between two different classes.

 Decision Rule: Compare the computed F-value with the critical F-value from the F-
distribution table. If the computed F-value is greater than the critical F-value, reject
the null hypothesis.
2. F-test in Analysis of Variance (ANOVA)
 Purpose: To test if there are any significant differences between the means of three or
more groups.
 Scenario: You want to determine whether different teaching methods lead to different
average scores among students.
 Null Hypothesis (H ): All group means are equal.
o H : μ1 =μ2 =μ3 =...=μk​
 Alternative Hypothesis (H ): At least one group mean is different.
3. F-test in Regression Analysis (Overall Significance)
 Purpose: To test if the overall regression model is significant. In other words,
whether at least one of the independent variables significantly explains the variability
in the dependent variable.
 Scenario: You want to determine whether the combination of independent variables
(e.g., hours studied and number of practice tests taken) predicts the dependent
variable (e.g., exam scores).
 Null Hypothesis (H ): All regression coefficients are equal to zero (i.e., the
independent variables have no effect).
  Decision Rule: If the computed F-statistic exceeds the critical value from the F-
distribution table, reject the null hypothesis. This would indicate that the independent
variables collectively explain a significant portion of the variation in the dependent
variable.
Example: You might perform an F-test to evaluate whether the number of study hours and
practice tests together predict exam scores.
Visualizing F-tests:
 F-distribution: The F-statistic follows the F-distribution, which is positively skewed
and depends on two degrees of freedom: one for the numerator and one for the
denominator.
 Critical F-value: The critical value is determined based on the significance level
(e.g., 0.05) and the degrees of freedom for both the numerator and denominator. If the
F-statistic exceeds the critical value, the null hypothesis is rejected.

Linear least squares

Linear least squares is a mathematical method used to find the best-fitting line or model to a
set of data points. The objective is to minimize the sum of the squared differences between
the observed values (data points) and the values predicted by the linear model. This is
commonly used for regression problems, where you want to fit a line (or hyperplane, in
higher dimensions) to your data.

Applications:
  Linear regression: Fit a line to a set of data points.
 Curve fitting: Fit more complex models (e.g., polynomials) to data.
 Signal processing: Estimate parameters of a model from noisy data.
Goodness Of Fit
Goodness of fit is a statistical measure used to assess how well a model (like a regression
model) fits the data. In the context of linear regression, the goodness of fit tells you how well
the predicted values from the model align with the observed data points.
Here are some key metrics commonly used to evaluate the goodness of fit:
 Testing a linear model – weighted resampling
Testing a linear model using weighted resampling involves adjusting how data points are
sampled or weighted during the model evaluation process. This technique can be particularly
useful when dealing with imbalanced data or when certain observations are considered more
important than others.
Weighted Resampling and its Purpose
In a linear regression model (or any statistical model), we may want to:
 Assign different importance (weights) to data points depending on factors like
reliability, frequency, or relevance.
 Handle imbalanced data where some classes or regions of the data might be
underrepresented.
 Perform resampling (such as bootstrap or cross-validation) in a way that gives more
influence to certain data points.
Weighted Resampling Process
Weighted resampling can be done in several ways, including:
1. Weighted Least Squares (WLS):
o This is a variant of ordinary least squares (OLS) where each data point is
given a weight. The idea is to give more importance to some points during the
fitting process. For example, points with smaller measurement errors might be
given higher weights, while noisy or less reliable data points might get lower
weights.

Performing regression using the Statsmodels library in Python is a common approach for
fitting and analyzing statistical models. Statsmodels provides a rich set of tools for linear
regression, generalized linear models, and other types of regression analysis.
Steps for Linear Regression using Statsmodels
Let’s walk through the basic steps for performing a linear regression using Statsmodels.
1. Install Statsmodels (if you haven't already):
You can install Statsmodels using pip:
pip install statsmodels
2. Import Required Libraries:
You'll need the following libraries:
import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf
 sm (from statsmodels.api): Used for general regression models and results.
 smf (from statsmodels.formula.api): Allows for a higher-level interface for specifying
models using formulas (similar to R).
3. Prepare Your Data:
Let’s assume you have a dataset with some independent variables (features) and a dependent
variable (target). For this example, let’s create a simple synthetic dataset.
# Create a synthetic dataset
data = {
'X1': [1, 2, 3, 4, 5],
'X2': [2, 4, 6, 8, 10],
'Y': [3, 6, 7, 8, 11]
}

df = pd.DataFrame(data)
 X1 and X2 are the independent variables (predictors).
 Y is the dependent variable (response).
4. Linear Regression Model:
We will use sm.OLS (Ordinary Least Squares) to fit a linear regression model. Before doing
this, we need to add a constant (intercept) to the features.
# Add a constant (intercept) to the model
X = df[['X1', 'X2']] # Independent variables
X = sm.add_constant(X) # Adds a column of ones to the matrix for the intercept

y = df['Y'] # Dependent variable

# Fit the OLS regression model

model = sm.OLS(y, X).fit()
  sm.add_constant(X): This adds the intercept (constant term) to the model.
5. Model Summary:
Once the model is fit, you can get a summary of the results by calling .summary() on the
fitted model.
# Display the model summary
print(model.summary())
This will print out the regression statistics, including:
 R-squared: The proportion of the variance in the dependent variable that is
predictable from the independent variables.
 p-values: Indicate whether the predictors are statistically significant.
 Coefficients: The estimated values for the intercept and the slopes (coefficients) for
each predictor.
 Standard errors: Estimate the variability of the coefficients.
Example Output from model.summary():
OLS Regression Results
==================================================================
============
Dep. Variable: Y R-squared: 0.996
Model: OLS Adj. R-squared: 0.993
Method: Least Squares F-statistic: 300.5
Date: Mon, 18 Mar 2025 Prob (F-statistic): 0.000234
Time: 15:45:22 Log-Likelihood: -4.2387
No. Observations: 5 AIC: 14.4774
Df Residuals: 3 BIC: 11.3545
Df Model: 1
Covariance Type: nonrobust
==================================================================
============
coef std err t P>|t| [0.025 0.975]
------------------------------------------------------------------------------
const 1.3000 0.500 2.600 0.049 0.010 2.590
X1 0.8000 0.400 2.000 0.078 -0.050 1.650
X2 0.5000 0.100 5.000 0.012 0.250 0.750
==================================================================
============
Key metrics to interpret from the summary:
 R-squared: In this case, it's 0.996, which means that the model explains 99.6% of the
variance in the dependent variable.
 p-values: For each predictor, this tells you whether the predictor is statistically
significant. A small p-value (usually < 0.05) means the variable is significant.
 Coefficients: The intercept (const) is 1.3, and the slopes for X1 and X2 are 0.8 and
0.5, respectively.
6. Predict Using the Model:
Once the model is fitted, you can use it to make predictions on new data.
# New data for prediction
new_data = pd.DataFrame({'X1': [6, 7], 'X2': [12, 14]})
new_data = sm.add_constant(new_data) # Add constant for intercept

# Make predictions
predictions = model.predict(new_data)
print(predictions)
7. Other Regression Types in Statsmodels:
Statsmodels also allows you to fit various other types of regression models, including:
 Logistic Regression: For binary or categorical outcomes.
logit_model = smf.logit('Y ~ X1 + X2', data=df).fit()
print(logit_model.summary())
 Poisson Regression: For count data.
poisson_model = smf.poisson('Y ~ X1 + X2', data=df).fit()
print(poisson_model.summary())
 Robust Regression: To handle outliers or heteroskedasticity.
robust_model = smf.ols('Y ~ X1 + X2', data=df).fit(cov_type='HC3')
print(robust_model.summary())
8. Model Diagnostics:
You can check various diagnostic measures to assess the quality of the model:
# Residuals plot
import matplotlib.pyplot as plt
plt.scatter(model.fittedvalues, model.resid)
plt.xlabel('Fitted Values')
plt.ylabel('Residuals')
plt.title('Residuals vs Fitted Values')
plt.show()

# Q-Q plot for normality of residuals

sm.qqplot(model.resid, line ='45')
plt.show()
Regression using Stats Models
To perform regression using Statsmodels in Python, you generally follow these steps:
1. Install and import required libraries
o Install statsmodels if you don't have it already using the command:
-pip install statsmodels
2. Prepare your data
o Your data should be structured with independent variables (predictors) and a
dependent variable (response).
3. Create the regression model
Here's an example of a simple linear regression using Statsmodels:
Example 1: Simple Linear Regression
import statsmodels.api as sm
import pandas as pd
# Sample data
data = {
'X': [1, 2, 3, 4, 5],
'Y': [2, 4, 5, 4, 5]
}
# Create a DataFrame
df = pd.DataFrame(data)
# Define the independent variable (X) and dependent variable (Y)
X = df['X'] # independent variable
Y = df['Y'] # dependent variable
# Add a constant to the independent variable for intercept
X = sm.add_constant(X)
# Create the model
model = sm.OLS(Y, X) # OLS = Ordinary Least Squares
results = model.fit()
# Print the summary of the regression
print(results.summary())
Example 2: Multiple Linear Regression
# Sample data for multiple regression
data = {
'X1': [1, 2, 3, 4, 5],
'X2': [5, 4, 3, 2, 1],
'Y': [2, 4, 5, 4, 5]
}
# Create a DataFrame
df = pd.DataFrame(data)
# Define independent variables (X1, X2) and dependent variable (Y)
X = df[['X1', 'X2']] # independent variables
Y = df['Y'] # dependent variable
# Add a constant to the independent variables for intercept
X = sm.add_constant(X)
# Create the model
model = sm.OLS(Y, X)
results = model.fit()
# Print the summary of the regression
print(results.summary())
Explanation of Output
The summary output from results.summary() will give you detailed statistics, including:
 R-squared: Measures the proportion of the variance in the dependent variable that is
explained by the independent variables.
 Coefficients: The estimated values for the model (intercept and slope).
 P-values: Show whether the coefficients are statistically significant.
 Confidence Intervals: For each coefficient, this shows the range in which the true
value might lie.
Key Points:
 sm.add_constant(X): Adds an intercept term to the model.
 sm.OLS(Y, X): Specifies an Ordinary Least Squares regression model.
 model.fit(): Fits the model to the data.
Multiple Regression
In multiple regression, you'll have more than one independent variable (predictor).
Steps to Perform Multiple Regression
1. Prepare the Data: You need a dataset with multiple independent variables
(predictors) and a dependent variable (response).
2. Fit the Model: Use statsmodels.OLS (Ordinary Least Squares) to fit a multiple
regression model.
3. Interpret the Results: The summary provides insights into how well the independent
variables explain the dependent variable.
Example: Multiple Linear Regression with Statsmodels
Let's assume you have a dataset with three predictors (independent variables) and one
response (dependent variable).
Sample Data:
 X1: Age
 X2: Years of Education
 X3: Work Experience
 Y: Salary (the dependent variable)
import statsmodels.api as sm
import pandas as pd
# Sample data
data = {
'Age': [25, 30, 35, 40, 45],
'Education': [12, 14, 16, 18, 20],
'Experience': [2, 5, 7, 10, 12],
'Salary': [40000, 50000, 60000, 70000, 80000]
}
# Create a DataFrame
df = pd.DataFrame(data)
# Define independent variables (Age, Education, Experience) and dependent variable (Salary)
X = df[['Age', 'Education', 'Experience']] # Independent variables
Y = df['Salary'] # Dependent variable
# Add a constant to the independent variables (this adds the intercept to the model)
X = sm.add_constant(X)
# Create and fit the OLS model
model = sm.OLS(Y, X)
results = model.fit()
# Print the summary of the regression
print(results.summary())
Explanation of the Code:
1. Data Preparation:
o The dataset contains columns for Age, Education, Experience, and Salary.
o We store the independent variables (X) and dependent variable (Y).
2. Adding the Constant:
o sm.add_constant(X) adds a column of ones to X, which represents the
intercept term in the regression.
3. Fitting the Model:
o sm.OLS(Y, X) creates the Ordinary Least Squares regression model.
o .fit() fits the model to the data.
4. Summary:
 o results.summary() provides a detailed summary with coefficients, p-values, R-
squared, etc.
Output Example:
The output from results.summary() might look like this:
OLS Regression Results
==================================================================
============
Dep. Variable: Salary R-squared: 0.998
Model: OLS Adj. R-squared: 0.997
Method: Least Squares F-statistic: 859.1
Date: Sat, 23 Mar 2025 Prob (F-statistic): 0.000
Time: 12:30:52 Log-Likelihood: -49.185
No. Observations: 5 AIC: 106.370
Df Residuals: 1 BIC: 106.152
Df Model: 3
Covariance Type: nonrobust
==================================================================
============
coef std err t P>|t| [0.025 0.975]
------------------------------------------------------------------------------
const 25000.0000 15833.333 1.580 0.173 -26455.138 76455.138
Age 1000.0000 2000.000 0.500 0.679 -5000.000 7000.000
Education 1500.0000 1000.000 1.500 0.203 -2000.000 5000.000
Experience 2000.0000 800.000 2.500 0.047 100.000 3900.000
==================================================================
============
Key Output Sections:
 R-squared: Indicates how much of the variance in the dependent variable (Salary) is
explained by the independent variables (Age, Education, Experience). Higher values
indicate a better fit.
 Coefficients: The estimated effect of each independent variable on the dependent
variable.
 o For example, the coefficient for Age suggests how much Salary increases per
year of age (though this might not always be statistically significant depending
on the p-value).
 P-values: Show the statistical significance of each predictor. If the p-value is less than
0.05, the corresponding predictor is considered statistically significant.
 Confidence Intervals: The range in which we expect the true coefficient to lie, with a
95% confidence level.
Nonlinear Relationships
Nonlinear relationships occur when the relationship between the independent and dependent
variables cannot be described by a straight line. In other words, the relationship isn't a simple
linear one, and the model's assumptions might need to be adjusted accordingly.
Common Methods for Modeling Nonlinear Relationships:
1. Polynomial Regression: Extending linear regression by including polynomial terms
(like squared or cubed terms) to model curved relationships.
2. Logarithmic or Exponential Regression: Applying logarithmic or exponential
transformations to the independent or dependent variables.
3. Logistic Regression: Used for binary dependent variables.
4. Generalized Additive Models (GAMs): A more flexible approach that can capture
complex nonlinear relationships.
We'll focus on Polynomial Regression using Statsmodels and explain how to handle
nonlinear relationships using polynomial terms.
Polynomial Regression in Python with Statsmodels
Polynomial regression is one of the simplest ways to model a nonlinear relationship. By
adding higher-degree terms of the independent variable(s) to your regression model, you
allow for more flexibility in how the model fits the data.
Example: Polynomial Regression
Suppose we have a dataset where the relationship between the independent variable X (e.g.,
experience) and the dependent variable Y (e.g., salary) is nonlinear. We can fit a polynomial
regression model.
1. Prepare the Data: We'll use polynomial terms (e.g., X^2, X^3) to fit a nonlinear
relationship.
2. Fit the Polynomial Model: Add these polynomial terms to the model and fit it using
statsmodels.
Code Example:
python
Copy
import statsmodels.api as sm
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Sample data
data = {
'X': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
'Y': [2, 4, 9, 16, 25, 36, 49, 64, 81, 100] # Quadratic relationship (Y = X^2)
}

# Create a DataFrame
df = pd.DataFrame(data)

# Define independent variable (X) and dependent variable (Y)

X = df['X']
Y = df['Y']

# Create polynomial features (X^2, X^3)

X_poly = np.column_stack([X, X**2, X**3])

# Add a constant for the intercept term

X_poly = sm.add_constant(X_poly)

# Fit the OLS model (Ordinary Least Squares regression)

model = sm.OLS(Y, X_poly)
results = model.fit()

# Print the summary of the regression

print(results.summary())
# Plot the data and the fitted polynomial curve
plt.scatter(X, Y, color='blue', label='Data')
plt.plot(X, results.fittedvalues, color='red', label='Polynomial fit (degree 3)')
plt.xlabel('X')
plt.ylabel('Y')
plt.legend()
plt.show()
Explanation:
 Polynomial Features: The line X_poly = np.column_stack([X, X**2, X**3]) creates
the polynomial terms. This means we're considering X, X^2 (squared term), and X^3
(cubed term) as independent variables.
 Model Fitting: We use sm.OLS to fit the model, just like in simple and multiple
linear regression.
 Plotting: After fitting the model, we plot the actual data points and the predicted
curve from the fitted polynomial model.
Output Example of the results.summary():
The summary will show the estimated coefficients for the intercept and the polynomial terms
(X, X^2, X^3). For example:
markdown
Copy
OLS Regression Results
==================================================================
============
Dep. Variable: Y R-squared: 0.998
Model: OLS Adj. R-squared: 0.997
Method: Least Squares F-statistic: 590.1
Date: Sat, 23 Mar 2025 Prob (F-statistic): 0.000
Time: 12:30:52 Log-Likelihood: -23.185
No. Observations: 10 AIC: 56.370
Df Residuals: 6 BIC: 57.526
Df Model: 3
Covariance Type: nonrobust
==================================================================
============
coef std err t P>|t| [0.025 0.975]
------------------------------------------------------------------------------
const -1.428e-15 7e-15 -0.204 0.844 -2.4e-14 2.1e-14
X 1.0000 0.016 62.500 0.000 0.968 1.032
X^2 0.0002 0.000 16.500 0.000 0.000 0.000
X^3 0.0000 0.000 6.800 0.000 0.000 0.000
==================================================================
============
Key Sections:
 Coefficients: The coefficients for X, X^2, and X^3 tell us how much each term
contributes to the predicted value of Y. For a quadratic relationship, you'd typically
see the coefficient for X^2 be significantly different from zero.
 R-squared: A high R-squared value suggests that the polynomial model explains
most of the variance in the data.
 Significance: The p-values for each coefficient should be low (usually < 0.05) to
indicate that these terms significantly contribute to the model.
Plot:
You will see a scatter plot of the actual data, and the fitted curve will be drawn over it,
showing how well the polynomial model captures the nonlinear relationship.
How to Handle More Complex Nonlinear Relationships?
1. Higher Degree Polynomial: You can use higher degrees (e.g., X^4, X^5) if the
relationship is more complex.
2. Logarithmic/Exponential Models: For data that grows exponentially or
logarithmically, you might consider fitting a model where the dependent or
independent variable is transformed (e.g., log of X or Y).
3. Generalized Additive Models (GAMs): For more flexibility in capturing nonlinear
relationships, you can use models like Generalized Additive Models (GAMs), but
they are typically available through libraries like pyGAM or scikit-learn's
preprocessing tools.
Logistic Regression
Logistic Regression is a statistical method used for binary classification tasks, where the
dependent variable is categorical (binary), typically with two outcomes (e.g., 0 or 1, true or
false, yes or no). Unlike linear regression, which predicts a continuous outcome, logistic
regression predicts the probability of an outcome.
The logistic regression model uses the logit function (the natural log of the odds) to model
the relationship between the independent variables and the probability of the binary outcome.

1. Interpretation of Coefficients: The coefficients in logistic regression are interpreted

as the log-odds of the outcome for each one-unit increase in the corresponding
predictor variable.
2. Prediction: The model predicts the probability that the dependent variable equals 1
(positive class). You can then choose a threshold (commonly 0.5) to classify
observations as 0 or 1.
Steps to Perform Logistic Regression in Python with Statsmodels:
1. Prepare the Data: The independent variables (predictors) should be numerical or
categorical (converted to dummy variables).
2. Fit the Model: We use sm.Logit for logistic regression.
3. Interpret the Results: Examine the coefficients, p-values, and other metrics.
Example: Logistic Regression with Statsmodels
Let’s assume we have a dataset of student scores (X) and whether they passed an exam (Y,
where 1 = passed and 0 = failed).
import statsmodels.api as sm
import pandas as pd

# Sample data: Score (X) vs Pass/Fail (Y)

data = {
'Score': [55, 70, 65, 80, 90, 85, 50, 60, 95, 100],
'Pass': [0, 1, 1, 1, 1, 1, 0, 0, 1, 1]
}
# Create a DataFrame
df = pd.DataFrame(data)
# Define independent variable (Score) and dependent variable (Pass)
X = df['Score']
Y = df['Pass']
# Add a constant to the independent variable (intercept term)
X = sm.add_constant(X)
# Fit the logistic regression model
model = sm.Logit(Y, X)
results = model.fit()
# Print the summary of the regression
print(results.summary())
Explanation of the Code:
1. Data Preparation:
o We create a simple dataset where Score is the independent variable and Pass is
the dependent binary variable.
o The Score represents the student’s score on an exam, and Pass represents
whether they passed the exam (1 = passed, 0 = failed).
2. Adding the Constant:
o sm.add_constant(X) adds a column of ones to X for the intercept term in the
logistic regression model.
3. Fitting the Model:
o sm.Logit(Y, X) creates the logistic regression model, where Y is the dependent
variable and X is the independent variable.
o .fit() fits the model to the data.
4. Printing the Summary:
o The summary provides key statistics like coefficients, p-values, and the
goodness of fit (e.g., Log-Likelihood, AIC).
Output Example of the results.summary():
Logit Regression Results
==================================================================
============
Dep. Variable: Pass No. Observations: 10
Model: Logit Df Residuals: 8
Method: MLE Df Model: 1
Date: Sat, 23 Mar 2025 Pseudo R-squ.: 0.231
Time: 12:30:52 Log-Likelihood: -4.1263
converged: True LL-Null: -5.3873
Covariance Type: nonrobust LLR p-value: 0.04813
==================================================================
============
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
const -6.0539 2.051 -2.949 0.003 -10.080 -2.028
Score 0.0904 0.034 2.660 0.008 0.024 0.157
==================================================================
============
Key Sections of the Output:
1. Coefficients:
o const: This is the intercept term of the logistic regression model.
o Score: This is the coefficient for the Score variable. In this case, for each one-
unit increase in Score, the log-odds of passing the exam increase by 0.0904.
2. P-values:
o The p-value for Score is 0.008, which is less than 0.05, indicating that Score is
statistically significant in predicting whether a student passes.
3. Log-Likelihood:
o The Log-Likelihood value (-4.1263) gives an idea of how well the model fits
the data. Higher values are better.
4. Pseudo R-squared:
 o This value (0.231) tells us how well the model explains the variability in the
data. For logistic regression, this is not directly comparable to R-squared in
linear regression.
5. Odds Ratio:
o To interpret the coefficients in terms of odds, we can exponentiate them (i.e.,
calculate the odds ratio). For Score, the odds ratio is exp(0.0904) ≈ 1.095,
meaning that for each unit increase in score, the odds of passing the exam
increase by 9.5%.
Making Predictions:
Once the model is fitted, we can use it to make predictions for new data points.
# New data for prediction
new_data = pd.DataFrame({'Score': [78, 60]})
new_data = sm.add_constant(new_data)
# Predict the probability of passing
predictions = results.predict(new_data)
print(predictions)
Logistic Regression is widely used for binary classification tasks and is easy to
implement using Statsmodels.
The coefficients provide log-odds of the outcome, and exponentiating them gives you the
odds ratios.
Logistic regression is useful when you need to predict probabilities for binary outcomes
and understand how the predictors impact the odds of a specific outcome.
Estimating Parameters
Estimating Parameters in Logistic Regression
In logistic regression, estimating parameters (coefficients) refers to determining the values of
the model’s weights (like b_0, b_1, etc.) that best fit the data. This is typically done by
maximizing the likelihood function using techniques like Maximum Likelihood
Estimation (MLE).
Key Steps in Estimating Parameters:
1. Logistic Function: The logistic function maps any input into the range [0, 1], which
is interpreted as a probability:
Maximum Likelihood Estimation (MLE):
 MLE is a method used to estimate the parameters (coefficients) by maximizing the
likelihood function. The likelihood function gives the probability of observing the
data given certain parameter values.
 The likelihood function for logistic regression is based on the Bernoulli distribution
(since the outcome is binary).

Estimating Parameters in Logistic Regression with Statsmodels

In statsmodels, you don't need to explicitly define the likelihood function or optimization
procedure; it automatically uses Maximum Likelihood Estimation to estimate the
parameters when you fit the logistic regression model.
Here’s how the parameters are estimated using Statsmodels:
Example: Estimating Parameters in Logistic Regression
Let’s continue with the student pass/fail dataset and estimate the parameters for the logistic
regression model.
import statsmodels.api as sm
import pandas as pd
# Sample data: Score (X) vs Pass/Fail (Y)
data = {
'Score': [55, 70, 65, 80, 90, 85, 50, 60, 95, 100],
'Pass': [0, 1, 1, 1, 1, 1, 0, 0, 1, 1]
}
# Create a DataFrame
df = pd.DataFrame(data)
# Define independent variable (Score) and dependent variable (Pass)
X = df['Score']
Y = df['Pass']
# Add a constant to the independent variable (intercept term)
X = sm.add_constant(X)
# Fit the logistic regression model
model = sm.Logit(Y, X)
results = model.fit()
# Print the summary of the regression
print(results.summary())
Explanation of the Output:
When you call results.summary(), it will display the estimated parameters along with the
statistical metrics like p-values and confidence intervals.
Logit Regression Results
==================================================================
============
Dep. Variable: Pass No. Observations: 10
Model: Logit Df Residuals: 8
Method: MLE Df Model: 1
Date: Sat, 23 Mar 2025 Pseudo R-squ.: 0.231
Time: 12:30:52 Log-Likelihood: -4.1263
converged: True LL-Null: -5.3873
Covariance Type: nonrobust LLR p-value: 0.04813
==================================================================
============
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
const -6.0539 2.051 -2.949 0.003 -10.080 -2.028
Score 0.0904 0.034 2.660 0.008 0.024 0.157
==================================================================
============
Interpreting the Estimated Parameters:
 Intercept (const): The estimated coefficient for the intercept (b_0). In this case, it's -
6.0539. This represents the log-odds of passing the exam when the score is 0 (though
a score of 0 doesn’t make sense in this context, it’s part of the model).
 Coefficient for Score: The estimated coefficient for Score is 0.0904. This means that
for each one-unit increase in score, the log-odds of passing increase by 0.0904.
 Log-Odds to Probability: To interpret this in terms of probability, we can use the
odds ratio, which is calculated by exponentiating the coefficient:
Odds ratio=e0.0904≈1.095\text{Odds ratio} = e^{0.0904} \approx
1.095Odds ratio=e0.0904≈1.095
This means that for each additional point in the score, the odds of passing increase by about
9.5%.
Maximum Likelihood Estimation (MLE) Process:
1. Initialization: Initially, the model starts with random values for the parameters.
2. Prediction: The model uses these initial parameters to predict probabilities.
3. Likelihood Function: The model calculates the likelihood function based on these
predictions and the observed data.
4. Optimization: The model adjusts the parameters iteratively to maximize the
likelihood function (or equivalently, the log-likelihood function).
5. Convergence: The optimization process continues until it converges to the parameter
values that maximize the likelihood function.
Using the Estimated Parameters to Make Predictions:
Once the model is fitted, you can use the estimated parameters to predict the probability of an
outcome.
# New data for prediction (e.g., score of 75)
new_data = pd.DataFrame({'Score': [75]})
new_data = sm.add_constant(new_data)
# Predict the probability of passing (probability of Y=1)
predicted_prob = results.predict(new_data)
print(f"Predicted probability of passing: {predicted_prob[0]:.4f}")
Time Series Analysis
Time series analysis is a statistical method used to analyze time-ordered data points. The
main goal of time series analysis is to model the underlying structure of the data, understand
its components, and make forecasts for future observations.
Key Components of Time Series Data:
Time series data often exhibit the following components:
1. Trend: The long-term movement in the data, which can be increasing or decreasing.
2. Seasonality: Regular and predictable fluctuations that repeat over a fixed period (e.g.,
daily, monthly, yearly).
3. Cyclic: Long-term fluctuations that do not occur at fixed intervals, often related to
economic cycles or other irregular events.
4. Noise: The random variations in the data that cannot be explained by the trend,
seasonality, or cyclic components.
Time Series Analysis Steps:
1. Plot the Data: Start by visualizing the data to understand its structure.
2. Decomposition: Break down the time series into its components (trend, seasonality,
and noise).
3. Stationarity Test: For most time series models (like ARIMA), the data needs to be
stationary (i.e., the statistical properties like mean and variance do not change over
time).
4. Modeling: Build models to forecast future values.
5. Validation: Evaluate the model's accuracy by comparing predicted values to actual
outcomes.
Common Time Series Models:
1. ARIMA (AutoRegressive Integrated Moving Average): A popular model for time
series forecasting. ARIMA has three components:
o AR (AutoRegressive): Uses the dependency between an observation and a
number of lagged observations (previous values).
 o I (Integrated): The differencing of raw observations to make the time series
stationary.
o MA (Moving Average): Uses dependency between an observation and a
residual error from a moving average model applied to lagged observations.
2. Exponential Smoothing: A forecasting method that gives more weight to recent
observations. It's widely used in short-term forecasting.
3. Seasonal ARIMA (SARIMA): A variation of ARIMA that includes seasonal
components in the model.
4. Prophet: A forecasting tool developed by Facebook, especially designed for daily,
weekly, and yearly seasonalities, as well as holidays.
Steps to Perform Time Series Analysis in Python (using statsmodels):
We'll walk through an example using ARIMA to model time series data.
1. Import Libraries and Load Data
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
from statsmodels.tsa.stattools import adfuller
# Example: Load a time series dataset (e.g., Monthly airline passengers)
data = sm.datasets.airline.load_pandas().data
data['Month'] = pd.to_datetime(data['Month'])
# Set the Month column as the index
data.set_index('Month', inplace=True)
# Plot the data
plt.figure(figsize=(10, 6))
plt.plot(data)
plt.title('Monthly Airline Passengers')
plt.xlabel('Date')
plt.ylabel('Number of Passengers')
plt.show()
2. Check for Stationarity
Most time series models, including ARIMA, require the data to be stationary. To check if the
data is stationary, we use the Augmented Dickey-Fuller (ADF) test.
# Augmented Dickey-Fuller test to check stationarity
result = adfuller(data['Passengers'])
print(f"ADF Statistic: {result[0]}")
print(f"p-value: {result[1]}")
# Interpretation
if result[1] < 0.05:
print("The series is stationary.")
else:
print("The series is not stationary.")
3. Make the Series Stationary (if needed)
If the series is not stationary, we can apply differencing to make it stationary.
# Apply first-order differencing to make the series stationary
data['Passengers_diff'] = data['Passengers'].diff().dropna()
# Plot the differenced series
plt.figure(figsize=(10, 6))
plt.plot(data['Passengers_diff'])
plt.title('Differenced Monthly Airline Passengers')
plt.xlabel('Date')
plt.ylabel('Differenced Passengers')
plt.show()
4. Fit the ARIMA Model
Now that the data is stationary, we can fit an ARIMA model. ARIMA is defined by three
parameters (p, d, q):
 p: The number of lag observations in the autoregressive model.
 d: The number of times that the raw observations are differenced.
 q: The size of the moving average window.
You can experiment with different values for p, d, and q using model selection techniques
like AIC or BIC.
# Fit ARIMA model (p=1, d=1, q=1)
from statsmodels.tsa.arima.model import ARIMA
model = ARIMA(data['Passengers'], order=(1, 1, 1)) # ARIMA(p=1, d=1, q=1)
fitted_model = model.fit()
# Summary of the model
print(fitted_model.summary())
5. Make Predictions
After fitting the model, you can use it to make forecasts. Let's forecast the next 12 months of
airline passenger numbers.
# Forecast the next 12 months
forecast = fitted_model.forecast(steps=12)
# Print the forecast
print(f"Forecasted Values: {forecast}")
# Plot the forecasted values
plt.figure(figsize=(10, 6))
plt.plot(data.index, data['Passengers'], label='Historical Data')
plt.plot(pd.date_range(start=data.index[-1], periods=13, freq='M')[1:], forecast,
label='Forecasted Data', color='red')
plt.title('Forecasted Monthly Airline Passengers')
plt.xlabel('Date')
plt.ylabel('Number of Passengers')
plt.legend()
plt.show()
6. Model Diagnostics and Validation
After fitting the model, it’s important to check the residuals to ensure that the model is a good
fit. We want the residuals (errors) to resemble white noise (random fluctuations).
# Plot the residuals
residuals = fitted_model.resid
plt.figure(figsize=(10, 6))
plt.plot(residuals)
plt.title('Residuals of ARIMA Model')
plt.xlabel('Date')
plt.ylabel('Residuals')
plt.show()
# Check the autocorrelation of residuals (should be close to zero)
sm.graphics.tsa.plot_acf(residuals, lags=40)
plt.show()
7. ARIMA Model Tuning
You can optimize the ARIMA model by experimenting with different combinations of p, d,
and q. One way to do this is by using grid search, where you systematically vary the
parameters and select the best model based on AIC or BIC.
# Use auto_arima from the pmdarima package to find the optimal (p, d, q)
import pmdarima as pm
Fit an ARIMA model using auto_arima to find the best p, d, q
auto_model = pm.auto_arima(data['Passengers'], seasonal=False, stepwise=True, trace=True)
# Print the summary of the best ARIMA model
print(auto_model.summary())
Moving Averages and Handling Missing Values in Time Series
Moving averages (MA) are a popular technique in time series analysis, used to smooth out
short-term fluctuations and highlight longer-term trends or cycles. A moving average is
calculated by averaging a window of past values in the series. It’s commonly used in
forecasting and trend analysis.
Types of Moving Averages
1. Simple Moving Average (SMA): The simple moving average is the most basic form,
which is calculated by averaging a fixed number of past observations. For a window
size nnn, the formula is:

where:
 yi is the observed value at time iii,
 t is the current time point.
Exponential Moving Average (EMA): The exponential moving average gives more
weight to more recent observations, making it more sensitive to recent changes in the data.

The formula for the EMA is:

where:
o α is the smoothing factor, typically between 0 and 1.
Handling Missing Values in Time Series Data
When applying moving averages, missing values can cause problems, as the calculation
requires a continuous set of data. There are several ways to handle missing values before
applying moving averages:
1. Imputation:
o Forward Fill: Replace missing values with the most recent non-missing
value.
o Backward Fill: Replace missing values with the next available non-missing
value.
o Linear Interpolation: Linearly interpolate between the nearest available
values.
o Mean/Median Imputation: Replace missing values with the mean or median
of the surrounding data.
2. Ignore Missing Values: Some functions, like pandas rolling or expanding,
automatically handle missing values by ignoring them while computing the moving
averages (i.e., only considering available data).
Handling Missing Values with Moving Averages in Python
Let’s explore how to handle missing values while computing moving averages using Python.
Example 1: Simple Moving Average (SMA) with Missing Values
Let’s create a sample time series with missing values and compute a simple moving average
using pandas.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# Create a sample time series with missing values
data = {'Date': pd.date_range(start='2020-01-01', periods=10, freq='D'),
'Value': [10, 15, np.nan, 20, 25, np.nan, 30, np.nan, 40, 45]}
df = pd.DataFrame(data)
df.set_index('Date', inplace=True)
# Plot the original data
plt.figure(figsize=(10, 6))
plt.plot(df.index, df['Value'], label='Original Data', marker='o')
plt.title('Time Series with Missing Values')
plt.xlabel('Date')
plt.ylabel('Value')
plt.show()
# Handle missing values by forward fill (propagate previous values forward)
df['Value_ffill'] = df['Value'].fillna(method='ffill')
# Calculate a simple moving average with a window size of 3
df['SMA'] = df['Value_ffill'].rolling(window=3).mean()
# Plot the results
plt.figure(figsize=(10, 6))
plt.plot(df.index, df['Value'], label='Original Data', marker='o', linestyle='--')
plt.plot(df.index, df['SMA'], label='Simple Moving Average', marker='x', linestyle='-',
color='red')
plt.title('Time Series with Moving Average (Handling Missing Values by Forward Fill)')
plt.xlabel('Date')
plt.ylabel('Value')
plt.legend()
plt.show()
Explanation:
1. Creating Data with Missing Values: We create a time series dataset with some
missing values (np.nan).
2. Forward Fill: We use fillna(method='ffill') to fill missing values by carrying forward
the previous value.
3. Simple Moving Average: We calculate a simple moving average using the
rolling(window=3).mean() function. It calculates the average over a window of 3 time
points, ignoring missing values that were forward-filled.
Example 2: Exponential Moving Average (EMA) with Missing Values
Let’s calculate the Exponential Moving Average (EMA) and handle missing values by
forward filling.
# Exponential Moving Average (EMA) with a span of 3 days
df['EMA'] = df['Value_ffill'].ewm(span=3, adjust=False).mean()
# Plot the results
plt.figure(figsize=(10, 6))
plt.plot(df.index, df['Value'], label='Original Data', marker='o', linestyle='--')
plt.plot(df.index, df['EMA'], label='Exponential Moving Average', marker='x', linestyle='-',
color='green')
plt.title('Time Series with Exponential Moving Average (Handling Missing Values by
Forward Fill)')
plt.xlabel('Date')
plt.ylabel('Value')
plt.legend()
plt.show()
Explanation:
1. Exponential Moving Average (EMA): The ewm(span=3) method computes the
exponential moving average, giving more weight to recent data.
2. Forward Fill for Missing Values: We forward fill missing values before computing
the EMA.
Other Imputation Methods for Missing Values
If you don't want to use forward or backward filling, you can use other imputation methods.
Here's an example of Linear Interpolation:
# Linear interpolation for missing values
df['Value_interp'] = df['Value'].interpolate(method='linear')
# Calculate moving average with interpolated values
df['SMA_interp'] = df['Value_interp'].rolling(window=3).mean()
# Plot the results
plt.figure(figsize=(10, 6))
plt.plot(df.index, df['Value'], label='Original Data', marker='o', linestyle='--')
plt.plot(df.index, df['SMA_interp'], label='SMA (Interpolated)', marker='x', linestyle='-',
color='purple')
plt.title('Time Series with Interpolation and Moving Average')
plt.xlabel('Date')
plt.ylabel('Value')
plt.legend()
plt.show()
Explanation:
1. Linear Interpolation: We use interpolate(method='linear') to estimate the missing
values based on the linear relationship between existing values.
2. Moving Average Calculation: We calculate the moving average using the
interpolated values.
Serial Correlation

Serial Correlation (Autocorrelation) in Time Series

Serial correlation, also known as autocorrelation, refers to the correlation of a time series
with its own past values. In simpler terms, it measures the relationship between a value in a
time series and its lagged (past) values. Serial correlation can indicate patterns, such as trends
or cycles, in a time series, or it can suggest that the data is influenced by past observations.
 Modeling: Serial correlation is crucial for building models. If the residuals (errors) of
a time series model exhibit serial correlation, this indicates that the model has not
fully captured the underlying pattern, and additional modeling may be required.
 Forecasting: Understanding serial correlation can improve forecasting by capturing
dependencies between past and future observations.
 Stationarity: A stationary time series often has zero or weak serial correlation. When
a series shows strong serial correlation, it might indicate the presence of trends or
seasonality.
Types of Autocorrelation
1. Positive Autocorrelation: This occurs when high values tend to be followed by high
values, and low values tend to be followed by low values. It suggests persistence or a
trend in the data.
2. Negative Autocorrelation: This occurs when high values tend to be followed by low
values, and low values tend to be followed by high values. This indicates an
alternating pattern in the data.
3. No Autocorrelation: When the correlation is close to zero, there is no relationship
between the values and their lagged values, indicating randomness or white noise.
Calculating Serial Correlation
To quantify serial correlation, you can use the autocorrelation function (ACF), which
measures the correlation between a time series and its lags.
Steps for Calculating Serial Correlation:
1. Autocorrelation Function (ACF): ACF measures the correlation between a time
series and its lagged values for different lag lengths.
2. Partial Autocorrelation Function (PACF): PACF measures the correlation between
the time series and its lags, after removing the effect of shorter lags. PACF helps
identify the order of the autoregressive (AR) part of ARIMA models.
Serial Correlation in Python
Using statsmodels and pandas, you can easily calculate serial correlation and visualize it.
1. Plotting the ACF and PACF
Let’s start by calculating and plotting the Autocorrelation Function (ACF) and Partial
Autocorrelation Function (PACF) to examine serial correlation.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
# Generate a sample time series with some random noise
np.random.seed(42)
data = np.random.randn(100) # 100 random values (simulating white noise)
ts = pd.Series(data)
# Plot the time series
plt.figure(figsize=(10, 6))
plt.plot(ts)
plt.title('Random Time Series (White Noise)')
plt.show()
# ACF and PACF plots
plt.figure(figsize=(12, 6))
# ACF Plot
plt.subplot(121)
plot_acf(ts, lags=20, ax=plt.gca())
plt.title('Autocorrelation Function (ACF)')
# PACF Plot
plt.subplot(122)
plot_pacf(ts, lags=20, ax=plt.gca())
plt.title('Partial Autocorrelation Function (PACF)')
plt.tight_layout()
plt.show()
Explanation:
 plot_acf: This function plots the ACF for different lags (how a value correlates with
its past values).
 plot_pacf: This function plots the PACF, which measures the correlation at a specific
lag after accounting for the correlations at shorter lags.
In the case of white noise (random data), we expect the autocorrelations at all lags to be close
to zero, and the plots should show no significant spikes.
2. ACF and PACF with Trend Data
Let’s create a time series with a trend and seasonality, which typically shows significant
autocorrelation.
# Simulating a time series with a trend (increasing values) and some noise
np.random.seed(42)
n = 100
trend = np.linspace(0, 10, n) # Trend component
seasonality = np.sin(np.linspace(0, 3*np.pi, n)) # Seasonal component
noise = np.random.randn(n) # Random noise
# Combining the components to create the time series
ts_trend = trend + seasonality + noise
# Plot the time series with trend
plt.figure(figsize=(10, 6))
plt.plot(ts_trend)
plt.title('Time Series with Trend and Seasonality')
plt.show()
# ACF and PACF plots for the time series with trend
plt.figure(figsize=(12, 6))
# ACF Plot
plt.subplot(121)
plot_acf(ts_trend, lags=20, ax=plt.gca())
plt.title('Autocorrelation Function (ACF) with Trend')
# PACF Plot
plt.subplot(122)
plot_pacf(ts_trend, lags=20, ax=plt.gca())
plt.title('Partial Autocorrelation Function (PACF) with Trend')
plt.tight_layout()
plt.show()
Explanation:
 The ACF plot will show significant autocorrelation at various lags, reflecting the
trend and seasonality in the data.
 The PACF plot is useful to identify how many lags (AR terms) to include in an
ARIMA model.
3. Durbin-Watson Test for Serial Correlation in Residuals
If you're modeling time series data, it’s essential to check if there is serial correlation in the
residuals. The Durbin-Watson test is used to detect the presence of autocorrelation in the
residuals of a regression model.
from statsmodels.tsa.arima.model import ARIMA
# Create a simple ARIMA model (for demonstration)
model = ARIMA(ts_trend, order=(1, 0, 0)) # AR(1) model
fitted_model = model.fit()
# Durbin-Watson test for residuals autocorrelation
from statsmodels.stats.stattools import durbin_watson
# Get the residuals from the fitted model
residuals = fitted_model.resid
# Perform the Durbin-Watson test
dw_stat = durbin_watson(residuals)
print(f'Durbin-Watson Statistic: {dw_stat}')
# Interpretation:
# A value of 2 means no autocorrelation.
# Values < 2 indicate positive autocorrelation.
# Values > 2 indicate negative autocorrelation.
Explanation:
 The Durbin-Watson statistic measures the degree of autocorrelation in the residuals.
A value close to 2 indicates no significant serial correlation, while values closer to 0
or 4 indicate strong positive or negative autocorrelation, respectively.
 1. Autoregressive (AR) Models: If significant autocorrelation is present in the data, you
might need to fit an Autoregressive (AR) model, where the value at time ttt depends
on previous values.
2. Differencing: In the case of trend or seasonality, differencing the series (i.e.,
subtracting the previous observation from the current one) can help eliminate serial
correlation by making the series stationary.
3. ARIMA (AutoRegressive Integrated Moving Average): ARIMA models combine
autoregression (AR), differencing (I), and moving averages (MA) to handle serial
correlation and forecast future values.
Introduction to survival analysis.

Survival analysis is a branch of statistics that deals with analyzing time-to-event data. The
primary goal is to understand the time it takes for an event of interest to occur. This type of
analysis is particularly useful when studying the duration until one or more events happen,
such as the time until a patient recovers from a disease, the time until a machine breaks down,
or the time until an individual defaults on a loan.
In survival analysis, the "event" typically refers to something of interest, like:
 Death (in medical research),
 Failure of a machine (in engineering),
 Default on a loan (in finance),
 Customer churn (in business).
1. Survival Function (S(t)): The survival function represents the probability that the
event of interest has not occurred by a certain time t. It is defined as:

1. Censoring: In survival analysis, censoring occurs when the event of interest has not
happened by the end of the observation period. There are two common types of
censoring:
 o Right censoring: When the subject has not yet experienced the event by the
end of the study.
o Left censoring: When the event occurred before the subject entered the study.
Censoring is an important feature of survival analysis, as it reflects the fact that we
don't always know the exact time of the event for every individual.
2. Kaplan-Meier Estimator: The Kaplan-Meier estimator is a non-parametric method
used to estimate the survival function from observed survival times, especially when
there is censoring. It provides an empirical estimate of the survival function.
3. Cox Proportional Hazards Model: The Cox model is a regression model that relates
the survival time to one or more predictor variables. It assumes that the hazard at any
time ttt is a baseline hazard multiplied by an exponential function of the predictor
variables. The model does not require the assumption of a specific survival
distribution, making it a widely used approach.
4. Log-Rank Test: The log-rank test is a statistical test used to compare the survival
distributions of two or more groups. It is commonly used in clinical trials to test
whether different treatment groups have different survival experiences.
Applications of Survival Analysis
 Medical Research: Estimating patient survival times after treatment or the time until
the onset of a disease.
 Engineering: Predicting the time until failure of machinery or components, such as
the lifespan of a battery or mechanical part.
 Business: Estimating the time until a customer churns or a product is returned.
 Finance: Analyzing the time until a loan defaults or the bankruptcy of a company.
Survival Analysis Example in Python
Here’s a simple example using Kaplan-Meier estimator and Cox Proportional
Hazards Model in Python.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from lifelines import KaplanMeierFitter, CoxPHFitter
from lifelines.datasets import load_rossi
# Example: Rossi dataset, a dataset on recidivism (criminal re-offending)
data = load_rossi()
# Kaplan-Meier Estimator: Estimate the survival function
kmf = KaplanMeierFitter()
 kmf.fit(durations=data['week'], event_observed=data['arrest'])
# Plot the Kaplan-Meier survival curve
plt.figure(figsize=(10, 6))
kmf.plot_survival_function()
plt.title("Kaplan-Meier Survival Curve")
plt.xlabel("Weeks")
plt.ylabel("Survival Probability")
plt.show()
# Cox Proportional Hazards Model: Fit the model
cph = CoxPHFitter()
cph.fit(data, duration_col='week', event_col='arrest')
# Display the summary of the Cox model
cph.print_summary()
# Plot the baseline survival function from the Cox model
cph.plot_baseline_survival()
plt.title("Baseline Survival Function (Cox Model)")
plt.show()
Explanation:
1. Kaplan-Meier Estimator: We use the KaplanMeierFitter from the lifelines package
to estimate the survival function for the dataset. This plot shows the survival
probability over time.
2. Cox Proportional Hazards Model: The CoxPHFitter is used to model the
relationship between the predictors (e.g., age, gender, etc.) and the time to event (e.g.,
recidivism).
Interpreting Results:
 Kaplan-Meier Curve: The plot shows how the survival probability decreases over
time.
 Cox Model Summary: The summary provides insights into how each predictor
variable influences the time to event (e.g., the effect of a specific treatment on
survival).