Machine Learning & Applications

ALARM INTELLIGENCE CASE

Assignment - I

Submitted to:
Prof. U Dinesh Kumar

Submitted By: Group 14

Faiz Alam - 2317069
Shreyash Baria - 2317054
Dodhia Kavit Ashok - 2317039
Asmita Johari - 2317021
V Sai Dheeraj Chandra – 2317033
EXECUTIVE SUMMARY

OSUM has teamed up with Drishya AI Labs to tackle a nagging problem: chattering alarms
wreaking havoc in its plants. These pesky interruptions not only disrupt operators but also
spell potential downtime and hefty losses for the company. To combat this, OSUM provided
data from its alarm systems, hoping to craft a predictive model to sniff out these troublesome
alarms.

However, the raw data necessitated meticulous feature engineering to distill meaningful
variables suitable for modeling. A notable hurdle was the absence of Active Time Duration
(ATD) for future alarms, prompting the application of advanced feature extraction techniques
to aggregate ATD values across various levels, leveraging metrics such as mean, median,
minimum, and maximum.

Initial attempts to construct models using logistic regression and decision tree algorithms
yielded suboptimal results, primarily characterized by inadequate sensitivity. Analysis
revealed a significant data imbalance ratio of 4, likely contributing to the model's
shortcomings. Consequently, various sampling methods were explored to rectify this
imbalance.

Furthermore, data normalization techniques were implemented to ensure consistency across

different ranges of values, thereby enhancing the accuracy of predictions. While the alarm tag
identifier emerged as a promising predictor of chattering behavior, integrating it directly into
the model posed challenges. To address this, additional feature extraction methods were
employed to effectively incorporate this crucial variable.

After undergoing iterative refinement processes, our efforts culminated in the development of
a decision tree model trained on oversampled data. This final model demonstrated significant
enhancements, with an AUC of 0.973 and sensitivity of 0.969. Furthermore, we successfully
identified the most influential features for predicting the dependent variable. These
performance metrics validate the model's effectiveness in accurately detecting potential
chattering alarms, thus aiding OSUM in ensuring seamless plant operations.

1
PROBLEM DESCRIPTION
The dependability of industrial alarm systems has a direct impact on industrial plant overhead
costs, product quality, and safety. The alarm system's ability to function correctly or whether
it causes operator distraction by sounding an excessive number of false and bothersome
alarms depends on the practice of alarm design. Alarm systems that are not set up correctly
are common in industry. This is frequently caused by having an alarm configuration that is
too extensive and by the time-consuming nature of the alarm rationalization process. Because
of this, there are typically more alarms during abnormalities in plant operations than the
operator can process and handle. During these alarm floods, most of the alarms are nuisance
(valid alarms with no useful information). Chattering or redundant alarms account for many
nuisance alarms. When one fault causes alarms to be triggered by multiple process variables,
this is known as a redundant alarm. In this instance, the operator is already aware that
something is wrong, but the constant alerts keep taking his or her attention away from fixing
the anomaly. A chattering alarm, on the other hand, is one that sounds repeatedly in a short
amount of time. Chattering alarms should be avoided as they can also be a distraction for the
operator.
OSUM, a privately held company in the oil and gas sector and a significant producer in
Western Canada, was facing the problem of chattering alarms and the subsequent downtime
of its plant. Plato, Information Systems Manager at OSUM, reached out to Drishya AI Labs
to predict the chattering of alarms. To improve the operator's performance, it is required to
predict the alarm chattering occurrence within a predetermined time frame. The plant
operator will be less distracted as a result, which will enhance his ability to respond to urgent
alarms. Consequently, there will be less chance of plant operations being disrupted, meaning
the company will suffer less financial loss.

MACHINE LEARNING ALGORITHMS

 Logistic Regression: Logistic regression is a statistical learning algorithm in which the

response variable takes a discrete value, and the feature (explanatory) variables can either
be continuous or discrete. LR is a supervised learning algorithm. In binary LR, the

2
 response variable takes only two values whereas in multinomial LR, the outcome variable
can take more than two values.
 Decision Trees: Decision Trees are a collection of predictive analytics techniques that use
tree-like graphs for predicting the value of a response variable based on the values of
explanatory variables. It is used for predicting both the discrete and the continuous
dependent variables.

CASE QUESTIONS

1. Use descriptive statistics and data visualization to gain insights from the data
provided. (2 Points)
 Following is a box plot of “ATD” variable. It shows that there are significant outliers
for this variable in the data.

 Following is the distribution of active time duration on the basis of CHB value:

3
 This can also be visualized on the log scale.

The observed pattern suggests that when 'CHB' equals 1, 'ATD' tends to be notably
low. However, utilizing 'ATD' directly in modeling isn't feasible as predicting alarm
chattering beforehand doesn't allow us to know its duration. Instead, we can leverage
aggregated historical 'ATD' data to construct the model.
 Next, we wanted to see frequency and chattering behavior of different alarm type tag.

4
 This indicates that certain alarms are prone to chatter more often, while others seldom
display this characteristic. Hence, incorporating this variable into the model appears
beneficial. However, given the presence of 460 unique values for this variable,
employing One Hot encoding could lead to a significant increase in the size of the
data table. An alternative approach could involve including the historical percentage
of chattering alarms as an additional column in the dataframe.
 Next, we plotted the alarm occurrences and chattering proportion by month.

5
 The above analysis has been shown for hour below.

 The above plot has been shown for week below.

 The above plot has been shown for Alarm tag type below.

6
 Following plot shows the correlation of the top 20 highest correlated columns with the
dependent variables, “CHB”.

7
 Most of the features having higher correlation values are derived features obtained
through feature extraction techniques.

2. Develop univariate and multivariate logistic regression (LR) models to generate

insights on chattering behaviour. (6 Points)

a) Which alarm types have the highest and lowest probability of alarm chattering?
GLM Summary
Null df Residual df AIC Dispersion Iterations
Deviance Deviance Residual
21488.5600 19999 18993.1100 19995 19003.1100 1 5

Coefficients

Estimate Std. Error z value Pr(>|z|)

(Intercept
-2.5546 0.0792 -32.2727 1.6879e-228 ***
)
Pressure 2.6143 0.0867 30.1684 6.1525e-200 ***
Flow 1.8982 0.0881 21.5408 6.4597e-103 ***
Others 1.2539 0.0876 14.3205 1.6303e-46 ***
Level 0.4981 0.0875 5.6953 1.2314e-08 ***
Note:
Signif. codes: 0 ' *** ' 0.001 ' ** ' 0.01 ' * ' 0.05 '.' 0.1 ' ' 1

log[P(CHB=1)/(1−P(CHB=1))]=α+β1(Flow)+β2(Level)+β3(Pressure)
+β4(Others)
log⎡⎣ˆP(CHB=1)/(1−ˆP(CHB=1))⎤⎦=−2.5546+1.8982(Flow)
+0.4981(Level)+2.6143(Pressure) +1.2539(Others)
All four features are significant.
 From the above table, the alarm-type pressure has the highest coefficient value. It
means that it has the highest probability of alarm chattering. The probability of alarm
chattering for this class is 0.5149.
 The alarm type temperature, which is the same as the base category, has the lowest
probability since the alpha value is negative and is the lowest. The probability of
alarm chattering for this class is 0.0721.

8
b) What is the model's sensitivity, specificity, and overall accuracy with an alarm
tag as a feature?
Confusion Matrix, positive level: 1
Prediction Reference
1 0
1 1658 1562 3220
0 2908 13872 16780
4566 15434 20000

Sensitivity 0.3631
Specificity 0.8988
Accuracy 0.7765

c) Check whether the time of the day significantly impacts chattering.

The above figure shows that all the values for time of the day significantly impacts
chattering. Therefore, time of the day significantly impacts chattering.

d) Build a logistic regression model after incorporating all the statistically

significant features.
All features are significant in the below model.

9
 GLM Summary
Null Deviance df Residual Deviance df Residual AIC Dispersion Iterations
21488.5600 19999 16212.3100 19984 16244.3100 1 8

Coefficients
Estimate Std. z value Pr(>|z|)
Error
(Intercept) -0.6459 0.0521 -12.4004 2.5992e-35 ***
ATD -0.0093 0.0004 -24.4966 1.6076e-132 ***
Pressure 1.1231 0.0571 19.657 5.0377e-86 ***
Level -0.9064 0.0572 -15.8532 1.3360e-56 ***
Temperature -1.0953 0.0936 -11.6983 1.2999e-31 ***
Month.Name_Nov 0.9265 0.0631 14.6832 8.2567e-49 ***
X1st.Week -0.5152 0.0641 -8.0331 9.5019e-16 ***
Month.Name_Dec 0.4358 0.0636 6.8473 7.5261e-12 ***
Hour.0.6 -0.2874 0.0493 -5.833 5.4439e-09 ***
Month.Name_Jul -0.444 0.0866 -5.1276 2.9345e-07 ***
Month.Name_Sep -0.6976 0.1209 -5.7711 7.8778e-09 ***
Month.Name_Oct -0.3553 0.0732 -4.8508 1.2296e-06 ***
Flow 0.3222 0.059 5.4604 4.7501e-08 ***
X3rd.week 0.2261 0.0448 5.0417 4.6143e-07 ***
Month.Name_Aug 0.3292 0.0741 4.4456 8.7647e-06 ***
Month.Name_Ma 0.3088 0.0794 3.889 0.0001 ***
y
Note:
Signif. codes: 0 ' *** ' 0.001 ' ** ' 0.01 ' * ' 0.05 '.' 0.1 ' ' 1

We also notice that the feature ATD cannot be used as is, considering we wouldn’t have
the ATD data of a potential alarm that will chatter.
So, we built a model without ATD and then ran the forward BIC. We get the variables
below as significant when we selected all the variables except for ATD.

Coefficients
Estimat Std. z Pr(>|z|)
e Error value
(Intercept) -2.4838 0.0866 - 4.6888e-

10
 28.692
181 ***
8
30.168 6.1295e-
Pressure 2.6902 0.0892
5 200 ***
20.193 1.1082e-90
Flow 1.8209 0.0902
9 ***
13.890 7.2313e-44
Others 1.2407 0.0893
5 ***
Month.Name_N 13.495 1.6593e-41
0.7766 0.0575
ov 6 ***
- 5.3584e-20
X1st.Week -0.5582 0.0610
9.1565 ***
Month.Name_D 3.4881e-15
0.4664 0.0592 7.8721
ec ***
- 9.6342e-17
Hour.0.6 -0.4083 0.0491
8.3092 ***
Month.Name_A 5.9632e-07
0.3449 0.0691 4.9924
ug ***
Month.Name_S - 4.0815e-11
-0.7600 0.1151
ep 6.6011 ***
Month.Name_Ju - 1.1777e-09
-0.5061 0.0832
l 6.0832 ***
Month.Name_O - 1.9892e-10
-0.4415 0.0694
ct 6.3622 ***
6.1347e-08
X3rd.week 0.2293 0.0423 5.4148
***
4.2462e-07
Level 0.4498 0.0889 5.0576
***
-
Hour.7.12 -0.1636 0.0441 0.0002 ***
3.7136
Note:
Signif. codes: 0 ' *** ' 0.001 ' ** ' 0.01 ' * ' 0.05 '.' 0.1 ' ' 1

3. Develop a classification tree model and infer insights based on the model. (4 Points)

11
CHB Mean is the ratio of the number of chattering and the total number of times the
alarm has been triggered.
Alarm.Tag.Type_H_Week_M_Median is the median of ATD of the groupby-ed
variables Alarm Tag Type, Hour, Week and Month.
Alarm.Tag.Type_M_Min is the minimum of the ATD of the groupby-ed variables
Alarm Tag Type & Month.
Alarm.Tag.Type_H_Week_Max is the maximum of the ATD of the groupby-ed
variables Alarm Tag Type, Hours & Week.
H_Week_M_Median is the median of the ATD of the groupby-ed variables Hour, Week
& Month
Alarm.Tag.Type_H_Min is the minimum of the ATD of the groupby-ed variables
Alarm Tag Type & Hours
Alarm.Tag.Type_H_M_Min is the minimum of the ATD of the groupby-ed variables
Alram Tag Type, Hour & Month
Rules:
Rule 1: If CHB Mean is < 0.38 then predicted class is 0 with support 72%.
Rule 2: If CHB Mean is >= 0.38, Alarm.Tag.Type_H_Week_M_Median >= 0.0043,
CHB Mean < 0.83, Alarm.Tag.Type_H_Week_Max >= 0.95 then the predicted class is
0 with support 2%.

12
Rule 3: If CHB Mean is >= 0.38, Alarm.Tag.Type_H_Week_M_Median >= 0.0043,
CHB Mean < 0.83, Alarm.Tag.Type_H_Week_Max < 0.95, H_Week_M_Median >=
0.081, then the predicted class is 0 with support 3%.
Rule 4: If CHB Mean is >= 0.38, Alarm.Tag.Type_H_Week_M_Median >= 0.0043,
CHB Mean < 0.83, Alarm.Tag.Type_H_Week_Max < 0.95, H_Week_M_Median <
0.081, then the predicted class is 1 with support 3%.
Rule 5: If CHB Mean is >= 0.38, Alarm.Tag.Type_H_Week_M_Median >= 0.0043,
CHB Mean >= 0.83, then the predicted class is 1 with support 1%.
Rule 6: If CHB Mean is >= 0.38, Alarm.Tag.Type_H_Week_M_Median < 0.0043,
Alarm.Tag.Type_H_Min >= 0.90, then the predicted class is 0 with support 1%.
Rule 7: If CHB Mean is >= 0.38, Alarm.Tag.Type_H_Week_M_Median < 0.0043,
Alarm.Tag.Type_H_Min < 0.90, Alarm.Tag.Type_H_M_Min < 0.0046, then the
predicted class is 0 with support 0%.
Rule 8: If CHB Mean is >= 0.38, Alarm.Tag.Type_H_Week_M_Median < 0.0043,
Alarm.Tag.Type_H_Min < 0.90, Alarm.Tag.Type_H_M_Min >= 0.0046, then the
predicted class is 1 with support 18%.
Confusion Matrix, positive Accuracy and Kappa
Additional Statistics
level: 1 Statistics
Value Value
Prediction Reference
s s
0.797 0.912
1 0 Sensitivity Accuracy
4 4
0.946 0.749
1 3641 826 Specificity Kappa
5 6
0.815 Accuracy: 0.908
0 925 14608 Pos Pred Value
1 95%CILower 4
0.940 Accuracy: 0.916
Neg Pred Value
4 95%CIUpper 3
0.815 Accuracy: No Info 0.771
Precision
1 Rate 7
0.797
Recall Accuracy: PValue 0
4
0.806 Mcnemar's Test: 0.019
F1
2 PValue 2
0.228
Prevalence
3
Detection Rate 0.182
Detection 0.223
Prevalence 4
Balanced 0.871
Accuracy 9

13
 The area under the curve (AUC) is 0.9069

4. Why is sensitivity low in LR and decision tree models? Revise your previous models
by using strategies that can be used to improve the sensitivity (correctly predicting
the positive class records). (6 Points)
In both the models, we noticed
that sensitivity was very low. For
example, in our LR model, the
sensitivity was 0.36 whereas the
accuracy of the model was 0.77.
This is most likely happening on
account of imbalance in the data.
The adjacent count plot shows
this imbalance.
To resolve this, we tried three different sampling methods: under-sampling, over-
sampling and SMOTE.

On top of that, we also tried some feature extraction techniques. Two prominent feature
extraction techniques have been described below:
 We can’t use Active time duration directly. However, we can use the historical data to
create new columns such as mean, median, max, and min of different combinations.
 We also noticed that chattering behavior is dependent on Alarm tag identifier. The
following plot shows the top 50 alarm tag identifiers on the basis of alarm frequency.

14
The left axis shows the frequency of the alarm and the right axis shows the proportion
of times alarm was chattering.

In the above plot, it is obvious that for some of the identifiers, chattering behavior is
always zero. However, this categorical column has 460 unique values and to use this
column in the model, we need to do one hot encoding which will make the data very
complex.
Therefore, instead of adding this column directly, we can create two new columns
which show the proportion and number of times chattering behavior was
demonstrated (No of times chattering behavior / Frequency of alarm) for each of the
alarm tag identifiers.
Also, we performed
BIC to identify the
statistically significant
columns to add in our
model. Following figure
shows the significant
columns.

15
Furthermore, all the continuous features were scaled using MinMaxScaler to bring the
data into the same range, ensuring consistency and facilitating better comparison and
analysis across variables.

Following table shows the metrics for all the models that we created after sampling
and feature extraction techniques.

In our scenario, both recall and precision hold significance. Sensitivity plays a crucial
role as we aim to avoid overwhelming our system with the resolution of chattering
alarms. Precision is equally vital because addressing non-existent issues could
potentially exacerbate complications and adversely impact our business operations.
Hence, our primary metrics for determining the optimal model include Sensitivity, F1
Score, and AUC. Consequently, the Decision Tree model with oversampled sampling
technique emerges as the most suitable choice. This model exhibits an AUC of 0.973,
F1 score of 0.947, precision of 0.927, and sensitivity of 0.969.
Following graphs shows the feature importance of the top 20 variables in terms of
their reduction in Gini impurity.

16
5. What will be your recommendations for OSUM to manage alarm chattering? (2
Points)
i) Usage of historical data and ML models in predicting future behavior – In this analysis,
we saw that we were able to successfully get sensitivity and AUC in excess of 0.95.
Therefore, this problem can be solved easily by using Machine learning model.
Therefore, company should incorporate ML models in their decision making analysis.
ii) In our EDA analysis, it was observed that when 'CHB' equals 1, 'ATD' tends to be
notably low. Therefore, for alarm tag types (or alarm tag identifiers to reduce False
Negatives), we can implement settings to trigger alerts only after a certain threshold of
time has elapsed. We will able to reduce False positives by applying this method.
iii)Some alarm tag identifiers exhibit significantly lower instances of chattering behavior
compared to others of the same type. Hence, it may be beneficial to prioritize the
acquisition or integration of these high-performing alarm tags to replace the ones
showing higher instances of chattering behavior.

17