Predictive Business Process Monitoring considering Reliability and Risk
Download as PPTX, PDF1 like256 views
The document discusses predictive business process monitoring, focusing on enhancing proactive adaptation by considering reliability and risk. It emphasizes the importance of prediction accuracy for minimizing unnecessary adaptations while preventing actual violations, using ensemble machine learning techniques. Experimental results indicate that integrating reliability estimates and risk factors can lead to significant cost savings in process adaptations.
![Considering Reliability
Experimental Results
10SE 2018, Ulm
EnsSize=100, BootsSize=80, Voting=simple, AdaptSuccRate=0.80
Tot - Lambda [%] 0 Tot - Lambda [%] 10 Tot - Lambda [%] 20 Tot - Lambda [%] 30
Tot - Lambda [%] 40 Tot - Lambda [%] 50 Tot - Lambda [%] 60 Tot - Lambda [%] 70
0,45 0,50 0,55 0,60 0,65 0,70 0,75 0,80 0,85 0,90 0,95 1,00 1,05
Reliability Threshold
60.000
70.000
80.000
90.000
100.000
110.000
120.000
130.000
Cost
(reliability threshold)
costs
= .9
= .8
= 1
= .7
= .6
= .5
= .4
= .3
= .2
= 0
= .1
.50 .55 .60 .65 .70 .75 .80 .85 .90 .95 1.0
130,000
120,000
110,000
100,000
90,000
80,000
70,000
60,000
No proactive
process adaptationProactive process adaptation w/out reliability
Key:
Negative effect
Positive effect
Optimum
alpha-0.1 alpha-0.2 alpha-0.3 alpha-0.4 alpha-0
alpha-0.8000001 alpha-0.9000001 alpha-1.0000001
0,45 0,50 0,55 0,60 0,65 0,70 0,75 0
Reliability Thresh
0,72
0,74
0,76
0,78
0,8
0,82
0,84
0,86
0,88
NonViolationRate
alpha-0.1 alpha-0.2 alpha-0.3 alpha-0.4 alpha-0
alpha-0.8000001 alpha-0.9000001 alpha-1.0000001
0,45 0,50 0,55 0,60 0,65 0,70 0,75 0
Reliability Thresh
0,72
0,74
0,76
0,78
0,8
0,82
0,84
0,86
NonViolationRate
= .8
Very high adaptation
costs, not compensated
by avoided penalties
Missed required adaptations
(even though cheap) due to
“loss” of predictions](https://image.slidesharecdn.com/se18prediction-180308084553/85/Predictive-Business-Process-Monitoring-considering-Reliability-and-Risk-10-320.jpg)
![Accuracy of individual prediction
Reliability estimate to quantify probability of violation
Severity of violation
• E.g., contractual penalties (such as stipulated in SLAs)
Estimated penalty to quantify severity (in terms of costs)
Risk = Probability of occurrence × Severity [ISO 31000:2009]
Risk = Reliability estimate × Estimated penalty
SE 2018, Ulm 14
Considering Risk
Factors impacting the success of the adaptation decision](https://image.slidesharecdn.com/se18prediction-180308084553/85/Predictive-Business-Process-Monitoring-considering-Reliability-and-Risk-13-320.jpg)
![Conclusions and Perspectives
Predictive business process monitoring
• Prediction of potential problems before they occur
• Proactive adaptation of processes
• Cost Savings
• Reliability: 14% on average
• Risk: + 23% on average
Deep Learning for Process Prediction
• Recurrent Neural Networks (RNNs) with LSTM
• Initial results: 27% higher accuracy than Multi-Layer Perceptron (MLP)
SE 2018, Ulm 18
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0 10 20 30 40 50 60 70 80 90
Diagrammtitel
num s2e noplannednopath mlp
Checkpoint [relative prefix]
Accuracy[MCC]
RNN
MLP](https://image.slidesharecdn.com/se18prediction-180308084553/85/Predictive-Business-Process-Monitoring-considering-Reliability-and-Risk-17-320.jpg)
Predictive Business Process Monitoring considering Reliability and Risk
- 1. Predictive Business Process Monitoring unter Berücksichtigung von Prognoseverlässlichkeit und Risiko Andreas Metzger, Philipp Bohn, Felix Föcker CAiSE 2017 - https://doi.org/10.1007/978-3-319-59536-8_28 ICSOC 2017 - https://doi.org/10.1007/978-3-319-69035-3_25
- 2. Motivation Predictive Monitoring and Proactive Adaptation 2SE 2018, Ulm monitor predict real-time decision proactive adaptation time t t + planned / acceptable situations = Violation = Non- Violation e.g., delay in freight delivery time e.g., schedule faster means of transport
- 3. Agenda 1. Considering Reliability 2. Considering Risk 3. Conclusions and Perspectives SE 2018, Ulm 3
- 4. Considering Reliability Prediction Accuracy SE 2018, Ulm 4 • Prediction accuracy is key for proactive process adaptation • Prediction accuracy = ability of prediction technique • to forecast as many true violations as possible, • while generating as few false alarms as possible • True violation triggering of required adaptations • Missed required adaptation = less opportunity for proactively preventing or mitigating a problem • False alarm triggering of unnecessary adaptation • Unnecessary adaptation = additional costs for executing the adaptations, while not addressing actual problems
- 5. Considering Reliability Prediction Accuracy • Research focused on aggregate accuracy • E.g., precision, recall, mean average prediction error, … • But: aggregate accuracy gives no direct information about error of an individual prediction • Prediction reliability estimates provide such information SE 2018, Ulm 5 Aggregate Accuracy 75% 75% 75% 75% Distinguish between more or less reliable predictions on case by case basis Prediction # 1 2 3 … Reliability Estimate 60% 90% 70% …
- 6. Considering Reliability Predictive Monitoring with Reliability Estimates SE 2018, Ulm 6 monitor predict real-time decision time t t + planned / acceptable situations = Violation = Non- Violation ≤ threshold no adaptation > threshold adaptation + Reliability estimate Reliability estimates offer more information for decision making proactive adaptation
- 7. Considering Reliability Computing Predictions and Reliability Estimates Foundation: Ensemble prediction using Machine Learning SE 2018, Ulm 7 Prediction T Reliability estimate Process Monitoring Data Classification Model 1 Classification Model m{ {{ Each model of ensemble trained differently (bagging) T1 Tm
- 8. Considering Reliability Experimental Design Cost Model and Experimental Variables SE 2018, Ulm 8 Costs Adaptation Cost Adaptation Cost + Penalty > ≤ No Adaptation Adaptation Reliability Violation Non-Violationeffective not effective 0 PenaltyViolation Non-Violation • Reliability threshold • Adaptation effectiveness • Relative adaptation costs : adaptation cost = · penalty
- 9. Considering Reliability Experimental Design Process Model and Data Set Airfreight process • 5 months of operational data • 3 942 process instances • 56 082 service invocations 9SE 2018, Ulm Point of Prediction
- 10. Considering Reliability Experimental Results 10SE 2018, Ulm EnsSize=100, BootsSize=80, Voting=simple, AdaptSuccRate=0.80 Tot - Lambda [%] 0 Tot - Lambda [%] 10 Tot - Lambda [%] 20 Tot - Lambda [%] 30 Tot - Lambda [%] 40 Tot - Lambda [%] 50 Tot - Lambda [%] 60 Tot - Lambda [%] 70 0,45 0,50 0,55 0,60 0,65 0,70 0,75 0,80 0,85 0,90 0,95 1,00 1,05 Reliability Threshold 60.000 70.000 80.000 90.000 100.000 110.000 120.000 130.000 Cost (reliability threshold) costs = .9 = .8 = 1 = .7 = .6 = .5 = .4 = .3 = .2 = 0 = .1 .50 .55 .60 .65 .70 .75 .80 .85 .90 .95 1.0 130,000 120,000 110,000 100,000 90,000 80,000 70,000 60,000 No proactive process adaptationProactive process adaptation w/out reliability Key: Negative effect Positive effect Optimum alpha-0.1 alpha-0.2 alpha-0.3 alpha-0.4 alpha-0 alpha-0.8000001 alpha-0.9000001 alpha-1.0000001 0,45 0,50 0,55 0,60 0,65 0,70 0,75 0 Reliability Thresh 0,72 0,74 0,76 0,78 0,8 0,82 0,84 0,86 0,88 NonViolationRate alpha-0.1 alpha-0.2 alpha-0.3 alpha-0.4 alpha-0 alpha-0.8000001 alpha-0.9000001 alpha-1.0000001 0,45 0,50 0,55 0,60 0,65 0,70 0,75 0 Reliability Thresh 0,72 0,74 0,76 0,78 0,8 0,82 0,84 0,86 NonViolationRate = .8 Very high adaptation costs, not compensated by avoided penalties Missed required adaptations (even though cheap) due to “loss” of predictions
- 11. Considering Reliability Experimental Results • Observations for full range of , , (= 5000 cases) • Striving balance between avoiding unnecessary proactive actions and rejecting required proactive actions • Cost savings due to proactive process adaptation • No, in 47.5% of the cases • Yes, in 52.5% of the cases • Cost savings due to considering reliability estimates • No, in 17,1% of the cases • Yes, in 82.9% of the cases 12SE 2018, Ulm Cost savings Frequency Savings from 2% to 54%, 14% on average
- 12. Agenda 1. Considering Reliability 2. Considering Risk 3. Conclusions and Perspectives SE 2018, Ulm 13
- 13. Accuracy of individual prediction Reliability estimate to quantify probability of violation Severity of violation • E.g., contractual penalties (such as stipulated in SLAs) Estimated penalty to quantify severity (in terms of costs) Risk = Probability of occurrence × Severity [ISO 31000:2009] Risk = Reliability estimate × Estimated penalty SE 2018, Ulm 14 Considering Risk Factors impacting the success of the adaptation decision
- 14. Considering Risk Risk-based Proactive Process Adaptation Decision SE 2018, Ulm 15 monitor predict real-time decision proactive adaptation time t t + planned / acceptable situations = Violation = Non- Violation R ≤ threshold no adaptation R > threshold adaptation + Risk R Risk estimate as basis for decision making
- 15. Ensemble Considering Risk Computing Penalty 16SE 2018, Ulm Penalty = Process Monitoring Data { = 1 𝑛 × 𝑖=1,…,𝑛 𝑎𝑖 − 𝐴 Regression Model 1 Regression Model n a1 an { Deviation δ Linear with cap clin c 0 1 δ Constantc 0 cconst 1 c Step-wise (s steps) δ 1/s 2/s cstep 1 2/s·cstep 1/s·cstep (s-1)/s … 0 𝑐()
- 16. Considering Risk Experimental Results Penalty R = 0.1 R = 0.3 R = 0.5 R = 0.7 R = 0.9 constant -19.0 -20.0 -17.0 -3.0 3.1 step-wise -14.0 12.0 20.0 20.0 8.6 linear 0.6 21.0 27.0 26.0 11.0 SE 2018, Ulm 17 Cost savings (averaged over α ={0.1, 0.2, 0.3, … ,1}) Constant penalty Step-wise penalty Linear penalty Risk threshold R Costsavings
- 17. Conclusions and Perspectives Predictive business process monitoring • Prediction of potential problems before they occur • Proactive adaptation of processes • Cost Savings • Reliability: 14% on average • Risk: + 23% on average Deep Learning for Process Prediction • Recurrent Neural Networks (RNNs) with LSTM • Initial results: 27% higher accuracy than Multi-Layer Perceptron (MLP) SE 2018, Ulm 18 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0 10 20 30 40 50 60 70 80 90 Diagrammtitel num s2e noplannednopath mlp Checkpoint [relative prefix] Accuracy[MCC] RNN MLP
- 18. Thank You / Danke SE 2018, Ulm 19 …the EFRE co-financed operational program NRW.Ziel2 http://www.lofip.de …the EU’s Horizon 2020 research and innovation programme under Objective ICT-15 ‘Big Data PPP: Large Scale Pilot Actions ‘ http://www.transformingtransport.eu Research leading to these results has received funding from…