See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/221447683

Exception Based Modeling and Forecasting.

Conference Paper · January 2008

Source: DBLP

CITATIONS READS
10 481

1 author:

Igor A. Trubin
Peter the Great St.Petersburg Polytechnic University
24 PUBLICATIONS   50 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

PERFOMALIST web app tool to build IT-Control charts for visualizing weekly patterns and anomalies View project

My paper "Is your Capacity Available?" has been accepted for the imPACt 2016 by CMG Conference to be held November 7 - 10, 2016 in La Jolla CA. View project

All content following this page was uploaded by Igor A. Trubin on 20 September 2014.

The user has requested enhancement of the downloaded file.

 Exception Based Modeling and Forecasting

Igor Trubin, Ph.D., SunTrust Bank

How often does the need arise for modeling and forecasting? Should it be done manually by ad-hoc,
by project requests or automatically? What tools and techniques are best for that? When is a trending
forecast enough and when is a correlation with business drivers required? The answers to these
questions are presented in this paper. The capacity management system should automatically
provide a small list of resources that need to be modeled or forecasted; a simple spreadsheet tool
can be used for that. This method is already implemented on the author's environment with
thousands of servers.

1. Introduction

One of the essential jobs of a Capacity Planner is to

produce forecasts based on current and future
situation models. This type of job is a bit dangerous.
Just like a weather or stock market forecasting job, if
the prediction is correct nobody usually gives you
credit for it, but in case a prediction is wrong,
everybody notices and blames you!

My first and quite unpleasant experience with this

type of activity was during one of my dead-end
career paths when I was hired by a stock brokerage Figure 1 – Old and Good Prediction Example
company to develop a forecasting system. I (Scanned Image from Personal Archive)
passionately tried to predict the unpredictable
market, and finally decided to give up because I Since that first good forecast, our Capacity Planning
realized that the management did not really need team kept a special focus on producing a different
accuracy, but just nice pictures and graphs to type of forecasting/trending analysis, automating the
persuade the customers to buy something. process as much as possible. The result was
automatically updated trend forecast charts for every
In spite of all those unpleasant side effects of server for several metrics and most of the charts
forecasting, I finally found this job to be fascinating looked nice and beautiful, but in most cases were
once I produced my very first precise forecast shown useless as you can see on Figure 2.
on Figure 1. That was a rather simple model, based
on common benchmarks (TPM) to recalculate UNIX
box CPU usage for a few possible upgrade
scenarios.

After one of my recommendations was accepted, I

collected performance data on the newly upgraded
configurations and - bingo - the server worked as I
had predicted! That was a moment of truth for me,
but of course I did not get any special appreciation
from the customer. Figure 2 - Beautiful Trend-Forecast Chart

(I have previously experienced similar excitement Why? Because in a good environment most of the
during my teaching career when I would see some production boxes are not trending much and you
sparks of understanding in the eyes of my students. could only enjoy this type of healthy future
I hope all the readers of this paper have those predictions. As a result, we had to scan all those
sparks as well.) thousands of charts manually (eyeballing) to select

1
only those that had real and possibly dangerous D. The starting historical time point should take
trends. into account the significant events such as
hardware upgrades; virtual machines,
Some of these experiences are discussed previously databases and application migrations; LPAR
in CMG papers [1] and [2] where “… authors used reconfigurations and so on.
the SAS language to automate the production of E. “Bad” data points should be excluded from
resource projections from business driver inputs historical samples as “outliers”.
(see Figure 3). If projections are stored in a
database, automatic “actual vs. projected” RULE A: “Right Summarization”. Data collection
comparisons can also be displayed…” must provide very granular data (10 sec -15 min at
least). However, for analysis data should be
summarized by hour, day, week or month. It is not a
good idea to try and produce trend forecast analysis
against the raw and very granular data even using
good “time series” algorithms. On the other hand, if
you have only hourly or daily snap-shots type of data
the forecast could be very misleading even after
summarization.

If correlation analysis is needed, all data

(independent and dependent variables) should be
normalized by the same interval, and usually the
less granular the summarized data is – the better the
Figure 3 – Business Driver Based Forecast correlation that can be found. Below in paragraph
five of this paper you can see an example of
However, some challenges were reported [2]: “…the correlation analysis of CPU usage vs. Web hits that
authors’ experience is that this approach appears to uses 5 min. interval data summarized by days with a
be more accurate for a single resource such as very good result (see Figures 16, 17 and 18).
CPU. When applied to multiple file systems’ disk I/O,
results may vary greatly…”. And there were really RULE B: “Do not mix shifts”. This is a pretty
great variations even for CPU resources! All in all, it obvious rule. The following real data example on
appeared that the maintenance and usage of the Figure 4 shows the difference between a trend
“bulk” forecasting with or without business driver forecast that includes weekends and one that
correlations becomes a nightmare. excludes weekends. Note that a “no-weekends”
chart forecast reaches the yellow zone sooner!
Living with all those challenges, I dreamed of
automating the process of selecting the really
dangerous up or down trends of resource
consumption and finally have discovered a way of
doing so.

2. Basic Forecasting Rules

First of all let’s review some basic and obvious rules

of how to avoid common mistakes in forecasting
models:
A. Historical data should have the right
summarization, including data for
correlation (e.g. business drivers).
B. Do not mix shifts: forecasts should be
done separately for working days and/or
hours or off-shifts.
C. The result depends on the statistical model Figure 4 – Trend forecast with and without
chosen. weekends

2
RULE C: “Statistical model choice”. Let’s leave (The stepar and trend=2 are default values and
the detailed discussion of this subject to real they mean “stepwise autoregressive” with the “linear
statisticians/mathematicians and here let’s just trend model”)
formulate the basic rule: start with linear trend. Play
with other statistical algorithms if they are available RULE D: “Significant Events”. The standard
and use them only if absolutely necessary. forecast procedure (time series algorithm based)
might work well where the history is consistently
The chart in Figure 5 shows the same data with reflecting some natural growth. However, often due
different future trends because three different to upgrades, workload shifts or consolidations, the
st
algorithms were used. That figure shows that the 1 historical data consists of phases with different
method tries to reflect weekly oscillation patterns. The forecasting method should be adjusted
(seasonality) while others try to keep up along with to take into consideration only the latest phase with
the most recent trends; the last one is the most a consistent pattern.
aggressive.

Figure 6 –Trend Forecasts Based on Whole and

Partial Data History

For instance, if the history shown in Figure 6 began

in October instead of July, the future trend would be
more realistic as shown by the dashed lines on the
“future” side of the chart.

RULE E: “Outliers”. Unfortunately, a server can

occasionally experience some pathological workload
events such as the run-away processes capturing all
spare CPU resources or memory leak situations
where the application consumes more memory than
it can actually use.

No doubt these workload defects cause some

Figure 5 – Different Statistical Forecasting
problems for the server. Even if the system’s
Algorithms Results Comparison
performance with a parasite process is still
acceptable, the real resource usage is hidden and
Using SAS language it is easy to play with these
the capacity planning for resource usage is
models, through changing the “method” and
unpredictable. When the automatic future trend chart
“trend“ parameters on the following proc statement:
generator is used, the historical data with
proc forecast pathologies causes inaccurate prediction, as shown
data=<actual_dataset> on Figure 7. To improve the quality of trend-forecast
interval=day lead=30 analyses such types of events along with some
method=<STEPAR|EXPO|WINTERS>
trend=<1|2|3> unexpected outages should be removed from the
out=<actual&predicted_dataset>; history as outliers.
run;

3
 It is a known approach to mark a resource as
potentially running out of capacity by using future
trend intersection with some obvious threshold.
However, this approach does not work when the
threshold is unknown. Below we will discuss another
method that does not have this weakness.

While my colleague was developing the forecasting

system, I worked on the Exception Detection System
using as a basis some of the CMG publications
starting with the first MASF paper [3]. The basic idea
I implemented was the following:
Figure 7 – Memory Leak Issue Spoiled the Forecast Take some representative historical reference data;
set it as a baseline and then compare it with the
most recent actual data. If the actual data exceeds
some statistical thresholds, (e.g. Upper (UCL) and
3. Forecasting vs. Exception Detecting
Lower (LCL) Control Limits are mean plus/minus 3
standard deviations), generate an exception (alert
About 7 years ago, the first manager of my first via e-mail) and build a control chart like shown on
Capacity Planning team faced a tough dilemma: he Figure 8 to publish on the web:
needed to assign the following tasks, one to me and
one to another mathematically well educated
analyst:
- Forecasting System development, and
- Exception Detection System development.
Naturally, I dreamed of working on the first task, but
unfortunately (or fortunately?) my manager assigned
the second task to me because of my mechanical
engineering background in which the use of Statistic
Process Control methodology is widespread.

My colleague (he is the co-author of my pervious

CMG paper [8]) did a great job developing a
forecasting system to produce trend-forecast and
business correlated forecast charts based on
performance data from hundreds of servers and Figure 8 – Exception Detection Control Chart
databases. His scripts are still used today to
generate high quality graphs. The charts shown in This weekly control chart is a powerful tool. It shows
Figures 3, 6 and 7 are based on code he wrote. I at a glance not only the last week’s worth of hourly
learned from him how to produce similar charts and I data, but also the daily and weekly profiles of the
greatly appreciated his help. metric’s behavior plus a historical view. It actually
shows you if the data are trending up or down and
He also implemented some of the basic forecasting when exactly the trend occurs. Gradually, I realized
rules described above including RULE D that I had developed a trending analyzer in addition
“Significant Events”. He developed scripts to to an Exception Detector!
forecast when the future trend of the “database used
space” metric intersects with the “database current While improving this approach, I also realized that
allocation” metric. His script does this task three some of the basic forecasting rules already apply to
times for short, medium and long history sample the Exception Detection System:
data to show the best and the worst scenarios. As a
result, only a few problematic databases and table - RULE A: “Summarization” says that all
spaces show up on his report which is automatically metrics and subsystems under the
colored yellow or red based on the future date Exception Detector should be consistently
threshold intersections. summarized and the best summarization
level is a 6-8 month history of hourly data.

4
 That, for instance, allows you to see where behavior and some examples of this analysis were
system performance and business driver published in another CMG paper [5].
metrics correlate simply by analyzing control
charts. But the most efficient use of this metric is to filter the
top most exceptional resources in terms of unusual
- RULE B: “Do Not Mix Shifts” is easily usage.
demonstrated by the weekly/hourly control
chart because it visualizes the separation of Publishing this top list in some way (e.g. bar charts
work or peak time and off time. shown on Figure 9 or 16) along with links to control
charts significantly reduces the number of servers
- RULE C: “Statistical Model Choice” that require the focus of Capacity Planning or
means playing with different statistical limits Performance Management analysts.
(e.g. 1 st. dev. vs. 3 or more st. dev.) to tune
the system and reduce the rate of false
positives.

- RULE D: “Significant Events” is another

important tuning parameter of the system.
RULE D is used to determine the depth of
the reference set. Even with a constant for
the reference set (e.g. 6 months), the
Exception Detector has the ability to adjust
itself statistically to some events because
the historical period follows (moving forward)
the actual data and every event will
occasionally be older than the oldest day in
the reference set.

- RULE E: “Outliers” are easily found

Figure 9 – Top list of servers with unusual CPU
statistically by the Exception Detector as all usage
workload pathologies are statistically
unusual. By adding some (non-statistical)
filters to the system, the most severe of 4. Exception Based Forecasting
these pathologies could and should be
excluded from the history to keep the The Exception Detector provides a list of resources
reference set free from outliers. [4] with highly unusual consumption. In so doing, the
Exception Detector provides a targeted list and
Finally, to increase the accuracy of the Exception helps to apply all of the forecasting rules described
Detector and to reduce the number of the false below.
positive situations (false alerting), a new meta-metric
was added - “ExtraValue” (EV) of exception. Here is the suggested method for implementing that
Geometrically speaking, it is the area between the approach:
actual data curve (black line on a control chart) and
the statistical limit curves (red and yellow lines on - The data for the Exception Detector and
the control chart) (Figure 9). This metric is basically forecasting system should be the same.
a magnitude of exception and is always equal to - The trend-forecast charts should be
zero, if actual data fluctuates within statistically generated only for resources listed in the
healthy boundaries. But if EV stays >0 or <0 for a Exception Detector outputs.
while that means there is an unusual growth - The data for trending analysis should be
(trending up) or drop (trending down), respectively. freed up from outliers based on Exception
(This metric was first introduced in a 2001 CMG Detector pathology filters (e.g., free from
paper [7].) See APPENDIX for a mathematical run-away and memory leak days or hours).
method for calculating EV. - The starting time point(s) in the historical
data for trending analysis can be found
Having this metric recorded to some exception based on exception database data with
database allows for in-depth analysis of the system

5
 “ExtraValue” or EV metric records, as the Which presentation is better? It depends. The
most recent negative value of this metric common recommendation is to use the daily peak
indicates time when the data actually started for OLTP or web application servers and daily
trending up. average for back-up and other batch oriented
application servers.
To illustrate how this works, let’s look at a couple of
case studies with actual data. One day, server But even looking at the Daily Peak trend forecast,
number 9 has hit the exception list as shown on the future looks good. Why? Because RULE D is not
Figure 9. Clicking on the control chart link, which the applied and the entire history was used for
web report should have on the same page, brings up forecasting. But the history is obviously more
the control chart (shown on Figure 8). That indeed complicated, and it’s a good idea to analyze only the
shows some signs of exceptional server behavior: last part of the history. It can be seen clearly just by
eyeballing historical trend chart. Could that decision
- Some hourly exceptions occurred on be done automatically? Certainly, if one looks at the
Monday. history of “ExtraValue” or EV metric on Figure 12.
- During the entire previous week the actual
CPU utilization was slightly higher than Note that the most recent negative value of
average (green mean curve). ExtraCPUtime metric (which is the EV meta-metric
- On Friday the upper limit (red curve) derived from CPU utilization metric) points exactly to
reached the 100% threshold for a few hours, the point of time when CPU utilization started to
which indicates that in the past the actual grow. Basically, to find a good starting point for
data might be at 100% level on other analyzing the history one needs to find the roots in
Fridays; and Friday average curve is higher the following equation:
than on the other days.
EV (t ) 0
Based on this information, it is a good idea to look at
the historical trend. But which metric statistic is
better suited for that: daily average or average of where EV for this example is “ExtraCPUtime”
peak hour? Let’s look at Figure 10 where both (unusual CPU time used) function of time t. (in days)
statistics are presented:
If EV metrics are recorded daily in some database,
this equation could be easily solved by developing a
simple program using one of the standard
algorithms. The solution for the real data example
shown above is t =~ 04/22.

The final trend-forecast chart is shown on Figure 11.

It predicts that after 06/16 the server might be
running out of CPU capacity:

Figure 10 – Trend Forecast Chart of Daily Average Figure 11 – Corrected Trend Forecast
vs. Daily Peak Hour Average CPU Utilization

6
 Figure 12 – History of “ExtraValue” Metric (ExtraCPUtime) vs.
Daily Average of CPU Utilization

Another example is presented in Figures 13-15. It is

interesting that the EV metric somewhat mimics the
original metric but makes trending more obvious.

Figure 15 – The Short History Based Trend-forecast

Figure 13 – The Whole History Based Forecast

But what about the opposite side of the exceptions
list that reports resources with unusually low usage?
Yes, the Exceptions Detector publishes that as well
and it makes perfect sense to build control and
trend-forecast charts for each resource from that list.
One example of such a web report is shown on
Figure 16 for the VM host from which some virtual
machines were recently moved to another host. The
Exception Detector captures that event perfectly and
gives an analyst the possibility to see how much
resource was released.

Figure 14 - ExtraTime data analysis One of the unique parts of this method is the
following. If a metric does not have an obvious
Some oscillations are seen around the most resent threshold (e.g. I/O, paging or Web hits rates) the
negative EV value, but that might be tuned out as approach works anyway and the trend-forecast will
those cases are using 1 st. dev. threshold which is be built only for resources (e.g. disk, memory or
too sensitive. And of course by the term “recent” one Web application) that recently started dangerously
should assume at least a few weeks or more to have trending up. Additional modeling may be needed to
enough data for meaningful trend analysis. estimate what drives the increase and how to avoid
potential problems.

7
 Figure 16 – Web Report about Top Servers that Released CPU Resource

One day, the Exception Detector notified me that

5. Exception Based Modeling and Forecasting some web application had an unusual number of
web hits producing the hits rate control chart similar
How often do we need to perform modeling? What to the one on Figure 8, but without obvious
tools are good for that? Obviously, if there is a threshold. The trend forecast for CPU usage was
project to upgrade hardware or to consolidate automatically generated for the server hosting this
resources or consumers (applications, servers, VMs application as shown on Figure 10. Finally, both
or databases), a good capacity management CPU and Web hits data were downloaded to a
process assumes that the capacity planner is Spreadsheet for modeling. Combining those metrics
involved as a project resource and should model in one scattered chart (Figure 17) shows excellent
2
various what-if scenarios. The capacity planner is correlation with R =0.96.
fortunate if he has good data and good modeling
tools (for instance, a queuing theory based analytical
tool).

A good capacity management service should be

able to initiate this type of project. The Exception
Detection System and the forecasting based on that
system can be very helpful in this process. In most
cases, if one has good data and some statistical and
capacity management experience, the modeling can
be done effectively with a spreadsheet.

Even control charts can be built using just a

spreadsheet, as demonstrated in one of the past
CMG sessions [6]. Trend analysis (forecasting) can
be also done by a spreadsheet. (One of these types
of techniques such as Forecast(…) formula usage
was also demonstrated at CMG – [1]). Let’s look at Figure 17 – Correlation between CPU utilization and
some real data in a case study to see how that Web hits rate
works. Here is the most recent data for a previous
example mentioned in the introduction; see Figure 1.

8
This simple correlation model shows us that the This model also allows for a more complex analysis.
maximum number of hits per second that this server To apply the method explained in the previous
can handle is about 18. This is a meaningful result paragraph of this paper – calculate the historical
and if the application’s support team anticipates a starting point based on the most recent negative EV,
higher hit rate in the near future based on and look at the most recent trend which is
specifications, stress test results, customer behavior, apparently the worst case scenario as shown on
forecast and/or business projections, this server will Figure 20:
need more processing capacity to meet the
requirements.

The model also shows (Figure 18) that if the pattern

of this application usage remains the same, the
server will be at capacity in about two months.

Figure 20 – Worst Case Scenario

What spreadsheet features were used for this

modeling exercise? All are pretty standard and
simple:
- Figure 17: “XY (scattered)” standard chart
type plus “Add trendline” wizard.
Figure 18 – Current Capacity Usage Projection
- Figure 18, 19: Just “Add trendline” wizard
with the coefficient =0.75 to reflect the
But what if the number of anticipated hits is higher?
proposed capacity increase.
How much additional capacity would be needed?
- Figure 20: In addition to “Add trendline”
This model can help someone to play out this type of
wizard the future data were populated by
scenario. For instance, Figure 19 shows that adding
just dragging down the selected range of
25% more processing power to that server gives it
historical data as shown on Figure 21.
the ability to handle 20 hits per sec and that capacity
will be reached in a period twice as long in about 4
months.

Figure 19 – Proposed Capacity Usage Projection

Figure 21 – Future Data Population Technique

9
6. Summary 7. References

Capacity management in a large IT environment [1] Merritt, Linwood, "Seeing the Forest AND
should perform forecasting and modeling only when the Trees: Capacity Planning for a Large
it is really needed. This saves a lot of man-hours Number of Servers", Proceedings of the
and computer resources. United Kingdom Computer Measurement
Group, 2003
Exception Detection techniques along with an
Exception Database could be used to automate the [2] Merritt, Linwood and Trubin, Igor, Ph. D.,
decision making process with regard to what needs “Disk Subsystem Capacity Management,
to be modeled/forecasted and when. Based on Business Drivers, I/O
Performance Metrics and MASF”, CMG2003
MASF Control Charts have the ability to uncover Proceedings.
some trends showing actual data deviations from an
historical baseline. The most recent negative EV [3] Jeffrey Buzen and Annie Shum, "MASF --
(ExtraValue of exceptions meta-metric first Multivariate Adaptive Statistical Filtering",
introduced in CMG’01) is an indicator of the moment CMG1995 Proceedings, pp. 1-10.
of time when it is good to start the trending analysis
of an historical sample. [4] Igor Trubin, “Capturing Workload Pathology
by Statistical Exception Detection System”,
A common way of raising a future capacity concern CMG2005 Proceedings.
by calculating future trend intersection with some
constant threshold does not work for metrics without [5] Igor Trubin, “Global and Application Levels
obvious thresholds. The Statistical Exception Exception Detection System, Based on
Detection approach helps to produce the trending MASF Technique”, CMG2002 Proceedings.
analysis necessary for those cases.
[6] Igor Trubin, “System Management by
Workload pathologies (e.g. run-aways or memory Exception Part 6”, CMG2006 Proceedings.
leaks) should be excluded from an historical sample
in order to improve the forecasting. The Exception [7] Kevin McLaughlin and Igor Trubin, “Exception
Detector provides data (dates and hours) for that. Detection System, Based on the Statistical
Process Control Concept", CMG2001
Application data (e.g. web-hits) vs. Server Proceedings.
performance data (e.g. CPU utilization) correlation
analysis gives a priceless opportunity to add some [8] Igor Trubin and Ray White: “System
meaning to forecasting/modeling studies and that Management by Exception, Part Final”,
analysis can be done using standard spreadsheet CMG2007 Proceedings.
tools.

10
 8. APENDIX:

ExtraVolume meta-metric calculation

ExtraVolume (let’s call it EV) is basically a

magnitude of exception that occurred at a particular
time with some metric [7].

Let’s look at a 2D model first. The flat and linear

model of some performance metric behavior is
shown on Figure 22, where U is the metric, t is time,
UCL is Upper Control Limit and LCL is Lower
Control Limit.

Figure 23 – 3D Model Example:

(Built by Spreadsheet Graph Wizard)

Where
(U ( h, t ) UCL( h, t ))dh,U UCL 0
S
0,U UCL 0
(U ( h, t ) LCL( h, t ))dh,U LCL 0
S
Figure 22 – 2D Model 0,U LCL 0

+ -
In a general case S and S as shown on Figure 24
For this model the formula for EV calculations is have the following geometrical meaning: it is the
area between the actual data curve (U) and the
S ,U (t ) UCL(t ) 0 statistical limit curves (UCL and LCL). They should
be calculated only on intervals where the actual
EV (t ) S ,U (t ) LCL(t ) 0 metric is outside of the UCL - LCL band. If the metric
+ -
0,UCL(t ) U (t ) LCL(t ) U is within the band, then both S and S as well as
EV are equal to zero.
+ -
where S = U(t)-UCL(t) and S =U(t)-LCL(t)

Three-dimensional model is more realistic: h – hours

of a day (or week) or days of a week, t - days (or
weeks) and U – performance metric. By the way, the
MASF control chart such as the one shown on
Figure 8 is the 2D cut (projection) on a particular
week. This 3D model was introduced in last year
CMG paper [8] and one example of that 3D view is
shown here (Figure 23). For the full 3D model case
EV(t) calculations are a bit more complex:

EV(t) = S+ + S- Figure 24 – EV Geometrical Meaning

11

View publication stats