CPSC 531:

System Modeling and Simulation

Carey Williamson
Department of Computer Science
University of Calgary
Fall 2017
 Motivational Quote

“If you can’t measure it, you can’t improve it.”

- Peter Drucker

2
 (Slightly Revised) Motivational Quote

model
“If you can’t measure it, you can’t improve it.”
- Peter Drucker

3
 Simulation Input Analysis

▪ Input models are the driving force for many simulations

▪ Quality of the output depends on the quality of inputs

▪ There are four main steps for input model development:

1. Collect data from the real system
2. Identify a suitable probability distribution to represent the
input process
3. Choose parameters for the distribution
4. Evaluate the goodness-of-fit for the chosen distribution and
parameters

4
 Data Collection

▪ Data collection is one of the biggest simulation tasks

▪ Beware of GIGO: Garbage-In-Garbage-Out

▪ Suggestions to facilitate data collection:

— Analyze the data as it is being collected: check adequacy
— Combine homogeneous data sets (e.g. successive time
periods, or the same time period on successive days)
— Be aware of inadvertent data censoring: quantities that are
only partially observed versus observed in their entirety;
gaps; outliers; risk of leaving out long processing times
— Collect input data, not performance data (i.e., output)

5
 Data Analysis Checklist (meta-level)

▪ Where did this data come from?

▪ How was it collected?
▪ What can it tell me?

▪ Do some exploratory data analysis (see next slide)

▪ Does this data make sense?

▪ Is it representative?
▪ What are the key properties?
▪ Does it resemble anything I’ve seen before?
▪ How best to model it?

6
 Data Analysis Checklist (detailed-level)

▪ How much data do I have? (N)

▪ Is it discrete or continuous?
▪ What is the range for the data? (min, max)
▪ What is the central tendency? (mean, median, mode)
▪ How variable is it? (mean, variance, std dev, CV)
▪ What is the shape of the distribution? (histogram)
▪ Are there gaps, outliers, or anomalies? (tails)
▪ Is it time series data? (time series analysis)
▪ Is there correlation structure and/or periodicity?
▪ Other interesting phenomena? (scatter plot)
7
 Identifying the Distribution

Non-Parametric Approach: does not care about the actual

distribution or its parameters; simply (re-)generates observations
from the empirically observed CDF for the distribution.
- less work for the modeler, but limited generative capability
(e.g., variety; length; repetitive; preserves flaws in data)

Parametric Approach: tries to find a compact, concise, and

parsimonious model that accurately represents the input data.
- more work, but potentially valuable model (parameterizable)
1. Histograms (visual/graphical approach)
2. Selecting families of distributions (logic/statistics)
3. Parameter estimation (statistical methods)
4. Goodness-of-fit tests (statistical/graphical methods)
8
 Histograms (1 of 3)

▪ Histogram: A frequency distribution plot useful in

determining the shape of a distribution
— Divide the range of data into (typically equal) intervals
or cells
— Plot the frequency of each cell as a rectangle

▪ For discrete data:

— Corresponds to the
probability mass function

▪ For continuous data:

— Corresponds to the
probability density function

9
 Histograms (2 of 3)

▪ The key problem is determining the cell size

— Small cells: large variation in the number of observations
per cell
— Large cells: details of the distribution are completely lost
— It is possible to reach very different conclusions about the
distribution shape

▪ The cell size depends on:

— The number of observations
— The dispersion of the data

▪ Guideline:
— The number of cells ≈ the square root of the sample size

10
 Histograms (3 of 3)

 Example: It is possible to reach very different conclusions

about the distribution shape by changing the cell size

Same data
with different
interval sizes

11
 Selecting the Family of Distributions (1 of 4)

▪ A family of distributions is selected based on:

— The context of the input variable
— Shape of the histogram

▪ Frequently encountered distributions:

— Easier to analyze: Exponential, Geometric, Poisson
— Moderate to analyze: Normal, Log-Normal, Uniform
— Harder to analyze: Beta, Gamma, Pareto, Weibull, Zipf

12
 Selecting the Family of Distributions (2 of 4)

▪ Use the physical basis of the distribution as a guide

▪ Examples:
— Binomial: number of successes in 𝑛 trials
— Poisson: number of independent events that occur in a
fixed amount of time or space
— Normal: distribution of a process that is the sum of a
number of (smaller) component processes
— Exponential: time between independent events, or a
processing time duration that is memoryless
— Discrete or continuous uniform: models the complete
uncertainty about the distribution (other than its range)
— Empirical: does not follow any theoretical distribution

13
 Selecting the Family of Distributions (3 of 4)

▪ Remember the physical characteristics of the process

— Is the process naturally discrete or continuous valued?
— Is it bounded?
— Is it symmetric, or is it skewed?

▪ No “true” distribution for any stochastic input

process

▪ Goal: obtain a good approximation that captures the

salient properties of the process (e.g., range, mean,
variance, skew, tail behavior)

14
 Selecting the Family of Distributions (4 of 4)

How to check if the chosen distribution is a good fit?

▪ Compare the shape of the pmf/pdf of the
distribution with the histogram:
— Problem: Difficult to visually compare probability curves
— Solution: Use Quantile-Quantile plots

Example: Oil change time at MinitLube

• Histogram suggests “exponential” dist.
• How well does Exponential fit the data?

15
 Quantile-Quantile Plots (1 of 8)

▪ Q-Q plot is a useful tool for evaluating distribution fit

— It is easy to visually inspect since we look for a straight line

▪ If 𝑋 is a random variable with CDF 𝐹(𝑥), then the 𝑞-

quantile of 𝑋 is given by 𝑥𝑞 such that:

𝐹 𝑥𝑞 = ℙ 𝑋 ≤ 𝑥𝑞 = 𝑞, 0<𝑞<1

▪ When 𝐹(𝑥) has an inverse, then 𝑥𝑞 = 𝐹 −1 (𝑞)

16
 Quantile-Quantile Plots (2 of 8)

▪ 𝑥𝑞𝑆 : empirical 𝑞-quantile from the sample

▪ 𝑥𝑞𝑀 : theoretical 𝑞-quantile from the model
▪ Q-Q plot: plot 𝑥𝑞𝑆 versus 𝑥𝑞𝑀 as a scatterplot of points

17
 Quantile-Quantile Plots (3 of 8)

▪ 𝑋: a random variable with CDF 𝐹(𝑥)

▪ {𝑋𝑖 , 𝑖 = 1, … , 𝑛}: a sample of 𝑋 consisting of 𝑛 observations
▪ Define 𝐹𝑛 (𝑥): empirical CDF of 𝑋,

number of 𝑋𝑖′ 𝑠 ≤ 𝑥
𝐹𝑛 𝑥 =
𝑛

▪ {𝑋 𝑗 , 𝑗 = 1, … , 𝑛}: observations ordered from smallest to largest

𝑋(1) ≤ 𝑋(2) ≤ ⋯ ≤ 𝑋(𝑛)

▪ It follows that
𝑗
𝐹𝑛 𝑥 =
𝑛
where 𝑗 is the rank or order of 𝑥, i.e., 𝑥 is the 𝑗-th value among 𝑋𝑖 ’s.
18
 Quantile-Quantile Plots (4 of 8)

▪ Problem:
— For finite value 𝑥 = 𝑋(𝑛) , we have 𝐹𝑛−1 1 = 𝑋(𝑛)
— But from the model we generally have: 𝐹 −1 1 = ∞
— How to resolve this mismatch?

▪ Solution: slightly modify the empirical distribution

0.5 𝑗 − 0.5
𝐹෨𝑛 𝑋 𝑗 = 𝐹𝑛 𝑋 𝑗 − =
𝑛 𝑛
▪ Therefore,
𝑗 − 0.5
𝐹෨𝑛−1 = 𝑋(𝑗)
𝑛
▪ and, thus,
𝑗−0.5
empirical −quantile of X = 𝑋(𝑗)
𝑛

19
 Quantile-Quantile Plots (5 of 8)

▪ 𝐹(𝑥): the CDF fitted to the observed data, i.e., the model

▪ Q-Q plot: plotting empirical quantiles vs. model quantiles

𝑗−0.5
— -quantiles for 𝑗 = 1, … , 𝑛
𝑛
▪ Empirical quantile = 𝑋(𝑗)
𝑗−0.5
▪ Model quantile = 𝐹 −1
𝑛

▪ Q-Q plot features:

— Approximately a straight line if 𝐹 is a member of an appropriate
family of distributions
— The line has slope 1 if 𝐹 is a member of an appropriate family of
distributions with appropriate parameter values

20
 Quantile-Quantile Plots (6 of 8)

▪ Example: Check whether the door installation times follow a normal

distribution.
— The observations are ordered from smallest to largest:

𝑗 value 𝑗 value 𝑗 value 𝑗 value

1 97.12 6 99.34 11 100.11 16 100.85
2 98.28 7 99.50 12 100.11 17 101.21
3 98.54 8 99.51 13 100.25 18 101.30
4 98.84 9 99.60 14 100.47 19 101.47
5 98.97 10 99.77 15 100.69 20 102.77

𝑗−0.5
— 𝑋(𝑗) ’s are plotted versus 𝐹 −1 where 𝐹 is the normal CDF with
𝑛
sample mean (99.93 sec) and sample STD (1.29 sec)

21
 Quantile-Quantile Plots (7 of 8)

▪ Example (continued):
Check whether the door installation
times follow a normal distribution.

Straight line,
supporting the
hypothesis of a
normal distribution

Superimposed density
function of the Normal
distribution scaled by the
number of observation,
that is 20 × 𝑓(𝑥)

22
 Quantile-Quantile Plots (8 of 8)

▪ Consider the following while evaluating the linearity

of a Q-Q plot:
— The observed values never fall exactly on a straight line
— Variation of the extremes is higher than the middle.
— Linearity of the points in the middle of the plot (the main
body of the distribution) is more important.

23
 Parameter Estimation (1 of 4)

Next step after selecting a family of distributions.

▪ If observations in a sample of size 𝑛 are

𝑋1 , 𝑋2 , … , 𝑋𝑛 (discrete or continuous), the sample
mean and variance are:
σ𝑛
𝑖=1 𝑋𝑖 σ𝑛 𝑋 − ത
𝑋 2 σ𝑛 𝑋 2
−𝑛 ത
𝑋 2
𝑋ത = , s2 = 𝑖=1 𝑖
= 𝑖=1 𝑖
𝑛 𝑛−1 𝑛−1

24
 Parameter Estimation (2 of 4)

▪ If the data are discrete and have been grouped into a

frequency distribution with 𝑘 distinct values:
σ𝑘
𝑗=1 𝑓𝑗 𝑋𝑗
𝑋ത = ,
𝑛
2
σ𝑘𝑗=1 𝑓𝑗 𝑋𝑗 − 𝑋ത σ𝑘𝑗=1 𝑓𝑗 𝑋𝑗2 − 𝑛 𝑋ത 2
s2 = =
𝑛−1 𝑛−1

where 𝑓𝑗 is the observed frequency of value 𝑋𝑗

25
 Parameter Estimation (3 of 4)

▪ Vehicle Arrival Example: number of vehicles arriving at an intersection

between 7: 00 am and 7: 05 am was monitored for 100 random workdays.
# Arrivals (𝑋𝑗 ) Frequency (𝑓𝑗 )
𝑛 = 100
𝑘 0 12
෍ 𝑓𝑗 𝑋𝑗 = 364 1 10
𝑗=1 2 19
𝑘
෍ 𝑓𝑗 𝑋𝑗2 = 2080 3 17
𝑗=1 4 10
5 8
6 7
— The sample mean and variance are
7 5
364
𝑋ത = = 3.64 8 5
100
2080−100∗ 3.64 2 9 3
𝑠2 = = 7.63 10 3
99
11 1

26
 Parameter Estimation (4 of 4)

▪ The histogram suggests 𝑋 is a Poisson distribution

— However, the sample mean is not equal to sample variance
— Reason: each estimator is a random variable (not perfect)

27
 Goodness-of-Fit Tests (1 of 2)

▪ Conduct hypothesis testing on input data distribution

using well-known statistical tests, such as:
— Chi-square test
— Kolmogorov-Smirnov test

▪ Note: you don’t always get a single unique correct

distributional result for any real application:
— If very little data are available, it is unlikely to reject any
candidate distributions
— If a lot of data are available, it is likely to reject all
candidate distributions

28
 Goodness-of-Fit Tests (2 of 2)

Objective: to determine how well a (theoretical)

statistical model fits a given set of empirical
observations (sample)

▪ Vehicle Arrival Example:

— The histogram suggests 𝑋 might be a Poisson distribution

— Hypothesis:
𝑋 has a Poisson distribution with rate 3.64

— How can we test the hypothesis?

29
 Chi-Square Test (1 of 11)

Intuition:

▪ It establishes whether an observed frequency

distribution differs from a model distribution
— Model distribution refers to the hypothesized distribution with
the estimated parameters
— Can be used for both discrete and continuous random variables
— Valid for large sample sizes

▪ If the difference between the distributions is smaller than

a critical value, the model distribution fits the observed
data well, otherwise, it does not.

30
 Chi-Square Test (2 of 11)

Concepts:

▪ Null hypothesis 𝐻0 :
The observed random variable 𝑋 conforms to the model distribution

▪ Alternative hypothesis 𝐻1 :
The observed random variable 𝑋 does not conform to the model distribution

▪ Test statistic 𝜒 2 :
The measure of the difference between sample data and the model
distribution

▪ Significance level 𝛼:
The probability of rejecting the null hypothesis when the null hypothesis is
true. Common values are 0.05 and 0.01.

31
 Chi-Square Test (3 of 11)

Approach:

▪ Arrange the 𝑛 observations into a set of 𝑘 intervals or cells, where interval

𝑖 is given by 𝑎𝑖−1 , 𝑎𝑖
— Suggestion: set the interval length such that at least 5 observations fall in each
interval

▪ Recommended number of class intervals (𝑘):

Sample Size, n Number of Class Intervals, k
20 Do not use the chi-square test
50 5 to 10
100 10 to 20
> 100 n1/2 to n/5

▪ Caution: Different grouping of data (i.e., 𝑘) can affect the hypothesis

testing result.

32
 Chi-Square Test (4 of 11)

Test Statistic:

▪ 𝑂𝑖 : the number of observations 𝑋𝑗 that fall in interval 𝑖

▪ 𝐸𝑖 : the expected number of observations in interval 𝑖 if taking 𝑛 samples

from the model distribution:

— Continuous model with fitted PDF 𝑓(𝑥):

𝑎𝑖
𝐸𝑖 = 𝑛 ⋅ න 𝑓 𝑥 𝑑𝑥
𝑎𝑖−1

— Discrete model with fitted PMF 𝑝(𝑥):

𝐸𝑖 = 𝑛 ⋅ ෍ 𝑝 𝑥
𝑎𝑖−1 ≤𝑥<𝑎𝑖

33
 Chi-Square Test (5 of 11)

Test Statistic:

▪ Test statistic 𝜒 2 is defined as

𝑘
2
2
𝑂𝑖 − 𝐸𝑖
𝜒 =෍
𝐸𝑖
𝑖=1

▪ 𝜒 2 approximately follows the chi-square distribution

with 𝑘 − 𝑠 − 1 degrees of freedom
— 𝑘: the number of intervals
— 𝑠: the number of parameters of the model (i.e., hypothesized distribution)
estimated by the sample statistics
▪ Uniform: 𝑠 = 0
▪ Poisson, Exponential, Bernoulli, Geometric: 𝑠 = 1
▪ Normal, Binomial: 𝑠 = 2

34
 Chi-Square Test (6 of 11)

▪ The distribution is not symmetric

▪ Minimum value is 0
▪ Mean = degrees of freedom
Chi-Square PDF

𝑑𝑓 = 2

𝑑𝑓 = 5

𝑑𝑓 = 10

35
 Chi-Square Test (7 of 11)

Intuition:

▪ 𝜒 2 measures the normalized squared difference between

the frequency distribution of the sample data and
hypothesized model

▪ A large 𝜒 2 provides evidence that the model is not a

good fit for the sample data:
— If the difference is greater than a critical value then reject the
null hypothesis
— Question: what is an appropriate critical value?
— Answer: it is pre-specified by the modeler.

36
 Chi-Square Test (8 of 11)

Critical Value:

2
▪ For significance level 𝛼, the critical value 𝜒𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙 is defined such that:

2 2
ℙ 𝜒𝑘−𝑠−1 ≥ 𝜒𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙 =𝛼

Chi-Square distributed random

variable with 𝑘 − 𝑠 − 1 Chi-square PDF
degrees of freedom.

2 2
▪ 𝜒𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙 = 𝜒𝑘−𝑠−1,1−𝛼
the (1 − 𝛼)-quantile of
chi-square distribution
2
with 𝑘 − 𝑠 − 1 𝜒𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙 Shaded area = 𝛼
degrees of freedom
Do not reject Reject
37
 Chi-Square Test (9 of 11)

▪ We say that the null hypothesis 𝐻0 is rejected at the significance level 𝛼,

if:
2
𝜒 2 > 𝜒𝑘−𝑠−1,1−𝛼

▪ Interpretation:
— The test statistic can be
as large as the critical value
Chi-square PDF
— If the test statistic is greater
than the critical value then,
the null hypothesis is rejected

— If the test statistic is not greater

than the critical value then, 2
the null hypothesis 𝜒𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙 Shaded area = 𝛼
can not be rejected
Do not reject Reject
38
Chi-Square Test (10 of 11)
 Chi-Square Test (11 of 11)

▪ Vehicle Arrival Example (continued):

𝐻0: the random variable is Poisson distributed (with 𝜆 = 3.64).
𝐻1: the random variable is not Poisson distributed.
0 12 2.6
7.87
1 10 9.6
2 19 17.4 0.15 Ei  np( x)
3 17 21.1 0.83
e   x
4 10 19.2 4.41 n
5 8 14.0 2.57 x!
6 7 8.5 0.26
7 5 4.4
8 5 2.0
9 3 0.8 11.63
10 3 0.3 Combined because
>11 1 0.1 of min 𝐸𝑖
100 100.0 27.72

— Degrees of freedom is 𝑘 − 𝑠 − 1 = 7 − 1 − 1 = 5, hence, the

hypothesis is rejected at the 0.05 level of significance:
2
𝜒 2 = 27.72 > 𝜒0.95,5 = 11.1

40
 Kolmogorov-Smirnov Test

▪ Intuition:
— Formalizes the idea behind examining a Q-Q plot
— The test compares the CDF of the hypothesized
distribution with the empirical CDF of the sample
observations based on the maximum distance between
two cumulative distribution functions.

▪ A more powerful test that is particularly useful when:

— Sample sizes are small
— No parameters have been estimated from the data

41
 Selecting Model without Data (1 of 2)

▪ If data is not available, some possible sources to

obtain information about the process are:
— Engineering data: often product or process has performance ratings
provided by the manufacturer or company that specify time or
production standards
— Expert option: people who are experienced with the process or similar
processes, often, they can provide optimistic, pessimistic and most-
likely times, and they may know the variability as well
— Physical or conventional limitations: physical limits on performance,
limits or bounds that narrow the range of the input process
— The nature of the process

▪ The uniform, triangular, and beta distributions are

often used as input models.

42
 Selecting Model without Data (2 of 2)

▪ Example: Production planning simulation.

— Input of sales volume of various products is required, salesperson of
product XYZ says that:
▪ No fewer than 1,000 units and no more than 5,000 units will
be sold.
▪ Given her experience, she believes there is a 90% chance of
selling more than 2,000 units, a 25% chance of selling more
than 3,000 units, and only a 1% chance of selling more than
4,000 units.
— Translating these information into a cumulative probability of being
less than or equal to those goals for simulation input:
i Interval (Sales) Cumulative Frequency, ci
1 1000  x 2000 0.10
2 2000 < x 3000 0.75
3 3000 < x 4000 0.99
4 4000 < x 5000 1.00
43
 Multivariate and Time-Series Models

▪ So far, we have considered:

— Single variate models for independent input parameters

▪ To model correlation among input parameters

— Multivariate models
— Time-series models

44