Dynamic programming

• In water resources planning, dynamism of the system is a major concern that must be taken into
account in modeling and optimization.
• In many actual field conditions, functions may not be so continuous, or so conveniently concave
for maximization or convex for minimization, making calculus-based methods for their solution
difficult.
• A possible solution method for constrained optimization problems containing continuous
and/or discontinuous functions of any shape is called Dynamic Programming (DP).
• DP is an approach that divides the original optimization problem, with all of its variables, into a
set of smaller optimization problems, each of which needs to be solved before the overall
optimum solution to the original problem can be identified.
• Each discrete dynamic programming problem can be represented by a network of nodes and
links.
• DP helps us find the best way to get to any node in that network.
• The nodes represent possible discrete states that can exist and the links represent the decisions
one could make to get from one state to another.
• Thus, DP models involve states and decisions.
• Let t denote time, where t = 0, 1, 2, …, T.
• Let Xt be the state of the system at time period t, and let Ut denote the decision (or control) at the
same time period.
• It is assumed that the state changes according to the state-transition relation:

• It is assumed that, at each time period, an objective function is given that is real valued, and we
are looking for a sequence of decisions such that an overall (total) objective is optimal.

Objective function

Constraint

The objective functions and constraint equations can take any form
• Assume that Q unit of water is to be shared among three users.
• Let’s assume that the benefit each firm obtains (V) as a function of water it uses (X) is known.

• What is needed is to maximize the total benefit:

• This means that:

• This can be reformulated as:
• Lets analyze user number 3:
• In order to maximize the benefit at this stage:

• User number 2:
• Let the state at this stage be S2.
• This state should maximize the benefits of user number 2 and 3.

• If X2 amount of S2 is given to user number 2, S2-X2 is given to user number 3.

• In order to maximize the total benefit of the water given to the three users:
These can be generalized as:

Examples:
1. The shortest path
• To find a shortest path in a multi-stage graph

• Apply the greedy method :

the shortest path from S to T :
1+2+5=8
The shortest path in multistage graphs

4
A D
1 18
11 9
• e.g.
2 5 13
S B E T
16 2

5
C 2
F

• The greedy method can not be applied to this case: (S, A, D, T) 1+4+18 = 23.
• The real shortest path is:
(S, C, F, T) 5+2+2 = 9.
Dynamic programming approach

• Dynamic programming approach (forward approach):

A
4
D 1 A
1 d(A, T)
18
11 9

2 d(B, T)
S
2
B
5
E
13
T S B T
16 2

5 d(C, T)
5
C 2
F C

• d(S, T) = min{1+d(A, T), 2+d(B, T), 5+d(C, T)}

4
A D
 d(A,T) = min{4+d(D,T), 11+d(E,T)} d(D, T)

= min{4+18, 11+13} = 22. 11 T

E d(E, T)
• d(B, T) = min{9+d(D, T), 5+d(E, T), 16+d(F, T)}
= min{9+18, 5+13, 16+2} = 18.
4
A D
9 D
1 18 d(D, T)
11 9

5 13 5 d(E, T)
S
2
B E T B E T
16 2

5 d(F, T)
16
C 2
F F

• d(C, T) = min{ 2+d(F, T) } = 2+2 = 4

• d(S, T) = min{1+d(A, T), 2+d(B, T), 5+d(C, T)}
= min{1+22, 2+18, 5+4} = 9.
• The above way of reasoning is called backward reasoning.
Backward approach (forward reasoning)
4
A D
1 18
11 9

2 5 13
S B E T
16 2

• d(S, A) = 1 C 2
F

d(S, B) = 2
d(S, C) = 5
• d(S,D)=min{d(S,A)+d(A,D), d(S,B)+d(B,D)}
= min{ 1+4, 2+9 } = 5
d(S,E)=min{d(S,A)+d(A,E), d(S,B)+d(B,E)}
= min{ 1+11, 2+5 } = 7
d(S,F)=min{d(S,B)+d(B,F), d(S,C)+d(C,F)}
= min{ 2+16, 5+2 } = 7
• d(S,T) = min{d(S, D)+d(D, T), d(S,E)+ d(E,T), d(S, F)+d(F, T)}
= min{ 5+18, 7+13, 7+2 }
=9

4
A D
1 18
11 9

2 5 13
S B E T
16 2

5
C 2
F
Example 2. Determine the pipe network that can be used to convey water from Point A to either
Points B1, B2 or B3 with minimum cost.
Therefore, the pipe network with the lowest cost is:

Assignment
Solve the question using the backward and forward
reasoning methods
Example 2.
5 units of water is to be shared among three users. The benefits the users obtain using various amounts
of water is given below:

Let’s solve the problem backward

Assignment-3
Eight units of water is to be shared among three users in such a way that total benefit is maximized.
Use the following table depicting the relationship between water use and benefit developed for the
three users and determine the optimum amounts of water to be given to the three users that maximize
the total benefit.

What would be the solution if the available water were seven units?
Multi-objective optimization

• Optimization problems can, generally, be either single-objective or multi-objective.

• The main concern of single-objective optimization is to define the minimum or maximum
value of an objective function, depending on the goal.
• The single-objective is just measuring the goal of operation of a single-purpose reservoir, or it
may weight different objectives using a fixed set of weights for operation of a multi-purpose
reservoir.
• It can provide operators with direct decisions for reservoir operation.

• In most cases, however, operation objectives have trade-offs, and hence single-objective
optimization cannot provide a unique optimum solution.
• In such situations, improvement of some objectives cannot be achieved without the sacrifice of
others.
• Multi-objective optimization refers to problems with several objectives that are expected be
fulfilled simultaneously.
• The objectives are, however, often in conflict with each other and measured by different units.
• An operation decision based on the solution of a multi-objective optimization problem requires
that the operator expresses his/her preference by choosing the most suitable solution in the set
of optimal solutions.
• In general, a multi-objective reservoir operation problem can be formulated as follows:
Min/max F(X) = [F1(X), F2(X)…, FN(X)]
Subject to
qi(X) 0 i = 1, 2, …, m
Where Fj(X), j=1,…, N are the objective functions, X is a vector of decision variables and qi(X)
are constraints that define the feasible solutions.

Difficulties related to Multi-objective optimization:

• Expressing objectives with one unit (Sometimes it is not possible to find a way to link
competing objectives; the differences are qualitative and the relative importance of these
objectives can’t be numerically quantified).
• The relative importance (weight) of the objectives is related to political decisions.
Techniques to solve multiple objective function optimization problems
• Can be classified under two groups:
Aggregation approaches: Weighting approach and Constraint approach
Pareto domination approach
• In aggregation approaches, priorities of the objectives are established.
• This means that, an objective function value is assigned to evaluate the dominance of solution
sets, hence transforming the multi-objective optimization problem into a single-objective
optimization.

Weighting approach
• This method transforms the various objectives into a single scalar objective function.
• The obtained optimal solution is based on the weights assigned to the objectives.
• Mathematically the objective function is given as follows:
Constraint approach

• The main idea of this method is to choose one objective as the dominating objective and treat
the rest of the objectives as constraints.

• In this method, the search is moving towards the direction of optimizing the chosen objective,
while satisfying the rest of the objectives.

Pareto domination
• No preference information is considered or is available before the search of an optimal solution
• It is not based on a single comparative value but on whether one solution is dominated by
others
• The trade-off among the different objectives is presented by the set of Pareto optimal
solutions.
What does this mean? Lets consider the following example:

• Suppose you need to fly on a long trip:

Should you choose the cheapest ticket (more connections) or shortest flying time (more
expensive)?

• It is impossible to put a value on time, so these two objectives can’t be linked.

• Also, the relative importance will vary.

• There may be a business emergency you need to go fix quickly.
• Or, maybe you are on a very tight budget.

• Even though we may not be able to assign numerical relative importance to the multiple
objectives, we can still classify some possible solutions as better than others.

• Suppose in our airplane-trip example from earlier, we find the following tickets:
 Ticket Travel Time Ticket Price
(hrs) ($)
A 10 1700
B 9 2000
C 8 1800
D 7.5 2300
E 6 2200

• If we compare tickets A & B, we can’t say that either is superior without knowing the relative
importance of Travel Time vs. Price.
• However, comparing tickets B & C shows that C is better than B in both objectives, so we can
say that C “dominates” B.
• So, as long as C is a feasible option, there is no reason we would choose B.
• Comparison also shows that D is dominated by E.
• Options A C and E have a trade-off associated with Time vs. Price, so none is clearly superior
to the others.
• These are called the “non-dominated” set of solutions become none of the solutions are dominated.

Usually, solutions of this type form a typical shape, shown in the chart below:
 • Solutions that lie along the line are non-dominated solutions while those that lie inside the
line are dominated because there is always another solution on the line that has at least one
objective that is better.
• The line is called the Pareto front and solutions on it are called Pareto-optimal.
• All Pareto-optimal solutions are non-dominated.
• Thus, it is important in multi-objective optimization to find the solutions as close as possible to
the Pareto front & as far along it as possible.

Graphical examples of feasible region with objectives f1 & f2 where both f1 & f2 are minimized:
How to determine the best option:
• Pareto feasible region and Pareto front are determined
• Co-benefit curves are determined (by decision makers)
• The line tangent to the Pareto front is tangent to the Co-benefit curve at the same point.
• The slope of the tangent line is equal to swap ratio.
• The optimal solution maximizes

• 1 and 2 are weights of V1 and V2 , respectively.

* *
W W

• The best option is the best compromise among the objectives (optimal solution).

Examples:
Water stored in a reservoir having a capacity of 6 million cubic meters is being used for irrigation
and water quality control purposes. The goal is to use a maximum of 4 million meter cube water
for irrigation. The objective functions given in terms of water use for irrigation (X1 in million meter
cubes) and quality control (X2 million meter cubes) are given as follows:
a. Formulate the problem using the weighting and constraint methods

Weighting method

Constraint method

OR
b. Pairs of feasible options in terms of X1 and X2

Values of V1 and V2 at points A, B and C

Pairs of feasible options in terms of
V1 and V2
c. Swap ratios

Along

Along
d. Optimum solution
This depends on the selected values of weights W1 and W2.
Assignment 4
In a certain water resource project having two objectives, the objective functions (V1 and V2) in
terms of water amounts (X1 and X2) are given as follows:
V1 = x1 – 3x2
V2 = -4x1 + x2

Contraints :
-x1 + x2 ≤ 3.5
x1 + x2 ≤ 5.5
2x1 + x2 ≤ 9
x1 ≤ 4
A. Using the weighting and constraints methods, formulate the multi-objective model
B. Draw feasibility region
C. Determine swap ratios of the objective functions
D. Determine optimum solutions ba taking into consideration various swap ratios