2 HW Solutions

2.1 Problem 2.1

R2
Find extremals for the functional J = 0 [2x2 (t) + x_ 2 (t)]dt with x(0) = 0, x(2) = 5.
p p
¡ 2t
The Euler-Lagrange equation is @V@x ¡ d @V
dt @ x_ = 0, from which Ä
x ¡ 2x = 0. Hence, x(t) = c1 e 2t
+ c2 e
p p
and ci are such that the B.C. are satis¯ed: c1 + c2 = 0 and c1 e 22 + c2 e¡ 22 = 5

2.2 Problem 2.6

R5
Find the optimal control for J = 12 0 u2 (t)dt subject to x
Ä = u, x(0) = 2, x(0)
_ = 2, x(5) = 0, x(5)
_ = 0.
Rewrite the system in state-space as
μ ¶ μ ¶ μ ¶ μ ¶
0 1 0 2 0
x_ = Ax + Bu; A = ; B= ; x(0) = ; x(5) = ;
0 0 1 2 0

The Hamiltonian is H(x; u; ¸) = 12 u2 +¸> (Ax+Bu). The Euler-Lagrange conditions for an extremal become

u = arg min H = ¡B > ¸

u
@H
¸_ = ¡ = ¡A> ¸
@x
@H
x_ = = Ax + Bu
@¸
Hence, 0 1 0 1
μ ¶ μ ¶μ ¶ μ ¶ 2 μ ¶ 0
x_ A ¡BB > x x B 2 C x B 0 C
= ; (0) = B C
@ ? A; (5) = B C
@ ? A
¸_ 0 ¡A> ¸ ¸ ¸
| {z } ? ?
H

where, H is the Hamiltonian matrix and ? denotes unspeci¯ed terms. This is a TPBVP and its solution
can be obtained through the state transition matrix (STM) that describes solutions of the state-costate
equations. That is, letting w = (x; ¸), we have w_ = Hw and, therefore,

w(t) = ©(t; t0 )w(t0 ); t0 = 0; ©(t; ¿ ) = eH(t¡¿ )

Suppressing the t0 -notation, we compute

0 1
1 5 20:833 ¡12:5
B 0 1 12:5 ¡5 C
©(5) = eH5 =B
@ 0
C
0 1 0 A
0 0 ¡5 1

Next, partition ©(5) according to the state-costate vectors and solve for the unknown initial conditions ¸(0).
We have that x(5) = ©11 (5)x(0) + ©12 (5)¸(0), from which
μ ¶μ ¶ μ ¶
¡1 ¡1 ¡0:096 ¡0:24 2 0:672
¸(0) = ©12 (5)x(5) ¡ ©12 (5)©11 (5)x(0) = ¡ =
¡0:24 ¡0:8 2 2:08

For the numerical computation of the solution, the MATLAB code is

t=[0:.01:5]';
H=[A -B*B';0*B*B' -A'];
p=expm(H*5);p11 = p(1:2,1:2);p12 = p(1:2,3:4);

1
S=ss(A,B,eye(2,2),zeros(2,1));
x0=[2;2;-(p12\p11)*[2;2]]
Sa=ss(-A',B,eye(2,2),zeros(2,1));
L=lsim(Sa,0*t,t,x0(3:4));plot(t,L),pause
u=-L*B; % lsim output: L rows are the costate transpose at each time instant
x=lsim(S,u,t,x0(1:2));
plot(t,x,t,u)

Not surprisingly, (the costate system is the homogeneous solution of a double integrator) the optimal
input is an a±ne function of time (straight line).
Alternatively, one can take advantage of the speci¯c form of equations to ¯nd the explicit solution:
¸1 (t) = c3 , ¸2 (t) = ¡c3 t + c4 , u(t) = ¡¸2 (t), etc., x2 (t) = ¢ ¢ ¢, x1 (t) = ¢ ¢ ¢. Then, the constants ci are found
by solving the four-equation, four-unknown linear system.

2.3 Problem 2.7

Rt
Find the optimal control for J = 0 f [x2 (t) + u2 (t)]dt subject to x_ = ¡x + u, x(0) = 5, x(tf ) = 0, and with
tf free.
The Hamiltonian is H(x; u; ¸) = x2 + u2 + ¸(¡x + u). The Euler-Lagrange conditions for an extremal
become
1
u = arg min H = ¡ ¸
u 2
@H
¸_ = ¡ = ¡2x + ¸
@x
@H
x_ = = ¡x + u
@¸
Hence, μ ¶ μ ¶μ ¶ μ ¶ μ ¶ μ ¶ μ ¶
x_ ¡1 ¡ 12 x x 5 x 0
= ; (0) = ; (tf ) =
¸_ ¡2 1 ¸ ¸ ? ¸ 0
| {z }
H

where, H is the Hamiltonian matrix and ? denotes unspeci¯ed terms.

Notice that, due to the free ¯nal time, the additional boundary condition is H¤ (tf ) = 0 Substituting the
optimal input in the Hamiltonian, H¤ = x2 ¡ ¸x ¡ 14 ¸2 . Evaluating at tf , x(tf ) = 0, so H¤ = ¡ 14 ¸(tf )2 = 0.
Thus, the extremals satisfy the state-costate equations: w = (x; ¸), w_ = Hw and, therefore, w(t) =
©(t; t0 )w(t0 ). The state transition matrix © is always nonsingular and an equivalent expression of the
solution is w(t) = ©¡1 (tf ; t)w(tf ) = ©(t; tf )w(tf ).
The problem now is that w(tf ) = 0 making the entire extremal trajectory equal to zero. The reason for
this is that we can obtain smaller values of the cost J as we allow tf ! 1, hence, the minimum does not
exist. Let us analyze this situation further and gain some insight.
RT
² Consider the ¯xed ¯nal time problem min JT = 0 V (x; u) dt, with x(T ) = 0. It follows that JT is
a decreasing function of T . (Suppose not; then JT 1 < JT 2 for some T 1 < T 2. Since x = 0 is an
equilibrium point, and V (x; u) ¸ 0, we can de¯ne a policy in the T2 interval as the T1-policy for
t < T 1 and zero afterwards. Since x(T 1) = 0, then the rest of the solution is u = 0, x = 0 for which
V (0; 0) = 0, so the corresponding cost is JT0 2 = JT 1 < JT 2 . This contradicts the optimality of the
original solution in the T2-interval.) Hence, unless there are multiple minima, the minimum does not
occur in ¯nite time.
RT
² Alternatively, consider the ¯xed ¯nal time problem min JT = (x> F x)(T ) + 0 x> P x + u> Ru dt with
free ¯nal state. Its solution is u = ¡R¡1 B > P x, where ¡P_ = A> P +P A+Q¡P BR¡1 B > P , P (T ) = F .

2
 The minimum JT = x> (0)P (0)x(0). Letting F = ½I, ½ ! 1 we approximate the problem of the ¯xed
¯nal state x(T ) = 0.
The optimal cost depends both on T and F through the solution of the Riccati. Rewriting the Riccati
as a forward-in-time ODE, we get (using the same symbol P ) P_ = A> P + P A + Q ¡ P BR¡1 B > P ,
P (0) = F , for which the optimal solution is JT = x(0)> P (T )x(0) and u(t) = ¡R¡1 B > P (T ¡ t)x(t).
We can explore the behavior of JT by solving this ODE, for di®erent initial conditions F . In our case,
P_ = ¡2P + 1 ¡ P 2 , P (0) = F , for which the optimal solution is peasily computed in
p SIMULINK. The
steady state value is the positive root of 0 = ¡2P + 1 ¡ P 2 , i.e., 2 ¡ 1. For F > 2 ¡ 1 the solution
decays monotonically to the steady-state value. This implies that for any given F (large enough), the
minimum JT with respect to T will occur at in¯nity.
Thus, we conclude that the optimal control is the limiting solution for the in¯nite interval

tf= 1
p
P = 2¡1
u = ¡P x
p
and min J = x(0)> P x(0) = 25( 2 ¡ 1) = 10:355.

2.4 Problem 2.11

Rt
Find the optimal control for J = 12 (x(tf ) ¡ xf )> F (x(tf ) ¡ xf ) + 12 0 f x> Qx + u> Ru dt subject to
μ ¶ μ ¶ μ ¶ μ ¶
0 1 0 1 1 0
x_ = Ax + Bu; A = ; B= ; x(0) = ; Q= ; R=4
¡2 0 3 2 0 2

for four di®erent cases.

Case 1: tf = 2, x(2) = [4; 6]. Here, the terminal cost (F; xf ) is irrelevant. The Hamiltonian is
H(x; u; ¸) = 12 (x> Qx + u> Ru) + ¸> (Ax + Bu). The Euler-Lagrange conditions for an extremal become

u = arg min H = ¡R¡1 B > ¸

u
@H
¸_ = ¡ = ¡A> ¸ ¡ Qx
@x
@H
x_ = = Ax + Bu
@¸
Hence, letting w = (x; ¸), we have w_ = Hw, where
0 1 0 1
μ ¶ 1 4
A ¡BR¡1 B > B 2 C B 6 C
H= ; w(0) = B C
@ ? A; w(2) = B C
@ ? A
¡Q ¡A>
? ?
Thw solution of this TPBVP can be obtained through the state transition matrix (STM) that describes
solutions of the state-costate equations w(t) = eHt w(0). From this, we compute the initial condition for the
costates as ¸(0) = ©¡1 12 [x(2) ¡ ©11 x(0)] = [4:8958; 2:7143]. The computation is similar to the procedure
listed in Problem 2.6:

t=[0:.01:2]';
Q=[1 0;0 2];R=4;
H=[A -B*inv(R)*B';-Q -A'];
p=expm(H*2);p11 = p(1:2,1:2);p12 = p(1:2,3:4);
S=ss(A,B,eye(2,2),zeros(2,1));

3
x0=[1;2;p12\([4;6]-p11*[1;2])]
% x-lambda are coupled; produce u by solving the entire state-costate eqn.
Sh=ss(H,[B;B],[0*B' -inv(R)*B'],0);
u=lsim(Sh,0*t,t,x0);
x=lsim(S,u,t,x0(1:2)); % verify solution
plot(t,x,t,u)

Case 2: tf free, x(tf ) = [4; 6]. Here, the terminal cost (F; xf ) is again irrelevant. With the same setup
as in Case 1, we now have the additional condition H(tf ) = 0 = 12 x> Qx ¡ 12 ¸> BR¡1 B > ¸ + ¸> Axjt=tf =
44 ¡ 1:125¸22 (tf ) ¡ 8¸2 (tf ) + 6¸1 (tf ).
One procedure to solve this problem is to begin by ¯xing tf , evaluate the optimal solution, check the
above transversality condition at tf , and then adjust tf and repeat. Alternatively, we could also simply
solve the the ¯xed-time optimal control problem JT and minimize JT with respect to T . Performing the
evaluation, we ¯nd that the transversality condition is satsi¯ed at several times: 0.759, 2.098, 5.2495,...
These are points where the necessary conditions for extremals are satis¯ed. For identifying the minimum, we
need to evaluate the cost. Plotting JT as a function of T we indeed ¯nd several local minima and maxima
with the one at tf = 0:759 yielding the least cost, J0:759 = 31:53.
The following function is useful in performing the necessary computations:
function [ER,J]=p211(tf);
t=[0:.01:tf]';
A=[0 1;-2 0];B=[0;3];
xf=[4;6];x0=[1;2];
Q=[1 0;0 2];R=4;
H=[A -B*inv(R)*B';-Q -A'];
p=expm(H*tf);p11 = p(1:2,1:2);p12 = p(1:2,3:4);
xx0=[x0;p12\(xf-p11*x0)];
% x-lambda are coupled; produce u by solving the state-costate eqn.
Sh=ss(H,[B;B],[eye(4,4);0*B' -inv(R)*B'],zeros(5,1));
U=lsim(Sh,0*t,t,xx0);
u=U(:,5);Lf=U(length(U),3:4)';
ER=xf'*Q*xf/2-Lf'*B*inv(R)*B'*Lf/2+Lf'*A*xf;
S=ss(A,B,eye(2,2),zeros(2,1));
x=lsim(S,u,t,x0(1:2));
%plot(t,x,t,u);
J=(norm(x(:,1))^2+norm(x(:,2))^2*2+norm(u)^2*4)/2*.01;
£ ¤>
Case 3: tf = 2, x(tf ) = [?; 6], F = 0. Here, @S@x ¡ ¸ t=tf ±xf = 0 implying that ¸1 (tf ) = 0. Again, we
express the state-costate equations as w_ = Hw, where
0 1 0 1
μ ¶ 1 ?
A ¡BR¡1 B > B 2 C B 6 C
H= ; w(0) = B C B C
@ ? A ; w(2) = @ 0 A
¡Q ¡A>
? ?

The rest of the problem is completely analogous to Case 1, except that a di®erent partition of the state
transition matrix (rows 2-3) is needed to compute the unknown initial conditions for ¸(0).
Case 4: tf free, x(tf ) on [4; ¡5tf + 15], F = diag(3; 5), xf = [4; 6]. Now, the boundary conditions for
the state-costate equations w_ = Hw become
0 1 0 1
1 4
B 2 C B ¡5tf + 15 C
w(0) = B C
@ ? A ; w(2) = @
B C
? A
? ?

4
 £ @S ¤>
and 0 = H(tf ) + @x ¡ ¸ t=t [0; ¡5] = H(tf ) + [F (x(tf ) ¡ xf ) ¡ ¸]>
t=tf [0; ¡5] = 0, yielding
f

H(tf ) + 5(¡5tf + 15 ¡ 6) ¡ ¸2 (tf )(¡5) = 0

This is a straightforward extension of Case 2, with the di®erence that the ¯nal states are functions of tf
instead of ¯xed constants. That presents no additional di±culty in the procedure outlined in Case 2.