2 The Linear Quadratic Regulator (LQR)

Problem:
Compute a state feedback controller

u(t) = Kx(t)

that stabilizes the closed loop system and minimizes

Z ∞
J := x(t)T Qx(t) + u(t)T Ru(t) dt
0

where x and u are the state and control of the LTI system

ẋ(t) = Ax(t) + Bu(t), x(0) = x0.

Assumptions:
a) Q  0, R ≻ 0;
b) (A, B) stabilizable;

A first step toward a solution:

The closed loop cost is
Z ∞
J= x(t)T (Q + K T RK)x(t) dt
0

and the closed loop system is

ẋ = (A + BK)x, x(0) = x0 .

But for a given K and x0

x(t) = e(A+BK)t x0.
Hence
Z ∞
T
J= xT0 e(A+BK) t (Q + K T RK)e(A+BK)tx0 dt
0 
Z ∞ 
T
= xT0 e(A+BK) t (Q + K T RK)e(A+BK)t dt x0.
0

MAE 280 B 23 Maurı́cio de Oliveira

This means that J can be computed as

J = xT0 Xx0

where X is the solution to the Lyapunov equation

(A + BK)T X + X(A + BK) + Q + K T RK = 0.

Before proceeding we need to learn how to solve the above Lyapunov equation
in X and K. This is not always possible. In this case, because R ≻ 0, we can
complete the squares, rewriting the above equation in the form

AT X + XA − XBR−1B T X + Q + (XBR−1 + K T )R(R−1B T X + K) = 0.

Note that K is confined to the term

(XBR−1 + K T )R(R−1B T X + K)  0

and that for

K = −R−1 B T X.

we have
Q + (XBR−1 + K T )R(R−1B T X + K) = Q.
This reduces the above equation to

AT X + XA − XBR−1 B T X + Q = 0.

This is an Algebraic Riccati Equation (ARE) in X.

As we learn more about AREs we shall prove that the above choice of K and X
is so that
a) A + BK is Hurwitz (asymptotically stable);
b) X is “minimum” in a certain sense;
c) The associated J is minimized.

MAE 280 B 24 Maurı́cio de Oliveira

2.1 Comparison Lemma
If S  0 and
Q2  Q1  0
then X1 and X2 , solutions to the Riccati equations

AT X1 + X1 A − X1 SX1 + Q1 = 0,
AT X2 + X2 A − X2 SX2 + Q2 = 0,

are such that

X2  X1
if A − SX2 is asymptotically stable.

Proof: Note that

AT X1 + X1 A − X1SX1 + Q1
= (A − SX2)T X1 + X1 (A − SX2 ) + X2 SX2 + Q1 − (X1 − X2 )S(X1 − X2 ),

and

AT X2 + X2A − X2 SX2 + Q2 = (A − SX2 )T X2 + X2 (A − SX2 ) + X2 SX2 + Q2

Now subtract the above equations to obtain the Lyapunov equation

(A − SX2)T X̄ + X̄(A − SX2) + Q̄ = 0

where

X̄ := X2 − X1 , Q̄ := (Q2 − Q1 ) + (X1 − X2 )S(X1 − X2 )  0.

Therefore, if A − SX2 is Hurwitz we conclude that X̄ = X2 − X1  0, that is

X2  X1 .

MAE 280 B 25 Maurı́cio de Oliveira

We can now use the comparison lemma to compare the two AREs

AT X2 + X2 A − X2BR−1 B T X2 + Q2 = 0

and
AT X1 + X1 A − X1BR−1 B T X1 + Q1 = 0
where

S = BR−1B T  0,

and

Q1 = Q, Q2 = Q + (X2BR−1 + K T )R(R−1B T X2 + K).

Note that for any X2 and stabilizing K that

Q2 = Q + (X2BR−1 + K T )R(R−1B T X2 + K)  Q = Q1

because R ≻ 0. Therefore, for any choice of

K 6= −R−1B T X1

we shall have

X2  X1 .

This proves that X1 is “minimum”. Of course this also implies that

J2 = xT0 X2 x0 ≥ xT0 X1 x0 = J1

so that J is also being minimized.

MAE 280 B 26 Maurı́cio de Oliveira

2.2 More on AREs
Warning: In this section we consider Riccati equations of the form
AT X + XA + XZX + Q = 0

Lemma 1: Consider the Hamiltonian matrix

 
A Z
H := .
−Q −AT

where A, Z = Z T and Q = QT ∈ Rn×n .

1. λ is an eigenvalue of H if and only if −λ is an eigenvalue of H.
2. If H has no eigenvalues on the imaginary axis then there exists a matrix
W ∈ Rn×n such that
HV1 = V1 W (2)
where W is Hurwitz.
Proof:
Item 1. H has eigenvalues pairs which are symmetric w.r.t the imaginary axis
because
 
0 −I
J −1HJ = −JHJ = −H T , J := , J −1 = −J
I 0
Item 2. Let HJ be the Jordan form of matrix H so that
HV = V HJ
where V ∈ R2n×2n is a matrix whose columns are the (generalized) eigenvectors
of H. Since the eigenvalues of H are symmetric with respect to the imaginary
axis and there are no eigenvalues on the imaginary axis, there exists at least two
distinct Jordan blocks
 
    HJ− 0
H V1 V2 = V1 V2
0 HJ+
where all n eigenvalues of HJ− have negative real part, i.e., HJ− is Hurwitz. The
first columns of the above equation are in the form (2) with W = HJ− Hurwitz.

MAE 280 B 27 Maurı́cio de Oliveira

Lemma 2: Consider the Algebraic Riccati Equation (ARE)

AT X + XA + XZX + Q = 0

where A, Z = Z T and Q = QT ∈ Rn×n and the associated Hamiltonian matrix

 
A Z
H := .
−Q −AT
which is assumed to have no eigenvalue on the imaginary axis.
1. Let  
X1
V1 = ∈ C2n×n
X2
be (generalized) eigenvectors of H associated with all n eigenvalues with
negative real part. If X1 is nonsingular then X = X2 X1−1 solves the ARE.
2. The solution obtained in item 1. is
(a) real,
(b) symmetric,
(c) unique stabilizing (A + ZX is Hurwitz).
3. If Z  0 (or Z  0) then X1 is invertible if and only if (A, Z) is
stabilizable.

Proof:
Item 1. From Item 2. of Lemma 1 there exists a Hurwitz matrix W such that

HV1 = V1 W

Then, multiplying the above by X1−1 on the right and by X −I on the left
 

we get    
I  I
X1 W X1−1 = 0
  
X −I H = X −I
X X
Note that
    
I  A Z I
= AT X + XA + XZX + Q
  
X −I H = X −I T
X −Q −A X

MAE 280 B 28 Maurı́cio de Oliveira

Item 2. (a) Since H is real, the columns of V1 can be chosen complex
conjugates in pairs so that
     
X̄1 X1 X1 P
V̄1 = = P =
X̄2 X2 X2 P
where P is some permutation matrix and

X̄ = X̄2 X̄1−1 = X2 P P −1 X1−1 = X2 X1−1 = X

that is X is real.

Item 2. (b) X is symmetric if

X = X2 X1−1 = (X1−1)∗X2∗ = X ∗

or in other words

T = X2∗ X1 − X1∗ X2 = 0.

Now note that

  
0 −I X1
T = V1∗JV1 = X1∗ X2∗
 
I 0 X2

Since JHJ = H T , H and J satisfy the Lyapunov equation

JH + H T J = 0

Multiplying the above equation by V1∗ on the left and V1 on the right and
using (2)

0 = V1∗ JHV1 + V1∗ H T JV1

= V1∗ JV1W + W ∗ V1∗ JV1
= T W + W ∗T.

Because W is Hurwitz T = 0.

MAE 280 B 29 Maurı́cio de Oliveira

Item 2. (c) Multiply (2) by X1−1 on the right and by I 0 on the left to obtain
 

    
I  A Z I
= A + ZX = X1 W X1−1 .
  
I 0 H = I 0 T
X −Q −A X
Therefore, A + ZX is stable because it is similar to a stable matrix. For
uniqueness assume X̃ is also a stabilizing solution to the ARE. Therefore
subtracting the two AREs

0 = (AT X + XA + XZX + Q) − (AT X̃ + X̃A + X̃Z X̃ + Q)

= AT (X − X̃) + (X − X̃)A + XZX − X̃Z X̃
= AT (X − X̃) + (X − X̃)A + XZX − X̃Z X̃ − XZ X̃ + XZ X̃
= AT (X − X̃) + (X − X̃)A + XZ(X − X̃) + (X − X̃)Z X̃
= (A + ZX)T (X − X̃) + (X − X̃)(A + Z X̃)

The last equation can be seen as a Sylvester equation in (X − X̃) and since
A + ZX and A + Z X̃ are both Hurwitz λi (A + ZX) + λj (A + Z X̃) < 0 so
that it admits only the trivial solution, that is, X − X̃ = 0.

Item 3. To prove sufficiency note that if X1 is invertible then A + ZX is stable

such that (A, Z) is stabilizable.
The proof of necessity is more complicated. Assume that Z  0 (or Z  0),
(A, Z) is stabilizable and that X 1 is singular, such that there exists x 6= 0 such
that X1x = 0. Multiply (2) by I 0 on the left to obtain
 
  X1
I 0 H = AX1 + ZX2 = X1 W
X2
Multiply on the left by x∗X2∗ and on the right by x

x∗X2∗ AX1 x + x∗X2∗ ZX2 x = x∗X2∗ X1 W x = x∗X1∗ X2 W x

and use the fact that X1 x = 0 to obtain

x∗X2∗ ZX2 x = 0

which implies ZX2x = 0 because Z  0 (or Z  0). Note that this also implies

X1 W x = 0.

MAE 280 B 30 Maurı́cio de Oliveira

Auxiliary lemma: Assume W is Hurwitz. There exists x 6= 0 such that
X1 x = X1 W x = 0 if and only if there exists x̃ 6= 0 such that

X1x̃ = 0, W x̃ = λ̃x̃, λ̃ + λ̃∗ < 0.

Proof (Auxiliary lemma): Sufficiency is immediate since

X1 W x̃ = λ̃X1 x̃ = 0.

Necessity follows by contradiction. If X1 is singular and

6 ∃x̃ : X1 x̃ = 0, W x̃ = λ̃x̃, λ̃ + λ̃∗ < 0

then

X1 W x̃ = λ̃X1 x̃ 6= 0

for any eigenvalue/eigenvector pair (λ̃, x̃). Because W is nonsingular this must
be true for n linearly independent vectors, which implies that X1 is not singular.
   (Auxiliary lemma)
Now multiply (2) by 0 I on the left to obtain
 
X1
= −QX1 − A∗X2 = X2 W
 
0 I H
X2
and multiply by x̃ such that X1 x̃ = 0 on the right

0 = (A∗X2 + X2 W )x̃
= (A∗ − λI)X2x̃, λ = −λ̃, λ + λ∗ > 0.

Since ZX2x̃ = 0, this implies that (A, Z) is not stabilizable, which is a

contradiction.

MAE 280 B 31 Maurı́cio de Oliveira

We now use Lemma 2 to prove that A + BK is Hurwitz. It amounts to apply
item 3 of Lemma 2 to the ARE

AT X + XA − XBR−1 B T X + Q = 0.

For that notice that

Z = −BR−1 B T  0

because R ≻ 0 then BR−1B T  0. Therefore, if (A, Z) = (A, −BR−1B T ) is

stabilizable then X1 in Lemma 2 should be invertible and the solution X should
be unique, symmetric and stabilizing.
With that in mind suppose that (A, B) is stabilizable but that (A, −BR−1B T )
is not, so that there exists z 6= 0 such that

z ∗ A = λz ∗ , z ∗ BR−1B T = 0, λ + λ∗ ≥ 0.

Therefore

z ∗ BR−1 B T z = 0.

Because R−1 ≻ 0 this can only be true if z ∗ B = 0, which contradicts the

hypothesis that (A, B) is stabilizable, proving that (A, −BR−1B T ) is
stabilizable and

A + BK = A + B(−R−1B T X) = A − BR−1B T X = A + ZX

is Hurwitz.

Warning: The optimal control gain K is independent from the initial

condition! However, the optimal cost is not!

MAE 280 B 32 Maurı́cio de Oliveira