Chapter 3

Design of Time Optimal Controllers

 Contents
• Application of the Minimum Principle for
design of time optimal control systems;
• Design of a time optimal feedback control
system
– The Maximum Principle
• State feedback control systems
– HJB theory,
Application of the maximum principle and DP

• The maximum principle is used to find a

bounded piecewise continuous control signal
u(t) over the time interval [t1 t2] such that
the system

is taken from initial state x(to) to final state

x(tf) and a performance index

is minimized
Application of the maximum principle and DP

• The Hamiltonian is then

H ( x, u,  , t )  L( x, x , t )  T f ( x, u , t )
• Hence for the control to be optimal, the
necessary condition is
H
x 
 1
 H

x
• At the optimal controller, the Hamiltonian has
the an absolute minimum
H ( x*, u*,  *, t )  H ( x*, u , *, t ) 2
 Application of the maximum …
• Furthermore, the Hamiltonian fulfils the
boundary condition at the end as follows
3

• Where p=
• The other necessary condition for the control
to cause the Hamiltonian to be minimum is
H 4
0
u
 Application of maximum principle …

• The sufficient condition for u to make the

Hamiltonian minimum is given by

• Where p*(t)=*(t)
• The above conditions work when the control
signal is both bounded and unbounded
Application of maximum principle
• The maximum principle can be applied to two
classes of time optimal problems
– Time optimal problems with fixed final state
• Fixed final time
• Free final time
– Time optimal problems with free final state
• Fixed final time
• Free final time
 Maximum principle for system with fixed
final state
• The formulation becomes find u for a system
given by

• So that the cost function J is minimized

 Maximum principle
• The two sub problems under this are:
– When tb=fixed and K(tb)=0
– When tb=free
• For both cases the steps to be followed are:
– Formulate the problem
– Check for existence of solution
– Formulate the necessary condition of Pontragian
– Solve Hamiltonian minimization and select controller
– Solve the boundary problem
– If possible convert the resulting open loop to closed loop
 Maximum principle for free final state
• Similarly we have two sub problems here
– Fixed final time
– Unknown final time
• Find a control signal which minimizes the
dynamic system

• With cost function being

Maximum principle with Free final state …

• The additional condition which is required

here is

• If the problem is again free final time problem,

additional condition is
 Example
• For the dynamic system given by

• Find a control signal and state trajectory x(t)

that minimize the performance index

• If both x1(tf)=0 and x2(tf)=0 while x1(0)=10 and

x2(0)=0
• If x1(tf)=0 and x2(tf)=unspecified with initial
condition as above
 Examples
• Case 1- applying the maximum principle the
state and control signals become

• Case 2- with new final condition, the

maximum principle gives
 Examples 2
• Rectilinear motion of a vehicle
– Consider the dynamics of the system to be given
by
– The control signal is u, which is the acceleration
has limits given by -<u<
– Initially the system has velocity xo and it is
required to find the optimal control signal which is
used to stop it in minimum time
 Examples
• The performance index is

• And hence the Hamiltonian is

• In order to minimize the Hamiltonian

• This gives a control law (bang-bang)

 Examples

• Applying the necessary condition for minimum

(maximum principle )

• Using the boundary condition, x(0)=to and

x(tf)=0, the solution becomes
 Examples
• Since u(t) is constant, we have

• Hence the optimal control signal is

• And the minimum time is given by

 Maximum principle for LQ regulator

• The LQ problem is a free final state problem

but the cost function is different
• Problem formulation
– Find a control signal that drives a linear system of
the form

– with initial state x(ta)=xa and x(tb) free

– Which minimizes the cost function
 Maximum principle for LQR
• The following points should be noted
– Final time is fixed
– The system is linear and cost is quadratic which
gives a linear controller as a function of the states,
i.e. u(t)=-G(t)x(t) state feedback control
• The Hamiltonian will be
 Maximum principle for LQR
• Necessary condition gives

• Minimizing the Hamiltonian

 Maximum principle for the LQR problem

• Using the control law into the differential

equations

• The two differential equations are

homogenous and the last equation is linear
• Hence assume
 Maximum principle for LQR
• Differentiating the last equation and
substituting in to the differential

• Since x is not zero, the term in the bracket

must be zero
• Hence

• With K(t) being solution of

 Maximum principle for LQR
• The boundary condition for the above solution
is

• Cases 1: when system is time invariant

• Case 2: when the matrices Q and R are
constant
• Case 3: when time horizon is infinite , i.e.
tb=infinite
 LQR example
• For the system given by the following state
space model, design an LQR controller
0 1 0 
A  B 
  1  2 1

• Assume that final time is 2sec and

1 0  2 1 
F   Q  R 1
0 1  1 2 
 Example 2
• Inverted pendulum control
 System modeling
• Free body diagram
 System model
• To change it to Linear model, assume the
deviation of the inverted pendulum to be
small, cos()=cos(+)=-1 and sin()= 
 Matlab design of LQR controller
• Substituting the system parameters
– M=0.5 m=0.2 b=0.1 l=0.3 J=0.006

– Enter the A B and C matrices

– Check for controllability
– Assume R=1 and Q=C’*C and enter them
– Design LQR controller k=lqr(A,B,Q,R);
Maximum principle for partially constrained
final state
• Here the final state is not fixed or left free but
the final state is given to fulfill some condition
or to lie with in a given area
• This is given us

• The time tb, here also may be fixed or free

• The Hamiltonian has the same form as free
state case
 Maximum principle for partially …
• The necessary condition has a new criteria
given by

• The other solutions are the same

 Maximum principle for systems with state
constraints
• The problems above were cases where the
state has to fulfill conditions at the beginning
and/ end of the trajectory
• The more advanced case is when x(t) is
constrained over the whole path from initial to
final
• The procedure for the solution is the same
with some differences
 Maximum principle for state constraint

• Problem : find control signal in [ta tb], tb being

free or fixed such that the performance index J
is minimum and constraints are fulfilled
– Constraints

– Performance index
 Optimal state feedback control
• The maximum principle usually results an
optimal control in open loop
• Such controller is not much useful unless
changed to a closed loop form
• State feedback control is a control law which
results in closed loop control
• A method which can be used to change the
open loop control is by using the Hamilton-
Jacobi-Bellman principle
 Bellman’s optimality principle (DP for
continuous case)
• Divide the time interval [ta tb] into two such
that the system is transferred first from [ta t]
and then from [t tb]
• Formulate the optimal control problem over
the two sections
• The principle of optimality states that
– Optimal solution of the first part coincide with the
optimal solution of the second part vise versa
 Bellman’s optimality principle
• According the above principle the cost
functional can be rewritten as

– Which is the optimal cost to go function

• The first part has also a modified cost function
 HJB theory
• Consider a system with

• And performance

• With the initial state known and final state being

arbitrary
• We want the states to be constrained with in a
given state space
• The solution of this type of problem is stated as
follows
 HJB theory
• The optimal controller which moves the
system from its initial position to an unknown
final state while minimizing the cost function

• With boundary condition

• The above equation is known as HJB

differential equation
 HJB theory
• A special case of the HJB is when the system is
linear and is given by

• With the cost function being

• The Hamiltonian of the system is given by

 HJB theory
• The controller which minimizes the
Hamiltonian is given by

• Applying the HJB theory

• With the boundary condition

 HJB theory
• Assume that

• Then

• Substituting into the HJB equation

 HJB theory
• The optimal controller is obtained as

• Where K(t) is obtained by solving the inner

part of the HJB equation given by

• At the boundary K(tb)=F

 Continued LQR
• Linear Systems with Quadratic cost
– Problem formulation
– Linear regulator
– Linear servomechanism
• LQR under constant disturbance
• Stochastic optimal control systems
 Linear system with quadratic cost
• Consider the system with the following
dynamics

• The optimal regulator problem is finding a

control signal which minimizes the cost
function
 Linear system with LQG
• Forming the Hamiltonian

• Following the maximum principle steps,

control signal becomes

• Where P(t) is a solution of the equation

 Some points about LQR solution
• 1. existence of solution
– If S is positive definite and Q is at least non
negative definite or vise versa and R is positive
definite, then a minimum J exists if P exists, is
bounded and Positive definite.
– If S,Q and R are symmetric, then P is also
symmetric
– For a linear time invariant case the Riccati
equation reduces to
PA  AT P  PBR 1 B T P  Q
 Some points about the LQR
• For the time invariant Riccati equation
– If there exists a matrix K, it is positive definite
– If there exists a matrix K, the closed loop is
asymptotically stable
– In practical implementation of LQR controller, the
states may not be measurable and hence use of
observer or some kind of state estimation will be
necessary
– LQR controller can be implemented using simulink,
microcontroller, digital computer etc
 LQGR system
• When the states of a system are not
measurable, LQR can be implemented by
using estimator
• An optimal LQR with estimator is known as
LQGR
• Consider the system given by
 LQGR
• The objective is to find an estimate of x which
minimizes the mean square error given by

• If we use an estimator given by

• The filter gain(Kalman filter gain Kf) minimizes

the MSE is
 LQGR
• The term Ve in the filter equation is obtained
by solving a Richatti equation given by
Optimal servomechanism for linear systems
• The tracking problem is driving a system so
that its output is close to a desired trajectory
• Consider a linear system given by

• Then the objective is to minimize the

performance index given by

Where e(t) is the difference between desired trajectory and

actual system output
Optimal servomechanism for linear systems

• Following the maximum principle, the control

signal becomes
u (t )   R 1 B T P (t ) x(t )   (t )
 (t )   R 1 B T  (t )

• The control signal has two parts:

– The first part where P(t) is obtained from the LQR
Riccati equation
– Second part (t) which comes due to the desired
trajectory and (t) is obtained from new equation
 Optimal servomechanism …
• The boundary condition for the new Riccati
equation is
 (t f )  C T (t f ) S (t f )
• Example: determine an optimal controller for
a linear system given by
0 1  0  1 0 
A  B  C 
 0 0  1
   0 1 
• And performance R=1 and Q=(1 0;0 0);
• Desire trajectory is 1=constant and 2=0;
 Optimal servomechanism
• Evaluating the Riccati equations, the optimal
controller is obtained as
u   x1  2 x2   2
 LQR under constant disturbance
• Consider a system under disturbance given by
x  A(t ) x  B (t )u   (t )
• Where B(t) is the input matrix and w(t) is the
disturbance with the following properties
– The expected value of w(t) is zero but its energy is
an impulse

– The the disturbance do not affect the average

value of the states and their energy
 LQR under constant disturbance
• Taking quadratic performance index as

• Since (t) is a random disturbance with short

time correlation, it is unpredictable
• Hence the controller we obtain will be the
same as the LQR with out noise

• To determine the average behavior of the

optimally controlled system, substitute the
controller into the system dynamics
 LQR with disturbance
• Substituting u(t) gives
x  A(t ) x  B(t )(  R 1 B (t ) P (t ) x(t ))   (t )
x  ( A(t )  BK (t )) x(t )   (t )

• If the mean square value of x(t) is taken as

• Then the mean square history of the states is

given by  T
X  ( A  BK ) X  X ( A  BK )  V

• Where V is the energy of the disturbance

 LQR under constant disturbance
• Similarly to determine the mean square history of the
control
T T
E[uu ]  KXK
• Where K  R 1 B (t ) P(t )
• Following a similar procedure to determine the mean
square average of the performance, we obtain that the
effect of the noise is to increase the performance index
 LQR under disturbance
• If the system is a linear time invariant one,
then
– The system matrices A and B are constant
– The energy of the Gaussian disturbance will also
be constant
– The matrices in the performance index, S, Q and R
will be constant and also the control gains P and K
– The mean value of the states X and control u
become constant is obtained from
X  ( A  BK ) X  X ( A  BK )T  V
 LQR with process and measurement noise

• When a linear system having disturbances is to

be used with LQR
– Output of the system will be measured using
sensors
– Sensors will have measurement noise
– System may also have disturbance
• In the controller implementation, the states
will not be available and hence have to be
estimated from noisy measurements
 LQR with measurement noise
• If the estimator is optimal and the LQR
without noise is used, then the closed loop
remains optimal
• The average value of the system remains the
same as the LQR without disturbance
 Example- LQR control of antenna with
disturbance
• Consider an antenna position system shown
below
 Example contd…
• The block diagram model of the system with
simple amplifier feedback control is
 State space model of the system
• The model is given by
A=[-6 0;1 0] B=[1 0] and C=[0 20]
• Design an LQR controller
– R=1;
– Q=C’*C;
– K=lqr(A,B,Q,R);
– Ac=A-B*K;
 Assignment I
• Solve problems 5.7, 5.8, 5.9, 5.11, 5.12
Optimal control and estimation pp 297