This** preview**
has intentionally

**sections.**

*blurred***to view the full version.**

*Sign up*
This** preview**
has intentionally

**sections.**

*blurred***to view the full version.**

*Sign up*
This** preview**
has intentionally

**sections.**

*blurred***to view the full version.**

*Sign up*
This** preview**
has intentionally

**sections.**

*blurred***to view the full version.**

*Sign up*
This** preview**
has intentionally

**sections.**

*blurred***to view the full version.**

*Sign up*
This** preview**
has intentionally

**sections.**

*blurred***to view the full version.**

*Sign up*
**Unformatted text preview: **52 VARI differentiai equation is described by
5:1 rÂ»: MU), 361(0) = 1 XII; 3 R XZ(0) 1:1. I '
ions and transversality conditions, â€™1. A linear second~order of the EulernLagrange equat â€˜nd, â€™0 use â€˜ . ' . '
FE y ntrol 11(1) which minimizes. the optimal co (a) J = I: 242 cit, an) = x20) == 9
(a) J a j: u2 dr, xlo) a 0
(c) J = j: at a, x4e) = can 2 at [Also determine t; and x1(rf).1 (d) J = j" u2 cit, any) =Â«Â» car) a ~I}, x203 w 0
D
3 2 4; ruin} dt.
(6) I Â«2 {llel ' .
For alt eases skeich both the optimal system trajectory 2:0!) and the optimal system control am. 8. "For the ï¬xed piant dynamics given by
. m u determine the optimal closedâ€”loop system which mmrmrzes J=ij:[u2+(x_r)2}dr where i0) = 1 ~ 9". â€˜ O M
9 For the ï¬xed plant dynamics giyen by x 3 MG), x( ) w x9,
closed-loop control which minimizes for ï¬xed If fr
J = %sz(tf) + i); L â€œ2. d: . . . . m
h re s is an arbitrary constant. Do this by ï¬rst determining the optima
e W I
:penâ€”loop control and traieetory and then let u(t) w k(t)x(t) determine the optimal The maximum principle and Hamiltonsjambi theory In the previous chapter, we formulated many problems in the classical
calculus of variations El]. A derivation of the Euler-Lagrange equations for
both the scalar and vector cases was presented. We discussed the associated
transversality conditions and some of the difï¬culties which we may encounter
if inequality constraints are present. Several simple optimal control problems
Were stated and solved. In this chapter we wish to reexamine many of the
problems presented in the previous chapter and obtain more general solutions
for some of them. in addition, we will develop methods for handling some
problems which could not be conveniently formulated by the methods in the
previous chapter. To these ends, we will present the Bolza formulation of the variational
calculus using Hamiltonian methods. This wiil lead us into a proof of the
Pontryagin maximum principle and the associated transversality conditions
12â€”5]. We will proceed then to a development of the HamiltonuJacobi equa-
tions [12â€”44], which are equivalent to Belimanâ€™s equations of continuous
dynamic programing. Finally, we will give brief mention to some iimitations
of dynamic programming. Examples to illustrate the methods will be preÂ»
sented. We will reserve the next chapter for a discussion of some of the many
problems which we can formulate and solve using the maximum principle. In order to fuiiy develop our approach to optimization theory where the
terminal time is not ï¬xed and where the control and state vectors are not
necessarily smooth functions, we must consider in more detail the ï¬rst variaâ€”
tion for such problems. 54 THE MAXIMUM PRINCIPLE AND HAMILTON-JACOB: THEORY CH. 4 4.1
The variational approach for functions
with terminal times not ï¬xed We now extend the variational approach introduced in Sec. 3.6 to prob-
lems having unspeciï¬ed terminal times. Consider extremlzrng J 2 Fence), in), :3 dz (4.1-1; with respect to the set of all admissible (see Sec. 2.1) trajectories. Let t, be the
terminai time associated with optimal trajectory is. Associated With. each
perturbation it away from the optimal trajectory is a perturbation 6:, in the
terminal time. Let the ï¬rst variation (SJ be the part of AI m J{x ~iâ€” 12,1:Jr + do) â€”â€” J(x, If) (4.1-2) which is linear in i1 and 52}. Substituting Eq. (4.1-1) into Eq. (41:2), taking
the linear terms of AJ (in h, 61}, and h), and performing the usual integration
by parts to reduce terms dependent on it to terms dependent on 11, we obtam aJ :2 <1:[x(:,),:t(z,.), 1:55;, + beef) {lï¬‚hlxegÃ©gra o] + 'r so a as _
I I170) (E â€” 2}â€œ 3;) d: (4.1 3) where for convenience, we have assumed that the initial conditiOn is ï¬xed and hence i195} : 0. . .
in order to rephrase Eq. (4.1-3) into a convenient form, we introduce the following notation. We deï¬ne From a Taylor series expansion of x(t,Â« + diff), we note that 6x(rf) is aâ€˜close
approximation to that part of [x(t, ~lâ€” Eff) +110, + 55,0] â€” x(t,) which is
linear in boy) and 61,. Submitting Eq. (4.1-4) into Ed. (4.1-3) and rearrangâ€”
ing, the ï¬rst variation becomes r a; : lento). no), tr] â€” ï¬‚aw} 61" + d 1m r) In much of our work, it will be convenient to deï¬ne a quantity, called the
Hamiltonian, by Htxtt), W), t} = (D Â«w itÂ»? 2 <1: + an. (4.1-6) aï¬‚oatâ€”semi. fantasies. (4.1-5) ..Â»..h._a..._...,,.m...._,.w..._.......-...t....... â€œâ€˜r. a CH. 4 THE MAXIMUM PRINCIPLE we Hammett-Janos; THEORY 55 where the Hamiltonian is not a function of}: i0), and Mt) are called the ' canonical Variables. In terms of the Hamiltonian, the ï¬rst variation of Eq. (4.1â€”1), which is Eq. (4.1-5) becomes
61 :- â€”6x7(t,)h(tf) + H {311(9), Mgr), if] 53,, + f lira) {93% -iâ€” ~33 dz.
(4.15!) To establish a necessary condition for-a minimum, it is necessary that
the integrand in Eqs. (43-5) and (4.1-7) vanish and also that the trans-
versality condition, as obtained from Eq. (4.1-7), â€”5XT(1â€˜I)M3I) â€lâ€˜ Htxm), Mfr), fr] 5â€˜? z 0 (4.1-8) be satisï¬ed. 4.2 Weierstrass-Erdmann conditions Thus far in our development, admissible trajectories have been constrained
to be continuously diiferentiabie with respect to x and t. This functional
constraint on the class of ail admissibie traiectories is often unrealistically
restrictive, as the following example will Show. For this example, an optima}
admissible soiution does not exist; however, if the functional restriction
on an admissibie trajectory is sufï¬ciently relaxed, existence of an optimal
admissible trajectory is assured. We now examine the consequences of our
new deï¬nition of an admissible trajectory which are the Weierstrass-
Erdmann conditions {i}. Let us consider the problem of minimizing the cost function J: fxzawxydz
0 subject to
x(0) = O, x(1) = l. Physically, it is clear that the absolute minimum for J is 0 and that this is
obtained for x(t) = 0, t E [0, i]
x(t) = 2; â€” 1, t e [121}
which is certainly a soiution to the Enter-Lagrange equation for this probiem
x255 â€”lâ€” 2655" â€”â€” 4x = 0. There is one disturbing feature about this solution, however, in that the
optimum x{t) has a â€œcornerâ€ or discontinuous ï¬rst derivative which gives 56 THE MAXIMUM Famctrrs AND HAMmTONâ€”IACOBI THEORY CH. 4 rise to formal difï¬culty since it is contained in the Euler-Lagrange equations.
Thus, the solution of the above problem is not admissible. Certainly, this
particular function x{t) is continuousiy differentiable everywhere except at
a ï¬nite number of points (in this case the single point r a: i). Thus, in relaxing
the set of admissibie trajectories to allow for functions which are piecewise
continuously differentiable, the function x0) is admissibie and the above inrobiem then has an optimal admissible control. .
The Weierstrass-Erdmann corner conditions furnish us with necessary conditions for an optimal trajectory to have a discontinuous derivative at
a point in the control interval of interest. Speciï¬cally, consider the probiem
of ï¬nding a trajectory among the class of all continuously differentiable
functions on [52, b] having a corner at c E (a, b) which satisï¬es ï¬xed initial
and ï¬nai boundary values such that the functional 3:
Jon 2 i one), in), t} dt
has an extremum. It is of course clear that, for t E Eu, 0] and r e [c, b}, the
function x(t) must satisfy the Euler-Lagrange equations for a minimum
ii. 993.. W â€˜1â€˜? ..â€”_ 0
dr 65: 6x
We may rewrite the cost function as a sum of two cost functions: a) = i: one), is), :1 at: + l: axe), so), :1 at : J1(x) + 14X). _ We may now take the ï¬rst variation 6J1(x) and 5.1291) separateiy. We
assume, for the moment oniy, that a and b are fixed, and we require that the Â£0) calculated from J1(x) and J2(x) is the same at t = c which is unknown.
Since 5 is arbitrary, the ï¬rst variation of J1(x) is 51K") = *i 51505:) WM + {we}, in), c] ._ Â«swagger c1} a: a sxrtciLWâ€”ngï¬â€™Ã©â€˜c)â€™ â€œ1 + .. do of as:
j, W) {as - a a} â€˜1â€œ Since x(t) satisï¬es the Euler-Lagrange equations for an extremal and since
8x(a) = 0, we have acme, ï¬t), I} i
drift) T {one}, to), '5} â€”â€”~ anoW} aÂ»: (for 1: = c _ 0). 6J,(x) 2 sxrm CH. 4 Tris Maxuatm PRINCIPLE AND HAMILTON-JACOB! THEORY 57 In a simiiar fashion, we can show that the ï¬rst variation for the extremal
solution of 1201;) is 6320â€˜) m â€”5XT(1)W _ {(Iâ€™lXCâ€˜F): 51(1): T] â€” 13(0W} 6': (for 'r :2 c + 0). In order to obtain the extremum, the extremal solution must satisfy
Â§J(x) 3 ages + 6.50;) = O Thus
â€˜1â€˜? 6(1)
63. irwcao m 33â€™? MPH) (4.2â€”1)
. dd) . aq,
â€˜33 _ Tâ€” = _ r__
x 6iiÂ¢=câ€”o q) x as: Wâ€ (4.22) since 5x and 5:, are arbitrary. These requirements, Eqs. (4.2â€”i) and (4.2-2),
are called the Weierstrassâ€”Erdmaan comer conditions and must hoid at any point c where the extremal has a corner. If we use the Hamiitonian canonical
variables -H=owf%Â§Â§mo+trx â€”_1 dis we immediately see that the Weierstrass-Erdmann conditions simpiy require Hand 24. to be continuous on the optimum trajectory at aii points where there
are corners. 4.3 The Bolza yrobiemâ€”mno inequality constraints In Sec. 3.7 We considered the solution of Lagrange problems with equality
constraints of the form g(x, it, t) 2 0 for alt r in the control intervai of interâ€” est. A speciai case of this-equality constraint which is wellâ€”recognized as a
model of a large and important class of physicai systems is so) = f[x(t), u(t), t}, (4.3-1) where the m-vector It represents the control function to be selected and the
nâ€”vector x represents the resulting trajectory. We wiii assume that f has con-
tinuous partial derivatives with respect to x and u. Often it is the case that
such smoothness assumptions guarantee that for any piecewise continuous
function a, there exists a unique, admissible trajectory x to Eq. (4.3-1). We
therefore deï¬ne the set of admissible control functions to be the ciass of pieces 58 THE MAXIMUM PRINCIPLE AND HAMILroN-IACOBI THEORY CH .4 wise continuous functions and assume that, for an admissible in and a given
initial condition x09), Eq. (4.3â€”1) deï¬nes a unique, admissible solution over the control interval of interest.
Throughout the remainder of this section and the following section, we wilt consider the development of necessary conditions for Bolza problems subject to the equality constraint in Eq. (4.3-1). Subsections (4.34) and (4.3â€”2)
will consider the fixed ï¬nai time and the unspeciï¬ed ï¬nal time cases when no restrictions are imposed on the value that u can take at each time if during
the control interval of interest. Section 4.4 considers two cases of inequality
constraints on the control function and its associated trajectory over the control interval. 4.3-]
Continuous optimal control problemsâ€œâ€” ï¬xed beginning and terminai timesâ€”â€”
no inequality constraints We now consider the problem of determining an admissible control
function u in order to minimize the criterion ' J = 9[x(r), t] ~+ l axe), no), :1 dr, (4.3â€”2) where 9 and git possess continuous partial derivatives in x and n.
We use the method of Lagrange multipliers discussed in the last chapter
to adjoin the system differential equaiity constraint to the cost function, which gives us
J x â€œml â€œi: t i," one), no), a + marine), um, :1 -- a} at
Â° Â° (4.3â€”3)
We deï¬ne a scalar function, the Hamiltonian, as
H [1(0), 11(1), Mt), Ii Â«re (ï¬lm), I10), I] aâ€œ â€(ï¬‚flxo'), 11(1), 1â€˜1. (4-3-4) Thus the cost function becomes J a can), t] + j {Htqu HQ), in), z] -â€” was} at. (4.3-5) If we integrate the East term in the integrand of Eq. (4.3â€”5) by parts, we
obtain J = {9[x(r), I] â€”> PKGXUH â€ + j " {Harri as), Mr). :1 4â€” texts); a.
2" Â° (4.3-6) We now take the ï¬rst variation of J for variations in the control vector
and, consequently, in the state vector about the optimai control: and optimai ::
f: CH. 4 THE MAXIMUM PRINCiPLn AND HAMiLTONâ€”JACOBI THEORY 59 state vector. This gives us
"â€˜I "â€™ 6H - 6H
+ J {Syria + A] + Barbâ€”u} dt. (4.3â€”7) A necessary condition for a minimum is that the ï¬rst variation in J vanish
for arbitrary variations 6x and ï¬‚u. Thus we have as the necessary condition
for a minimum the very important relations 61 .7, {ext 3% w it} 5x1" 2;: ._ 1a 0, for r a i,,:, (4.3-8) - 6H .
r a "3;, x m f(x, 1:, t) a ï¬g (4.3-9)
:1: a 0. C are/L3 (4.3-10) _ We now consider in more detail the transversaiity conditions expressed
in Eq. (4.3-8). . F ora large class of optimal control problems, the initial state of the system
is specrfied but the terminal state is unspeciï¬ed. In that case, Eq. (4.3â€”8)
yields the transversality conditions as Mo) : Lgâ€˜ï¬gb â€™1'} (4.3Â»! 1)
since Sabra) m 0, x03) is ï¬xed, and 5x09) is completely arbitrary. In another
broad class of problems x00) and x(t,-) are ï¬xed. In this case 8x00) and 61:0,)
must be zero, and x03) and x03) are the boundary conditions for the two-
point boundary value problem. For rnany estimation probierns, neither
x(r,,) nor x03) are fixed and 9 = 0. In that case, Eq. (4.3â€”8) yields Mtg) m
sz) = O as the boundary conditions for the problem since 5116,) and Â§X(tâ€˜,â€”)
are arbitrary. In still another case, we might have x00) Lâ€œ X5, 6 2 O, and
il x(tf) E? m i. In this event, it is easy for us to show that the final transversality conditions are obtained if we solve the two scalar equations, each in :1
variables xtto) 2 xi, 5XT(tf)x(fj-) : 0, waging.) a c. (4.342) We now give a more general and precise interpretation to the transversality
conditions. For the generai case where the initiai manifold is Mixua), to] m 0 (4.3-13)
and the terminal manifold is
Ntxui), If} 2 0, (4.344) we adjoin these conditions to the (9 function by means of Lagrange multipiiers,
eff and v and obtain for the cost function 60 THE MAXIMUM PRINCIPLE mo HemaroN-Jacoar THEORY CH. 4 J n axe), :11: Â«m were), :01 + vTNtxea a] + jâ€ {H{X(t), u(t), to), a] _ iron} dz. (4. 3â€”15) We now apply the usual variational techniques to obtain for the trans-
versality conditions at the initial time: _ 69 dMT
Jâ€œ'(lâ€˜o) 'â€” a; 'iâ€” ( 6X )Â§9
The n initial conditions are obtained from this, with r parameters to be found
in Eq. (4.3â€”16) such that we satisfy the r conditions of Eq. (4.3-1.3). In a
similar fashion, the terminal condition is
W 69 dNT w, ,m . - Mg.) ~ 3â€˜; (73v, N{x(t), t] _ o, t _ i,, (4.3 17)
n terminal conditions are obtained from this with g parameters v found in
Eq. (4.3-1?) such that the q conditions of Eq. (43-14) are satisï¬ed. The n vector diï¬erential equation obtained from Eq. (4.3-9) will be called
the adioint equation. Equation (4.3-10) provides the coupling relation be
tween the original plant dynamics, Eq. (4.34), and the adj oint equation, the
1. equation of Eq. (4.3â€”9). This coupling equation was obtained from are +Â£f{Â§u7â€˜%%+ ...}dz, and it is important to note that Bu must be completely arbitrary in order for
us to draw the conclusion that 6876:: x: O to obtain the optimal control.
For the problem posed here where the admissible control set is inï¬nite, Eu
can be completely arbitrary. Where the admissible control is bounded, 8â€œ
cannot be completely arbitrary, and 619/611 m 0 may not be the correct
requirement. We will have more to say about this later. The solution we have obtained for this problem is a specral case of the Pontryagin maximum principle. _
It is also interesting to note that, since H =2 93 +â€˜Wf, we may compute the total derivative with respect to time as
dH _ do . 695 GE . do QB: -' 143
t x" Â«n +( it] â€Tiaâ€”u 4"â€œ (will t W "iâ€œ t a: M[x(r), r} = 0, r 2 re. (4.346) .3? M 3? 6);;
(4.3-18)
but from Eqs. (4.3-9) and (4.34) we have-
- .. 3H _... do dfT _
l r" "a; _ â€as â€" lair)" - (4'3 â€˜9) and from Eq. (4.3-4) CH. 4 THE MAXIMUM PRINClPLE AND HAMILTON-JACOB: THEORY 61 Thus, since fl. = 1T1â€œ, Eq. (43-18) becomes dH 6 6f . d
We see that, if to and f are not explicit functions of time, the Hamiltonian
is constant along an optimal trajectory where 63/61! x 0. It can be shown
that this 13 always true along an optimal trajectory, even if we cannot require
dH/du m 0. We will make use of this fact in a later development. . In order that J be a minimum, the second variation of J must be nonnegaÂ»
tive along all trajectories such that Eq. (4.3â€”1) is satisï¬ed. Therefore we need
to compute the second variation of J in Eq. (4.3-6) and impose the require-
ment that the variation of Eq. (4.3-1) is zero, or that as; _.. (72%) 5x â€” (3%) 3:: = e. (4.3Â»22) Applying this condition and taking the quadratic part of the Taylor series
expansron of J(x â€”Â§â€” ï¬x, it â€”Â§- 511) â€” J(x, 11), We have for the second variation (4.3â€”7.1) 1 62 tax,
so: â€œï¬‚axrmgsï¬‚ +
@611 a an
.h â€ T x2 (ii-1793? 5x
2 J: an SM] 6 M T 62H [quâ€œ? (4.3â€”23)
{3&3}? W and this must be nonnegative for a minimum. This will be the case if the g :1â€” m square matrix under the integral Sign and rind/(3x2 are nonnegative
e nite. Example 4.3-1. We are given the dilferentiai system consisting of three
cascaded integrators ft; m x; 351(0) = 0
32 m x; xz(0) m G
xâ€˜3 w a 33(9) m 0.
We wish to drive the system so that we reach the terminal manifoid
Xiti) + 2:30) m I
such that the cost function J=Â§Â£uzdr is minimized. The solution to the problem proceeds as follows. We compute
the Hamiltonian from Eq. (4.3-4) as
H = Â£412 + Iâ€˜ll-1'2 + 11.23% + 131! and determine the coupling relation, Eq. (4.340), 62. THE MAXIMUM PRENCliâ€™LE AND HAMILTON-Incest anonv CH. 4 ï¬m0=u+lg ï¬n
and the adjoint Eq. (4.3-9),
1 w _Â§_Â£:i = 0
i W 6X}
61â€˜! _ _
12 â€- â€”E52 M 11 From Eqs. (4.3â€”14) and (4.347) we see that the transversality condition at the terminal time is
xï¬‚l) 4.- xiï¬‚) m 1 NT
in) m 12% + (6â€”6;)v, are t: where
Nixtrr), oi 3: xi(?;) + xiï¬‚r) M 1 = 0: If e 1- 112(1) 2171(1)?
1(1) = [212(1) â€œ [2362(1)]? -
3(1) 0 Thus the probiern of ï¬nding the optimai controi and associated trajectories
for this example is completeiy resoived when we solve the two-pornt boundary vaiue problem represented by Thus i; m x; x;(0) = 0 222 = x3 are) = 0 .333 w â€”13 953(0) = 0 it a o M1) = 2xltnv}xm + ï¬g) = 1
3.; a â€”11; M1) = man is m ~22 . 213(1) =0 Although the six ï¬rst-order differential equations represented above are perâ€”
fectly linear and time invariant, the solution to this problem'is complicated
by the nonlinear nature of the terminal conditions. We shail discover various iterative schemes for overcoming this difï¬culty in Chapter 10. 4.3â€”2
Continuous optimal control problemsâ€”- ï¬xed beginning and unspeciï¬ed terminal times-â€”
720 inequality constraints The materiai of the previous subsection may be easiiy extended to. the
case where the terminai manifold equation is a function of the terminal tame, CH. 4 Tire MAXiMUM PRINCIPLE AND HAMILTON-Jamar THEORY 63 and the terminal time is unspeciï¬ed. For convenience we will assume that
the initiai time and the initial state vector are speciï¬ed. Soiution may then
easily be obtained for the case where the initiai time and initial state vector
are unspeciï¬ed. Therefore the problem becomes one of minimizing the cost
function J = Bins), n] + if axe), no), :1 dz (43â€”24)
for the system described by
i = fEXOâ€˜), â€œ(1â€˜), 3}, X09) = Xe (4-3â€”25) where ta is ï¬xed and where, at the unspeciï¬ed terminal time t m If, the q
vector terminal manifold equation N[X(tf), if} = 9 (4.3-26) is satisï¬ed. It may be noted here that the terminal manifold line, x(tf) = .205.)
of the previous chapter becomes N[x(t,), if] M: 0, which is more genera}. We
adjoin the equality constraints to the cost function via Lagrange multipliers
to obtain J :- axon, ta + vrutxen, s] + if {axe}, an), r] +
7LT(r)[f{x(r), 110), t} -â€” 31]} dr. As before, we define the Hanuitonian H [31(3), 60), Mt), fi ï¬ MW), "0), I] + â€(OHXG), â€œ(0, t] and integrate a portion of the cost function, Eq. (43-27), to obtain
J m 956;Â»): If] + VTNDiUr), If] â€œ- VCfrkOf) + moxie) i- f {Hint}, at), Mr), a + true} at (4.3Â»27) (43-28) We again form the ï¬rst variation by ietting
x0) m 510) + ha), :10) : ii(r) + 51103), If = ff ~iâ€” 5t, (4.3â€”29) and then we form the difference J[x, u, If} - J [i 13, ff} and retain only the
iinear terms. Thus we have, after dropping the s notation for convenience, at a Maine), non. Mo), s]...

View
Full Document

- Winter '09
- Bamieh