L. Karp
Notes for Dynamics
VI. Two Stochastic Control Problems
1) Stochastic Control of Jump Process
(i) Derivation of DPE
(ii) Application to pollution control problem in section 4
2) Finite Markov chanins
i) Discuss range management application.
ii) Review finite Markov Chains.
iii) Formulate optimal control problem.
iv) Existence and uniqueness theorem.
v)
Three
solution
algorithms:
a)successive
approximation;
b)policy
iteration;
c)linear
programming.
vi) A suggestion for estimating DP problem.
The most popular analytic stochastic control model in economics uses a continuous random
variable, in continuous time.
This model is based on Brownian motion.
I will not discuss it.
1. Jump processes
: Stochastic control of Jump Processes, with an application to pollution control
Now we consider optimal control problems that involve a particular kind of stochastic process,
called a "jump process".
If the state variable z follows a jump process, it evolves along path
whose time derivative is f(z,u,t) "most of the time", where u is the control variable.
But
occasionally the variable z jumps up or down by a discrete amount. For example, z may be the
stock of a resource, say fish or oil.
Occasionally an event occurs, such as an epidemic which
drastically reduces the fish population, or a discovery which increases the stock of oil, causing
a discontinuous change in the variable. In other situations, the state variable might be the
exchange rate which depreciates at a deterministic rate most of the time; but some event might
trigger an attack on the currency, causing an abrupt devaluation.
We’ll consider a simple case
in which the discrete change is a(z,u,t). (More generally, there could be other possible discrete
changes, say b(z,u,t). For example, the stock of oil might increase due to a discovery, so a > 0,
or it might decrease discretely due to an environmental law which closes offshore drilling, so
b < 0.)
i) Derivation of DPE for simple case.
The probability that the state variable increases by the
discrete amount a(z,u,t) over a small interval of time dt is A(z,u,t)dt.
The state equation is
(1)
dz
=
f
(
z
,
u
)
dt
+
d
π
Pr (
d
π
=
a
) =
A
(
z
,
u
)
dt
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Pr (
d
π
= 0) = 1 
A
(
z
,
u
)
dt
This equation implies that there is "very little" chance that the jump happens over any small
interval.
In order to derive the DPE we go through the same steps as with the deterministic problem. We
define the value function, split up the program into two parts, the first part over (t, t+dt) and the
second part over (t+dt, T). However in the present case we have to take expectations, to account
for the uncertainty.
(2)
J
(
z
,
t
) =
max
{
u
}
E
⌡
⌠
T
t
L
(
z
,
u
,
τ
)
d
τ
max
{
u
}
E
⌡
⌠
t
dt
t
L
(
Z
,
u
,
τ
)
d
τ
⌡
⌠
T
t
dt
L
(
z
,
u
,
τ
)
d
τ
(
Note:
)
E
t
dt
⌡
⌠
T
t
dt
L
(
z
,
u
,
τ
)
d
τ
J
(
z
dz
,
t
dt
)
max
u
L
(
z
,
u
,
t
)
dt
J
(
z
t
f
( )
dt
a
,
t
dt
)
A
( )
dt
1
J
(
z
t
f
( )
dt
,
t
dt
)[1
A
( )
dt
]
2
Depending on whether or not the jump occurs, the state changes by f( )dt + a or by f( )dt.
We
have to weight each of those events by the probability that it occurs.
This is the end of the preview.
Sign up
to
access the rest of the document.
 Fall '06
 KARP
 Optimization, The Land, Probability theory, Markov chain, DPE

Click to edit the document details