L. Karp
Notes for Dynamics
VI. Two Stochastic Control Problems
1) Stochastic Control of Jump Process
(i) Derivation of DPE
(ii) Application to pollution control problem in section 4
2) Finite Markov chanins
i) Discuss range management application.
ii) Review finite Markov Chains.
iii) Formulate optimal control problem.
iv) Existence and uniqueness theorem.
v) Three solution algorithms: a)successive approximation; b)policy iteration; c)linear
programming.
vi) A suggestion for estimating DP problem.
The most popular analytic stochastic control model in economics uses a continuous random
variable, in continuous time. This model is based on Brownian motion. I will not discuss it.
1. Jump processes
: Stochastic control of Jump Processes, with an application to pollution control
Now we consider optimal control problems that involve a particular kind of stochastic process,
called a "jump process". If the state variable z follows a jump process, it evolves along path
whose time derivative is f(z,u,t) "most of the time", where u is the control variable.
But
occasionally the variable z jumps up or down by a discrete amount. For example, z may be the
stock of a resource, say fish or oil. Occasionally an event occurs, such as an epidemic which
drastically reduces the fish population, or a discovery which increases the stock of oil, causing
a discontinuous change in the variable. In other situations, the state variable might be the
exchange rate which depreciates at a deterministic rate most of the time; but some event might
trigger an attack on the currency, causing an abrupt devaluation. We’ll consider a simple case
in which the discrete change is a(z,u,t). (More generally, there could be other possible discrete
changes, say b(z,u,t). For example, the stock of oil might increase due to a discovery, so a > 0,
or it might decrease discretely due to an environmental law which closes off-shore drilling, so
b < 0.)
i) Derivation of DPE for simple case. The probability that the state variable increases by the
discrete amount a(z,u,t) over a small interval of time dt is A(z,u,t)dt. The state equation is
(1)
dz
=
f
(
z
,
u
)
dt
+
d
π
Pr (
d
π
=
a
)=
A
(
z
,
u
)
dt