This preview shows page 1. Sign up to view the full content.
Unformatted text preview: o reasonable limit. It
also blurs the distinction between time and ensembleaverages, so we won’t use it in what
follows.
We next investigate the ensembleaverage, E [R(t)] of a reward function, and in particular how it vh
aries for large t and whether limt→1 E [R(t)] exists. We could also ﬁnd
i
Rt
limt→1 (1/t)E 0 R(τ ) dτ , which can be interpreted as the limiting expected value of reward, averaged over both time and ensemble. Not surprisingly, this is equal to E [Rn ] /E [X ])
(See [16], Theorem 6.6.1). We will not bother with that here, however, since what is more
important is ﬁnding E [R(t)] and limt→1 E [R(τ )] if it exists. In concrete terms, if E [R(t)]
varies signiﬁcantly with t, even for large t, it means, for example, that our waiting time for
a bus depends strongly on our arrival time. In other words, it is important to learn if the
eﬀect of the original arrival at t = 0 gradually dies out as t → 1. 3.5 Renewalreward processes; ensembleaverages As in the last section, {N (t); t ≥ 0} is a renewal counting process, Z (t) and X (t), t > 0,
are the age and duration random variables, R(z , x) is a real valued function of the real
variables z and x, and {R(t); t ≥ 0} is a reward process with R(t) = R[Z (t), X (t)]. Our
ob jective is to ﬁnd (and to understand) limt→1 E [R(t)]. We start out with an intuitive
derivation which assumes that the interrenewal intervals {Xn ; n ≥ 1} have a probability
density fX (x). Also, rather than ﬁnding E [R(t)] for a ﬁnite t and then going to the limit,
we simply assume that t is so large that m(τ + δ ) − m(τ ) = δ /X for all τ in the vicinity of
t (i.e., we ignore the limit in (3.17)). After this intuitive derivation, we return to look at
the limiting issues more carefully.
Since R(t) = R[Z (t), X (t)], we start by ﬁnding the joint probability density, fZ (t),X (t) (z , x),
of Z (t), X (t). Since the duration at t is equal to the age plus residual life at t, we must
have X (t) ≥ Z (t), and the joint probability density can be nonzero only in the triangular
region shown in Figure 3.12.
From (3.19), the probability of a renewal in a small interval [t − z , t − z + δ ) is δ /X − o(δ ).
Note that, although Z (t) = z implies a renewal at t − z , a renewal at t − z does not imply
that Z (t) = z , since there might be other renewals between t − z and t. Given a renewal at
t − z , however, the subsequent interrenewal interval has probability density fX (x). Thus, 3.5. RENEWALREWARD PROCESSES; ENSEMBLEAVERAGES 113 the joint probability of a renewal in [t − z , t − z + δ ) and a subsequent interrenewal interval
X between x and x + δ is δ 2 fX (x)/X + o(δ 2 ), i.e.,
Pr {renewal ∈ [t − z , t − z + δ ), X ∈ (x, x + δ ]} = δ 2 fX (x)
+ o(δ 2 ).
X This is valid for arbitrary x. For x > z , however, the joint event above is the same as the
joint event {Z (t) ∈ (z − δ, z ], X (t) ∈ (x, x + δ ]}. Thus, going to the limit δ → 0, we have
fZ (t),X (t) (z , x) = fX (x)
, x > z;
X fZ (t),X (t) (z , x) = 0 elsewhere. (3.38) This joint density is illustrated in Figure 3.12. Note that the argument z does not appear
except in the condition x > z ≥ 0, but this condition is very important. The marginal
densities for Z (t) and X (t) can be found by integrating (3.38) over the constraint region,
Z1
fX (x) dx
1 − FX (z )
fZ (t) (z ) =
=
.
(3.39)
X
X
x=z
fX (t) (x) = Z x z =0 fX (x) dz
xfX (x)
=
.
X
X The mean age can be calculated from (3.39) by integration by parts, yielding
£§
E X2
E [Z (t)] =
.
2X
♣♣♣♣♣♣♣♣♣♣
°♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣
♣♣
°♣♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣
♣
°♣♣♣ ♣♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣
♣♣
z
°♣♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣
♣
°Z (t),X (t) (z , x) = fX (x) ;
f
X
♣♣ ♣°♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣
♣♣ ♣♣♣
♣ ♣°♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣
♣♣
♣♣ ♣°♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣
♣♣ ♣♣♣
♣ ♣°
°♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ x ♣♣
♣♣
♣♣
♣♣
♣
♣♣
♣♣
♣♣
♣♣ ♣♣
♣♣
♣♣
♣♣
♣
♣♣
♣♣
♣♣
♣♣ ♣♣
♣♣
♣♣
♣♣
♣
♣♣
♣♣
♣♣
♣...
View Full
Document
 Spring '09
 R.Srikant

Click to edit the document details