Discrete-time stochastic processes

E the time customer n spends in the system is the

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: o reasonable limit. It also blurs the distinction between time and ensemble-averages, so we won’t use it in what follows. We next investigate the ensemble-average, E [R(t)] of a reward function, and in particular how it vh aries for large t and whether limt→1 E [R(t)] exists. We could also find i Rt limt→1 (1/t)E 0 R(τ ) dτ , which can be interpreted as the limiting expected value of reward, averaged over both time and ensemble. Not surprisingly, this is equal to E [Rn ] /E [X ]) (See [16], Theorem 6.6.1). We will not bother with that here, however, since what is more important is finding E [R(t)] and limt→1 E [R(τ )] if it exists. In concrete terms, if E [R(t)] varies significantly with t, even for large t, it means, for example, that our waiting time for a bus depends strongly on our arrival time. In other words, it is important to learn if the effect of the original arrival at t = 0 gradually dies out as t → 1. 3.5 Renewal-reward processes; ensemble-averages As in the last section, {N (t); t ≥ 0} is a renewal counting process, Z (t) and X (t), t > 0, are the age and duration random variables, R(z , x) is a real valued function of the real variables z and x, and {R(t); t ≥ 0} is a reward process with R(t) = R[Z (t), X (t)]. Our ob jective is to find (and to understand) limt→1 E [R(t)]. We start out with an intuitive derivation which assumes that the inter-renewal intervals {Xn ; n ≥ 1} have a probability density fX (x). Also, rather than finding E [R(t)] for a finite t and then going to the limit, we simply assume that t is so large that m(τ + δ ) − m(τ ) = δ /X for all τ in the vicinity of t (i.e., we ignore the limit in (3.17)). After this intuitive derivation, we return to look at the limiting issues more carefully. Since R(t) = R[Z (t), X (t)], we start by finding the joint probability density, fZ (t),X (t) (z , x), of Z (t), X (t). Since the duration at t is equal to the age plus residual life at t, we must have X (t) ≥ Z (t), and the joint probability density can be non-zero only in the triangular region shown in Figure 3.12. From (3.19), the probability of a renewal in a small interval [t − z , t − z + δ ) is δ /X − o(δ ). Note that, although Z (t) = z implies a renewal at t − z , a renewal at t − z does not imply that Z (t) = z , since there might be other renewals between t − z and t. Given a renewal at t − z , however, the subsequent inter-renewal interval has probability density fX (x). Thus, 3.5. RENEWAL-REWARD PROCESSES; ENSEMBLE-AVERAGES 113 the joint probability of a renewal in [t − z , t − z + δ ) and a subsequent inter-renewal interval X between x and x + δ is δ 2 fX (x)/X + o(δ 2 ), i.e., Pr {renewal ∈ [t − z , t − z + δ ), X ∈ (x, x + δ ]} = δ 2 fX (x) + o(δ 2 ). X This is valid for arbitrary x. For x > z , however, the joint event above is the same as the joint event {Z (t) ∈ (z − δ, z ], X (t) ∈ (x, x + δ ]}. Thus, going to the limit δ → 0, we have fZ (t),X (t) (z , x) = fX (x) , x > z; X fZ (t),X (t) (z , x) = 0 elsewhere. (3.38) This joint density is illustrated in Figure 3.12. Note that the argument z does not appear except in the condition x > z ≥ 0, but this condition is very important. The marginal densities for Z (t) and X (t) can be found by integrating (3.38) over the constraint region, Z1 fX (x) dx 1 − FX (z ) fZ (t) (z ) = = . (3.39) X X x=z fX (t) (x) = Z x z =0 fX (x) dz xfX (x) = . X X The mean age can be calculated from (3.39) by integration by parts, yielding £§ E X2 E [Z (t)] = . 2X ♣♣♣♣♣♣♣♣♣♣ °♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣ °♣♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣ °♣♣♣ ♣♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ z °♣♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ °Z (t),X (t) (z , x) = fX (x) ; f X ♣♣ ♣°♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣ ♣♣♣ ♣ ♣°♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣♣ ♣♣ ♣°♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣♣ ♣♣ ♣♣♣ ♣ ♣° °♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ x ♣♣ ♣♣ ♣♣ ♣♣ ♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣♣ ♣ ♣♣ ♣♣ ♣♣ ♣...
View Full Document

Ask a homework question - tutors are online