Discrete-time stochastic processes

# The important thing here is for the reward function

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: robability of a renewal at time kd is given by lim Pr {N (kd) − N (kd − d) = 1} = k→1 3.4 d . X (3.20) Renewal-reward processes; time-averages There are many situations in which, along with a renewal counting process {N (t); t ≥ 0}, there is another randomly varying function of time, called a reward function {R(t); t ≥ 0}. 106 CHAPTER 3. RENEWAL PROCESSES R(t) models a rate at which the process is accumulating a reward. We shall illustrate many examples of such processes and see that a “reward” could also be a cost or any randomly varying quantity of interest. The important restriction on these reward functions is that R(t) at a given t depends only on the particular inter-renewal interval containing t. We start with several examples to illustrate the kinds of questions addressed by this type of process. Example 3.4.1. (time-average Residual Life) For a renewal counting process {N (t), t ≥ 0}, let Y (t) be the residual life at time t. The residual life is deﬁned as the interval from t until the next renewal epoch, i.e., as SN (t)+1 − t. For example, if we arrive at a bus stop at time t and buses arrive according to a renewal process, Y (t) is the time we have to wait for a bus to arrive (see Figure 3.6). We interpret {Y (t); t ≥ R } as a reward function. The time0 6 (1/t) t Y (τ )dτ . We are interested in the average of Y (t), over the interval (0, t], is given by 0 limit of this average as t → 1 (assuming that it exists in some sense). Figure 3.6 illustrates a sample function of a renewal counting process {N (t); t ≥ 0} and shows the residual life Y (t) for R that sample function. Note that, for a given sample function {Y (t) = y (t)}, the t integral 0 y (τ ) dτ is simply a sum of isosceles right triangles, with part of a ﬁnal triangle at the end. Thus it can be expressed as Z t 0 n(t) 1X 2 y (τ )dτ = xi + 2 i=1 Z t y (τ )dτ τ =sn(t) where {xi ; 0 < i < 1} is the set of sample values for the inter-renewal intervals. Since this relationship holds for every sample point, we see that the random variable Rt 0 Y (τ )dτ can b e expressed in terms of the inter-renewal random variables Xn as Z t Y (τ )dτ = τ =0 Zt N (t) 1X 2 Xn + Y (τ )dτ . 2 n=1 τ =SN (t) Although the ﬁnal term above can be easily evaluated for a given SN (t) (t), it is more convenient to use the following bound: Z N (t) N (t)+1 1X 2 1 t 1X 2 X≤ Y (τ )dτ ≤ Xn . 2t n=1 n t τ =0 2t n=1 (3.21) The term on the left can now be evaluated in the limit t → 1 (for all sample functions except a set of probability zero) as follows: lim t→1 6 Rt PN (t) n=1 2t 2 Xn = lim t→1 £§ 2 E X2 Xn N (t) = . N (t) 2t 2E [X ] PN (t) n=1 (3.22) Y (τ )dτ is a random variable just like any other function of a set of variables. It has a sample 0 value for each sample function of {N (t); t ≥ 0}, and its distribution function could be calculated in a straightforward but tedious way. For arbitrary stochastic processes, integration and diﬀerentiation can require great mathematical sophistication, but none of those subtleties occur here. 3.4. RENEWAL-REWARD PROCESSES; TIME-AVERAGES 107 N (t) ✛ X2 ✲ X1 S1 S2 S3 S4 S5 S6 X5 ❅ ❅ ❅Y (t) X2 X4 ❅ ❅ ❅ X6 ❅ ❅ X1 ❅ ❅ ❅ ❅ ❅ X3 ❅ ❅ ❅❅ ❅ ❅❅ t ❅ ❅❅ ❅ ❅ S1 S2 S3 S4 S5 S6 Figure 3.6: Residual life at time t. For any given sample function of the renewal process, the sample function of residual life decreases linearly with a slope of −1 from the beginning to the end of each inter-renewal interval. The second equality above follows by applying the strong law of large numbers (Theorem P 2 1.5) to n≤N (t) Xn /N (t) as N (t) approaches inﬁnity, and by applying the strong law for renewal processes (Theorem 3.1) to N (t)/t as t → 1. The right hand term of (3.21) is handled almost the same way: £§ PN (t)+1 2 PN (t)+1 2 E X2 Xn Xn N (t) + 1 N (t) n=1 n=1 lim = lim = . (3.23) t→1 t→1 2t N (t) + 1 N (t) 2t 2E [X ] Combining these two results, we see that, with probability 1 (abbreviated as W.P.1), the time-average residual life is given by £§ Rt E X2 τ =0 Y (τ ) dτ lim = W.P.1. (3.24) t→1 t 2E [X ] 2 2 Note that this time-average depends on the second moment of X ; this is X + σ 2 ≥ X , so the time-average residual life is at least half the expected inter-renewal interval (which is not surprising). On the other hand, the second moment of X can be arbitrarily large (even inﬁnite) for any given value of E [X ], so that the time average residual life can be arbitrarily large relative to E [X ]. This can be explained intuitively by observing that large inter-renewal intervals are weighted more heavily in this time-average than small interrenewal intervals. As an example, consider an inter-renewal random variable X that takes on value ≤ with § probability 1 − ≤ and value 1/≤ with probability ≤. Then, for small ≤, £ E [X ] ∼ 1, E X 2 ∼ 1/≤, and the time average residual life is approximately 1/(2≤) (see Figure 3.7...
View Full Document

## This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online