2.2 The Powers of the Transition Matrix
2.2.1 A Formula for the Powers of the Transition Matrix
Last time we saw that probabilities of future events in a Markov chain can be computed from the powers
P
2
Markov Chains
2.1 Basic Principles.
2.1.1 The Transition Matrix.
A stochastic process is a mathematical model of a situation in the real world that evolves in time in a
probabilistic fashion, i.e. w
2.5 Hitting Times
2.5.1 Hitting Times
Sometimes we want to know things like
What is the probability that the system will have been in a certain state by a certain time?
or
What is the probability that
1.12 Single Period Inventory
We buy some item wholesale and sell it retail. The demand for the item varies, so we treat it a random variable.
We want to know how many to buy wholesale so as to maximiz
2.7 Number of Visits to a State
2.6.1 Number of Visits to a State
In the previous section we calculated the probability of reaching a state. In this section we want to
calculate the expected number of
2.3 Steady State Probabilities
2.3.1 Finding Steady State Probabilities
In the previous section we saw how to compute the powers Pn of the transition matrix P. We saw that
each element of Pn was a con