Formula Sheet, E3106, Introduction to Stochastic Models
Columbia University, Fall 2005, Professor S. Kou.
Ch. 4
1. Chapman-Kolmogorov equation. Consider a discrete Markov Chain Xn.Let P(n)
ij =
P(Xn+m=j|Xm=i).ThenP(n+m)
ij =P∞
k=0 P(n)
ik P(m)
kj .Hence P(n)=Pn,whereP(n)=(P(n)
ij )
and P=(Pij ).
2. Classification of states. Accessible, i→j:ifP(n)
ij >0for some n≥0. Communicate,
i←→ j:ifi→jand j→i. Class: all states that communicate. Irreducible: if there is only
one class.
3. Recurrent and transient. Let fi=P(ever come back to i|starts at i).Transient:if
fi<1.Recurrent:iffi=1.
(1) Suppose state iis transient. Then the probability that the MC will be in state ifor
exactly ntime period is fn−1
i(1 −fi);andE[X]= 1
1−fi,whereXisthenumberoftimeperiod
theMCwillbeinstatei.
(2) State iis recurrent if P∞
n=1 P(n)
ii =∞, and transient if P∞
n=1 P(n)
ii <∞.
(3) If iis recurrent and i←→ j,thenjis also recurrent. Therefore, recurrence and transience
are class properties.
(4) Positive recurrent: if E(T)<∞,whereTis the time until the process returns to state
i. Null recurrent if E(T)=∞.
(5) Positive recurrent aperiodic states are call ergodic states.
4. πiis defined to be the solution of πj=P∞
i=0 πiPij,j≥0,P∞
i=0 πi=1.Four interpreta-
tions of πij.
(1) “Limiting probabilities”. For a irreducible ergodic MC, limn→∞ P(n)
ij =πj.
(2) “Stationary probabilities”. If P(X0=j)=πj,thenP(Xn=j)=πj.
(3) “Long-run average frequencies”. Let aj(N)be the number of periods a irreducible MC
spends in state jduring time periods 1,2,...,N. Then as N→∞,aj(N)
N→πj.
(4) mjj =1/πj,where mjj is the expectation of the number of transitions until the MC
returns back to j,startingatj.
5. Gambler’s ruin problem.
(1)ItisaMCwithP00 =PNN =1,andPi,i+1 =p=1−Pi,i−1,i=1,...,N −1.
(2) Let Pibe the probability of reaching Nbefore 0, starting with $i.ThenPi=pPi+1 +
qPi−1.Furth erm ore ,
Pi=1−(q/p)i
1−(q/p)N,if p6=1
2;Pi=i
N,if p=1
2.
(3) Let PTbe the transition matrix for transient states only. Then S=(I−PT)−1,where
S=(sij)and sij is the expected time periods that the MC is in state j, starting at i,for
transient states iand j.
(4) Suppose iand jare transient states. Let fij be the probability of ever making a transition
into state j,startingati.Thenfij =sij−δij
sjj ,where δij =1if i=j,andδij =0if i6=j.
Ch. 5
1. Exponential distribution: basic properties.
