Download Discrete Time Markov Chains and more Summaries Probability and Stochastic Processes in PDF only on Docsity!
Mor Harchol-Balter. Introduction to Probability for Computing,
Part VIII
Discrete-Time Markov
Chains
This final part of the book is devoted to the topic of Markov chains. Markov chains are an extremely powerful tool used to model problems in computer sci- ence, statistics, physics, biology, and business – you name it! They are used extensively in AI/machine learning, computer science theory, and in all areas of computer system modeling (analysis of networking protocols, memory manage- ment protocols, server performance, capacity provisioning, disk protocols, etc.). Markov chains are also very common in operations research, including supply chain, call center, and inventory management.
Our goal in discussing Markov chains is two-fold. On the one hand, as always, we are interested in applications and particularly applications to computing. On the other hand, Markov chains are a core area of probability theory and thus we have chosen to cover the theory of Markov chains in some depth here.
In Chapter 24, we introduce finite-state Markov chains, limiting distributions, and stationary distributions.
In Chapter 25, we delve into the theory of finite-state Markov chains, discussing whether the limiting distribution exists and whether the stationary distribution is unique. We also introduce time reversibility, time averages, and mean passage times. A more elementary class might choose to skip this chapter, but it is my experience that undergraduates are fully capable of understanding this material if they proceed slowly and focus on examples to help illustrate the concepts.
In Chapter 26, we turn to infinite-state Markov chains. These are great for modeling the number of packets queued at a router, or the number of jobs at a data center. Although we skip the hardest proofs here, there is still a lot of intuition to be gained just in understanding definitions like transient and positive-recurrent.
All these chapters are full of examples of the application of Markov chains for modeling and solving problems. However, it is the final chapter, Chapter 27 on queueing theory, which really ties it all together. Through queueing theory, we see a real-world application of all the abstract concepts introduced in the Markov chain chapters.
Mor Harchol-Balter. Introduction to Probability for Computing,
24 Discrete-Time Markov
Chains: Finite-State
This chapter begins our study of Markov chains, specifically discrete-time Markov chains. In this chapter and the next, we limit our discussion to Markov chains with a finite number of states. Our focus in this chapter will be on under- standing how to obtain the limiting distribution for a Markov chain.
Markov chains come up in almost every field. As we study Markov chains, be on the lookout for Markov chains in your own work and the world around you. They are everywhere!
24.1 Our First Discrete-Time Markov Chain
Love is complicated. Figure 24.1 depicts the day-by-day relationship status of CMU students.
0.95 Single 0.
0.5 0.5^ 0.
com^ It’s
plica ted
In a relation- ship
Figure 24.1 The states of love, according to Facebook.
There are three possible states for the relationship status. We assume that the relationship status can change only at the end of each day, according to the probabilities shown. For example, if we’re “single” today, with probability 0. we will still be single tomorrow. When entering the “relationship” state, we stay there on average for five days (note the Geometric distribution), after which we
Mor Harchol-Balter. Introduction to Probability for Computing,
422 24 Discrete-Time Markov Chains: Finite-State
Definition 24.3 The transition probability matrix associated with any DTMC is a matrix, P , whose (𝑖, 𝑗 ) th entry, 𝑃𝑖 𝑗 , represents the probability of moving to state 𝑗 on the next transition, given that the current state is 𝑖.
Observe that, by definition,
Í
𝑗 𝑃𝑖 𝑗^ =^ 1,^ ∀𝑖, because, given that the DTMC is in state 𝑖, it must next transition to some state 𝑗.
Finite state versus infinite state: This chapter and the next will focus on DTMCs with a finite number of states, 𝑀. In Chapter 26, we will generalize to DTMCs with an infinite (but still countable) number of states.
DTMCs versus CTMCs: In a DTMC, the state can only change at synchronized (discrete) time steps. This book focuses on DTMCs. In a continuous-time Markov chain (CTMC) the state can change at any moment of time. CTMCs are outside the scope of this book, but we refer the interested reader to [35].
Ergodicity issues: In working with Markov chains, we will often be trying to understand the “limiting probability” of being in one state as opposed to another (limiting probabilities will be defined very soon). In this chapter, we will not dwell on the question of whether such limiting probabilities exist (called ergodicity issues). Instead we simply assume that there exists some limiting probability of being in each state of the chain. We defer all discussion of ergodicity to Chapter 25.
The three Ms: Solving Markov chains typically requires solving large systems of simultaneous equations. We therefore recommend taking the time to familiarize yourself with tools like Matlab [52], Mathematica [80], or Maple [50].
24.3 Examples of Finite-State DTMCs
We start with a few examples of simple Markov chains to illustrate the key concepts.
24.3.1 Repair Facility Problem
A machine is either working or is in the repair center. If it is working today, then there is a 95% chance that it will be working tomorrow. If it is in the repair center today, then there is a 40% chance that it will be working tomorrow. We are interested in questions like, “What fraction of time does my machine spend in the repair shop?”
Mor Harchol-Balter. Introduction to Probability for Computing,
24.3 Examples of Finite-State DTMCs 423
Question: Describe the DTMC for the repair facility problem.
Answer: There are two states, “Working” and “Broken,” where “Broken” denotes that the machine is in repair. The transition probability matrix is
P =
[
𝑊 𝐵 𝑊 0.95 0. 𝐵 0.40 0.
]
The Markov chain diagram is shown in Figure 24.2.
Working Broken
0.95 0.
Figure 24.2 Markov chain for the repair facility problem.
Question: Now suppose that after the machine remains broken for four days, the machine is replaced with a new machine. How does the DTMC diagram change?
Answer: The revised DTMC is shown in Figure 24.3.
0.05 0.6 0.6 0.
ÊÊ
Working BrokenDay 1 BrokenDay 2 BrokenDay 3 BrokenDay 4
Figure 24.3 Markov chain for the repair facility problem with a four-day limit.
24.3.2 Umbrella Problem
An absent-minded professor has two umbrellas that she uses when commuting from home to office and back. If it rains and an umbrella is available in her location, she takes it. If it is not raining, she always forgets to take an umbrella. Suppose that it rains with probability 𝑝 each time she commutes, independently of prior commutes. Our goal is to determine the fraction of commutes during which the professor gets wet.
Mor Harchol-Balter. Introduction to Probability for Computing,
24.4 Powers of P: n -Step Transition Probabilities 425
24.4 Powers of P: n -Step Transition Probabilities
Definition 24.4 Let P 𝑛^ = P · P · · · P , multiplied n times. Then ( P 𝑛)𝑖 𝑗 denotes the (𝑖, 𝑗 ) th entry of matrix P 𝑛. Occasionally, we will use the shorthand: 𝑃𝑛𝑖 𝑗 ≡ ( P 𝑛)𝑖 𝑗.
Question: What does ( P 𝑛)𝑖 𝑗 represent?
Answer: To answer this, we first consider two examples.
Example 24.5 (Back to the umbrellas)
Consider the umbrella problem from before, where the chance of rain on any given day is 𝑝 = 0.4. We then have:
P =
P^5 =
P^30 =
Observe that all the rows become the same! Note also that, for all the above powers, each row sums to 1.
Example 24.6 (Back to the repair facility)
Now, consider again the simple repair facility problem, with general transition probability matrix P :
P =
[
]
You should be able to prove by induction that
P 𝑛^ =
[ 𝑏+𝑎( 1 −𝑎−𝑏)𝑛
𝑎+𝑏
𝑎−𝑎( 1 −𝑎−𝑏)𝑛 𝑏−𝑏( 1 −𝑎−𝑏)𝑛^ 𝑎+𝑏 𝑎+𝑏
𝑎+𝑏( 1 −𝑎−𝑏)𝑛 𝑎+𝑏
]
lim 𝑛→∞
P 𝑛^ =
[ 𝑏
𝑎+𝑏
𝑎 𝑏^ 𝑎+𝑏 𝑎+𝑏
𝑎 𝑎+𝑏
]
Question: Again, all rows are the same. Why? What is the meaning of the row?
Hint: Consider a DTMC in state 𝑖. Suppose we want to know the probability that it will be in state 𝑗 two steps from now. To go from state 𝑖 to state 𝑗 in two steps,
Mor Harchol-Balter. Introduction to Probability for Computing,
426 24 Discrete-Time Markov Chains: Finite-State
the DTMC must have passed through some state 𝑘 after the first step. Below we condition on this intermediate state 𝑘.
For an 𝑀-state DTMC, as shown in Figure 24.5, ( P^2
𝑖 𝑗
𝑀Õ − 1
𝑘= 0
= Probability of being in state 𝑗 in two steps, given we’re in state 𝑖 now.
i
j
Pij^2 =
Figure 24.
( P^2
) 𝑖 𝑗
.
Likewise, the 𝑛-wise product can be viewed by conditioning on the state 𝑘 after 𝑛 − 1 time steps:
( P 𝑛)𝑖 𝑗 =
𝑀Õ − 1
𝑘= 0
P 𝑛−^1
𝑖𝑘
= Probability of being in state 𝑗 in 𝑛 steps, given we are in state 𝑖 now.
24.5 Limiting Probabilities
We now move on to looking at the limit. Consider the (𝑖, 𝑗 )th entry of the power matrix P 𝑛^ for large 𝑛:
lim 𝑛→∞
( P 𝑛)𝑖 𝑗 ≡
lim 𝑛→∞
P 𝑛
𝑖 𝑗
This quantity represents the limiting probability of being in state 𝑗 infinitely far into the future, given that we started in state 𝑖.
Question: So what is the limiting probability of having zero umbrellas?
Answer: According to P^30 , it is 0.23.
Question: The fact that the rows of lim𝑛→∞ P 𝑛^ are all the same is interesting because it says what?
Mor Harchol-Balter. Introduction to Probability for Computing,
428 24 Discrete-Time Markov Chains: Finite-State
24.6 Stationary Equations
Question: Based only on what we have learned so far, how do we determine 𝜋 (^) 𝑗 = lim𝑛→∞ ( P 𝑛)𝑖 𝑗?
Answer: We take the transition probability matrix P and raise it to the 𝑛th power for some large 𝑛 and look at the 𝑗 th column, any row.
Question: Multiplying P by itself many times sounds quite onerous. Also, it seems one might need to perform a very large number of multiplications if the Markov chain is large. Is there a more efficient way?
Answer: Yes, by solving stationary equations, given in Definition 24.8.
Definition 24.8 A probability distribution 𝜋® = (𝜋 0 , 𝜋 1 ,... , 𝜋𝑀 − 1 ) is said to be stationary for the Markov chain with transition matrix P if
𝜋 ® · P = 𝜋® 𝑎𝑛𝑑
𝑀Õ − 1
𝑖= 0
Figure 24.7 provides an illustration of 𝜋® · P = 𝜋®.
π (^) 0 π (^) 1 π 2 P 00 P 01 P 02 π 0 π (^) 1 π 2
P 20
P 11 P 12
P 21 P 22
=
P 10
Figure 24.7 Visualization of 𝜋® · P = 𝜋® for the case of 𝑀 = 3 states.
Doing the row-by-column multiplication in Figure 24.7 results in the following stationary equations :
𝜋 0 · 𝑃 00 + 𝜋 1 · 𝑃 10 + 𝜋 2 · 𝑃 20 = 𝜋 0 𝜋 0 · 𝑃 01 + 𝜋 1 · 𝑃 11 + 𝜋 2 · 𝑃 21 = 𝜋 1 𝜋 0 · 𝑃 02 + 𝜋 1 · 𝑃 12 + 𝜋 2 · 𝑃 22 = 𝜋 2 𝜋 0 + 𝜋 1 + 𝜋 2 = 1.
These stationary equations can be written more compactly as follows: 𝑀Õ − 1
𝑖= 0
𝜋𝑖 𝑃𝑖 𝑗 = 𝜋 (^) 𝑗 , ∀ 𝑗 and
𝑀Õ − 1
𝑖= 0
Mor Harchol-Balter. Introduction to Probability for Computing,
24.7 The Stationary Distribution Equals the Limiting Distribution 429
Question: What does the left-hand side of the first equation in (24.1) represent?
Answer: The left-hand side represents the probability of being in state 𝑗 one transition from now, given that the current probability distribution on the states is 𝜋®. So (24.1) says that if we start out distributed according to 𝜋®, then one step later our probability of being in each state will still follow distribution 𝜋®. Thus, from then on we will always have the same probability distribution on the states. Hence, we call the distribution “stationary,” which connotes the fact that we stay there forever.
24.7 The Stationary Distribution Equals the Limiting
Distribution
The following theorem relates the limiting distribution to the stationary distribu- tion for a finite-state DTMC. Specifically, the theorem says that for a finite-state DTMC, the stationary distribution obtained by solving (24.1) is unique and rep- resents the limiting probabilities of being in each state, assuming these limiting probabilities exist.
Theorem 24.9 (Stationary distribution = limiting distribution) In a finite- state DTMC with 𝑀 states, let 𝜋 (^) 𝑗 = lim 𝑛→∞
( P 𝑛)𝑖 𝑗
be the limiting probability of being in state 𝑗 (independent of the starting state 𝑖 ) and let
𝜋 ® = (𝜋 0 , 𝜋 1 ,... , 𝜋𝑀 − 1 ), where
𝑀Õ − 1
𝑖= 0
be the limiting distribution. Assuming that 𝜋® exists, then 𝜋® is also a stationary distribution and no other stationary distribution exists.
Question: What’s the intuition behind Theorem 24.9?
Answer: Intuitively, given that the limiting distribution, 𝜋®, exists, it makes sense that this limiting distribution should be stationary, because we’re not leaving the limit once we get there. It’s not as immediately obvious that this limiting distribution should be the only stationary distribution.
Question: What’s the impact of Theorem 24.9?
Answer: Assuming that the limiting distribution exists, Theorem 24.9 tells us
Mor Harchol-Balter. Introduction to Probability for Computing,
24.7 The Stationary Distribution Equals the Limiting Distribution 431
Looking at the 𝑗 th entry of 𝜋®′^ in (24.2), we have:
𝑀Õ − 1
𝑘= 0
𝜋′ 𝑘 ( P 𝑛)𝑘 𝑗 = 𝜋′ 𝑗.
Taking the limit as 𝑛 goes to infinity of both sides, we have:
𝑛lim→∞
𝑀Õ − 1
𝑘= 0
𝜋′ 𝑘 ( P 𝑛)𝑘 𝑗 = (^) 𝑛lim→∞ 𝜋′ 𝑗 = 𝜋′ 𝑗.
We are now ready to prove that 𝜋′ 𝑗 = 𝜋 (^) 𝑗 , ∀ 𝑗 :
𝜋′ 𝑗 = (^) 𝑛lim→∞
𝑀Õ − 1
𝑘= 0
𝜋′ 𝑘 ( P 𝑛)𝑘 𝑗 =
𝑀Õ − 1
𝑘= 0
𝜋′ 𝑘 (^) 𝑛lim→∞ ( P 𝑛)𝑘 𝑗
𝑀Õ − 1
𝑘= 0
𝑀Õ − 1
𝑘= 0
Note that we were allowed to pull the limit into the summation sign in both parts because we had finite sums (𝑀 is finite).
One more thing: In the literature you often see the phrase “consider a stationary Markov chain,” or “consider the following Markov chain in steady state ...”
Definition 24.10 A Markov chain for which the limiting probabilities exist is said to be stationary or in steady state if the initial state is chosen according to the stationary probabilities.
Summary: Finding the limiting probabilities in a finite-state DTMC :
By Theorem 24.9, provided the limiting distribution 𝜋® = (𝜋 0 , 𝜋 1 , 𝜋 2 ,... , 𝜋𝑀 − 1 ) exists, we can obtain it by solving the stationary equations:
𝜋 ® · P = 𝜋® and
𝑀Õ − 1
𝑖= 0
where 𝜋® = (𝜋 0 , 𝜋 1 ,... , 𝜋𝑀 − 1 ).
Mor Harchol-Balter. Introduction to Probability for Computing,
432 24 Discrete-Time Markov Chains: Finite-State
24.8 Examples of Solving Stationary Equations
Example 24.11 (Repair facility problem with cost)
Consider again the repair facility problem represented by the finite-state DTMC shown again in Figure 24.8.
Working Broken
0.95 0.
Figure 24.8 Markov chain for the repair facility problem.
We are interested in the following type of question.
Question: The help desk is trying to figure out how much to charge me for maintaining my machine. They figure that it costs them $300 every day that my machine is in repair. What will be my annual repair bill?
To answer this question, we first derive the limiting distribution 𝜋® = (𝜋𝑊 , 𝜋𝐵) for this chain. We solve the stationary equations to get 𝜋® as follows:
𝜋 ® = 𝜋® · P , where P =
This translates to the following equations: 𝜋𝑊 = 𝜋𝑊 · 0.95 + 𝜋𝐵 · 0. 𝜋𝐵 = 𝜋𝑊 · 0.05 + 𝜋𝐵 · 0. 𝜋𝑊 + 𝜋𝐵 = 1.
Question: What do you notice about the first two equations above?
Answer: They are identical! In general, if 𝜋® = 𝜋® · P results in 𝑀 equations, only 𝑀 − 1 of these will be linearly independent (this is because the rows of P all sum to 1). Fortunately, the last equation above (the normalization condition) is there to help us out. Solving, we get 𝜋𝑊 = 89 and 𝜋𝐵 = 19.
Mor Harchol-Balter. Introduction to Probability for Computing,
434 24 Discrete-Time Markov Chains: Finite-State
24.2 Powers of transition matrix Given any finite-state transition matrix, P , prove that for any positive integer 𝑛, P 𝑛^ maintains the property that each row sums to 1.
24.3 Random walk on clique You are given a clique on 𝑛 > 1 nodes (a clique is a graph where there is an edge between every pair of nodes). At every time step, you move to a uniformly random node other than the node you’re in. You start at node 𝑣. Let 𝑇 denote the time (number of hops) until you first return to 𝑣. (a) What is E [𝑇]? (b) What is Var (𝑇)?
24.4 Card shuffling You have 𝑛 distinct cards, arranged in an ordered list: 1, 2, 3,... , 𝑛. Every minute, you pick a card at random and move it to the front of the ordered list. We can model this process as a DTMC, where the state is the ordered list. Derive a stationary distribution for the DTMC. [Hint: Make a guess and then prove it.]
24.5 Doubly stochastic matrix A doubly stochastic matrix is one in which the entries in each row sum up to 1, and the entries in each column sum up to 1. Suppose you have a finite-state Markov chain whose limiting probabilities exist and whose transition matrix is doubly stochastic. What can you prove about the stationary distribution of this Markov chain? [Hint: Start by writing some examples of doubly stochastic transition matrices.]
24.6 Randomized chess In chess, a rook can move either horizontally within its row (left or right) or vertically within its column (up or down) any number of squares. Imagine a rook that starts at the lower left corner of an 8 × 8 chess board. At each move, a bored child decides to move the rook to a random legal location (assume that the “move” cannot involve staying still). Let 𝑇 denote the time until the rook first lands in the upper right corner of the board. Compute E [𝑇] and Var (𝑇).
24.7 Tennis match [Proposed by William Liu] Abinaya and Misha are playing tennis. They’re currently tied at deuce, meaning that the next person to lead by two points wins the game. Suppose that Misha wins each point independently with probability 23 (where Abinaya wins with probability 13 ). (a) What is the probability that Misha wins the game? (b) What is the expected number of remaining points played until someone wins?
Mor Harchol-Balter. Introduction to Probability for Computing,
24.9 Exercises 435
24.8 Markovopoly [Proposed by Tai Yasuda] Suppose you are playing a board game where the board has 28 locations arranged as shown in Figure 24.10. You start at the “Go” square, and, at each turn, you roll a six-sided die and move forward in the clockwise direction whatever number you roll. However, the dark squares in the corners are jail states, and once you land there, you must sit out for the next three turns (for the next three turns, you stay in jail instead of rolling a die and moving). On the fourth turn, you can roll the die again and move. Your goal is to figure out the fraction of the turns that you are in jail. (You are “in jail” if you are in a jail square at the end of your turn.) Write stationary equations to determine this fraction.
Go
Markovopoly
Figure 24.10 Markovopoly for Exercise 24.8.
24.9 Axis & Allies In the game Axis & Allies, the outcome of a two-sided naval battle is decided by repeated rolling of dice. Until all ships on at least one side are destroyed, each side rolls one six-sided die for each of its existing ships. The die rolls determine casualties inflicted on the opponent; these casualties are removed from play and cannot fire (roll) in subsequent rounds. There are two types of ships: battleships and destroyers. For a battleship, a die roll of four or lower is scored as a “hit” on the opponent. For a destroyer, a die roll of three or lower is scored as a “hit” on the opponent. It takes two hits (not necessarily in the same round) to destroy a battleship and only one hit to destroy a destroyer. (Note: Battleships are twice as expensive as destroyers.) For example: Suppose side A has two destroyers and one battleship. Suppose side B has one destroyer and three battleships. Side A rolls two dice for its destroyers (rolling, say, 3 and 6) and one die for its battleship (rolling, say, 5). This means that side A generates one hit against side B.
Mor Harchol-Balter. Introduction to Probability for Computing,
24.9 Exercises 437
(a) How many states are there in this system? (b) How many absorbing states are there in this system, and what are they? Absorbing states are states that you never leave once you enter them. [Hint: What is 𝑛𝐼 for an absorbing state?] (c) Derive the transition probability from state (2, 1, 0) to (1, 1, 1). Be careful to think about all the ways that this transition can happen. Plug in the values of 𝑛𝐼 and 𝑛 and use 𝑝 = 0.5 so that your final answer is a constant. (d) Use a computer program to generate the entire transition matrix P. Assume that 𝑝 = 0.5. Print out the row corresponding to state (2, 1, 0). Now raise P to some very high power and watch what happens to row (2, 1, 0). You’ll want a high enough power that most of your entries are smaller than 0.01. What is the meaning of the row corresponding to state (2, 1, 0)? (e) The parameter 𝑝 can be thought of as a social distancing parameter, where lower 𝑝 represents better social distancing practices. Consider values of 𝑝 between 0 and 1. For each value of 𝑝, determine the expected fraction of the population who are left in the susceptible state when the outbreak is over (you will do this by conditioning on the probability of ending up in each absorbing state). Assume that you start in state (2, 1, 0). Your final output will be a graph with 𝑝 on the x-axis, but you can alternatively create a table with values of 𝑝 spaced out by 0.05.
