[PDF] [PDF] Chapter 8: Markov Chains

[PDF] 9 Markov Chains: Introduction

Markov Chains: A discrete-time stochastic process X is said to be a Markov Chain if it has the Markov Property: Markov Property (version 1): For any s, i0, ,in−1 ∈ S and any n ≥ 1, P(Xn = sX0 = i0, ,Xn−1 = in−1) = P(Xn = sXn−1 = in−1)

[PDF] Markov Chains - Department of Statistics and Data Science

Exercise 51 shows that E[β1X(0) = 0] = q(1 − p)/(q − p) − 1/p 1 6 Classification of States Depending on its transition probabilities, a Markov chain may visit 

[PDF] 5 Markov Chains

The process X is a Markov chain if it satisfies the Markov property: P(Xn+1 Show that every transition matrix on a finite state space has at least one closed 

[PDF] Markov chains

(i) Show that the Xn form a Markov chain (ii) Find its transition probabilities Solution (i) Fix a time n ≥ 1 Suppose that you know that Xn = x The goal is to show 

[PDF] Part IB - Markov Chains (Theorems with proof) - Dexter Chua

since P(X1 = · X0 = i) is a probability distribution function Theorem Let λ be a distribution (on S) and P a stochastic matrix The sequence X = (X0,X1, 

[PDF] Introduction to Stochastic Processes - University of Kent

Theorem 2 25 Let X denote an irreducible, positive recurrent Markov chain Then X has a unique stationary distribution Proof: Existence has been shown in 

[PDF] Chapter 8: Markov Chains

The transition diagram above shows a system with 7 possible states: state space Definition: The state space of a Markov chain, S, is the set of values that each

17 Markov Chains

Proof If the conditional distributions exist, then, by Theorem 17 9, the equation ( 17 4) is equivalent to X being a Markov chain Hence we only have to show that

[PDF] 01 Markov Chains

that X◦,X1,X2 ททท is a Markov chain with state space Z/n = {0,1,2,ททท ,n − 1} Show that the sequence of random variables Y◦,Y1,Y2,ททท where Yj = Sj 

[PDF] show that x is a random variable

[PDF] show that [0

[PDF] show the mechanism of acid hydrolysis of ester

[PDF] show time zone cisco

[PDF] show ∞ n 2 1 n log np converges if and only if p > 1

[PDF] shredded workout plan pdf

[PDF] shredding diet

[PDF] shredding workout plan

[PDF] shrm furlough

[PDF] shuttle paris

[PDF] si clauses french examples

[PDF] si clauses french exercises

[PDF] si clauses french practice

[PDF] si clauses french practice pdf

[PDF] si present

149

Chapter 8: Markov Chains

A.A.Markov

1856-19228.1

Introduction

So far, we have examined several stochastic processes using transition diagrams and First-Step Analysis.

The processes can be written as012,

wheretis the state at time.

On the transition diagram,tcorresponds to

which box we are in at step. In the Gambler"s Ruin (Section 2.7),tis the amount of money the gambler possesses after toss. In the model for gene spread (Section 3.7),tis the number of animals possessing the harmful allele A in generation. The processes that we have looked at via the transition diagram have a crucial property in common:t+1 depends only ont.

It does notdepend upon01t?1.

Processes like this are called

Markov Chains.

Example:Random Walk (see Chapter 4)

time tnone of these steps matter for time t+1 time t+1

In a Markov chain, the

future depends only upon the present:

NOT upon the past.

150
5 67 1 1 1 3 24 1 312
3 1 ...............13 515
1

5The text-book imageof a Markov chain hasa flea hopping about atrandom on the verticesof the transition diagram,according to the probabilities shown.The transition diagram above shows a system with 7 possible states:

state space=1234567

Questions of interest

Starting from state 1, what is the probability of ever reaching state 7? Starting from state 2, what is the expected time taken to reach state 4? Starting from state 2, what is the long-run proportion of time spent in state 3? Starting from state 1, what is the probability of being in state 2 at time ? Does the probability converge as , and if so, to what? We have been answering questions like the first two using first-step analysis since the start of STATS 325. In this chapter we develop a unified approach to all these questions using the matrix of transition probabilities, called the transition matrix. 151
8.2

Definitions

The Markov chain is the process012.

Definition:Thestateof a Markov chain at timeis thevalue oft. For example, ift= 6, we saythe process is in state6at time. Definition:Thestate spaceof a Markov chain,, is the set of values that each tcan take. For example,=1234567

Lethave size(possibly infinite).

Definition:Atrajectory

of a Markov chain isa particular set of values for 012. For example, if0= 1,1= 5, and2= 6, then the trajectory up to time = 2 is 156 More generally, if we refer to the trajectory0123, we mean that 0=0 ,1=1,2=2,3=3, ... 'Trajectory" is just a word meaning`path'.

Markov Property

The basic property of a Markov chain is thatonly the most recent point in the trajectory affects what happens next.

This is called theMarkov Property.

It means thatt+1dependsupont,butitdoesnotdependupont?1,10 152
We formulate the Markov Property in mathematical notation as follows:

P(t+1=t=tt?1=t?10=0) =P(t+1=t=t)

for all= 123and for all states01t.

Explanation:

P(t+1=t=t t?1=t?1t?2=t?21=10=0)

distribution oft+1depends ontbut whatever happened before time doesn't matter. Definition:Let012be a sequence of discrete random variables. Then

012is aMarkov chain

ifit satisfies the Markov property:

P(t+1=t=t0=0) =P(t+1=t=t)

for all= 123and for all states01t.

8.3The Transition Matrix

We have seen many examples oftransition diagramsto describe Markov chains. The transition diagram is so-called because it shows the transitions between different states.

0.6Hot

Cold

We can also summarize the probabilities

in amatrix: 02 08

06 04Hot

ColdtHot Cold

t+1 153
The matrix describing the Markov chain is called the transition matrix. It is the most important tool for analysing Markov chains.

Transition Matrix

list all states tlist all states t+1 insert probabilities ijrows add to 1? rows add to 1 The transition matrix is usually given the symbol= (ij)

In the transition matrix:

the ROWS represent NOW, or FROM (t); the COLUMNS represent NEXT, or TO (t+1); entry()is the CONDITIONAL probability that NEXT=, given that NOW =: the probability of going FROM stateTO state. ij=P(t+1=t=) Notes:1. The transition matrixmust listallpossible states in the state space.

2.is asquare matrix(), becauset+1andtboth take values in the

same state space(of size).

3. Therows

ofshould eachsum to 1: N j=1 ij=N j=1P(t+1=t=) =N j=1P {Xt=i}(t+1=) = 1 This simply states thatt+1musttake one of the listed values.

4. Thecolumns

ofdonotin general sum to 1. 154
Definition:Let012be a Markov chain with state space, where has size(possibly infinite). Thetransition probabilities of the Markov chain are ij=P(t+1=t=) for = 012

Definition:Thetransition matrix

of the Markov chain is= (ij)

8.4Example: setting up the transition matrix

We can create a transition matrix for any of the transition diagrams we have seen in problems throughout the course. For example, check the matrix below.

Example:Tennis game at Deuce.

VENUSWINS (W)VENUSAHEAD (A)

VENUS

BEHIND (B)

pq p p q qVENUSLOSES (L)DEUCE (D) 0 0 0 0 00 0 0 0

0 0 0 1 0

0 0 0 0 1

8.5Matrix Revision

Notation

col j a ijrow i N N byA

Letbe anmatrix.

We write= (ij),

i.e.comprises elementsij.

The () element ofis written both asijand ()ij:

e.g. for matrix2we might write (2)ij. 155

Matrix multiplication

Let= (ij) and= (ij)

bematrices.

The product matrix is=, with elements ()ij=N

k=1 ikkj

Summation notation for a matrix squared

Letbe anmatrix. Then

(2)ij=N k=1()ik()kj=N k=1 ikkj

Pre-multiplication of a matrix by a vector

Letbe anmatrix, and letπbe an1 column vector:π= 1... N

We can pre-multiplybyπTto get a 1row vector,

T=(πT)1(πT)N, with elements

(πT)j=N i=1 iij

8.6Thet-step transition probabilities

Let012be a Markov chain with state space=12.

Recall that the elements of the transition matrixare defined as: ()ij=ij=P(1=0=) =P(n+1=n=) for anyquotesdbs_dbs19.pdfusesText_25