Markov Processes
a measure on I
Definition 1. By a measure on I we mean a non-negative vector ν = (νi ∶i ∈ I), that is: 0 ≤ νi < ∞ for all i ∈ I. Note: When ν ≡/ 0 and Z ∶= ∑i∈I νi < ∞, it is possible to normalize ν to a distribution on I, by setting πi ∶= νi/Z, i ∈ I.
Exponential RV
Definition. Let 0 ≤ λ < ∞. We say that T ∼ Exp(λ), if P[T > t] = e^−λt for all t ≥ 0. (Note: we allow λ = 0, in which case T ≡ ∞.)
Compound Poisson process
Definition. The process (Xt)t≥0 is called a compound Poisson process, if it can be written in the form: Xt = ∑(i=1,Nt) Yi , t ≥ 0, where (Nt)t≥0 is a Poisson process, and Y1, Y2, . . . are i.i.d. RVs that are independent of (Nt)t≥0. Note that we interpret the sum as 0, on the event when {Nt = 0}. e.g. Customers leave a supermarket according to a Poisson process. Let Yi = amount spent by the i-th customer (in pounds). Then Xt = revenue of the supermarket by time t.
Theorem about PP(λ) definitions
Definitions (a), (b) and (c) of a Poisson process are equivalent.
Proposition. For large t, Xt is
approximately Norm(λtE[Y1], λtE[Y1²]) Don't get sketch proof
Why do we need non diagonal entried of Q to be non-negative
because the diagonal entries are non-positive and the row sum has to be 0
Let (Xn)n≥0 be a random process Fn =
collection of events determined by X0, . . . , Xn = "state of knowledge at time n" = "history up to time n" = σ(X0, . . . , Xn) There may be other equivalent ways to express Fn (with other events that are expressed in terms of Xi)
We call (Xt)0≤t≤T reversible (with respect to λ), if
for all T > 0, the time-reversal (X̂t)0≤t≤T is also Markov(λ, Q). Xt)t≥0 is reversible ⇐⇒ Q̂ = Q ⇐⇒ λ and Q are in detailed balance
A process (Mn)n≥0 is called adapted if...
for all n ≥ 0, the value of Mn only depends on X0, . . . , Xn (that is, only depends on events observed up to time n).
o(h) as h → 0
function of the variable h such that o(h)/h → 0 as h ↓ 0.
Definition 2. (i) We say that the state i ∈ I is transient,
if qi > 0 and Pi[Ti < ∞] < 1
Lemma 1. A compound Poisson process has ...
independent and stationary increments. Proof on pages 12-14
Theorem 1. (Betting on increments of a martingale) Suppose that (Xn)n≥0 is a martingale with respect to the filtration (Fn)n≥0. Let Bn−1 be a RV determined by Fn−1, n = 1, 2, . . ., and assume that ∣Bn−1∣ ≤ Kn−1 for some constants Kn−1, n = 1, 2, . . .. Let M0 be a RV determined by F0. Then Mn = M₀ + ∑(k=1,n)Bk−1(Xk − Xk−1), n ≥ 0,
is a martingale E[Mn+1 ∣ Fn] = E[Mn + Bn(Xn+1 − Xn) ∣ Fn] = Mn + Bn E[Xn+1 − Xn ∣ Fn] = Mn. Let us also check the two other requirements: Mn is determined by Fn, because it is determined by M0 and the terms Bk−1(Xk −Xk−1) for k = 1, . . . , n, and Bk−1 and Xk, Xk−1 are determined by Fn−1. Also, we have E∣Mn∣ ≤ E∣M0∣ +∑(k=1,n) E∣Bk−1∣∣Xk − Xk−1∣ ≤ E∣M0∣ +∑(k=1,n)Kk−1(E∣Xk∣ + E∣Xk−1∣) < ∞, so (Mn)n≥0 is integrable
work out what i have to write to show P(T<∞)=1
okie dokie
The mean M/M/1 queue length in equilibrium
one less than the mean of a Geom(1 − ρ) RV, so it is: ∑(i=0,∞)iνi = 1/(1 − ρ) − 1 = [1 − (1 − ρ)]/(1 − ρ)= ρ/(1 − ρ)=(λ/µ)/(1 −λ/µ)= λ/(µ − λ).
Open migration process
particles are also allowed to enter or leave the system "from the outside world". General setup: Let I = I ∪ {∆} = {1, . . . , J} ∪ {∆}. We call the newly added Node ∆ the "outside world" We call I the set of nodes for the open migration process, but note that we will not keep track of any particles in the outside world (imagine it has an infinite supply of particles, not influenced by the outside world) The state-space is: Ĩ∶= N J = {n = (n1, . . . , nJ ) ∶ n1, . . . , nJ ≥ 0} . We will need notation for particles being moved between all the nodes: Tj∆n = (n1, . . . , nj − 1, . . . , nJ ), j ∈ I; T∆kn = (n1, . . . , nk + 1, . . . , nJ ), k ∈ I; and as before: Tjkn = (n1, . . . , nj − 1, . . . , nk + 1, . . . , nJ ), j, k ∈ I. The Q-matrix Q̃ for the open migration process takes the form: q̃(n, Tjkn) = qjkφj(nj) j, k ∈ I,(as before); q̃(n, Tj∆n) = µjφj(nj) j ∈ I,(departures j → ∆); q̃(n, T∆kn) = λk k ∈ I,(arrivals ∆ → k) The Markov chain (X(t))t≥0 with Q-matrix Q̃ is called a closed (surely it should be open) migration process. Here X(t) = (X1(t), . . . , XJ (t)) ∈ N J = I, ̃ Xj(t) = # particles at Node j at time t.
Why do we need the diagonal entries of Q to be non-positive?
pii(t) = (I)ii + t(Q)ii + O(t²) = 1 + tqii + O(t²), so we need qii ≤ 0 for pii(t) ≤ 1.
When is the M/M/S queue transient and when is it recurrent?
we have for i > s the ratio νi+1/νi =λ/sµ. So the ratio test implies that if λ < sµ, then Z ∶= ∑i≥0 νi < ∞, and πi ∶= νi/Z is a stationary distribution Therefore, due to Theorem 2.5.2 and Example 2.3.1 the M/M/s queue is positive recurrent, when λ < sµ. When λ > sµ the M/M/s queue is transient. We can see this from the jump chain: for all i > s we have πi,i+1 =λ/λ + sµ 1/2 πi,i−1 =sµ/λ + sµ<1/2 Hence starting from state i = s+1, there is positive probability that the jump chain never returns to s + 1. Hence s + 1 is a transient state for the jump chain (but due to irreducibility, then all states are transient). By Theorem 2.4, transience of the jump chain implies transience of the M/M/s queue.
General form of the stationary distribution for an open migration process
πj(nj) ∶= bj πj(bar)^nj / ∏(r=1,nj) φj(r) , nj = 0, 1, 2, . . . , (*) where bj ∶= (∑(nj=0, ∞) πj(bar)^nj / ∏ (r=1,nj) φj(r) )^−1 . The normalisation constant bj ensures that (πj(nj) ∶ nj = 0, 1, 2, . . .) is a probability distribution. When the infinite sum diverges, we define bj = 0 (in this case it is not possible to normalise).
Traffic intensity
ρ ∶= λ/µ the traffic intensity (arrival rate to service rate ratio).
communicating classes for Q
the communicating classes for jump matrix Π.
An Exp(λ) RV
a continuous RV with probability density f(x) = λe^(−λx) if x ≥ 0; 0 if x < 0. It has mean 1/λ and variance 1/λ².
A Poisson(λ) RV
a discrete RV with probability mass function p(k) = e^−λ (λ^k)/k! , k = 0, 1, 2, . . . . It has mean λ and variance λ.
Poisson process definition (b) (infinitesimal definition)
(Xt)t≥0 has independent increments, X₀ = 0, and we have P[Xt+h − Xt = 0] = 1 − λh + o(h), as h → 0; P[Xt+h − Xt = 1] = λh + o(h), as h → 0; uniformly in t ≥ 0.
Poisson process definition (c) (transition probability definition)
(Xt)t≥0 has stationary and independent increments and for each t > 0 we have Xt ∼ Poisson(λt).
Let Q be an irreducible, non-explosive Q-matrix on I with invariant distribution λ Let (Xt)0≤t≤T be Markov(λ, Q). Set X̂t ∶= XT−t , 0 ≤ t ≤ T. Then
(X̂t)0≤t≤T is Markov(λ,Q̂), and Q̂ is also non-explosive. We call (X̂t)0≤t≤T the time-reversal of (Xt)0≤t≤T .
Definition. Let I be a countable set (finite or countably infinite). A Qmatrix on I is a matrix Q = (qij ∶ i, j ∈ I) satisfying:
(i) 0 ≤ −qii < ∞ for all i ∈ I; (ii) qij ≥ 0 for all i =/ j; (iii) ∑j∈I qij = 0 for all i ∈ I. We denote qi ∶= −qii, for short.
Theorem 1. Let (Xt)t≥0 be Markov(λ, Q). If any one of the following conditions hold, then (Xt)t≥0 does not explode:
(i) I is finite; or (ii) supi∈I qi < ∞; or (iii) X0 = i and i is a recurrent state for the jump chain.
Theorem 3. If (Xt)t≥0 is a birth process with rates (qj ∶ j ≥ 0), starting from 0, then:
(i) If ∑(j=0,∞)1/qj < ∞, then P[ζ < ∞] = 1 (explosion w.p. 1); (ii) If ∑(j=0,∞)1/qj = ∞, then P[ζ = ∞] = 1 (no explosion w.p. 1).
Wright-Fisher model
(i) Population stays constant so there are 2N alleles in the following generation. (ii) Each allele in the next generation is a random sample from the population of alleles in the current generation. That is, if the proportion of A and B alleles in the current generation are pA and pB (pA + pB = 1), then in the next generation each allele, independently, will be of type A or B with probabilities pA and pB, respectively. (No selective advantage to either)
M/M/1 queue stationary distribution
(i) We saw that the M/M/1 queue is positive recurrent when λ < µ (that is ρ < 1) and then the equilibrium queue length is given by: νi = (1 −λ/µ) (λ/µ)^i = (1 − ρ)ρ^i , i ≥ 0. Hence in equilibrium the queue length is a Geom(1 − ρ) RV minus 1. (The minus 1 is present because the usual definition of Geom(1 − ρ) requires the possible values to be 1, 2, 3, . . . , whereas in our case the possible values are 0, 1, 2, . . . .)
Let Q be an irreducible Q-matrix. The following are equivalent:
(i) every state is positive recurrent; (ii) some state i is positive recurrent; (iii) Q has an invariant distribution λ and it is non-explosive (for all i ∈ I, Pi[ζ = ∞] = 1). When (iii) holds, we have mi = 1/λiqi , i ∈ I.
Theorem 1. We have: (About recurrence and transience)
(i) if i is recurrent for the jump chain (Yn)n≥0, then i is recurrent for (Xt)t≥0; (ii) if i is transient for the jump chain (Yn)n≥0, then i is transient for (Xt)t≥0; (iii) every state is either recurrent or transient; (iv) recurrence and transience are class properties. Proof. (i) If i is recurrent for the jump chain (Yn)n≥0, then it is either because πii = 1 (in which case qi = 0), or otherwise πii = 0, and we have that Pi[Yn = i for some n ≥ 1] = 1. The latter implies that Pi[Ti <] = 1. (ii) If i is transient for the jump chain (Yn)n≥0, then necessarily qi > 0 (since πii < 1), and we have Pi[Yn = i for some n ≥ 1] < 1. The latter implies that Pi[Ti < ∞] < 1. (iii) This follows immediately from our definitions of recurrence and transience, as they cover all possibilities. (iv) The statement is true for discrete time, so this follows from parts (i) and (ii).
Definition 3. Among recurrent states we make the following distinction between two types of recurrence:
(iia) i is positive recurrent, if either qi = 0, or qi > 0 and mi ∶= Ei[Ti] < ∞; (iib) i is null recurrent, if qi > 0 and mi ∶= Ei[Ti] = ∞.
Give three equivalent formulations of the martingale property.
1. An adapted, integrable process (Mn)n≥0 is called a martingale, if for all n ≥ 0, and all events of the form: {X0 = x0, . . . , Xn = xn} ∈ Fn of positive probability, we have E[Mn+1 − Mn ∣ X0 = x0, . . . , Xn = xn] = 0. 2. Since all events in Fn are finite or countable unions of elementary events of the form {X0 = x0, . . . , Xn = xn}, the formulation above is equivalent to requiring that E[(Mn+1 − Mn)1A] = 0 (28) for all events A ∈ Fn. 3. A third formulation can be given by defining the conditional expectation of a RV Y given Fn. This is a random variable that is obtained by replacing Y by its average value on each elementary event {X0 = x0, . . . , Xn = xn}. That is, we define: E[Y ∣ Fn] = ∑(x0,...,xn)E[Y ∣ X0 = x0, . . . , Xn=xn]1{X0=x0,...,Xn=xn}. The value of E[Y ∣ Fn] gives the average value of Y , knowing the information in Fn. In this notation, the martingale property reads: E[Mn+1 ∣ Fn] = Mn, n ≥ 0. Important for calculations is the tower property: E[E[Y ∣ Fn]] = ∑(x0,...,xn)E[Y ∣ X0 = x0, . . . , Xn = xn]P[X0 = x0, . . . , Xn = xn] = E[Y ]. If (Mn)n≥0 is a martingale, we always have: E[Mn] = E[E[Mn+1 ∣ Fn]] = E[Mn+1]. Therefore, by induction, E[Mn] = E[M0]. That is, the average value of a martingale at a given time n equals its average initial value
(Linearity) For any RVs W1 and W2 we have: E[W1 + W2 ∣ Fn] =...
= E[W1 ∣ Fn] + E[W2 ∣ Fn].
birth process
A birth process is a generalization of the Poisson process where the rate at which we jump from state j to state j+1 is allowed to depend on the current state j, so it is given by a set of parameters 0 ≤ qj < ∞, j = 0, 1, 2, . . . . It is also convenient to allow the initial state X0 to be any state in {0, 1, 2, . . . }. More formally, the process is defined by the requirement, that conditional on X0 = i, the holding times S1, S2, . . . are independent exponential RVs with parameters qi , qi+1, qi+2, . . . , and the jump chain is Yn = i+n (recall this is the sequence of states visited).
Gambler's ruin
A gambler makes bets of £1 on the outcomes of fair coin tosses. If she guesses correctly, she wins £1, if not, she loses £1. She starts with £x0, where 0 < x0 < b, and her goal is to reach £b. She plays until either her fortune reaches £b, or she lost all her money. The fortune of the gambler can be modelled by the symmetric simple random walk (Xn)n≥0 started at x0: Xn = x0 +∑(i=1,n) Zi, n = 0, 1, 2, . . . , where Z1, Z2, . . . are i.i.d. with P[Z1 = 1] =1/2 = P[Z1 = −1]. We saw in Section 4.1 that Mn ∶= Xn is a martingale. The time when the gambler stops playing is T ∶= inf{n ≥ 0 ∶ Xn = 0 or Xn = b}. This is a stopping time, because {T = n} = {X0 ∈/ {0, b}, . . . , Xn−1 ∈/ {0, b}, Xn ∈ {0, b}}, and this event is in Fn. We check that condition (ii) of the OST, Theorem 1 applies. We have T < ∞ with probability 1, as can be seen in many ways. For example, (Xn)n≥0 is a Markov chain, and {0, b} can be reached from each state in the finite set: {1, . . . , b − 1}, so {0, b} is visited eventually. Whenever n ≤ T, we have ∣Mn∣ ≤ b =∶ C. Therefore Theorem 1(ii) gives: x0 = EM0 = EMT = 0P[XT = 0] + bP[XT = b] = bP[XT = b], and we deduce that the probability that the gambler reaches her goal is P[XT = b] = x0/b.
Genetic drift model
Assume the Wright-Fisher model Xn = # A alleles in the n-th generation, n ≥ 0. s ∶= P[Xn = 2N eventually] Here we assume X0 = x0, where x0 is a constant. (Note that once Xn = 0 or Xn = 2N is reached, the process remains at that value.) The evolution of (Xn)n≥0 is given as follows. Let p^(n)A∶= proportion of A alleles in the n-th generation = Xn/2N. Conditional on X0, . . . , Xn, the distribution of Xn+1 is given by: P[Xn+1 = k ∣ X0, . . . , Xn] = (2NChoosek)(p^(n)A))^k(1 − p^(n)A)^(2N−k) In other words, the conditional distribution of Xn+1 is Binom(2N, p(n)A). Fn ∶= events determined by X0, . . . , Xn = σ(X0, . . . , Xn). Then it is clear that Xn is determined by Fn. Also, E∣Xn∣ ≤ 2N < ∞, so (Xn)n≥0 is integrable. For the martingale property, note that the mean of a Binom(2N, p(n)A ) RV is 2N p^(n)A so E[Xn+1 ∣ Fn] = 2Np^(n)A = 2N Xn/2N = Xn. To apply OST: T = inf{n ≥ 0 ∶ Xn = 0 or Xn = 2N}. This is a stopping time, for reasons seen earlier. We show that T < ∞ with probability 1. This is because as long as 0 < Xn < 2N, at each step, we have probability ≥ (p^(n)A) 2N ≥ (1/2N)^2N > 0 that all alleles will be A in the next step, irrespective of the past history. This implies that eventually we must have Xn ∈ {0, 2N}. The martingale is bounded up to stopping: ∣Xn∣ ≤ 2N whenever n ≤ T. From the Optional Stopping Theorem we get x0 = E[X0] = E[XT ] = 0P[XT = 0] + 2N P[XT = 2N] = 0(1 − s) + 2Ns = 2Ns. (36) That is: s=x0/2N
define e^B for a finite square matrix B
B ∶=∑(n=0,∞) Bⁿ/n! = I + B +B²/2! +B³/3! + . . . . See Probability 2B or [2, Chapter 2] for the proof of convergence
mean length of busy period in an M/M/1 queue
By a busy period we mean a maximal time period during which there is at least one customer in the system. We can compute the mean length of a busy period from the relation: m0 = mean return time to 0 = mean time taken to leave 0 + mean length of busy period = 1/q0 + mean length of busy period. This gives: mean length of busy period = m0 −1/q0 = 1/ν0q0 −1/q0 =µ/[(µ − λ)λ] −1/λ =[µ − (µ − λ)]/[(µ − λ)λ]= 1/(µ − λ).
Theorem 1. (Markov property)
Let (Xt)t≥0 be a PP(λ). Then for any s ≥ 0, the process (Xt+s − Xs)t≥0 is also a PP(λ) and is independent of (Xr ∶ r ≤ s). Proof: page 18/19, worth another look.
Learn answers to PS9 Q2
CCAAAAAAAN DOOOO!
The exponential martingale
Consider an asymmetric simple random walk: Xn = x0 +∑(i=1,n) Zi, n ≥ 0, where Z1, Z2, . . . are i.i.d. with P[Z1 = 1] = p, P[Z1 = −1] = 1 − p, with 0 < p < 1, p ≠ 1/2. In this case, an exponential function yields a useful martingale. In order to find it, let us fix a number β ∈ R. We calculate: E[e^βXn+1∣ Fn] = E[e^β(Xn+Zn+1)∣ Fn] = E[e^βXn e^βZn+1∣ Fn] = e^βXn E[e^βZn+1∣ Fn] = e βXn E[e^βZn+1] = e^βXn[e^βp + e^−β(1 − p)] . The computation shows that if we can choose β in such a way that e^βp + e^−β(1 − p) = 1 then Mn = eβXn is a martingale, because we have E[Mn+1 ∣ Fn] = Mn Mn. Writing y = e^β, we need to solve the equation: yp + 1/y (1 − p) = 1 py² − y + (1 − p) = 0 y =[1 ± √1 − 4p(1 − p)]/2p = [1 ± ∣2p − 1∣]/2p. Hence we have the roots: y = 1 and y = (1−p)/p. The root y = 1 gives the uninteresting martingale Mn ≡ 1, but the other root y = (1−p)/p = e^β gives something non-trivial: Mn = ((1 − p)/p)^Xn is a martingale. For completeness, note that Mn is clearly adapted, and E∣Mn∣ ≤ max{( (1−p)/p)ⁿ, ((1−p)/p)⁻ⁿ} < ∞.
Consequences of time-reversal for open migration processes.
Corollary. For an open migration process in equilibrium, the departure processes from each node j ∈ I to Node ∆ are independent Poisson processes with rates: ̂λj = πjµj.
Mean of a compound poisson process Xt
E[Xt∣ Nt = n] = E[∑(i=1,Nt)Yi∣Nt = n] = E[∑(i=1,n))Yi∣Nt = n] (Y1,Y2,... indep. of Nt) = E[∑(i=1,n)Yi] = nE[Y1]. Therefore E[Xt∣ Nt] = (EY1)Nt . It follows that E[Xt] (tower property of cond. expectation) = E[E[Xt∣ Nt]] = E[(EY1)Nt] = (EY1)E[ Nt ¯ Poisson(λt) RV ] = λt ⋅ (EY1).
2. Suppose that Z is independent of X0, . . . , Xn. Then E[Z ∣ Fn] = ...
E[Z].
We say that (Mn)n≥0 is integrable if...
E∣Mn∣ < ∞ for all n ≥ 0.
How to compute P(t) from the Q-matrix?
Find the eigenvalues of Q (Note that there is always a factor x due to the row sums of Q being 0) Diagonalize Q with an invertible matrix U and the matrix of eigenvalues on the diagonal e.g. Q = U * (0 0 0 0 −2 0 0 0 −4) * U^−1 =∶ UDU^−1 Q^k = UD^kU^−1 e^tQ = ∑(k=0,∞)(tQ)^k/k! =U(∑(k=0,∞)t^k/k! * (0^k00 0(-2)^k)0 00(-4)^k)* U^-1 = U* (100 0e^-2t0 00e^4t) Since the entries of U and U^−1 are simply constants, we have: p₁₁(t) = (e^tQ)₁₁ = a + be−2t + ce−4t The constants can be determined from the derivatives of P(t) at t = 0:# 1 = p11(0) = a + b + c −2 = q11 = p′11(0) = −2b − 4c 7 = (Q^2)11 = p′′11(0) = 4b + 16c Solving for a, b, c we get: a = 3/8, b = 1/4, c = 3/8.
1.6 Poisson process in R^d
For a set B ⊂ R^d, we denote by vol(B) its d-dimensional volume. Given a countable set of points V ⊂ R^d , we denote by #(V ∩B) the number of points of V that fall in B. A random collection of points V ⊂ R^d is called a Poisson process in R^d of intensity λ, if: (i) for any subset B ⊂ R^d with 0 < vol(B) < ∞, we have #(V ∩ B) ∼ Poisson(λvol(B)); (ii) for any pairwise disjoint sets B1, . . . , Bn ⊂ R^d , the n RVs #(V ∩ B1), . . . , #(V ∩ Bn) are independent.
Sums of independent Poisson Processes Theorem 1.
If (Xt)t≥0 is a PP(λ) and (Yt)t≥0 is a PP(µ), and if the two processes are independent, then (Xt + Yt)t≥0 is a PP(λ + µ). Proof. Let Zt = Xt + Yt . We use the infinitesimal definition (b). We first check that (Zt)t≥0 has independent increments. For this, observe that if 0 ≤ s1 ≤ t1 ≤ ⋅ ⋅ ⋅ ≤ sn ≤ tn, then the 2n RVs Xt1 − Xs1 , . . . , Xtn − Xsn , Yt1 − Ys1 , . . . , Ytn − Ysn are independent (use that both the X-process and the Y -process has independent increments and that they are independent of each other). It follows that the n RVs Zt1 − Zs1 = (Xt1 − Xs1) + (Yt1 − Ys1) ⋮ Ztn − Zsn = (Xtn − Xsn) + (Ytn − Ysn) are independent. Hence, (Zt)t≥0 has independent increments. Next we check the other two requirements in the infinitesimal definition, using what they say about the X-process and the Y -process. First we have: P[Zt+h − Zt = 0] = P[Xt+h − Xt = 0, Yt+h − Yt = 0] indep. assump. = P[Xt+h − Xt = 0]P[Yt+h − Yt = 0] = (1 − λh + o(h))(1 − µh + o(h)) = 1 − (λ + µ)h + o(h), as h ↓ 0. Second we have: P[Zt+h − Zt = 1] = P[Xt+h − Xt = 1, Yt+h − Yt = 0] + P[Xt+h − Xt = 0, Yt+h − Yt = 1] indep. assump. = P[Xt+h − Xt = 1]P[Yt+h − Yt = 0] + P[Xt+h − Xt = 0]P[Yt+h − Yt = 1] = (λh + o(h))(1 − µh + o(h)) + (1 − λh + o(h))(µh + o(h)) = (λ + µ)h + o(h), as h ↓ 0. Therefore (Zt)t≥0 is a PP(λ + µ).
Definition. Let Q be a Q-matrix on I, and Π its associated jump matrix. Let λ be an initial distribution on I. We define the jump chain (Yn)n≥0 to be a discrete-time Markov chain on I with transition matrix Π and initial distribution λ. We will write Markov(λ,Π) to denote such a Markov chain. Conditional on (Yn)n≥0, the holding times S1, S2, . . . are defined to be independent exponential RVs with rates q(Y0), q(Y1), q(Y2), . . . . The continuous-time Markov chain (Xt)t≥0 is then defined as follows:
J0 ∶= 0, Jn ∶= S1 + ⋅ ⋅ ⋅ + Sn, n ≥ 1. We set: Xt ∶= Yn if Jn ≤ t < Jn+1; ∞ otherwise (The second possibility can only occur if there is explosion, otherwise it is vacuous.) We say that the process (Xt)t≥0 is Markov(λ, Q).
Detailed balance lemma 1
Lemma 1. We have νQ = 0 ⇐⇒ for all j ∈ I: ∑(i∈I, i≠j) νjqji = ∑(i∈I, i≠j) νiqij . (balance equations) Proof of Lemma 1. Start with the right hand statement in the balance equations. We rewrite the left hand side as: ∑(i∈I, i≠j) νjqji = νjqj = −νjqjj. Bringing this term to the other side gives 0 = νjqjj + ∑(i∈I, i≠j) νiqij = ∑(i∈I)νiqij = (νQ)j . So the right hand statement is equivalent to νQ = 0, which is the left-hand statement. When detailed balance holds, it is usually much easier to find the invariant distribution ν by solving the detailed balance equations, than solving the equations νQ = 0.
Optional stopping theorem
Let (Mn)n≥0 be a martingale and let T be a stopping time. Suppose that at least one of the following conditions hold: (i) T ≤ n with probability 1 for some fixed 0 ≤ n < ∞. (ii) T < ∞ with probability 1 and for some constant C we have ∣Mn∣ ≤ C whenever n ≤ T. Then EMT = EM0. proof on page 61
Theorem 3. (Strong Markov property)
Let (Xt)t≥0 be a PP(λ), and let T be a stopping time of (Xt)t≥0. Conditional on {T < ∞}, the process (Xt+T − XT )t≥0 is also a PP(λ) and is independent of (Xr ∶ r ≤ T).
The colouring theorem.
Let (Xt)t≥0 be a PP(λ). Let 0 < p < 1. Colour each jump time (event time) of (Xt)t≥0, independently of each other either red or white, with probabilities p and 1 − p, respectively. For example, the jumps / events of (Xt)t≥0 may represent the visits of birds at a bird feeder, where each visiting bird is, independently, red with probability p, or white with probability 1 − p. Let Ytred = # of red events in [0, t] (# red birds during [0, t]) Ytwhite = # of white events in [0, t] (# white birds during [0, t]). Theorem 1. (Ytred)t≥0 is a PP(λp), (Ytwhite)t≥0 is a PP(λ(1 − p)), and the two processes are independent. Sketch of the proof. We use the transition probability definition (c). Consider first the increments over the time interval (0, t]. We have: P[Ytred = r, Ytwhitet = w] = P[Xt = r + w, Ytred = r] = P[Xt = r + w]P[Y tredt = r ∣ Xt = r + w] = e^−λt ((λt)^r+w)/ (r + w)! ⋅ (r + w choose r)p^r(1 − p)^w = e^−λpt * e^−λ(1−p)t(λt)^r (λt)^w/(r + w)! (r + w)!/r!w! p^r (1 − p)^w = e^−λpt ((λpt)^r)/r! ⋅ e^−λ(1−p)t ((λ(1 − p)t)^w)/w! (3) The right hand side is the product of the probabilities for a Poisson(λpt) RV to take the value r and a Poisson(λ(1 − p)t) RV to take the value w. This shows that for any t ≥ 0 the RVs Ytred and Ytwhite t are independent Poisson RVs with parameters λpt and λ(1 − p)t. A computation similar to (3) can be carried out for the increments over intervals (0, t1], . . . , (tn−1, tn] for 0 < t1 < ⋅ ⋅ ⋅ < tn, showing that the two processes are independent, and (Ytred)t≥0 is a PP(λp), (Ytwhite)t≥0 is a PP(λ(1−p)) (this is a part we do not detail here).
Jump times are "as random as possible" Theorem 1.
Let (Xt)t≥0 be a PP. Conditional on there being exactly one jump in [s, s+t], that jump occurs at a time uniformly distributed in [s, s+t]. It can also be shown (see [2, Theorem 2.4.6]), that conditional on there being exactly n jumps in [s, s + t], those jumps occur at times distributed as n independent uniform points in [s, s+t] (taken in increasing order). Proof of Theorem. We use the transition probability definition (c). Stationarity of increments means that we may assume s = 0. Let 0 ≤ u ≤ t. We have P[J1 ≤ u ∣ Xt = 1] = (P[J1 ≤ u, Xt = 1])/P[Xt = 1] = (P[Xu = 1, Xt − Xu = 0])P[Xt = 1] = (P[Xu = 1]*P[Xt − Xu = 0])/P[Xt = 1] = ((λu)e^−λu ⋅ e^−λ(t−u))/(λt)e^−λt =u/t. The computation shows that the distribution of the time of the jump is uniform on [0, t], as required.
A continuous-time random process
Let I be a countable set, called the state space. A continuous-time random process (Xt)t≥0 = (Xt∶ 0 ≤ t < ∞) is a family of random variables Xt ∶ Ω → I We restrict attention to rightcontinuous processes, i.e. for all ω ∈ Ω and for all t ≥ 0 there exists ε > 0, such that Xs(ω) = Xt(ω) for t ≤ s ≤ t + ε. e let J₀ = 0, and Jn = time of the n-th jump Sn = Jn − Jn−1 = n-th holding time, n ≥ 1, Yn = XJn = n-th state visited = "jump process / jump chain"
binomial model for asset prices
Mn = price of stock at time n, n = 0, 1, 2, . . . . We assume that every time step, Mn either increases or decreases by a fixed factor (1 + α), with fixed probabilities. That is, Mn+1 = (1 + α)Mn with probability p and Mn+1 = 1/(1+α) Mn with probability 1 − p. A common model used in mathematical finance is when p is such that (Mn)n≥0 is a martingale. This requires that (Mn)n≥0 be the exponential martingale of Section 4.9, and so (1 − p)/p = e^β = 1 + α. i.e. 1 − p = p + αp 1 = 2p + pα p =1/(2 + α).
Important note about PP(λ)
Note that a PP(λ) is not a single RV ! It is a family of RVs, indexed by time: for each time t ≥ 0 we have a random variable Xt . A lot of mistakes on problems arise from failing to make the distinction.
Consequence 1 of the reversibility of the M/M/S queue
Observe that when we reverse time, an upward (resp. downward) jump in the queue length turns into a downward (resp. upward) jump. Therefore, if A ∶= set of arrival times for (Xt)0≤t≤T ; D ∶= set of departure times for (Xt)0≤t≤T , then  ∶= set of arrival times for (X̂t)0≤t≤T = D; D̂ ∶= set of departure times for (X̂t)0≤t≤T = A. (14) That is: the departure process D becomes the arrival process  for the time-reversal. The latter is a PP(λ), and we get the following consequence: Consequence 1: For an M/M/s queue in equilibrium (1 ≤ s ≤ ∞), the departure process is a PP(λ).
semi-group property of P(t) = e^tQ
P(t + s) = P(t)P(s) for all s, t ≥ 0. This can be checked from e (s+t)Q = e^sQ+tQ, and (sQ + tQ)ⁿ = ∑(k=0,n)(n choose k) s^kQ^kt^(n−k)Q^(n−k). As for scalar exponentials, this implies that e^sQ+tQ = e^sQe^tQ. Some caution is necessary with matrix exponentials: when A and B do not commute, it can happen that e A+B ≠ e^A e^B.
Why do we need the row sums of Q to be 0?
P(t) = e^tQ =∑(n=0,∞)tⁿQⁿ/n! P(t) = I + tQ + O(t²), o, since I already has row sums equal to 1, Q has to have row sums equal to 0, for P(t) to have row sums equal to 1. if Q has row sums equal to 0, it is easy to check that so do all higher powers of Q, so this condition, due to the series (7) is also sufficient to ensure that P(t) has row sums equal to 1.
i being recurrent for the jump chain means
P[Yn = i for some n ≥ 1 ∣ Y0 = i] = 1.
Theorem 1. Let S1, S2, . . . be independent RVs, Sn ∼ Exp(λn), 0 < λn < ∞. (i) If ∑(n=1,∞)1/λn < ∞,... (ii) If ∑(n=1,∞)1/λn = ∞,...
P[∑(n=1,∞) Sn < ∞] = 1. P[∑(n=1,∞) Sn = ∞] = 1. Proof of Theorem 1. (i) E[∑(n=1,∞)Sn] = (MCT) ∑(n=1,∞) E[Sn] = ∑(n=1,∞) 1/λn < ∞. Since ∑n Sn is a non-negative RV, if it has finite expectation, it is finite with probability 1. So we get P[∑(n=1,∞) Sn < ∞] = 1. (ii) We have ∏(n=1,∞) (1 +1/λn)) ≥ 1 +∑(n=1,∞) 1/λn = ∞. Therefore, we have E[exp {−∑(n=1,∞)Sn}] = E[∏(n=1,∞)exp(−Sn)] = (independence + MCT) ∏(n=1,∞)E[e^−Sn] Calc. = ∏(n=1,∞)1/(1 + 1/λn) = 0. Here in Step "Calc.", we used thatE[e^−Sn] = ∫(0,∞)e^−x λe^−λx dx = λ/(1 + λ)= 1/(1 +1/λ). The non-negative RV exp{− ∑n Sn} can only have 0 expectation, if it equals 0 with probability 1. So we get P[∑(n=1,∞) Sn = ∞] = 1.
Theorem 3 (Convergence to equilibrium). If Q is irreducible and positive recurrent, then for all states i, j ∈ I we have
Pi[Xt = j]→ λj as (t→∞)=1/mjqj In view of the above convergence, the invariant distribution λ is also called the equilibrium distribution. Yet another name for it is stationary distribution (in view of the fact that λ = λP(t) implies that if (Xt)t≥0 is Markov(λ, Q) then P[Xt = i] = λi for all i ∈ I, t ≥ 0).
Let Q be an irreducible, non-explosive Q-matrix on I with invariant distribution λ Define the matrix Q̂ = (q̂ij ∶ i, j ∈ I) by requiring that λjq̂ji = λiqij for all i, j ∈ I. (13) Then
Q̂ is also a Q-matrix, it is also irreducible, and also has invariant distribution λ. Then Q̂ is also a Q-matrix, it is also irreducible, and also has invariant distribution λ. Proof. We first check the requirements for Q̂ to be a Q-matrix: (i) 0 ≤ −q̂ii < ∞ holds, because q̂ii = qii. (ii) q̂ij ≥ 0 holds, because λi , λj > 0 and qji ≥ 0. (iii) For the row-sums of Q̂ we have: ∑(i∈I)q̂ji =1/λj ∑(i∈I)λiqij =1/λj (λQ)j = 0. (Since λQ=0) Regarding irreducibility, we observe that Q̂ allows a jump from i to j if and only if Q allows a jump from j to i: due to (13) we have q̂ij > 0 if and only if qji > 0. Since Q was assumed irreducible, for any pair of states i, j ∈ I, i ≠ j, Q allows some sequence of jumps from j to i. The reversal of this sequence is then allowed for Q̂, so Q̂ allows a sequence of jumps from i to j. Therefore Q̂ is irreducible. Finally, invariance of λ follows from: (λQ̂)i = ∑(j∈I) λjq̂ji (13) = ∑(j∈I) λiqij = λi ∑(j∈I)qij row sum of Q = 0.
Theorem 4 (Ergodic theorem)
Suppose Q is irreducible, and ν is any initial distribution. Let (Xt)t≥0 be Markov(ν, Q). Then with probability 1 1/t ∫(0,t) 1{Xs=i} ds →1/miqi as t→∞. In the positive recurrent case, for any bounded function f ∶ I → R, with probability 1, 1/t ∫(0,t) f(Xs) ds → ¯f ∶= ∑(i∈I) λif(i) as t→∞ where λ is the invariant distribution.
Memoryless property
T is an exponential RV ⇐⇒ P[T > t + s ∣ T > s] = P[T > t], ∀s, t ≥ 0.
Time reversal of an open migration theorem 1.
The time-reversal of an open migration process in equilibrium, with single particle motion Q is an open migration process with single particle motion ̂Q, and the same population pressures at each node. We have ̂Q(bar) = (̂q(bar)jk ∶ j, k ∈ I) defined by ̂q(bar)jk = πk/πj q(bar)kj , j, k ∈ I = I ∪ {∆}. Proof on page 54
Theorem 1. The following are equivalent The following theorem gives an easy way to translate between invariance with respect to Q (continuous time) and invariance with respect to Π (discrete time).
Theorem 1. The following are equivalent: (i) λ is an invariant measure for Q; (ii) µΠ = µ, where µi = λiqi, i ∈ I. Proof. We compute: (µΠ − µ)j = ∑(i∈I)µiπij − µj = ∑(i∈I, i≠j)λiqi qij/qi − λj qj = ∑(i∈I, i≠j) λiqij + λjqjj = ∑(i∈I)λiqij = (λQ)j . Hence we have LHS ≡ 0 if and only if RHS ≡ 0
Which flows between nodes in an open migration process are Poisson processes?
The crucial observation is that each communicating class behaves as an open migration so leaving and arriving at a nodes in a communicating class are Poisson processes but movements between nodes in a communicating class are not.
the filtration of the process (Xn)n≥0
The growing family (Fn)n≥0
Poisson process definition (a) (jump chain / holding time definition)
The process (Xt)t≥0 with values in I = {0, 1, 2, . . . } is a Poisson process with rate λ (with 0 < λ < ∞ a fixed parameter), if S₁, S₂, . . . are i.i.d. Exp(λ) RVs, and Yn = n. We will shorten this as: (Xt)t≥0 is PP(λ)
first passage time to state
Ti ∶= inf{t ≥ J1 ∶ Xt = i}. The reason for only looking at t after the first jump is that when X0 = i, we want Ti to be equal to the first return time to state i (and not equal to 0).
Conditional variance formula
Var(X ∣ Y = y) = E[(X − E[X ∣ Y = y])²∣ Y = y] = E[X²∣ Y = y] − (E[X ∣ Y = y])²
Lemma 1. For any random variables X and Y , such that X has finite variance, we have:
Var(X) = E[Var(X ∣ Y )] + Var(E[X ∣ Y ]) Proof. On the one hand: E[Var(X ∣ Y )] = E[E[X²∣ Y ] − (E[X ∣ Y ])²] = E[E[X²∣ Y ]] − E[E[X ∣ Y ]²]. (4) On the other hand: Var(E[X ∣ Y ]) = E[E[X ∣ Y ]²] − (E[E[X ∣ Y ]])²= E[E[X ∣ Y ]²] − (E[X])². Adding (4) and (5) we get on the right hand side: E[X²] − (E[X])² = Var(X).
Variance of a compound poisson process Xt
Var(Xt∣ Nt = n) = Var(∑(i=1,Nt)Yi∣Nt = n) = Var(∑(i=1,n)Yi∣Nt = n) (Y1,Y2,... indep. of Nt) = Var(∑(i=1,n)Yi) (Y1,Y2,... are i.i.d). = nVar(Y1). This shows that Var(Xt∣ Nt) = Var(Y1) ⋅ Nt. Using the conditional variance formula we get: Var(Xt) = E[Var(Xt∣ Nt)] + Var(E[Xt∣ Nt]) = E[Var(Y1) ⋅ Nt] + Var(E[Y1] ⋅ Nt) = Var(Y1) ⋅ E[Nt]+(E[Y1])²⋅ Var(Nt) = λt (Var(Y1) + (E[Y1])²) = (λt) ⋅ E[Y²1].
Consequence 2 of the reversibility of the M/M/S queue
We can say more. Let us write A _‖_ B to denote that the random objects A and B (events or RVs) are independent. Consider some 0 < T0 < T, and let T̂0 = T − T0. Then we have: (Xt)0≤t≤T0 _‖_ A ∩ [T0, T], since arrivals occurring after time T0 do not influence the behaviour of the queue up to time T0. Due to (14), we can rewrite this as: (X̂t)T̂0≤t≤T _‖_ D̂ ∩ [0, T̂0]. 38 But since (X̂t)0≤t≤T is also an M/M/s queue in equilibrium this implies the following consequence: Consequence 2: For an M/M/s queue in equilibrium (1 ≤ s ≤ ∞), the behaviour of the queue after any fixed time t0 is independent of the departures that occurred before time t0. In particular, we have the surprising feature that the queue length at any given time t0 is independent of past departures (that occurred up to time t0). In particular, we have the surprising feature that the queue length at any given time t0 is independent of past departures (that occurred up to time t0).
General setup for a closed migration process.
We have J ≥ 2 nodes, labelled by the set I ∶= {1, . . . , J}. The possible jumps are encoded by a Q-matrix Q on I. If there is a single particle, N = 1, then I is the statespace, and Q describes the motion of that single particle on the network. Suppose now the general case that there are N ≥ 1 particles. Then the state-space is: SN ∶={n ∈ N^J ∶ ∑(j=1,J)nj = N}. We assume that service rates are of the following general form: when there are nj particles at Node j, service is provided at rate qjφj(nj), where the function φj satisfies: φj(0) = 0 (no jump is possible from Node j if nj = 0), and φj(nj) > 0 for nj ≥ 1 (a jump is always possible when there is at least one particle). After completing service at Node j, a particle moves to Node k with probability qjk/qj . The functions φj is called the population pressure at Node j. It expresses the influence of the number of particles present at the node on the total service rate. Denote Tjkn ∶= (n1, . . . , nj−1, nj −1, . . . , nk +1, . . . , nJ ), when n ∈ SN , nj ≥ 1. The entries of Q̃ are given by: q̃(n, Tjkn) ∶= qjkφj(nj), j, k ∈ I, j =/ k, n ∈ SN , nj > 0 That is, the only possible transitions are that one particle moves from some Node j to some Node k, the rate for this transition being qjkφj(nj). Write X(t) = (X1(t), . . . , XJ (t)), t ≥ 0 for the Markov chain on SN with Q-matrix Q̃. Then (X(t))t≥0 is called a closed migration process.
When and why is the M/M/S queue reversible?
We saw: if 1 ≤ s ≤ ∞ and λ < sµ, then the M/M/s queue has an invariant distribution π = (π ∶ i ≥ 0) satisfying detailed balance: πiqij = πjqji, for all i, j ∈ I = {0, 1, 2, . . . }. It follows that Q̂ = Q. Therefore, if (Xt)0≤t≤T is an M/M/s queue in equilibrium, then due to Theorem 2.8, the time-reversal X̂t = XT−t , 0 ≤ t ≤ T is also an M/M/s queue in equilibrium. This has some rather non-trivial consequences
stopping time
We say that a RV T ∶ Ω → [0,∞] is a stopping time for the process (Xt)t≥0, if for all t ≥ 0, the event {T ≤ t} only depends on (Xs ∶ s ≤ t).
Independent incremenets
We say that a process (Xt)t≥0 has independent increments, if for all choices of 0 ≤ s1 ≤ t1 ≤ s2 ≤ t2 ≤ ⋅ ⋅ ⋅ ≤ sn ≤ tn the n RVs Xt1 − Xs1 (increment over the interval (s1, t1]), . . . , Xtn − Xsn (increment over the interval (sn, tn]) are independent. Here Xti − Xsi is the increment of the process over the interval (si , ti].
Stationary increments
We say that a process (Xt)t≥0 has stationary increments, if the distribution on Xt − Xs only depends on t − s.
What does it mean for the birth process to explode/not explode?
We say that the process explodes, when ζ < ∞ and it does not explode, when ζ = ∞ where the RV ζ ∶= S1 + S2 + ⋅ ⋅ ⋅ = "explosion time"
Finding the expexted duration of the gambler's game
We want to use the quatratic martingale since the OTS would give us x₀² = E[M₀] = E[MT ] = E[X²T] − E[T] and we could derive E[T] from this. However, neither of the conditions of the OST hold. The RV T is not bounded by a constant, and although T < ∞ with probability 1, ∣Mn∣ is not bounded for n ≤ T: although X²n ≤ b², the negative term −n can make the absolute value ∣X²n − n∣ arbitrarily large. Instead we fix x a large finite time N, and replace T by the stopping time TN ∶= min{N, T}. That is, the stopping rule TN says that the gambler will stop playing either if she reaches b, or of she is ruined, or by time N, whichever happen sooner It is easy to see that TN is indeed a stopping time. Since we have TN ≤ N with probability 1, the OST does apply to TN , and gives: x₀² = E[M₀] = E[MTN] = E[X²TN] − E[TN ]. E[X²TN] = 0² P[XT = 0, T ≤ N] + b² P[XT = b, T ≤ N] + E[X²N 1T>N ]. Now we let N → ∞. The left hand side converges to E[T] (by the Monotone Convergence Theorem) The second term in the right hand side of (34) converges to P[XT = b] =x0/b. The third term goes to 0, because 0 ≤ X²N ≤ b² whenever T > N: ∣E[X²N 1T>N ]∣ ≤ b² P[T > N] → 0, as N → ∞. This gives the formula: E[T] = −x₀² + b² x₀/b = x0(b − x0). Therefore, the expected duration of the gambler's game is x0(b − x0).
compare the M/M/∞ and M/M/s queues by realizing them as coordinates of a single Markov chain. Consider the Q-matrix Q on the state space I = {(i, i′ ) ∶ 0 ≤ i ≤ i ′ < ∞} given by the following jump rates.
When 0 ≤ i ≤ i′ < s: (i, i′) → (i + 1, i′ + 1) at rate λ; → (i − 1, i′ − 1) at rate iµ; → (i, i′ − 1) at rate (i′ − i)µ; When 0 ≤ i < s ≤ i′ < ∞: (i, i′) → (i + 1, i′ + 1) at rate λ; → (i − 1, i′ − 1) at rate iµ; → (i, i′ − 1) at rate (s − i)µ; When s ≤ i ≤ i′ < ∞: (i, i′)→ (i + 1, i′ + 1) at rate λ; → (i − 1, i′ − 1) at rate sµ; → (i − 1, i′) at rate (i − s)µ
minimal process
When explosion does happen, we augment the state space as I = {0, 1, 2, . . . } ∪ {∞}, and set Xt = ∞ for all t ≥ ζ. This is called the minimal process associated with the given rates. (All activity takes place on the minimal length time interval possible. Non-minimal processes would be obtained by "re-starting" the process in some finite state i′ at time ζ, according to a probabilistic rule.
When does the conclusion of the OST fail
When lim(N-> infinity) E[XT I[T<=N]] does not go to 0
Gambler's ruin in the asymmetric case
Xn = x0 + ∑(i=1,n) Zi, n ≥ 0, where Z1, Z2, . . . are i.i.d. taking values +1 and −1 with probabilities p and 1 − p, respectively. T = inf{n ≥ 0 ∶ Xn ∈ {0, b}}. apply the OST to the exponential martingale of Section 4.9. we have T < ∞ with probability 1. This can be seen for example from the fact that (Xn)n≥0 is a Markov chain, and {0, b} can be reached from any of the finitely many states 1, . . . , b−1. Also, Mn is bounded up to the stopping time: since ∣Xn∣ ≤ b for n ≤ T, we have ∣Mn∣ ≤ max{((1−p)/p)^b, 1} for n ≤ T. ((1 − p)/p)^x0 = E[M0] = E[MT ] = E[((1 − p)/p)^XT] = ((1 − p)/p)^0 P[XT = 0] + ((1 − p)/p)^bP[XT = b] = P[XT = 0] + ((1 − p)/p)^b P[XT = b] = 1 − P[XT = b] + ((1 − p)/p)^b P[XT = b]. Rearranging gives: P[XT = b] = [((1−p)/p)^x0 − 1]/[((1−p)/p)^b − 1]
Investing on the binomial model for asset prices
Xn ∶=∑(i=1,n) Zi where Z1, Z2, . . . are i.i.d. with P[Z1 = +1] = p, P[Z1 = −1] = 1 − p, where 0 < p < 1, p ≠ 1/2. Recall that Mn = ((1 − p)/p)^Xn, n ≥ 0 is a martingale. Suppose that Mn models the price of a stock at time n, where we take M0 = 1 as the unit price. The information available at time n is Fn = events determined by Z1, . . . , Zn, which gives the up and down movements of the stock up to time n Suppose that an investor holds Yn−1 units of the stock at time n − 1, n = 1, 2, . . .. Depending on the information Fn−1 available at time n − 1, she decides to buy Cn−1 units of the stock. Here Cn−1 < 0 is allowed, and corresponds to selling that many units of the stock. Let Nn = worth of investment at time n. N0 = Y0M0 At time n − 1 the investor has stocks worth Yn−1Mn−1. She pays Cn−1Mn−1 for her purchase and at time n she has stocks worth (Yn−1 + Cn−1)Mn (here Yn−1 + Cn−1 = Yn). Thus the net change is: (Yn−1 + Cn−1)Mn − Yn−1Mn−1 − Cn−1Mn−1 = (Yn−1 + Cn−1) (Mn − Mn−1) and hence Nn = Y0M0 +∑(k=1,n)(Yk−1 + Ck−1)(Mk − Mk−1), n ≥ 0 Due to Theorem 4.8.1, with Bk−1 = Yk−1 + Ck−1, the value of the investment (Nn)n≥0 is a martingale: E[Nn+1 ∣ Fn] = E[Nn + (Yn + Cn)(Mn+1 − Mn) ∣ Fn] = Nn + (Yn + Cn)E[Mn+1 − Mn ∣ Fn] = Nn
Suppose that the RV Y is determined by X0, . . . , Xn. Then E[Y ∣ Fn] = ... If W is another RV, then E[Y W ∣ Fn] =...
Y since Y is constant on each elementary event. Y E[W ∣ Fn]. This is the rule of "taking out what is known"
Martingale property
conditional on the entire past up to a given time m, the average value of the process at a future time n > m equals the current value. This can be expressed as: E[Xn − Xm ∣ X0 = xo, . . . , Xm = xm] = E[∑(i=m+1,n) Zi∣ X0 = x0, . . . , Xm = xm] = E[∑(i=m+1,n)Zi] =∑(i=m+1,n) E[Zi] = 0. In the second equality we used that the steps Zm+1, . . . , Zn are independent of the history up to time m.
Theorem 2. Let (Xt)t≥0 be a birth process of rates (qj ∶ j ≥ 0). Then...
conditionally on Xs = i, the process (Xs+t)t≥0 is a birth process of rates (qj ∶ j ≥ 0) starting from i and independent of (Xr)r≤s.
(forward equation) of P(t) = e^tQ
d dtP(t) = P(t)Q, P(0) = I. Indeed: d/dt (∑(n=0,∞)tⁿQⁿ/n! =∑(n=1,∞)nt^(n−1)Qⁿ/n! =∑(n=1,∞)t^(n−1)Q^(n−1)/(n − 1)! Q
(backward equation) of P(t) = e^tQ
d/dtP(t) = QP(t), P(0) = I. ● P(0) = I, d/dtP∣t=0 = Q, d²/dt²P∣t=0 = Q²
Theorem 1. If b1, . . . , bJ > 0, the open migration process ...
has invariant distribution ̃π(n) = J ∏ j=1 πj(nj), n = (n1, . . . , nJ ) ∈ N J , where πj(nj) is given by (*). Proof of Theorem. We check that at each node "flux out" equals "flux in". Node ∆. We need to verify the following: flux out of state n due to departures from Node ∆ = flux into state n due to arrivals to Node ∆ ̃π(n) ∑(k∈I) q̃(n, T∆kn) = ∑ k∈I ̃π(T∆kn) q̃(T∆kn,n) ̃π(n) ∑ k∈I q∆k = ∑(k∈I) ̃π(T∆kn) qk∆φk(nk + 1) [̃π(T∆kn) = π(n) πk/ φk (nk +1) π(bar)∆ ∑(k∈I) q∆k = ∑(k∈I) π(bar)kqk∆. The last equality holds, because πQ = 0 Rest of proof quite long (page 53)
(ii) We say that the state i ∈ I is recurrent
if either qi = 0, or qi > 0 and Pi[Ti < ∞] = 1.
Definition 1. We call a Q-matrix irreducible if...
if its jump matrix Π is irreducible, that is, if for any pair of states i, j ∈ I, i =/ j there exists a sequence of states i = i0, i1, . . . , ik = j, such that qir−1,ir > 0 for each r = 1, . . . , k
Lemma 1. If Q is irreducible on I then for all N ≥ 1, the Q-matrix Q̃
is irreducible on SN. We have to show that given any two configurations n, m ∈ SN , there is a sequence of allowed jumps of the closed migration process that leads from n to m. Let n∗ be the configuration in which all N particles are at Node 1. Each particle in configuration n that is not at Node 1 can be moved to Node 1 via a sequence of jumps that are allowed by Q for a single particle. Such a sequence of jumps is also always allowed in the closed migration process, since population pressures are assumed positive whenever there is at least one particle at the node. Hence we can move all particles to Node 1 one-by-one. Likewise, from configuration n∗ , we can move particles to other nodes one-by-one, to obtain configuration m. Hence there is a sequence of jumps from n to m.
probabilistic interpretation of the Qmatrix
its entries tell us the short time behaviour of the Markov chain (up to an error that is o(t) as t ↓ 0). On the one hand: pii(t) = 1 − qit + o(t), so qi is the "rate of leaving state i". On the other hand: pij(t) = qij t + o(t), so qij is the "rate of moving from state i to state j"
ii) Starting with an empty M/M/1 queue, the mean return time to an empty queue is
m0 = 1/ν0q0 = 1/[(1 −λ/µ)λ] = µ/[(µ − λ)λ]
Tutorial 3
question 3 MADNESS
Let (Tk ∶ k ∈ I) be a countable collection of independent RVs with Tk ∼ Exp(qk) and 0 < q ∶= ∑k∈I qk < ∞. Put T = infk∈I Tk. Then...
with probability 1, there exists a unique random index K ∈ I, such that T = TK. Moreover, T and K are independent with distributions T ∼ Exp(q) and P[K = k] = qk/q. Proof. Set K = k, whenever the infimum defining T is a minimum, occurring at the index k (otherwise leave K undefined). We calculate, by conditioning on the value of Tk and using independence: P[K = k and T ≥ t] = P[Tk ≥ t and Tj > Tk for all j ≠ k] = ∫(t,∞) qk e^−qks P[Tj > s for all j ≠ k] ds = ∫(t,∞) qk e^−qks ∏(j≠k)e^−qjs ds = ∫(t,∞) qk e ^−qs ds = qk/q × e^−qt. Here the first factor only depends on k and the second one only on t. This gives that K and T are independent, P[K = k] = qk/q and T ∼ Exp(q).
Tutorial 2 question 3
worth another look
Explosion time
ζ ∶= S1 + S2 + ⋅ ⋅ ⋅ = "explosion time".
Definition 2. A measure or distribution λ on I is called invariant, if
λQ = 0. Motivation for this definition: When I is finite, we can calculate using the backward equation: λP(t) = λ ∀t ≥ 0 ⇐⇒ 0 = d/dt (λ P(t))) = λQP(t) ∀t ≥ 0 ⇐⇒ λQ = 0. (since P(t) = e^tQ) When I is infinite, proving the equivalence in (11) is more tricky, but it is possible
the long run proportion of time the M/M/1 queue is empty is
ν0 = 1 − ρ.
Lemma 2. If ν and Q are in detailed balance, then
νQ = 0 sum both sides of νjqji = νiqij for all i, j ∈ I. over i ∈ I, i =/ j to see that the balance equations are satisfied, and use Lemma 1.
Definition. A measure ν and a Q-matrix Q are in detailed balance, if
νjqji = νiqij for all i, j ∈ I
Let π be the unique distribution on I such that πQ = 0. Theorem 1. The stationary distribution of the closed migration process is given by
π(n) = BN ⋅ ∏(j=1,J) [πj^(nj)/∏(r=1,nj)φj(r)] , n ∈ SN , (19) where BN is a normalising constant
M/M/∞ queue invariant distribution
πi e−ρ (ρ^i)//i! page 34
Let Q be a Q-matrix on I, where we now allow I to be countably infinite. We define the jump matrix of Q to be the matrix Π = (πij ∶ i, j ∈ I), where
πij = qij/qi if j ≠ i, qi > 0; 0 if j ≠ i, qi = 0; πii = 0 if qi > 0; 1 if qi = 0.