MAST30001 Chap.4 Continuous-Time Markov Chains
Continuous-Time Markov Chains
Continuous-time Markov chains move among states at random times. Their transition family P(t) obeys Chapman-Kolmogorov, while the generator Q stores instantaneous jump rates. This chapter separates holding-time rates from jump destinations, develops forward and backward transition equations, and uses stationary flow to solve long-run probabilities.
Birth-death chains provide a reusable adjacent-balance structure that leads directly into queueing systems. A generator row encodes two distinct objects: its diagonal determines how long the process remains, while its off-diagonal proportions determine the destination after departure. Reading those components separately is the safest route from rates to a probabilistic story.
What this chapter covers
- 01
Continuous-time Markov property and homogeneity
- 02
Chapman-Kolmogorov for the transition family P(t)
- 03
Generator rows, holding times and embedded jumps
- 04
Forward and backward transition equations
- 05
Stationary equations and long-run flow
- 06
Birth-death chains and adjacent balance
Worked example · free
Find reliability availability from two rates
- 1The generator is [[-0.10,0.10],[0.40,-0.40]].
- 1Stationary flow gives 0.10 pi_W = 0.40 pi_F.
- 1Normalisation gives pi_W + pi_F = 1.
- 2Therefore pi_F=0.20 and pi_W=0.80.
Key terms
- Jump Process
- A continuous-time process that remains in a state for a holding time and then changes state.
- Holding Time
- The random duration spent in the current state before the next jump occurs.
- Jump Rate
- An instantaneous transition intensity that is not itself a one-step probability.
- Embedded Chain
- The discrete-time chain formed by recording only the sequence of states visited at jump epochs.
- Birth-Death Chain
- A continuous-time chain whose moves change the state by one step upward or downward.
- Generator Matrix
- A rate matrix whose off-diagonal entries are jump intensities and whose rows sum to zero.
- Stationary Flow
- The equality of total probability flow entering and leaving each state under a stationary law.
Continuous-Time Markov Chains FAQ
How is a generator different from a transition matrix?
A transition matrix contains probabilities and has rows summing to one. A generator contains instantaneous rates, has nonnegative off-diagonal entries and rows summing to zero, with negative diagonal entries equal to minus the total departure rate from each state.
How do generator rates determine the next jump?
Negate the diagonal to obtain the total departure rate and the exponential holding-time law. Divide each off-diagonal rate by that total to obtain the conditional probability of the next destination after a departure occurs.
Why are birth-death chains easier to solve?
Their nearest-neighbour structure often yields adjacent flow equations that express each stationary probability as a ratio times the previous one. After building those relative weights, a single normalisation equation completes the distribution if the weights are summable.
What checks should every proposed generator pass?
Every off-diagonal entry must be nonnegative, each row must sum to zero and every diagonal entry must be nonpositive. Its magnitude equals the total departure rate. These checks are dimensional as well as algebraic: generator entries have units of inverse time rather than units of probability.
How are the forward and backward equations different?
Both describe how transition probabilities change with elapsed time, but the generator multiplies on different sides. The forward equation evolves destination probabilities, while the backward equation conditions through the initial state. Fix the course row-vector convention before multiplying so that dimensions and indices remain consistent.
When may adjacent balance be used?
Adjacent balance is natural for a birth-death chain because transitions occur only between neighbours. Write the flow across each edge and derive relative stationary weights. For a more general chain, pairwise detailed balance may not hold even when a stationary distribution exists, so use the full balance equation instead.
Exam move
Practise decoding each generator row into a holding-time rate and destination probabilities. Check off-diagonal signs and zero row sums before doing any other work. For stationary questions, use flow balance to express state weights relatively and normalise last. Draw birth-death chains with upward and downward rates over each edge, because a missing index causes many algebra errors.
Keep P(t) and Q conceptually separate: P(t) is stochastic for an elapsed duration, while Q is the rate matrix that generates those probabilities. Use short generator drills in both directions: given a matrix, narrate holding times and jump destinations; given a verbal rate model, construct every row and its diagonal. On transition-equation problems, state whether vectors are rows or columns before writing a product.
On stationary problems, verify nonnegativity, total mass one and zero net flow. For an infinite birth-death chain, also check that the relative weights form a finite sum before normalising them.
Working through Continuous-Time Markov Chains in MAST30001? Sia is AskSia’s AI Statistics tutor — ask any MAST30001 Continuous-Time Markov Chains question and get a clear, step-by-step explanation grounded in how MAST30001 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
