COMP4318 · Machine Learning and Data Mining
Reinforcement Learning
Week 12 introduces reinforcement learning: the agent–environment loop, Markov decision processes, value functions and optimal policies, Q-learning, and deep Q-learning with experience replay — learning from reward rather than labels. There is no weekly quiz this week, but both sample exam papers include a reinforcement-learning question (an optimal policy and value function, or the RL-versus-supervised contrast and experience replay), so it is exam-relevant despite closing the semester.
What this chapter covers
- 01The agent–environment loop: states, actions, rewards R[s, a, s'] and the discount factor γ
- 02The Markov decision process (MDP) as the reinforcement-learning setting
- 03The value function V*(s) = cumulative (discounted) reward under the optimal policy
- 04The optimal policy: the action in each state that maximises long-run reward
- 05Q-learning: learning action-values Q(s, a) from experience
- 06Deep Q-learning: a neural network approximates Q; experience replay stabilises training
- 07Reinforcement learning vs supervised learning: learning from a reward signal through interaction, not from labelled outputs
The discounted return of a trajectory
- +1The discounted return from the start is G = r0 + γ·r1 + γ²·r2, with γ = 0.9 down-weighting later rewards.
- +1Substitute the rewards r0 = +2, r1 = 0, r2 = −1: G = 2 + 0.9·0 + 0.9²·(−1) = 2 + 0 − 0.81·1.
- +1Compute: 0.9² = 0.81, so G = 2 − 0.81 = 1.19.
Key terms
- Markov decision process (MDP)
- The reinforcement-learning framework of states, actions, transition dynamics and rewards R[s,a,s'] with a discount factor γ; the agent seeks a policy maximising long-run reward.
- Reward & discount factor γ
- The scalar signal the environment returns for each step; γ ∈ [0,1) down-weights future rewards so nearer rewards count more in the return.
- Policy
- A rule mapping each state to an action (or a distribution over actions); the optimal policy maximises the expected return from every state.
- Value function V*(s)
- The expected cumulative (discounted) reward obtainable from state s under the optimal policy.
- Q-learning
- A value-based method that learns action-values Q(s,a) from experience, converging toward the optimal policy without a model of the environment.
- Experience replay
- Storing past transitions and sampling them randomly to train a deep Q-network, which breaks correlations between consecutive samples and stabilises learning.
Reinforcement Learning FAQ
How is reinforcement learning different from supervised learning?
Supervised learning trains on labelled examples with known correct outputs. Reinforcement learning has no labels — the agent interacts with an environment and learns from a scalar reward signal, discovering which actions are good by trial and error over time. The objective is a policy that maximises long-run reward, not matching given labels.
What does the discount factor γ do?
γ ∈ [0,1) controls how much future rewards count relative to immediate ones. A small γ makes the agent short-sighted (it cares mostly about the next reward); a γ near 1 makes it far-sighted. It also keeps the infinite-horizon return finite by geometrically shrinking distant rewards.
What is experience replay in deep Q-learning?
It is a buffer of past transitions (state, action, reward, next state) that the agent samples from at random to train its Q-network. Because consecutive experiences are highly correlated, learning from them in order is unstable; replaying random past samples decorrelates the updates and reuses data efficiently, improving stability over vanilla deep Q-learning.
Is reinforcement learning on the exam?
Yes — both mined sample papers include a reinforcement-learning question, even though there is no weekly quiz in Week 12. Expect an optimal-policy/value-function part on a small MDP, or a short-answer on the RL-versus-supervised distinction and the advantages of experience replay. Confirm the current structure on Canvas.
Exam move
Because Week 12 has no quiz, it is easy to under-prepare — but it is exam-relevant, so lock in the core ideas: the agent–environment loop, the MDP components, the discount factor's effect, the optimal policy and V*(s), Q-learning, and experience replay in deep Q-learning. Rehearse a small MDP calculation (a discounted return or a best-action choice) and be able to write the RL-versus-supervised contrast in one clean line. When the discounting arithmetic slips, ask Sia to recompute a return with a fresh reward sequence and γ and check your powers of γ.
Working through Reinforcement Learning in COMP4318? Sia is AskSia’s AI Machine Learning tutor — ask any COMP4318 Reinforcement Learning question and get a clear, step-by-step explanation grounded in how COMP4318 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
