AML Reinforcement Learning & Decision Making 2

Question 1

In Q-learning, what does the Q-value Q(s, a) represent?

Accepted Answer

The expected cumulative discounted reward when taking action a in state s and following the optimal policy thereafter

Answer

The probability of selecting action a when in state s

Answer

The immediate reward received after taking action a in state s

Answer

The number of times action a has been taken in state s

Question 2

What fundamentally distinguishes SARSA from Q-learning?

Accepted Answer

SARSA is on-policy and updates using the actual next action taken; Q-learning is off-policy and updates using the greedy max-Q action

Answer

SARSA uses neural networks while Q-learning uses lookup tables

Answer

SARSA requires an environment model while Q-learning does not

Answer

SARSA targets continuous action spaces while Q-learning targets discrete ones

Question 3

What is the primary motivation for using Deep Q-Networks (DQN) over tabular Q-learning?

Accepted Answer

DQN uses neural networks to approximate Q-values, enabling generalization over large or continuous state spaces

Answer

DQN converges faster in all environments regardless of state space size

Answer

DQN eliminates the need for a replay buffer during training

Answer

DQN removes the requirement for a discount factor

Question 4

What problem does experience replay solve in Deep Q-Networks?

Accepted Answer

It reduces correlation between consecutive training samples, stabilizing neural network learning

Answer

It prevents catastrophic forgetting of previously learned tasks

Answer

It increases the exploration rate during early training stages

Answer

It allows DQN to train simultaneously across multiple environments

Question 5

In the Actor-Critic architecture, what is the primary role of the Critic?

Accepted Answer

The Critic estimates a value function to evaluate the Actor's actions and reduce gradient variance

Answer

The Critic selects and executes actions in the environment

Answer

The Critic explores the environment by generating random action sequences

Answer

The Critic directly computes the policy gradient

Question 6

What is the key innovation that makes Proximal Policy Optimization (PPO) stable during training?

Accepted Answer

PPO clips the policy update ratio to prevent excessively large policy changes that could destabilize learning

Answer

PPO uses Monte Carlo returns exclusively instead of TD estimates

Answer

PPO requires a separate world model for environment planning

Answer

PPO eliminates the value function baseline to simplify optimization

Question 7

What is the 'deadly triad' in reinforcement learning, and why is it problematic?

Accepted Answer

Function approximation, bootstrapping, and off-policy training — their combination can cause divergence and instability

Answer

Exploration, exploitation, and generalization — they create oscillating reward signals

Answer

Reward, policy, and value function — optimizing all three simultaneously leads to conflicting gradients

Answer

Speed, accuracy, and memory — hardware constraints prevent jointly optimizing all three