Reinforcement Learning

Reinforcement learning (RL) trains an agent to act in an environment by trial and error. Unlike supervised learning there is no labeled dataset; instead the agent observes a state, takes an action, and receives a numeric reward. The goal is a policy that maximizes cumulative reward over time. RL powers game-playing agents, robotics controllers, recommendation policies, and resource schedulers.

The Markov Decision Process

Most RL problems are modeled as a Markov Decision Process (MDP) with five elements: states S, actions A, transition probabilities P, a reward function R, and a discount factor gamma that trades immediate rewards against future ones. Key objects to learn are the value function (how good a state is) and the Q-function (how good a state-action pair is). The Bellman equation defines consistency between a Q-value and the next one, and every tabular algorithm below is a way of converging to that fixed point.

Q-Learning from Scratch

Q-learning is the classic tabular algorithm. The agent balances exploration (try something new, epsilon-greedy) against exploitation (reuse the best known action) while updating a table of Q-values with the learning rate alpha.

import numpy as np

n_states, n_actions = 16, 4
Q = np.zeros((n_states, n_actions))
alpha, gamma, eps = 0.1, 0.99, 0.2

for episode in range(2000):
    s, exp_env = reset_env()          # your environment's reset
    done = False
    while not done:
        a = np.random.randint(n_actions) if np.random.rand() < eps \
            else int(np.argmax(Q[s]))
        s2, r, done = exp_env.step(a) # returns next state, reward, terminal
        target = r + gamma * Q[s2].max() * (not done)
        Q[s, a] += alpha * (target - Q[s, a])
        s = s2
print(Q.argmax(axis=1))  # greedy policy

Gymnasium Environments

You rarely write environments by hand. Gymnasium (formerly Gym — treat import names accordingly) provides a standard reset()/step() API and hundreds of environments, from CartPole-v1 to Atari. Deep Q-Networks replace the Q-table with a Keras or PyTorch network when the state space explodes, plus a replay buffer of past transitions for sample reuse.

import gymnasium as gym

env = gym.make("CartPole-v1")
obs, info = env.reset(seed=42)
total = 0
done, truncated = False, False

while not (done or truncated):
    action = 1 if obs[2] + obs[3] > 0 else 0  # naive balancing heuristic
    obs, reward, done, truncated, info = env.step(action)
    total += reward
print("episode reward:", total)
env.close()

For a real training run, a minimal DQN in PyTorch keeps a policy network, copies it to a target network every C steps, and minimizes the squared Bellman error on minibatches sampled from replay memory. Stable-Baselines3 wraps these details behind DQN("MlpPolicy", env).learn(total_timesteps=100_000), which is the fastest way to get a baseline.

Policy Gradient Methods

Value methods need discrete actions. Policy gradient methods parameterize the policy directly, often with a neural network that outputs action probabilities, and increase the probability of actions that led to high rewards. REINFORCE is the simplest variant; actor-critic methods such as A2C and PPO combine a value critic for lower-variance updates. PPO is the de-facto general-purpose algorithm and the one most production RL libraries default to.

# tabular policy-gradient sketch: REINFORCE on one trajectory
logp_grad = 0.0
for (s, a, r) in trajectory:
    logp_grad += grad_log_pi(s, a)    # gradient of log policy
    returns = discounted_return(trajectory)
pi.update(+lr * returns * logp_grad)  # maximize expected return

Common Pitfalls

RL training is notoriously unstable. Reduce complexity first: too large a network, too high a learning rate, or too little exploration noise makes training diverge silently. Track reward per episode, always seed environments, and benchmark against a random agent so you know what improvement actually means.

Summary

Reinforcement learning frames decisions as an MDP, learned through reward signals rather than labels. Tabular Q-learning introduces exploration-exploitation, Gymnasium supplies standardized environments, and deep variants — DQN, REINFORCE, PPO — extend the same ideas to neural policies with Keras, PyTorch, or Stable-Baselines3. Keep experiments small, seeded, and instrumented, because RL rewards careful iteration.