AI Tech Inspire

DQN Agent Learns to Speedrun Super Mario World in 24 Hours

What happens when classic platformers meet modern reinforcement learning? At AI Tech Inspire, this project jumped out: a Deep Q-Network learns to speedrun Yoshi’s Island 1 from Super Mario World in about a day of wall-clock time, reaching near-human consistency. It’s a tight, engineering-focused exploration of reward shaping, exploration hacks, and temporal modeling—exactly the kind of work that makes developers think, “I could adapt that.”

TL;DR: Key Facts

  • Environment: Super Mario World (SNES), Yoshi’s Island 1; observations downsampled to 84×84 grayscale with 4–8 frame stacking; discrete 12-action space.
  • Algorithm: Deep Q-Network (DQN) with experience replay and a target network; extended with an LSTM for temporal dependencies.
  • Network: 3×Conv2D (32, 64, 64), ReLU; FC layers 512 → 256 → 12; ~1.2M parameters.
  • Training: 1,180,000 steps over ~24 hours (~20 GPU-hours active); replay buffer 100k, batch size 32, Adam lr=1e-4, γ=0.99, target update every 1,000 steps, epsilon decay from 1.0→0.1 over 100k steps.
  • Rewards: -0.1/frame (speed), +1 per screen, +100 on completion, -10 on death; shaped bonuses for ducking Banzai Bills (+2) and staged stomping reward via curriculum.
  • Key challenges: Banzai Bill timing, Rex stomp vs jump choice, exploration collapse, and screen-transition memory—addressed via reward shaping, curriculum, Noisy DQN + periodic epsilon resets, and LSTM.
  • Results: Completion rate reached 94% over the last 1,000 episodes; convergence near 900k steps; stable policy; no significant overfitting observed.
  • Hardware: Ryzen 5900X and RTX 4070 Ti; 64GB RAM; ~50GB checkpoints; saves every 10k steps.
  • Repo: GitHub project and code; fixed seed noted.
  • Metrics not specified: average completion time, best single run, and world-record percentage were left as placeholders.

Why this setup works (and why it’s interesting)

Side-scrollers are deceptively rich RL environments. They demand temporal awareness, momentum control, and split-second action selection. This project leans into that with:

  • Frame stacking (4→8) to carry short-term velocity and enemy state forward.
  • A compact CNN that’s big enough to capture spatial patterns but small enough to train within 24 hours.
  • A reward function laser-focused on speed: a small per-frame penalty forces the agent to trade safety for pace, while screen and completion bonuses anchor longer-term goals.
  • An LSTM on top of DQN to stabilize behavior across screen transitions, where naive Markov assumptions break down.

The result is a policy that doesn’t just survive; it learns timing, ducking, stomping, and route selection that resemble how strong human players approach the level.

Training decisions that moved the needle

Several practical adjustments turned early failure into a consistent speedrun policy:

  • Shaped Rewards for Banzai Bills: The agent reportedly jumped into Banzai Bills 847 times in a row before a specific +2.0 ducking reward nudged it toward the correct timing-and-hold behavior.
  • Curriculum for Rex enemies: Early attempts to jump over Rex led to a local optimum. Delaying stomping rewards until 200k steps prevented brittle early strategies and built a stronger base policy.
  • Exploration resets: Even with Noisy DQN, the policy drifted toward safe but slow play. Periodic epsilon resets every 100k steps reintroduced discovery without derailing training.
  • Temporal modeling via LSTM: Screen transitions require remembering what just happened (e.g., maintaining run momentum). Adding sequence modeling stabilized decisions that span multiple screens.

Key takeaway: small, targeted interventions—reward tweaks, curricula, and episodic exploration resets—can resolve specific brittleness without overhauling the entire stack.

Architecture and hyperparameters at a glance

The network follows a familiar Atari-style template: 3 convolutional layers with 8×8, 4×4, and 3×3 kernels, then fully connected layers (512 → 256 → 12). With ~1.2M parameters, it sits in a sweet spot for single-GPU training on a consumer card. Training details worth highlighting:

  • Adam at 1e-4 and γ=0.99.
  • Replay buffer of 100k, batch size 32, and a target network update every 1,000 steps keep learning stable.
  • Epsilon decays from 1.0 to 0.1 over 100k steps, with periodic resets to counter premature convergence.

If you’re thinking about reusing this setup, the core components map neatly to common RL stacks like PyTorch, TensorFlow, or Stable-Baselines3. The environment wrapper references Gym Retro, a popular path for integrating emulator-based games. A CUDA-capable GPU (see NVIDIA CUDA) is assumed for reasonable training times.

Results, caveats, and failure modes

By ~900k steps, rewards plateaued and the policy stabilized. Over the final 1,000 episodes, completion hit 94%. The agent learned to maintain momentum through jump timing and to bias toward safer, consistent routes over risky time-savers. A few caveats:

  • Precision limits: Sub-pixel positioning caused ~6% failures—common in emulator control where timing resolution is tight and frame skip amplifies errors.
  • Spawn edge cases: Rare enemy patterns still tripped the policy, suggesting limited coverage in the replay distribution.
  • Unreported time metrics: Average completion time, best single run, and world-record comparisons were not specified.

Given the 24-hour budget and ~1.18M steps, those omissions are understandable. If you pick this up, instrument timing from day one to track whether stability translates into speed, not just success rate.

Why it matters for developers and engineers

This is a compact blueprint for sequential decision-making under latency and memory constraints. Three patterns stand out:

  • Reward shaping as engineering: Nudging the agent toward duck instead of jump, or staged stomping, is exactly how RL gets practical in messy environments.
  • Temporal extensions to value-based methods: An LSTM on top of DQN (a DRQN-style move) will often beat pure CNNs when the environment hides state across screens or events.
  • Exploration schedules that reset: Treat exploration like scheduled maintenance. Rebooting epsilon on a cadence can prevent “safe-but-slow” traps.

These lessons transfer to robotics tasks with partial observability, simulator-based control, and even synthetic benchmarks where frame_skip, action_repeat, or delayed rewards complicate learning.

Reproducing and extending the work

Interested in trying it? A few practical pointers:

  • Start with the repo: SuperMarioWorldSpeedRunAI. Check the exact framework used and verify dependencies. Respect legal requirements for game ROMs.
  • Track the right metrics: Log completion time, deaths, and risky shortcut attempts—not just reward and success rate.
  • Modularize rewards: Keep shaping terms togglable so you can A/B test curricula and ducking/stomping incentives quickly.
  • Stress-test temporal modeling: Compare 4 vs 8 frame stacking, and LSTM vs no-LSTM. If you switch to Stable-Baselines3, try recurrent policies where available.
  • Profile performance: Record wall-clock vs environment steps to see if bottlenecks are in emulation, preprocessing, or GPU compute.

Extensions worth exploring include imitation learning via expert traces, double/dueling DQN variants, and curiosity or entropy bonuses when exploration collapses.

A final thought

Speedrunning is a neat proxy objective: it compresses many mechanics—timing, pathing, and risk management—into a single metric that matters to both humans and agents. Projects like this remind us that the difference between “works on paper” and “works in play” often comes down to a few well-placed reward tweaks and a fresh approach to exploration. If you’re building agents for environments with tight control loops and partial observability, this is a template worth cloning—and then pushing further.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.