The Arcade Learning Environment has a reputation for making reinforcement learning look approachable. Feed it a ROM name, get back a reward signal and a frame. What it hides is that someone already found the score register, verified the end-of-game condition, and wired it all to a clean Python API for 100-plus Atari 2600 games. The PPO algorithm. The learning part everyone talks about. Is 340 lines in a CleanRL single-file trainer. The unglamorous half is everything ALE already did for you. This post is about rebuilding that half for a game ALE doesn't cover.
The project lives in ~/Projects/Orange-Robot/Tron (directory name left over from a scrapped Tron light-cycle idea), Python package retrorl. The session ran over 2026-09-06 and 2026-09-07 on a machine with an NVIDIA RTX 3050 Ti Laptop GPU (4 GB), 16 CPU cores, and 62 GB of RAM. Stack: Python 3.11.15, PyTorch 2.14.0+cu130, Gymnasium 1.3.0, ale-py 0.12.1. Algorithm: PPO, CleanRL-derived, deliberately chosen over Stable-Baselines3 because the end goal is code that gets read and modified. SB3's abstraction layers work against that.
First order of business: confirm Dig Dug isn't in ALE. It isn't. Dig Dug is an arcade game, not an Atari 2600 title, so it was never in Stella's scope and was never bundled. The right response to that finding isn't to swap in a different game. It's to validate the training stack on a game that does have published baselines before touching anything novel. Ms. Pac-Man was the choice, because the SLM Lab benchmark reports DQN at 2,311 and PPO at 2,278 at 10M frames under comparable conditions. If the stack hits roughly those numbers, subsequent failures on Dig Dug are the emulator's fault, not the trainer's.
The Ms. Pac-Man run went 10M agent steps in 73 minutes at about 2,265 steps per second across 16 async environments. Final training return: 2,302. Explained variance: 0.986. Evaluated over 15 episodes per checkpoint: random play scored 265 ± 130, the 500k checkpoint scored 733 ± 424, and the 10M checkpoint scored 2,145 ± 877. That 2,145 sits close enough to the SLM Lab DQN figure of 2,311 to call the stack validated. The more interesting number is in the right-hand columns. Points-per-step went from 0.50 at random to 1.14 at 500k to 2.42 at 10M, a 2.1x improvement over the second half of training. Survival improved only 1.38x over the same interval. The agent learned pellet routing first and ghost avoidance second. And this ordering isn't an accident. Pellet reward is immediate and dense. Dying is a delayed signal that arrives once per episode, which makes credit assignment much harder. At 500k steps the agent was scoring 3.8x random while surviving only 1.2x longer: it had learned to eat efficiently and was still dying like a random player.
Now for the Dig Dug work. The first emulator choice was stable-retro, which was wrong: it bundles cores for Atari, NES, SNES, and Genesis, but no arcade MAME core. The second attempt was MAME 2003-Plus, which loads Dig Dug correctly but returns 0 for every retro_get_memory_size call across IDs 0 through 5. No memory exposed, no reward signal, useless for RL. The working solution was FBNeo (fbneo_libretro.so), the currently active FinalBurn Neo fork. It loads the same ROM and exposes 5,120 bytes of SYSTEM_RAM. The integration is a direct ctypes binding to the libretro C API: retro_run per frame, retro_get_memory_data for RAM reads, retro_serialize and retro_unserialize for episode resets. No subprocess, no Lua bridge, no socket. Measured throughput: 3,336 emulator fps on a single instance.
One design choice that makes the environment usable: rebooting the arcade board every episode would dominate run time. Instead the core boots once, through the self-test and coin insert, and that savestate is restored on every reset() call. Then there's the score. ramsearch.py records all 5,120 RAM bytes every frame while a random policy plays, then ranks byte groups that behave like a score: non-decreasing, static most of the time, incrementing by values from a small set. Every candidate is decoded as BCD and as binary, both byte orders. The Dig Dug score lives at addresses 0x416, 0x415, 0x414, BCD little-endian. Lives are at 0x40A. Verification was direct: the on-screen display showed 430 and 480 at two points where the decode returned exactly 430 and 480. The stronger validation was done on Ms. Pac-Man, where ALE exposes both 128 bytes of RAM and a ground-truth score. Ramsearch independently proposed [0x7A, 0x79, 0x78] BCD little-endian, and across 1,921 frames there were zero mismatches. Every increment was exactly 10, one pellet. That is the number that proves the tool.
Episodes terminate when lives hits 0, which cuts the final life short. That is deliberate. After game over, Dig Dug's attract mode plays itself and racks up score the agent never earned. Training on that would be far worse than losing a third of each episode. The transition was confirmed by watching 0x40A go 2 → 1 → 0 and then jump back to 3 at frame 7,079 as the demo restarted.
The Dig Dug training run went 10M steps in 2.7 hours at 861 steps per second on 16 async environments. Final training return: roughly 1,032. Explained variance: 0.982. Training return climbed monotonically across all 10,000 episodes with no plateau. Evaluation over 15 episodes per checkpoint: random scored 117 ± 135, surviving 622 steps at 0.19 points per step. The 51,200-step checkpoint scored 132 ± 58. But survived only 436 steps, worse than random, while already earning 0.30 points per step. By 2.56M steps the agent was scoring 599 ± 108 with 662 steps alive. The final checkpoint at roughly 10M steps scored approximately 1,164, surviving 899 steps. Score x9.98 over random. Survival x1.44. Three things in that table are worth more than the headline score. First, the score column cannot resolve the last 5M steps: the 5.07M and 10M checkpoints differ by 75 points with a combined standard error of ±273, so the right summary is "roughly 1,100" rather than any specific improvement claim. Second, survival is the low-variance signal and it keeps climbing monotonically after an early dip. Where score is too noisy to show progress, time-alive still shows it. Third, the 51,200-step agent is actively worse at staying alive than random, while already more efficient per step. It has learned to move toward things before learning which things kill it. Random play mills around harmlessly; a half-trained agent walks into Pookas. This is the same pellet-routing-before-ghost-avoidance pattern seen in Ms. Pac-Man, but Dig Dug shows it more starkly because here the early agent regresses on survival rather than merely stalling.
The bugs were the most expensive part of the work, and every one of them failed silently or gave a confident wrong answer rather than a clean error. Gymnasium 1.x defaults to NEXT_STEP autoreset: the step following an episode end returns a reset observation with reward 0 and discards the action you sent. A junk sample that a straight CleanRL port trains on without complaint. Verified empirically against a CartPole A/B comparison with SAME_STEP. The trainer masks those steps from the policy loss, value loss, entropy term, and advantage normalisation, and logs dead_transition_frac so the masking stays visible; measured at 0.0020, one per roughly 500-step episode. Wrapper ordering produced a different class of silent failure: RecordEpisodeStatistics must wrap the raw environment, inside ClipReward, so the logs show real game scores while the network trains on clipped rewards. With the ordering reversed, a perfectly healthy run reported clipped scores. The same episode measured 160 raw against 23 clipped. MAME 2003-Plus segfaulted inside retro_load_game because the core asks the frontend for a log callback and calls the null pointer if none is provided; passing a real log callback fixed it immediately, but the failure left zero Python traceback to start from.
The RAM search failed twice before it worked. The first run returned no candidates because the recorder only pressed directional buttons. Ms. Pac-Man sat on its title screen for 6,000 frames and never scored. The second run, on Dig Dug, returned high-score-table values like 750,000 because the pump is the only way to score in Dig Dug, and the random policy was still never pressing fire. Adding fire to the random action set took the search from 4 jumps in 12,000 frames to 60. The failure mode was not "no answer"; it was "a confident wrong answer," which is the more dangerous kind. The fork bug was structural: AsyncVectorEnv constructs one throwaway environment in the parent process before forking, regardless of kwargs, to read its observation and action spaces. For a libretro core, that loads and then deinits the core, so every forked child inherits dirty global state. Clean parent, forked child: fine, 5,120 bytes of RAM. Parent touches the core first, then forks: double free or corruption, SIGABRT. Fix: run the libretro backend under the spawn start method, which means env factories must be picklable classes rather than closures.
Two video-tooling bugs only surfaced because the pipeline was tested on Dig Dug early. progression.py tiles multiple checkpoints in one process; creating a libretro core a second time in one process dies with free(): invalid size. Each rollout now runs in its own spawned process and hands frames back through a compressed .npz. The rendered video came out upside down: Dig Dug is a portrait cabinet, so the core emits a 224×288 landscape frame that needs rotating. np.rot90 is counter-clockwise, so a 90-degree rotation produced readable text that was upside down; 270 is correct. A subtler trap: progression.py and watch.py read their rotation config from the checkpoint, not the YAML, so editing the YAML mid-run changes nothing. Both tools now take an explicit --rotate override.
The measurement mistake worth logging: the first progression video played one episode per checkpoint and reported scores of 260 / 1060 / 450 / 350, which reads as the agent getting worse after 200k steps. That was pure noise. Ms. Pac-Man scores have a standard deviation around 227, so a single episode routinely inverts the ordering. Five episodes per checkpoint with the median gives 254 / 718 / 776 / 774. The real curve. A tool that reports a single sample will lie to you with complete confidence.
The packaged result isn't the trained checkpoint. At 6.8 MB and 1.69M parameters it does exactly one task, transfers to nothing, and is reproducible in under four hours. The reusable artifact is the integration file: four lines specifying score_addrs, lives_addr, start_buttons, and action_set. That is hours of memory archaeology nobody else has to repeat. Each integration records rom_sha1 and core_version; a mismatched ROM warns rather than errors, because a different revision may still work, but reading a score from wrong offsets produces a reward signal that looks plausible and is nonsense. The checkpoint is the trophy. The integration is the map.