TinyWorldFormer Dreams Up Pac-Man and Doom-Lite on a Raspberry Pi Pico 2 W

I spent way too long this morning playing Pac-Man inside a dream. TinyWorldFormer, a hobby project shared by developer SheepSeb, runs a small AI world model that invents each game frame on the fly. It's built to run on a Raspberry Pi Pico 2 W, a microcontroller board that costs a few dollars.

Two tiny networks

The setup is wonderfully simple. One network is the world model, a looped transformer stored as int8 that takes the current frame plus a button press and predicts the next frame and the reward. The other is a small actor that only looks at those generated frames and picks the next button. On the chip, neither one ever touches the real game.

The world models range from about 26,000 to 131,000 parameters, depending on the game. There are five games: Breakout, Flappy Bird, Pong, a Pac-Man lite and a raycast Doom-lite with a minimap. Breakout fits in 361 KB of flash, and even Pac-Man, the biggest, stays inside the board's 520 KB of SRAM.

The part I loved

The write-up is honest about how agents break their own dream. An agent trained only inside the world model scored 210 in Breakout, where the real maximum is 56, and then scored just 28 in the real game. Each round of recording real play and fine-tuning closed the exploit it had found. An early model even drew two balls at once until a decoding fix forced exactly one.

There's also an agent that can learn from scratch right on the board, one transition at a time. The author notes they haven't measured the Pico frame rate yet, but on a desktop CPU a frame takes about 0.7 to 1.5 ms.

You can play the browser version yourself and grab the code from the project page. Tiny AI that dreams retro games is exactly my kind of weekend rabbit hole.