Kingdom Rush Agent
I got the logic of a tower-defense game running without a screen, far faster than real time, and trained one network to play its levels.
What it is
Kingdom Rush is one of the games I played as a kid. I liked it then and I still do, so going back to it has been a nice thing to work on. The levels turn out to be easier than I remembered, though some of them are still hard to win in one go, and it is a bit nostalgic to see the game again from a different angle. Enemies walk along a road, and you build towers beside it to stop them before they get through. I wanted to find out whether a program could teach itself to play it, and not just one level but a whole run of them. Watching the strategies it comes up with is half the fun. The program is a neural network, which is just a program that learns from its own mistakes, and I used the same one for every level.
The game belongs to Ironhide Game Studio and I don't publish any of it, so there are no screenshots here, only my own diagrams and charts.
The catch
A program learns like this by playing an enormous number of games and noticing what went wrong. At normal speed that would take forever. So I pulled the rules of the game out and made them run as plain calculation, with no graphics, sound or menus. A level 1 game now takes about a second, around 260 times faster than real time.
Fast isn't enough. The program also has to be able to repeat a game exactly, and the real game can't: even its sound effects share the dice rolls behind the fights, so the same moves can end differently. My screenless version has no sound, so that problem is gone, and I fixed the other places where it could drift. Now the same moves always give the same battle in it.
Repeatable games let me set the board back to a few minutes before things went wrong and let the program try that stretch again, the way a chess player sets the pieces back to their mistake.
What happened
At its best, it wins the first seven levels every time in my test games and the eighth most of the time. That best result is the luckiest of about 150 rounds of testing, so treat it as rough. The scores bounce around a lot: a version that wins a level every time can lose it in the next round.
The ninth level ends in a boss fight, and it has never won that. It gets to the boss in every game, and the boss usually still walks out with about half its health or more. I tried all 13 heroes and 46 ways of spending upgrades, and none of them won. I've only tested nine of the game's 26 levels, so the rest are still ahead of it.
I stopped the ninth-level training myself. It was using a lot of computing time and barely getting anywhere, so carrying on seemed pointless.
What's next
First I need to step back, do some reading and change how I train it, because more of the same clearly isn't going to beat the boss. After that comes the ninth level, and then the other 17. The simulator can already run all of them, but I haven't tried the program on any level past nine.
The problem
Kingdom Rush is one of my childhood games, and I enjoyed it then and still do. Coming back to it as an adult is a mix of nostalgia and a different point of view: the levels are easier than they seemed at the time, but some are still hard to win in one attempt, and it is interesting to see what strategies a trained network settles on.
I wanted to know whether one network could learn to play many levels of a tower-defense game, rather than one level tuned to death. Reinforcement learning needs millions of games, and the real game plays at one second per second, so the real work was a simulator fast enough to learn from.
The game is Kingdom Rush, by Ironhide Game Studio. I worked from my own copy of the game's logic. The code and assets belong to Ironhide, so none of it is published and there is nothing of the game to download. What I'm showing is the research side: the simulator, the training and the results, in diagrams and charts I made. I trained it in October 2026.
How it works
The headless version runs the game's own systems, scripts and data tick by tick in plain LuaJIT, with no window, sound or UI. Only engine services with no effect on gameplay are stubbed out. The agent's actions go through code that copies the battle UI, with the same legality rules, and only one action is allowed per tick, as for a human clicking. Building charges gold on the next tick, so two builds in one tick could overspend.
Search and rebuilding states both depend on determinism, and the game isn't deterministic. Its sound system draws from the same random stream as gameplay, so which sounds play changes the battle, and the same seed doesn't give the same fight. The headless version has no sound, so that one never touches it, but I found four other sources of divergence. Two places iterate over tables keyed by memory address (simultaneous waves, and signal handlers), and those now run in creation and registration order. The pathfinder's cache is read by a gameplay script, and even a failed call clears it, so observations use a side-effect-free flood fill instead, which agreed with the game's A* on 117,936 of 117,936 checks. A save slot changes during play. And the seeds have to be set at exactly the right points. With all that fixed, the same seed and actions give the same game, tick for tick. A record-and-replay of all 26 levels matched at 769 of 769 checkpoints, and against the real game on level 2, a replay matched 32 of 32.
Why does it matter that the game repeats exactly?
Everything else here leans on it. If the same moves always give the same game, I can rebuild any earlier moment by replaying them, check a rebuild against a fingerprint of the state, and compare two networks on identical games. Without that, a lost game could be bad luck as easily as bad play.
Drawing took 47% of every tick on a busy level. A lean render step skips the drawing-only work and keeps the animation state that gameplay reads, and a bot played all 26 levels both ways with identical state fingerprints. A busy tick went from about 350 to about 170 microseconds.
| What was measured | Ticks/s × real time |
|---|---|
| Simulation only, level 1, one process | ~15,000 ~260× |
| Simulation only, all 26 levels at worst case (unlimited gold), 20 processes | ~60,000 ~1,000× |
| With the Python environment, a decision every tick | ~2,500 ~40× |
| Same, a decision every 60 ticks | ~34,500 ~575× |
The 1,000 times figure is the simulation alone, at worst case. Once the environment builds observations for the agent and takes its actions (random legal actions, in that test), it is 40 to 575 times. Building one observation costs about 1.3 ms, roughly 25 game ticks, which is what limits it at high decision rates.
One network plays every level. It reads a set of tokens, one for each enemy, unit, tower, path segment and upcoming wave plus one for every legal action, and picks an action, then how long to wait before the next decision. The reward is +1 for a win, -1 for a loss and -0.05 per life lost, plus 0.5 spread over the waves reached so that losing later beats losing earlier, and up to +1 per boss for the share of its health removed (from update 8000). There is also a tiny cost per decision: without it, deciding every tick is free and the agent drifts there. Training is PPO with a curriculum, and the next level unlocks when the newest one reaches a 50% win rate.
What is PPO?
Proximal Policy Optimization. The network plays a batch of games, then gets nudged towards the moves that scored better than it expected and away from the ones that scored worse. The "proximal" part caps how far one batch can move it, so a single odd batch can't wreck what it already knows.
What is an entity transformer?
A game has a different number of things on screen at every moment, so I give each enemy, tower, unit and legal action its own token, the way a sentence is a row of word tokens. Attention lets every token look at all the others, so a tower can weigh the enemies near it. The network is 3.9M parameters, 4 layers, d=256.
Because the game is deterministic, any earlier moment can be rebuilt by replaying its actions. The game's scripts run on coroutines, which can't be saved, so replay is the only way back. Restoring 500 steps takes about 2 seconds, and a rebuilt state gave an identical fingerprint and observation in three different workers, so spare workers do it in the background. A share of the games on an unsolved level (30% by default) start from a rebuilt point one to three waves before where an earlier game was lost. The same trick drives a search step.
What does the search step actually do?
It is lookahead followed by learning. Every 40 seconds of game time, eight copies of the game play the next stretch in different ways, and each is scored by what it collected plus the network's own guess at how good the ending state is. The best copy is kept, and the network is trained to imitate its moves. It is the same idea as AlphaZero: search, then learn from the search.
Results
The test is a set of seeded games per level, with the star upgrades a player would have at that point. I switched it to play the network's top choice instead of a sample, because sampled moves depend on how many games share a batch, and with 8 games per level the same network could score up to 3 games apart. At update 4001 the agent won levels 1 to 4, 6 and 7 every time, level 5 in 75% of games and level 8 in 25%. Continuing from that checkpoint, the best score is 7.88 at update 12204.
| Level | Bot | Agent, 4001 | Agent, 12204 |
|---|---|---|---|
| 1 to 3 | – | 100% | 100% |
| 4 | 25% | 100% | 100% |
| 5 | 0% | 75% | 100% |
| 6 | 0% | 100% | 100% |
| 7 | 0% | 100% | 100% |
| 8 | – | 25% | 88% |
| 9 | – | 0% | 0% |
The test covers 9 of the game's 26 levels. Level 9 ends in a boss fight, and the agent has never won it on this test.
What didn't work
The 7.88 is the best of about 150 evaluations of one run, and the same seeded games also pick which checkpoint I keep, so it is an optimistic number. With 8 games per level, single levels swing a lot: level 6 went from 0% to 100% between neighbouring checkpoints. After update 4000 the scores in the chart have a median of 6.9 and range from 5.9 to 7.9. I haven't tested on held-out seeds.
The early runs failed in specific ways. One collapsed when its teacher pull ended. In another, a 64% win rate on level 4 turned out to be a stale average, and on the fixed test it won 0 of 4. I also removed a kill bonus that was worth as much as winning on levels the agent was losing (+1.3 on level 4).
Search was slow to help. Repeated low-memory stops meant it barely contributed in two runs. A single shared imitation buffer let lost level-9 games crowd out level 6, which fell from 8 wins out of 8 to 1 out of 8, so each level now has its own buffer, fed from won games only. The exception is whole-game searches on level 9, which imitate their best copy even when it lost. I haven't run a clean comparison of search against plain PPO, so I can't say how much of the gain is search.
Level 9 is the open failure. The network reaches the boss in every game, but in the final level-9 run the boss still left with 64 to 76% of its health in training games, and there are no wins in 191 logged fixed-test evaluations of level 9 across the runs (up to 8 games each). A boss that reaches the exit costs 20 lives, so the level comes down to one fight, and the defence the network builds is mostly level-1 archer towers even when level-4 upgrades are legal. I tried 13 heroes and 46 star allocations. No game won, and the best hero left the boss at about half its health. With every upgrade maxed, one diagnostic still left it at about 56%.
In one test of 8 copies of level 9, the best copy left the boss at 52% against about 59% for the greedy network. Later whole-game searches in the last run varied widely, with the best copy leaving between 17% and 99% of the boss's health, and none of those 60 games was won.
What is unfinished
Level 9 is not beaten, and levels 10 to 26 are untrained. The simulation runs all 26, but the agent has only been tested on the first 9, and I haven't tried it on anything above level 9. I still need held-out seeds, and a proper comparison of search against plain PPO. The last run continued from the update-12204 best checkpoint to train on level 9 only (it isn't in the chart). I stopped it myself near update 14,000: it was burning a lot of compute for almost no progress on the boss, and I haven't restarted it. Before I do, I want to read up on how others handle this kind of sparse, one-fight bottleneck and change the training itself, rather than run more of the same. The training machine is my own PC, so training resizes itself or steps aside when RAM is short or a game is running.
- Stack
- LuaJIT (headless simulation), Python, PyTorch, PPO
- Hardware
- Ryzen 9 9900X; the later runs trained under WSL
- Credit
- Game by Ironhide Game Studio
- Status
- In progress
- Code
- Not public.