Whom do you serve?

Dungeon Mastery

Hi, I’m Joe, and I’m a forever-DM.

Chorus: “Hi, Joe.”

I’ve been forever-DMing since around 2015, I played a bit in college before that and ran a one-shot or several, but it wasn’t until then that I’d tried to run anything homebrew. Up until then I’d played canned encounters that had been extensively playtested, I’m sure, by folks who were quite good at the game and who made sure that the encounters would be balanced nicely.

It’s not, therefore, my or my player’s fault when I would routinely trounce them. I had all the advantages. I knew their characters, I knew the monsters; it simply wasn’t my fault that playing the game well meant also not playing for very long because it turns out ‘being everything else’ means you get to create pretty favorable conditions. I did what all DMs do, then, I started to pull punches.

It wasn’t hard to justify, but it wasn’t fun for me anymore. I instead focused on big narratives that were occasionally peppered with quite combat. This led to burnout as I’d plan for one set of things and have to replan when the players went right instead of left. Then I thought, “I’ll go back to modules; better balance there.” Except I don’t like modules, because it turns out what had been balancing them for me wasn’t the design, but rather my own inability with the game. I practiced, practice perfects, and so suddenly because I was playing, essentially, 5x the D&D as anyone else; managing dozens of characters and learning statblocks and rules deeply and confidently, I was simply better at the game than my players. It became a problem to the point of needing to nerf fights preemptively, make a lot of “well this monster does have a -1 int modifier so maybe they’re just stupid and die.”

It sucks. There has to be a better way.

Go

In 2013, a company in London published some work about the use of reinforcement learning to play some simple (in terms of size-of-inputspace games) like Breakout, Pong, and others (which were nonetheless complex in their outputspace¹). This technique led to extremely good play, and moreover, it generalized to other games like Space Invaders, Beamrider, and Q*bert. In 2015, this same company decided to take on a real giant, Go.

Go is a two-player abstract game about placing stones in a grid. The rules are simple and I do not know them. I am sure Go is a lovely game to play, I don’t particularly care for it. I have trouble reading a go board, worse than a chess board (and I’m bad at reading those too). I don’t know if it’s the dyslexia or if I’m just the wrong kind of nerd for it, but Go has always bounced off my brain.

What hasn’t bounced off, though, is the fact that Go is a very deep game. For some perspective, we can think about it in the context of Chess. Chess, ultimately has only 32 pieces, with limited movesets, on a 64 square board. Number of possible games of chess is astronomical, and the most powerful engines, playing in real time, might search 30 plies (about 15 full moves) ahead. That search might take a few seconds on a modern computer.

Comparatively, Go is much simpler, there are only two kinds of pieces, but the board is much bigger at 19x19, and the number of possible moves is essentially 19*19 - number-of-occupied-squares, which is extremely large after even a few ply. The technical term here is that Go has a high ‘branching factor’, and there are many more moves to be made. As a result, good play still needs to look very far ahead, and looking far ahead is much harder, and in fact since the pieces do less, looking even further is more necessary to determine good play. For many years, Go was unassailable by computer; even the best programs only achieving slightly-better-than-amateur level play.

We’re talking Stockfish Level 3 on lichess kind of play. I remember trying to learn go and being able to occasionally beat the best engine on it’s hardest setting in 2012, so the gap was wide.

In 2015 though, DeepMind released AlphaGo, which combined statistical analysis and RL techniques to play Go significantly better than humans, famously beating very strong Go players like Lee Sedol and Ke Jie.

Why am I telling you about AlphaGo?

Complexity requires Learning

D&D is a complex game. There are a dozen classes, each of which has unique mechanics, spells, abilities, and customizations. There are hundreds of monsters with their own abilities and mechanics. There is an entire environment to interact with, move through, and exploit. The combat system itself introduces effects, optional moves, resources to manage, and many more besides.

D&D is also a simple game. Almost everything comes down to a few multiple choice questions. “I have some resources, I want to accomplish Y goals, I can choose to spend resources on abilities to alter the state of the game.” Every action is built out of the same parts. Checks and Saves measure a roll against a threshold; Contested Checks roll against a dynamic threshold; Value rolls tell you how much of something happens. There are no other real roll types in D&D, and almost every action is some cipher of “Roll some combination of Check or Saves, Contested Check, or Value rolls and make a state update based on the result.”

It’s not hard to play D&D because the mechanics are easy. It’s very hard to be good at D&D because there are so many variants of the mechanics and so many variables to consider in how the results of your actions might interact with the world.

That is to say, D&D has pretty limited input space, but a wide output space. Sound familiar?

AlphaDM Uruk-hai

I actually don’t want to replace myself entirely, I’m reminded of a Brennan Lee Mulligan conversation where he lays out the primary reason he uses D&D as opposed to some other ruleset is precisely because D&D combat is deep and rich, and it’s an area where he doesn’t have a lot of experience. His battle scenes are relatively few in number, they’re generally set pieces, and I am positive he has access to actual playtesting, which makes this make lots of sense. If you’re trying to do all that on your own though, it’s quite difficult.

I also want a combat system I don’t have to think about. To be honest, there are times where it’d be nice to say, “Alright combat is happening I need to jump in for the occasional bit of narration but otherwise it runs itself.”

To that end, I started on uruk-hai. This is a collection of projects, with the goal, ultimately, it so ‘solve’ balance by experimentation. lurtz is a D&D combat engine, supporting both the 2024 and 2014 editions of the 5e ruleset. Eventually I’d like to try to include other combat systems (you’ll understand how and why in a moment). uruk learns to play optimal D&D by self-training via lurtz, orthanc records an analog of “Elo” score based on an open arena model. You create an encounter and let it fight hundreds or thousands of times against ‘perfect’ players, and determine how it fairs against different party and power compositions.

I intend to integrate this with FoundryVTT at some point and allow it to manage combat to some extent directly; the GM sets an encounter, simulate it against their player’s party composition, and then adjust the agent to modify difficulty on the fly.

It’s also just a pretty fun data problem, the design of lurtz is cool as it is intended to be runnable on the GPU directly.

lurtz is also a fully autonomous project. I wrote precisely none of it’s code (and it definitely looks like agent-written code), but instead spent quite a long time building a suite of validation tests without ever having written any actual code. Instead, each action is compiled to an abstract machine that Lurtz can run. An action like “Cast Firebolt at $Monster” gets turned into a sequence of Opcodes like:

    set_caster $Current
    set_target $Monster
    spend_action $Current
    if attack hits $monster:
        set_damage_type(Fire)
        do_damage(1d10)

I built these in tandem with the machine, I’d write a narrative, walk through what the engine should do, and then write some opcodes. The machine kept track of which opcodes we added, we’d iterate a bit to ensure some minimal set, and design Cucumber scenarios to capture that the engine produced the expected output sequence. Once I was sure these validation tests were roughly correct, I set the machine off to implement the underlying spec.

This underlying VM is generic across editions and aims to be system agnostic more generally. Since each action is essentially an instantiation of some underlying template, this allows us to minimize the actual number of needed opcodes to go from “I cast Firebolt at Monster” to an actual implementation. It also let’s us be extensible since future mechanics just need to provide a compilation path.

The other upside of this is the “VM” here can be made quite lean; and since we already depend on a seeded RNG, we can potentially run many of these on a single GPU along with the RL agents being trained on it. By locating many of these on a GPU, we can have a single RL process playing many games simultaneously without ever leaving the GPU; which I hope will improve training speed.

My plan is to use an open-arena approach to training; early experiments show this is a quite servicable way of running training that gives good convergence rate while allowing for some other training benefits; in particular drop-in/drop-out compositions. This model allows for some very convenient experiments:

1. Given a fixed party of PCs, assuming they play optimally, what encounters exist that are within a few hundred Elo of the party?
2. Given a fixed party of PCs, what is the effect on the party’s Elo of giving one specific PC an item with a unique combat ability?
3. What is an optimal composition of PCs given a fixed XP budget to spend?
4. Given an enemy encounter, what is the incremental effect of swapping one monster for another of equivalent CR?
5. How closely does CR map to the statistical reality of the monster it describes? That is, across some large number of compositions, what is the net effect of adding/removing the monster on any such encounter, and how closely does that effect match other monsters of that CR?
6. Given two encounters of equivalent CR, how closely do they perform against a wide variety of party compositions of appropriate level?

I dislike CR because it is a one-dimensional description of a very high dimensional problem, and in many ways uruk-hai is way to try to understand exactly what CR measures and how well it actually does it.

lurtz being fully-autonomous/‘vibed’ also serves another purpose. I suspect that this class of code will shortly be the majority of code. Agents can easily produce far more LOC than any human, even the most hyperlexic among us, and because it works even a little, it will almost certainly dominate every codebase. The question is not ‘if’, but ‘when’, and ‘how do we deal with it.’

Practice perfects, and one cannot practice without examples. I hate toy examples, and lurtz is not a toy example. It currently is running around 100KLOC, I suspect if I were working from scratch and doing it by hand I’d be in the 50KLOC region. Many of those lines are Claudes insufferable commenting habit, but I would guess there is around 80-90KLOC of actual code there, and much of it is quite bad. My goal was not to produce good code here, just working code. The spec took about two weeks of conscious effort and back and forth iteration with the machine, but the engine has been getting churned on for about 2-3 solid weeks of a majority of my token budget from a 20x Max plan of Claude.

Part of the relative slowness in development (and incidentally while this doesn’t feel particularly ‘vibecoded’ to me, despite me having written no code and read fairly little of it) is how I’ve approach coding standards, but part of it I think is intrinsic to the way LLMs work with codebases. Right now the standard is “0 failing tests, >99% line and branch coverage, 0 warnings, 0 lints, ~0 mutant survivors.” These are much higher standards than I set for myself (I am personally quite pleased with ~80-90% coverage and ‘relatively few’ mutant survivors), but in this context they have managed to keep the output results reasonably sane despite being fully hands off. This led to a whole sidequest of getting mutant runs sharded across my R730s and other equipment so I could shrink the runtime from 4d -> 4h².

My plan now is to finish the remaining items on lurtz’s implementation; and get my ~400 scenario suite passing. Then I’m going to take some stock before letting the machine refactor the heck out of it, and trying to find any corners I can to flesh out the suite. Once that’s done, I plan to write a second implementation using my preferred LLM-involved workflow, which is “I implement some stuff, then when it’s tedious refactor time, the machine takes over.” I’ll probably do this while getting the RL side (uruk) hooked back up (early iterations of lurtz implemented a simple version with only melee combat and did the full training flow with open arena³.

Once I can reproduce a similar result with the current engine, and I’m confident it’s actually compliant and mostly bug-free, I’ll be able to start doing some of those big training items and probably porting to GPU as I mentioned.