28-AUG-2026

28-AUG-2026

1900

I’m in the ‘time to split the monolith up’ stage of this. Some interesting datapoints.

First, I’m roughly 2:1 lines-of-test to lines-of-code. Part of this is double-specification, the Cuke spec covers all the SRD content, and I have similar cukes for some of the PHB content I’m interested in testing (more on that in a minute). The agents also write tests, and they also frequently are made to chase mutants and coverage. This results in a pretty sizable codebase. The whole subproject is about 100KLOC and only about 30KLOC is the actual engine. This is pretty close to where I likely would’ve landed. If anything, I’d’ve guessed a handbuilt engine would be like, 50KLOC of engine code and more like 10-20KLOC of tests, so net smaller but probably more mutants and stuff.

One thing that has been extremely effective in fighting the tendency for agents to write complex logic that is prone to mutation is, of course, mutant testing. Agents are fond of defensive programming techniques in lieu of structural changes to eliminate classes of bugs. Mutation routinely revealed these areas, and coupled with a draconian 5% comments-by-total-LOC, it hasn’t been able to yap its way through. I noticed that comments were often used to mark ‘traps’ which were actually just bad design in disguise. At one point the agent complained about having to remove one of seven separate trap comments on a function.

The current work is splitting everything out, the architecture is, as mentioned previously, a compilation pipeline from intended-action to some bytecode that’ll run on an interpreter. Worth noting this is 100% rust until you get to the RL stuff later (uruk), which is python. There is, therefore, a py03 binding to all this. The thing is broadly split into the following crates:

• lurtz: A prelude/common re-export site.
• lurtz-engine: the bytecode interpreter
• lurtz-python: py03 bindings
• lurtz-srd: The core compilers for action -> bytecode; implements the 2014 and 2024 SRDs.
• lurtz-$class: Implementation of class-specific, non-SRD features/mechanics/etc from the PHB.
• lurtz-{equipment,monsters,spells}-{srd,phb}: specific implementation of spells/monsters from the srd/phb, per crate.

There are internal crate dependencies (lurtz-$class will probably depend on lurtz-spells-* when it enumerates a spell list for a given class), but I suspect most of those crates to be pretty small. The pattern will extend to cover splatbooks/UA/homebrew stuff as needed.

Once I’ve got the split done, I need to start working out how to do some more end-to-end validations. Once tricky thing here is that I am doing largely a opaque validation here. I need to craft some scenarios that will guarantee various mechanics get exercised and then validate they’re done correctly. I trust the Cukes because I wrote them, but they aren’t the obvious “Here is a battle that exercises all the SRD mechanics and ensures they are properly tracked in at least this instance.” In aggregate these cukes are useful because they ensure the agent stays trying to build the system I want instead of whatever it invents (and it really likes to invent stuff), but I don’t think it properly validates the engine.

Instead, I have a different idea. I know some basic facts about D&D. Given a party of a particular composisition and size, presented with an enemy encounter that relies heavily on a particular underlying feature to provide combat synergy, absent the correct implementation of the feature the battle will statistically go one way or another. Consider a party of 3 level 1 fighters, and an encounter with 10 zombies. If the Zombies lack a proper implementation of Undead Fortitude, then this is roughly equivalent to a nerfed encounter with Wild Dogs¹.

I’m not limited to ‘real’ monsters, though, I can instead create arbitrary fixture monsters that have a very narrow set of features that I can incrementally add and remove. In the same way we expect a test to fail under mutation, we should expect to see the average winrate of some standard party rise or fall under the presence of a new feature added or removed. My plan is to use exactly this – track objective statistics, like round length, damage done/recieved/healed, resource spend, etc. For some features we should see a dramatic change (Undead Fortitude is quite good), but an extreme change (compared to the change created by other mechanics) indicates an area that needs further review.

The nice thing about this is it’s pretty straightforward to automate. I just need to run a sampling of tests for… every configuration of party, at every level, against every enemy encounter.

Obviously that’s a little over the top to simply brute force, but I already want to do essentially this anyway for the big arena training/Elo measurement idea; this is just an alternative probe that can help validate the engine is implemented correctly. Approaching the validation statistically also means that I can generalize a hand validation of some subset of runs (i.e., I review the produced log and validate it went the way the SRD/PHB expects), and then generalize to all the others from those validated runs.

I have to be careful of interaction corner cases, but my hope is those are relatively few and ideally relatively well-known already.

Essentially the idea is “Identify some pier points, build the validation on them by statistical self-comparison.”