disturbed-circles

28-AUG-2026

2026-08-28T00:00:00+00:00
28-AUG-2026</h1>

1900</h2>
I’m in the ‘time to split the monolith up’ stage of this. Some interesting datapoints.
First, I’m roughly 2:1 lines-of-test to lines-of-code. Part of this is double-specification, the Cuke spec covers all the SRD content, and I have similar cukes for some of the PHB content I’m interested in testing (more on that in a minute). The agents also write tests, and they also frequently are made to chase mutants and coverage. This results in a pretty sizable codebase. The whole subproject is about 100KLOC and only about 30KLOC is the actual engine. This is pretty close to where I likely would’ve landed. If anything, I’d’ve guessed a handbuilt engine would be like, 50KLOC of engine code and more like 10-20KLOC of tests, so net smaller but probably more mutants and stuff.
One thing that has been extremely effective in fighting the tendency for agents to write complex logic that is prone to mutation is, of course, mutant testing. Agents are fond of defensive programming techniques in lieu of structural changes to eliminate classes of bugs. Mutation routinely revealed these areas, and coupled with a draconian 5% comments-by-total-LOC, it hasn’t been able to yap</a> its way through. I noticed that comments were often used to mark ‘traps’ which were actually just bad design in disguise. At one point the agent complained about having to remove one of seven separate trap comments on a function.
The current work is splitting everything out, the architecture is, as mentioned previously</a>, a compilation pipeline from intended-action to some bytecode that’ll run on an interpreter. Worth noting this is 100% rust until you get to the RL stuff later (`uruk</code>), which is python. There is, therefore, a py03 binding to all this. The thing is broadly split into the following crates:`
`lurtz</code>: A prelude/common re-export site.</li> lurtz-engine</code>: the bytecode interpreter</li> lurtz-python</code>: py03 bindings</li> lurtz-srd</code>: The core compilers for action -> bytecode; implements the 2014 and 2024 SRDs.</li> lurtz-$class</code>: Implementation of class-specific, non-SRD features/mechanics/etc from the PHB.</li>`
lurtz-{equipment,monsters,spells}-{srd,phb}</code>: specific implementation of spells/monsters from the srd/phb, per crate.</li> </ul> There are internal crate dependencies (lurtz-$class</code> will probably depend on lurtz-spells-*</code> when it enumerates a spell list for a given class), but I suspect most of those crates to be pretty small. The pattern will extend to cover splatbooks/UA/homebrew stuff as needed. Once I’ve got the split done, I need to start working out how to do some more end-to-end validations. Once tricky thing here is that I am doing largely a opaque validation here. I need to craft some scenarios that will guarantee various mechanics get exercised and then validate they’re done correctly. I trust the Cukes because I wrote them, but they aren’t the obvious “Here is a battle that exercises all the SRD mechanics and ensures they are properly tracked in at least this instance.” In aggregate these cukes are useful because they ensure the agent stays trying to build the system I want instead of whatever it invents (and it really likes to invent stuff), but I don’t think it properly validates the engine.Instead, I have a different idea. I know some basic facts about D&D. Given a party of a particular composisition and size, presented with an enemy encounter that relies heavily on a particular underlying feature to provide combat synergy, absent the correct implementation of the feature the battle will statistically go one way or another. Consider a party of 3 level 1 fighters, and an encounter with 10 zombies. If the Zombies lack a proper implementation of Undead Fortitude</a>, then this is roughly equivalent to a nerfed encounter with Wild Dogs</a>1</a>. I’m not limited to ‘real’ monsters, though, I can instead create arbitrary fixture monsters that have a very narrow set of features that I can incrementally add and remove. In the same way we expect a test to fail under mutation, we should expect to see the average winrate of some standard party rise or fall under the presence of a new feature added or removed. My plan is to use exactly this – track objective statistics, like round length, damage done/recieved/healed, resource spend, etc. For some features we should see a dramatic change (Undead Fortitude is quite good), but an extreme change (compared to the change created by other mechanics) indicates an area that needs further review. The nice thing about this is it’s pretty straightforward to automate. I just need to run a sampling of tests for… every configuration of party, at every level, against every enemy encounter. Obviously that’s a little over the top to simply brute force, but I already want to do essentially this anyway for the big arena training/Elo measurement idea; this is just an alternative probe that can help validate the engine is implemented correctly. Approaching the validation statistically also means that I can generalize a hand validation of some subset of runs (i.e., I review the produced log and validate it went the way the SRD/PHB expects), and then generalize to all the others from those validated runs. I have to be careful of interaction corner cases, but my hope is those are relatively few and ideally relatively well-known already. Essentially the idea is “Identify some pier points, build the validation on them by statistical self-comparison.” Sorry for two different stat sources there, I’m just a DM, a simple innocent DM. ↩</a> </li> </ol> </section>



26-AUG-2026
2026-08-26T00:00:00+00:00
26-AUG-2026</h1>
0940</h2>
I’m taking some vacation before Mayhem</em> heads off to kindergarten. This morning is orientation and she is nervous and
excited and off to orientation this morning so I had some time to do some review of lurtz</code>, and something cool
happened.</p>
The design of lurtz</code> has always been a little haphazard owing to the vibe-y nature of it, I knew that going in. I also
knew that it was likely I would step over the SRD/PHB line as I am sure Claude is trained – like all the perpetrators
of the grandest larceny in the history of theft – on all the PHB stuff, and it was going to leak in. My understanding
of the SRD and OGL and all that is this:</p>

Build whatever you like, these things govern what you distribute</em>.</li>
Distributing stuff based on the SRD content is fine.</li>
If you distribute PHB content, or any other content from WOTC, the full force of the legal department of Hasbro will
set ruin upon thee and thy house for ten generations.</li>
</ol>
I do not want to piss off Hasbro, but I am sure lurtz</code> has like, PHB and some Arcana and such.</p>
Now, this may seem like a problem if you want to build a D&D engine to manage balance for all</em> of D&D. I agree it is a
constraint, but lurtz</code> can be clever, and part of why I vibecoded it was to sort out how best to design around this.</p>
I knew I wanted a bytecode machine, I wasn’t sure why. I try not to make design decisions like that without having a
reason more than, “Because they’re cool.” When I first started learning Rust, I tried the
synacor</a>1</a></sup> challenge, which required building a weird little bytecode
machine among some other fun puzzles I won’t spoil. I’ve really liked these little guys since taking a low-level systems
course in college based on some weird little 16b ISA that had a little simulator and stuff, it was pretty rad. I liked
the architecture enough I started building hazel</a> along those lines, before getting
distracted by other shinies and landing here.</p>
In any case, I knew I wanted to take that approach, but that didn’t really satisfy me for a reason, I decided eventually
though that a good enough reason was to see how the tokenmachine managed with the architecture.</p>
Overall, fine, I guess. Not great, not terrible. It took a lot of nudging to push it towards that model, and it took a
lot more to keep it there and get it not to re-invent every single wheel. Normally I think these systems have good
API-sense, but in reality what they have is majority</em> API sense. That is, whatever the greatest common subset of APIs
people use, that is what it will inevitably lean to use; and since the APIs themselves are probably part of it’s
training set, there is a huge</em> sample of DIY in there. Since the sorts of libraries for this less common architecture
aren’t as readily available or used as libraries for a more ‘ad hoc/amorphous’ architecture2</a></sup>, it is more likely to
grab a generic, frequently used library for which it has lots of examples, or just DIY everything. I had to point it to
bitmask</code> a bunch of times.</p>
As I was starting to dig through some of the generated code, I started seeing it – PHB-sign.</em> A little scattering of
Battle Master rules has snuck it’s way into the API. I need this code, it’s not going anywhere, by #1 above I can keep
it I just can’t give it away, so the obvious solution is to stick it in a crate. In a lurtz-fighter</code> crate it is safely
tucked away where I can avoid publishing it, but how can I keep it’s PHB-ness off the core engine?</p>
When you design an ISA, you don’t take responsibility for every program written in that ISA, you just design the ISA.
When Intel had its FDIV bug, it was responsible because the ISA was broken. When your software crashes because someone
divided by zero, the author of the software is at fault, not Intel.</p>
In the same way, lurtz</code> doesn’t need to provide direct APIs for any of the mechanics – in any system –
lurtz</code>-as-a-VM should be agnostic to the game system entirely. We need that for 2014/2024 support anyway. In a way, you
can think of a battle as an incremental compilation problem. We have a battle-log, I need to turn that sequence of
actions into a sequence of opcodes on a VM such that the state of the VM after executing those codes contains sufficient
information to reconstruct the battle at that state.</p>
That is to say, I can manage some auxillary structure to manage all the things a Battle-Master class would need without
actually implementing the class directly in lurtz</code>. This is a much better reason and I retroactively use this</em> one as
justification for this model. It has some nice knock-on effects too.</p>
Right now lurtz</code> is one big crate with everything piled in. It has an action API for all the things you can do in D&D,
it has the engine itself, all the opcode stuff, everything, one big pile. Obviously I can split out lurtz-$class</code> like
mentioned, but also lurtz-monsters</code> and lurtz-players</code> can be split out. Players and Monster statblocks both translate
to ‘intialization’ opcodes that start a battle log and configure the environment. These translate into datastructures
inside the VM as well. I can then further split out lurtz-srd</code>, which contains the core SRD actions, upon which all
these other crates depend, now lurtz</code> is the core</em> crate with only</em> the engine in it. Each crate can have it’s own
validation set, which we compile and assert on the engine state. These now track the implementations much more tightly,
which will greatly improve my ever growing mutant problem.</p>
It has a natural extension pattern as well, new UA or splatbooks can be included as another crate, independently tested.
It means that lurtz</code> can be optimized aggressively for the GPU and the ‘compiler’ crates can expect to run more or less
uninterrupted on the CPU. I suppose in the final tally the training process probably looks like some number of agents
playing as many games as they can on the GPU; they make a big pile of next move decisions while the CPU grabs those
moves, compiles them, and sends them back to the engine to update in parallel. I’m not sure if I could get 100% of that
loop on the GPU but it doesn’t seem impossible</em>. That’s either inexperience or very clever, I’m not sure which.</p>
It also leaves open a door to test other combat systems. Pathfinder is an obvious target, but there are lots of systems
that look like D20-style combat.</p>
All this would’ve probably been findable with a whole bunch of anxious thought about how things might work. By the time
I got there I’d’ve been so worried about implementing it wrong I’d’ve given up. I’ve put about two proper weeks of
thought into lurtz</code> at this point and have a working program. It’s not good, it’s not super well factored (yet), but
it’s reasonably functional and it’s easy enough to change. Because I had a working program, I was able to see</em> this
refactor, instead of having to imagine</em> it ahead of time. I can look at this code and this structure falls out.</p>
I will definitely</em> need to rewrite every one of these crates at some point3</a></sup>, but having them to hand when finding
the architecture makes code feel much</em> more play-doh than it used to.</p>
I quit working on hazel</code> partially because I was getting lost down a rabbit hole of screwing with the trait system to
do some weird message passing stuff. I also quit on it because I kept running into problems that needed a big fat
refactor that would take days of churning over error messages and making tiny adjustments. If it weren’t for the fact
that I do</em> want to keep one project as agent-free as I can, I’d be using an LLM for those changes today and probably
still be working on it. That hypothesis is supported by this result with lurtz. My favorite part of coding is
refactoring, because it is really difficult to hold both the observed and the desired in your head at the same time;
much less to plot a course from one to the other. My strategy has been to rely on good observation skills and git to
slowly move from one to another. LLMs offer a new option here. lurtz</code> has been sort of ‘3d printed’ into place. It’s
cheap plastic, it’s all one mold instead of several moving parts, but it gives me an object I can hold in my hands.</p>
I occasionally turn wood; mostly to make handles for stuff. One of the things about turning handles is that – while you
need a template to get a repeatable result, no two handles are ever finished exactly alike. You always use the same
process, or at least I do:</p>

Turn stock to the template dimensions</li>
Cleaning pass with a skew</li>
Rough pass with 60 grit</li>
Stop lathe, feel the handle.</li>
Start lath, skew or sand.</li>
Repeat 4 and 5 until it’s right, adjust grit as necessary.</li>
Friction polish with BLO, Wax on the Lathe, Buff.</li>
</ol>
You can’t make a handle for something without trying to use it first, there is no substitute for holding it in your
hand. Trying to divine this architecture for lurtz</code> ahead of time, especially given my vanishingly close to zero prior
experience in game making, would’ve been like trying to make a perfect handle template – not possible.</p>
In any case, this was a very nice validation of what I hoped would happen when I started this, I had hoped that LLMs
would work nicely this way and it has. I have a natural skeleton to build the next version of this against, and all it
took was two weeks of prompting, not exclusively on this but majority on this, most of which were status report requests
/ disambiguating parts of the plan. I’ll talk about how I set up this project another time. I kept all the conversations
archived so I should be able to do some forensics on it.</p>



The original challenge page is down, this is a mirror I found
here</a>. I hope it
comes back at some point, but maybe someone oughta just build a new one. :) ↩</a></p>
</li>

I couldn’t find a way to make that not sound perjorative. I don’t mean it that way. Ad hoc architecture is not a
crime, it’s a choice with trade-offs. ↩</a></p>
</li>

I wrote that and immediately went to hedge – maybe this is true, but I could see a world where I rewrite some</em>
of the crates, then make the machine copy</em> that style to the others. Still a rewrite, but by what fraction? ↩</a></p>
</li>
</ol>
</section>


25-AUG-2026
2026-08-25T00:00:00+00:00
25-AUG-2026</h1>
1352</h2>
Added some quotes to the rotation, I don’t love how I have it set up right now, probably will move to a single
structured file that compiles to the multiple files at build time or something, good enough for now though.</p>
Work on stuff here has slowed as I’ve been working much more on getting the kiddo ready for kindergarten, and planning
an office move</a> to a much larger space. I’ve started thinking again about a whole-lab
sound</em> setup as well. It’s been on my punchlist for several years now to build some sort of network-attached-sound
system. I have a number of SFFs and RPi’s that connect to most of the noisemaking things in my house and a many-to-many
mapping system would be a very nice thing.</p>
It coincides with me wanting to push my current daily driver machine, a M2 Studio named Hedges</code>, to be more of a
dedicated music/recording machine and less of a ‘do everything’ machine. When I initially concieved of it, I had planned
an analog approach of just sending everything’s sound to a central mixing console, and then having the channels AUX into
Hedges</code> UMC1820, but this would only ever really be useful in relatively small area, as I don’t want to try to run the
50m of speakerwire through walls, especially given it’d need to be shielded. I looked into using
JACK</a> at the time and plan to look again, but I planned all of this in the days where wiring up
JACK required what felt like a PhD in audio engineering, I’m hoping LLMs can take some sting out of it for me. I’m not
looking to be a pro audio routing engineer here, just get my noises all filed into one box where I can mangle them and
spit them out into my ears.</p>
Much of that work may be hidden as I haven’t sorted out a way to expose my monorepo that doesn’t also expose the
absolute mess</em> that is my monorepo. Most of my projects start out as subprojects of angband</code>. angband</code> currently
contains the live version of minas-tarwon</code>, the homelab infrastructure from metal -> k8s running services; truncheon</code>
and all it’s constituent parts (including an army worthy</em> of Mordor</a>); and a
bunch of other stuff that’s worth talking about, but not necessarily sharing code for (yet). I set up
josh</a> a while ago and have half an idea about using it to extract some of the
code to mirror repos. The main issue is most of the monorepo is unsharable because I don’t build those projects with the
intent of public use, so I’m quite happy to hardcode all sorts of stuff about my system and network. For instance, I
have in that repo a tool called thrak</code>. thrak</code> is a CLI for the monorepo itself, it is not a very well made CLI, but
more because it has no specific remit other than, “I need a place for this specific tool to live that I can use from my
laptop when I’m in this repo.” Some people use just</code> or make</code> or rake</code> or whatever, I’ve been all those people, but
at some point I always end up thinking, “If only I’d just written a damn CLI, I wouldn’t be wondering about the call
order or having to write so much friggin’ shell.”</p>
I suppose probably the better thing to do is a careful migration where the tool is built first and projects imported,
but that is a lot of work for a tool that has a constituency of one.</p>
Since ‘constituency of one’ describes most of those projects, most of those projects are not really the sort of thing
I’d feel great putting out into the world, and so much remains hidden.</p>
I don’t have a solution I just felt like writing a bit today and this is what came out.</p>


Whom do you serve?
2026-08-25T00:00:00+00:00

  </iframe>
</div>
Dungeon Mastery</h1>
Hi, I’m Joe, and I’m a forever-DM.</p>

Chorus: “Hi, Joe.”</p>
</blockquote>
I’ve been forever-DMing since around 2015, I played a bit in college before that and ran a one-shot or several, but it
wasn’t until then that I’d tried to run anything homebrew. Up until then I’d played canned encounters that had been
extensively playtested, I’m sure, by folks who were quite good at the game and who made sure that the encounters would
be balanced nicely.</p>
It’s not, therefore, my or my player’s fault when I would routinely trounce them. I had all the advantages. I knew their
characters, I knew the monsters; it simply wasn’t my fault that playing the game well</em> meant also not playing for very
long</em> because it turns out ‘being everything else’ means you get to create pretty favorable conditions. I did what all
DMs do, then, I started to pull punches.</p>
It wasn’t hard to justify, but it wasn’t fun for me anymore. I instead focused on big narratives that were occasionally
peppered with quite combat. This led to burnout as I’d plan for one set of things and have to replan when the players
went right instead of left. Then I thought, “I’ll go back to modules; better balance there.” Except I don’t like</em>
modules, because it turns out what had been balancing them for me wasn’t the design, but rather my own inability with
the game. I practiced, practice perfects, and so suddenly because I was playing, essentially, 5x the D&D as anyone else;
managing dozens of characters and learning statblocks and rules deeply and confidently, I was simply better</em> at the
game than my players. It became a problem to the point of needing to nerf fights preemptively, make a lot of “well this
monster does</em> have a -1 int modifier so maybe they’re just stupid and die.”</p>
It sucks. There has to be a better way.</p>
Go</h1>
In 2013, a company in London published some work about the use of reinforcement learning to play some simple (in terms
of size-of-inputspace games) like Breakout</em>, Pong</em>, and others (which were nonetheless complex in their
outputspace</em>1</a></sup>). This technique led to extremely good play, and moreover, it generalized to other games like Space
Invaders</em>, Beamrider</em>, and Q*bert</em>. In 2015, this same company decided to take on a real giant, Go</em>.</p>
Go is a two-player abstract game about placing stones in a grid. The rules are simple and I do not know them. I am sure
Go is a lovely game to play, I don’t particularly care for it. I have trouble reading a go board, worse than a chess
board (and I’m bad at reading those too). I don’t know if it’s the dyslexia or if I’m just the wrong kind of nerd for
it, but Go has always bounced off my brain.</p>
What hasn’t bounced off, though, is the fact that Go is a very deep</em> game. For some perspective, we can think about it
in the context of Chess. Chess, ultimately has only 32 pieces, with limited movesets, on a 64 square board. Number of
possible games of chess is astronomical, and the most powerful engines, playing in real time, might search 30 plies
(about 15 full moves) ahead. That search might take a few seconds on a modern computer.</p>
Comparatively, Go is much simpler, there are only two kinds of pieces, but the board is much</em> bigger at 19x19, and the
number of possible moves is essentially 19*19 - number-of-occupied-squares</code>, which is extremely</em> large after even a
few ply. The technical term here is that Go has a high ‘branching factor’, and there are many more moves to be made. As
a result, good play still needs to look very far ahead, and looking far ahead is much harder, and in fact since the
pieces</em> do less, looking even</em> further is more necessary to determine good play. For many years, Go was unassailable
by computer; even the best programs only achieving slightly-better-than-amateur level play.</p>
We’re talking Stockfish Level 3 on lichess kind of play. I remember trying to learn go and being able to occasionally
beat the best engine on it’s hardest setting in 2012, so the gap was wide.</p>
In 2015 though, DeepMind released AlphaGo, which combined statistical analysis and RL techniques to play Go
significantly better than humans, famously beating very strong Go players like Lee Sedol and Ke Jie.</p>
Why am I telling you about AlphaGo?</p>
Complexity requires Learning</h1>
D&D is a complex game. There are a dozen classes, each of which has unique mechanics, spells, abilities, and
customizations. There are hundreds of monsters with their own abilities and mechanics. There is an entire environment to
interact with, move through, and exploit. The combat system itself introduces effects, optional moves, resources to
manage, and many more besides.</p>
D&D is also a simple game. Almost everything comes down to a few multiple choice questions. “I have some resources, I want to
accomplish Y goals, I can choose to spend resources on abilities to alter the state of the game.” Every action is built
out of the same parts. Checks and Saves measure a roll against a threshold; Contested Checks roll against a dynamic
threshold; Value rolls tell you how much of something happens. There are no other real roll types in D&D, and almost
every action is some cipher of “Roll some combination of Check or Saves, Contested Check, or Value rolls and make a
state update based on the result.”</p>
It’s not hard to play D&D because the mechanics are easy. It’s very</em> hard to be good at D&D because there are so many
variants of the mechanics and so many variables to consider in how the results of your actions might interact with the
world.</p>
That is to say, D&D has pretty limited</em> input space, but a wide</em> output space. Sound familiar?</p>
AlphaDM</del> Uruk-hai</h1>
I actually don’t want to replace myself entirely, I’m reminded of a Brennan Lee Mulligan conversation where he lays out
the primary reason he uses D&D as opposed to some other ruleset is precisely because D&D combat is deep and rich, and
it’s an area where he doesn’t have a lot of experience. His battle scenes are relatively few in number, they’re
generally set pieces, and I am positive he has access to actual</em> playtesting, which makes this make lots of sense. If
you’re trying to do all that on your own though, it’s quite difficult.</p>
I also want a combat system I don’t have to think about. To be honest, there are times where it’d be nice to say,
“Alright combat is happening I need to jump in for the occasional bit of narration but otherwise it runs itself.”</p>
To that end, I started on uruk-hai</code>. This is a collection of projects, with the goal, ultimately, it so ‘solve’ balance
by experimentation. lurtz</code> is a D&D combat engine, supporting both the 2024 and 2014 editions of the 5e ruleset.
Eventually I’d like to try to include other combat systems (you’ll understand how and why in a moment). uruk</code> learns to
play optimal D&D by self-training via lurtz</code>, orthanc</code> records an analog of “Elo” score based on an open arena model.
You create an encounter and let it fight hundreds or thousands of times against ‘perfect’ players, and determine how it
fairs against different party and power compositions.</p>
I intend to integrate this with FoundryVTT at some point and allow it to manage combat to some extent directly; the GM
sets an encounter, simulate it against their player’s party composition, and then adjust the agent to modify difficulty
on the fly.</p>
It’s also just a pretty fun data problem, the design of lurtz</code> is cool as it is intended to be runnable on the GPU
directly.</p>
lurtz</code> is also</em> a fully autonomous project. I wrote precisely none of it’s code (and it definitely looks</em> like
agent-written code), but instead spent quite a long time building a suite of validation tests without ever having
written any actual code. Instead, each action is compiled</em> to an abstract machine that Lurtz can run. An action like
“Cast Firebolt at $Monster” gets turned into a sequence of Opcodes like:</p>
    set_caster $Current</span></span>
    set_target $Monster</span></span>
    spend_action $Current</span></span>
    if attack hits $monster:</span></span>
        set_damage_type(Fire)</span></span>
        do_damage(1d10)</span></span></code></pre>
I built these in tandem with the machine, I’d write a narrative, walk through what the engine should do, and then write
some opcodes. The machine kept track of which opcodes we added, we’d iterate a bit to ensure some minimal set, and
design Cucumber scenarios to capture that the engine produced the expected output sequence. Once I was sure these
validation tests were roughly correct, I set the machine off to implement the underlying spec.</p>
This underlying VM is generic across editions and aims to be system agnostic more generally. Since each action is
essentially an instantiation of some underlying template, this allows us to minimize the actual number of needed
opcodes to go from “I cast Firebolt at Monster” to an actual implementation. It also let’s us be extensible since
future mechanics just need to provide a compilation path.</p>
The other</em> upside of this is the “VM” here can be made quite lean; and since we already depend on a seeded RNG, we can
potentially run many of these on a single GPU along with the RL agents being trained on it. By locating many of these on
a GPU, we can have a single RL process playing many games simultaneously without ever leaving the GPU; which I hope will
improve training speed.</p>
My plan is to use an open-arena approach to training; early experiments show this is a quite servicable way of running
training that gives good convergence rate while allowing for some other training benefits; in particular
drop-in/drop-out compositions. This model allows for some very convenient experiments:</p>

Given a fixed party of PCs, assuming they play optimally, what encounters exist that are within a few hundred Elo of
the party?</li>
Given a fixed party of PCs, what is the effect on the party’s Elo of giving one specific PC an item with a unique
combat ability?</li>
What is an optimal composition of PCs given a fixed XP budget to spend?</li>
Given an enemy encounter, what is the incremental effect of swapping one monster for another of equivalent CR?</li>
How closely does CR map to the statistical reality of the monster it describes? That is, across some large number of
compositions, what is the net effect of adding/removing the monster on any such encounter, and how closely does that
effect match other monsters of that CR?</li>
Given two encounters of equivalent CR, how closely do they perform against a wide variety of party compositions of
appropriate level?</li>
</ol>
I dislike CR because it is a one-dimensional description of a very high dimensional problem, and in many ways uruk-hai</code>
is way to try to understand exactly what CR measures and how well it actually does it.</p>
lurtz</code> being fully-autonomous/‘vibed’ also serves another purpose. I suspect that this class of code will shortly be
the majority of code. Agents can easily produce far more LOC than any human, even the most hyperlexic among us, and
because it works even a little, it will almost certainly dominate every codebase. The question is not ‘if’, but ‘when’,
and ‘how do we deal with it.’</p>
Practice perfects, and one cannot practice without examples. I hate toy examples, and lurtz</code> is not a toy example. It
currently is running around 100KLOC, I suspect if I were working from scratch and doing it by hand I’d be in the 50KLOC
region. Many of those lines are Claudes insufferable</em> commenting habit, but I would guess there is around 80-90KLOC of
actual code there, and much of it is quite bad. My goal was not to produce good code here, just working code. The spec
took about two weeks of conscious effort and back and forth iteration with the machine, but the engine has been getting
churned on for about 2-3 solid weeks of a majority of my token budget from a 20x Max plan of Claude.</p>
Part of the relative slowness in development (and incidentally while this doesn’t feel particularly ‘vibecoded’ to me,
despite me having written no code and read fairly little of it) is how I’ve approach coding standards, but part of it I
think is intrinsic to the way LLMs work with codebases. Right now the standard is “0 failing tests, >99% line and branch
coverage, 0 warnings, 0 lints, ~0 mutant survivors.” These are much higher standards than I set for myself (I am
personally quite pleased with ~80-90% coverage and ‘relatively few’ mutant survivors), but in this context they have
managed to keep the output results reasonably sane despite being fully hands off. This led to a whole sidequest of
getting mutant runs sharded across my R730s and other equipment so I could shrink the runtime from 4d -> 4h2</a></sup>.</p>
My plan now is to finish the remaining items on lurtz</code>’s implementation; and get my ~400 scenario suite passing. Then
I’m going to take some stock before letting the machine refactor the heck out of it, and trying to find any corners I
can to flesh out the suite. Once that’s done, I plan to write a second</em> implementation using my preferred LLM-involved
workflow, which is “I implement some stuff, then when it’s tedious refactor time, the machine takes over.” I’ll probably
do this while getting the RL side (uruk</code>) hooked back up (early iterations of lurtz</code> implemented a simple version with
only melee combat and did the full training flow with open arena3</a></sup>.</p>
Once I can reproduce a similar result with the current engine, and I’m confident it’s actually compliant and mostly
bug-free, I’ll be able to start doing some of those big training items and probably porting to GPU as I mentioned.</p>



By this I mean; there are many simple actions that can be taken, the immediate effects of which are easy to
predict, but the distant effects of which are much harder to predict. In Breakout</em>, this is the number of bricks
and the exact pattern you have made allowing for different, more efficient strategies in quickly clearing bricks. In
Pong</em>, it’s the fact that while returning the ball is easy, knowing how your opponent might return it is not. ↩</a></p>
</li>

Along the way I accidentally reinvented the concept used by the mutagen</a>
project, which seems to be pretty dead (and also tied to a proc-macro which makes it hard to maintain, I imagine).
I have some hazy thoughts about trying something similar but re-using the mutator code in cargo-mutants</a>
to tag the IR in the right spots and hopefully avoid the need for a proc-macro/syntax hacking by directly modifying
the IR. The majority of time spent mutation-testing is compilation, so even avoiding a few such recompilations
could save many minutes in the long run. Even doing this solely</em> for the binops would probably be enough to take
my current 4h sharded run to like, an hour. ↩</a></p>
</li>

This is the only image I have from those runs. You can see it correctly sorts, though there are some monsters ‘out
of place’ with respect to their CRs. For the most part I think this is unimplemented mechanics for those monsters,
and the fact that this is all monsters-fighting-monsters, which is it’s own area of research.</p>



  
</a>
 ↩</a></li>
</ol>
</section>


17-AUG-2026
2026-08-17T00:00:00+00:00

[~] other projects in the log section that come from content managed here

mostly for projects that don’t really have natural repos of their own.</li>

Project Nobby</li>

The Garden</li>

The Office</li>
</ul>
</li>
[~] Music section?


Links to playlists</li>

Guitar inventory</li>

The Work</li>
Might could harvest some stuff from the old site?</li>
</ul>
</li>

Gehenna

A place where i drop ideas that I probably will never get to. I have very many of these.</li>
</ul>
</li>

FTA


HTML Gen

probably should follow the reorg/citation thing. Idea would be to take each env and include citation info
in it directly, then extract to some input format. Later I can add equivalent citation (i.e., where in
Munkres a def appears or something).</li>
not sure if I should keep it as latex as the primary language or extract to a different format/convention</li>
aim is Temu nLab, all my notes + cited definitions across my various books. One big map. Queryability would be
nice. Maybe a graphdb – ideally an atlas integration?</li>
</ul>
</li>
</ul>
</li>
</ul>
17-AUG-2026</h1>
1528</h2>
Work in the lab on getting my jellyfin container properly pinned to the GPU node in my cluster. I need to think about
GPU sharing at some point (everything just pins down to one of the cards in the fleet, occupying the whole thing).</p>
I’ve got some more subsections set up, and I need to spend some time porting stuff from the old site here, but I’m
pretty happy with how this is working out. Most of the sidebar is just pointing directly at repos and builds dynamically
from the LOG.md there, so I can keep my notes local and still have them centralize here. Much nicer than the obsidian
site I was running for a while.</p>
I am in the final stretch for my topology refresher and then I think I’m going to do the same with Hungerford, I’m
debating if I want to formalize it as I go or not. I really like algebra so it would be cool to formalize it, but I’m
starting to think I’m just procrastinating because I’m not confident the idea I have for GIYF is going to work how I
want. Anxiety is a complicated and frustrating thing.</p>


Music isn't the sound that music makes
2026-08-14T00:00:00+00:00
When I was first learning guitar, I went through a series of teachers. The first, a young man named Chris, he was in his
20s and wildly unprepared for the degree of weird I was going to bring. I was raised Evangelical1</a></sup> So my musical
horizons were limited to ‘approved’ music: hymns, classical music, and ‘CCM’2</a></sup>. CCM, if you are unfamiliar, is the sort
of sacchyrine, flavorless drivel that takes a bad love song and replaces the object of affection with Jesus. There are certainly some tunes that rise above the tepid dreck, but only few, and never by much.</p>
Chris liked to play Rock and Roll, Chris liked the Blues. Chris was confused when I asked him to teach me an Amy Grant
tune. Eventually Chris gave up. Though he did introduce me to Delta Blues, so six-of-one.</p>
After a brief interregnum, I ended up with my second teacher. Shirley was a very nice lady from Church, she played piano
and guitar, and thought that meant she could teach it. This is not an uncommon mistake people make, teaching is a
different problem than simple knowing. Shirley failed like all the others, I only had lessons with her for a few months.
Chris outlasted her.</p>
Another few months, another teacher from Church, she taught piano to my sister. She lasted two lessons and told me that
it was impossible to play the guitar without a pick. She wasn’t a very good player, either, having only picked up the
instrument a few months prior. Come to think of it I might’ve actually been playing longer than she had when she tried
to teach me.</p>
There was a long gap after that. I first got the guitar around 8 or 9, cheap Samick that probably cost 80$ at the time.
The action was too high, and the intonation was trash. I broke a string, but no one ever explained that that was normal
and you could just change them, so I played with no high E for a year or so. Finally I started taking lessons with Ron
from a music store one town over when I was about thirteen or so.</p>
Ron was, and remains, probably the best teacher I’ve had in any subject in my life. I had great teachers in college, but
Ron was the first teacher to take an honest interest in what I wanted out of guitar.</p>
Our first lesson was pretty funny. I kept my guitar in the cardboard box it came in. No gig bag, certainly no hardcase.
I dragged it up the stairs to the small practice room and pulled it out, ancient strings, one missing.</p>
“Okay, ooh, nice guitar, looks like you broke a string though.”</p>
“Yah, few months ago.”</p>
“You didn’t replace it?”</p>
“You can replace them?”</p>
For the next year or so, Ron taught me how to learn guitar. He didn’t merely teach me how to play, but how to think
about playing. I think of this now whenever I think about teaching anything, the hard part of teaching is not the
content of the subject, but the narrative of the subject. He introduced this concept early with one of my favorite music
exercises.</p>
“Play it wrong.”</p>
“What? What do you mean play it wrong?”</p>
“Take what you’re doing, the notes you’re playing, and screw up. I want you to play it off by fret or three.”</p>
“But it’ll sound terrible.”</p>
“Not in your head, that’s where I want it, don’t worry about the sound you’re making, listen to the music and just do
whatever feels natural. Be okay with it sounding wrong, that’s going to happen a lot.”</p>
“But it sounds bad when I do that.”</p>
“Play it anyway, even if it sounds bad, Music isn’t the sound that music makes, Music is in your mind, you have to
listen for it, chase it, sit in it, let it soak into your brain. Don’t try to force it, just let it move your hands, let
it move you, worry about the mechanics later.”</p>
Ringing in my mind, ever since, has been those words, “Music isn’t the sound that music makes.”</p>
I didn’t understand that lesson till many years later. After a few years of therapy for (what I assumed was) unrelated
Anxiety, I began to notice something happening to my guitar playing.</p>
I sounded good, like really good. I was enjoying what I played more, I was listening to what I played more. I didn’t
feel the same need to control it anymore, to make it perfect, I could just play. As I sat in front of my daughter, I
remember hearing myself play her tune3</a></sup>, and in my mind like a siren, “Music isn’t the sound that music makes.”</p>
I’ve been afraid for a majority of my adult life that if anyone ever heard me play, that the facade would break, and I
would be revealed an imposter. I’ve worried that they would tell me I was awful, and this thing which I love so much
would be ruined by their judgement. I still worry about that, but I’m working on it.</p>



This is a very deep hole I crawled out of; maybe someday I’ll talk more about how. ↩</a></p>
</li>

“Christian Contemporary Music” ↩</a></p>
</li>

This</a> is a good enough version as any to share. There is a specific motive I
use a lot, that’s what I mean when I say this. ↩</a></p>
</li>
</ol>
</section>


So what sound will I make?
2026-08-14T00:00:00+00:00
Well, I’m going to start by posting this, my youtube channel</a>.</p>
It’s on the frontpage, obviously, but worth posting the direct link.</p>
Right now I’m prepping to move my office where I currently record.</a>, so I probably won’t
have much in the way of new music in here till after I get settled down there. I’ve got plans to build out some better
recording space, there is too much stuff in my office to do any kind of decent recording configuration without having to
play three dimensional Twister while playing.</p>
I do have some old content from a previous iteration of the site which I’ll import. I’ve got a bunch of records left to
archive. So I’ll be bringing those over over time. For now this section will probably be quiet, but eventually I’m
hoping to make my Sesshin</code> style video a more regular occurence.</p>


The Move
2026-08-14T00:00:00+00:00
It’s quite small for the number of things I try to do in here. 10x15 or so. The basement is really just the whole first
floor of my house, a 20x25 untreated garage and a 40x25 basement floor with about 12x10 cutout for right-angle stairs,
and another 8x17 for mechanical. In this space I am planning to break into sections. The layout is roughly:</p>
 ------------------------------------------------------------------------------------</span></span>
 |            D       I                     .     |                                 |</span></span>
 |            I   2   W                     .     |                                 |</span></span>
 |            H       I                     .     |                                 |</span></span>
 |     1      I=.=D=.=I         4           .  M  |                                 |</span></span>
 |            I                             .     |                                 |</span></span>
 |            I                             .     |                                 |</span></span>
 |            I                             .     |            GARAGE               |</span></span>
 |=.=.=D=.=.=.o. . D . . ------------- . D . . . .|                                 |</span></span>
 |                       |  STAIR    |            |                                 |</span></span>
 |                       |           |            |                                 |</span></span>
 |                       -------|    |   5        D                                 |</span></span>
 |          3            .      |    |            |                                 |</span></span>
 |=.=.=.=                .  6               . . . |                                 |</span></span>
 |  SR  |                .                 .  OIL |                                 |</span></span>
 ---------------------------------------------------D--GGGGGGGGGGGGGGGGGGGGGGGGG-----</span></span>
</span>
#1 - Main Office Area</span></span>
#2 - Recording Booth</span></span>
#3 - Lab</span></span>
#4 - Seating</span></span>
#5 - Entry</span></span>
#6 - Pantry</span></span>
M  - Mechanical Room</span></span>
SR - Server Room / Main Panel</span></span>
</span>
 . . .              - stud wall, no sound isolation</span></span>
=.=.=.=             - stud wall, sound isolation</span></span>
`I` when vertical   - stud wall, sound isolation</span></span>
D                   - door</span></span>
H                   - half-rack with rear mounted in #2 and front in #1.</span></span>
</span>
the rest should be obvious enough</span></span></code></pre>
I’ll come back to the garage part another time1</a></sup>. The plan is to have a single-stud false floor under the main office
area, but leave the rest bare concrete. I’ll carpet in the booth, but keep all the electronics out of it mod some
control panels and pedal controls. I play mostly acoustic but I have a kemper that’ll probably be stationed in there, it
currently lives on a half-rack and the plan is to install that so half is in the booth and I can have the frontend
controls for me in there, along with enough controls and monitors to manage the session. There are some water pipes there but I mostly record at night and between that and the sound insulation I can hopefully minimize the noise to acceptable levels.</p>
I wanted to keep the easy flow between the lab and the seating area, but I do want them to be separate. I suspect that
door may be a pocket door. There is some storage underneath the stairway, I’m unsure if I want to set up that area to be
availbe to area #6 or #3. The former would require the removal of the sheathing and may not be straightforward to
accomplish, so I’m leaning #3. I want to avoid a 3-6 door2</a></sup>, as it would greatly reduce the usable configurations of
#6, but I will likely have a passthrough or double-sided cabinet since #6 is going to be where my snacks are. Likely a
minifridge under the stairs; unfortunate to have in the lab, but the most convenient place to locate it for sure.</p>
Area #5 is just general entry and will extend the pantry storage from #6. The OIL tank there is similarly poorly studded
in and takes up a ton of unneeded space because Kevin couldn’t work a square to save his life.</p>
I have two racks to manage all my equipment, they’ll end up in the ‘Server Room’, which is actually not</em> going to be a
studwall. I plan to have two walls-on-wheels that just cover the area but can be fully rolled out of the way when
needed. The design will just be a normal set of stud-walls in a L</code> or U</code> shape (depending on stability), set on
low-profile omni-directional wheels. Maybe a skirt/weatherstrip if it’s needed, but ideally there is a slight gap to
encourage some convective airflow3</a></sup>. When I need to work on the rack, just roll it out of the way. The false wall can
still benefit from the heat sink of the foundation of the house, but when I need to work on the rack it’s just ‘slide
the thing out of the way and go to work’.</p>
It also avoids having a very small room that would definitely not meet code.</p>
We plan to move our pool table down here too (I am sure that will be an ordeal covered across many entries), that’ll end
up in #4. Getting it down there may be tricky but fortunately the wall around M is (bad) stud-wall left from the
previous owner (my arch-nemesis, Kevin4</a></sup>). I don’t think that stud wall will survive my hatred of it, I’ve already had
to remove about a third of the shitty stud job he did down there. So I can gain a couple feet if the movers need it. The
lower part of that wall doesn’t have a ton of clearance from the furnace, but I can likely re-use my movable-wall idea
here and just have a wall I can roll out of the way when we need to access the area behind it. Gaining access to the
extra space at the cost of a little engineering. That wall could be simpler than the server false wall, since it just
needs to roll out of the way very infrequently.</p>
I may be able to gain a foot of that space back, and could also gain the space at the top of that room as it’s just
covering off the water main and water heater, which are annoyingly separated (honestly the boiler location in this house
makes no sense to me. I blame Kevin, it’s not his fault, but I blame him anyway).</p>
Otherwise that section is probably going to be bare concrete, I’ll fit a couch and TV in there probably, hopefully have
my simracing setup in there as well. I’ll probably put some rugs down. I considered doing more false floors like in
#1 and #2, but concrete-with-a-rug has the advantage of being extremely durable and still pretty comfortable if you
don’t have to stand on it for hours and hours, which is sort of the definitional opposite of what will be done there.</p>
The ceiling is currently open-bay-and-pink-insulation; that will certainly need to change before proceeding, but I think
it’s going to just be a ‘sheet-rock-and-call-it’. I considered doing some amount of drop-ceiling as it would aid in
cable management5</a></sup>, but I decided it’d be cooler to run LED strips with the main cable runs (there won’t be too many)
so I can use the LED strip later to identify cables.</p>
Also I can do the blinkenlights with them.</p>
This is basically the plan. I’m at 0% complete, and it’s probably going to take a year or more to get this done, but I
plan to move down there sometime later this year, so the other complexity here is doing this all ‘live’ to some extent.</p>
It’s gonna be a fun project.</p>



Mostly because I have no idea what I’m going to do in there, it’s a mess. ↩</a></p>
</li>

That is, an X-Y door is a door from section X to section Y, so the 4-5 door is the door between the mechanical
room and stairwell. ↩</a></p>
</li>

I’m not an aero guy but my understanding is the heat rising will vent through the top of this false wall, that’ll
need to pull air from somewhere, and pulling it along the cold cement floor sounds like free cooling to me. I plan
to model this space in CAD and maybe I can figure out how to fluid sim it. ↩</a></p>
</li>

Not his real name. ↩</a></p>
</li>

In the sense that I wouldn’t need to see</em> the mess I made ↩</a></p>
</li>
</ol>
</section>


12-AUG-2026
2026-08-12T00:00:00+00:00


Link GIYF’s atlas page.


DNS needs to change for that too

Should be giyf.disturbed-circles.com</li>
</ul>
</li>
</ul>
</li>

GH


Repo</li>

action


point at repos everywhere in CI</li>

point at paths everywhere locally</li>
</ul>
</li>

Pages setup</li>

DNS for DC generally


disturbed-circles.com -> root site</li>

giyf.disturbed-circles.com -> atlas</li>
</ul>
</li>
</ul>
</li>

other projects in the log section that come from content managed here

mostly for projects that don’t really have natural repos of their own.</li>

Project Nobby</li>

The Garden</li>

The Office</li>
</ul>
</li>

Music section?


Links to playlists</li>

Guitar inventory</li>

The Work</li>
Might could harvest some stuff from the old site?</li>
</ul>
</li>

Gehenna

A place where i drop ideas that I probably will never get to. I have very many of these.</li>
</ul>
</li>

FTA needs it’s PDF generated and linked, might be nice to turn the latex -> html at some point


PDF Gen</li>

PDF Link</li>

HTML Gen

probably should follow the reorg/citation thing. Idea would be to take each env and include citation info
in it directly, then extract to some input format. Later I can add equivalent citation (i.e., where in
Munkres a def appears or something).</li>
not sure if I should keep it as latex as the primary language or extract to a different format/convention</li>
aim is Temu nLab, all my notes + cited definitions across my various books. One big map. Queryability would be
nice. Maybe a graphdb – ideally an atlas integration?</li>
</ul>
</li>
</ul>
</li>
</ul>
12-AUG-2026</h1>
1405</h2>
I have a big TODO list but I’m starting to whittle it down. I’ve got most of the CI stuff done, but still need to get
DNS wired up. I updated the profile to point to projects which do not yet exist, but all things in time.</p>
1449</h2>
Waiting on tickets leaves lots of time for scutwork. Pages setup done for DC.</p>
1559</h2>
Good enough for now, I think. HTML Gen for FTA is going to need some thinking about how I notate everything; I’ll
probably need to deeply restructure that repo.</p>
I suspect Gehenna and the other projects and stuff will come after I finish my next big project, which is moving my
office from the current little room I’m in to the basement, where I’ll have much more room to spread things out and set
them up in a way that means I’ll actually work on those projects.</p>


10-AUG-2026
2026-08-10T00:00:00+00:00
10-AUG-2026</h1>
1002</h2>
I have no idea what I’m doing today. Probably worth setting up some kind of DNS stuff for this site, which also means
making the repo and stuff on GH.</p>


Link GIYF’s atlas page.


DNS needs to change for that too</li>
</ul>
</li>

DNS for DC</li>

GH


Repo</li>

action


point at repos everywhere in CI</li>

point at paths everywhere locally</li>
</ul>
</li>
</ul>
</li>

other projects in the log section that come from content managed here

mostly for projects that don’t really have natural repos of their own.</li>

Project Nobby</li>

The Garden</li>

The Office</li>
</ul>
</li>

Music section?


Links to playlists</li>

Guitar inventory</li>

The Work</li>
Might could harvest some stuff from the old site?</li>
</ul>
</li>

Gehenna

A place where i drop ideas that I probably will never get to. I have very many of these.</li>
</ul>
</li>

FTA needs it’s PDF generated and linked, might be nice to turn the latex -> html at some point


PDF Gen</li>

PDF Link</li>

HTML Gen

probably should follow the reorg/citation thing. Idea would be to take each env and include citation info
in it directly, then extract to some input format. Later I can add equivalent citation (i.e., where in
Munkres a def appears or something).</li>
not sure if I should keep it as latex as the primary language or extract to a different format/convention</li>
aim is Temu nLab, all my notes + cited definitions across my various books. One big map. Queryability would be
nice. Maybe a graphdb – ideally an atlas integration?</li>
</ul>
</li>
</ul>
</li>
</ul>
Work is about to involve a lot of waiting due to a forced reimage, so I want to have some stuff lined up to keep the
tires warm while I wait.</p>


10-AUG-2026
2026-08-10T00:00:00+00:00
1107</h2>
Thinking about the plan here a bit. I think my aim is just going to be to finish out point set here for now, which is
only another 30 pages or so, then pause to reorganize the notes. I will probably switch to Hatcher to match the course
I’ve been watching for the homotopy stuff. I haven’t been following along the syllabus directly, just watching the
videos, so it’ll be good to have some exercises, and to be honest, I’m kinda just disliking the G&G book a bit. The
authors are inconsistent in some annoying ways (interchanging basis and base freely, for instance), and they also have a
habit of making some intuitive leaps I don’t think are good leaps to take. I don’t think any of it is wrong, just – if
I were formalizing it I think the ratio of lines-of-Lean to lines-of-Proof would be pretty skewed towards Lean.</p>
It has been good for being a pretty dense</em> listing of the various names and structures in point-set though. I feel like
I have a pretty good glossary of definitions now. I like the working mode I’ve come around to of ‘take informal notes on
one thing; formalize another; watch a class; pick two.’ It’s a good pace that’s relatively easy to keep, while feeling
quick. The goal is to build endurance, not speed. Speed only comes through hours spent, endurance comes through
consistently spending them.</p>


9-AUG-2026
2026-08-09T00:00:00+00:00
9-AUG-2026</h1>
1609</h2>
I’ve mostly been setting up this site, my intent is to use this version of my LOG.md</code> note-taking to start to
centralize all those LOG.md</code>s here in some sense. This list of posts will be LOG.md</code> for this site + general project
overview/thoughts, the individual LOG.md</code>s per project retain their presence so they stay project-local; and then I’m
planning to pull and render all my LOG.md</code>s here as well so they’re legible for me for later.</p>
1854</h2>
That hooked up pretty quickly. LLMs are pretty magic at this kind of thing.</p>
I can pretty easily enumerate projects with a LOG.md and get them pretty seamlessly integrated to the sidebar. At this
point, then, it’s mostly just pixel pushing and at some point getting this stood up on GH and building. Then updates
should be seamless.</p>
1935</h2>
Did some cleanup of GIYF’s log. Now that I can plan these logs to end up here, I can actually use proper footnotes and
katex and stuff. Nice bonus I hadn’t really thought about. I need to get better at running spellcheck over this stuff
though.</p>
2200</h2>
Pixel pushing, lots of pixel pushing.</p>
2239</h2>
Still TODO:</p>


Link GIYF’s atlas page.


DNS needs to change for that too</li>
</ul>
</li>

DNS for DC</li>

GH action


point at repos everywhere in CI</li>

point at paths everywhere locally</li>
</ul>
</li>

other projects in the log section that come from content managed here

mostly for projects that don’t really have natural repos of their own.</li>
</ul>
</li>
</ul>


9-AUG-2026
2026-08-09T00:00:00+00:00
2233</h2>
Started this so it’ll end up in the site eventually. I noticed I’m about halfway through the book and I’m coming close
to the end of it’s point-set material. That lines up in parallel with the Algebraic Topology course I’ve been watching
here</a>, which is very good. I know this book
(Gamelin & Green) jumps right into the Fundamental Group1</a></sup>, but that course starts with intuitions about homotopy and
chain complexes and I found that a much nicer way to start.</p>
In any case, that’s going to help out a bunch with reading The Rising Sea</em>, I started taking these notes as a way to
sharpen my topology tools a bit and remind myself of the vocabulary, but I think this is going to be a collection of
notes on a bunch of stuff, maybe do some cross-referencing as I go through. I have a \ggref</code> macro I’ve been using to
just keep general track of where things are, but properly threading these things through and maybe doing some nicer
rendering would be cool.</p>
It’ll be like Temu nLab.</p>



Know is a strong word, I think I remember that it does from like… 25 years ago when I last read it. ↩</a></p>
</li>
</ol>
</section>


23-JUN-2026
2026-06-23T00:00:00+00:00
0940</h2>
I had been working off the 2017 draft, which I think I probably grabbed in</em> 2017 and just never came around to work on
until recently. I ordered a print copy. Color me surprised when I saw the numbering misaligned. This means
‘sane-up-to-1.2.’ above should’ve been ‘sane-up-to-1.1.</em>’ which it still is but I misnamed it. :)</p>
I renumbered everything and finished out up to 1.1.B, this required a lengthy sidequest to upgrade atlas to better
encode the approach here. When formalizing text, authors like to pack like-results together and my approach in GIYF was
not scaling well with this. Theorems felt separate even when they were the same item in the book, names were getting
complicated, big pain.</p>
Atlas v2 has a very different structural UI then the version currently used by GIYF, so another ‘big giant refactor’
will be coming soon I suspect.</p>
High Tide</em> started as a way to get enough context on something I stumbled on while thinking about GIYF’s Line
structure. I think</em> what I found is what Vakil describes in chapter 15, “Line Bundles” – I do not speak enough of the
language to be sure of this, but my idea is roughly matching the description of a ‘Sheaf of Bounded Tanglings’ in my
parlance, and just based on the names I suspect my ‘Tangling’ corresponds to the ‘set of base points’ or ‘base locus’ of
L, the idea is that the lines in the sheaf are the global sections?</p>
I have truly</em> no idea if any of what I just said is correct, but that’s why I’m in the book, it’s a race to Chapter 15
to understand if this is the structure I spotted or if I was looking at something else.</p>
To that end, my plan so far is basically to sprint from Chapter 1 to Chapter 15; with the early chapters receiving a
pretty thorough treatment, and growing more sparse as I get closer to the target chapters, then probably a pretty
thorough work through of Line Bundles. My hope is mostly to gain some technical skill around working with
categories/sheaves/schemes as a structure; and to verify/deny my intuition regarding the ‘sheaves of bounded tanglings’
idea for lines.</p>
I will say, the print version of this book is very nice. I have a softcover on a little 3d printed stand while I’m
working and the print quality is very nice. The cover is colorful in a way that doesn’t scream “Overpriced undergrad
Calculus Book” and is more of a “Someone might mistake this for a book about something that isn’t math.” Which I mean in
the most complementary sense I can.</p>
1034</h2>
I wouldn’t be surprised if this, too, ends up getting shelved for me to go do some algebra and topology, I have an
Algebra</em> by Hungerford which is not</em> the GTM version that probably covers enough for my purposes and should be pretty
light work. I have another book, Topology: A Categorical approach</em> by Bradely et al, that I’ve worked through part of
before and might be a decent way to extend HT’s Category theory and also rough in some Topology stuff.</p>


18-JUN-2026
2026-06-18T00:00:00+00:00
1251</h2>
I started this a couple days back, aiming to do something a little closer to the Mathlib standard and also get a handle
on the necessary CT to get to sheaves, which I think will play back into the GIYF work around handling Lines/Tangles/the
like.</p>
So far I really like the book. I’ve taken a couple survey passes over it in the past, and Vakil’s writing style is nice,
and the commitment to a sane numbering system (or, at least, sane-up-to-1.2.*, there is a lot of book here) is pretty
nice. It is going to be a testbed for a new iteration of atlas’ syntax and hopefully should help drive out some more of
its features.</p>


27-MAY-2026
2026-05-27T00:00:00+00:00
1156</h2>
Working on adding ‘Arrangements’ to the theory. This extends the betweenness syntax to arbitrary ordered lists of
points. In the text, Greenberg uses figures to disambiguate these arrangements, which is unfortunately not really
available to Lean.</p>
The implementation of Arrangements involves a great deal of fiddly list math, and so it’s implementation is largely
Agent-built, but works off the properties proved in Prop 3.3 and its compatriots. The syntax works thusly:</p>
given some set of betweenness statements, you can attempt to prove or reduce another betweenness/arrangement by the
following pattern:</p>
have</span> h : A - B - C - D := </span>by</span> organize ABC BCD</span></span></code></pre>
In this case, it will close the proof by the simple application of P3.5. However, if you had A-B-C</code> and A-P-C</code>, the
case is ambiguous – there are three possible placements of P (to the left, right, or on top of B), so you get a proof
like:</p>
have</span> h : A - P - B ∨ P = B ∨ B - P - C := </span>by</span> organize ABC APC</span></span>
rcases h </span>with</span> APB | rfl | BPC</span></span>
all_goals obvious</span></span></code></pre>
This is fine, but I wanted something a little bit cleaner for that as well as for the other branch in 3.5 which results
in 6 different cases to close, all of which are obvious up to some re-arranging / deduction of the arrangement.</p>
In some sense this means I didn’t write the proofs for 3.5 really, they fall out of the work the agent did to encode
arrangements; but one of the nice things about Lean and math more generally is that so long as the proof proofs, then we
know the thing works. I don’t necessarily plan to dig in and rebuild this apparatus unless I have to, but I suppose some
human-fidelity is lost in this.</p>
Given that the apparatus is really just ‘bookkeeping applications of Prop 3.3 and topological sorting’, I’m pretty
alright with that state of affairs; but wanted to be clear.</p>
The syntax I settled on was:</p>
arranging <hypos> [into <rcases-like-breakdown>]</span></span>
. proof-</span>by</span>-cases-</span>if</span>-necessary</span></span></code></pre>
This takes the given hypos and optionally a naming cluster like rcases</code>, then it attempts to obvious</code> off any produced
goals before finally returning a minimal set of nonobvious cases. Under the hood this uses the organize</code> apparatus, but
arranged more like the distinguish</code> tactic, which uses available proofstate to try to weed through simple goals. I’ve
sort of mentally decided that anything that just needs to sweep the proofstate once is fundamentally not expensive and
well worth doing proactively. So far compile times haven’t suffered to the point of being noticable, like with prior
experience with tauto</code> and simp</code>.</p>
As part of it, I also implemented CoeDep</code> (value-dependent coercion) for arrangements down to betweenesses; which means
they are drop-in wherever a Betweenness is allowed. This makes it pretty gentle to use overall. Derive the full
arrangement then drop it into place. The mechanism here is good old macros</em>, like the C-preproc kind (except done with
elab). This generates a finite suite of instances to do each coercion for each finite arrangement size. We can’t
natively support arbitrary arrangements without introducing a tactic, which would ruin the drop-in ability, but
practially arrangements of up to 7 points are supported for about ~2s of compile time over the n=5 case. The book never
gets above 5 points in a line from my brief survey, so should be fine.</p>
1602</h2>
Currently renaming all of the lemmas/exercises and reorganizing again. LLMs are a godsend for fiddling with this kind of
thing.</p>


24-MAY-2026
2026-05-24T00:00:00+00:00
1219</h2>
I’ve been doing a series of largely mechanical refactorings and restructurings. The main thrust of which is a typed
heirarchy for line-parts (which is not exactly</em> what I wanted, but is a significant improvement over the prior state).
As well as trying to capture what Greenberg considers ‘obvious’ in a tactic (creatively named obvious</code> as well), which
I’m hoping will leverage atlas</code> at some point to do a phased ‘intuit like Greenberg’ automated theorem proof attempt.
The aim isn’t necessarily to replace every place Greenberg calls a ‘clearly X holds’ or ‘obviously Y is true’ with this,
but most of the cases where he does, the result is really trivial and should be inferred without argument. Sometimes</em>
(as in Pasch’s ‘A and B are clearly not on L’) the argument isn’t necessarily so obvious, though, and these are
interesting places where Greenberg’s intuition might’ve been making ‘leaps of faith’ which should have been justified
more thoroughly. In the end I hope to capture all the little intuitive, subconscious arguments that Greenberg used and
develop something like a ‘If Greenberg were arguing here, he would consider this subtextually true / true without
argument’ as a sort of research artifact.</p>
The process of getting there has been an interesting application of LLMs in this space.</p>
In particular, the process has been one of “Let’s build up a structure that ties into the existing mathlib machinery to
represent lineparts.” This involved fighting with Coe and the elaborator and some other syntax fighting. I estimate it
would’ve taken a few weeks of solid effort to get done, Claude finished in a day. That’s an honest-to-goodness savings
on work I really didn’t want to do. Most of it was weedy, mechanical shit of wiring up and rewriting after each
iteration to get the corpus compiling. I’m barely 20% through the book and have about 4KLOC of lean, most of which are
lemmas-in-anger, so being able to mechanically replace a component like that was a very legitimate value add.</p>
I don’t think, however, it would’ve come up with the structure on it’s own. It was struggling to understand why</em> I
cared about making these parts flow together; and frequently suggested some flavor of “We’re at the limits of what Lean
can do.” until I reminded it of some other extension we could try. We did eventually hit the limit (there is an
irreducible type ascription necessary in some few cases of intra-type equality comparisons), but it was only after
blowing past a few others that the machine thought existed.</p>
Once the apparatus was built and installed, the really interesting step happened. The obvious</code> tactic was able to close
out entire lemmas that were proved before, resulting in a significant reduction in the total surface area of nontrivial
theorems. If a proof is just := obvious</code>, then every ref</code> to that theorem can be replaced by obvious</code> to drop one
layer of indirection. So I instructed the machine to do the following:</p>

Go to each lemma, replace it by obvious</code> and see if it closes, if it does, mark it for inlining.</li>
Go to each marked lemma and inline it, replacing all of it’s occurences with a obvious</code>. If the inline fails, raise a flag</li>
</ol>
No flags were raised, and a bunch of lemmas (about a dozen) dropped away. Crucially, though, some of them didn’t</em> and
in fact, couldn’t</em>, these were the core underlying ‘obvious’ facts that Greenberg takes without proof, but are
necessary in formalization. I landed on something very neat, by ‘shaking the tree’ to try to refactor proofs to use more
of Greenbergs intuition. I got proofs back that were closer to the book, but also got proofs that were irreducible and
unstated by the author, meaning they represent some fundamental component of his intuition. They were required by
obvious</code> to close the other goals.</p>
It also led me down this other interesting path adjacent to but distinct from other automatic theorem prover tactics.
Consider the humble tauto</code>, which has this problem:</p>

Given an arbitrary type with universal and existential quantifiers, a proofstate full of hypotheses of arbitrary
shape (also with various quantification) and a corpus of theorems about as wide as @[simp]</code>’ll get you, resolve the
goal.</p>
</blockquote>
There is no context on what kinds of theorems there are, no insight into the relevant corpus of math, no ability to
filter or prune anything. It’s a really hard problem, and people have come up with extremely clever ways to optimize it,
but math isn’t a ‘prove any theorem’ game, it’s a ‘prove this</em> theorem’ game. When proving, one has the context</em> to
rely on. I know I’m proving theorems in synthetic geometry here, I don’t need to consider calculus or number theory
(yet). I can limit my scope much more aggressively if I look at the proof goal and see it involves intersections, I can
make more complicated, geometry-local arguments that wouldn’t necessarily translate wholesale to other fields. In a
sense, the progress in the unconstrained world is stymied by</em> the lack of constraint. It is much harder to do math if
you don’t specialize.</p>
So obvious</code> is sort of a start towards a specialized “geometry” tactic, but it’s even more specific than that, it’s
Greenberg</em> that I’m modelling here. It’s a specific</em> intuition (as captured by the book and my formalization of it,
which means it is not without some bias), and I think that’s pretty cool. I can imagine building up a little library for
designing these bespoke tactics per book. Then when the time comes to prove something new I can throw a corpus and it’s
corresponding intuition at it for the low cost of translating it to the language that tactic understands.</p>
I’m sure the idea isn’t novel, but it is fun, and blends nicely with the atlas</code> thing I’ve worked on. I should be able
to incorporate some ranking/searching directly in the graph, which means I can have the structure inform how best to use
the structure (a theorem with a high pagerank after a specific filter is applied might be a good way to search corpii
efficiently).</p>


18-MAY-2026
2026-05-18T00:00:00+00:00
1454</h2>
Time to talk about Agents and LLMs and this codebase.</p>
First things first, all the math is me. No agents involved in the primary proving of anything beyond the occasional
‘remind me how this syntax works’. I want to be clear that humans are doing the math here, not agents.</p>
Second, the distinct</code> and collinear</code> syntaxes, as well as most of the ‘small’ syntax tweaks (obvious</code>,
by_exhaustion</code>, etc) are mostly me, similar to the above, with the occasional ‘add the six other cases following the
same pattern’ level prompting. Edits, but small and proscribed.</p>
Third, atlas</code>, Atlas is almost entirely LLM-coded. I guided it through what I wanted the parser to do and what kind of
information I wanted to record. I also had it mechanically convert all the various callsites to use the system and
generate some ‘pithy’ names for things.</p>
Now let’s talk about why I think this is okay, as these tools are controversial in a way that feels hysterical.</p>
Pneumatic Nailers and Table Saws, Lathes and Agents</h3>
The Pneumatic Nailer, “Nailgun,” was introduced in 1950. It was designed to speed up the construction of houses by
making floor and wall sheathing much simpler. Instead of manually hammering things, you’d use instead a bulky,
unreliable, expensive and power-hungry pneumatic nailer to do the work that a journeyman carpenter could do in about the
same amount of time as a journeyman carpenter could do it with a hammer and practice. You can imagine how large house
building companies would embrace this technology, even as their more seasoned workers hated using it. They were
dangerous, the broke a lot, they cheapened the trade and reduced the skill floor. Some punk kid on the jobsite could
pick up a nailgun on day one and be installing subfloor, where was the craftsmanship, where was the quality? Are we just
gonna not build good houses anymore?</em></p>
No one thinks using a nailgun isn’t homebuilding anymore. No one thinks that a nailgun-based home construction technique
builds a materially worse</em> home. They might argue homes are worse today than in the past, but as someone who has been
around old and new construction my whole life; it’s always been about as bad as it’s always been.</p>
Table Saws probably were similar. No more need for proper form and structure; no need to practice technique. Just
measure twice, cut once, make sure you know how to avoid kickback.</p>
I heard a story once about Norm Abrams, the host of New Yankee Workshop</em> and a personal hero. He got his first
on-screen job after having worked as a journeyman carpenter on a job for the producer of This Old House</em> before it came
on the air. In the late ’70s, the use of power tools would’ve been common, but it wouldn’t’ve necessarily been the only
kind of homebuilding, and while a radial-arm probably would’ve been the standard power saw on a jobsite, it is easy to
imagine such a thing leading some to wastefulness and carelessness. It’s so simple</em> to make a cut, it’s so easy</em> to
get it to the right size. Norm got the job because of his efficiency with the tools at hand, and the rest is TV history.</p>
In my shop, I have a woodlathe. A lovely Grizzly that spins large hunks of frequently unstable wood by means of various
chucks and mounts and a big ass electric motor on the back. I can twirl a hunk of hickory at 2500RPM and stick a tool
into it. It is by far the most dangerous tool I own; it actively wants to kill me, it wants to see me dead, it wants it
to hurt the whole time. In the old days, a lathe would’ve been made of wood and rope, and powered by a ‘spring pole’ –
a green stick that would reciprocate the work back and forth in time with your foot movements. It was safer than my
lathe, slower too. I imagine the introduction of the electric, motor-driven lathe might’ve left spring-pole enthusiasts
worried about the craftsmanship, the reduction of the skill floor, the worsening of their trade.</p>
But I can turn beautiful work on my lathe, even though it wants to kill me, even though it is more dangerous. To be
honest, I’m not sure I could produce the same quality of work on a springpole, even if I started with it. There are
limitations inherent in the tool. The reciprocation means different techniques are available and different constraints
are in place.</p>
Tools and Weapons</h3>
My path through LLMs has been pretty bouncy. I started with the same curiousity as everyone. I tried GPT4 and early
copilot-style autocomplete. I thought it was cool. I used it in hazel</code> for a bit, but rapidly found, like most people,
that the code was poor quality and not much better than cut-pasting. I spent a lot of energy cleaning up that mess, and
Hazel is better for it, but almost none of that code remains[1]. As I learned more about the ethical… well let’s just
go ahead an call it what it is instead of euphemizing it as an ethics problem. The big “AI” companies committed the
largest act of copyright infringement to have ever been done. It makes Napster look like shoplifting. It was simple
larceny. They stole petabytes of art, literature, and music; packaged it, and sold it as if it were theirs. Depriving
the original creator of the value of their labor. This is unconscionable and wrong in a way that I’m comfortable
thinking of as Sinful, with all the weight applied to it as I would have when I was an Evangelical Nationalist Scumbag.</p>
It was simply evil behavior, performed by evil men (it is always men, it seems), and they should be shamed, pilloried,
and launched into the sun for having done it. These facts, and the existential threat to my job and career, the threat
to my family by dint of that, and all the myriad anxieties the “AI” tulip craze that was and is and will continue to
spiral until crashing – as it inevitably will – led me towards a sort of deep hatred for the technology. It wasn’t
until I realized how I started talking about it that I understood the mistake I was making.</p>
I started calling LLMs “the False Prophet,” I thought of them with irrational anger and disgust. I transferred the shock
and horror and dissapointment with the actions of their creators with the tool itself. I was mad</em> about it. Then I got
tagged by my therapist, “It sounds like you’re pretty heated, Joe. But don’t you always say, ‘it’s a poor craftsman that
blames the tool?’”</p>
Here’s the reality we live in, folks, the deed is done</em>. We can mourn the past, but not return to it. We are here</em>
now, alea iacta est</em>. The constant cry of ‘WeLl NoOnE iS mAkInG yOu UsE iT!’ falls on deaf ears the moment my families
livelihood is in peril. You are right, no one is making</em> me do it, just like how no one will make</em> food appear on the
table. There is no ability for me to respond against the crushing weight of the machine that has been so long built by
capitalists and bastards (sorry to repeat myself), so we can have no responsiblity other than to ask now, “What do we
fucking do</em> with the monster they made?” “Is it the monster’s fault that Frankenstein made it?” “Should we kill it simply
because its creation was abominable to us?” “Isn’t that cruelty too?” “Isn’t that wrong?”</p>
I don’t mean to say we would be killing a literal, living thing, but I refuse the notion that a tool itself is ethical
or not. I suppose, if pressed, I’d have to say that the argument that has always seemed strongest to me about gun
control is the ‘Guns don’t kill people’ argument. It’s simply correct, guns don’t</em> kill people, it’s irrefutable in
some sense. Whether through intent, negligence, or mistake, it requires action on the part of someone</em> to make the tool
into a weapon and for that weapon to harm another[2].</p>
I don’t want LLMs to be weapons, but the only way to make them not</em> weapons is to find out how to use them like
tools</em>. I can’t assign ethical or moral value to the machine-what-is-made. I can only try to understand the effect of
its existence and how I can use it to find ways to be kind.</p>
I have a lot of thoughts on how that might be done, but the first step is to learn how to use the tool in ways that feel
like I’m producing something that might actually benefit people sufficiently to justify the cost of the tool. I have
spent some months now unpacking and understanding this technology. I think it has broad, mostly unexplored uses, uses
that could legitimately benefit more than the cost. I think that there are ways to get these tools to produce good
output that can be genuinely useful and improve peoples lives. If you don’t, cool, don’t use the tools, but I will
accept criticism of my use in precisely one form, &> /dev/null</code>.</p>
[1] I caveat only because I truly don’t remember which code was generated anymore, almost all of it has been touched at
this point though so it’s probably gone.</p>
[2] This not to say that it is a good enough argument to justify all the things people try to use this to justify, we
still need gun control for lots of good reasons, but it’s complicated by the fact that this is, ultimately, a pretty
good fact the gun control advocate needs to reckon with.</p>
What is good code?</h3>
Once I started trying these tools I started to ask a new kind of question, not “Is this good code?”, nor “Can I make
this produce good code?”, but “What is good code?”</p>
In mathematics, we focus on properties. We might say that a line has the property of being defined by two distinct
points; and conversely that two distinct points have the joint-property of defining a unique line; and so on. I started
to ask “What are the properties of ‘good’ code?”</p>
I was easy to enumerate many features</em> of good code. It’s well-factored, it’s well-tested, the tests are resilient and
easy to understand and change. Really the whole codebase should be well documented and easy to understand and change. It
should be well-specified and it’s edge cases well tested. I should be able to trust the code and when my expectations
about it’s output fail I should be able to easily identify, isolate, reproduce and change the code to address the
failing.</p>
There was a theme developing, it’s not hard to see, good code is:</p>

Easy to understand</li>
Easy to change</li>
Well specified and well tested.</li>
</ol>
None of those things constrain the shape</em> of the code, or its implementation, the only constrain what the code does</em>
and what you can do to it</em>. Does it matter how I answer those questions? If I can read a good codebase, let’s take
ripgrep</code> as an example, I can see structure present there, I can build up a mental model. I might have to spend a bunch
of time learning the machine’s innerworkings, but the code is good</em>, it is well factored</em> and precise</em>, it minimizes
mental load as much as it can for being a complex tool with many precision engineered parts.</p>
It is easy to understand (relative to your understanding of Rust and the techniques involved), it is easy to change
(because it is well-factored, and assuming you have a good existing understanding of the design of the system), and it
is well specified and tested. It is Good Code (tm).</p>
If I think about the Atlas</code> thing I just built, it is also easy to understand – if I have a question about the
structure I can ask the LLM, I can get a line-by-line walkthrough if I want. It is much</em> more adept and API-aware than
me, and while it can often write code I don’t immediately understand, there isn’t any part of it I am incapable</em> of
understanding, and with sufficient curiousity I can learn the deep parts fo the system just as before.</p>
Arguably, it’s easier</em> to change than a handrolled implementation, I just have to precisely describe what I want. Often
that means giving it instructions about how to rewire the machine’s internals (not the oft-assumed “write me a tool”
prompt, rather a “adjust how the syntax is parsed here to use this API in this way.” It’s certainly less typing than
before, and as someone who spends most of their waking life typing things into this infernal box, less typing is a
massive</em> win for my physical and mental health.</p>
It’s lean, well-specified here means ‘comes with mathematical proof’, but even in other languages you can pretty easily
build up a comprehensive test suite; optimize it; measure its coverage; mutant and fuzz test it – the incremental cost
of testing your code is now nearly zero, the incremental cost of changing it is zero. The places where we have trouble
are in getting insight into the internal, abstract structure of the code. Before the bottleneck was writing</em> code, now
the bottleneck is understanding</em> code and its structure.</p>
So, no matter the implementation, if I can answer those three questions in the affirmative, isn’t whatever I have – by
definition – good code? It is not pretty</em> code – I don’t like the look of LLM-generated code more than most people,
but in the fully ‘vibecoded’ ecosystem, the reality is the codebases are perfectly cromulent, so long as I don’t really
read the code too deeply. The skeptics among you clutch their keyboards and say “But how do you know it works?” To which
I reply, “My test suites are more thorough than anything I could write on my own, I have every possible path covered and
specified, I built tests which test my tests to ensure the tests are testing what they are supposed to test. I have
mutation testing, fuzz testing, property testing, unit and integration tests, I test in situ, I test end to end. Some of
this stuff is literal, mathematical proof</em> What more could you want to prove it to you? How much do you have? Why
should I trust your</em> software? Do you have formal proof it works?”</p>
The refrain of “It’s not good code” relies on a notion of ‘good code’ that is optimized not for what good code is</em>, but
for what good code is for a handwritten, human codebase</em>. The issue is that we have mistaken the limitations of our
tools</em> for the indications of quality</em>. We are overgrown monkeys doing mathematics on specialially arranged rocks; we
have designed practices to accomodate for our weaknesses; but those practices are, themselves, no different than the
hours you’d spend hucking a hammer at a nail installing sheathing and subfloor; they’re no different than the time spent
structuring your body to properly cast and return a handsaw. We have new tools now, we need new practices, and that
means new understanding of what good looks like in a new context.</p>
I’m not saying every project is a vibe project. Indeed, this is an example of a project which is emphatically not</em> a
vibe project. Why? Because the point of this project is for me to engage the math and build something mostly by hand.
LLMs enter into it more like a orbital sander. Is hand sanding “better”? No, it’s different, it leaves a different
finish, people may be able to spot the tell-tale circular pattern, but it’s not better</em> to sand by hand, it’s not
worse</em> to sand by machine. It’s not better</em> for me to manually replace every theorem <name> : <statement> := by <proof></code>, an LLM can do that work for me in much less effort (though not much less time, if I’m honest); it doesn’t
detract from the product to automate away those sorts of changes, and indeed it allows them much better and richer
tooling later.</p>
My point is simply: If you are reacting to these tools with anger at the loss of the old ways of doing things, I feel
you, but you don’t get to uninvent the nailgun. You can’t un-table the saw. The only question is ‘do these things have
any use at all’ and ‘is the juice worth the squeeze’? At this pint, I think at this point, I can answer at least the
former, and I’m beginning to see the answer to the latter too.</p>
What’s the use?</h2>
The uses of LLMs in this repo have been stated, but at a higher level, I use LLMs in the following ways to good effect:</p>
Mundanity</h3>
Pure, Shameless Vibecoding:</h4>
Ask for a thing, recieve a buggy, halfworking thing that mostly just looks the part. This is entirely for the purpose of
candy. A cardboard version of a product is a useful thing for me. I am not skilled in visualizing the final product. I
may be aphantasic[3], but in any case, being able to get a cardboard cutout of a thing to look at and talk about with
people is extremely useful.</p>
[3] I don’t know, I’ve never been not me, so I can’t tell you whether I’m seeing more or less in my ‘mind’s eye’, I can
say that I have always assumed that to be much more metaphor than other people I know</p>
Data gathering and scutwork.</h4>
There is a large class of things for which I need the product of a process but have no interest in its performance / I
do not stand to gain a skill I want from practice there. This ranges from truly mundane questions like finding facts
about pricing or API shape, to researching bugs in my editor configs. I cannot express to you how little I care about my
dotfiles. I simply couldn’t give a shit if they look nice or are wellmaintained. I want my editor to always do exactly
what I want, I don’t care who maintains it or how, I don’t care about it at all, it just has to always do exactly what I
expect and otherwise be a transparent thing. The most common cause of project abandonment for me is not</em> an overhard
problem, it is always the same thing. Some externality – editor config, testing/hosting infra, etc – gets in the way
of me continuing to work on the interesting part of the project. WLOG, let’s call that class of work scutwork.</p>
Scutwork is a necessary evil, with greater emphasis on the ‘evil’. It is weapons-grade boredom, and the answer to this
has always been “Use this framework, set it up once, and try not to think about it.” or “Just fucking do it.”</p>
Fuck every square millimeter of that. I don’t want to maintain that shit, and frankly I don’t know anyone who actually
does. If you are one such person. I think you’re weird (complementary), and I do not aspire to be like you. I’m here to
do math, not convince Neovim to stop being so fucking stupid.</p>
Targeted Refactors / Draw the rest of the fucking Owl</h3>
Prop 3.3 in this repo has a case where LLMs helped greatly with the math. There is a common, underlying argument in that
setup for the Line Separation property (3.4), and I could see it, but extracting it from the haze of specific variables
was really, really hard. I could smell it, but I couldn’t hunt it down. Excitingly, this intuition came first from the
code, and I aligned it with the intuition in the math. It was a delightful moment when I linked the two things together
and felt the intuition in the math spill onto the code and the real power</em> that Lean might offer was spread out. Then I
realized how fucking awful</em> it was going to be to extract that common argument, so I put down the project for 2 months
and dealt with my anger around LLMs.</p>
Then I asked Claude to highlight the extractable argument after describing what I saw, it made a suggestion, I verified
it, did the proof, and then ported it back into the proof. From there it was much, much simpler to complete the other
branches of the proof complex, and thus 3.3 was closed.</p>
In another project, I inherited a pile of 50 or so jenkins pipelines. I do not know groovy, but I can hack together
something that is almost valid groovy and expresses the intent accurately. I left the code littered with little todos
and then told Claude to paint inside the lines I left it. Twenty minutes later I had the rest of the fucking owl, and
all my pipelines ported. I genuinely would have taken days or weeks to do that alone.</p>
I don’t need or want to be a groovy expert. I don’t need or want to spend hours meticulously adjusting incantations to
satisfy Lean. I want to spend my mental energy on the overall CI ecosystem. I want to spend my time in the Math. LLMs
let me be efficient with what skills I’m reinforcing</em> and what I delegate to someone (or something) else</em>. If I had,
at my disposal, an army of grad-students and interns and junior engineers, it is the class of work I would send them off
to do not because I thought it would improve their ability to do their main jobs, but because I would not like to do
that work and I have an army at my behest. I did not have such an army before, now I do. Now if I acquire such an army,
I can send them too</em> on missions of actual</em> import to their actual</em> goals.</p>
How much juice?</h2>
LLMs, Agents, “AI”, whatever you prefer to call it, it is not useless</em>, it’s not perfect, it’s a machine that guesses,
and guesses are not useless things. You can</em> get these tools to produce good code (under both understandings), though
human-good code is much more difficult to achieve, it is possible to do so with sufficient prompting and oversight. In
this project already it has more or less covered a month or two of weekends worth of scutwork that, frankly, probably
would’ve killed this project.</p>
Is this project valuable? It certainly is valuable to me. It feels good to work on this, I like that I found a way to do
real</em> math again, not the idle pining for a life not lived, not the silly doodles of differential equations in
meetings; but real, honest-to-goodness math.</p>
Is it valuable to others? I don’t know, I like to think Atlas will be a generally useful thing, and maybe someone,
someday will ready the body of GIYF or tour through the Atlas site and learn something useful. I don’t know that what
I’m doing is “Good”</em>. I’m not sure that Good</em> is even a meaningful concept so much as a pleasant fiction – a lie to
children – that we tell to make ourselves feel better about a reality that is fully indifferent to us at all levels,
times, and spaces we inhabit. I can’t tell you to like this work, I can’t tell you to accept it, or even to not hate it.
If my use of agents bothers you, I have empathy for you, but for me – their use lets me do the things I want to do in
life, and none of us asked to be here, or be thrust into all this. For me, I’m going to use the tools available to me in
the most responsible and efficient way I can. I promise I won’t act carelessly and wastefully in their use, and I won’t
treat them with the kind of flippancy that many do. If that is insufficient for you, I’ll give you the same response I
put in a recent commit removing the anti-LLM canary.</p>
“I do not care.”</p>


13-MAY-2026
2026-05-13T00:00:00+00:00
1600</h2>
I spent a bit of time refactoring 3.3, extracted a helper, had a small think about how much to tweak the arguments
Greenberg made for Lean convenience. I went with a hybrid approach, some theorems will get refactored for my own sanity,
some will be kept all inline so it is clear which chunks of lean correspond to which statements. I think that will help
identify where Greenberg is making intuitive leaps that are worth examining closely, while still leaving me with tools
ready for proving other things.</p>
I left this project for a bit while I worked on other things, but I’m going to come back to it now. I’m pleasantly
surprised at how well Lean has remained in my brain despite a couple months of hiatus.</p>


5-MAR-2026
2026-03-05T00:00:00+00:00
1316</h2>
Finished 3.3, there are some corrolaries to wind up, but the main bulk is done. The proof is very</em> long and pretty
nasty, so it probably needs an intermediate or ten to clean it up.</p>
I started thinking about a couple tasks I want to do:</p>

Alignment/Concurrence – a generalization of the collinear</code> condition to accept an arbitrary set of points,
convertible from collinear</code>, but covers all the line equality constraints automatically.</li>
Separating Theory</code> a bit more, creating namespaces for Ray</code>, Segment</code>, etc. Cleaning up naming, and ideally
getting aesop</code> tags set up?</li>
A construction</code> or diagram</code> DSL for describing how to construct a diagram that can then be reasoned about; building
up the types of propositions by a series of operations that guarantees we’re not assuming invalid constructions in
the type.</li>
Getting leanblueprint</code> ‘working’, which probably means replacing it. I have scripts/DumpDecls.lean</code> which outputs
some JSON, I don’t like how manual the maintenance of LBP is, so something automated might just obviate the need for
all the TeX stuff. Whither plantuml, graphviz, or otherwise.</li>
Extending Betweenness</code> to arbitrary length, inferring all the internal conditions; this notation would simplify the
density axiom a bit, and is pretty natural.</li>
</ol>
I think the plan is to prove the corollaries, merge, then figure out which way to extend; I definitely need to do some
more refactoring and cleaning, I think the blueprint stuff will help.</p>


4-MAR-2026
2026-03-04T00:00:00+00:00
0020</h2>
Some thoughts before guitar and bed.</p>
I think I need some work around concurrence</code>s, which are groups of concurrent lines.</p>
A frequent headache is managing various appelates of the same line. A set of collinear points induces O(n^2)</code> ‘line
throughs’ by picking pairs of their points. All of these lines are geometrically identical; but in the prover, they’re a
big ol’ pain in the ass. I’ve been thinking about this in the context of more automation for distinct / collinear, but
there are a couple properties of these things that are kind of interesting.</p>

Any subset of a distinct/collinear/concurrent set is distinct/collinear/concurrent – follows from the underlying pairwise equality/inequality</li>
for a ‘negative’ property, like distinct, combining two distinct hypotheses is pretty difficult, it requires M * N</code>
proofs, where M</code> and N</code> are the numbers of points in the structure.</li>
for positive properties, it’s often possible to satisfy entry much more easily. Collinear points only need to prove
they lie on at two induced lines; Concurrent lines only need to prove they’re equal to any of the other lines in the
set.</li>
Concurrency and collinearity are related – all those linethroughs are concurrent, and concurrent lines share the
same underlying collinear set.</li>
</ol>
It’s neat how being forced into hyperformality here makes it really clear to see some of the dualities – theorems have
very similar proofs despite the type change, and even the underlying plumbing is really just considering the
relationships between points and sets of points.</p>
In any case, the aim is to build some more structures, probably divide up the theory</code> section more, I’m not super happy
with the way it’s broken by chapter, I think it’s maybe better to try to further divide up things; I believe there
should be a way to re-export from the various files; so that I don’t have to import everything by hand; but to be honest
the module system (or import system, not really sure which is which) is a mystery.</p>


3-MAR-2026
2026-03-03T00:00:00+00:00
2245</h2>
Prop 3.3 is done, but nasty; it’s actually been done for a bit but I’ve been working on something else. A couple things,
actually.</p>
First I got separate</code> working so now distinct</code> conditions are pretty easy to deal with. Ex 3.1 is much improved as a
result.</p>
I started working on generating the blueprint; but there’s a ways to go.</p>
The proof of 3.3 is very long, and I have to replicate it pretty heavily to prove the other half of the condition; which
is pretty gnar. It’s the same up to some renaming, but I am having trouble seeing what I could extract. I did do a bit
to allow collinear</code> conditions to stand in directly for lines in all but the case where I’m trying to prove A on cL</code>,
where cL</code> is a collinear condition. The prover gets confused about what kind of membership it should use, which breaks
stuff.</p>
I might try just winging it with the other half and see if I find a shorter approach, I have an advantage of not minding
tedious cases that maybe would scare off someone in a setting where case-reasoning is cheaper.</p>


On Proving with Lean
2026-03-02T00:00:00+00:00
2-MAR-2026</p>
Hey Shae,</p>
We (very) briefly chatted on Mastodon about my excitement around Lean tactic metaprogramming, and you mentioned it would
make a good blogpost, and I wanted to agree; but I kind of hate blogging. I’ve been doing this “LOG.md” thing for a bit;
and I like that, but it doesn’t feel</em> right for an explanatory blog. I use it more as a journal or engineering log or
scratchpad. I’ve been trying to embrace my hyperlexia, and this is a good way to do it, but it does feel a little weird
to write a blog-ish explainer/experience report in it.</p>
I guess it’s because I don’t like the sort of anonymous audience of a blogpost; I know someone might</em> read it, but it
feels itchy to write something someone only might</em> read. I spend a lot of time thinking about writing and writing about
thinking, and I care</em> about it a lot. I think my identity is in it a bit, so it feels bad if I don’t know someone</em>
will read it. With LOG.md, I know I’ll</em> read it, so I don’t mind writing – a blogpost, though? All those hits could be
bots and crawlers, the comments just LLMs. It just feels lousy to not know</em> someone has really read it. It feels like I
wasted my time.</p>
But, as sometimes happens when my anxiety spikes about stuff, I had an idea. I may not be able to ensure that an
anonymous audience is nonempty, as a blogpost would require, but an open letter? I can guarantee at least one</em> person
reads it, the recipient1</a></sup>.</p>
A long time ago in #haskell-cafe I posted my then-current phone number in hex because I, being a college freshman,
hadn’t yet learned that obfuscation is not the same as encryption. You called my cell. I answered and was confused. I
realized the lesson. I remember chatting a bunch in those salad days on IRC, and I have the sense that you would
appreciate this approach to ‘blogging’; I hope you don’t mind being my test subject.</p>
Anyway, the meat of the letter, tactic programming in Lean4.</p>
Geometry</h1>
I’ve been working on this thing (geometry_is_your_friend</code>) as a little project to learn Lean. It’s formalizing Martin
Greenberg’s Euclidean and Noneuclidean Geometry</em>. In geometry, one frequently deals with small collections of objects
with simple (often pairwise) relations, distinct</code> and collinear</code> are the two big ones, and the former is especially
annoying when you have more than a few distinct points; The number of hypotheses in a list of distinct points is
O(n^2)</code>, so it rapidly becomes very messy in a proof to bookkeep all those hypotheses, and proving a distinct</code> goal is
a real</em> bear. Lean offers good facilities for creating custom syntax, and so it was quite easy to create a custom proof
term:</p>
distinct A B C D ... := A ≠ B ∧ A ≠ C ∧ A ≠ D ∧ A ≠ ... ∧ B ≠ C ∧ B ≠ ... ∧ C ≠ ... ∧ ...</span></span></code></pre>
which I can use like:</p>
example</span> (A B C D : Point) (h : distinct A B C D) : A ≠ B ∧ B ≠ C ∧ A ≠ D := </span>by</span></span>
  distinguish</span></span></code></pre>
That example also shows the tactic counterpart, distinguish</code>; which tries to automatically close goals based on the
distinct</code> hypotheses it can find in the proofstate. The initial version was just the distinct</code> term, and it barely
worked, resulting in a lot of proofs like this one:</p>
/-- p146. Given A-B-C and A-C-D:</span></span>
  (a) Prove that A,B,C, and D are four distinct points (the proof requires an axiom)</span></span>
-/</span></span>
theorem</span> Ex1.a</span> : A - B - C ∧ A - C - D -> distinct A B C D := </span>by</span></span>
  intro ⟨ABC, ACD⟩</span></span>
  have</span> distinctABC := Betweenness.abc_imp_distinct ABC</span></span>
  have</span> distinctACD := Betweenness.abc_imp_distinct ACD</span></span>
  simp only [ne_eq, List.pairwise_cons, List.mem_cons, List.not_mem_nil, or_false, forall_eq_or_imp,</span></span>
    forall_eq, IsEmpty.forall_iff, implies_true, List.Pairwise.nil, and_self, and_true]</span></span>
  have</span> AneB : A ≠ B := </span>by</span> distinguish distinctABC A B</span></span>
  have</span> AneC : A ≠ C := </span>by</span> distinguish distinctABC A C</span></span>
  have</span> BneC : B ≠ C := </span>by</span> distinguish distinctABC B C</span></span>
  have</span> AneD : A ≠ D := </span>by</span> distinguish distinctACD A D</span></span>
  have</span> CneD : C ≠ D := </span>by</span> distinguish distinctACD C D</span></span>
  refine ⟨⟨AneB, AneC, AneD⟩, ⟨BneC, ?_⟩, CneD⟩</span></span>
  -- </span>have</span> BneD : B ≠ D := </span>by</span> distinguish distinctACD B D</span></span>
  by_contra! BeqD</span></span>
  rw [<- BeqD] at ACD</span></span>
  exact Betweenness.absurdity_abc_acb ⟨ABC, ACD⟩</span></span></code></pre>
This uses an earlier version of distinguish</code> which just automates a single proof of a single inequality fact; and that
only after</p>
simp only [ne_eq, List.pairwise_cons, List.mem_cons, List.not_mem_nil, or_false, forall_eq_or_imp,</span></span>
forall_eq, IsEmpty.forall_iff, implies_true, List.Pairwise.nil, and_self, and_true]</span></span></code></pre>
has cluttered my proof. This is ugly, and so I dug into the guts of the tactic mode to figure it out. Eventually, after
a whole bunch of cajoling, and while only truly understanding like, 20% of it; but it works. I went through the Lean
Tactic Programming Guide</em></a> and cannot speak highly
enough of it as a significantly better introduction to the topic then this will be. The Lean tactic system is really</em>
neat; and also a headtrip</em> to understand. It’s very easy to confuse yourself in terms of what level you’re thinking at;
and I spent a fair few hours confused because I hadn’t realized I needed to convert from the LocalDecl</code> to an actual
Expr</code> in order to reference a hypothesis inside the inequality proof.</p>
Actually, let me back up a little and talk about the Proof environment and the proofstate. This is based on having read
through the Lean Tactic Programming Guide</em> over the course of an evening. It is with that hubris the rest of this
letter is written.</p>
Proofstate</h1>
A proof, as you know, is a program; and in particular it is a program of both values and types. Values have Types, and
Types, in Lean, can depend on Values. The nice thing about Lean’s syntax is that it makes heavy use of unicode, so you
can encode a statement using relatively natural mathematical syntax without having to resort to LaTeX. Lean uses some
clever editor features and a bit of muscle memory which develops pretty quickly. So something like:</p>
example</span> : distinct A B C D E -> A ≠ X -> A ≠ B ∧ (B ≠ C ∧ X ≠ A) ∧ (∀ P : Nat, P = </span>3</span> -> P > </span>1</span>) ∧ (C ≠ D ∨ V = W) := </span>by</span></span></code></pre>
gets entered like</p>
example</span> : distinct A B C D E -> A \ne X -> A \ne B \and (B \ne C \and X \ne A) \and (\</span>forall</span> P : Nat, P = </span>3</span> -> P > </span>1</span>) \and (C \ne D \or V = W) := </span>by</span></span></code></pre>
and your preferred editor will do the right thing, assuming your plugins are in order.</p>
This statement gets translated to a ‘proofstate’ like the following:</p>
A B C D E X : α✝¹</span></span>
V W : α✝</span></span>
h : Distinct [A, B, C, D, E]</span></span>
AneX : A ≠ X</span></span>
⊢ A ≠ B ∧ (B ≠ C ∧ X ≠ A) ∧ (∀ (P : ℕ), P = </span>3</span> → P > </span>1</span>) ∧ (C ≠ D ∨ V = W)</span></span></code></pre>
This is split into two parts, the Context</code>, which contains all the facts you know, and the Goal</code>, which comes after
the ⊢</code> and is the term you have to construct out of your pieces, you can have multiple simultaneous goals; which just
means you have multiple branches of a conjunction to prove. All this is the ‘proofstate’ of your theorem. When proving,
you can either directly construct terms using the underlying dependently typed functional language, like in this part of
a proof of prop 2.2:</p>
-- This </span>lemma</span> was not suggested by the author, but is handy. The proof is not long and simply establishes the</span></span>
-- '</span>Parallel' fact for each pair of lines. We need the unique point and the negative condition to build</span></span>
-- </span>these</span></span>
have hABnotparBC</span> : (AB ∦ BC) := Line.intersecting_lines_are_not_parallel hPonAB hPonBC</span></span>
have</span> hABnotparAC : (AB ∦ AC) := Line.intersecting_lines_are_not_parallel hPonAB hPonAC</span></span>
have</span> hBCnotparAC : (BC ∦ AC) := Line.intersecting_lines_are_not_parallel hPonBC hPonAC</span></span></code></pre>
where I construct a proof that a bunch of lines are not parallel to one another by directly invoking the function
provided by the Line.intersecting_lines_are_not_parallel</code> lemma. This is very similar to plain old functional
programming. Line.intersecting_lines_are_not_parallel</code> is a ‘lemma’, sure:</p>
lemma</span> intersecting_lines_are_not_parallel</span> {L M : Line} {P : Point} : (P on L) -> (P on M) -> (L ∦ M)</span></span></code></pre>
… in the sense that it is a logical statement about lines, but it’s also a function that takes two facts (and some
implicit variables those facts operate on) to produce a new fact.</p>
lemma</span> intersecting_lines_are_not_parallel</span> </span></span>
    -- </span>Implicit arguments to the function are searched for in context when the function is called</span></span>
    {L M : Line} {P : Point} :</span></span>
    -- input arguments, these </span>have</span> _type_ `P on L`, which is itself of type `</span>Prop</span>`</span></span>
    (P on L) -></span></span>
    (P on M) -></span></span>
    -- output, a fact of type `(L ∦ M)`, which also has type `</span>Prop</span>`</span></span>
    (L ∦ M)</span></span></code></pre>
In this mode, “term mode”, we act much more like a Haskell-adjacent language. It’s quite similar to the old pointfree
golf I assume people still play with Haskell.</p>

Aside: You can also see some custom syntax there for ‘not parallel’, this is very simple to achieve using the simplest
notation-related tool in Lean, notation</code></p>
notation</span>:</span>20</span> L </span>" ∥ "</span> M => Parallel L M</span></span>
notation</span>:</span>20</span> L </span>" ∦ "</span> M => ¬(Parallel L M)</span></span></code></pre>
The syntax side of things is a deep</em> well, I’ll talk about it a little more at the end, but it suffices to say that I
have not yet found too many areas where I think the syntax couldn’t be bent to whatever will pulls on it.</p>
</blockquote>
You can also use tactic</code> mode to prove theorems, and this is what I wanted to do for the distinct</code> term. Tactics are
small programs that run against the ‘proofstate’, they can inspect known facts, manipulate the goal, and create new
goals by ‘proving’ old goals. Those proofs themselves can have interior goals, which allows a tactic to incrementally
break down a big type into a series of smaller types. Equivalently, it separates a complicated logical statement into
many simpler ones.</p>
In Haskell, type-directed programming is this sort of delightful flow state where you gain the ‘if it compiles, it
probably works’ feeling that is occasional but fleeting in Haskell, and rarely even felt outside of languages not in its
lineage. Lean is quite good at eliciting that feeling, and this makes sense because the type system is powerful enough
to accurately represent mathematics directly, and mathematics is only ever that feeling (for me, at least).</p>
Tactics are quite powerful, but they’re also a little magical. A tactic like tauto</code> or simp</code> or aesop</code> might prove
your goal, but the reasoning might be opaque, or the proof it generates highly complicated; so it’s worthwhile, IMO, to
learn how to build up from terms, as it’s usually a lot more parsimonious. Once you get the hang, it’s a lot easier to
think about it in terms of simple function applications. When building tactics, term-mode proving is handy for
constructing proofs without having to drop into the evalTactic</code> mode. Once you start invoking nested tactics you are,
at least in spirit, ceding some control over what the prover is doing (which I’ll show later when we get to the meat of
the distinguish</code> and separate</code> tactics).</p>
What distinct should do</h1>
In particular, I need three things:</p>

notation for small collections of pairwise distinct values of some type; I suspect it’ll almost always be points, but
it could be useful for lines at some point too.</li>
a way to automatically prove inequality goals using available hypotheses in the context</li>
a way to destructure a distinct goal into a right-associated (“flat”) conjunction of inequality goals</li>
</ol>
This would allow me to work with the distinct</code> condition as a goal, known fact, and tool to prove these simple cases
automatically by automatically looking at the lists of known-distinct items and trying to prove the goal without
cluttering the proofstate (which is important for performance reasons).</p>
So far I’ve found that there are really only three ‘kinds’ of things you can do in Lean while proving; you can
manipulate and add facts to the context, you can manipulate and split or glue goals together, or you can close goals.
Tactics, my intuition says, should have an answer for how they work in each of those states. In my case, distinct</code> is a
type that can be used to express pairwise distinct relationships between (relatively) small, finite lists of variables;
distinguish</code> is a tactic that can prove inequalities using distinct</code> hypotheses, and separate</code> is a tactic that
deconstructs a distinct</code> term into an equivalent set of inequalities. This covers the three cases.</p>
How it works</h1>
Here’s the lede I’ve kept buried: the proofstate is just a Monad</strong>. Essentially, it’s a little virtual machine that
tracks the hypotheses in the LocalContext</code> as a bunch of LocalDecls</code>. These can be easily turned into Expr</code>s, which
is the type of Lean Statements in the AST after elaboration; the Lean parser is a multilayer system, and Expr</code> is the
last step before running actual code^{2</a></sup>. This machine is exposed via the TacticM</code> monad, which is essentially a DSL
for manipulating proofstates. Proofstates themselves are just a collection of variables that get wired together; they
come in three flavors of interest:</p>}

FVars

Free variables, these are the A B C D E X V W</code> above, but also h</code> is an FVar.</li>
I prefer to think of it as ‘FactVariable’, which I think is mostly okay.</li>
</ul>
</li>
BVars

Bound variables, these are the arguments to functions.</li>
The P</code> in the (∀ (P : ℕ), P = 3 → P > 1)</code> above is a BVar</li>
</ul>
</li>
MVars

Meta variables, or holes, these are associated with the current goal(s).</li>
Each must get assigned a term of a matching type to be closed.</li>
</ul>
</li>
</ul>
The TacticM</code> monad exposes a simple programming language to interact with these variables, and the Qq</code> library and
Lean</code> namespace provide the API to manipulate it. I started^{3</a></sup> with the following to cover #1 above:</p>}
structure</span> Distinct</span> {α : </span>Type</span>*} (points : List α) : </span>Prop</span> where</span></span>
  pairwise : List.Pairwise (· ≠ ·) points</span></span>
</span>
namespace</span> Distinct</span></span>
</span>
-- ... snip ...</span></span>
</span>
-- Custom </span>syntax for</span> distinct/distinguish</span></span>
declare_syntax_cat</span> distinct_binder</span></span>
syntax</span> ident+ </span>" : "</span> term : distinct_binder</span></span>
</span>
syntax</span> "distinct"</span> ident+ : term</span></span>
macro_rules</span></span>
  | `(distinct $xs*) => `(Distinct [$xs,*])</span></span>
</span>
-- ... snip ...</span></span>
</span>
end</span> Distinct</span></span></code></pre>
This gets me to distinct A B C D E</code> as a proof term, I can also write Distinct [A,B,C,D,E]</code> but that’s much more
cumbersome. The syntax category is probably not necessary, but it feels nice to give everything its own little home.
Next I added a couple example statements to test with:</p>
</span>
example</span> : distinct A B C D E -> A ≠ X -> A ≠ B ∧ B ≠ C ∧ X ≠ A ∧ (∀ P : Nat, P = </span>2</span> -> P > </span>1</span>) ∧ C ≠ D := </span>by</span></span>
  intro h AneX</span></span>
  distinguish</span></span>
  -- this part doesn't matter, the extra assertions are just to make sure the distinguish step doesn't oversolve</span></span>
  exact AneX.symm</span></span>
  intro P Peq2</span></span>
  rw [Peq2]; trivial</span></span>
</span>
example</span> : distinct A B C D E -> A ≠ X -> A ≠ B ∧ (B ≠ C ∧ X ≠ A) ∧ (∀ P : Nat, P = </span>3</span> -> P > </span>1</span>) ∧ (C ≠ D ∨ V = W) := </span>by</span></span>
  intro h AneX</span></span>
  distinguish</span></span>
  · exact AneX.symm</span></span>
  · intro P Peq3; rw [Peq3]; trivial</span></span>
  · </span>have</span> CneD : C ≠ D := </span>by</span> distinguish</span></span>
    left; trivial</span></span>
</span>
example</span> (A B C D : Point) (h : distinct A B C D) : A ≠ B ∧ B ≠ C ∧ A ≠ D := </span>by</span></span>
  distinguish</span></span>
</span>
example</span> (h : distinct A B) : A ≠ B := </span>by</span> distinguish</span></span>
</span></code></pre>
This shows how distinguish</code> should work; in particular it should:</p>

Flatten and split the goal tree so that every part of the conjunction is its own goal.</li>
Make a list of all the goals that are just an inequality statement</li>
Find all the hypotheses that are created by distinct</code></li>
Use the hypotheses from #3 to try to prove every marked goal from #2</li>
Leave the rest</li>
</ol>
It is certainly possible to do this in other ways, or to extend the traversal through disjunctions or quantifiers, but
the 80% case is ‘flatten things and prove a bunch of single inequalities’ so I started there. The first step is
splitting the goals, this went through a couple iterations, but landed at:</p>
/-- Split conjunction goal into MVars and track which are inequalities -/</span></span>
partial</span> def</span> splitAndTagGoals</span> : TacticM (List MVarId × List Nat) := </span>do</span></span>
  let</span> goal ← getMainGoal</span></span>
</span>
  let rec</span> splitAndExtract (g : MVarId) (idx : Nat) : TacticM (List MVarId × List (Nat × Bool)) := </span>do</span></span>
    let</span> goalType ← g.getType</span></span>
    have</span> goalTypeProp : Q(</span>Prop</span>) := goalType</span></span>
</span>
    match</span> goalTypeProp </span>with</span></span>
    | ~q($a ∧ $b) => </span>do</span></span>
      -- Split conjunction</span></span>
      setGoals [g]</span></span>
      evalTactic (← `(tactic| constructor))</span></span>
      let</span> [leftGoal, rightGoal] ← getGoals | </span>throwError</span> "Expected two goals after constructor"</span></span>
</span>
      -- Recursively process both sides</span></span>
      let</span> (leftMvars, leftTags) ← splitAndExtract leftGoal idx</span></span>
      let</span> rightIdx := idx + leftMvars.length</span></span>
      let</span> (rightMvars, rightTags) ← splitAndExtract rightGoal rightIdx</span></span>
</span>
      return</span> (leftMvars ++ rightMvars, leftTags ++ rightTags)</span></span>
</span>
    | ~q(@Ne _ $a $b) =></span></span>
      -- Inequality - mark as such</span></span>
      return</span> ([g], [(idx, </span>true</span>)])</span></span>
</span>
    | _ =></span></span>
      -- Other goal - not an inequality</span></span>
      return</span> ([g], [(idx, </span>false</span>)])</span></span>
</span>
  let</span> (mvars, tags) ← splitAndExtract goal </span>0</span></span>
  let</span> ineqIndices := tags.filterMap (</span>fun</span> (idx, isIneq) => </span>if</span> isIneq </span>then</span> some idx </span>else</span> none)</span></span>
</span>
  return</span> (mvars, ineqIndices)</span></span></code></pre>
The majority of tactic writing, it seems, is bookkeeping. There is a great deal of bookkeeping in this, but it’s a
pretty simple idea – grab the goal (getMainGoal</code>), then try to break it into its components using the
splitAndExtract</code> helper defined inline; it does this by simple pattern matching on the goal – if it’s a conjunction
(q($a ∧ $b)</code>, which is a quotation of the literal A ∧ B</code> with match placeholder for the left and right sides of the
conjunction), then recurse down each side, updating the index of the goal, if it’s an inequality (~q(@Ne _ $a $b)</code>),
mark it as such, and otherwise mark it as a non-equality.</p>
The other part of this function generates (via the constructor</code> tactic in the first branch of the match) an MVar</code> for
each new goal; so the result is a list of MVarId</code>s, one for each goal; and a second list which indexes the first to
tell us where all the inequality goals are (ineqIndices</code>). This could maybe be done via pattern matching, but
constructor</code> is a nice, predictable tactic, and shouldn’t cause trouble.</p>
Now we’ve successfully decomposed the goal from a single conjunction tree (that is, it’s in some freely associated form,
not uniformly right-associated) to a ‘flat’ (right-associated) list of goals and an index of where the ones we can
prove with distinct are.</p>
One bit that took me a little while to wrap my head around. When you get the goal it is an MVarId</code>, but we need to
deal with its type as a syntactic element that represents a fixed proposition. A goal, ultimately, is just a slot that
expects a construction assigned to it of the appropriate type. It doesn’t carry any information directly with it, so in
order to talk about its type we have to look it up with goal.getType</code>. This gives us an Expr</code>, which seems like what
we need, because Expr</code> is the representation of the syntax, but in fact we need to go one step further and work with
the Q</code>-quoted version of this Expr</code>, which allows us to pick it apart without worrying about the constraints of the
Expr</code> type.</p>
From the LTPG:</p>

– The data structure that is used to represent Lean expressions is Lean.Expr</code>.
– Due to the nature of dependent type theory, Lean.Expr</code> is used to encode types, terms and proofs.
– Thus, Lean.Expr</code> is also what is checked by the Lean kernel when checking proofs.
– ctrl-click on Lean.Expr</code> below to see its definition in the library.
#check Lean.Expr</p>
– Lean has a handy library Qq</code> to help you build Lean.Expr</code> terms with a convenient notation.
open Qq</p>
– Q(...)</code> is a type annotation of an expression, and
– q(...)</code> is an expression</p>
</blockquote>
This took me a bit to grok, and I’m not sure it feels natural yet, but then again quotes and quasiquotes don’t seem like
the sort of thing you’re ever supposed to feel comfortable with, so I suppose it’s feeling the way it’s supposed to. The
type of goalTypeProp</code> in:</p>
let</span> goalType ← g.getType</span></span>
have</span> goalTypeProp : Q(</span>Prop</span>) := goalType</span></span></code></pre>
is the quotation</em> of an arbitrary proposition, not a proposition itself. It’s a syntactic version of, e.g., A ≠ B</code>,
not the statement A ≠ B</code> itself. The same as the difference between: "A ≠ B"</code> and A ≠ B</code>. 🤯 I suppose there is a
reason they call it quoting.</p>
Another thing this function doesn’t do is try to disambiguate situations which are not simple conjunctions; consider a
proposition like:</p>
distinct A B C D -> A ≠ B ∧ (C ≠ D ∨ V = W)</span></span></code></pre>
In this example, it is easy to resolve: we know A ≠ B := by distinguish</code> and C ≠ D := by distinguish</code>, but the
splitAndTagGoals</code> above will treat the (C ≠ D ∨ V = W)</code> disjunction as unassailable. Similarly a condition like:</p>
distinct A B -> ∀ P : </span>Prop</span>, A ≠ B ∨ P</span></span></code></pre>
is obviously true regardless of the choice of P</code>, but the current tool won’t look inside the quantifier to reduce that
to a proof of ∀ P : Prop, True</code>, even though one could conclude A ≠ B</code> easily.</p>
Limitations noted, now I have the broken up goal. Next, I need to gather up the tools I’ll need; I don’t want to try to
use every available fact-in-evidence, just the distinct</code> properties that I know about; this is not the best</em> way to do
things, as there might be ambient facts that could be used to prove the property, but hunting for them involves doing a
lot more mangling of the proofstate, and things were already complicated enough. Here’s the function:</p>
/-- Finds all `Distinct` hypotheses in the local context -/</span></span>
def</span> findDistinctHypos</span> : TacticM (List LocalDecl) := </span>do</span></span>
  let</span> lctx ← getLCtx</span></span>
  let mut</span> distinctHypos : List LocalDecl := []</span></span>
  for</span> decl </span>in</span> lctx </span>do</span></span>
    if</span> decl.isImplementationDetail </span>then continue</span></span>
    let</span> declType ← instantiateMVars decl.type</span></span>
    if</span> declType.isAppOfArity ``Distinct </span>2</span> then</span></span>
      distinctHypos := decl :: distinctHypos</span></span>
  return</span> distinctHypos</span></span></code></pre>
This produces a list of LocalDecl</code>s that match the signature of the distinct structure. The declType.isAppofArity</code>
line is what does that, it’s looking for a literal Distinct</code> of arity 2; which I don’t think is the best</em> way to find
these things, but it is a</em> way and the first one that worked; I suspect there are better tools for this (probably
pattern matching on the type), but for a simple structure like Distinct</code>, it’s easy enough to just search for it. It’s
of arity 2 because it has two parameters, the list of points (its second parameter) is an explicit parameter, but the
type of the values in that list is not constrained to only</em> points, and so there is an implicit type parameter.</p>
Lean is pretty aggressive about making parts of the language implicit, which is a big step in making the language more
ergonomic than other languages I’ve tried. Ergonomics rarely are the only reason I stop using a prover, but it’s
frequently the thing that makes it impossible to get over whatever other struggles I have. Lean does a very good job
here of making it possible – even easy – to let context fill in details you don’t want to care about but need to care
about, probably my favorite part of the language.</p>
The code itself is simple; it walks over each fact in the LocalContext</code> and builds up a list; the let mut</code> is a little
bit of a lie, as I understand. It hides a State</code>-monad-adjacent thing that lets you pretend the variable is mutable
even in this pure-ish environment. This makes it pretty pleasant to write these little programs that query the
proof-state; it’s not too far away from a regular imperative language about as wieldy as shell; not powerful, but
powerful enough.</p>
Once it finds a declaration of the correct type, it adds it to a list of hypotheses that we use in the following
monster^{4</a></sup></p>}
def</span> runDistinct</span> : TacticM Unit := withMainContext </span>do</span></span>
    let</span> (allGoals, ineqIndices) ← splitAndTagGoals</span></span>
    let</span> distinctHypos ← findDistinctHypos</span></span>
    let mut</span> solvedIndices : List Nat := []</span></span>
</span>
    for</span> idx </span>in</span> ineqIndices </span>do</span></span>
      let</span> goalMVar := allGoals[idx]!</span></span>
      setGoals [goalMVar]</span></span>
</span>
      for</span> hypo </span>in</span> distinctHypos </span>do</span></span>
        -- </span>1</span>. </span>break</span> into the fvars on either side</span></span>
        let</span> goalType ← goalMVar.getType</span></span>
        have</span> goalTypeProp : Q(</span>Prop</span>) := goalType</span></span>
        if let</span> ~q(@Ne $typ $lhs $rhs) := goalTypeProp </span>then</span></span>
          -- 2a. </span>if</span> the fvars are the same, </span>then</span> the two things are equal, reject</span></span>
          if</span> lhs.fvarId! != rhs.fvarId! </span>then</span></span>
            -- </span>3</span>. now search the `points` of the `distinct` condition and we can conclude inequality based</span></span>
            -- on the pairwise relationship</span></span>
            if let</span> some points ← Distinct.getPointsExpr hypo.toExpr </span>then</span></span>
              -- establish that both lhs and rhs are </span>in</span> the list of distinct variables</span></span>
              let</span> lhsIn := points.any (</span>fun</span> p => p.isFVar && p.fvarId! == lhs.fvarId!)</span></span>
              let</span> rhsIn := points.any (</span>fun</span> p => p.isFVar && p.fvarId! == rhs.fvarId!)</span></span>
              if</span> lhsIn && rhsIn </span>then</span></span>
                -- we can prove it, we </span>have</span> the technology</span></span>
                let</span> proofGoal ← mkFreshExprMVar goalType</span></span>
                let</span> proofMVar := proofGoal.mvarId!</span></span>
                setGoals [proofMVar]</span></span>
                let</span> hypoName := mkIdent hypo.userName</span></span>
</span>
                -- prove </span>using</span> aesop + simp, this is not ideal, it should be possible to construct a direct</span></span>
                -- proof, but it's not low-effort, so FIXME some other time.</span></span>
                evalTactic (← `(tactic| (</span></span>
                  have</span> h := ($hypoName).pairwise</span></span>
                  simp only [List.Pairwise, List.mem_cons] at h</span></span>
                  aesop</span></span>
                )))</span></span>
</span>
                -- Check </span>if</span> it was solved, </span>then</span> assign the goal </span>if</span> it is.</span></span>
                if</span> ← proofMVar.isAssigned </span>then</span></span>
                  let</span> proof ← instantiateMVars proofGoal</span></span>
                  goalMVar.assign proof</span></span>
          else</span></span>
            -- </span>in</span> this case, lhs is _literally the same </span>variable</span> reference_ as rhs, so we are trying to prove</span></span>
            -- ¬(rfl A), which is just </span>false</span>, so the whole conjunction is </span>false</span> and we should replace the goal</span></span>
            -- </span>with</span> false</span>. I'm not doing that now, but it's doable, I think.</span></span>
            throwError</span> "lhs is identical to rhs, you're trying to prove A ≠ A, and that's no bueno"</span></span>
        else</span></span>
          logInfo </span>m!</span>"</span>{</span>goalType</span>}</span>"</span></span>
          -- 2b. </span>do</span> nothing, this case is not possible because we're only inspecting inequalities.</span></span>
          throwError</span> "not possible"</span></span>
      -- bookkeeping to make sure we set the goals correctly later.</span></span>
      -- FIXME: I think this could probably be based on whether or not the mvar is assigned?</span></span>
      if</span> ← goalMVar.isAssigned </span>then</span></span>
        solvedIndices := idx :: solvedIndices</span></span>
</span>
    -- Collect unsolved goals</span></span>
    let mut</span> unsolvedGoals : List MVarId := []</span></span>
    for</span> i </span>in</span> [:allGoals.length] </span>do</span></span>
      if</span> !solvedIndices.contains i </span>then</span></span>
        unsolvedGoals := unsolvedGoals ++ [allGoals[i]!]</span></span>
</span>
    setGoals unsolvedGoals</span></span></code></pre>
This integrates everything together along with a couple other helpers; but the strategy is really simple and mostly
contained in the -- 2a</code> branch, and within that, the main branch is the one that actually does the work. After the
initial preamble which splits the goal, gathers the facts, and allocates a list for all the goals we automatically
solve; I iterate over each inequality and try to close it with one of the available distinct</code> hypotheses I gathered. I
do this by first checking that the two sides are not ‘literally’ the same, that is, they are different fVars</code>. Recall
that an fVar</code> represents a ‘fact’ variable, and in my case they’ll usually represent named points, so right now I want
to fail if any of the facts we’re trying to prove is a simple falsehood of A ≠ A</code>, as something like simp</code> will
already cover that case. Once we’re in the branch, the process is simple – if the respective fVars</code> for the left and
right sides of the inequality are present in the same list of distinct variables, we know there is a proof they are not
equal, so I construct it using simp</code> and aesop</code>; two existing automated tactics that can resolve most simply proofs.</p>
The simp</code> reduces the distinct</code> goal to the relevant inequalities, and aesop</code> is able to use that to close. I could,
and probably should, directly construct the proof here; but doing so is a little messy and this works alright for most
cases. Since the number of variables is relatively small, it’s unlikely this will become a huge performance issue except
in extreme cases. The reason to be wary here is aesop</code> and simp</code> both iterate over available theorems in the
environment, not</em> just the hypotheses in your proofstate. Tagging a theorem with @[simp]</code>, or equivalently with one of
the tags aesop</code> cares about, will cause each of these tactics to consider them when running, which it then considers
recursively up to some user-defined limit. This can lead to two spooky-action-at-a-distance</code> problems, like:</p>

adding a new simp</code> tag or aesop</code> tag can result in working proofs failing as they overflow the recursion stack</li>
adding a new simp</code> or aesop</code> can also result in a performance problem on random theorems. If the new entry adds a
new path that can be recursed through, it may result in the prover spending time chasing rabbits.</li>
</ol>
The moral of the story is to be judicious about the use of simp</code> and aesop</code> both as tags and tactics. In this case,
the scope of available and relevant theorems is relatively</em> small, and it should be okay. In principle there should be
a finitely constructible proof for these facts, and as the FIXME</code> mentions, I should probably use it; but this is
certified Good Enough(tm).</p>

Aside: It’s really</em> interesting how performance of the prover becomes a concern for the math side of things. Earlier,
while working more directly on the geometry, I was pretty liberal using the tauto</code> tactic; which more or less
brute-forces its way through a proof by trying to apply simp</code>, contradiction</code>, and any available hypothesis to the
goal to try to resolve it. In small contexts with few active hypotheses, it’s quick enough; but on one proof where I
had two dozen or so active facts, it ground the machine to a halt and my file would take tens of seconds to evaluate.
Real nasty.</p>
The cool thing, though, is that good programming</em> practices seem to translate naturally to good mathematical</em>
practices. Spending time refactoring really does translate to better quality arguments that feel more elegant. I’ve
long associated refactoring with the feeling of a good proof technique or clever mathematical argument, it’s very
validating to see it play out in code.</p>
</blockquote>
Once I’ve attempted the proof, I check to see if I actually managed to assign the variable (it always should, another
benefit to a direct construction would be eliminating this branch), and then assign the goal to the MVar associated with
the proof; closing the goal.</p>
The rest is bookkeeping, which Lean doesn’t do on its own. The actual tactic is very simple, but updating
the state is tricky.</p>
Syntax and tooling</h1>
All of this gets us through the code</em> part of this, but there is a whole second step. At the beginning, I showed a bit
of the notation</code> created for distinct</code>, in fact, there are three items of syntax I need; I have the tools, but they
are not yet integrated, I could run a proof like:</p>
example</span> (A B C D : Point) (h : distinct A B C D) : A ≠ B ∧ B ≠ C ∧ A ≠ D := </span>by</span></span>
  run_tac runDistinct</span></span></code></pre>
but this is ugly, I prefer the distinguish</code> keyword I set before. In fact, I need to be able to use this syntax in
three contexts as prior mentioned:</p>

As a hypothesis in a theorem statement, e.g. distinct A B C -> ...</code></li>
As a conclusion of a theorem statement, e.g., ... -> distinct A B C := by ...</code></li>
As a tool for intermediate proof steps, e.g., have h := by distinguish</code>, or separate at distinctABC</code> (where
distinctABC : distinct A B C</code>), or even separate</code> to separate a goal of distinct A B C ...</code> into a conjunction of
inequalities.</li>
</ol>
To do this, we must dive into the darkest magic of this whole thing, the syntax</code> and elab</code> APIs. The first two items
are easy, they just construct the Distinct</code> structure while eliding the need for brackets and commas (in the case of
distinct</code>, which I showed at the beginning):</p>
-- Custom </span>syntax for</span> distinct/distinguish</span></span>
declare_syntax_cat</span> distinct_binder</span></span>
syntax</span> ident+ </span>" : "</span> term : distinct_binder</span></span>
</span>
syntax</span> "distinct"</span> ident+ : term</span></span>
macro_rules</span></span>
  | `(distinct $xs*) => `(Distinct [$xs,*])</span></span></code></pre>
The distinguish</code> tactic just runs the runDistinct</code> function</p>
syntax</span> "distinguish"</span> : tactic</span></span>
</span>
macro_rules</span></span>
  | `(tactic| distinguish) => `(tactic| run_tac runDistinct)</span></span></code></pre>
But the last item, separate</code>, which deals with destructuring a distinct</code> goal or fact into its constituent inequalities is… well it’s a lot.</p>
This is definitely the most hobbled-together, hasty and unkempt part of this whole thing, so strap in:</p>
/-- Extract points from a List Expr at the meta level -/</span></span>
partial</span> def</span> extractPoints</span> (e : Expr) : List Expr :=</span></span>
  if</span> e.isAppOfArity ``List.cons </span>3</span> then</span></span>
    let</span> head := e.appFn!.appArg!</span></span>
    let</span> tail := e.appArg!</span></span>
    head :: extractPoints tail</span></span>
  else</span></span>
    []</span></span>
</span>
syntax</span> "separate"</span> (</span>" at "</span> ident)? : tactic</span></span></code></pre>
This quick helper just gets the list of Exprs inside a list, those exprs need to get pairwise connected into a
conjunction of inequality statements. I also declare the syntax to accept separate</code> and separate at h</code>, where h</code> is
some fact in the context. Then I use the elab_rules</code> to create a macro expansion. elab_rules</code> is similar to
macro_rules</code>, but is much more powerful, quoting the Lean Metaprogramming
Guide</em></a></p>

In principle, you can do with a macro (almost?) anything you can do with the elab function. Just write what you would
have in the body of your elab as a syntax within macro. However, the rule of thumb here is to only use macros when the
conversion is simple and truly feels elementary to the point of aliasing. As Henrik Böving puts it: “as soon as types
or control flow is involved a macro is probably not reasonable anymore”.</p>
</blockquote>
separate</code> has types and control flow, so we’re above a macro; and the only thing above a macro is elab. The rules break
into two parts, the at h</code> side is handled first, goal side (which is simpler) is second. The main difference is the
first branch needs to name individual hypotheses it’s adding to the local context, while the second just needs to
introduce new goals. I suspect there is more refactoring here, but I’ll explain it as it is; take a look at the whole
thing:</p>
elab_rules</span> : tactic</span></span>
  | `(tactic| separate $[at $h]?) => </span>do</span></span>
  match</span> h </span>with</span></span>
    | some hId => </span>do</span></span>
      withMainContext </span>do</span></span>
        let</span> hExpr ← elabTerm hId none</span></span>
        let</span> hType ← instantiateMVars (← inferType hExpr)</span></span>
        let</span> hType ← whnf hType</span></span>
        if</span> !hType.isAppOfArity ``Distinct </span>2</span> then</span></span>
          throwError</span> "separate: {hId} is not a `Distinct` hypothesis"</span></span>
        let</span> some points ← Distinct.getPointsExpr hExpr</span></span>
          | </span>throwError</span> "separate: could not extract points from {hId}"</span></span>
</span>
        for</span> i </span>in</span> [:points.length] </span>do</span></span>
          for</span> j </span>in</span> [i+</span>1</span>:points.length] </span>do</span></span>
            let</span> pi := points[i]!</span></span>
            let</span> pj := points[j]!</span></span>
            let</span> ineqType ← mkAppM ``Ne #[pi, pj]</span></span>
            let</span> ineqStx ← PrettyPrinter.delab ineqType</span></span>
            -- Build a name like `AneB` </span>from</span> the fvar usernames</span></span>
            let</span> iName := (← FVarId.getUserName pi.fvarId!).toString</span></span>
            let</span> jName := (← FVarId.getUserName pj.fvarId!).toString</span></span>
            let</span> hypName := mkIdent (Name.mkSimple (iName ++ </span>"ne"</span> ++ jName))</span></span>
            evalTactic (← `(tactic|</span></span>
              have</span> $hypName : $ineqStx := </span>by</span></span>
                have</span> hp := ($hId).pairwise</span></span>
                simp only [List.pairwise_cons, List.mem_cons, List.mem_singleton,</span></span>
                           List.not_mem_nil, List.Pairwise.nil] at hp</span></span>
                aesop))</span></span>
</span>
    | none => </span>do</span></span>
      withMainContext </span>do</span></span>
        let</span> goal ← getMainGoal</span></span>
        let</span> goalType ← instantiateMVars (← goal.getType)</span></span>
        let</span> goalType ← whnf goalType</span></span>
        if</span> !goalType.isAppOfArity ``Distinct </span>2</span> then</span></span>
          throwError</span> "separate: goal is not of the form `Distinct [...]`"</span></span>
</span>
        let</span> listExpr := goalType.getArg! </span>1</span></span>
        let</span> points := extractPoints listExpr</span></span>
</span>
        evalTactic (← `(tactic| apply Distinct.mk))</span></span>
</span>
        for</span> _ </span>in</span> [:points.length] </span>do</span></span>
          try</span> evalTactic (← `(tactic| rw [List.pairwise_cons]))</span></span>
          catch</span> _ => pure ()</span></span>
</span>
        evalTactic (← `(tactic| simp only [List.mem_cons, List.mem_singleton, List.Pairwise.nil,</span></span>
                                            List.not_mem_nil, forall_eq_or_imp, forall_eq,</span></span>
                                            forall_const, IsEmpty.forall_iff, forall_true_iff,</span></span>
                                            and_true, true_and, and_assoc] at *))</span></span>
        try</span> evalTactic (← `(tactic| exact List.Pairwise.nil))</span></span>
        catch</span> _ => pure ()</span></span></code></pre>
notice the at h</code> side uses simp</code> and aesop</code>, whereas the goal side is just simp</code>. Additionally, I restricted simp</code>
to a fixed list of theorems; which can help address the performance issues. In both cases the simp</code> flattens out the
goal into a series of inequalities, and the aesop</code> line will then close the inequalities.</p>
I’ll focus at the second half, as it’s easy to explain:</p>
      withMainContext </span>do</span></span>
        let</span> goal ← getMainGoal</span></span>
        let</span> goalType ← instantiateMVars (← goal.getType)</span></span>
        let</span> goalType ← whnf goalType</span></span>
        if</span> !goalType.isAppOfArity ``Distinct </span>2</span> then</span></span>
          throwError</span> "separate: goal is not of the form `Distinct [...]`"</span></span>
</span>
        let</span> listExpr := goalType.getArg! </span>1</span></span>
        let</span> points := extractPoints listExpr</span></span>
</span>
        evalTactic (← `(tactic| apply Distinct.mk))</span></span>
</span>
        for</span> _ </span>in</span> [:points.length] </span>do</span></span>
          try</span> evalTactic (← `(tactic| rw [List.pairwise_cons]))</span></span>
          catch</span> _ => pure ()</span></span>
</span>
        evalTactic (← `(tactic| simp only [List.mem_cons, List.mem_singleton, List.Pairwise.nil,</span></span>
                                            List.not_mem_nil, forall_eq_or_imp, forall_eq,</span></span>
                                            forall_const, IsEmpty.forall_iff, forall_true_iff,</span></span>
                                            and_true, true_and, and_assoc] at *))</span></span>
        try</span> evalTactic (← `(tactic| exact List.Pairwise.nil))</span></span>
        catch</span> _ => pure ()</span></span></code></pre>
This is similar to the runDistinct</code> function – I grab the context, and look at the goal. I’m going to ‘prove’ this
goal by reducing it from a ‘prove this conjunction’ to ‘the conjunction is proved only if its constituent terms are
proved’. To do that, I instantiate the MVar and inspect its type, apply a whnf</code> to ensure it’s in a normal form, and then
verify it is indeed a distinct</code> goal with the same arity method as runDistinct</code>. Worth noting whnf</code> is another
recursive tool which can have similar recursion depth issues, though it’s less susceptible in my experience.</p>
Next, I grab the list of points via the extractPoints</code> helper, and walk over each one, rewriting with:</p>
@[simp]</span> theorem</span> pairwise_cons</span> : Pairwise R (a::l) ↔ (∀ a', a' ∈ l → R a a') ∧ Pairwise R l :=</span></span></code></pre>
which says, “If an item is in a List.Pairwise R</code>, then it has the relationship R</code> with all other items in the pairwise
relationship following its position in the list”</p>
If this fails for any reason, the point is skipped and we won’t be able to prove everything, but that case is actually
impossible; because we know the list is only items in a pairwise relationship anyway. The last try evalTactic</code> covers a
case where there aren’t enough elements in the distinct set, and in this case if the tactic fails we don’t care, it’s a
convenience to not have loose, trivial goals floating around.</p>
All that, ultimately, allows for this to work:</p>
example</span> : A ≠ B ∧ A ≠ C ∧ B ≠ C -> distinct A B C := </span>by</span></span>
  intro ⟨AneB, AneC, BneC⟩</span></span>
  separate</span></span>
  exact ⟨AneB, AneC, BneC⟩</span></span>
</span>
example</span> : D ≠ A ∧ D ≠ B ∧ D ≠ C -> distinct A B C -> distinct A B C D := </span>by</span></span>
  intro ⟨DneA, DneB, DneC⟩ distinctABC</span></span>
  separate</span></span>
  distinguish</span></span>
  repeat</span> tauto -- tauto covers the .symm</span></span>
</span>
example</span> : D ≠ A ∧ D ≠ B ∧ D ≠ C -> distinct A B C -> distinct A B C D := </span>by</span></span>
  intro ⟨DneA, DneB, DneC⟩ distinctABC</span></span>
  separate at distinctABC</span></span>
  separate</span></span>
  repeat</span> tauto -- tauto covers the .symm</span></span></code></pre>Effect on the initial exercise</h1>
This was the original:</p>
/-- p146. Given A-B-C and A-C-D:</span></span>
  (a) Prove that A,B,C, and D are four distinct points (the proof requires an axiom)</span></span>
-/</span></span>
theorem</span> Ex1.a</span> : A - B - C ∧ A - C - D -> distinct A B C D := </span>by</span></span>
  intro ⟨ABC, ACD⟩</span></span>
  have</span> distinctABC := Betweenness.abc_imp_distinct ABC</span></span>
  have</span> distinctACD := Betweenness.abc_imp_distinct ACD</span></span>
  simp only [ne_eq, List.pairwise_cons, List.mem_cons, List.not_mem_nil, or_false, forall_eq_or_imp,</span></span>
    forall_eq, IsEmpty.forall_iff, implies_true, List.Pairwise.nil, and_self, and_true]</span></span>
  have</span> AneB : A ≠ B := </span>by</span> distinguish distinctABC A B</span></span>
  have</span> AneC : A ≠ C := </span>by</span> distinguish distinctABC A C</span></span>
  have</span> BneC : B ≠ C := </span>by</span> distinguish distinctABC B C</span></span>
  have</span> AneD : A ≠ D := </span>by</span> distinguish distinctACD A D</span></span>
  have</span> CneD : C ≠ D := </span>by</span> distinguish distinctACD C D</span></span>
  refine ⟨⟨AneB, AneC, AneD⟩, ⟨BneC, ?_⟩, CneD⟩</span></span>
  -- </span>have</span> BneD : B ≠ D := </span>by</span> distinguish distinctACD B D</span></span>
  by_contra! BeqD</span></span>
  rw [<- BeqD] at ACD</span></span>
  exact Betweenness.absurdity_abc_acb ⟨ABC, ACD⟩</span></span></code></pre>
To be honest, this isn’t too bad, but it is a little bad; the simp</code> line is ugly, the have</code>s are a little unsightly,
and I have to hand-hold the old tactic a lot to get it to work. This should</em> be simple to conclude, and that’s what
distinguish</code> is for, here it is after all that work:</p>
/-- p146. Given A-B-C and A-C-D:</span></span>
  (a) Prove that A,B,C, and D are four distinct points (the proof requires an axiom)</span></span>
-/</span></span>
theorem</span> Ex1.a</span> : A - B - C ∧ A - C - D -> distinct A B C D := </span>by</span></span>
  intro ⟨ABC, ACD⟩</span></span>
  have</span> distinctABC := Betweenness.abc_imp_distinct ABC</span></span>
  have</span> distinctACD := Betweenness.abc_imp_distinct ACD</span></span>
  -- The majority of cases are handled </span>by</span> the custom tactics</span></span>
  separate; distinguish</span></span>
  -- The remaining case is to disprove BeqD under the betweenness hypotheses</span></span>
  by_contra! BeqD</span></span>
  rw [<- BeqD] at ACD</span></span>
  exact Betweenness.absurdity_abc_acb ⟨ABC, ACD⟩</span></span></code></pre>
Much</em> nicer. I can grab a couple distinctness hypotheses using existing theorems about Betweenness of points. I then separate</code> the main goal into a list of inequalities like:</p>
A ≠ B ∧ A ≠ C ∧ A ≠ D ∧ B ≠ C ∧ B ≠ D ∧ C ≠ D</span></span></code></pre>
most of which are dispatched by the distinguish</code> command immediately following it. The only remaining case is that B</code>
and D</code> are distinct, which is easy to prove by contradiction and a known absurdity.</p>
What’s left</h1>
There are a lot of improvements to make, I think; my use of the elab</code> facilities is pretty naive, and I think a lot
could be cleaned up if I had a better understanding of the API, but no progress without practice, and this really
simplifies a lot of annoying bookkeeping. The fact that the syntax is so directly adjustable and easy to extend and
tweak is really satisfying. I spent a lot of time writing Ruby, and the reason I liked it was really down to how easy it
was to get the interpreter to read the way I wanted. Lean is that but with even more flexibility and a really powerful
API for both extending the language and also manipulating the proof state. Since the language is so focused on the proof
state anyway, being able to automate the interactions really feels like a superpower. Truly cool stuff.</p>
I have some plans to add lemmas around how to extend the distinct</code> condition, ‘adding’ a point to the condition
requires a proof that the new item is distinct from all other items, which is a goal-state that should be automatable to
create, but would certainly be a pain to do by hand. Removing items from a distinct</code> condition is comparatively easy
(it requires no real ‘proof’), but proving a general theorem that doesn’t regard the length of the list is a little
tricky.</p>
I also have another condition, collinear</code>, that needs similar syntactic treatment, and probably some custom tactics as
well.</p>
There is also, of course, extending distinguish</code> and separate</code> to look past disjunctions and quantifiers to try to
resolve goals.</p>
One thing I didn’t mention was the delaborator</code>. I didn’t talk about it mostly because I barely understand it, but it’s
similar to a prettyprinter. It helps display facts and goals in a ‘nice’ way. Right now the facts look like Distinct [A, B, C]</code>, not distinct A B C</code>, which is not too</em> confusing, but originally they looked like distinctABC : List.Pairwise (· ≠ ·) [A, B, C]</code> which was</em> pretty weird, the delaborator is the thing that fixes that, I think; I
haven’t really dug in.</p>
It’d also be nice to grab other known inequality-related facts and try to use those, or reassemble things back down to a
single goal after splitting; these are quality-of-life things. It’s often the case that you’ll have a couple spare facts
floating around that could resolve more goals, but distinguish</code> will ignore them, or you’ll have a fact like h : (∀ P : Nat, P = 3 -> P > 1) ∧ (C ≠ D ∨ V = W)</code>, and if you end up with a pair of goals you have to resolve them across two
lines with exact h.left; exact h.right</code> instead of just exact h</code>. Low frequency, but possible.</p>
But overall that’s it. That’s how it works. It’s just a Monad. It’s got a little scripting language in it, and now I’ve
taken about ~2000 words more than the LTPG to explain the same things but worse, but it was fun to do it and that’s what
matters.</p>
Thanks for nudging me towards writing about it.</p>
/Joe</p>
P.S.</p>
A stunning addendum to my suggested trichotomy of things a keyword might operate on – another has been found!</p>
example</span> : distinct A B C D -> distinct A B C := </span>by</span></span>
  intro dABCD</span></span>
  exact dABCD forgetting D</span></span></code></pre>
This is like a little wrapped up proof, similar to the distinguish</code> proof, but instead of being a tactic it’s a sort of
term-mode operator. The implementation works for both collinear</code> and distinct</code> by doing type-dispatch in the
elaboration step. It constructs the proof dynamically, similar to distinguish</code>, but without needing aesop</code>, which is
pretty handy. Operators on proof terms like this is a whole other world I hadn’t considered for syntax trickery. Lean is
so cool.</p>
P.P.S. (13-MAY-2026)</p>
I got distracted and left this project alone for a bit, but coming back to it, I noticed an interesting,
mostly-unrelated-to-syntax-hacking thing that needed a place to live.</p>
The project this repo encapsulates is one of in-place formalization, trying to capture Greenberg’s original argument as
made</em> instead of the most idiomatic Lean formalization of the fact. Terence Tao makes a similar caveat in his Analysis</em>
project:</p>

The files in this directory contain a formalization of my text Analysis I into Lean. The formalization is intended to
be as faithful a paraphrasing as possible to the original text</strong>, while also showcasing Lean’s features and syntax. In
particular, the formalization is not optimized for efficiency, and in some cases may deviate from idiomatic Lean
usage.</strong></p>

README.md of teorth/analysis</a>, emph mine</li>
</ul>
</blockquote>
I ran into a similar issue with P3.3, which uses (as I ultimately extracted in Intersection.between_splits</code>) the
converse of Intersection.splits_points</code> to dispatch which points end up on which side of the splitting line to
construct, ultimately, (most of) the apparatus needed to define an arbitrary A - B - ... - Z</code> betweenness
specification. This is its own little syntax rabbit warren to explore, but what was really interesting was the decision
point in how I’m formalizing Greenberg.</p>
On the one hand, if I were to formalize a theory of Geometry in Lean, I don’t think I’d pick this text to work from.
This makes some sense, Greenberg wasn’t optimizing to be formalized, he was optimizing for human intuition. As such, he
relies occasionally on ‘big’ (in the sense of # of lines of Lean) steps that are deeply intuitive</em>, but frustrating to
formalize.</p>
There’s been a lot</a> of
talk</a> recently about
how</a>
LLMs/Agents/“AI”</a> are going to
obsoletize</a> us in the context of
mathematics. I think this distinction points at a specific place where, at least for now, it can’t.</p>
I’ve worked a little bit here with some LLM tools; and while I was, at first, extremely antagonistic (irrationally so)
about them, I think I’m coming around a bit. In working on refactoring out that lemma, in particular, it did a great job
of immediately spotting and extracting the common lemma (which makes sense, it wasn’t hard to spot the duplication, just
a little tricky to extract amidst all the tactics). Probing it even a little made it clear that the process was
mechanical and it had no intuition</em> for the structure, just the ability to see the pattern and extract it.</p>
I ended up in a conundrum: how much do I leave in the proof to mimic Greenberg’s initial argument, which relies on a
deeply intuitive and natural conclusion, but which – at least in the context of my abilities with Lean – is not a
trivial conclusion, and how much do I refactor to make the argument ‘nice’? On the one hand, my goal is to line-for-line
confirm the original arguments, filling in the areas that were not ‘fully rigorous’ (as far as you trust Lean, anyway),
and not necessarily to write ‘good Lean’ necessarily that has the kinds of qualities we might look for in a standard
codebase (e.g., Mathlib). No shade to the standards, I’m just a novice and I only learn by making all the mistakes
myself.</p>
I offer no conclusion or deeper insight other than to say my solution was to punt as hard and far as I could. I left the
original argument intact on one branch, and used (or plan to use, at time-of-writing I have two corollaries left to
prove) the extracted between_splits</code> lemma on the others, but I think it’s interesting to note that; while we may be
able to ‘harvest the overhang’ as David Bessis put it in his article linked above, the consequence is not merely a loss
of the ‘Theorem Economy’, but that since this will give us only</em> the factored, narrow, intuition-free proofs; I’m not
even sure how useful the result would really be to understanding mathematics more deeply. The point</em> of all of this is
to find the ‘big but intuitive’ jumps that we can use not just here, but elsewhere. The jump that Greenberg makes is one
that requires the reader to grab for the intuition that ‘betweenness’ is really just a kind of ordered listing of the
names of points. It points towards a lighthouse far in the distance and says “This ship will sail us all the way to
where Dedekind was when he made his cuts.” I don’t think this small lemma would be enough to break the intuition, but it
certainly shortens its long gaze, and if I focused instead on the particular arrangement of lines and halyards, I’d’ve
missed seeing the shoreline approaching, and that seems like more than just a simple mechanical disaster; it seems more
like we’re building a tunnel through mathematics to get to the destination, with no concern for what we missed on our
long, boring drive there. That makes me sad.</p>



Of course, this is equal parts real – I do mean this as a direct letter to you; but also it’s just going to be
stuck in the repo and anyone’ll be able to read it, and I’ve written explanations of stuff I’m sure you already know
well from experience with Haskell and the like, so it’s still also kind of a blog post? Left as an exercise to the
reader. ↩</a></p>
</li>

It is, of course more complicated than
that</a>.
You can bypass Expr. You can convert things back and forth. The parser is extremely</em> flexible. ↩</a></p>
</li>

This is a lie, I started in a much less structured way that mostly involved me copying large chunks of Mathlib and
hitting it with my face until it mostly worked. I cannot emphasize enough how far from comprehension some of this is to
me; it’s delightful. ↩</a></p>
</li>

I am positive</em> there is a cleaner way to do this; I don’t know what it is, but I’m sure that spending more time
building up a better structure over the proof would help simplify the monster here, but the critical thing is this does
work for most of what I need, and so the tyranny of the local optimum is definitely</em> going to win. ↩</a></p>
</li>
</ol>
</section>


2-MAR-2026
2026-03-02T00:00:00+00:00
2145</h2>
Got distinguish</code> working, I need a lemma, though, to decompose a distinct</code> term in the goal to it’s relevant ineq
goals; and then I get to do more repairs on proofs to get them running again. The distinguish</code> stuff was certainly a
trip to get working; it’s simultaneously very easy and very hard to think about metaprogramming Lean. collinear</code> is
going to be another headtrip, I think.</p>
Skunks are out, though, so spring is coming soon; and spring means open windows, and open windows are good for math.</p>


28-FEB-2026
2026-02-28T00:00:00+00:00
2220</h2>
I’m getting into the weeds of API design here and I’m finding an increasing need to learn how to actually use the Elab
and stop cobbling together snippets and screwing around until I get it working. There’s a cool project about
metaprogramming in Lean which is itself a lean project, so I might take a little time and go through that.</p>
The main struggle right now is with coercing lean into an ergonomic API for talking about distinct</code> points and
collinear</code> points; in particular, I’d like to have theorems that are something like:</p>
distinct A B C ... /\ D =/= ...</span></span></code></pre>
I decided I’d talk about this in a vlog, so if I ever post that anywhere I’ll edit a link in here.</p>


27-FEB-2026
2026-02-27T00:00:00+00:00
0041</h2>
Working with collinearity is a pain; I currently have it limited to a triple, so reasoning about larger bodies of
collinear points is a pain (I have to manually extract the induced line for each triple and manually correlate them). I
have a similar problem with distinct</code>, which is for pairwise distinct things-with-equality.</p>
I need to build some better tools for reasoning about these, because it makes proofs a pain to follow and is a common
place where I burn tauto</code> time.</p>


26-FEB-2026
2026-02-26T00:00:00+00:00
2145</h2>
I’m making good headway on formalizing Greenberg’s proof of 3.3; there was mostly just small stuff missing; my proof of
step (5) is close to done, but the proof is quite big as a result, and it’s a conjuction of two similar but very
slightly different arguments that is going to be hard to suffice</code> away.</p>
I also ran into a new problem, speed. I made liberal use of tauto</code> across this code base and it is starting to hurt.
It’s mostly just laziness and a misunderstanding of cost; I’ve started taking it out where possible. I’ll need to add
another script (or maybe a lint) to count these and chase them away. I’ll also need to add a profile step to the CI. The
stats it produces are pretty basic, but it’s more than enough to chase away performance blowups. The issue does seem
local to the proof (which makes sense, as it’s essentially just hammering away using the available hypothesis, which
don’t cut across proofs), it’s interesting how proof complexity becomes a performance hazard; you lose a powerful tool
when the proof is too large. Even if you don’t intend to keep the tauto, knowing that your environment has all the
equipment needed to finish the proof is quite useful, and if the proof is so complex that tauto takes forever, you are
essentially blinded by your own inefficiency.</p>
Refactoring has always felt like an ‘elegant’ thing to do; and it’s neat to see that mathematical parsimony, clever
argument, and smart lemma choices; things that feel themselves like elegance, are directly correlated with each other in
such a nice way.</p>
I fucking love math.</p>
Going to take a break from 3.3 and clean up tautos for a few commits.</p>


25-FEB-2026
2026-02-25T00:00:00+00:00
0028</h2>
I can’t believe I didn’t have this going already.</p>
I’m trying to reorganize things a bit. I have a huge pile of general theory that I had to build up to formalize 3.2, and
I learned a bunch about the namespace system, and I have a rough idea of what I want to do, but I’m trying to think of
the best way to structure things.</p>
My current setup has</p>
Geometry/</span></span>
    ChX/ -- directory per chapter</span></span>
        Prop/</span></span>
            PY.lean -- file per proposition and related correlaries. Currently contains a bunch of loose theory</span></span>
        Ex/</span></span>
            ExZ.lean -- file per exercise (if multipart, it's all here).</span></span>
        Prop.lean</span></span>
        Ex.lean</span></span>
        README.md</span></span>
    Theory/</span></span>
        Concept/ -- A directory for each "Concept", e.g., Line, Betweenness, Collinearity, Intersection, etc</span></span>
            ChX.lean -- A file for all the general theory needed for that chapter. May depend on previous chapters.</span></span>
            ChX/ -- if there is a directory, ChX.lean should exist to bring in it's contents</span></span>
    Syntax.lean -- any custom syntax I need at the topmost level, mostly empty</span></span>
    Tactics.lean -- imports mathlib tactics</span></span>
    Theory.lean -- imports everything, main entry point to the theory</span></span>
Geometry.lean -- pulls in the chapter propositions and exercises via the `Prop.lean`</span></span></code></pre>
This mostly lets me interleave and get all the ordering right, but it’s a bear to build the main import file
(Theory.lean) because the order of the tree is wrong, chapters should parent concepts, but then I have the opposite
problem in the directory structure – I have to replicate the concept folders over and over.</p>
I really</em> wish that lean just jammed everything into a database, workedout the dependencies, and complained when it
couldn’t. Maybe the module system will fix this? I haven’t really looked too much into it except for my initial attempt,
I’ll have to investigate.</p>
For now I suppose I can write a script to generate the theory file.</p>
I added a flake.nix and set up python, so I suppose I’ll use that. I saw the blueprint tool uses python to interact with
the lean code to generate the graphviz file, so maybe I can crib from them a bit.</p>
Unrelated, it’d be very cool to build some kind of geogebra/lean connection (maybe using
proofwidgets</a>?), not sure the feasibility (especially since I’m
not on vscode which I suspect is required, but haven’t found an explicit statement thereof).</p>
2235</h2>
I got pretty much everything reorganized, it wasn’t too</em> painful. I also rewrote a bunch of proofs in a much nicer way;
in particular Line.line_trichotomy</code> is a really handy tool.</p>
The debate now is whether I’m going to automate generating the Geometry.lean and Theory.lean files; and more generally
it’d be nice to have some scaffolding scripts for creating new chapters/etc in the correct structure. I had a ChX</code>
chapter-template, but I wasn’t sure what I was doing when I originally made it. The things I’d like some kind of
template/script for are:</p>

Constructing the Geometry.lean, which includes all the actual chapters</li>
Constructing the Theory.lean, which is where all the general theory and lemmas go</li>
Constructing the web</code> and print</code> blueprint files via some introspection over all the theorems/lemmas in the
codebase</li>
Creating a new chapter/theory section according to whatever strictures I need.</li>
</ol>
I don’t think I’m going to do this now, I’d like to get back to proving, but something to think on.</p>


6-JAN-2026
2026-01-06T00:00:00+00:00
1337</h2>
lmao funny number.</p>
I have the initial input handler more or less working, I need to commit off what I have for the image pipeline side so
it’s saved somewhere then remove it.</p>
A critical next step is building a better build/test/repeat loop. I’ve been sitting on this work for a while because
testing it feels bad, lots of waiting, lots of actions-per-loop.</p>
bevy_testing</code> and headless mode are probably the way to go, but it’d be cool if I can do some amount of input testing,
maybe get insta working somehow, not sure.</p>
Hot code reloading would be nice but I don’t want to fight through the ratatui component of things to get it working.</p>
I’m leaning towards refactoring the input manager to be all-bevy, then feed back to the ratatui side, but not sure if
that makes sense yet.</p>


1-JAN-2026
2026-01-01T00:00:00+00:00
1158</h2>
I’ve been off working on upgrades to the editor and general improvements to my workflow, but I’ve also been reading a
bunch about climate modeling because I am constantly drawn towards the fun part and not the boring stuff it takes to get
there, which is why I’m writing this instead of getting the IO widget working. I’m going to go work on the IO widget
now, and hopefully come back and write another entry when I finish it.</p>


29-DEC-2025
2025-12-29T00:00:00+00:00
1709</h2>
Merged a pile of stuff into glamdring, using edgy</code> to get some sidebars in place and make better use of the ultrawide.
The log</code> section is pretty crap at the moment but I’m hoping I can eventually get it to scroll through and show all the
content that would otherwise be displayed via :Notifications</code> in a buffer that auto-refreshes (a la VSCode’s version of
same).</p>
I also set up lean4</code> support (in principle) and stubbed in modelica</code> support. These will be for general use and
specific use, respectively; lean</code> just looks like a good tool to learn, and modelica</code> is going to factor into
truncheon</code> in short order.</p>


9-DEC-2025
2025-12-09T00:00:00+00:00
1136</h2>
Got logging and the picker updating better, still having trouble with the camera widget, likely down to not
understanding exactly the scope of the ‘strategy’ system that BRC is using.</p>
22-DEC-2025</h2>
1144</h2>
Alright, pretty much everything is ‘working’ now, I switched back to the standard BRC output for now, with a plan to
replace with the ‘full resolution’ version in the future. For now it’s good enough to proceed with the rest of things.</p>
I think the next steps are the following:</p>

Input/Command parsing</li>
Representing “Spaces”, the first of which is probably as standard “Overlay”-based hexgrid (similar to what’s on offer
in Foundry or R20), followed by a sparse “HexTopology” – which stores individual hexes and their locations and will
(eventually) support something like WFC for generating more terrain</li>
diagnostic components rendering in the ratatui interface – e.g., the rest of the TUI Logger widget, a proper command
IO widget, etc</li>
Mouse integration + logged click events.</li>
</ol>
#2 is what I’ve been chipping at, but I think #1/#3 and to a lesser extent #4 are the better things to work on right
now, as the interaction I want to have is largely command-driven.</p>


8-DEC-2025
2025-12-08T00:00:00+00:00
1419</h2>
Fighting with nixos and dioxus and bevy and friends. Makes me think of building a VM or container for just dev. NixOS is
a fantastic thing for static environments, but it’s very frustrating to use in a dev environment.</p>
An Arch box w/ home-manager may be in the future.</p>


4-DEC-2025
2025-12-04T00:00:00+00:00
1457</h2>
Feeling a bit frustrated trying to cajole bevy_ratatui_camera</code> into ratatui_image</code> and get something sensible to pop
out. I suspect many things, but I’ve also been thinking a bit about how to approach this if I can’t get it working at
the CLI level. In particular, getting it to run in a browser is appealing from a delivery perspective, it would be
significantly simpler to embed it as a WebGL/WSGL + WASM binary, at which point I could ostensibly have hot-reloading as
well; it minimizes the testability to some extent (insta</code> probably won’t work), but it would mean I could just use
bevy</code> as intended, rather than trying to force it into a ratatui_image</code> block.</p>
Another alternative is to move development to be directly on dragon rather than over SSH. I like the idea of trying to
get something that uses kitty’s image protocol to do real ‘graphics’ over SSH, but I also just want the thing to work so
I can get on with the actual point of this project.</p>


2-DEC-2025
2025-12-02T00:00:00+00:00
1220</h2>
I’m working on getting everything organized. I decided to just drop the current state into a commit, and then start
cutting through things till I have something functional again. I’m going to rip out all the SVG stuff after that commit
as I don’t intend to use it, and I think the more I pare down the better time I’m going to have.</p>


1-DEC-2025
2025-12-01T00:00:00+00:00
1334</h2>
I think I have settled in on what this is, it is a campaign system. A GM of a TTRPG chooses a TTRPG system, and creates
maps/players/etc and then tracks their goings on via this system, which manages hex/depth crawls, and tracks multiple
parties of both PCs and NPCs and generates “Decisions” which end up assigned to different players. Most will be left to
the GM, who can use a scripting language to make decisions automatically or intervene to make specific decisions when
they care to. Others get assigned to the players and the GM presents those decisions organically as a part of play. The
point of the system is to track the timelines of PCs and NPCs so that any individual’s history is inspectable by the GM
so that the GM can have a full understanding of how things occurred within the game world. The idea is that it provides
tools to track the tedious parts of the game, separate from any particular representation of the game. It is system
agnostic, it still requires a GM to intervene and encode the effects of decisions that players make, but it tries to
tuck away all the die rolling to generate encounters, terrain, etc, and helps track the overall difficulty of the world
and accrue details on what players are doing as they do it.</p>
I think I’m going to switch to using Bevy with a Ratatui frontend. I still want this to be usable via CLI, I still want
it to integrate with Foundry, though I’m less concerned with the SVG output-based approach and more interested in the
already existing bevy_ratatui and bevy_ratatui_camera, which I think cover my needs nicely.</p>


20-NOV-2025
2025-11-20T00:00:00+00:00
1441</h2>
All of this does nothing in the face of the actual current problem which is rendering slowness, but it has been fun to
dive down the rabbit hole a bit.</p>


19-NOV-2025
2025-11-19T00:00:00+00:00
1220</h2>
I think I have a theory of how it should work.</p>
Each Pop and Locality is equipped with a Timeline. Timelines are sequences of a shared pool of Events which are
Simultaneous</em> across those timelines but not necessarily across others. When an event occurs, all those present for it
mark it in their timeline as an event.</p>
Each Locality is part of a Region, multiple Regions are part of a World, and a World has a special timeline that all
inhabitants of that World share. The World Timeline (“Worldline”) contains every event that happens in the world on a
shared reference clock; but other worlds can proceed at different paces to it. So for my setup I currently have 4
worlds:</p>

Eret</li>
Ynn</li>
Stygia</li>
The IM’s lair</li>
</ol>
Time moves differently in Ynn, but is synced with Eret in the other cases. Each Worldline advances according to GM Fiat;
generally, instead, timelines calculate where they should be on the other timeline</em> when travelling between worlds.
Take this simpler example:</p>
Ratio</th> World A</th> World B</th></tr></thead>

World A</td> 1</td> 2</td></tr>
World B</td> 0.5</td> 1</td></tr>
</tbody></table>
So if a Pop is in World A, then enters World B and stays for 1 hour on world B, then returns to World A, they will
return at T1 = T0 + 0.5h</code>, where T0</code> is the time of the previous event on world A. If they wait an hour on A, then return to B,
they will arrive at T1+2h</code>, or T0+2.5h</code>.</p>
So long as we know the sequence of events undertaken by a Pop, we can always calculate where we should arrive on the
target timeline.</p>
The goal is to just keep a consistent history of each Pop, the timeline of each world is just a function of the
relationship between two timeline’s clock-speed and the events which cause them to co-align. Events get tagged with the
Locality</code> they occur in, and the Worldlines are thus calculated. An event has a Start</code> time local to its worldline,
which will account for gaps.</p>
This creates a final structure like this:</p>

Universe

Eret

Upper Carpathia</li>
Ulm</li>
Edom</li>
Corinth</li>
</ul>
</li>
Ynn

3x Eret</li>
</ul>
</li>
Stygia

1x Eret</li>
</ul>
</li>
The IM’s Lair

1x Eret</li>
</ul>
</li>
</ul>
</li>
</ul>
With connections such that Eret is the Hub, and the others are spokes, which simplifies the matrix above.</p>
It is possible, likely even, that this will allow for paradoxes. That’s intentional. When a pardox occurs, the worlds
should split into new, independent worlds, with that Event representing the branching point. Any timeline can split.
This occurs naturally as people travel across different worlds.</p>
This model also allows for pseudorelativistic effects between worlds, since travel is ‘instant’ inasmuch as there is no
space to travel, but the two worlds run their clocks at ‘different rates’ so things change slightly as you move ‘further
away’. The relative clock speed on Mercury, for instance, is somewhat slower than an equivalent clock on Earth, losing
about 1 minute every 50 years or so, but varying considerably in the exact rate of exchange between an terran minute and
a mercurial one. This would model that as a fixed ratio (though in actuality it is not fixed, but based on the relative
positions of the various involved bodies. A future enhancement might turn the relativity matrix above into a dynamically
calculated function, but that’d require additional work to make the various ‘worlds’ more like physical objects.</p>


18-NOV-2025
2025-11-18T00:00:00+00:00
1050</h2>
EU V has been eating my whole life. Factorio too.</p>
I’ve also been thinking about this and what I want to do. Ultimately, and I recognize this is a function of having been
playing EU V pretty much nonstop since it came out, I think I want to push truncheon in the direction of having some
pop-based simulatation component. It’s not exactly EU, but it’s definitely adjacent.</p>
The idea would be to have a few classes of ’pop’s, which would be user definable with ‘promotion rules’ between them. So
for my D&D campaign I might have two general classes of pops, “Mundane” and “Sorted”, with no promotion between them
(generally), and within “Sorted” I might have a subclass for each level with promotion rules about how muse “Applied
Vesper” the Sorted has acquired. As they acquire enough, they promote to the next level.</p>
I can then make arbitrary rules for how those pops behave, move around the world, etc. Mundane pops would tend to
congregate with other Mundanes, give them logic to form villages and stuff; and have the sorted run around. Maybe
another class for Monsters, etc.</p>
Then I give them a big hex grid to hang around in and let them do their thing; I can still use it to track a party in
the same world, except now as they move, so does everything else.</p>
The world would have a bunch of Good</code>s as well, produced by the various pop classes by doing Job</code>s. Jobs produce goods
or alter a Locality</code>, of which a Region</code> may have many. Regions</code> are fields of Hex</code>es associated with a certain
Scale</code>. Regions exist within a World</code>, and are connected to other regions within a specific hex, and may be contained
within other regions. The connections are arbitrary scale.</p>
1105</h2>
Another way of approaching this that might be interesting. Hexes are nice for rendering, but representation as a fixed
field is a little limiting. In particular, it would be nice to just think in terms of topology, and then maybe
superimpose some hex system when it’s appropriate?</p>
1231</h2>
I roughed out some ideas on paper, but rapidly realized I’ll also need some sense of a ‘timeline’ for each party, as
different places within the worlds I manage may move at different rates.</p>


26-OCT-2025
2025-10-26T00:00:00+00:00
1244</h2>
I had an idea last night that I think squares everything up. Instead of pulling images in and rendering ‘on-demand’, the
renderer should instead run as an independent process that allows ‘subscribers’</p>
The pipeline would then be for an internal model to update, which the renderer can observe; the renderer creates a new
frame and sends a pointer to each subscriber, the UI is a subscriber. On application start, the model is booted, then
the renderer, then the UI ‘subscribes’ to the renderer and passes the imageref to whatever widgets it likes.</p>
Ideally it’s a queue, so the UI event loop runs, places the new image in the ‘right slot’, but only renders the actual
UI every 1/f</code> seconds (f = framerate), So if the renderer ends up generating 3 frames before the render call happens,
it should only copy the image for the last of those frames, the rest just being an Arc/other ref to the image data. This
also means the renderer can be ‘smart’ in the sense that I can create an SVG of the whole map, and dynamically grab
portions of it after rendering it once. I can also have the renderer manage multiple layers and other tweaks by making
the image reference a little smarter. This nets me the SVG-based approach I wanted (for transportability) and the speed
I’m looking for (since the map should change pretty rarely).</p>


20-OCT-2025
2025-10-20T00:00:00+00:00
1352</h2>
I don’t know that it fixed it, but it is slightly less laggy now. I need to hook up console or something to get data
out, but I think the issue is that svg</code> is fundamentally going to be slow because it’s just a long, unaccelerated
pipeline and ultimately it’s not even really necessary; I could just work directly on the image buffer.</p>
The upside of svg is that it is lightweight for a render-once-clientside application, which isn’t my intent; and it’s
nice to convert to other image formats and resolutions. The downside is that it is unlikely that I’ll ever be able to
render it quickly enough to ‘work’. I think next step is to start sticking probes around it and seeing if I can get some
timing data out, I suspect it will quickly show more blocking processes that need to be handled; and it will also
probably show that each of those blocking processes still take a long time to render. Low framerates for the thing are
tolerable, but dismal ones won’t be, so I might still go for a ‘render via ratatui-image’ approach, but just manage a
DynamicImage (or some wrapping structure) which I copy out when rendering each frame. Then I can have a process
continually buffer a new frame and just asynchronously ask for the current one each actual rendered frame. Should be
significantly simpler and avoids me having to write a template file.</p>
1425</h2>
I hooked up tokio_console and found some interesting info. In particular there are, as expected, quite a few blocking
functions. In particular there seem to be 4 copies of the render step running all the time, Only one task seems to ever
actually be running; which shouldn’t be the case, and because of this I seem to remain pretty far down in the queue. I
think tokio is basically shoving blocking tasks onto threads, those tasks take a while to return so it creates another
one w/o cancelling the first, and so it quickly exhausts all it’s available worker threads and spins waiting for all
it’s stuff to finish. Trying to feed people in half the time by cooking twice as much food.</p>
It seems that the problem ‘should’ be solved with messages, as I had been moving towards before. Sending a message can
be ignored if the render is already in progress, so you just have a little state machine on the far side and if it’s
rendering, it adds another return address to the queue.</p>
I want to benchmark the actual SVG rendering time for a reasonably complicated SVG, so I think I’m going to create a
separate map renderer that doesn’t do any of the ratatui stuff, then I can figure out a best-possible render time; if
that is acceptable, then I like the idea of trying to keep the SVG-based approach since being able to keep the actual
content of the game in pure xml/text seems nice from a longevity point of view. Easy to store and compress, easy to
build another interpreter for, etc. If it’s not, then using image</code> and writing a simple rasterizer (I think that’s the
right term, a thing that takes abstract model of space and turns it into pixels is what I mean, I’m not a graphics guy)
should be the path, since I’ll have all the control I can manage and can make the pipeline async-throughout, which would
hopefully resolve the issue as well. In any case I think it’s a good opportunity to use the state-machine</code> port.</p>


10-OCT-2025
2025-10-10T00:00:00+00:00
0033</h2>
I think I’ve found the actual issue with the svg rendered, it’s blocking. It needs to run in it’s own thread and the UI
talk to it, separate from tokio</code>, started independently and globally available. tokio</code> can run the UI and provide
requests to the renderer, updating as new frames are available. This may</em> be what spawn_blocking</code> does, I haven’t read
the docs enough to grok it yet. I grepped for it and it’s not in current use anyway, so worth looking through, an old
stackoverflow here</a>
also points toward it. My initial thought, however, was to either spawn a separate thread with an async channel, then
start tokio, obviating the #[tokio::main]</code> handle and thus DIYing it with this additional thread. The renderer side
would also have to directly manage the Future I think, that seems messy.</p>
I could also just try to implement Future for the svg rendering step itself, so that it’s callable in the async context
directly. I think the implementation would just be a spawn_blocking</code> call.</p>
I may need to grant some access to the tokio runtime to do this?</p>
https://tokio.rs/tokio/topics/bridging#spawning-things-on-a-runtime</p>
Seems relevant.</p>


26-AUG-2025
2025-08-26T00:00:00+00:00
1313</h2>
I think I need to focus on building a separate svg renderer that does a better job of separating the update/render
stuff. The SVG</code> widget should basically first-and-forget the SVG render, and the update should look it up after.
Probably the ‘simplest’ way to do that is to hash the SVG Template after compiling the template, and have the renderer
cache based on that hash. I’d push the template.render()</code> step up</em> to the update</code> call, then on the render</code> call
look up based on the current hash – so that I don’t clone the png data, I just retrieve an ARC to it from the
renderer.</p>
1313</h2>
I think I need to focus on building a separate svg renderer that does a better job of separating the update/render
stuff. The SVG</code> widget should basically first-and-forget the SVG render, and the update should look it up after.
Probably the ‘simplest’ way to do that is to hash the SVG Template after compiling the template, and have the renderer
cache based on that hash.</p>


25-AUG-2025
2025-08-25T00:00:00+00:00
1538</h2>
Apparently ratatui_image has something for separating rendering, might be able to use that as an example of how to
structure the SVG rendering thread. The current HEAD</code> of ratatui_image has a tokio example as well.</p>


21-AUG-2025
2025-08-21T00:00:00+00:00
0102</h2>
Still not super happy with the PNG rendering performance, but there is much to do to improve it, and it’s mostly
asynchronous now, all the major issues are worked out with the wiring, just needs a bunch of polish and tests.</p>
I dislike how I’m managing the layout on multiple levels, but it’s the most flexible thing for right now, eventually it
probably makes sense to have it have a fixed initial size so that the render loop can start without a blank screen. I’ve
already hardcoded it to use the kitty</code> protocol to avoid all the picker</code> nonsense that broke input for quite a while.</p>
0924</h2>
I decided to go to sleep mid thought when I realized what time it was.</p>
The way the layout works right now, it drops everything into a (fresh) hashmap on each call to build_layout</code>, which is
pretty frequent. This means that I’ve got strings littered everywhere. In this form, it’s quite easy to get hard crashes
due to uncatchable typos, which is not ideal. The obvious solution is to use an enum, but that adds some overhead to
re-arranging sections while I work on it, so I think I’m going to tolerate the pain until I’ve firmed up the UI design,
then encode it.</p>
Next step is to chase down the scaling issue, and then work to send the actual png generation as an async task, which I
think should make the UI stay responsive during rendering.</p>
1028</h2>
I think I need to create an entirely separate thread that communicates over a channel. I’ve been trying to get
tokio::task</code> to work, but I can’t seem to find the right invocation to get it to be non-blocking. Instead I could start
up the App</code> and UI</code> as separate threads that communicate, alternatively, a PNG renderer that runs in a separate
thread, maybe using a shared memory space? Not quite sure how I want to proceed.</p>
1145</h2>
https://stackoverflow.com/questions/61752896/how-to-create-a-dedicated-threadpool-for-cpu-intensive-work-in-tokio</p>
Seems relevant.</p>
I also need to do some kind of caching/don’t re-render if the template hasn’t changed. Probably a simple hash scheme’d
do it.</p>


18-AUG-2025
2025-08-18T00:00:00+00:00
0921</h2>
The process for rendering a hexfield is going to involve the following parts:</p>


Map from image dimensions (which I should look up on load and store in the cache) to nominal hex dimensions, scaling
everything to the same scale. This’ll require some math to determine the anchor points in the SVG for the image</p>
</li>

Figure out the viewbox/viewport/svg size stuff. This might be just ensuring I’m not working near the SVG origin
(since I think negative x/y values are not supported? Not sure. Might be able to solve this by making matching
adjustments to the viewport offset when I select subsections).
- Indeed, my example images have a big pile of surrounding transparency, the hexes are centered, though, so it
should be easy enough to find the center dynamically. which should allow placement relative to the center, it is
absolutely top-left anchored atm (by experimentation).
- This may also explain why the fill</code> wasn’t working, it was, we were just only seeing the transparent section.
I’ll need to experiment to see if adding back the defs</code> saves time once there are a lot of hexes being shown.
- an upshot might be that I could make a texture</em> to apply to the hex, instead of a hex image; contiguous
regions of hexes could pull from coordinated parts of the texture and get semi-random changes to add variety.</p>
</li>

Use existing hex math stuff to place all the hexes, this will probably need some more math to go from nominal hexes
in radial space to the image dimensions which will be weird.</p>
</li>
</ol>
Before any of that I have</em> to make this shit async because it is so annoying to wait for the thing. It’d also be good
to get it to automatically rerender every few seconds or whatever.</p>
1024</h2>
Looking at the async stuff, it may be that I need to separate the drawing side of the SVG</code> stuff and the display. The
widget can have an async ‘update’ function which stores the pngdata in the struct, the render just splats the pngdata to
the screen.</p>
1113</h2>
I started to tease apart the update/render parts of the svg so I can async it. The picker</code> part is a little tricky to
locate, and I’ve got a lot of state that needs managing, but I think I should be able to pass through the picker
information (perhaps as part of the state for SVG?).</p>
It feels like there should be a trait for ‘Async’ widgets (similar to stateful widgets), which are widgets with state
that endure a separate update loop. Each would get all the current layout information, so that, for instance, I can
pre-render the next SVG frame to the correct size, but one would get called to manage FPS and the other for actual state
upate?</p>
IDK, feels like there’s a pattern here, haven’t found it yet.</p>
I definitely think the layout should get cached, maybe I just don’t like immediate-mode UI? :) It’s just a set of rects
that get calculated, it’s another thing that I could stick in an async update loop and reference everywhere.</p>
One crazy idea here would be to refactor so the whole UI</em> is done as an SVG. At that point I’m just sort of streaming
graphics over the terminal protocol so it’s a little bit silly, but I suppose that’s probably what, e.g., canvas is
doing in a browser over HTTP.</p>
1319</h2>
I think I’m moving inexorably towards an event-loop style system with a bunch of asynchronous handlers with a
framerate-managed render loop. The current method does a loop like:</p>
1. Check for exit</span></span>
2. No exit, then handle events (sync)</span></span>
3. Update (async)</span></span>
4. synchronize</span></span>
5. Draw</span></span></code></pre>
What would be ideal is if the startup ran two systems, the frontend UI which just does the draw</code> step on a
fixed-framerate-target loop, and a ‘backend’ component which is asynchronous. The frontend would capture events and send
them to the backend, and request data from the backend when needed to render.</p>


17-AUG-2025
2025-08-17T00:00:00+00:00
0130</h2>
Good progress. I factored out some caches, though the image one isn’t working right just yet. I still need to review the
render function to make sure I got it wired up correctly.</p>
I need to get the UI async for this approach to work. I don’t need particularly high framerates for the map especially
(since it will be mostly static most of the time), and in fact a longer render time for the ‘static’ portion should make
the non-static portion easier to make it’s own, transparent layer that gets dropped on top. Pulling the caches out to be
async also just makes sense, they’re already tossing Arcs around with impugnity.</p>
The image cache is currently caching the whole ImageKind, but I think it should actually cache the tag and data
separately, this would allow me to reconstruct the imagekind at the last minute, avoiding an extra Arc and an unpleasant
.as_ref().clone()</code></p>
I’m still fighting to get it to fill the polyline hexagon with the image, but practically speaking I can also probably
just use image</code> tags instead, and layer the hexgrid on top of it.</p>
As it is, the process takes about 3s to render the svg as a png, I still can’t actually see the image that’s slowing it
down, but the cache works, I just eat a 3s render time.</p>
Hey, it’s down from the 53 minutes it took on the first run.</p>
It also appears that the use of defs</code> deeply slows the rendering down. So maybe just writing the image</code> directly makes
sense. This is inline with what I want to do with SVGTemplate</code> anyway, so I think that’s the way I’ll go.</p>
I did notice that when working w/ the webp images, it would frequently miscalculate the size of the grid it should
render. I need to have this derive not from the SVG dimensions, but set the offset based on input values from the
widget, as well as the visible space and ‘zoom’ level of the widget. Some transform exists from the ratatui-image</code>
widget’s Rect</code> and the SVG’s native bounding box based on some scalar zoom level which would map coordinates and
heights and stuff. I’ll need to figure that out and that should resolve the issue. It doesn’t seem to happen with .pngs,
I assume because they are more strict about their bounding box. webp is a mystery to me, all file formats are mysteries,
really.</p>
For now I’ll just use pngs.</p>
using images instead of the polyline does mean that I’ll need to figure out how to convert from where-ever the png</code> is
anchored to the center of the hex it contains. There may be a way to control where the drag point is in SVG, I don’t
know.</p>
1438</h2>
Definitely a render-box issue, I’ve added a hardcoded 100% rect to the template and it actually should have excluded
some stuff, it appears the width/height is controlled by parameters on the 
https://www.w3.org/TR/SVG/coords.html#TransformProperty</p>
Seems to be useful for this</p>


14-AUG-2025
2025-08-14T00:00:00+00:00
1510</h2>
I got the svg pipeline working, it seems pretty snappy but I’m also not drawing it most of the time. Next step is to
extend the template stuff to allow for drawing shapes and whatnot.</p>


23-JUL-2025
2025-07-23T00:00:00+00:00
1305</h2>
I’m thinking I might want to actually pull all the helm stuff into terraform in earendil</code> – right now I’m using k3s,
but I’d prefer the actual install to be agnostic with respect to k8s host. Instead the k3s install would be barebones,
and just focus on the necessaries to get k3s up and clustered.</p>
Terraform would drive helm, standing up the infrastructure and configuring, e.g., vault and such as well. This would
make it easier to bootstrap vault, as well, and would clarify earendil</code> as all the ‘service’ level automation, and the
rest of minas-tarwon</code> for the ‘infrastructure’ level automation.</p>
1405</h2>
Started to move things, need to stop and think through how to organize it. I could have a single helm chart with
subcharts, or move the dependencies up into terraform, not sure how I want to run it just yet.</p>


16-JUL-2025
2025-07-16T00:00:00+00:00
1205</h2>
Trying to use IngressRoute for routing, right now have some draft in earendil</code> that should move back to laurelin since
it’s the ingress for vault; which is laurelin managed, for the moment though it’s handy to keep all the k8s stuff in one
spot. I might wrap it in nix so I can import it via flake.</p>
For the moment I can just leave it in place to work out how that automation will be wired up</p>


13-JUL-2025
2025-07-13T00:00:00+00:00
1048</h2>
I split off earendil</code> for housing helm charts, it presently has nothing in it. I’ll store instructions on setup there
as well.</p>


12-JUL-2025
2025-07-12T00:00:00+00:00
1456</h2>
I set up a vault auto-chart, and will set up a vault-operator autochart, but I need to figure out how I’m storing all my
helm charts first, I don’t like where they live now (in telperion south of the emerald.city VM folder), since they
should be relatively agnostic.</p>
I could store them in laurelin, but they aren’t going to be nix files so they are a little out of place.</p>
I could store them in a new repo, and tbh that’s where I’m leaning, but it means if I need a secrets-repo for it I’ve
added another narya, which I’d like to keep to a minimum.</p>
I think they’re probably going to get split into their own repo, my plan is to largely have the k3s</code> module in laurelin
set up the initial state that the repo would expect, and then the repo will contain an ‘initialize’ script that runs
through all the bootstrapping, and the rest will just be ‘normal’ helm-files with a parent ‘install/upgrade’ script.</p>
Eventually I suspect I’ll build an operator for all my stuff, but today is not that day.</p>
An alternative would be to have some of the bootstrap script contained in the module itself, since I’m going to have it
minimally be able to set up:</p>

Vault</li>
Cert-manager</li>
NFS</li>
Longhorn</li>
</ol>
Then it could drop a bunch of systemd oneshots that depend on a parent job that does the initial install
semi-automatically. You’d deploy a single node in ‘server’ mode, log in, run the script via systemctl start emerald-city-k8s-install</code> or whatever, then it’d spit back the secret you need for all the other client machines.</p>
The other chart repo would then just contain install files for the services on top of the substrate.</p>
Right now I’m also using an ‘out of band’ docker registry, so it’d probably be good to add an option to run said
registry and automatically generate the relevant trust files; the true ideal would be to run it within k8s</em> and have it
loop back, though I suspect that might be not recommended, because cool things rarely are.</p>


9-JUL-2025
2025-07-09T00:00:00+00:00
0044</h2>
I’m making a distinction – most of what I want to run on the k3s cluster is going to be ‘ephemeral’, in the sense that
I don’t necessarily know what I want to run and am going to iterate on it reasonably quickly, and don’t want to have to
version it against the server’s</em> configuration. However, some items are required-on-first-run, e.g., setting up storage
drivers, GPU passthrough where it’s appropriate, etc.</p>
I’m writing my own charts mostly for the practice for all the ephemeral stuff, but for the more ‘structural’ stuff, that
will be encoded into the module itself.</p>
NFS and Longhorn both should get set up, nfs should be easy to get running with nancy which will unblock my foundry
chart progress.</p>


8-JUL-2025
2025-07-08T00:00:00+00:00
1045</h2>
Got ody-01 fixed, needed to fully delete the k3s stuff by following:
this</a> doc
to clear everything; this is likely down to the way I built the machines having a pile of crap left on them.</p>
I’m going to build a ‘mode’ option into the service that will add an option to set the service to ‘clean’ mode which
should rip out all the relevant stuff – or maybe just a script, idk.</p>
Next step is to get helm setup.</p>


7-JUL-2025
2025-07-07T00:00:00+00:00
2131</h2>
I’m planning to move minas-tarwon & co to dragon</code>, which should improve average build times. Archi is going to be an AV
machine after that, to help support some efforts for infinite-dungeon</code>, which has driven most of the recent changes
here and occupies a good portion of my time.</p>
I’m also working on setting up the k3s environment on ody</code>&co, the intent is that my various RPis and SFF machines will
be dragooned into helping with the workloads eventually, but building out the modules in laurelin</code> will be easier with
some VMs that are easy to blow away and reset.</p>


19-JUN-2025
2025-06-19T00:00:00+00:00
1814</h2>
Got affine</code> set up on barge</code>, began to look into reconfiguring my cloudflare tunnel to point into barge, probably
easiest to expose it on the host itself, and then configure it in nixos.</p>
I still want to migrate all this stuff to k8s, as well, but for the moment it’s just simpler to drop it all here. I need
to get foundry stood up there as well, which I suspect will be comparatively simpler.</p>
I’m going to extend the container definition to accept an arbitrary RP domain, since the incoming traffic over the
tunnel should only go to a few services. Would prefer k8s namespacing, but the plan is to turn the thing off/allowlist
IPs, so should be alright.</p>
Overall, things are proceeding apace.</p>


14-JUN-2025
2025-06-14T00:00:00+00:00
1609</h2>
I did some refactoring and ended up with 3 identical VMs for the k3s cluster, all running on BTG right now, but once
I’ve got it working I can migrate to multiple machines. I’m still using the current VM model, I plan to test the
build-a-vm-directly on DOP for ease, and then figure out how to share it everywhere after the fact.</p>
Next step is to stand up the k3s cluster and do any adjustments to horsepower needed. Then I can migrate services over
to be managed by some helm charts and glue.</p>
That should let me retire barge, and I should be able to retire daktylos as well.</p>


18-MAY-2025
2025-05-18T00:00:00+00:00
1140</h2>
I swear I didn’t plan to start this 23h59m after the last one, I just got distracted and managed to land on it.</p>
So – doing a bit of reading, I think I should be able to start by standing up 3 copies of odysseus</code> (mod a rename) and
get something wired together, from there I can just run an ‘all-leader’ cluster and probably do just fine. Eventually
I’ll spread those managers to toto</code>, maiasaura</code>, and the other small machines I have scattered, and just run workers
on BTG, DOP, and Nancy (if I decide to rebuild her as a nixos machine, anyway, I don’t plan to fight it if setting up a
k3s worker there is hard; maybe synology has a package for it but w/e). I can definitely use the iscsi/nfs storage
drivers at least, but it’d be cool to run some work directly on the nancy machine where it will be colocated w/ the data
it needs and could ostensibly use local-file storage drivers.</p>
One issue I need to work out not that I’m not going to netboot things is how to build whole a new VM quickly and ideally
without too much manual work. Right now the process is:</p>

Create a disk in BTG’s config</li>
Create a VM definition that mounts that disk, sets cpu/mem/etc params</li>
Attach the install ISO in that VM def as boot order 1</li>
Deploy BTG (sometimes provoking a network failure because of the weird bonding situation I have set up)</li>
SSH to the new VM on whatever address it grabbed from DHCP, install a base that includes my user, basic nix config,
etc.</li>
After install is done, down the machine, update the vm definition to not boot the install iso</li>
Update BTG again with the new def.</li>
Boot the new VM</li>
Apply it’s final config via the boot</code> option</li>
Reboot the machine to it’s final config.</li>
</ol>
This is quite tedious, and also relies on having the data stored right on disk on BTG, no option for offering disks from
a remote source (e.g., iSCSI). It also involves a deploy to either BTG (#4, 7) or the VM (#5, 9) a total of four times.</p>
I can solve some of this with UEFI bouncing, I think, I can build the whole image as a UEFI partitioned image and then
EFI bounce like I am with Barge. This costs at boot time, but only a little, and it cuts out several steps.</p>
Even better, I think nixos-generators</code> will basically do this for me.</p>
Next, if I can get iSCSI cooking, then I can build this into a remote-root/local-scratch setup, where there is a iSCSI
based root system that is fairly minimal, and then overlay a local disk (or another iSCSI) as an overlayfs which will
get subsequent updates to that machine. I think the procedure would then just be:</p>

Build a nix definition for a specific VM</li>
Build a UEFI-bootable image to an iSCSI target

In this mode, the image should not try to boot a second overlay drive from the next step</li>
</ul>
</li>
Create a second iSCSI target drive</li>
Build a libvirt definition that mounts both the #2 image and the #3 disk as an overlay</li>
Update BTG to run the new VM.</li>
</ol>
This still requires running an update against the whole hypervisor, but it’s a hell of a lot simpler than my current
methodology. I can start by building up a pinky</code> image with this model, hosting the iSCSI on Nancy as I already have
some test disks set up over there.</p>
Another option is to use nixosgenerators and drop the whole VM image as a qemu image. This may be much easier but does
mean rsyncing around the VM image.</p>
Practically it would also mean I’d want to probably tag images with some relevant sha of the repo they were built from,
not sure how I want to do that yet though. This could also still use the above EFI bounce approach, every vm definition
would just load rEFInd and point to the underlying thing. There is an option to install a bootloader as well, or
netboot, so worth exploring on pinky for sure I think.</p>


17-MAY-2025
2025-05-17T00:00:00+00:00
1141</h2>
It’s all fixed, basically.</p>
I switched to a proper SSD based install on BTG by swapping the Optical Drive for an SSD. I have a UEFI based install
there because initially I was planning to just EFI bounce via rEFInd</code>, but it turned out not to be necessary.</p>
Now I need to get DOP set up the same way, get the GPU installed there as well, and then get to work bringning up a k3s
cluster up and running.</p>
This also meant I removed the netbooting infrastructure from the VMs as well, which means I need to convert the other
services over to a proper install based system, or have everything like, boot to refind, which is what barge</code> is doing
right now.</p>
I think I’ll probably just start with three VMs on the same host with a single worker, that should be enough to get
everything set up, I can migrate services over, and then I can work on migrating to a more resilient distribution of
workers and leaders.</p>


30-APR-2025
2025-04-30T00:00:00+00:00
0051</h2>
I’ven’t made any progress on the above, and have justified what may otherwise be described as laziness as an impromptu
experiment in seeing how long it takes bitrot or force majeure to break something; turns out the latter won. Some work
on the powerlines down the street yeilded a brownout, which tripped my NAS into a strange state which required some
effort to recover. This left the netboot image mounts in a state of corruption, it appears, and my main VM host also
felt the sting, leading to a situation where all my VMs aren’t available.</p>
This was at least somewhat expected, partly because even though I have a rackmount UPS for the servers I haven’t
installed it because it requires also installing some electrical and I simply haven’t had the gumption. In any case, the
unfortunate situation is that the netboot apparatus to generate new</em> images is also</em> broken, so I need to either fix
what’s there or convert to something new, neither of which are super exciting, but at least the latter means I’ll have
some time to play with iSCSI, which appeals to me as a potentially interesting alternative to the NFS</code> route I had
planned. There is also tvix</code> to consider.</p>
At the moment, the state is that BTG and DOP both can’t boot up; but everything else is running alright. Rewriting the
netboot to avoid the image size issue will also make netboot-everything viable again, which will be nice for automating
VM creation, which in turn should make moving services to k8s much simpler.</p>
1238</h2>
What was originally a small issue has grown. In attempting to untangle things, I needed to get my laptop, Archimedes,
updated and working, however he is now crashing on update</em>, which is a new thing for me in Nix world. I think he might
just need a bit of cleaning, but I’m getting all my stuff committed and pushed before continuing.</p>
The fun never really stops.</p>


1-APR-2025
2025-04-01T00:00:00+00:00
0031 - spring-cleaning-1</h2>

Phase 1


Tag audit</li>

Complete ticket making from audit.</li>

Add additional tasks to P2 from audit (if any)</li>

Merge P1</li>
</ul>
</li>
Phase 2


Cache Refactor</li>

Variation Refactor to use Spell + Tape->Spell rename</li>

Better CI Pipeline</li>

Test Refactor</li>

Fastchess UCI test</li>

Benchmark harness</li>
</ul>
</li>
</ul>
I’m gonna make a pile of issues because I don’t really want to work on this stuff necessarily right now, but I knew
there were only a few logical issues in all the tags that I just kept running into, so I wanted to gather them up.</p>
I don’t love making issues, it feels uncomfortably organized and rational, and I strive to avoid any appearance of
rationality.</p>
It is a good opportunity to play with a new vim plugin though, so I suppose that will have to sustain me through my
discomfort.</p>
1003 - spring-cleaning-1</h2>

Phase 1


Tag audit</li>

Complete ticket making from audit.</li>

Add additional tasks to P2 from audit (if any)</li>

Merge P1</li>
</ul>
</li>
Phase 2


Cache Refactor</li>

Variation Refactor to use Spell + Tape->Spell rename</li>

Better CI Pipeline</li>

Test Refactor</li>

Fastchess UCI test</li>

Benchmark harness</li>
</ul>
</li>
</ul>
Just need to merge, then start a new branch. I didn’t see any tickets that need</em> to be done in P2. I need to go through
one last time and remove all the things I said I’d remove, and then I should be ready to merge.</p>
1025 - spring-cleaning-1</h2>

Phase 1


Tag audit</li>

Complete ticket making from audit.</li>

Add additional tasks to P2 from audit (if any)</li>

Merge P1</li>
</ul>
</li>
Phase 2


Cache Refactor</li>

Variation Refactor to use Spell + Tape->Spell rename</li>

Better CI Pipeline</li>

Test Refactor</li>

Fastchess UCI test</li>

Benchmark harness</li>
</ul>
</li>
</ul>
Phase 1 complete, now onto the real refactoring tasks. I’m going to start with Spell/Variation, as I think it’s probably
the longest pole of the bunch and will close two tickets if I’m efficient.</p>


31-MAR-2025
2025-03-31T00:00:00+00:00
0002 - spring-cleaning-1</h2>

Cache needs to be refactored to a generic key, ultimately this will be a LRU cache with a generic key type.</li>
Spell needs to have Familiar refactored to rely on a SpellState trait</li>
Tape needs to be renamed to Spell</li>

Move BEN</code> -> -basic

This means figuring out the to_fen_position</code> extension, which might be able to stay put? IDK. In theory
that means this can just be moved over and no api change, since query is already over there.</li>
</ul>
</li>
FIXMEs, TODOs, and the like need an audit.</li>

Reorganize directory structure</li>
Rename -core to -representation</li>
rename -basic to -core</li>

Get all the tests uncommented (in place) and passing.

With caveats, there are still some tests pending a refactor, but I think I got everything that could be easily
re-enabled.</li>
</ul>
</li>
nix run #ci</li>

-parser extraction</li>
</ol>
I think I might go ahead and merge this after a few final checks, and start spring-cleaning-2</code>, maybe after the
renames? In -core (right now), there is still:</p>
types/log  # Goes away with the Familiar/Variation rewrite.</span></span>
coup/rep # should be renamed to `coup` and be a top level module</span></span>
game/* # needs heavy refactoring to use the new familiar system</span></span>
interface/ # probably can just live in `game`</span></span></code></pre>
That should simplify it a lot and I think I’m done reorganizing after that.</p>
0052 - spring-cleaning-1</h2>
I went ahead and got fastchess</code> installed via a derivation in nix/fastchess.nix</code>. This’ll build on the first entry,
which sucks a bit, but maybe I can delay that later by hiding the fastchess</code> acceptance tests working behind a flake
app or something.</p>
Speaking of, the main purpose for fastchess</code> is going to be the UCI</code> compliance tool that just came out, that’ll help
a lot with getting the engine working correctly and saves me a ton of time trying to decipher what the semantics should
be, I can just focus on getting the suite to pass.</p>
That’s enough for one day though, the rest of the renaming tomorrow I think, then on to refactoring.</p>
1005 - spring-cleaning-1</h2>
        FAIL [   0.017s] hazel-test::zobrist zobrist::zobrist_update_is_idempotent</span></span>
──── STDOUT:             hazel-test::zobrist zobrist::zobrist_update_is_idempotent</span></span>
</span>
running 1 test</span></span>
test zobrist::zobrist_update_is_idempotent ... FAILED</span></span>
</span>
failures:</span></span>
</span>
failures:</span></span>
    zobrist::zobrist_update_is_idempotent</span></span>
</span>
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 8 filtered out; finished in 0.00s</span></span>
</span>
──── STDERR:             hazel-test::zobrist zobrist::zobrist_update_is_idempotent</span></span>
[tests/zobrist_test.rs:126:13] z1 = Z|0x0000000000000000|</span></span>
[tests/zobrist_test.rs:126:13] z2 = Z|0x0000000000000000|</span></span>
</span>
thread 'zobrist::zobrist_update_is_idempotent' panicked at /home/jfredett/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/quickcheck-1.0.3/src/tester.rs:165:28:</span></span>
[quickcheck] TEST FAILED. Arguments: (Assert <b Kk - 8 55390>, Assert <b Kk - 8 31393>)</span></span></code></pre>
This is an interesting failure, look at those metadata asserts resulting in a null. Very interesting. It is not
particularly common, I assume this is a natural collision, and I’m mostly curious about what components of the thing
it’s sensitive too. This test is pretty simple, it takes two random alterations and then calculates:</p>
z1</span> ==</span> z1</span> ^</span> z2</span> ^</span> z2</span></span></code></pre>
Verifying that the zobrist self-inverts, basically. I only log the values out, not the alterations, so I’ve changed
that, but since the test is flaky, it’s hard to reproduce, so I added additional logging and hopefully I’ll catch one
before too long.</p>
1514 - spring-cleaning-1</h2>

Cache needs to be refactored to a generic key, ultimately this will be a LRU cache with a generic key type.</li>
Spell needs to have Familiar refactored to rely on a SpellState trait</li>
Tape needs to be renamed to Spell</li>
FIXMEs, TODOs, and the like need an audit.</li>

Move BEN</code> -> -basic

This means figuring out the to_fen_position</code> extension, which might be able to stay put? IDK. In theory
that means this can just be moved over and no api change, since query is already over there.</li>
</ul>
</li>

Reorganize directory structure</li>

Rename -core to -representation</li>

rename -basic to -core</li>

Get all the tests uncommented (in place) and passing.

With caveats, there are still some tests pending a refactor, but I think I got everything that could be easily
re-enabled.</li>
</ul>
</li>

nix run #ci</li>

-parser extraction</li>
</ol>
I’m gonna rewrite my TODO list now in terms of phase 2.</p>

Phase 1


Tag audit</li>
</ul>
</li>
Phase 2


Cache Refactor</li>

Variation Refactor to use Spell + Tape->Spell rename</li>

Better CI Pipeline</li>

Test Refactor</li>

Fastchess UCI test</li>

Benchmark harness</li>
</ul>
</li>
</ul>
I’ll probably split some</em> of that out to a phase 3, but I don’t know what yet. The UCI test from fastchess</code> is a real
windfall, so I definitely want to get that going quickly, but it’s probably the most ‘phase-3’-y of the bunch. The list
is in rough order, but I rarely work in order.</p>


30-MAR-2025
2025-03-30T00:00:00+00:00
1514 - spring-cleaning-1</h2>
I pulled out the integration test, and right now I think I’m just going to focus on moving tests into their own modules
where it makes sense. Eventually most tests will not be co-located with the code-under-test, as I talked about before. I
haven’t brought in any new frameworks just yet, as I’m still determining what the best approach is going to be.</p>
Now that I’ve got everything else pulled out though, I think it’s time to return to extracting the parser</code> stuff, at
which point I think -core</code> is sufficiently gutted so as to be ready for the great renaming.</p>
I’m leaving the unused dependencies in place for the test crate for now, I suspect it’ll be worthwhile to reduce it at
some point.</p>
The only other item I need to actually add</em> to the list above is “10. move to a nix run #ci</code> like tabitha</code> has”, I
think that should greatly simplify my CI pipeline and make it easier to add in lints and git hooks and stuff later.</p>
So the current TODO list is:</p>

Cache needs to be refactored to a generic key, ultimately this will be a LRU cache with a generic key type.</li>
Spell needs to have Familiar refactored to rely on a SpellState trait</li>
Tape needs to be renamed to Spell</li>

Move BEN</code> -> -basic

This means figuring out the to_fen_position</code> extension, which might be able to stay put? IDK. In theory
that means this can just be moved over and no api change, since query is already over there.</li>
</ul>
</li>
FIXMEs, TODOs, and the like need an audit.</li>

Reorganize directory structure</li>
Rename -core to -representation</li>
rename -basic to -core</li>
Get all the tests uncommented (in place) and passing.</li>
nix run #ci</li>
</ol>
I think #9 is the next port of call, I can use it as an opportunity to move some tests around as well.</p>
1536 - spring-cleaning-1</h2>
One of the things I need to sit and think about is the way I convert between representations.</p>
I was reading a bit about the fearless-simd</code> project. It uses ZSTs as ‘tokens’ representing the capability</em> of SIMD
independent of the implementation</em> of SIMD, in a relatively ‘low cost’ sort of way to encode the intent</em> into the type
system and then use it to determine an optimal implementation using whatever SIMD primitives are locally available. I
think I’d like to try something similar at some point for my representations. A recurring frustration is the need to
pick a</em> representation as ‘canonical’ when no such representation really deserves that title. Bitboards are useful
sometimes, rank-file tuples others, strings others, it’s not natural to pick any one. There is no good type to give the
raw number or tuple-of-numbers that doesn’t include generally unnecessary bound checking. Bitboards are fine for what
they can do but quite limited in other ways, same with strings. Ideally I should be able to pick whatever representation
is ‘best’ for the current situation, leaving the others behind, but I don’t know what the ‘best’ rep is ahead of time.</p>
To square this (hah), I rely on impl Into<Square></code> frequently, but htis is also a representation with a downside. It
says, “I can become</em> a specific</em> known implemnetation, but you will have to pay at least the .into()</code> cost and maybe
a second to convert it to the ‘real’ type you want.” I can justfy paying something</em>, but I hate paying it twice.</p>
Relying on From/Into</code> also leads to a lot of weird situations where I can’t rely on type inference to ‘do the right
thing’ and guess the type correctly, leaving me to add annotations inconveniently.</p>
If I instead had something like Square</code> as a collection of ZSTs, each with constant</em> implementations of what are now
From/Into</code>s, I could have an API like:</p>
</span>
#[</span>quickcheck</span>] {</span></span>
    A1</span>.</span>bitboard</span>()</span>.</span>is_set</span>((</span>A1</span>.</span>rank</span>(),</span> A1</span>.</span>file</span>()))</span> ==</span> true</span></span>
}</span></span>
</span></code></pre>
Which isn’t too bad, and it means that the actual type I’m passing around is a finite type family, something like:</p>
// This presumes I can const-trait...</span></span>
trait</span> Square</span> {</span></span>
    const</span> fn</span> bitboard</span>()</span> -></span> Bitboard</span>;</span></span>
    const</span> fn</span> rank</span>()</span> -></span> usize</span>;</span></span>
    // ...</span></span>
</span>
}</span></span>
</span>
</span>
struct</span> A1</span>;</span> impl</span> Square</span> for</span> A1</span>;</span></span>
// ...</span></span></code></pre>
There aren’t too many representations to target, but I could do similar things with Piece</code>, Color</code>, etc where it was
needed. Essentially this is the same thing as how I tend to rely on Query</code>, but where Query</code> is really an open
specification with a potentially unbounded number of implementors, Square</code> is really</em> finite and basically will never
be extended.</p>
Something to chew on, anyway.</p>
1620 (nice) - spring-cleaning-1</h2>

Cache needs to be refactored to a generic key, ultimately this will be a LRU cache with a generic key type.</li>
Spell needs to have Familiar refactored to rely on a SpellState trait</li>
Tape needs to be renamed to Spell</li>

Move BEN</code> -> -basic

This means figuring out the to_fen_position</code> extension, which might be able to stay put? IDK. In theory
that means this can just be moved over and no api change, since query is already over there.</li>
</ul>
</li>
FIXMEs, TODOs, and the like need an audit.</li>

Reorganize directory structure</li>
Rename -core to -representation</li>
rename -basic to -core</li>

Get all the tests uncommented (in place) and passing.

With caveats, there are still some tests pending a refactor, but I think I got everything that could be easily
re-enabled.</li>
</ul>
</li>
nix run #ci</li>
-parser extraction</li>
</ol>


29-MAR-2025
2025-03-29T00:00:00+00:00
0730 - spring-cleaning-1</h2>
First things first, pretty sure I had dates wrong for previous log entries, so I adjusted.</p>
Second, I’m working on extracting spell</code>, but it’s more complicated because of the way I was managing specific state
update stuff, basically because I’m not using an interface for state objects. In particular I have a number of impls for
Familiar’s which fix a particular generic parameter, which I can’t do outside of the home crate; this is a good thing
ultimately, but will make extraction harder.</p>
For now, I’m just cranking away at what errors I can resolve, and then once I’ve reduced to the minimum, I’ll figure out
how to sort things.</p>
I also need to refactor the old Variation-specific familiar and really Variation itself to a Tape</code>/Spell</code> based
system, but I want to get the crate extracted and the name change performed first.</p>
1001 - spring-cleaning-1</h2>
Couple remaining open refactoring jobs now:</p>

Cache needs to be refactored to a generic key, ultimately this will be a LRU cache with a generic key type.</li>
Spell needs to have Familiar refactored to rely on a trait</li>
Tape needs to be renamed to Spell</li>
Move BEN</code> -> -basic

This means figuring out the to_fen_position</code> extension, which might be able to stay put? IDK. In theory
that means this can just be moved over and no api change, since query is already over there.</li>
</ul>
</li>
FIXMEs, TODOs, and the like need an audit.</li>
Reorganize directory structure</li>
Rename -core to -representation</li>
rename -basic to -core</li>
Get all the tests uncommented (in place) and passing.</li>
</ol>
Once all these are done (and I just want to audit tags for now, not necessarily do anything to resolve them unless it’s
very easy), I can get started on the test refactoring, which’ll be spring-cleaning-2, I think.</p>
1517 - spring-cleaning-1</h2>
Finished extracting spell</code> and moving ben</code> to -basic</code>. Dependencies are already minimized, and I can see the end
approaching. This ticks off #4 from above.</p>
1527 - spring-cleaning-1</h2>
Reorganized the crates a bit, deeper directory structure but cleaner lines. Finishes #6 above.</p>


28-MAR-2025
2025-03-28T00:00:00+00:00
1002 - spring-cleaning-1</h2>
Got -engine</code> pulled out, started pulling on -generator</code>, and found that it tangles a bit into the Cache/ATM</code> thing,
which can be extracted, but tangles (due to a shortcut) to the Zobrist</code> type, and so I’m left with the question of
‘where should Zobrist</code> go?’ Zobrist currently depends on both hazel_basic</code> types, which are easy to bring across, but
also Alter</code>, Query</code>, and the like, and the CastleRights</code> structure.</p>
I think -basic</code> is rapidly coinciding with -core</code>, and probably has a better name than either out there. I think there
are these approaches:</p>

Make Cache</code> key-agnostic, instead of doing any more reorganizing, just make it generic. Originally I had wanted the
key-building part to be transparent, but I don’t really do that anymore, so making it fully generic may be worth it.</li>
Move Zobrist</code> to -util</code> or it’s own crate, moving the various stuff it depends on in -core</code> to -basic</code>; this’d
mean interface</code>, PositionMetadata</code>, and CastleRights</code> move to -basic</code>, then -core</code> would pull in and set
HazelZobrist</code>, everything else would end up in the destination crate.</li>
</ol>
After all this is done, it leaves:</p>

Board Representation</li>
Move Representation</li>
Position Representation</li>
GameState Representation</li>
some more types that are definitely moving out (log/tape/etc -> spell, witch -> witch, movesheet -> devnull)</li>
</ol>
in -core</code>, at which point I’m forced to ask, ‘what is -core</code> really? -representation</code>’, then hazel-basic</code> ->
hazel-core</code>, and</p>
I also am starting to lean towards having a crates/</code> folder, something like:</p>
assets/</span></span>
crates/</span></span>
    hazel/              # Main library</span></span>
        bitboard/       # bitboard/pextboards</span></span>
        core/           # core types, Square, Color, Direction, Occupant; but also interface stuff (Alter et al)</span></span>
        engine/         # Actor that integrates evaluator and generator and other stuff to play chess</span></span>
        evaluator/      # Actor that evaluates a set of positions as efficiently as possible.</span></span>
        generator/      # Actor that generates all moves for positions as efficiently as possible.</span></span>
        parser/         # Parsers for various chess storage formats (PGN, SAN, etc)</span></span>
        representation/ # Representation types for Boards, GameState, Moves, etc.</span></span>
        ui/             # A debugging TUI designed to view the internals of the engine in operation</span></span>
        util/           # Various utility types.</span></span>
    spell/              # Generic 'spell' type for reading/writing/running familiars</span></span>
    witch/              # Generic actor framework</span></span>
    witchhazel/         # Integration point for engine + ui, configuration management, etc.</span></span>
doc/</span></span>
hooks/</span></span>
quals/</span></span>
tests/</span></span></code></pre>
I’m inclined to push more things down into the core, since I do think a lot of those won’t undergo too much movement
once settled. Alteration</code> is more or less ‘done’, and that is the most mobile of the bunch.</p>


27-MAR-2025
2025-03-27T00:00:00+00:00
1945 - spring-cleaning-1</h2>
I took a stab over on spring-cleaning-1-parser-twig</code> at pulling out the -parser</code> library, but ran into a number of
issues of where things should be sorted; once I got everything detangled, I was left with some parser bugs and a bad
taste; so I decided to rewind and try again.</p>
Along the way I found that the -engine</code> components are probably going to be simpler to extract first, and then the
parsers after. In particular it’s difficult to have these crates cross boundaries too much before you enter a circular
dependency hell.</p>
I’m going to keep working to split out mostly along module lines, if nothing else it’s telling me where all the tight
coupling is.</p>
Another thing I discovered is that the core representation (everything up to and including coup::rep::Move</code>) really
needs to ‘stick together’ – the first logical split point I think is probably at Position</code>. This is a place where
tabitha</a> would probably help, since I could see how different modules and structs
tangle with each other.</p>
I also found that my more or less arbitrary use of From/Into is resulting in some weird semantics. I think the right way
is to support converting into</em> primitive-ish types (the stuff in -basic), and then have other methods for converting
those things into specific subclasses of a given type; e.g., BEN</code> should have a as_fen_string</code> method; but not a
From<String></code> implementation. Whereas (Rank, File)</code> should have a From<Square></code> implementation.</p>
Most of the time I’m asking for anything squarelike</code>, not anything BENlike</code>; though impl Into<BEN></code> is not uncommon,
I prefer to lean on the trait</em> for that, via impl Query</code>.</p>
I’ll leave the twig up as it may be useful later, but I return to extracting other things now.</p>


25-MAR-2025
2025-03-25T00:00:00+00:00
0012 - atm</h2>
I’ve been babysitting a file upload most of the night, good time to kick on the playlist and hack.</p>
I got perft fixed up by finding a silly bug in how I reported length</code> from the tape. It’s a little ambiguous, but at
least it works.</p>
I can now fail perft_4 in ~35s, wall-time for the unoptimized test binary, anyway. That’s down from ‘well over 3
minutes’ by a factor of 6 or so, not bad for a HashMap</code> backed cache it only took me a full month to hack together. :)</p>
Next, spring cleaning. I’m going to live with the lack of tests because I’m going to move to rs-test, and start building
some fixtures for more efficient testing.</p>
1122 - spring-cleaning-1</h2>
Started spring cleaning, this is phase 1, which is just getting the workspace going and starting to split out crates and
cleanup dependencies. I’ve done the easiest split so far, into -core</code> and -ui</code> crates; and ran cargo-udeps</code> to clear
out unused dependencies. So far, I’ve brought a clean compile (measured by cargo clean ; time cargo build</code>) from ~37s
-> ~25s, about a 40% reduction.</p>
I’m going to continue splitting, but the overall workspace is going to look something like:</p>
hazel/</span></span>
    hazel-core/</span></span>
        src/</span></span>
            lib.rs</span></span>
            some-module/</span></span>
                mod.rs      # Quickcheck properties in an embedded module "properties"</span></span>
                test.rs     # Unit tests for the module, driven by rstest fixtures</span></span>
        test/</span></span>
            # core-specific integration tests, driven by rstest</span></span>
    hazel-ui/</span></span>
        # similar layout to the above, but backed also by `insta`</span></span>
    hazel-*/            # other crates as needed</span></span>
    hooks/              # Profile hook programs</span></span>
    quals/              # Qualification tests, comparing to other engines, establishing Elo, etc.</span></span>
    benchmarks/         # Performance tests/data gathering</span></span>
    tests/              # Integration tests</span></span></code></pre>
I’m debating following rust-gpu</code>’s example and putting the crates south of a crates/</code> folder, but for the moment I
don’t mind just having them toplevel.</p>
Part of the work so far moved over some ratatui stuff to the ui crate, which changed some apis around, but so far it’s
mostly been mechanical.</p>
1150 - spring-cleaning-1</h2>
I tried to extract the constants</code> module, but it tangles pretty tightly with other chunks of the system, and extracting
them will be a little tricky. It’s actually a little tricky to extract anything right now because it all
cross-references pretty freely. I think the way out of that is to push more of the code to rely on the interfaces in
src/interface</code>, and have that</em> become hazel-core</code>, and then the other hazel crates rely on the interface, and we have
a delightfully decoupled thing. Ultimately this will mean there is a hazel-representation</code> crate which implements the
interfaces, and then that should be replacable when I come up with better representations.</p>
He said, full of dumb hope.</p>
1727 - spring-cleaning-1</h2>
Okay, two more crates split out, hazel-bitboard</code> has all the pretty static bitboard related stuff, hazel-basic</code> has
all the piece/color/square/etc types.</p>
Next I think I’m going to start scaffolding the integration test crate, which can hold the last item in -core</code>’s
constants module. I’ll then turn that module into a re-export module maybe, idk.</p>
I want to split the parser stuff as well, as that should reduce more of -core</code>’s dependencies, but getting -basic</code> and
-bitboard</code> extracted will make that process easier I think.</p>
The end of all this should be a hazel::toolbox</code> module that I can import and get all the basic constants and types for
doing ‘normal’ stuff in hazel</code> the eventual engine</code> crate will have the bits for running the game using the
representation provided by that toolbox</code> module.</p>
Separating the parser should eliminate the nom</code> and similar deps. Separating the engine will probably involve
separating out the witch</code> thing to it’s own crate as well, and hopefully that’ll leave the types</code> module pretty thin
and maybe removable.</p>
So far so good, phase 2 will be rstest</code> and maybe some preliminary benchmarking work.</p>
2227 - spring-cleaning-1</h2>
I’m settling in on the split, I’m favoring lots of little crates, perhaps to the point of overkill, but at the moment I
prefer to overdo it and pare it back later, it has already paid dividends in terms of clarifying responsibility
barriers, and in service to that I’m going to make a little diagram.</p>
</p>
It’s in assets/hazel-crates.png</code> if that isn’t rendering for whatever reason.</p>


24-MAR-2025
2025-03-24T00:00:00+00:00
1406 - atm</h2>
I’m getting close to scrapping the current Position implementation. I think in order to move to the Familiar-based
approach, it may be sensible to take that route, since it removes some of the internal hash calculation stuff and
should</em> result in a smoother experience. Right now I end up with a lot of little logical errors everywhere because
multiple different codepaths are trying to keep track of a gamestate. I started extracting this (based on the
InnerPosition</code> struct), and I think this will make it easier to transfer state around (just send a quintessence and a
reference-by-hash to the relevant tape.</p>
The ergonomics are such that I may try blanking the relevant tests, merging as-is, and then doing the split. I’m going
to see how close I can get to working again, but I’m itching to do some sanding on this to get the rough spots down.</p>
1519 - atm</h2>
There is definitely an errant cache hit, and naturally the easiest thing to do is to create a UI widget to see the
internal state of the cache, but once again that returns me to the same problem of the UI side of things being pretty
messy.</p>
# In the `zobrist_is_same_for_transposition` test</span></span>
</span>
# ... before this, I construct a transposed version of the position. The final position is the same FEN, but the route</span></span>
# to get there is different</span></span>
</span>
2025-03-24T19:14:34.610727Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Making move: d2 (11) -> d4 (27) (QUIET) [0o026660]</span></span>
2025-03-24T19:14:34.610836Z DEBUG zobrist_is_same_for_transposition: hazel::game::chess::position: making: d2 (11) -> d4 (27) (QUIET) [0o026660]</span></span>
2025-03-24T19:14:34.610928Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Caching</span></span>
</span>
# Right here I get a cache hit but it's wrong, this has cached the position _after 1... d5_ as if it were the position</span></span>
# after 1. d4.</span></span>
2025-03-24T19:14:34.611145Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Cache hit Z|0x49B41FB4723326D3| -> "rnbqkbnr/pppppppp/8/8/3P4/8/PPP1PPPP/RNBQKBNR b KQkq d3 0 1"</span></span>
2025-03-24T19:14:34.611206Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Making move: g8 (62) -> f6 (45) (QUIET) [0o175320]</span></span>
2025-03-24T19:14:34.611249Z DEBUG zobrist_is_same_for_transposition: hazel::game::chess::position: making: g8 (62) -> f6 (45) (QUIET) [0o175320]</span></span>
2025-03-24T19:14:34.611292Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Caching</span></span>
2025-03-24T19:14:34.611397Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Cache hit Z|0x7AE74C1F751CA0F1| -> "rnbqkbnr/ppp1pppp/8/3p4/3P4/8/PPP1PPPP/RNBQKBNR w KQkq d6 0 2"</span></span>
2025-03-24T19:14:34.611442Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Making move: c1 (02) -> f4 (29) (QUIET) [0o004720]</span></span>
2025-03-24T19:14:34.611481Z DEBUG zobrist_is_same_for_transposition: hazel::game::chess::position: making: c1 (02) -> f4 (29) (QUIET) [0o004720]</span></span></code></pre>
Not yet sure why it’s calculating the same hash for that position, I suspect it’s probably some side-to-move malarky. I
should be able to drop the Turn</code> marker and instead use the fact that Assert/Inform wraps each move, that should mean I
can do my STM on/off through those.</p>
I definitely could benefit from a Cache-browser widget for this, but I think getting into the Actor-y stuff seems most
interesting right now anyway, so I’ll probably pack this up and merge it if I can’t fix this easily.</p>
1834 - atm</h2>
I tweaked to remove the update on Turn</code> and replaced it with full metadata updates on Inform/Assert</code>, as I mentioned
before. Doesn’t solve the immediate problem but does resolve one of the hash_familiar tests.</p>
1953 - atm</h2>
2025-03-24T23:51:55.072518Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Making move: d2 (11) -> d4 (27) (QUIET) [0o026660]</span></span>
2025-03-24T23:51:55.072750Z DEBUG zobrist_is_same_for_transposition: hazel::game::chess::position: making: d2 (11) -> d4 (27) (QUIET) [0o026660]</span></span>
2025-03-24T23:51:55.072892Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Caching</span></span>
2025-03-24T23:51:55.073106Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Cache miss</span></span>
2025-03-24T23:51:55.073196Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Applying: Turn</span></span>
2025-03-24T23:51:55.073243Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Applying: Assert <w KQkq - 0 1></span></span>
2025-03-24T23:51:55.073324Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position_metadata: My meta: w KQkq - 0 1, their meta: w KQkq - 0 1</span></span>
2025-03-24T23:51:55.073370Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Applying: Remove P @ d2</span></span>
2025-03-24T23:51:55.073430Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Applying: Place P @ d4</span></span>
2025-03-24T23:51:55.073472Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Applying: Inform <b KQkq d3 0 1></span></span>
2025-03-24T23:51:55.073547Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: updated inner</span></span>
2025-03-24T23:51:55.073650Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Cache set Z|0x175357BEF46D03ED| -> "rnbqkbnr/pppppppp/8/8/3P4/8/PPP1PPPP/RNBQKBNR b KQkq d3 0 1"</span></span>
2025-03-24T23:51:55.073721Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Tape is: 0x42356324111111110000000000000000000000000000000077777777a89bc98a:w KQkq - 0 1</span></span>
PositionZobrist { current: Z|0xB2390A8808563F88|, position: Z|0x175357BEF46D03ED| }</span></span>
RwLock { data: </span></span>
TAPE(head_hash: Z|0xB2390A8808563F88|, head: 0x27, hwm: 0x27)</span></span>
0x000000 | Clear | Z|0x0000000000000000|</span></span>
0x000001 | Place R @ a1 | Z|0xF8F96A2C09E4AF7E|</span></span>
0x000002 | Place N @ b1 | Z|0xEA58B07658FE5BEA|</span></span>
0x000003 | Place B @ c1 | Z|0x4929572CE96AC736|</span></span>
0x000004 | Place Q @ d1 | Z|0xD80F29D8B72E614D|</span></span>
0x000005 | Place K @ e1 | Z|0x407EFBE89EB626EA|</span></span>
0x000006 | Place B @ f1 | Z|0x82E8D8D87B2DF6B8|</span></span>
0x000007 | Place N @ g1 | Z|0x246E9B437DAE2756|</span></span>
0x000008 | Place R @ h1 | Z|0xDA8B17A14E032698|</span></span>
0x000009 | Place P @ a2 | Z|0xE13AC46130BCA406|</span></span>
0x00000A | Place P @ b2 | Z|0xC09F69247A322979|</span></span>
0x00000B | Place P @ c2 | Z|0x423E7386EE621B94|</span></span>
0x00000C | Place P @ d2 | Z|0x06F4427BB3BBFA4B|</span></span>
0x00000D | Place P @ e2 | Z|0xF7C662F78033FE6D|</span></span>
0x00000E | Place P @ f2 | Z|0x33FF1E11437B3BB5|</span></span>
0x00000F | Place P @ g2 | Z|0x4488A0019704DF24|</span></span>
0x000010 | Place P @ h2 | Z|0xF9569451BA88BDBE|</span></span>
0x000011 | Place p @ a7 | Z|0x560A2845D85A8564|</span></span>
0x000012 | Place p @ b7 | Z|0x174477C88AD15BCD|</span></span>
0x000013 | Place p @ c7 | Z|0x52788A0CCA3D5708|</span></span>
0x000014 | Place p @ d7 | Z|0x864346235FD05CED|</span></span>
0x000015 | Place p @ e7 | Z|0x44F8C4E79659D080|</span></span>
0x000016 | Place p @ f7 | Z|0x136DFFF91DDB872C|</span></span>
0x000017 | Place p @ g7 | Z|0x857658D3FD60A795|</span></span>
0x000018 | Place p @ h7 | Z|0x6E7BB19382E73A5A|</span></span>
0x000019 | Place r @ a8 | Z|0x18603AEEAAEC7478|</span></span>
0x00001A | Place n @ b8 | Z|0xA020A28F27D57B0A|</span></span>
0x00001B | Place b @ c8 | Z|0xBC72A17411A7839D|</span></span>
0x00001C | Place q @ d8 | Z|0x0FD353607E034BD1|</span></span>
0x00001D | Place k @ e8 | Z|0xF93226BEA0E8522A|</span></span>
0x00001E | Place b @ f8 | Z|0x5FD9F939C87C647E|</span></span>
0x00001F | Place n @ g8 | Z|0xB1484F2F5287B45C|</span></span>
0x000020 | Place r @ h8 | Z|0x816A59230F79B9AA|</span></span>
0x000021 | Inform <w KQkq - 0 1> | Z|0x175357BEF46D03ED|</span></span>
0x000022 | Turn | Z|0x175357BEF46D03ED|</span></span>
0x000023 | Assert <w KQkq - 0 1> | Z|0x816A59230F79B9AA|</span></span>
0x000024 | Remove P @ d2 | Z|0xC5A068DE52A05875|</span></span>
0x000025 | Place P @ d4 | Z|0x46776A46D78D19A7|</span></span>
0x000026 | Inform <b KQkq d3 0 1> | Z|0xB2390A8808563F88|</span></span>
END-OF-TAPE</span></span>
=================================</span></span>
, poisoned: false, .. }</span></span>
</span>
2025-03-24T23:51:55.074220Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Making move: d7 (51) -> d5 (35) (QUIET) [0o147060]</span></span>
2025-03-24T23:51:55.074287Z DEBUG zobrist_is_same_for_transposition: hazel::game::chess::position: making: d7 (51) -> d5 (35) (QUIET) [0o147060]</span></span>
2025-03-24T23:51:55.074354Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Caching</span></span>
2025-03-24T23:51:55.074452Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Cache miss</span></span>
2025-03-24T23:51:55.074500Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Applying: Turn</span></span>
2025-03-24T23:51:55.074530Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Applying: Assert <b KQkq d3 0 1></span></span>
2025-03-24T23:51:55.074572Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position_metadata: My meta: b KQkq - 0 1, their meta: b KQkq d3 0 1</span></span>
2025-03-24T23:51:55.074642Z ERROR zobrist_is_same_for_transposition: hazel::game::chess::position_metadata: Incorrect metadata, Found: b KQkq d3 0 1, expected b KQkq - 0 1</span></span>
2025-03-24T23:51:55.074723Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Applying: Remove p @ d7</span></span>
2025-03-24T23:51:55.074760Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Applying: Place p @ d5</span></span>
2025-03-24T23:51:55.074793Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Applying: Inform <w KQkq d6 0 2></span></span>
2025-03-24T23:51:55.074837Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: updated inner</span></span>
</span>
### Note the hash, 0xB239... this hash is associated with the move after 1... d5.</span></span>
2025-03-24T23:51:55.074911Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Cache set Z|0xB2390A8808563F88| -> "rnbqkbnr/ppp1pppp/8/3p4/3P4/8/PPP1PPPP/RNBQKBNR w KQkq d6 0 2" </span></span>
</span>
</span>
2025-03-24T23:51:55.074985Z TRACE zobrist_is_same_for_transposition: hazel::game::chess::position: Tape is: 0x42356324111111110000000000000000000000000000000077777777a89bc98a:w KQkq - 0 1</span></span>
PositionZobrist { current: Z|0x75AB520EB7FCDCCB|, position: Z|0xB2390A8808563F88| }</span></span>
RwLock { data: </span></span>
TAPE(head_hash: Z|0x75AB520EB7FCDCCB|, head: 0x2c, hwm: 0x2c)</span></span>
0x000000 | Clear | Z|0x0000000000000000|</span></span>
0x000001 | Place R @ a1 | Z|0xF8F96A2C09E4AF7E|</span></span>
0x000002 | Place N @ b1 | Z|0xEA58B07658FE5BEA|</span></span>
0x000003 | Place B @ c1 | Z|0x4929572CE96AC736|</span></span>
0x000004 | Place Q @ d1 | Z|0xD80F29D8B72E614D|</span></span>
0x000005 | Place K @ e1 | Z|0x407EFBE89EB626EA|</span></span>
0x000006 | Place B @ f1 | Z|0x82E8D8D87B2DF6B8|</span></span>
0x000007 | Place N @ g1 | Z|0x246E9B437DAE2756|</span></span>
0x000008 | Place R @ h1 | Z|0xDA8B17A14E032698|</span></span>
0x000009 | Place P @ a2 | Z|0xE13AC46130BCA406|</span></span>
0x00000A | Place P @ b2 | Z|0xC09F69247A322979|</span></span>
0x00000B | Place P @ c2 | Z|0x423E7386EE621B94|</span></span>
0x00000C | Place P @ d2 | Z|0x06F4427BB3BBFA4B|</span></span>
0x00000D | Place P @ e2 | Z|0xF7C662F78033FE6D|</span></span>
0x00000E | Place P @ f2 | Z|0x33FF1E11437B3BB5|</span></span>
0x00000F | Place P @ g2 | Z|0x4488A0019704DF24|</span></span>
0x000010 | Place P @ h2 | Z|0xF9569451BA88BDBE|</span></span>
0x000011 | Place p @ a7 | Z|0x560A2845D85A8564|</span></span>
0x000012 | Place p @ b7 | Z|0x174477C88AD15BCD|</span></span>
0x000013 | Place p @ c7 | Z|0x52788A0CCA3D5708|</span></span>
0x000014 | Place p @ d7 | Z|0x864346235FD05CED|</span></span>
0x000015 | Place p @ e7 | Z|0x44F8C4E79659D080|</span></span>
0x000016 | Place p @ f7 | Z|0x136DFFF91DDB872C|</span></span>
0x000017 | Place p @ g7 | Z|0x857658D3FD60A795|</span></span>
0x000018 | Place p @ h7 | Z|0x6E7BB19382E73A5A|</span></span>
0x000019 | Place r @ a8 | Z|0x18603AEEAAEC7478|</span></span>
0x00001A | Place n @ b8 | Z|0xA020A28F27D57B0A|</span></span>
0x00001B | Place b @ c8 | Z|0xBC72A17411A7839D|</span></span>
0x00001C | Place q @ d8 | Z|0x0FD353607E034BD1|</span></span>
0x00001D | Place k @ e8 | Z|0xF93226BEA0E8522A|</span></span>
0x00001E | Place b @ f8 | Z|0x5FD9F939C87C647E|</span></span>
0x00001F | Place n @ g8 | Z|0xB1484F2F5287B45C|</span></span>
0x000020 | Place r @ h8 | Z|0x816A59230F79B9AA|</span></span>
0x000021 | Inform <w KQkq - 0 1> | Z|0x175357BEF46D03ED|</span></span>
0x000022 | Turn | Z|0x175357BEF46D03ED|</span></span>
0x000023 | Assert <w KQkq - 0 1> | Z|0x816A59230F79B9AA|</span></span>
0x000024 | Remove P @ d2 | Z|0xC5A068DE52A05875|</span></span>
0x000025 | Place P @ d4 | Z|0x46776A46D78D19A7|</span></span>
0x000026 | Inform <b KQkq d3 0 1> | Z|0xB2390A8808563F88|</span></span>
</span>
### Note the hash here corresponds to the above, but is incorrect -- it should be further down, on 0x00002B</span></span>
</span>
0x000027 | Turn | Z|0xB2390A8808563F88|</span></span>
0x000028 | Assert <b KQkq d3 0 1> | Z|0x46776A46D78D19A7|</span></span>
0x000029 | Remove p @ d7 | Z|0x924CA66942601242|</span></span>
0x00002A | Place p @ d5 | Z|0xDF027ACAEE79DFDA|</span></span>
0x00002B | Inform <w KQkq d6 0 2> | Z|0x75AB520EB7FCDCCB|</span></span>
END-OF-TAPE</span></span>
=================================</span></span>
, poisoned: false, .. }</span></span></code></pre>
That wall of text shows the bug in real time. We can see that the Zobrist it’s using is based on the start of the turn,
not the end of it. So instead of caching the position associated after applying</em> the move, we associate the updated
position</em> with the hash prior to applying</em> the move, which is an off-by-one, essentially. I could fix this by
reintroducing the end</code> command, or by instead ensuring I’m updating the position hash at the Inform</code> step. In reverse,
this will still work, since it will be examining alter.inverse()</code>, which for Assert</code> is Inform</code>, and thus on the
reverse it will still do the right thing cache-wise.</p>
I think I’m going to try to remove Turn</code>, fewer tokens is better, since ultimately I want to encode this into a
bytestream.</p>
2300 - atm</h2>
Had to put things down, but came back, changed the line, and that was it, that was the whole bug. Still 3 broken tests
remaining but very happy to not have to chase that one anymore.</p>


23-MAR-2025
2025-03-23T00:00:00+00:00
1317 - atm</h2>
I don’t think there’s going to be an easy way to get the debug frontend set up until I overhaul the UI heavily to be
more async-first. Bear with me while I talk it out.</p>
Basically, in order to handle the input, I need to include some sort of ‘focus’ mechanism so I can still scroll through
the tape, but also enter values, etc. In order to do this I need to:</p>

Track which subsection is chosen, easy – I already have a modal enum I can just add another enum to tell me which
part of the UI I’ve got selected.</li>
Harder, I need to then hand off event handling to that subsection</li>
Harder still, I need to do this without interrupting the engine.</li>
</ol>
This is doable if I move to a message-passing/multi-process setup, because then I can just have each subcomponent have a
channel for event handling, etc. Getting there is it’s own little project entirely though.</p>
So on that front, I think I’m going to try to do something a little crazy.</p>
I should be able to fix unmake by basically saying, “Recacluate the boardstate from the beginning of the tape each
time.” I don’t think I’m at any risk of overwriting the end of the tape, and I can always make it bigger in the short
term, but I think that might get all the calculations right at the expense of not actually caching out efficiently?</p>
Eventually I want to store the running-hash alongside the alteration in the tape, and I think that model will work a bit
better anyway, most of my widget code is written at the tape level and not the position level, as well, and I think that
might be wrong.</p>
I think I’m going to spend a few cycles trying to fix the code as is, if I can get it right, then I should be able to
get the tests passing and back to working on the engine, I can follow through with my above plan for either the UI or
the more general plan I set up on 14-MAR.</p>


22-MAR-2025
2025-03-22T00:00:00+00:00
1227 - atm</h2>
Setting up the input section, and I’m finding the layouts result in the Input-line is off the end of the screen, I’m not
quite sure why as I’m using percentages for almost everything, might just be something with padding? not sure.</p>
Getting input wired up probably also comes with getting better event handling and widget-selection, which leads me
further from the point of this branch, but I suppose I’m just committed to the UI stuff now. I’d rather not merge with
broken tests.</p>
I did put a little work into the perft stuff, but I’m getting a bug where it unwinds too much and ends up running off
the end of the tape. I think fixing that should get perft working back up to perft3.</p>
I have no</em> idea how I’m going to get the perft-in-the-ui</code> working, I think I’ll have to build a specific message +
handler that sends position updates back to the UI as it runs, then the UI can render it and the perft can continue,
it’ll be quite slow, but this isn’t intended for performance, but debugging. Something like a:</p>
</span>
struct</span> Perft</span> {</span></span>
    initial</span>:</span> BEN</span>,</span></span>
    depth</span>:</span> usize</span>,</span></span>
    reply_to</span>:</span> TxChannel</span></span>
}</span></span>
</span>
impl</span> Perft</span> {</span></span>
    pub fn</span> new</span>(</span>initial</span>:</span> impl</span> Into</span><</span>BEN</span>>)</span> -></span> (</span>Box</span><</span>Self</span>>,</span> RxChannel</span>) {</span></span>
        // returns both the message to send and the callback channel</span></span>
        // UI then hands the callback channel to the UI widget, which </span></span>
        // waits asynchronously for a message and updates when it recieves one.</span></span>
    }</span></span>
}</span></span>
</span></code></pre>
I’m a little concerned that this approach is going to rapidly devolve into a “Simply rebuild the entire UI to be fully
async in all parts from the get-go”, at which point I may abandon this till I can at least get the tests passing again
and then take a better stab at refactoring. Before I do that level of work on the UI I want to at least (in no
particular order yet):</p>

Split up the crates</li>
move to rs-test</li>
Unify the variation stuff under the new familiar</code> system and remove the old implementation.</li>
</ol>
Once that’s done, I can move to a fully asynchronous UI which ‘observes’ the engine state over (ideally) a channel that
is otherwise ignorant of the internals, this should</em> allow the UI to work as I upgrade the engine code</em> via something
like a hot-reload.</p>
My intent is to first support all of UCI, but quickly this will mean a custom protocol for communicating internal engine
state to the UI asynchronously. Ultimately I want the UI to basically hold a finite buffer of updates and try to just
‘keep up’ with the engine, discarding states it can’t get too as it does so.</p>
Input will be collected, categorized, and then notifications sent to various subcomponents of the UI.</p>


19-MAR-2025
2025-03-19T00:00:00+00:00
1600 - atm</h2>
I really need to split this thing into multiple crates, if only so I can more easily skip all the UI tests which, owing
to their ‘compare to a text screenshot’ approach, are pretty brittle when I start working on the UI.</p>
I’m back to working through some of the failed tests, going to hopefully get it back down to just perft failures before
continuing with the UI debugging, ideally to minimize the number of moving bugs when I’m trying to squash the zobrist
caching part of movegen, which is what this whole thing was supposed to be about in the first place.</p>


18-MAR-2025
2025-03-18T00:00:00+00:00
2343 - atm</h2>
The end-of-tape-nullmove bug does introduce a secondary bug, where it overwrites the A1 square with a knight that does
not unwind when you run back over it. I think I might just clamp the end of the tape so you can’t scroll past the end.
The way the tapereader work sucks in a lot of ways, but it is pretty cool to watch the pieces wiggle around.</p>
The UI is quite messy, and I need to wire up some more controls so I can make better use of the log widget, and also
send messages into the system, then I should be able to get perft working in the UI.</p>


14-MAR-2025
2025-03-14T00:00:00+00:00
1056 - atm</h2>
I got it working, mostly. The Table</code> widget is not working as I’d expect, and in particular it seems to have very
little correlation in the number of rows I fetch to the number of rows it renders. I think I may DIY this, I’m not
entirely sure what I’m doing wrong, so building it myself might be an improvement. Also an opportunity to work out the
remaining bugs with the thing. Mostly UX issues, a couple of weird UI bugs as well, and at least one O(n)</code>-growing
repeated call problem when you scroll past the end of the tape.</p>
Separately, I want to integrate a board representation and other diagnostic info before getting to the actual point,
which is debugging perft</code> and the game representation.</p>
I had to compromise a lot on the Tapelike</code> design, mostly due to the way Tape</code> don’t have central
allocation/management. Ultimately Tape</code> should reside in some stable memory-space that I can easily refer to across the
application. I’d even like if they were CoW and the ‘TapeDeck</code>’[1]</p>
[1] Name pending, I went with Tape</code> for the tame now, but it doesn’t fit the rest of the naming scheme w/ familiars and
stuff. Tape</code> is a very Computer-Sciency name for it, I want something more like Computer Witchcraft, so this might get
renamed.</p>
1538 - atm</h2>
Continuing: Barring some naming stuff, once I have a situation where I’m storing Tape</code>s in some broader allocation, I
can share reference to subsections of Tape</code> that way. Eventually I think the result will be some storage entity a la
Cache</code> that holds all the tapes and dynamically grabs the right ‘section’ of tape (identified by Zobrist of said tape)
and dynamically creates a Tape</code> with that content as needed. This will also allow the UI a stable reference to a
section of tape.</p>
Ideally this thing is actually just a big Vec</code> of larger contiguous chunks of memory, maybe adding some kind of Jump</code>
alteration to skip around to different subsections within the Tape</code> (so long as the destination had a reversible
Landing</code> instruction, a la COME FROM</code> in Intercal, of all things).</p>
I think getting a board/metadata rep connected is the next step, and then I can rig up an IO section for controlling the
engine via uci commands. Once I can do that I should be able to get it hooked up to perft</code> output.</p>
I will need to extend this to a nested reader at some point, connecting a variation-tape to a position-tape and warping
everything around appropriately.</p>
I also really need to overhaul both how communication is handled via Witch</code> (it’s presently handled in a way to mimic a
STDIO/UCI style thing, and I’d prefer to have most internal messages be a direct-response-over-a-oneshot-channel, which
will avoid all sorts of race conditions if multiple clients are reading the engine simultaneously), and also refactor
the UI to be a fully asynchronous update/event/draw loop. Ideally the drawing would be independent of the other two and
support a high-sample-rate of the underlying state, so that the engine is not limited by the UI update speed, which I
suspect will be the case in the current model.</p>
Last, I need to think about what columns make sense in the TapeReader, and also how to render alterations. I know for
sure I need to add a Inform/Assert(EnPassant(None))</code> at least for now to make the forward/reverse simple, but I suspect
in the longer term I’ll be able to elide much of the metadata alterations.</p>
For now, though, the tapereader basically works (mod bugs), and things are starting to look like how I want them.</p>
2352 - atm</h2>
Just documenting some bugs w/ the tapereaderwidget:</p>

On first load, shows nothing, this is probably due to how it’s initialized, but the unloaded state should be better
represented.</li>
After loading by pressing ‘down’, it loads and sits on 0x00001, can’t scroll up to 0x00000.

I can scroll to 0x00020 later, without issue, so it’s just the 0th entry, not the first entry of a page</li>
</ul>
</li>
0x00049 is the actual end of tape, 0x0004A shows a null-place (Place N @ a1</code> maps to 0</code> in all fields), so I’m
getting a null that’s interpreted as an alteration even though there isn’t an alteration there (or shouldn’t be,
anyway).</li>
After scrolling past the end of the tape, it seems to have an O(n) increasing number of calls to the page_range
function, which is an issue only if I allow scrolling past the final page/scrolling arbitrarily far forward.</li>
</ol>
Next tasks:</p>

Hook up and provide a gamestate representation attached to the current position. Board, FEN, at least.</li>
Input/output section routing to the engine.

UCI messages should get rebuilt into WitchHazel</code> messages, ideally with a oneshot return channel, but they also
need to obey the STDIO stuff as well which is going to use the #write</code> method.</li>
Internal messages are more free, and the main ones I’m looking for are probably going to be something like
‘load-from-file’ and ‘perft’ to start. That’ll make it possible to build the stockfish integration test I
described… way back in November…</li>
</ul>
</li>
Make things look nicer.</li>
</ol>


11-MAR-2025
2025-03-11T00:00:00+00:00
0042 - atm</h2>
I’m finding it quite difficult to live with my, perhaps ill-adviced/anxiety-informed choice to try to avoid Arc</code>/Rc</code>
for the Cursor</code>/Familiar</code> system, but I think I have met my match with the way things are going with the tape widget.</p>
After a brief diversion to get logging exposed in the UI, I now have an interactive way to debug the damn tape widget.</p>
I finally want to get it hooked up to the actual engine backend using the Witch</code> actor system, but I’m running into a
borrowing issue since I’m now in the multithreaded environment where it gets tricky to pass the data around to be used
by both the engine and the various UI components. Since I want the UI to be able to expose parts of the engine directly
for debugging, I don’t want to rely on copying state across or otherwise processing it, I really want a reference as
close as possible to the ‘real’ thing.</p>
I’ve been putting off using any kind of thread-safe stuff because I couldn’t imagine what joint it would make sense to
cut at in terms of thread safety. Too coarse and the system won’t scale, too fine and the system won’t perform. Polya
says if you have a hard problem you can’t solve, find a simpler one you can’t solve; thus everything else up till now.</p>
I have already established one boundary, Hazel</code> is the engine state struct, with Witch</code> surrounding it and managing
message passing. Hazel ultimately will contain a Variation</code>, which encodes a multi-variation game structure as a flat
file[1]. A Familiar</code> on that structure will calculate some chosen Position</code> based on the contents of the Variation</code>
at the chosen point. Similarly, Position</code> compiles Moves</code> and writes them to a Tape</code>[2]. This tape also uses a
Familiar</code> to calculate the current boardstate, potentially jumping to specific states via Zobrist Cache</code> hits.
PositionFamiliar</code>s, in particular, use Bitboard</code>, the Query</code> trait, and ultimately use the Alter</code> trait and it’s
companion Alteration</code> enum to specify incremental, reversible updates. Since Cursor</code>s (and therefore Familiar</code>s) can
also read ranges of the underlying tape, eventually these Familiars can optimize across multiple Alteration</code>s to
minimize computation (e.g., the PawnStructure</code> familiar might ignore all the non-pawn related alterations, and further
use simd</code> to apply multiple alterations at once where possible). All of this is provided via the hazel</code> libary.</p>
main.rs</code> is then just whatever is necessary to set up a tokio</code> environment and run either the UI or just the UCI
subsystem. The UCI</code> subsytem uses the UCIMessage</code> struct (which sucks) to send UCI messages to the engine via
WitchHazel</code>’s MessageFor</code> trait implementors. Eventually each message will be it’s own type and can contain whatever
logic is needed to move hazel into the implied state in their various impls. Most will be ZSTs, but where natural types
like BEN</code> and such can be used. This will also make it easy to add custom commands to Hazel</code>, which I need for the
eventual database-y features I intend to add.</p>
The Familiar</code>s in particular need to be self-contained little guys that ideally can be quickly transferred to any
thread or client Hazel</code> engine. They rely on the Cursor</code> object, and ultimately I think that’s the joint where I need
to start cutting. Cursor</code> presently relies on a simple borrowed reference to talk about it’s target Tapelike</code> object,
but it should probably use an Arc</code>.</p>
Additionally, I only have familiars that don’t mutate the underlying variation, I don’t know that I’ll run into a need
to have concurrent writes just</em> yet, and it’s another joint in any case. I think</em> there is some combination of Arc</code>
and RwLock</code> that might make sense for cursor, but I need to think on it.</p>
I think this should also clean up the types a bit and avoid all the lifetime math I’ve been doing, since I can just
clone an Arc</code> around without the same anxiety that cloning a big Tape</code> gives me.</p>
Once all</em> of that refactoring is done, I will finally</em> be able to finish the TapeReader</code> widget, since it will rely on
a TapeFamiliar</code>[3] instead of this crazy reference lifetime rigamarole I’ve been trying to massage into shape.</p>
The remaining problems to solve I see are:</p>

Getting the UCI Implementation working (as always).</li>
Some mechanism for managing a familiars and states

In particular, the TapeReader</code> can only show it’s own position, but I will want to be able to have the result of
other familiars operating on the tape simultaneously shown on a single widget.</li>
I also want to be able to independently manage a particular familiar (as in, the API wrapping a particular
state-object with a cursor onto some other object) with the actual cursor that ‘powers’ it. In this way I can better
batch different needs of the engine. If multiple lines all want to calculate through some particular position, I can
prioritize unblocking those threads dynamically. This is the plan, anyway.</li>
It will also</em> enable the UI to have a Menagerie</code> of familiars it can then loan the state of to each of the
widgets.</li>
</ul>
</li>
Hook up the Board</code> widget to a syncronized Familiar</code> with TapeReader</code>, and we should get the engine’s idea of the
current selected boardstate.</li>
Write some command that runs perft</code> in the engine so we can watch it blow up.</li>
…</li>
Well, I can’t say profit, but it’ll be cool to watch the pieces move around. Very blinkenlights</code>, very cool.</li>
</ol>
Anyway – the point of this is that’s it</em>, that’s the whole system I have conceived up to this point. I took time to
work on tabitha</code> trying to figure out that, and though I haven’t finished her yet, she ended up not being totally
necessary. There are a few missing types I didn’t mention in there (Square</code> is probably the only notable omission, but
it’s not really a type, just a notational convenience in the engine), but it covers the design as is. I am pretty close,
I think, to getting that working and getting to at least step 1 (or 2) with a little more refactoring and a finished
cache implementation, which should come quickly when I finish the debugging tool intended to fix it.</p>
Let’s hope I don’t go walkabout first.</p>

[1] Variation presently uses an early iteration of the Tapelike</code> interface and all that, it needs to be unified to this
model and also to use the Tape</code> structure, enhanced with some write-to-disk functionality added.</p>
[2] The same cache-to-disk functionality can be applied to any particular Tape</code>, but with a target of an in-memory
cache.</p>
[3] type TapeFamiliar<S> = forall T : Tapelike, Familiar<T, S></code> – or whatever rust has for that, I think.</p>
1033 - atm</h2>
Started working on the refactor, I decided to just go ahead an do an Arc<RwLock<T>></code> on Cursor</code>, but not provide a
write API for now. I think ultimately I’ll want to put the write API on the cursor itself, but for now it mostly aids in
sharing the already RwLock’d tape inside position. I am also going to Arc</code> that tape</code> instance in Position</code>, which
makes creating the cursor very easy (cursor just clones the tape</code>, but the tape is owned by Position</code>).</p>
1046 - atm</h2>
I’m also ripping out all the hash calculation on tape</code>, I’m going to manage this via familiars still, but I want them
to exist outside tape itself, which should be agnostic with respect to it’s contents.</p>
The only place where I’m a little hesitant is the head_hash</code>, which is used for tape-level cache-out, but I think the
cacheout can just calculate this at cache-out time from scratch (no incremental maintenance), since the array is small
this works out to a pretty small number of operations done rarely, which I think is probably fine.</p>
2049 - atm</h2>
I’ve found that, in fact, what I’m trying to do is impossible, and needs some additional state. In particular, I’m
reinventing the owning_ref</code> crate. Ultimately, I want to only allocate the underlying Tape</code>’s memory once</em>, living in
an (eventually not 'static</code> Cache</code> shared amongst threads and owned by the parent thread behind a tokio wrapper so
that it is accessible across networked clients as well).</p>
I’m also starting to get frustrated with the compile time cycle. I think there are a few things to look into. First is
this</a> and it’s followup to
investigate. I’m already on nightly with all the stripes painted on the side, so I don’t mind being experimental. I
think mold</code> was set up at least when I was still using the other shell framework, but I’ll have to spend a cycle or two
refactoring. I also suspect moving to a workspace and separating out some of the crates might mean I can cut some
dependencies out and speed things up.</p>
Add it to the bucketlist.</p>
2119 - atm</h2>
Okay, I think I’ve ended up on a rabbit trail. I’m also in a bit of a pickle with respect to the repo, I need to commit
off some of the UI work, and then I think revert a bit and take a fresh stab at the Familiar/Cursor/Tape stuff.</p>
In particular, this arose while trying to get the TapeReader</code> widget working, which was exhibiting extremely strange
bugs, that ultimately were down to the scaffolded way I had wired it in re-building the widget every draw</code> call.</p>
The idea was to connect it to the engine, which exposed issues with tokio</code> and how the UI connects to the backend, and
thus all this other work.</p>
I’m going to get the commits done, then start over on this setup. I think the rough heirarchy still makes sense, but I
need to push some of the concurrency around and that requires a redesign.</p>


9-MAR-2025
2025-03-09T00:00:00+00:00
0034 - atm</h2>
Working a bit more, I’ve got the tape widget kind of working, but the Table</code> widget has some slightly strange behavior
that I can’t quite</em> say is it’s fault, but I don’t see where the issue is on my side.</p>
It’s an easy problem to fix, though, DIY.</p>
I am starting to think that I should build my actual Widgets to be the payload of a Familiar, using a similar
update-override, I could have a Widgety</code> trait that adds an advance/retreat logic to update it’s internal state, and
then a parent widget can just manage advancing/retreating whatever widget needs said advancing/retreating.</p>


6-MAR-2025
2025-03-06T00:00:00+00:00
1354 - atm</h2>
Noticed a little bug, if you place two places to the same square in a row, it will happily ‘place’ the piece in the
square, screw up the zobrist, and then move along. This is happening in the current (46da817) SHA, though I honestly
don’t know why</em> it’s making multiple place</code> commands there, it is. Maybe because it’s initializing to zeroes and
trying to read them as None</code>s? IDK. Might be worthwhile to add an Alteration::Noop</code> in the mix and get rid of the
Option</code> type entirely in Tape</code>.</p>
Next steps are to continue refactoring, I realized that by adding some additional methods to the familiar, I can get the
best of both worlds, update the state via the simple Alter</code> code, then when the UI needs to read context data, it talks
to the familiar, which can specialize the methods according to it’s current position/state. This means that Familiar</code>
will proxy some of Cursorlike</code> and Tapelike</code> to the end user, but with the added context of the familiar’s state,
which I think should be plenty for what I need. I am going to aim to build up the TapeReaderState</code> with the new
familiar system and see how that fares.</p>
1936 - atm</h2>
I was thinking a bit about Zobrist</code> while organizing code, and I think I want to tweak the design. I’m planning to use
Zobrist</code> for much more than just position hashing, since I can certainly use them for Tape</code> caching at least, and
perhaps for other things as well. I also expect I’ll want to use them primarily in the context of a Cache, so I want to
tie those things together and also ensure the type system prevents using the wrong calculated hash, since they all look
pretty much the same. I’d also like to be able to tweak the seed per cache, since the behavior is quite sensitive to
what it is you’re hashing.</p>
I’m thinking of taking a phantom type, at the very least, on both Cache</code> and Zobrist</code>. Cache will accept lookups only
with matching type. I can also use that phantom type information to report back to the debug UI and log as telemetry
about the hashes, which I can then use to start analyzing hash quality, which I think will be important. I think this
will be a good way to implement things when I get to actually implementing a non-stub version of Cache</code>.</p>
Unrelated, I need to put some kind of prelude module together, there is a set of common things (Square constants,
macros, some types) that I simply need all the time, and more generally it’d be good to have some sort of toolbox module
that just reexports all the stuff I need. I have been idling on splitting up the thing into multiple crates, LSP is
getting quite slow and I think it’s potentially due to project structure – it’s also just getting a bit messy, and some
parts of the system are quite stable and unlikely to change until I get into the optimization loop later.</p>
1953 - atm</h2>
I’m procrastinating, but I thought of a potentially interesting game/tool for teaching chess to children. My kid is
almost four, and loves watching me play chess, despite constant protestation about not playing illegal moves. I’ve been
thinking about how I will teach the game, and when it comes to teaching, my style is somewhat idiosyncratic. I dislike
direct instruction and generic or trite metaphors. Rarely do these appeal to the actual individual learning in front of
you, and they always seem to focus on creating a superficial understanding that does not really drive nor reward
discovering better abstractions later.</p>
Take, for example, chess puzzles. I love them now, but hated them at first because they didn’t make sense – I didn’t
understand what they were aiming to reinforce or teach, because I was hung up on the idea that I was simply supposed to
play ‘well’ and not ‘accurately’, puzzles reward accurate identification of tactical sequences and execution of those
sequences (“combination”). By directing a student to a puzzle which has the full rules of chess enforced, probably with
many pieces to consider, and then to expect them to just ‘caclulate’ leads to frustration. This is mitigated either by
replacing the puzzle with simpler ones, which is demoralizing for many students (myself included); or by starting with
trivial puzzles (which don’t really help develop the muscles for real games, which is the typical goal of students
(myself not included)) and progressively increasing difficulty, which delays the value of the puzzle and makes it hard
to understand how it’s helping; or by carefully walking them through the puzzle and emphasizing calculation and patience
and how to look for tactics and so on. This is probably our best try, but it still leaves the game badly unbalanced, the
amount of work</em> it takes to drag a student through the weight of all the possibilities is highly inefficient.</p>
The problem is not the student nor the teacher, it’s the rules. There are two many of them to easily isolate tactics in
a narrow space that makes calculation easy. Most small-piececount puzzles in chess are either trivial mates/forks, or
pawn-and-piece endgames which are renowned for being calculation intense, as the branching factor shrinks, allowing a
great deal of depth, paradoxically increasing</em> the number of positions that must be considered.</p>
The rules are also against us because the best</em> tactical positions are also the narrowest</em>. These offer the most
excitement, snatching victory from the jaws of utter defeat. A mate-or-be-mated position is thrilling</em>, and excitement
cements memory like nothing else. So we want to reduce the resistance and ensure early and frequent narrow wins. We, as
teachers, can then slowly increase the difficulty and lower the win rate, and ultimately develop the mental muscles of
our student.</p>
So what if we isolate tactics into smaller rulesets built around simpler games? Let’s also try to make the game slightly
asymmetric, such that the student’s role is easier, we will rely on the skill differential to help balance the game.
We’ll also use an asymmetric clock, so that the stronger player has less time.</p>
The rules of the game are simple, the stronger player takes a king and piece of their choice (not a pawn). The student
takes a piece assigned to them by the stronger player. The stronger player is given a clock, the student may be given a
clock, depending on the stage of training.</p>
Student places their piece on the board.</p>
The teacher goes second and places both of their pieces on the board.</p>
The student must fork the teacher’s pieces. As soon as this happens the game ends and the student wins. If the student’s
piece is captured, they lose. When the student wins, the teacher gains an increment on their clock and the game resets.
When the teacher wins, they lose an increment. Play until you’re done.</p>
The point of the game is just to practice recognizing fork configurations of these pieces on instinct. There is no
attempt to teach deeper chess concepts, it’s just lifting weights.</p>
I don’t know if it’s a good game, but I might make hazel play it someday.</p>


5-MAR-2025
2025-03-05T00:00:00+00:00
0058 - atm</h2>
I’m working on the UI a bit to help get the perft bugs sorted out. I am running into an unfortunate design decision
which leads to some unfortunate type stuff.</p>
Ideally I want a tape to be a static allocation, and have Alterations</code> be encoded to some relatively small
representation (maybe a u16). Encoding this in the type will make it (I hope) easy to tweak later to appease the memory
gods. It’s not premature optimization, it’s optimizating prior to maturity.</p>
In any case, using a const generic seems natural, except that my Familiar</code> type needs to borrow a reference to it, but
that means it needs to name the specific</em> type, not just one of any size, though the reference will be the same for all
of them.</p>
I’m not sure how to fix that, logically I think it’s sound to drop the knowledge of the size of the tape here, but I
can’t think of a way to do that in the type system (mostly due to ignorance, I think it should be possible).</p>
I suppose I could push the generic into the method, which would possible make things easier, I think the easiest way to
address it would be dynamic allocation, but that would set up a bit of a can of worms of how configuration will work. I
still want it to be a single static allocation so I think that’s probably the easiest thing.</p>
1022 - atm</h2>
I found dynamic-array</code>, which I’m going to use for now, I think at some point I’ll probably want to implement it by
hand, but for now I just want a dynamic array I don’t have to think about.</p>
1435 - atm</h2>
dynamic-array</code> was drop in, which was nice, but I am straining against how I’m managing these Familiar</code>s and I think I
need to just bite the bullet and do the refactor. In concept it’s straightforward, it’s a common API for working with
linear data on some kind of tape. Everything is a file</code> extended into Hazel.</p>
// Calculates a state based on the content of some tapelike. Importantly, the `cursor` should be _replacable_, so that</span></span>
// if a familiar runs off the end of a tape, and we have a continuation for that tape in cache, we can replace it's</span></span>
// cursor with a new one on the new tape and maintain the state. These should ultimately be sendable between threads, so</span></span>
// all their state is maintained internally in a thread-safe way.</span></span>
struct</span> Familiar</span><</span>T</span>,</span> S</span>> {</span></span>
    cursor</span>:</span> Cursor</span><</span>T</span>>,</span></span>
    state</span>:</span> S</span></span>
}</span></span>
</span>
// A zipper-type over the Tapelike. Hides locking details from familiar, so that familiar can RW safely.</span></span>
// Eventually will tie into some kind of transaction tool to coordinate concurrent writes</span></span>
struct</span> Cursor</span><</span>T</span>,</span> E</span>></span> where</span> T</span> :</span> Tapelike</span><</span>E</span>> {</span></span>
    ref</span>:</span> Arc</span><</span>RwLock</span><</span>T</span>>>,</span></span>
    position</span>:</span> usize</span></span>
}</span></span>
</span>
// Covers all the IO operations on the tape, without an explicit read/write head being maintained.</span></span>
trait</span> Tapelike</span><</span>T</span>> {</span></span>
    // locking ranges is internal, ideally takes place transactionally (eventually)</span></span>
    fn</span> cursor</span>(</span>&</span>self</span>)</span> -></span> Cursor</span><</span>Self</span>,</span> T</span>>;</span></span>
    fn</span> length</span>(</span>&</span>self</span>)</span> -></span> usize</span>;</span></span>
    fn</span> read_address</span>(</span>&</span>self</span>,</span> address</span>:</span> usize</span>)</span> -></span> Option</span><</span>&</span>T</span>>;</span></span>
    fn</span> read_range</span>(</span>&</span>self</span>,</span> range</span>:</span> Range</span><</span>usize</span>>)</span> -></span> Option</span><</span>&</span>[</span>T</span>]>;</span></span>
    fn</span> write_address</span>(</span>&</span>mut</span> self</span>,</span> address</span>:</span> usize</span>,</span> data</span>: &</span>T</span>);</span></span>
    fn</span> write_range</span>(</span>&</span>mut</span> self</span>,</span> start</span>:</span> usize</span>,</span> data</span>: &</span>[</span>T</span>]);</span></span>
}</span></span>
</span>
fn</span> conjure</span><</span>S</span>,</span> E</span>>(</span>tape</span>: &</span>T</span>)</span> -></span> Familiar</span><</span>E</span>,</span> S</span>></span> where</span> T</span> :</span> Tapelike</span><</span>E</span>>,</span> S</span> :</span> Default</span> {</span></span>
    Familiar</span> {</span></span>
        cursor</span>:</span> tape</span>.</span>cursor</span>(),</span></span>
        state</span>:</span> S</span>::</span>default</span>()</span></span>
    }</span></span>
}</span></span></code></pre>
this is my rough sketch, I’m not sure I’ve got all the concurrency stuff right there but that’s what compiler errors are
for.</p>
1532 - atm</h2>
Extending the above, the Cursor</code> object would be able to access the whole Tapelike</code> API, which makes me think maybe I
should use an associated constant for the entry type, similar to Iterator, which is kind of what I’m aping here.</p>


4-MAR-2025
2025-03-04T00:00:00+00:00
0937 - atm</h2>
I’m back to the perft bugs with the new Tape</code> implementation. It’s not perfect and needs a ton of sanding, but it’s a
much better and more complete start than I had.</p>
I think the next step is actually going to be wiring this into the UI, so I can manually walk around and watch the perft
happen and try to catch the Tape misbehaving ‘live’. The current bug is definitely just an unmake</code> issue, but hard to
guess from simple trace output, a TapeWidget</code> would be helpful, as well as a better way to manage all the Familiars</code>
surrounding the Tape.</p>
I’m also not correctly processing metadata at the moment, the Inform</code> blocks don’t change the metadata.</p>
So basically, it still doesn’t work, but it doesn’t doesn’t work as doesn’tly as it did didn’t before.</p>
I’m going to try to do a little fix-in-place so I can maybe merge this branch before doing more UI stuff, as that’s
going to require building up the communication side of things, and movegen working would help a lot there.</p>


28-FEB-2025
2025-02-28T00:00:00+00:00
1836 - atm</h2>
I’m starting to think about Log</code>, Tape</code>, Cursor</code> and Familiar</code> and I think I’m honing in on the API I want.</p>
I’m essentially dealing with two APIs.</p>

A ‘thing what moves through a file’ API, i.e Cursor/Familiar, both of which have file-like APIs to non-file things.</li>
A ‘thing what stores variants of some enum’, i.e., Log or Tape.</li>
</ol>
A familiar in particular accrues state as it goes, while a cursor is a ‘stateless’ familiar.</p>
Realistically I don’t care about the #2 thing as much as the #1. #2 is just for storing data and should be ‘pretty dumb’
most of the time, however, I do need to store it somewhere so that I can run my familiar over it to accrue whatever
state it likes. Ultimately a familiar is a type:</p>
struct</span> Familiar</span><'</span>a</span>,</span> F</span>,</span> E</span>,</span> S</span>></span> where</span> E</span> :</span> Invertible</span> {</span></span>
    state</span>:</span> S</span>,</span></span>
    source</span>: &</span>'</span>a F</span><</span>E</span>>,</span></span>
    update</span>:</span> fn</span>(</span>&</span>mut</span> S</span>,</span> E</span>)</span></span>
}</span></span></code></pre>
As the familiar is advanced or retreated, the update function is called with the E</code> (entry) item or it’s inverse if
retreating. The F</code> here is some arbitrary container, I don’t think I can actually specify a type function as a type
parameter but practically that’s a nonissue since this is almost always either Vec</code> (as in Log</code> or a finite array (in
the case of Tape</code>). This parent structure is responsible for updating/writing things to the log, the familiar is
responsible for interpreting it’s contents and producing whatever items we like on demand.</p>
I think I want to extend this a bit, since familiars may be ‘left behind’ over time, they are responsible for
incrementally updating themselves. When a cache-out happens (for Tape</code>), their states should also be caught up, cached
off, and put in the ‘initial’ state.</p>
These familiars are initially going to be useful for calculating various zobrist hashes of things for which we can
zobrist. In particular, the Tape</code> backend is made of Alteration</code>s, which ultimately it what implements the zobrist
hashing, so a familiar which is maintaining a hash as I edit the storage makes sense.</p>
Similarly, it makes sense to keep track of the hash at the position of the write head. Since we can just have a Familiar
copy that when the need arises.</p>
I’m just tossing a function pointer in the above, but I suspect that should be some kind of trait. My thought was this
is not really intended to be called outside the context of the familiar, and it’s possible I’ll want to build these
dynamically, especially since these generic familiars will be able to interact with the existing Variation</code> structure
and the Position</code> structure as well.</p>


28-FEB-2025
2025-02-28T00:00:00+00:00
Some days, all you can manage is an update to a log.</p>
Dropped those AI subscriptions, I find that qwen</code> on the local P40 is more than enough for my relatively limited LLM
use. Not having Copilot</code> adjacent autocomplete has been a little rough on the hands but ultimately isn’t really costing
time.</p>
I think my next project for the editor is probably to pare down the number of plugins, set up more autocomand/language
specific configuration, and continue to improve the ergonomics a bit. I’d like to get all my keybindings centralized and
documented as well, as right now they’re pretty ad hoc from years of accrual.</p>
For the lab, I think the next projects include geting the k3s ‘production’ cluster up and running, and starting to
helmify all my various service-level stuff. That gets me a good balance, more ‘permanent’ services can live as full vms
in NixOS, then I can take my smaller services and use either OTS docker containers or DIY nix-based containers. K8s can
handle distributing across multiple machines.</p>
Ideally I want to build in autoscaling/migration, so that my projects are run in a container on my ‘main’ development
machine w/ remote storage backing them. In the evening when I finish working, I want those containers to migrate
(without losing any internal state) to a single, low power machine, so that I can shut off all my other machines. Then
in the morning I can have them automatically scale up a bit before I need them.</p>
The idea is that – during peak hours of the day I want to have the full horsepower available, but once the family is
asleep I don’t need most of the services running at all. I should be able to turn off a good chunk of the infrastructure
so that I’m only using it when I’m generating the maximum from my solar roof. Combined with a off-grid battery system
and I think it should be possible to get all the horsepower I want out of the 730s as compile-farm machines, while more
or less not using any grid-power for them.</p>
Dare to dream.</p>
Anyway, it’s a rough day here, hopefully it’s better for you.</p>


27-FEB-2025
2025-02-27T00:00:00+00:00
0017 - atm</h2>
My initial idea was to build something like the Cursor</code> struct with ATM</code>, and have a sort of type-agnostic cache, that
I could then have each method use. The cache would be transparent to the caller, you ask for the board, you get it, if
it’s cached you pull it from there, if it’s not you build it right then and cache it. More like memoization/transparent
caching.</p>
That’s not a great way to do this, with make</code> and unmake</code> in particular, I have a lot of opportunity for incremental
updates and minimizing halfply-to-halfply work, so it makes sense to do opportunistic caching during this process, and
manage when cache is updated more directly.</p>
Right now that cache is a HashMap, but there are smarter structures for this, and memory will be a concern quickly if I
keep all positions.</p>
So I’ve opted for a much simpler global cache, centralizing all the cache logic in Position</code> for now. Ideally the
eventual model will be something like:</p>


Position</code> provides an API to talk about the position, it builds it’s methods out of methods provided by
PositionInner</code>. It provides the pleasant API for implementing, e.g., movegen.</p>
</li>

PositionInner</code> provides the bitboard caclulation and has more memoization-adjacent caching via RwLocked bitboards,
I’ll probably build some custom wrapper around bitboard for that, not sure. Ideally most of the math gets pushed back
here, and this can be a site for optimization of board querys.</p>
</li>
</ul>
The PositionInner</code> should ultimately ideally be Copy</code>, just a big pile of bitboards and the like.</p>
I currently have the Alteration cached there as well, but I think I’d prefer that to be first-class on Position, since
most of my incremental opportunities involve tooling around a Log of Alterations.</p>
To wit, another thing I’ve been sorely missing has been a good structure for a collection of alterations, and I realized
as I was writing this that I have the perfect thing already built, that little Log</code> structure backing Variation</code> would
work perfectly here, and it’s transaction</code> feature (which is moreso named for it’s aspiration than it’s actuality) may
see some use.</p>
Finally, I’m pretty sure ChessGame</code> is going to get factored away and replaced with Position</code>, and I think that will
net out to a much nicer flow than is currently there.</p>
2354 - atm</h2>
make/unmake is such a drag, but I think the issue may be on the side of the cached_alterations</code> (and more general
alteration system). I’m currently just using a Vec</code>, but it makes sense to have something closer to Log</code>, except I
don’t want all the transaction stuff. Log initially was intended for branching trees of moves, this is something a bit
simpler, and probably finite/ring-buffer-ish.</p>
I also sort of want to change how metadata is tracked, right now I have to Assert</code> the whole metadata into the
alteration stream (though technically it shouldn’t be needed), ideally I would only flag when events happen that alter
the metadata state; that would also make it easier to unwind (since I just update the PositionMetadata</code> incrementally),
but it unifies the whole system. This also makes zobrist-ing these guys pretty easy (and very amenable to SIMD
calculation of the zobrist later).</p>
Ultimately what I want to do when doing make/unmake</code> is something like:</p>
make(mov):</span></span>
    push the move onto the stack</span></span>
    compile the move to alterations</span></span>
    apply each alteration to the zobrist of this position.</span></span>
    check cache:</span></span>
        if hit, load the contents</span></span>
        if miss,</span></span>
            use current contents and incrementally update the position,</span></span>
            populate the cache</span></span>
</span>
unmake:</span></span>
    unwind alterations until and including the previous START_TURN</span></span>
        apply each alteration to update the zobrist of the position</span></span>
    pop the move off the list</span></span>
    check cache for this position</span></span>
        if hit, copy contents.</span></span>
        if miss, populate the cache</span></span></code></pre>
In both cases, the Move</code> is more of a convenience representation for the less compact but much simpler Alteration</code>,
which specifies how to update specific squares, and how metadata updates. In the current model I just dump all 4 bytes
of PositionMetadata</code> into the stream, but I’d prefer if things were a bit more incremental. Turns are already counted,
but I need flags for at least:</p>

Losing Castle Rights (4 bits)</li>
Which is the EP file (if any, it’s 3 bits for the file, absence can indicate no EP)</li>
50 move rule reset (1 bit)</li>
</ol>
We can calculate movecount and side-to-move from the pre-existing StartTurn</code> variant.</p>
Eventually I want to be able to encode this to a bytestream, so it’s handy to keep aligned to bytes. For now they can
just be full variants, but eventually this will be represented as a bytestream that can be stored and gamestates
recovered without any need to rule checking or any more advanced understanding of chess.</p>
Ideally the make/unmake</code> will eventually only store partial alteration tapes, especially if they’re finite, when we hit
some threshold we can take the current zobrist (even in the middle of a partially-applied turn) and clear our buffer and
continue. Essentially allowing us to zobrist</code> whole alteration buffers in and out of cache.</p>
The structure I’m using now (a Vec<Alteration></code>) doesn’t have the ability to seek around, and ultimately I need to seek
through and do something on each alteration, so the whole thing is much more turing-machine than stack machine. The
latter of which is how I’ve been considering it so far, but it’s inconvenient since it’s not a lot of stack bookkeeping
and manually updating state.</p>
The Log</code> datastructure is what I want, but I want to tweak how it works. I’m going to calculate two Zobrist</code> values –
one is the current actual position up to a StartTurn</code> marker, this is the ‘real’ zobrist of the legal chess position.
The other is a zobrist of the sum of all instructions in the buffer. When the buffer is filled, this zobrist is used to
cache the state and then clear the cache, that hash is also stored in the tape as the ‘previous’ state, so unwinding is
always possible. This hash can be computed incrementally along with the other hash. Once in cache, it may be possible to
connect hashes together so that more chunks can be served from cache, but I’ve missed my exit.</p>
This Tape</code> structure is really just responsible for managing the alterations, the Position</code> will compile moves to it,
and maintain it’s own cache based on the zobrist it calculates.</p>


23-FEB-2025
2025-02-23T00:00:00+00:00
0034 - movegen</h2>
Hazel once again correctly calculates the number of moves up to depth 3 from the starting position.</p>
She’s still missing a bunch of rules (castling, checking for pins, etc), but the way I break this up is much better now,
and I think it will be much easier to extend. Once I got going I was able to whip out most of the generators quite
quickly from the old code.</p>
The generator is extremely</em> slow right now. Well over 3 minutes to fail perft 4. I think the initial fix may be to huck
rayon</code> at it so I can get the accuracy down, but I may do some caching if the mood strikes.</p>
See you on the next branch.</p>
1132 - atm</h2>
Why ATM?</p>
Because it’s all about managing the Cache.</p>
This is going to focus on building some caching into the movegenerator, to hopefully speed us up a bit. I am going to
avoid adding more rule implementation till after I’ve got the caching in place.</p>
The main bit of trickiness is going to be hiding the mutable bits so that Position</code> transparently uses or populates the
cache, while the caller only sees the immutable interface – from their perspective Position</code> is just a data object.</p>
I have a basic idea of how to do this. First, setting up some kind of Zobrist</code> style hashing (or maybe BCH, I have a
lot of reading to do) is obvious. Once I have that it should make the unmake</code> step a lot faster (since it won’t be
recalculating from scratch each time.</p>
I’m not sure if I’m going to try setting up benchmarks just yet, but it would make it a bit easier to measure
improvements. I think for now I can just go with wall time, bringing everything up to, say, perft 5</code> to within a couple
minutes would be ideal.</p>
Once I’ve got something relatively quick, I can finish the implementation for accuracy, then come back to refactor tests
and add proper benchmarks.</p>
The cache itself will live on the MoveGenerator, and Position</code> will have a borrowed reference to the ATM</code>, which
itself has a borrowed, mutable reference to the MoveGen. This immutable</em> reference will centralize all the cache
retrieve/populate operations, so that multiple Position</code> instances ask a single ATM</code> for items from the
MoveGenerator</code> cache, and the ATM</code> either populates or retrieves those items for the Position</code>. This should</em> mean
that Position</code> stays this fully immutable looking thing and MoveGenerator</code> does all the mutation over time. Since
MoveGenerator</code> is eventually going to be wrapped up in a Witch</code> and sent off to live on a thread, this means
thread-local cache with an immutable interface for the caller, which would be pretty sweet.</p>
I’ve never tried to do anything like this in Rust, the closest I’ve come is the Cursor</code> stuff on what I’ve come to
think of as my “Log-unstructured-messy-tree” datastructure.</p>
It’s on the list for a rewrite too.</p>
Anyway, this is the plan, it remains to see if it survives contact.</p>


19-FEB-2025
2025-02-19T00:00:00+00:00
1027 - movegen</h2>
I think I’ve figured out the right split for the move generator. The Position</code> struct is going to be responsible for
fully representing a position with every useful representation, lazily updated.</p>
A position is modified by a move</code>, which is compiled to alterations</code> and those alterations are tagged with metadata
that allows them to be filtered by different subrepresentations. A good example is when calculating pawn moves.</p>
There are six ways a pawn can move.</p>

Push</li>
Double Push</li>
Capture</li>
Promotion</li>
Capture-Promotion</li>
En Passant</li>
</ol>
It makes sense to try to calculate some of these in aggregate via bitboards (you can calculate all the pushes in very
short order by a simple shift of a bitboard representing pawns of a given color, and a second representing all the
blockers on the board). Others (en passant) might be easier in another representation since it’s only ever a few squares
to check.</p>
When describing moves as an alteration, we lose some data about the move, in particular what piece was moving, what
color it was, etc. In order to efficiently create these representations, I want to really just ignore</em> a bunch of the
alterations, in particular I only care about alterations affecting pawns, so I really only need to apply a subset of
those alterations to the representation.</p>
This makes for a pretty efficient way to get bespoke representations for different optimizations. The path is something
like:</p>
Describe a particular subrepresentation of the board, e.g., “Positions of all White Pawns”, turn this into a filter over
the set of alterations, which can then be cached, only updating the representation as new alterations meeting the filter
criteria are encountered and</em> only then when asked.</p>
This scales nicely with the Variation</code> structure, I can attach all the same metadata to the compiled result and just
push a copy of the current representation when encountering a branch.</p>
Once we have the subrepresentation tagged, we can translate the bitboard operations back</em> to the predicates that form
the filter, meaning a more complex representation can – I think – be built incrementally without having to repeat the
underlying bitboard operations.</p>
If that works the way I think it will, it would be pretty cool. If you think of positions of all the pawns</code> and
positions of all the white pieces</code> as two subrepresentations, we would find the alterations for the first by searching
for any alteration involving any pawn. When a pawn is captured by a knight, the remove</code> alteration would be tagged pawn</code> because
of the capture, but the place</code> alteration would not be. Similarly, the second would be searching for any alteration
involving any white piece. A move would have it’s place/remove both tagged by the source color, on a capture the remove
would be tagged with both</em> colors.</p>
When combining these bitboards, we might pawn_locs & white_locs</code>, but logically this is the same as combining the above
two predicates to look for any alteration tagged both white</em> and pawn</em>. This will result in fewer total alterations,
and so should</em> be faster than either of the constituents.</p>
Position</code> can then start to choose which caches it wants to keep, trying to minimize the amount of work it has to do to
answer all the incoming queries.</p>
The path to this I think is something like:</p>

Build ad-hoc representations in Position’s impl. These won’t do any of this caching, they’ll just build a set of
representations worth using.</li>
Once the movegenerator works, start to extract this Move -> Alteration -> Tagged Alteration -> Cached Alteration
Query idea. Simultaneously set up benchmarks calculating whatever people usually calculate for movegen.</li>
See if this nets out to good performance.</li>
</ol>


19-FEB-2025
2025-02-19T00:00:00+00:00
Nearly deleted a bunch of things, fortunately I had just commited everything off.</p>
Lemony snickets these last few weeks.</p>


16-JAN-2025
2025-01-16T00:00:00+00:00
1118 - movegen</h2>
The irony is that I kinda need this tool for the ruby stuff as well now, hard to keep track of it’s</em> structure.</p>
I’m not gonna do that though. I think I can get this close enough to unblock my movegen work, then I definitely need to
rethink how to do this in Rust instead of Ruby, the way I have the Ruby structured is ‘right’ I think, so it should just
be a port to Rust.</p>
I want to look at the syn</code> crate over treesitter</code>, as well. I have a feeling it’s not going to work the way I want,
because I still want to be able to generate diagrams even for incorrect/invalid syntax when possible, but it’s
definitely worth a peek.</p>


15-JAN-2025
2025-01-15T00:00:00+00:00
1111 - movegen</h2>
Made a lot of progress on the ruby/treesitter stuff, and integrated PlantUML</code> which I really like so far. I think I’m
at the point where I need to start treating it as it’s own thing, but it’s still very tied to this project, so I’m not
sure I want to move it out of the repo yet. I also have a long-term plan to make hazel</code> a multi-crate thing, in
particular the ui</code>, the types</code> and this ruby thing (which lacks a name) should all be separate projects, but probably
under the same git repo? Not sure.</p>
I know cargo supports this OOB, so I’m gonna take some time soon and work on a prototype of this, but I think the crate
split is probably after movegen is done.</p>
Right now I can generate some UML ‘cards’ for each of the ADTs in the repo (more or less), there is still some missing
information here and there, but the concept is working alright and it’s mostly about marshalling things to my internal
representation.</p>
I would</em> like to rewrite a bunch of this in Rust at some point, the ruby is quite slow, but I definitely want to get it
producing a diagram with at least all the cards, and maybe adding some relationships where it makes sense.</p>
1456 - movegen</h2>
Two steps back, one step forward on the UML thing, I broke it up, started refactoring to support grabbing fields, and
ended up turning a bunch of things off in the meantime. I think I’m getting pretty close to the ‘just extract it’ and
maybe even the ‘riir’ phase of this thing, but for now I’m just going to keep chipping away at it.</p>


11-JAN-2025
2025-01-11T00:00:00+00:00
2137 - movegen</h2>
I merged, started on movegen, then immediately began procrastinating.</p>
I roughed in a little script to hook up to tree-sitter</code> for querying across the source. My aim is to build up some
scripts to collect some topological information about the codebase.</p>
In particular, I’m beginning to get close to the point where I don’t have sufficient spare memory to dedicate to this to
hold the whole project in my head, which means my organization is starting to break down. I know</em> there is a tangled
web of redundant representations, and I know sort of</em> where all the wiring goes, but I need to get a lower resolution
view of the codebase to start to untangle it.</p>
My first step is to build up something that can pull out the traits, structs, and enums and all their relevant APIs to
some set of files. I can think of ‘easier’ ways to do this (leveraging docgen, for instance, could get me partly there),
but I’d like to sharpen my tree-sitter</code> tool, and I haven’t written ruby in a while. I’ll also need to have it crawl
the src</code> directory and apply the query to every file and build whatever index file I need.</p>
I figure I can dump the relevant information out to a file, then ideally incrementally update it based on filechange.
From there I’d like to generate some graphics I can use to start show the topology; I spent a good hour or so looking
for an off-the-shelf tool to just do the diagramming, but I can’t stand literally every single tool, and I don’t have
the desk space to do it by hand.</p>
2156 - movegen</h2>
Worked a bit and generalized things, I’ll need to build up a little model, but I think it could work pretty well. I
should be able to calculate the relevant module from the file path, since I essentially never have a module nested
without it being a separate file other than for tests.</p>
What works now is a dump of the definition of every struct, enum, and trait, along with it’s path.</p>
Ideally I’ll generate something approximating a UML card for each of these types, then I can figure out how to link it
all together.</p>


9-JAN-2025
2025-01-09T00:00:00+00:00
1500 - ui</h2>
I think I’ve got the headless side of this about where I need it. I still don’t have a movegen or evaluator or anything,
but I should be able to get the UI side of things up and running, then I just want to get things set up so I can run the
tokio-console tool, have tracing go to a log file, and also show up in the UI, and then also be able to take commands
from the UI.</p>
That’s not too much, right?</p>
In any case, I think</em> I might actually merge this once I know CI is good. The UI change is probably a total rewrite,
the structure I used in the last iteration was too broken up, and I should really aim to have more ‘everything in one
bucket’ model to start.</p>
Ultimately the UI, STDIO, and Error Log threads should all branch off of WitchHazel</code>, Hazel</code> is the chess side,
Witch</code> is the communication engine, and WitchHazel</code> is the handle that manages all the IO. I should be able to move
the run_with_io</code> code into the WitchHazel</code> type as an associated function, and I can similarly move the UI code there
as well.</p>
I would like to take some time to write up a move generator, if only so I can have it play random chess and actually
‘use’ the engine, but I don’t want to get too distracted from current progress. There is a lot of fleshing out to do of
the WitchHazel</code> stuff, including custom messages for debugging, and also getting some kind way of having messages
allocate resources in various subsystems. One of the things I found is I will definitely want to be able to ‘defer’
messages – so that the main queue is actually just the input stream, and the first step is delineating between, e.g.,
UCI commands to update the engine state, versus Hazel-specific stuff to control what the engine is doing internally.</p>
I suppose this might mean I have a few actors to build, I could have a UCI frontend actor that forwards messages to the
WitchHazel actor. WitchHazel can happily talk to itself, but the UCI actor only has state sufficient for the current UCI
game.</p>
2302 - ui</h2>
I got distracted. I’m stuck debating how I want to approach the next bit of work. I’m leaning towards movegen, I’ve done
it once, I should be able to port it over relatively quickly. The tools I’ve got in place now make it much nicer to work
with. Getting anything in place gets me closer to a working engine, which was originally my EOY goal.</p>
I do want to rebuild the UI, but I think it’s still going to be flawed until I have a working engine to back it. I
considered building something that would just talk to stockfish, but the point of the UI is to have a backend look into
Hazel, so building against stockfish doesn’t really get me anything.</p>
Building the movegen will necessarily build out the DSL for WitchHazel as well, so I think it’s time to merge this and
go back to procrastinating on UI stuff. This branch has been exceedingly misnamed for it’s entire life, and I think that
might be my favorite thing about it. It’s final act, in fact, was one in which it was ultimately decided that not only
should most of that code be deleted,</p>
but also that I should do it later.</p>
Larry Wall said that the three virtues of the good programmer are Laziness, Impatience, and Hubris. I am a good
programmer.</p>


30-DEC-2024
2024-12-30T00:00:00+00:00
2249</h2>
Working on setting a few things up in glamdring</code>; in particualr I’m working on getting set up on a new terminal
emulator, probably ghostty</code>, though kitty</code> remains in contention. In particular, I figured out (via ghostty</code>) how to
get kitty to work alright with tmux</code> using the same (instructions)[https://ghostty.org/docs/help/terminfo] as
ghostty</code>. That solved the immediate issue with kitty</code> everywhere, but I do like the defaults of ghostty</code> better?</p>
In any case, the main motivation is getting image-in-terminal to work. kitty</code> has the edge there right now, on account
of it working correctly in a nested neovim-with-image.nvim-in-kitty-mode in tmux in kitty</code> situation, which ghostty</code>
presently doesn’t. I want to try the alternate backend for image.nvim</code> to see if it works any better, but I know
kitty</code> already has some existing integration with nix</code> anyway, so it may be the better choice.</p>
I also set up stickybuf</code> and started working on avante</code> integration with a local ollama</code> hosted model; but I have
some networking that needs doing there as well.</p>
Hoping I can drop my ChatGPT and Copilot subscriptions in favor of the self-hosted setup.</p>
I also switched over to lix</code> after getting stung by 9708</a>; I don’t know if
I’ll stick with it, but it seems to be a drop in for now.</p>


28-DEC-2024
2024-12-28T00:00:00+00:00
1514 - ui</h2>
As everyone else seems to be doing at TOW, I’m trying out the ghostty</code> terminal emulator; and I have to say, the hype
seems pretty reasonable. Coming from iTerm2, it definitely ‘feels’ faster. I’m not sure if that’s just placebo, but it
feels like it renders at a higher framerate.</p>
The relevance to this project comes down to protocol wrt images-in-the-terminal. ghostty</code> uses the kitty</code> protocol,
iTerm2 uses it’s own. Barring significant issues, I’m already thinking ghostty</code> might win out in at least the near
term. I do want to investigate alacritty</code> for this as well. Depending on how I go, the choice of protocol will be made.
I like iTerm, I’ve been a longterm user, but the fact that ghostty</code> is already well integrated with nix</code> from the repo
up is very appealing.</p>


27-DEC-2024
2024-12-27T00:00:00+00:00
2337 - ui</h2>
I’m working a bit on the UI and a bit on the game driver itself, I’m thinking I’m going to tear up and rebuild the UI
from scratch. The initial design is a little bit too split up and I think a less layered approach makes sense. It should
really be more like a three layer tree, a collection of forms, containing widgets, which contain widgets.</p>
I’d like to experiment with a image-in-the-terminal API, I think designing the Driver and UI in tandem will help as
well, since I think I want the Driver to ultimately house the UI startup code anyway, because ultimately the main entry
point is going to configure an instance of the driver, and then run it.</p>
That makes it so the Driver is the primary entry point, and everything can ultimately speak it’s little language to
drive whatever it needs to do. This will also make it easier to expose driver internals to the UI, which is it’s main
goal.</p>
Ideally the UI would work by setting up some arbitrary query on the hazel engine and then use the output of that query
to render itself. That would reduce the UI to a Query</code> widget and a bunch of glue widgets that glue different
Query</code>-derived widgets together. Query</code> can then also be used outside the UI to drive the engine in general.</p>
The idea would be that the query specifies a particular state you want the engine to converge on, and then it will
return an iterator of solutions to that query. So a query might be logically “Find the best move after a 6 ply search =
the position at turn 10 for black of the current game.”. The Query widget would then break the query into the ‘command’
part (‘in the position at turn 10 for black of the current game.’) and the ‘constraint’ part (’the best move after a 6
ply search); and it would then create a ‘job’ that the driver would pick up and run based on some scheduler (ideally the
job would contain metadata about what it intends to do so the scheduler can be efficient in how it runs the jobs to save
work).</p>
Ultimately I’m interested more in making hazel</code> good at the statistical analysis of large bodies of chess games, not
so much necessarily an engine that is good at beating you at chess; there’s stockfish for that. I am aiming to build
something closer to a “Chess Machine” that can take a script targeting a simple command/query language and then generate
games for further analysis. It should be good enough to play and be a reliable, if weak, engine.</p>
The engine what actually solves the constraint I think</em> might just be a MCTS; I can say with some probability what,
e.g., an evaluation distribution looks like. If I get the itch I do want to take a stab at a home-made NNUE
implementation which would ostensibly give me a very strong eval function, so coupled with some SIMD MCTS I can at least
implement a big subset of what I’m interested in, which is statistical views subject to some clever evaluation function.
That evaluation function can generate richer evaluation structures, and I think it could be interesting to see what kind
of trouble you can get up to with that.</p>


20-DEC-2024
2024-12-20T00:00:00+00:00
1249</h2>
I went ahead and picked up a SSD for DOP to run it’s OS off of, this should allow me to get moved back</em> over to DOP as
a dev machine, randy will still be used as a dev box, but more for GPU workloads and the like until I get it tuned up.
It’ll probably be getting my test configs set against it as well, etc.</p>
I’m going to put the 2070 in DOP as well so there is some GPU horses there, but ultimately I do want to run both DOP and
BTG as hypervisors running off iSCSI for boot. I think that gets me the best of both worlds in terms of keeping the
disks separated from the machines that manage them, and sets me up for a move to a DIY SAN eventually.</p>


17-DEC-2024
2024-12-17T00:00:00+00:00
1042 - srailimaf</h2>
This is idle thoughts about how to break up what’s in hazel</code> rn. I don’t think I want to keep everything in the same
codebase longterm, since large parts of it will become quite static (e.g,. the PGN parser, the UCI parser, etc); and
other parts are really not particularly chess related (the Log</code> type, in particular, and the UI to some extent may be
worth separating, if only for namespace friendliness).</p>
This would leave the core chess functionality in the main hazel</code> crate.</p>
Right now my compile times are fine, couple seconds, maybe a minute or two for a fresh build, but I can see the writing
on the wall. It’s also just getting cluttered in the project, so I’m itching to organize.</p>
I think the Log</code> type is going to be the first out, especially because I expect it will see re-use outside of this
project, and because I have some ideas on how to extend it to be more useful in terms of how it manages the data it
stores. In particular, I want to extend it to dynamically load segments into and out of the log, so that I can store
logs bounded by disk space and not memory. To do this, I’ll need to have the Cursor</em> types be responsible for holding
the Log data in memory, and the main Log</code> type will instead use a WriteHead</code> to write to an abstract log which is
stored in segments on disk. Finally making it all async would be nice, but I’m not sure if this design’ll scale to that.
Probably a single-writer multi-reader would be ‘fine’, or some kind of segment-based locking, IDK.</p>
It’s not a new idea but it’s a very nice one, and it seems like it’d be fun to build.</p>


15-DEC-2024
2024-12-15T00:00:00+00:00
0037</h2>
I’ve got things sorted I think. I have a spare 2070 I can use for DOP, I’ll pull a bit of it’s memory (half, 64GB) and
put it in BTG. randy</code> will stay there for doing GPU workloads and running ollama</code> models and the like for selfhosted
copilot. DOP will become my main dev machine again, with 64GB of memory and 2070 to run upstairs to my office. To
simplify things I may get a SSD for DOP, that way I can just directly install nixos on the SSD. It abandons entirely my
netbooting experiment, really, but I think it’s probably the right move – I can revisit netbooting in the future when I
have a more stable environment with all my systems running what I need them to run.</p>


14-DEC-2024
2024-12-14T00:00:00+00:00
0049 - srailimaf</h2>
I’m really not liking how the rewind stuff is coming together, and I think I need to change the approach and think about
how I’m going to use Familiar. Initially I was thinking of it as a being fully bidirectional, but trying to track all
the necessary state is looking very complicated, and trying to pack it inside the class itself is looking tricky.</p>
I’m going to keep trying just to see if the additional state calculation needed is really as bad as I think, but I’m
pretty sure it’s going to get out of hand, mostly around variation management.</p>


12-DEC-2024
2024-12-12T00:00:00+00:00
0004 - srailimaf</h2>
Starting in on #2 from above, the first step will be doing some work working on the Unplay</code> side of things and maybe
doing some genericizing.</p>
1000 - srailimaf</h2>
I’ve been poking around at some other tools I might want to integrate, and came across insta</code>, which I think would
greatly improve the tests for the UI. More generally, I want to start considering how to refactor the test code. I like
being able to put tests right alongside the code it’s testing, but I think ultimately I’d prefer if the test code were a
little less heavily duplicated.</p>
I found rstest</code> which offers a little extra, and I think I have a plan for how to refactor longterm.</p>

Keep developing using #[test]</code> and the builtin runner along side the source. These are ‘in situ’ unit tests, they
are intended to simply do whatever is necessary to make whatever assertion is intended.</li>
tests/unit/</code> contains an equivalent folder structure, but uses rstest</code> to run the tests, and is more heavily
factored and intended to be the place where in situ</code> tests are moved to once the functionality is stablized (and
really, once I get tired of the long file size).</li>
tests/integration/</code> can contains integration tests</li>
tests/ui</code> for ui tests via insta</code></li>
</ol>
and so on.</p>
The goal would be to slowly extract to some generic ‘spec’ – especially for integration tests, relying on
non-hazel-specific tools to implement the test as much as possible will allow for easy cross-comparison with knonw-good
engines. The spec will ideally be engine-agnostic to some extent, so that it should be relatively low maintenance as I
keep tweaking stuff in hazel.</p>
I have some loose refactoring that happens in the unit tests as of right now, and coverage is pretty good, but
integration is a little weak. I’ve been thinking about other metrics that could be useful with respect to coverage, in
particular I’ve been thinking about two in particular:</p>

Lines / Test Coverage - How many lines does a particular test touch in our code?</li>
assertions / line - How many assertions are made per line of code?</li>
</ol>
Ideally we want a test to cover as few</em> lines as possible (tests should be precise), while making as many (good)
assertions as possible per line. The former is probably easy to calculate based on the .lcov information, but the latter
is a little trickier. I suspect that these are the underlying metrics that mutant</code>-style testing reveals. Having a high
assertion-per-line ratio would mean that random changes to code are more likely to be caught by some</em> test, reducing
the class of ‘tests-missing-obvious-logic-error’ mutant, but not the ‘subtle-logical-change’ mutant. The latter is far
more rare than the former. Fewer lines touched per test means that the test is making more targeted assertions, which
should address the latter.</p>
I may take a sidequest at some point to investigate how to calculate these metrics, if nothing else than because I like
a good metric.</p>


11-DEC-2024
2024-12-11T00:00:00+00:00
0047 - familiars</h2>
I am, gods help me, thinking about changing some names again. I think I’m starting to see that I really should try to
pack the metadata in the ‘alterations’ stream. I rejected it because it kept alterations very</em> simple, but I think I
need to allow for some kind of arbitrary metadata encoding that would record ‘events’ that alter some metadata flag,
this would make it easy to undo and track metadata state, at the cost of making the alterations a little more
complicated.</p>
I don’t think I’m going to chase this rabbit yet</em>, but I am not loving how my current design is managing metadata –
which is to say, it’s not really, it’s just assuming partial representations will ‘work out’ and shoving metadata into
the right spots in whatever way works. I think my goal is to try to get the thing to be able to represent a full PGN
with variations. My main target for Hazel is high-depth analysis / big data, not necessarily chess playing proper, so
once I can do that, I can start to replace the ugly bits with more confidence since I’ll have a test suite to work
against.</p>
The current plan is something like:</p>

Finish PGN parser via the Familiar system</li>
Implement a recalculation-based ‘backwards movement’ system for the Familiar.</li>
Wire the Variation to a UI widget that connects to the boardstate shown in the UI.</li>
Finish implementing UCI protocol</li>
Implement an extremely</em> bad evaluator/movegen system that can arguably play chess.</li>
Take a break</li>
Kill every single mutant and get to 100% coverage.</li>
Start to refactor the design to something comfortable.</li>
</ol>
I’ll chip away at that before getting into evaluators and UI and all that. I have plans for that but I think I’ve
settled on the design and I just need to finish building it so I can get to the polish phase.</p>
2128</h2>
I merged, finishing #1 from above. I think #2 will require some work on the ChessGame</code> component, I’m pretty unhappy
with how BEN</code> and FEN</code> are sort of scattered around, I would prefer to treat the board rep more abstractly, and this
gets back to the Alter</code> system and what a ‘board representation’ really is. I think it might start to reveal itself as
I start to remove the hard Move</code> and BEN</code> types in Action.</p>
I could approach this in the ‘dumb’ way of replacing ChessGame with a trait that captures it’s current API. This would
at least gather the API into one spot so I could look at it.</p>
In any case, I’m just going to start tweaking stuff and see what happens. Can’t overthink if you stay busy breaking
stuff.</p>


10-DEC-2024
2024-12-10T00:00:00+00:00
1314 - familiars</h2>
I’ve been chipping away at this and I’ve gone for a bit of a tweak, embedding a lot of what was ChessAction</code> into the
Play trait first class. It’s still tied to specific move/boardrep types, but I think that’s okay for the moment. I need
the boardrep because I need to support ‘setup’ commands, and obviously I need the movetype, but I dislike how things
work in the current model and would prefer it to be more generic.</p>
The next step is to get Familiar</code> correctly evaluating to arbitrary positions in the variation; once I can do that, I
can start working to extract the type assumptions.</p>


30-NOV-2024
2024-11-30T00:00:00+00:00
1049 - familiars</h2>
I’m thinking about some reorg. Part of the process of building Familiar</code> has made it clear that what I have is quite
capable of representing pretty generically any kind of abstract perfect information game, and I’d like to preserve that
property and make it more explicit in the organization. I’m thinking of these changes:</p>


Move src/game/</code> tree to have a src/game/<name of abstract game>/<contents here></code> structure, so I can represent
other games.</li>
Extract the PositionMetadata</code> struct to this new location for the chess</code> subdirectory.</li>

Move board/interface</code> to the top level.</li>

Move compiles_to</code> to the interface section, though presently it’s unused and may remain there, I may leave this on a
twig and remove it from the trunk, haven’t decided yet.</li>
</ol>
Hazel</code> can then have (ideally) a relatively abstract idea of what a game is, and can hopefully lead to some reuse with
some of the scaffolding (e.g., whatever alpha-beta/minimax/mcts/nnue bullshit I come up with) with other games. I’m
thinking primarily for fairychess, but also even something like nim</code> for testing purposes could be handy. nim</code> is an
extremely simple game, so it’s possibly valuable for debugging and testing purposes, remains to be seen.</p>
All of this should also provide a nice place to put a Game</code> structure that can then implement Play</code>, this should be
generic with respect to board representation and metadata representation, but canonically should use the
PositionMetadata</code> struct for metadata, and any Query + Alter</code> capable rep.</p>
1114 - familiars</h2>
As of now, all but #2 is done, tests are passing, so time for a big commit.</p>
2321 - familiars</h2>
I’m working on the last move from above, and I’m thinking of some further tweaks I want to consider.</p>
In particular, I think I’m going to want to build an ‘Index’/‘Cache’ system for Log</code>. Eventually I want to be able to
refer to different parts of the log by different criteria. For instance, I might want to search the log for a specific
turn’s position in a specific variation, I need to find the place in the log where that is, and I may want to cache that
position for later use, so I’m starting to think of what that might look like. In particular I suspect something like
the familiar</code> system would be used to maintain that index and cache. Ultimately this sort of works as a file format for
a database, and hazel</code> acts as an interface to that database.</p>


27-NOV-2024
2024-11-27T00:00:00+00:00
1214</h2>
I’m thinking I might move to an iSCSI-based approach for VMs for the near term.</p>
The netboot experiment has been cool, but it’s finicky with total available space and also it’s a really slow process to
rebuild the images. I think I can get a lot of the same benefits (centralized storage, decoupled configuration, etc)
with iSCSI and eventually some kind of overlay system.</p>
I’m going to build an iSCSI gold-image that can be cloned. It will have a baseline install of a recent NixOS, and will
be periodically mounted and updated. When creating a new machine, I’ll have scripts that:</p>

COW Clone this iSCSI base image</li>
Create a new RW iSCSI disk</li>
Configure these disks for the new machine.</li>
</ol>
The machine would then mount the two disks in an overlay, booting form the base COW clone and then writing to the RW
disk with any subsequent updates.</p>
Machines can be periodically recreated, and any dedicated data can be placed on an independent iSCSI, so the COW clone
is just the common shared underlayment, the RW disk is for machine-specific config, and additional independent disks for
application data.</p>
This will require a lot of automation around iSCSI creation, but for the immediate term I’ll probably just make disks
per machine and install that way, a hard clone of the base image should do it. I’ll serve them out of Nancy and once
it’s working I can work on building a better iSCSI server.</p>
To that end, I think I’m targeting 100GBe Ethernet out of the gate, I considered other protocols, and they may be useful
if I want to split out a backplane for data, but in the short term I’m going to keep it simple and have a single flat
network on 100GBe between all physical components that support it (currently only 3 such machines, but I expect that I
will have a few more in the future, so I’m aiming for something in the 16 port range on the switch).</p>
Ultimately the 100GBe switch will link over a pair of 10GBe links to Condorcet, and then the majority of ‘normal’
machines in the lab will still route through there, but the 100GBe backend will be used for the Racked machines to share
data, the traffic is low enough that a flat network should be fine, and I do want to move to IPv6 eventually anyway, at
which point the routing will be much simpler.</p>
I may</em> start-from-v6 on this project, and use it as a lever to excise v4 from my lab, but tbh I’m not sure it’s worth
it in the short term.</p>
All of this mess brought to you by frustrating issues with my prometheus server not mounting NFS the way I like. Hell of
a lot easier when it just looks like a disk.</p>
1619</h2>
I think I’m going to start by replacing some of the NFS-as-a-target stuff with iSCSI. NFS is really not the right tool
for what I want, which is a pile of bits that the VM can read/write from on it’s own. I can still netboot the machines
but have all the data live in iSCSI volumes. I should be able to extend my existing network-storage code for this, and
I’ll have to manually recreate the luns, but I should be able to query the infra code to generate an appropriate list
for Nancy at least, and then that list can eventually drive a proper iSCSI host.</p>
1940</h2>
Okay, actual plan:</p>


I’ll create NixOS-24.11 as a LUN in my Synology (Nancy)</li>
[~] I’ll mount this LUN on pinky</code>, my lab VM, and install NixOS 24.11 on it.</li>

I’ll then clone this LUN to a new LUN, PINKY-24.11-ROOT, and mount it on pinky as its new root disk</li>

I’ll alter Pinky’s configuration till it can boot from this iSCSI disk.</li>
</ol>
After this, I can recreate the other VMs using this new system, the clones will be RW per machine and full copies (since
COW doesn’t seem straightforward here), but it’ll get the system running and I can work on it from there.</p>
I’m a little ways into the plan above and I don’t expect it’ll take to long to get running.</p>


26-NOV-2024
2024-11-26T00:00:00+00:00
1252 - familiars</h2>
Continuing from previous entry, I’m starting to solidify the design of the playing-end of this thing.</p>
I have this setup:</p>
</span>
Log is a way to store a sequence of anything, in our case ChessActions</span></span>
</span>
Chess Action supports a notion of 'Variation'</span></span>
</span>
The Log lays out it's contents as single, seekable stream, similar to the File API.</span></span>
</span>
Log can produce a Cursor or WriteHead (mutable cursor) on itself, which can be used to navigate the log.</span></span>
</span>
A Familiar takes a Cursor and calculates some useful values, in particular the current board state and current metadata</span></span>
information.</span></span>
</span>
Familiars are generic, they only care that the type they work over implements `Play`, which itself is a trait that is</span></span>
generic over the Rule and Metadata types that govern whatever abstract game they define.</span></span>
</span>
A Familiar can be specialized to a specific representation type to allow for faster/more efficient calculation, it might</span></span>
be responsible for caching important results, etc. Most directly, it's responsible for calculating the current board</span></span>
and metadata state.</span></span>
</span></code></pre>
The final engine will essentially be a Log, a bunch of Familiars that can be created/destroyed as needed, and insodoing
I can have multiple representations that can all benefit from intraconversion. Familiars should have a ‘Set Position
with cached state’ option which allows one familiar to transfer it’s state to another; so that a Familiar optimized for
fast scanning can then feed one that’s designed for fast querying, etc.</p>
This also means new representations can be easily compared apples-to-apples with exisitng representations.</p>


23-NOV-2024
2024-11-23T00:00:00+00:00
0048 - pgn</h2>
I added a fixme to mark where I left off, but I’ve managed a lot of progress on this parser. I switched to a different
approach that essentially focused on tokenizing the PGN to a much more convenient representation that was easy to shove
into the Variation structure. I’m only missing a few things:</p>

Annotations need to be tokenized so they can be ignored correctly.</li>
Comments similar</li>
Better Section marking, I’m just marking start/end of file, the naming is bad, etc.</li>
The current_position calculation for Variation needs to be modified/replaced to be able to calculate the position at
the tip of the log.</li>
</ol>
I think those things get me enough to parse PGNs. It should also be possible to reverse this and generate a sequence of
tokens from a variation, which would make it very useful for comparing game trees and the like later on, also handy for
the UI.</p>
I’m currently ignoring the turn marker, but embedding that in the variation could be a pretty handy way to track
different positions within the tree. I want to look into having a sort of ‘detached’ cursor that could be used to have
multiple referents to a single Variation, I think this will be helpful in lots of ways but it starts to venture into
bits of the borrow checker I’ve largely avoided.</p>
Sounds like a problem for future me though.</p>
1116 - pgn</h2>
Finished the annotations and comments from above.</p>
2309 - pgn</h2>
I need to work a lot on how I navigate around the log.</p>
There is a case, I think, for a specialized cursor that acts as a state machine over the log, that is where the game
tracking lives, a single log can have multiple cursors, which can be constantly growing and being pruned as the engine
works. It tweaks my design a little, but I think that’s the next natural place to put effort. I think the parser is
properly handling PGNs now, so it’s really ‘done’ and should be merged, the conversion to a variation is what’s broken,
but it’s separate from the ‘current_position’ problem that I think this will solve.</p>
I also want to consider embedding the number of moves contained in a variation as part of the variation itself. This
would have to be done in a second pass after tokenization, but it would make the skip-ahead logic much easier.</p>
I’m going to split this work into a new branch, then kill mutants until I can merge pgn</code>. The tests for the actual
pgn</code> class will be lacking, but I think I’ll just have to make it up elsewhere.</p>


20-NOV-2024
2024-11-20T00:00:00+00:00
1141 - pgn</h2>
Having thought about it more overnight, I think my plan is thus:</p>

Get the existing PGN parser to the closest thing to a working state as I can. Hack as needed</li>
Merge</li>
Extract and unify this ‘minilanguage’ thing I have going on into it’s own abstraction (preparing for eventual parser
writing for the Witchlang)</li>
Build a better MoveGen system based on the enum-based approach</li>
</ol>
Ultimately Hazel (the engine bit) is going to have a WitchLang</code>, which compiles to WitchASM</code>, which is an Enum-y
language like what I have now. WitchLang will be a small scripting language that can be used to create more complex
queries that can then be optimized, similar to how a database query-plans.</p>
Hazel (the engine) will be a small VM with some tools to alter it’s scale, what represnetations are active, etc. It will
produce a stream of instructions to configure itself and solve any presented chess problem.</p>
This will also allow for more asynchronous processing, e.g., the movegen can request a bunch of calculations, but the
engine can batch and cache these things, reach into existing cache, etc – behind the scenes.</p>

Get perft</code> working for a few positions, matching stockfish</li>
</ol>
This milestone will be the big ‘I’ve got a system working’ moment; since from here it’s just a matter of adding eval and
pruning tools to the results of the movegen. Ideally I’ll be able to express some basic evaluation functions in the
WitchLang and then start looking to extend it to NNUE and the like. Ideally it’s something like:</p>
Engine Tune:</span></span>
    Set parameters here</span></span>
Search <Some FEN></span></span>
    Depth <some plycount></span></span>
    Prune With:</span></span>
        Some subprogram</span></span>
    Filter Final:</span></span>
        Some subprogram</span></span>
    Group By:</span></span>
        Some subprogram</span></span>
# etc</span></span></code></pre>
This gets compiled down to a series of WitchASM</code> instructions that can be run on the engine, and then ideally the
language can express self-retuning as it iterates, etc. Ideally all the chess-related logic ends up in this language and
we then attack the problem like a compiler problem, optimizing to intermediate reps and building an engine that can
solve the given script optimally.</p>
I think this will make for something very flexible, since most of the chess logic will live in the language and not the
engine itself. I suspect I may see some overhead, but I’m hoping the translation layers should mitigate some of that,
since the final set of instructions that the thing needs to execute should be somewhat smaller. Translating further down
to bytecode (and perhaps SIMD bytecode, since most of these operations should be parallelish) should be doable and
hopefully keep the speed sufficient to justify the flexibility on offer.</p>
For Eval, I’m planning to build a bunch of different eval functions, but I’m particularly interested in NNUE and messing
around with different architectures using the NNUE concept. More SIMD in my future.</p>
Ideally I’d like to get to the point where I have a suite of integration tests that:</p>

Use WitchLang to load a PGN, do evaluation to it, and report statistics about the results.</li>
Use WitchLang to perft from multiple different positions and compare statically to stockfish</li>
Use WitchLang to generate a random position by playing random moves, then dynamically compare perft results with
stockfish to the maximum depth achievable in a reasonable time.</li>
</ol>
Those three tests should fully exercise any movegen code I’m using; especially if I can set the number of PGNs pretty
high.</p>
2136 - pgn</h2>
I ticked off #1 of the above and got past the ambiguity with sliding pieces. I even got to use some of the old bitboard
implementation.</p>
I need to do some work to get variations parsing and the like, but I think the move generation is probably ‘good enough’
that I shouldn’t run into an ambiguity problem again.</p>
I’m not tempting the gods, you’re</em> tempting the gods!</p>


19-NOV-2024
2024-11-19T00:00:00+00:00
2313 - pgn</h2>
I’m frustrated by the fact that PGN requires, essentially, an entire movegen system to parse. I’m going to hack in
something in the ‘good enough’ category so I can flesh out the Variation stuff and then probably put it down and go to
work on the MoveGen. I have an idea for it that I think will work well with the design I’m aiming for. In a surprise to
no one, it’s copying the Alter</code> system. I’m going to use the old movegen as a guide, and focus on building a system
that can implement a rich language for querying a boardstate. I can then use this to build a movegen system that is
abstract with respect to boardstate and</em> can abstractly describe different movegen calculation strategies that can then
be run against multiple backend boards.</p>
An implementor would then have to implement some minimal set of operations, which all others must be expressed in terms
of, and then I can build different backend representations designed to make some operations faster.</p>
Ultimately this will build up to a general language that can describe how to arrive at specific boardstates, and how to
do analysis downstream.</p>
You’d have a script that describes some algorithm to tell the engine how to proceed from it’s current position, what to
evaluate (e.g., maybe “find the top 100 lines from this position for white at depth n</code> and then calculate the relative
power of the black bishop in each lines and report the distribution as a graph”), and then hazel would haul off and do
the work.</p>
Each little language is really a part of this bigger language that ultimately ‘compiles’ to some glue language.</p>
I was hoping to get pgns</code> more fully and comfortably supported, but I don’t think that’s going to be possible right
now. I might still work on the Variation -> PGN (at least the mainline) so I can display it in the UI, but I’ll have to
think about how much I want to keep writing parser/printer code.</p>


15-NOV-2024 - pgn
2024-11-15T00:00:00+00:00
2157</h2>
I’m writing the parser, it’s going alright. I’m finding myself in want of a lot of QoL stuff so I’m splitting between them.</p>
In particular, not being able to have a cheap, copy-able board representation like FEN was killing me, so I started
working on BEN, which is a relatively compact binary-encoded FEN equivalent. Insodoing I added Alter</code> to FEN</code> so it
technically counts as a whole Board Representation now. This led me to realize I’ve been using PieceBoard</code> pretty
liberally as a way to make FEN alter-able, and I’ve unwittingly bound it quite tightly to a particular board
representation internally. It occured to me it would be pretty cheap to abstract this to a type alias that only claims
it’s traits and no particular internal representation. So in principle something like:</p>
type</span> Board</span> :</span> impl</span> Alter</span> +</span> Default</span> +</span> Query</span> +</span> Clone</span> +</span> Into</span><</span>FEN</span>></span> =</span> PieceBoard</span>;</span></span></code></pre>
Then I can use Board</code> everywhere, and if I want to switch to a different representation, I can just change the type to
anything that implements the traits. Later I can add additional traits like:</p>
type</span> MoveGenOptimizedBoard</span> :</span> impl</span> Alter</span> +</span> Default</span> +</span> Query</span> +</span> Clone</span> +</span> Into</span><</span>FEN</span>></span> =</span> Bitboard</span>;</span></span>
type</span> UIOptimized</span> :</span> impl</span> Alter</span> +</span> Default</span> +</span> Query</span> +</span> Clone</span> +</span> Into</span><</span>FEN</span>></span> =</span> CharBoard</span>;</span></span></code></pre>
I suspect I’ll want to introduce some kind of granularity here, I’m not sure the best way to do it, but I want to rely
on no specific board representation interally, but rather on a set of traits that can be implemented by any board
representation.</p>
I’m premature in my optimization, but I can see that there will be a point where board representation becomes an
optimization path and I want to approach that in a structured way.</p>
I suppose figuring out how to extract the generic Board</code> type is the first step.</p>
2038 - pgn</h2>
I’m thinking a bit more about Alteration</code> and what I should encode there. I think ultimately I do want to try to
encode the entire gamestate in the Alteration</code> stream, which means encoding some sense of metadata, as well as game
events, and so on.</p>
Each board representation implementation is going to be good at “something”, it may be optimized for efficient movegen,
or for easy evaluation, etc. I think each Alter</code> type should advertise which subset of the commands it implements.
During ‘compilation’, the ‘compiler’ will check this list and try, where possible, to provide implementations of
whatever keywords are missing. So for instance, if a BoardRep doesn’t implement the ‘Clear’ command, the compiler will
replace it with 64 ‘Remove’ tags, assuming it implements ‘remove’.</p>
As I build up the engine’s abilities, I can add new commands, and so long as I can implement them in terms of the older
commands, I should be able to use the new command with any older board representation still, even if a bit slower. This
matters for the UI and non-engine-y parts of Hazel. Since the UI is a big part of how I plan to develop Hazel, I want to
build the UI to also read along the Log of ChessActions, which ultimately become a list of alterations.</p>
Different consumers of that log stream can maintain a Cursor</code> into the log and then ideally rewind/fastforward to any
state in the log. The composed object will implement Alter</code> as the sum of it’s internal implementations, and dispatch
commands to subcomponents as it pleases. So the UI’s implementation of Alter</code> might update it’s UI oriented internal
state, while the Engine might instead be grabbing many alter’s at once and applying them in batches somehow.</p>


10-NOV-2024
2024-11-10T00:00:00+00:00
1219 - gamerep</h2>
I’ve got Variation</code> (formerly  Game, Line, HalfPly and Ply… it’s been through a few iterations) basically working.
It can at least represent a simple variation, and while the API is going to need some finishwork, I think the design
works and I should be able to flatten a PGN into it.</p>
I’m debating now between merging and tackling that in a separate PR, or just pushing through and getting it done. I
think I might try to push through and see how painful it is. If it comes together quickly then I’ll go for it, but
otherwise I’ll plan to merge and then tackle it in a separate PR.</p>
I have an existing ‘no-variations’ PGN example I can use, and I think I should probably have the test exist in the
tests/</code> subdir as it’s more of an integration test than a unit test. I plan to use the Shakmaty</code> PGN parser for now,
but would like to build my own eventually.</p>
1628 - gamerep</h2>
I think I’m going to merge.</p>
I also think I’m going to write my own parser (probably with nom</code>, maybe fully by hand).</p>
Here’s why.</p>

The next step really is a big change in design, I’m going to be doing parser stuff, it’s not going to be particularly
chess-y, and it should</em> be pretty simple</em> believe it or not.</li>
The model I have and the model the pgn-reader/shakmaty use are very different. I should</em> be able to directly read a
variation in a single pass, no visitors or whatever, just a single read to translate each move to the correct format.
The OTS dependencies expect a visitor-pattern approach because they (rightly) believe most people won’t be doing weird
bytecode shit.</li>
Doing it myself means I can drop a couple dependencies, which is very cool.</li>
</ol>
I’ll probably use nom</code>, but I may even try a simple RD parser myself, since the format is pretty simple. I will
probably build a PGN</code> object that holds all the metadata and the actual variation, which can then be produced by/handed
off to the actual Engine.</p>


31-OCT-2024
2024-10-31T00:00:00+00:00
2007 - gamerep</h2>
Spooky season is ’pon us, and I have deleted a bunch of code. I’ve been working on the Game</code> representation, and I
think I have the design down (almost)</p>
The main problem, it seems, is metadata. In order to know, for instance, who is allowed to castle, I need to know if any
one of a number of events has occured over the course of the game up to this point. This can be calculated each time I
need it, and it is admittedly not the most expensive thing in the world, but it gets tricky when I need to know the
metadata state in a particular variation of a particular position and it can very rapidly get hairy.</p>
The Alter</code> system works really nicely for this, I have an Alteration</code> struct that has much simpler primitive
operations to the board, so I though, “Can I extend this language?” and the answer is “Actually I should just build
another language.”</p>
So now I have ChessRule</code>, which includes the higher level actions that ‘compile’ to this lower level ‘alteration’,
I’ve also added a couple alterations variants. Clear</code> simply instructs the implementor to reset the state of the board
to nothing, and Tag</code> allows a 4 byte ‘tag’ in the output that is otherwise ignored. These allow me to easily generate a
single stream of Alterations that can:</p>

Describe the change to the boardstate over time</li>
Represent arbitrary trees of games</li>
Be easily ‘compressed’ to a ‘compact’ form (distinct from, but equivalent too, FEN) for any single boardstate</li>
</ol>
Since Alteration</code>s are reversible (mod Clear, more in a sec), I can easily scroll forward and backward within the space
of a single ‘clear’ call, and if I need to rewind to before that, I just rewind back to the previous clear and rerun.</p>
This leaves the existing Alteration</code> stuff more or less unchanged, and allows me to use this new higher level language
to describe a chess game as a stream of ChessRule</code>s, which can track higher level details. The Game Representation is
just a vector of ChessRule</code> that gets Move</code>s #make</code>d on it, and when the move is made, the representation can also
add any number of metadata variants too, these then get ‘compiled’ in the current context and added to the vector of
Alteration</code>s that then represent the game. The metadata gets added as Tag</code> variants in the GameRep.</p>
All this still requires a bunch of work and testing, in particular getting the Tag</code> parts right is proving a little
tricky. I initially thought of sticking a [u8] slice in the tag, and I still, tbh, want to do that, but it ends up
touching a bunch of the system with lifetime annotations and that seems really lousy.</p>
Current plan is just to power through till I get something that can represent a PGN and get that wired up to the UI,
maybe with some kind of insight from PGN -> ChessRule -> Alteration</p>
A practical effect of this is also that many things currently implementing Engine</code> really shouldn’t, since they can’t
track metadata, which makes a ton</em> of sense in retrospect. I kept trying to figure out how to reconcile why it felt so
weird that any board rep was a kind of ‘engine’, but in this model, they’re not, they’re just Query + Alter</code>,
make and unmake are really part of another trait, Play</code>, which is different than something which can be merely
‘altered’; it implies an understanding and ability to track metadata for the given game. In my case, there is only one,
but I have a Play</code> trait which specifies a particular included type, Rule</code>, which takes the equivalent of the
Alteration</code> struct (in this case, ChessRule) and has an apply</code> and unwind</code> method (and _mut variants) to apply and
… unwind the given ‘Rule’[1] and track the metadata internal to the structure, the trait doesn’t care about the
content or how that metadata is stored, just that it is.</p>
Movegen can then take a state as compiled to Alterations</code>, then start creating whatever moves it likes as ChessRule</code>
variants piled on top, the ChessRule</code> includes the idea of a Variation</code> which is a delimited sequence of moves
branching off from the previous. These can be nested to create arbitrary structures. Seeking to a particular variation
involves simply unwinding from the variation backward until you find a Clear</code>, and then reading back the result.</p>
Now we get back to engines, Engines take UCI commands, Play</code>able objects take ChessRules</code> (or whatever other variant),
our system says every UCI command should translate to some set of ChessRules</code> that can then be compiled to
Alteration</code>s and applied to some Alter + Query</code> object to represent the game. This allows a single representation
which can easily be tracked through, the resulting state can be ‘compressed’ to a Clear</code>, a series ofPlace</code>
operations, and then a set of Tag</code>s to define the metadata, and this representation is easy to turn into any other
board representation you like. So if you want to use SIMDified bitboards, you just need to tell it how to process the
Alteration</code> stream, and once it is in your domain, how you use that representation is your business. Once you want to
send results back, compile them to Alteration</code> or ChessRule</code>s (or even a mixed stream of them) and send them back.</p>
I think this’ll work well for my purpose, but I do expect I’m adding some amount of overhead compared to a direct
implementation. Fortunately, I should be able to test that later if I like by directly interpreting the ChessRule</code>
without the Alteration</code> layer, and that will give me some sense of how much overhead I’m adding by the extra jump.</p>
[1] Here I’m doing a classic math thing of taking a word with a well known and well-understood meaning and using it in a
way which almost</em> fits that meaning but stretches it just a little past comfort. In normal terms, a ‘rule’ is an
assertion toward obedience. You have a rule that says “Thou shalt X”, and you better be X-ing or else you’re breaking
the rule.</p>
However, rules can also be ‘applied’, as in, “Use L’Hopital’s Rule to solve this limit”. In this case, the rule is an
algorithm or technique or manipulation.</p>
Rules can also be ‘observed’, as in “The rule of law”, where the rule is a principle or standard that is generally
understood to be ‘the way things work’ and is used to guide behavior, though not strictly enforce obedience to some
particular interpretation.</p>
In Hazel</code>, a rule is simple ‘An operation that can be applied to a gamestate’. It’s a little bit of all three, but
really it’s closest analog is an ‘Arrow’ between objects. In the “Category”[2] of all Chess games, where arrows are
game-legality-preserving moves between boardstates, a rule is associated with every one of those arrows.</p>
[2] I don’t actually know if this is a proper category, so maybe it’s just a Graph with some extra steps, but I like to
think of it this way.</p>


27-OCT-2024
2024-10-27T00:00:00+00:00
1109</h2>
Finished the passthrough setup for randy, next steps are to get BTG set up w/ the 2070 I have and to run cables from the
rack to my office for HDMI and DP from BTG. Then I can use looking-glass</code> and BTG for near-native performance on the
VMs.</p>
Once these are done I’ll look towards migrating mirzakhani</code> into this model as well; and maybe move it down to the rack
as a result. Then I’ll have two hosts with plenty of GPU horsepower I can eventually MDev up.</p>
I haven’t dug too deeply into the MDev side of things, but from the little I’ve read so far, I’m hoping I can have a
setup which gangs up all these GPUs, less for the gaming side of things (if I have to pull one out of the gang to run
whatever, so be it), but for the compute side it’ll be important. A quick test with hashcat</code> got me 13GHashes/s on the
P40 alone. I can’t get it to run on the ‘CPU’ on the VM, since it’s a QEMU vCPU it doesn’t see it as a ‘real’ CPU, but I
suspect 13GHashes/s is going to translate to a pretty good rate of speed on Brocard. Assuming I get about the same in
candidates-tested-per-second, it’s 11Y of compute time, but I have good reason to believe it’ll be faster (since MD5 is
compartively more complicated than what I’m doing with brocard) and hopefully I can clear the 2e64 space somewhat more
quickly without having to scale out too much.</p>


16-OCT-2024
2024-10-16T00:00:00+00:00
2104</h2>
I merged it. I got everything working again and I merged it. I was able to delete a ton of old code that wasn’t needed,
and really improve the ergonomics of working in hazel</code>. Now that it’s at least partially done, let me take stock of
where things live and what it cost.</p>
Costs:</h3>
I binned the NOTATION_TO_INDEX</code> and related constant lookup tables, and replaced them with a common Square</code> type that
is tucked behind a SquareNotation</code> trait. This leaves a door open to change this later, but for the moment having a
consistent type for referring to squares everywhere, and allowing me to easily convert between rank/file/index is very
handy. Saves me writing octal constants everywhere, too.</p>
I also deleted, more impactffully, the Game</code> and Plytypes, these were trying to do way too much, and I have a better approach in mind. This does mean I gave up my progress towards</code>perft`, but I think I can bet back there in a more
stable configuration and not have to worry about carrying the mess with me.</p>
Two weeks of work, and I’m technically behind where I was, but I think it was worth it.</p>
Oh, and the benchmarks are toast. I changed a lot of the bitboard API and that’s all that really had benchmarks. I’m
going to get the thing fully working and then go back and write benches. Originally I had started this as a reason to
learn criterion</code>, but rapidly I found I was less interested in chasing performance then in the core problem of a chess
engine itself, so it bitrot.</p>
Benefits and new Design</h3>
The main benefit is that this enables moving forward with a new design. The organization is slightly improved. It looks
like this now.</p>
</span>
src/</span></span>
    board/</span></span>
        # Board Representation</span></span>
        interface/</span></span>
        <board reps></span></span>
    brain/</span></span>
        # Primary Subsystem, manages all the other processes.</span></span>
    constants/</span></span>
        # Useful _constants_</span></span>
    coup/</span></span>
        # Move related code</span></span>
        rep/</span></span>
            # Move representation</span></span>
        gen/</span></span>
            # Move Generation</span></span>
    engine/</span></span>
        # UCI Interface Substystem</span></span>
    evaluator/</span></span>
        # Position Evaluation Subsystem</span></span>
    game/</span></span>
        # Game State Representation</span></span>
    notation/</span></span>
        # Notation Representation + Conversion to internal canonical formats.</span></span>
        fen/</span></span>
        uci/</span></span>
        square/</span></span>
        pgn/</span></span>
        ...</span></span>
    types/</span></span>
        # General types that don't fit in other subsystems.</span></span>
        bitboard/</span></span>
        pextboard/</span></span>
        color.rs</span></span>
        direction.rs</span></span>
        occupant.rs</span></span>
        piece.rs</span></span>
    ui/</span></span>
        # UI code</span></span>
        model/</span></span>
            # View Models</span></span>
        widgets/</span></span>
            # Widgets</span></span>
        app.rs</span></span>
    util/</span></span>
        # Utility functions and some types</span></span></code></pre>
Ultimately, the hazel</code> lib will provide a single tokio process hazel</code> which spawns a ``brainprocess, which in turn spawns whatever other subsystems it wants according to it's configuration. This will include a gamestate represnetaiton, some way to speak UCI, evaluation and movegen, etc. The idea is that</code>brain` can dynamically adjust how it’s set up,
connect with other hazel instances, etc.</p>
The hazel</code> process can also optionally spawn a UI, which will allow deeper insight into the engine state.</p>
Already existing is an ability to wrap other engines, so another subproject here is to build some cross-testing tools as
well.</p>
So the thing looks roughly like:</p>
</span>
Hazel</span></span>
|               /--A- Engine Driver Backend</span></span>
|               |</span></span>
|------ Brain --|--A- Move Generator(s)</span></span>
|               |</span></span>
|               |---- Game State</span></span>
|               |</span></span>
|               \--A- Evaluator(s)</span></span>
|</span></span>
|</span></span>
|</span></span>
\------ UI</span></span>
</span></code></pre>
The Hazel</code> process will be the main entrypoint, and will have a VM+scripting language to control it and the other
processes. This might be custom, for fun, not sure. I might just embed lua or something, like a responsible adult.</p>
The goal would be to have a DSL that can load games, do analysis, process that analysis, etc. Less an engine to beat
other engines, more an engine to study positions efficiently, etc.</p>
The next step is to rebuild the Game Representation, it’s mostly done, but I want to get Move and PGN represented in the
notation module and Move’s abstracted towards the MoveNotation</code> trait similar to what I did forSquareNotation</code>, and
Square</code>. This is already underway as well.</p>
After that, MoveGen and Perft. I still have the old Ply</code> code so I can borrow from it where it makes sense. This is new
territory, I want to build it as a tokio</code> process that speaks over some API. Ultimately I want hazel</code> to be natively
multi-system, so it can run it’s components across any system you like, so building up the engine as a bunch of
independent components is going to be necessary. I’ll probably take that opportunity to scaffold in the whole process,
and also get a diagnostic environment set up.</p>
Somewhere in there I also want to make more progress on the UI.</p>
My plan is to have two worktrees, UI</code> and whatever thing I’’m working on at the time. As I implement stuff, I can keep
chipping at the UI as I go.</p>


13-OCT-2024
2024-10-13T00:00:00+00:00
2322 - movegen-v2</h2>
A great culling has occurred as I move things towards the unified Square</code> approach to notation. I’ve more or less got
everything lined up, and am just chasing out bugs from the various bits that I tweaked incorrectly to return to
compiling. At time of writing, six failing tests remain.</p>
The approach I took seems alright, though it was tricky in spots. Essentially I have everything rely on an implementor
of SquareNotation</code>, which requires the implementor be capable of converting to a Square</code>, which is a simple newtype
around usize</code> that constrains to 0..64</code> and provides some convenient const-time functions for working with it. The
trait wraps and provides those functions for general use, eventually it would be nice to push as much to const-time
evaluation as possible.</p>
This should make it easy to have alternative implementations I can experiment with if I find better represnetaitons
later, and should smooth the cutover should the need arise.</p>
After getting Square</code> setup throughout, I plan to do the same thing with Move</code>, building my current, compact Move</code>
representation into the ‘Default’ MoveNotation</code> implementation, similar to SquareNotation</code> and Square</code>. Also FEN</code>
needs to be factoryed in throughout. Here it will be much more likely that thr trait will be valuable, as there are lots
of different move representation schemes, so by implementing the trait, I can set up a canonical pipeline to convert
between different representations. Something like:</p>
</span>
pub fn</span> convert</span><</span>M</span>,</span> N</span>>(</span>m</span>:</span> M</span>)</span> -></span> N</span> where</span> M</span>:</span> MoveNotation</span>,</span> N</span>:</span> MoveNotation</span> {</span></span>
    let</span> i</span> =</span> Move</span>::</span>from</span><</span>m</span>>;</span></span>
    N</span>::</span>from</span>(</span>i</span>)</span></span>
}</span></span>
</span></code></pre>
I don’t think it’s possible to write the more general:</p>
</span>
impl</span><</span>M</span>,</span> N</span>></span> From</span><</span>N</span>></span> for</span> M</span> where</span> M</span>:</span> MoveNotation</span>,</span> N</span>:</span> MoveNotation</span> {</span></span>
    fn</span> from</span>(</span>n</span>:</span> N</span>)</span> -></span> M</span> {</span></span>
        let</span> i</span> =</span> Move</span>::</span>from</span>(</span>n</span>);</span></span>
        M</span>::</span>from</span>(</span>i</span>)</span></span>
    }</span></span>
}</span></span>
</span></code></pre>
because of orphan instances, but I think the first is good enough for now. I can always add more implementations later,
and within hazel, it’s most likely that I’ll want to move to Move</code> anyway.</p>
Lots of gruntwork to do, but I think the result will be worth it.</p>


12-OCT-2024
2024-10-12T00:00:00+00:00
1500 - movegen-v2</h2>
I think I have to commit to the reorg now, but some things have changed which makes the above not quite</em> right. I
started to change it and I think I’m just going to get in and start moving things around on a clean commit of this
branch. After that I think I’m going to merge and go back to working on main. It turns out I’m not so great at sticking
to a single topic on a side project so branching isn’t really what I need to do here, at least not till I’ve cleaned up
some of the tech mess I’ve left for myself.</p>
Next update will hopefully have all that done and I can document it here.</p>
Also on the list is going to be getting mkdocs</code> set up for a wiki, and maybe a local docserver, if only to motivate me
to document more.</p>


12-OCT-2024
2024-10-12T00:00:00+00:00
2157</h2>
I’ve been having an issue on DOP (and BTG, for that matter) where after running an in-place update via just deploy dragon-of-perdition</code>, for instance, it will restart some part of the networking stack and then fail to bring it back up.
Killing my SSH connection and making it a real pain to return to function.</p>
I think this has to do with the silly ‘bond everything’ thing I did, but I like the silly thing so I’ve just gotta
figure out how to fix it and I didn’t want to forget.</p>
1335</h2>
I’ve been doing some work on the the nvim config today, getting DAP set up and generally tuning up the editor, which
I’ve neglected for a bit now while I got used to all the changes I made the last time I was stuck on other things.</p>
I’m thinking I need to adjust all my keybindings to be more consistent and more focused. Right now I manage them poorly,
and there is no mnemonic consistency to them. I do prefix most of them with leader, and I generally have 2 ‘kinds’ of
bindings, <leader><something><leader></code> and <leader><something><space></code>. I can ‘arpeggiate’ these, so I don’t have to
hold them all down (chord), but if I type them quickly, it feels like one ‘action’. I like commands to alternate between
hands, as well, so ,l,</code> is bad, because I hit ,</code> with my right hand, and then have to hit l</code> with my right as well,
which is uncomfortable.</p>
What I’d like to do is establish two kinds of commands with two separate leader keys. A “Right leads” uses my current
leader and starts and ends on the right side with the leader, a “Left leads” uses a different leader key and starts and
ends on the left hand. The third kind of command would be a ‘double key’ command, like <leftleader>ll</code> or
<rightleader>dd</code>, these need to be reserved for the most common commands, since they are by far the easiest to type so
long as I ensure that the keys are on opposite sides of the keyboard.</p>
The goal is to have commands grouped mnemonically both by letter (i.e., the first letter of the command should be mapped
consistently to some subfunction, so all my telescopes might be <rightleader>t<something></code> (where <something></code> might
be several characters), and my Debugger commands file under <rightleader>d<something></code>. Ideally I also</em> sort command
‘kinds’ by hand, so that, maybe, all the commands for altering editor state (e.g., opening files or whatever) are ‘left
hand’ commands, and all the commands for inspecting the state of the object being edited (e.g., running tests, checking
coverage, etc) are ‘right hand’, not sure how to break that up yet.</p>
This, however, will make it easier to remember commands and build up a little cheatsheet, at the moment I routinely have
to poke through my code to figure out where it all lives.</p>
This should also make it a little easier to build up some library functions for this kind of thing. Something like:</p>
{</span></span>
    rightleader</span> =</span> {</span></span>
        d</span> =</span> {</span></span>
            "</span>o</span>"</span> =</span> "</span>lua require('dap-ui').toggle()</span>";</span></span>
            "</span>c</span>"</span> =</span> "</span>lua require('dap').continue()</span>";</span></span>
            "</span>n</span>"</span> =</span> "</span>lua require('dap').step_over()</span>";</span></span>
            "</span>i</span>"</span> =</span> "</span>lua require('dap').step_into()</span>";</span></span>
            "</span>o</span>"</span> =</span> "</span>lua require('dap').step_out()</span>";</span></span>
            "</span>d</span>"</span> =</span> "</span>lua require('dap').toggle_breakpoint()</span>";</span></span>
        };</span></span>
        f</span> =</span> {</span></span>
            "</span>f</span>"</span> =</span> "</span>Telescope find_files</span>";</span></span>
            "</span>b</span>"</span> =</span> "</span>Telescope buffers</span>";</span></span>
            # ...</span></span>
        };</span></span>
    };</span></span>
    leftleader</span> =</span> {</span></span>
        e</span> =</span> {</span></span>
            x</span> =</span> {</span></span>
                modes</span> =</span> "</span>nox</span>";</span></span>
                action</span> =</span> "</span>some action</span>";</span></span>
            }</span></span>
        }</span></span>
        # snip</span></span>
    }</span>;</span></span>
}</span></span></code></pre>
If the value on the RHS of the assignment is a string, it’s interpreted as the action</code> with default mode and all that,
if the value is an attrset, it must be the submodule specifying modes/actions/etc.</p>
by default, it will assume the command is for nox</code> and that it’s a normal mode command ended with <CR></code>. I can then
build this object as a nix module, then use it to create the keymap and the cheatsheet. Each plugin can specify it’s own
keybindings locally, which get included conditionally with the plugin.</p>


10-OCT-2024
2024-10-10T00:00:00+00:00
0017 - movegen-v2</h2>
Thinking about the Nota type problems I was can convert internally 
Another option is one called UCI</code>, but with a const into()</code> 
I truly have no idea starting to hurt.</p> 
0100 I think I have a 
I’m going to reorganize 
                                                    It’s a bit easier and interact with. It scripting language/VM everything, I want to subcomponents on the 
I’m not going to evaluator, movegen, duplication I have now.</p> 
  8-OCT-2024
 2024-10-08T00:00:00+00:00

tion</code> idea again. I think there is a different way I can approach it that won’t encounter the having. In particular I can use an associated type to track the format of the notation, so that I between the formats.</p> just to have an unassociated family of types with mutual From</code> implementations. One called Index</code>, and so on, then I can have them all be const impls, and everything in the crate can just use Index</code>, method which converts to the correct type at compile time?</p> if that’ll work, but it makes half sense in my head, and my lack of types for these things is - movegen-v2</h2> better plan now.</p> the repo like this:</p> data-lang="plain"></span> src/</span></span> board/</span></span> bit/         # Implementation of a bitboard-based representation. Not the bitboard type itself.</span></span> piece/       # Implementation of a simpler array-of-objects representation.</span></span> planned/     # ... there may be other representations I put here eventually as well.</span></span> mod.rs       # Interface definitions and module stuff, Alter and Query live here.</span></span> notation/</span></span> # Some kind of notation management object, ideally mostly const-time, as a QoL thing. Much</span></span> # easier to have a consistent way to represent notation, instead of the mix of octal and text and coords.</span></span> index/    # Index-based move notation</span></span> uci/      # UCI Move notation</span></span> san/      # SAN Move notation</span></span> fen/      # FEN gamestate parsing and generation</span></span> pgn/      # PGN game history parsing and generation</span></span> mod.rs</span></span> move/</span></span> rep/</span></span> # This contains various compact move representations, and probably also the `alteration` type and it's</span></span> # friends.</span></span> gen/</span></span> # Async Move generator, legality checker, etc.</span></span> game/</span></span> # This tracks the gamestate, it's home to:</span></span> halfply/    # Represents a single move/</span></span> line/</span></span> variation/</span></span> mod.rs       # The Game Representation object itself.</span></span> interface.rs # The `Chess` trait goes here</span></span> evaluator/</span></span> # evaluation here, someday</span></span> brain/</span></span> # Async Hazel Engine</span></span> engine/</span></span> # UCI interface</span></span> uci/</span></span> driver/</span></span> stockfish.rs # Wrapper around stockfish, for integration testing</span></span> hazel.rs     # Hazel Engine implementation wrapper (wrapping brain.</span></span> mod.rs</span></span> mod.rs</span></span> ui/</span></span> model/</span></span> widgets/</span></span> app.rs</span></span> mod.rs</span></span> types/</span></span> # generic/crossfunctional types that are used everywhere.</span></span> bitboard/  # bitboard type implementation</span></span> pextboard/ # pextboard type implementation</span></span> mask/      # async process wrapper.</span></span> constants/</span></span> # Constants used throughout the program</span></span></code></pre> to see the structure of the thing that way. The ‘brain’ is the main process that the UI will spawn will spawn the other components and maintain the engine state. I’d like it to have a little sort of thing to control it via the UI in richer ways than UCI would allow. Ultimately, like make it’s parts hotloadable so I can leave it running all the time and just reload the fly. Everything must scale to infinite machines, uptime must grow.</p> do this till I finish the new movegen stuff, which should get all the raw materials above (mod the and brain bits, I suppose) in place, and then I can start to reorganize and clean up some of the

1827 - movegen-v2</h2>
Got some initial work done getting a better game-tracking structure in place (HalfPly</code>) and tying it to the existing
Move</code> struct. I also was able to abstract away a lot of the board rep stuff behind some interfaces.</p>
Remaining is to organize the various move</code> related code and start tearing out the old crappy stuff and building
something better. I briefly took a look in notation-spike</code> at doing some const-time notation stuff, but I’m stymied by
the half-implementation there.</p>
I’ve got plenty to do on this side now that the HalfPly work is done, I can start to build up the Line</code> struct and
build up to something that’ll replace the current Game</code> struct.</p>
Once that’s done, I can start work on a movegen/legality checker. I think I want to build this as a collection of
small services, one to generate all moves (no legality checking), a second to check for legality independently, and
later evaluation services that can provide various evaluations, these evaluation tools can be tagged and run
independently to compare them to each other in situ. The main process will spawn a process which has the engine backend
and the some frontend that tracks boardstate, options-per-engine, etc. The main process can then spawn whatever child
processes it likes based on the request from the Engine.</p>
It makes sense in my head, which means it almost certainly won’t make sense anywhee else, but that’s how it always goes.</p>


8-OCT-2024
2024-10-08T00:00:00+00:00
2140</h2>
Full disclosure, I scrapped an old entry that was incoherent, I was jotting down a TODO List and not making a lot of
sense.</p>
I got a P40 installed in BTG, and I’m working on figuring out if I can set it up as an MDEV or if I just need to
passthru to randy</code> and call it. I’m leaning toward the latter. I have a 2080 also hanging around that I’ll install once
the P40 is working. Between these I should have plenty of GPU horsepower to insource my LLM usage and also get a proper
dev VM built for both randy</code> (for normal work) and hardie</code> (for security/pentesting).</p>
I’m hoping that if I can get the MDEV set up, I’ll be able to run a few VMs backed by at least the P40, ideally it’s the
main ‘development’ card; and then I can use later hardware for the actual compute task on DOP.</p>
Spricht, DOP is getting set up to work as a development machine right now, the build times on hazel</code> are getting out of
hand and it’s not like DOP is doing anything else right now.</p>
Besides these minor roadblocks (like having no idea how MDEV works), things proceed apace. I’ve been focusing much more
on hazel</a> recently, so the lab work has slowed. I’m using it as a testbed for some
tools I need to build for brocard</code>. In particular working with ratatui</code> has been pretty fun, and tokio</code> less
terrifying than advertised.</p>


5-OCT-2024
2024-10-05T00:00:00+00:00
0015</h2>
These too:</p>
</span>
const</span> EMPTY</span> : &</span>'</span>static str</span> =</span> " ";</span></span>
</span>
const</span> HORIZONTAL</span> : &</span>'</span>static str</span> =</span> "</span>─</span>";</span></span>
const</span> TOP_LEFT_CORNER</span> : &</span>'</span>static str</span> =</span> "</span>┌</span>";</span></span>
const</span> TOP_RIGHT_CORNER</span> : &</span>'</span>static str</span> =</span> "</span>┐</span>";</span></span>
const</span> BOTTOM_LEFT_CORNER</span> : &</span>'</span>static str</span> =</span> "</span>└</span>";</span></span>
const</span> BOTTOM_RIGHT_CORNER</span> : &</span>'</span>static str</span> =</span> "</span>┘</span>";</span></span>
const</span> VERTICAL</span> : &</span>'</span>static str</span> =</span> "</span>│</span>";</span></span>
const</span> CROSS</span> : &</span>'</span>static str</span> =</span> "</span>┼</span>";</span></span>
const</span> TEE_RIGHT</span> : &</span>'</span>static str</span> =</span> "</span>├</span>";</span></span>
const</span> TEE_LEFT</span> : &</span>'</span>static str</span> =</span> "</span>┤</span>";</span></span>
const</span> TEE_DOWN</span> : &</span>'</span>static str</span> =</span> "</span>┬</span>";</span></span>
const</span> TEE_UP</span> : &</span>'</span>static str</span> =</span> "</span>┴</span>";</span></span>
</span>
const</span> TRANSPARENT_ROOK</span> : &</span>'</span>static str</span> =</span> "</span>♖</span>";</span></span>
const</span> TRANSPARENT_KNIGHT</span> : &</span>'</span>static str</span> =</span> "</span>♘</span>";</span></span>
const</span> TRANSPARENT_BISHOP</span> : &</span>'</span>static str</span> =</span> "</span>♗</span>";</span></span>
const</span> TRANSPARENT_QUEEN</span> : &</span>'</span>static str</span> =</span> "</span>♕</span>";</span></span>
const</span> TRANSPARENT_KING</span> : &</span>'</span>static str</span> =</span> "</span>♔</span>";</span></span>
const</span> TRANSPARENT_PAWN</span> : &</span>'</span>static str</span> =</span> "</span>♙</span>";</span></span>
</span></code></pre>
They may be useful again someday.</p>
2143</h2>
I got things cleaned up and merged, I think the board is basically done for now. It’s pointed out a need to extract some
of the UI to configuration, so I can change it more easily. I dislike the unicode pieces a lot, I imagine it’s just the
font I’m using not being particular legible, but I think I want to aim in a direction of sprites rendered using a
terminal image api. This is a little bit trickier, and think my time is better spent elsewhere right now. When I do
that, I’ll spend some time looking at custom font patching as well, as keeping it all text is pretty nice, and I don’t
think it’d be too difficult to make some better looking pieces.</p>
I think next I want to set up an integration test, the goal will be to evaluate some deep perft</code>, and send the games
off to be ‘validated’ by stockfish</code>, ideally in parallel (on both sides).</p>
The idea would require building some better abstractions around Engine</code>s, and would also work towards finishing the
movegen component. I do want to build the Alteration</code> based model; but I think I’m close enough with my initial, naive
approach that it’s worth bringing to completion if I can before starting on the next version.</p>
I do want to get the UI finished soon, but it needs some design review, as the current deeply nested approach is a
little tricky without some distinct ‘main loop’. This should be straightforward to build with tokio</code>, but I need to
take some time to do it.</p>
I’ve been working with worktrees</code> recently; and I’m debating just having three branches running so I can switch between
subtasks. It’ll make the LOG</code> a little weird to maintain though – parallel universes. Maybe I’ll start adding the
branch to the entry? It should be a straightforward merge process.</p>
I suppose I’ll know by the next entry.</p>
2201 - movegen-v2</h2>
Hi from a worktree. This worktree will track work on the ‘movegen-v2’, which is a new approach to tracking gamestate and
generating moves. I expect this branch to merge once over it’s lifespan. I’m running a few of these worktrees
simultaneously, experimenting with the model, we’ll see if it works.</p>


4-OCT-2024
2024-10-04T00:00:00+00:00
2339</h2>
While working on the UI yesterday and a bit today, I thought I was going to have the opportunity to do some extremely
cursed things with unicode variables and stuff, but alas it was not to be. I did, however, make this at one point:</p>
</span>
let</span> table</span> =</span> Table</span>::</span>new</span>([</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>┌</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┬</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┬</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┬</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┬</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┬</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┬</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┬</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┐</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>│</span>", " ", "</span>♜</span>", " ", "</span>│</span>", " ", "</span>♞</span>", " ", "</span>│</span>", " ", "</span>♝</span>", " ", "</span>│</span>", " ", "</span>♛</span>", " ", "</span>│</span>", " ", "</span>♚</span>", " ", "</span>│</span>", " ", "</span>♝</span>", " ", "</span>│</span>", " ", "</span>♞</span>", " ", "</span>│</span>", " ", "</span>♜</span>", " ", "</span>│</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>├</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┤</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>│</span>", " ", "</span>♟</span>", " ", "</span>│</span>", " ", "</span>♟</span>", " ", "</span>│</span>", " ", "</span>♟</span>", " ", "</span>│</span>", " ", "</span>♟</span>", " ", "</span>│</span>", " ", "</span>♟</span>", " ", "</span>│</span>", " ", "</span>♟</span>", " ", "</span>│</span>", " ", "</span>♟</span>", " ", "</span>│</span>", " ", "</span>♟</span>", " ", "</span>│</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>├</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┤</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>├</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┤</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>├</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┤</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>├</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┤</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>", " ", " ", " ", "</span>│</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>├</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┤</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>│</span>", " ", "</span>♙</span>", " ", "</span>│</span>", " ", "</span>♙</span>", " ", "</span>│</span>", " ", "</span>♙</span>", " ", "</span>│</span>", " ", "</span>♙</span>", " ", "</span>│</span>", " ", "</span>♙</span>", " ", "</span>│</span>", " ", "</span>♙</span>", " ", "</span>│</span>", " ", "</span>♙</span>", " ", "</span>│</span>", " ", "</span>♙</span>", " ", "</span>│</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>├</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┼</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┤</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>│</span>", " ", "</span>♖</span>", " ", "</span>│</span>", " ", "</span>♘</span>", " ", "</span>│</span>", " ", "</span>♗</span>", " ", "</span>│</span>", " ", "</span>♕</span>", " ", "</span>│</span>", " ", "</span>♔</span>", " ", "</span>│</span>", " ", "</span>♗</span>", " ", "</span>│</span>", " ", "</span>♘</span>", " ", "</span>│</span>", " ", "</span>♖</span>", " ", "</span>│</span>"]),</span></span>
    Row</span>::</span>new</span>(</span>vec!</span>["</span>└</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┴</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┴</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┴</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┴</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┴</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┴</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┴</span>", "</span>─</span>", "</span>─</span>", "</span>─</span>", "</span>┘</span>"]),</span></span>
</span>
</span>
],</span> Constraint</span>::</span>from_maxes</span>(</span>vec!</span>[</span>1</span>]</span>.</span>repeat</span>(</span>33</span>)))</span></span>
.</span>column_spacing</span>(</span>0</span>)</span></span>
.</span>style</span>(</span></span>
    Style</span>::</span>default</span>()</span>.</span>bg</span>(</span>Color</span>::</span>White</span>)</span>.</span>fg</span>(</span>Color</span>::</span>Black</span>)</span></span>
);</span></span></code></pre>
Which is beautiful in both rendering and absolute piss-elegance. I did not want it to go to waste in /dev/null, so I
have recorded it here.</p>


3-OCT-2024
2024-10-03T00:00:00+00:00
2049</h2>
Working on getting the UI to be ‘reactive’, and my heirarchy is making for a real annoying problem of having to thread
the various bits of state through the UI, it would be convenient but clumsy to have it all in a single object on Hazel,
laying out the UI a little more rigidly there would probably work, I think I could simply render to an internal
‘infinite’ buffer and then grab a XxY view to actually render to the screen.</p>
I think a more elegant way, though, is to make the main UI object an async object and have it run a little message queue
that each widget can listen for events addressed to it on. So the handler would grab the event, parse it based on the
current UI state, and enqueue a message which can be seen by any widget and reacted to. This would allow for each widget
to manage all it’s own internal state, just by subscribing to this message queue. Leaving the object structure to
represent the DOM, essentially.</p>
Did I accidentally re-invent React? I don’t know, just seems logical to me, I’m not a UI programmer.</p>
For the moment I’m just going to eat crow and thread everything through, it’s ugly but it will work.</p>


1-OCT-2024
2024-10-01T00:00:00+00:00
1739</h2>
A little idea I had to improve board management.</p>
I want to have a single MoveType</code> that is respected across boards, and to me, that really comes down to that
Alteration</code> idea I used in the Pieceboard</code> type. In particular, I reduced from thinking about a move as a monolithic
enum like:</p>
pub</span> enum</span> MoveType</span> {</span></span>
    QUIET</span> =</span> 0b0000</span>,</span></span>
    DOUBLE_PAWN</span> =</span> 0b0001</span>,</span></span>
    SHORT_CASTLE</span> =</span> 0b0010</span>,</span></span>
    LONG_CASTLE</span> =</span> 0b0011</span>,</span></span>
    CAPTURE</span> =</span> 0b0100</span>,</span></span>
    EP_CAPTURE</span> =</span> 0b0101</span>,</span></span>
    UNUSED_1</span> =</span> 0b0110</span>,</span>   // NOTE: unused at the moment</span></span>
    ///</span></span>
    /// UCI sends moves in a simplified long algebraic notation that simply specifies:</span></span>
    ///</span></span>
    /// 1. source square</span></span>
    /// 2. target square</span></span>
    /// 3. promotion piece (if any)</span></span>
    ///</span></span>
    /// For Hazel, more is expected of the move metadata, but calculating it requires knowing the</span></span>
    /// game state. This value is used to indicate that the move metadata is ambiguous and needs to</span></span>
    /// be calculated.</span></span>
    /// </span></span>
    UCI_AMBIGUOUS</span> =</span> 0b0111</span>,</span></span>
    PROMOTION_KNIGHT</span> =</span> 0b1000</span>,</span></span>
    PROMOTION_BISHOP</span> =</span> 0b1001</span>,</span></span>
    PROMOTION_ROOK</span> =</span> 0b1010</span>,</span></span>
    PROMOTION_QUEEN</span> =</span> 0b1011</span>,</span></span>
    PROMOTION_CAPTURE_KNIGHT</span> =</span> 0b1100</span>,</span></span>
    PROMOTION_CAPTURE_BISHOP</span> =</span> 0b1101</span>,</span></span>
    PROMOTION_CAPTURE_ROOK</span> =</span> 0b1110</span>,</span></span>
    PROMOTION_CAPTURE_QUEEN</span> =</span> 0b1111</span>,</span></span>
}</span></span></code></pre>
Coupled with some location information, I can instead think of a move as entirely defined by a series of simpler
Alteration</code>s to the board.</p>
enum Alteration {</span></span>
    Remove(Location, Color, Piece),</span></span>
    Place(Location, Color, Piece),</span></span>
    Comment(&'static str)</span></span>
}</span></span></code></pre>
A location is a compact representation of a spot on the board, the Piece is a compact representation of a piece, since
there are 64 squares and 6 possible pieces, and 2 colors, then there these can be tightly packed into a single byte. A
move may be multiple ‘alterations’ e.g., a Capture a black pawn on e4</code> with a Knight by White would be:</p>
Comment</span>("</span>1. Ne4 ...</span>")</span></span>
Remove</span>(</span>e4</span>,</span> Black</span>,</span> Pawn</span>)</span>   // Remove the target piece, the piece metadata here is used for unmoving.</span></span>
Remove</span>(</span>d2</span>,</span> White</span>,</span> Knight</span>)</span> // Remove the source piece from it's original location.</span></span>
Place</span>(</span>e4</span>,</span> White</span>,</span> Knight</span>)</span>  // Place it in it's new location.</span></span></code></pre>
The Comment</code> allows for some clever tricks later. This representation means we can represent the game state as a series
of trivially reversible alterations, and we can use the Comment</code> to act as a checkpoint to know when we’ve unwound
enough. Every board representation can, ultimately, just implement some parser for these alterations, and then we can
decouple make/unmake from the board representation entirely. Movegen would query a representation of the board, generate
a bunch of Move</code>s, which can then be ‘compiled’ as alterations.</p>
It also means that internally, the engine can happily keep a couple different representations in various stages of the
game tree depending on what the engine might need for evaluation. It can have a pieceboard rep, perhaps, for doing one
kind of analysis where the pieceboard is convenient, and so long as it is only working on that analysis, it can keep it
up to date to the exclusion of other reps. Similarly, it can roll forward or back as needed.</p>
This also makes it much easier to maintain UI-friendly represnetations that can benefit from the faster internal
representations. We can pick up the list of alterations and apply them unchanged.</p>
Ultimately this boils down to a BoardRepresentation</code> trait which looks like:</p>
type</span> Move</span> =</span> Vec</span><</span>Alteration</span>>;</span></span>
</span>
trait</span> BoardRepresentation</span> {</span></span>
    fn</span> make</span>(</span>&</span>mut</span> self</span>,</span> alteration</span>:</span> Alteration</span>);</span></span>
}</span></span>
</span>
pub fn</span> unmake</span><</span>BR</span>>(</span>board</span>: &</span>mut</span> BR</span>,</span> alteration</span>:</span> Alteration</span>)</span> where</span> BR</span>:</span> BoardRepresentation</span> {</span></span>
    board</span>.</span>make</span>(</span>!</span>alteration</span>);</span></span>
}</span></span>
</span>
trait</span> MoveGenerator</span> {</span></span>
    // Calculate _all_ (not necessarily only legal) forward moves from the current position.</span></span>
    fn</span> moves</span>(</span>&</span>self</span>)</span> -></span> Vec</span><</span>Move</span>>;</span></span>
}</span></span>
</span>
trait</span> LegalMoveGenerator</span> {</span></span>
    // Calculate only legal moves from the current position.</span></span>
    fn</span> legal_moves</span>(</span>&</span>self</span>)</span> -></span> Vec</span><</span>Move</span>>;</span></span>
}</span></span>
</span></code></pre>
This will make it easy to differentiate between UI/Non-UI reps (e.g., whether it implements MoveGenerator or
LegalMoveGenerator). Meanwhile, !</code> starts to do some heavy lifting because:</p>
impl Not<Alteration> for Alteration {</span></span>
    fn not(self) -> Alteration {</span></span>
        match self {</span></span>
            Remove(loc, color, piece) => Place(loc, color, piece),</span></span>
            Place(loc, color, piece) => Remove(loc, color, piece),</span></span>
            Comment(c) => Comment(c),</span></span>
        }</span></span>
    }</span></span>
}</span></span></code></pre>
For completeness, every move type is represented:</p>
Quiet(src, dest):</span></span>
    src_color, src_piece = get(src)</span></span>
    Place(dest, src_color, src_piece)</span></span>
    Remove(dest, src_color, src_piece)</span></span>
</span>
ShortCastle(color):</span></span>
    if color == White:</span></span>
        Place(6, White, King)</span></span>
        Remove(4, White, King)</span></span>
        Place(5, White, Rook)</span></span>
        Remove(7, White, Rook)</span></span>
    else:</span></span>
        Place(62, Black, King)</span></span>
        Remove(60, Black, King)</span></span>
        Place(61, Black, Rook)</span></span>
        Remove(63, Black, Rook)</span></span>
</span>
LongCastle(color):</span></span>
    if color == White:</span></span>
        Place(2, White, King)</span></span>
        Remove(4, White, King)</span></span>
        Place(3, White, Rook)</span></span>
        Remove(0, White, Rook)</span></span>
    else:</span></span>
        Place(58, Black, King)</span></span>
        Remove(60, Black, King)</span></span>
        Place(59, Black, Rook)</span></span>
        Remove(56, Black, Rook)</span></span>
</span>
Capture(src, dest):</span></span>
    src_color, src_piece = get(src)</span></span>
    Place(dest, src_color, src_piece)</span></span>
    Remove(dest, src_color, src_piece)</span></span>
    Remove(src, src_color, src_piece)</span></span>
</span>
EnPassant(src, dest):</span></span>
    src_color, src_piece = get(src)</span></span>
    Place(dest, src_color, src_piece)</span></span>
    Remove(dest, src_color, Pawn)</span></span>
    Remove(dest, !src_color, Pawn)</span></span>
</span>
</span>
Promotion(src, dest, piece):</span></span>
    src_color, src_piece = get(src)</span></span>
    Place(dest, src_color, piece)</span></span>
    Remove(dest, src_color, src_piece)</span></span>
</span>
PromotionCapture(src, dest, piece):</span></span>
    src_color, src_piece = get(src)</span></span>
    Place(dest, src_color, piece)</span></span>
    Remove(dest, src_color, src_piece)</span></span>
    Remove(dest, !src_color, piece)</span></span></code></pre>
These rely on an assumed internal ‘get’ function that retrieves the piece at a location, but I think that’s too low
level to include in the trait, since get</code> might be different depending on context and how you want to implement things.</p>
In any case, this should (I think) make for a bit of an easier make/unmake system. It may</em> be possible to make it fully
piece/color agnostic, but I haven’t thought it that far through yet.</p>


29-SEP-2024
2024-09-29T00:00:00+00:00
1014</h2>
I’ve added a ‘placeholder’ widget so I can work more from the top-down on the UI. I designed a UI in the ui-sketch.md</code>
and I’m going to start working towards that now that I have a concept of what I want things to look like. My hope is
that I can start with the layout of mostly placeholder</code> invokes. The whole layout expects a static size, and eventually
I want the UI to just be a window on a larger, statically rendered ‘infinite canvas’ type of thing.</p>
I should emphasize that I am not a UI/UX guy and I’m basically making this up as I go, but I think it’ll be cool.</p>


27-SEP-2024
2024-09-27T00:00:00+00:00
0145</h2>
I have foolishly stayed up too late working on this, but I’ve got the UI in a good spot. I think I’ve identified the
mistake in my approach, in particular, I’m trying to extract modules/structs too early, I should start with state living
on Hazel and extract the widgets I need as methods, then abstract the state away in bigger chunks.</p>


26-SEP-2024
2024-09-26T00:00:00+00:00
2034</h2>
I set up a mask</code> util object, but this is going to require refactoring Engine</code> before I can use
it. I think I can create a second trait, AsyncEngine</code>, which wraps Engine</code>, allowing for both.</p>
I think I need to switch to only returning messages one at a time, rather than trying to eagerly
read. All this needs to factor into the UI as well, which is going to be a bunch of work as well.</p>
For now, I think I can progress with the blocking version and try to get the UI ‘working’ to some
extent with a stockfish backend. Thus back to the UI I guess. Logging output and text-to-stdin input
next, I think.</p>


24-SEP-2024
2024-09-24T00:00:00+00:00
1932</h2>
I’ve got things looking a bit more operational now. I’ve refactored the UI a lot to be more aligned
with the tutorial, and I’ve created a skeleton for the datamodel.</p>
I also refactored some of the terminal handling stuff, and chased some bugs in the stockfish wrapper
implementation.</p>
I started on a mask.rs</code> which is going to supercede that stockfish module and be able to manage any
STDIO/unix-y process, which should help with future scaling, but it’s tricky and I’m debating
whether I should just bite the tokio</code> bullet or not.</p>
Next step is to improve things by getting state tracking set up on the Hazel side, since engines
generally don’t have ways to display their boardstate (stockfish being a notable exception).</p>
I think I do have enough now to write a test that tries to ‘play’ random moves (generated by Hazel)
for both sides and tries to see if stockfish and hazel ever disagree (including the PGN used to get
there as well).</p>
I did a little coverage testing and I need to get that automated, so somewhere in here there’s going
to be some github action setup to do.</p>
I think I’m close to mergable, though the UI is total trash, I’d prefer to keep developing on main,
so I’ll probably merge soon, perhaps after taking a detour to chase out that damn en passant bug.</p>
2121</h2>
I chased down that en passant bug, I believe it was a bad metadata provided in the unmake call, but
probably I should be calculating the ‘correct’ metadata from the ‘unmade’ moves metadata. The whole
Ply</code> struct is pretty cooked, tbh.</p>
Anyway, back to main</code> I go.</p>


20-SEP-2024
2024-09-20T00:00:00+00:00
0104</h2>
I’ve roughed in the UI model/viewmodel layout, and also made a quick ‘driver’ that talks to
stockfish over my UCI protocol implementation. I also set up an implementation of Stockfish’s d</code>
command and more generally I think I’m going to work on building up a UCI+ style implementation for
now, with an eventual goal of supporting multiple distinct protocols and transports.</p>
I’m already seeing the UI side of this being separable, but for the moment I think there is some
benefit to keeping it embedded, in particular for building deep analysis tooling into the engine.</p>
The next step is to start building widgets that query against the RaceControl</code> backend. I want to
try to set up threading early in the UI so I suspect the way the widget is managing state might be
subject to a lot of change.</p>


18-SEP-2024
2024-09-18T00:00:00+00:00
1505</h2>
Stuck on a work problem, decided to commit off some work here while thinking about it. Several
things are in flight here but I haven’t been back fore a while because I simply can’t bring myself
to work on UI code. I’m procrastinating even now.</p>
I’m debating whether I want a special ‘Hazel’ widget that can bypass the UCI interface, or if I want
to support a sort of UCI+ interface that allows for custom commands to be passed through, so I can
just focus on using an existing UCI interface. I’m probably going to branch off and work on both,
because making decisions is hard.</p>


2-SEP-2024
2024-09-02T00:00:00+00:00
1355</h2>
https://nix.dev/manual/nix/2.18/advanced-topics/distributed-builds.html</p>
I need to set up a builder-file that I can put into the justfile that uses DOP, maybe a VM on BTG,
Archi, Hedges (eventually), and whatever other hardware seems reasonable to use as a builder. I
don’t think I have hyprland set up correctly to use the cache, so I’m stuck compiling. I don’t – in
principle – hate the idea of compiling it, but it is slow and it’d be nice to have it build on the
giant R730 that is otherwise unoccupied.</p>
https://nixos.wiki/wiki/Distributed_build</p>
That wiki indicates I might be able to specify this in nix directly (rather than through a separate
file), that’d be ideal since I can probably calculate it and have an option set within laurelin</code>
that allows a machine to mark itself as a builder. That’ll move nicely into the gehenna</code> setup</p>
2107</h2>
I think I have an idea for how to improve the way I’m handling cross-flake concerns (e.g.,
calculating DNS or scrapeTargets), these should live in a library provided by minas-tarwon</code> itself,
it can pass along the other flakes for reference, but otherwise be a ‘normal’ nix library. Then I
can set up a just job to start a repl with this library preloaded; and be able to directly interact
with the library; and additionally I can create a ‘run’ job that takes an argument, evaluates it,
and shows the result, just wrapping the nix-instantiate</code> command.</p>
That’ll can be used as plumbing for a nicer CLI tool at some point, but for now it should make it
easier to organize the various scripts that currently just live in the flake.</p>


31-AUG-2024
2024-08-31T00:00:00+00:00
2150</h2>
I’ve been working on getting some more exporters set up, and I am finding a ‘next-domino’ problem as
I roll out the ipmi</code> exporter, I need to deploy each machine with the new config, but once I do
that I need to deploy daktylos</code> to update it’s configuration to search. This made me think about
how the eventual ‘real’ methodology rolls out; I want to be able to develop an abstract
configuration of machines such that the configuration will be the same no matter how it is
deployed, so that I can then specify the resources necessary for a machine to play a particular
role, and then separately have code that tries to efficiently deploy it using some configuration
data. Essentially separating design from deployment and creating something like a ‘high level’
terraform.</p>
This problem – knowing what to deploy when some configuration changes – is very similar to the
incremental compilation problem, and I think it’ll be an interesting thing to build strategies for
in the operations context. For now, I just have to remember to deploy them sequentially; I think
once I set up the CI server I’ll add some simple logic to allow me to trigger common build patterns
with flags or something. Ideally I could have some way to specify what parts of the config changing
it cares about, but that’s probably much more complicated.</p>


27-AUG-2024
2024-08-27T00:00:00+00:00
I got a bunch of things set up in glamdring over the last couple days, I’m hoping to get my workflow
more contained inside of vim itself, so that ultimately I can more or less obviate the need for tmux
and rely only on a vim server. To this end I added neogit</code> (and gitblame</code>, gitgutter</code>, and
diffview</code> to support it), neotest</code>, and toggleterm</code> to my setup. The quake-style terminal is
perfect for what I need most of the time, and once I sort out a good system for backgrounding
processes from it I shouldn’t really need more than the one anyway. A glance of the help did suggest
it is possible to run a couple independent terminals via the plugin.</p>


24-AUG-2024
2024-08-24T00:00:00+00:00
0101</h2>
nix-instantiate</span> --</span>eval</span> -</span>E</span> '</span>builtins</span>.</span>fromJSON</span> (</span>builtins</span>.</span>readFile</span> ./telperion/emerald.city/configs/grafana/dashboards/system-view.json</span>)</span>' |</span> nixfmt</span></span></code></pre>
and</p>
nix-instantiate</span> --</span>eval</span> -</span>E</span> '</span>let template</span> =</span> import</span> ./telperion/emerald.city/configs/grafana/dashboards/system-view-template.nix</span>; in</span> builtins</span>.</span>toJSON</span> (</span>template</span> "</span>daklytos</span>")</span>'</span></span></code></pre>
are my handy snippets of the day, the former I saw on a discourse answer, for how to quickly convert
json -> nix. I have some grafana dashboards I want to templatize, and the first step is getting them
converted. This made short work. Now I took this simple system dashboard and made it into a
template, I can then rattle off a bunch of basic dashboards for each of my services, and then
specialize them and store the specialized versions in the configs</code> directory of telperion to
override it. daktylos</code> uses a library function from laurelin</code> to invoke the dashboards that are
already stored over there, and I’ll build up some helper functions as it makes sense. Once I’ve got
the initial version built, when I specialize them I can import them back into the repo, and
eventually I’ll set up something automated to push them to their own git repo or something.</p>


20-AUG-2024
2024-08-20T00:00:00+00:00
1738</h2>
I rigged up vpn routing, but ran into an issue where because the containers now routed through a
VPN, they cannot see my local DNS for local DNS queries. I think it should be possible to bypass
this but I don’t really want to at the moment and the whole setup needs some more significant
refactoring anyway, so for the moment I’m just going to turn it all back off but leave my changes in
place.</p>
All this happens in narya</code>, though, so it’s not really visible to anyone but me. Maybe someday I’ll
open source that bit but for now the mystery shall remain.</p>
Independently and in addition, I stood up daktylos</code> in the emerald.city</code> domain to be the
loki/prom server, and wired up exporters for nginx, systemd, and the node</code> exporter. There is
definitely some overlap to unpack for these, and I’m beginning to dislike how fat telperion</code>’s
flake is growing, but I think the plan will be to move the logic to laurelin</code>’s library, then have
a leaner telperion</code> library which does the various calculations. Domains eventually should be
flakes unto themselves, I think, so for the moment that stuff can stay put, but the scrapeTargets</code>
key is definitely in need of refactoring.</p>
I also added additional horsepower to barge</code> while I work on getting things configured, I really
don’t care for the *arr</code> apps, they feel bloated and it’s odd to have them split the way they are,
but there doesn’t seem to be any reasonable alternative out there, so I think I’m stuck with them
for now.</p>
Remaining, I need to get my dev VMs built, and I need to start thinking about NFS mounting the nix</code>
store, the two things are not unrelated, but it is a big job so I might take a break from this for a
bit after I finish the work I’ve got going now.</p>
1821</h2>
I’m exhausted. There are performance issues to chase down, and things to fix, but I’m tired so I’m
going to put this down for a few days, I think.</p>
2049</h2>
One last thought before I put things down, I suspect the majority of my problem is that barge is
overloaded with IO, network, and cpu intensive tasks. I spent a little time setting up some panels
so I could better diagnose, and this not to say I didn’t see it coming, I think if I split out the
rev-proxy into it’s own VM that’s shared between daktylos</code> and barge</code>, and then split the
torrent/vpn component into it’s own VM(s), that should get me much closer to where I want to be. I
also suspect at least some of it has to do with the way networking is setup on BTG; since all the VM
traffic has to go in and out through the same pipe, things are simply not very quick when I’m
saturating said pipe. I may temporarily create a secondary bridge that can be dedicated to the
torrent side.</p>
I also suspect I’ve got the CPUs configured in a poor topology that is confusing things and perhaps
slowing them down a bit. I should also review and ensure all the relevant kernel modules and such
are loaded – I am at least two layers of virtualization deep, I suppose I could also try running
these as containers directly on the host, but I’m inclined to try to get this working (as eventually
I plan to run all this on k8s anyway).</p>


19-AUG-2024
2024-08-19T00:00:00+00:00
0006</h2>
I’ve got loki running, promtail modules roughed in, and I just need to get the prometheus exporters
set up the same way. I’ve set them to opt out, since I’d rather have the logs then remain dogmatic
to the ‘explicit-is-better-than-implicit’ principle.</p>
I like how promtail converts the nix code directly to JSON, and I need to steal that for Dashy, as
that gets me exactly what I want, each service can automatically provide a dashy snippet that I can
opt into on a per-dashy-instance basis.</p>


18-AUG-2024
2024-08-18T00:00:00+00:00
1013</h2>
As I work on getting the remaining barge</code> services running, I’m thinking about the CI side of this
and how to keep all these machines updated.</p>
I could set up hydra</code>, but I think that’s overkill for my use right now; it is nice that it’s a
bespoke ‘build nixos images’ kind of tool, but I think I can probably get most of the value with
less setup using some laminar</code> scripts that introspect on the minas-tarwon</code> flake; and have it
periodically rebuild each of the canon</code> images as vm</code>s into a centralized nix store, then if the
target machines mount /nix</code> over nfs</code> and are otherwise mostly static (i.e., they don’t have any
local OS state stored on whatever disks they may have), then it should be possible to verify the
image in CI, then automatically deploy when those images work alright.</p>
Independently, I need to set up a metrics machine, and I think I’m going to create a second,
dedicated VM for it, and probably just set up the services using the other, ‘just give it a
nixos-config’ style container, more as an experiment than anything else. I should be able to re-use
the definitions in laurelin</code> inside these containers, which would make it easier to ‘promote’ them
to full VMs if I wanted to, which is kind of cool – same configuration in multiple locations.</p>
I’d also like to experiment with microvm.nix</code>, which I’ve read a bit about, and more generally with
microvms.</p>
Lastly, I’ve removed elenta</code> for now, as I’m not really doing anything with it and I don’t think
it’ll be necessary for a while. Right now building in place is easier, and working out the
deployment strategy for multiple domains is a problem for future Joe.</p>


17-AUG-2024
2024-08-17T00:00:00+00:00
1346</h2>
Moved dashy to a raw YAML config in the repo, I’d like to next move this to a mashable nix-based one
using the existing nix->yaml conversion stuff, so I can have various parts of the infra
automatically contribute their dashy configs when dashy is enabled.</p>
</span>
{</span></span>
    pageInfo</span> =</span> {</span></span>
        title</span> =</span> "</span>Emerald.City</span>";</span></span>
        description</span> =</span> "</span>Off to see the wizard</span>";</span></span>
        navLinks</span> =</span> [</span></span>
            {</span> title</span> =</span> "</span>Github</span>";</span> path</span> =</span> "</span>https://github.com/jfredett</span>"; }</span></span>
            {</span> title</span> =</span> "</span>NixOS</span>";</span> path</span> =</span> "</span>https://nixos.org</span>"; }</span></span>
            {</span> title</span> =</span> "</span>Dashy Docs</span>";</span> path</span> =</span> "</span>https://dashy.to/docs</span>"; }</span></span>
        ];</span></span>
    };</span></span>
</span>
    # ... snip for other sections</span></span>
}</span></span>
</span></code></pre>
Should translate naturally, then the section</code> section is just an attrset of section-name -> other
config.</p>


15-AUG-2024
2024-08-15T00:00:00+00:00
1243</h2>
Got some stuff reorganized and pushed up. I need to build an installer image I can have randy</code> boot
off of, so I can install on the ‘real’ disk and get to something functional, I also need to set up
backups on just the randy image independent of other things on nancy</code>’s side. So some kind of
rsync</code> handoff to network storage, etc.</p>
1746</h2>
I think next step is a grafana/prometheus/loki stack, I’m tired of having to jump all over creation
to find logs.</p>


12-AUG-2024
2024-08-12T00:00:00+00:00
2033</h2>
Got some more stuff working w/ barge, needed to add recommendedProxySettings</code> to each of the
proxied items. I still need to setup outbound VPN for these as well.</p>


11-AUG-2024
2024-08-11T00:00:00+00:00
1512</h2>
Working on the hostkey problem, I’m starting simple, just getting it set for the non-persistent
(i.e., netbooted) machines, I also include a ‘generic’ one that I can use in case I don’t want to
generate one right away (more for testing new netbooted machines).</p>
I’m planning to try for a “Nix Store over NFS” approach since my netbooted machines should be more
or less RO. I think that should make ‘most’ of the infrastructure non-persistent; though I haven’t
worked out exactly how I’m going to do the complicated dance there.</p>
One upside of putting all my hostkeys in a single bucket is I should</em> be able to pregenerate a
‘known-hosts’ file which I can actually trust to some extent, so that if I get an ‘unknown host’ it
would actually indicate something has gone awry.</p>
At time of writing I’m just finishing up the experiment with pinky</code> and will roll out the change
once it’s ready. I’ve kept all the keys in narya</code>, so they’re out of the way for the moment.
Eventually these’ll move to Vault and be pulled in via turnkey</code>, so they won’t need to be in any
repo.</p>
This is also nice because it makes rotation pretty pleasant; I can just generate a new key, update
the relevant machines, and (eventually) regen my knownhost file appropriately.</p>
2313</h2>
Got storage pools set up, I think I need to write something to sync the xmls back and forth, since
the easiest way to manage these right now is just to copy the xml back and forth and do the
adjusting in virt-manager.</p>


10-AUG-2024
2024-08-10T00:00:00+00:00
1419</h2>
Working on the DNS issue with toto, I’ve set it up so all of KANSAS is bypassing my internal DNS so
that normal operations are maintained, and only my laptop has DNS issues, while my main machine
(Hedges) is not so-encumbered, that way I’m able to work freely on toto</code> without knocking everyone
offline, maintaining the most important invariant my lab must maintain:</p>

Do not interrupt the Bluey.</li>
</ol>
I think the issue is probably with the new NSD stuff, I can still resolve some .canon</code> addresses
but I suspect that’s just cache.</p>
It’s always DNS that gets you.</p>
2251</h2>
This turned out to be a simple bit of misconfiguration, using .canon</code> instead of .emerald.city</code>,
and a missing port open.</p>
I like it when it’s simple.</p>


7-AUG-2024
2024-08-07T00:00:00+00:00
(and a bit of 6-AUG too)</p>
0141</h2>
I’m doing something wrong with the filesystem definitions, I need to rebuild the pool again.</p>
I got the script using wwns to make sure it persists, but that wasn’t why the thing was breaking on
boot, to be honest, I’m not sure why it’s breaking on boot, but I think I might be able to skirt all
of this because of this</a>, but that needs a
rebuild.</p>
1142</h2>
I’ve got it mostly working with the automount, but I’d still like to understand what I was doing
wrong w/ the filesystem definitions; probably I needed to set different options or something, the
emergency shell never managed to start, so it’s hard to figure out exactly what went wrong.</p>
In any case, I like the way I laid out the disks and want to extend the idea of the cadaster in that
way, building a large nixos-style module system that describes all my hardware (and maybe integrates
with the nascent nixos facter</a> tool) and then can be used
to populate other parts of the configuration by reference seems pretty handy.</p>
Ultimately I think it also points at a way out of the nix ecosystem while still maintaining the
‘single common configuration system’ that I like about NixOS. There’s no reason to couple it to Nix
besides the convenience of the package manager, a declarative description of one’s hardware and
virtual hardware architecture is useful in it’s own right and could ostensibly be compiled to
instructions to configure virtually anything from a known starting state. My interest in it is
actually simpler, to boot, I just want to be able to simulate arbitrary architectures and employ
some kind of static analysis to surface issues at ‘compile time’; I think getting anywhere near that
for architecture design could be extremely valuable.</p>
1532</h2>
I’m going to do some tweaking to the cadaster</code> idea, I’m going to split it between narya</code> and
telperion</code>, two items with the same underlying schema, both provided as flake outputs, telperion</code>
will pull in and merge narya</code>’s definitions into it’s own at build time, but will otherwise be
unaware, that structure will then be available to all subsequent definitions. I’ll probably just
keep the hardware defs in telperion</code> for now, but they may be extracted to laurelin</code> later if that
seems prudent.</p>
The structure will be something like:</p>
</span>
{</span></span>
    compute</span> =</span> {</span></span>
        "</span>hostname</span>"</span> =</span> {</span></span>
            # metadata</span></span>
        };</span></span>
        # ...</span></span>
    };</span></span>
    storage</span> =</span> {</span></span>
        disks</span> =</span> {</span></span>
            "</span>diskname</span>"</span> =</span> {</span></span>
                # metadata</span></span>
                # includes a reference to `cadaster.compute.hostname` for where it's installed</span></span>
            };</span></span>
            # ...</span></span>
        };</span></span>
    };</span></span>
    network</span> =</span> {</span></span>
        switches</span> =</span> {</span></span>
            "</span>switchname</span>"</span> =</span> {</span></span>
                # metadata</span></span>
                # Should include port maps and the like.</span></span>
            };</span></span>
            # ...</span></span>
        };</span></span>
        routers</span> =</span> {</span></span>
            "</span>routername</span>"</span> =</span> {</span></span>
                # metadata</span></span>
            };</span></span>
            # ...</span></span>
        };</span></span>
        firewalls</span> =</span> {</span></span>
            "</span>firewallname</span>"</span> =</span> {</span></span>
                # metadata</span></span>
            };</span></span>
            # ...</span></span>
        };</span></span>
    };</span></span>
    racks</span> =</span> {</span></span>
</span>
    };</span></span>
    power</span> =</span> {</span></span>
</span>
    };</span></span>
    # ...</span></span>
}</span></span></code></pre>
The idea is to represent each physical item in this with all it’s metadata, but none of it’s
configuration. Configuration then consumes this to build the actual configuration, and the
resulting configuration is then used to build the actual system images.</p>
Essentially it’s DCIM-as-code. I’m sure this has been done before, but reinventing wheels is fun.</p>
I’m going to build each chunk as nixos modules, and then I can use that to attach additional
computed information to each section. Ideally I can use this to drive some sort of
testing/monitoring tools as well.</p>


5-AUG-2024
2024-08-05T00:00:00+00:00
1144</h2>
I’ve got the format-disks stuff wrapped up in some Nix code now, I need to extend this into a
module, but essentially it just writes the format-disks</code> script based on the Attrset I feed it; it
should be extensible to other machines, but does expect to be able to wipe all specified disks so
it’s still an imperative management strategy, I doubt there is a good declarative one for disks
since they are essentially entirely made of statefulness.</p>
In any case, I plan to manage these disks imperatively and track changes via the CHANGELOG in the
cadaster, so it’s not a big issue.</p>
Remaining on the TODO for this is:</p>

Backups to Nancy (and thus to cloud)</li>
Porting all the VMs over to use the disk I’ve set up</li>
Setting up the NFS store on BTG</li>
</ol>
2331</h2>
I started moving the ZFS configuration into a dedicated module. This is pretty unnecessary but I
wanted to start it and leave a branch to come back to later. I set up the existing configuration
with a simple ‘mode’ flage to control whether it leaves the ‘prepare’ script in place, or if it just
tries to mount the datasets. I’m not automatically calculating the filesystems, but I think that
will be easier to do when it’s properly a module and not just the shim.</p>


3-AUG-2024
2024-08-03T00:00:00+00:00
0026</h2>
Had to revert toto</code>, I think I remember having an issue where it would serve DNS without issue for
a while, then start refusing connections thereafter. I remember going to unstable</code> helped with
that, but I don’t remember the details enough. I think there is a way to simply make NSD do the pass
through, but I need to go do some reading to remember how.</p>
2036</h2>
I have run into an issue with my netboot setup, which points to a more general issue I had been
punting off until now.</p>
I have a bunch of machines I’d like to netboot that, ultimately, really shouldn’t be netbooted.
“Standard” (non-EFI) netbooting has a hard limit at a 4GB Ramdisk, and tbh, I don’t want to go
beyond that generally since things get quite memory hungry quickly, so while moving to an .efi-based
boot system is probably ‘good’ in the long term, it helps entrench a bit of bad design I admitted
while building things.</p>
Now I’m caught out a little earlier than I wanted.</p>
Initially I had planned to build gehenna</code>, a machine that would be a nix-store-over-NFS server, but
I read that this was nontrivial and projects like tvix</code> exist but aren’t yet ready for production.</p>
Ideally the system would work thus:</p>
Boot every machine to a common nix image that is stripped to the barest essentials, and contains
only one post-boot script that asks it to be configured by some remote configurator. This base image
would mount a nix store over NFS specific to that machine, so that it can then be treated exactly as
if it were a ‘regular’ machine with a persistent nix store, but where that store is actually located
on the far side of a network connection. This has an obvious duplication issue (different machines
will likely share the same nix store objects, so they will be duplicated across the network), but I
figured:</p>

Probably whatever native dedup happens would be sufficient to minimize the pain</li>
I don’t really have a big enough architecture in mind to warrant fixing that storage problem
until I have it.</li>
</ol>
Then I read this</a> which made me think I could do something
about halfway between what the author does and my idea. A main .ro store with a .rw store per
machine mounted over it, I didn’t actually realize this was possible with NFS, but it actually seems
pretty straightforward. The author there uses a tmpfs, but there is no reason I couldn’t use a
persistent store if I preferred, then I could factor out common values periodically into the .ro
side to minimize duplication.</p>
I think this is a good idea, but I think it’s going to take a pile of work to get right, so I’m
going to focus for now on getting BTG formatted with ‘real’ storage so I can carve off a chunk for
my dev VM and do things ‘the old fashioned way’ with a real backing disk, but I definitely need to
do some more research on how to approach building gehenna</code> and the NFS store.</p>
I really like the idea of a sort of ‘stem cell’ VM that I can differentiate simply by mounting the
right underlying store and doing some command at it to ‘activate’ whatever configuration lives on
the other side of that, I don’t think such a thing would be too difficult. I could also see having a
simple .ro-only approach and having builds happen entirely on the gehenna</code> side rather than the
client side, design is needed.</p>
2102</h2>
A brief thought on the UEFI netbooting, in particular, this solves a major defect in terms of
security, BIOS netbooting (what I’m doing now) is only over insecure protocols, fine for a closed
lab, bad for anything that might ever be exposed to the internet, UEFI netbooting can be done over
HTTPS, so definitely needs doing, for now I have BTG and DOP both standing up on a netbooted image
via BIOS netbooting, so hopefully that unblocks me and I can return to UEFI later.</p>


2-AUG-2024
2024-08-02T00:00:00+00:00
2335</h2>
I’m working on committing off a bunch of work, I solved the DNS issue without adding a second phase,
it ultimately boiled down to needing to split and calculate each zone via the nixosConfigTree</code>
instead of using the nixosConfigurations</code> directly.</p>
I have an annoying reversal that happens when calculating the nixosConfigurations</code> that results in
the domain coming first – I haven’t fixed that yet but it’s so ugly I feel obligated to mention it
in the log.</p>
I refactored some of the deploy</code> scripts but just</code> is really not up to what I’m putting it
through, I think. In particular the nix flake update</code> logic is a bit of a mess and often runs a
couple times. This isn’t a huge issue, but it burns time; I suppose moving development to a VM will
help with that, but that would require I actually get that built, which isn’t as interesting as
chasing infinite recursion bugs.</p>
In any case, I’m getting toto</code> updated tonight, which unblocks barge</code> and puts me back to where I
was with ereshkigal</code> before cutting over. This whole process took about a month, but I think it was
a good move, except that I truly hate dealing with multiple git repos like this. I’ve looked at
using worktrees</code> and they aren’t quite</em> what I want.</p>
Really what I want is a monorepo that isn’t a monorepo. I want each project to have an independent
history, but I also want to be able to tie commits together across repos. Right now when I make an
update in telperion</code>, nothing coordinates it with laurelin</code>. I can</em> do this through flakes (make
each flake track via the remote git repo, update to specific SHAs, then do everything the ‘intended’
way, as I understand it), but this, plainly, sucks. Iteration on a change becomes difficult when
building it, and while it is very handy to create ‘canonical’ deployments of a suite of tools, it
isn’t great for hacking on a lab, where purity isn’t as valuable. I still want to be able to track
that compatibility information, but most of the time I want to do it outside of my actual IaC; and
instead have my ‘monorepo’ actually just be a bunch of independent git trees in the same git
database</em>, but with an additional way to tie two commits together when they are ‘compatible’.
Ideally also allowing that tie to be revoked if a later discovery shows they aren’t compatible.</p>
The idea is still a little nascent, but I think this is doable in the context of git</code> but with
different semantics to the standard git</code> repo.</p>
In any case, I need another project like I need to increase my topological genus, so I’m just going
to focus on standing up some services and getting btg</code> sorted.</p>


1-AUG-2024
2024-08-01T00:00:00+00:00
0112</h2>
Ran into an infinite recursion with my DNS generation, by referencing it directly, it has to query
it’s own configuration to calculate it’s DNS. I think this only happens for the canon</code> domain
because of how I have split up the options. Since the canon</code> domain is calculated from the configs,
and the config of the dns server wants to include the canon</code> domain, it has to calculate it’s own
config and down the rabbit hole went Alice.</p>
I think this is straightforward to fix, but I lack the will tonight. I can rely on precomputed files
and make this a compilation step that just runs when I run a deploy. It’s relatively quick to
calculate statically, then I can refer to the static copy. That also make it easy to inspect when
there are issues, and solves the ‘serial number in the git repo’ problem I haven’t yet mentioned
(the idea of trying to remember to maintain that number will result in many many many tiny commits
full of more swear words than delta).</p>


29-JUL-2024
2024-07-29T00:00:00+00:00
1748</h2>
Over the weekend, I mostly refactored and tweaked things, and got DOP + it’s VMs up and netbooting.</p>
I ran into an issue where building a bridge over a VLAN-split bond connection didn’t seem to work. I
spent almost no time debugging in it favor of getting the lab back to a working state; and just
pulled an interface to dedicate to it. I suspect this is mostly a switching problem (enabling trunk
ports or w/e), but I want to converge both the WiFi and LAN networks eventually and split everything
via VLAN instead, so there’s no point in spending time on it now when I’m just going to radically
change it later.</p>
barge</code> and pinky</code> are both building but need a bit of work on their networking setup, eventually
I’ll pull that config into laurelin as a module; I’m thinking about how I want to organize the
domains and get more of the libvirt config ported back into nix; but the idea is hazy right now, I
think ideally the individual networks per domain will have some nix</code> representation that I can pass
in, so the function’d be something like:</p>
laurelin</span> =</span> {</span></span>
    networks</span> =</span> [</span></span>
        {</span> link</span> =</span> self</span>.</span>domains</span>.</span>"</span>foo.bar</span>"</span>.</span>networks</span>.</span>"</span>foo-net</span>";</span> ip</span> =</span> "</span>1.2.3.4</span>"; }</span></span>
        {</span> link</span> =</span> self</span>.</span>domains</span>.</span>"</span>foo.bar</span>"</span>.</span>networks</span>.</span>"</span>foo-net2</span>";</span> ip</span> =</span> "</span>1.2.3.5</span>"; }</span></span>
        {</span> link</span> =</span> self</span>.</span>domains</span>.</span>"</span>foo.bar</span>"</span>.</span>networks</span>.</span>"</span>foo-net3</span>";</span> ip</span> =</span> "</span>1.2.3.6</span>"; }</span></span>
    ];</span></span>
}</span>;</span></span></code></pre>
This would then use the content of the foo-net</code> attrset to populate all the networking config. It
should</em> be possible to re-use this model for ‘real’ networks, but I’ll probably limit it to virtual
networks to start. Ideally I can work out my abovementioned issue in virtuo and then port back.</p>
I can do a similar thing for the domain</code> side of things, have a few ‘standard’ domains, then set a
key in laurelin</code> just for data’s sake, to be used when deploying the config to a particular
host.</p>
Unrelated to that, I started working on setting up neotest</code> and want to start trying to get more of
the workflow into vim</code> so I need to jump between fewer windows. tmux</code> makes it hurt less, but I
want to have the lab integrated into the editor so I can more fluently manage, e.g., rebuilds and so
on. I think that comes on the back of setting up a laminar server and moving some of the
build/deploy stuff there. That also means moving towards purity/no more --impure</code> builds, which is
it’s own can of worms.</p>
Finally, I need to get some way to provide DNS includes, maybe creating a more abstract ‘reverse
proxy’ module that both configures an nginx service but also generates a zone-file w/ appropriate
CNAMEs would work here? Something like:</p>
laurelin</span> =</span> {</span></span>
    services</span> =</span> {</span></span>
        reverse-proxy</span> =</span> {</span></span>
            "</span>parent.domain</span>"</span> =</span> {</span></span>
                "</span>subdomain</span>"</span> =</span> {</span></span>
                    frontendPort</span> =</span> 80</span>;</span></span>
                    backendPort</span> =</span> 8080</span>;</span></span>
                }</span></span>
            }</span>;</span></span>
            # ...</span></span>
        };</span></span>
    };</span></span>
};</span></span></code></pre>
Then I would have that generate both an nginxconfig and a key that could then be pulled by the
show-dns</code> function in the flake? Ideally this would allow the ‘parent’ to exist as hostname.canon</code>,
then have it populate hostname.parent.domain</code> as a CNAME to hostname.canon</code> and then have the
others similarly populate based on the desired subdomain.</p>
1850</h2>
I may be getting slightly ahead of myself w/ barge, I think I need to write a more general ‘docker
container’ module that I then feed additional items into, perhaps?</p>
I also need a dev machine that isn’t this poor old laptop.</p>


27-JUL-2024
2024-07-27T00:00:00+00:00
0026</h2>
Big pile of stuff done today, I did a ton of refactoring and cleaning, and I did a bit of secret
scrubbing as well. I also ported over all the modules from glamdring, so it should be ready to turn
public on github after a last once-over.</p>
laurelin and telperion are also almost there, which is moderately terrifying, I get anxious about
these things.</p>
Rather than process those emotions, I instead work on getting the domain xmls at least temporarily
loaded with some hacky work in the flake. I also have another hack from the refactoring to pull
common configuration out of the cadaster</code>, a module in telperion</code> where it does not belong. I plan
to move it over to laurelin after I’m done auditing the existing cadaster and pulling all the common
config out. Ideally cadaster</code> is really only the hardware configuration and stuff related to
building the base image. In the end all my machines will be a collection of laurelin</code>, glamdring</code>,
and narya</code> configuration, everything else gets split up.</p>
0042</h2>
Fixed an issue where unfree packages weren’t allowed and that meant 1pass couldn’t build. However,
that got me thinking that it’d be better to export NIXPKGS_ALLOW_UNFREE</code>, since there is no reason
to litter the codebase with unfree tags when I can just export a variable in the devshell once that
I can turn off if I want to see if I’m using anything unfree.</p>
1227</h2>
I just made public all the component repos except elenta</code> and narya</code>, the former because there is
nothing there (and TBH I may collapse it away into minas-tarwon</code> proper), and the latter for
obvious reasons.</p>
I started thinking about making minas-tarwon</code> a bit more of a general monorepo for my projects,
which would point to moving a lot of my existing automation out to elenta</code> and having just
project-management stuff in this repo, but I haven’t really decided on organization yet.</p>


25-JUL-2024
2024-07-25T00:00:00+00:00
2341</h2>
Got dragon-of-perdition fixed up and booting. I also got all his interfaces bonded and running as
one unit. I need to read about the other options, but they all look pretty straightforward.</p>
I also refactored the Justfile a bunch, and got netboot image creation moved into a flake ‘app’.
It mostly works, but is quite messy, so I’m going to think about some ways to clean it up.</p>


24-JUL-2024
2024-07-24T00:00:00+00:00
1445</h2>
I set up canon IPs for both of the DRACs on the two R730s (DOP and BTG), but for whatever reason I
can’t get to them via the DNS address, only by their IP, not a big deal for now, but annoying.</p>
Archimedes is hooked back up to the barrier server, cleaned up some stuff in the Justfile</code> and
started in on the netboot image. I need to figure out what to include in the imports path (if
anything), and then go from there.</p>
2248</h2>
I think I want to rewrite a bunch of the image-building stuff.</p>
The script I have is very clunky, and I think I could pretty easily adapt the existing script in nix
to just work with my setup directly, but I’d have to introduce a couple new ideas.</p>
In particular, There is a sort of ‘hyperparameter’ that I need to introduce, something that is
specific to my setup, namely – where I store the netboot images – but which is not really a detail
that fits in any of the existing flakes in the blizzard, so to speak.</p>
I suppose minas-tarwon</code> is the natural place for such things, but I could also make an argument
that much of what I plan to do in elenta</code> is similar.</p>
I think, however, my idea has evolved somewhat. Where elenta</code> was originally going to be the
flying-monkey</code> replacement, and sort of the operational arm of the thing (with laurelin</code> being the
abstract library from which telperion</code> both describes the physical architecture (the ‘cadaster’,
and also describes abstract configurations of that architecture (‘domains’). I also had intended for
elenta</code> to manage mapping those domains</code> onto the physical architecture.</p>
I think I’m going to purpose elenta</code> solely towards that goal, and pull all the operational stuff
into minas-tarwon</code>, additionally, it can contain the ‘hyperconfiguration’ which can be passed into
scripts and so on (but ideally isn’t referred to in and of the other flakes). The first such element
of the hyperconfiguration is the minas-tarwon.netboot_store</code>, which is some structured
configuration that referents will use to know where netboot images are stored, e.g.:</p>
minas-tarwon</span>.</span>netboot_store</span> =</span> {</span></span>
    path</span> =</span> "</span>/path/to/store</span>";</span></span>
    # ...</span></span>
}</span>;</span></span></code></pre>
I’ll try to encode as much of my scripting to nix, so it can natively refer to things, then wrap
these outputs with a just script.</p>


23-JUL-2024
2024-07-23T00:00:00+00:00
0014</h2>
Got deploys tested, and am now serving the .canon</code> domain on toto</code>. I realized I need to get
ifrit</code> added to the cadaster, and it presents a wrinkle, as it is on multiple networks, and thus
would need multiple canon IPs, since bonding these is not really reasonable since they need</em> to
serve actually indepentent networks in my architecture. My topology splits lab and non-lab networks,
and I want my firewall to maintain that separation, while having, ideally, a single flat network
that stretches across both. Moving the point of failure to my firewall, instead of the lab switch.</p>
This is the theory, and it means that ifrit</code> will need at least two entries. I think that’s okay,
and ultimately it makes convenient the fact that I’m stealing a whole TLD for this. I can just use
subdomains.</p>
!!! CAUTION</p>
This, by the way, is not a good way to use DNS, there are a variety of ways to exploit this kind of
thing if you aren’t careful, but this lab is confined to me, for the most part, and ultimately I’m
not super worried as I do plan to make sure requests for these domains never leave the network. This
can itself be bad because if the domains are ‘real’ (i.e., if you’re split-horizoning a real
domain), then you can block an otherwise ‘correct’ lookup. Generally you’ll be fine if you use weird
TLDs and block all outgoing DNS for those TLDs (which is what I do). You can be much safer by just
buying a domain and using that, but I like punned names and I don’t want to pay for a problem I’m
unlikely to ever actually have.</p>
Each independent network on ifrit can have it’s own subdomain to point to the gateway. This means
that when I start serving multiple domains and layering VLANs, I can have as many subdomains as I
like.</p>
In any case, next step is to get ifrit</code> into the cadaster and update the DNS stuff to allow for
multiple subdomains for ifrit. Then I can update ifrit</code> to point it’s clients at the DNS IP and I
think I’m basically back up and running in the new setup.</p>
1220</h2>
Looking at a new issue, toto</code> has two IPs on the same interface, one is virtual, and iptables</code> nor
nftables</code>-based local firewalls seem to be able to handle this case with the built-in networking
module. I’m not sure I understand exactly how things are getting generated just yet, but my guess is
that this is not a typical way to cut up a network and so the implementation is not robust to my
particular breed of weirdo.</p>
If I turn off the firewall, everything works, and so I know the issue is local to the machine. I
tried just inspecting both the configurations generated by my config and they both look ‘fine’ in
the sense that they should happily be sending packets to the specified interface, so after some
consternation, I turned my attention further up the stack.</p>
The tipoff to this was looking at the dmesg</code> output as I tried to hit port 53 w/ an nmap scan; it
shows that when I run the nmap</code> against that port, it sees the incoming connection as originating
from the enp2s0</code> interface, and not the enp2s0_dns</code> virtual interface. enp2s0</code> does not have 53</code>
open.</p>
I tried opening 53</code> on enp2s0</code> itself and sure enough everything works fine. So for now I suppose
I’ll just have to open the sum of all the ports I want on the enp2s0</code> interface and then later do
something bespoke to handle my weird networking idea.</p>
I’m now seeing what I suspect is a very straightforward “need to turn some ports on” issue on the
switch, so that’s tonight’s project.</p>
1738</h2>
Truly confusing, even with the extra port, it actually just shows the port as open, doesn’t actually
respond correctly. For now I’ve just disabled the firewall entirely, I’m sure I’m doing something
silly, I just can’t track down what at the moment.</p>
2205</h2>
Got the DNS configured for both KANSAS and CONDORCET. I should probably add the wifi router and
cable modem to the cadaster (and really everything in the racks, I suppose it should try to cover
everything that touches my network, at least).</p>
2212</h2>
I also updated NFS rules for nancy. This gets network storage back to functional.</p>


22-JUL-2024
2024-07-22T00:00:00+00:00
0916</h2>
I think I’ve got a better idea on how to organize things.</p>
A deployment of a constellation is a derivation. It’s build phase creates and caches all the
cross-environment artifacts (zonefiles, images, etc), it’s install phase is a script that gathers
those artifacts and deploys them to the target machines and ensures all netboot images are up to
date.</p>
elenta</code> should provide these derivations given the constellation input, and then minas-tarwon</code> can
take those derivations and apply them to the target machines.</p>
Two different derivations are ‘compatible’ if they do not:</p>

Overallocate any underlying machine</li>
Have contradictory configurations (i.e., colliding ports, colliding filesystem
assignments/mounts, etc).</li>
Can be mkMerge</code>d with each other and still have any assertions/tests pass</li>
</ol>
Ultimately this would allow breaking down that ‘main’ derivation into, e.g., a derivation per
machine or however else I want to break things up. It also makes the deployment pipeline
straightforward. “Here are a bunch of constellations that elenta</code> compiles down to a single script
that you run from a provisioning machine to deploy everything specified.” Rollback is just applying
the previous derivation + whatever backup restoration should be done.</p>
Automating a pre-deploy backup can be part of the installPhase</code> of the derivation.</p>
Eventually it’d be cool to have the merge</code> bit be smart and try to create, e.g., VMs and stuff
dynamically, so it just becomes a ‘map machine roles to resources, don’t care about where they are
hosted’</p>
1054</h2>
I think I’m just going to leverage just</code> in the minas-tarwon</code> repo to port over the
flying-monkey</code> stuff. It can do the basic deployment from telperion</code> -> machine config for now.</p>
2317</h2>
I set up just</code> to do deploys now, porting over flying-monkey</code> was pretty easy, I rather like
just</code>.</p>
Next I need to get the canon</code> domain hosted. I just need to see how to refer to my own flake, I am
pretty sure I can just pass it down as a special arg. Not sure.</p>


21-JUL-2024
2024-07-21T00:00:00+00:00
1306</h2>
I was thinking about the constellation file a little more last night, and I think I prefer this
design instead:</p>
rec</span> {</span></span>
    "</span>machine.canon.lan</span>"</span> =</span> {</span></span>
        hostConfig</span> =</span> "</span>null | { literal, module } | import ./path/to/machine-role</span>";</span></span>
        vms</span> =</span> {</span></span>
            "</span>vm1</span>"</span> =</span> "</span>{ literal, module } | import</span>";</span></span>
            # ...</span></span>
        };</span></span>
        containers</span> =</span> {</span></span>
            "</span>container1</span>"</span> =</span> "</span>{ literal, module } | import</span>";</span></span>
            # ...</span></span>
        };</span></span>
</span>
        # arbitrary config available to be placed and used here.</span></span>
    };</span></span>
    "</span>machine2.canon.wifi</span>"</span> =</span> {</span></span>
        # ...</span></span>
    };</span></span>
    # ...</span></span>
}</span></span></code></pre>
The "machine"</code> key is the canon name of the machine, and there are options to configure items at
the host level, to spin up vms, or containers. vms</code> and containers</code> both take literal
configurations and spin up containers/vms on the target. Additionally if these items are non-empty,
it will contribute appropriate configuration to the host to support them.</p>
Canon addresses are also extended to include protocol type, this is something I’m toying with, I’d
prefer to have all interfaces bonded and then have a single logical address, but I’m not sure how
good of an idea it is to bond wireless and LAN interfaces, so I’m toying with separating them at
least at that level.</p>
Unrelated, the issue I saw with toto</code> seems to have resolved itself, so I’m guessing it was just
stale DHCP leases causing trouble.</p>
1410</h2>
One of the things I’m going to want to figure out is some mechanism to figure out when machines need
to be updated to match a specific SHA within the context of a specific constellation. Ultimately I
want a tool that can take a constellation and figure out how to rebuild everything to bring the
constellation into compliance with the config. Not sure how to do that yet, I’m sure there is some
git magic to be done, but I’m not sure what it is yet.</p>
1645</h2>
I’m a little stuck, trying to build up to the constellation/role/machine mapping system is going to
take a lot longer than I’d like, and I’m considering just manually managing the machines directly
for the short term and then extracting configurations out after the fact. This will inevitably leave
a big mess in the cadaster, but I think I can limit it by putting all the bespoke config in it’s own
file and relying on the laurelin modules directly there.</p>
If I do that, then I can develop all the modules, get them tested and wired up, then the
constellation stuff won’t have the issue of needing to have both the modules and wireup system
debugged.</p>
I’m going to go that route, and just add a bespoke.nix</code> into which all the stuff that ‘should’ be a
machine role will go into, then later I can start to unpack that.</p>
This also means that I can build everything around the .canon domain for now.</p>
2132</h2>
Definitely liking the idea from earlier, I’m just placing it in the config.nix</code> for now, but I
should be able to stand up everything pretty quickly this way, I can just use a flying-monkey</code>
style script right now. I have a ton of pending changes to close in telperion</code> and laurelin</code> now,
so I’m going to work on that tonight and then probably take some time this week to start getting the
deployment pipeline working and nfs</code> and dns</code> working for the canon</code> domain.</p>


20-JUL-2024
2024-07-20T00:00:00+00:00
2307</h2>
Overall today I’ve got toto</code>, nancy</code>, and archimedes</code> all on the 10.255.0.0/16</code> network, mostly
manually configured. I need to build the deploy script in elenta</code> and get the netboot image
generation and direct configuration application working.</p>
I saw a recurrence of an old issue where ssh connections hang after some amount of time, I think it
relates to old DHCP leases, and I believe I’ve resolved it.</p>
The other thing I am missing is some tooling to manage the switches. I’m hopeful I can repair the
old switch which died (prompting all this work happening so suddenly, I had planned a more gentle
transition), but I need a way to better manage the switch configuration, so I’d like to add tooling
in elenta</code> to manage that as well.</p>
In the short term I may just port flying-monkey</code> and build-img</code> from ereshkigal</code> and serve them
out of elenta</code>; elenta</code> can pull in narya</code> and telperion</code> and provide a version of
flying-monkey</code> to manually manage machines based on their canon</code> address. I’d like to improve the
ergonomics to just need the canon</code> name of the machine, then have it look up the IP at least
optionally without relying on DNS.</p>
I can also have it introspect and get the MAC</code> of the relevant machine for image building, the CLI
should be something like:</p>
</span>
# Build an image and place it at the specified location</span></span>
$</span> elenta build-image</span> --config</span> '</span>.#machine-role</span>'</span> --out</span> /path/to/store/image</span></span>
# Build an image for the given MAC, store it under that MAC in the given 'store'</span></span>
$</span> elenta build-image</span> --config</span> '</span>.#machine-role</span>'</span> --mac</span> 00:11:22:33:44:55</span> --store</span> /path/to/netboot/store</span></span>
# Build an image for the given machine by it's canon, with the given role.</span></span>
$</span> elenta build-image</span> --config</span> '</span>.#machine-role</span>'</span> --canon</span> '</span>machine.canon</span>'</span> --store</span> /path/to/netboot/store</span></span>
</span>
# Deploy a role to a machine by it's canon name, do not boot to it on reboot (equiv to `nixos-rebuild test`)</span></span>
$</span> elenta deploy</span> --config</span> '</span>.#machine-role</span>'</span> --canon</span> '</span>machine.canon</span>'</span></span>
# Deploy a role to a machine and set it to boot to it on reboot (equiv to `nixos-rebuild switch`)</span></span>
$</span> elenta switch</span> --config</span> '</span>.#machine-role</span>'</span> --canon</span> '</span>machine.canon</span>'</span></span>
</span>
# Optionally use `--no-dns` to look up the mapping directly in telperion (useful for IP changes)</span></span>
$</span> elenta deploy</span> --config</span> '</span>.#machine-role</span>'</span> --canon</span> '</span>machine.canon</span>'</span> --no-dns</span></span></code></pre>
elenta</code> is going to have a concept of constellations</code>, which I suppose are similar to helm charts
or compose files, excepting that it’s a mapping of machine roles to machines, and expects that
configurations support whatever other systems you want to run.</p>
The upshot of these is that you gain a separation between arbitrary physical hardware and the
logical design of your architecture. You can spend all your time designing an entirely abstract
architecture, and then distribute roles to machines independently.</p>
The API I think looks a bit like</p>
# Note this is a sketch, I don't think you can import flake components like that.</span></span>
rec</span> {</span></span>
    roles</span> =</span> {</span></span>
        reverse-proxy</span> =</span> import</span> "</span>flake:telperion#domains.example.tld.roles.reverse-proxy</span>";</span></span>
        app-server</span> =</span> import</span> "</span>flake:telperion#domains.example.tld.roles.app-server</span>";</span></span>
        database</span> =</span> import</span> "</span>flake:telperion#domains.example.tld.roles.database</span>";</span></span>
    };</span></span>
    mapping</span> =</span> {</span></span>
        domain</span> =</span> "</span>example.tld</span>";</span></span>
        redirect_bare_to</span> =</span> "</span>www</span>";</span></span>
        machines</span> =</span> {</span></span>
            "</span>www</span>"</span> =</span> {</span></span>
                role</span> =</span> roles</span>.</span>reverse-proxy</span>;</span></span>
                host</span> =</span> "</span>server-a.canon</span>";</span></span>
                extraOpts</span> =</span> {</span></span>
                    # Arbitrary extra options to pass to the role</span></span>
                };</span></span>
            };</span></span>
            "</span>app</span>"</span> =</span> {</span></span>
                role</span> =</span> roles</span>.</span>app-server</span>;</span></span>
                # Since these two are on the same machine, the mapping will try to merge the two</span></span>
                # configurations, if they can't be merged, it will fail with an error.</span></span>
                host</span> =</span> "</span>server-a.canon</span>";</span></span>
            };</span></span>
</span>
            # Here we might assume that the roles have modes which tell it how to look for other db</span></span>
            # instances, again, the goal of this is to try to only think about the allocation of</span></span>
            # roles to hardware, and only pass information as is strictly necessary.</span></span>
            "</span>db-a</span>"</span> =</span> {</span></span>
                role</span> =</span> roles</span>.</span>database</span>;</span></span>
                host</span> =</span> "</span>server-b.canon</span>";</span></span>
                extraOpts</span> =</span> {</span></span>
                    mode</span> =</span> "</span>primary</span>";</span></span>
                };</span></span>
            };</span></span>
            "</span>db-b</span>"</span> =</span> {</span></span>
                role</span> =</span> roles</span>.</span>database</span>;</span></span>
                host</span> =</span> "</span>server-c.canon</span>";</span></span>
                extraOpts</span> =</span> {</span></span>
                    mode</span> =</span> "</span>secondary</span>";</span></span>
                };</span></span>
            };</span></span>
</span>
        }</span></span>
    }</span>;</span></span>
}</span></span></code></pre>
elenta</code> will have access to the full mapping and can pass that down to the role configurations, so
roles can look up items in the constellation and use that for static service discovery; elenta can
also use this mapping to generate DNS configuration that creates a zonefile for the particular
domain that creates all the necessary records.</p>
Eventually it’d be cool to include switch configurations as well.</p>


4-JUL-2024
2024-07-04T00:00:00+00:00
1713</h2>
Been picking at this all day, managed to get Arena to connect to Hazel and for Hazel to dump it’s
logs somewhere other than STDOUT. I imagine I could have the thing continue to dump logs to STDOUT
but I’m not sure if that’ll cause weirdness later with the UI, so I’m just going to leave it sending
to /tmp/hazel.log</code> until I have some UI in place.</p>
I think I’m going to work towards getting it making random moves and generally implementing the rest
of UCI for now. I also want to get it so that when debug</code> is turned on, the debug</code> log level is
set on the subscriber, and otherwise leave it at info</code>. That way I can continue to use Arena as a
debugging mechanism.</p>
I slapped together a couple scripts for running cargo test</code> and cargo build</code> with warnings
suppressed. I wish this was an option on the command itself instead of a flag. I recognize the ‘good
practice’ of always leaving these on, but I think it’s a little misguided, most of the time I
prioritize fixing the thing I’m working on, and because the error occasionally pop up in the middle
of all the warnings, it’s hard to track down the error amidst them.</p>
I did a little experimenting with watchman</code> and I think it may be worth a little hackery to come up
with something that will automatically, incrementally rerun tests and give you a little grid display</p>

log output of failing tests. Mostly I want it to exist, I don’t know that I actually want to build
such a thing.</li>
</ul>
In any case, good progress today, hopefully I can get most of the protocol implemented and see if it
can help me debug any of the weird EP bugs I’ve been seeing.</p>


3-JUL-2024
2024-07-03T00:00:00+00:00
0941</h2>
I think the UI is in a good spot to leave for a minute while I bring in some of the components that
need the UI. The first round is a simple UCI REPL; I should be able to type raw UCI into the UI and
have it respond (probably with dummy info for now). It should expose an option “current engine” that
points to the relevant backend by name.</p>
I also managed to re-introduce an en passant bug, I’m going to leave it for now; I started on a
refactored movegen module before I put the project down last time, I think that’s probably the
‘right’ way to chase out these bugs, and I think having a better debugging tool will help a lot in
figuring out what the issue is.</p>
1145</h2>
I’ve roughed in the UCI repl a bit, just the parsing stuff. I’m going to get a basic
parse-to-an-enum, then refactor to use Hazel types for Moves/etc. Those are generic enough that it
should help with forwarding UCI around but also being ergonomic enough to use directly in the UI and
for Hazel.</p>
1320</h2>
Fully roughed in, no type fanciness yes, but I’ll enrich the parsing as I need it. I think the next
step is to get the OCI connection started up, and then work on a backend which proxies the OCI
connection to the backend engine.</p>
1335</h2>
I think I need to learn tokio</code>. I keep hemming and hawing about just writing my own threadpool
thing, but I should probably learn the ‘real’ tool instead of rolling my own here. Time for more
tutorial reading, I guess.</p>
Or not, this article</a> suggests that maybe threads are just fine. I
don’t know that this needs to be such a giant thing, I’m comfortable having lots of otherwise idle
threads with connections open. I suppose maybe a better approach would be to have a
thread-per-engine, and then keep them in a pool. I suppose I could have Grid</code> be abstract across
any Engine</code>, and then have Engine</code> be a trait which, e.g., HazelEngine</code> might implement.</p>
The trait would just expect a way to send it UCI messages and recieve them in return. Eventually I
can place another variant which supports, e.g., a protobuf-defined API or something. The API would
have a raw_uci</code> message so you could always fall back to just parsing UCI, but for richer queries I
could extend it.</p>
I think I’ll skip tokio</code> for now and stick with threads until it starts hurting.</p>
2143</h2>
Definitely a pass on tokio</code>, the docs say that I’m probably not the right case for it right now,
and if I can go with a threaded approach, go with a threaded approach, so that sounds like a good
reason to me.</p>
I’m going to work on getting the ‘Driver’ part of Hazel working, that’s going to be the thing that
actually takes UCI messages and plays the game. My goal is going to be to integrate the UCI parser I
just wrote with the Game</code> struct, and implement whichever commands feel easy and bail on the rest.
That should be enough for me to hook hazel up to an OTS GUI, and then I can get something relatively
bug free working on a known-good GUI before working to implement my own. I can also hopefully hook
up my UI and the OTS one at the same time, so I can compare when debugging.</p>


1-JUL-2024
2024-07-01T00:00:00+00:00
2242</h2>
Got the Ratatui UI roughed in; also sketched out a plan for the debugging
interface/engine-tournament-manager thing. I’m off on a rabbit trail, but it seems fun so probably
worth doing.</p>
The first thing I want to do is get some sort of widget built to display a chessboard, I don’t think
that should be too difficult, since I’ve got half an implementation in the Debug impl for Ply.</p>
I can’t say I know enough about ratatui</code> so far to have an opinion, but I’ll say for sure that the
markup-in-Rust thing is not my favorite. Copilot actually helps a bit here, these APIs kind of suck
to remember; I frequently get simple transpotition errors where I miscapitalize or whatever. Copilot
eventually figures out what I’m trying to do and at least makes the mistakes for me, so I’m just
fixing what it fucks up instead of me making the mistake and feeling bad about it.</p>
2307</h2>
Futzing more, I think it makes sense to include all the UI stuff in the library proper, and maybe
just gate it by a feature someday. I can implement widgets directly against the types in the
library, which seems nicer than having to wrap things in a newtype to widget them.</p>
I’m definitely not quite doing it right w/ rendering, this is because I’m procrastinating reading
the rest of this tutorial.</p>


30-JUN-2024
2024-06-30T00:00:00+00:00
0102</h2>
Been working on the EP_CAPTURE issue, I ended up just deleting the implementation, walking away for
a bit, then reimplementing. I ended up doing it a slightly different way than the inital bit
mangling trick I was trying, and I think I’m just going to leave it as is for now. I think the bit
mangling will be faster and I was probably just doing it wrong, but I can leave the optimization for
another day.</p>
I added some better error output for the unrecoverable error branch, and managed to get the perft</code>
test running (but getting a different count of positions than it should).</p>
I need to set up some kind of integration test w/ a known-good engine. I think I might try switching
off the movegen for a bit and work on the UCI implementation and maybe a little TUI for debugging. I
could then ostensibly get it and stockfish talking to each other, and it could then start perfting
and having SF verify the results, so I can start to see the game state that led to it overcounting.</p>
In any case, for now, EP_CAPTURE I think is working, and I won’t really be able to track down the
bug until I build up some better tools for debugging.</p>
1838</h2>
A rough design:</p>
Hazel: The main thread, spawns:
- Grid: A list of engine instances, defaulting to Hazel, but allowing arbitrary UCI connections
to other engines.
- UI: The UI thread
- Race Control: Which sends commands between engines / the UI, may be part of the main Hazel thread
instead of it’s own thing, not really sure.</p>
The idea is to let Hazel have a sort of ‘tournament management’ feature, where it can run multiple
different engines and manage games between them, and also allow for deep integration tests as above.</p>
The Engine Instance is a UCI Socket for communicating to the engine, which Hazel manages and the UI
allows selection between, messages are proxied down to the actual engine instance, which is just
running it’s own UCI client reading from the same socket.</p>


22-JUN-2024
2024-06-22T00:00:00+00:00
0143</h2>
Worked a bit on EP_CAPTURE, but it appears that the unmake</code> method marked todo!()</code> that I started
implementing isn’t actully getting called in my test that explicitly exercises the unmake function
for this. It seems to be failing further up the chain, so I managed to stumble on a different bug
while trying to get perft working up to 6-ply.</p>
1221</h2>
I think the issue may be that it’s not properly recording that the previous move was an en passant?</p>
1318</h2>
I’ve got it figured out, I just needed to do some calculation to get all the pieces moved correctly.
I need to add another couple tests for en passant on the edge files, one for an EP by black on
white, and a few other cases, but I suspect perft</code> will reveal if there are any such issues.</p>
2117</h2>
2024-06-22T21:48:00.245032Z DEBUG perft_start_position_to_depth_6:perft{depth=5}:perft{depth=4}:perft{depth=3}:perft{depth=2}:perft{depth=1}: hazel::game: </span></span>
8 | r . b q k b n r</span></span>
7 | p . p p p p p p</span></span>
6 | n P . . . . . .</span></span>
5 | . . . . . . . .</span></span>
4 | . . . . . . . .</span></span>
3 | . . . . . . . .</span></span>
2 | . P P P P P P P</span></span>
1 | R N B Q K B N R</span></span>
    a b c d e f g h</span></span>
    Black to play</span></span>
</span>
2024-06-22T21:48:00.245161Z DEBUG perft_start_position_to_depth_6:perft{depth=5}:perft{depth=4}:perft{depth=3}:perft{depth=2}:perft{depth=1}: hazel::game: </span></span>
8 | r . b q k b n r</span></span>
7 | p . p p p p p p</span></span>
6 | n P . . . . . .</span></span>
5 | p P . . . . . .</span></span>
4 | . . . . . . . .</span></span>
3 | . P . . . . . .</span></span>
2 | . . P P P P P P</span></span>
1 | R N B Q K B N R</span></span>
    a b c d e f g h</span></span>
    Black to play</span></span></code></pre>
This is where the bug is introduced. In the first Ply, White plays axb6, and in the second Ply, the
unmake</code> function has been called but did not properly accomplish the unmove. By the time we see it,
White has played b3, but the board is in an incorrect state because the en passant didn’t unwind
properly.</p>
I’m not 100% sure what I did wrong, but I think the answer is probably to add a bunch more tests.</p>
One thing I could do is have each ply have a pointer to a ‘previous’ ply, rather than maintaining a
full history in the ply itself. This would trivialize the unmake</code> function, but at the cost of way
more memory use (I think).</p>
I think the first step is to analyze the unmake</code> side of the en passant, and make sure it works
consistently. I’m looking forward to when I have the movegen side of this working correctly, so I
can refactor it to be less of a giant megamethod, I think at least some of my pain is found there.</p>


21-JUN-2024
2024-06-21T00:00:00+00:00
1610</h2>
Working on getting this fixed up and functional, I updated dependencies, cleaned up some syntax
stuff, got a flake</code> set up and rustfmt</code> configured and running. I’ll add precommits at some point
maybe.</p>
I’m working on the pgn parsing issue, it appears to be failing to generate blocking moves for check.
It gets stuck on 26. .. Rf7 in the example fixture at tests/fixtures/no-variations.pgn. The perft</code>
test is also over-counting the number of moves, so I need to figure out why it isn’t seeing the rook
move.</p>
I’m going to check out bugstalker</a> to see if it helps suss
out what’s happening.</p>
2115</h2>
I found the issue. I think my movegen was more or less fine? I added some logic about looking for
‘blocking squares’, but I don’t actually know if it’s necessary. I’ll have to come back and test it
when I get to the refactor stage. Right now I’ve chased down a different bug which appears to be due
to Game::from_fen/1</code> not calculating metadata from the FEN string. It ultimately delegates to
Ply::from_fen/1</code>, which appears to incorrectly grant castling rights to black, which is incorrect.
I suspect this is because Metadata::default()</code> grants castling rights, but in the event I see the
-</code> value, I don’t do anything, which means it breaks.</p>
2133</h2>
I got everything pushed up onto main</code>, switched the branch over, and need to just do a few more
cleanup tasks before I can set the repo to open.</p>
2147</h2>
Added a LICENSE (AGPLv3), README, and uncommented the next tier of perft</code>, which is failing due to
unimplemented EP_CAPTURE</code> in the src/ply/make.rs</code> module.</p>
For now, it’s good enough to go live I think. I still need to chase out some dependabot warnings,
but that shouldn’t be too hard</p>


3-JUN-2024
2024-06-03T00:00:00+00:00
1041</h2>
I’m in between meetings so I did some cleanup and got the benches working again. I’m going to wipe
out the old results and start fresh this evening.</p>


2-JUN-2024
2024-06-02T00:00:00+00:00
1457</h2>
Took a brief break to work on some other stuff</a>, and will
probably continue to do so for a bit, but I also</em> looked a bit at
rust-gpu</a> and I think there is a natural way to
translate the montgomery stuff to use that, so I may take a brief stab at implementing my MMP
solution there instead of via SIMD. If I could get this to run on a GPU I suspect I’d sweep the
speedrun targets pretty quickly, and it’s a good way to get a pipeline set up to exploit the GPU for
other similar problems.</p>


31-MAY-2024
2024-05-31T00:00:00+00:00
0802</h2>
I got this all working last night and started it on my R730, Dragon-of-Perdition. It is not fast,
and it is not yet producing results I would feel confident counting on, but it is running and broke
past the Berndt-Galway% barrier overnight.</p>
At this point, the next step is to work on improving the solver speed and that means SIMD.</p>
I will later need to build up another layer above the broker (or move some of the broker stuff down
to another thread, IDK yet), to allow for some cross-system coordination; but for now I can scale to
consume all available cores on a machine and that is pretty rad.</p>
I’ve been thinking about how to SIMD and I’m leaning towards the MMP solution with SIMD happening
within the MMP implementation. I considered trying to SIMD across primes, and that might still be
the ‘right’ way (more on that in a second) in some sense, but ultimately I want to build an
upgradable thing for all</em> numbers, and that’ll work better with MMP.</p>
I think it’ll also scale better to GPUs.</p>
I’d like to build something that is a sort of ‘progressive’ multiprecision, so slowly expanding the
amount of SIMD resources spent on the multiprecision-ness, but also allowing for a SIMD-ified Elt</code>
object, which means we’d have SIMD both inside</em> the object and ‘across’ the object. Essentially
having a EltVector<R_EXP, WIDTH></code> where WIDTH</code> is the number of MP objects contained in the
structure, which can then be vectorized as well. Not quite sure how to make that happen just yet,
but one of the things my current implementation does which I think might not be so clever as I hoped
is calculate a range of primes for each chunk that is ‘just outside’ the range. This seems to result
in a lot of ‘near misses’ where the high watermarks on passing items is frequently 40+ primes. I
can’t tell if that’s a bug or if the choice of primes are such that this is ‘normal’.</p>
Fortunately, each chunk publishes the range of primes used to calculate it. Unfortunately it doesn’t
currently print what the start/span is, so another area to improve is the reporting and general
output functionality.</p>
I think chosing primes at the start may be net faster, as I should, in principle, be able to cache
and share the Montgomery Space</code> objects across everything as immutable objects and eliminate some
amount of churn when spinning up new threads. I suspect that’ll be quicker and will give more
consistent results. If I set it to read those primes from a file I should be able to exactly
replicate some results which should help me verify my code is working equivalently to the prior
work.</p>
This also opens some alternative Speedrun categories in the form of ‘what’s the set of primes for
which the number of passed tests is minimized in the worst case?’</p>
That question is actually extremely interesting once you try asking why</em> those primes might result
in a minimized NSW passed-tests metric.</p>
I also think the ‘how many tests did you actually pass’ metric is a little harder on me than it has
been on prior work, since I’m always testing across all 60 primes, where I believe they only tested
across as many as it took to get a NSW. So prior works ‘maximum passed tests’ might be hiding the
fact that the tests were run sequentially, and they just ‘lucked out’ and had most terminate early,
the alternative prime modes should support that model of checking as well.</p>
So, the TODO list as it stands, in no particular order:</p>

Improve report output to include:

SHA of code used to run it</li>
Timestamp of start time, duration information</li>
Range searched</li>
</ul>
</li>
Extract broker’s RX Loop to a separate thread</li>
Build multi-broker coordinator</li>
Implement alternative prime-selection modes

Existing mode finds primes via miller rabin</li>
Also support providing a list of primes

preinclude prime lists from all prior work</li>
</ul>
</li>
Also support sequential scan/short circuiting vs full test mode</li>
</ul>
</li>
SIMDify an MMP implementation

This is, y’know, pretty hard</li>
</ul>
</li>
</ol>
I think that covers it, I’m going to continue to let it run on my other machine, but at the moment
it looks to be doing about 4-5 billion values per day, so I’ve got a long way to go to beat the
~5 month time jhg</code> set.</p>
1213</h2>
I killed the run on my other machine, the 2,3 search wasn’t converging particularly nicely, and I
think I’ll probably need to adjust my approach, it will help when the reciever loop is independent
of the main loop, so that all my control logic is separate from my output/adjustment logic and solve
logic.</p>
I managed to get up to around 1e10 in about 12 hours, so I’m within spitting distance of Matson%.</p>
I want to improve the report output as it seems to be the lowest hanging fruit, and improving it
should help me come up with a better time-tuning algorithm.</p>
I may try just refactoring to use an existing BigInt instead of u128, and maybe my existing impl
will translate cleanly, not sure yet.</p>


30-MAY-2024
2024-05-30T00:00:00+00:00
2042</h2>
Okay, I did a bunch of work to start implementing the parallelized version of this. Yes, I am
putting off learning the SIMD stuff. I’m trying to sort out the right way to vectorize it, and I
haven’t decided if I want to try to just outscale the pain in performance that comes w/ the MMP
branch, or if I want to try to just knock out 2^64%</code>[1]. I’m leaning MMP but I have to make sure
I’ve got that dog in me.</p>
So, naturally I turn to the other thing, parallelization.</p>
I have mostly implemented the thing, but I’m chasing the borrow-checker around and I need to work
out how this thing is going to function.</p>
I want to make sure that if the program crashes, I can recover with minimal lost work. We can see
why in a bit from jhg</code>’s work, they note:</p>

We tested the first 1x10^15 (1 quadrillion) values over a period of ~5 months (January-May, 2020),
but no additional solutions were found.</p>
</blockquote>
5 months is a long time, and indeed the time will grow exponentially for me. I’m attacking that
problem through scaling out, but I also need to consider how often I want to spend ‘saving’ the
work I’ve already done so that I don’t waste it in an unexpected crash.</p>
I also want to be able to upgrade this over time, I’m going to be continuing to run this well past
the initial 2^64%</code> goal, so I want to be able to reliably stop without wasting work. My plan is to
build to the point where I have some</em> implementation going that can reliably generate new NSWs as
quickly as possible, so that I can then slowly iterate and replace parts as it runs.</p>
To accomplish this, I really want to think in terms of an RPO, not in terms of how it’s going to
work; if I never want to lose more than some target time – say a minute – of data, then it’s easy
enough to break off a ‘chunk’ of my target range and search it; identifying some set of primes just
outside the range dynamically, it can employ any number of algorithms for searching that range that
it likes, so long as I record:</p>

The primes used</li>
The number of solutions/nonsolutions in the range</li>
The SHA of the version of code being run, so we can replicate it after the fact</li>
Build information (dependencies, etc) associated with the code</li>
</ol>
A future skeptic could then check out the same code at the same versions, compile the code, and with
reasonable certainty replicate the chunk.</p>
I’d conjecture that larger chunks will generally be more efficient, but smaller chunks are safer
from a crash-perspective. Because I also expect to scale to new hardware in this process, I won’t
always know how large of a range will take a minute to process, so ultimately I want to have some
dynamic ‘tuning’ capability.</p>
Since I’m already breaking things into chunks, it makes sense to me to have a Broker/Worker design
to fan this work out across cores. The ‘broker’ hands out ‘chunks’ to the ‘spans’ along with some
primes that are just slightly outside the span. I’m not sure that’ll make things faster, but it’s
very aesthetic.</p>
I’ve roughed in a version of that, but it’s still missing a couple pieces.</p>
First, I’m pretty sure as written it’ll only run one thread; but I was fighting a lot with the
borrow checker and wanted to get to some psuedo-stable point.</p>
Second, I haven’t implemented the ‘take_primes’ function which should return a fixed number of
primes upon request.</p>
I have a lot of ideas for optimizations, but I want to get to the point where I can start running
this in some capacity and start figuring out how to upgrade it in situ.</p>

[1] An footnote: I am going to name some things.</p>
First, an acronym: WNS or NSW, both mean Witness of Nonsolution</code> or Nonsolution-Witness</code>. These
are the primes for which the Legendre test returns a non-residue result.</p>
Second, I’m going to assign some ‘speedrun’ categories to different bounds. The obvious ones are:</p>

Gupta% : Wall time to 63</li>
Berndt-Galway% : Wall time to 1e9</li>
Matson% : Wall time to 1e12</li>
jhg% : Wall time to 1e15</li>
2^64% : Wall time to solve for every value less than 2^64</li>
Brocard% : Time To Novel Proof there are/are no more solutions.</li>
</ol>
Other categories may be introduced in the future. I have assigned names as seem appropriate, the
first person to set a record in the category should, I think, have a right to name it, so if any of
these fine folks want to change the name I’m happy to oblige as official self-nominated benevolent
record-keeper for as long as the position suits me.</p>
Thank you for coming to my footnote.</p>


26-MAY-2024
2024-05-26T00:00:00+00:00
1223</h2>
I have two big problems:</p>

Get an ASM dump of the montgomery math to verify there are no divisions.</li>
SIMDify montgomery math.</li>
</ol>
For #2, I think the actual algorithm isn’t really SIMD friendly until I’m doing MP, but some of the
math functions might need some attention. I’m going to look over the C++ code from the prior work to
see if I can glean how they approached it, but my suspicion is that the SIMD will really come in on
the brocard side of the problem, not the montgomery side.</p>
I could potentially SIMD to do some calculations in parallel, but I don’t think it’ll gain much.</p>
The ASM dump to verify no divisions is a bit more straightforward. The #new code definitely has at
least a couple divisions by r</code>, but I should be able to replace them w/ &’s. There’s a % n</code> in
there as well but I don’t think that’s going anywhere.</p>
I replaced the r</code> divisions and kicked off the benchmark, I expect little gain.</p>
I’m going to refactor the montgomery math to use const generics for R_EXP, and then refactor the
benchmark to try every relevant R_EXP. The redc</code> algorithm currently has to recalculate r-1</code>
every time, and a const R_EXP would prevent that. It might also be worth genericizing over u128</code> to
see if a u64 is faster.</p>
1256</h2>
One way I might approach SIMDing the montgomery stuff is SIMDing the .enter</code> side, and having a
single n</code> and constant R</code> for the space. So that a single Space can handle, e.g., Cache - Overhead</code> values.</p>
1424</h2>
Refactoring to a const R_EXP appears to have improved things slightly, at least on Legendre symbol
benchmarking. Many of the benchmarks don’t scale nicely to const generics so it’s a little tricky to
tell. Next step is to refactor out the u128, but that is going to be a massive effort since every
math function relies on u128 and will need to be tweaked.</p>
Refactoring that should transparently support SIMD-lanes, which would probably improve the existing
BrocardSpan implementation.</p>
Overally there is a ~1-2% improvement from the R_EXP refactor, which is pretty good for such a small
change. Avoiding the extra calculations on each redc</code> should pay off significantly on the brocard
benchmark. I’ll run that next to see how it goes. I would expect to reap significant benefit since
redc</code> is run on each multiply, and factorial necessarily multiplies n</code> times, so a 1%</code> move on
n</code> multiplies is a 1.01^n</code> move on the total time.</p>
I can also consider refactoring just</em> the factorial function to SIMD to see if that adds an extra
bit of speed to brocard.</p>
I also need to refactor TestCase to have a const R_EXP, and then do some more cleanup.</p>
I did finish re-organizing the code, the tests are all still clumped in the montgomery/mod.rs</code>
file, I’ll probably pull them out to the tests</code> directory since they’re a little tough to organize
otherwise.</p>
1440</h2>
I can’t actually refactor the factorial because I can’t use u128 w/ simd. ugh.</p>
1809</h2>
Slight regression</em> for brocard, not sure why, really need to look at the ASM output. I suppose it
might be down to the fact that I make a bunch of these Space</code> objects and the cost is in doing some
of the same precalculations over and over? Not sure.</p>
Next step is to Genericize, I think, that’ll at least allow me to process a small range of n</code>’s
simultaneously, if I set it to some generic LANE size I can do some benchmarking to see what the
best size is, but I can relatively cheaply calculate a vector of n_i!</code> over some small range i</code> in
[k, k+LANES]</code> since each n_i = (n_i-1)i</code> which means I calculate n_k!</code> and then each subsequent
element in the range is easily constructed in LANES</code> operations, then I can legendre test all of
them; since each will ‘terminate’ at different times I can just have a LANES</code>-sized mask that I
check to determine when to stop, meaning the Legendre test will take a worst-case time of the
slowest test in the range. I can also short circuit if I get a Non-witness report, since it doesn’t
matter how many it passes, only that it fails on at least one.</p>
2305</h2>
I did some work w/ flamegraph</code> to chase out some remaining div_mod calls, that worked quite well.</p>
I also found that the fairly naive splattering of rayon</code> primitives around the solve function was
probably resulting in some deadlock or something, because it simply made my machine hot for 30m
while the sequential version runs to 1 billion in ~11m or so. Which, if I were to parallelize it
perfectly efficiently would mean I’m right in the same ballpark as the prior solution in terms of
speed.</p>
One optimization I made that wasn’t simply avoiding div_rem instructions was to avoid a .exit()</code>
call in the Legendre symbol calculation. Since 0</code> is the same value for every R</code>, we can just
check for 0</code> without leaving montgomery form, and it turns out that entering</em> montgomery form is
faster (despite having more steps) because there are only shifts and multiplies due to redc</code>, so
you can minimize the math by comparing the result to see if it’s equal to space(1)</code>, rather than
exit</code>ing as I was.</p>


25-MAY-2024
2024-05-25T00:00:00+00:00
0012</h2>
Alright, I’ve got a brocard benchmark in place, I’m slow as molasses. I think the next step is
profiling, I don’t want to jump to a SIMD conversion until I’m sure I’ve got the algorithm right.</p>
Another thing that occurs to me to check is the generated code to ensure there aren’t any division
instructions being written. Ultimately, the point of montgomery arithmetic is to dodge divisions by
replacing them with shifts; so I would assume an audit of the generated code to verify there are
only divisions where I think there should be divisions makes sense.</p>
I’m getting this committed, then I’m going to read
this</a>,
which seems like an interesting article. Right now, I think there are two paths forward with SIMD.</p>
First, I could rewrite the existing implementation to parameterize it’s Elt</code> type, instead of u128,
it’s be T. I have no idea what effect that’ll have on performance, and I imagine it will deeply
change how much I have to appease the trait system, but I think it’s a theoretical K</code>-times speedup for
essentially a bunch of gruntwork when I sub in u64xK</code> or whatever it is. The downside is this
cannot exceed some value slightly below 2^64, since I’ll need primes larger than n</code>, and those
primes have to be represented as a 64b number to make good use of SIMD in this model. I don’t see
any 128b types allowed in the docs I’ve looked at, but I haven’t looked particularly hard since
signac-vector</code>.</p>
Second, I could focus on something past the 2^64 (~1e19)</code> barrier. While it is 4 orders of
magnitude ahead, and it took months to go from 1e12 to 1e15, I am approaching this with a scaling
game in mind; I’m honestly not all that interested in Brocard itself, I’m interested in building out
the thing that scales the solution and making that efficient, so it may be wise to focus on the
multiprecision brocard implementation. That should be easy to make SIMD/vectorizing friendly, to the
point where I would expect most of the relevant loops to be vectorized by the compiler. A naive
(not explicitly SIMDified) implementation would obviate the barrier and also potentially be
sufficiently quick (with the added horsepower I have) to not hinder progress.</p>
Multiprecision is also the way to go for GPU implementation, which is still somewhere in the cards.</p>
In any case, the final thing I need to do is some cleanup in the repo, get things organized, tests
moved to their rightful place, some documentation work, and hopefully by the time I’m done I’ll feel
like figuring out performance tracing for what feels like the thousandth time.</p>
0038</h2>
Reading this article was a very good idea, SIMD does not work the way I thought it did anymore.</p>
0052</h2>
Finished reading (well, skimming through) the article. I think I know the approach, the answer is
‘both’. I can optimize the existing algorithm using this new core::simd</code> stuff mentioned in the
article to get familiar with it. This will allow me to hit 2^61 - 1</code> as my largest prime, with
R_EXP=61</code>. That’s about 1e18, or 3 orders of magnitude from the previous attempt.</p>
Next, I’ll implement a multiprecision montgomery version using SIMD while the initial version is
running. The algorithm with generate Brocard Spans and dump all the relevant metadata to disk.
Periodically they will run the same span to verify they are both producing the same result. I think
the managing apparatus will just be a couple systemd-wrapping-sinatra or something to drive them and
report results to some parent process I can run on toto.</p>


21-MAY-2024
2024-05-21T00:00:00+00:00
1555</h2>
Current status: Hauling ass.</p>
I set up a brocard run up to 1e9, as an initial test. It’s currently quite a bit slower than the jhg
version, but I think for an unoptimized bit of hackery it’s doing alright. rayon</code> is handling
parallelizing, but it’s the stupidest implementation I could come up with so I think there’s more to
gain by just improving the code a bit.</p>
As mentioned above, there’s an outer loop that’s causing a lot of slowness on the inner loop by
recalculating factorial over and over. I’m also not being particularly efficient with how I break up
work. I should be controlling the work-unit size and assigning threads to a smaller sub-span of
whatever input span I get.</p>
I can then precompute all the relevant montgomery spaces, and share some of the work of calculating
the factorial of all those spaces.</p>
I suppose since each number is only 16B wide, if I plan to precompute all the factorials for a given
range, then the total memory allocation is only 16*(UPPER_BOUND - START)</code>. If I split off the
parent range into ‘sections’, I can probably manage the memory use and then simply precompute the
factorial for the whole range, then iterate over each to do the series of legendre tests,
essentially turning the loop inside out and saving all that time.</p>
I need to set up proper benchmarks for that, and probably learn a bit more about Rayon.</p>
I also need to look into SIMD optimization for this, but it’s limited inasmuch as I want to use 128b
values in most places, but nothing appears to support lanes >64b.</p>
I want to push the boundary to arbitrary heights, but I suppose refactoring the existing impl to at
least by type-generic might be worth it if it doesn’t hurt performance. It may be faster to cover
all 64b numbers first, then move to a multi-precision model which will be slower but can proceed
unbounded.</p>


8-JAN-2024
2024-01-08T00:00:00+00:00
1541 - ui</h2>
I took some time to write a little Actor-adjacent (I don’t think these technically</em> qualify as proper actors, but
whatever) system for Hazel</code> to use, it should be enough to separate out the communication logic from the chess logic,
which I hope will make it easier to finally get it done. It also supports an open ended ‘message’ system. To implement a
new message, a struct implements MessageFor<W></code> where W</code> is, in theory, a Witch</code> backend, but could be whatever if I
want to create a new engine later. The Witch</code> type is a little message reactor machine thing. It’s somewhere between a
genserver, SmallTalk object, Ruby class, and twisted nightmare that only the damned may dream of.</p>
I used to be a ruby guy, so I’ll let you guess where I got the inspiration.</p>
In any case, the next step is to refactor the Driver to use this backend, and that means extracting out the stateful
bits from the communicating bits.</p>
One other tidbit about the design is that the request and response types are separate. The response type is arbitrary
because I explicitly did not</em> include a general way to get at a Witch’s state object (which is also arbitrary) from the
Handle that controls it. I don’t want to make an assumption about how one should interact with their state, and very
often returning a whole object would be silly anyway. Messages do not generally allow for direct response (although such
a thing could be encoded in a message type easily to bypass the built in write</code> mechanism), all output from the Actor
is through the broadcast channel. It’s assumed every client will want to recieve every message in lockstep.</p>
I don’t intend to really have more than one or two of these at present, though I could see using this as a stepping
stone towards multiple separate instances across multiple machines, all communicating. Ultimately my goal was to learn
tokio</code>, and I think I’ve got it to hand now. Reasoning about it is tricky for sure, but it honestly just reminds me of
Haskell, it’s just a matter of understanding how the laziness works and then chasing the compiler around until
everything type and borrowchecks.</p>
It’s a pretty nice system, I am interested to eventually investigate tower</code> for this or another project, but time will
tell.</p>
1901 - ui</h2>
I’m thinking a bit about refactoring the tests. I think I’d like to break out of the habit of writing the tests in the
same file; it makes the files quite large, and I much prefer lots of small files, even if it means jumping around some
more. I suspect I’ll want to add some editor shortcuts for alternating between test and source files, and maybe also add
in some kind of shortcut for running relevant benchmarks as well, not quite sure how I’ll do that, but when the time
comes.</p>
In any case. I think I’m going to look towards centralizing and better codifying the tests I have, and once I have
something that can, ostensibly, play chess, I’ll probably take the time to move to rstest</code> and get something with good
coverage and few mutants. I generally aim to have the testing part of the codebase be 1/3rd to 1/2 the total size of the
codebase, and I’m sitting close to the 1/2 mark, so I definitely want to take some time to build something that can be
extended naturally and easily.</p>
disturbed-circles

28-AUG-2026

28-AUG-2026</h1>

26-AUG-2026

26-AUG-2026</h1>

25-AUG-2026

25-AUG-2026</h1>

1352</h2> Added some quotes to the rotation, I don’t love how I have it set up right now, probably will move to a single structured file that compiles to the multiple files at build time or something, good enough for now though.</p>

Whom do you serve?

Dungeon Mastery</h1> Hi, I’m Joe, and I’m a forever-DM.</p>

17-AUG-2026

17-AUG-2026</h1>

Music isn't the sound that music makes

So what sound will I make?

The Move

12-AUG-2026

12-AUG-2026</h1>

1405</h2> I have a big TODO list but I’m starting to whittle it down. I’ve got most of the CI stuff done, but still need to get DNS wired up. I updated the profile to point to projects which do not yet exist, but all things in time.</p>

1449</h2> Waiting on tickets leaves lots of time for scutwork. Pages setup done for DC.</p>

10-AUG-2026

10-AUG-2026</h1>

10-AUG-2026

9-AUG-2026

9-AUG-2026</h1>

1935</h2> Did some cleanup of GIYF’s log. Now that I can plan these logs to end up here, I can actually use proper footnotes and katex and stuff. Nice bonus I hadn’t really thought about. I need to get better at running spellcheck over this stuff though.</p>

2200</h2> Pixel pushing, lots of pixel pushing.</p>

9-AUG-2026

23-JUN-2026

18-JUN-2026

27-MAY-2026

1602</h2> Currently renaming all of the lemmas/exercises and reorganizing again. LLMs are a godsend for fiddling with this kind of thing.</p>

24-MAY-2026

18-MAY-2026

Mundanity</h3>

13-MAY-2026

5-MAR-2026

4-MAR-2026

3-MAR-2026

On Proving with Lean

Effect on the initial exercise</h1> This was the original:</p>

2-MAR-2026

28-FEB-2026

27-FEB-2026

26-FEB-2026

25-FEB-2026

6-JAN-2026

1-JAN-2026

29-DEC-2025

9-DEC-2025

1136</h2> Got logging and the picker updating better, still having trouble with the camera widget, likely down to not understanding exactly the scope of the ‘strategy’ system that BRC is using.</p>

22-DEC-2025</h2>

8-DEC-2025

1419</h2> Fighting with nixos and dioxus and bevy and friends. Makes me think of building a VM or container for just dev. NixOS is a fantastic thing for static environments, but it’s very frustrating to use in a dev environment.</p> An Arch box w/ home-manager may be in the future.</p>

4-DEC-2025

2-DEC-2025

1-DEC-2025

20-NOV-2025

1441</h2> All of this does nothing in the face of the actual current problem which is rendering slowness, but it has been fun to dive down the rabbit hole a bit.</p>

19-NOV-2025

18-NOV-2025

1231</h2> I roughed out some ideas on paper, but rapidly realized I’ll also need some sense of a ‘timeline’ for each party, as different places within the worlds I manage may move at different rates.</p>

26-OCT-2025

20-OCT-2025

10-OCT-2025

26-AUG-2025

25-AUG-2025

1538</h2> Apparently ratatui_image has something for separating rendering, might be able to use that as an example of how to structure the SVG rendering thread. The current HEAD</code> of ratatui_image has a tokio example as well.</p>

21-AUG-2025

1145</h2> https://stackoverflow.com/questions/61752896/how-to-create-a-dedicated-threadpool-for-cpu-intensive-work-in-tokio</p> Seems relevant.</p> I also need to do some kind of caching/don’t re-render if the template hasn’t changed. Probably a simple hash scheme’d do it.</p>

18-AUG-2025

17-AUG-2025

14-AUG-2025

1510</h2> I got the svg pipeline working, it seems pretty snappy but I’m also not drawing it most of the time. Next step is to extend the template stuff to allow for drawing shapes and whatnot.</p>

23-JUL-2025

1405</h2> Started to move things, need to stop and think through how to organize it. I could have a single helm chart with subcharts, or move the dependencies up into terraform, not sure how I want to run it just yet.</p>

16-JUL-2025

13-JUL-2025

1048</h2> I split off earendil</code> for housing helm charts, it presently has nothing in it. I’ll store instructions on setup there as well.</p>

12-JUL-2025

1405</h2>
I have a big TODO list but I’m starting to whittle it down. I’ve got most of the CI stuff done, but still need to get DNS wired up. I updated the profile to point to projects which do not yet exist, but all things in time.</p>

1449</h2>
Waiting on tickets leaves lots of time for scutwork. Pages setup done for DC.</p>

1935</h2>
Did some cleanup of GIYF’s log. Now that I can plan these logs to end up here, I can actually use proper footnotes and katex and stuff. Nice bonus I hadn’t really thought about. I need to get better at running spellcheck over this stuff though.</p>

2200</h2>
Pixel pushing, lots of pixel pushing.</p>

`1602</h2> Currently renaming all of the lemmas/exercises and reorganizing again. LLMs are a godsend for fiddling with this kind of thing.</p>`

1136</h2>
Got logging and the picker updating better, still having trouble with the camera widget, likely down to not understanding exactly the scope of the ‘strategy’ system that BRC is using.</p>

1419</h2>
Fighting with nixos and dioxus and bevy and friends. Makes me think of building a VM or container for just dev. NixOS is a fantastic thing for static environments, but it’s very frustrating to use in a dev environment.</p>
An Arch box w/ home-manager may be in the future.</p>

1441</h2>
All of this does nothing in the face of the actual current problem which is rendering slowness, but it has been fun to dive down the rabbit hole a bit.</p>

1231</h2>
I roughed out some ideas on paper, but rapidly realized I’ll also need some sense of a ‘timeline’ for each party, as different places within the worlds I manage may move at different rates.</p>

1538</h2>
Apparently ratatui_image has something for separating rendering, might be able to use that as an example of how to structure the SVG rendering thread. The current `HEAD</code> of ratatui_image has a tokio example as well.</p>`

`1145</h2> https://stackoverflow.com/questions/61752896/how-to-create-a-dedicated-threadpool-for-cpu-intensive-work-in-tokio</p> Seems relevant.</p> I also need to do some kind of caching/don’t re-render if the template hasn’t changed. Probably a simple hash scheme’d do it.</p>`

1510</h2>
I got the svg pipeline working, it seems pretty snappy but I’m also not drawing it most of the time. Next step is to extend the template stuff to allow for drawing shapes and whatnot.</p>

`1405</h2> Started to move things, need to stop and think through how to organize it. I could have a single helm chart with subcharts, or move the dependencies up into terraform, not sure how I want to run it just yet.</p>`

1048</h2>
I split off `earendil</code> for housing helm charts, it presently has nothing in it. I’ll store instructions on setup there as well.</p>`
