27-NOV-2024

1214

I’m thinking I might move to an iSCSI-based approach for VMs for the near term.

The netboot experiment has been cool, but it’s finicky with total available space and also it’s a really slow process to rebuild the images. I think I can get a lot of the same benefits (centralized storage, decoupled configuration, etc) with iSCSI and eventually some kind of overlay system.

I’m going to build an iSCSI gold-image that can be cloned. It will have a baseline install of a recent NixOS, and will be periodically mounted and updated. When creating a new machine, I’ll have scripts that:

  1. COW Clone this iSCSI base image
  2. Create a new RW iSCSI disk
  3. Configure these disks for the new machine.

The machine would then mount the two disks in an overlay, booting form the base COW clone and then writing to the RW disk with any subsequent updates.

Machines can be periodically recreated, and any dedicated data can be placed on an independent iSCSI, so the COW clone is just the common shared underlayment, the RW disk is for machine-specific config, and additional independent disks for application data.

This will require a lot of automation around iSCSI creation, but for the immediate term I’ll probably just make disks per machine and install that way, a hard clone of the base image should do it. I’ll serve them out of Nancy and once it’s working I can work on building a better iSCSI server.

To that end, I think I’m targeting 100GBe Ethernet out of the gate, I considered other protocols, and they may be useful if I want to split out a backplane for data, but in the short term I’m going to keep it simple and have a single flat network on 100GBe between all physical components that support it (currently only 3 such machines, but I expect that I will have a few more in the future, so I’m aiming for something in the 16 port range on the switch).

Ultimately the 100GBe switch will link over a pair of 10GBe links to Condorcet, and then the majority of ‘normal’ machines in the lab will still route through there, but the 100GBe backend will be used for the Racked machines to share data, the traffic is low enough that a flat network should be fine, and I do want to move to IPv6 eventually anyway, at which point the routing will be much simpler.

I may start-from-v6 on this project, and use it as a lever to excise v4 from my lab, but tbh I’m not sure it’s worth it in the short term.

All of this mess brought to you by frustrating issues with my prometheus server not mounting NFS the way I like. Hell of a lot easier when it just looks like a disk.

1619

I think I’m going to start by replacing some of the NFS-as-a-target stuff with iSCSI. NFS is really not the right tool for what I want, which is a pile of bits that the VM can read/write from on it’s own. I can still netboot the machines but have all the data live in iSCSI volumes. I should be able to extend my existing network-storage code for this, and I’ll have to manually recreate the luns, but I should be able to query the infra code to generate an appropriate list for Nancy at least, and then that list can eventually drive a proper iSCSI host.

1940

Okay, actual plan:

  1. I’ll create NixOS-24.11 as a LUN in my Synology (Nancy)
  2. [~] I’ll mount this LUN on pinky, my lab VM, and install NixOS 24.11 on it.
  3. I’ll then clone this LUN to a new LUN, PINKY-24.11-ROOT, and mount it on pinky as its new root disk
  4. I’ll alter Pinky’s configuration till it can boot from this iSCSI disk.

After this, I can recreate the other VMs using this new system, the clones will be RW per machine and full copies (since COW doesn’t seem straightforward here), but it’ll get the system running and I can work on it from there.

I’m a little ways into the plan above and I don’t expect it’ll take to long to get running.