20-AUG-2024

1738

I rigged up vpn routing, but ran into an issue where because the containers now routed through a VPN, they cannot see my local DNS for local DNS queries. I think it should be possible to bypass this but I don’t really want to at the moment and the whole setup needs some more significant refactoring anyway, so for the moment I’m just going to turn it all back off but leave my changes in place.

All this happens in narya, though, so it’s not really visible to anyone but me. Maybe someday I’ll open source that bit but for now the mystery shall remain.

Independently and in addition, I stood up daktylos in the emerald.city domain to be the loki/prom server, and wired up exporters for nginx, systemd, and the node exporter. There is definitely some overlap to unpack for these, and I’m beginning to dislike how fat telperion’s flake is growing, but I think the plan will be to move the logic to laurelin’s library, then have a leaner telperion library which does the various calculations. Domains eventually should be flakes unto themselves, I think, so for the moment that stuff can stay put, but the scrapeTargets key is definitely in need of refactoring.

I also added additional horsepower to barge while I work on getting things configured, I really don’t care for the *arr apps, they feel bloated and it’s odd to have them split the way they are, but there doesn’t seem to be any reasonable alternative out there, so I think I’m stuck with them for now.

Remaining, I need to get my dev VMs built, and I need to start thinking about NFS mounting the nix store, the two things are not unrelated, but it is a big job so I might take a break from this for a bit after I finish the work I’ve got going now.

1821

I’m exhausted. There are performance issues to chase down, and things to fix, but I’m tired so I’m going to put this down for a few days, I think.

2049

One last thought before I put things down, I suspect the majority of my problem is that barge is overloaded with IO, network, and cpu intensive tasks. I spent a little time setting up some panels so I could better diagnose, and this not to say I didn’t see it coming, I think if I split out the rev-proxy into it’s own VM that’s shared between daktylos and barge, and then split the torrent/vpn component into it’s own VM(s), that should get me much closer to where I want to be. I also suspect at least some of it has to do with the way networking is setup on BTG; since all the VM traffic has to go in and out through the same pipe, things are simply not very quick when I’m saturating said pipe. I may temporarily create a secondary bridge that can be dedicated to the torrent side.

I also suspect I’ve got the CPUs configured in a poor topology that is confusing things and perhaps slowing them down a bit. I should also review and ensure all the relevant kernel modules and such are loaded – I am at least two layers of virtualization deep, I suppose I could also try running these as containers directly on the host, but I’m inclined to try to get this working (as eventually I plan to run all this on k8s anyway).