Wow, that was the most confusing computer error I have had for a long time. I have a little box here, x86 based, one of those Chinese mini PC things, running as the home server. It's configured with a pair of SSDs in RAID1 so one of them breaking doesn't stop the Internet from happening, and it runs a bunch of VMs for the actual functionality: one for Home Assistant, one as the home router, one for 'misc stuff'. I wanted to mess around with something that takes up a m.2 slot and I had none left, but I can simply take out one of the SSDs; as they're in RAID1 it should just start running solely from the other SSD, right? I knew in practice it would probably be a bit harder because of UEFI stuff, but not much. So I do that. Indeed I run into UEFI stuff, but putting the m.2 disk back and messing with Grub makes the spare SSD bootable and away we go.
Machine comes up fine, so I go and check on the thing I installed. Yes, it sees it. But why do I not have Internet yet? That worked for like forever and survived a reboot last time and I didn't change anything, why is it broken? I log into the router VM; it seems perfectly content aside from not getting onto the Internet. Hm, that's weird. There's a pppd setup there that seems properly configured, I didn't touch those files in half an eternity, so why can't it get Internet? Additionally, there's a DHCP server running on the thing. DHCP server also seems pretty content, but my laptop gets no IP? What is going on there?
So I run tcpdump on the eth interface that goes to the Internet, expecting pppd packets to the fibre modem, at least when I reset pppd because that is how the connection to the Internet is configured. That is not what I see. I see UDP (and some fragments of TCP) traffic on the raw unencapsulated interface, addressed to our public IP. Whut? Cue head scratching. I really do see the main Internet interface configured to use pppd... I take another look at the pppd config. That username and password matches with what I remember I had... before we moved to Singapore. I also check the system log, and before I scroll down, I see a date in January. I'm decently sure the logs rotate more often than that.
And then it hits me: It doesn't work because the machine time-travelled a bit more than two years back in time. During our move, something must have happened that knocked one of the two SSDs out of the RAID1, and while I normally have scripts poking me when that happens, those either didn't work or I didn't pay notice to it because we were moving countries. The machine happily ran from the still OK SSD from then on, like nothing happened, exactly what a RAID1 should do. Until I removed an SSD, and that happened to be the active one... Seems there was nothing particularly wrong with the remaining SSD, so it still had the same Linux install on there, just stuck two years and a bit in the past. After the move, the network situation changed, so it was very confused when it woke up and found itself in an entirely new network situation, leading to the weirdness I saw.
Computers, man...
