use Perl

Ugh just about sums up my weekend. I spent most of Saturday day, all of Saturday night, and almost all of Sunday, upgrading Ted, the computer that runs axkit.org.

This was problematic for a few reasons:

1. I could no longer get security updates for the version of Linux I was running (Mandrake 7.0). This upsets me a lot, but I guess you get what you pay for (yeah, I know MS also deprecates versions of their OS, so I guess the whole world sucks today ;-). The upshot is I needed to upgrade my entire OS.

2. I don't own a CD burner, so no CD upgrades. Must be a network upgrade.

3. I wanted to move to an "apt" capable distribution. urpmi is nice, up2date is nice. apt is just nicer.

4. I wanted to play with KDE3. I have this on my laptop and it makes me more productive. Really.

5. I wanted to do the update directly via apt. This worked OK on my laptop so it would work ok on my server, right?

6. All this with minimal downtime required.

I decided to go for a completely new Linux distribution - Conectiva Linux. They are the guys who ported apt to support rpm (instead of dpkg) and they're fully behind apt - to the point that their snapshot distribution is available via apt. So that's what I went for - Conectiva Snapshot. (oh, and Conectiva are so far the only guys who ship AxKit with their distro ;-)

So I installed apt and set about doing the upgrade. It took most of the day to download the 500 rpms that needed updating, then a good couple of hours sorting out the broken dependencies. Finally it installed everything. I did a lot of checks to make sure it would come back up, and rebooted. Up she came. But no X. Next time up everything seemed OK, and I installed some necessary bits and bobs to get X up. I got a few lock ups along the way, but I dismissed them as configuration issues that I would iron out.

Once I got X up, I discovered that I couldn't su. It just locked up solid. I tried from the console. Same thing. Then I tried logging on multiple times - and got lock ups once again. Eventually I found out that it was /bin/login locking up. An strace revealed it was going:

ioctl(0, SNDCTL_TMR_CONTINUE, {B38400 opost isig icanon echo ...}) = ? ERESTARTSYS (To be restarted)
--- SIGTTOU (Stopped (tty output)) ---
--- SIGTTOU (Stopped (tty output)) ---

Over and over and over ad-infinitum.

A search revealed very little - two people had the same problem but with different setups. Two. Bizarre. I'd never seen such strangeness. And I had no idea what SNDCTL_TMR_CONTINUE on STDOUT was supposed to do (there are no docs to be found on this).

I did everything to try and debug this. I installed different kernels (including the stable Red Hat linux 7.2 one). I re-installed RPMs. I recreated /dev. I tried to use devfs. It was truly horrible. In the end at 2am I slunk into bed without a clue how to fix it.

Of course, as always happens, I woke up with ideas. I completely cleaned out /etc/rc.d/rc.* and re-installed all the RPMs for those files. Reboot. It worked. IT WORKED!. Wooohooo! Actually I was too tired to celebrate that loudly, but whatever...

So after that I just had to get everything working again that wasn't RPM based: Perl, Apache, mod_perl, AxKit, qmail, firewall, tinydns, courier imapd, and a few other bits. Oh and of course make sure everything worked from both inside and outside the network.

Well it all took me way too long, but I'm very happy with the results. My desktop feels much faster - like a new computer. I'm now on Perl 5.6.1 (after all that time on 5.00503). And I'm running KDE3 here, which is nice to use.

After all that, our radiators have been making a hell of a racket lately. So I ventured up into the loft to check out the header tank. Needless to say it was full of crap. So I emptied it with a bucket (because I didn't want to send all the crap down the system) and now will have to get some radiator flush. My hands are now full of fibreglass splinters from moving the insulation around.

Ugh.