My six month contract is up. It's been a busy six months.
I tell people that consulting is feast or famine. I've enjoyed the feast. I've been eating good food from the farmer's market. Before that, I've killed a 25 pound sack of rice and a 50 pound one along with crazy amounts of dried black beans. I've put on weight. I was down to 155 pounds for a while.
The firewall/gateway machine was apparently suffering some severe hardware problem that caused it to crash more and more frequently. I got it booting memtest only to see that memtest was reporting the same unhandled page faults that Linux was reporting. Daaamn. That Toughbook was replaced with another. I have to find somewhere to recycle a 20 pound blob of magnesium now. That's just one misadventure of mine these guys had to endure that's now taken care of and the next guys won't.
Speaking of Toughbooks, poor old fluffy, the CF-R1, got replaced and retired, first with a CF-51 then with a CF-73. The 51 proved too bulky to haul around under my own power all over creation. It actually worked the backpack zipper loose once and leapt out hitting the sidewalk at a good speed (and survived, minus a chink in its armour). I could have pulled the 32gb Compact Flash card out of fluffy and stuck it directly into the 51 then 73 but fluffy was still on a 2.4 kernel and upgrading to 2.6 was one of the goals. 2.6 wasn't setting up AGP correctly on the thing so I had to keep it on 2.4 and glibc had long since passed 2.4 by and that was making it difficult to use things like alien and klik to install software on it. That and the 256 megs of RAM made upgrading critical. I decided to try to upgrade the existing OS and keep the software. That proved to be a huge time sync and a disaster. If I had it to do over again, I'd have just done a clean install. I used to be able to manage that but low level Linux deps have gotten far more complex. Worse, Slackware on these Toughbooks with the Intel WiFi hardware was -- and this blows my mind but it's perfectly replicatable -- looses my packets in Chicago. I don't think it's the Intel WiFi either though the firmware crashes constantly and constantly gets restarted. traceroute in Knoppix on the same machine shows almost no packet loss or lag; traceroute in Slackware on the same machine shows massive packet loss, terrible network performance, and high ping times. It may or may not impact wired connects. It may or may really have anything to do with the router in Chicago; the WiFi stack may have just been systematically losing replies and leaning heavily on retransmits of unack'd packets. This problem proved disastrous. Trips to Seattle and Minnesota as well as in-town coffee shops (often when the home network was down!) left me still stranded without network. An income left me the chance to upgrade hardware and that chance bit me in the ass. But, on the bright side, it's sorted out and next go, I won't be fighting with this one. I should be able to squeeze a couple years of this machine. And I have a spare. These puppies cost me $60 each on eBay. I've been bashing the Linux users that act like Microsoft users in reinstalling the latest version of the OS at the first sign of trouble, but I have to give it to these guys for being plucky and jumping into battle with just the tools de jour and doing a fantastic job of wielding them.
The thing with Slackware and Chicago reminds me of a certain FreeBSD NAT at a small company in Scottsdale that absolutely would not speak to a Windows webserver that one of their clients needed for work.
Then I got to spend some quality time with git. I did sit down and read about its internal datastructures. Bootstrapping knowledge is hard. A lot of stuff on the Web is misleading. This happens with young technologies -- people who really shouldn't pretend to be experts. Tutorials assume overly simplistic scenarios that get blown away when you're working with people that know what they're doing. You need to know a certain amount to be able to see signs that something is misleading or incorrect. I think git's reputation stems from the sort of alpha geek who early adopted git. These people get excited by cool technology but lack some human-human teaching instinct. They declare things to be "easy!" very readily and rattle off strings of commands that can easily go wrong if considerations aren't taken into account. They have no idea they're doing this. I'm generalizing several git users I've been exposed to for some time, here. I think Perl users tend to fit this same class. We're so good at what we do and we've been doing it for so long, we forget the pitfalls that novices step into. Every problem is "easy!". We're jumping to give the correct bit of code but failing to communicate what's needed to conceptualize what's happening or to otherwise generalize the knowledge. To those on the outside, this creates the impression that the technology in question is overly fickle and overly complex -- somewhat ironically, since the attempt was to portray it as easy. Anything unpredictable and hard to conceptualize is going to seem "hard". But I'm beginning to be able to conceptualize this thing. At least one individual in the git camp showed enough self awareness here to communicate that understanding git's datastructures is the key to understanding git. git does awesome things, no doubt, and with power comes a certain amount of necessary complexity. This complexity, be it in git or Perl, cannot be swept under the rug.
Then there's Zeus. I spent a week, on and off, just looking all over creation for the WSDL, to use as a cheat to figure out what the data being sent to the thing was supposed to look like. Turns out that even though the Zeus site insists that it has it, it doesn't but it can be found in the most unlikely place -- the help screen of the Zeus machine itself. Even though Zeus's admin and API are implemented in Perl, there are no Perl examples in the docs of using the datastructures. The various resources that must get built to set up a functioning loadbalancer through the API are numerous, haphazard, and badly non-normalized. The documentation lists almost every function as taking (Char names, Char values). Bloody thanks a lot. Names of which other resource? What's valid for the values? Sorting out the API took a lot of the same sort of reverse engineering I was doing right around 2000 trying to puzzle out the numerous bank credit card processing gateways before Authorize.net came along, published some *good* documentation, and ran everyone else out of business overnight (even though Authorize.net ran on Windows and had outages that would sometimes be long enough that the credit card companies reversed the charges -- something like 5 days). It's always good to get the chance to work with a product that's *expensive*. I can play with Linux all day long but you have to get a job at the right place to get to touch certain things. I should do an online article detailing what I've learned.
Oh yeah. And I got to spend a little time with Nagios. The logic for sucking Perl plugins in is just cranky. It violates an anti-pattern in caching failure and it doesn't signal failure in any useful way. I actually doubt that it knows the difference itself.
I have to thank these guys for investing in me to learn one thing after another. I wish I could repay that investment.
I haven't lost my touch in tickling bizarre bugs.
I practiced being nice, at the urging of friends. I'm still a deer in the headlights when confronted with hu-mans; I can never conceptualize what's going on in their heads, and I think my stress in facing them stresses them back out. It's like if you encounter an alien in the woods and you're all like "ARHGH!" and he's all like "ARHGGH!" and so you're all like "ARGHGH!". It's just no good. Even being nice, I have to learn how to distribute warm fuzzies. My normal model seems to be to antagonize people into answering questions. If things don't make sense, I tease people for being apparently silly. People *hate* being unintentionally, apparently silly, so this is a fantastic way to get them to answer things -- they stop what they're doing and vigorously explain away. Putting down that tool was hard. Learning other tools I expect will also be hard. Anyway, this was a wanted chance to experiment with that one.