Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

joedoc (3603)

joedoc
  (email not shown publicly)
http://attaboy.tommydoc.net/
AOL IM: joedoc21 (Add Buddy, Send Message)

Journal of joedoc (3603)

Wednesday November 13, 2002
10:51 AM

why technology is just....friggin' awesome

As long as I've worked in this industry, I'm still awed by what we're able to do in regards to technology and how it affects us so casually. Maybe it's my age (almost 48) and the fact that I've spanned generations of not having all this stuff, then having it all.

Case in point: I have a DirecTV satellite dish at my home. Last week, they began offering local channels on the dish. I really wanted to get this because I currently have a YAGI-style antenna on my house, and it's just a big lightning rod.

{Go ahead, lecture me on proper grounding. I have all kinds of grounding, and yet the previous antenna too a direct hit this summer, taking down many electronic devices in my house. All the grounding in the world doesn't mean manure to a 50,000 volt whacking from the sky.}

Anyway, in order to get this new feature, I had to upgrade a receiver and install a slightly larger dish that can point to two satellites. I was installing this last night, and managed to get it pointed and drawing a signal pretty easily.

I was tightening down the bolts that hold everything in place when the amazement occurred to me. I recall an event...jeez, can it be 40, 42 years ago...when the first satellite TV transmission between Japan and the US was broadcast on one of the networks, on a morning news show. I remember the brief interview done with what I believe was a member of the Japanese government.

The transmission was a bit fuzzy and occasionally dropped out for a second, (plus it was in black-and-white, which is what Mom and Dad provided for us back then), but I have this vivid memory of this and how it seemed like a big deal...live TV from around the world via space.

Now, here I am, 40-something years later, dropping a 23" platter to a mounting pole in my backyard, and twisting it to an approximate location until the receiver in my bedroom gives me a nice strong signal. The two LNBs on the arm are about six inches apart, yet they pull down all this stuff from two satellites, 25,000 miles up and 18 degrees apart in space. One can't help but to pause at the wonder of it all. And, yet we often take all of this for granted.

On this satellite web forum I visit, I hear people bitching about the occasional picture problems they have (DBS uses a lot of compression to get everything on the bird, and this can cause some occasional idiosyncracies in picture quality). Or moaning because there aren't enough HDTV channels on certain satellites. Please...

In 1968, I was swearing at my kid brother for leaving my copy of the Beatles' white album on the turntable with the cover off where it could get dust in the grooves. So, shut up about what you can't get on satellite yet. At the rate we're going, you'll have your own show on the bird before you know it.

Oh, yeah, Date::Calc is friggin' awsome, too. Thank you, Steffan. You rock.

Tuesday November 12, 2002
11:54 AM

Back to work

Thanks goodness for Date::Calc. I'm working on a log roll procedure that needs to run after midnight, while saving the log file with yesterday's date. No need to reinvent the wheel again.

If you've never laid 1000 square feet of St. Augustine Floritam sod, one square at a time, it isn't easy. Crap, I'm only 47, but Mr. Body sure feels old today.

I don't know if anyone here went to the University of Florida, but if so, I need not tell you of the religious ferver Gator fans hold for their football team and the Ol' Ball Coach, Steve Spurrier (or Mohamar Spurrier, as Jim Rome calls him). Football is a religion here, and Spurrier is a god, despite his exit to the NFL this year.

He brought his Washington Redskins to town Sunday, and there were a lot of soap operas associated with his visit. The gator faithful believe he should be coaching the Jaguars and that Jaguars owner Wayne Weaver should have sacked current coach Tom Coughlin in the off-season and hired Spurrier when he had the chance. Coughlin settled that argument by working his team a little harder this week (they lost 4 straight going into Sunday's game after starting 3-1). Then, they proceeded to pound the Redskins, winning 26-7 and sending the Spurrier faithful home, drunk and tails between legs. Funny how those things work.

Monday November 11, 2002
04:57 AM

Sod and Football

I suppose folks in the north would call this weather we're having "indian summer." I look at it as a break from the heat we have to deal with from May through September.

The weather was nice enough for working in the yard on Saturady, so I began the monster lawn repair project by laying in some sod on the right-of-way in front of my house. Two pallets of St. Augustine Floritam in 1 square yard pieces. Watching it go onto the ground, one long row at a time is pretty amazing. Instant lawn. Where I grew up, on Long Island, you threw seed down in the spring and hoped it would take root and get green by the summer. This is very different. I've lived down here for 18 years, and this is the first time I've ever laid in my own lawn. I've done little patches and some sprigging, but this is a trip.

Oh, yeah, it's a lot of work, too. Only half way done. Thank goodness Monday's a holiday.

The job would be done, but we had a Jaguars' home game today, and that usually kills the greater part of my entire Sunday. They've been on a skid lately, so it was nice to see them rally and beat Washington.

Friday November 08, 2002
11:15 AM

an error checking lesson

Heh. I'm so...grrr.

We have a linux firewall (iptables). We generate a lot of logfile data, naturally, so I roll them nightly and store them, in case I ever decided to look at them. Which I probably never will.

Recently built a new firewall system to replace the old one. Rack mounted, dual processor, RAID, etc. Decided to keep the same hostname and configuration as the old system, so I wouldn't have to reconfigure all the client machines to point to a new box. Copied all the necessary scripts, config files, etc, to a CD. Installed on new system. Flipped. All was well, until...

I rolled into the server room yesterday and found all the log output from iptables going to the console. WTF! I ran the syslogd script to restart syslogd and klogd, and things quieted down. Now to hunt for the problem.

The syslogd.conf file was fine. Checked the logs and noticed that the last entry to the logs was made at midnight last night/this AM. Found that the logging stopped and klogd never properly restarted.

Looked at my logroll perl script. The culprit turned out to be me and my lack of error checking. My logroll script created a file with the current date, gzipped it, then was supposed to store it in a subdirectory named after the month of the file (nov, dec, etc). Part of the routine was to roll through a hash to find the right month name and match the directory with the filename.

The problem occurred when the script looked for the "nov" subdirectory and it wasn't in the right place. When I wrote the original script, I had already manually created the directories ahead of time, so the script didn't have to check for their existence.

When I moved everything to the new firewall system, I neglected to create to directories, and the script didn't error-check for their existence and create them. Bad boy. The script stopped (or paused, actually) klogd and syslogd, and never restarted them because of the file error. And the iptables process kept on logging, only it all went to the console.

Amazing how something so fundamental could slip by me. You learn each day.

05:53 AM

IIS web long file names, and other new officers...

[ #8838 ]

We have two web servers at my job. Well, actually five, but only two matter right now.

One of the two is our main server, our image to the community and the world. My design, runs on a home-made rackmount box running Linux, Apache, Perl and all good things OSS.

The other is the box from which we serve our product, meteorological images and data. The product is either produced or stored on that system. That the product is served from it is really secondary to some folks around here, but when I suggest moving the product to the Linux system, I hear shudders. The box is an NT system. Those shudders are fear of *nix by the people who "manage" that system. So, I have to live with it.

We have a new executive officer. He's young, energetic, and a pain in my ass. He arrived right before the new commanding officer, who's a bit older, more laid back, and away a lot.

The new XO is asking for a lot of things, including something he keeps referring to as "web metrics."

{He likes to toss around phrases like "paradigm shift" and "think out of the box" and "we have to many legacy systems." These three things translate, as you all know, to "doing stuff differently", "changing everything because nothing works" and "we have too much old crap."}

I asked him what he means by "web metrics." Turns out he wants web statistics, which is what I figured, but I wanted to make him say the right words. Simple. I grab the log files from the main server, run analog, toss the resulting page on the web site, viola.

But, wait. We needs stats for the [i]products[/i] on the other server...that's our bread-and-butter. OK, no problem. Grab a log file from the NT box, run analog...oops.

Here's where I find out more about how Microsoft's way of doing things is fundamentally flawed in every way. The log files on the NT box (runs IIS version 4.0) go back three, four years. Daily logs. Gigs of them. One problem: when IIS was set up, they used the default settings for the logs, which don't put the date/time with each log entry (only at the top of each file). This makes analog choke, rendering four years of log files pretty useless. There is a utility that will add the date, but who has the time...maybe I'll write a script.

After explaining this to them, I reconfigured the NT box to do weekly logs in W3C Extended format. I gave it a couple of weeks to build up some data, then began working on the scripts to roll the logs to the intranet server and run analog on them.

{Now, I know some of you are wondering why I would go to all the trouble of building new scripts, from scratch, to do all this, when there's probably some very fine material out on the net that's already been written. Remember, I'm still pretty new at a lot of this, so building this setup will, hopefully, make me a better perl programmer. Which is the point, right?}

Now, here's Microsoft/IIS screwup number two. My plan is to have all the log entries on the Apache system go to a single log file. On Sunday night, my perl procedure will move the active long file, restsrt Apache to recreate new files, pull the rolled file to the intranet box, run analog, add to stats page, etc., etc. Not complicated.

However, there's a problem with the way IIS does weekly files. First, it names the file based on the month and the week, and you can't configure it to just give it a generic name, as you can with Apache. The second problem is their concept of a "weekly" log file. If the calendar month ends in the middle of a week, IIS closes the active log file and opens a new one. Apparently, they assume that if the month changes in mid-week, the first week of the new month must only have days from that month included. {Don't they understand the concept of flexibility?}

So, if you automate the process of grabbing those weekly files, you have to account for November beginning on Wednesday, then search for two files, cat them together, then run analog.

Naturally, perl has enough magic to permit me to figure out a way around all this. For right now, the simple way is to just run daily logs on the IIS server, have the perl script run nightly to get the daily file to the intranet server, and cat it to a weekly file there. Then, I can analog it on Sunday morning along with the Apache file.

Perhaps this is another example of evil plotting by Microsoft for world domination, blah, blah, blah, but I still have to work with their stuff. And I really can't imagine the IIS developers sitting around saying "let's make this log file thing really nasty for joedoc and every other OSS freak out there."

Besides, this is why they pay us the big bucks, right?

Thursday October 31, 2002
05:55 AM

My first

This is my first time, so be gentle.

I'm a perl newcomer when lined up with most of you, just using it regularly for a little over a year, and sporadically for a bit longer then that. I'm rapidly turning from user to advocate...

I manage a small IT department for a US Navy command. Like most other Navy sites, we have the impending doom of the Navy-Marine Corps Intranet (NMCI) deployment shadowing our every move.

What's amusing about it is that as this Windows-centric, heavily single-minded concept gets deployed (slowly and at great cost to the taxpayer), the use of open source systems and tools is experiencing a quiet but effective surge from within the dark crevices of IT shops and developer's desks.

The big initiative in the Navy's information management future is the concept of fully-standardized web portals, using concepts like XML and languages like Perl and PHP. One layer below that is the push to use an open source foundation to run all this stuff. So, the use of Linux and the BSDs and Apache is slowly becoming more apparent.

I've been doing it for some time now. About time the rest of them caught up. ;)