Slash Boxes
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Matts (1087)

  (email not shown publicly)

I work for MessageLabs [] in Toronto, ON, Canada. I write spam filters, MTA software, high performance network software, string matching algorithms, and other cool stuff mostly in Perl and C.

Journal of Matts (1087)

Tuesday July 06, 2004
04:16 PM

Really stupid programmer error

[ #19718 ]

So today I rolled out an RPM to 2500 linux servers that had a post install script that tried to HUP my spam scanner. It had this section of code:

    my $post = <<"EOT";
ps afxww |grep my_proc_name|grep -v grep|tail --lines=+2|awk '{print$1}'|xargs kill -HUP

You have to be pretty awake to spot the problem - the $1 in the awk script is interpolated in the string, so the awk script ends up being "awk '{print}'". This is very bad indeed. 2500 linux servers had to be rebooted. I owe a lot of beers.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
More | Login | Reply
Loading... please wait.
  • Bummer... :-)

    Random thoughts, which may or may not be helpful:

    • Skip the "grep -v grep" step:

      ps afxww |grep 'm[y]_proc_name'|tail --lines=+2|awk '{print$1}'|xargs kill -HUP

    • Replace the whole line with killall (if available):

      killall -HUP 'm[y]_proc_name'

    • Don't wrap commands in doublequotes :-)


    • I once thought I could get around the "grep", "grep -v", AND the "tail" by doing:
      ps ... | awk 'NR > 1 && /[m]y_proc_name/{print $1}'
      But (maybe depending on your awk) awk blows up when there's more than 99 fields, and there were often 'ps' lines which had very long command lines. So it was good to filter before piping to awk (or maybe just do the whole thing in perl :).
    • Even better, change: ps afxww |grep 'm[y]_proc_name'|tail --lines=+2|awk '{print$1}'|xargs kill -HUP to ps afxww |grep 'm[y]_proc_name'|tail --lines=+2|awk '{print$1}'|less the first time you run it, and make sure those are the results you expected!
  • that programmers should never be given root...not /ever/. I hope those sysadmins guilt you out of enough beer that you begin to feel queasy the next time you even think about root. :)
    • While that's valid, it's not strictly relevant here as the RPM has to do this, and the programmers have to write the RPM generation code.

      The real issue is it not going through QA (and this happened because the code to build this RPM has been in the QA queue for 6 months and keeps getting overlooked in favour of more important projects).

      It also shows that developers need sleep. I'm very tired right now and not sleeping well and am making stupid decisions because of this.
    • Because, you know, there have never been any sysadmins who've made a mistake while they're working as root :-)