scot's Journal

Automated downloading of NOAA radar images

scot — 2007-06-13T21:47:15+00:00

Inspired by Develop your own weather maps and alerts with Perl and GD, I have put together a script which downloads the base radar image from a specific site at a specified interval.

use strict; use LWP::Simple; use POSIX; my $refresh = 1800; do { my $base="http://radar.weather.gov/ridge/RadarImg/N0R/MTX_N0R_0.gif"; my $timestring = strftime( "%Y%m%d_%H%M%S", gmtime() ); my $savefilename = "SLCRadar_".$timestring.".gif"; my $status = getstore($base,$savefilename); print $status."\n" ; sleep $refresh; } while ($refresh); <blockquote>

See my Standard Code Disclaimer

Automated extraction of NOAA weather satellite images

scot — 2007-05-29T22:45:27+00:00

NOAA updates its weather satellite images on the GOES website every 30 minutes, with images in infrared, visible light, and water vapor wavelengths; for both the eastern and western halves of the US. The script below grabs those images and saves them to a local file with a timestamp. Could prove useful for doing your own weather forecasting or whatever....I have updated the script to use the "strftime" function from the POSIX module,and the filenames now are very precise (for example "eastconusir20070530_074348.jpg"). I also updated the script to timestamp the images with Greenwich Mean Time...thanks to graff and jasonk over at Perlmonks for their feedback...

use strict; use LWP::Simple; use POSIX; my $image; my $url; my %images = ( "eastconusir" => "http://www.goes.noaa.gov/GIFS/ECIR.JPG", "eastconusvis" => "http://www.goes.noaa.gov/GIFS/ECVS.JPG", "eastconuswv" => "http://www.goes.noaa.gov/GIFS/ECWV.JPG", "westconusir" => "http://www.goes.noaa.gov/GIFS/WCIR.JPG", "westconusvis" => "http://www.goes.noaa.gov/GIFS/WCVS.JPG", "westconusvwv" => "http://www.goes.noaa.gov/GIFS/WCWV.JPG", ); foreach my $key (keys %images) { print $key."\n"; my $epoch = ( head( $images{$key} ) )[ 2 ] || next; my $timestring = strftime( "%Y%m%d_%H%M%S", gmtime( $epoch ) ); print $timestring."\n"; print $images{$key}."\n"; my $status = getstore($images{$key},$key.$timestring.".jpg"); print $status."\n" ; };

Sun's Java Web Server superior to Apache?

scot — 2007-04-23T18:42:03+00:00

Serverwatch has posted a review of Sun's Java Web Server which is very positive. My summary of the review is here.

Method for extracting urls for filetypes + auto-retrieval

scot — 2007-04-04T22:51:12+00:00

This post reminded me of a problem which I have been trying to solve involving extracting URL's pointing to a specific filetype (say a gz archive) from a web page. It turns out that at CPAN there is a page which contains an alphabetical list of all modules, with a hyperlink to the tar.gz file of each module.

The following code (given appropriate substitution of the command line input; ie gz for pdf) will create a text file with all of the URL's for the tar.gz files:

use strict; use LWP::Simple; use HTML::SimpleLinkExtor; #usage getfileoftype http://www.example.com pdf > urllist.txt my $url = shift; my $filetype = shift; my $filetypelen = length($filetype); my $offset = -$filetypelen; #print $filetypelen."\n"; #print $offset."\n"; my $fileget = getstore($url,"tempfile.html"); my $extor = HTML::SimpleLinkExtor->new(); $extor->parse_file("tempfile.html"); my @a_hrefs = $extor->a; for my $element (@a_hrefs) { # print $element; # print "\n"; my $suffix = substr($element,$offset,$filetypelen); # print $suffix; # print "\n"; if ($suffix =~ m/$filetype/){ print $element; print "\n"; } }

Once you have that, you can then use the following code to automatically download all of the modules if you so choose, or whatever subset of the modules you wish to extract from the text file created by the above code:

use strict; use LWP::Simple; use File::Basename; open (DATA, "urllist.txt") || die "File open failure!"; while (my $downloadurl = <DATA>){ (my $name, my $path, my $suffix) = fileparse($downloadurl); my $finurl = $downloadurl; print $finurl."\n"; my $savefilename = $name.$suffix; print $savefilename; print "\n"; my $status = getstore($finurl,$savefilename); print $status."\n" }

Both pieces of code work nicely on my WinXP box. Yes, I know that "tempfile.html" gets clobbered, but I was just glad to get this code working, and WinXP doesn't seem to care. In any case, one can now generate a local repository of modules. Suggestions for improvement in my code are welcome!

use.perl posts hit google quickly

scot — 2007-04-03T19:11:12+00:00

use.perl posts frequently make it into the top hits of a google search.

Remote computing with a Linux app server farm

scot — 2007-02-13T16:44:01+00:00

IBM Developerworks writes up a slick setup at the University of California where they replaced Windows XP boxes with ITX Mini-boxes in a couple of computer labs and got positive feedback from the endusers plus significant cost savings.

Script to calculate digits of pi

scot — 2007-01-24T21:31:51+00:00

Introducing my blog on Perl, Red Cloud Research, with this entry.

Dev language poll

scot — 2007-01-11T21:11:09+00:00

From here: Developers Embrace Java, Drop Visual Basic

By Gregg Keizer

Use of Visual Basic has dropped 35% since the spring, says a poll of more than 430 North American developers done by research company Evans Data Corp. Developers have abandoned Microsoft's Visual Basic in droves during the last six months, and they're using Java more than any other development language, according to a recently published survey.
Use of Visual Basic has dropped 35% since the spring, says a poll of more than 430 North American developers done by research company Evans Data. "Microsoft has dominated languages since the early 90s, but we are seeing much more parity now," said John Andrews, president of Evans Data, in a statement. "The use of scripting languages, as well as Java, appears to have limited VB's future market potential."
Developers aren't only leaving Visual Basic, they're also less likely to work in the Visual Basic.Net environment; VB.Net use is down 26%, the survey shows.
Java now holds the top spot, with 45% of the polled developers saying they used Java during some part of the last six months. C/C++, meanwhile, was used by 40% of the coders, and C# was used by 32%.
The survey also indicates that use of Ajax, short for Asynchronous JavaScript and XML, is up 10% since the spring 2006 poll, with 28% of the developers involved in Ajax-style Web interface development at some point during the last six months.

"A Java Front-End To CGI/Perl "

scot — 2007-01-09T18:48:28+00:00

Found this JavaCGI Bridge Presentation which based on a cursory read looks like an alternative to AJAX. The gist is this:

JavaCGIBridge Solution

CGI/Perl advantages

* CGI/Perl can provide database connectivity more easily
* Application related "business-rules" are centrally maintained and executed in CGI/Perl scripts
* Browsers that don't support Java can still use the application
* Leverage existing "legacy" CGI/Perl code
* Many Internet Service Providers only allow CGI/Perl access (no daemons)

Java advantages

* Java applet becomes a truly thin client
- only requires GUI code and GUI logic
- JavaCGIBridge class adds ~5k overhead
- No need to learn the entire Java class library.
- You can get by with AWT (Forms) and some utility classes
* Java applet can maintain state between all CGI script calls
* Java applet can cache retrieved data
- eliminates the need to constantly get redundant data from the server

Check the date of the presentation: 1997 :) !!

Simple extraction of links from web page

scot — 2007-01-02T21:35:02+00:00

It took me far longer than I thought it would to come up with this code that grabs a web page and stuffs all the page's hyperlinks into a text file.

Updated...

use strict; use WWW::Mechanize; #usage perl linkextractor.pl http://www.example.com/ > output.txt my $url = shift; my $mech = WWW::Mechanize->new(); $mech->get($url); my $status=$mech->status(); print $status." OK-URL request succeeded."."\n"; my @links = $mech->links; print STDOUT ($_->url, $/) foreach $mech->links;

Simple automated file retriever

scot — 2007-01-02T18:27:24+00:00

I whipped this together to grab web pages on a daily basis:

use strict; use LWP::Simple; use Time::localtime; use File::Basename;

open (DATA, "urllist.txt") || die "File open failure!"; while (my $downloadurl = <DATA>){ (my $name, my $path, my $suffix) = fileparse($downloadurl); my $finurl = $downloadurl; print $finurl."\n"; my $lt = localtime(); my $now_string = $lt->mon." ".$lt->mday;

my $savefilename = $now_string.$name.$suffix; print $savefilename; print "\n"; my $status = getstore($finurl,$savefilename); print $status."\n" }

Java + Perl = Good stuff

scot — 2006-12-22T23:37:43+00:00

This article is definitely worth a look. From the article: "The key to using JCR from Perl is Inline::Java. This library allows a Perl program to call Java methods with very little effort."

Today's quote from St. Larry

scot — 2006-12-14T18:17:37+00:00

Much as I hate to say it, the Computer Science view of language design has gotten too inbred in recent years. The Computer Scientists should pay more attention to the Linguists, who have a much better handle on how people prefer to communicate. --Larry Wall

Amen!

Pugs-Perl6 on Windows is impressive

scot — 2006-12-14T18:09:26+00:00

I've been meaning to take a look at Perl 6 and Pugs and found Jonathon Worthington's site. Installing and running Perl6-Pugs on my Windows box from Jonathan's binary was astoundingly quick and easy. He's got a nice "Quick Start" tutorial as well. ++JW.

Review of "Perl in a Nutshell" 2nd edition

scot — 2006-12-13T19:49:19+00:00

This journal entry is my working draft for eventually submitting a review as a story. My thesis is that this book is a solid addition for any Perl coder's bookshelf. Why?
-someone with experience in another language could pick this book and be productive fairly quickly
-plenty of useful code snippets
-section with usage summary for all of the standard modules
-sections on CGI gives you more than just what's in CGI.pm but form a good intro to CGI in general
Book details
Title: Perl in a Nutshell
by Stephen Spainhour, Ellen Siever, Nathan Patwardhan Second Edition: June 2002
Series: In a Nutshell
ISBN 10: 0-596-00241-6
ISBN 13: 9780596002411
Pages: 760
Current price at oreilly.com: $39.95

This site's daily Wall quote

scot — 2006-11-30T16:15:34+00:00

This morning's quote from Larry got me chuckling: "Let's say the docs present a simplified view of reality... :-)". I created myself a little file to paste these quotes into for future enjoyment. Nice site feature!

Smart move by Sun Microsystems

scot — 2006-11-29T21:55:53+00:00

The following popped up on the Sun Developer Network RSS feed a few days ago:

"For a limited time, Sun is offering a free DVD media kit which includes the Solaris 10 Operating System for both SPARC and x86 platforms as well as Sun Studio 11 software.
Take this opportunity to get familiar with the most advanced operating system on the planet and the tools which enable the highest optimizations and best runtime performance on the Solaris Operating System, bar-none.
Sun Studio software provides optimizing C, C++ and Fortan compilers, visual performance tools, and high performance libraries to enhance your Solaris development environment."

The cost to Sun to distribute DVD's will be minimal compared to the potential increase in market share for/usage of Solaris. There's no question that Solaris has always been a rock-solid *nix, and if Sun can get the open source community using Solaris the benefits to Sun are obvious.

EnterpriseDB-PostGreSQL rebranded

scot — 2006-11-29T19:47:15+00:00

I stumbled onto the website of EnterpriseDB which markets a product called EnterpriseDB which appears to be essentially a re-branded PostGreSQL. I will be very interested to see if CPAN modules targeted to PostGreSQL work out of the box with this database. Presumably, they would. My super search of Perlmonks.org turned up exactly zero references to EnterpriseDB, which was mildly surprising.

The biggest roadblock to more widespread adoption of PostGreSQL in my opinion has been its horrible name. Any branding or marketing consultant worth their salt would blanch in horror if a client suggested giving a product such a hard to spell, difficult to pronounce name. I don't know how any CIO could sell switching to PostGreSQL; you can just imagine the CEO saying "you want us to switch to Post-what? post-grrsklll? post grease-kill? Forget it."