I spent all day poking at writing a chapter on spiders.
I think there's roughly four kinds of LWP programs:
I constantly write programs that do the first two, but I don't generally call them "spiders". I generally reserve the term "spider" for the last two.
I can imagine writing a program that does the third (a single-site spider), and indeed I'm doing so for the chapter, and I think it'll comprise the meat of the chapter, showing off LWP::RobotUA.
But it seems that many people want to write a program like the fourth -- a freely-traversing spider -- and for them I'm trying to muster something more useful than just saying "DON'T! [endchapter]".
After spending much of the day watching the cursor blink ON and OFF and ON and OFF, I think that that section will be "Don't, because... [fifty good reasons]".
Because just about everyone I know who admins a server, has had some numb-skull's useless aimless spider come and hammer their server senseless for just no good reason. "I was just searching the whole web for pages about Duran Duran, is all! How was I supposed to know your host would contain an infinite URL-space, in its events calendar site?"