Originally this seems to have been part of an application to do some kind of web crawling or something. I haven't even looked at that. The reason I am commenting is that a large part of this package is a collection of other people's modules, dated to when this package was released in 1997. As a result, a collection of extremely old versions of modules, which were not written by lcwa's author, have been sitting on CPAN unchanged since 1997. Maybe that would not be a problem if nobody knew about them, but in fact these extremely antiquated versions of modules turn up in the search results of search.cpan.org.
At the time of writing this review, an ancient version of WWW::RobotRules turns up as the second hit if you search for the term "robots.txt" on search.cpan.org:
Even worse, it's actually the sixth module on the list if you search for WWW::RobotRules.
When I searched for a module to read "robots.txt", I landed up at the version of WWW::RobotRules bundled in with this package. I spent quite a few minutes wondering why something like that had not been updated since 1997. I started to look for some more modern alternative, and I found my way to
which comes below the above broken module but above the legitimate WWW::RobotRules. Now it says right at the top of this, "Unlike WWW::RobotRules (which is very cool),". However, since the version of WWW::RobotRules which I'd discovered before that was obviously not very cool, I was puzzled, so I went back to the search box of search.cpan.org and found my way back to the 2011 version of WWW:::RobotRules. This surely proves that there is something wrong with my method of searching search.cpan.org and what a fool I am, but I notice that this package even got some bug reports at rt.cpan.org which should have gone to the legitimate modules, so it seems there are other fools like me.
So this package is really a distraction and a nuisance. I don't know if it was necessary to bundle up third-party modules in a package in this way in 1997, but in 2011 it is a really bad practice, so my request is that this package should somehow be removed from CPAN, made invisible, have the ancient third party modules removed, or otherwise downgraded in the search.cpan.org search results.