0 out of 1 found this review helpful:
WWW-RobotRules
(6.01)
WWW-RobotRules
(6.01)
This module parses "robots.txt" and tells you if your url is allowed for your user agent.
(Contact me at bkb@cpan.org if you have a comment about this review.)
Ben Bullock - 2011-04-26T21:13:25
Was this review helpful to you?
Yes
No
