RSS | Module Info | Add a review of WWW-RobotRules
WWW-RobotRules
(6.01)
This module parses "robots.txt" and tells you if your url is allowed for your user agent.
(Contact me at bkb@cpan.org if you have a comment about this review.)
Ben Bullock - 2011-04-26T21:13:25 (permalink)
0 out of 1 found this review helpful.
Was this review helpful to you?
Yes
No
