lib::LWP::RobotUA
Section: User Contributed Perl Documentation
(3)
Updated: perl 5.004, patch 55
Index
Return to Main Contents
NAME
LWP::RobotUA - A class for Web Robots
SYNOPSIS
require LWP::RobotUA;
$ua = new LWP::RobotUA 'my-robot/0.1', 'me@foo.com';
$ua->delay(10); # be very nice, go slowly
...
# just use it just like a normal LWP::UserAgent
$res = $ua->request($req);
DESCRIPTION
This class implements a user agent that is suitable for robot
applications. Robots should be nice to the servers they visit. They
should consult the robots.txt file to ensure that they are welcomed
and they should not send too frequent requests.
But, before you consider writing a robot take a look at
<URL:http://info.webcrawler.com/mak/projects/robots/robots.html>.
When you use a LWP::RobotUA as your user agent, then you do not
really have to think about these things yourself. Just send requests
as you do when you are using a normal LWP::UserAgent and this
special agent will make sure you are nice.
METHODS
The LWP::RobotUA is a sub-class of LWP::UserAgent and implements the
same methods. The use_alarm() method also desides whether we will
wait if a request is tried too early (if true), or will return an error
response (if false).
In addition these methods are provided:
$ua = LWP::RobotUA->new($agent_name, $from, [$rules])
Your robot's name and the mail address of the human responsible for
the robot (i.e. you) is required by the constructor.
Optionally it allows you to specify the WWW::RobotRules object to
use.
$ua->delay([$minutes])
Set the minimum delay between requests to the same server. The
default is 1 minute.
$ua->use_sleep([$boolean])
Get/set a value indicating wether the UA should sleep() if request
arrive to fast (before $ua->delay minutes has passed). The default is
TRUE. If this value is FALSE then an internal SERVICE_UNAVAILABLE
response will be generated. It will have an Retry-After header that
indicate when it is OK to send another request to this server.
$ua->rules([$rules])
Set/get which WWW::RobotRules object to use.
$ua->no_visits($netloc)
Returns the number of documents fetched from this server host. Yes I
know, this method should probably have been named num_visits() or
something like that :-(
$ua->host_wait($netloc)
Returns the number of seconds (from now) you must wait before you can
make a new request to this host.
$ua->as_string
Returns a text that describe the state of the UA.
Mainly useful for debugging.
SEE ALSO
the LWP::UserAgent manpage, the WWW::RobotRules manpage
AUTHOR
Gisle Aas <aas@sn.no>
Index
- NAME
-
- SYNOPSIS
-
- DESCRIPTION
-
- METHODS
-
- $ua = LWP::RobotUA->new($agent_name, $from, [$rules])
-
- $ua->delay([$minutes])
-
- $ua->use_sleep([$boolean])
-
- $ua->rules([$rules])
-
- $ua->no_visits($netloc)
-
- $ua->host_wait($netloc)
-
- $ua->as_string
-
- SEE ALSO
-
- AUTHOR
-
This document was created by
man2html,
using the manual pages.
Time: 23:58:15 GMT, February 15, 2023