ꦫꦣꦺꦤ꧀ꦄꦤ꧀ꦠꦱꦺꦤ

PATH: usr / local / share / man / man3

.\" Automatically generated by Pod::Man 2.27 (Pod::Simple 3.28)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings.  \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote.  \*(C+ will
.\" give a nicer C++.  Capital omega is used to do unbreakable dashes and
.\" therefore won't be available.  \*(C` and \*(C' expand to `' in nroff,
.\" nothing in troff, for use with C<>.
.tr \(*W-
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
.    ds -- \(*W-
.    ds PI pi
.    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
.    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
.    ds L" ""
.    ds R" ""
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds -- \|\(em\|
.    ds PI \(*p
.    ds L" ``
.    ds R" ''
.    ds C`
.    ds C'
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el       .ds Aq '
.\"
.\" If the F register is turned on, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.\"
.\" Avoid warning from groff about undefined register 'F'.
.de IX
..
.nr rF 0
.if \n(.g .if rF .nr rF 1
.if (\n(rF:(\n(.g==0)) \{
.    if \nF \{
.        de IX
.        tm Index:\\$1\t\\n%\t"\\$2"
..
.        if !\nF==2 \{
.            nr % 0
.            nr F 2
.        \}
.    \}
.\}
.rr rF
.\" ========================================================================
.\"
.IX Title "LWP::RobotUA 3"
.TH LWP::RobotUA 3 "2020-08-18" "perl v5.16.3" "User Contributed Perl Documentation"
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH "NAME"
LWP::RobotUA \- a class for well\-behaved Web robots
.SH "SYNOPSIS"
.IX Header "SYNOPSIS"
.Vb 4
\&  use LWP::RobotUA;
\&  my $ua = LWP::RobotUA\->new(\*(Aqmy\-robot/0.1\*(Aq, \*(Aqme@foo.com\*(Aq);
\&  $ua\->delay(10);  # be very nice \-\- max one hit every ten minutes!
\&  ...
\&
\&  # Then just use it just like a normal LWP::UserAgent:
\&  my $response = $ua\->get(\*(Aqhttp://whatever.int/...\*(Aq);
\&  ...
.Ve
.SH "DESCRIPTION"
.IX Header "DESCRIPTION"
This class implements a user agent that is suitable for robot
applications.  Robots should be nice to the servers they visit.  They
should consult the \fI/robots.txt\fR file to ensure that they are welcomed
and they should not make requests too frequently.
.PP
But before you consider writing a robot, take a look at
<URL:http://www.robotstxt.org/>.
.PP
When you use an \fILWP::RobotUA\fR object as your user agent, then you do not
really have to think about these things yourself; \f(CW\*(C`robots.txt\*(C'\fR files
are automatically consulted and obeyed, the server isn't queried
too rapidly, and so on.  Just send requests
as you do when you are using a normal \fILWP::UserAgent\fR
object (using \f(CW\*(C`$ua\->get(...)\*(C'\fR, \f(CW\*(C`$ua\->head(...)\*(C'\fR,
\&\f(CW\*(C`$ua\->request(...)\*(C'\fR, etc.), and this
special agent will make sure you are nice.
.SH "METHODS"
.IX Header "METHODS"
The LWP::RobotUA is a sub-class of LWP::UserAgent and implements the
same methods. In addition the following methods are provided:
.SS "new"
.IX Subsection "new"
.Vb 3
\&    my $ua = LWP::RobotUA\->new( %options )
\&    my $ua = LWP::RobotUA\->new( $agent, $from )
\&    my $ua = LWP::RobotUA\->new( $agent, $from, $rules )
.Ve
.PP
The LWP::UserAgent options \f(CW\*(C`agent\*(C'\fR and \f(CW\*(C`from\*(C'\fR are mandatory.  The
options \f(CW\*(C`delay\*(C'\fR, \f(CW\*(C`use_sleep\*(C'\fR and \f(CW\*(C`rules\*(C'\fR initialize attributes
private to the RobotUA.  If \f(CW\*(C`rules\*(C'\fR are not provided, then
\&\f(CW\*(C`WWW::RobotRules\*(C'\fR is instantiated providing an internal database of
\&\fIrobots.txt\fR.
.PP
It is also possible to just pass the value of \f(CW\*(C`agent\*(C'\fR, \f(CW\*(C`from\*(C'\fR and
optionally \f(CW\*(C`rules\*(C'\fR as plain positional arguments.
.SS "delay"
.IX Subsection "delay"
.Vb 2
\&    my $delay = $ua\->delay;
\&    $ua\->delay( $minutes );
.Ve
.PP
Get/set the minimum delay between requests to the same server, in
\&\fIminutes\fR.  The default is \f(CW1\fR minute.  Note that this number doesn't
have to be an integer; for example, this sets the delay to \f(CW10\fR seconds:
.PP
.Vb 1
\&    $ua\->delay(10/60);
.Ve
.SS "use_sleep"
.IX Subsection "use_sleep"
.Vb 2
\&    my $bool = $ua\->use_sleep;
\&    $ua\->use_sleep( $boolean );
.Ve
.PP
Get/set a value indicating whether the \s-1UA\s0 should \*(L"sleep\*(R" in LWP::RobotUA if
requests arrive too fast, defined as \f(CW\*(C`$ua\->delay\*(C'\fR minutes not passed since
last request to the given server.  The default is true.  If this value is
false then an internal \f(CW\*(C`SERVICE_UNAVAILABLE\*(C'\fR response will be generated.
It will have a \f(CW\*(C`Retry\-After\*(C'\fR header that indicates when it is \s-1OK\s0 to
send another request to this server.
.SS "rules"
.IX Subsection "rules"
.Vb 2
\&    my $rules = $ua\->rules;
\&    $ua\->rules( $rules );
.Ve
.PP
Set/get which \fIWWW::RobotRules\fR object to use.
.SS "no_visits"
.IX Subsection "no_visits"
.Vb 1
\&    my $num = $ua\->no_visits( $netloc )
.Ve
.PP
Returns the number of documents fetched from this server host. Yeah I
know, this method should probably have been named \f(CW\*(C`num_visits\*(C'\fR or
something like that. :\-(
.SS "host_wait"
.IX Subsection "host_wait"
.Vb 1
\&    my $num = $ua\->host_wait( $netloc )
.Ve
.PP
Returns the number of \fIseconds\fR (from now) you must wait before you can
make a new request to this host.
.SS "as_string"
.IX Subsection "as_string"
.Vb 1
\&    my $string = $ua\->as_string;
.Ve
.PP
Returns a string that describes the state of the \s-1UA.\s0
Mainly useful for debugging.
.SH "SEE ALSO"
.IX Header "SEE ALSO"
LWP::UserAgent, WWW::RobotRules
.SH "COPYRIGHT"
.IX Header "COPYRIGHT"
Copyright 1996\-2004 Gisle Aas.
.PP
This library is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.

[+] ..
[-] ExtUtils::Packlist.3pm [edit]
[-] DBI::Const::GetInfoType.3pm [edit]
[-] DBD::Sponge.3pm [edit]
[-] DBD::mysql::INSTALL.3pm [edit]
[-] ExtUtils::Install.3pm [edit]
[-] IO::Tty.3pm [edit]
[-] JSON::backportPP.3pm [edit]
[-] ExtUtils::Liblist.3pm [edit]
[-] DBI::DBD::Metadata.3pm [edit]
[-] Net::SSL.3pm [edit]
[-] libwww::lwpcook.3pm [edit]
[-] CPAN::Meta::History.3pm [edit]
[-] JSON::backportPP::Compat5006.3pm [edit]
[-] ExtUtils::Installed.3pm [edit]
[-] DBI::Gofer::Serializer::Storable.3pm [edit]
[-] IO::Tty::Constant.3pm [edit]
[-] ExtUtils::MM_UWIN.3pm [edit]
[-] LWP::MemberMixin.3pm [edit]
[-] LWP::UserAgent.3pm [edit]
[-] DBD::Gofer::Transport::Base.3pm [edit]
[-] DBD::File::Developers.3pm [edit]
[-] POD2::PT_BR::local::lib.3pm [edit]
[-] ExtUtils::MakeMaker::FAQ.3pm [edit]
[-] JSON::backportPP::Boolean.3pm [edit]
[-] DBI::Gofer::Request.3pm [edit]
[-] DBI::DBD::SqlEngine::Developers.3pm [edit]
[-] DBI::ProfileDumper.3pm [edit]
[-] LWP::Simple.3pm [edit]
[-] Bundle::DBI.3pm [edit]
[-] DBD::Gofer::Policy::pedantic.3pm [edit]
[-] DBI::ProfileData.3pm [edit]
[-] Net::HTTP.3pm [edit]
[-] libwww::lwptut.3pm [edit]
[-] Expect.3pm [edit]
[-] CPAN::Meta::Spec.3pm [edit]
[-] CPAN::Meta::Feature.3pm [edit]
[-] JSON.3pm [edit]
[-] JSON::backportPP::Compat5005.3pm [edit]
[-] Win32::DBIODBC.3pm [edit]
[-] ExtUtils::MakeMaker.3pm [edit]
[-] ExtUtils::MM_VOS.3pm [edit]
[-] DBD::Gofer::Transport::corostream.3pm [edit]
[-] ExtUtils::MM_OS2.3pm [edit]
[-] DBD::Gofer::Transport::null.3pm [edit]
[-] DBD::Gofer::Policy::classic.3pm [edit]
[-] IO::Socket::INET6.3pm [edit]
[-] ExtUtils::MM_Cygwin.3pm [edit]
[-] JSON::PP.3pm [edit]
[-] HTTP::Tiny.3pm [edit]
[-] Types::Serialiser.3pm [edit]
[-] ExtUtils::MM_Darwin.3pm [edit]
[-] Canary::Stability.3pm [edit]
[-] DBI::Util::CacheMemory.3pm [edit]
[-] DBI::PurePerl.3pm [edit]
[-] ExtUtils::MakeMaker::Tutorial.3pm [edit]
[-] DBI::Gofer::Transport::pipeone.3pm [edit]
[-] DBI::W32ODBC.3pm [edit]
[-] Path::Class::File.3pm [edit]
[-] CPAN::Meta::Requirements.3pm [edit]
[-] Net::HTTP::Methods.3pm [edit]
[-] DBI::Gofer::Serializer::DataDumper.3pm [edit]
[-] DBI::SQL::Nano.3pm [edit]
[-] ExtUtils::MM_BeOS.3pm [edit]
[-] ExtUtils::MM_QNX.3pm [edit]
[-] JSON::Syck.3pm [edit]
[-] version::Internals.3pm [edit]
[-] ExtUtils::MY.3pm [edit]
[-] IO::Pty.3pm [edit]
[-] DBI::Gofer::Response.3pm [edit]
[-] DBI::Gofer::Transport::Base.3pm [edit]
[-] ExtUtils::MM_MacOS.3pm [edit]
[-] ExtUtils::MM_DOS.3pm [edit]
[-] LWP::Authen::Ntlm.3pm [edit]
[-] DBI::ProxyServer.3pm [edit]
[-] DBI::Gofer::Serializer::Base.3pm [edit]
[-] Log::NullLogLite.3pm [edit]
[-] JSON::XS::Boolean.3pm [edit]
[-] LWP.3pm [edit]
[-] common::sense.3pm [edit]
[-] DBD::File::HowTo.3pm [edit]
[-] Types::Serialiser::Error.3pm [edit]
[-] LWP::Protocol::https.3pm [edit]
[-] Crypt::SSLeay.3pm [edit]
[-] Net::HTTPS.3pm [edit]
[-] POD2::DE::local::lib.3pm [edit]
[-] ExtUtils::testlib.3pm [edit]
[-] DBD::Gofer::Transport::stream.3pm [edit]
[-] ExtUtils::Command::MM.3pm [edit]
[-] ExtUtils::MM_AIX.3pm [edit]
[-] DBI::Const::GetInfo::ANSI.3pm [edit]
[-] JSON::XS.3pm [edit]
[-] ExtUtils::MM_Win32.3pm [edit]
[-] LWP::Debug.3pm [edit]
[-] Path::Class.3pm [edit]
[-] ExtUtils::MM_Any.3pm [edit]
[-] DBD::Gofer::Policy::Base.3pm [edit]
[-] DBD::Gofer::Transport::pipeone.3pm [edit]
[-] DBI::Profile.3pm [edit]
[-] IO::LockedFile::Flock.3pm [edit]
[-] CPAN::Meta.3pm [edit]
[-] Path::Class::Dir.3pm [edit]
[-] ExtUtils::MakeMaker::Locale.3pm [edit]
[-] LWP::ConnCache.3pm [edit]
[-] DBI::Gofer::Transport::stream.3pm [edit]
[-] CPAN::Meta::YAML.3pm [edit]
[-] DBD::Gofer.3pm [edit]
[-] ExtUtils::Command.3pm [edit]
[-] JSON::PP::Boolean.3pm [edit]
[-] lib::core::only.3pm [edit]
[-] LWP::Protocol.3pm [edit]
[-] LWP::RobotUA.3pm [edit]
[-] Try::Tiny.3pm [edit]
[-] DBI::ProfileSubs.3pm [edit]
[-] DBD::DBM.3pm [edit]
[-] Crypt::SSLeay::Version.3pm [edit]
[-] ExtUtils::MM_Win95.3pm [edit]
[-] ExtUtils::MM.3pm [edit]
[-] CPAN::Meta::Prereqs.3pm [edit]
[-] IO::LockedFile.3pm [edit]
[-] YAML::Syck.3pm [edit]
[-] ExtUtils::Mkbootstrap.3pm [edit]
[-] DBI::Const::GetInfo::ODBC.3pm [edit]
[-] DBD::Gofer::Policy::rush.3pm [edit]
[-] DBI::DBD.3pm [edit]
[-] ExtUtils::MakeMaker::Config.3pm [edit]
[-] version.3pm [edit]
[-] Devel::CheckLib.3pm [edit]
[-] CPAN::Meta::Converter.3pm [edit]
[-] DBI::DBD::SqlEngine::HowTo.3pm [edit]
[-] ExtUtils::MM_NW5.3pm [edit]
[-] CPAN::Meta::Validator.3pm [edit]
[-] DBD::File::Roadmap.3pm [edit]
[-] Log::LogLite.3pm [edit]
[-] DBD::mysql.3pm [edit]
[-] DBI::Gofer::Execute.3pm [edit]
[-] DBD::File.3pm [edit]
[-] DBD::Proxy.3pm [edit]
[-] Mozilla::CA.3pm [edit]
[-] ExtUtils::Mksymlists.3pm [edit]
[-] ExtUtils::MM_Unix.3pm [edit]
[-] local::lib.3pm [edit]
[-] Net::HTTP::NB.3pm [edit]
[-] ExtUtils::Manifest.3pm [edit]
[-] ExtUtils::MM_VMS.3pm [edit]
[-] DBI::ProfileDumper::Apache.3pm [edit]
[-] Bundle::DBD::mysql.3pm [edit]
[-] Parse::CPAN::Meta.3pm [edit]
[-] CPAN::Meta::Merge.3pm [edit]
[-] Path::Class::Entity.3pm [edit]
[-] DBI::DBD::SqlEngine.3pm [edit]
[-] DBI::Const::GetInfoReturn.3pm [edit]
[-] DBI.3pm [edit]
[-] DBD::Mem.3pm [edit]