robotparser

robotparser - Navigation index modules | next | previous |...

Info iconThis preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon
Navigation index modules | next | previous | Python v2.6.5c1 documentation » The Python Standard Library » 14. File Formats » 14.3. robotparser — Parser for robots.txt Note The robotparser module has been renamed urllib.robotparser in Python 3.0. The 2to3 tool will automatically adapt imports when converting your sources to 3.0. This module provides a single class, RobotFileParser , which answers questions about whether or not a particular user agent can fetch a URL on the Web site that published the robots.txt file. For more details on the structure of robots.txt files, see http://www.robotstxt.org/orig.html . class robotparser.RobotFileParser This class provides a set of methods to read, parse and answer questions about a single robots.txt file. set_url ( url ) Sets the URL referring to a robots.txt file. read () Reads the robots.txt URL and feeds it to the parser. parse
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 2
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 03/07/2010 for the course CS 6913 taught by Professor Torsensuel during the Spring '10 term at NYU Poly.

Page1 / 2

robotparser - Navigation index modules | next | previous |...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online