urllib
This module provides a high-level interface for fetching data across
the World-Wide Web. In particular, the urlopen function is
similar to the built-in function open, but accepts URLs
(Universal Resource Locators) instead of filenames. Some restrictions
apply — it can only open URLs for reading, and no seek operations
are available.
it defines the following public functions:
Restrictions:
- Currently, only the following protocols are supported: HTTP, (versions
0.9 and 1.0), Gopher (but not Gopher-+), FTP, and local files.
- The caching feature of urlretrieve() has been disabled until I
find the time to hack proper processing of Expiration time headers.
- There should be a function to query whether a particular URL is in
the cache.
- For backward compatibility, if a URL appears to point to a local file
but the file can't be opened, the URL is re-interpreted using the FTP
protocol. This can sometimes cause confusing error messages.
- The urlopen() and urlretrieve() functions can cause
arbitrarily long delays while waiting for a network connection to be
set up. This means that it is difficult to build an interactive
web client using these functions without using threads.
- The data returned by urlopen() or urlretrieve() is the
raw data returned by the server. This may be binary data (e.g. an
image), plain text or (for example) HTML. The HTTP protocol provides
type information in the reply header, which can be inspected by
looking at the Content-type header. For the Gopher protocol,
type information is encoded in the URL; there is currently no easy way
to extract it. If the returned data is HTML, you can use the module
htmllib to parse it.
htmllib
- Although the urllib module contains (undocumented) routines to
parse and unparse URL strings, the recommended interface for URL
manipulation is in module urlparse.
urlparse