Python HOWTO Documents

Python HOWTOs are documents that cover a single, specific topic, and attempt to cover it fairly completely. Modelled on the Linux Documentation Project's HOWTO collection, this collection is an effort to foster documentation that's more detailed than the Python Library Reference. Why not write a HOWTO of your own?

  • Advocacy:
    Discusses reasons to use Python, strategies for winning acceptance, facts and arguments you can use, and cases where you shouldn't try to suggest Python as the solution.
  • Bobo and Medusa, by Richard Jones:
    Demonstrates how to install the Bobo WWW object publishing system on either a Win32 or UNIX system.
  • Curses Programming with Python:
    An introduction to writing text-mode programs using Python and curses.
  • Idioms and Anti-Idioms in Python:
    This document can be considered a companion to the tutorial. It shows how to use Python, and even more importantly, how not to use Python.
  • Editor Configuration for Python, by Andrés Corrada-Emmanuel:
    Describes how to configure your favorite editor for writing Python code.
  • Restricted Execution (rexec) HOWTO:
    A guide to Python's restricted execution features, which allow running untrusted code inside a "cage" or "sandbox" where it can't damage the system.
  • Regular Expression HOWTO:
    An introduction to using regular expressions and the re module to process text.
  • regex-to-re HOWTO:
    Discusses how to convert Python code that uses the regex module to the newer and fancier re module.
  • Socket Programming HOWTO, by Gordon McMillan:
    Sockets are used nearly everywhere, but are often misunderstood. This is a high-level overview of sockets, that doesn't cover all the fine points (there are a lot of them), but will give you enough background to begin using them decently.
  • Sorting Mini-HOWTO, by Andrew Dalke:
    A little tutorial showing a half dozen ways to sort a list with the built-in sort() method.
  • XML HOWTO:
    Discusses XML processing with Python.


Python Advocacy HOWTO

A.M. Kuchling

amk1@bigfoot.com

Abstract:

It's usually difficult to get your management to accept open source software, and Python is no exception to this rule. This document discusses reasons to use Python, strategies for winning acceptance, facts and arguments you can use, and cases where you shouldn't try to use Python.

This document is available in several formats, including PostScript, PDF, HTML and plain ASCII, from the Python HOWTO page at http://www.python.org/doc/howto/.

  • Contents
  • 1 Reasons to Use Python
    • 1.1 Programmability
    • 1.2 Prototyping
    • 1.3 Simplicity and Ease of Understanding
    • 1.4 Java Integration
  • 2 Arguments and Rebuttals
  • 3 Useful Resources
  • About this document ...

Up One Level Next Page Python Advocacy HOWTO Contents
Up: HOWTOs Next: Contents
Release 0.03.


1 Reasons to Use Python

There are several reasons to incorporate a scripting language into your development process, and this section will discuss them, and why Python has some properties that make it a particularly good choice.

Subsections
  • 1.1 Programmability
  • 1.2 Prototyping
  • 1.3 Simplicity and Ease of Understanding
  • 1.4 Java Integration

Previous Page Up One Level Next Page Python Advocacy HOWTO Contents
Previous: Contents Up: Python Advocacy HOWTO Next: 1.1 Programmability
Release 0.03.


1.1 Programmability

Programs are often organized in a modular fashion. Lower-level operations are grouped together, and called by higher-level functions, which may in turn be used as basic operations by still further upper levels.

For example, the lowest level might define a very low-level set of functions for accessing a hash table. The next level might use hash tables to store the headers of a mail message, mapping a header name like "Date" to a value such as "Tue, 13 May 1997 20:00:54 -0400". A yet higher level may operate on message objects, without knowing or caring that message headers are stored in a hash table, and so forth.

Often, the lowest levels do very simple things; they implement a data structure such as a binary tree or hash table, or they perform some simple computation, such as converting a date string to a number. The higher levels then contain logic connecting these primitive operations. Using the approach, the primitives can be seen as basic building blocks which are then glued together to produce the complete product.

Why is this design approach relevant to Python? Because Python is well suited to functioning as such a glue language. A common approach is to write a Python module that implements the lower level operations; for the sake of speed, the implementation might be in C, Java, or even Fortran. Once the primitives are available to Python programs, the logic underlying higher level operations is written in the form of Python code. The high-level logic is then more understandable, and easier to modify.

John Ousterhout wrote a paper that explains this idea at greater length, entitled ``Scripting: Higher Level Programming for the 21st Century''. I recommend that you read this paper; see the references for the URL. Ousterhout is the inventor of the Tcl language, and therefore argues that Tcl should be used for this purpose; he only briefly refers to other languages such as Python, Perl, and Lisp/Scheme, but in reality, Ousterhout's argument applies to scripting languages in general, since you could equally write extensions for any of the langauges mentioned above.

Previous Page Up One Level Next Page Python Advocacy HOWTO Contents
Previous: 1 Reasons to Use Up: 1 Reasons to Use Next: 1.2 Prototyping
Release 0.03.


1.2 Prototyping

In The Mythical Man-Month, Fredrick Brooks suggests the following rule when planning software projects: ``Plan to throw one away; you will anyway.'' Brooks is saying that the first attempt at a software design often turns out to be wrong; unless the problem is very simple or you're an extremely good designer, you'll find that new requirements and features become apparent once development has actually started. If these new requirements can't be cleanly incorporated into the program's structure, you're presented with two unpleasant choices: hammer the new features into the program somehow, or scrap everything and write a new version of the program, taking the new features into account from the beginning.

Python provides you with a good environment for quickly developing an initial prototype. That lets you get the overall program structure and logic right, and you can fine-tune small details in the fast development cycle that Python provides. Once you're satisfied with the GUI interface or program output, you can translate the Python code into C++, Fortran, Java, or some other compiled language.

Prototyping means you have to be careful not to use too many Python features that are hard to implement in your other language. Using eval(), or regular expressions, or the pickle module, means that you're going to need C or Java libraries for formula evaluation, regular expressions, and serialization, for example. But it's not hard to avoid such tricky code, and in the end the translation usually isn't very difficult. The resulting code can be rapidly debugged, because any serious logical errors will have been removed from the prototype, leaving only more minor slip-ups in the translation to track down.

This strategy builds on the earlier discussion of programmability. Using Python as glue to connect lower-level components has obvious relevance for constructing prototype systems. In this way Python can help you with development, even if end users never come in contact with Python code at all. If the performance of the Python version is adequate and corporate politics allow it, you may not need to do a translation into C or Java, but it can still be faster to develop a prototype and then translate it, instead of attempting to produce the final version immediately.

One example of this development strategy is Microsoft Merchant Server. Version 1.0 was written in pure Python, by a company that subsequently was purchased by Microsoft. Version 2.0 began to translate the code into C++, shipping with some C++code and some Python code. Version 3.0 didn't contain any Python at all; all the code had been translated into C++. Even though the product doesn't contain a Python interpreter, the Python language has still served a useful purpose by speeding up development.

This is a very common use for Python. Past conference papers have also described this approach for developing high-level numerical algorithms; see David M. Beazley and Peter S. Lomdahl's paper ``Feeding a Large-scale Physics Application to Python'' in the references for a good example. If an algorithm's basic operations are things like "Take the inverse of this 4000x4000 matrix", and are implemented in some lower-level language, then Python has almost no additional performance cost; the extra time required for Python to evaluate an expression like m.invert() is dwarfed by the cost of the actual computation. It's particularly good for applications where seemingly endless tweaking is required to get things right. GUI interfaces and Web sites are prime examples.

The Python code is also shorter and faster to write (once you're familiar with Python), so it's easier to throw it away if you decide your approach was wrong; if you'd spent two weeks working on it instead of just two hours, you might waste time trying to patch up what you've got out of a natural reluctance to admit that those two weeks were wasted. Truthfully, those two weeks haven't been wasted, since you've learnt something about the problem and the technology you're using to solve it, but it's human nature to view this as a failure of some sort.

Previous Page Up One Level Next Page Python Advocacy HOWTO Contents
Previous: 1.1 Programmability Up: 1 Reasons to Use Next: 1.3 Simplicity and Ease
Release 0.03.


1.3 Simplicity and Ease of Understanding

Python is definitely not a toy language that's only usable for small tasks. The language features are general and powerful enough to enable it to be used for many different purposes. It's useful at the small end, for 10- or 20-line scripts, but it also scales up to larger systems that contain thousands of lines of code.

However, this expressiveness doesn't come at the cost of an obscure or tricky syntax. While Python has some dark corners that can lead to obscure code, there are relatively few such corners, and proper design can isolate their use to only a few classes or modules. It's certainly possible to write confusing code by using too many features with too little concern for clarity, but most Python code can look a lot like a slightly-formalized version of human-understandable pseudocode.

In The New Hacker's Dictionary, Eric S. Raymond gives the following definition for "compact":

Compact adj. Of a design, describes the valuable property that it can all be apprehended at once in one's head. This generally means the thing created from the design can be used with greater facility and fewer errors than an equivalent tool that is not compact. Compactness does not imply triviality or lack of power; for example, C is compact and FORTRAN is not, but C is more powerful than FORTRAN. Designs become non-compact through accreting features and cruft that don't merge cleanly into the overall design scheme (thus, some fans of Classic C maintain that ANSI C is no longer compact).

(From http://sagan.earthspace.net/jargon/jargon_18.html#SEC25)

In this sense of the word, Python is quite compact, because the language has just a few ideas, which are used in lots of places. Take namespaces, for example. Import a module with import math, and you create a new namespace called "math". Classes are also namespaces that share many of the properties of modules, and have a few of their own; for example, you can create instances of a class. Instances? They're yet another namespace. Namespaces are currently implemented as Python dictionaries, so they have the same methods as the standard dictionary data type: .keys() returns all the keys, and so forth.

This simplicity arises from Python's development history. The language syntax derives from different sources; ABC, a relatively obscure teaching language, is one primary influence, and Modula-3 is another. (For more information about ABC and Modula-3, consult their respective Web sites at http://www.cwi.nl/~steven/abc/ and http://www.m3.org.) Other features have come from C, Icon, Algol-68, and even Perl. Python hasn't really innovated very much, but instead has tried to keep the language small and easy to learn, building on ideas that have been tried in other languages and found useful.

Simplicity is a virtue that should not be underestimated. It lets you learn the language more quickly, and then rapidly write code, code that often works the first time you run it.

Previous Page Up One Level Next Page Python Advocacy HOWTO Contents
Previous: 1.2 Prototyping Up: 1 Reasons to Use Next: 1.4 Java Integration
Release 0.03.


1.4 Java Integration

If you're working with Java, JPython (http://www.python.org/jpython/) is definitely worth your attention. JPython is a re-implementation of Python in Java that compiles Python code into Java bytecodes. The resulting environment has very tight, almost seamless, integration with Java. It's trivial to access Java classes from Python, and you can write Python classes that subclass Java classes. JPython can be used for prototyping Java applications in much the same way CPython is used, and it can also be used for test suites for Java code, or embedded in a Java application to add scripting capabilities.

Previous Page Up One Level Next Page Python Advocacy HOWTO Contents
Previous: 1.3 Simplicity and Ease Up: 1 Reasons to Use Next: 2 Arguments and Rebuttals
Release 0.03.


2 Arguments and Rebuttals

Let's say that you've decided upon Python as the best choice for your application. How can you convince your management, or your fellow developers, to use Python? This section lists some common arguments against using Python, and provides some possible rebuttals.

Python is freely available software that doesn't cost anything. How good can it be?

Very good, indeed. These days Linux and Apache, two other pieces of open source software, are becoming more respected as alternatives to commercial software, but Python hasn't had all the publicity.

Python has been around for several years, with many users and developers. Accordingly, the interpreter has been used by many people, and has gotten most of the bugs shaken out of it. While bugs are still discovered at intervals, they're usually either quite obscure (they'd have to be, for no one to have run into them before) or they involve interfaces to external libraries. The internals of the language itself are quite stable.

Having the source code should be viewed as making the software available for peer review; people can examine the code, suggest (and implement) improvements, and track down bugs. To find out more about the idea of open source code, along with arguments and case studies supporting it, go to http://www.opensource.org.

Who's going to support it?

Python has a sizable community of developers, and the number is still growing. The Internet community surrounding the language is an active one, and is worth being considered another one of Python's advantages. Most questions posted to the comp.lang.python newsgroup are quickly answered by someone.

Should you need to dig into the source code, you'll find it's clear and well-organized, so it's not very difficult to write extensions and track down bugs yourself. If you'd prefer to pay for support, there are companies and individuals who offer commercial support for Python.

Who uses Python for serious work?

Lots of people; one interesting thing about Python is the surprising diversity of applications that it's been used for. People are using Python to:

  • Run Web sites
  • Write GUI interfaces
  • Control number-crunching code on supercomputers
  • Make a commercial application scriptable by embedding the Python interpreter inside it
  • Process large XML data sets
  • Build test suites for C or Java code

Whatever your application domain is, there's probably someone who's used Python for something similar. Yet, despite being useable for such high-end applications, Python's still simple enough to use for little jobs.

See http://www.python.org/psa/Users.html for a list of some of the organizations that use Python.

What are the restrictions on Python's use?

They're practically nonexistent. Consult the Misc/COPYRIGHT file in the source distribution, or http://www.python.org/doc/Copyright.html for the full language, but it boils down to three conditions.

  • You have to leave the copyright notice on the software; if you don't include the source code in a product, you have to put the copyright notice in the supporting documentation.

  • Don't claim that the institutions that have developed Python endorse your product in any way.

  • If something goes wrong, you can't sue for damages. Practically all software licences contain this condition.

Notice that you don't have to provide source code for anything that contains Python or is built with it. Also, the Python interpreter and accompanying documentation can be modified and redistributed in any way you like, and you don't have to pay anyone any licensing fees at all.

Why should we use an obscure language like Python instead of well-known language X?

I hope this HOWTO, and the documents listed in the final section, will help convince you that Python isn't obscure, and has a healthily growing user base. One word of advice: always present Python's positive advantages, instead of concentrating on language X's failings. People want to know why a solution is good, rather than why all the other solutions are bad. So instead of attacking a competing solution on various grounds, simply show how Python's virtues can help.

Previous Page Up One Level Next Page Python Advocacy HOWTO Contents
Previous: 1.4 Java Integration Up: Python Advocacy HOWTO Next: 3 Useful Resources
Release 0.03.


3 Useful Resources

http://www.fsbassociates.com/books/pythonchpt1.htm

The first chapter of Internet Programming with Python also examines some of the reasons for using Python. The book is well worth buying, but the publishers have made the first chapter available on the Web.

http://www.scriptics.com/people/john.ousterhout/scripting.html

John Ousterhout's white paper on scripting is a good argument for the utility of scripting languages, though naturally enough, he emphasizes Tcl, the language he developed. Most of the arguments would apply to any scripting language.

http://www.python.org/workshops/1997-10/proceedings/beazley.html

The authors, David M. Beazley and Peter S. Lomdahl, describe their use of Python at Los Alamos National Laboratory. It's another good example of how Python can help get real work done. This quotation from the paper has been echoed by many people:

Originally developed as a large monolithic application for massively parallel processing systems, we have used Python to transform our application into a flexible, highly modular, and extremely powerful system for performing simulation, data analysis, and visualization. In addition, we describe how Python has solved a number of important problems related to the development, debugging, deployment, and maintenance of scientific software.

http://www.pythonjournal.com/volume1/art-interview/

This interview with Andy Feit, discussing Infoseek's use of Python, can be used to show that choosing Python didn't introduce any difficulties into a company's development process, and provided some substantial benefits.

http://www.python.org/psa/Commercial.html

Robin Friedrich wrote this document on how to support Python's use in commercial projects.

http://www.python.org/workshops/1997-10/proceedings/stein.ps

For the 6th Python conference, Greg Stein presented a paper that traced Python's adoption and usage at a startup called eShop, and later at Microsoft.

http://www.opensource.org

Management may be doubtful of the reliability and usefulness of software that wasn't written commercially. This site presents arguments that show how open source software can have considerable advantages over closed-source software.

http://sunsite.unc.edu/LDP/HOWTO/mini/Advocacy.html

The Linux Advocacy mini-HOWTO was the inspiration for this document, and is also well worth reading for general suggestions on winning acceptance for a new technology, such as Linux or Python. In general, you won't make much progress by simply attacking existing systems and complaining about their inadequacies; this often ends up looking like unfocused whining. It's much better to point out some of the many areas where Python is an improvement over other systems.

Previous Page Up One Level Next Page Python Advocacy HOWTO Contents
Previous: 2 Arguments and Rebuttals Up: Python Advocacy HOWTO Next: About this document ...
Release 0.03.


About this document ...

Python Advocacy HOWTO

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.

Previous Page Up One Level Python Advocacy HOWTO Contents
Previous: 3 Useful Resources Up: Python Advocacy HOWTO
Release 0.03.


Bobo and Medusa

Richard Jones

richard.jones@bom.gov.au

Abstract:

This document will attempt to demonstrate how to install the Bobo WWW object publishing system (CGI made easy) on either a Win32 or UNIX system. All components in this HOWTO are written in Python (except, of course, the interpreter itself).

This document is available in several formats, including PostScript, PDF, HTML and plain ASCII, from the Python HOWTO page at http://www.python.org/doc/howto/.

  • Contents
  • 1 Python
  • 2 Bobo
  • 3 Medusa
  • 4 Giving it a go!
  • About this document ...

Up One Level Next Page Bobo and Medusa Contents
Up: HOWTOs Next: Contents
Release 0.01.


1 Python

Just follow the instructions that come with your Python installation package.

Previous Page Up One Level Next Page Bobo and Medusa Contents
Previous: Contents Up: Bobo and Medusa Next: 2 Bobo
Release 0.01.


2 Bobo

I have created a directory to do all my Bobo work in.

  1. On UNIX, this is /home/richard/src/bobo

  2. On Win95, this is C:\web\bobo\

You should create a similar directory to work in. For a simple test, create a directory in there called hello. In it place the file hello.py

def hello(name='World'):
    """Display a greeting"""
    return "Hello %s!" % name

From the Bobo distribution, copy cgi_module_publisher.py and CGIResponse.py to the hello directory.

Previous Page Up One Level Next Page Bobo and Medusa Contents
Previous: 1 Python Up: Bobo and Medusa Next: 3 Medusa
Release 0.01.


3 Medusa

Medusa is a snap to install - simply unpack the medusa distribution. You can place it anywhere. I put it in /home/richard/src/medusa on Unix and C:\web\medusa\ on win95, but it's entirely up to you.

Copy the bobo_handler.py file into the Medusa directory.

Make a backup of the start_medusa.py script. Edit the start_medusa.py script down at the "HTTP Server" section (search for that string). Just after the line hs.install_handler(dh), insert the following lines (depending on operating system):

On UNIX:

sys.path.append("/home/richard/src/bobo/hello")
import bobo_handler
import hello
bfh = bobo_handler.bobo_handler(hello,
                                uri_base="/cgi-bin/hello",
                                debug=1)
hs.install_handler(bfh)

On Win95:

sys.path.append("C:\\web\\bobo\\hello")
import bobo_handler
import hello
bfh = bobo_handler.bobo_handler(hello,
                                uri_base="/cgi-bin/hello",
                                debug=1)
hs.install_handler(bfh)

I've defined the debug=1 flag so that I can play around with editing the hello.py script without having to restart the Medusa server. If you don't include the flag, then you'll have to kill and restart the server each time you change the source for your application. This has a major benefit in that you can edit and test the source and not affect the running application.

Note that I've specified the URI base in these examples. It's perfectly OK to leave them out -- Medusa's Bobo handler will figure out what the base name should be (in this case, "/hello"). Unfortunately, some of the most popular web browsers do special cache tricks if the URI doesn't start with "/cgi-bin/". It's better to include a "cgi-bin" even if we're lying :)

Previous Page Up One Level Next Page Bobo and Medusa Contents
Previous: 2 Bobo Up: Bobo and Medusa Next: 4 Giving it a
Release 0.01.


4 Giving it a go!

Start Medusa with the command python start_medusa.py. On win32, you can also just double-click the script or type start start_medusa.py.

To access this from your browser, enter something like this:

http://angua/cgi-bin/hello/hello
or this:
http://angua/cgi-bin/hello/hello?name=Richard

For your labors, you are rewarded with a fine message:

Hello, World!
or
Hello, Richard!

Previous Page Up One Level Next Page Bobo and Medusa Contents
Previous: 3 Medusa Up: Bobo and Medusa Next: About this document ...
Release 0.01.


About this document ...

Bobo and Medusa

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.

Previous Page Up One Level Bobo and Medusa Contents
Previous: 4 Giving it a Up: Bobo and Medusa
Release 0.01.


Curses Programming with Python

Eric S. Raymond

esr@thyrsus.com

Abstract:

This document describes how to write text-mode programs with Python, using the curses extension module to control the display.

  • Contents
  • 1 What is curses?
    • 1.1 The Python curses module
  • 2 Starting and ending a curses Application
  • 3 Windows and Pads
  • 4 Displaying Text
    • 4.1 Attributes and Colour
  • 5 User Input
  • 6 For More Information
  • About this document ...

Up One Level Next Page Curses Programming with Python Contents
Up: HOWTOs Next: Contents
Release 2.0.


1 What is curses?

The curses library supplies a terminal-independent screen-painting and keyboard-handling facility for text-based terminals; such terminals include VT100s, the Linux console, and the simulated terminal provided by X11 programs such as xterm and rxvt. Display terminals support various control codes to perform common operations such as moving the cursor, scrolling the screen, and erasing areas. Different terminals use widely differing codes, and often have their own minor quirks.

In a world of X displays, one might ask ``why bother''? It's true that character-cell display terminals are an obsolete technology, but there are niches in which being able to do fancy things with them are still valuable. One is on small-footprint or embedded Unixes that don't carry an X server. Another is for tools like OS installers and kernel configurators that may have to run before X is available.

The curses library hides all the details of different terminals, and provides the programmer with an abstraction of a display, containing multiple non-overlapping windows. The contents of a window can be changed in various ways-adding text, erasing it, changing its appearance-and the curses library will automagically figure out what control codes need to be sent to the terminal to produce the right output.

The curses library was originally written for BSD Unix; the later System V versions of Unix from AT&T added many enhancements and new functions. BSD curses is no longer maintained, having been replaced by ncurses, which is an open-source implementation of the AT&T interface. If you're using an open-source Unix such as Linux or FreeBSD, your system almost certainly uses ncurses. Since most current commercial Unix versions are based on System V code, all the functions described here will probably be available. The older versions of curses carried by some proprietary Unixes may not support everything, though.

No one has made a Windows port of the curses module. On a Windows platform, try the Console module written by Fredrik Lundh. The Console module provides cursor-addressable text output, plus full support for mouse and keyboard input, and is available from http://effbot.org/efflib/console.

Subsections
  • 1.1 The Python curses module

Previous Page Up One Level Next Page Curses Programming with Python Contents
Previous: Contents Up: Curses Programming with Python Next: 1.1 The Python curses
Release 2.0.


1.1 The Python curses module

Thy Python module is a fairly simple wrapper over the C functions provided by curses; if you're already familiar with curses programming in C, it's really easy to transfer that knowledge to Python. The biggest difference is that the Python interface makes things simpler, by merging different C functions such as addstr, mvaddstr, mvwaddstr, into a single addstr() method. You'll see this covered in more detail later.

This HOWTO is simply an introduction to writing text-mode programs with curses and Python. It doesn't attempt to be a complete guide to the curses API; for that, see the Python library guide's serction on ncurses, and the C manual pages for ncurses. It will, however, give you the basic ideas.

Previous Page Up One Level Next Page Curses Programming with Python Contents
Previous: 1 What is curses? Up: 1 What is curses? Next: 2 Starting and ending
Release 2.0.


2 Starting and ending a curses Application

Before doing anything, curses must be initialized. This is done by calling the initscr() function, which will determine the terminal type, send any required setup codes to the terminal, and create various internal data structures. If successful, initscr() returns a window object representing the entire screen; this is usually called stdscr, after the name of the corresponding C variable.

import curses
stdscr = curses.initscr()

Usually curses applications turn off automatic echoing of keys to the screen, in order to be able to read keys and only display them under certain circumstances. This requires calling the noecho() function.

curses.noecho()

Applications will also commonly need to react to keys instantly, without requiring the Enter key to be pressed; this is called cbreak mode, as opposed to the usual buffered input mode.

curses.cbreak()

Terminals usually return special keys, such as the cursor keys or navigation keys such as Page Up and Home, as a multibyte escape sequence. While you could write your application to expect such sequences and process them accordingly, curses can do it for you, returning a special value such as curses.KEY_LEFT. To get curses to do the job, you'll have to enable keypad mode.

stdscr.keypad(1)

Terminating a curses application is much easier than starting one. You'll need to call

curses.nobreak(); curses.keypad(0); curses.echo()

to reverse the curses-friendly terminal settings. Then call the endwin() function to restore the terminal to its original operating mode.

curses.endwin()

A common problem when debugging a curses application is to get your terminal messed up when the application dies without restoring the terminal to its previous state. In Python this commonly happens when your code is buggy and raises an uncaught exception. Keys are no longer be echoed to the screen when you type them, for example, which makes using the shell difficult.

In Python you can avoid these complications and make debugging much easier by importing the module curses.wrapper. It supplies a function wrapper that takes a hook argument. It does the initializations described above, and also initializes colors if color support is present. It then runs your hook, and then finally deinitializes appropriately. The hook is called inside a try-catch clause which catches exceptions, performs curses deinitialization, and then passes the exception upwards. Thus, your terminal won't be left in a funny state on exception.

Previous Page Up One Level Next Page Curses Programming with Python Contents
Previous: 1.1 The Python curses Up: Curses Programming with Python Next: 3 Windows and Pads
Release 2.0.


3 Windows and Pads

Windows are the basic abstraction in curses. A window object represents a rectangular area of the screen, and supports various methods to display text, erase it, allow the user to input strings, and so forth.

The stdscr object returned by the initwin() function is a window object that covers the entire screen. Many programs may need only this single window, but you might wish to divide the screen into smaller windows, in order to redraw or clear them separately. The newwin() function creates a new window of a given size, returning the new window object.

begin_x = 20 ; begin_y = 7
height = 5 ; width = 40
win = curses.newwin(height, width, begin_y, begin_x)

A word about the coordinate system used in curses: coordinates are always passed in the order y,x, and the top-left corner of a window is coordinate (0,0). This breaks a common convention for handling coordinates, where the x coordinate usually comes first. This is an unfortunate difference from most other computer applications, but it's been part of curses since it was first written, and it's too late to change things now.

When you call a method to display or erase text, the effect doesn't immediately show up on the display. This is because curses was originally written with slow 300-baud terminal connections in mind; with these terminals, minimizing the time required to redraw the screen is very important. This lets curses accumulate changes to the screen, and display them in the most efficient manner. For example, if your program displays some characters in a window, and then clears the window, there's no need to send the original characters because they'd never be visible.

Accordingly, curses requires that you explicitly tell it to redraw windows, using the refresh() method of window objects. In practice, this doesn't really complicate programming with curses much. Most programs go into a flurry of activity, and then pause waiting for a keypress or some other action on the part of the user. All you have to do is to be sure that the screen has been redrawn before pausing to wait for user input, by simply calling stdscr.refresh() or the refresh() method of some other relevant window.

A pad is a special case of a window; it can be larger than the actual display screen, and only a portion of it displayed at a time. Creating a pad simply requires the pad's height and width, while refreshing a pad requires giving the coordinates of the on-screen area where a subsection of the pad will be displayed.

pad = curses.newpad(100, 100)
#  These loops fill the pad with letters; this is
# explained in the next section
for y in range(0, 100):
    for x in range(0, 100):
        try: pad.addch(y,x, ord('a') + (x*x+y*y) % 26 )
        except curses.error: pass

#  Displays a section of the pad in the middle of the screen
pad.refresh( 0,0, 5,5, 20,75)

The refresh() call displays a section of the pad in the rectangle extending from coordinate (5,5) to coordinate (20,75) on the screen;the upper left corner of the displayed section is coordinate (0,0) on the pad. Beyond that difference, pads are exactly like ordinary windows and support the same methods.

If you have multiple windows and pads on screen there is a more efficient way to go, which will prevent annoying screen flicker at refresh time. Use the methods noutrefresh() and/or noutrefresh() of each window to update the data structure representing the desired state of the screen; then change the physical screen to match the desired state in one go with the function doupdate(). The normal refresh() method calls doupdate() as its last act.

Previous Page Up One Level Next Page Curses Programming with Python Contents
Previous: 2 Starting and ending Up: Curses Programming with Python Next: 4 Displaying Text
Release 2.0.


4 Displaying Text

From a C programmer's point of view, curses may sometimes look like a twisty maze of functions, all subtly different. For example, addstr() displays a string at the current cursor location in the stdscr window, while mvaddstr() moves to a given y,x coordinate first before displaying the string. waddstr() is just like addstr(), but allows specifying a window to use, instead of using stdscr by default. mvwaddstr() follows similarly.

Fortunately the Python interface hides all these details; stdscr is a window object like any other, and methods like addstr() accept multiple argument forms. Usually there are four different forms.

Form | Description
str or ch | Display the string str or character ch
str or ch, attr | Display the string str or character ch, using attribute attr
y, x, str or ch | Move to position y,x within the window, and display str or ch
y, x, str or ch, attr | Move to position y,x within the window, and display str or ch, using attribute attr

Attributes allow displaying text in highlighted forms, such as in boldface, underline, reverse code, or in colour. They'll be explained in more detail in the next subsection.

The addstr() function takes a Python string as the value to be displayed, while the addch() functions take a character, which can be either a Python string of length 1, or an integer. If it's a string, you're limited to displaying characters between 0 and 255. SVr4 curses provides constants for extension characters; these constants are integers greater than 255. For example, ACS_PLMINUS is a +/- symbol, and ACS_ULCORNER is the upper left corner of a box (handy for drawing borders).

Windows remember where the cursor was left after the last operation, so if you leave out the y,x coordinates, the string or character will be displayed wherever the last operation left off. You can also move the cursor with the move(y,x) method. Because some terminals always display a flashing cursor, you may want to ensure that the cursor is positioned in some location where it won't be distracting; it can be confusing to have the cursor blinking at some apparently random location.

If your application doesn't need a blinking cursor at all, you can call curs_set(0) to make it invisible. Equivalently, and for compatibility with older curses versions, there's a leaveok(bool) function. When bool is true, the curses library will attempt to suppress the flashing cursor, and you won't need to worry about leaving it in odd locations.

Subsections
  • 4.1 Attributes and Colour

Previous Page Up One Level Next Page Curses Programming with Python Contents
Previous: 3 Windows and Pads Up: Curses Programming with Python Next: 4.1 Attributes and Colour
Release 2.0.


4.1 Attributes and Colour

Characters can be displayed in different ways. Status lines in a text-based application are commonly shown in reverse video; a text viewer may need to highlight certain words. curses supports this by allowing you to specify an attribute for each cell on the screen.

An attribute is a integer, each bit representing a different attribute. You can try to display text with multiple attribute bits set, but curses doesn't guarantee that all the possible combinations are available, or that they're all visually distinct. That depends on the ability of the terminal being used, so it's safest to stick to the most commonly available attributes, listed here.

Attribute | Description
A_BLINK | Blinking text
A_BOLD | Extra bright or bold text
A_DIM | Half bright text
A_REVERSE | Reverse-video text
A_STANDOUT | The best highlighting mode available
A_UNDERLINE | Underlined text

So, to display a reverse-video status line on the top line of the screen, you could code:

stdscr.addstr(0, 0, "Current mode: Typing mode",
          curses.A_REVERSE)
stdscr.refresh()

The curses library also supports colour on those terminals that provide it, The most common such terminal is probably the Linux console, followed by colour xterms.

To use colour, you must call the start_color() function soon after calling initscr(), to initialize the default colour set (the curses.wrapper.wrapper() function does this automatically). Once that's done, the has_colors() function returns TRUE if the terminal in use can actually display colour. (Note from amk: that curses uses the American spelling 'color', instead of the Canadian/British spelling 'colour'; if you're like me, you'll have to resign yourself to misspelling it for the sake of these functions.)

The curses library maintains a finite number of colour pairs, containing a foreground (or text) colour and a background colour. You can get the attribute value corresponding to a colour pair with the COLOR_PAIR() function; this can be bitwise-OR'ed with other attributes such as A_REVERSE, but again, such combinations are not guaranteed to work on all terminals.

An example, which displays a line of text using colour pair 1:

stdscr.addstr( "Pretty text", curses.COLOR_PAIR(1) )
stdscr.refresh()

As I said before, a colour pair consists of a foreground and background colour. start_color() initializes 8 basic colours when it activates colour mode. They are: 0:black, 1:red, 2:green, 3:yellow, 4:blue, 5:magenta, 6:cyan, and 7:white. The curses module defines named constants for each of these colours: curses.COLOR_BLACK, curses.COLOR_RED, and so forth.

The init_pair(n, f, b) function changes the definition of colour pair n, to foreground colour f and background colour b. Colour pair 0 is hard-wired to white on black, and cannot be changed.

Let's put all this together. To change colour 1 to red text on a white background, you would call:

curses.init_pair(1, curses.COLOR_RED, curses.COLOR_WHITE)

When you change a colour pair, any text already displayed using that colour pair will change to the new colours. You can also display new text in this colour with:

stdscr.addstr(0,0, "RED ALERT!", curses.COLOR_PAIR(1) )

Very fancy terminals can change the definitions of the actual colours to a given RGB value. This lets you change colour 1, which is usually red, to purple or blue or any other colour you like. Unfortunately, the Linux console doesn't support this, so I'm unable to try it out, and can't provide any examples. You can check if your terminal can do this by calling can_change_color(), which returns TRUE if the capability is there. If you're lucky enough to have such a talented terminal, consult your system's man pages for more information.

Previous Page Up One Level Next Page Curses Programming with Python Contents
Previous: 4 Displaying Text Up: 4 Displaying Text Next: 5 User Input
Release 2.0.


5 User Input

The curses library itself offers only very simple input mechanisms. Python's support adds a text-input widget that makes up some of the lack.

The most common way to get input to a window is to use its getch() method. that pauses, and waits for the user to hit a key, displaying it if echo() has been called earlier. You can optionally specify a coordinate to which the cursor should be moved before pausing.

It's possible to change this behavior with the method nodelay(). After nodelay(1), getch() for the window becomes non-blocking and returns ERR (-1) when no input is ready. There's also a halfdelay() function, which can be used to (in effect) set a timer on each getch(); if no input becomes available within the number of milliseconds specified as the argument to halfdelay(), curses throws an exception.

The getch() method returns an integer; if it's between 0 and 255, it represents the ASCII code of the key pressed. Values greater than 255 are special keys such as Page Up, Home, or the cursor keys. You can compare the value returned to constants such as curses.KEY_PPAGE, curses.KEY_HOME, or curses.KEY_LEFT. Usually the main loop of your program will look something like this:

while 1:
    c = stdscr.getch()
    if c == ord('p'): PrintDocument()
    elif c == ord('q'): break  # Exit the while()
    elif c == curses.KEY_HOME: x = y = 0

The curses.ascii module supplies ASCII class membership functions that take either integer or 1-character-string arguments; these may be useful in writing more readable tests for your command interpreters. It also supplies conversion functions that take either integer or 1-character-string arguments and return the same type. For example, curses.ascii.ctrl() returns the control character corresponding to its argument.

There's also a method to retrieve an entire string, getstr(). It isn't used very often, because its functionality is quite limited; the only editing keys available are the backspace key and the Enter key, which terminates the string. It can optionally be limited to a fixed number of characters.

curses.echo()            # Enable echoing of characters

# Get a 15-character string, with the cursor on the top line
s = stdscr.getstr(0,0, 15)

The Python curses.textpad module supplies something better. With it, you can turn a window into a text box that supports an Emacs-like set of keybindings. Various methods of Textbox class support editing with input validation and gathering the edit results either with or without trailing spaces. See the library documentation on curses.textpad for the details.

Previous Page Up One Level Next Page Curses Programming with Python Contents
Previous: 4.1 Attributes and Colour Up: Curses Programming with Python Next: 6 For More Information
Release 2.0.


6 For More Information

This HOWTO didn't cover some advanced topics, such as screen-scraping or capturing mouse events from an xterm instance. But the Python library page for the curses modules is now pretty complete. You should browse it next.

If you're in doubt about the detailed behavior of any of the ncurses entry points, consult the manual pages for your curses implementation, whether it's ncurses or a proprietary Unix vendor's. The manual pages will document any quirks, and provide complete lists of all the functions, attributes, and ACS_* characters available to you.

Because the curses API is so large, some functions aren't supported in the Python interface, not because they're difficult to implement, but because no one has needed them yet. Feel free to add them and then submit a patch. Also, we don't yet have support for the menus or panels libraries associated with ncurses; feel free to add that.

If you write an interesting little program, feel free to contribute it as another demo. We can always use more of them!

The ncurses FAQ: http://dickey.his.com/ncurses/ncurses.faq.html

Previous Page Up One Level Next Page Curses Programming with Python Contents
Previous: 5 User Input Up: Curses Programming with Python Next: About this document ...
Release 2.0.


About this document ...

Curses Programming with Python

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.

Previous Page Up One Level Curses Programming with Python Contents
Previous: 6 For More Information Up: Curses Programming with Python
Release 2.0.


Idioms and Anti-Idioms in Python

Moshe Zadka

howto@zadka.site.co.il

This document is placed in the public doman.

Abstract:

This document can be considered a companion to the tutorial. It shows how to use Python, and even more importantly, how not to use Python.

  • Contents
  • 1 Language Constructs You Should Not Use
    • 1.1 from module import *
      • 1.1.1 Inside Function Definitions
      • 1.1.2 At Module Level
      • 1.1.3 When It Is Just Fine
    • 1.2 Unadorned exec, execfile and friends
    • 1.3 from module import name1, name2
    • 1.4 except:
  • 2 Exceptions
  • 3 Using the Batteries
  • About this document ...

Up One Level Next Page Idioms and Anti-Idioms in Python Contents
Up: HOWTOs Next: Contents
Release 0.00.


1 Language Constructs You Should Not Use

While Python has relatively few gotchas compared to other languages, it still has some constructs which are only useful in corner cases, or are plain dangerous.

Subsections
  • 1.1 from module import *
    • 1.1.1 Inside Function Definitions
    • 1.1.2 At Module Level
    • 1.1.3 When It Is Just Fine
  • 1.2 Unadorned exec, execfile and friends
  • 1.3 from module import name1, name2
  • 1.4 except:

Previous Page Up One Level Next Page Idioms and Anti-Idioms in Python Contents
Previous: Contents Up: Idioms and Anti-Idioms in Next: 1.1 from module import
Release 0.00.


1.1 from module import *

Subsections
  • 1.1.1 Inside Function Definitions
  • 1.1.2 At Module Level
  • 1.1.3 When It Is Just Fine

Previous Page Up One Level Next Page Idioms and Anti-Idioms in Python Contents
Previous: 1 Language Constructs You Up: 1 Language Constructs You Next: 1.1.1 Inside Function Definitions
Release 0.00.


1.2 Unadorned exec, execfile and friends

The word ``unadorned'' refers to the use without an explicit dictionary, in which case those constructs evaluate code in the current environment. This is dangerous for the same reasons from import * is dangerous -- it might step over variables you are counting on and mess up things for the rest of your code. Simply do not do that.

Previous Page Up One Level Next Page Idioms and Anti-Idioms in Python Contents
Previous: 1.1.3 When It Is Up: 1 Language Constructs You Next: 1.3 from module import
Release 0.00.


1.3 from module import name1, name2

This is a ``don't'' which is much weaker then the previous ``don't''s but is still something you should not do if you don't have good reasons to do that. The reason it is usually bad idea is because you suddenly have an object which lives in two seperate namespaces. When the binding in one namespace changes, the binding in the other will not, so there will be a discrepancy between them. This happens when, for example, one module is reloaded, or changes the definition of a function at runtime.

Previous Page Up One Level Next Page Idioms and Anti-Idioms in Python Contents
Previous: 1.2 Unadorned exec, execfile Up: 1 Language Constructs You Next: 1.4 except:
Release 0.00.


1.4 except:

Python has the except: clause, which catches all exceptions. Since every error in Python raises an exception, this makes many programming errors look like runtime problems, and hinders the debugging process.

The following code shows a great example:

try:
    foo = opne("file") # misspelled "open"
except:
    sys.exit("could not open file!")

The second line triggers a NameError which is caught by the except clause. The program will exit, and you will have no idea that this has nothing to do with the readability of "file".

There are some situations in which the except: clause is useful: for example, in a framework when running callbacks, it is good not to let any callback disturb the framework.

Previous Page Up One Level Next Page Idioms and Anti-Idioms in Python Contents
Previous: 1.3 from module import Up: 1 Language Constructs You Next: 2 Exceptions
Release 0.00.


2 Exceptions

Exceptions are a useful feature of Python. You should learn to raise them whenever something unexpected occurs, and catch them only where you can do something about them.

The following is a very popular anti-idiom

def get_status(file):
    if not os.path.exists(file):
        print "file not found"
        sys.exit(1)
    return open(file).readline()

Consider the case the file gets deleted between the time the call to os.path.exists is made and the time open is called. That means the last line will throw an IOError. The same would happen if file exists but has no read permission. Since testing this on a normal machine on existing and non-existing files make it seem bugless, that means in testing the results will seem fine, and the code will get shipped. Then an unhandled IOError escapes to the user, who has to watch the ugly traceback.

Here is a better way to do it.

def get_status(file):
    try:
        return open(file).readline()
    except (IOError, OSError):
        print "file not found"
        sys.exit(1)

In this version, *either* the file gets opened and the line is read (so it works even on flaky NFS or SMB connections), or the message is printed and the application aborted.

Still, get_status makes too many assumptions -- that it will only be used in a short running script, and not, say, in a long running server. Sure, the caller could do something like

try:
    status = get_status(log)
except SystemExit:
    status = None

So, try to make as few except clauses in your code -- those will usually be a catch-all in the main, or inside calls which should always succeed.

So, the best version is probably

def get_status(file):
    return open(file).readline()

The caller can deal with the exception if it wants (for example, if it tries several files in a loop), or just let the exception filter upwards to its caller.

The last version is not very good either -- do to implementation details, the file would not be closed when an exception is raised until the handler finishes, and perhaps not at all in non-C implementations (e.g., Jython).

def get_status(file):
    fp = open(file)
    try:
        return fp.readline()
    finally:
        fp.close()

Previous Page Up One Level Next Page Idioms and Anti-Idioms in Python Contents
Previous: 1.4 except: Up: Idioms and Anti-Idioms in Next: 3 Using the Batteries
Release 0.00.


3 Using the Batteries

Every so often, people seem to be writing stuff in the Python library again, usually poorly. While the occasional module has a poor interface, it is usually much better to use the rich standard library and data types that come with Python then inventing your own.

A useful module very few people know about is os.path. It always has the correct path arithmetic for your operating system, and will usually be much better then whatever you come up with yourself.

Compare:

# ugh!
return dir+"/"+file
# better
return os.path.join(dir, file)

More useful functions in os.path: basename, dirname and splitext.

There are also many useful builtin functions people seem not to be aware of for some reason: min() and max() can find the minimum/maximum of any sequence with comparable semantics, for example, yet many people write they own max/min. Another highly useful function is reduce(). Classical use of reduce() is something like

import sys, operator
nums = map(float, sys.argv[1:])
print reduce(operator.add, nums)/len(nums)

This cute little script prints the average of all numbers given on the command line. The reduce() adds up all the numbers, and the rest is just some pre- and postprocessing.

On the same note, note that float(), int() and long() all accept arguments of type string, and so are suited to parsing -- assuming you are ready to deal with the ValueError they raise.

Previous Page Up One Level Next Page Idioms and Anti-Idioms in Python Contents
Previous: 2 Exceptions Up: Idioms and Anti-Idioms in Next: About this document ...
Release 0.00.


About this document ...

Idioms and Anti-Idioms in Python

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.

Previous Page Up One Level Idioms and Anti-Idioms in Python Contents
Previous: 3 Using the Batteries Up: Idioms and Anti-Idioms in
Release 0.00.


Editor Configuration for Python

Andrés Corrada-Emmanuel, Mamey

andres@corrada.com

  • Front Matter
  • Contents
    • 1 Alpha
      • 1.1 Features
      • 1.2 Installation
    • 2 CodeWright
      • 2.1 Features
      • 2.2 Installation
    • 3 Emacs
    • 4 jed
    • 5 vi
    • 1 How to construct a Python-mode for your editor
      • 1.1 Indentation
      • 1.2 Strings
  • About this document ...

Up One Level Next Page Editor Configuration for Python Contents
Up: HOWTOs Next: Front Matter
Release 2.0.


Front Matter

Abstract:

This document describes how to configure your favorite editor for writing Python code. Detailed instructions are provided for editors that are supported by the Python community. If your editor is not here and you would like to configure it for Python, we have included a discussion of issues specific to Python's syntax.

This document is available in several formats, including PostScript, PDF, HTML and plain ASCII, from the Python HOWTO page at http://www.python.org/doc/howto/.

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: Editor Configuration for Python Up: Editor Configuration for Python Next: Contents
Release 2.0.


1 Alpha

The Alpha editor for the Mac OS platform has a python-mode authored by Tom Fetherston. He currently maintains it, although he has neglected it in recent times and would welcome any volunteers to continue its development. The mode is implemented in Tcl.

Subsections
  • 1.1 Features
  • 1.2 Installation

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: Contents Up: Contents Next: 1.1 Features
Release 2.0.


1.1 Features

The mode supports:

  • Automatic indentation
  • Colorization of keywords and variable types

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: 1 Alpha Up: 1 Alpha Next: 1.2 Installation
Release 2.0.


1.2 Installation

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: 1.1 Features Up: 1 Alpha Next: 2 CodeWright
Release 2.0.


2 CodeWright

The CodeWright editor in the Windows platform has a Python-mode implemented as a DLL by Robert L. Pyron. The mode is implemented in C using CodeWright's API for extensions.

Subsections
  • 2.1 Features
  • 2.2 Installation

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: 1.2 Installation Up: Contents Next: 2.1 Features
Release 2.0.


2.1 Features

The mode supports:

  • Automatic and smart indentation
  • Colorization of keywords and variable types
  • Language templates

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: 2 CodeWright Up: 2 CodeWright Next: 2.2 Installation
Release 2.0.


2.2 Installation

A zip file with the source code, a compiled dll, and make file can be downloaded from:

  • ftp://www.premia.com/pub/addons/ver60/python.zip for version 6.0 users.
  • ftp://www.premia.com/pub/addons/ver51/python.zip for version 5.1 users.

The zip file can be expanded in the CodeWright directory, where it will create a Python sub-directory. Upon start-up of CodeWright the dll will be recognized and the editor can be configured by navigating to Tools Customize Language. Click ``New Type'' to add the Python extensions for which you want automatic loading of the mode. Select the Python from the lexer list to bind the mode to your extensions. Click on the ``Settings'' button to customize the features of the mode.

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: 2.1 Features Up: 2 CodeWright Next: 3 Emacs
Release 2.0.


3 Emacs

The first editor supported by the Python community was Emacs back in 1992. Tim Peters authored the first version of a major mode for editing Python programs. Currently the mode is maintained by Barry A. Warsaw. This editor currently ranks as the best supported of all by the Python community. The mode is implemented in Emacs Lisp.

A page dedicated to Emacs support can be found at http://www.python.org/emacs/python-mode. This page points you to the python-mode.el file you need to implement Python support in emacs and discusses how to install it. Enough said.

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: 2.2 Installation Up: Contents Next: 4 jed
Release 2.0.


4 jed

jed (http://space.mit.edu/%7Edavis/jed.html) is a highly programmable editor for Unix, Windows, and VMS, with much of the configurability of Emacs but little of its bulk; jed is about as large as current versions of vi. jed is distributed with a number of editing modes, including a python-mode written by Harri Pasanen and Brien Barton. The mode supports:

  • Automatic and smart indentation
  • Colorization of keywords and variable types

With current versions of jed, no special configuration steps are required to enable python-mode, as it will be automatically used when editing a file with an extension of .py.

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: 3 Emacs Up: Contents Next: 5 vi
Release 2.0.


5 vi

There are several different implementations of the vi editor for various platforms. The vim implementation (http://www.vim.org) provides syntax coloring, and a Python syntax file has been written by Neil Schemenauer.

You can automatically enable syntax coloring and automatic indentation for Python code by adding the following lines to your ~/.vimrc file:

autocmd BufRead,BufNewFile *.py syntax on
autocmd BufRead,BufNewFile *.py set ai

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: 4 jed Up: Contents Next: 1 How to construct
Release 2.0.


1 How to construct a Python-mode for your editor

Python's syntax requires special handling in the construction of a Python-mode. Two issues are particular to Python:

  • Indentation.
  • Strings

Subsections
  • 1.1 Indentation
  • 1.2 Strings

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: 5 vi Up: Contents Next: 1.1 Indentation
Release 2.0.


1.1 Indentation

Python uses indentation to group statements. A well designed mode should offer support for smart indentation of code. For example, a line beginning with the keywork ``def'' and ending with ``:'' should cause the editor to indent the following line upon hitting of the return key.

Code examples or references on how to deal with indentation should follow.

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: 1 How to construct Up: 1 How to construct Next: 1.2 Strings
Release 2.0.


1.2 Strings

Python supports three types of string literals. Strings can be enclosed in single quotes ('), double quotes ("). They can also be enclosed in groups of three single or double quotes (and are then referred to as triple-quoted strings). A well designed mode should support all three types.

In particular, a mode should correctly recognize that triple-quoted strings can include unescaped newlines and quotes.

Code examples or references on how to deal with strings should follow.

Previous Page Up One Level Next Page Editor Configuration for Python Contents
Previous: 1.1 Indentation Up: 1 How to construct Next: About this document ...
Release 2.0.


About this document ...

Editor Configuration for Python

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.

Previous Page Up One Level Editor Configuration for Python Contents
Previous: 1.2 Strings Up: Editor Configuration for Python
Release 2.0.


Regular Expression HOWTO

A.M. Kuchling

amk1@bigfoot.com

Abstract:

This document is an introductory tutorial to using regular expressions in Python with the re module. It provides a gentler introduction than the corresponding section in the Library Reference.

This document is available in several formats, including PostScript, PDF, HTML and plain ASCII, from the Python HOWTO page at http://www.python.org/doc/howto/.

  • Contents
  • 1 Introduction
  • 2 Simple Patterns
    • 2.1 Matching Characters
    • 2.2 Repeating Things
  • 3 Using Regular Expressions
    • 3.1 Compiling Regular Expressions
    • 3.2 The Backslash Plague
    • 3.3 Performing Matches
    • 3.4 Module-Level Functions
    • 3.5 Compilation Flags
  • 4 More Pattern Power
    • 4.1 More Metacharacters
    • 4.2 Grouping
    • 4.3 Non-capturing, and Named Groups
    • 4.4 Other Assertions
  • 5 Modifying Strings
    • 5.1 Splitting Strings
    • 5.2 Search and Replace
  • 6 Common Problems
    • 6.1 Know About the string Module
    • 6.2 match() versus search()
    • 6.3 Greedy versus Non-Greedy
    • 6.4 Not using re.VERBOSE
  • 7 Feedback
  • About this document ...

Up One Level Next Page Regular Expression HOWTO Contents
Up: HOWTOs Next: Contents
Release 0.03.


1 Introduction

The re module was added in Python 1.5, and provides Perl-style regular expressions. Earlier versions of Python provided the regex module, which provides Emacs-style expressions. Emacs-style is slightly less readable, and doesn't provide as many features, so there's not much reason to use the regex module when writing new code, though you should be aware of it in order to read older code.

Regular expressions (or REs) are essentially a tiny, highly specialized programming language embedded inside Python and made available through the re module. Using this little language, you specify the rules for the set of possible strings that you want to match; this set might contain English sentences, or e-mail addresses, or TeX commands, or anything you like. You can then ask questions such as ``Does this string match the pattern?'', or ``Is there a match for the pattern anywhere in this string?''. You can also use REs to modify a string, or to split it apart in various ways.

Regular expression patterns are compiled into a series of bytecodes, which are then executed by a matching engine written in C. For advanced use, it may be necessary to pay careful attention to how the engine will execute a given RE, and optimize the RE in order to produce bytecode that runs faster. Optimization isn't really covered in this document, because it requires that you have a good understanding of the matching engine's internals.

The regular expression language is relatively small and restricted, so not all possible string processing tasks can be done using regular expressions. There are also tasks that can be done with regular expressions, but the expressions turn out to be very complicated. In these cases, you may be better off writing Python code to do the processing; while Python code will be slower than an elaborate regular expression, it will also probably be more understandable.

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: Contents Up: Regular Expression HOWTO Next: 2 Simple Patterns
Release 0.03.


2 Simple Patterns

We'll start by learning about the simplest possible regular expressions. Since regular expressions are used to operate on strings, we'll start with the most common task: matching characters.

For a detailed explanation of the computer science underlying regular expressions (deterministic and non-deterministic finite automata), you can refer to almost any textbook on writing compilers.

Subsections
  • 2.1 Matching Characters
  • 2.2 Repeating Things

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 1 Introduction Up: Regular Expression HOWTO Next: 2.1 Matching Characters
Release 0.03.


2.1 Matching Characters

Most letters and characters will simply match themselves. For example, the regular expression test will match the string "test" exactly. (You can enable a case-insensitive mode that would let this RE match "Test" or "TEST" as well; more about this later.)

There are exceptions to this rule; some characters are special, and don't match themselves. Instead, they signal that some out-of-the-ordinary thing should be matched, or they affect other portions of the RE by repeating them. Much of this document is devoted to discussing various metacharacters and what they do.

Here's a complete list of the metacharacters; their meanings will be discussed in the rest of this HOWTO.

. ^ $ * + ? { [ \ | ( )
The first metacharacter we'll look at is "["; it's used for specifying a character class, which is a set of characters that you wish to match. Characters can be listed individually, or a range of characters can be indicated by giving two characters and separating them by a "-". For example, [abc] will match any of the characters "a", "b", or "c"; this is the same as [a-c], which uses a range to express the same set of characters. If you wanted to match only lowercase letters, your RE would be [a-z].

Metacharacters are not active inside classes. For example, [akm$] will match any of the characters "a", "k", "m", or "$"; "$" is usually a metacharacter, but inside a character class it's stripped of its special nature.

You can match the characters not within a range by complementing the set. This is indicated by including a "^" as the first character of the class; "^" elsewhere will simply match the "^" character. For example, [^5] will match any character except "5".

Perhaps the most important metacharacter is the backslash, "\". As in Python string literals, the backslash can be followed by various characters to signal various special sequences. It's also used to escape all the metacharacters so you can still match them in patterns; for example, if you need to match a "[" or "\", you can precede them with a backslash to remove their special meaning: \[ or \\.

Some of these special sequences represent predefined sets of characters that are often useful, such as the set of digits, or the set of letters, or the set of anything that isn't whitespace. The following predefined special sequences are available:

\d
Matches any decimal digit; this is equivalent to the class [0-9].
\D
Matches any non-digit character; this is equivalent to the class [^0-9].
\s
Matches any whitespace character; this is equivalent to the class [ \t\n\r\f\v].
\S
Matches any non-whitespace character; this is equivalent to the class [^ \t\n\r\f\v].
\w
Matches any alphanumeric character; this is equivalent to the class [a-zA-Z0-9_].
\W
Matches any non-alphanumeric character; this is equivalent to the class [^a-zA-Z0-9_].

These sequences can be included inside a character class. For example, [\s,.] is a character class that will match any whitespace character, or "," or ".".

The final metacharacter in this section is .. It matches anything except a newline character, and there's an alternate mode (re.DOTALL) where it will match even a newline. "." is often used where you want to match ``any character''.

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 2 Simple Patterns Up: 2 Simple Patterns Next: 2.2 Repeating Things
Release 0.03.


2.2 Repeating Things

Being able to match varying sets of characters is the first thing regular expressions can do that isn't already possible with Python's string module. However, if that was the only additional capability of regexes, they wouldn't be much of an advance over string. The second capability is that you can specify that portions of the RE must be repeated a certain number of times.

The first metacharacter for repeating things that we'll look at it is *. * doesn't match the literal character "*"; instead, it specifies that the previous character can be matched zero or more times, instead of exactly once.

For example, ca*t will match "ct" (0 "a"characters), "cat" (1 "a"), "caaat" (3 "a"characters), and so forth. The RE engine has various internal limitations, stemming from the size of C's int type, that will prevent it from matching over 2 billion "a" characters; you probably don't have enough memory to construct a string that large, so you shouldn't run into that limit.

Repetitions such as * are greedy; when repeating a RE, the matching engine will try to repeat it as many times as possible. If later portions of the pattern don't match, the matching engine will then back up and try again with few repetitions.

A step-by-step example will make this more obvious. Let's consider the expression a[bcd]*b. This matches the letter "a", zero or more letters from the class [bcd], and finally ends with a "b". Now imagine matching this RE against the string "abcbd".

Step | Matched | Explanation
1 | a | The a in the RE matches.
2 | abcbd | The engine matches [bcd]*, going as far as it can, which is to the end of the string.
3 | Failure | The engine tries to match b, but the current position is at the end of the string, so it fails.
4 | abcb | Back up, so that [bcd]* matches one less character.
5 | Failure | Try b again, but the current position is at the last character, which is a "d".
6 | abc | Back up again, so that [bcd]* is only matching "bc".
6 | abcb | Try b again. This time but the character at the current position is "b", so it succeeds.

The end of the RE has now been reached, and it has matched "abcb". This demonstrates how the matching engine goes as far as it can at first, and if no match is found, it will then progressively back up and retry the rest of the RE again and again. It will back up until it's tried zero matches for [bcd]*, and if that subsequently fails, it will conclude that the string doesn't match the RE at all.

Another repeating metacharacter is +, which matches one or more times. Pay careful attention to the difference between * and
regexp+; * matches zero or more times, so whatever's being repeated may not be present at all, while + requires at least one occurrence. To use a similar example, ca+t will match "cat" (1 "a"), "caaat" (3 "a"'s), but won't match "ct".

There are two more repeating qualifiers. The question mark character, ?, matches either once or zero times; you can think of it as marking something as being optional. For example, home-?brew matches either "homebrew" or "home-brew".

The most complicated repeated qualifier is {m,n}, where m and n are decimal numbers. This means there must be at least m repetitions, and at most n. For example, a/{1,3}b will match "a/b", "a//b", and "a///b". It won't match "ab", which has no slashes, or "a////b", which has four.

You can omit either m or n; in that case, a reasonable value is assumed for the missing value. Omitting m is interpreted as a lower limit of 0, while omitting n results in an upper bound of infinity -- actually, the 2 billion limit mentioned earlier, but that might as well be infinity.

Readers of a reductionist bent may notice that the 3 other qualifiers can all be expressed using this notation. {0,} is the same as *, {1,} is equivalent to +, and {0,1} is the same as ?. It's better to use *, +, or ?, simply because they're shorter and easier to read.

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 2.1 Matching Characters Up: 2 Simple Patterns Next: 3 Using Regular Expressions
Release 0.03.


3 Using Regular Expressions

Now that we've looked at some simple regular expressions, how do we actually use them in Python? The re module provides an interface to the regular expression engine, allowing you to compile REs into objects and then perform matches with them.

Subsections
  • 3.1 Compiling Regular Expressions
  • 3.2 The Backslash Plague
  • 3.3 Performing Matches
  • 3.4 Module-Level Functions
  • 3.5 Compilation Flags

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 2.2 Repeating Things Up: Regular Expression HOWTO Next: 3.1 Compiling Regular Expressions
Release 0.03.


3.1 Compiling Regular Expressions

Regular expressions are compiled into RegexObject instances, which then have methods for various operations such as searching for pattern matches or performing string substitutions.

>>> import re
>>> p = re.compile('ab*')
>>> print p
<re.RegexObject instance at 80b4150>
re.compile() also accepts an optional flags argument, used to enable various special features and syntax variations. We'll go over the available settings later, but for now a single example will do:

>>> p = re.compile('ab*', re.IGNORECASE)
The RE is passed to re.compile() as a string. REs are handled as strings because regular expressions aren't part of the core Python language, and no special syntax was created for expressing them. (There are applications that don't need REs at all, so there's no need to bloat the language specification by including them.) Instead, the re module is just a C extension module, just like the string module.

Putting REs in strings keeps the Python language simpler, but has one disadvantage which is the topic of the next section.

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 3 Using Regular Expressions Up: 3 Using Regular Expressions Next: 3.2 The Backslash Plague
Release 0.03.


3.2 The Backslash Plague

As stated earlier, regular expressions use the backslash character ("\") to indicate special forms or to allow special characters to be used without invoking their special meaning. This conflictts with Python's usage of the same character for the same purpose in string literals.

Let's say you want to write a RE that matches the string "\section", which might be found in a LATEX file. To figure out what to write in the program code, start with the desired string to be matched. Next, you must escape any backslashes and other metacharacters by preceding them with a backslash, resulting in the string "\\section". The resulting string that must be passed to re.compile() must be \\section. However, to express this as a Python string literal, both backslashes must be escaped again.

Characters | Stage
\section | Text string to be matched
\\section | Escaped backslash for re.compile
"\\\\section" | Escaped backslashes for a string literal

In short, to match a literal backslash, one has to write '\\\\' as the RE string, because the regular expression must be "\\", and each backslash must be expressed as "\\" inside a regular Python string literal. In REs that feature backslashes repeatedly, this leads to lots of repeated backslashes which makes the resulting strings difficult to understand.

The solution is to use Python's raw string notation for regular expressions; backslashes are not handled in any special way in a string literal prefixed with "r", so r"\n" is a two-character string containing "\" and "n", while "\n" is a one-character string containing a newline. Frequently regular expressions will be expressed in Python code using this raw string notation.

Regular String | Raw string
"ab*" | r"ab*"
"\\\\section" | r"\\section"
"\\w+\\s+\\1" | r"\w+\s+\1"

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 3.1 Compiling Regular Expressions Up: 3 Using Regular Expressions Next: 3.3 Performing Matches
Release 0.03.


3.3 Performing Matches

Once you have an object representing a compiled regular expression, what do you do with it? RegexObject instances have several methods and attributes. Only the most significant ones will be covered here; consult the Library Reference for a complete listing.

Method/Attribute | Purpose
match | Determine if the RE matches at the beginning of the string.
search | Scan through a string, looking for any location where this RE matches.
split | Split the string into a list, splitting it wherever the RE matches
sub | Find all substrings where the RE matches, and replace them with a different string
subn | Does the same thing as sub(), except you can limit the number of replacements

These methods return None if no match can be found. If they're successful, a MatchObject instance is returned, containing information about the match: where it starts and ends, the substring it matched, and more.

You can learn about this by interactively experimenting with the re module. (If you have Tkinter available, you may also want to look at redemo.py, a demonstration program included with the Python distribution. It allows you to enter REs and strings, and displays whether the RE matches or fails. redemo.py can be quite useful when trying to debug a complicated RE.)

First, run the Python interpreter, import the re module, and compile a RE:

Python 1.5.1 (#6, Jul 17 1998, 20:38:08)  [GCC 2.7.2.3] on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import re
>>> p = re.compile('[a-z]+')
>>> p
<re.RegexObject instance at 80c3c28>
Now, you can try matching various strings against the RE [a-z]+. An empty string shouldn't match at all, since + means 'one or more repetitions'. match() should return None in this case, which will cause the interpreter to print no output. You can explicitly print the result of match() to make this clear.

>>> p.match( "" )
>>> print p.match( "" )
None
Now, let's try it on a string that it should match, such as "tempo". In this case, match() will return a MatchObject, so you should store the result in a variable for later use.

>>> m = p.match( 'tempo')
>>> print m
<re.MatchObject instance at 80c4f68>
Now you can query the MatchObject for information about the matching string. MatchObject instances also have several methods and attributes; the most important ones are:

Method/Attribute | Purpose
group() | Return the string matched by the RE
start() | Return the starting position of the match
end() | Return the ending position of the match
span() | Return a tuple containing the (start, end) of the match

Trying these methods will soon clarify their meaning:

>>> m.group()
'tempo'
>>> m.start(), m.end()
(0, 5)
>>> m.span()
(0, 5)
group() returns the substring that was matched by the RE. start() and end() return the starting and ending index of the match. span() returns both start and end indexes in a single tuple. Since the match method only checks if the RE matches at the start of a string, start() will always be zero. However, the search method of RegexObject instances scans through the string, so the match may not start at zero in that case.

>>> print p.match('::: message')
None
>>> m = p.search('::: message') ; print m
<re.MatchObject instance at 80c9650>
>>> m.group()
'message'
>>> m.span()
(4, 11)
In actual programs, the most common style is to store the MatchObject in a variable, and then check if it was None. This usually looks like:

p = re.compile( ... )
m = p.match( 'string goes here' )
if m:
    print 'Match found: ', m.group()
else:
    print 'No match'

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 3.2 The Backslash Plague Up: 3 Using Regular Expressions Next: 3.4 Module-Level Functions
Release 0.03.


3.4 Module-Level Functions

You don't have to produce a RegexObject and call its methods; the re module also provides top-level functions called match(), search(), sub(), and so forth. These functions take the same arguments as the corresponding RegexObject method, with the RE string added as the first argument, and still return either None or a MatchObject instance.

>>> print re.match(r'From\s+', 'Fromage amk')
None
>>> re.match(r'From\s+', 'From amk Thu May 14 19:12:10 1998')
<re.MatchObject instance at 80c5978>
Under the hood, these functions simply produce a RegexObject for you and call the appropriate method on it. They also store the compiled object in a cache, so future calls to them using the same RE are faster.

Should you use these module-level functions, or should you get the RegexObject and call its methods yourself? That choice depends on how frequently the RE will be used, and on your personal coding style. If a RE is being used at only one point in the code, then the module functions are probably more convenient. If a program contains a lot of regular expressions, or re-uses the same ones in several locations, then it might be worthwhile to collect all the definitions in the same place, in a section of code that compiles all the REs ahead of time. To take an example from the standard library, here's an extract from xmllib.py:

ref = re.compile( ... )
entityref = re.compile( ... )
charref = re.compile( ... )
starttagopen = re.compile( ... )
(I generally prefer to work with the compiled object, even for one-time uses, but few people will be as much of a purist about this as I am.)

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 3.3 Performing Matches Up: 3 Using Regular Expressions Next: 3.5 Compilation Flags
Release 0.03.


3.5 Compilation Flags

Compilation flags let you modify some aspects of how regular expressions work. Flags are available in the re module under two names, a long name such as IGNORECASE, and a short, one-letter form such as I. (If you're familiar with Perl's pattern modifiers, the one-letter forms use the same letters; the short form of re.VERBOSE is re.X.) Multiple flags can be specified by bitwise OR-ing them; re.I | re.M sets both the I and M flags, for example.

Here's a table of the available flags, followed by a more detailed explanation of each one.

Flag | Meaning
DOTALL, S | Make . match any character, including newlines
IGNORECASE, I | Do case-insensitive matches
LOCALE, L | Do a locale-aware match
MULTILINE, M | Multi-line matching, affecting ^ and $
VERBOSE, X | Enable verbose REs, which can be organized more cleanly and understandably.

I
IGNORECASE
Perform case-insensitive matching; character class and literal strings will match letters by ignoring case. For example, [A-Z] will match lowercase letters, too, and Spam will match "Spam", "spam", or "spAM".1This doesn't take the current locale into account.

L
LOCALE
Make \w, \W, \b, and \B, dependent on the current locale.

Locales are a feature of the C library intended to help in writing programs that take account of language differences. For example, if you're processing French text, you'd want to be able to write \w+ to match words, but \w only matches the character class [A-Za-z]; it won't match "é" or "ç". If your system is configured properly, and a French locale is selected, certain C functions will tell the program that "é" should also be considered a letter. Setting the LOCALE flag when compiling a regular expression will cause the resulting compiled object to use these C functions for \w; this is slower, but also enables \w+ to match French words as you'd expect.

M
MULTILINE
Usually ^ matches only at the beginning of the string, and $ only at the end of the string and immediately before the newline (if any) at the end of the string. When this flag is specified, ^ matches at the beginning of the string and at the beginning of each line within the string, immediately following each newline. Similarly, the $ metacharacter matches either at the end of the string and at the end of each line (immediately preceding each newline).

S
DOTALL
Makes the "." special character match any character at all, including a newline; without this flag, "." will match anything except a newline.

X
VERBOSE
This flag allows you to write regular expressions that are more readable, by giving you more flexibility in how you can format them. When this flag has been specified, whitespace within the RE string is ignored, except when in a character class or preceded by an unescaped backslash; this lets you organize and indent the RE more clearly. It also enables you to put comments within a RE; comments are marked by a "#" that's neither in a character class or preceded by an unescaped backslash. Comments are simply ignored.

For example, here's a RE that uses re.VERBOSE; see how much easier it is to read?

charref = re.compile(r"""
 &#		     # Start of a numeric entity reference
 (?P<char>
   [0-9]+[^0-9]      # Decimal form
   | 0[0-7]+[^0-7]   # Octal form
   | x[0-9a-fA-F]+[^0-9a-fA-F] # Hexadecimal form
 )
""", re.VERBOSE)
Without the verbose setting, the RE would look like this:
charref = re.compile("&#(?P<char>[0-9]+[^0-9]"
                     "|0[0-7]+[^0-7]"
                     "|x[0-9a-fA-F]+[^0-9a-fA-F])")
In the above example, Python's automatic concatenation of string literals has been used to break up the RE into smaller pieces, but it's still more difficult to understand than the version using re.VERBOSE.

Footnotes

....1
"Spam", "spam", "spam", "spam", "spam", ...

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 3.4 Module-Level Functions Up: 3 Using Regular Expressions Next: 4 More Pattern Power
Release 0.03.


4 More Pattern Power

So far we've only covered a part of the features of regular expressions. In this section, we'll cover some new metacharacters, and how to use groups to retrieve portions of the text that was matched.

Subsections
  • 4.1 More Metacharacters
  • 4.2 Grouping
  • 4.3 Non-capturing, and Named Groups
  • 4.4 Other Assertions

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 3.5 Compilation Flags Up: Regular Expression HOWTO Next: 4.1 More Metacharacters
Release 0.03.


4.1 More Metacharacters

There are some metacharacters that we haven't covered yet. Most of them will be covered in this section.

Some of the remaining metacharacters to be discussed are zero-width assertions. They don't cause the engine to advance through the string at all; instead, they consume no characters at all, and simply succeed or fail. For example, \b is an assertion that the current position is located at a word boundary; the position isn't changed by the \b at all. This means that zero-width assertions should never be repeated, because if they match once at a given location, they can obviously be matched an infinite number of times.

|
Alternation, or the ``or'' operator. If A and B are regular expressions, A|B will match any string that matches either "A" or "B". | has very low precedence, in order to make it work reasonably when you're alternating multi-character strings. Crow|Servo will match either "Crow" or "Servo", not "Cro", a "w" or an "S", and "ervo".

To match a literal "|", use \|, or enclose it inside a character class, as in [|].

^
Matches at the beginning of lines. Unless the MULTILINE flag has been set, this will only match at the beginning of the string. In MULTILINE mode, this also matches immediately after each newline within the string.

For example, if you wish to match the word "From" only at the beginning of a line, the RE to use is ^From.

>>> print re.search('^From', 'From Here to Eternity')
<re.MatchObject instance at 80c1520>
>>> print re.search('^From', 'Reciting From Memory')
None

To match a literal "^", use \^, or enclose it inside a character class, as in [\].

$
Matches at the end of lines, which is defined as either the end of the string, or any location followed by a newline character.

>>> print re.search('}$', '{block}')
<re.MatchObject instance at 80adfa8>
>>> print re.search('}$', '{block} ')
None
>>> print re.search('}$', '{block}\n')
<re.MatchObject instance at 80adfa8>
To match a literal "$", use \$, or enclose it inside a character class, as in [$].
\A
Matches only at the start of the string. When not in MULTILINE mode, \A and ^ are effectively the same. In MULTILINE mode, however, they're different; \A still matches only at the beginning of the string, but ^ may match at several locations inside the string (anywhere following a newline character).
\Z
Matches only at the end of the string.
\b
Word boundary. This is a zero-width assertion that matches only at the beginning or end of a word. A word is defined as a sequence of alphanumeric characters, so the end of a word is indicated by whitespace or a non-alphanumeric character.

The following example matches "class" only when it's a complete word; it won't match when it's contained inside another word.

>>> p = re.compile(r'\bclass\b')
>>> print p.search('no class at all')
<re.MatchObject instance at 80c8f28>
>>> print p.search('the declassified algorithm')
None
>>> print p.search('one subclass is')
None
There are two subtleties you should remember when using this special sequence. First, this is the worst collision between Python's string literals and regular expression sequences. In Python's string literals, "\b" is the backspace character, ASCII value 8. If you're not using raw strings, then Python will convert the "\b" to a backspace, and your RE won't match as you expect it to. The following example looks the same as our previous RE, but omits the "r" in front of the RE string.

>>> p = re.compile('\bclass\b')
>>> print p.search('no class at all')
None
>>> print p.search('\b' + 'class' + '\b')
<re.MatchObject instance at 80c3ee0>
Second, inside a character class, where there's no use for this assertion, \b represents the backspace character, for compatibility with Python's string literals.
\B
Another zero-width assertion, this is the opposite of \b, only matching when the current position is not at a word boundary.

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 4 More Pattern Power Up: 4 More Pattern Power Next: 4.2 Grouping
Release 0.03.


4.2 Grouping

Frequently you need to obtain more information than just whether the RE matched or not. Regular expressions are also often used to dissect strings by writing a RE divided into several subgroups which match different components of interest. For example, an RFC-822 header line is divided into a header name and a value, separated by a ":". This can be handled by writing a regular expression which matches an entire header line, and has one group which matches the header name, and another group which matches the header's value.

Groups are marked by the "(", ")" metacharacters. "(" and ")" have much the same meaning as they do in mathematical expressions; they group together the expressions contained inside them. For example, you can repeat the contents of a group with a repeating qualifier, such as *, +, ?, or {m,n}. For example, (ab)* will match zero or more repetitions of "ab".

>>> p = re.compile('(ab)*')
>>> print p.match('ababababab').span()
(0, 10)

Groups indicated with "(", ")" also capture the starting and ending index of the text that they match; this can be retrieved by passing an argument to group(), start(), end(), and span(). (Later we'll see how to express groups that don't capture the span of text that they match.) Groups are numbered starting with 0. Group 0 is always present; it's the whole RE. The methods all have group 0 as their default argument.

>>> p = re.compile('(a)b')
>>> m = p.match('ab')
>>> m.group()
'ab'
>>> m.group(0)
'ab'

Subgroups are numbered from left to right, from 1 upward. Groups can be nested; to determine the number, just count the opening parenthesis characters, going from left to right.

>>> p = re.compile('(a(b)c)d')
>>> m = p.match('abcd')
>>> m.group(0)
'abcd'
>>> m.group(1)
'abc'
>>> m.group(2)
'b'

group() can be passed multiple group numbers at a time, in which case it will return a tuple containing the corresponding values for those groups.

>>> m.group(2,1,2)
('b', 'abc', 'b')

The groups() method returns a tuple containing the strings for all the subgroups, from 1 up to however many there are.

>>> m.groups()
('abc', 'b')

Backreferences allow you to specify that the contents of an earlier capturing group must also be found at the current location in the string. For example, \1 will succeed if the exact contents of group 1 can be found at the current position, and fails otherwise. Remember that Python's string literals also use a backslash followed by numbers to allow including arbitrary characters in a string, so be sure to use a raw string when incorporating backreferences in a RE.

For example, the following RE detects doubled words in a string.

>>> p = re.compile(r'(\b\w+)\s+\1')
>>> p.search('Paris in the the spring').group()
'the the'

Backreferences like this aren't often useful for just searching through a string -- there are few text formats which repeat data in this way -- but you'll soon find out that they're very useful when performing string substitutions.

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 4.1 More Metacharacters Up: 4 More Pattern Power Next: 4.3 Non-capturing, and Named
Release 0.03.


4.3 Non-capturing, and Named Groups

Elaborate REs may use many groups, both to capture substrings of interest, and to group and structure the RE itself. In complex REs, it becomes difficult to keep track of the group numbers. There are two features which help with this problem. Both of them use a common syntax for regular expression extensions, so we'll look at that first.

Perl 5 added several additional features to standard regular expressions, and the Python re module supports most of them. It would have been difficult to choose new single-keystroke metacharacters or new special sequences beginning with "\" to represent the new features, without making Perl's regular expressions confusingly different from standard REs. If you chose "&" as a new metacharacter, for example, old expressions would be assuming that "&" was a regular character and wouldn't have escaped it by writing \& or [&]. The solution chosen was to use (?...) as the extension syntax. "?" immediately after a parenthesis was a syntax error, because the "?" would have nothing to repeat, so this doesn't introduce any compatibility problems. The characters immediately after the "?" indicate what extension is being used, so (?=foo) is one thing (a positive lookahead assertion) and (?:foo) is something else (a non-capturing group containing the subexpression foo).

Python adds an extension syntax to Perl's extension syntax. If the first character after the question mark is a "P", you know that it's a extension that's specific to Python. Currently there are two such extensions: (?P<name>...) defines a named group, and (?P=name) is a backreference to a named group. If future versions of Perl 5 add similar features using a different syntax, the re module will be changed to support the new syntax, while preserving the Python-specific syntax for compatibility's sake.

Now that we've looked at the general extension syntax, we can return to the features that simplify working with groups in complex REs. Since groups are numbered from left to right, and a complex expression may use many groups, it can become difficult to keep track of the correct numbering, and modifying such a complex RE is annoying. Insert a new group near the beginning, and you change the numbers of everything that follows it.

First, sometimes you'll want to use a group to collect a part of a regular expression, but aren't interested in retrieving the group's contents. You can make this fact explicit by using a non-capturing group: (?:...), where you can put any other regular expression inside the parentheses.

>>> m = re.match("([abc])+", "abc")
>>> m.groups()
('c',)
>>> m = re.match("(?:[abc])+", "abc")
>>> m.groups()
()

Except for the fact that you can't retrieve the contents of what the group matched, a non-capturing group behaves exactly the same as a capturing group; you can put anything inside it, repeat it with a repetition metacharacter such as "*", and nest it within other groups (capturing or non-capturing). (?:...) is particularly useful when modifying an existing group, since you can add new groups without changing how all the other groups are numbered. It should be mentioned that there's no performance difference in searching between capturing and non-capturing groups; neither form is any faster than the other.

The second, and more significant, feature, is named groups; instead of referring to them by numbers, groups can be referenced by a name.

The syntax for a named group is one of the Python-specific extensions: (?P<name>...). name is, obviously, the name of the group. Except for associating a name with a group, named groups also behave identically to capturing groups. The MatchObject methods that deal with capturing groups all accept either integers, to refer to groups by number, or a string containing the group name. Named groups are still given numbers, so you can retrieve information about a group in two ways:

>>> p = re.compile(r'(?P<word>\b\w+\b)')
>>> m = p.search( '(((( Lots of punctuation )))' )
>>> m.group('word')
'Lots'
>>> m.group(1)
'Lots'
Named groups are handy because they let you use easily-remembered names, instead of having to remember numbers. Here's an example RE from the imaplib module:

InternalDate = re.compile(r'INTERNALDATE "'
        r'(?P<day>[ 123][0-9])-(?P<mon>[A-Z][a-z][a-z])-'
    r'(?P<year>[0-9][0-9][0-9][0-9])'
        r' (?P<hour>[0-9][0-9]):(?P<min>[0-9][0-9]):(?P<sec>[0-9][0-9])'
        r' (?P<zonen>[-+])(?P<zoneh>[0-9][0-9])(?P<zonem>[0-9][0-9])'
        r'"')
It's obviously much easier to retrieve m.group('zonem'), instead of having to remember to retrieve group 9.

Since the syntax for backreferences refers to the number of the group, in an expression like (...)\1, there's naturally a variant that uses the group name instead of the number. This is also a Python extension: (?P=name) indicates that the contents of the group called name should again be found at the current point. The regular expression for finding doubled words, (\b\w+)\s+\1 can also be written as (?P<word>\b\w+)\s+(?P=word):

>>> p = re.compile(r'(?P<word>\b\w+)\s+(?P=word)')
>>> p.search('Paris in the the spring').group()
'the the'

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 4.2 Grouping Up: 4 More Pattern Power Next: 4.4 Other Assertions
Release 0.03.


4.4 Other Assertions

Another zero-width assertion is the lookahead assertion. Lookahead assertions are available in both positive and negative form, and look like this:

(?=...)
Positive lookahead assertion. This succeeds if the contained regular expression, represented here by ..., successfully matches at the current location, and fails otherwise. But, once the contained expression has been tried, the matching engine doesn't advance at all; the rest of the pattern is tried right where the assertion started.

(?!...)
Negative lookahead assertion. This is the opposite of the positive assertion; it succeeds if the contained expression doesn't match at the current position in the string.

An example will help make this concrete, and will demonstrate a case where a lookahead is useful. Consider a simple pattern to match a filename, and split it apart into a base name and an extension, separated by a ".". For example, in "news.rc", "news"is the base name, and "rc" is the filename's extension.

The pattern to match this is quite simple: .*[.].*$. (Notice that the "." needs to be treated specially because it's a metacharacter; I've put it inside a character class. Also notice the trailing $; this is added to ensure that all the rest of the string must be included in the extension.) This regular expression matches "foo.bar" and "autoexec.bat" and "sendmail.cf" and "printers.conf".

Now, consider complicating the problem a bit; what if you want to match filenames where the extension is not "bat"? Some incorrect attempts:

.*[.][^b].*$

First attempt: Exclude "bat" by requiring that the first character of the extension is not a "b". This is wrong, because it also doesn't match "foo.bar".

.*[.]([b]..|.[â].|..[t])$

The expression gets messier when you try to patch up the first solution by requiring one of the following cases to match: the first character of the extension isn't "b"; the second character isn't "a"; or the third character isn't "t". This accepts "foo.bar" and rejects "autoexec.bat", but it requires a three-letter extension, and doesn't accept "sendmail.cf". Another bug, so we'll complicate the pattern again in an effort to fix it.

.*[.]([b].?.?|.[â]?.?|..?[t]?)$

In the third attempt, the second and third letters are all made optional in order to allow matching extensions shorter than three characters, such as "sendmail.cf".

The pattern's getting really complicated now, which makes it hard to read and understand. When you write a regular expression, ask yourself: if you encountered this expression in a program, how hard would it be to figure out what the expression was intended to do? Worse, this solution doesn't scale well; if the problem changes, and you want to exclude both "bat" and "exe" as extensions, the pattern would get still more complicated and confusing.

A negative lookahead cuts through all this. Go back to the original pattern, and, before the .* which matches the extension, insert (?!bat$). This means: if the expression bat doesn't match at this point, try the rest of the pattern; if bat$ does match, the whole pattern will fail. (The trailing $ is required to ensure that something like "sample.batch", where the extension only starts with "bat", will be allowed.

After this modification, the whole pattern is .*[.](?!bat$).*$. Excluding another filename extension is now easy; simply add it as an alternative inside the assertion. .*[.](?!bat$|exe$).*$ excludes both "bat" and "exe".

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 4.3 Non-capturing, and Named Up: 4 More Pattern Power Next: 5 Modifying Strings
Release 0.03.


5 Modifying Strings

Up to this point, we've simply performed searches against a static string. Regular expressions are commonly used to modify a string in various ways.

Subsections
  • 5.1 Splitting Strings
  • 5.2 Search and Replace

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 4.4 Other Assertions Up: Regular Expression HOWTO Next: 5.1 Splitting Strings
Release 0.03.


5.1 Splitting Strings

The split() method of a RegexObject splits a string apart wherever the RE matches, returning a list of the pieces. It's similar to string.split(), providing much more generality in the delimiters that you can split by; string.split() only supports splitting by whitespace, or by a fixed string. As you'd expect, there's a module-level re.split() function, too.

split(string [, maxsplit = 0])
Split string by the matches of the regular expression. If capturing parentheses are used in the RE, then their contents will also be returned as part of the resulting list. If maxsplit is nonzero, at most maxsplit splits are performed.

You can limit the number of splits made, by passing a value for maxsplit. When maxsplit is nonzero, at most maxsplit splits will be made, and the remainder of the string is returned as the final element of the list. In the following example, the delimiter will be any sequence of non-alphanumeric characters.

>>> p = re.compile(r'\W+')
>>> p.split('This is a test, short and sweet, of split().')
['This', 'is', 'a', 'test', 'short', 'and', 'sweet', 'of', 'split', '']
>>> p.split('This is a test, short and sweet, of split().', 3)
['This', 'is', 'a', 'test, short and sweet, of split().']
Sometimes you're not only interested in what the text between delimiters is, but also need to know what the delimiter was. If capturing parentheses are used in the RE, then their values are also returned as part of the list. Compare the following calls:

>>> p = re.compile(r'\W+')
>>> p2 = re.compile(r'(\W+)')
>>> p.split('This... is a test.')
['This', 'is', 'a', 'test', '']
>>> p2.split('This... is a test.')
['This', '... ', 'is', ' ', 'a', ' ', 'test', '.', '']
The module-level function re.split() adds the RE to be used as the first argument, but is otherwise the same.

>>> re.split('[\W]+', 'Words, words, words.')
['Words', 'words', 'words', '']
>>> re.split('([\W]+)', 'Words, words, words.')
['Words', ', ', 'words', ', ', 'words', '.', '']
>>> re.split('[\W]+', 'Words, words, words.', 1)
['Words', 'words, words.']

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 5 Modifying Strings Up: 5 Modifying Strings Next: 5.2 Search and Replace
Release 0.03.


5.2 Search and Replace

Another common task is to find all the matches for a pattern, and replace them with a different string. The sub() method takes a replacement value, which can be either a string or a function, and the string to be processed. Python strings are immutable, so this function will return a new string.

sub(replacement, string[, count = 0])
Returns the string obtained by replacing the leftmost non-overlapping occurrences of the RE in string by the replacement replacement. If the pattern isn't found, string is returned unchanged.

The optional argument count is the maximum number of pattern occurrences to be replaced; count must be a non-negative integer. The default value of 0 means to replace all occurrences.

Here's a simple example of using the sub() method.

>>> p = re.compile( '(blue|white|red)')
>>> p.sub( 'colour', 'blue socks and red shoes')
'colour socks and colour shoes'
>>> p.sub( 'colour', 'blue socks and red shoes', 1)
'colour socks and red shoes'
Empty matches are replaced only when not they're not adjacent to a previous match.

>>> p = re.compile('x*')
>>> p.sub('-', 'abxd')
'-a-b-d-'
If replacement is a string, any backslash escapes in it are processed. That is, "\n" is converted to a single newline character, "\r" is converted to a carriage return, and so forth. Unknown escapes such as "\j" are left alone. Backreferences, such as "\6", are replaced with the substring matched by the corresponding group in the RE. This lets you incorporate portions of the original text in the resulting replacement string.

>>> p = re.compile('section{ ( [^}]* ) }', re.VERBOSE)
>>> p.sub(r'subsection{\1}','section{First} section{second}')
'subsection{First} subsection{second}'
In addition to character escapes and backreferences as described above, "\g<name>" will use the substring matched by the group named "name", as defined by the (?P<name>...) syntax. "\g<number>" uses the corresponding group number. "\g<2>" is therefore equivalent to "\2", but isn't ambiguous in a replacement string such as "\g<2>0". ("\ 20" would be interpreted as a reference to group 20, not a reference to group 2 followed by the literal character "0".) The following substitutions are all equivalent, but use all three variations of the replacement string.

>>> p = re.compile('section{ (?P<name> [^}]* ) }', re.VERBOSE)
>>> p.sub(r'subsection{\1}','section{First}')
'subsection{First}'
>>> p.sub(r'subsection{\g<1>}','section{First}')
'subsection{First}'
>>> p.sub(r'subsection{\g<name>}','section{First}')
'subsection{First}'
replacement can also be a function, which gives you even more powerful control. If replacement is a function, the function is called for every non-overlapping occurance of pattern, and is passed a MatchObject argument. The function can use that information to compute the desired replacement string and return it. For example:
>>> def hexrepl( match ):
...     "Return the hex string for a decimal number"
...     value = int( match.group() )
...     return hex(value)
...
>>> p = re.compile(r'\d+')
>>> p.sub(hexrepl, 'Call 65490 for printing, 49152 for user code.')
'Call 0xffd2 for printing, 0xc000 for user code.'
When using the module-level re.sub() function, the pattern is passed as the first argument. The pattern may be a string or a RegexObject; if you need to specify regular expression flags, you must either use a RegexObject as the first parameter, or use embedded modifiers in the pattern, e.g. sub("(?i)b+", "x", "bbbb BBBB") returns 'x x'.

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 5.1 Splitting Strings Up: 5 Modifying Strings Next: 6 Common Problems
Release 0.03.


6 Common Problems

Regular expressions are a powerful tool for some applications, but in some ways their behaviour isn't intuitive. At times they don't behave the way you may expect them to. This section will point out some of the most common pitfalls.

Subsections
  • 6.1 Know About the string Module
  • 6.2 match() versus search()
  • 6.3 Greedy versus Non-Greedy
  • 6.4 Not using re.VERBOSE

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 5.2 Search and Replace Up: Regular Expression HOWTO Next: 6.1 Know About the
Release 0.03.


6.1 Know About the string Module

Sometimes, using the re module at all is a mistake. If you're matching a fixed string, or a single character class, and you're not using any re features such as the IGNORECASE flag, then the full power of regular expressions may not be required. The string module has several functions for performing operations with fixed strings and it's usually much faster, because the implementation is a single small C loop that's been optimized for the purpose, instead of the large, more generalized regular expression engine.

One example might be replacing a single fixed string with another one; for example, you might replace "word"with "deed". re.sub() seems like the function to use for this, but consider string.replace(). Note that string.replace() will also replace "word" inside words, turning "swordfish" into "sdeedfish", but the naïve RE word would have done that, too. (To avoid performing the substitution on parts of words, the pattern would have to be \bword\b, in order to require that "word" have a word boundary on either side. This takes the job beyond the string module's abilities.)

Another common task is deleting every occurrence of a single character from a string, or replacing it with another single character. You might do this with something like re.sub('\n', ' ', S), but string.translate() is capable of doing both these tasks, and will be much faster that any regular expression operation can ever be.

In short, before turning to the re module, consider whether your problem can be solved with the faster and simpler string module.

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 6 Common Problems Up: 6 Common Problems Next: 6.2 match() versus search()
Release 0.03.


6.2 match() versus search()

The match() function only checks if the RE matches at the beginning of the string, while search() will scan forward through the string for a match. It's important to keep this distinction in mind. Remember, match() will only report a successful match which will start at 0; if the match wouldn't start at zero, match() will not report it.

>>> print re.match('super', 'superstition').span()
(0, 5)
>>> print re.match('super', 'insuperable')
None
On the other hand, search() will scan forward through the string, reporting the first match it finds.

>>> print re.search('super', 'superstition').span()
(0, 5)
>>> print re.search('super', 'insuperable').span()
(2, 7)
Sometimes you'll be tempted to keep using re.match(), and just add .* to the front of your RE. Resist this tempation, and use re.search() instead. The regular expression compiler does some analysis of REs in order to speed up the process of looking for a match. One such analysis figures out what the first character of a match must be; for example, a pattern starting with Crow must match starting with a "C". The analysis lets the engine quickly scan through the string looking for the starting character, and only try the full match if one is found.

Adding .* defeats this optimization, and requires scanning to the end of the string and then backtracking to find a match for the rest of the RE. Use re.search() instead.

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 6.1 Know About the Up: 6 Common Problems Next: 6.3 Greedy versus Non-Greedy
Release 0.03.


6.3 Greedy versus Non-Greedy

When repeating a regular expression, as in a*, the resulting action is to consume as much of the pattern as possible. This fact often bites you when you're trying to match a pair of balanced delimiters, such as the angle brackets surrounding an HTML tag. The naïve pattern for matching a single HTML tag doesn't work because of the greedy nature of .*.

>>> s = '<html><head><title>Title</title>'
>>> len(s)
32
>>> print re.match('<.*>', s).span()
(0, 32)
>>> print re.match('<.*>', s).group()
<html><head><title>Title</title>
The RE matches the "<" in "<html>", and the .* consumes the rest of the string. There's still more left in the RE, though, and the > can't match at the end of the string, so the regular expression engine has to backtrack character by character until it finds a match for the >. The final match extends from the "<" in "<html>"to the ">" in "</title>", which isn't what you want.

In this case, the solution is to use the non-greedy qualifiers *?, +?, ??, or {m,n}?, which match as little text as possible. In the above example, the ">" is tried immediately after the first "<" matches, and when it fails, the engines advances a character at a time, retrying the ">" at every step. This produces just the right result:

>>> print re.match('<.*?>', s).group()
<html>

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 6.2 match() versus search() Up: 6 Common Problems Next: 6.4 Not using re.VERBOSE
Release 0.03.


6.4 Not using re.VERBOSE

By now you've probably noticed that regular expressions are a very compact notation, but they're not terribly readable. REs of moderate complexity can become lengthy collections of backslashes, parentheses, and metacharacters, making them difficult to read and understand.

For such REs, specifying the re.VERBOSE flag when compiling the regular expression can be helpful, because it allows you to format the regular expression more clearly.

The re.VERBOSE flag has several effects. Whitespace in the regular expression that isn't inside a character class is ignored. This means that an expression such as dog | cat is equivalent to the less readable dog|cat, but [a b] will still match the characters "a", "b", or a space. In addition, you can also put comments inside a RE; comments extend from a "#" character to the next newline. When used with triple-quoted strings, this enables REs to be formatted more neatly:

pat = re.compile(r"""
 \s*                 # Skip leading whitespace
 (?P<header>[^:]+)   # Header name
 \s* :               # Whitespace, and a colon
 (?P<value>.*?)      # The header's value -- *? used to
                     # lose the following trailing whitespace
 \s*$                # Trailing whitespace to end-of-line
""", re.VERBOSE)
This is far more readable than:

pat = re.compile(r"\s*(?P<header>[^:]+)\s*:(?P<value>.*?)\s*$")

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 6.3 Greedy versus Non-Greedy Up: 6 Common Problems Next: 7 Feedback
Release 0.03.


7 Feedback

Regular expressions are a complicated topic. Did this document help you understand them? Were there parts that were unclear, or Problems you encountered that weren't covered here? If so, please send suggestions for improvements to amk1@bigfoot.com.

The most complete book on regular expressions is almost certainly Jeffrey Friedl's Mastering Regular Expressions, published by O'Reilly. Unfortunately, the Python material in this book dates from before the re module -- all the examples use the old regex module -- but it covers writing good regular expressions in great detail.

Previous Page Up One Level Next Page Regular Expression HOWTO Contents
Previous: 6.4 Not using re.VERBOSE Up: Regular Expression HOWTO Next: About this document ...
Release 0.03.


About this document ...

Regular Expression HOWTO

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.

Previous Page Up One Level Regular Expression HOWTO Contents
Previous: 7 Feedback Up: Regular Expression HOWTO
Release 0.03.


regex-to-re HOWTO

A.M. Kuchling

amk1@bigfoot.com

Abstract:

This document will show you how to port Python code that uses the regex module to the new re module, introduced with Python 1.5. This discussion will only cover the highlights; for full documentation, see the re module's documentation in the Python Library Reference.

This document is available in several formats, including PostScript, PDF, HTML and plain ASCII, from the Python HOWTO page at http://www.python.org/doc/howto/.

  • Contents
  • 1 Why Should You Port?
  • 2 Module-level differences
  • 3 Pattern Differences
  • 4 String Substitutions
  • About this document ...

Up One Level Next Page regex-to-re HOWTO Contents
Up: HOWTOs Next: Contents
Release 2.0.


1 Why Should You Port?

While the regex module will continue to be supported for backwards compatibility, there are several reasons to modify your code to use the re module. re provides a more powerful regular expression notation, mostly compatible with Perl's notation, though there are some extensions and omissions. There are new operations in re, such as minimal matching, that cannot be performed using the regex module. re will be the focus of new development and enhancements and improvements in future versions; there are already a few speed optimizations on the TODO list that didn't make it into 1.5.

re code is also automatically thread-safe, while the regex module requires that you explicitly lock pattern objects. (More about this below.)

Previous Page Up One Level Next Page regex-to-re HOWTO Contents
Previous: Contents Up: regex-to-re HOWTO Next: 2 Module-level differences
Release 2.0.


2 Module-level differences

The top-level functions in the re module are quite similar to those in the regex module; there are a few new functions and some new optional arguments, but these can mostly be ignored when converting regex code.

re is now the only module required to access all the available functionality. regsub has been swallowed by re, and is available as the sub(), subn(), and split() functions. There's no equivalent of regex_syntax; re supports only one syntax, and you can't change it. If you want alternative regex syntaxes, you'll have to manually parse the syntax and convert it to the basic Perl-like syntax; sorry!

The functions in the regex module return an integer giving the length or position of any match, or -1 if no match was found. The subgroups from the match are then available as attributes of the compiled pattern object: regs, last, and so forth. This doesn't interact well with threads, because two threads may use the same pattern object at almost the same time; the results from the second thread's operation will then stomp on the first thread's results.

To fix this problem, functions in the re module return a MatchObject instance, or None if the match failed. Pattern objects now have no attributes that change after the object is created. Code must therefore be converted to store the MatchObject instance in a variable, and check for None to determine if a match was found:

regex code:

pat = regex.compile('[0-9]+')
if pat.match(strvar) == -1: print 'No match'

re code:

pat = re.compile('[0-9]+')
m = pat.match(strvar)
if m == None: print 'No match'

The search() and match() functions have the same parameters in both modules; the re module returns None or a MatchObject instance instead of an integer, and adds an optional flags argument. Of course, the re versions use the new regular expression syntax (see "Pattern Differences", below).

regex code:

result = regex.match('\\w+', 'abc abc')

re code:

result = re.match('\\w+', 'abc abc')

Another thing that's disappeared is the translate argument to the compile() function; in the regex module, a 256-character string can be passed to indicate how characters should be translated before matching them. This feature was often used to perform case-insensitive matching, or to map the digits 0-9 to 0 to simplify patterns that matched digits. With the re.IGNORECASE and re.LOCALE flags, and special sequences such as "\d", the need for this functionality is greatly reduced; since the on-the-fly translation complicated the matching engine and made it slower, the feature was dropped. If you still need it, you'll have to explicitly call string.translate() on your target string before running your regular expression on it.

regex code:

pat = regex.compile('[abc]', translation)
result = pat.match(str_var)

re code:

pat = re.compile('[abc]')
result = pat.match( string.translate(str_var, translation) )

Some programs use a translation string to convert foreign characters such as é or øo to characters in the range A-Za-z so they can be matched by "\w". These programs can specify the re.LOCALE flag, which causes "\w" to match the alphabetic characters defined in the current locale.

The most common use of the translation string is to do a case-insensitive match by passing regex.casefold. The re equivalent is to pass re.IGNORECASE (or re.I, which is the same thing) as the flags argument to re.compile().

regex code:

pat = regex.compile('[abc]', regex.casefold)

re code:

pat = re.compile('[abc]', re.IGNORECASE)

regex.symcomp() is no longer required; named groups are always available using the (?P<name>...) syntax.

regex code:

pat = regex.symcomp( "(<integer>[0-9]+)" )

re code:

pat = re.compile( "(?P<integer>[0-9]+)" )

With the regex module, the group(args) method of pattern objects takes zero or more arguments, either integers or strings containing symbolic group names. group(args) returns a tuple containing the corresponding groups from the last pattern match. With no arguments, .group() returns a tuple containing all the groups in the pattern. In all these cases, if the tuple would only contain a single element, just that single string is returned. This is inconsistent, but convenient:

x = pat.group(1)
x,y = pat.group(1, 2)

The re module moves this method to MatchObject instances. The action of group() with no arguments has been changed for consistency with the start(), end(), and span() methods; they all assume group 0 as the default. To get a tuple containing all the subgroups, use the groups() method.

regex code:

substring = pattern.group(0)

re code:

substring = match.group(0)
or
match.group()

regex code:

substring = pattern.group()

re code:

substring = match.groups()

Previous Page Up One Level Next Page regex-to-re HOWTO Contents
Previous: 1 Why Should You Up: regex-to-re HOWTO Next: 3 Pattern Differences
Release 2.0.


3 Pattern Differences

The regex module uses Emacs-style regular expressions, while the re module uses Perl-style expressions. The largest difference between the two is that in Emacs-style syntax, all metacharacters have a backslash in front of them. For example, groups are delimited by "\(" and "\)"; "(" and ")" match the literal characters. This clutters moderately complicated expressions with lots of backslashes. Unfortunately, Python's string literals also use the backslash as an escape character, so it's frequently necessary to add backslashes in front of backslashes; "\\\\" is required to match a single "\", for example.

In Perl-style expressions, things are just the opposite; "\(" and "\)"match the literal characters "(" and ")", while "(" and ")" in a pattern denote grouping. This makes patterns neater, since you'll rarely need to match literal "( )" characters, but will often be using grouping.

regex pattern:

\(\w+\|[0-9]+\)

re pattern:

(\w+|[0-9]+)

The Perl syntax also has more character classes that allow simplifying some expressions. The regex module only supports "\w" to match alphanumeric characters, and "\W" to match non-alphanumeric characters. The re module adds "\d" and "\D" for digits and non-digits, and "\s" and "\S"for whitespace and non-whitespace characters.

regex pattern:

[0-9]+[ \t\n]+

re pattern:

\d+\s+

Regular expressions can get very complicated and difficult to understand. To make expressions clearer, the re.VERBOSE flag can be specified. This flag causes whitespace outside of a character class to be ignored, and a "#" symbol outside of a character class is treated as the start of a comment, extending to the end of the line. This means a pattern can be put inside a triple-quoted string, and then formatted for clarity.

re code:

pat = re.compile("""
(?P<command> # A command contains ...
  \w+)       # ... a word ...
  \s+        # ... followed by whitespace ...
(?P<var>     # ... and an optional variable name
  (?!\d)     # Lookahead assertion: can't start with a digit
  \w+        # Match a word
)""", re.VERBOSE)

If the re.VERBOSE flag seems a bit easy to overlook, off at the end of the statement, you can put (?x) inside the pattern; this has the same effect as specifying re.VERBOSE, but makes the flag setting part of the pattern. There are similar extensions to specify re.DOTALL, re.IGNORECASE, re.LOCALE, and re.MULTILINE: (?s), (?i), (?L), and (?m).

A module to automatically translate expressions from the old regex syntax to the new syntax has been written, and is available as reconvert.py in Python 1.5b2. (I haven't really looked at it yet.)

Previous Page Up One Level Next Page regex-to-re HOWTO Contents
Previous: 2 Module-level differences Up: regex-to-re HOWTO Next: 4 String Substitutions
Release 2.0.


4 String Substitutions

regsub.sub(pattern, replacement, string) performs a single string substitution, replacing the substring matched by pattern with the replacement string. regsub.gsub is more commonly used; it replaces all non-overlapping matches to the pattern in the string. Both functions have been replaced by re.sub(pattern, replacement, string, count). count is optional, and gives the number of substitutions to perform; without a count argument, the default action is to perform all the substitutions possible.

regsub code:

regsub.sub('[ \t\n]+', ' ', s)

re code:

re.sub('[ \t\n]+', ' ', s, 1)

regsub code:

regsub.gsub('[ \t\n]+', ' ', s)

re code:

re.sub('[ \t\n]+', ' ', s)

regsub.split(string, pattern) splits the string apart into fields separated by delimiters matching the pattern, and returns a list containing the fields. The optional maxsplit parameter allows limiting the number of splits performed. The splitx() function also returns the delimiters as part of the list.

Both tasks are performed by the re.split(pattern, string) function. Note that the order of the arguments has been reversed! Since all the other re functions have the pattern first, the order was changed for consistency. There's still an optional maxsplit parameter, with the same meaning and the same default value. To keep the delimiters, put a group in the pattern; after each delimiter is found, the value of .groups() for the MatchObject instance will be appended to the list.

regsub code:

regsub.split(s, ',[ \t]*')

re code:

re.split(',[ \t]*', s)

regsub code:

regsub.splitx(s, ',[ \t]*')

re code:

re.split('(,[ \t]*)', s)

Previous Page Up One Level Next Page regex-to-re HOWTO Contents
Previous: 3 Pattern Differences Up: regex-to-re HOWTO Next: About this document ...
Release 2.0.


About this document ...

regex-to-re HOWTO

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.

Previous Page Up One Level regex-to-re HOWTO Contents
Previous: 4 String Substitutions Up: regex-to-re HOWTO
Release 2.0.


Restricted Execution HOWTO

A.M. Kuchling

amk1@bigfoot.com

Abstract:

Python provides a restricted execution mode for running untrusted code that will prevent the code from performing dangerous operations. This HOWTO explains how to use restricted execution mode, and how to customize the restricted environment for your application. It aims to provide a gentler introduction than the corresponding section in the Python Library Reference.

This document is available in several formats, including PostScript, PDF, HTML and plain ASCII, from the Python HOWTO page at http://www.python.org/doc/howto/.

  • Contents
  • 1 Basic use of RExec
  • 2 Frequently Asked Questions
  • 3 Customizing The Restricted Environment
    • 3.1 Inserting Variables
    • 3.2 Allowing Access to Unrestricted Objects
    • 3.3 Modifying Built-ins
  • 4 References
  • 5 Version History
  • About this document ...

Up One Level Next Page Restricted Execution HOWTO Contents
Up: HOWTOs Next: Contents
Release 2.0.


1 Basic use of RExec

For some applications, it's desirable to execute chunks of Python code that come from an outside source. The most obvious example is a Web browser such as Grail, which can download and execute applets written in Python.

An obvious danger of downloading and running code from anywhere is that someone might write a malicious applet that appears to be harmless, but silently erases files, makes copies of sensitive data, or gives the applet's author a back door into your system. The solution is to run the code in a restricted environment, where it's prevented from performing any operations that could be used maliciously.

Java does this by using the Java Virtual Machine, which executes Java bytecode. The virtual machine, or VM, has complete control over the running applet, and any dangerous operations must go through the VM in order to be performed. The VM can therefore trap suspicious activity, and stop the applet's execution, if a strict security policy is used, or ask the user if the operation should be permitted, if the policy is somewhat looser.

Python already has a virtual machine that executes Python byte codes, so creating a restricted execution environment simply requires sealing off dangerous built-in functions such as open(), and dangerous modules, such as the socket module. This can be done by creating new namespaces, removing any dangerous functions, and forcing code to be executed in those namespaces. While a simple idea, in practice it's fairly complicated to implement. Luckily, the required features have been present in Python for a while, and it's already been implemented for you as a standard module.

Code for using a restricted execution environment is in the rexec module. The base class is called RExec; in a later section of this HOWTO, we'll show you how to create your own subclasses of RExec to customize the functions and modules that are available. Here's the documentation for creating a new RExec instance:

RExec([hooks], [verbose] )
Returns a RExec instance. The verbose parameter is a Boolean value, defaulting to false. If true, the RExec instance will execute in verbose mode, which will print a debugging message when modules are imported, as if the -v option was given to the Python interpreter.

The hooks parameter can be an instance of the RHooks class, or of some subclass of RHooks; a default instance will be used if the parameter is omitted. This is only required when creating particularly exotic restricted environments that import modules in new ways. If you need to use this, you'll have to consult the source code (or Guido) for a complete picture of what's going on.

The RExec instance has r_exec(), r_eval(), and r_execfile() functions, which do the same thing as Python's built-in exec(), eval(), and execfile() functions, performing them in the restricted environment. (There are also s_exec(), s_eval(), and s_execfile() methods which replace the restricted environment's standard input, output, and error files with StringIO objects that allow you to control the input and capture any output generated.)

Here's a sample usage of a restricted environment. First, the RExec instance has to be created.

r_env = rexec.RExec()

Now, we can execute code and evaluate expressions in the environment:

r_env.r_exec('import string')
expr = 'string.upper("This is a test")'
print r_env.r_eval( expr )

The first line executes a statement, importing the string module. Since it's considered a safe module, the operation succeeds. The second and third lines create a string containing an expression, and evaluates the expression in the restricted environment; it prints out "THIS IS A TEST", as you'd expect.

Unsafe operations trigger an exception. For example:

r_env.r_exec('import socket')

The previous line will cause an ImportError exception to be raised, with an associated string value that reads "untrusted dynamic module: _socket". Trying to open a file for writing is also forbidden:

r_env.r_exec('file = open("/tmp/a.out", "w")')

This will raise an IOError exception, with an assocated string value that reads "can't open files for writing in restricted mode". The restricted code can catch the exception in a try...except block and continue running; this is useful for writing code which works in both restricted and unrestricted mode. Opening files for reading will work, however.

Exactly what restrictions does the base RExec impose? It limits the modules that can be imported to the following safe list:

audioop, array, binascii, cmath, errno, imageop,
marshal, math, md5, operator, parser, regex,
pcre, rotor, select, strop, struct, time

In general, these are modules that can't affect anything outside of the executing code; they allow various forms of computation, but don't allow operations that change the filesystem or use network connections to other machines. (The pcre module may be unfamiliar. It's an internal module used by the re module, so restricted code can still use the re to perform regular expression matches.)

It also restricts the variables and functions that are available from the sys and os modules. The sys module only contains the following symbols:

ps1, ps2, copyright, version, platform, exit, maxint

The os module is reduced to the following functions:

error, fstat, listdir, lstat, readlink,
stat, times, uname, getpid, getppid,
getcwd, getuid, getgid, geteuid, getegid

Note that restricted code has some read-only access to the filesystem via functions like os.stat and os.readlink; if you wish to forbid all access to the filename, these functions must be removed.

In restricted mode, there are various attributes of function and class objects that are no longer accessible: the __dict__ attribute of class, instance and module objects; the __self__ attribute of method objects; and most of the attributes of function objects, namely func_code, func_defaults, func_doc, func_globals, and func_name.

The __import__() and reload() functions are replaced by versions which implement the above restrictions. Finally, Python's usual open() function is removed and replaced by a restricted version that only allows opening files for reading.

To change any of these policies, whether to be stricter or looser, see the section below on customizing the restricted environment.

Previous Page Up One Level Next Page Restricted Execution HOWTO Contents
Previous: Contents Up: Restricted Execution HOWTO Next: 2 Frequently Asked Questions
Release 2.0.


2 Frequently Asked Questions

[I want more questions to be answered in this section; please send them to amk1@bigfoot.com.]

How do I guard against denial-of-service attacks? Or, how do I keep restricted code from consuming a lot of memory?

Even if restricted code can't open sockets or write files, it can still cause problems by entering an infinite loop or consuming lots of memory; this is as easy as coding while 1: pass or 'a' * 12345678901. Unfortunately, there's no way at present to prevent restricted code from doing this. The Python process may therefore encounter a MemoryError exception, loop forever, or be killed by the operating system.

One solution would be to perform os.fork() to get a child process running the interpreter. The child could then use the resource module to set limits on the amount of memory, stack space, and CPU time it can consume, and run the restricted code. In the meantime, the parent process can set a timeout and wait for the child to return its results; if the child takes too long, the parent can conclude that the restricted code looped forever, and kill the child process.

If restricted code returns a class instance via r_eval(), can that class instance do nasty things if unrestricted code calls its methods?

You might be worried about the handling of values returned by r_eval(). For example, let's say your program does this:

value = r_env.r_eval( expression )
print str(value)

If value is a class instance, and has a __str__ method, that method will get called by the str() function. Is it possible for the restricted code to return a class instance where the __str__ function does something nasty? Does this provide a way for restricted code to smuggle out code that gets run without restrictions?

The answer is no. If restricted code returns a class instance, or a function, then, despite being called by unrestricted code, those functions will always be executed in the restricted environment. You can see why if you follow this little exercise. Run the interpreter in interactive mode, and create a sample class with a single method.

>>> class C:
...   def f(self): print "Hi!"
...

Now, look at the attributes of the unbound method C.f:

>>> dir(C.f)
['__doc__', '__name__', 'im_class', 'im_func', 'im_self']

im_func is the attribute we're interested in; it contains the actual function for the method. Look at the function's attributes using the dir() built-in function, and then look at the func_globals attribute.

>>> dir(C.f.im_func)
['__doc__', '__name__', 'func_code', 'func_defaults', 'func_doc',
 'func_globals', 'func_name']
>>> C.f.im_func.func_globals
{'__doc__': None, '__name__': '__main__',
 '__builtins__': <module '__builtin__'>,
 'f': <function f at 1201a68b0>,
 'C': <class __main__.C at 1201b35e0>,
 'a': <__main__.C instance at 1201a6b10>}

See how the function contains attributes for its __builtins__ module? This means that, wherever it goes, the function will always use the same __builtin__ module, namely the one provided by the restricted environment.

This means that the function's module scope is limited to that of the restricted environment; it has no way to access any variables or methods in the unrestricted environment that is calling into the restricted environment.

r_env.r_exec('def f(): g()\n')
f = r_env.r_eval('f')
def g(): print "I'm unrestricted."

If you execute the f() function in the unrestricted module, it will fail with a NameError exception, because f() doesn't have access to the unrestricted namespace. To make this work, you'd must insert g into the restricted namespace. Be careful when doing this, since g will be executed without restrictions; you have to be sure that g is a function that can't be used to do any damage. (Or is an instance with no methods that do anything dangerous. Or is a module containing no dangerous functions. You get the idea.)

What happens if restricted code raises an exception?

The rexec module doesn't do anything special for exceptions raised by restricted code; they'll be propagated up the call stack until a try...except statement is found that catches it. If no exception handler is found, the interpreter will print a traceback and exit, which is its usual behaviour. To prevent untrusted code from terminating the program, you should surround calls to r_exec(), r_execfile(), etc. with a try...except statement.

Python 1.5 introduced exceptions that could be classes; for more information about this new feature, consult http://www.python.org/doc/essays/stdexceptions.html. Class-based exceptions present a problem; the separation between restricted and unrestricted namespaces may cause confusion. Consider this example code, suggested by Jeff Rush.

t1.py:

# t1.py

from rexec import RHooks, RExec
from t2 import MyException
r= RExec( )

print 'MyException class:', repr(MyException)
try:
    r.r_execfile('t3.py')
except MyException, args:
    print 'Got MyException in t3.py'
except:
    print 'Missed MyException "%s" in t3.py' % repr(MyException)

t2.py

#t2.py

class MyException(Exception): pass
def myfunc():
    print 'Raising', `MyException`
    raise MyException, 5

print 't2 module initialized'

t3.py:

#t3.py
import sys
from t2 import MyException, myfunc
myfunc()

So, t1.py imports the MyException class from t2.py, and then executes some restricted code that also imports t2.py and raises MyException. However, because of the separation between restricted and unrestricted code, t2.py is actually imported twice, once in each mode. Therefore two distinct class objects are created for MyException, and the except statement doesn't catch the exception because it seems to be of the wrong class.

The solution is to modify t1.py to pluck the class object out of the restricted environment, instead of importing it. The following code will do the job, if added to t1.py:

module = r.add_module('__main__')
mod_dict = module.__dict__
MyException = mod_dict['MyException']

The first two lines simply get the dictionary for the __main__ module; this is a usage pattern discussed above. The last line simply gets the value corresponding to 'MyException', which will be the class object for MyException.

Previous Page Up One Level Next Page Restricted Execution HOWTO Contents
Previous: 1 Basic use of Up: Restricted Execution HOWTO Next: 3 Customizing The Restricted
Release 2.0.


3 Customizing The Restricted Environment

Subsections
  • 3.1 Inserting Variables
  • 3.2 Allowing Access to Unrestricted Objects
  • 3.3 Modifying Built-ins

Previous Page Up One Level Next Page Restricted Execution HOWTO Contents
Previous: 2 Frequently Asked Questions Up: Restricted Execution HOWTO Next: 3.1 Inserting Variables
Release 2.0.


3.1 Inserting Variables

While restricted code may be completely self-contained, it's common for it to require other data: perhaps a tuple listing various available plug-ins, or a dictionary mapping symbols to values. For simple Python data types, such as numbers and strings, the natural solution is to insert variables into one of the namespaces used by the restricted environment, binding the desired variable name to the value.

Continuing from the examples above, you can get the dictionary corresponding to the restricted module named module_name with the following code:

module = r_env.add_module(module_name)
mod_dict = module.__dict__

Despite its name, the add_module() method actually only adds the module if it doesn't already exist; it returns the corresponding module object, whether or not the module had to be created.

Most commonly, you'll insert variable bindings into the __main__ or __builtins__ module, so these will be the most frequent values of module_name.

Once you have the module's dictionary, you need only insert a key/value pair for the desired variable name and value. For example, to add a username variable:

mod_dict['username'] = "Kate Bush"

Restricted code will then have access to this variable.

Previous Page Up One Level Next Page Restricted Execution HOWTO Contents
Previous: 3 Customizing The Restricted Up: 3 Customizing The Restricted Next: 3.2 Allowing Access to
Release 2.0.


3.2 Allowing Access to Unrestricted Objects

Often, the code being executed will need access to various objects that exist outside the restricted environment. For example, an applet should be able to read some attributes of the object representing the browser, or needs access to the Tkinter module to provide a GUI display. But the browser object, or the Tkinter module aren't safe, so what can be done?

The solution is in the Bastion module, which lets you create class instances that represent some other Python object, but deny access to certain sensitive attributes or methods.

Bastion(object, [filter], [name], [class] )

Return a Bastion instance protecting the class instance object. Any attempt to access one of the object's attributes will have to be approved by the filter function; if the access is denied an AttributeError exception will be raised.

If present, filter must be a function that accepts a string containing an attribute name, and returns true if access to that attribute will be permitted; if filter returns false, the access is denied. The default filter denies access to any function beginning with an underscore "_". The bastion's string representation will be <Bastion for name> if a value for name is provided; otherwise, repr(object) will be used.

class, if present, would be a subclass of BastionClass; see the code in bastion.py for the details. Overriding the default BastionClass will rarely be required.

So, to safely make an object available to restricted code, create a Bastion object protecting it, and insert the Bastion instance into the restricted environment's namespace.

For example, the following code will create a bastion for an instance, named S, that simulates a dictionary. We want restricted code to be able to set and retrieve values from S, but no other attributes or methods should be accessible.

import Bastion
maindict = r_env.modules['__main__'].__dict__
maindict['S'] = Bastion.Bastion(SS,
          filter = lambda name: name in ['__getitem__', '__setitem__'] )

Previous Page Up One Level Next Page Restricted Execution HOWTO Contents
Previous: 3.1 Inserting Variables Up: 3 Customizing The Restricted Next: 3.3 Modifying Built-ins
Release 2.0.


3.3 Modifying Built-ins

Often you'll wish to customize the restricted environment in various ways, most commonly by adding or subtracting variables or functions from the modules available. At a more advanced level, you might wish to write replacements for existing functions; for example, a Web browser that executes Python applets would have an import function that allows retrieving modules via HTTP and importing them.

An easy way to add or remove functions is to create the RExec instance, get the namespace dictionary for the desired module, and add or delete the desired function. For example, the RExec class provides a restricted open() that allows opening files for reading. If you wish to disallow this, you can simply delete 'open' from the RExec instance's __builtin__ module.

module = r_env.add_module('__builtin__')
mod_dict = module.__dict__
del mod_dict['open']

(This isn't enough to prevent code from accessing the filesystem; the RExec class also allows access via some of the functions in the posix module, which is usually aliased to the os module. See below for how to change this.)

This is fine if only a single function is being added or removed, but for more complicated changes, subclassing the RExec class is a better idea.

Subclassing can potentially be quite simple. The RExec class defines some class attributes that are used to initialize the restricted versions of modules such as os and sys. Changing the environment's policy then requires just changing the class attribute in your subclass. For example, the default environment allows restricted code to use the posix module to get its process and group ID. If you decide to disallow this, you can do it with the following custom class:

class MyRExec(rexec.RExec):
    ok_posix_names = ('error', 'fstat', 'listdir', 'lstat', 'readlink',
              'stat', 'times', 'uname')

More elaborate customizations may require overriding one of the methods called to create the corresponding module. The functions to be overridden are make_builtin, make_main, make_osname, and make_sys. The r_import, r_open, and r_reload methods are made available to restricted code, so by overriding these functions, you can change the capabilities available.

For example, defining a new import function requires overriding r_import:

class MyRExec(rexec.RExec):
    def r_import(self, mname, globals={}, locals={}, fromlist=[]):
        raise ImportError, "No imports allowed--ever"

Obviously, a less trivial function could import modules using HTTP, or do something else of interest.

Previous Page Up One Level Next Page Restricted Execution HOWTO Contents
Previous: 3.2 Allowing Access to Up: 3 Customizing The Restricted Next: 4 References
Release 2.0.


4 References

See some of the papers on the Knowbot Programming Environment on CNRI's publications page: ``Knowbot programming: System support for mobile agents'', at http://www.cnri.reston.va.us/home/koe/papers/iwooos-full.html, and ``Using the Knowbot Operating Environment in a Wide-Area Network'', at http://www.cnri.reston.va.us/home/koe/papers/mos.html.

For information on Java's security model, consult the Java Security FAQ at http://java.sun.com/sfaq/index.html.

Perl supports similar features, via a software package called Penguin developed by Felix Gallo. Humberto Ortiz Zuazaga wrote a paper called "The Penguin Model for Secure Distributed Internet Scripting", at http://www.hpcf.upr.edu/~humberto/documents/penguin-safe-scripting.html. Thanks to Fred Drake for bringing it to my attention.

Work has also been done on Safe-Tcl; see ``The Safe-Tcl Security Model'', by Jacob Y. Levy, Laurent Demailly, John K. Ousterhout, and Brent B. Welch, in the Proceedings of the 1998 USENIX Annual Technical Conference. Usenix members can access the paper online at http://www.usenix.org/publications/library/proceedings/usenix98/levy.html.

The Janus project provides a secure environment for untrusted helper applications by trapping unsafe system calls. The project page is http://www.cs.berkeley.edu/~daw/janus/. Thanks to Paul Prescod for suggesting it.

Can you suggest other links, or some academic references, for this section?

Previous Page Up One Level Next Page Restricted Execution HOWTO Contents
Previous: 3.3 Modifying Built-ins Up: Restricted Execution HOWTO Next: 5 Version History
Release 2.0.


5 Version History

Sep. 12, 1998: Minor revisions and added the reference to the Janus project.

Feb. 26, 1998: First version. Suggestions are welcome; please send them to amk1@bigfoot.com.

Mar. 16, 1998: Made some revisions suggested by Jeff Rush. Some minor changes and clarifications, and a sizable section on exceptions added.

Oct. 4, 2000: Checked with Python 2.0. Minor rewrites and fixes made. Version number increased to 2.0.

Previous Page Up One Level Next Page Restricted Execution HOWTO Contents
Previous: 4 References Up: Restricted Execution HOWTO Next: About this document ...
Release 2.0.


About this document ...

Restricted Execution HOWTO

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.

Previous Page Up One Level Restricted Execution HOWTO Contents
Previous: 5 Version History Up: Restricted Execution HOWTO
Release 2.0.


Socket Programming HOWTO

Gordon McMillan

gmcm@hypernet.com

Abstract:

Sockets are used nearly everywhere, but are one of the most severely misunderstood technologies around. This is a 10,000 foot overview of sockets. It's not really a tutorial - you'll still have work to do in getting things operational. It doesn't cover the fine points (and there are a lot of them), but I hope it will give you enough background to begin using them decently.

This document is available in several formats, including PostScript, PDF, HTML and plain ASCII, from the Python HOWTO page at http://www.python.org/doc/howto/.

  • Contents
  • 1 Sockets
    • 1.1 History
  • 2 Creating a Socket
    • 2.1 IPC
  • 3 Using a Socket
    • 3.1 Binary Data
  • 4 Disconnecting
    • 4.1 When Sockets Die
  • 5 Non-blocking Sockets
    • 5.1 Performance
  • About this document ...

Up One Level Next Page Socket Programming HOWTO Contents
Up: HOWTOs Next: Contents
Release 0.00.


1 Sockets

Sockets are used nearly everywhere, but are one of the most severely misunderstood technologies around. This is a 10,000 foot overview of sockets. It's not really a tutorial - you'll still have work to do in getting things working. It doesn't cover the fine points (and there are a lot of them), but I hope it will give you enough background to begin using them decently.

I'm only going to talk about INET sockets, but they account for at least 99% of the sockets in use. And I'll only talk about STREAM sockets - unless you really know what you're doing (in which case this HOWTO isn't for you!), you'll get better behavior and performance from a STREAM socket than anything else. I will try to clear up the mystery of what a socket is, as well as some hints on how to work with blocking and non-blocking sockets. But I'll start by talking about blocking sockets. You'll need to know how they work before dealing with non-blocking sockets.

Part of the trouble with understanding these things is that "socket" can mean a number of subtly different things, depending on context. So first, let's make a distinction between a "client" socket - an endpoint of a conversation, and a "server" socket, which is more like a switchboard operator. The client application (your browser, for example) uses "client" sockets exclusively; the web server it's talking to uses both "server" sockets and "client" sockets.

Subsections
  • 1.1 History

Previous Page Up One Level Next Page Socket Programming HOWTO Contents
Previous: Contents Up: Socket Programming HOWTO Next: 1.1 History
Release 0.00.


1.1 History

Of the various forms of IPC (Inter Process Communication), sockets are by far the most popular. On any given platform, there are likely to be other forms of IPC that are faster, but for cross-platform communication, sockets are about the only game in town.

They were invented in Berkeley as part of the BSD flavor of Unix. They spread like wildfire with the Internet. With good reason -- the combination of sockets with INET makes talking to arbitrary machines around the world unbelievably easy (at least compared to other schemes).

Previous Page Up One Level Next Page Socket Programming HOWTO Contents
Previous: 1 Sockets Up: 1 Sockets Next: 2 Creating a Socket
Release 0.00.


2 Creating a Socket

Roughly speaking, when you clicked on the link that brought you to this page, your browser did something like the following:

    #create an INET, STREAMing socket
    s = socket.socket(
        socket.AF_INET, socket.SOCK_STREAM)
    #now connect to the web server on port 80
    # - the normal http port
    s.connect("www.mcmillan-inc.com", 80)

When the connect completes, the socket s can now be used to send in a request for the text of this page. The same socket will read the reply, and then be destroyed. That's right - destroyed. Client sockets are normally only used for one exchange (or a small set of sequential exchanges).

What happens in the web server is a bit more complex. First, the web server creates a "server socket".

    #create an INET, STREAMing socket
    serverscket = socket.socket(
        socket.AF_INET, socket.SOCK_STREAM)
    #bind the socket to a public host,
    # and a well-known port
    serversocket.bind((socket.gethostname(), 80))
    #become a server socket
    serversocket.listen(5)

A couple things to notice: we used socket.gethostname() so that the socket would be visible to the outside world. If we had used s.bind(('', 80)) or s.bind(('localhost', 80)) or s.bind(('127.0.0.1', 80)) we would still have a "server" socket, but one that was only visible within the same machine.

A second thing to note: low number ports are usually reserved for "well known" services (HTTP, SNMP etc). If you're playing around, use a nice high number (4 digits).

Finally, the argument to listen tells the socket library that we want it to queue up as many as 5 connect requests (the normal max) before refusing outside connections. If the rest of the code is written properly, that should be plenty.

OK, now we have a "server" socket, listening on port 80. Now we enter the mainloop of the web server:

    while 1:
        #accept connections from outside
        (clientsocket, address) = serversocket.accept()
        #now do something with the clientsocket
        #in this case, we'll pretend this is a threaded server
        ct = client_thread(clientsocket)
        ct.run()

There's actually 3 general ways in which this loop could work - dispatching a thread to handle clientsocket, create a new process to handle clientsocket, or restructure this app to use non-blocking sockets, and mulitplex between our "server" socket and any active clientsockets using select. More about that later. The important thing to understand now is this: this is all a "server" socket does. It doesn't send any data. It doesn't receive any data. It just produces "client" sockets. Each clientsocket is created in response to some other "client" socket doing a connect() to the host and port we're bound to. As soon as we've created that clientsocket, we go back to listening for more connections. The two "clients" are free to chat it up - they are using some dynamically allocated port which will be recycled when the conversation ends.

Subsections
  • 2.1 IPC

Previous Page Up One Level Next Page Socket Programming HOWTO Contents
Previous: 1.1 History Up: Socket Programming HOWTO Next: 2.1 IPC
Release 0.00.


2.1 IPC

If you need fast IPC between two processes on one machine, you should look into whatever form of shared memory the platform offers. A simple protocol based around shared memory and locks or semaphores is by far the fastest technique.

If you do decide to use sockets, bind the "server" socket to 'localhost'. On most platforms, this will take a shortcut around a couple of layers of network code and be quite a bit faster.

Previous Page Up One Level Next Page Socket Programming HOWTO Contents
Previous: 2 Creating a Socket Up: 2 Creating a Socket Next: 3 Using a Socket
Release 0.00.


3 Using a Socket

The first thing to note, is that the web browser's "client" socket and the web server's "client" socket are identical beasts. That is, this is a "peer to peer" conversation. Or to put it another way, as the designer, you will have to decide what the rules of etiquette are for a conversation. Normally, the connecting socket starts the conversation, by sending in a request, or perhaps a signon. But that's a design decision - it's not a rule of sockets.

Now there are two sets of verbs to use for communication. You can use send and recv, or you can transform your client socket into a file-like beast and use read and write. The latter is the way Java presents their sockets. I'm not going to talk about it here, except to warn you that you need to use flush on sockets. These are buffered "files", and a common mistake is to write something, and then read for a reply. Without a flush in there, you may wait forever for the reply, because the request may still be in your output buffer.

Now we come the major stumbling block of sockets - send and recv operate on the network buffers. They do not necessarily handle all the bytes you hand them (or expect from them), because their major focus is handling the network buffers. In general, they return when the associated network buffers have been filled (send) or emptied (recv). They then tell you how many bytes they handled. It is your responsibility to call them again until your message has been completely dealt with.

When a recv returns 0 bytes, it means the other side has closed (or is in the process of closing) the connection. You will not receive any more data on this connection. Ever. You may be able to send data successfully; I'll talk about that some on the next page.

A protocol like HTTP uses a socket for only one transfer. The client sends a request, the reads a reply. That's it. The socket is discarded. This means that a client can detect the end of the reply by receiving 0 bytes.

But if you plan to reuse your socket for further transfers, you need to realize that there is no "EOT" (End of Transfer) on a socket. I repeat: if a socket send or recv returns after handling 0 bytes, the connection has been broken. If the connection has not been broken, you may wait on a recv forever, because the socket will not tell you that there's nothing more to read (for now). Now if you think about that a bit, you'll come to realize a fundamental truth of sockets: messages must either be fixed length (yuck), or be delimited (shrug), or indicate how long they are (much better), or end by shutting down the connection. The choice is entirely yours, (but some ways are righter than others).

Assuming you don't want to end the connection, the simplest solution is a fixed length message:

    class mysocket:
        '''demonstration class only
          - coded for clarity, not efficiency'''
        def __init__(self, sock=None):
            if sock is None:
                self.sock = socket.socket(
                    socket.AF_INET, socket.SOCK_STREAM)
            else:
                self.sock = sock
        def connect(host, port):
            self.sock.connect((host, port))
        def mysend(msg):
            totalsent = 0
            while totalsent < MSGLEN:
                sent = self.sock.send(msg[totalsent:])
                if sent == 0:
                    raise RuntimeError, \\
                        "socket connection broken"
                totalsent = totalsent + sent
        def myreceive():
            msg = ''
            while len(msg) < MSGLEN:
                chunk = self.sock.recv(MSGLEN-len(msg))
                if chunk == '':
                    raise RuntimeError, \\
                        "socket connection broken"
                msg = msg + chunk
            return msg

The sending code here is usable for almost any messaging scheme - in Python you send strings, and you can use len() to determine its length (even if it has embedded \0 characters). It's mostly the receiving code that gets more complex. (And in C, it's not much worse, except you can't use strlen if the message has embedded \0s.)

The easiest enhancement is to make the first character of the message an indicator of message type, and have the type determine the length. Now you have two recvs - the first to get (at least) that first character so you can look up the length, and the second in a loop to get the rest. If you decide to go the delimited route, you'll be receiving in some arbitrary chunk size, (4096 or 8192 is frequently a good match for network buffer sizes), and scanning what you've received for a delimiter.

One complication to be aware of: if your conversational protocol allows multiple messages to be sent back to back (without some kind of reply), and you pass recv an arbitrary chunk size, you may end up reading the start of a following message. You'll need to put that aside and hold onto it, until it's needed.

Prefixing the message with it's length (say, as 5 numeric characters) gets more complex, because (believe it or not), you may not get all 5 characters in one recv. In playing around, you'll get away with it; but in high network loads, your code will very quickly break unless you use two recv loops - the first to determine the length, the second to get the data part of the message. Nasty. This is also when you'll discover that send does not always manage to get rid of everything in one pass. And despite having read this, you will eventually get bit by it!

In the interests of space, building your character, (and preserving my competitive position), these enhancements are left as an exercise for the reader. Lets move on to cleaning up.

Subsections
  • 3.1 Binary Data

Previous Page Up One Level Next Page Socket Programming HOWTO Contents
Previous: 2.1 IPC Up: Socket Programming HOWTO Next: 3.1 Binary Data
Release 0.00.


3.1 Binary Data

It is perfectly possible to send binary data over a socket. The major problem is that not all machines use the same formats for binary data. For example, a Motorola chip will represent a 16 bit integer with the value 1 as the two hex bytes 00 01. Intel and DEC, however, are byte-reversed - that same 1 is 01 00. Socket libraries have calls for converting 16 and 32 bit integers - ntohl, htonl, ntohs, htons where "n" means network and "h" means host, "s" means short and "l" means long. Where network order is host order, these do nothing, but where the machine is byte-reversed, these swap the bytes around appropriately.

In these days of 32 bit machines, the ascii representation of binary data is frequently smaller than the binary representation. That's because a surprising amount of the time, all those longs have the value 0, or maybe 1. The string "0" would be two bytes, while binary is four. Of course, this doesn't fit well with fixed-length messages. Decisions, decisions.

Previous Page Up One Level Next Page Socket Programming HOWTO Contents
Previous: 3 Using a Socket Up: 3 Using a Socket Next: 4 Disconnecting
Release 0.00.


4 Disconnecting

Strictly speaking, you're supposed to use shutdown on a socket before you close it. The shutdown is an advisory to the socket at the other end. Depending on the argument you pass it, it can mean "I'm not going to send anymore, but I'll still listen", or "I'm not listening, good riddance!". Most socket libraries, however, are so used to programmers neglecting to use this piece of etiquette that normally a close is the same as shutdown(); close(). So in most situations, an explicit shutdown is not needed.

One way to use shutdown effectively is in an HTTP-like exchange. The client sends a request and then does a shutdown(1). This tells the server "This client is done sending, but can still receive." The server can detect "EOF" by a receive of 0 bytes. It can assume it has the complete request. The server sends a reply. If the send completes successfully then, indeed, the client was still receiving.

Python takes the automatic shutdown a step further, and says that when a socket is garbage collected, it will automatically do a close if it's needed. But relying on this is a very bad habit. If your socket just disappears without doing a close, the socket at the other end may hang indefinitely, thinking you're just being slow. Please close your sockets when you're done.

Subsections
  • 4.1 When Sockets Die

Previous Page Up One Level Next Page Socket Programming HOWTO Contents
Previous: 3.1 Binary Data Up: Socket Programming HOWTO Next: 4.1 When Sockets Die
Release 0.00.


4.1 When Sockets Die

Probably the worst thing about using blocking sockets is what happens when the other side comes down hard (without doing a close). Your socket is likely to hang. SOCKSTREAM is a reliable protocol, and it will wait a long, long time before giving up on a connection. If you're using threads, the entire thread is essentially dead. There's not much you can do about it. As long as you aren't doing something dumb, like holding a lock while doing a blocking read, the thread isn't really consuming much in the way of resources. Do not try to kill the thread - part of the reason that threads are more efficient than processes is that they avoid the overhead associated with the automatic recycling of resources. In other words, if you do manage to kill the thread, your whole process is likely to be screwed up.

Previous Page Up One Level Next Page Socket Programming HOWTO Contents
Previous: 4 Disconnecting Up: 4 Disconnecting Next: 5 Non-blocking Sockets
Release 0.00.


5 Non-blocking Sockets

If you've understood the preceeding, you already know most of what you need to know about the mechanics of using sockets. You'll still use the same calls, in much the same ways. It's just that, if you do it right, your app will be almost inside-out.

In Python, you use socket.setblocking(0) to make it non-blocking. In C, it's more complex, (for one thing, you'll need to choose between the BSD flavor O_NONBLOCK and the almost indistinguishable Posix flavor O_NDELAY, which is completely different from TCP_NODELAY), but it's the exact same idea. You do this after creating the socket, but before using it. (Actually, if you're nuts, you can switch back and forth.)

The major mechanical difference is that send, recv, connect and accept can return without having done anything. You have (of course) a number of choices. You can check return code and error codes and generally drive yourself crazy. If you don't believe me, try it sometime. Your app will grow large, buggy and suck CPU. So let's skip the brain-dead solutions and do it right.

Use select.

In C, coding select is fairly complex. In Python, it's a piece of cake, but it's close enough to the C version that if you understand select in Python, you'll have little trouble with it in C.

    ready_to_read, ready_to_write, in_error = \\
                   select.select(
                      potential_readers,
                      potential_writers,
                      potential_errs,
                      timeout)

You pass select three lists: the first contains all sockets that you might want to try reading; the second all the sockets you might want to try writing to, and the last (normally left empty) those that you want to check for errors. You should note that a socket can go into more than one list. The select call is blocking, but you can give it a timeout. This is generally a sensible thing to do - give it a nice long timeout (say a minute) unless you have good reason to do otherwise.

In return, you will get three lists. They have the sockets that are actually readable, writable and in error. Each of these lists is a subset (possbily empty) of the corresponding list you passed in. And if you put a socket in more than one input list, it will only be (at most) in one output list.

If a socket is in the output readable list, you can be as-close-to-certain-as-we-ever-get-in-this-business that a recv on that socket will return something. Same idea for the writable list. You'll be able to send something. Maybe not all you want to, but something is better than nothing. (Actually, any reasonably healthy socket will return as writable - it just means outbound network buffer space is available.)

If you have a "server" socket, put it in the potential_readers list. If it comes out in the readable list, your accept will (almost certainly) work. If you have created a new socket to connect to someone else, put it in the ptoential_writers list. If it shows up in the writable list, you have a decent chance that it has connected.

One very nasty problem with select: if somewhere in those input lists of sockets is one which has died a nasty death, the select will fail. You then need to loop through every single damn socket in all those lists and do a select([sock],[],[],0) until you find the bad one. That timeout of 0 means it won't take long, but it's ugly.

Actually, select can be handy even with blocking sockets. It's one way of determining whether you will block - the socket returns as readable when there's something in the buffers. However, this still doesn't help with the problem of determining whether the other end is done, or just busy with something else.

Portability alert: On Unix, select works both with the sockets and files. Don't try this on Windows. On Windows, select works with sockets only. Also note that in C, many of the more advanced socket options are done differently on Windows. In fact, on Windows I usually use threads (which work very, very well) with my sockets. Face it, if you want any kind of performance, your code will look very different on Windows than on Unix. (I haven't the foggiest how you do this stuff on a Mac.)

Subsections
  • 5.1 Performance

Previous Page Up One Level Next Page Socket Programming HOWTO Contents
Previous: 4.1 When Sockets Die Up: Socket Programming HOWTO Next: 5.1 Performance
Release 0.00.


5.1 Performance

There's no question that the fastest sockets code uses non-blocking sockets and select to multiplex them. You can put together something that will saturate a LAN connection without putting any strain on the CPU. The trouble is that an app written this way can't do much of anything else - it needs to be ready to shuffle bytes around at all times.

Assuming that your app is actually supposed to do something more than that, threading is the optimal solution, (and using non-blocking sockets will be faster than using blocking sockets). Unfortunately, threading support in Unixes varies both in API and quality. So the normal Unix solution is to fork a subprocess to deal with each connection. The overhead for this is significant (and don't do this on Windows - the overhead of process creation is enormous there). It also means that unless each subprocess is completely independent, you'll need to use another form of IPC, say a pipe, or shared memory and semaphores, to communicate between the parent and child processes.

Finally, remember that even though blocking sockets are somewhat slower than non-blocking, in many cases they are the "right" solution. After all, if your app is driven by the data it receives over a socket, there's not much sense in complicating the logic just so your app can wait on select instead of recv.

Previous Page Up One Level Next Page Socket Programming HOWTO Contents
Previous: 5 Non-blocking Sockets Up: 5 Non-blocking Sockets Next: About this document ...
Release 0.00.


About this document ...

Socket Programming HOWTO

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.

Previous Page Up One Level Socket Programming HOWTO Contents
Previous: 5.1 Performance Up: Socket Programming HOWTO
Release 0.00.


Sorting Mini-HOWTO

Andrew Dalke

dalke@bioreason.com

Abstract:

This document is a little tutorial showing a half dozen ways to sort a list with the built-in sort() method.

This document is available in several formats, including PostScript, PDF, HTML and plain ASCII, from the Python HOWTO page at http://www.python.org/doc/howto/.

  • Contents
  • 1 Sorting basic data types
  • 2 Comparing classes
  • About this document ...

Up One Level Next Page Sorting Mini-HOWTO Contents
Up: HOWTOs Next: Contents
Release 0.01.


1 Sorting basic data types

A simple ascending sort is easy; just call the sort() method of a list.

>>> a = [5, 2, 3, 1, 4]
>>> a.sort()
>>> print a
[1, 2, 3, 4, 5]

Sort takes an optional function which can be called for doing the comparisons. The default sort routine is equivalent to

>>> a = [5, 2, 3, 1, 4]
>>> a.sort(cmp)
>>> print a
[1, 2, 3, 4, 5]

where cmp is the built-in function which compares two objects, x and y, and returns -1, 0 or 1 depending on whether , , or . During the course of the sort the relationships must stay the same for the final list to make sense.

If you want, you can define your own function for the comparison. For integers (and numbers in general) we can do:

>>> def numeric_compare(x, y):
>>>    return x-y
>>>
>>> a = [5, 2, 3, 1, 4]
>>> a.sort(numeric_compare)
>>> print a
[1, 2, 3, 4, 5]

By the way, this function won't work if result of the subtraction is out of range, as in sys.maxint - (-1).

Or, if you don't want to define a new named function you can create an anonymous one using lambda, as in:

>>> a = [5, 2, 3, 1, 4]
>>> a.sort(lambda x, y: x-y)
>>> print a
[1, 2, 3, 4, 5]

If you want the numbers sorted in reverse you can do

>>> a = [5, 2, 3, 1, 4]
>>> def reverse_numeric(x, y):
>>>     return y-x
>>>
>>> a.sort(reverse_numeric)
>>> print a
[5, 4, 3, 2, 1]

(a more general implementation could return cmp(y,x) or -cmp(x,y)).

However, it's faster if Python doesn't have to call a function for every comparison, so if you want a reverse-sorted list of basic data types, do the forward sort first, then use the reverse() method.

>>> a = [5, 2, 3, 1, 4]
>>> a.sort()
>>> a.reverse()
>>> print a
[1, 2, 3, 4, 5]

Here's a case-insensitive string comparison using a lambda function:

>>> import string
>>> a = string.split("This is a test string from Andrew.")
>>> a.sort(lambda x, y: cmp(string.lower(x), string.lower(y)))
>>> print a
['a', 'Andrew.', 'from', 'is', 'string', 'test', 'This']

This goes through the overhead of converting a word to lower case every time it must be compared. At times it may be faster to compute these once and use those values, and the following example shows how.

>>> words = string.split("This is a test string from Andrew.")
>>> offsets = []
>>> for i in range(len(words)):
>>>     offsets.append( (string.lower(words[i]), i) )
>>>
>>> offsets.sort()
>>> new_words = []
>>> for dontcare, i in offsets:
>>>      new_words.append(words[i])
>>>
>>> print new_words

The offsets list is initialized to a tuple of the lower-case string and its position in the words list. It is then sorted. Python's sort method sorts tuples by comparing terms; given x and y, compare x[0] to y[0], then x[1] to y[1], etc. until there is a difference.

The result is that the offsets list is ordered by its first term, and the second term can be used to figure out where the original data was stored. (The for loop assigns dontcare and i to the two fields of each term in the list, but we only need the index value.)

Another way to implement this is to store the original data as the second term in the offsets list, as in:

>>> words = string.split("This is a test string from Andrew.")
>>> offsets = []
>>> for word in words:
>>>     offsets.append( (string.lower(word), word) )
>>>
>>> offsets.sort()
>>> new_words = []
>>> for word in offsets:
>>>     new_words.append(word[1])
>>>
>>> print new_words

This isn't always appropriate because the second terms in the list (the word, in this example) will be compared when the first terms are the same. If this happens many times, then there will be the unneeded performance hit of comparing the two objects. This can be a large cost if most terms are the same and the objects define their own __cmp__ method, but there will still be some overhead to determine if __cmp__ is defined.

Still, for large lists, or for lists where the comparison information is expensive to calculate, the last two examples are likely to be the fastest way to sort a list. It will not work on weakly sorted data, like complex numbers, but if you don't know what that means, you probably don't need to worry about it.

Previous Page Up One Level Next Page Sorting Mini-HOWTO Contents
Previous: Contents Up: Sorting Mini-HOWTO Next: 2 Comparing classes
Release 0.01.


2 Comparing classes

The comparison for two basic data types, like ints to ints or string to string, is built into Python and makes sense. There is a default way to compare class instances, but the default manner isn't usually very useful. You can define your own comparison with the __cmp__ method, as in:

>>> class Spam:
>>>     def __init__(self, spam, eggs):
>>>         self.spam = spam
>>>         self.eggs = eggs
>>>     def __cmp__(self, other):
>>>         return cmp(self.spam+self.eggs, other.spam+other.eggs)
>>>     def __str__(self):
>>>         return str(self.spam + self.eggs)
>>>
>>> a = [Spam(1, 4), Spam(9, 3), Spam(4,6)]
>>> a.sort()
>>> for spam in a:
>>>   print str(spam)
5
10
12

Sometimes you may want to sort by a specific attribute of a class. If appropriate you should just define the __cmp__ method to compare those values, but you cannot do this if you want to compare between different attributes at different times. Instead, you'll need to go back to passing a comparison function to sort, as in:

>>> a = [Spam(1, 4), Spam(9, 3), Spam(4,6)]
>>> a.sort(lambda x, y: cmp(x.eggs, y.eggs))
>>> for spam in a:
>>>   print spam.eggs, str(spam)
3 12
4 5
6 10

If you want to compare two arbitrary attributes (and aren't overly concerned about performance) you can even define your own comparison function object. This uses the ability of a class instance to emulate an function by defining the __call__ method, as in:

>>> class CmpAttr:
>>>     def __init__(self, attr):
>>>         self.attr = attr
>>>     def __call__(self, x, y):
>>>         return cmp(getattr(x, self.attr), getattr(y, self.attr))
>>>
>>> a = [Spam(1, 4), Spam(9, 3), Spam(4,6)]
>>> a.sort(CmpAttr("spam"))  # sort by the "spam" attribute
>>> for spam in a:
>>>    print spam.spam, spam.eggs, str(spam)
1 4 5
4 6 10
9 3 12

>>> a.sort(CmpAttr("eggs"))   # re-sort by the "eggs" attribute
>>> for spam in a:
>>>    print spam.spam, spam.eggs, str(spam)
9 3 12
1 4 5
4 6 10

Of course, if you want a faster sort you can extract the attributes into an intermediate list and sort that list.

So, there you have it; about a half-dozen different ways to define how to sort a list:

  • sort using the default method
  • sort using a comparison function
  • reverse sort not using a comparison function
  • sort on an intermediate list (two forms)
  • sort using class defined __cmp__ method
  • sort using a sort function object

Previous Page Up One Level Next Page Sorting Mini-HOWTO Contents
Previous: 1 Sorting basic data Up: Sorting Mini-HOWTO Next: About this document ...
Release 0.01.


About this document ...

Sorting Mini-HOWTO

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.

Previous Page Up One Level Sorting Mini-HOWTO Contents
Previous: 2 Comparing classes Up: Sorting Mini-HOWTO
Release 0.01.


Python/XML HOWTO

The Python/XML Special Interest Group

xml-sig@python.org
(edited by amk1@bigfoot.com)

Abstract:

XML is the eXtensible Markup Language, a subset of SGML, intended to allow the creation and processing of application-specific markup languages. Python makes an excellent language for processing XML data. This document is a tutorial for the Python/XML package. It assumes you're already familiar with the structure and terminology of XML.

This is a draft document; 'XXX' in the text indicates that something has to be filled in later, or rewritten, or verified, or something.

  • Contents
  • 1 Introduction to XML
    • 1.1 Related Links
  • 2 Installing the XML Toolkit
    • 2.1 Related Links
  • 3 SAX: The Simple API for XML
    • 3.1 Starting Out
    • 3.2 Error Handling
    • 3.3 Searching Element Content
    • 3.4 Related Links
  • 4 DOM: The Document Object Model
    • 4.1 Getting A DOM Tree
    • 4.2 Manipulating The Tree
      • 4.2.1 The Node class
      • 4.2.2 Document, Element, and Text nodes
    • 4.3 Walking Over The Entire Tree
    • 4.4 Building A Document
    • 4.5 Processing HTML
    • 4.6 Related Links
  • 5 Glossary
  • About this document ...

Up One Level Next Page Python/XML HOWTO Contents
Up: HOWTOs Next: Contents
Release 0.6.1.


1 Introduction to XML

XML, the eXtensible Markup Language, is a simplified dialect of SGML, the Standardized General Markup Language. XML is intended to be reasonably simple to implement and use, and is already being used for specifying markup languages for various new standards: MathML for expressing mathematical equations, Synchronized Multimedia Integration Language for multimedia presentations, and so forth.

SGML and XML represent a document by tagging the document's various components with their function, or meaning. For example, an academic paper contains several parts: it has a title, one or more authors, an abstract, the actual text of the paper, a list of references, and so forth. A markup languge for writing such papers would therefore have tags for indicating what the contents of the abstract are, what the title is, and so forth. This should not be confused with the physical details of how the document is actually printed on paper. The abstract might be printed with narrow margins in a smaller font than the rest of the document, but the markup usually won't be concerned with details such as this; other software will translate from the markup language to a typesetting language such as TEX, and will handle the details.

A markup language specified using XML looks a lot like HTML; a document consists of a single element, which contains sub-elements, which can have further sub-elements inside them. Elements are indicated by tags in the text. Tags are always inside angle brackets < >. There are two forms of elements. An element can contain content between opening and closing tags, as in <name>Euryale</name>, which is a name element containing the data "Euryale". This content may be text data, other XML elements, or a mixture of both. Elements can also be empty, in which case they contain nothing, and are represented as a single tag ended with a slash, as in <stop/>, which is an empty stop element. Unlike HTML, XML element names are case-sensitive; stop and Stop are two different element types.

Opening and empty tags can also contain attributes, which specify values associated with an element. For example, text such as <name lang='greek'>Herakles</name>, the name element has a lang attribute which has a value of "greek". This would contrast with <name lang='latin'>Hercules</name>, where the attribute's value is "latin".

A given XML language is specified with a Document Type Definition, or DTD. The DTD declares the element names that are allowed, and how elements can be nested inside each other. The DTD also specifies the attributes that can be provided for each element, their default values, and if they can be omitted. For example, to take an example from HTML, the LI element, representing an entry in a list, can only occur inside certain elements which represent lists, such as OL or UL. A validating parser can be given a DTD and a document, and verify whether a given document is legal according to the DTD's rules, or determine that one or more rules have been violated.

Applications that process XML can be classed into two types. The simplest class is an application that only handles one particular markup language. For example, a chemistry program may only need to process Chemical Markup Language, but not MathML. This application can therefore be written specifically for a single DTD, and doesn't need to be capable of handling multiple markup languages. This type is simpler to write, and can easily be implemented with the available Python software.

The second type of application is less common, and has to be able to handle any markup language you throw at it. An example might be a smart XML editor that helps you to write XML that conforms to a selected DTD; it might do so by not letting you enter an element where it would be illegal, or by suggesting elements that can be placed at the current cursor location. Such an application needs to handle any possible XML-defined markup, and therefore must be able to obtain a data structure embodying the DTD in use. XXX This type of application can't currently be implemented in Python without difficulty (XXX but wait and see if a DTD module is included...)

For the full details of XML's syntax, the one definitive source is the XML 1.0 specification, available on the Web at http://www.w3.org/TR/xml-spec.html. However, like all specifications, it's quite formal and isn't intended to be a friendly introduction or a tutorial. The annotated version of the standard, at http://www.xml.com/xml/pub/axml/axmlintro.html, is quite helpful in clarifying the specification's intent. There are also various informal tutorials and books available to introduce you to XML.

The rest of this HOWTO will assume that you're familiar with the relevant terminology. Most section will use XML terms such as element and attribute; section 4 on the Document Object Model will assume that you've read the relevant Working Draft, and are familiar with things like Iterators and Nodes. Section 3 does not require that you have experience with the Java SAX implentations.

Subsections
  • 1.1 Related Links

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: Contents Up: Python/XML HOWTO Next: 1.1 Related Links
Release 0.6.1.


1.1 Related Links

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 1 Introduction to XML Up: 1 Introduction to XML Next: 2 Installing the XML
Release 0.6.1.


2 Installing the XML Toolkit

Windows users should get the precompiled version at http://sourceforge.net/projects/pyxml; Mac users will use the corresponding precompiled version at XXX. Linux users may wish to use either the Debian package from XXX, or the RPM from http://sourceforge.net/projects/pyxml. To compile from source on a Unix platform, simply perform the following steps.

  1. If you have are using Python 1.5, you need to install the distutils first, which are available from http://www.python.org/sigs/distutils-sig. Python 1.6 and later already includes the distutils, so you can skip this step.

  2. Get a copy of the source distribution from http://sourceforge.net/projects/pyxml. Unpack it with the following command.

    gzip -dc xml-package.tgz | tar -xvf -

  3. Run:
    python setup.py install

    To properly execute this operation, a C compiler is required - the same that was used to build Python itself. On a Unix system, this operation may require superuser permissions. setup.py supports a number of different commands and options, invoke setup.py without any arguments to obtain help.

If you have difficulty installing this software, send a problem report to <xml-sig@python.org> describing the problem, or submit a bug report at http://sourceforget.net/projects/pyxml.

There are various demonstration programs in the demo/ directory of the source distribution. You may wish to look at them next to get an impression of what's possible with the XML tools, and as a source of example code.

Subsections
  • 2.1 Related Links

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 1.1 Related Links Up: Python/XML HOWTO Next: 2.1 Related Links
Release 0.6.1.


2.1 Related Links

http://www.python.org/topics/xml/
This is the starting point for Python-related XML topics; it is updated to refer to all software, mailing lists, documentation, etc.

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 2 Installing the XML Up: 2 Installing the XML Next: 3 SAX: The Simple
Release 0.6.1.


3 SAX: The Simple API for XML

The Simple API for XML isn't a standard in the formal sense, but an informal specification designed by David Megginson, with input from many people on the xml-dev mailing list. SAX defines an event-driven interface for parsing XML. To use SAX, you must create Python class instances which implement a specified interface, and the parser will then call various methods of those objects.

This howto describes version 2 of SAX (also referred to as SAX2). Earlier versions of this text did explain SAX1, which is primarily of historical interest only.

SAX is most suitable for purposes where you want to read through an entire XML document from beginning to end, and perform some computation, such as building a data structure representating a document, or summarizing information in a document (computing an average value of a certain element, for example). It's not very useful if you want to modify the document structure in some complicated way that involves changing how elements are nested, though it could be used if you simply wish to change element contents or attributes. For example, you would not want to re-order chapters in a book using SAX, but you might want to change the contents of any name elements with the attribute lang equal to 'greek' into Greek letters.

One advantage of SAX is speed and simplicity. Let's say you've defined a complicated DTD for listing comic books, and you wish to scan through your collection and list everything written by Neil Gaiman. For this specialized task, there's no need to expend effort examining elements for artists and editors and colourists, because they're irrelevant to the search. You can therefore write a class instance which ignores all elements that aren't writer.

Another advantage is that you don't have the whole document resident in memory at any one time, which matters if you are processing really huge documents.

SAX defines 4 basic interfaces; an SAX-compliant XML parser can be passed any objects that support these interfaces, and will call various methods as data is processed. Your task, therefore, is to implement those interfaces that are relevant to your application.

The SAX interfaces are:

Interface | Purpose
ContentHandler | Called for general document events. This interface is the heart of SAX; its methods are called for the start of the document, the start and end of elements, and for the characters of data contained inside elements.
DTDHandler | Called to handle DTD events required for basic parsing. This means notation declarations (XML spec section 4.7) and unparsed entity declarations (XML spec section 4).
EntityResolver | Called to resolve references to external entities. If your documents will have no external entity references, you won't need to implement this interface.
ErrorHandler | Called for error handling. The parser will call methods from this interface to report all warnings and errors.

Python doesn't support the concept of interfaces, so the interfaces listed above are implemented as Python classes. The default method implementations are defined to do nothing--the method body is just a Python pass statement-so usually you can simply ignore methods that aren't relevant to your application.

Pseudo-code for using SAX looks something like this:

# Define your specialized handler classes
from xml.sax import Contenthandler, ...
class docHandler(ContentHandler):
    ...

# Create an instance of the handler classes
dh = docHandler()

# Create an XML parser
parser = ...

# Tell the parser to use your handler instance
parser.setContentHandler(dh)

# Parse the file; your handler's method will get called
parser.parse(sys.stdin)

Subsections
  • 3.1 Starting Out
  • 3.2 Error Handling
  • 3.3 Searching Element Content
  • 3.4 Related Links

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 2.1 Related Links Up: Python/XML HOWTO Next: 3.1 Starting Out
Release 0.6.1.


3.1 Starting Out

Following the earlier example, let's consider a simple XML format for storing information about a comic book collection. Here's a sample document for a collection consisting of a single issue:

<collection>
  <comic title="Sandman" number='62'>
    <writer>Neil Gaiman</writer>
    <penciller pages='1-9,18-24'>Glyn Dillon</penciller>
    <penciller pages="10-17">Charles Vess</penciller>
  </comic>
</collection>

An XML document must have a single root element; this is the "collection" element. It has one child comic element for each issue; the book's title and number are given as attributes of the comic element, which can have one or more children containing the issue's writer and artists. There may be several artists or writers for a single issue.

Let's start off with something simple: a document handler named FindIssue that reports whether a given issue is in the collection.

from xml.sax import saxutils

class FindIssue(saxutils.DefaultHandler):
    def __init__(self, title, number):
        self.search_title, self.search_number = title, number

The DefaultHandler class inherits from all four interfaces: ContentHandler, DTDHandler, EntityResolver, and ErrorHandler. This is what you should use if you want to use one class for everything. When you want separate classes for each purpose, or if you want to implement only a single interface, you can just subclass each interface individually. Neither of the two approaches is always ``better'' than the other; their suitability depends on what you're trying to do, and on what you prefer.

Since this class is doing a search, an instance needs to know what to search for. The desired title and issue number are passed to the FindIssue constructor, and stored as part of the instance.

Now let's look at the function which actually does all the work. This simple task only requires looking at the attributes of a given element, so only the startElement method is relevant.

    def startElement(self, name, attrs):
        # If it's not a comic element, ignore it
        if name != 'comic': return

        # Look for the title and number attributes (see text)
        title = attrs.get('title', None)
        number = attrs.get('number', None)
        if title == self.search_title and number == self.search_number:
            print title, '#'+str(number), 'found'

The startElement() method is passed a string giving the name of the element, and an instance containing the element's attributes. The latter implements the AttributeList interface, which includes most of the semantics of Python dictionaries. Therefore, the function looks for comic elements, and compares the specified title and number attributes to the search values. If they match, a message is printed out.

startElement() is called for every single element in the document. If you added print 'Starting element:', name to the top of startElement(), you would get the following output.

Starting element: collection
Starting element: comic
Starting element: writer
Starting element: penciller
Starting element: penciller

To actually use the class, we need top-level code that creates instances of a parser and of FindIssue, associates them, and then calls a parser method to process the input.

from xml.sax import make_parser
from xml.sax.handler import feature_namespaces

if __name__ == '__main__':
    # Create a parser
    parser = make_parser()
    # Tell the parser we are not interested in XML namespaces
    parser.setFeature(feature_namespaces, 0)

    # Create the handler
    dh = FindIssue('Sandman', '62')

    # Tell the parser to use our handler
    parser.setContentHandler(dh)

    # Parse the input
    parser.parse(file)

The make_parser class can automate the job of creating parsers. There are already several XML parsers available to Python, and more might be added in future. xmllib.py is included with Python 1.5, so it's always available, but it's also not particularly fast. A faster version of xmllib.py is included in xml.parsers. The xml.parsers.expat module is faster still, so it's obviously a preferred choice if it's available. make_parser determines which parsers are available and chooses the fastest one, so you don't have to know what the different parsers are, or how they differ. (You can also tell make_parser to try a list of parsers, if you want to use a specific one).

In SAX2, XML namespace are supported. Parsers will not call startElement, but startElementNS if namespace processing is active. Since our content handler does not implement the namespace-aware methods, we request that namespace processing is deactivated. The default of this setting varies from parser to parser, so you should always set it to a safe value - unless your handlers support either method.

Once you've created a parser instance, calling setContentHandler tells the parser what to use as the handler.

If you run the above code with the sample XML document, it'll output Sandman #62 found.

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 3 SAX: The Simple Up: 3 SAX: The Simple Next: 3.2 Error Handling
Release 0.6.1.


3.2 Error Handling

Now, try running the above code with this file as input:

<collection>
  &foo;
  <comic title="Sandman" number='62'>
</collection>

The &foo; entity is unknown, and the comic element isn't closed (if it was empty, there would be a "/" before the closing ">". As a result, you get a SAXParseException, e.g.

xml.sax._exceptions.SAXParseException: undefined entity at None:2:2

The default code for the ErrorHandler interface automatically raises an exception for any error; if that is what you want in case of an error, you don't need to change the error handler. Otherwise, you should provide your own version of the ErrorHandler interface, and at minimum override the error() and fatalError() methods. The minimal implementation for each method can be a single line. The methods in the ErrorHandler interface-warning, error, and fatalError-are all passed a single argument, an exception instance. The exception will always be a subclass of SAXException, and calling str() on it will produce a readable error message explaining the problem.

So, to re-implement a variant of ErrorRaiser, simply define one of the three methods to print the exception they're passed:

    def error(self, exception):
        import sys
        sys.stderr.write("\%s\n" \% exception)

With this definition, non-fatal errors will result in an error message, whereas fatal errors will continue to produce a traceback.

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 3.1 Starting Out Up: 3 SAX: The Simple Next: 3.3 Searching Element Content
Release 0.6.1.


3.3 Searching Element Content

Let's tackle a slightly more complicated task, printing out all issues written by a certain author. This now requires looking at element content, because the writer's name is inside a writer element: <writer>Peter Milligan</writer>.

The search will be performed using the following algorithm:

  1. The startElement method will be more complicated. For comic elements, the handler has to save the title and number, in case this comic is later found to match the search criterion. For writer elements, it sets a inWriterContent flag to true, and sets a writerName attribute to the empty string.

  2. Characters outside of XML tags must be processed. When inWriterContent is true, these characters must be added to the writerName string.

  3. When the writer element is finished, we've now collected all of the element's content in the writerName attribute, so we can check if the name matches the one we're searching for, and if so, print the information about this comic. We must also set inWriterContent back to false.

Here's the first part of the code; this implements step 1.

from xml.sax import ContentHandler
import string

def normalize_whitespace(text):
    "Remove redundant whitespace from a string"
    return string.join(string.split(text), ' ')

class FindWriter(ContentHandler):
    def __init__(self, search_name):
        # Save the name we're looking for
        self.search_name = normalize_whitespace(search_name)

        # Initialize the flag to false
        self.inWriterContent = 0

    def startElement(self, name, attrs):
        # If it's a comic element, save the title and issue
        if name == 'comic':
            title = normalize_whitespace(attrs.get('title', ""))
            number = normalize_whitespace(attrs.get('number', ""))
            self.this_title = title
            self.this_number = number

        # If it's the start of a writer element, set flag
        elif name == 'writer':
            self.inWriterContent = 1
            self.writerName = ""

The startElement() method has been discussed previously. Now we have to look at how the content of elements is processed.

The normalize_whitespace() function is important, and you'll probably use it in your own code. XML treats whitespace very flexibly; you can include extra spaces or newlines wherever you like. This means that you must normalize the whitespace before comparing attribute values or element content; otherwise the comparision might produce a wrong result due to the content of two elements having different amounts of whitespace.

    def characters(self, ch):
        if self.inWriterContent:
            self.writerName = self.writerName + ch

The characters() method is called for characters that aren't inside XML tags. ch is a string of characters. It is not necessarily a byte string; parsers may also provide a buffer object that is a slice of the full document, or they may pass Unicode objects (as the expat parser does in Python 2.0).

You also shouldn't assume that all the characters are passed in a single function call. In the example above, there might be only one call to characters() for the string "Peter Milligan", or it might call characters() once for each character. More realistically, if the content contains an entity reference, as in "Wagner &amp; Seagle", the parser might call the method three times; once for "Wagner ", once for "&", represented by the entity reference, and again for " Seagle".

For step 2 of FindWriter, characters() only has to check inWriterContent, and if it's true, add the characters to the string being built up.

Finally, when the writer element ends, the entire name has been collected, so we can compare it to the name we're searching for.

    def endElement(self, name):
        if name == 'writer':
            self.inWriterContent = 0
            self.writerName = normalize_whitespace(self.writerName)
            if self.search_name == self.writerName:
                print 'Found:', self.this_title, self.this_number

To avoid being confused by differing whitespace, the normalize_whitespace() function is called. This can be done because we know that leading and trailing whitespace are insignificant for this element, in this DTD.

End tags can't have attributes on them, so there's no attrs parameter. Empty elements with attributes, such as "<arc name="Season of Mists"/>", will result in a call to startElement(), followed immediately by a call to endElement().

XXX how are external entities handled? Anything special need to be done for them?

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 3.2 Error Handling Up: 3 SAX: The Simple Next: 3.4 Related Links
Release 0.6.1.


3.4 Related Links

http://www.megginson.com/SAX/
The SAX home page. This has the most recent copy of the specification, and lists SAX implementations for various languages and platforms. At the moment it's somewhat Java-centric.

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 3.3 Searching Element Content Up: 3 SAX: The Simple Next: 4 DOM: The Document
Release 0.6.1.


4 DOM: The Document Object Model

The Document Object Model specifies a tree-based representation for an XML document. A top-level Document instance is the root of the tree, and has a single child which is the top-level Element instance; this Element has children nodes representing the content and any sub-elements, which may have further children, and so forth. Functions are defined which let you traverse the resulting tree any way you like, access element and attribute values, insert and delete nodes, and convert the tree back into XML.

The DOM is useful for modifying XML documents, because you can create a DOM tree, modify it by adding new nodes and moving subtrees around, and then produce a new XML document as output. You can also construct a DOM tree yourself, and convert it to XML; this is often a more flexible way of producing XML output than simply writing <tag1>...</tag1> to a file.

While the DOM doesn't require that the entire tree be resident in memory at one time, the Python DOM implementation currently does keep the whole tree in RAM. It's possible to write an implementation that stores most of the tree on disk or in a database, and reads in new sections as they're accessed, but this hasn't been done yet. This means you may not have enough memory to process very large documents as a DOM tree. A SAX handler, on the other hand, can potentially churn through amounts of data far larger than the available RAM.

Subsections
  • 4.1 Getting A DOM Tree
  • 4.2 Manipulating The Tree
    • 4.2.1 The Node class
    • 4.2.2 Document, Element, and Text nodes
  • 4.3 Walking Over The Entire Tree
  • 4.4 Building A Document
  • 4.5 Processing HTML
  • 4.6 Related Links

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 3.4 Related Links Up: Python/XML HOWTO Next: 4.1 Getting A DOM
Release 0.6.1.


4.1 Getting A DOM Tree

The easiest way to get a DOM tree is to have it built for you. PyXML offers two alternative implementations of the DOM, xml.dom.minidom and 4DOM. xml.dom.minidom is included in Python 2. It is a minimalistic implementation, which means it does not provide all interfaces and operations required by the DOM standard. 4DOM (XXX reference) is a complete implementation of DOM Level 2 (which is currently work in progress), so we will use that in the examples.

One of the modules in the xml.dom package is xml.dom.ext.reader.Sax2, which provides the functions FromXmlStream, FromXml, FromXmlFile, and FromXmlUrl which will construct a DOM tree from their input (a file-like object, a string, a file name, and a URL, respectively). They all return a DOM Document object.

import sys
from xml.dom.ext.reader.Sax import FromXmlStream
from xml.dom.ext import PrettyPrint

# parse the document
doc = FromXmlStream(sys.stdin)

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 4 DOM: The Document Up: 4 DOM: The Document Next: 4.2 Manipulating The Tree
Release 0.6.1.


4.2 Manipulating The Tree

This HOWTO can't be a complete introduction to the Document Object Model, because there are lots of interfaces and lots of methods. Luckily, the DOM Recommendation is quite a readable document, so I'd recommend that you read it to get a complete picture of the available interfaces; this will only be a partial overview.

The Document Object Model represents a XML document as a tree of nodes, represented by an instance of some subclass of the Node class. Some subclasses of Node are Element, Text, and Comment.

We'll use a single example document throughout this section. Here's the sample:

<?xml version="1.0" encoding="iso-8859-1"?>
<xbel>
  <?processing instruction?>
  <desc>No description</desc>
  <folder>
    <title>XML bookmarks</title>
    <bookmark href="http://www.python.org/sigs/xml-sig/" >
      <title>SIG for XML Processing in Python</title>
    </bookmark>
  </folder>
</xbel>

Converted to a DOM tree, this document could produce the following tree:

Element xbel None
   Text #text '  \012  '
   ProcessingInstruction processing 'instruction'
   Text #text '\012  '
   Element desc None
      Text #text 'No description'
   Text #text '\012  '
   Element folder None
      Text #text '\012    '
      Element title None
         Text #text 'XML bookmarks'
      Text #text '\012    '
      Element bookmark None
         Text #text '\012      '
         Element title None
            Text #text 'SIG for XML Processing in Python'
         Text #text '\012    '
      Text #text '\012  '
   Text #text '\012'

This isn't the only possible tree, because different parsers may differ in how they generate Text nodes; any of the Text nodes in the above tree might be split into multiple nodes.)

Subsections
  • 4.2.1 The Node class
  • 4.2.2 Document, Element, and Text nodes

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 4.1 Getting A DOM Up: 4 DOM: The Document Next: 4.2.1 The Node class
Release 0.6.1.


4.3 Walking Over The Entire Tree

The xml.dom package also includes various helper classes for common tasks such as walking over trees.

The Walker class

Introduction to the walker class

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 4.2.2 Document, Element, and Up: 4 DOM: The Document Next: 4.4 Building A Document
Release 0.6.1.


4.4 Building A Document

Intro to builder

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 4.3 Walking Over The Up: 4 DOM: The Document Next: 4.5 Processing HTML
Release 0.6.1.


4.5 Processing HTML

Intro to HTML builder

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 4.4 Building A Document Up: 4 DOM: The Document Next: 4.6 Related Links
Release 0.6.1.


4.6 Related Links

http://www.w3.org/DOM/
The World Wide Web Consortium's DOM page.

http://www.w3.org/TR/1998/REC-DOM-Level-1-19981001/
The DOM Level 1 Recommendation. Unlike most standards, this one is actually pretty readable, particularly if you're only interested in the Core XML interfaces.

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 4.5 Processing HTML Up: 4 DOM: The Document Next: 5 Glossary
Release 0.6.1.


5 Glossary

XML has given rise to a sea of acronyms and terms. This section will list the most significant terms, and sketch their relevance.

Many of the following definitions are taken from Lars Marius Garshol's SGML glossary, at http://www.stud.ifi.uio.no/&#126;larsga/download/diverse/sgmlglos.html.

DOM (Document Object Model)
The Document Object Model is intended to be a platform- and language-neutral interface that will allow programs and scripts to dynamically access and update the content, structure and style of documents. Documents will be represented as tree structures which can be traversed and modified.

DTD (Document Type Definition)
A Document Type Definition (nearly always called DTD) defines an XML document type, complete with element types, entities and an XML declaration.

In other words: a DTD completely describes one particular kind of XML document, such as, for instance, HTML 3.2.

SAX (Simple API for XML)
SAX is a simple standardized API for XML parsers developed by the contributors to the xml-dev mailing list. The interface is mostly language-independent, as long as the language is object-oriented; the first implementation was written for Java, but a Python implementation is also available. SAX is supported by many XML parsers.

XML (eXtensible Markup Language)
XML is an SGML application profile specialized for use on the web and has its own standards for linking and stylesheets under development.

XSL (eXtensible Style Language)
XSL is a proposal for a stylesheet language for XML, which enables browsers to lay out XML documents in an attractive manner, and also provides a way to convert XML documents to HTML.

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 4.6 Related Links Up: Python/XML HOWTO Next: About this document ...
Release 0.6.1.


About this document ...

Python/XML HOWTO

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.

Previous Page Up One Level Python/XML HOWTO Contents
Previous: 5 Glossary Up: Python/XML HOWTO
Release 0.6.1.


Python/XML Reference Guide

The Python/XML Special Interest Group

xml-sig@python.org
(edited by amk1@bigfoot.com)

Abstract:

XML is the eXtensible Markup Language, a subset of SGML, intended to allow the creation and processing of application-specific markup languages. Python makes an excellent language for processing XML data. This document is the reference manual for the Python/XML package, containing several XML modules.

This is a draft document; 'XXX' in the text indicates that something has to be filled in later, or rewritten, or verified, or something.

This document is available in several formats, including PostScript, PDF, HTML and plain ASCII, from the Python HOWTO page at http://www.python.org/doc/howto/.

  • Contents
  • 1 xml.arch.xmlarch
    • 1.1 ArchDocHandler methods
    • 1.2 Architecture methods
    • 1.3 ArchParseState methods
    • 1.4 ArchException methods
    • 1.5 AttributeParser methods
    • 1.6 Normalizer methods
  • 2 xml.parsers.xmllib
  • 3 xml.sax.saxexts
    • 3.1 ExtendedParser methods
    • 3.2 ParserFactory methods
  • 4 xml.sax.saxlib
    • 4.1 AttributeList methods
    • 4.2 DocumentHandler methods
    • 4.3 DTDHandler methods
    • 4.4 EntityResolver methods
    • 4.5 ErrorHandler methods
    • 4.6 Locator methods
    • 4.7 Parser methods
    • 4.8 SAXException methods
    • 4.9 SAXParseException methods
  • 5 xml.sax.saxutils
    • 5.1 Location methods
  • 6 xml.utils.iso8601
  • About this document ...

Up One Level Next Page Python/XML Reference Guide Contents
Up: HOWTOs Next: Contents
Release 0.06.


1 xml.arch.xmlarch

The xmlarch module contains an XML architectural forms processor written in Python. It allows you to process XML architectural forms using any parser that uses the SAX interfaces. The module allows you to process several architectures in one parsing pass. Architectural document events for an architecture can even be broadcasted to multiple DocumentHandlers. (e.g. you can have 2 handlers for the RDF architecture, 3 for the XLink architecture and perhaps one for the HyTime architecture.)

The architecture processor uses the SAX DocumentHandler interface which means that you can register the architecture handler (ArchDocHandler) with any SAX 1.0 compliant parser.

It currently does not process any meta document type definition documents (meta-DTDs). When a DTD parser module is available the code will be modified to use that in order to process meta-DTD information.

Please note that validating and well-formed parsers may report different SAX events when parsing documents.

The xmlarch module contains six classes: ArchDocHandler, Architecture, ArchParseState, ArchException, AttributeParser and Normalizer.

  • ArchDocHandler is a subclass of the saxlib.DocumentHandler interface. This is the class used for processing an architectural document.

  • Architecture contains information about an architecture.

  • ArchParseState holds information about an architecture's parse state when parsing a document.

  • AttributeParser parses architecture use declaration PIs (attribute strings).

  • ArchException holds information about an architectural exception thrown by an ArchDocHandler instance.

  • Normalizer is a document handler that outputs "normalized" XML.

Subsections
  • 1.1 ArchDocHandler methods
  • 1.2 Architecture methods
  • 1.3 ArchParseState methods
  • 1.4 ArchException methods
  • 1.5 AttributeParser methods
  • 1.6 Normalizer methods

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: Contents Up: Python/XML Reference Guide Next: 1.1 ArchDocHandler methods
Release 0.06.


1.1 ArchDocHandler methods

Still undocumented.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 1 xml.arch.xmlarch Up: 1 xml.arch.xmlarch Next: 1.2 Architecture methods
Release 0.06.


1.2 Architecture methods

Still undocumented.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 1.1 ArchDocHandler methods Up: 1 xml.arch.xmlarch Next: 1.3 ArchParseState methods
Release 0.06.


1.3 ArchParseState methods

Still undocumented.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 1.2 Architecture methods Up: 1 xml.arch.xmlarch Next: 1.4 ArchException methods
Release 0.06.


1.4 ArchException methods

Still undocumented.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 1.3 ArchParseState methods Up: 1 xml.arch.xmlarch Next: 1.5 AttributeParser methods
Release 0.06.


1.5 AttributeParser methods

Still undocumented.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 1.4 ArchException methods Up: 1 xml.arch.xmlarch Next: 1.6 Normalizer methods
Release 0.06.


1.6 Normalizer methods

Still undocumented.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 1.5 AttributeParser methods Up: 1 xml.arch.xmlarch Next: 2 xml.parsers.xmllib
Release 0.06.


2 xml.parsers.xmllib

This is a version of the xmllib module from Python 1.5, modified to use the sgmlop C extension when it's available. This produces a significant speedup, amounting to about a factor of 5. The interface is unchanged from the original xmllib module; consult the Python Library Reference documentation for that module.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 1.6 Normalizer methods Up: Python/XML Reference Guide Next: 3 xml.sax.saxexts
Release 0.06.


3 xml.sax.saxexts

make_parser([parser])
A utility function that returns a Parser object for a non-validating XML parser. If parser is specified, it must be a parser name; otherwise, a list of available parsers is checked and the fastest available is chosen.

HTMLParserFactory
An instance of the ParserFactory class that's already been prepared with a list of HTML parsers. Simply call its make_parser() method to get a Parser object.

class ParserFactory()
A general class to be used by applications for creating parsers on foreign systems where the list of installed parsers is unknown.

SGMLParserFactory
An instance of the ParserFactory class that's already been prepared with a list of SGML parsers. Simply call its make_parser() method to get a parser object.

XMLParserFactory
An instance of the ParserFactory class that's already been prepared with a list of nonvalidating XML parsers. Simply call its make_parser() method to get a parser object.

XMLValParserFactory
An instance of the ParserFactory class that's already been prepared with a list of validating XML parsers. Simply call its make_parser() method to get a parser object.

class ExtendedParser()
This class is an experimental extended parser interface, that offers additional functionality that may be useful. However, it's not specified by the SAX specification.

Subsections
  • 3.1 ExtendedParser methods
  • 3.2 ParserFactory methods

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 2 xml.parsers.xmllib Up: Python/XML Reference Guide Next: 3.1 ExtendedParser methods
Release 0.06.


3.1 ExtendedParser methods

close()
Called after the last call to feed, when there are no more data.

feed(data)
Feeds data to the parser.

get_parser_name()
Returns a single-word parser name.

get_parser_version()
Returns the version of the imported parser, which may not be the one the driver was implemented for.

is_dtd_reading()
True if the parser is non-validating, but conforms to the XML specification by reading the DTD.

is_validating()
Returns true if the parser is validating, false otherwise.

reset()
Makes the parser start parsing afresh, discarding any buffered input data.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 3 xml.sax.saxexts Up: 3 xml.sax.saxexts Next: 3.2 ParserFactory methods
Release 0.06.


3.2 ParserFactory methods

get_parser_list()
Returns the list of possible drivers. Currently this starts out as ["xml.sax.drivers.drv_xmltok", "xml.sax.drivers.drv_xmlproc", "xml.sax.drivers.drv_xmltoolkit", "xml.sax.drivers.drv_xmllib"].

make_parser([driver_name])
Returns a SAX driver for the first available parser of the parsers in the list. Note that the list contains drivers, so it first tries the driver and, if that exists, imports it to determine if the parser also exists. If no parsers are available a SAXException is thrown.

Optionally, driver_name can be a string containing the name of the driver to be used; the stored parser list will then not be used at all.

set_parser_list(list)
Sets the driver list to list.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 3.1 ExtendedParser methods Up: 3 xml.sax.saxexts Next: 4 xml.sax.saxlib
Release 0.06.


4 xml.sax.saxlib

class AttributeList()
Interface for an attribute list. This interface provides information about a list of attributes for an element (only specified or defaulted attributes will be reported). Note that the information returned by this object will be valid only during the scope of the DocumentHandler.startElement() callback, and the attributes will not necessarily be provided in the order declared or specified.

class DocumentHandler()
Handle general document events. This is the main client interface for SAX: it contains callbacks for the most important document events, such as the start and end of elements. You need to create an object that implements this interface, and then register it with the Parser. If you do not want to implement the entire interface, you can derive a class from HandlerBase, which implements the default functionality. You can find the location of any document event using the Locator interface supplied by setDocumentLocator().

class DTDHandler()
Handle DTD events. This interface specifies only those DTD events required for basic parsing (unparsed entities and attributes). If you do not want to implement the entire interface, you can extend HandlerBase, which implements the default behaviour.

class EntityResolver()
This is the basic interface for resolving entities. If you create an object implementing this interface, then register the object with your Parser instance, the parser will call the method in your object to resolve all external entities. Note that HandlerBase implements this interface with the default behaviour.

class ErrorHandler()
This is the basic interface for SAX error handlers. If you create an object that implements this interface, then register the object with your Parser, the parser will call the methods in your object to report all warnings and errors. There are three levels of errors available: warnings, (possibly) recoverable errors, and unrecoverable errors. All methods take a SAXParseException as the only parameter.

class HandlerBase()
Default base class for handlers. This class implements the default behaviour for four SAX interfaces, inheriting from them all: EntityResolver, DTDHandler, DocumentHandler, and ErrorHandler. Rather than implementing those full interfaces, you may simply extend this class and override the methods that you need. Note that the use of this class is optional, since you are free to implement the interfaces directly if you wish.

class Locator()
Interface for associating a SAX event with a document location. A locator object will return valid results only during calls to methods of the SAXDocumentHandler class; at any other time, the results are unpredictable.

class Parser()
Basic interface for SAX parsers. All SAX parsers must implement this basic interface: it allows users to register handlers for different types of events and to initiate a parse from a URI, a character stream, or a byte stream. SAX parsers should also implement a zero-argument constructor.

class SAXException(msg, exception, locator)
Encapsulate an XML error or warning. This class can contain basic error or warning information from either the XML parser or the application: you can subclass it to provide additional functionality, or to add localization. Note that although you will receive a SAXException as the argument to the handlers in the ErrorHandler interface, you are not actually required to throw the exception; instead, you can simply read the information in it.

class SAXParseException(msg, exception, locator)
Encapsulate an XML parse error or warning.

This exception will include information for locating the error in the original XML document. Note that although the application will receive a SAXParseException as the argument to the handlers in the ErrorHandler interface, the application is not actually required to throw the exception; instead, it can simply read the information in it and take a different action.

Since this exception is a subclass of SAXException, it inherits the ability to wrap another exception.

Subsections
  • 4.1 AttributeList methods
  • 4.2 DocumentHandler methods
  • 4.3 DTDHandler methods
  • 4.4 EntityResolver methods
  • 4.5 ErrorHandler methods
  • 4.6 Locator methods
  • 4.7 Parser methods
  • 4.8 SAXException methods
  • 4.9 SAXParseException methods

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 3.2 ParserFactory methods Up: Python/XML Reference Guide Next: 4.1 AttributeList methods
Release 0.06.


4.1 AttributeList methods

The AttributeList class supports some of the behaviour of Python dictionaries; the len(), has_key(), items(), keys() and values() methods are available, and attr['href'] will retrieve the value of the href attribute. There are also additional methods specific to AttributeList:

getLength()
Return the number of attributes in the list.

getName(i)
Return the name of attribute i in the list.

getType(i)
Return the type of an attribute in the list. i can be either the integer index or the attribute name.

getValue(i)
Return the value of an attribute in the list. i can be either the integer index or the attribute name.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 4 xml.sax.saxlib Up: 4 xml.sax.saxlib Next: 4.2 DocumentHandler methods
Release 0.06.


4.2 DocumentHandler methods

characters(ch, start, length)
Handle a character data event.

endDocument()
Handle an event for the end of a document.

endElement(name)
Handle an event for the end of an element.

ignorableWhitespace(ch, start, length)
Handle an event for ignorable whitespace in element content.

processingInstruction(target, data)
Handle a processing instruction event.

setDocumentLocator(locator)
Receive an object for locating the origin of SAX document events. You'll probably want to store the value of locator as an attribute of the handler instance.

startDocument()
Handle an event for the beginning of a document.

startElement(name, attrs)
Handle an event for the beginning of an element.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 4.1 AttributeList methods Up: 4 xml.sax.saxlib Next: 4.3 DTDHandler methods
Release 0.06.


4.3 DTDHandler methods

notationDecl(name, publicId, systemId)
Handle a notation declaration event.

unparsedEntityDecl(publicId, systemId, notationName)
Handle an unparsed entity declaration event.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 4.2 DocumentHandler methods Up: 4 xml.sax.saxlib Next: 4.4 EntityResolver methods
Release 0.06.


4.4 EntityResolver methods

resolveEntity(name, publicId, systemId)
Resolve the system identifier of an entity.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 4.3 DTDHandler methods Up: 4 xml.sax.saxlib Next: 4.5 ErrorHandler methods
Release 0.06.


4.5 ErrorHandler methods

error(exception)
Handle a recoverable error.

fatalError(exception)
Handle a non-recoverable error.

warning(exception)
Handle a warning.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 4.4 EntityResolver methods Up: 4 xml.sax.saxlib Next: 4.6 Locator methods
Release 0.06.


4.6 Locator methods

getColumnNumber()
Return the column number where the current event ends.

getLineNumber()
Return the line number where the current event ends.

getPublicId()
Return the public identifier for the current event.

getSystemId()
Return the system identifier for the current event.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 4.5 ErrorHandler methods Up: 4 xml.sax.saxlib Next: 4.7 Parser methods
Release 0.06.


4.7 Parser methods

parse(systemId)
Parse an XML document from a system identifier.

parseFile(fileobj)
Parse an XML document from a file-like object.

setDocumentHandler(handler)
Register an object to receive basic document-related events.

setDTDHandler(handler)
Register an object to receive basic DTD-related events.

setEntityResolver(resolver)
Register an object to resolve external entities.

setErrorHandler(handler)
Register an object to receive error-message events.

setLocale(locale)
Allow an application to set the locale for errors and warnings.

SAX parsers are not required to provide localisation for errors and warnings; if they cannot support the requested locale, however, they must throw a SAX exception. Applications may request a locale change in the middle of a parse.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 4.6 Locator methods Up: 4 xml.sax.saxlib Next: 4.8 SAXException methods
Release 0.06.


4.8 SAXException methods

getException()
Return the embedded exception, if any.

getMessage()
Return a message for this exception.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 4.7 Parser methods Up: 4 xml.sax.saxlib Next: 4.9 SAXParseException methods
Release 0.06.


4.9 SAXParseException methods

The SAXParseException class has a locator attribute, containing an instance of the Locator class, which represents the location in the document where the parse error occurred. The following methods are delegated to this instance.

getColumnNumber()
Return the column number of the end of the text where the exception occurred.

getLineNumber()
Return the line number of the end of the text where the exception occurred.

getPublicId()
Return the public identifier of the entity where the exception occurred.

getSystemId()
Return the system identifier of the entity where the exception occurred.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 4.8 SAXException methods Up: 4 xml.sax.saxlib Next: 5 xml.sax.saxutils
Release 0.06.


5 xml.sax.saxutils

class Canonizer(writer)
A SAX document handler that produces canonicalized XML output. writer must support a write() method which accepts a single string.

class ErrorPrinter()
A simple class that just prints error messages to standard error (sys.stderr).

class ESISDocHandler(writer)
A SAX document handler that produces naive ESIS output. writer must support a write() method which accepts a single string.

class EventBroadcaster(list)
Takes a list of objects and forwards any method calls received to all objects in the list. The attribute list holds the list and can freely be modified by clients.

class Location(locator)
Represents a location in an XML entity. Initialized by being passed a locator, from which it reads off the current location, which is then stored internally.

Subsections
  • 5.1 Location methods

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 4.9 SAXParseException methods Up: Python/XML Reference Guide Next: 5.1 Location methods
Release 0.06.


5.1 Location methods

getColumnNumber()
Return the column number of the location.

getLineNumber()
Return the line number of the location.

getPublicId()
Return the public identifier for the location.

getSystemId()
Return the system identifier for the location.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 5 xml.sax.saxutils Up: 5 xml.sax.saxutils Next: 6 xml.utils.iso8601
Release 0.06.


6 xml.utils.iso8601

The xml.utils.iso8601 module provides conversion routines between the ISO 8601 representations of date/time values and the floating point values used elsewhere in Python. The floating point represtentation is particularly useful in conjunction with the standard time module.

Currently, this module supports a small superset of the ISO 8601 profile described by the World Wide Web Consortium (W3C). This is a subset of ISO 8601, but covers the cases expected to be used most often in the context of XML processing and Web applications. Future versions of this module may support a larger subset of ISO 8601-defined formats.

parse(s)
Parse an ISO 8601 date representation (with an optional time-of-day component) and return the date in seconds since the epoch.

parse_timezone(timezone)
Parse an ISO 8601 time zone designator and return the offset relative to Universal Coordinated Time (UTC) in seconds. If timezone is not valid, ValueError is raised.

tostring(t[, timezone])
Return formatted date/time value according to the profile described by the W3C. If timezone is provided, it must be the offset from UTC in seconds specified as a number, or time zone designator which can be parsed by parse_timezone(). If timezone is specified as a string and cannot be parsed by parse_timezone(), ValueError will be raised.

ctime(t)
Return formatter date/time value using the local timezone. This is equivalent to "tostring(t, time.timezone)".

See Also:

International Organization for Standardization. Data elements and interchange formats -- Information interchange -- Representation of dates and times. International Organization for Standardization, 1988.

Gary Houston. ISO 8601 date/time representations. January 1993. Available online as compressed PostScript: ftp://ftp.informatik.uni-erlangen.de/pub/doc/ISO/ISO8601.ps.Z.

Markus Kuhn. A Summary of the International Standard Dateand Time Notation. Available online at http://www.cl.cam.ac.uk/~mgk25/iso-time.html.

Misha Wolf and Charles Wicksteed. Date and Time Formats. World Wide Web Consortium Technical Note, September 1998. Available online at http://www.w3.org/TR/NOTE-datetime.

Previous Page Up One Level Next Page Python/XML Reference Guide Contents
Previous: 5.1 Location methods Up: Python/XML Reference Guide Next: About this document ...
Release 0.06.


About this document ...

Python/XML Reference Guide

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.

Previous Page Up One Level Python/XML Reference Guide Contents
Previous: 6 xml.utils.iso8601 Up: Python/XML Reference Guide
Release 0.06.


Writing a HOWTO Document

There are many other possible topics for HOWTOs; contributions of new HOWTOs would be very welcome.

Currently HOWTOs are written in LaTeX, using the same macros and style as the Library Reference, and the other documentation included with the standard interpreter. This lets us easily generate PostScript, PDF, and HTML versions of documentation. You're not forced to use LaTeX, though; the HOWTO editor will cheerfully work with you to convert from whatever format is most convenient for you.

If using LaTeX isn't a problem for you, you can grab Fred Drake's documentation source distribution, and use templates/howto.tex as a template for your own document. The howto document class isn't just useful for HOWTOs; it's also useful for documenting your own packages, or for writing Python-related papers.

Possible topics

Some topics that have been suggested for future HOWTOs:

  • Using cgi.py
  • Using the Python profiler
  • Using the Python debugger
  • Using Pipermail
  • Socket programming & inter-process communication in general
  • PythonWin
  • NumPy
  • individual GUI toolkits such as Gtk, Qt, or Tkinter

If you're interested in writing a HOWTO, whether about a topic listed above or a topic of your own choosing, inform the HOWTO editor.


Contents

  • 1 Reasons to Use Python
    • 1.1 Programmability
    • 1.2 Prototyping
    • 1.3 Simplicity and Ease of Understanding
    • 1.4 Java Integration
  • 2 Arguments and Rebuttals
  • 3 Useful Resources
  • About this document ...

Previous Page Up One Level Next Page Python Advocacy HOWTO
Previous: Python Advocacy HOWTO Up: Python Advocacy HOWTO Next: 1 Reasons to Use
Release 0.03.


Contents

  • 1 Python
  • 2 Bobo
  • 3 Medusa
  • 4 Giving it a go!
  • About this document ...

This is possibly the easiest way of getting WWW CGI running on both Unix and win32. Some of the benefits of using this combination are:

  • incredible speed,
  • ease of implementation that Bobo gives, and
  • totally portable code between Unix and win95.

This document will attempt to demonstrate how to install the Bobo WWW object publishing system (CGI made easy) on either a Win32 or UNIX system. All components in this HOWTO are written in Python (except, of course, the interpreter itself).

Python is a useful interpreted language for, among many other tasks, writing Internet applications. Consult http://www.python.org for more information.

Bobo is an object publisher written in Python. The Bobo product is a cross between an object request broker, an object database, and a publishing system to reference web material as a collection of distributed objects. Bobo is provided by Digital Creations under very generous (free) terms. Consult the Bobo distribution page at http://www.digicool.com/releases/bobo/.

Medusa is a high-performance, extensible Internet (HTTP, FTP, NNTP, ...) server framework. Medusa is provided under a more restrictive license that requires an arrangement to be made if it is to be used to make money. Consult the Medusa Web page at http://www.nightmare.com/medusa.

To get Bobo and Medusa working, you'll need to obtain the following components:

  1. Python 1.5 from the Python Language Website at http://www.python.org/. Follow the links to obtain a binary or source for your platform.

  2. Medusa dated Jan 24 1998 or later, from http://www.nightmare.com/medusa.

  3. Bobo release 2.0 or later (I recommend you get DocumentTemplate and the Bobo Persistent Object System also), from http://www.digicool.com/releases/bobo/.

  4. Amos Lattemeier's bobo_handler.py for Medusa, from http://starship.skyport.net/crew/amos/.

Previous Page Up One Level Next Page Bobo and Medusa
Previous: Bobo and Medusa Up: Bobo and Medusa Next: 1 Python
Release 0.01.


Contents

  • 1 What is curses?
    • 1.1 The Python curses module
  • 2 Starting and ending a curses Application
  • 3 Windows and Pads
  • 4 Displaying Text
    • 4.1 Attributes and Colour
  • 5 User Input
  • 6 For More Information
  • About this document ...

Previous Page Up One Level Next Page Curses Programming with Python
Previous: Curses Programming with Python Up: Curses Programming with Python Next: 1 What is curses?
Release 2.0.


Contents

  • 1 Language Constructs You Should Not Use
    • 1.1 from module import *
    • 1.2 Unadorned exec, execfile and friends
    • 1.3 from module import name1, name2
    • 1.4 except:
  • 2 Exceptions
  • 3 Using the Batteries
  • About this document ...

Previous Page Up One Level Next Page Idioms and Anti-Idioms in Python
Previous: Idioms and Anti-Idioms in Up: Idioms and Anti-Idioms in Next: 1 Language Constructs You
Release 0.00.


1.1.1 Inside Function Definitions

from module import * is invalid inside function definitions. While many versions of Python do no check for the invalidity, it does not make it more valid, no more then having a smart lawyer makes a man innocent. Do not use it like that ever. Even in versions where it was accepted, it made the function execution slower, because the compiler could not be certain which names are local and which are global. In Python 2.1 this construct causes warnings, and sometimes even errors.

Previous Page Up One Level Next Page Idioms and Anti-Idioms in Python Contents
Previous: 1.1 from module import Up: 1.1 from module import Next: 1.1.2 At Module Level
Release 0.00.


1.1.2 At Module Level

While it is valid to use from module import * at module level it is usually a bad idea. For one, this loses an important property Python otherwise has -- you can know where each toplevel name is defined by a simple "search" function in your favourite editor. You also open yourself to trouble in the future, if some module grows additional functions or classes.

One of the most awful question asked on the newsgroup is why this code:

f = open("www")
f.read()

does not work. Of course, it works just fine (assuming you have a file called "www".) But it does not work if somewhere in the module, the statement from os import * is present. The os module has a function called open() which returns an integer. While it is very useful, shadowing builtins is one of its least useful properties.

Remember, you can never know for sure what names a module exports, so either take what you need -- from module import name1, name2, or keep them in the module and access on a per-need basis -- import module;print module.name.

Previous Page Up One Level Next Page Idioms and Anti-Idioms in Python Contents
Previous: 1.1.1 Inside Function Definitions Up: 1.1 from module import Next: 1.1.3 When It Is
Release 0.00.


1.1.3 When It Is Just Fine

There are situations in which from module import * is just fine:

  • The interactive prompt. For example, from math import * makes Python an amazing scientific calculator.

  • When extending a module in C with a module in Python.

  • When the module advertises itself as from import * safe.

Previous Page Up One Level Next Page Idioms and Anti-Idioms in Python Contents
Previous: 1.1.2 At Module Level Up: 1.1 from module import Next: 1.2 Unadorned exec, execfile
Release 0.00.


Contents

  • Front Matter
    • 1 Alpha
      • 1.1 Features
      • 1.2 Installation
    • 2 CodeWright
      • 2.1 Features
      • 2.2 Installation
    • 3 Emacs
    • 4 jed
    • 5 vi
    • 1 How to construct a Python-mode for your editor
      • 1.1 Indentation
      • 1.2 Strings
  • About this document ...

Python writers have two choices: use a Python IDE or install a python-mode for their favorite editor. This HOWTO shows you how to do the latter. Editor support for the Python language can range from simple automatic indentation to the ability to execute a file. If an editor is currently supported by the Python community, pointers are provided for configuring the editor. In addition an appendix discussing issues specific to Python in implementing editor support is included. If your editor does not support Python, we hope you will consider expanding our offerings by implementing such support.

Previous Page Up One Level Next Page Editor Configuration for Python
Previous: Front Matter Up: Editor Configuration for Python Next: 1 Alpha
Release 2.0.


Contents

  • 1 Introduction
  • 2 Simple Patterns
    • 2.1 Matching Characters
    • 2.2 Repeating Things
  • 3 Using Regular Expressions
    • 3.1 Compiling Regular Expressions
    • 3.2 The Backslash Plague
    • 3.3 Performing Matches
    • 3.4 Module-Level Functions
    • 3.5 Compilation Flags
  • 4 More Pattern Power
    • 4.1 More Metacharacters
    • 4.2 Grouping
    • 4.3 Non-capturing, and Named Groups
    • 4.4 Other Assertions
  • 5 Modifying Strings
    • 5.1 Splitting Strings
    • 5.2 Search and Replace
  • 6 Common Problems
    • 6.1 Know About the string Module
    • 6.2 match() versus search()
    • 6.3 Greedy versus Non-Greedy
    • 6.4 Not using re.VERBOSE
  • 7 Feedback
  • About this document ...

Previous Page Up One Level Next Page Regular Expression HOWTO
Previous: Regular Expression HOWTO Up: Regular Expression HOWTO Next: 1 Introduction
Release 0.03.


Contents

  • 1 Why Should You Port?
  • 2 Module-level differences
  • 3 Pattern Differences
  • 4 String Substitutions
  • About this document ...

Previous Page Up One Level Next Page regex-to-re HOWTO
Previous: regex-to-re HOWTO Up: regex-to-re HOWTO Next: 1 Why Should You
Release 2.0.


Contents

  • 1 Basic use of RExec
  • 2 Frequently Asked Questions
  • 3 Customizing The Restricted Environment
    • 3.1 Inserting Variables
    • 3.2 Allowing Access to Unrestricted Objects
    • 3.3 Modifying Built-ins
  • 4 References
  • 5 Version History
  • About this document ...

Previous Page Up One Level Next Page Restricted Execution HOWTO
Previous: Restricted Execution HOWTO Up: Restricted Execution HOWTO Next: 1 Basic use of
Release 2.0.


Contents

  • 1 Sockets
    • 1.1 History
  • 2 Creating a Socket
    • 2.1 IPC
  • 3 Using a Socket
    • 3.1 Binary Data
  • 4 Disconnecting
    • 4.1 When Sockets Die
  • 5 Non-blocking Sockets
    • 5.1 Performance
  • About this document ...

Previous Page Up One Level Next Page Socket Programming HOWTO
Previous: Socket Programming HOWTO Up: Socket Programming HOWTO Next: 1 Sockets
Release 0.00.


Contents

  • 1 Sorting basic data types
  • 2 Comparing classes
  • About this document ...

Python lists have a built-in sort() method. There are many ways to use it to sort a list and there doesn't appear to be a single, central place in the various manuals describing them, so I'll do so here.

Previous Page Up One Level Next Page Sorting Mini-HOWTO
Previous: Sorting Mini-HOWTO Up: Sorting Mini-HOWTO Next: 1 Sorting basic data
Release 0.01.


Contents

  • 1 Introduction to XML
    • 1.1 Related Links
  • 2 Installing the XML Toolkit
    • 2.1 Related Links
  • 3 SAX: The Simple API for XML
    • 3.1 Starting Out
    • 3.2 Error Handling
    • 3.3 Searching Element Content
    • 3.4 Related Links
  • 4 DOM: The Document Object Model
    • 4.1 Getting A DOM Tree
    • 4.2 Manipulating The Tree
    • 4.3 Walking Over The Entire Tree
    • 4.4 Building A Document
    • 4.5 Processing HTML
    • 4.6 Related Links
  • 5 Glossary
  • About this document ...

Previous Page Up One Level Next Page Python/XML HOWTO
Previous: Python/XML HOWTO Up: Python/XML HOWTO Next: 1 Introduction to XML
Release 0.6.1.


4.2.1 The Node class

We'll start by considering the basic Node class. All the other DOM nodes -- Document, Element, Text, and so forth -- are subclasses of Node. It's possible to perform many tasks using just the interface provided by Node.

XXX table of attributes and methods
            readonly attribute  DOMString            nodeName;
                     attribute  DOMString            nodeValue;
                                                           // raises(DOMException) on setting
                                                           // raises(DOMException) on retrieval
            readonly attribute  unsigned short       nodeType;
            readonly attribute  Node                 parentNode;
            readonly attribute  NodeList             childNodes;
            readonly attribute  Node                 firstChild;
            readonly attribute  Node                 lastChild;
            readonly attribute  Node                 previousSibling;
            readonly attribute  Node                 nextSibling;
            readonly attribute  NamedNodeMap         attributes;
            readonly attribute  Document             ownerDocument;

Node insertBefore(in Node newChild, in Node refChild) raises(DOMException); Node replaceChild(in Node newChild, in Node oldChild) raises(DOMException); Node removeChild(in Node oldChild) raises(DOMException); Node appendChild(in Node newChild) raises(DOMException); boolean hasChildNodes(); Node cloneNode(in boolean deep);

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 4.2 Manipulating The Tree Up: 4.2 Manipulating The Tree Next: 4.2.2 Document, Element, and
Release 0.6.1.


4.2.2 Document, Element, and Text nodes

The base of the entire tree is the Document node. Its documentElement attribute contains the Element node for the root element. The Document node may have additional children, such as ProcessingInstruction nodes; the complete list of children XXX.

Previous Page Up One Level Next Page Python/XML HOWTO Contents
Previous: 4.2.1 The Node class Up: 4.2 Manipulating The Tree Next: 4.3 Walking Over The
Release 0.6.1.


Contents

  • 1 xml.arch.xmlarch
    • 1.1 ArchDocHandler methods
    • 1.2 Architecture methods
    • 1.3 ArchParseState methods
    • 1.4 ArchException methods
    • 1.5 AttributeParser methods
    • 1.6 Normalizer methods
  • 2 xml.parsers.xmllib
  • 3 xml.sax.saxexts
    • 3.1 ExtendedParser methods
    • 3.2 ParserFactory methods
  • 4 xml.sax.saxlib
    • 4.1 AttributeList methods
    • 4.2 DocumentHandler methods
    • 4.3 DTDHandler methods
    • 4.4 EntityResolver methods
    • 4.5 ErrorHandler methods
    • 4.6 Locator methods
    • 4.7 Parser methods
    • 4.8 SAXException methods
    • 4.9 SAXParseException methods
  • 5 xml.sax.saxutils
    • 5.1 Location methods
  • 6 xml.utils.iso8601
  • About this document ...

Previous Page Up One Level Next Page Python/XML Reference Guide
Previous: Python/XML Reference Guide Up: Python/XML Reference Guide Next: 1 xml.arch.xmlarch
Release 0.06.