PEP 0 -- Index of Python Enhancement Proposals (PEPs)

PEP: | 0
Title: | Index of Python Enhancement Proposals (PEPs)
Version: | $Revision: 1.90 $
Author: | barry@digicool.com (Barry A. Warsaw)
Status: | Active
Type: | Informational
Created: | 13-Jul-2000
Post-History: |

Introduction

    The PEP contains the index of all Python Enhancement Proposals,
    known as PEPs.  PEP numbers are assigned by the PEP Editor, and
    once assigned are never changed.

Index by Category

     num  filename      title                                  owner
     ---  ------------  -----                                  -----

 Meta-PEPs (PEPs about PEPs or Process)

 I     0  pep-0000.txt  Index of Python Enhancement Proposals  Warsaw
 I     1  pep-0001.txt  PEP Guidelines                         Warsaw, Hylton
 I     3  pep-0003.txt  Guidelines for Handling Bug Reports    Hylton
 I     4  pep-0004.txt  Deprecation of Standard Modules        von Loewis
 I     5  pep-0005.txt  Guidelines for Language Evolution      Prescod
 I     6  pep-0006.txt  Bug Fix Releases                       Aahz

 Other Informational PEPs

 I   247  pep-0247.txt  API for Cryptographic Hash Functions   Kuchling
 I   248  pep-0248.txt  Python Database API Specification v1.0 Lemburg
 I   249  pep-0249.txt  Python Database API Specification v2.0 Lemburg

 Active PEPs (under consideration for Python 2.2)

 I    42  pep-0042.txt  Small Feature Requests                 Hylton
 S   234  pep-0234.txt  Iterators                              Yee, van Rossum
 S   237  pep-0237.txt  Unifying Long Integers and Integers    Zadka
 S   238  pep-0238.txt  Non-integer Division                   Zadka
 S   239  pep-0239.txt  Adding a Rational Type to Python       Zadka
 S   240  pep-0240.txt  Adding a Rational Literal to Python    Zadka
 S   241  pep-0241.txt  Metadata for Python Software Packages  Kuchling
 S   242  pep-0242.txt  Numeric Kinds                          Dubois
 S   243  pep-0243.txt  Module Repository Upload Mechanism     Reifschneider
 S   244  pep-0244.txt  The `directive' Statement              von Loewis
 S   245  pep-0245.txt  Python Interface Syntax                Pelletier
 S   246  pep-0246.txt  Object Adaptation                      Evans
 S   250  pep-0250.txt  Using site-packages on All Platforms   Moore
 I   251  pep-0251.txt  Python 2.2 Release Schedule            Warsaw
 S   252  pep-0252.txt  Making Types Look More Like Classes    van Rossum
 S   253  pep-0253.txt  Subtyping Built-in Types               van Rossum

 Py-in-the-sky PEPs (not ready; may become active yet)

 I   206  pep-0206.txt  2.0 Batteries Included                 Zadka
 S   209  pep-0209.txt  Adding Multidimensional Arrays        Barrett, Oliphant
 SD  211  pep-0211.txt  Adding New Linear Algebra Operators    Wilson
 SD  212  pep-0212.txt  Loop Counter Iteration                 Schneider-Kamp
 SD  213  pep-0213.txt  Attribute Access Handlers              Prescod
 S   215  pep-0215.txt  String Interpolation                   Yee
 I   216  pep-0216.txt  Docstring Format                       Zadka
 SD  218  pep-0218.txt  Adding a Built-In Set Object Type      Wilson
 SD  219  pep-0219.txt  Stackless Python                       McMillan
 S   222  pep-0222.txt  Web Library Enhancements               Kuchling
 SD  225  pep-0225.txt  Elementwise/Objectwise Operators       Zhu, Lielens
 S   228  pep-0228.txt  Reworking Python's Numeric Model       Zadka

 Finished PEPs (done, implemented)

 IF  160  pep-0160.txt  Python 1.6 Release Schedule            Drake
 IF  200  pep-0200.txt  Python 2.0 Release Schedule            Hylton
 SF  201  pep-0201.txt  Lockstep Iteration                     Warsaw
 S   202  pep-0202.txt  List Comprehensions                    Peters
 SF  203  pep-0203.txt  Augmented Assignments                  Wouters
 S   205  pep-0205.txt  Weak References                        Drake
 SF  207  pep-0207.txt  Rich Comparisons                   van Rossum, Ascher
 SF  208  pep-0208.txt  Reworking the Coercion Model       Schemenauer, Lemburg
 SF  214  pep-0214.txt  Extended Print Statement               Warsaw
 SF  217  pep-0217.txt  Display Hook for Interactive Use       Zadka
 SF  221  pep-0221.txt  Import As                              Wouters
 SF  223  pep-0223.txt  Change the Meaning of \x Escapes       Peters
 I   226  pep-0226.txt  Python 2.1 Release Schedule            Hylton
 S   227  pep-0227.txt  Statically Nested Scopes               Hylton
 S   229  pep-0229.txt  Using Distutils to Build Python        Kuchling
 SF  230  pep-0230.txt  Warning Framework                      van Rossum
 SF  232  pep-0232.txt  Function Attributes                    Warsaw
 SF  235  pep-0235.txt  Import on Case-Insensitive Platforms   Peters
 S   236  pep-0236.txt  Back to the __future__                 Peters

 Empty PEPs (or containing only an abstract)

 ID    2  pep-0002.txt  Procedure for Adding New Modules       Raymond
 SD  210  pep-0210.txt  Decoupling the Interpreter Loop        Ascher
 ID  220  pep-0220.txt  Coroutines, Generators, Continuations  McMillan

 Deferred, Abandoned, and Rejected PEPs

 SR  204  pep-0204.txt  Range Literals                         Wouters
 SR  224  pep-0224.txt  Attribute Docstrings                   Lemburg
 SR  231  pep-0231.txt  __findattr__()                         Warsaw
 S   233  pep-0233.txt  Python Online Help                     Prescod

Numerical Index

     num  filename      title                                  owner
     ---  ------------  -----                                  -----
 I     0  pep-0000.txt  Index of Python Enhancement Proposals  Warsaw
 I     1  pep-0001.txt  PEP Guidelines                         Warsaw, Hylton
 I     2  pep-0002.txt  Procedure for Adding New Modules       Raymond
 I     3  pep-0003.txt  Guidelines for Handling Bug Reports    Hylton
 I     4  pep-0004.txt  Deprecation of Standard Modules        von Loewis
 I     5  pep-0005.txt  Guidelines for Language Evolution      Prescod
 I     6  pep-0006.txt  Bug Fix Releases                       Aahz
 I    42  pep-0042.txt  Small Feature Requests                 Hylton

 IF  160  pep-0160.txt  Python 1.6 Release Schedule            Drake

 IF  200  pep-0200.txt  Python 2.0 Release Schedule            Hylton
 SF  201  pep-0201.txt  Lockstep Iteration                     Warsaw
 S   202  pep-0202.txt  List Comprehensions                    Peters
 SF  203  pep-0203.txt  Augmented Assignments                  Wouters
 SR  204  pep-0204.txt  Range Literals                         Wouters
 S   205  pep-0205.txt  Weak References                        Drake
 I   206  pep-0206.txt  2.0 Batteries Included                 Zadka
 SF  207  pep-0207.txt  Rich Comparisons                   van Rossum, Ascher
 SF  208  pep-0208.txt  Reworking the Coercion Model       Schemenauer, Lemburg
 S   209  pep-0209.txt  Adding Multidimensional Arrays        Barrett, Oliphant
 SD  210  pep-0210.txt  Decoupling the Interpreter Loop        Ascher
 SD  211  pep-0211.txt  Adding New Linear Algebra Operators    Wilson
 SD  212  pep-0212.txt  Loop Counter Iteration                 Schneider-Kamp
 SD  213  pep-0213.txt  Attribute Access Handlers              Prescod
 SF  214  pep-0214.txt  Extended Print Statement               Warsaw
 SD  215  pep-0215.txt  String Interpolation                   Yee
 I   216  pep-0216.txt  Docstring Format                       Zadka
 SF  217  pep-0217.txt  Display Hook for Interactive Use       Zadka
 SD  218  pep-0218.txt  Adding a Built-In Set Object Type      Wilson
 SD  219  pep-0219.txt  Stackless Python                       McMillan
 I   220  pep-0220.txt  Coroutines, Generators, Continuations  McMillan
 SF  221  pep-0221.txt  Import As                              Wouters
 S   222  pep-0222.txt  Web Library Enhancements               Kuchling
 SF  223  pep-0223.txt  Change the Meaning of \x Escapes       Peters
 SD  224  pep-0224.txt  Attribute Docstrings                   Lemburg
 SD  225  pep-0225.txt  Elementwise/Objectwise Operators       Zhu, Lielens
 I   226  pep-0226.txt  Python 2.1 Release Schedule            Hylton
 S   227  pep-0227.txt  Statically Nested Scopes               Hylton
 S   228  pep-0228.txt  Reworking Python's Numeric Model       Zadka
 S   229  pep-0229.txt  Using Distutils to Build Python        Kuchling
 S   230  pep-0230.txt  Warning Framework                      van Rossum
 SR  231  pep-0231.txt  __findattr__()                         Warsaw
 SF  232  pep-0232.txt  Function Attributes                    Warsaw
 S   233  pep-0233.txt  Python Online Help                     Prescod
 S   234  pep-0234.txt  Iterators                              Yee, van Rossum
 SF  235  pep-0235.txt  Import on Case-Insensitive Platforms   Peters
 S   236  pep-0236.txt  Back to the __future__                 Peters
 S   237  pep-0237.txt  Unifying Long Integers and Integers    Zadka
 S   238  pep-0238.txt  Non-integer Division                   Zadka
 S   239  pep-0239.txt  Adding a Rational Type to Python       Zadka
 S   240  pep-0240.txt  Adding a Rational Literal to Python    Zadka
 S   241  pep-0241.txt  Metadata for Python Software Packages  Kuchling
 S   242  pep-0242.txt  Numeric Kinds                          Dubois
 S   243  pep-0243.txt  Module Repository Upload Mechanism     Reifschneider
 S   244  pep-0244.txt  The `directive' Statement              von Loewis
 S   245  pep-0245.txt  Python Interface Syntax                Pelletier
 S   246  pep-0246.txt  Object Adaptation                      Evans
 I   247  pep-0247.txt  API for Cryptographic Hash Functions   Kuchling
 I   248  pep-0248.txt  Python Database API Specification v1.0 Lemburg
 I   249  pep-0249.txt  Python Database API Specification v2.0 Lemburg
 S   250  pep-0250.txt  Using site-packages on All Platforms   Moore
 I   251  pep-0251.txt  Python 2.2 Release Schedule            Warsaw
 S   252  pep-0252.txt  Making Types Look More Like Classes    van Rossum
 S   253  pep-0253.txt  Subtyping Built-in Types               van Rossum

Key

    I - Informational PEP
    S - Standards Track PEP
    A - Accepted proposal
    R - Rejected proposal
    D - Deferred proposal
    F - Final proposal

Owners

    name                     email address
    ----------------         -------------
    Aahz                     aahz@pobox.com
    Ascher, David            davida@activestate.com
    Barrett, Paul            barrett@stsci.edu
    Drake, Fred              fdrake@acm.org
    Dubois, Paul F.          paul@pfdubois.com
    Evans, Clark C.          cce@clarkevans.com
    Hylton, Jeremy           jeremy@digicool.com
    Kuchling, Andrew         akuchlin@mems-exchange.org
    Lemburg, Marc-Andre      mal@lemburg.com
    Lielens, Gregory         gregory.lielens@fft.be
    von Loewis, Martin       loewis@informatik.hu-berlin.de
    McMillan, Gordon         gmcm@hypernet.com
    Moore, Paul              gustav@morpheus.demon.co.uk
    Oliphant, Travis         oliphant@ee.byu.edu
    Pelletier, Michel        michel@digicool.com
    Peters, Tim              tim@digicool.com
    Prescod, Paul            paul@prescod.net
    Raymond, Eric            esr@snark.thyrsus.com
    Reifschneider, Sean      jafo-pep@tummy.com
    van Rossum, Guido        guido@python.org
    Schemenauer, Neil        nas@arctrix.com
    Schneider-Kamp, Peter    nowonder@nowonder.de
    Warsaw, Barry            barry@digicool.com
    Wilson, Greg             gvwilson@nevex.com
    Wouters, Thomas          thomas@xs4all.net
    Yee, Ka-Ping             ping@lfw.org
    Zadka, Moshe             moshez@zadka.site.co.il
    Zhu, Huaiyu              hzhu@users.sourceforge.net


PEP 1 -- PEP Purpose and Guidelines

[index]
PEP: | 1
Title: | PEP Purpose and Guidelines
Version: | $Revision: 1.19 $
Author: | barry@digicool.com (Barry A. Warsaw), jeremy@digicool.com (Jeremy Hylton)
Status: | Active
Type: | Informational
Created: | 13-Jun-2000
Post-History: | 21-Mar-2001

What is a PEP?

    PEP stands for Python Enhancement Proposal.  A PEP is a design
    document providing information to the Python community, or
    describing a new feature for Python.  The PEP should provide a
    concise technical specification of the feature and a rationale for
    the feature.

    We intend PEPs to be the primary mechanisms for proposing new
    features, for collecting community input on an issue, and for
    documenting the design decisions that have gone into Python.  The
    PEP author is responsible for building consensus within the
    community and documenting dissenting opinions.

    Because the PEPs are maintained as plain text files under CVS
    control, their revision history is the historical record of the
    feature proposal[1].

Kinds of PEPs

    There are two kinds of PEPs.  A standards track PEP describes a
    new feature or implementation for Python.  An informational PEP
    describes a Python design issue, or provides general guidelines or
    information to the Python community, but does not propose a new
    feature.

PEP Work Flow

    The PEP editor, Barry Warsaw <barry@digicool.com>, assigns numbers
    for each PEP and changes its status.

    The PEP process begins with a new idea for Python.  Each PEP must
    have a champion -- someone who writes the PEP using the style and
    format described below, shepherds the discussions in the
    appropriate forums, and attempts to build community consensus
    around the idea.  The PEP champion (a.k.a. Author) should first
    attempt to ascertain whether the idea is PEP-able.  Small
    enhancements or patches often don't need a PEP and can be injected
    into the Python development work flow with a patch submission to
    the SourceForge patch manager[2] or feature request tracker[3].

    The PEP champion then emails the PEP editor with a proposed title
    and a rough, but fleshed out, draft of the PEP.  This draft must
    be written in PEP style as described below.

    If the PEP editor approves, he will assign the PEP a number, label
    it as standards track or informational, give it status 'draft',
    and create and check-in the initial draft of the PEP.  The PEP
    editor will not unreasonably deny a PEP.  Reasons for denying PEP
    status include duplication of effort, being technically unsound,
    or not in keeping with the Python philosophy.  The BDFL
    (Benevolent Dictator for Life, Guido van Rossum
    <guido@python.org>) can be consulted during the approval phase,
    and is the final arbitrator of the draft's PEP-ability.

    The author of the PEP is then responsible for posting the PEP to
    the community forums, and marshaling community support for it.  As
    updates are necessary, the PEP author can check in new versions if
    they have CVS commit permissions, or can email new PEP versions to
    the PEP editor for committing.

    Standards track PEPs consists of two parts, a design document and
    a reference implementation.  The PEP should be reviewed and
    accepted before a reference implementation is begun, unless a
    reference implementation will aid people in studying the PEP.
    Standards Track PEPs must include an implementation - in the form
    of code, patch, or URL to same - before it can be considered
    Final.

    PEP authors are responsible for collecting community feedback on a
    PEP before submitting it for review.  A PEP that has not been
    discussed on python-list@python.org and/or python-dev@python.org
    will not be accepted.  However, wherever possible, long open-ended
    discussions on public mailing lists should be avoided.  A better
    strategy is to encourage public feedback directly to the PEP
    author, who collects and integrates the comments back into the
    PEP.

    Once the authors have completed a PEP, they must inform the PEP
    editor that it is ready for review.  PEPs are reviewed by the BDFL
    and his chosen consultants, who may accept or reject a PEP or send
    it back to the author(s) for revision.

    Once a PEP has been accepted, the reference implementation must be
    completed.  When the reference implementation is complete and
    accepted by the BDFL, the status will be changed to `Final.'

    A PEP can also be assigned status `Deferred.'  The PEP author or
    editor can assign the PEP this status when no progress is being
    made on the PEP.  Once a PEP is deferred, the PEP editor can
    re-assign it to draft status.

    A PEP can also be `Rejected'.  Perhaps after all is said and done
    it was not a good idea.  It is still important to have a record of
    this fact.

    PEPs can also be replaced by a different PEP, rendering the
    original obsolete.  This is intended for Informational PEPs, where
    version 2 of an API can replace version 1.

    PEP work flow is as follows:

        Draft -> Accepted -> Final -> Replaced
          ^
          +----> Rejected
          v
        Deferred

    Some informational PEPs may also have a status of `Active' if they
    are never meant to be completed.  E.g. PEP 1.

What belongs in a successful PEP?

    Each PEP should have the following parts:

    1. Preamble -- RFC822 style headers containing meta-data about the
       PEP, including the PEP number, a short descriptive title
       (limited to a maximum of 38 characters), the names contact info
       for each author, etc.

    2. Abstract -- a short (~200 word) description of the technical
       issue being addressed.

    3. Copyright/public domain -- Each PEP must either be explicitly
       labelled in the public domain or the Open Publication
       License[4].

    4. Specification -- The technical specification should describe
       the syntax and semantics of any new language feature.  The
       specification should be detailed enough to allow competing,
       interoperable implementations for any of the current Python
       platforms (CPython, JPython, Python .NET).

    5. Rationale -- The rationale fleshes out the specification by
       describing what motivated the design and why particular design
       decisions were made.  It should describe alternate designs that
       were considered and related work, e.g. how the feature is
       supported in other languages.

       The rationale should provide evidence of consensus within the
       community and discuss important objections or concerns raised
       during discussion.

    6. Reference Implementation -- The reference implementation must
       be completed before any PEP is given status 'Final,' but it
       need not be completed before the PEP is accepted.  It is better
       to finish the specification and rationale first and reach
       consensus on it before writing code.

       The final implementation must include test code and
       documentation appropriate for either the Python language
       reference or the standard library reference.

PEP Style

    PEPs are written in plain ASCII text, and should adhere to a
    rigid style.  There is a Python script that parses this style and
    converts the plain text PEP to HTML for viewing on the web[5].

    Each PEP must begin with an RFC822 style header preamble.  The
    headers must appear in the following order.  Headers marked with
    `*' are optional and are described below.  All other headers are
    required.

        PEP: <pep number>
        Title: <pep title>
        Version: <cvs version string>
        Author: <list of authors' email and real name>
      * Discussions-To: <email address>
        Status: <Draft | Active | Accepted | Deferred | Final | Replaced>
        Type: <Informational | Standards Track>
      * Requires: <pep numbers>
        Created: <date created on, in dd-mmm-yyyy format>
      * Python-Version: <version number>
        Post-History: <dates of postings to python-list and python-dev>
      * Replaces: <pep number>
      * Replaced-By: <pep number>

    Standards track PEPs must have a Python-Version: header which
    indicates the version of Python that the feature will be released
    with.  Informational PEPs do not need a Python-Version: header.

    While a PEP is in private discussions (usually during the initial
    Draft phase), a Discussions-To: header will indicate the mailing
    list or URL where the PEP is being discussed.  No Discussions-To:
    header is necessary if the PEP is being discussed privately with
    the author, or on the python-list or python-dev email mailing
    lists.

    PEPs may have a Requires: header, indicating the PEP numbers that
    this PEP depends on.

    PEPs may also have a Replaced-By: header indicating that a PEP has
    been rendered obsolete by a later document; the value is the
    number of the PEP that replaces the current document.  The newer
    PEP must have a Replaces: header containing the number of the PEP
    that it rendered obsolete.

    PEP headings must begin in column zero and the initial letter of
    each word must be capitalized as in book titles.  Acronyms should
    be in all capitals.  The body of each section must be indented 4
    spaces.  Code samples inside body sections should be indented a
    further 4 spaces, and other indentation can be used as required to
    make the text readable.  You must use two blank lines between the
    last line of a section's body and the next section heading.

    Tab characters must never appear in the document at all.  A PEP
    should include the Emacs stanza included by example in this PEP.

    A PEP must contain a Copyright section, and it is strongly
    recommended to put the PEP in the public domain.

    You should footnote any URLs in the body of the PEP, and a PEP
    should include a References section with those URLs expanded.

References and Footnotes

    [1] This historical record is available by the normal CVS commands
    for retrieving older revisions.  For those without direct access
    to the CVS tree, you can browse the current and past PEP revisions
    via the SourceForge web site at

    http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=python

    [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470

    [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse

    [4] http://www.opencontent.org/openpub/

    [5] The script referred to here is pep2html.py, which lives in
    the same directory in the CVS tree as the PEPs themselves.  Try
    "pep2html.py --help" for details.

    The URL for viewing PEPs on the web is
    http://python.sourceforge.net/peps/

Copyright

    This document has been placed in the public domain.


PEP 2 -- Procedure for Adding New Modules

PEP: | 2
Title: | Procedure for Adding New Modules
Version: | $Revision: 1.2 $
Author: | esr@snark.thyrsus.com (Eric S. Raymond)
Status: | Deferred
Type: | Informational
Created: | 07-Aug-2000
Post-History: |

Abstract

    This PEP will describes review and voting procedures for
    incorporating candidate modules and extensions into the Python
    core.


PEP 3 -- Guidelines for Handling Bug Reports

PEP: | 3
Title: | Guidelines for Handling Bug Reports
Version: | $Revision: 1.2 $
Author: | jeremy@alum.mit.edu (Jeremy Hylton)
Status: | Active
Type: | Informational
Created: | 25-Sep-2000
Post-History: |

Introduction

    This PEP contains guidelines for handling bug reports to the
    Python project on SourceForge[1].  Still to be done is to collect
    a list of people willing to handle bug reports and their areas of
    expertise.

Guidelines

    1. Make sure the bug category and bug group are correct.  If they
       are correct, it is easier for someone interested in helping to
       find out, say, what all the open Tkinter bugs are.

    2. If it's a minor feature request that you don't plan to address
       right away, add it to PEP 42[2] or ask the owner to add it for
       you.  If you add the bug to PEP 42, mark the bug as "feature
       request", "later", and "closed"; and add a comment to the bug
       saying that this is the case (mentioning the PEP explicitly).

    3. Assign the bug a reasonable priority.  We don't yet have a
       clear sense of what each priority should mean, except than 9 is
       highest and 1 is lowest.  One rule, however, is that bugs with
       priority seven or higher must be fixed before the next release.

    4. If a bug report doesn't have enough information to allow you to
       reproduce or diagnose it, send email to the original submittor
       and ask for more information.  If the original report is really
       thin and your email doesn't get a response after a reasonable
       waiting period, you can close the bug.

    5. If you fix a bug, mark the status as "Fixed" and close it.  In
       the comments, including the CVS revision numbers of the
       affected files.  In the CVS checkin message, include the
       SourceForge bug number *and* a normal description of the
       change.

    6. If you are assigned a bug that you are unable to deal with
       assign it to someone else.  The guys at PythonLabs get paid to
       fix these bugs, so pick one of them if there is no other
       obvious candidate.

References

    [1] http://sourceforge.net/projects/python
    [2] pep-0042.txt


PEP 4 -- Deprecation of Standard Modules

PEP: | 4
Title: | Deprecation of Standard Modules
Version: | $Revision: 1.1 $
Author: | Martin v. Loewis
Status: | Active
Type: | Informational
Created: | 1-Oct-2000

Introduction

    When new modules where added to the standard Python library in the
    past, it was not possible to foresee whether they would still be
    useful in the future.  Even though Python "Comes With Batteries
    Included", batteries may discharge over time.  Carrying old
    modules around is a burden on the maintainer, especially when
    there is no interest in the module anymore.

    At the same time, removing a module from the distribution is
    difficult, as it is not known in general whether anybody is still
    using it.  This PEP defines a procedure for removing modules from
    the standard Python library.  Usage of a module may be
    `deprecated', which means that it may be removed from a future
    Python release.  The rationale for deprecating a module is also
    collected in this PEP.  If the rationale turns out faulty, the
    module may become `undeprecated'.

Procedure for declaring a module deprecated

    Since the status of module deprecation is recorded in this PEP,
    proposals for deprecating modules MUST be made by providing a
    change to the text of this PEP, which SHOULD be a patch posted to
    SourceForge, or sent by mail to the author of this PEP.

    A proposal for deprecation of the module MUST include the date of
    the proposed deprecation and a rationale for deprecating it.  In
    addition, the proposal MUST include a change to the documentation
    of the module; deprecation is indicated by saying that the module
    is "obsolete" or "deprecated".  The proposal MAY include a patch
    for the module's source code to indicate deprecation there as
    well.

    It is expected that deprecated modules are included in the Python
    releases that immediately follows the deprecation; later releases
    may ship without the deprecated modules.

Procedure for declaring a module undeprecated

    When a module becomes deprecated, a rationale is given for its
    deprecation.  In some cases, an alternative interface for the same
    functionality is provided, so the old interface is deprecated.  In
    other cases, the need for having the functionality of the module
    may not exist anymore.

    If the rationale is faulty, again a change to this PEP's text MUST
    be submitted.  This change MUST include the date of undeprecation
    and a rationale for undeprecation.  Modules that are undeprecated
    under this procedure MUST be listed in this PEP for at least one
    major release of Python.

Obsolete modules

    A number of modules are already listed as obsolete in the library
    documentation.  These are listed here for completeness.

    addpack, cmp, cmpcache, codehack, dircmp, dump, fmt, lockfile,
    newdir, Para, poly, regex, regsub, tb, timing, util, whatsound,
    tzmod, find, grep, packmail, ni, rand, soundex, cl, sv

    All these modules have been declared as obsolete in Python 2.0,
    some even earlier.

Deprecated modules

    Module name:   posixfile
    Rationale:     Locking is better done by fcntl.lockf().
    Date:          Before 1-Oct-2000.
    Documentation: Already documented as obsolete.

    Module name:   gopherlib
    Rationale:     The gopher protocol is not in active use anymore.
    Date:          1-Oct-2000.
    Documentation: TBD

    Module name:   audioop
    Rationale:     The module is not actively being used, no mentioning
                   of it was found on deja.com on Oct-1-2000.
    Date:          1-Oct-2000
    Documentation: TBD

    Module name:   rgbimgmodule
    Rationale:     The module is not actively being used, no mentioning
                   of it was found on deja.com on Oct-1-2000.
    Date:          1-Oct-2000
    Documentation: TBD

Undeprecated modules

    None.


PEP 5 -- Guidelines for Language Evolution

PEP: | 5
Title: | Guidelines for Language Evolution
Version: | $Revision: 1.1 $
Author: | paul@prescod.net (Paul Prescod)
Status: | Active
Type: | Informational
Created: | 26-Oct-2000

Abstract

    In the natural evolution of programming languages it is sometimes
    necessary to make changes that modify the behavior of older
    programs.  This PEP proposes a policy for implementing these
    changes in a manner respectful of the installed base of Python
    users.

Implementation Details

    Implementation of this PEP requires the addition of a formal
    warning and deprecation facility that will be described in another
    proposal.

Scope

    These guidelines apply to future versions of Python that introduce
    backward-incompatible behavior.  Backward incompatible behavior is
    a major deviation in Python interpretation from an earlier
    behavior described in the standard Python documentation.  Removal
    of a feature also constitutes a change of behavior.

    This PEP does not replace or preclude other compatibility
    strategies such as dynamic loading of backwards-compatible
    parsers.  On the other hand, if execution of "old code" requires a
    special switch or pragma then that is indeed a change of behavior
    from the point of view of the user and that change should be
    implemented according to these guidelines.

    In general, common sense must prevail in the implementation of
    these guidelines.  For instance changing "sys.copyright" does not
    constitute a backwards-incompatible change of behavior!

Steps For Introducing Backwards-Incompatible Features

    1. Propose backwards-incompatible behavior in a PEP.  The PEP must
       include a section on backwards compatibility that describes in
       detail a plan to complete the remainder of these steps.

    2. Once the PEP is accepted as a productive direction, implement
       an alternate way to accomplish the task previously provided by
       the feature that is being removed or changed.  For instance if
       the addition operator were scheduled for removal, a new version
       of Python could implement an "add()" built-in function.

    3. Formally deprecate the obsolete construct in the Python
       documentation.

    4. Add an an optional warning mode to the parser that will inform
       users when the deprecated construct is used.  In other words,
       all programs that will behave differently in the future must
       trigger warnings in this mode.  Compile-time warnings are
       preferable to runtime warnings.  The warning messages should
       steer people from the deprecated construct to the alternative
       construct.

    5. There must be at least a one-year transition period between the
       release of the transitional version of Python and the release
       of the backwards incompatible version.  Users will have at
       least a year to test their programs and migrate them from use
       of the deprecated construct to the alternative one.


PEP 6 -- Bug Fix Releases

PEP: | 6
Title: | Bug Fix Releases
Version: | $Revision: 1.4 $
Author: | aahz@pobox.com (Aahz)
Status: | Draft
Type: | Informational
Created: | 15-Mar-2001
Post-History: | 15-Mar-2001

Abstract

    Python has historically had only a single fork of development,
    with releases having the combined purpose of adding new features
    and delivering bug fixes (these kinds of releases will be referred
    to as "feature releases").  This PEP describes how to fork off
    patch releases of old versions for the primary purpose of fixing
    bugs.

    This PEP is not, repeat NOT, a guarantee of the existence of patch
    releases; it only specifies a procedure to be followed if patch
    releases are desired by enough of the Python community willing to
    do the work.

Motivation

    With the move to SourceForge, Python development has accelerated.
    There is a sentiment among part of the community that there was
    too much acceleration, and many people are uncomfortable with
    upgrading to new versions to get bug fixes when so many features
    have been added, sometimes late in the development cycle.

    One solution for this issue is to maintain the previous feature
    release, providing bug fixes until the next feature release.  This
    should make Python more attractive for enterprise development,
    where Python may need to be installed on hundreds or thousands of
    machines.

Prohibitions

    Patch releases are required to adhere to the following restrictions:

    1. There must be zero syntax changes.  All .pyc and .pyo files
       must work (no regeneration needed) with all patch releases
       forked off from a feature release.

    2. There must be zero pickle changes.

    3. There must be no incompatible C API changes.  All extensions
       must continue to work without recompiling in all patch releases
       in the same fork as a feature release.

    Breaking any of these prohibitions requires a BDFL proclamation
    (and a prominent warning in the release notes).

Version Numbers

    Starting with Python 2.0, all feature releases are required to
    have a version number the form X.Y; patch releases will always be
    of the form X.Y.Z.

    The current feature release under development is referred to as
    release N; the just-released feature version is referred to as
    N-1.

Procedure

    The process for managing patch releases is modeled in part on the
    Tcl system [1].

    The Patch Czar is the counterpart to the BDFL for patch releases.
    However, the BDFL and designated appointees retain veto power over
    individual patches.

    As individual patches get contributed to the feature release fork,
    each patch contributor is requested to consider whether the patch is
    a bug fix suitable for inclusion in a patch release.  If the patch is
    considered suitable, the patch contributor will mail the SourceForge
    patch (bug fix?) number to the maintainers' mailing list.

    In addition, anyone from the Python community is free to suggest
    patches for inclusion.  Patches may be submitted specifically for
    patch releases; they should follow the guidelines in PEP 3 [2].

    The Patch Czar decides when there are a sufficient number of
    patches to warrant a release.  The release gets packaged up,
    including a Windows installer, and made public.  If any new bugs
    are found, they must be fixed immediately and a new patch release
    publicized (with an incremented version number).

    Patch releases are expected to occur at an interval of roughly one
    month.  In general, only the N-1 release will be under active
    maintenance at any time.

Patch Czar History

    Thomas Wouters (thomas@xs4all.net) is the Patch Czar for 2.1.1.

    Moshe Zadka (moshez@zadka.site.co.il) is the Patch Czar for 2.0.1.

Issues To Be Resolved

    What is the equivalent of python-dev for people who are
    responsible for maintaining Python?  (Aahz proposes either
    python-patch or python-maint, hosted at either python.org or
    xs4all.net.)

    Does SourceForge make it possible to maintain both separate and
    combined bug lists for multiple forks?  If not, how do we mark
    bugs fixed in different forks?  (Simplest is to simply generate a
    new bug for each fork that it gets fixed in, referring back to the
    main bug number for details.)

History

    This PEP started life as a proposal on comp.lang.python.  The
    original version suggested a single patch for the N-1 release to
    be released concurrently with the N release.  The original version
    also argued for sticking with a strict bug fix policy.

    Following feedback from the BDFL and others, the draft PEP was
    written containing an expanded patch release cycle that permitted
    any previous feature release to obtain patches and also relaxed
    the strict bug fix requirement (mainly due to the example of PEP
    235 [3], which could be argued as either a bug fix or a feature).

    Discussion then mostly moved to python-dev, where BDFL finally
    issued a proclamation basing the Python patch release process on
    Tcl's, which essentially returned to the original proposal in
    terms of being only the N-1 release and only bug fixes, but
    allowing multiple patch releases until release N is published.

References

    [1] http://dev.scriptics.com:8080/cgi-bin/tct/tip/28.html

    [2] http://python.sourceforge.net/peps/pep-0003.html

    [3] http://python.sourceforge.net/peps/pep-0235.html

Copyright

    This document has been placed in the public domain.


PEP 42 -- Feature Requests

PEP: | 42
Title: | Feature Requests
Version: | $Revision: 1.50 $
Author: | Jeremy Hylton
Status: | Active
Type: | Informational
Created: | 12-Sep-2000

Introduction

    This PEP contains a list of feature requests that may be
    considered for future versions of Python.  Large feature requests
    should not be included here, but should be described in separate
    PEPs; however a large feature request that doesn't have its own
    PEP can be listed here until its own PEP is created.  See
    pep-0000.txt for details.

    This PEP was created to allow us to close bug reports that are really
    feature requests.  Marked as Open, they distract from the list of real
    bugs (which should ideally be less than a page).  Marked as Closed, they
    tend to be forgotten.  The procedure now is:  if a bug report is really
    a feature request, add the feature request to this PEP; mark the bug as
    "feature request", "later", and "closed"; and add a comment to the bug
    saying that this is the case (mentioning the PEP explicitly).  It is
    also acceptable to move large feature requests directly from the bugs
    database to a separate PEP.

Core Language / Builtins

    - A builtin function that returns the number of bytes an object
      uses internally.  Apparently mxTools has a sizeof function that
      returns the size of the object struct itself.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110835&group_id=5470

    - Add C API functions to help Windows users who are building
      embedded applications where the FILE * structure does not match
      the FILE * the interpreter was compiled with.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110821&group_id=5470

      See this bug report for a specific suggestion that will allow a
      Borland C++ builder application to interact with a python.dll
      build with MSVC.

    - The parser should handle more deeply nested parse trees.

      The following will fail -- eval("["*50 + "]"*50) -- because the
      parser has a hard-coded limit on stack size.  This limit should
      be raised or removed.  Removal would be hard because the
      current compiler can overflow the C stack if the nesting is too
      deep.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=115555&group_id=5470

    - The buffer interface could be smarter when a buffer object is
      created that depends on another buffer object -- if the
      original buffer object has no base, the depending object will
      have no base either.  It could be argued that the depending
      object should have the original object as a base.  Or not.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=116405&group_id=5470

Standard Library

    - The test suite is incomplete (and probably always will be).
      This is a reminder to people that more regression tests are
      needed.

      http://sourceforge.net/bugs/?func=detailbug&group_id=5470&bug_id=110819

    - The urllib module should support proxies which require
      authenication.  See SourceForge bug #110619 for information:

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110619&group_id=5470

    - Use objects with attributes in place of tuples of values for
      return values in several places in the library.  Two specific
      APIs which could use this treatment include os.stat() and
      os.statvfs(); see SourceForge bug #111481:

      http://sourceforge.net/bugs/?func=detailbug&bug_id=111481&group_id=5470

      which shows very specifically why this approach is better than
      the tuple approach: additional attributes can be added as needed
      without having as detrimental an affect on existing code.

    - os.rename() should be modified to handle EXDEV errors on
      platforms that don't allow rename() to operate across filesystem
      boundaries by copying the file over and removing the original.
      Linux is one system that requires this treatment.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=112317&group_id=5470

    - signal handling doesn't always work as expected.  E.g. if
      sys.stdin.readline() is interrupted by a (returning) signal
      handler, it returns "".  It would be better to make it raise an
      exception (corresponding to EINTR) or to restart.  But these
      changes would have to applied to all places that can do blocking
      interruptable I/O.  So it's a big project.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110599&group_id=5470

    - Ensure that all .py files in the std library use 4-space indents and
      no hard tabs.  This was actually a PEP200 precondition for the
      release of 2.0b1, but got misinterpreted as the weaker condition that
      tabnanny not complain.  Tim Peters will do this now, but, since about
      250 files are affected, will wait until after 2.0final is released.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=114557&group_id=5470

    - Port the Python SSL code to Windows.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110683&group_id=5470

    - Extend Windows utime to accept directory paths.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=114245&group_id=5470

    - Extend copy.py to class, module & function types.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=114553&group_id=5470

    - Better checking for bad input to marshal.load*().

      http://sourceforge.net/bugs/?func=detailbug&bug_id=114754&group_id=5470

    - Generalize eval to accept any mapping objects for locals and globals.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=115126&group_id=5470

    - Add a portable implementation of time.strptime() that works in
      clearly defined ways on all platforms.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=115146&group_id=5470
      http://sourceforge.net/bugs/?func=detailbug&bug_id=112244&group_id=5470

    - rfc822.py should be more lenient than the spec in the types of
      address fields it parses.  Specifically, an invalid address of
      the form "From: Amazon.com <delivers-news2@amazon.com>" should
      be parsed correctly.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110678&group_id=5470

    - cgi.py's FieldStorage class should be more conservative with
      memory in the face of large binary file uploads.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110674&group_id=5470

      There are two issues here: first, because
      read_lines_to_outerboundary() uses readline() it is possible
      that a large amount of data will be read into memory for a
      binary file upload.  This should probably look at the
      Content-Type header of the section and do a chunked read if it's
      a binary type.

      The second issue was related to the self.lines attribute, which
      was removed in revision 1.56 of cgi.py (see also):

      http://sourceforge.net/bugs/?func=detailbug&bug_id=119806&group_id=5470

    - urllib should support proxy definitions that contain just the
      host and port

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110849&group_id=5470

    - urlparse should be updated to comply with RFC 2396, which
      defines optional parameters for each segment of the page.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110834&group_id=5470

    - The exceptions raised by pickle and cPickle are currently
      different; these should be unified (probably the exceptions
      should be defined in a helper module that's imported by both).
      [No bug report; I just thought of this.]

    - More standard library routines should support Unicode.  For
      example, urllib.quote() could convert Unicode strings to UTF-8
      and then do the usual %HH conversion.  But this is not the only
      one!

      http://sourceforge.net/bugs/?func=detailbug&bug_id=116716&group_id=5470

    - There should be a way to say that you don't mind if str() or
      __str__() return a Unicode string object.  Or a different
      function -- ustr() has been proposed.  Or something...

      http://sourceforge.net/patch/?func=detailpatch&patch_id=101527&group_id=5470

    - A UTF-7 codec.  It's not a great encoding, but it's required in
      some places (e.g. IMAP mailbox names with Asian language
      support).  [No bug report; private email from Seo Junwon
      <linuxqna@chollian.net>.]

    - Killing a thread from another thread.  Or maybe sending a
      signal.  Or maybe raising an asynchronous exception.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=121115&group_id=5470

    - The debugger (pdb) should understand packages.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110631&group_id=5470

    - The cmath library should be rewritten in Python.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110838&group_id=5470

    - Could use a more comprehensive email module.  (But then again,
      that may be better done as a 3rd party project.)

      http://sourceforge.net/bugs/?func=detailbug&bug_id=121208&group_id=5470

    - every built-in function or method (including all core
      extensions) that accepts a string, dict, or list, should also
      accept a UserString, UserDict, or UserList.  (The latter two
      should more generally accept all mappings, all sequences.)

      http://sourceforge.net/bugs/?func=detailbug&bug_id=132493&group_id=5470

Tools

    - IDLE should reload & recompile modules changed externally.  To
      be done properly, scripts will have to be run in a separate
      process.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110841&group_id=5470

    - Python could use a GUI builder.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110820&group_id=5470

    - IDLE's key bindings should be revised.  Some of the advertised
      bindings don't even work!

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110659&group_id=5470

    - IDLE has deep problems running threaded programs.  Re-architect.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=121963&group_id=5470

    - Would be nice if IDLE showed which structure was being closed upon
      typing one of "})]".  Heck, a string too, for that matter.
      The bug report also says something about TAB I didn't grok.

      http://sourceforge.net/tracker/?func=detail&aid=404444&group_id=5470&atid=105470

Building and Installing

    - You should be able to configure and build Python with a
      cross-compiler.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=110836&group_id=5470

    - Modules/makesetup should make sure the 'config.c' file it
      generates from the various Setup files, is valid C. It currently
      accepts module names with characters that are not allowable in
      Python or C identifiers.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=116326&group_id=5470

    - We should rename and rationalize the API getopt.c provides, and
      always use this, instead of trying to rely on the platform
      getopt and fighting with its prototype.

      http://www.python.org/pipermail/python-dev/2000-October/016620.html

    - There's a name conflict on case insensitive filesystems (in
      particular Mac OSX) between the directory "Python" and the key
      build target "python".  That's currently solved by abusing the
      --with-suffix option, but that's not ideal (since that also
      causes the binary to be *installed* as python.exe).  What should
      be the solution?

      http://sourceforge.net/bugs/?func=detailbug&bug_id=122215&group_id=5470

    - Building from source should not attempt to overwrite the
      Include/graminit.h and Parser/graminit.c files, at least for
      people downloading a source release rather than working from CVS
      or snapshots.  Some people find this a problem in unusual build
      environments.

      http://sourceforge.net/bugs/?func=detailbug&bug_id=119221&group_id=5470


PEP 160 -- Python 1.6 Release Schedule

PEP: | 160
Title: | Python 1.6 Release Schedule
Version: | $Revision: 1.9 $
Author: | Fred L. Drake, Jr.
Status: | Finished
Type: | Informational
Created: | 25-Jul-2000
Python-Version: | 1.6
Post-History: |

Introduction

    This PEP describes the Python 1.6 release schedule.  The CVS
    revision history of this file contains the definitive historical
    record.

    This release will be produced by BeOpen PythonLabs staff for the
    Corporation for National Research Initiatives (CNRI).

Schedule

    August 1     1.6 beta 1 release (planned).
    August 3     1.6 beta 1 release (actual).
    August 15    1.6 final release (planned).
    September 5  1.6 final release (actual).

Features

    A number of features are required for Python 1.6 in order to
    fulfill the various promises that have been made.  The following
    are required to be fully operational, documented, and forward
    compatible with the plans for Python 2.0:

    * Unicode support: The Unicode object defined for Python 2.0 must
      be provided, including all methods and codec support.

    * SRE: Fredrik Lundh's new regular expression engine will be used
      to provide support for both 8-bit strings and Unicode strings.
      It must pass the regression test used for the pcre-based version
      of the re module.

    * The curses module was in the middle of a transformation to a
      package, so the final form was adopted.

Mechanism

    The release will be created as a branch from the development tree
    rooted at CNRI's close of business on 16 May 2000.  Patches
    required from more recent checkins will be merged in by moving the
    branch tag on individual files whenever possible in order to
    reduce mailing list clutter and avoid divergent and incompatible
    implementations.

    The branch tag is "cnri-16-start".

    Patches and features will be merged to the extent required to pass
    regression tests in effect on 16 May 2000.

    The beta release is tagged "r16b1" in the CVS repository, and the
    final Python 1.6 release is tagged "release16" in the repository.

Copyright

    This document has been placed in the public domain.


PEP 200 -- Python 2.0 Release Schedule

PEP: | 200
Title: | Python 2.0 Release Schedule
Version: | $Revision: 1.47 $
Author: | Jeremy Hylton
Python-Version: | 2.0
Status: | Final

Introduction

    This PEP describes the Python 2.0 release schedule, tracking the
    status and ownership of the major new features, summarizes
    discussions held in mailing list forums, and provides URLs for
    further information, patches, and other outstanding issues.  The
    CVS revision history of this file contains the definitive
    historical record.

Release Schedule

    [revised 5 Oct 2000]

    26-Sep-2000: 2.0 beta 2
     9-Oct-2000: 2.0 release candidate 1 (2.0c1)
    16-Oct-2000: 2.0 final

Previous milestones

    14-Aug-2000: All 2.0 PEPs finished / feature freeze
     5-Sep-2000: 2.0 beta 1

What is release candidate 1?

    We believe that release candidate 1 will fix all known bugs that
    we intend to fix for the 2.0 final release.  This release should
    be a bit more stable than the previous betas.  We would like to
    see even more widespread testing before the final release, so we
    are producing this release candidate.  The final release will be
    exactly the same unless any show-stopping (or brown bag) bugs are
    found by testers of the release candidate.

Guidelines for submitting patches and making changes

    Use good sense when committing changes.  You should know what we
    mean by good sense or we wouldn't have given you commit privileges
    <0.5 wink>.  Some specific examples of good sense include:

    - Do whatever the dictator tells you.

    - Discuss any controversial changes on python-dev first.  If you
      get a lot of +1 votes and no -1 votes, make the change.  If you
      get a some -1 votes, think twice; consider asking Guido what he
      thinks.

    - If the change is to code you contributed, it probably makes
      sense for you to fix it.

    - If the change affects code someone else wrote, it probably makes
      sense to ask him or her first.

    - You can use the SF Patch Manager to submit a patch and assign it
      to someone for review.

    Any significant new feature must be described in a PEP and
    approved before it is checked in.

    Any significant code addition, such as a new module or large
    patch, must include test cases for the regression test and
    documentation.  A patch should not be checked in until the tests
    and documentation are ready.

    If you fix a bug, you should write a test case that would have
    caught the bug.

    If you commit a patch from the SF Patch Manager or fix a bug from
    the Jitterbug database, be sure to reference the patch/bug number
    in the CVS log message.  Also be sure to change the status in the
    patch manager or bug database (if you have access to the bug
    database).

    It is not acceptable for any checked in code to cause the
    regression test to fail.  If a checkin causes a failure, it must
    be fixed within 24 hours or it will be backed out.

    All contributed C code must be ANSI C.  If possible check it with
    two different compilers, e.g. gcc and MSVC.

    All contributed Python code must follow Guido's Python style
    guide.  http://www.python.org/doc/essays/styleguide.html

    It is understood that any code contributed will be released under
    an Open Source license.  Do not contribute code if it can't be
    released this way.

Failing test cases need to get fixed

    We need to resolve errors in the regression test suite quickly.
    Changes should not be committed to the CVS tree unless the
    regression test runs cleanly with the changes applied.  If it
    fails, there may be bugs lurking in the code.  (There may be bugs
    anyway, but that's another matter.)  If the test cases are known
    to fail, they serve no useful purpose.

    test case         platform    date reported
    ---------         --------    -------------
    test_mmap         Win ME      03-Sep-2000       Windows 2b1p2 prelease
        [04-Sep-2000 tim
         reported by Audun S. Runde mailto:audun@mindspring.com
         the mmap constructor fails w/
            WindowsError: [Errno 6] The handle is invalid
         since there are no reports of this failing on other
         flavors of Windows, this looks like to be an ME bug
        ]

Open items -- Need to be resolved before 2.0 final release

    Decide whether cycle-gc should be enabled by default.

    Resolve compatibility issues between core xml package and the
    XML-SIG XML package.

    Update Tools/compiler so that it is compatible with list
    comprehensions, import as, and any other new language features.

    Improve code coverage of test suite.

    Finish writing the PEPs for the features that went out with
    2.0b1(! sad, but realistic -- we'll get better with practice).

    Major effort to whittle the bug database down to size.  I've (tim)
    seen this before: if you can keep all the open bugs fitting on one
    screen, people will generally keep it that way.  But let it
    slobber over a screen for a month, & it just goes to hell (no
    "visible progress" indeed!).

Accepted and in progress

    * Currently none left. [4-Sep-2000 guido]

Open: proposed but not accepted or rejected

    * There are a number of open patches again.  We need to clear
      these out soon.

Previously failing test cases

    If you find a test bouncing between this section and the previous one,
    the code it's testing is in trouble!

    test case         platform    date reported
    ---------         --------    -------------
    test_fork1        Linux       26-Jul-2000
        [28-aug-2000 fixed by cgw; solution is to create copies of
        lock in child process]
        [19-Aug-2000 tim
         Charles Waldman whipped up a patch to give child processes a new
         "global lock":
         http://sourceforge.net/patch/?func=detailpatch&patch_id=101226&group_id=5470
         While this doesn't appear to address the symptoms we *saw*, it
         *does* so far appear to be fixing the failing cases anyway
        ]

    test_parser       all         22-Aug-2000
    test_posixpath    all         22-Aug-2000

    test_popen2       Win32       26-Jul-2000
        [31-Aug-2000 tim
         This died again, but for an entirely different reason:  it uses a
         dict to map file pointers to process handles, and calls a dict
         access function during popen.close().  But .close releases threads,
         which left the internal popen code accessing the dict without a
         valid thread state.  The dict implementation changed so that's no
         longer accepted.  Fixed by creating a temporary thread state in the
         guts of popen's close routine, and grabbing the global lock with
         it for the duration]
        [20-Aug-2000 tim
         changed the popen2.py _test function to use the "more" cmd
         when os.name == "nt".  This makes test_popen2 pass under
         Win98SE.
         HOWEVER, the Win98 "more" invents a leading newline out
         of thin air, and I'm not sure that the other Windows flavors
         of "more" also do that.
         So, somebody please try under other Windows flavors!
        ]
        [still fails 15-Aug-2000 for me, on Win98 - tim
             test test_popen2 crashed -- exceptions.AssertionError :
         The problem is that the test uses "cat", but there is
         no such thing under Windows (unless you install it).
         So it's the test that's broken here, not (necessarily)
         the code.
        ]

    test_winreg        Win32      26-Jul-2000
        [works 15-Aug-2000 for me, on Win98 - tim]

    test_mmap          Win32      26-Jul-2000
        [believe that was fixed by Mark H.]
        [works 15-Aug-2000 for me, on Win98 - tim]

    test_longexp      Win98+?     15-Aug-2000
        [fails in release build,
         passes in release build under verbose mode but doesn't
             look like it should pass,
         passes in debug build,
         passes in debug build under verbose mode and looks like
             it should pass
        ]
        [18-Aug-2000, tim:  can't reproduce, and nobody else
         saw it.  I believe there *is* a subtle bug in
         regrtest.py when using -v, and I'll pursue that,
         but can't provoke anything wrong with test_longexp
         anymore; eyeballing Fred's changes didn't turn up
         a suspect either
         19-Aug-2000, tim: the "subtle bug" in regrtest.py -v is
         actually a feature:  -v masks *some* kinds of failures,
         since it doesn't compare test output with the canned
         output; this is what makes it say "test passed" even
         in some cases where the test fails without -v
        ]

    test_winreg2      Win32       26-Jul-2000
        [20-Aug-2000 tim - the test has been removed from the project]
        [19-Aug-2000 tim
         This test will never work on Win98, because it's looking for
         a part of registry that doesn't exist under W98.
         The module (winreg.py) and this test case will be removed
         before 2.0 for other reasons, though.
        ]
        [still fails 15-Aug-2000 for me, on Win98 - tim
         test test_winreg2 failed -- Writing: 'Test Failed: testHives',
         expected: 'HKEY_PERFORMANCE_DATA\012'
        ]

Open items -- completed/fixed

    [4-Sep-2000 guido: Fredrik finished this on 1-Sep]
    * PyErr_Format - Fredrik Lundh
      Make this function safe from buffer overflows.

    [4-Sep-2000 guido: Fred has added popen2, popen3 on 28-Sep]
    Add popen2 support for Linux -- Fred Drake

    [4-Sep-2000 guido: done on 1-Sep]
    Deal with buffering problem with SocketServer

    [04-Sep-2000 tim:  done; installer runs; w9xpopen not an issue]
    [01-Sep-2000 tim:  make a prerelease availabe]
    Windows ME:  Don't know anything about it.  Will the installer
    even run?  Does it need the w9xpopen hack?

    [04-Sep-2000 tim:  done; tested on several Windows flavors now]
    [01-Sep-2000 tim:  completed but untested except on Win98SE]
    Windows installer:  If HKLM isn't writable, back off to HKCU (so
    Python can be installed on NT & 2000 without admin privileges).

    [01-Sep-200 tim - as Guido said, runtime code in posixmodule.c doesn't
     call this on NT/2000, so no need to avoid installing it everywhere.
     Added code to the installer *to* install it, though.]
    Windows installer:  Install w9xpopen.exe only under Win95/98.

    [23-Aug-2000 jeremy - tim reports "completed recently"]
    Windows:  Look for registry info in HKCU before HKLM - Mark
    Hammond.

    [20-Aug-2000 tim - done]
    Remove winreg.py and test_winreg2.py.  Paul Prescod (the author)
    now wants to make a registry API more like the MS .NET API.  Unclear
    whether that can be done in time for 2.0, but, regardless, if we
    let winreg.py out the door we'll be stuck with it forever, and not
    even Paul wants it anymore.

    [24-Aug-2000 tim+guido - done]
    Win98 Guido:  popen is hanging on Guido, and even freezing the
    whole machine.  Was caused by Norton Antivirus 2000 (6.10.20) on
    Windows 9x.  Resolution: disable virus protection.

Accepted and completed

    * Change meaning of \x escapes - PEP 223 - Fredrik Lundh

    * Add \U1234678 escapes in u"" strings - Fredrik Lundh

    * Support for opcode arguments > 2**16 - Charles Waldman
      SF Patch 100893

    * "import as" - Thomas Wouters
      Extend the 'import' and 'from ... import' mechanism to enable
      importing a symbol as another name. (Without adding a new keyword.)

    * List comprehensions - Skip Montanaro
      Tim Peters still needs to do PEP.

    * Restore old os.path.commonprefix behavior
      Do we have test cases that work on all platforms?

    * Tim O'Malley's cookie module with good license

    * Lockstep iteration ("zip" function) - Barry Warsaw

    * SRE - Fredrik Lundh
      [at least I *think* it's done, as of 15-Aug-2000 - tim]

    * Fix xrange printing behavior - Fred Drake
      Remove the tp_print handler for the xrange type; it produced a
      list display instead of 'xrange(...)'.  The new code produces a
      minimal call to xrange(), enclosed in (... * N) when N != 1.
      This makes the repr() more human readable while making it do
      what reprs are advertised as doing.  It also makes the xrange
      objects obvious when working in the interactive interpreter.

    * Extended print statement - Barry Warsaw
      PEP 214
      http://python.sourceforge.net/peps/pep-0214.html
      SF Patch #100970
      http://sourceforge.net/patch/?func=detailpatch&patch_id=100970&group_id=5470

    * interface to poll system call - Andrew Kuchling
      SF Patch 100852

    * Augmented assignment - Thomas Wouters
      Add += and family, plus Python and C hooks, and API functions.

    * gettext.py module - Barry Warsaw

Postponed

    * Extended slicing on lists - Michael Hudson
      Make lists (and other builtin types) handle extended slices.

    * Compression of Unicode database - Fredrik Lundh
      SF Patch 100899
      At least for 2.0b1.  May be included in 2.0 as a bug fix.

    * Range literals - Thomas Wouters
      SF Patch 100902
      We ended up having a lot of doubt about the proposal.

    * Eliminated SET_LINENO opcode - Vladimir Marangozov
      Small optimization achieved by using the code object's lnotab
      instead of the SET_LINENO instruction.  Uses code rewriting
      technique (that Guido's frowns on) to support debugger, which
      uses SET_LINENO.

      http://starship.python.net/~vlad/lineno/
      for (working at the time) patches

      Discussions on python-dev:

      - http://www.python.org/pipermail/python-dev/2000-April/subject.html
        Subject: "Why do we need Traceback Objects?"

      - http://www.python.org/pipermail/python-dev/1999-August/002252.html

    * test harness for C code - Trent Mick

Rejected

    * 'indexing-for' - Thomas Wouters
      Special syntax to give Python code access to the loop-counter in 'for'
      loops. (Without adding a new keyword.)


PEP 201 -- Lockstep Iteration

PEP: | 201
Title: | Lockstep Iteration
Version: | $Revision: 1.18 $
Author: | barry@digicool.com (Barry A. Warsaw)
Status: | Final
Type: | Standards Track
Python-Version: | 2.0
Created: | 13-Jul-2000
Post-History: | 27-Jul-2000

Introduction

    This PEP describes the `lockstep iteration' proposal.  This PEP
    tracks the status and ownership of this feature, slated for
    introduction in Python 2.0.  It contains a description of the
    feature and outlines changes necessary to support the feature.
    This PEP summarizes discussions held in mailing list forums, and
    provides URLs for further information, where appropriate.  The CVS
    revision history of this file contains the definitive historical
    record.

Motivation

    Standard for-loops in Python iterate over every element in a
    sequence until the sequence is exhausted[1].  However, for-loops
    iterate over only a single sequence, and it is often desirable to
    loop over more than one sequence in a lock-step fashion.  In other
    words, in a way such that nthe i-th iteration through the loop
    returns an object containing the i-th element from each sequence.

    The common idioms used to accomplish this are unintuitive.  This
    PEP proposes a standard way of performing such iterations by
    introducing a new builtin function called `zip'.

    While the primary motivation for zip() comes from lock-step
    iteration, by implementing zip() as a built-in function, it has
    additional utility in contexts other than for-loops.

Lockstep For-Loops

    Lockstep for-loops are non-nested iterations over two or more
    sequences, such that at each pass through the loop, one element
    from each sequence is taken to compose the target.  This behavior
    can already be accomplished in Python through the use of the map()
    built-in function:

    >>> a = (1, 2, 3)
    >>> b = (4, 5, 6)
    >>> for i in map(None, a, b): print i
    ...
    (1, 4)
    (2, 5)
    (3, 6)
    >>> map(None, a, b)
    [(1, 4), (2, 5), (3, 6)]

    The for-loop simply iterates over this list as normal.

    While the map() idiom is a common one in Python, it has several
    disadvantages:

    - It is non-obvious to programmers without a functional
      programming background.

    - The use of the magic `None' first argument is non-obvious.

    - It has arbitrary, often unintended, and inflexible semantics
      when the lists are not of the same length: the shorter sequences
      are padded with `None'.

      >>> c = (4, 5, 6, 7)
      >>> map(None, a, c)
      [(1, 4), (2, 5), (3, 6), (None, 7)]

    For these reasons, several proposals were floated in the Python
    2.0 beta time frame for syntactic support of lockstep for-loops.
    Here are two suggestions:

    for x in seq1, y in seq2:
        # stuff

    for x, y in seq1, seq2:
        # stuff

    Neither of these forms would work, since they both already mean
    something in Python and changing the meanings would break existing
    code.  All other suggestions for new syntax suffered the same
    problem, or were in conflict with other another proposed feature
    called `list comprehensions' (see pep-0202.txt).

The Proposed Solution

    The proposed solution is to introduce a new built-in sequence
    generator function, available in the __builtin__ module.  This
    function is to be called `zip' and has the following signature:

    zip(seqa, [seqb, [...]])

    zip() takes one or more sequences and weaves their elements
    together, just as map(None, ...) does with sequences of equal
    length.  The weaving stops when the shortest sequence is
    exhausted.

Return Value

    zip() returns a real Python list, the same way map() does.

Examples

    Here are some examples, based on the reference implementation
    below.

    >>> a = (1, 2, 3, 4)
    >>> b = (5, 6, 7, 8)
    >>> c = (9, 10, 11)
    >>> d = (12, 13)

    >>> zip(a, b)
    [(1, 5), (2, 6), (3, 7), (4, 8)]

    >>> zip(a, d)
    [(1, 12), (2, 13)]

    >>> zip(a, b, c, d)
    [(1, 5, 9, 12), (2, 6, 10, 13)]

    Note that when the sequences are of the same length, zip() is
    reversible:

    >>> a = (1, 2, 3)
    >>> b = (4, 5, 6)
    >>> x = zip(a, b)
    >>> y = zip(*x) # alternatively, apply(zip, x)
    >>> z = zip(*y) # alternatively, apply(zip, y)
    >>> x
    [(1, 4), (2, 5), (3, 6)]
    >>> y
    [(1, 2, 3), (4, 5, 6)]
    >>> z
    [(1, 4), (2, 5), (3, 6)]
    >>> x == z
    1

    It is not possible to reverse zip this way when the sequences are
    not all the same length.

Reference Implementation

    Here is a reference implementation, in Python of the zip()
    built-in function.  This will be replaced with a C implementation
    after final approval.

    def zip(*args):
        if not args:
            raise TypeError('zip() expects one or more sequence arguments')
        ret = []
        i = 0
        try:
            while 1:
                item = []
                for s in args:
                    item.append(s[i])
                ret.append(tuple(item))
                i = i + 1
        except IndexError:
            return ret

BDFL Pronouncements

    Note: the BDFL refers to Guido van Rossum, Python's Benevolent
    Dictator For Life.

    - The function's name.  An earlier version of this PEP included an
      open issue listing 20+ proposed alternative names to zip().  In
      the face of no overwhelmingly better choice, the BDFL strongly
      prefers zip() due to its Haskell[2] heritage.  See version 1.7
      of this PEP for the list of alternatives.

    - zip() shall be a built-in function.

    - Optional padding.  An earlier version of this PEP proposed an
      optional `pad' keyword argument, which would be used when the
      argument sequences were not the same length.  This is similar
      behavior to the map(None, ...) semantics except that the user
      would be able to specify pad object.  This has been rejected by
      the BDFL in favor of always truncating to the shortest sequence,
      because of the KISS principle.  If there's a true need, it is
      easier to add later.  If it is not needed, it would still be
      impossible to delete it in the future.

    - Lazy evaluation.  An earlier version of this PEP proposed that
      zip() return a built-in object that performed lazy evaluation
      using __getitem__() protocol.  This has been strongly rejected
      by the BDFL in favor of returning a real Python list.  If lazy
      evaluation is desired in the future, the BDFL suggests an xzip()
      function be added.

    - zip() with no arguments.  the BDFL strongly prefers this raise a
      TypeError exception.

    - zip() with one argument.  the BDFL strongly prefers that this
      return a list of 1-tuples.

    - Inner and outer container control.  An earlier version of this
      PEP contains a rather lengthy discussion on a feature that some
      people wanted, namely the ability to control what the inner and
      outer container types were (they are tuples and list
      respectively in this version of the PEP).  Given the simplified
      API and implementation, this elaboration is rejected.  For a
      more detailed analysis, see version 1.7 of this PEP.

References

    [1] http://www.python.org/doc/current/ref/for.html
    [2] http://www.haskell.org/onlinereport/standard-prelude.html#$vzip

    Greg Wilson's questionaire on proposed syntax to some CS grad students
    http://www.python.org/pipermail/python-dev/2000-July/013139.html

Copyright

    This document has been placed in the public domain.


PEP 202 -- List Comprehensions

PEP: | 202
Title: | List Comprehensions
Version: | $Revision: 1.6 $
Author: | tim@digicool.com (Tim Peters)
Status: | Draft
Type: | Standards Track
Python-Version: | 2.0
Created: | 13-Jul-2000
Post-History: |

Introduction

    This PEP describes a proposed syntactical extension to Python,
    list comprehensions.

The Proposed Solution

    It is proposed to allow conditional construction of list literals
    using for and if clauses.  They would nest in the same way for
    loops and if statements nest now.

Rationale

    List comprehensions provide a more concise way to create lists in
    situations where map() and filter() and/or nested loops would
    currently be used.

Examples

    >>> print [i for i in range(10)]
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

    >>> print [i for i in range(20) if i%2 == 0]
    [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

    >>> nums = [1,2,3,4]
    >>> fruit = ["Apples", "Peaches", "Pears", "Bananas"]
    >>> print [(i,f) for i in nums for f in fruit]
    [(1, 'Apples'), (1, 'Peaches'), (1, 'Pears'), (1, 'Bananas'),
     (2, 'Apples'), (2, 'Peaches'), (2, 'Pears'), (2, 'Bananas'),
     (3, 'Apples'), (3, 'Peaches'), (3, 'Pears'), (3, 'Bananas'),
     (4, 'Apples'), (4, 'Peaches'), (4, 'Pears'), (4, 'Bananas')]
    >>> print [(i,f) for i in nums for f in fruit if f[0] == "P"]
    [(1, 'Peaches'), (1, 'Pears'),
     (2, 'Peaches'), (2, 'Pears'),
     (3, 'Peaches'), (3, 'Pears'),
     (4, 'Peaches'), (4, 'Pears')]
    >>> print [(i,f) for i in nums for f in fruit if f[0] == "P" if i%2 == 1]
    [(1, 'Peaches'), (1, 'Pears'), (3, 'Peaches'), (3, 'Pears')]
    >>> def zip(*args):
    ...    return apply(map, (None,)+args)
    ...
    >>> print [i for i in zip(nums,fruit) if i[0]%2==0]
    [(2, 'Peaches'), (4, 'Bananas')]

Reference Implementation

    SourceForge contains a patch that adds list comprehensions to Python[1].

BDFL Pronouncements

    Note: the BDFL refers to Guido van Rossum, Python's Benevolent
    Dictator For Life.

    - The syntax proposed above is the Right One.

    - The form [x, y for ...] should be disallowed; one should be
      required to write [(x, y) for ...].

    - The form [... for x... for y...] nests, with the last index
      varying fastest, just like nested for loops.

Open Issues

    Syntax

        Several people proposed connecting or separating syntax
        between the various clauses, for example, requiring a
        semicolon between them to set them apart:

            [i,f; for i in nums; for f in fruit; if f[0]=="P"; if i%2==1]

        To minimize strain on the Python parser, Guido has suggested
        requiring parentheses around the initial tuple:

            [(i,f) for i in nums for f in fruit if f[0]=="P" if i%2==1]

    Semantics

        The semantics of multiple for clauses is not obvious to many
        people.  Currently, it nests, so that

            [i,f for i in nums for f in fruit]

        is functionally equivalent to

            tmp = []
            for i in nums:
                for f in fruit:
                    tmp.append((i,f))

        Other people would read it as if it executed

            map(None, nums, fruit)

        It's not clear that this is necessary.  The newly proposed
        zip() builtin[2] takes care of that case.

    Stability of the Implementation

        The current reference implementation is simply an adaptation
        of Greg Ewing's original demonstration of the concept.  Other
        than tracking changes to the source code to keep it a valid
        patch, reindenting the code and switching to function
        prototypes, nothing has been done to it.  This obviously
        raises some questions about how stable the code is.  It has
        not had a lot of exercise, though the patch does include a few
        test cases.

References

    [1] https://sourceforge.net/patch/?func=detailpatch&patch_id=100654&group_id=5470
    [2] Lockstep Iteration, pep-0201.txt


PEP 203 -- Augmented Assignments

PEP: | 203
Title: | Augmented Assignments
Version: | $Revision: 1.13 $
Author: | thomas@xs4all.net (Thomas Wouters)
Status: | Final
Type: | Standards Track
Python-Version: | 2.0
Created: | 13-Jul-2000
Post-History: | 14-Aug-2000

Introduction

    This PEP describes the `augmented assignment' proposal for Python
    2.0.  This PEP tracks the status and ownership of this feature,
    slated for introduction in Python 2.0.  It contains a description
    of the feature and outlines changes necessary to support the
    feature.  This PEP summarizes discussions held in mailing list
    forums, and provides URLs for further information where
    appropriate.  The CVS revision history of this file contains the
    definitive historical record.

Proposed semantics

    The proposed patch that adds augmented assignment to Python
    introduces the following new operators:

       += -= *= /= %= **= <<= >>= &= ^= |=

    They implement the same operator as their normal binary form,
    except that the operation is done `in-place' when the left-hand
    side object supports it, and that the left-hand side is only
    evaluated once.

    They truly behave as augmented assignment, in that they perform
    all of the normal load and store operations, in addition to the
    binary operation they are intended to do. So, given the expression:

       x += y

    The object `x' is loaded, then `y' is added to it, and the
    resulting object is stored back in the original place. The precise
    action performed on the two arguments depends on the type of `x',
    and possibly of `y'.

    The idea behind augmented assignment in Python is that it isn't
    just an easier way to write the common practice of storing the
    result of a binary operation in its left-hand operand, but also a
    way for the left-hand operand in question to know that it should
    operate `on itself', rather than creating a modified copy of
    itself.

    To make this possible, a number of new `hooks' are added to Python
    classes and C extension types, which are called when the object in
    question is used as the left hand side of an augmented assignment
    operation. If the class or type does not implement the `in-place'
    hooks, the normal hooks for the particular binary operation are
    used.

    So, given an instance object `x', the expression

        x += y

    tries to call x.__iadd__(y), which is the `in-place' variant of
    __add__. If __iadd__ is not present, x.__add__(y) is attempted,
    and finally y.__radd__(x) if __add__ is missing too.  There is no
    `right-hand-side' variant of __iadd__, because that would require
    for `y' to know how to in-place modify `x', which is unsafe to say
    the least. The __iadd__ hook should behave similar to __add__,
    returning the result of the operation (which could be `self')
    which is to be assigned to the variable `x'.

    For C extension types, the `hooks' are members of the
    PyNumberMethods and PySequenceMethods structures.  Some special
    semantics apply to make the use of these methods, and the mixing
    of Python instance objects and C types, as unsurprising as
    possible.

    In the generic case of `x <augop> y' (or a similar case using the
    PyNumber_InPlace API functions) the principal object being
    operated on is `x'.  This differs from normal binary operations,
    where `x' and `y' could be considered `co-operating', because
    unlike in binary operations, the operands in an in-place operation
    cannot be swapped.  However, in-place operations do fall back to
    normal binary operations when in-place modification is not
    supported, resuling in the following rules:

    - If the left-hand object (`x') is an instance object, and it
      has a `__coerce__' method, call that function with `y' as the
      argument. If coercion succeeds, and the resulting left-hand
      object is a different object than `x', stop processing it as
      in-place and call the appropriate function for the normal binary
      operation, with the coerced `x' and `y' as arguments. The result
      of the operation is whatever that function returns.

      If coercion does not yield a different object for `x', or `x'
      does not define a `__coerce__' method, and `x' has the
      appropriate `__ihook__' for this operation, call that method
      with `y' as the argument, and the result of the operation is
      whatever that method returns.

    - Otherwise, if the left-hand object is not an instance object,
      but its type does define the in-place function for this
      operation, call that function with `x' and `y' as the arguments,
      and the result of the operation is whatever that function
      returns.

      Note that no coercion on either `x' or `y' is done in this case,
      and it's perfectly valid for a C type to receive an instance
      object as the second argument; that is something that cannot
      happen with normal binary operations.

    - Otherwise, process it exactly as a normal binary operation (not
      in-place), including argument coercion. In short, if either
      argument is an instance object, resolve the operation through
      `__coerce__', `__hook__' and `__rhook__'. Otherwise, both
      objects are C types, and they are coerced and passed to the
      appropriate function.

    - If no way to process the operation can be found, raise a
      TypeError with an error message specific to the operation.

    - Some special casing exists to account for the case of `+' and
      `*', which have a special meaning for sequences: for `+',
      sequence concatenation, no coercion what so ever is done if a C
      type defines sq_concat or sq_inplace_concat. For `*', sequence
      repeating, `y' is converted to a C integer before calling either
      sq_inplace_repeat and sq_repeat. This is done even if `y' is an
      instance, though not if `x' is an instance.

    The in-place function should always return a new reference, either
    to the old `x' object if the operation was indeed performed
    in-place, or to a new object.

Rationale

    There are two main reasons for adding this feature to Python:
    simplicity of expression, and support for in-place operations. The
    end result is a tradeoff between simplicity of syntax and
    simplicity of expression; like most new features, augmented
    assignment doesn't add anything that was previously impossible. It
    merely makes these things easier to do.

    Adding augmented assignment will make Python's syntax more complex.
    Instead of a single assignment operation, there are now twelve
    assignment operations, eleven of which also perform an binary
    operation. However, these eleven new forms of assignment are easy
    to understand as the coupling between assignment and the binary
    operation, and they require no large conceptual leap to
    understand. Furthermore, languages that do have augmented
    assignment have shown that they are a popular, much used feature.
    Expressions of the form

        <x> = <x> <operator> <y>

    are common enough in those languages to make the extra syntax
    worthwhile, and Python does not have significantly fewer of those
    expressions. Quite the opposite, in fact, since in Python you can
    also concatenate lists with a binary operator, something that is
    done quite frequently. Writing the above expression as

        <x> <operator>= <y>

    is both more readable and less error prone, because it is
    instantly obvious to the reader that it is <x> that is being
    changed, and not <x> that is being replaced by something almost,
    but not quite, entirely unlike <x>.

    The new in-place operations are especially useful to matrix
    calculation and other applications that require large objects. In
    order to efficiently deal with the available program memory, such
    packages cannot blindly use the current binary operations. Because
    these operations always create a new object, adding a single item
    to an existing (large) object would result in copying the entire
    object (which may cause the application to run out of memory), add
    the single item, and then possibly delete the original object,
    depending on reference count.

    To work around this problem, the packages currently have to use
    methods or functions to modify an object in-place, which is
    definitely less readable than an augmented assignment expression.
    Augmented assignment won't solve all the problems for these
    packages, since some operations cannot be expressed in the limited
    set of binary operators to start with, but it is a start. A
    different PEP[2] is looking at adding new operators.

New methods

    The proposed implementation adds the following 11 possible `hooks'
    which Python classes can implement to overload the augmented
    assignment operations:

        __iadd__
        __isub__
        __imul__
        __idiv__
        __imod__
        __ipow__
        __ilshift__
        __irshift__
        __iand__
        __ixor__
        __ior__

    The `i' in `__iadd__' stands for `in-place'.

    For C extension types, the following struct members are added:

    To PyNumberMethods:
        binaryfunc nb_inplace_add;
        binaryfunc nb_inplace_subtract;
        binaryfunc nb_inplace_multiply;
        binaryfunc nb_inplace_divide;
        binaryfunc nb_inplace_remainder;
        binaryfunc nb_inplace_power;
        binaryfunc nb_inplace_lshift;
        binaryfunc nb_inplace_rshift;
        binaryfunc nb_inplace_and;
        binaryfunc nb_inplace_xor;
        binaryfunc nb_inplace_or;

    To PySequenceMethods:
        binaryfunc sq_inplace_concat;
        intargfunc sq_inplace_repeat;

    In order to keep binary compatibility, the tp_flags TypeObject
    member is used to determine whether the TypeObject in question has
    allocated room for these slots. Until a clean break in binary
    compatibility is made (which may or may not happen before 2.0)
    code that wants to use one of the new struct members must first
    check that they are available with the `PyType_HasFeature()'
    macro:

    if (PyType_HasFeature(x->ob_type, Py_TPFLAGS_HAVE_INPLACE_OPS) &&
        x->ob_type->tp_as_number && x->ob_type->tp_as_number->nb_inplace_add) {
            /* ... */

    This check must be made even before testing the method slots for
    NULL values! The macro only tests whether the slots are available,
    not whether they are filled with methods or not.

Implementation

    The current implementation of augmented assignment[1] adds, in
    addition to the methods and slots already covered, 13 new bytecodes
    and 13 new API functions.

    The API functions are simply in-place versions of the current
    binary-operation API functions:

        PyNumber_InPlaceAdd(PyObject *o1, PyObject *o2);
        PyNumber_InPlaceSubtract(PyObject *o1, PyObject *o2);
        PyNumber_InPlaceMultiply(PyObject *o1, PyObject *o2);
        PyNumber_InPlaceDivide(PyObject *o1, PyObject *o2);
        PyNumber_InPlaceRemainder(PyObject *o1, PyObject *o2);
        PyNumber_InPlacePower(PyObject *o1, PyObject *o2);
        PyNumber_InPlaceLshift(PyObject *o1, PyObject *o2);
        PyNumber_InPlaceRshift(PyObject *o1, PyObject *o2);
        PyNumber_InPlaceAnd(PyObject *o1, PyObject *o2);
        PyNumber_InPlaceXor(PyObject *o1, PyObject *o2);
        PyNumber_InPlaceOr(PyObject *o1, PyObject *o2);
        PySequence_InPlaceConcat(PyObject *o1, PyObject *o2);
        PySequence_InPlaceRepeat(PyObject *o, int count);

    They call either the Python class hooks (if either of the objects
    is a Python class instance) or the C type's number or sequence
    methods.

    The new bytecodes are:
        INPLACE_ADD
        INPLACE_SUBTRACT
        INPLACE_MULTIPLY
        INPLACE_DIVIDE
        INPLACE_REMAINDER
        INPLACE_POWER
        INPLACE_LEFTSHIFT
        INPLACE_RIGHTSHIFT
        INPLACE_AND
        INPLACE_XOR
        INPLACE_OR
        ROT_FOUR
        DUP_TOPX

    The INPLACE_* bytecodes mirror the BINARY_* bytecodes, except that
    they are implemented as calls to the `InPlace' API functions. The
    other two bytecodes are `utility' bytecodes: ROT_FOUR behaves like
    ROT_THREE except that the four topmost stack items are rotated.

    DUP_TOPX is a bytecode that takes a single argument, which should
    be an integer between 1 and 5 (inclusive) which is the number of
    items to duplicate in one block. Given a stack like this (where
    the right side of the list is the `top' of the stack):

        [1, 2, 3, 4, 5]

    "DUP_TOPX 3" would duplicate the top 3 items, resulting in this
    stack:

        [1, 2, 3, 4, 5, 3, 4, 5]

    DUP_TOPX with an argument of 1 is the same as DUP_TOP. The limit
    of 5 is purely an implementation limit. The implementation of
    augmented assignment requires only DUP_TOPX with an argument of 2
    and 3, and could do without this new opcode at the cost of a fair
    number of DUP_TOP and ROT_*.

Open Issues

    The PyNumber_InPlace API is only a subset of the normal PyNumber
    API: only those functions that are required to support the
    augmented assignment syntax are included. If other in-place API
    functions are needed, they can be added later.

    The DUP_TOPX bytecode is a conveniency bytecode, and is not
    actually necessary. It should be considered whether this bytecode
    is worth having. There seems to be no other possible use for this
    bytecode at this time.

Copyright

    This document has been placed in the public domain.

References

    [1] http://www.python.org/pipermail/python-list/2000-June/059556.html
    [2] http://sourceforge.net/patch?func=detailpatch&patch_id=100699&group_id=5470
    [3] PEP 211, Adding New Linear Algebra Operators,
        http://python.sourceforge.net/peps/pep-0211.html


PEP 204 -- Range Literals

PEP: | 204
Title: | Range Literals
Version: | $Revision: 1.7 $
Author: | thomas@xs4all.net (Thomas Wouters)
Status: | Rejected
Type: | Standards Track
Python-Version: | 2.0
Created: | 14-Jul-2000
Post-History: |

Introduction

    This PEP describes the `range literal' proposal for Python 2.0.
    This PEP tracks the status and ownership of this feature, slated
    for introduction in Python 2.0.  It contains a description of the
    feature and outlines changes necessary to support the feature.
    This PEP summarizes discussions held in mailing list forums, and
    provides URLs for further information, where appropriate.  The CVS
    revision history of this file contains the definitive historical
    record.

List ranges

    Ranges are sequences of numbers of a fixed stepping, often used in
    for-loops.  The Python for-loop is designed to iterate over a
    sequence directly:

        >>> l = ['a', 'b', 'c', 'd']
        >>> for item in l:
        ...     print item
        a
        b
        c
        d

    However, this solution is not always prudent.  Firstly, problems
    arise when altering the sequence in the body of the for-loop,
    resulting in the for-loop skipping items.  Secondly, it is not
    possible to iterate over, say, every second element of the
    sequence.  And thirdly, it is sometimes necessary to process an
    element based on its index, which is not readily available in the
    above construct.

    For these instances, and others where a range of numbers is
    desired, Python provides the `range' builtin function, which
    creates a list of numbers.  The `range' function takes three
    arguments, `start', `end' and `step'.  `start' and `step' are
    optional, and default to 0 and 1, respectively.

    The `range' function creates a list of numbers, starting at
    `start', with a step of `step', up to, but not including `end', so
    that `range(10)' produces a list that has exactly 10 items, the
    numbers 0 through 9.

    Using the `range' function, the above example would look like
    this:

        >>> for i in range(len(l)):
        ...     print l[i]
        a
        b
        c
        d

    Or, to start at the second element of `l' and processing only
    every second element from then on:

        >>> for i in range(1, len(l), 2):
        ...     print l[i]
        b
        d

    There are several disadvantages with this approach:

    - Clarity of purpose: Adding another function call, possibly with
      extra arithmetic to determine the desired length and step of the
      list, does not improve readability of the code.  Also, it is
      possible to `shadow' the builtin `range' function by supplying a
      local or global variable with the same name, effectively
      replacing it.  This may or may not be a desired effect.

    - Efficiency: because the `range' function can be overridden, the
      Python compiler cannot make assumptions about the for-loop, and
      has to maintain a separate loop counter.

    - Consistency: There already is a syntax that is used to denote
      ranges, as shown below.  This syntax uses the exact same
      arguments, though all optional, in the exact same way.  It seems
      logical to extend this syntax to ranges, to form `range
      literals'.

Slice Indices

    In Python, a sequence can be indexed in one of two ways:
    retrieving a single item, or retrieving a range of items.
    Retrieving a range of items results in a new object of the same
    type as the original sequence, containing zero or more items from
    the original sequence.  This is done using a `range notation':

        >>> l[2:4]
        ['c', 'd']

    This range notation consists of zero, one or two indices separated
    by a colon.  The first index is the `start' index, the second the
    `end'.  When either is left out, they default to respectively the
    start and the end of the sequence.

    There is also an extended range notation, which incorporates
    `step' as well.  Though this notation is not currently supported
    by most builtin types, if it were, it would work as follows:

        >>> l[1:4:2]
        ['b', 'd']

    The third `argument' to the slice syntax is exactly the same as
    the `step' argument to range().  The underlying mechanisms of the
    standard, and these extended slices, are sufficiently different
    and inconsistent that many classes and extensions outside of
    mathematical packages do not implement support for the extended
    variant.  While this should be resolved, it is beyond the scope of
    this PEP.

    Extended slices do show, however, that there is already a
    perfectly valid and applicable syntax to denote ranges in a way
    that solve all of the earlier stated disadvantages of the use of
    the range() function:

    - It is clearer, more concise syntax, which has already proven to
      be both intuitive and easy to learn.

    - It is consistent with the other use of ranges in Python
      (e.g. slices).

    - Because it is built-in syntax, instead of a builtin function, it
      cannot be overridden.  This means both that a viewer can be
      certain about what the code does, and that an optimizer will not
      have to worry about range() being `shadowed'.

The Proposed Solution

    The proposed implementation of range-literals combines the syntax
    for list literals with the syntax for (extended) slices, to form
    range literals:

        >>> [1:10]
        [1, 2, 3, 4, 5, 6, 7, 8, 9]
        >>> [:5]
        [0, 1, 2, 3, 4]
        >>> [5:1:-1]
        [5, 4, 3, 2]

    There is one minor difference between range literals and the slice
    syntax: though it is possible to omit all of `start', `end' and
    `step' in slices, it does not make sense to omit `end' in range
    literals.  In slices, `end' would default to the end of the list,
    but this has no meaning in range literals.

Reference Implementation

    The proposed implementation can be found on SourceForge[1].  It
    adds a new bytecode, BUILD_RANGE, that takes three arguments from
    the stack and builds a list on the bases of those.  The list is
    pushed back on the stack.

    The use of a new bytecode is necessary to be able to build ranges
    based on other calculations, whose outcome is not known at compile
    time.

    The code introduces two new functions to listobject.c, which are
    currently hovering between private functions and full-fledged API
    calls.

    PyList_FromRange() builds a list from start, end and step,
    returning NULL if an error occurs.  Its prototype is:

        PyObject * PyList_FromRange(long start, long end, long step)

    PyList_GetLenOfRange() is a helper function used to determine the
    length of a range.  Previously, it was a static function in
    bltinmodule.c, but is now necessary in both listobject.c and
    bltinmodule.c (for xrange).  It is made non-static solely to avoid
    code duplication.  Its prototype is:

        long PyList_GetLenOfRange(long start, long end, long step)

Open issues

    - One possible solution to the discrepancy of requiring the `end'
      argument in range literals is to allow the range syntax to
      create a `generator', rather than a list, such as the `xrange'
      builtin function does.  However, a generator would not be a
      list, and it would be impossible, for instance, to assign to
      items in the generator, or append to it.

      The range syntax could conceivably be extended to include tuples
      (i.e. immutable lists), which could then be safely implemented
      as generators.  This may be a desirable solution, especially for
      large number arrays: generators require very little in the way
      of storage and initialization, and there is only a small
      performance impact in calculating and creating the appropriate
      number on request.  (TBD: is there any at all? Cursory testing
      suggests equal performance even in the case of ranges of length
      1)

      However, even if idea was adopted, would it be wise to `special
      case' the second argument, making it optional in one instance of
      the syntax, and non-optional in other cases ?

    - Should it be possible to mix range syntax with normal list
      literals, creating a single list?  E.g.:

          >>> [5, 6, 1:6, 7, 9]

    to create

          [5, 6, 1, 2, 3, 4, 5, 7, 9]

    - How should range literals interact with another proposed new
      feature, `list comprehensions'[2]?  Specifically, should it be
      possible to create lists in list comprehensions?  E.g.:

          >>> [x:y for x in (1, 2) y in (3, 4)]

      Should this example return a single list with multiple ranges:

          [1, 2, 1, 2, 3, 2, 2, 3]

      Or a list of lists, like so:

          [[1, 2], [1, 2, 3], [2], [2, 3]]

      However, as the syntax and semantics of list comprehensions are
      still subject of hot debate, these issues are probably best
      addressed by the `list comprehensions' PEP.

    - Range literals accept objects other than integers: it performs
      PyInt_AsLong() on the objects passed in, so as long as the
      objects can be coerced into integers, they will be accepted.
      The resulting list, however, is always composed of standard
      integers.

      Should range literals create a list of the passed-in type?  It
      might be desirable in the cases of other builtin types, such as
      longs and strings:

          >>> [ 1L : 2L<<64 : 2<<32L ]
          >>> ["a":"z":"b"]
          >>> ["a":"z":2]

      However, this might be too much `magic' to be obvious.  It might
      also present problems with user-defined classes: even if the
      base class can be found and a new instance created, the instance
      may require additional arguments to __init__, causing the
      creation to fail.

    - The PyList_FromRange() and PyList_GetLenOfRange() functions need
      to be classified: are they part of the API, or should they be
      made private functions?

Rejection

    After careful consideration, and a period of meditation, this
    proposal has been rejected. The open issues, as well as some
    confusion between ranges and slice syntax, raised enough questions
    for Guido not to accept it for Python 2.0, and later to reject the
    proposal altogether. The new syntax and its intentions were deemed
    not obvious enough.

    [ TBD: Guido, ammend/confirm this, please. Preferably both; this
      is a PEP, it should contain *all* the reasons for rejection
      and/or reconsideration, for future reference. ]

Copyright

    This document has been placed in the Public Domain.

References:

    [1] http://sourceforge.net/patch/?func=detailpatch&patch_id=100902&group_id=5470
    [2] PEP 202, List Comprehensions, pep-0202.txt


PEP 205 -- Weak References

PEP: | 205
Title: | Weak References
Version: | $Revision: 1.12 $
Author: | Fred L. Drake, Jr.
Python-Version: | 2.1
Status: | Incomplete
Type: | Standards Track
Post-History: | 11 January 2001

Motivation

    There are two basic applications for weak references which have
    been noted by Python programmers: object caches and reduction of
    pain from circular references.

    Caches (weak dictionaries)

        There is a need to allow objects to be maintained that represent
        external state, mapping a single instance to the external
        reality, where allowing multiple instances to be mapped to the
        same external resource would create unnecessary difficulty
        maintaining synchronization among instances.  In these cases,
        a common idiom is to support a cache of instances; a factory
        function is used to return either a new or existing instance.

        The difficulty in this approach is that one of two things must
        be tolerated: either the cache grows without bound, or there
        needs to be explicit management of the cache elsewhere in the
        application.  The later can be very tedious and leads to more
        code than is really necessary to solve the problem at hand,
        and the former can be unacceptable for long-running processes
        or even relatively short processes with substantial memory
        requirements.

        - External objects that need to be represented by a single
          instance, no matter how many internal users there are.  This
          can be useful for representing files that need to be written
          back to disk in whole rather than locked & modified for
          every use.

        - Objects that are expensive to create, but may be needed by
          multiple internal consumers.  Similar to the first case, but
          not necessarily bound to external resources, and possibly
          not an issue for shared state.  Weak references are only
          useful in this case if there is some flavor of "soft"
          references or if there is a high likelihood that users of
          individual objects will overlap in lifespan.

    Circular references

        - DOMs require a huge amount of circular (to parent & document
          nodes), but these could be eliminated using a weak
          dictionary mapping from each node to it's parent.  This
          might be especially useful in the context of something like
          xml.dom.pulldom, allowing the .unlink() operation to become
          a no-op.

    This proposal is divided into the following sections:

        - Proposed Solution
        - Implementation Strategy
        - Possible Applications
        - Previous Weak Reference Work in Python
        - Weak References in Java

    The full text of one early proposal is included as an appendix
    since it does not appear to be available on the net.

Aspects of the Solution Space

    There are two distinct aspects to the weak references problem:

        - Invalidation of weak references
        - Presentation of weak references to Python code

    Invalidation:

    Past approaches to weak reference invalidation have often hinged
    on storing a strong reference and being able to examine all the
    instances of weak reference objects, and invalidating them when
    the reference count of their referent goes to one (indicating that
    the reference stored by the weak reference is the last remaining
    reference).  This has the advantage that the memory management
    machinery in Python need not change, and that any type can be
    weakly referenced.

    The disadvantage of this approach to invalidation is that it
    assumes that the management of the weak references is called
    sufficiently frequently that weakly-referenced objects are noticed
    within a reasonably short time frame; since this means a scan over
    some data structure to invalidate references, an operation which
    is O(N) on the number of weakly referenced objects, this is not
    effectively amortized for any single object which is weakly
    referenced.  This also assumes that the application is calling
    into code which handles weakly-referenced objects with some
    frequency, which makes weak-references less attractive for library
    code.

    An alternate approach to invalidation is that the de-allocation
    code to be aware of the possibility of weak references and make a
    specific call into the weak-reference management code to all
    invalidation whenever an object is deallocated.  This requires a
    change in the tp_dealloc handler for weakly-referencable objects;
    an additional call is needed at the "top" of the handler for
    objects which support weak-referencing, and an efficient way to
    map from an object to a chain of weak references for that object
    is needed as well.

    Presentation:

    Two ways that weak references are presented to the Python layer
    have been as explicit reference objects upon which some operation
    is required in order to retrieve a usable reference to the
    underlying object, and proxy objects which masquerade as the
    original objects as much as possible.

    Reference objects are easy to work with when some additional layer
    of object managemenet is being added in Python; references can be
    checked for liveness explicitly, without having to invoke
    operations on the referents and catching some special exception
    raised when an invalid weak reference is used.

    However, a number of users favor the proxy appoach simply because
    the weak reference looks so much like the original object.

Proposed Solution

    Weak references should be able to point to any Python object that
    may have substantial memory size (directly or indirectly), or hold
    references to external resources (database connections, open
    files, etc.).

    A new module, weakref, will contain new functions used to create
    weak references.  weakref.ref() will create a "weak reference
    object" and optionally attach a callback which will be called when
    the object is about to be finalized.  weakref.mapping() will
    create a "weak dictionary".  A third function, weakref.proxy(),
    will create a proxy object that behaves somewhat like the original
    object.

    A weak reference object will allow access to the referenced object
    if it hasn't been collected and to determine if the object still
    exists in memory.  Retrieving the referent is done by calling the
    reference object.  If the referent is no longer alive, this will
    return None instead.

    A weak dictionary maps arbitrary keys to values, but does not own
    a reference to the values.  When the values are finalized, the
    (key, value) pairs for which it is a value are removed from all
    the mappings containing such pairs.  Like dictionaries, weak
    dictionaries are not hashable.

    Proxy objects are weak references that attempt to behave like the
    object they proxy, as much as they can.  Regardless of the
    underlying type, proxies are not hashable since their ability to
    act as a weak reference relies on a fundamental mutability that
    will cause failures when used as dictionary keys -- even if the
    proper hash value is computed before the referent dies, the
    resulting proxy cannot be used as a dictionary key since it cannot
    be compared once the referent has expired, and comparability is
    necessary for dictionary keys.  Operations on proxy objects after
    the referent dies cause weakref.ReferenceError to be raised in
    most cases.  "is" comparisons, type(), and id() will continue to
    work, but always refer to the proxy and not the referent.

    The callbacks registered with weak references must accept a single
    parameter, which will be the weak reference or proxy object
    itself.  The object cannot be accessed or resurrected in the
    callback.

Implementation Strategy

    The implementation of weak references will include a list of
    reference containers that must be cleared for each weakly-
    referencable object.  If the reference is from a weak dictionary,
    the dictionary entry is cleared first.  Then, any associated
    callback is called with the object passed as a parameter.  Once
    all callbacks have been called, the object is finalized and
    deallocated.

    Many built-in types will participate in the weak-reference
    management, and any extension type can elect to do so.  The type
    structure will contain an additional field which provides an
    offset into the instance structure which contains a list of weak
    reference structures.  If the value of the field is <= 0, the
    object does not participate.  In this case, weakref.ref(),
    <weakdict>.__setitem__() and .setdefault(), and item assignment will
    raise TypeError.  If the value of the field is > 0, a new weak
    reference can be generated and added to the list.

    This approach is taken to allow arbitrary extension types to
    participate, without taking a memory hit for numbers or other
    small types.

    XXX -- Need to determine the exact list of standard types which
    should support weak references.  This should include instances and
    dictionaries.  Whether it should include strings, buffers, lists,
    files, or sockets is debatable.

    The implementation has already been added to the Python CVS
    repository.

Possible Applications

    PyGTK+ bindings?

    Tkinter -- could avoid circular references by using weak
    references from widgets to their parents.  Objects won't be
    discarded any sooner in the typical case, but there won't be so
    much dependence on the programmer calling .destroy() before
    releasing a reference.  This would mostly benefit long-running
    applications.

    DOM trees.

Previous Weak Reference Work in Python

    Dianne Hackborn has proposed something called "virtual references".
    'vref' objects are very similar to java.lang.ref.WeakReference
    objects, except there is no equivalent to the invalidation
    queues.  Implementing a "weak dictionary" would be just as
    difficult as using only weak references (without the invalidation
    queue) in Java.  Information on this has disappeared from the Web,
    but is included below as an Appendix.

    Marc-André Lemburg's mx.Proxy package:

        http://www.lemburg.com/files/python/mxProxy.html

    The weakdict module by Dieter Maurer is implemented in C and
    Python.  It appears that the Web pages have not been updated since
    Python 1.5.2a, so I'm not yet sure if the implementation is
    compatible with Python 2.0.

        http://www.handshake.de/~dieter/weakdict.html

    PyWeakReference by Alex Shindich:

        http://sourceforge.net/projects/pyweakreference/

    Eric Tiedemann has a weak dictionary implementation:

        http://www.hyperreal.org/~est/python/weak/

Weak References in Java

    http://java.sun.com/j2se/1.3/docs/api/java/lang/ref/package-summary.html

    Java provides three forms of weak references, and one interesting
    helper class.  The three forms are called "weak", "soft", and
    "phantom" references.  The relevant classes are defined in the
    java.lang.ref package.

    For each of the reference types, there is an option to add the
    reference to a queue when it is invalidated by the memory
    allocator.  The primary purpose of this facility seems to be that
    it allows larger structures to be composed to incorporate
    weak-reference semantics without having to impose substantial
    additional locking requirements.  For instance, it would not be
    difficult to use this facility to create a "weak" hash table which
    removes keys and referents when a reference is no longer used
    elsewhere.  Using weak references for the objects without some
    sort of notification queue for invalidations leads to much more
    tedious implementation of the various operations required on hash
    tables.  This can be a performance bottleneck if deallocations of
    the stored objects are infrequent.

    Java's "weak" references are most like Dianne Hackborn's old vref
    proposal: a reference object refers to a single Python object,
    but does not own a reference to that object.  When that object is
    deallocated, the reference object is invalidated.  Users of the
    reference object can easily determine that the reference has been
    invalidated, or a NullObjectDereferenceError can be raised when
    an attempt is made to use the referred-to object.

    The "soft" references are similar, but are not invalidated as soon
    as all other references to the referred-to object have been
    released.  The "soft" reference does own a reference, but allows
    the memory allocator to free the referent if the memory is needed
    elsewhere.  It is not clear whether this means soft references are
    released before the malloc() implementation calls sbrk() or its
    equivalent, or if soft references are only cleared when malloc()
    returns NULL.

    "Phantom" references are a little different; unlike weak and soft
    references, the referent is not cleared when the reference is
    added to it's queue.  When all phantom references for an object
    are dequeued, the object is cleared.  This can be used to keep an
    object alive until some additional cleanup is performed which
    needs to happen before the objects .finalize() method is called.

    Unlike the other two reference types, "phantom" references must be
    associated with an invalidation queue.

Appendix -- Dianne Hackborn's vref proposal (1995)

    [This has been indented and paragraphs reflowed, but there have be
    no content changes.  --Fred]

    Proposal: Virtual References

    In an attempt to partly address the recurring discussion
    concerning reference counting vs. garbage collection, I would like
    to propose an extension to Python which should help in the
    creation of "well structured" cyclic graphs.  In particular, it
    should allow at least trees with parent back-pointers and
    doubly-linked lists to be created without worry about cycles.

    The basic mechanism I'd like to propose is that of a "virtual
    reference," or a "vref" from here on out.  A vref is essentially a
    handle on an object that does not increment the object's reference
    count.  This means that holding a vref on an object will not keep
    the object from being destroyed.  This would allow the Python
    programmer, for example, to create the aforementioned tree
    structure tree structure, which is automatically destroyed when it
    is no longer in use -- by making all of the parent back-references
    into vrefs, they no longer create reference cycles which keep the
    tree from being destroyed.

    In order to implement this mechanism, the Python core must ensure
    that no -real- pointers are ever left referencing objects that no
    longer exist.  The implementation I would like to propose involves
    two basic additions to the current Python system:

    1. A new "vref" type, through which the Python programmer creates
       and manipulates virtual references.  Internally, it is
       basically a C-level Python object with a pointer to the Python
       object it is a reference to.  Unlike all other Python code,
       however, it does not change the reference count of this object.
       In addition, it includes two pointers to implement a
       doubly-linked list, which is used below.

    2. The addition of a new field to the basic Python object
       [PyObject_Head in object.h], which is either NULL, or points to
       the head of a list of all vref objects that reference it.  When
       a vref object attaches itself to another object, it adds itself
       to this linked list.  Then, if an object with any vrefs on it
       is deallocated, it may walk this list and ensure that all of
       the vrefs on it point to some safe value, e.g. Nothing.

    This implementation should hopefully have a minimal impact on the
    current Python core -- when no vrefs exist, it should only add one
    pointer to all objects, and a check for a NULL pointer every time
    an object is deallocated.

    Back at the Python language level, I have considered two possible
    semantics for the vref object --

    ==> Pointer semantics:

      In this model, a vref behaves essentially like a Python-level
      pointer; the Python program must explicitly dereference the vref
      to manipulate the actual object it references.

      An example vref module using this model could include the
      function "new"; When used as 'MyVref = vref.new(MyObject)', it
      returns a new vref object such that that MyVref.object ==
      MyObject.  MyVref.object would then change to Nothing if
      MyObject is ever deallocated.

      For a concrete example, we may introduce some new C-style syntax:

      & -- unary operator, creates a vref on an object, same as vref.new().
      * -- unary operator, dereference a vref, same as VrefObject.object.

      We can then define:

      1.     type(&MyObject) == vref.VrefType
      2.        *(&MyObject) == MyObject
      3. (*(&MyObject)).attr == MyObject.attr
      4.          &&MyObject == Nothing
      5.           *MyObject -> exception

      Rule #4 is subtle, but comes about because we have made a vref
      to (a vref with no real references).  Thus the outer vref is
      cleared to Nothing when the inner one inevitably disappears.

    ==> Proxy semantics:

      In this model, the Python programmer manipulates vref objects
      just as if she were manipulating the object it is a reference
      of.  This is accomplished by implementing the vref so that all
      operations on it are redirected to its referenced object.  With
      this model, the dereference operator (*) no longer makes sense;
      instead, we have only the reference operator (&), and define:

      1.  type(&MyObject) == type(MyObject)
      2.        &MyObject == MyObject
      3. (&MyObject).attr == MyObject.attr
      4.       &&MyObject == MyObject

      Again, rule #4 is important -- here, the outer vref is in fact a
      reference to the original object, and -not- the inner vref.
      This is because all operations applied to a vref actually apply
      to its object, so that creating a vref of a vref actually
      results in creating a vref of the latter's object.

    The first, pointer semantics, has the advantage that it would be
    very easy to implement; the vref type is extremely simple,
    requiring at minimum a single attribute, object, and a function to
    create a reference.

    However, I really like the proxy semantics.  Not only does it put
    less of a burden on the Python programmer, but it allows you to do
    nice things like use a vref anywhere you would use the actual
    object.  Unfortunately, it would probably an extreme pain, if not
    practically impossible, to implement in the current Python
    implementation.  I do have some thoughts, though, on how to do
    this, if it seems interesting; one possibility is to introduce new
    type-checking functions which handle the vref.  This would
    hopefully older C modules which don't expect vrefs to simply
    return a type error, until they can be fixed.

    Finally, there are some other additional capabilities that this
    system could provide.  One that seems particularily interesting to
    me involves allowing the Python programmer to add "destructor"
    function to a vref -- this Python function would be called
    immediately prior to the referenced object being deallocated,
    allowing a Python program to invisibly attach itself to another
    object and watch for it to disappear.  This seems neat, though I
    haven't actually come up with any practical uses for it, yet... :)

    -- Dianne

Copyright

    This document has been placed in the public domain.


PEP 206 -- 2.0 Batteries Included

PEP: | 206
Title: | 2.0 Batteries Included
Version: | $Revision: 1.12 $
Author: | moshez@zadka.site.co.il (Moshe Zadka)
Python-Version: | 2.0
Status: | Draft

Introduction

    This PEP describes the `batteries included' proposal for Python
    2.0, the fat distribution containing commonly used third party
    extension modules.  This PEP tracks the status and ownership of
    this proposal, slated for introduction in Python 2.0.  It contains
    a description of the proposal and outlines how to support it.

Batteries Included Philosophy

    The Python source distribution always maintained the philosophy of
    "batteries included" -- having a rich and versatile standard
    library which is immediately available, without making the user
    download separate packages.  This gives the Python language a head
    start in many projects.  However, the Python standard library
    often relies on important Open Source libraries which might be
    unavailable on many computers, so that the user has to separately
    download and compile those.  Some examples are the zlib
    compression library, and the gmp number manipulation library.

    The original coinage of the term "batteries included" can be found
    here:

    http://www.uk.research.att.com/~fms/ipc7/tr-1998-9.html

The Proposed Solution

    The proposed solution is to maintain an alternate form of distribution,
    nicknamed the "sumo" interpreter. It will be structured as a core
    interpreter + extra useful libraries. In addition, the build procedure
    will be changed to build those libraries by default, and build the Python
    modules which rely on them linked against those libraries.  Individual
    users will still be able to link the Python modules against already
    installed libraries, or disable them completely.

    Additionally, some Open Source third-party Python modules will
    also be distributed in the "sumo" distribution.  The
    difference between those and external libraries is that the former
    are distributed in order to make the Python distribution self
    contained, and the latter are added so there will be new
    functionality in out-of-the-box Python.

Python Advanced Library

    Since it is decided that the additions to the interpreter will live
    in a seperate directory, they can even be distributed seperately,
    as an additional library. This is called the "Python Advanced Library",
    or PAL. This will also solve the problem of an advanced library module
    failing -- the core interpreter will still be built, tested and installed.
    Since the compilation of the other modules be helped by Python scripts,
    it is quite possible that it will even install as many modules as it
    can.

Suggested Libraries and Modules

    Here is the tentative list of libraries which are proposed to be
    dealt with in this manner, and where they can be downloaded:

    zlib -- http://www.info-zip.org/pub/infozip/zlib/zlib.tar.gz
    expat -- ftp://ftp.jclark.com/pub/xml/expat.zip
    Tcl -- http://dev.scriptics.com:80/download/tcl/tcl8_3/tcl8.3.1.tar.gz
    Tk -- http://dev.scriptics.com:80/download/tcl/tcl8_3/tk8.3.1.tar.gz
    PIL -- http://www.pythonware.com/downloads/Imaging-1.1.tar.gz
    libjpeg -- ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6b.tar.gz
    ncurses -- ftp://dickey.his.com/ncurses/ncurses.tar.gz

    TBD, the following:

    NumPy -- http://download.sourceforge.net/numpy/Numerical-15.3.tgz
    Pmw --  ftp://ftp.dscpl.com.au/pub/pmw/Pmw.0.8.4.tar.gz
    BLT -- ftp://ftp.tcltk.com/aa004735/pub/blt/BLT2.4u.tar.gz
    piddle -- http://download.sourceforge.net/piddle/piddle-1.0b7.tgz

Connection to Distutils

    Since Python 2.0 comes with a new mechanism of distributing Python
    modules, called ``distutils'', we should be able to use it. Specificially,
    since distutils has the capability or building Python modules in an
    automatic fashion, it is hoped that some of the work will consist in
    creating a new directory with the source, and having the ``super-make''
    run "./setup --install" in those directories. This should at least take
    care of PIL, NumPy and Pmw. In addition, it is hoped that this mechanism
    will be easy enough to add other Python modules as user requests direct
    us.

Software covered by the GNU General Public License

    While many Python modules rely on software distributed under the
    GNU General Public License and the GNU Lesser General Public
    License, no such sources are proposed here to be added to the
    sumo interpreter.  Currently, the Python interpreter is
    distributed under a closed-source-friendly license, which means
    that it is possible to include it in a closed source product.
    Making redistributers worry about which parts they need to remove
    to make closed-source redistribution legal might cost more then
    the benefits.

How It Is Implemented

    Sumo-Python is basically be a set of scripts, which download
    a Python source tarball, and source tarballs for all the modules.
    Then it unpacks everything, modifies Setup.in (or whatever mechanism
    will be in place by Python 2.1, and puts in a little
    shell script which builds everything. Then it will repack it
    into the huge tarball which will be a sumo-python-version.tar.gz,
    which users can then download and install.

    Inside the tarball, running the script

    build-everything [--prefix ...]

    Will compile and install a sumo interpreter into specified prefix.

Open Issues

    ESR also mentioned libpng, but I don't know of a Python module
    that uses it.

    How to make PAL as easy to build as the whole Sumo Python?


PEP 207 -- Rich Comparisions

PEP: | 207
Title: | Rich Comparisions
Version: | $Revision: 1.10 $
Author: | guido@python.org (Guido van Rossum), DavidA@ActiveState.com (David Ascher)
Python-Version: | 2.1
Status: | Final

Abstract

    This PEP proposes several new features for comparisons:

    - Allow separately overloading of <, >, <=, >=, ==, !=, both in
      classes and in C extensions.

    - Allow any of those overloaded operators to return something else
      besides a Boolean result.

Motivation

    The main motivation comes from NumPy, whose users agree that A<B
    should return an array of elementwise comparison outcomes; they
    currently have to spell this as less(A,B) because A<B can only
    return a Boolean result or raise an exception.

    An additional motivation is that frequently, types don't have a
    natural ordering, but still need to be compared for equality.
    Currently such a type *must* implement comparison and thus define
    an arbitrary ordering, just so that equality can be tested.

    Also, for some object types an equality test can be implemented
    much more efficiently than an ordering test; for example, lists
    and dictionaries that differ in length are unequal, but the
    ordering requires inspecting some (potentially all) items.

Previous Work

    Rich Comparisons have been proposed before; in particular by David
    Ascher, after experience with Numerical Python:

      http://starship.python.net/crew/da/proposals/richcmp.html

    It is also included below as an Appendix.  Most of the material in
    this PEP is derived from David's proposal.

Concerns

    1 Backwards compatibility, both at the Python level (classes using
      __cmp__ need not be changed) and at the C level (extensions
      defining tp_compare need not be changed, code using
      PyObject_Compare() must work even if the compared objects use
      the new rich comparison scheme).

    2 When A<B returns a matrix of elementwise comparisons, an easy
      mistake to make is to use this expression in a Boolean context.
      Without special precautions, it would always be true.  This use
      should raise an exception instead.

    3 If a class overrides x==y but nothing else, should x!=y be
      computed as not(x==y), or fail?  What about the similar
      relationship between < and >=, or between > and <=?

    4 Similarly, should we allow x<y to be calculated from y>x?  And
      x<=y from not(x>y)?  And x==y from y==x, or x!=y from y!=x?

    5 When comparison operators return elementwise comparisons, what
      to do about shortcut operators like A<B<C, ``A<B and C<D'',
      ``A<B or C<D''?

    6 What to do about min() and max(), the 'in' and 'not in'
      operators, list.sort(), dictionary key comparison, and other
      uses of comparisons by built-in operations?

Proposed Resolutions

    1 Full backwards compatibility can be achieved as follows.  When
      an object defines tp_compare() but not tp_richcompare(), and a
      rich comparison is requested, the outcome of tp_compare() is
      used in the ovious way.  E.g. if "<" is requested, an exception if
      tp_compare() raises an exception, the outcome is 1 if
      tp_compare() is negative, and 0 if it is zero or positive.  Etc.

      Full forward compatibility can be achieved as follows.  When a
      classic comparison is requested on an object that implements
      tp_richcompare(), up to three comparisons are used: first == is
      tried, and if it returns true, 0 is returned; next, < is tried
      and if it returns true, -1 is returned; next, > is tried and if
      it returns true, +1 is returned.  If any operator tried returns
      a non-Boolean value (see below), the exception raised by
      conversion to Boolean is passed through.  If none of the
      operators tried returns true, the classic comparison fallbacks
      are tried next.

      (I thought long and hard about the order in which the three
      comparisons should be tried.  At one point I had a convincing
      argument for doing it in this order, based on the behavior of
      comparisons for cyclical data structures.  But since that code
      has changed again, I'm not so sure that it makes a difference
      any more.)

    2 Any type that returns a collection of Booleans instead of a
      single boolean should define nb_nonzero() to raise an exception.
      Such a type is considered a non-Boolean.

    3 The == and != operators are not assumed to be each other's
      complement (e.g. IEEE 754 floating point numbers do not satisfy
      this).  It is up to the type to implement this if desired.
      Similar for < and >=, or > and <=; there are lots of examples
      where these assumptions aren't true (e.g. tabnanny).

    4 The reflexivity rules *are* assumed by Python.  Thus, the
      interpreter may swap y>x with x<y, y>=x with x<=y, and may swap
      the arguments of x==y and x!=y.  (Note: Python currently assumes
      that x==x is always true and x!=x is never true; this should not
      be assumed.)

    5 In the current proposal, when A<B returns an array of
      elementwise comparisons, this outcome is considered non-Boolean,
      and its interpretation as Boolean by the shortcut operators
      raises an exception.  David Ascher's proposal tries to deal
      with this; I don't think this is worth the additional complexity
      in the code generator.  Instead of A<B<C, you can write
      (A<B)&(B<C).

    6 The min() and list.sort() operations will only use the
      < operator; max() will only use the > operator.  The 'in' and
      'not in' operators and dictionary lookup will only use the ==
      operator.

Implementation Proposal

    This closely follows David Ascher's proposal.

    C API

    - New functions:

      PyObject *PyObject_RichCompare(PyObject *, PyObject *, enum cmp_op)

      This performs the requested rich comparison, returning a Python
      object or raising an exception.  The 3rd argument must be one of
      Py_LT, Py_LE, Py_EQ, Py_NE, Py_GT or Py_GE.

      int PyObject_RichCompareBool(PyObject *, PyObject *, enum cmp_op)

      This performs the requested rich comparison, returning a
      Boolean: -1 for exception, 0 for false, 1 for true.  The 3rd
      argument must be one of Py_LT, Py_LE, Py_EQ, Py_NE, Py_GT or
      Py_GE.  Note that when PyObject_RichCompare() returns a
      non-Boolean object, PyObject_RichCompareBool() will raise an
      exception.

    - New typedef:

      typedef PyObject *(*richcmpfunc) (PyObject *, PyObject *, int);

    - New slot in type object, replacing spare tp_xxx7:

      richcmpfunc tp_richcompare;

      This should be a function with the same signature as
      PyObject_RichCompare(), and performing the same comparison.
      At least one of the arguments is of the type whose
      tp_richcompare slot is being used, but the other may have a
      different type.  If the function cannot compare the particular
      combination of objects, it should return PyExc_NotImplemented.

    - PyObject_Compare() is changed to try rich comparisons if they
      are defined (but only if classic comparisons aren't defined).

    Changes to the interpreter

    - Whenever PyObject_Compare() is called with the intent of getting
      the outcome of a particular comparison (e.g. in list.sort(), and
      of course for the comparison operators in ceval.c), the code is
      changed to call PyObject_RichCompare() or
      PyObject_RichCompareBool() instead; if the C code needs to know
      the outcome of the comparison, PyObject_IsTrue() is called on
      the result (which may raise an exception).

    - Most built-in types that currently define a comparison will be
      modified to define a rich comparison instead.  (This is
      optional; I've converted lists, tuples, complex numbers, and
      arrays so far, and am not sure whether I will convert others.)

    Classes

    - Classes can define new special methods __lt__, __le__, __eq__,
      __ne__,__gt__, __ge__ to override the corresponding operators.
      (I.e., <, <=, ==, !=, >, >=. You gotta love the Fortran
      heritage.)  If a class defines __cmp__ as well, it is only used
      when __lt__ etc. have been tried and return NotImplemented.

Copyright

    This document has been placed in the public domain.

Appendix

    Here is most of David Ascher's original proposal (version 0.2.1,
    dated Wed Jul 22 16:49:28 1998; I've left the Contents, History
    and Patches sections out).  It addresses almost all concerns
    above.

Abstract

    A new mechanism allowing comparisons of Python objects to return
    values other than -1, 0, or 1 (or raise exceptions) is
    proposed. This mechanism is entirely backwards compatible, and can
    be controlled at the level of the C PyObject type or of the Python
    class definition. There are three cooperating parts to the
    proposed mechanism:

    - the use of the last slot in the type object structure to store a
    pointer to a rich comparison function

    - the addition of special methods for classes

    - the addition of an optional argument to the builtin cmp()
    function.

Motivation

    The current comparison protocol for Python objects assumes that
    any two Python objects can be compared (as of Python 1.5, object
    comparisons can raise exceptions), and that the return value for
    any comparison should be -1, 0 or 1. -1 indicates that the first
    argument to the comparison function is less than the right one, +1
    indicating the contrapositive, and 0 indicating that the two
    objects are equal. While this mechanism allows the establishment
    of a order relationship (e.g. for use by the sort() method of list
    objects), it has proven to be limited in the context of Numeric
    Python (NumPy).

    Specifically, NumPy allows the creation of multidimensional
    arrays, which support most of the numeric operators. Thus:

             x = array((1,2,3,4))        y = array((2,2,4,4))

    are two NumPy arrays. While they can be added elementwise,:

             z = x + y   # z == array((3,4,7,8))

    they cannot be compared in the current framework - the released
    version of NumPy compares the pointers, (thus yielding junk
    information) which was the only solution before the recent
    addition of the ability (in 1.5) to raise exceptions in comparison
    functions.

    Even with the ability to raise exceptions, the current protocol
    makes array comparisons useless. To deal with this fact, NumPy
    includes several functions which perform the comparisons: less(),
    less_equal(), greater(), greater_equal(), equal(),
    not_equal(). These functions return arrays with the same shape as
    their arguments (modulo broadcasting), filled with 0's and 1's
    depending on whether the comparison is true or not for each
    element pair. Thus, for example, using the arrays x and y defined
    above:

             less(x,y)

    would be an array containing the numbers (1,0,0,0).

    The current proposal is to modify the Python object interface to
    allow the NumPy package to make it so that x < y returns the same
    thing as less(x,y). The exact return value is up to the NumPy
    package -- what this proposal really asks for is changing the
    Python core so that extension objects have the ability to return
    something other than -1, 0, 1, should their authors choose to do
    so.

Current State of Affairs

    The current protocol is, at the C level, that each object type
    defines a tp_compare slot, which is a pointer to a function which
    takes two PyObject* references and returns -1, 0, or 1. This
    function is called by the PyObject_Compare() function defined in
    the C API. PyObject_Compare() is also called by the builtin
    function cmp() which takes two arguments.

Proposed Mechanism

    1. Changes to the C structure for type objects

    The last available slot in the PyTypeObject, reserved up to now
    for future expansion, is used to optionally store a pointer to a
    new comparison function, of type richcmpfunc defined by:

           typedef PyObject *(*richcmpfunc)
                Py_PROTO((PyObject *, PyObject *, int));

    This function takes three arguments. The first two are the objects
    to be compared, and the third is an integer corresponding to an
    opcode (one of LT, LE, EQ, NE, GT, GE). If this slot is left NULL,
    then rich comparison for that object type is not supported (except
    for class instances whose class provide the special methods
    described below).

    The above opcodes need to be added to the published Python/C API
    (probably under the names Py_LT, Py_LE, etc.)

    2. Additions of special methods for classes

    Classes wishing to support the rich comparison mechanisms must add
    one or more of the following new special methods:

             def __lt__(self, other):
                ...
             def __le__(self, other):
                ...
             def __gt__(self, other):
                ...
             def __ge__(self, other):
                ...
             def __eq__(self, other):
                ...
             def __ne__(self, other):
                ...

    Each of these is called when the class instance is the on the
    left-hand-side of the corresponding operators (<, <=, >, >=, ==,
    and != or <>). The argument other is set to the object on the
    right side of the operator. The return value of these methods is
    up to the class implementor (after all, that's the entire point of
    the proposal).

    If the object on the left side of the operator does not define an
    appropriate rich comparison operator (either at the C level or
    with one of the special methods, then the comparison is reversed,
    and the right hand operator is called with the opposite operator,
    and the two objects are swapped. This assumes that a < b and b > a
    are equivalent, as are a <= b and b >= a, and that == and != are
    commutative (e.g. a == b if and only if b == a).

    For example, if obj1 is an object which supports the rich
    comparison protocol and x and y are objects which do not support
    the rich comparison protocol, then obj1 < x will call the __lt__
    method of obj1 with x as the second argument. x < obj1 will call
    obj1's __gt__ method with x as a second argument, and x < y will
    just use the existing (non-rich) comparison mechanism.

    The above mechanism is such that classes can get away with not
    implementing either __lt__ and __le__ or __gt__ and
    __ge__. Further smarts could have been added to the comparison
    mechanism, but this limited set of allowed "swaps" was chosen
    because it doesn't require the infrastructure to do any processing
    (negation) of return values. The choice of six special methods was
    made over a single (e.g. __richcmp__) method to allow the
    dispatching on the opcode to be performed at the level of the C
    implementation rather than the user-defined method.

    3. Addition of an optional argument to the builtin cmp()

    The builtin cmp() is still used for simple comparisons. For rich
    comparisons, it is called with a third argument, one of "<", "<=",
    ">", ">=", "==", "!=", "<>" (the last two have the same
    meaning). When called with one of these strings as the third
    argument, cmp() can return any Python object. Otherwise, it can
    only return -1, 0 or 1 as before.

Chained Comparisons

    Problem

    It would be nice to allow objects for which the comparison returns
    something other than -1, 0, or 1 to be used in chained
    comparisons, such as:

             x < y < z

    Currently, this is interpreted by Python as:

             temp1 = x < y
             if temp1:
               return y < z
             else:
               return temp1

    Note that this requires testing the truth value of the result of
    comparisons, with potential "shortcutting" of the right-side
    comparison testings. In other words, the truth-value of the result
    of the result of the comparison determines the result of a chained
    operation. This is problematic in the case of arrays, since if x,
    y and z are three arrays, then the user expects:

            x < y < z

    to be an array of 0's and 1's where 1's are in the locations
    corresponding to the elements of y which are between the
    corresponding elements in x and z. In other words, the right-hand
    side must be evaluated regardless of the result of x < y, which is
    incompatible with the mechanism currently in use by the parser.

    Solution

    Guido mentioned that one possible way out would be to change the
    code generated by chained comparisons to allow arrays to be
    chained-compared intelligently. What follows is a mixture of his
    idea and my suggestions. The code generated for x < y < z would be
    equivalent to:

             temp1 = x < y
             if temp1:
               temp2 = y < z
               return boolean_combine(temp1, temp2)
             else:
               return temp1

    where boolean_combine is a new function which does something like
    the following:

             def boolean_combine(a, b):
                 if hasattr(a, '__boolean_and__') or \
                    hasattr(b, '__boolean_and__'):
                     try:
                         return a.__boolean_and__(b)
                     except:
                         return b.__boolean_and__(a)
                 else: # standard behavior
                     if a:
                         return b
                     else:
                         return 0

    where the __boolean_and__ special method is implemented for
    C-level types by another value of the third argument to the
    richcmp function. This method would perform a boolean comparison
    of the arrays (currently implemented in the umath module as the
    logical_and ufunc).

    Thus, objects returned by rich comparisons should always test
    true, but should define another special method which creates
    boolean combinations of them and their argument.

    This solution has the advantage of allowing chained comparisons to
    work for arrays, but the disadvantage that it requires comparison
    arrays to always return true (in an ideal world, I'd have them
    always raise an exception on truth testing, since the meaning of
    testing "if a>b:" is massively ambiguous.

    The inlining already present which deals with integer comparisons
    would still apply, resulting in no performance cost for the most
    common cases.


PEP 208 -- Reworking the Coercion Model

PEP: | 208
Title: | Reworking the Coercion Model
Version: | $Revision: 1.8 $
Author: | nas@arctrix.com (Neil Schemenauer), mal@lemburg.com (Marc-André Lemburg)
Status: | Final
Type: | Standards Track
Created: | 04-Dec-2000
Post-History: |
Python-Version: | 2.1

Abstract

    Many Python types implement numeric operations.  When the arguments of
    a numeric operation are of different types, the interpreter tries to
    coerce the arguments into a common type.  The numeric operation is
    then performed using this common type.  This PEP proposes a new type
    flag to indicate that arguments to a type's numeric operations should
    not be coerced.  Operations that do not support the supplied types
    indicate it by returning a new singleton object.  Types which do not
    set the type flag are handled in a backwards compatible manner.
    Allowing operations handle different types is often simpler, more
    flexible, and faster than having the interpreter do coercion.

Rationale

    When implementing numeric or other related operations, it is often
    desirable to provide not only operations between operands of one type
    only, e.g. integer + integer, but to generalize the idea behind the
    operation to other type combinations as well, e.g. integer + float.

    A common approach to this mixed type situation is to provide a method
    of "lifting" the operands to a common type (coercion) and then use
    that type's operand method as execution mechanism.  Yet, this strategy
    has a few drawbacks:

        * the "lifting" process creates at least one new (temporary)
          operand object,

        * since the coercion method is not being told about the operation
          that is to follow, it is not possible to implement operation
          specific coercion of types,

        * there is no elegant way to solve situations were a common type
          is not at hand, and

        * the coercion method will always have to be called prior to the
          operation's method itself.

    A fix for this situation is obviously needed, since these drawbacks
    make implementations of types needing these features very cumbersome,
    if not impossible.  As an example, have a look at the DateTime and
    DateTimeDelta[1] types, the first being absolute, the second
    relative.  You can always add a relative value to an absolute one,
    giving a new absolute value.  Yet, there is no common type which the
    existing coercion mechanism could use to implement that operation.

    Currently, PyInstance types are treated specially by the interpreter
    in that their numeric methods are passed arguments of different types.
    Removing this special case simplifies the interpreter and allows other
    types to implement numeric methods that behave like instance types.
    This is especially useful for extension types like ExtensionClass.

Specification

    Instead of using a central coercion method, the process of handling
    different operand types is simply left to the operation.  If the
    operation finds that it cannot handle the given operand type
    combination, it may return a special singleton as indicator.

    Note that "numbers" (anything that implements the number protocol, or
    part of it) written in Python already use the first part of this
    strategy - it is the C level API that we focus on here.

    To maintain nearly 100% backward compatibility we have to be very
    careful to make numbers that don't know anything about the new
    strategy (old style numbers) work just as well as those that expect
    the new scheme (new style numbers).  Furthermore, binary compatibility
    is a must, meaning that the interpreter may only access and use new
    style operations if the number indicates the availability of these.

    A new style number is considered by the interpreter as such if and
    only it it sets the type flag Py_TPFLAGS_NEWSTYLENUMBER.  The main
    difference between an old style number and a new style one is that the
    numeric slot functions can no longer assume to be passed arguments of
    identical type.  New style slots must check all arguments for proper
    type and implement the necessary conversions themselves.  This may seem
    to cause more work on the behalf of the type implementor, but is in
    fact no more difficult than writing the same kind of routines for an
    old style coercion slot.

    If a new style slot finds that it cannot handle the passed argument
    type combination, it may return a new reference of the special
    singleton Py_NotImplemented to the caller.  This will cause the caller
    to try the other operands operation slots until it finds a slot that
    does implement the operation for the specific type combination.  If
    none of the possible slots succeed, it raises a TypeError.

    To make the implementation easy to understand (the whole topic is
    esoteric enough), a new layer in the handling of numeric operations is
    introduced.  This layer takes care of all the different cases that need
    to be taken into account when dealing with all the possible
    combinations of old and new style numbers.  It is implemented by the
    two static functions binary_op() and ternary_op(), which are both
    internal functions that only the functions in Objects/abstract.c
    have access to.  The numeric API (PyNumber_*) is easy to adapt to
    this new layer.

    As a side-effect all numeric slots can be NULL-checked (this has to be
    done anyway, so the added feature comes at no extra cost).

    The scheme used by the layer to execute a binary operation is as
    follows:

      v        | w          | Action taken
      ---------+------------+----------------------------------
      new      | new        | v.op(v,w), w.op(v,w)
      new      | old        | v.op(v,w), coerce(v,w), v.op(v,w)
      old      | new        | w.op(v,w), coerce(v,w), v.op(v,w)
      old      | old        | coerce(v,w), v.op(v,w)

    The indicated action sequence is executed from left to right until
    either the operation succeeds and a valid result (!=
    Py_NotImplemented) is returned or an exception is raised.  Exceptions
    are returned to the calling function as-is.  If a slot returns
    Py_NotImplemented, the next item in the sequence is executed.

    Note that coerce(v,w) will use the old style nb_coerce slot methods
    via a call to PyNumber_Coerce().

    Ternary operations have a few more cases to handle:

      v   | w   | z   | Action taken
      ----+-----+-----+------------------------------------
      new | new | new | v.op(v,w,z), w.op(v,w,z), z.op(v,w,z)
      new | old | new | v.op(v,w,z), z.op(v,w,z), coerce(v,w,z), v.op(v,w,z)
      old | new | new | w.op(v,w,z), z.op(v,w,z), coerce(v,w,z), v.op(v,w,z)
      old | old | new | z.op(v,w,z), coerce(v,w,z), v.op(v,w,z)
      new | new | old | v.op(v,w,z), w.op(v,w,z), coerce(v,w,z), v.op(v,w,z)
      new | old | old | v.op(v,w,z), coerce(v,w,z), v.op(v,w,z)
      old | new | old | w.op(v,w,z), coerce(v,w,z), v.op(v,w,z)
      old | old | old | coerce(v,w,z), v.op(v,w,z)

    The same notes as above, except that coerce(v,w,z) actually does:

        if z != Py_None:
            coerce(v,w), coerce(v,z), coerce(w,z)
        else:
            # treat z as absent variable
            coerce(v,w)

    The current implementation uses this scheme already (there's only one
    ternary slot: nb_pow(a,b,c)).

    Note that the numeric protocol is also used for some other related
    tasks, e.g. sequence concatenation.  These can also benefit from the
    new mechanism by implementing right-hand operations for type
    combinations that would otherwise fail to work.  As an example, take
    string concatenation: currently you can only do string + string.  With
    the new mechanism, a new string-like type could implement new_type +
    string and string + new_type, even though strings don't know anything
    about new_type.

    Since comparisons also rely on coercion (every time you compare an
    integer to a float, the integer is first converted to float and then
    compared...), a new slot to handle numeric comparisons is needed:

        PyObject *nb_cmp(PyObject *v, PyObject *w)

    This slot should compare the two objects and return an integer object
    stating the result.  Currently, this result integer may only be -1, 0,
    1. If the slot cannot handle the type combination, it may return a
    reference to Py_NotImplemented.  [XXX Note that this slot is still
    in flux since it should take into account rich comparisons
    (i.e. PEP 207).]

    Numeric comparisons are handled by a new numeric protocol API:

        PyObject *PyNumber_Compare(PyObject *v, PyObject *w)

    This function compare the two objects as "numbers" and return an
    integer object stating the result.  Currently, this result integer may
    only be -1, 0, 1.  In case the operation cannot be handled by the given
    objects, a TypeError is raised.

    The PyObject_Compare() API needs to adjusted accordingly to make use
    of this new API.

    Other changes include adapting some of the built-in functions (e.g.
    cmp()) to use this API as well.  Also, PyNumber_CoerceEx() will need to
    check for new style numbers before calling the nb_coerce slot.  New
    style numbers don't provide a coercion slot and thus cannot be
    explicitly coerced.

Reference Implementation

    A preliminary patch for the CVS version of Python is available through
    the Source Forge patch manager[2].

Credits

    This PEP and the patch are heavily based on work done by Marc-André
    Lemburg[3].

Copyright

    This document has been placed in the public domain.

References

    [1] http://www.lemburg.com/files/python/mxDateTime.html
    [2] http://sourceforge.net/patch/?func=detailpatch&patch_id=102652&group_id=5470
    [3] http://www.lemburg.com/files/python/CoercionProposal.html


PEP 209 -- Multi-dimensional Arrays

PEP: | 209
Title: | Multi-dimensional Arrays
Version: | $Revision: 1.3 $
Author: | barrett@stsci.edu (Paul Barrett), oliphant@ee.byu.edu (Travis Oliphant)
Python-Version: | 2.2
Status: | Draft
Type: | Standards Track
Created: | 03-Jan-2001
Post-History: |

Abstract

    This PEP proposes a redesign and re-implementation of the multi-
    dimensional array module, Numeric, to make it easier to add new
    features and functionality to the module.  Aspects of Numeric 2
    that will receive special attention are efficient access to arrays
    exceeding a gigabyte in size and composed of inhomogeneous data
    structures or records.  The proposed design uses four Python
    classes: ArrayType, UFunc, Array, and ArrayView; and a low-level
    C-extension module, _ufunc, to handle the array operations
    efficiently.  In addition, each array type has its own C-extension
    module which defines the coercion rules, operations, and methods
    for that type.  This design enables new types, features, and
    functionality to be added in a modular fashion.  The new version
    will introduce some incompatibilities with the current Numeric.

Motivation

    Multi-dimensional arrays are commonly used to store and manipulate
    data in science, engineering, and computing.  Python currently has
    an extension module, named Numeric (henceforth called Numeric 1),
    which provides a satisfactory set of functionality for users
    manipulating homogeneous arrays of data of moderate size (of order
    10 MB).  For access to larger arrays (of order 100 MB or more) of
    possibly inhomogeneous data, the implementation of Numeric 1 is
    inefficient and cumbersome.  In the future, requests by the
    Numerical Python community for additional functionality is also
    likely as PEPs 211: Adding New Linear Operators to Python, and
    225: Elementwise/Objectwise Operators illustrate.

Proposal

    This proposal recommends a re-design and re-implementation of
    Numeric 1, henceforth called Numeric 2, which will enable new
    types, features, and functionality to be added in an easy and
    modular manner.  The initial design of Numeric 2 should focus on
    providing a generic framework for manipulating arrays of various
    types and should enable a straightforward mechanism for adding new
    array types and UFuncs.  Functional methods that are more specific
    to various disciplines can then be layered on top of this core.
    This new module will still be called Numeric and most of the
    behavior found in Numeric 1 will be preserved.

    The proposed design uses four Python classes: ArrayType, UFunc,
    Array, and ArrayView; and a low-level C-extension module to handle
    the array operations efficiently.  In addition, each array type
    has its own C-extension module which defines the coercion rules,
    operations, and methods for that type.  At a later date, when core
    functionality is stable, some Python classes can be converted to
    C-extension types.

    Some planned features are:

    1.  Improved memory usage

    This feature is particularly important when handling large arrays
    and can produce significant improvements in performance as well as
    memory usage.  We have identified several areas where memory usage
    can be improved:

        a.  Use a local coercion model

        Instead of using Python's global coercion model which creates
        temporary arrays, Numeric 2, like Numeric 1, will implement a
        local coercion model as described in PEP 208 which defers the
        responsibility of coercion to the operator.  By using internal
        buffers, a coercion operation can be done for each array
        (including output arrays), if necessary, at the time of the
        operation.  Benchmarks [1] have shown that performance is at
        most degraded only slightly and is improved in cases where the
        internal buffers are less than the L2 cache size and the
        processor is under load.  To avoid array coercion altogether,
        C functions having arguments of mixed type are allowed in
        Numeric 2.

        b.  Avoid creation of temporary arrays

        In complex array expressions (i.e. having more than one
        operation), each operation will create a temporary array which
        will be used and then deleted by the succeeding operation.  A
        better approach would be to identify these temporary arrays
        and reuse their data buffers when possible, namely when the
        array shape and type are the same as the temporary array being
        created.  This can be done by checking the temporary array's
        reference count.  If it is 1, then it will be deleted once the
        operation is done and is a candidate for reuse.

        c.  Optional use of memory-mapped files

        Numeric users sometimes need to access data from very large
        files or to handle data that is greater than the available
        memory.  Memory-mapped arrays provide a mechanism to do this
        by storing the data on disk while making it appear to be in
        memory.  Memory- mapped arrays should improve access to all
        files by eliminating one of two copy steps during a file
        access.  Numeric should be able to access in-memory and
        memory-mapped arrays transparently.

        d.  Record access

        In some fields of science, data is stored in files as binary
        records.  For example in astronomy, photon data is stored as a
        1 dimensional list of photons in order of arrival time.  These
        records or C-like structures contain information about the
        detected photon, such as its arrival time, its position on the
        detector, and its energy.  Each field may be of a different
        type, such as char, int, or float.  Such arrays introduce new
        issues that must be dealt with, in particular byte alignment
        or byte swapping may need to be performed for the numeric
        values to be properly accessed (though byte swapping is also
        an issue for memory mapped data).  Numeric 2 is designed to
        automatically handle alignment and representational issues
        when data is accessed or operated on.  There are two
        approaches to implementing records; as either a derived array
        class or a special array type, depending on your point-of-
        view.  We defer this discussion to the Open Issues section.

    2.  Additional array types

    Numeric 1 has 11 defined types: char, ubyte, sbyte, short, int,
    long, float, double, cfloat, cdouble, and object.  There are no
    ushort, uint, or ulong types, nor are there more complex types
    such as a bit type which is of use to some fields of science and
    possibly for implementing masked-arrays.  The design of Numeric 1
    makes the addition of these and other types a difficult and
    error-prone process.  To enable the easy addition (and deletion)
    of new array types such as a bit type described below, a re-design
    of Numeric is necessary.

        a.  Bit type

        The result of a rich comparison between arrays is an array of
        boolean values.  The result can be stored in an array of type
        char, but this is an unnecessary waste of memory.  A better
        implementation would use a bit or boolean type, compressing
        the array size by a factor of eight.  This is currently being
        implemented for Numeric 1 (by Travis Oliphant) and should be
        included in Numeric 2.

    3.  Enhanced array indexing syntax

    The extended slicing syntax was added to Python to provide greater
    flexibility when manipulating Numeric arrays by allowing
    step-sizes greater than 1.  This syntax works well as a shorthand
    for a list of regularly spaced indices.  For those situations
    where a list of irregularly spaced indices are needed, an enhanced
    array indexing syntax would allow 1-D arrays to be arguments.

    4.  Rich comparisons

    The implementation of PEP 207: Rich Comparisons in Python 2.1
    provides additional flexibility when manipulating arrays.  We
    intend to implement this feature in Numeric 2.

    5. Array broadcasting rules

    When an operation between a scalar and an array is done, the
    implied behavior is to create a new array having the same shape as
    the array operand containing the scalar value.  This is called
    array broadcasting.  It also works with arrays of lesser rank,
    such as vectors.  This implicit behavior is implemented in Numeric
    1 and will also be implemented in Numeric 2.

Design and Implementation

    The design of Numeric 2 has four primary classes:

    1.  ArrayType:

    This is a simple class that describes the fundamental properties
    of an array-type, e.g. its name, its size in bytes, its coercion
    relations with respect to other types, etc., e.g.

    > Int32 = ArrayType('Int32', 4, 'doc-string')

    Its relation to the other types is defined when the C-extension
    module for that type is imported.  The corresponding Python code
    is:

    > Int32.astype[Real64] = Real64

    This says that the Real64 array-type has higher priority than the
    Int32 array-type.

    The following attributes and methods are proposed for the core
    implementation.  Additional attributes can be added on an
    individual basis, e.g. .bitsize or .bitstrides for the bit type.

    Attributes:
        .name:                  e.g. "Int32", "Float64", etc.
        .typecode:              e.g. 'i', 'f', etc.
                                (for backward compatibility)
        .size (in bytes):       e.g. 4, 8, etc.
        .array_rules (mapping): rules between array types
        .pyobj_rules (mapping): rules between array and python types
        .doc:                   documentation string
    Methods:
        __init__():             initialization
        __del__():              destruction
        __repr__():             representation

    C-API:
        This still needs to be fleshed-out.

    2.  UFunc:

    This class is the heart of Numeric 2.  Its design is similar to
    that of ArrayType in that the UFunc creates a singleton callable
    object whose attributes are name, total and input number of
    arguments, a document string, and an empty CFunc dictionary; e.g.

    > add = UFunc('add', 3, 2, 'doc-string')

    When defined the add instance has no C functions associated with
    it and therefore can do no work.  The CFunc dictionary is
    populated or registered later when the C-extension module for an
    array-type is imported.  The arguments of the register method are:
    function name, function descriptor, and the CUFunc object.  The
    corresponding Python code is

    > add.register('add', (Int32, Int32, Int32), cfunc-add)

    In the initialization function of an array type module, e.g.
    Int32, there are two C API functions: one to initialize the
    coercion rules and the other to register the CFunc objects.

    When an operation is applied to some arrays, the __call__ method
    is invoked.  It gets the type of each array (if the output array
    is not given, it is created from the coercion rules) and checks
    the CFunc dictionary for a key that matches the argument types.
    If it exists the operation is performed immediately, otherwise the
    coercion rules are used to search for a related operation and set
    of conversion functions.  The __call__ method then invokes a
    compute method written in C to iterate over slices of each array,
    namely:

    > _ufunc.compute(slice, data, func, swap, conv)

    The 'func' argument is a CFuncObject, while the 'swap' and 'conv'
    arguments are lists of CFuncObjects for those arrays needing pre-
    or post-processing, otherwise None is used.  The data argument is
    a list of buffer objects, and the slice argument gives the number
    of iterations for each dimension along with the buffer offset and
    step size for each array and each dimension.

    We have predefined several UFuncs for use by the __call__ method:
    cast, swap, getobj, and setobj.  The cast and swap functions do
    coercion and byte-swapping, respectively and the getobj and setobj
    functions do coercion between Numeric arrays and Python sequences.

    The following attributes and methods are proposed for the core
    implementation.

    Attributes:
        .name:                  e.g. "add", "subtract", etc.
        .nargs:                 number of total arguments
        .iargs:                 number of input arguments
        .cfuncs (mapping):      the set C functions
        .doc:                   documentation string
    Methods:
        __init__():             initialization
        __del__():              destruction
        __repr__():             representation
        __call__():             look-up and dispatch method
        initrule():             initialize coercion rule
        uninitrule():           uninitialize coercion rule
        register():             register a CUFunc
        unregister():           unregister a CUFunc

    C-API:
        This still needs to be fleshed-out.

    3.  Array:

    This class contains information about the array, such as shape,
    type, endian-ness of the data, etc..  Its operators, '+', '-',
    etc. just invoke the corresponding UFunc function, e.g.

    > def __add__(self, other):
    >     return ufunc.add(self, other)

    The following attributes, methods, and functions are proposed for
    the core implementation.

    Attributes:
        .shape:                 shape of the array
        .format:                type of the array
        .real (only complex):   real part of a complex array
        .imag (only complex):   imaginary part of a complex array
    Methods:
        __init__():             initialization
        __del__():              destruction
        __repr_():              representation
        __str__():              pretty representation
        __cmp__():              rich comparison
        __len__():
        __getitem__():
        __setitem__():
        __getslice__():
        __setslice__():
        numeric methods:
        copy():                 copy of array
        aslist():               create list from array
        asstring():             create string from array

    Functions:
        fromlist():             create array from sequence
        fromstring():           create array from string
        array():                create array with shape and value
        concat():               concatenate two arrays
        resize():               resize array

    C-API:
        This still needs to be fleshed-out.

    4.  ArrayView

    This class is similar to the Array class except that the reshape
    and flat methods will raise exceptions, since non-contiguous
    arrays cannot be reshaped or flattened using just pointer and
    step-size information.

    C-API:
        This still needs to be fleshed-out.

    5.  C-extension modules:

    Numeric2 will have several C-extension modules.

        a.  _ufunc:

        The primary module of this set is the _ufuncmodule.c.  The
        intention of this module is to do the bare minimum,
        i.e. iterate over arrays using a specified C function.  The
        interface of these functions is the same as Numeric 1, i.e.

        int (*CFunc)(char *data, int *steps, int repeat, void *func);

        and their functionality is expected to be the same, i.e. they
        iterate over the inner-most dimension.

        The following attributes and methods are proposed for the core
        implementation.

        Attributes:

        Methods:
            compute():

        C-API:
            This still needs to be fleshed-out.

        b.  _int32, _real64, etc.:

        There will also be C-extension modules for each array type,
        e.g. _int32module.c, _real64module.c, etc.  As mentioned
        previously, when these modules are imported by the UFunc
        module, they will automatically register their functions and
        coercion rules.  New or improved versions of these modules can
        be easily implemented and used without affecting the rest of
        Numeric 2.

Open Issues

    1.  Does slicing syntax default to copy or view behavior?

    The default behavior of Python is to return a copy of a sub-list
    or tuple when slicing syntax is used, whereas Numeric 1 returns a
    view into the array.  The choice made for Numeric 1 is apparently
    for reasons of performance: the developers wish to avoid the
    penalty of allocating and copying the data buffer during each
    array operation and feel that the need for a deep copy of an array
    to be rare.  Yet, some have argued that Numeric's slice notation
    should also have copy behavior to be consistent with Python lists.
    In this case the performance penalty associated with copy behavior
    can be minimized by implementing copy-on-write.  This scheme has
    both arrays sharing one data buffer (as in view behavior) until
    either array is assigned new data at which point a copy of the
    data buffer is made.  View behavior would then be implemented by
    an ArrayView class, whose behavior be similar to Numeric 1 arrays,
    i.e. .shape is not settable for non-contiguous arrays.  The use of
    an ArrayView class also makes explicit what type of data the array
    contains.

    2.  Does item syntax default to copy or view behavior?

    A similar question arises with the item syntax.  For example, if a
    = [[0,1,2], [3,4,5]] and b = a[0], then changing b[0] also changes
    a[0][0], because a[0] is a reference or view of the first row of
    a.  Therefore, if c is a 2-d array, it would appear that c[i]
    should return a 1-d array which is a view into, instead of a copy
    of, c for consistency.  Yet, c[i] can be considered just a
    shorthand for c[i,:] which would imply copy behavior assuming
    slicing syntax returns a copy.  Should Numeric 2 behave the same
    way as lists and return a view or should it return a copy.

    3.  How is scalar coercion implemented?

    Python has fewer numeric types than Numeric which can cause
    coercion problems.  For example when multiplying a Python scalar
    of type float and a Numeric array of type float, the Numeric array
    is converted to a double, since the Python float type is actually
    a double.  This is often not the desired behavior, since the
    Numeric array will be doubled in size which is likely to be
    annoying, particularly for very large arrays.  We prefer that the
    array type trumps the python type for the same type class, namely
    integer, float, and complex.  Therefore an operation between a
    Python integer and an Int16 (short) array will return an Int16
    array.  Whereas an operation between a Python float and an Int16
    array would return a Float64 (double) array.  Operations between
    two arrays use normal coercion rules.

    4.  How is integer division handled?

    In a future version of Python, the behavior of integer division
    will change.  The operands will be converted to floats, so the
    result will be a float.  If we implement the proposed scalar
    coercion rules where arrays have precedence over Python scalars,
    then dividing an array by an integer will return an integer array
    and will not be consistent with a future version of Python which
    would return an array of type double.  Scientific programmers are
    familiar with the distinction between integer and float-point
    division, so should Numeric 2 continue with this behavior?

    5.  How should records be implemented?

    There are two approaches to implementing records depending on your
    point-of-view.  The first is two divide arrays into separate
    classes depending on the behavior of their types.  For example
    numeric arrays are one class, strings a second, and records a
    third, because the range and type of operations of each class
    differ.  As such, a record array is not a new type, but a
    mechanism for a more flexible form of array.  To easily access and
    manipulate such complex data, the class is comprised of numeric
    arrays having different byte offsets into the data buffer.  For
    example, one might have a table consisting of an array of Int16,
    Real32 values.  Two numeric arrays, one with an offset of 0 bytes
    and a stride of 6 bytes to be interpreted as Int16, and one with an
    offset of 2 bytes and a stride of 6 bytes to be interpreted as
    Real32 would represent the record array.  Both numeric arrays
    would refer to the same data buffer, but have different offset and
    stride attributes, and a different numeric type.

    The second approach is to consider a record as one of many array
    types, albeit with fewer, and possibly different, array operations
    than for numeric arrays.  This approach considers an array type to
    be a mapping of a fixed-length string.  The mapping can either be
    simple, like integer and floating-point numbers, or complex, like
    a complex number, a byte string, and a C-structure.  The record
    type effectively merges the struct and Numeric modules into a
    multi-dimensional struct array.  This approach implies certain
    changes to the array interface.  For example, the 'typecode'
    keyword argument should probably be changed to the more
    descriptive 'format' keyword.

        a.  How are record semantics defined and implemented?

        Which ever implementation approach is taken for records, the
        syntax and semantics of how they are to be accessed and
        manipulated must be decided, if one wishes to have access to
        sub-fields of records.  In this case, the record type can
        essentially be considered an inhomogeneous list, like a tuple
        returned by the unpack method of the struct module; and a 1-d
        array of records may be interpreted as a 2-d array with the
        second dimension being the index into the list of fields.
        This enhanced array semantics makes access to an array of one
        or more of the fields easy and straightforward.  It also
        allows a user to do array operations on a field in a natural
        and intuitive way.  If we assume that records are implemented
        as an array type, then last dimension defaults to 0 and can
        therefore be neglected for arrays comprised of simple types,
        like numeric.

    6.  How are masked-arrays implemented?

    Masked-arrays in Numeric 1 are implemented as a separate array
    class.  With the ability to add new array types to Numeric 2, it
    is possible that masked-arrays in Numeric 2 could be implemented
    as a new array type instead of an array class.

    7.  How are numerical errors handled (IEEE floating-point errors in
        particular)?

    It is not clear to the proposers (Paul Barrett and Travis
    Oliphant) what is the best or preferred way of handling errors.
    Since most of the C functions that do the operation, iterate over
    the inner-most (last) dimension of the array.  This dimension
    could contain a thousand or more items having one or more errors
    of differing type, such as divide-by-zero, underflow, and
    overflow.  Additionally, keeping track of these errors may come at
    the expense of performance.  Therefore, we suggest several
    options:

        a.  Print a message of the most severe error, leaving it to
        the user to locate the errors.

        b.  Print a message of all errors that occurred and the number
        of occurrences, leaving it to the user to locate the errors.

        c.  Print a message of all errors that occurred and a list of
        where they occurred.

        d.  Or use a hybrid approach, printing only the most severe
        error, yet keeping track of what and where the errors
        occurred.  This would allow the user to locate the errors
        while keeping the error message brief.

    8.  What features are needed to ease the integration of FORTRAN
        libraries and code?

    It would be a good idea at this stage to consider how to ease the
    integration of FORTRAN libraries and user code in Numeric 2.

Implementation Steps

    1.  Implement basic UFunc capability

        a.  Minimal Array class:

        Necessary class attributes and methods, e.g. .shape, .data,
        .type, etc.

        b.  Minimal ArrayType class:

        Int32, Real64, Complex64, Char, Object

        c.  Minimal UFunc class:

        UFunc instantiation, CFunction registration, UFunc call for
        1-D arrays including the rules for doing alignment,
        byte-swapping, and coercion.

        d.  Minimal C-extension module:

        _UFunc, which does the innermost array loop in C.

        This step implements whatever is needed to do: 'c = add(a, b)'
        where a, b, and c are 1-D arrays.  It teaches us how to add
        new UFuncs, to coerce the arrays, to pass the necessary
        information to a C iterator method and to do the actually
        computation.

    2.  Continue enhancing the UFunc iterator and Array class

        a.  Implement some access methods for the Array class:
            print, repr, getitem, setitem, etc.

        b.  Implement multidimensional arrays

        c.  Implement some of basic Array methods using UFuncs:
            +, -, *, /, etc.

        d.  Enable UFuncs to use Python sequences.

    3.  Complete the standard UFunc and Array class behavior

        a.  Implement getslice and setslice behavior

        b.  Work on Array broadcasting rules

        c.  Implement Record type

    4.  Add additional functionality

        a.  Add more UFuncs

        b.  Implement buffer or mmap access

Incompatibilities

    The following is a list of incompatibilities in behavior between
    Numeric 1 and Numeric 2.

    1.  Scalar coercion rules

    Numeric 1 has single set of coercion rules for array and Python
    numeric types.  This can cause unexpected and annoying problems
    during the calculation of an array expression.  Numeric 2 intends
    to overcome these problems by having two sets of coercion rules:
    one for arrays and Python numeric types, and another just for
    arrays.

    2.  No savespace attribute

    The savespace attribute in Numeric 1 makes arrays with this
    attribute set take precedence over those that do not have it set.
    Numeric 2 will not have such an attribute and therefore normal
    array coercion rules will be in effect.

    3.  Slicing syntax returns a copy

    The slicing syntax in Numeric 1 returns a view into the original
    array.  The slicing behavior for Numeric 2 will be a copy.  You
    should use the ArrayView class to get a view into an array.

    4.  Boolean comparisons return a boolean array

    A comparison between arrays in Numeric 1 results in a Boolean
    scalar, because of current limitations in Python.  The advent of
    Rich Comparisons in Python 2.1 will allow an array of Booleans to
    be returned.

    5.  Type characters are deprecated

    Numeric 2 will have an ArrayType class composed of Type instances,
    for example Int8, Int16, Int32, and Int for signed integers.  The
    typecode scheme in Numeric 1 will be available for backward
    compatibility, but will be deprecated.

Appendices

    A.  Implicit sub-arrays iteration

    A computer animation is composed of a number of 2-D images or
    frames of identical shape.  By stacking these images into a single
    block of memory, a 3-D array is created.  Yet the operations to be
    performed are not meant for the entire 3-D array, but on the set
    of 2-D sub-arrays.  In most array languages, each frame has to be
    extracted, operated on, and then reinserted into the output array
    using a for-like loop.  The J language allows the programmer to
    perform such operations implicitly by having a rank for the frame
    and array.  By default these ranks will be the same during the
    creation of the array.  It was the intention of the Numeric 1
    developers to implement this feature, since it is based on the
    language J.  The Numeric 1 code has the required variables for
    implementing this behavior, but was never implemented.  We intend
    to implement implicit sub-array iteration in Numeric 2, if the
    array broadcasting rules found in Numeric 1 do not fully support
    this behavior.

Copyright

    This document is placed in the public domain.

Related PEPs

    PEP 207: Rich Comparisons
        by Guido van Rossum and David Ascher

    PEP 208: Reworking the Coercion Model
        by Neil Schemenauer and Marc-Andre' Lemburg

    PEP 211: Adding New Linear Algebra Operators to Python
        by Greg Wilson

    PEP 225: Elementwise/Objectwise Operators
        by Huaiyu Zhu

    PEP 228: Reworking Python's Numeric Model
        by Moshe Zadka

References

    [1] P. Greenfield 2000. private communication.


PEP 210 -- Decoupling the Interpreter Loop

PEP: | 210
Title: | Decoupling the Interpreter Loop
Version: | $Revision: 1.3 $
Author: | davida@activestate.com (David Ascher)
Status: | Deferred
Type: | Standards Track
Python-Version: | 2.1
Created: | 15-Jul-2000
Post-History: |


PEP 211 -- Adding New Linear Algebra Operators to Python

PEP: | 211
Title: | Adding New Linear Algebra Operators to Python
Version: | $Revision: 1.5 $
Author: | gvwilson@nevex.com (Greg Wilson)
Status: | Draft
Type: | Standards Track
Python-Version: | 2.1
Created: | 15-Jul-2000
Post-History: |

Introduction

    This PEP describes a conservative proposal to add linear algebra
    operators to Python 2.0.  It discusses why such operators are
    desirable, and why a minimalist approach should be adopted at this
    point.  This PEP summarizes discussions held in mailing list
    forums, and provides URLs for further information, where
    appropriate.  The CVS revision history of this file contains the
    definitive historical record.

Summary

    Add a single new infix binary operator '@' ("across"), and
    corresponding special methods "__across__()", "__racross__()", and
    "__iacross__()".  This operator will perform mathematical matrix
    multiplication on NumPy arrays, and generate cross-products when
    applied to built-in sequence types.  No existing operator
    definitions will be changed.

Background

    The first high-level programming language, Fortran, was invented
    to do arithmetic.  While this is now just a small part of
    computing, there are still many programmers who need to express
    complex mathematical operations in code.

    The most influential of Fortran's successors was APL [1].  Its
    author, Kenneth Iverson, designed the language as a notation for
    expressing matrix algebra, and received the 1980 Turing Award for
    his work.

    APL's operators supported both familiar algebraic operations, such
    as vector dot product and matrix multiplication, and a wide range
    of structural operations, such as stitching vectors together to
    create arrays.  Even by programming's standards, APL is
    exceptionally cryptic: many of its symbols did not exist on
    standard keyboards, and expressions have to be read right to left.

    Most subsequent work numerical languages, such as Fortran-90,
    MATLAB, and Mathematica, have tried to provide the power of APL
    without the obscurity.  Python's NumPy [2] has most of the
    features that users of such languages expect, but these are
    provided through named functions and methods, rather than
    overloaded operators.  This makes NumPy clumsier than most
    alternatives.

    The author of this PEP therefore consulted the developers of GNU
    Octave [3], an open source clone of MATLAB.  When asked how
    important it was to have infix operators for matrix solution,
    Prof. James Rawlings replied [4]:

        I DON'T think it's a must have, and I do a lot of matrix
        inversion. I cannot remember if its A\b or b\A so I always
        write inv(A)*b instead. I recommend dropping \.

    Rawlings' feedback on other operators was similar.  It is worth
    noting in this context that notations such as "/" and "\" for
    matrix solution were invented by programmers, not mathematicians,
    and have not been adopted by the latter.

    Based on this discussion, and feedback from classes at the US
    national laboratories and elsewhere, we recommend only adding a
    matrix multiplication operator to Python at this time.  If there
    is significant user demand for syntactic support for other
    operations, these can be added in a later release.

Requirements

    The most important requirement is minimal impact on existing
    Python programs and users: the proposal must not break existing
    code (except possibly NumPy).

    The second most important requirement is the ability to handle all
    common cases cleanly and clearly.  There are nine such cases:

        |5 6| *   9   = |45 54|      MS: matrix-scalar multiplication
        |7 8|           |63 72|

          9   * |5 6| = |45 54|      SM: scalar-matrix multiplication
                |7 8|   |63 72|

        |2 3| * |4 5| = |8 15|       VE: vector elementwise multiplication

        |2 3| *  |4|  =   23         VD: vector dot product
                 |5|

         |2|  * |4 5| = | 8 10|      VO: vector outer product
         |3|            |12 15|

        |1 2| * |5 6| = | 5 12|      ME: matrix elementwise multiplication
        |3 4|   |7 8|   |21 32|

        |1 2| * |5 6| = |19 22|      MM: mathematical matrix multiplication
        |3 4|   |7 8|   |43 50|

        |1 2| * |5 6| = |19 22|      VM: vector-matrix multiplication
                |7 8|

        |5 6| *  |1|  =   |17|       MV: matrix-vector multiplication
        |7 8|    |2|      |23|

    Note that 1-dimensional vectors are treated as rows in VM, as
    columns in MV, and as both in VD and VO.  Both are special cases
    of 2-dimensional matrices (Nx1 and 1xN respectively).  We will
    therefore define the new operator only for 2-dimensional arrays,
    and provide an easy (and efficient) way for users to treat
    1-dimensional structures as 2-dimensional.

    Third, we must avoid confusion between Python's notation and those
    of MATLAB and Fortran-90.  In particular, mathematical matrix
    multiplication (case MM) should not be represented as '.*', since:

    (a) MATLAB uses prefix-'.' forms to mean 'elementwise', and raw
        forms to mean "mathematical"; and

    (b) even if the Python parser can be taught how to handle dotted
        forms, '1.*A' will still be visually ambiguous.

Proposal

    The meanings of all existing operators will be unchanged.  In
    particular, 'A*B' will continue to be interpreted elementwise.
    This takes care of the cases MS, SM, VE, and ME, and ensures
    minimal impact on existing programs.

    A new operator '@' (pronounced "across") will be added to Python,
    along with special methods "__across__()", "__racross__()", and
    "__iacross__()", with the usual semantics.  (We recommend using
    "@", rather than the times-like "><", because of the ease with
    which the latter could be mis-typed as inequality "<>".)

    No new operators will be defined to mean "solve a set of linear
    equations", or "invert a matrix".

    (Optional) When applied to sequences, the "@" operator will return
    a tuple of tuples containing the cross-product of their elements
    in left-to-right order:

    >>> [1, 2] @ (3, 4)
    ((1, 3), (1, 4), (2, 3), (2, 4))

    >>> [1, 2] @ (3, 4) @ (5, 6)
    ((1, 3, 5), (1, 3, 6),
     (1, 4, 5), (1, 4, 6),
     (2, 3, 5), (2, 3, 6),
     (2, 4, 5), (2, 4, 6))

    This will require the same kind of special support from the parser
    as chained comparisons (such as "a<b<c<=d").  However, it will
    permit:

    >>> for (i, j) in [1, 2] @ [3, 4]:
    >>>     print i, j
    1 3
    1 4
    2 3
    2 4

    as a short-hand for the common nested loop idiom:

    >>> for i in [1, 2]:
    >>>    for j in [3, 4]:
    >>>        print i, j

    Response to the 'lockstep loop' questionnaire [5] indicated that
    newcomers would be comfortable with this (so comfortable, in fact,
    that most of them interpreted most multi-loop 'zip' syntaxes [6]
    as implementing single-stage nesting).

Alternatives

    01. Don't add new operators.

    Python is not primarily a numerical language; it may not be worth
    complexifying it for this special case.  NumPy's success is proof
    that users can and will use functions and methods for linear
    algebra.  However, support for real matrix multiplication is
    frequently requested, as:

    * functional forms are cumbersome for lengthy formulas, and do not
      respect the operator precedence rules of conventional mathematics;
      and

    * method forms are asymmetric in their operands.

    What's more, the proposed semantics for "@" for built-in sequence
    types would simplify expression of a very common idiom (nested
    loops).  User testing during discussion of 'lockstep loops'
    indicated that both new and experienced users would understand
    this immediately.

    02. Introduce prefixed forms of all existing operators, such as
        "~*" and "~+", as proposed in PEP 0225 [7].

    This proposal would duplicate all built-in mathematical operators
    with matrix equivalents, as in numerical languages such as
    MATLAB.  Our objections to this are:

    * Python is not primarily a numerical programming language.  While
      the (self-selected) participants in the discussions that led to
      PEP 0225 may want all of these new operators, the majority of
      Python users would be indifferent.  The extra complexity they
      would introduce into the language therefore does not seem
      merited. (See also Rawlings' comments, quoted in the Background
      section, about these operators not being essential.)

    * The proposed syntax is difficult to read (i.e. passes the "low
      toner" readability test).

    03. Retain the existing meaning of all operators, but create a
        behavioral accessor for arrays, such that:

            A * B

        is elementwise multiplication (ME), but:

            A.m() * B.m()

        is mathematical multiplication (MM).  The method "A.m()" would
        return an object that aliased A's memory (for efficiency), but
        which had a different implementation of __mul__().

    This proposal was made by Moshe Zadka, and is also considered by
    PEP 0225 [7].  Its advantage is that it has no effect on the
    existing implementation of Python: changes are localized in the
    Numeric module.  The disadvantages are

    * The semantics of "A.m() * B", "A + B.m()", and so on would have
      to be defined, and there is no "obvious" choice for them.

    * Aliasing objects to trigger different operator behavior feels
      less Pythonic than either calling methods (as in the existing
      Numeric module) or using a different operator.  This PEP is
      primarily about look and feel, and about making Python more
      attractive to people who are not already using it.

Related Proposals

    0207 :  Rich Comparisons

            It may become possible to overload comparison operators
            such as '<' so that an expression such as 'A < B' returns
            an array, rather than a scalar value.

    0209 :  Adding Multidimensional Arrays

            Multidimensional arrays are currently an extension to
            Python, rather than a built-in type.

    0225 :  Elementwise/Objectwise Operators

            A larger proposal that addresses the same subject, but
            which proposes many more additions to the language.

Acknowledgments

    I am grateful to Huaiyu Zhu [8] for initiating this discussion,
    and for some of the ideas and terminology included below.

References

    [1] http://www.acm.org/sigapl/whyapl.htm
    [2] http://numpy.sourceforge.net
    [3] http://bevo.che.wisc.edu/octave/
    [4] http://www.egroups.com/message/python-numeric/4
    [5] http://www.python.org/pipermail/python-dev/2000-July/013139.html
    [6] PEP-0201.txt "Lockstep Iteration"
    [7] http://www.python.org/pipermail/python-list/2000-August/112529.html

Appendix: NumPy

    NumPy will overload "@" to perform mathematical multiplication of
    arrays where shapes permit, and to throw an exception otherwise.
    Its implementation of "@" will treat built-in sequence types as if
    they were column vectors.  This takes care of the cases MM and MV.

    An attribute "T" will be added to the NumPy array type, such that
    "m.T" is:

    (a) the transpose of "m" for a 2-dimensional array

    (b) the 1xN matrix transpose of "m" if "m" is a 1-dimensional
        array; or

    (c) a runtime error for an array with rank >= 3.

    This attribute will alias the memory of the base object.  NumPy's
    "transpose()" function will be extended to turn built-in sequence
    types into row vectors.  This takes care of the VM, VD, and VO
    cases.  We propose an attribute because:

    (a) the resulting notation is similar to the 'superscript T' (at
        least, as similar as ASCII allows), and

    (b) it signals that the transposition aliases the original object.

    NumPy will define a value "inv", which will be recognized by the
    exponentiation operator, such that "A ** inv" is the inverse of
    "A".  This is similar in spirit to NumPy's existing "newaxis"
    value.


PEP 212 -- Loop Counter Iteration

PEP: | 212
Title: | Loop Counter Iteration
Version: | $Revision: 1.4 $
Author: | nowonder@nowonder.de (Peter Schneider-Kamp)
Status: | Draft
Type: | Standards Track
Created: | 22-Aug-2000
Python-Version: | 2.1

Introduction

    This PEP describes the often proposed feature of exposing the loop
    counter in for-loops.  This PEP tracks the status and ownership of
    this feature.  It contains a description of the feature and
    outlines changes necessary to support the feature.  This PEP
    summarizes discussions held in mailing list forums, and provides
    URLs for further information, where appropriate.  The CVS revision
    history of this file contains the definitive historical record.

Motivation

    Standard for-loops in Python iterate over the elements of a
    sequence[1].  Often it is desirable to loop over the indices or
    both the elements and the indices instead.

    The common idioms used to accomplish this are unintuitive.  This
    PEP proposes two different ways of exposing the indices.

Loop counter iteration

    The current idiom for looping over the indices makes use of the
    built-in 'range' function:

        for i in range(len(sequence)):
            # work with index i

    Looping over both elements and indices can be achieved either by the
    old idiom or by using the new 'zip' built-in function[2]:

        for i in range(len(sequence)):
            e = sequence[i]
            # work with index i and element e

    or

        for i, e in zip(range(len(sequence)), sequence):
            # work with index i and element e

The Proposed Solutions

    There are three solutions that have been discussed.  One adds a
    non-reserved keyword, the other adds two built-in functions.
    A third solution adds methods to sequence objects.

Non-reserved keyword 'indexing'

    This solution would extend the syntax of the for-loop by adding
    an optional '<variable> indexing' clause which can also be used
    instead of the '<variable> in' clause..

    Looping over the indices of a sequence would thus become:

        for i indexing sequence:
            # work with index i

    Looping over both indices and elements would similarly be:

        for i indexing e in sequence:
            # work with index i and element e

Built-in functions 'indices' and 'irange'

    This solution adds two built-in functions 'indices' and 'irange'.
    The semantics of these can be described as follows:

        def indices(sequence):
            return range(len(sequence))

        def irange(sequence):
            return zip(range(len(sequence)), sequence)

    These functions could be implemented either eagerly or lazily and
    should be easy to extend in order to accept more than one sequence
    argument.

    The use of these functions would simplify the idioms for looping
    over the indices and over both elements and indices:

        for i in indices(sequence):
            # work with index i

        for i, e in irange(sequence):
            # work with index i and element e

Methods for sequence objects

    This solution proposes the addition of 'indices', 'items'
    and 'values' methods to sequences, which enable looping over
    indices only, both indices and elements, and elements only
    respectively.

    This would immensely simplify the idioms for looping over indices
    and for looping over both elements and indices:

        for i in sequence.indices():
            # work with index i

        for i, e in sequence.items():
            # work with index i and element e

    Additionally it would allow to do looping over the elements
    of sequences and dicitionaries in a consistent way:

        for e in sequence_or_dict.values():
            # do something with element e

Implementations

    For all three solutions some more or less rough patches exist
    as patches at SourceForge:

        'for i indexing a in l': exposing the for-loop counter[3]
        add indices() and irange() to built-ins[4]
        add items() method to listobject[5]

    All of them have been pronounced on and rejected by the BDFL.

    Note that the 'indexing' keyword is only a NAME in the
    grammar and so does not hinder the general use of 'indexing'.

Backward Compatibility Issues

    As no keywords are added and the semantics of existing code
    remains unchanged, all three solutions can be implemented
    without breaking existing code.

Copyright

    This document has been placed in the public domain.

References

    [1] http://www.python.org/doc/current/ref/for.html
    [2] Lockstep Iteration, pep-0201.txt
    [3] http://sourceforge.net/patch/download.php?id=101138
    [4] http://sourceforge.net/patch/download.php?id=101129
    [5] http://sourceforge.net/patch/download.php?id=101178


PEP 213 -- Attribute Access Handlers

PEP: | 213
Title: | Attribute Access Handlers
Version: | $Revision: 1.4 $
Author: | paul@prescod.net (Paul Prescod)
Status: | Draft
Type: | Standards Track
Python-Version: | 2.1
Created: | 21-Jul-2000
Post-History: |

Introduction

     It is possible (and even relatively common) in Python code and
     in extension modules to "trap" when an instance's client code
     attempts to set an attribute and execute code instead. In other
     words it is possible to allow users to use attribute assignment/
     retrieval/deletion syntax even though the underlying implementation
     is doing some computation rather than directly modifying a
     binding.

     This PEP describes a feature that makes it easier, more efficient
     and safer to implement these handlers for Python instances.

Justification

    Scenario 1:

        You have a deployed class that works on an attribute named
        "stdout". After a while, you think it would be better to
        check that stdout is really an object with a "write" method
        at the moment of assignment. Rather than change to a
        setstdout method (which would be incompatible with deployed
        code) you would rather trap the assignment and check the
        object's type.

    Scenario 2:

        You want to be as compatible as possible with an object
        model that has a concept of attribute assignment. It could
        be the W3C Document Object Model or a particular COM
        interface (e.g. the PowerPoint interface). In that case
        you may well want attributes in the model to show up as
        attributes in the Python interface, even though the
        underlying implementation may not use attributes at all.

    Scenario 3:

        A user wants to make an attribute read-only.

    In short, this feature allows programmers to separate the
    interface of their module from the underlying implementation
    for whatever purpose. Again, this is not a new feature but
    merely a new syntax for an existing convention.

Current Solution

    To make some attributes read-only:

    class foo:
       def __setattr__( self, name, val ):
          if name=="readonlyattr":
             raise TypeError
          elif name=="readonlyattr2":
             raise TypeError
          ...
          else:
             self.__dict__["name"]=val

     This has the following problems:

     1. The creator of the method must be intimately aware of whether
        somewhere else in the class hiearchy __setattr__ has also been
        trapped for any particular purpose. If so, she must specifically
        call that method rather than assigning to the dictionary. There
        are many different reasons to overload __setattr__ so there is a
        decent potential for clashes. For instance object database
        implementations often overload setattr for an entirely unrelated
        purpose.

     2. The string-based switch statement forces all attribute handlers
        to be specified in one place in the code. They may then dispatch
        to task-specific methods (for modularity) but this could cause
        performance problems.

     3. Logic for the setting, getting and deleting must live in
        __getattr__, __setattr__ and __delattr__. Once again, this can be
        mitigated through an extra level of method call but this is
        inefficient.

Proposed Syntax

    Special methods should declare themselves with declarations of the
    following form:

    class x:
        def __attr_XXX__(self, op, val ):
            if op=="get":
                return someComputedValue(self.internal)
            elif op=="set":
                self.internal=someComputedValue(val)
            elif op=="del":
                del self.internal

    Client code looks like this:

    fooval=x.foo
    x.foo=fooval+5
    del x.foo

Semantics

     Attribute references of all three kinds should call the method.
     The op parameter can be "get"/"set"/"del". Of course this string
     will be interned so the actual checks for the string will be
     very fast.

     It is disallowed to actually have an attribute named XXX in the
     same instance as a method named __attr_XXX__.

     An implementation of __attr_XXX__ takes precedence over an
     implementation of __getattr__ based on the principle that
     __getattr__ is supposed to be invoked only after finding an
     appropriate attribute has failed.

     An implementation of __attr_XXX__ takes precedence over an
     implementation of __setattr__ in order to be consistent. The
     opposite choice seems fairly feasible also, however. The same
     goes for __del_y__.

Proposed Implementation

    There is a new object type called an attribute access handler.
    Objects of this type have the following attributes:

       name (e.g. XXX, not __attr__XXX__
       method (pointer to a method object

    In PyClass_New, methods of the appropriate form will be detected and
    converted into objects (just like unbound method objects). These are
    stored in the class __dict__ under the name XXX. The original method
    is stored as an unbound method under its original name.

    If there are any attribute access handlers in an instance at all,
    a flag is set. Let's call it "I_have_computed_attributes" for
    now. Derived classes inherit the flag from base classes. Instances
    inherit the flag from classes.

    A get proceeds as usual until just before the object is returned.
    In addition to the current check whether the returned object is a
    method it would also check whether a returned object is an access
    handler. If so, it would invoke the getter method and return
    the value. To remove an attribute access handler you could directly
    fiddle with the dictionary.

    A set proceeds by checking the "I_have_computed_attributes" flag. If
    it is not set, everything proceeds as it does today. If it is set
    then we must do a dictionary get on the requested object name. If it
    returns an attribute access handler then we call the setter function
    with the value. If it returns any other object then we discard the
    result and continue as we do today. Note that having an attribute
    access handler will mildly affect attribute "setting" performance for
    all sets on a particular instance, but no more so than today, using
    __setattr__. Gets are more efficient than they are today with
    __getattr__.

    The I_have_computed_attributes flag is intended to eliminate the
    performance degradation of an extra "get" per "set" for objects not
    using this feature. Checking this flag should have miniscule
    performance implications for all objects.

    The implementation of delete is analogous to the implementation
    of set.

Caveats

    1. You might note that I have not proposed any logic to keep
       the I_have_computed_attributes flag up to date as attributes
       are added and removed from the instance's dictionary. This is
       consistent with current Python. If you add a __setattr__ method
       to an object after it is in use, that method will not behave as
       it would if it were available at "compile" time. The dynamism is
       arguably not worth the extra implementation effort. This snippet
       demonstrates the current behavior:

    >>> def prn(*args):print args
    >>> class a:
    ...    __setattr__=prn
        >>> a().foo=5
    (<__main__.a instance at 882890>, 'foo', 5)

    >>> class b: pass
    >>> bi=b()
    >>> bi.__setattr__=prn
    >>> b.foo=5

     2. Assignment to __dict__["XXX"] can overwrite the attribute
    access handler for __attr_XXX__. Typically the access handlers will
        store information away in private __XXX variables

     3. An attribute access handler that attempts to call setattr or getattr
        on the object itself can cause an infinite loop (as with __getattr__)
        Once again, the solution is to use a special (typically private)
        variable such as __XXX.


PEP 214 -- Extended Print Statement

PEP: | 214
Title: | Extended Print Statement
Version: | $Revision: 1.11 $
Author: | barry@digicool.com (Barry A. Warsaw)
Python-Version: | 2.0
Status: | Final
Created: | 24-Jul-2000
Post-History: | 16-Aug-2000

Introduction

    This PEP describes a syntax to extend the standard `print'
    statement so that it can be used to print to any file-like object,
    instead of the default sys.stdout.  This PEP tracks the status and
    ownership of this feature.  It contains a description of the
    feature and outlines changes necessary to support the feature.
    This PEP summarizes discussions held in mailing list forums, and
    provides URLs for further information, where appropriate.  The CVS
    revision history of this file contains the definitive historical
    record.

Proposal

    This proposal introduces a syntax extension to the print
    statement, which allows the programmer to optionally specify the
    output file target.  An example usage is as follows:

        print >> mylogfile, 'this message goes to my log file'

    Formally, the syntax of the extended print statement is

        print_stmt: ... | '>>' test [ (',' test)+ [','] ] )

    where the ellipsis indicates the original print_stmt syntax
    unchanged.  In the extended form, the expression just after >>
    must yield an object with a write() method (i.e. a file-like
    object).  Thus these two statements are equivalent:

    print 'hello world'
        print >> sys.stdout, 'hello world'

    As are these two statements:

        print
        print >> sys.stdout

    These two statements are syntax errors:

        print ,
        print >> sys.stdout,

Justification

    `print' is a Python keyword and introduces the print statement as
    described in section 6.6 of the language reference manual[1].
    The print statement has a number of features:

    - it auto-converts the items to strings
    - it inserts spaces between items automatically
    - it appends a newline unless the statement ends in a comma

    The formatting that the print statement performs is limited; for
    more control over the output, a combination of sys.stdout.write(),
    and string interpolation can be used.

    The print statement by definition outputs to sys.stdout.  More
    specifically, sys.stdout must be a file-like object with a write()
    method, but it can be rebound to redirect output to files other
    than specifically standard output.  A typical idiom is

        save_stdout = sys.stdout
        try:
            sys.stdout = mylogfile
            print 'this message goes to my log file'
        finally:
            sys.stdout = save_stdout

    The problem with this approach is that the binding is global, and
    so affects every statement inside the try: clause.  For example,
    if we added a call to a function that actually did want to print
    to stdout, this output too would get redirected to the logfile.

    This approach is also very inconvenient for interleaving prints to
    various output streams, and complicates coding in the face of
    legitimate try/except or try/finally clauses.

Reference Implementation

    A reference implementation, in the form of a patch against the
    Python 2.0 source tree, is available on SourceForge's patch
    manager[2].  This approach adds two new opcodes, PRINT_ITEM_TO and
    PRINT_NEWLINE_TO, which simply pop the file like object off the
    top of the stack and use it instead of sys.stdout as the output
    stream.

    (This reference implementation has been adopted in Python 2.0.)

Alternative Approaches

    An alternative to this syntax change has been proposed (originally
    by Moshe Zadka) which requires no syntax changes to Python.  A
    writeln() function could be provided (possibly as a builtin), that
    would act much like extended print, with a few additional
    features.

    def writeln(*args, **kws):
        import sys
        file = sys.stdout
        sep = ' '
        end = '\n'
        if kws.has_key('file'):
        file = kws['file']
        del kws['file']
        if kws.has_key('nl'):
        if not kws['nl']:
            end = ' '
        del kws['nl']
        if kws.has_key('sep'):
        sep = kws['sep']
        del kws['sep']
        if kws:
        raise TypeError('unexpected keywords')
        file.write(sep.join(map(str, args)) + end)

    writeln() takes a three optional keyword arguments.  In the
    context of this proposal, the relevant argument is `file' which
    can be set to a file-like object with a write() method.  Thus

        print >> mylogfile, 'this goes to my log file'

    would be written as

        writeln('this goes to my log file', file=mylogfile)

    writeln() has the additional functionality that the keyword
    argument `nl' is a flag specifying whether to append a newline or
    not, and an argument `sep' which specifies the separator to output
    in between each item.

More Justification by the BDFL

    The proposal has been challenged on the newsgroup.  One series of
    challenges doesn't like '>>' and would rather see some other
    symbol.

    Challenge: Why not one of these?

        print in stderr items,....
        print + stderr items,.......
        print[stderr] items,.....
        print to stderr items,.....

    Response: If we want to use a special symbol (print <symbol>
    expression), the Python parser requires that it is not already a
    symbol that can start an expression -- otherwise it can't decide
    which form of print statement is used.  (The Python parser is a
    simple LL(1) or recursive descent parser.)

    This means that we can't use the "keyword only in context trick"
    that was used for "import as", because an identifier can start an
    expression.  This rules out +stderr, [sterr], and to stderr.  It
    leaves us with binary operator symbols and other miscellaneous
    symbols that are currently illegal here, such as 'import'.

    If I had to choose between 'print in file' and 'print >> file' I
    would definitely choose '>>'.  In part because 'in' would be a new
    invention (I know of no other language that uses it, while '>>' is
    used in sh, awk, Perl, and C++), in part because '>>', being
    non-alphabetic, stands out more so is more likely to catch the
    reader's attention.

    Challenge: Why does there have to be a comma between the file and
    the rest?

    Response: The comma separating the file from the following expression is
    necessary!  Of course you want the file to be an arbitrary
    expression, not just a single word.  (You definitely want to be
    able to write print >>sys.stderr.)  Without the expression the
    parser would't be able to distinguish where that expression ends
    and where the next one begins, e.g.

        print >>i +1, 2
        print >>a [1], 2
        print >>f (1), 2

    Challenge: Why do you need a syntax extension?  Why not
    writeln(file, item, ...)?

    Response: First of all, this is lacking a feature of the print
    statement: the trailing comma to print which suppresses the final
    newline.  Note that 'print a,' still isn't equivalent to
    'sys.stdout.write(a)' -- print inserts a space between items, and
    takes arbitrary objects as arguments; write() doesn't insert a
    space and requires a single string.

    When you are considering an extension for the print statement,
    it's not right to add a function or method that adds a new feature
    in one dimension (where the output goes) but takes away in another
    dimension (spaces between items, and the choice of trailing
    newline or not).  We could add a whole slew of methods or
    functions to deal with the various cases but that seems to add
    more confusion than necessary, and would only make sense if we
    were to deprecate the print statement altogether.

    I feel that this debate is really about whether print should have
    been a function or method rather than a statement.  If you are in
    the function camp, of course adding special syntax to the existing
    print statement is not something you like.  I suspect the
    objection to the new syntax comes mostly from people who already
    think that the print statement was a bad idea.  Am I right?

    About 10 years ago I debated with myself whether to make the most
    basic from of output a function or a statement; basically I was
    trying to decide between "print(item, ...)" and "print item, ...".
    I chose to make it a statement because printing needs to be taught
    very early on, and is very important in the programs that
    beginners write.  Also, because ABC, which lead the way for so
    many things, made it a statement.  In a move that's typical for
    the interaction between ABC and Python, I changed the name from
    WRITE to print, and reversed the convention for adding newlines
    from requiring extra syntax to add a newline (ABC used trailing
    slashes to indicate newlines) to requiring extra syntax (the
    trailing comma) to suppress the newline.  I kept the feature that
    items are separated by whitespace on output.

    Full example: in ABC,

        WRITE 1
        WRITE 2/

    has the same effect as

        print 1,
        print 2

    has in Python, outputting in effect "1 2\n".

    I'm not 100% sure that the choice for a statement was right (ABC
    had the compelling reason that it used statement syntax for
    anything with side effects, but Python doesn't have this
    convention), but I'm also not convinced that it's wrong.  I
    certainly like the economy of the print statement.  (I'm a rabid
    Lisp-hater -- syntax-wise, not semantics-wise! -- and excessive
    parentheses in syntax annoy me.  Don't ever write return(i) or
    if(x==y): in your Python code! :-)

    Anyway, I'm not ready to deprecate the print statement, and over
    the years we've had many requests for an option to specify the
    file.

    Challenge: Why not > instead of >>?

    Response: To DOS and Unix users, >> suggests "append", while >
    suggests "overwrite"; the semantics are closest to append.  Also,
    for C++ programmers, >> and << are I/O operators.

    Challenge: But in C++, >> is input and << is output!

    Response: doesn't matter; C++ clearly took it from Unix and
    reversed the arrows.  The important thing is that for output, the
    arrow points to the file.

    Challenge: Surely you can design a println() function can do all
    what print>>file can do; why isn't that enough?

    Response: I think of this in terms of a simple programming
    exercise.  Suppose a beginning programmer is asked to write a
    function that prints the tables of multiplication.  A reasonable
    solution is:

        def tables(n):
            for j in range(1, n+1):
                for i in range(1, n+1):
                    print i, 'x', j, '=', i*j
                print

    Now suppose the second exercise is to add printing to a different
    file.  With the new syntax, the programmer only needs to learn one
    new thing: print >> file, and the answer can be like this:

        def tables(n, file=sys.stdout):
            for j in range(1, n+1):
                for i in range(1, n+1):
                    print >> file, i, 'x', j, '=', i*j
                print >> file

    With only a print statement and a println() function, the
    programmer first has to learn about println(), transforming the
    original program to using println():

        def tables(n):
            for j in range(1, n+1):
                for i in range(1, n+1):
                    println(i, 'x', j, '=', i*j)
                println()

    and *then* about the file keyword argument:

        def tables(n, file=sys.stdout):
            for j in range(1, n+1):
                for i in range(1, n+1):
                    println(i, 'x', j, '=', i*j, file=sys.stdout)
                println(file=sys.stdout)

    Thus, the transformation path is longer:

        (1) print
        (2) print >> file

    vs.

        (1) print
        (2) println()
        (3) println(file=...)

    Note: defaulting the file argument to sys.stdout at compile time
    is wrong, because it doesn't work right when the caller assigns to
    sys.stdout and then uses tables() without specifying the file.
    This is a common problem (and would occur with a println()
    function too).  The standard solution so far has been:

        def tables(n, file=None):
            if file is None:
                file = sys.stdout
            for j in range(1, n+1):
                for i in range(1, n+1):
                    print >> file, i, 'x', j, '=', i*j
                print >> file

    I've added a feature to the implementation (which I would also
    recommend to println()) whereby if the file argument is None,
    sys.stdout is automatically used.  Thus,

        print >> None, foo bar

    (or, of course, print >> x where x is a variable whose value is
    None) means the same as

        print foo, bar

    and the tables() function can be written as follows:

        def tables(n, file=None):
            for j in range(1, n+1):
                for i in range(1, n+1):
                    print >> file, i, 'x', j, '=', i*j
                print >> file

    [XXX this needs more justification, and a section of its own]

References

    [1] http://www.python.org/doc/current/ref/print.html
    [2] http://sourceforge.net/patch/download.php?id=100970


PEP 215 -- String Interpolation

PEP: | 215
Title: | String Interpolation
Version: | $Revision: 1.3 $
Author: | ping@lfw.org (Ka-Ping Yee)
Status: | Draft
Type: | Standards Track
Python-Version: | 2.1
Created: | 24-Jul-2000
Post-History: |

Abstract

    This document proposes a string interpolation feature for Python
    to allow easier string formatting.  The suggested syntax change
    is the introduction of a '$' prefix that triggers the special
    interpretation of the '$' character within a string, in a manner
    reminiscent to the variable interpolation found in Unix shells,
    awk, Perl, or Tcl.

Copyright

    This document is in the public domain.

Specification

    Strings may be preceded with a '$' prefix that comes before the
    leading single or double quotation mark (or triplet) and before
    any of the other string prefixes ('r' or 'u').  Such a string is
    processed for interpolation after the normal interpretation of
    backslash-escapes in its contents.  The processing occurs just
    before the string is pushed onto the value stack, each time the
    string is pushed.  In short, Python behaves exactly as if '$'
    were a unary operator applied to the string.  The operation
    performed is as follows:

    The string is scanned from start to end for the '$' character
    (\x24 in 8-bit strings or \u0024 in Unicode strings).  If there
    are no '$' characters present, the string is returned unchanged.

    Any '$' found in the string, followed by one of the two kinds of
    expressions described below, is replaced with the value of the
    expression as evaluated in the current namespaces.  The value is
    converted with str() if the containing string is an 8-bit string,
    or with unicode() if it is a Unicode string.

    1.  A Python identifier optionally followed by any number of
        trailers, where a trailer consists of:
            - a dot and an identifier,
            - an expression enclosed in square brackets, or
            - an argument list enclosed in parentheses
        (This is exactly the pattern expressed in the Python grammar
        by "NAME trailer*", using the definitions in Grammar/Grammar.)

    2.  Any complete Python expression enclosed in curly braces.

    Two dollar-signs ("$$") are replaced with a single "$".

Examples

    Here is an example of an interactive session exhibiting the
    expected behaviour of this feature.

        >>> a, b = 5, 6
        >>> print $'a = $a, b = $b'
        a = 5, b = 6
        >>> $u'uni${a}ode'
        u'uni5ode'
        >>> print $'\$a'
        5
        >>> print $r'\$a'
        \5
        >>> print $'$$$a.$b'
        $5.6
        >>> print $'a + b = ${a + b}'
        a + b = 11
        >>> import sys
        >>> print $'References to $a: $sys.getrefcount(a)'
        References to 5: 15
        >>> print $"sys = $sys, sys = $sys.modules['sys']"
        sys = <module 'sys' (built-in)>, sys = <module 'sys' (built-in)>
        >>> print $'BDFL = $sys.copyright.split()[4].upper()'
        BDFL = GUIDO

Discussion

    '$' is chosen as the interpolation character within the
    string for the sake of familiarity, since it is already used
    for this purpose in many other languages and contexts.

    It is then natural to choose '$' as a prefix, since it is a
    mnemonic for the interpolation character.

    Trailers are permitted to give this interpolation mechanism
    even more power than the interpolation available in most other
    languages, while the expression to be interpolated remains
    clearly visible and free of curly braces.

    '$' works like an operator and could be implemented as an
    operator, but that prevents the compile-time optimization
    and presents security issues.  So, it is only allowed as a
    string prefix.

Security Issues

    "$" has the power to eval, but only to eval a literal.  As
    described here (a string prefix rather than an operator), it
    introduces no new security issues since the expressions to be
    evaluated must be literally present in the code.

Implementation

    The Itpl module at http://www.lfw.org/python/Itpl.py provides a
    prototype of this feature.  It uses the tokenize module to find
    the end of an expression to be interpolated, then calls eval()
    on the expression each time a value is needed.  In the prototype,
    the expression is parsed and compiled again each time it is
    evaluated.

    As an optimization, interpolated strings could be compiled
    directly into the corresponding bytecode; that is,

        $'a = $a, b = $b'

    could be compiled as though it were the expression

        ('a = ' + str(a) + ', b = ' + str(b))

    so that it only needs to be compiled once.


PEP 216 -- Docstring Format

PEP: | 216
Title: | Docstring Format
Version: | $Revision: 1.8 $
Author: | moshez@zadka.site.co.il (Moshe Zadka)
Status: | Draft
Type: | Informational
Created: | 31-Jul-2000

Abstract

    Named Python objects, such as modules, classes and functions, have a
    string attribute called __doc__. If the first expression inside
    the definition is a literal string, that string is assigned
    to the __doc__ attribute.

    The __doc__ attribute is called a documentation string, or docstring.
    It is often used to summarize the interface of the module, class or
    function. However, since there is no common format for documentation
    string, tools for extracting docstrings and transforming those into
    documentation in a standard format (e.g., DocBook) have not sprang
    up in abundance, and those that do exist are for the most part
    unmaintained and unused.

Perl Documentation

    In Perl, most modules are documented in a format called POD -- Plain
    Old Documentation. This is an easy-to-type, very low level format
    which integrates well with the Perl parser. Many tools exist to turn
    POD documentation into other formats: info, HTML and man pages, among
    others. However, in Perl, the information is not available at run-time.

Java Documentation

    In Java, special comments before classes and functions function to
    document the code. A program to extract these, and turn them into
    HTML documentation is called javadoc, and is part of the standard
    Java distribution. However, the only output format that is supported
    is HTML, and JavaDoc has a very intimate relationship with HTML.

Python Docstring Goals

    Python documentation string are easy to spot during parsing, and are
    also available to the runtime interpreter. This double purpose is
    a bit problematic, sometimes: for example, some are reluctant to have
    too long docstrings, because they do not want to take much space in
    the runtime. In addition, because of the current lack of tools, people
    read objects' docstrings by "print"ing them, so a tendancy to make them
    brief and free of markups has sprung up. This tendancy hinders writing
    better documentation-extraction tools, since it causes docstrings to
    contain little information, which is hard to parse.

High Level Solutions

    To counter the objection that the strings take up place in the running
    program, it is suggested that documentation extraction tools will
    concatenate a maximum prefix of string literals which appear in the
    beginning of a definition. The first of these will also be available
    in the interactive interpreter, so it should contain a few summary
    lines.

Docstring Format Goals

    These are the goals for the docstring format, as discussed ad neasum
    in the doc-sig.

    1. It must be easy to type with any standard text editor.
    2. It must be readable to the casual observer.
    3. It must not contain information which can be deduced from parsing
       the module.
    4. It must contain sufficient information so it can be converted
       to any reasonable markup format.
    5. It must be possible to write a module's entire documentation in
       docstrings, without feeling hampered by the markup language.

Docstring Contents

    For requirement 5. above, it is needed to specify what must be
    in docstrings.

    At least the following must be available:

    a. A tag that means "this is a Python ``something'', guess what"

    Example: In the sentence "The POP3 class", we need to markup "POP3"
    so. The parser will be able to guess it is a class from the contents
    of the poplib module, but we need to make it guess.

    b. Tags that mean "this is a Python class/module/class var/instance var..."

    Example: The usual Python idiom for singleton class A is to have _A
    as the class, and A a function which returns _A objects. It's usual
    to document the class, nonetheless, as being A. This requires the
    strength to say "The class A" and have A hyperlinked and marked-up
    as a class.

    c. An easy way to include Python source code/Python interactive sessions
    d. Emphasis/bold
    e. List/tables

Docstring Basic Structure

    The documentation strings will be in StructuredTextNG
    (http://www.zope.org/Members/jim/StructuredTextWiki/StructuredTextNG)
    Since StructuredText is not yet strong enough to handle (a) and (b)
    above, we will need to extend it. I suggest using
    '[<optional description>:python identifier]'.
    E.g.: [class:POP3], [:POP3.list], etc. If the description is missing,
    a guess will be made from the text.

Unresolved Issues

    Is there a way to escape characters in ST? If so, how?
    (example: * at the beginning of a line without being bullet symbol)

    Is my suggestion above for Python symbols compatible with ST-NG?
    How hard would it be to extend ST-NG to support it?

    How do we describe input and output types of functions?

    What additional constraint do we enforce on each docstring?
    (module/class/function)?

    What are the guesser rules?

Rejected Suggestions

    XML -- it's very hard to type, and too cluttered to read it
           comfortably.


PEP 217 -- Display Hook for Interactive Use

PEP: | 217
Title: | Display Hook for Interactive Use
Version: | $Revision: 1.5 $
Author: | peps@zadka.site.co.il (Moshe Zadka)
Status: | Final
Type: | Standards Track
Python-Version: | 2.1
Created: | 31-Jul-2000
Post-History: |

Abstract

    Python's interactive mode is one of the implementation's great
    strengths -- being able to write expressions on the command line
    and get back a meaningful output.  However, the output function
    cannot be all things to all people, and the current output
    function too often falls short of this goal.  This PEP describes a
    way to provides alternatives to the built-in display function in
    Python, so users will have control over the output from the
    interactive interpreter.

Interface

    The current Python solution has worked for many users, and this
    should not break it. Therefore, in the default configuration,
    nothing will change in the REPL loop. To change the way the
    interpreter prints interactively entered expressions, users
    will have to rebind sys.displayhook to a callable object.
    The result of calling this object with the result of the
    interactively entered expression should be print-able,
    and this is what will be printed on sys.stdout.

Solution

    The bytecode PRINT_EXPR will call sys.displayhook(POP())
    A displayhook() will be added to the sys builtin module, which is
    equivalent to

    import __builtin__
    def displayhook(o):
        if o is None:
            return
        __builtin__._ = None
        print `o`
        __builtin__._ = o

Jython Issues

    The method Py.printResult will be similarily changed.


PEP 218 -- Adding a Built-In Set Object Type

PEP: | 218
Title: | Adding a Built-In Set Object Type
Version: | $Revision: 1.5 $
Author: | gvwilson@ddj.com (Greg Wilson)
Status: | Draft
Type: | Standards Track
Python-Version: | 2.2
Created: | 31-Jul-2000
Post-History: |

Introduction

    This PEP proposes adding a Set module to the standard Python
    library, and to then make sets a built-in Python type if that
    module is widely used.  After explaining why sets are desirable,
    and why the common idiom of using dictionaries in their place is
    inadequate, we describe how we intend built-in sets to work, and
    then how the preliminary Set module will behave.  The penultimate
    section discusses the mutability (or otherwise) of sets and set
    elements, and the solution which the Set module will implement.
    The last section then looks at alternatives that were considered,
    but discarded.

Rationale

    Sets are a fundamental mathematical structure, and are very
    commonly used in algorithm specifications.  They are much less
    frequently used in implementations, even when they are the "right"
    structure.  Programmers frequently use lists instead, even when
    the ordering information in lists is irrelevant, and by-value
    lookups are frequent.  (Most medium-sized C programs contain a
    depressing number of start-to-end searches through malloc'd
    vectors to determine whether particular items are present or
    not...)

    Programmers are often told that they can implement sets as
    dictionaries with "don't care" values.  Items can be added to
    these "sets" by assigning the "don't care" value to them;
    membership can be tested using "dict.has_key"; and items can be
    deleted using "del".  However, the other main operations on sets
    (union, intersection, and difference) are not directly supported
    by this representation, since their meaning is ambiguous for
    dictionaries containing key/value pairs.

Long-Term Proposal

    The long-term goal of this PEP is to add a built-in set type to
    Python.  This type will be an unordered collection of unique
    values, just as a dictionary is an unordered collection of
    key/value pairs.  Constant sets will be represented using the
    usual mathematical notation, so that "{1, 2, 3}" will be a set of
    three integers.

    In order to avoid ambiguity, the empty set will be written "{-}",
    rather than "{}" (which is already used to represent empty
    dictionaries).  We feel that this notation is as reasonable as the
    use of "(3,)" to represent single-element tuples; a more radical
    strategy is discussed in the "Alternatives" section, and more
    readable than the earlier proposal "{,}".

    Iteration and comprehension will be implemented in the obvious
    ways, so that:

        for x in S:

    will step through the elements of S in arbitrary order, while:

        {x**2 for x in S}

    will produce a set containing the squares of all elements in S,
    Membership will be tested using "in" and "not in", and basic set
    operations will be implemented by a mixture of overloaded
    operators:

        |               union
        &               intersection
        ^               symmetric difference
        -               asymmetric difference

    and methods:

        S.add(x)        Add "x" to the set.

        S.update(s)     Add all elements of sequence "s" to the set.

        S.remove(x)     Remove "x" from the set.  If "x" is not
                        present, this method raises a LookupError
                        exception.

        S.discard(x)    Remove "x" from the set if it is present, or
                        do nothing if it is not.

        S.popitem()     Remove and return an arbitrary element,
                        raising a LookupError if the element is not
                        present.

    and one new built-in conversion function:

        set(x)          Create a set containing the elements of the
                        collection "x".

    Notes:

    1. We propose using the bitwise operators "|&" for intersection
       and union.  While "+" for union would be intuitive, "*" for
       intersection is not (very few of the people asked guessed what
       it did correctly).

    2. We considered using "+" to add elements to a set, rather than
       "add".  However, Guido van Rossum pointed out that "+" is
       symmetric for other built-in types (although "*" is not).  Use
       of "add" will also avoid confusion between that operation and
       set union.

    3. Sets raise "LookupError" exceptions, rather than "KeyError" or
       "ValueError", because set elements are neither keys nor values.

Short-Term Proposal

    In order to determine whether there is enough demand for sets to
    justify making them a built-in type, and to give users a chance to
    try out the semantics we propose for sets, our short-term proposal
    is to add a "Set" class to the standard Python library.  This
    class will have the operators and methods described above; it will
    also have named methods corresponding to all of the operations: a
    "union" method for "|", and a "union_update" method for "|=", and
    so on.

    This class will use a dictionary internally to contain set values.
    In order to avoid having to duplicate values (e.g. for iteration
    through the set), the class will rely on the iterators which are
    scheduled to appear in Python 2.2.

    Tim Peters believes that the class's constructor should take a
    single sequence as an argument, and populate the set with that
    sequence's elements.  His argument is that in most cases,
    programmers will be created sets from pre-existing sequences, so
    that common case should be usual.  However, this would require
    users to remember an extra set of parentheses when initializing a
    set with known values:

    >>> Set((1, 2, 3, 4))       # case 1

    On the other hand, feedback from a small number of novice Python
    users (all of whom were very experienced with other languages)
    indicates that people will find a "parenthesis-free" syntax more
    natural:

    >>> Set(1, 2, 3, 4)         # case 2

    On the other, other hand, if Python does adopt a dictionary-like
    notation for sets in the future, then case 2 will become
    redundant.  We have therefore adopted the first strategy, in which
    the initializer takes a single sequence argument.

Mutability

    The most difficult question to resolve in this proposal was
    whether sets ought to be able to contain mutable elements.  A
    dictionary's keys must be immutable in order to support fast,
    reliable lookup.  While it would be easy to require set elements
    to be immutable, this would preclude sets of sets (which are
    widely used in graph algorithms and other applications).

    At Tim Peters' suggestion, we will implement the following
    compromise.  A set may only contain immutable elements, but is
    itself mutable *until* its hash code is calculated.  As soon as
    that happens, the set is "frozen", i.e. becomes immutable.  Thus,
    a set may be used as a dictionary key, or as a set element, but
    cannot be updated after this is done.  Peters reports that this
    behavior rarely causes problems in practice.

Alternatives

    An alternative to the notation "{-}" for the empty set would be to
    re-define "{}" to be the empty collection, rather than the empty
    dictionary.  Operations which made this object non-empty would
    silently convert it to either a dictionary or a set; it would then
    retain that type for the rest of its existence.  This idea was
    rejected because of its potential impact on existing Python
    programs.  A similar proposal to modify "dict.keys" and
    "dict.values" to return sets, rather than lists, was rejected for
    the same reasons.

Copyright

    This document has been placed in the Public Domain.


PEP 219 -- Stackless Python

PEP: | 219
Title: | Stackless Python
Version: | $Revision: 1.5 $
Author: | gmcm@hypernet.com (Gordon McMillan)
Status: | Draft
Type: | Standards Track
Created: | 14-Aug-2000
Python-Version: | 2.1
Post-History: |

Introduction

    This PEP discusses changes required to core Python in order to
    efficiently support generators, microthreads and coroutines. It is
    related to PEP 220, which describes how Python should be extended
    to support these facilities. The focus of this PEP is strictly on
    the changes required to allow these extensions to work.

    While these PEPs are based on Christian Tismer's Stackless[1]
    implementation, they do not regard Stackless as a reference
    implementation.  Stackless (with an extension module) implements
    continuations, and from continuations one can implement
    coroutines, microthreads (as has been done by Will Ware[2]) and
    generators. But in more that a year, no one has found any other
    productive use of continuations, so there seems to be no demand
    for their support.

    However, Stackless support for continuations is a relatively minor
    piece of the implementation, so one might regard it as "a"
    reference implementation (rather than "the" reference
    implementation).

Background

    Generators and coroutines have been implemented in a number of
    languages in a number of ways. Indeed, Tim Peters has done pure
    Python implementations of generators[3] and coroutines[4] using
    threads (and a thread-based coroutine implementation exists for
    Java). However, the horrendous overhead of a thread-based
    implementation severely limits the usefulness of this approach.

    Microthreads (a.k.a "green" or "user" threads) and coroutines
    involve transfers of control that are difficult to accommodate in
    a language implementation based on a single stack. (Generators can
    be done on a single stack, but they can also be regarded as a very
    simple case of coroutines.)

    Real threads allocate a full-sized stack for each thread of
    control, and this is the major source of overhead. However,
    coroutines and microthreads can be implemented in Python in a way
    that involves almost no overhead.  This PEP, therefor, offers a
    way for making Python able to realistically manage thousands of
    separate "threads" of activity (vs. todays limit of perhaps dozens
    of separate threads of activity).

    Another justification for this PEP (explored in PEP 220) is that
    coroutines and generators often allow a more direct expression of
    an algorithm than is possible in today's Python.

Discussion

    The first thing to note is that Python, while it mingles
    interpreter data (normal C stack usage) with Python data (the
    state of the interpreted program) on the stack, the two are
    logically separate. They just happen to use the same stack.

    A real thread gets something approaching a process-sized stack
    because the implementation has no way of knowing how much stack
    space the thread will require. The stack space required for an
    individual frame is likely to be reasonable, but stack switching
    is an arcane and non-portable process, not supported by C.

    Once Python stops putting Python data on the C stack, however,
    stack switching becomes easy.

    The fundamental approach of the PEP is based on these two
    ideas. First, separate C's stack usage from Python's stack
    usage. Secondly, associate with each frame enough stack space to
    handle that frame's execution.

    In the normal usage, Stackless Python has a normal stack
    structure, except that it is broken into chunks. But in the
    presence of a coroutine / microthread extension, this same
    mechanism supports a stack with a tree structure.  That is, an
    extension can support transfers of control between frames outside
    the normal "call / return" path.

Problems

    The major difficulty with this approach is C calling Python. The
    problem is that the C stack now holds a nested execution of the
    byte-code interpreter. In that situation, a coroutine /
    microthread extension cannot be permitted to transfer control to a
    frame in a different invocation of the byte-code interpreter. If a
    frame were to complete and exit back to C from the wrong
    interpreter, the C stack could be trashed.

    The ideal solution is to create a mechanism where nested
    executions of the byte code interpreter are never needed. The easy
    solution is for the coroutine / microthread extension(s) to
    recognize the situation and refuse to allow transfers outside the
    current invocation.

    We can categorize code that involves C calling Python into two
    camps: Python's implementation, and C extensions. And hopefully we
    can offer a compromise: Python's internal usage (and C extension
    writers who want to go to the effort) will no longer use a nested
    invocation of the interpreter. Extensions which do not go to the
    effort will still be safe, but will not play well with coroutines
    / microthreads.

    Generally, when a recursive call is transformed into a loop, a bit
    of extra bookkeeping is required. The loop will need to keep it's
    own "stack" of arguments and results since the real stack can now
    only hold the most recent. The code will be more verbose, because
    it's not quite as obvious when we're done. While Stackless is not
    implemented this way, it has to deal with the same issues.

    In normal Python, PyEval_EvalCode is used to build a frame and
    execute it. Stackless Python introduces the concept of a
    FrameDispatcher. Like PyEval_EvalCode, it executes one frame. But
    the interpreter may signal the FrameDispatcher that a new frame
    has been swapped in, and the new frame should be executed. When a
    frame completes, the FrameDispatcher follows the back pointer to
    resume the "calling" frame.

    So Stackless transforms recursions into a loop, but it is not the
    FrameDispatcher that manages the frames. This is done by the
    interpreter (or an extension that knows what it's doing).

    The general idea is that where C code needs to execute Python
    code, it creates a frame for the Python code, setting its back
    pointer to the current frame. Then it swaps in the frame, signals
    the FrameDispatcher and gets out of the way. The C stack is now
    clean - the Python code can transfer control to any other frame
    (if an extension gives it the means to do so).

    In the vanilla case, this magic can be hidden from the programmer
    (even, in most cases, from the Python-internals programmer). Many
    situations present another level of difficulty, however.

    The map builtin function involves two obstacles to this
    approach. It cannot simply construct a frame and get out of the
    way, not just because there's a loop involved, but each pass
    through the loop requires some "post" processing. In order to play
    well with others, Stackless constructs a frame object for map
    itself.

    Most recursions of the interpreter are not this complex, but
    fairly frequently, some "post" operations are required. Stackless
    does not fix these situations because of amount of code changes
    required. Instead, Stackless prohibits transfers out of a nested
    interpreter. While not ideal (and sometimes puzzling), this
    limitation is hardly crippling.

Advantages

    For normal Python, the advantage to this approach is that C stack
    usage becomes much smaller and more predictable. Unbounded
    recursion in Python code becomes a memory error, instead of a
    stack error (and thus, in non-Cupertino operating systems,
    something that can be recovered from).  The price, of course, is
    the added complexity that comes from transforming recursions of
    the byte-code interpreter loop into a higher order loop (and the
    attendant bookkeeping involved).

    The big advantage comes from realizing that the Python stack is
    really a tree, and the frame dispatcher can transfer control
    freely between leaf nodes of the tree, thus allowing things like
    microthreads and coroutines.

References

    [1] http://www.stackless.com
    [2] http://world.std.com/~wware/uthread.html
    [3] Demo/threads/Generator.py in the source distribution
    [4] http://www.stackless.com/coroutines.tim.peters.html


PEP 220 -- Coroutines, Generators, Continuations

PEP: | 220
Title: | Coroutines, Generators, Continuations
Version: | $Revision: 1.1 $
Author: | gmcm@hypernet.com (Gordon McMillan)
Status: | Active
Type: | Information
Created: | 14-Aug-2000
Post-History: |

Abstract

    Demonstrates why the changes described in the stackless PEP are
    desirable.  A low-level continuations module exists.  With it,
    coroutines and generators and "green" threads can be written.  A
    higher level module that makes coroutines and generators easy to
    create is desirable (and being worked on).  The focus of this PEP
    is on showing how coroutines, generators, and green threads can
    simplify common programming problems.


PEP 221 -- Import As

PEP: | 221
Title: | Import As
Version: | $Revision: 1.5 $
Author: | thomas@xs4all.net (Thomas Wouters)
Status: | Final
Type: | Standards Track
Python-Version: | 2.0
Created: | 15-Aug-2000
Post-History: |

Introduction

    This PEP describes the `import as' proposal for Python 2.0.  This
    PEP tracks the status and ownership of this feature.  It contains
    a description of the feature and outlines changes necessary to
    support the feature.  The CVS revision history of this file
    contains the definitive historical record.

Rationale

    This PEP proposes an extention of Python syntax regarding the
    `import' and `from <module> import' statements.  These statements
    load in a module, and either bind that module to a local name, or
    binds objects from that module to a local name.  However, it is
    sometimes desirable to bind those objects to a different name, for
    instance to avoid name clashes.  This can currently be achieved
    using the following idiom:

        import os
        real_os = os
        del os

    And similarly for the `from ... import' statement:

        from os import fdopen, exit, stat
        os_fdopen = fdopen
        os_stat = stat
        del fdopen, stat

    The proposed syntax change would add an optional `as' clause to
    both these statements, as follows:

        import os as real_os
        from os import fdopen as os_fdopen, exit, stat as os_stat

    The `as' name is not intended to be a keyword, and some trickery
    has to be used to convince the CPython parser it isn't one.  For
    more advanced parsers/tokenizers, however, this should not be a
    problem.

    A slightly special case exists for importing sub-modules.  The
    statement

        import os.path

    stores the module `os' locally as `os', so that the imported
    submodule `path' is accessible as `os.path'.  As a result,

        import os.path as p

    stores `os.path', not `os', in `p'.  This makes it effectively the
    same as

        from os import path as p

Implementation details

    This PEP has been accepted, and the suggested code change has been
    checked in.  The patch can still be found in the SourceForge patch
    manager[1].  Currently, a NAME field is used in the grammar rather
    than a bare string, to avoid the keyword issue.  It introduces a
    new bytecode, IMPORT_STAR, which performs the `from module import
    *' behaviour, and changes the behaviour of the IMPORT_FROM
    bytecode so that it loads the requested name (which is always a
    single name) onto the stack, to be subsequently stored by a STORE
    opcode. As a result, all names explicitly imported now follow the
    `global' directives.

    The special case of `from module import *' remains a special case,
    in that it cannot accomodate an `as' clause, and that no STORE
    opcodes are generated; the objects imported are loaded directly
    into the local namespace. This also means that names imported in
    this fashion are always local, and do not follow the `global'
    directive.

    An additional change to this syntax has also been suggested, to
    generalize the expression given after the `as' clause.  Rather
    than a single name, it could be allowed to be any expression that
    yields a valid l-value; anything that can be assigned to.  The
    change to accomodate this is minimal, as the patch[2] proves, and
    the resulting generalization allows a number of new constructs
    that run completely parallel with other Python assignment
    constructs. However, this idea has been rejected by Guido, as
    `hypergeneralization'.

Copyright

    This document has been placed in the Public Domain.

References

    [1] http://sourceforge.net/patch/?func=detailpatch&patch_id=101135&group_id=5470

    [2] http://sourceforge.net/patch/?func=detailpatch&patch_id=101234&group_id=5470


PEP 222 -- Web Library Enhancements

PEP: | 222
Title: | Web Library Enhancements
Version: | $Revision: 1.9 $
Author: | akuchlin@mems-exchange.org (Andrew Kuchling)
Status: | Deferred
Type: | Standards Track
Python-Version: | 2.1
Created: | 18-Aug-2000
Post-History: | 22-Dec-2000

Abstract

    This PEP proposes a set of enhancements to the CGI development
    facilities in the Python standard library.  Enhancements might be
    new features, new modules for tasks such as cookie support, or
    removal of obsolete code.

    The original intent was to make improvements to Python 2.1.
    However, there seemed little interest from the Python community,
    and time was lacking, so this PEP has been deferred to some future
    Python release.

Open Issues

    This section lists changes that have been suggested, but about
    which no firm decision has yet been made.  In the final version of
    this PEP, this section should be empty, as all the changes should
    be classified as accepted or rejected.

    cgi.py: We should not be told to create our own subclass just so
    we can handle file uploads. As a practical matter, I have yet to
    find the time to do this right, so I end up reading cgi.py's temp
    file into, at best, another file. Some of our legacy code actually
    reads it into a second temp file, then into a final destination!
    And even if we did, that would mean creating yet another object
    with its __init__ call and associated overhead.

    cgi.py: Currently, query data with no `=' are ignored.  Even if
    keep_blank_values is set, queries like `...?value=&...' are
    returned with blank values but queries like `...?value&...' are
    completely lost.  It would be great if such data were made
    available through the FieldStorage interface, either as entries
    with None as values, or in a separate list.

    Utility function: build a query string from a list of 2-tuples

    Dictionary-related utility classes: NoKeyErrors (returns an empty
    string, never a KeyError), PartialStringSubstitution (returns
    the original key string, never a KeyError)

New Modules

    This section lists details about entire new packages or modules
    that should be added to the Python standard library.

    * fcgi.py : A new module adding support for the FastCGI protocol.
      Robin Dunn's code needs to be ported to Windows, though.

Major Changes to Existing Modules

    This section lists details of major changes to existing modules,
    whether in implementation or in interface.  The changes in this
    section therefore carry greater degrees of risk, either in
    introducing bugs or a backward incompatibility.

    The cgi.py module would be deprecated.  (XXX A new module or
    package name hasn't been chosen yet: 'web'?  'cgilib'?)

Minor Changes to Existing Modules

    This section lists details of minor changes to existing modules.
    These changes should have relatively small implementations, and
    have little risk of introducing incompatibilities with previous
    versions.

Rejected Changes

    The changes listed in this section were proposed for Python 2.1,
    but were rejected as unsuitable.  For each rejected change, a
    rationale is given describing why the change was deemed
    inappropriate.

    * An HTML generation module is not part of this PEP.  Several such
      modules exist, ranging from HTMLgen's purely programming
      interface to ASP-inspired simple templating to DTML's complex
      templating.  There's no indication of which templating module to
      enshrine in the standard library, and that probably means that
      no module should be so chosen.

    * cgi.py: Allowing a combination of query data and POST data.
      This doesn't seem to be standard at all, and therefore is
      dubious practice.

Proposed Interface

    XXX open issues: naming convention (studlycaps or
    underline-separated?); need to look at the cgi.parse*() functions
    and see if they can be simplified, too.

    Parsing functions: carry over most of the parse* functions from
    cgi.py

    # The Response class borrows most of its methods from Zope's
    # HTTPResponse class.

    class Response:
        """
        Attributes:
        status: HTTP status code to return
        headers: dictionary of response headers
        body: string containing the body of the HTTP response
        """

        def __init__(self, status=200, headers={}, body=""):
            pass

        def setStatus(self, status, reason=None):
            "Set the numeric HTTP response code"
            pass

        def setHeader(self, name, value):
            "Set an HTTP header"
            pass

        def setBody(self, body):
            "Set the body of the response"
            pass

        def setCookie(self, name, value,
                      path = '/',
                      comment = None,
                      domain = None,
                      max-age = None,
                      expires = None,
                      secure = 0
                      ):
            "Set a cookie"
            pass

        def expireCookie(self, name):
            "Remove a cookie from the user"
            pass

        def redirect(self, url):
            "Redirect the browser to another URL"
            pass

        def __str__(self):
            "Convert entire response to a string"
            pass

        def dump(self):
            "Return a string representation useful for debugging"
            pass

        # XXX methods for specific classes of error:serverError,
        # badRequest, etc.?

    class Request:

        """
        Attributes:

        XXX should these be dictionaries, or dictionary-like objects?
        .headers : dictionary containing HTTP headers
        .cookies : dictionary of cookies
        .fields  : data from the form
        .env     : environment dictionary
        """

        def __init__(self, environ=os.environ, stdin=sys.stdin,
                     keep_blank_values=1, strict_parsing=0):
            """Initialize the request object, using the provided environment
            and standard input."""
            pass

        # Should people just use the dictionaries directly?
        def getHeader(self, name, default=None):
            pass

        def getCookie(self, name, default=None):
            pass

        def getField(self, name, default=None):
            "Return field's value as a string (even if it's an uploaded file)"
            pass

        def getUploadedFile(self, name):
            """Returns a file object that can be read to obtain the contents
            of an uploaded file.  XXX should this report an error if the
            field isn't actually an uploaded file?  Or should it wrap
            a StringIO around simple fields for consistency?
            """

        def getURL(self, n=0, query_string=0):
            """Return the URL of the current request, chopping off 'n' path
            components from the right.  Eg. if the URL is
            "http://foo.com/bar/baz/quux", n=2 would return
            "http://foo.com/bar".  Does not include the query string (if
            any)
            """

        def getBaseURL(self, n=0):
            """Return the base URL of the current request, adding 'n' path
            components to the end to recreate more of the whole URL.

            Eg. if the request URL is
            "http://foo.com/q/bar/baz/qux", n=0 would return
            "http://foo.com/", and n=2 "http://foo.com/q/bar".

            Returned URL does not include the query string, if any.
            """

        def dump(self):
            "String representation suitable for debugging output"
            pass

        # Possibilities?  I don't know if these are worth doing in the
        # basic objects.
        def getBrowser(self):
            "Returns Mozilla/IE/Lynx/Opera/whatever"

        def isSecure(self):
            "Return true if this is an SSLified request"

    # Module-level function
    def wrapper(func, logfile=sys.stderr):
        """
        Calls the function 'func', passing it the arguments
        (request, response, logfile).  Exceptions are trapped and
        sent to the file 'logfile'.
        """
        # This wrapper will detect if it's being called from the command-line,
        # and if so, it will run in a debugging mode; name=value pairs
        # can be entered on standard input to set field values.
        # (XXX how to do file uploads in this syntax?)

Copyright

    This document has been placed in the public domain.


PEP 223 -- Change the Meaning of \x Escapes

PEP: | 223
Title: | Change the Meaning of \x Escapes
Version: | $Revision: 1.6 $
Author: | tim@digicool.com (Tim Peters)
Status: | Final
Type: | Standards Track
Python-Version: | 2.0
Created: | 20-Aug-2000
Post-History: | 23-Aug-2000

Abstract

    Change \x escapes, in both 8-bit and Unicode strings, to consume
    exactly the two hex digits following.  The proposal views this as
    correcting an original design flaw, leading to clearer expression
    in all flavors of string, a cleaner Unicode story, better
    compatibility with Perl regular expressions, and with minimal risk
    to existing code.

Syntax

    The syntax of \x escapes, in all flavors of non-raw strings, becomes

        \xhh

    where h is a hex digit (0-9, a-f, A-F).  The exact syntax in 1.5.2 is
    not clearly specified in the Reference Manual; it says

        \xhh...

    implying "two or more" hex digits, but one-digit forms are also
    accepted by the 1.5.2 compiler, and a plain \x is "expanded" to
    itself (i.e., a backslash followed by the letter x).  It's unclear
    whether the Reference Manual intended either of the 1-digit or
    0-digit behaviors.

Semantics

    In an 8-bit non-raw string,
        \xij
    expands to the character
        chr(int(ij, 16))
    Note that this is the same as in 1.6 and before.

    In a Unicode string,
        \xij
    acts the same as
        \u00ij
    i.e. it expands to the obvious Latin-1 character from the initial
    segment of the Unicode space.

    An \x not followed by at least two hex digits is a compile-time error,
    specifically ValueError in 8-bit strings, and UnicodeError (a subclass
    of ValueError) in Unicode strings.  Note that if an \x is followed by
    more than two hex digits, only the first two are "consumed".  In 1.6
    and before all but the *last* two were silently ignored.

Example

    In 1.5.2:

        >>> "\x123465"  # same as "\x65"
        'e'
        >>> "\x65"
        'e'
        >>> "\x1"
        '\001'
        >>> "\x\x"
        '\\x\\x'
        >>>

    In 2.0:

        >>> "\x123465" # \x12 -> \022, "3456" left alone
        '\0223456'
        >>> "\x65"
        'e'
        >>> "\x1"
        [ValueError is raised]
        >>> "\x\x"
        [ValueError is raised]
        >>>

History and Rationale

    \x escapes were introduced in C as a way to specify variable-width
    character encodings.  Exactly which encodings those were, and how many
    hex digits they required, was left up to each implementation.  The
    language simply stated that \x "consumed" *all* hex digits following,
    and left the meaning up to each implementation.  So, in effect, \x in C
    is a standard hook to supply platform-defined behavior.

    Because Python explicitly aims at platform independence, the \x escape
    in Python (up to and including 1.6) has been treated the same way
    across all platforms:  all *except* the last two hex digits were
    silently ignored.  So the only actual use for \x escapes in Python was
    to specify a single byte using hex notation.

    Larry Wall appears to have realized that this was the only real use for
    \x escapes in a platform-independent language, as the proposed rule for
    Python 2.0 is in fact what Perl has done from the start (although you
    need to run in Perl -w mode to get warned about \x escapes with fewer
    than 2 hex digits following -- it's clearly more Pythonic to insist on
    2 all the time).

    When Unicode strings were introduced to Python, \x was generalized so
    as to ignore all but the last *four* hex digits in Unicode strings.
    This caused a technical difficulty for the new regular expression engine:
    SRE tries very hard to allow mixing 8-bit and Unicode patterns and
    strings in intuitive ways, and it no longer had any way to guess what,
    for example, r"\x123456" should mean as a pattern:  is it asking to match
    the 8-bit character \x56 or the Unicode character \u3456?

    There are hacky ways to guess, but it doesn't end there.  The ISO C99
    standard also introduces 8-digit \U12345678 escapes to cover the entire
    ISO 10646 character space, and it's also desired that Python 2 support
    that from the start.  But then what are \x escapes supposed to mean?
    Do they ignore all but the last *eight* hex digits then?  And if less
    than 8 following in a Unicode string, all but the last 4?  And if less
    than 4, all but the last 2?

    This was getting messier by the minute, and the proposal cuts the
    Gordian knot by making \x simpler instead of more complicated.  Note
    that the 4-digit generalization to \xijkl in Unicode strings was also
    redundant, because it meant exactly the same thing as \uijkl in Unicode
    strings.  It's more Pythonic to have just one obvious way to specify a
    Unicode character via hex notation.

Development and Discussion

    The proposal was worked out among Guido van Rossum, Fredrik Lundh and
    Tim Peters in email.  It was subsequently explained and disussed on
    Python-Dev under subject "Go \x yourself", starting 2000-08-03.
    Response was overwhelmingly positive; no objections were raised.

Backward Compatibility

    Changing the meaning of \x escapes does carry risk of breaking existing
    code, although no instances of incompabitility have yet been discovered.
    The risk is believed to be minimal.

    Tim Peters verified that, except for pieces of the standard test suite
    deliberately provoking end cases, there are no instances of \xabcdef...
    with fewer or more than 2 hex digits following, in either the Python
    CVS development tree, or in assorted Python packages sitting on his
    machine.

    It's unlikely there are any with fewer than 2, because the Reference
    Manual implied they weren't legal (although this is debatable!).  If
    there are any with more than 2, Guido is ready to argue they were buggy
    anyway <0.9 wink>.

    Guido reported that the O'Reilly Python books *already* document that
    Python works the proposed way, likely due to their Perl editing
    heritage (as above, Perl worked (very close to) the proposed way from
    its start).

    Finn Bock reported that what JPython does with \x escapes is
    unpredictable today.  This proposal gives a clear meaning that can be
    consistently and easily implemented across all Python implementations.

Effects on Other Tools

    Believed to be none.  The candidates for breakage would mostly be
    parsing tools, but the author knows of none that worry about the
    internal structure of Python strings beyond the approximation "when
    there's a backslash, swallow the next character".  Tim Peters checked
    python-mode.el, the std tokenize.py and pyclbr.py, and the IDLE syntax
    coloring subsystem, and believes there's no need to change any of
    them.  Tools like tabnanny.py and checkappend.py inherit their immunity
    from tokenize.py.

Reference Implementation

    The code changes are so simple that a separate patch will not be produced.
    Fredrik Lundh is writing the code, is an expert in the area, and will
    simply check the changes in before 2.0b1 is released.

BDFL Pronouncements

    Yes, ValueError, not SyntaxError.  "Problems with literal interpretations
    traditionally raise 'runtime' exceptions rather than syntax errors."

Copyright

    This document has been placed in the public domain.


PEP 224 -- Attribute Docstrings

PEP: | 224
Title: | Attribute Docstrings
Version: | $Revision: 1.5 $
Author: | mal@lemburg.com (Marc-Andre Lemburg)
Status: | Rejected
Type: | Standards Track
Python-Version: | 2.1
Created: | 23-Aug-2000
Post-History: |

Introduction

    This PEP describes the "attribute docstring" proposal for Python
    2.0.  This PEP tracks the status and ownership of this feature.
    It contains a description of the feature and outlines changes
    necessary to support the feature.  The CVS revision history of
    this file contains the definitive historical record.

Rationale

    This PEP proposes a small addition to the way Python currently
    handles docstrings embedded in Python code.

    Python currently only handles the case of docstrings which appear
    directly after a class definition, a function definition or as
    first string literal in a module.  The string literals are added
    to the objects in question under the __doc__ attribute and are
    from then on available for introspection tools which can extract
    the contained information for help, debugging and documentation
    purposes.

    Docstrings appearing in locations other than the ones mentioned
    are simply ignored and don't result in any code generation.

    Here is an example:

        class C:
            "class C doc-string"

            a = 1
            "attribute C.a doc-string (1)"

            b = 2
            "attribute C.b doc-string (2)"

    The docstrings (1) and (2) are currently being ignored by the
    Python byte code compiler, but could obviously be put to good use
    for documenting the named assignments that precede them.

    This PEP proposes to also make use of these cases by proposing
    semantics for adding their content to the objects in which they
    appear under new generated attribute names.

    The original idea behind this approach which also inspired the
    above example was to enable inline documentation of class
    attributes, which can currently only be documented in the class's
    docstring or using comments which are not available for
    introspection.

Implementation

    Docstrings are handled by the byte code compiler as expressions.
    The current implementation special cases the few locations
    mentioned above to make use of these expressions, but otherwise
    ignores the strings completely.

    To enable use of these docstrings for documenting named
    assignments (which is the natural way of defining e.g. class
    attributes), the compiler will have to keep track of the last
    assigned name and then use this name to assign the content of the
    docstring to an attribute of the containing object by means of
    storing it in as a constant which is then added to the object's
    namespace during object construction time.

    In order to preserve features like inheritance and hiding of
    Python's special attributes (ones with leading and trailing double
    underscores), a special name mangling has to be applied which
    uniquely identifies the docstring as belonging to the name
    assignment and allows finding the docstring later on by inspecting
    the namespace.

    The following name mangling scheme achieves all of the above:

        __doc_<attributename>__

    To keep track of the last assigned name, the byte code compiler
    stores this name in a variable of the compiling structure.  This
    variable defaults to NULL.  When it sees a docstring, it then
    checks the variable and uses the name as basis for the above name
    mangling to produce an implicit assignment of the docstring to the
    mangled name.  It then resets the variable to NULL to avoid
    duplicate assignments.

    If the variable does not point to a name (i.e. is NULL), no
    assignments are made.  These will continue to be ignored like
    before.  All classical docstrings fall under this case, so no
    duplicate assignments are done.

    In the above example this would result in the following new class
    attributes to be created:

        C.__doc_a__ == "attribute C.a doc-string (1)"
        C.__doc_b__ == "attribute C.b doc-string (2)"

    A patch to the current CVS version of Python 2.0 which implements
    the above is available on SourceForge at [1].

Caveats of the Implementation

    Since the implementation does not reset the compiling structure
    variable when processing a non-expression, e.g. a function
    definition, the last assigned name remains active until either the
    next assignment or the next occurrence of a docstring.

    This can lead to cases where the docstring and assignment may be
    separated by other expressions:

        class C:
            "C doc string"

            b = 2

            def x(self):
                "C.x doc string"
                y = 3
                return 1

            "b's doc string"

    Since the definition of method "x" currently does not reset the
    used assignment name variable, it is still valid when the compiler
    reaches the docstring "b's doc string" and thus assigns the string
    to __doc_b__.

    A possible solution to this problem would be resetting the name
    variable for all non-expression nodes in the compiler.

Possible Problems

    Even though highly unlikely, attribute docstrings could get
    accidentally concatenated to the attribute's value:

    class C:
      x = "text" \
      "x's docstring"

    The trailing slash would cause the Python compiler to concatenate
    the attribute value and the docstring.

    A modern syntax highlighting editor would easily make this
    accident visible, though, and by simply inserting emtpy lines
    between the attribute definition and the docstring you can avoid
    the possible concatenation completely, so the problem is
    negligible.

    Another possible problem is that of using triple quoted strings as
    a way to uncomment parts of your code.

    If there happens to be an assignment just before the start of the
    comment string, then the compiler will treat the comment as
    docstring attribute and apply the above logic to it.

    Besides generating a docstring for an otherwise undocumented
    attribute there is no breakage.

Comments from our BDFL

    Early comments on the PEP from Guido:

        I "kinda" like the idea of having attribute docstrings (meaning
        it's not of great importance to me) but there are two things I
        don't like in your current proposal:

        1. The syntax you propose is too ambiguous: as you say,
        stand-alone string literal are used for other purposes and could
        suddenly become attribute docstrings.

        2. I don't like the access method either (__doc_<attrname>__).

    The author's reply:

        > 1. The syntax you propose is too ambiguous: as you say, stand-alone
        >    string literal are used for other purposes and could suddenly
        >    become attribute docstrings.

        This can be fixed by introducing some extra checks in the
        compiler to reset the "doc attribute" flag in the compiler
        struct.

        > 2. I don't like the access method either (__doc_<attrname>__).

        Any other name will do. It will only have to match these
        criteria:

        * must start with two underscores (to match __doc__)
        * must be extractable using some form of inspection (e.g. by using
          a naming convention which includes some fixed name part)
        * must be compatible with class inheritence (i.e. should be
          stored as attribute)

    Later on in March, Guido pronounced on this PEP in March 2001 (on
    python-dev). Here are his reasons for rejection mentioned in
    private mail to the author of this PEP:

        ...

        It might be useful, but I really hate the proposed syntax.

            a = 1
            "foo bar"
            b = 1

        I really have no way to know whether "foo bar" is a docstring
        for a or for b.

        ...

        You can use this convention:

            a = 1
            __doc_a__ = "doc string for a"

        This makes it available at runtime.

        > Are you completely opposed to adding attribute documentation
        > to Python or is it just the way the implementation works ? I
        > find the syntax proposed in the PEP very intuitive and many
        > other users on c.l.p and in private emails have supported it
        > at the time I wrote the PEP.

        It's not the implementation, it's the syntax.  It doesn't
        convey a clear enough coupling between the variable and the
        doc string.

Copyright

    This document has been placed in the Public Domain.

References

    [1] http://sourceforge.net/patch/?func=detailpatch&patch_id=101264&group_id=5470


PEP 225 -- Elementwise/Objectwise Operators

PEP: | 225
Title: | Elementwise/Objectwise Operators
Version: | $Revision: 1.2 $
Author: | hzhu@users.sourceforge.net (Huaiyu Zhu), gregory.lielens@fft.be (Gregory Lielens)
Status: | Draft
Type: | Standards Track
Python-Version: | 2.1
Created: | 19-Sep-2000
Post-History: |

Introduction

    This PEP describes a proposal to add new operators to Python which
    are useful for distinguishing elementwise and objectwise
    operations, and summarizes discussions in the news group
    comp.lang.python on this topic.  See Credits and Archives section
    at end.  Issues discussed here include:

    - Background.
    - Description of proposed operators and implementation issues.
    - Analysis of alternatives to new operators.
    - Analysis of alternative forms.
    - Compatibility issues
    - Description of wider extensions and other related ideas.

    A substantial portion of this PEP describes ideas that do not go
    into the proposed extension.  They are presented because the
    extension is essentially syntactic sugar, so its adoption must be
    weighed against various possible alternatives.  While many
    alternatives may be better in some aspects, the current proposal
    appears to be overall advantageous.

    The issues concerning elementwise-objectwise operations extends to
    wider areas than numerical computation.  This document also
    describes how the current proposal may be integrated with more
    general future extensions.

Background

    Python provides six binary infix math operators: + - * / % **
    hereafter generically represented by "op".  They can be overloaded
    with new semantics for user-defined classes.  However, for objects
    composed of homogeneous elements, such as arrays, vectors and
    matrices in numerical computation, there are two essentially
    distinct flavors of semantics.  The objectwise operations treat
    these objects as points in multidimensional spaces.  The
    elementwise operations treat them as collections of individual
    elements.  These two flavors of operations are often intermixed in
    the same formulas, thereby requiring syntactical distinction.

    Many numerical computation languages provide two sets of math
    operators.  For example, in MatLab, the ordinary op is used for
    objectwise operation while .op is used for elementwise operation.
    In R, op stands for elementwise operation while %op% stands for
    objectwise operation.

    In Python, there are other methods of representation, some of
    which already used by available numerical packages, such as

    - function:   mul(a,b)
    - method:     a.mul(b)
    - casting:    a.E*b

    In several aspects these are not as adequate as infix operators.
    More details will be shown later, but the key points are

    - Readability: Even for moderately complicated formulas, infix
      operators are much cleaner than alternatives.

    - Familiarity: Users are familiar with ordinary math operators.

    - Implementation: New infix operators will not unduly clutter
      Python syntax.  They will greatly ease the implementation of
      numerical packages.

    While it is possible to assign current math operators to one
    flavor of semantics, there is simply not enough infix operators to
    overload for the other flavor.  It is also impossible to maintain
    visual symmetry between these two flavors if one of them does not
    contain symbols for ordinary math operators.

Proposed extension

    - Six new binary infix operators ~+ ~- ~* ~/ ~% ~** are added to
      core Python.  They parallel the existing operators + - * / % **.

    - Six augmented assignment operators ~+= ~-= ~*= ~/= ~%= ~**= are
      added to core Python.  They parallel the operators += -= *= /=
      %= **= available in Python 2.0.

    - Operator ~op retains the syntactical properties of operator op,
      including precedence.

    - Operator ~op retains the semantical properties of operator op on
      built-in number types.

    - Operator ~op raise syntax error on non-number builtin types.
      This is temporary until the proper behavior can be agreed upon.

    - These operators are overloadable in classes with names that
      prepend "t" (for tilde) to names of ordinary math operators.
      For example, __tadd__ and __rtadd__ work for ~+ just as __add__
      and __radd__ work for +.

    - As with exiting operators, the __r*__() methods are invoked when
      the left operand does not provide the appropriate method.

    It is intended that one set of op or ~op is used for elementwise
    operations, the other for objectwise operations, but it is not
    specified which version of operators stands for elementwise or
    objectwise operations, leaving the decision to applications.

    The proposed implementation is to patch several files relating to
    the tokenizer, parser, grammar and compiler to duplicate the
    functionality of corresponding existing operators as necessary.
    All new semantics are to be implemented in the classes that
    overload them.

    The symbol ~ is already used in Python as the unary "bitwise not"
    operator.  Currently it is not allowed for binary operators.  The
    new operators are completely backward compatible.

Prototype Implementation

    Greg Lielens implemented the infix ~op as a patch against Python
    2.0b1 source[1].

    To allow ~ to be part of binary operators, the tokenizer would
    treat ~+ as one token.  This means that currently valid expression
    ~+1 would be tokenized as ~+ 1 instead of ~ + 1.  The parser would
    then treat ~+ as composite of ~ +.  The effect is invisible to
    applications.

    Notes about current patch:

    - It does not include ~op= operators yet.

    - The ~op behaves the same as op on lists, instead of raising
      exceptions.

    These should be fixed when the final version of this proposal is
    ready.

    - It reserves xor as an infix operator with the semantics
      equivalent to:

        def __xor__(a, b):
            if not b: return a
            elif not a: return b
            else: 0

   This preserves true value as much as possible, otherwise preserve
   left hand side value if possible.

   This is done so that bitwise operators could be regarded as
   elementwise logical operators in the future (see below).

Alternatives to adding new operators

    The discussions on comp.lang.python and python-dev mailing list
    explored many alternatives.  Some of the leading alternatives are
    listed here, using the multiplication operator as an example.

    1. Use function mul(a,b).

       Advantage:
       -  No need for new operators.

       Disadvantage:
       - Prefix forms are cumbersome for composite formulas.
       - Unfamiliar to the intended users.
       - Too verbose for the intended users.
       - Unable to use natural precedence rules.

    2. Use method call a.mul(b)

       Advantage:
       - No need for new operators.

       Disadvantage:
       - Asymmetric for both operands.
       - Unfamiliar to the intended users.
       - Too verbose for the intended users.
       - Unable to use natural precedence rules.

    3. Use "shadow classes".  For matrix class define a shadow array
       class accessible through a method .E, so that for matrices a
       and b, a.E*b would be a matrix object that is
       elementwise_mul(a,b).

       Likewise define a shadow matrix class for arrays accessible
       through a method .M so that for arrays a and b, a.M*b would be
       an array that is matrixwise_mul(a,b).

       Advantage:
       - No need for new operators.
       - Benefits of infix operators with correct precedence rules.
       - Clean formulas in applications.

       Disadvantage:
       - Hard to maintain in current Python because ordinary numbers
         cannot have user defined class methods; i.e. a.E*b will fail
         if a is a pure number.
       - Difficult to implement, as this will interfere with existing
         method calls, like .T for transpose, etc.
       - Runtime overhead of object creation and method lookup.
       - The shadowing class cannot replace a true class, because it
         does not return its own type.  So there need to be a M class
         with shadow E class, and an E class with shadow M class.
       - Unnatural to mathematicians.

    4. Implement matrixwise and elementwise classes with easy casting
       to the other class.  So matrixwise operations for arrays would
       be like a.M*b.M and elementwise operations for matrices would
       be like a.E*b.E.  For error detection a.E*b.M would raise
       exceptions.

       Advantage:
       - No need for new operators.
       - Similar to infix notation with correct precedence rules.

       Disadvantage:
       - Similar difficulty due to lack of user-methods for pure numbers.
       - Runtime overhead of object creation and method lookup.
       - More cluttered formulas
       - Switching of flavor of objects to facilitate operators
         becomes persistent.  This introduces long range context
         dependencies in application code that would be extremely hard
         to maintain.

    5. Using mini parser to parse formulas written in arbitrary
       extension placed in quoted strings.

       Advantage:
       - Pure Python, without new operators

       Disadvantage:
       - The actual syntax is within the quoted string, which does not
         resolve the problem itself.
       - Introducing zones of special syntax.
       - Demanding on the mini-parser.

    6. Introducing a single operator, such as @, for matrix
       multiplication.

       Advantage:
       - Introduces less operators

       Disadvantage:
       - The distinctions for operators like + - ** are equally
         important.  Their meaning in matrix or array-oriented
         packages would be reversed (see below).
       - The new operator occupies a special character.
       - This does not work well with more general object-element issues.

    Among these alternatives, the first and second are used in current
    applications to some extent, but found inadequate.  The third is
    the most favorite for applications, but it will incur huge
    implementation complexity.  The fourth would make applications
    codes very context-sensitive and hard to maintain.  These two
    alternatives also share significant implementational difficulties
    due to current type/class split.  The fifth appears to create more
    problems than it would solve.  The sixth does not cover the same
    range of applications.

Alternative forms of infix operators

    Two major forms and several minor variants of new infix operators
    were discussed:

    - Bracketed form

        (op)
        [op]
        {op}
        <op>
        :op:
        ~op~
        %op%

    - Meta character form

        .op
        @op
        ~op

      Alternatively the meta character is put after the operator.

    - Less consistent variations of these themes.  These are
      considered unfavorably.  For completeness some are listed here

        - Use @/ and /@ for left and right division
        - Use [*] and (*) for outer and inner products
        - Use a single operator @ for multiplication.

    - Use __call__ to simulate multiplication.
      a(b)  or (a)(b)

    Criteria for choosing among the representations include:

        - No syntactical ambiguities with existing operators.

        - Higher readability in actual formulas.  This makes the
          bracketed forms unfavorable.  See examples below.

        - Visually similar to existing math operators.

        - Syntactically simple, without blocking possible future
          extensions.

    With these criteria the overall winner in bracket form appear to
    be {op}.  A clear winner in the meta character form is ~op.
    Comparing these it appears that ~op is the favorite among them
    all.

    Some analysis are as follows:

        - The .op form is ambiguous: 1.+a would be different from 1 .+a

        - The bracket type operators are most favorable when standing
          alone, but not in formulas, as they interfere with visual
          parsing of parenthesis for precedence and function argument.
          This is so for (op) and [op], and somewhat less so for {op}
          and <op>.

        - The <op> form has the potential to be confused with < > and =

        - The @op is not favored because @ is visually heavy (dense,
          more like a letter): a@+b is more readily read as a@ + b
          than a @+ b.

        - For choosing meta-characters: Most of existing ASCII symbols
          have already been used.  The only three unused are @ $ ?.

Semantics of new operators

    There are convincing arguments for using either set of operators
    as objectwise or elementwise.  Some of them are listed here:

    1. op for element, ~op for object

       - Consistent with current multiarray interface of Numeric package
       - Consistent with some other languages
       - Perception that elementwise operations are more natural
       - Perception that elementwise operations are used more frequently

    2. op for object, ~op for element

       - Consistent with current linear algebra interface of MatPy package
       - Consistent with some other languages
       - Perception that objectwise operations are more natural
       - Perception that objectwise operations are used more frequently
       - Consistent with the current behavior of operators on lists
       - Allow ~ to be a general elementwise meta-character in future
         extensions.

    It is generally agreed upon that

       - there is no absolute reason to favor one or the other
       - it is easy to cast from one representation to another in a
         sizable chunk of code, so the other flavor of operators is
         always minority
       - there are other semantic differences that favor existence of
         array-oriented and matrix-oriented packages, even if their
         operators are unified.
       - whatever the decision is taken, codes using existing
         interfaces should not be broken for a very long time.

    Therefore not much is lost, and much flexibility retained, if the
    semantic flavors of these two sets of operators are not dictated
    by the core language.  The application packages are responsible
    for making the most suitable choice.  This is already the case for
    NumPy and MatPy which use opposite semantics.  Adding new
    operators will not break this.  See also observation after
    subsection 2 in the Examples below.

    The issue of numerical precision was raised, but if the semantics
    is left to the applications, the actual precisions should also go
    there.

Examples

    Following are examples of the actual formulas that will appear
    using various operators or other representations described above.

    1. The matrix inversion formula:

       - Using op for object and ~op for element:

         b = a.I - a.I * u / (c.I + v/a*u) * v / a

         b = a.I - a.I * u * (c.I + v*a.I*u).I * v * a.I

       - Using op for element and ~op for object:

         b = a.I @- a.I @* u @/ (c.I @+ v@/a@*u) @* v @/ a

         b = a.I ~- a.I ~* u ~/ (c.I ~+ v~/a~*u) ~* v ~/ a

         b = a.I (-) a.I (*) u (/) (c.I (+) v(/)a(*)u) (*) v (/) a

         b = a.I [-] a.I [*] u [/] (c.I [+] v[/]a[*]u) [*] v [/] a

         b = a.I <-> a.I <*> u </> (c.I <+> v</>a<*>u) <*> v </> a

         b = a.I {-} a.I {*} u {/} (c.I {+} v{/}a{*}u) {*} v {/} a

       Observation: For linear algebra using op for object is preferable.

       Observation: The ~op type operators look better than (op) type
       in complicated formulas.

       - using named operators

         b = a.I @sub a.I @mul u @div (c.I @add v @div a @mul u) @mul v @div a

         b = a.I ~sub a.I ~mul u ~div (c.I ~add v ~div a ~mul u) ~mul v ~div a

       Observation: Named operators are not suitable for math formulas.

    2. Plotting a 3d graph

       - Using op for object and ~op for element:

         z = sin(x~**2 ~+ y~**2);    plot(x,y,z)

       - Using op for element and ~op for object:

         z = sin(x**2 + y**2);   plot(x,y,z)

        Observation: Elementwise operations with broadcasting allows
        much more efficient implementation than MatLab.

        Observation: It is useful to have two related classes with the
        semantics of op and ~op swapped.  Using these the ~op
        operators would only need to appear in chunks of code where
        the other flavor dominates, while maintaining consistent
        semantics of the code.

    3. Using + and - with automatic broadcasting

         a = b - c;  d = a.T*a

       Observation: This would silently produce hard-to-trace bugs if
       one of b or c is row vector while the other is column vector.

Miscellaneous issues:

    - Need for the ~+ ~- operators.  The objectwise + - are important
      because they provide important sanity checks as per linear
      algebra.  The elementwise + - are important because they allow
      broadcasting that are very efficient in applications.

    - Left division (solve).  For matrix, a*x is not necessarily equal
      to x*a.  The solution of a*x==b, denoted x=solve(a,b), is
      therefore different from the solution of x*a==b, denoted
      x=div(b,a).  There are discussions about finding a new symbol
      for solve.  [Background: MatLab use b/a for div(b,a) and a\b for
      solve(a,b).]

      It is recognized that Python provides a better solution without
      requiring a new symbol: the inverse method .I can be made to be
      delayed so that a.I*b and b*a.I are equivalent to Mat lab's a\b
      and b/a.  The implementation is quite simple and the resulting
      application code clean.

    - Power operator.  Python's use of a**b as pow(a,b) has two
      perceived disadvantages:

      - Most mathematicians are more familiar with a^b for this purpose.
      - It results in long augmented assignment operator ~**=.

      However, this issue is distinct from the main issue here.

    - Additional multiplication operators.  Several forms of
      multiplications are used in (multi-)linear algebra.  Most can be
      seen as variations of multiplication in linear algebra sense
      (such as Kronecker product).  But two forms appear to be more
      fundamental: outer product and inner product.  However, their
      specification includes indices, which can be either

      - associated with the operator, or
      - associated with the objects.

      The latter (the Einstein notation) is used extensively on paper,
      and is also the easier one to implement.  By implementing a
      tensor-with-indices class, a general form of multiplication
      would cover both outer and inner products, and specialize to
      linear algebra multiplication as well.  The index rule can be
      defined as class methods, like,

          a = b.i(1,2,-1,-2) * c.i(4,-2,3,-1)   # a_ijkl = b_ijmn c_lnkm

      Therefore one objectwise multiplication is sufficient.

    - Bitwise operators.

      - The proposed new math operators use the symbol ~ that is
        "bitwise not" operator.  This poses no compatibility problem
        but somewhat complicates implementation.

      - The symbol ^ might be better used for pow than bitwise xor.
        But this depends on the future of bitwise operators.  It does
        not immediately impact on the proposed math operator.

      - The symbol | was suggested to be used for matrix solve.  But
        the new solution of using delayed .I is better in several
        ways.

      - The current proposal fits in a larger and more general
        extension that will remove the need for special bitwise
        operators.  (See elementization below.)

    - Alternative to special operator names used in definition,

          def "+"(a, b)      in place of       def __add__(a, b)

      This appears to require greater syntactical change, and would
      only be useful when arbitrary additional operators are allowed.

Impact on general elementization

    The distinction between objectwise and elementwise operations are
    meaningful in other contexts as well, where an object can be
    conceptually regarded as a collection of elements.  It is
    important that the current proposal does not preclude possible
    future extensions.

    One general future extension is to use ~ as a meta operator to
    "elementize" a given operator.  Several examples are listed here:

    1. Bitwise operators.  Currently Python assigns six operators to
       bitwise operations: and (&), or (|), xor (^), complement (~),
       left shift (<<) and right shift (>>), with their own precedence
       levels.

       Among them, the & | ^ ~ operators can be regarded as
       elementwise versions of lattice operators applied to integers
       regarded as bit strings.

           5 and 6                # 6
           5 or 6                 # 5

           5 ~and 6               # 4
           5 ~or 6                # 7

       These can be regarded as general elementwise lattice operators,
       not restricted to bits in integers.

       In order to have named operators for xor ~xor, it is necessary
       to make xor a reserved word.

    2. List arithmetics.

           [1, 2] + [3, 4]        # [1, 2, 3, 4]
           [1, 2] ~+ [3, 4]       # [4, 6]

           ['a', 'b'] * 2         # ['a', 'b', 'a', 'b']
           'ab' * 2               # 'abab'

           ['a', 'b'] ~* 2        # ['aa', 'bb']
           [1, 2] ~* 2            # [2, 4]

       It is also consistent to Cartesian product

           [1,2]*[3,4]            # [(1,3),(1,4),(2,3),(2,4)]

    3. List comprehension.

           a = [1, 2]; b = [3, 4]
           ~f(a,b)                # [f(x,y) for x, y in zip(a,b)]
           ~f(a*b)                # [f(x,y) for x in a for y in b]
           a ~+ b                 # [x + y for x, y in zip(a,b)]

    4. Tuple generation (the zip function in Python 2.0)

          [1, 2, 3], [4, 5, 6]   # ([1,2, 3], [4, 5, 6])
          [1, 2, 3]~,[4, 5, 6]   # [(1,4), (2, 5), (3,6)]

    5. Using ~ as generic elementwise meta-character to replace map

          ~f(a, b)               # map(f, a, b)
          ~~f(a, b)              # map(lambda *x:map(f, *x), a, b)

       More generally,

          def ~f(*x): return map(f, *x)
          def ~~f(*x): return map(~f, *x)
          ...

    6. Elementwise format operator (with broadcasting)

          a = [1,2,3,4,5]
          print ["%5d "] ~% a
          a = [[1,2],[3,4]]
          print ["%5d "] ~~% a

    7.  Rich comparison

          [1, 2, 3]  ~< [3, 2, 1]  # [1, 0, 0]
          [1, 2, 3] ~== [3, 2, 1]  # [0, 1, 0]

    8. Rich indexing

          [a, b, c, d] ~[2, 3, 1]  # [c, d, b]

    9. Tuple flattening

          a = (1,2);  b = (3,4)
          f(~a, ~b)                # f(1,2,3,4)

    10. Copy operator

          a ~= b                   # a = b.copy()

        There can be specific levels of deep copy

          a ~~= b                  # a = b.copy(2)

    Notes:

    1. There are probably many other similar situations.  This general
       approach seems well suited for most of them, in place of
       several separated extensions for each of them (parallel and
       cross iteration, list comprehension, rich comparison, etc).

    2. The semantics of "elementwise" depends on applications.  For
       example, an element of matrix is two levels down from the
       list-of-list point of view.  This requires more fundamental
       change than the current proposal.  In any case, the current
       proposal will not negatively impact on future possibilities of
       this nature.

    Note that this section describes a type of future extensions that
    is consistent with current proposal, but may present additional
    compatibility or other problems.  They are not tied to the current
    proposal.

Impact on named operators

    The discussions made it generally clear that infix operators is a
    scarce resource in Python, not only in numerical computation, but
    in other fields as well.  Several proposals and ideas were put
    forward that would allow infix operators be introduced in ways
    similar to named functions.  We show here that the current
    extension does not negatively impact on future extensions in this
    regard.

    1. Named infix operators.

        Choose a meta character, say @, so that for any identifier
        "opname", the combination "@opname" would be a binary infix
        operator, and

        a @opname b == opname(a,b)

        Other representations mentioned include .name ~name~ :name:
        (.name) %name% and similar variations.  The pure bracket based
        operators cannot be used this way.

        This requires a change in the parser to recognize @opname, and
        parse it into the same structure as a function call.  The
        precedence of all these operators would have to be fixed at
        one level, so the implementation would be different from
        additional math operators which keep the precedence of
        existing math operators.

        The current proposed extension do not limit possible future
        extensions of such form in any way.

    2. More general symbolic operators.

        One additional form of future extension is to use meta
        character and operator symbols (symbols that cannot be used in
        syntactical structures other than operators).  Suppose @ is
        the meta character.  Then

            a + b,    a @+ b,    a @@+ b,  a @+- b

        would all be operators with a hierarchy of precedence, defined by

            def "+"(a, b)
            def "@+"(a, b)
            def "@@+"(a, b)
            def "@+-"(a, b)

        One advantage compared with named operators is greater
        flexibility for precedences based on either the meta character
        or the ordinary operator symbols.  This also allows operator
        composition.  The disadvantage is that they are more like
        "line noise".  In any case the current proposal does not
        impact its future possibility.

        These kinds of future extensions may not be necessary when
        Unicode becomes generally available.

        Note that this section discusses compatibility of the proposed
        extension with possible future extensions.  The desirability
        or compatibility of these other extensions themselves are
        specifically not considered here.

Credits and archives

    The discussions mostly happened in July to August of 2000 on news
    group comp.lang.python and the mailing list python-dev.  There are
    altogether several hundred postings, most can be retrieved from
    these two pages (and searching word "operator"):

        http://www.python.org/pipermail/python-list/2000-July/
        http://www.python.org/pipermail/python-list/2000-August/

    The names of contributers are too numerous to mention here,
    suffice to say that a large proportion of ideas discussed here are
    not our own.

    Several key postings (from our point of view) that may help to
    navigate the discussions include:

        http://www.python.org/pipermail/python-list/2000-July/108893.html
        http://www.python.org/pipermail/python-list/2000-July/108777.html
        http://www.python.org/pipermail/python-list/2000-July/108848.html
        http://www.python.org/pipermail/python-list/2000-July/109237.html
        http://www.python.org/pipermail/python-list/2000-July/109250.html
        http://www.python.org/pipermail/python-list/2000-July/109310.html
        http://www.python.org/pipermail/python-list/2000-July/109448.html
        http://www.python.org/pipermail/python-list/2000-July/109491.html
        http://www.python.org/pipermail/python-list/2000-July/109537.html
        http://www.python.org/pipermail/python-list/2000-July/109607.html
        http://www.python.org/pipermail/python-list/2000-July/109709.html
        http://www.python.org/pipermail/python-list/2000-July/109804.html
        http://www.python.org/pipermail/python-list/2000-July/109857.html
        http://www.python.org/pipermail/python-list/2000-July/110061.html
        http://www.python.org/pipermail/python-list/2000-July/110208.html
        http://www.python.org/pipermail/python-list/2000-August/111427.html
        http://www.python.org/pipermail/python-list/2000-August/111558.html
        http://www.python.org/pipermail/python-list/2000-August/112551.html
        http://www.python.org/pipermail/python-list/2000-August/112606.html
        http://www.python.org/pipermail/python-list/2000-August/112758.html

        http://www.python.org/pipermail/python-dev/2000-July/013243.html
        http://www.python.org/pipermail/python-dev/2000-July/013364.html
        http://www.python.org/pipermail/python-dev/2000-August/014940.html

    These are earlier drafts of this PEP:

        http://www.python.org/pipermail/python-list/2000-August/111785.html
        http://www.python.org/pipermail/python-list/2000-August/112529.html
        http://www.python.org/pipermail/python-dev/2000-August/014906.html

    There is an alternative PEP (with official PEP number 211) by Greg
    Wilson, titled "Adding New Linear Algebra Operators to Python".

    Its first (and current) version is at:

        http://www.python.org/pipermail/python-dev/2000-August/014876.html
        http://python.sourceforge.net/peps/pep-0211.html

Additional References

    [1] http://MatPy.sourceforge.net/Misc/index.html


PEP 226 -- Python 2.1 Release Schedule

PEP: | 226
Title: | Python 2.1 Release Schedule
Version: | $Revision: 1.11 $
Author: | Jeremy Hylton
Status: | Incomplete
Type: | Informational
Created: | 16-Oct-2000
Python-Version: | 2.1
Post-History: |

Abstract

    This document describes the post Python 2.0 development and
    release schedule.  According to this schedule, Python 2.1 will be
    released in April of 2001.  The schedule primarily concerns
    itself with PEP-size items.  Small bug fixes and changes will
    occur up until the first beta release.

Release Schedule

    Tentative future release dates

    [bugfix release dates go here]

    Past release dates:

    17-Apr-2001: 2.1 final release
    15-Apr-2001: 2.1 release candidate 2
    13-Apr-2001: 2.1 release candidate 1
    23-Mar-2001: Python 2.1 beta 2 release
    02-Mar-2001: First 2.1 beta release
    02-Feb-2001: Python 2.1 alpha 2 release
    22-Jan-2001: Python 2.1 alpha 1 release
    16-Oct-2000: Python 2.0 final release

Open issues for Python 2.0 beta 2

    Add a default unit testing framework to the standard library.

Guidelines for making changes for Python 2.1

    The guidelines and schedule will be revised based on discussion in
    the python-dev@python.org mailing list.

    The PEP system was instituted late in the Python 2.0 development
    cycle and many changes did not follow the process described in PEP
    1.  The development process for 2.1, however, will follow the PEP
    process as documented.

    The first eight weeks following 2.0 final will be the design and
    review phase.  By the end of this period, any PEP that is proposed
    for 2.1 should be ready for review.  This means that the PEP is
    written and discussion has occurred on the python-dev@python.org
    and python-list@python.org mailing lists.

    The next six weeks will be spent reviewing the PEPs and
    implementing and testing the accepted PEPs.  When this period
    stops, we will end consideration of any incomplete PEPs.  Near the
    end of this period, there will be a feature freeze where any small
    features not worthy of a PEP will not be accepted.

    Before the final release, we will have six weeks of beta testing
    and a release candidate or two.

General guidelines for submitting patches and making changes

    Use good sense when committing changes.  You should know what we
    mean by good sense or we wouldn't have given you commit privileges
    <0.5 wink>.  Some specific examples of good sense include:

    - Do whatever the dictator tells you.

    - Discuss any controversial changes on python-dev first.  If you
      get a lot of +1 votes and no -1 votes, make the change.  If you
      get a some -1 votes, think twice; consider asking Guido what he
      thinks.

    - If the change is to code you contributed, it probably makes
      sense for you to fix it.

    - If the change affects code someone else wrote, it probably makes
      sense to ask him or her first.

    - You can use the SourceForge (SF) Patch Manager to submit a patch
      and assign it to someone for review.

    Any significant new feature must be described in a PEP and
    approved before it is checked in.

    Any significant code addition, such as a new module or large
    patch, must include test cases for the regression test and
    documentation.  A patch should not be checked in until the tests
    and documentation are ready.

    If you fix a bug, you should write a test case that would have
    caught the bug.

    If you commit a patch from the SF Patch Manager or fix a bug from
    the Jitterbug database, be sure to reference the patch/bug number
    in the CVS log message.  Also be sure to change the status in the
    patch manager or bug database (if you have access to the bug
    database).

    It is not acceptable for any checked in code to cause the
    regression test to fail.  If a checkin causes a failure, it must
    be fixed within 24 hours or it will be backed out.

    All contributed C code must be ANSI C.  If possible check it with
    two different compilers, e.g. gcc and MSVC.

    All contributed Python code must follow Guido's Python style
    guide.  http://www.python.org/doc/essays/styleguide.html

    It is understood that any code contributed will be released under
    an Open Source license.  Do not contribute code if it can't be
    released this way.


PEP 227 -- Statically Nested Scopes

PEP: | 227
Title: | Statically Nested Scopes
Version: | $Revision: 1.9 $
Author: | jeremy@digicool.com (Jeremy Hylton)
Status: | Draft
Type: | Standards Track
Python-Version: | 2.1
Created: | 01-Nov-2000
Post-History: |

Abstract

    This PEP describes the addition of statically nested scoping
    (lexical scoping) for Python 2.2, and as an source level option
    for python 2.1.  In addition, Python 2.1 will issue warnings about
    constructs whose meaning may change when this feature is enabled.

    The old language definition (2.0 and before) defines exactly three
    namespaces that are used to resolve names -- the local, global,
    and built-in namespaces.  The addition of nested scopes allows
    resolution of unbound local names in enclosing functions'
    namespaces.

    The most visible consequence of this change is that lambdas (and
    other nested functions) can reference variables defined in the
    surrounding namespace.  Currently, lambdas must often use default
    arguments to explicitly creating bindings in the lambda's
    namespace.

Introduction

    This proposal changes the rules for resolving free variables in
    Python functions.  The new name resolution semantics will take
    effect with Python 2.2.  These semantics will also be available in
    Python 2.1 by adding "from __future__ import nested_scopes" to the
    top of a module.  (See PEP 236.)

    The Python 2.0 definition specifies exactly three namespaces to
    check for each name -- the local namespace, the global namespace,
    and the builtin namespace.  According to this definition, if a
    function A is defined within a function B, the names bound in B
    are not visible in A.  The proposal changes the rules so that
    names bound in B are visible in A (unless A contains a name
    binding that hides the binding in B).

    This specification introduces rules for lexical scoping that are
    common in Algol-like languages.  The combination of lexical
    scoping and existing support for first-class functions is
    reminiscent of Scheme.

    The changed scoping rules address two problems -- the limited
    utility of lambda expressions (and nested functions in general),
    and the frequent confusion of new users familiar with other
    languages that support nested lexical scopes, e.g. the inability
    to define recursive functions except at the module level.

    The lambda expression yields an unnamed function that evaluates a
    single expression.  It is often used for callback functions.  In
    the example below (written using the Python 2.0 rules), any name
    used in the body of the lambda must be explicitly passed as a
    default argument to the lambda.

      from Tkinter import *
      root = Tk()
      Button(root, text="Click here",
             command=lambda root=root: root.test.configure(text="..."))

    This approach is cumbersome, particularly when there are several
    names used in the body of the lambda.  The long list of default
    arguments obscures the purpose of the code.  The proposed
    solution, in crude terms, implements the default argument approach
    automatically.  The "root=root" argument can be omitted.

    The new name resolution semantics will cause some programs to
    behave differently than they did under Python 2.0.  In some cases,
    programs will fail to compile.  In other cases, names that were
    previously resolved using the global namespace will be resolved
    using the local namespace of an enclosing function.  In Python
    2.1, warnings will be issued for all statements that will behave
    differently.

Specification

    Python is a statically scoped language with block structure, in
    the traditional of Algol.  A code block or region, such as a
    module, class definition, or function body, is the basic unit of a
    program.

    Names refer to objects.  Names are introduced by name binding
    operations.  Each occurrence of a name in the program text refers
    to the binding of that name established in the innermost function
    block containing the use.

    The name binding operations are argument declaration, assignment,
    class and function definition, import statements, for statements,
    and except clauses.  Each name binding occurs within a block
    defined by a class or function definition or at the module level
    (the top-level code block).

    If a name is bound anywhere within a code block, all uses of the
    name within the block are treated as references to the current
    block.  (Note: This can lead to errors when a name is used within
    a block before it is bound.)

    If the global statement occurs within a block, all uses of the
    name specified in the statement refer to the binding of that name
    in the top-level namespace.  Names are resolved in the top-level
    namespace by searching the global namespace, i.e. the namespace of
    the module containing the code block, and in the builtin
    namespace, i.e. the namespace of the __builtin__ module.  The
    global namespace is searched first.  If the name is not found
    there, the builtin namespace is searched.  The global statement
    must precede all uses of the name.

    If a name is used within a code block, but it is not bound there
    and is not declared global, the use is treated as a reference to
    the nearest enclosing function region.  (Note: If a region is
    contained within a class definition, the name bindings that occur
    in the class block are not visible to enclosed functions.)

    A class definition is an executable statement that may uses and
    definitions of names.  These references follow the normal rules
    for name resolution.  The namespace of the class definition
    becomes the attribute dictionary of the class.

    The following operations are name binding operations.  If they
    occur within a block, they introduce new local names in the
    current block unless there is also a global declaration.

    Function definition: def name ...
    Argument declaration: def f(...name...), lambda ...name...
    Class definition: class name ...
    Assignment statement: name = ...
    Import statement: import name, import module as name,
        from module import name
    Implicit assignment: names are bound by for statements and except
        clauses

    There are several cases where Python statements are illegal when
    used in conjunction with nested scopes that contain free
    variables.

    If a variable is referenced in an enclosed scope, it is an error
    to delete the name.  The compiler will raise a SyntaxError for
    'del name'.

    If the wild card form of import (import *) is used in a function
    and the function contains a nested block with free variables, the
    compiler will raise a SyntaxError.

    If exec is used in a function and the function contains a nested
    block with free variables, the compiler will raise a SyntaxError
    unless the exec explicitly specifies the local namespace for the
    exec.  (In other words, "exec obj" would be illegal, but
    "exec obj in ns" would be legal.)

    If a name bound in a function scope is also the name of a module
    global name or a standard builtin name, and the function contains
    a nested function scope that references the name, the compiler
    will issue a warning.  The name resolution rules will result in
    different bindings under Python 2.0 than under Python 2.2.  The
    warning indicates that the program may not run correctly with all
    versions of Python.

Discussion

    The specified rules allow names defined in a function to be
    referenced in any nested function defined with that function.  The
    name resolution rules are typical for statically scoped languages,
    with three primary exceptions:

        - Names in class scope are not accessible.
        - The global statement short-circuits the normal rules.
        - Variables are not declared.

    Names in class scope are not accessible.  Names are resolved in
    the innermost enclosing function scope.  If a class definition
    occurs in a chain of nested scopes, the resolution process skips
    class definitions.  This rule prevents odd interactions between
    class attributes and local variable access.  If a name binding
    operation occurs in a class definition, it creates an attribute on
    the resulting class object.  To access this variable in a method,
    or in a function nested within a method, an attribute reference
    must be used, either via self or via the class name.

    An alternative would have been to allow name binding in class
    scope to behave exactly like name binding in function scope.  This
    rule would allow class attributes to be referenced either via
    attribute reference or simple name.  This option was ruled out
    because it would have been inconsistent with all other forms of
    class and instance attribute access, which always use attribute
    references.  Code that used simple names would have been obscure.

    The global statement short-circuits the normal rules.  Under the
    proposal, the global statement has exactly the same effect that it
    does for Python 2.0.  It is also noteworthy because it allows name
    binding operations performed in one block to change bindings in
    another block (the module).

    Variables are not declared.  If a name binding operation occurs
    anywhere in a function, then that name is treated as local to the
    function and all references refer to the local binding.  If a
    reference occurs before the name is bound, a NameError is raised.
    The only kind of declaration is the global statement, which allows
    programs to be written using mutable global variables.  As a
    consequence, it is not possible to rebind a name defined in an
    enclosing scope.  An assignment operation can only bind a name in
    the current scope or in the global scope.  The lack of
    declarations and the inability to rebind names in enclosing scopes
    are unusual for lexically scoped languages; there is typically a
    mechanism to create name bindings (e.g. lambda and let in Scheme)
    and a mechanism to change the bindings (set! in Scheme).

    XXX Alex Martelli suggests comparison with Java, which does not
    allow name bindings to hide earlier bindings.

Examples

    A few examples are included to illustrate the way the rules work.

    XXX Explain the examples

    >>> def make_adder(base):
    ...     def adder(x):
    ...         return base + x
    ...     return adder
    >>> add5 = make_adder(5)
    >>> add5(6)
    11

    >>> def make_fact():
    ...     def fact(n):
    ...         if n == 1:
    ...             return 1L
    ...         else:
    ...             return n * fact(n - 1)
    ...     return fact
    >>> fact = make_fact()
    >>> fact(7)
    5040L

    >>> def make_wrapper(obj):
    ...     class Wrapper:
    ...         def __getattr__(self, attr):
    ...             if attr[0] != '_':
    ...                 return getattr(obj, attr)
    ...             else:
    ...                 raise AttributeError, attr
    ...     return Wrapper()
    >>> class Test:
    ...     public = 2
    ...     _private = 3
    >>> w = make_wrapper(Test())
    >>> w.public
    2
    >>> w._private
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    AttributeError: _private

    An example from Tim Peters demonstrates the potential pitfalls of
    nested scopes in the absence of declarations:

    i = 6
    def f(x):
        def g():
            print i
        # ...
        # skip to the next page
        # ...
        for i in x:  # ah, i *is* local to f, so this is what g sees
            pass
        g()

    The call to g() will refer to the variable i bound in f() by the for
    loop.  If g() is called before the loop is executed, a NameError will
    be raised.

    XXX need some counterexamples

Backwards compatibility

    There are two kinds of compatibility problems caused by nested
    scopes.  In one case, code that behaved one way in earlier
    versions behaves differently because of nested scopes.  In the
    other cases, certain constructs interact badly with nested scopes
    and will trigger SyntaxErrors at compile time.

    The following example from Skip Montanaro illustrates the first
    kind of problem:

    x = 1
    def f1():
        x = 2
        def inner():
            print x
        inner()

    Under the Python 2.0 rules, the print statement inside inner()
    refers to the global variable x and will print 1 if f1() is
    called.  Under the new rules, it refers to the f1()'s namespace,
    the nearest enclosing scope with a binding.

    The problem occurs only when a global variable and a local
    variable share the same name and a nested function uses that name
    to refer to the global variable.  This is poor programming
    practice, because readers will easily confuse the two different
    variables.  One example of this problem was found in the Python
    standard library during the implementation of nested scopes.

    To address this problem, which is unlikely to occur often, the
    Python 2.1 compiler (when nested scopes are not enabled) issues a
    warning.

    The other compatibility problem is caused by the use of 'import *'
    and 'exec' in a function body, when that function contains a
    nested scope and the contained scope has free variables.  For
    example:

    y = 1
    def f():
        exec "y = 'gotcha'" # or from module import *
        def g():
            return y
        ...

    At compile-time, the compiler cannot tell whether an exec that
    operates on the local namespace or an import * will introduce
    name bindings that shadow the global y.  Thus, it is not possible
    to tell whether the reference to y in g() should refer to the
    global or to a local name in f().

    In discussion of the python-list, people argued for both possible
    interpretations.  On the one hand, some thought that the reference
    in g() should be bound to a local y if one exists.  One problem
    with this interpretation is that it is impossible for a human
    reader of the code to determine the binding of y by local
    inspection.  It seems likely to introduce subtle bugs.  The other
    interpretation is to treat exec and import * as dynamic features
    that do not effect static scoping.  Under this interpretation, the
    exec and import * would introduce local names, but those names
    would never be visible to nested scopes.  In the specific example
    above, the code would behave exactly as it did in earlier versions
    of Python.

    Since each interpretation is problematic and the exact meaning
    ambiguous, the compiler raises an exception.  The Python 2.1
    compiler issues a warning when nested scopes are not enabled.

    A brief review of three Python projects (the standard library,
    Zope, and a beta version of PyXPCOM) found four backwards
    compatibility issues in approximately 200,000 lines of code.
    There was one example of case #1 (subtle behavior change) and two
    examples of import * problems in the standard library.

    (The interpretation of the import * and exec restriction that was
    implemented in Python 2.1a2 was much more restrictive, based on
    language that in the reference manual that had never been
    enforced.  These restrictions were relaxed following the release.)

Compatibility of C API

    The implementation causes several Python C API functions to
    change, including PyCode_New().  As a result, C extensions may
    need to be updated to work correctly with Python 2.1.

locals() / vars()

    These functions return a dictionary containing the current scope's
    local variables.  Modifications to the dictionary do not affect
    the values of variables.  Under the current rules, the use of
    locals() and globals() allows the program to gain access to all
    the namespaces in which names are resolved.

    An analogous function will not be provided for nested scopes.
    Under this proposal, it will not be possible to gain
    dictionary-style access to all visible scopes.

Warnings and Errors

    The compiler will issue warnings in Python 2.1 to help identify
    programs that may not compile or run correctly under future
    versions of Python.  Under Python 2.2 or Python 2.1 if the
    nested_scopes future statement is used, which are collectively
    referred to as "future semantics" in this section, the compiler
    will issue SyntaxErrors in some cases.

    The warnings typically apply when a function that contains a
    nested function that has free variables.  For example, if function
    F contains a function G and G uses the builtin len(), then F is a
    function that contains a nested function (G) with a free variable
    (len).  The label "free-in-nested" will be used to describe these
    functions.

    import * used in function scope

        The language reference specifies that import * may only occur
        in a module scope.  (Sec. 6.11)  The implementation of C
        Python has supported import * at the function scope.

        If import * is used in the body of a free-in-nested function,
        the compiler will issue a warning.  Under future semantics,
        the compiler will raise a SyntaxError.

    bare exec in function scope

        The exec statement allows two optional expressions following
        the keyword "in" that specify the namespaces used for locals
        and globals.  An exec statement that omits both of these
        namespaces is a bare exec.

        If a bare exec is used in the body of a free-in-nested
        function, the compiler will issue a warning.  Under future
        semantics, the compiler will raise a SyntaxError.

    local shadows global

        If a free-in-nested function has a binding for a local
        variable that (1) is used in a nested function and (2) is the
        same as a global variable, the compiler will issue a warning.

Rebinding names in enclosing scopes

    There are technical issues that make it difficult to support
    rebinding of names in enclosing scopes, but the primary reason
    that it is not allowed in the current proposal is that Guido is
    opposed to it.  His motivation: it is difficult to support,
    because it would require a new mechanism that would allow the
    programmer to specify that an assignment in a block is supposed to
    rebind the name in an enclosing block; presumably a keyword or
    special syntax (x := 3) would make this possible.  Given that this
    would encourage the use of local variables to hold state that is
    better stored in a class instance, it's not worth adding new
    syntax to make this possible (in Guido's opinion).

    The proposed rules allow programmers to achieve the effect of
    rebinding, albeit awkwardly.  The name that will be effectively
    rebound by enclosed functions is bound to a container object.  In
    place of assignment, the program uses modification of the
    container to achieve the desired effect:

    def bank_account(initial_balance):
        balance = [initial_balance]
        def deposit(amount):
            balance[0] = balance[0] + amount
            return balance
        def withdraw(amount):
            balance[0] = balance[0] - amount
            return balance
        return deposit, withdraw

    Support for rebinding in nested scopes would make this code
    clearer.  A class that defines deposit() and withdraw() methods
    and the balance as an instance variable would be clearer still.
    Since classes seem to achieve the same effect in a more
    straightforward manner, they are preferred.

Implementation

    XXX Jeremy, is this still the case?

    The implementation for C Python uses flat closures [1].  Each def
    or lambda expression that is executed will create a closure if the
    body of the function or any contained function has free
    variables.  Using flat closures, the creation of closures is
    somewhat expensive but lookup is cheap.

    The implementation adds several new opcodes and two new kinds of
    names in code objects.  A variable can be either a cell variable
    or a free variable for a particular code object.  A cell variable
    is referenced by containing scopes; as a result, the function
    where it is defined must allocate separate storage for it on each
    invocation.  A free variable is referenced via a function's
    closure.

    The choice of free closures was made based on three factors.
    First, nested functions are presumed to be used infrequently,
    deeply nested (several levels of nesting) still less frequently.
    Second, lookup of names in a nested scope should be fast.
    Third, the use of nested scopes, particularly where a function
    that access an enclosing scope is returned, should not prevent
    unreferenced objects from being reclaimed by the garbage
    collector.

    XXX Much more to say here

References

    [1] Luca Cardelli.  Compiling a functional language.  In Proc. of
    the 1984 ACM Conference on Lisp and Functional Programming,
    pp. 208-217, Aug. 1984
        http://citeseer.nj.nec.com/cardelli84compiling.html

Copyright

    XXX


PEP 228 -- Reworking Python's Numeric Model

PEP: | 228
Title: | Reworking Python's Numeric Model
Version: | $Revision: 1.4 $
Author: | pep@zadka.site.co.il (Moshe Zadka)
Status: | Draft
Type: | Standards Track
Python-Version: | ??
Created: | 4-Nov-2000
Post-History: |

Abstract

    Today, Python's numerical model is similar to the C numeric model:
    there are several unrelated numerical types, and when operations
    between numerical types are requested, coercions happen.  While
    the C rationale for the numerical model is that it is very similar
    to what happens at the hardware level, that rationale does not
    apply to Python.  So, while it is acceptable to C programmers that
    2/3 == 0, it is surprising to Python programmers.

Rationale

    In usability studies, one of the least usable aspect of Python was
    the fact that integer division returns the floor of the division.
    This makes it hard to program correctly, requiring casts to
    float() in various parts through the code.  Python's numerical
    model stems from C, while an model that might be easier to work with
    can be based on the mathematical understanding of numbers.

Other Numerical Models

    Perl's numerical model is that there is one type of numbers --
    floating point numbers.  While it is consistent and superficially
    non-surprising, it tends to have subtle gotchas.  One of these is
    that printing numbers is very tricky, and requires correct
    rounding.  In Perl, there is also a mode where all numbers are
    integers.  This mode also has its share of problems, which arise
    from the fact that there is not even an approximate way of
    dividing numbers and getting meaningful answers.

Suggested Interface For Python's Numerical Model

    While coercion rules will remain for add-on types and classes, the
    built in type system will have exactly one Python type -- a
    number.  There are several things which can be considered "number
    methods":

    1. isnatural()
    2. isintegral()
    3. isrational()
    4. isreal()
    5. iscomplex()

    a. isexact()

    Obviously, a number which answers true to a question from 1 to 5, will
    also answer true to any following question. If "isexact()" is not true,
    then any answer might be wrong.
    (But not horribly wrong: it's close to the truth.)

    Now, there is two thing the models promises for the field operations
    (+, -, /, *):

    - If both operands satisfy isexact(), the result satisfies
      isexact().

    - All field rules are true, except that for not-isexact() numbers,
      they might be only approximately true.

    One consequence of these two rules is that all exact calcutions
    are done as (complex) rationals: since the field laws must hold,
    then

        (a/b)*b == a

    must hold.

    There is built-in function, inexact() which takes a number
    and returns an inexact number which is a good approximation.
    Inexact numbers must be as least as accurate as if they were
    using IEEE-754.

    Several of the classical Python functions will return exact numbers
    even when given inexact numbers: e.g, int().

Coercion

    The number type does not define nb_coerce
    Any numeric operation slot, when receiving something other then PyNumber,
    refuses to implement it.

Inexact Operations

    The functions in the "math" module will be allowed to return
    inexact results for exact values.  However, they will never return
    a non-real number.  The functions in the "cmath" module are also
    allowed to return an inexact result for an exact argument, and are
    furthermore allowed to return a complex result for a real
    argument.

Numerical Python Issues

    People who use Numerical Python do so for high-performance vector
    operations.  Therefore, NumPy should keep its hardware based
    numeric model.

Unresolved Issues

    Which number literals will be exact, and which inexact?

    How do we deal with IEEE 754 operations? (probably, isnan/isinf should
    be methods)

Copyright

    This document has been placed in the public domain.


PEP 229 -- Using Distutils to Build Python

PEP: | 229
Title: | Using Distutils to Build Python
Version: | $Revision: 1.3 $
Author: | akuchlin@mems-exchange.org (A.M. Kuchling)
Status: | Draft
Type: | Standards
Created: | 16-Nov-2000
Post-History: |

Introduction

    The Modules/Setup mechanism has some flaws:

    * People have to remember to uncomment bits of Modules/Setup in
      order to get all the possible modules.

    * Moving Setup to a new version of Python is tedious; new modules
      have been added, so you can't just copy the older version, but
      have to reconcile the two versions.

    * Users have to figure out where the needed libraries, such as
      zlib, are installed.

Proposal

    Use the Distutils to build the modules that come with Python.

    The changes can be broken up into several pieces:

    1. The Distutils needs some Python modules to be able to build
       modules.  Currently I believe the minimal list is posix, _sre,
       and string.

       These modules will have to be built before the Distutils can be
       used, so they'll simply be hardwired into Modules/Makefile and
       be automatically built.

    2. A top-level setup.py script will be written that checks the
       libraries installed on the system and compiles as many modules
       as possible.

    3. Modules/Setup will be kept and settings in it will override
       setup.py's usual behavior, so you can disable a module known
       to be buggy, or specify particular compilation or linker flags.
       However, in the common case where setup.py works correctly,
       everything in Setup will remain commented out.  The other
       Setup.* become unnecessary, since nothing will be generating
       Setup automatically.

Implementation

    Patch #102588 on SourceForge contains the proposed patch.
    Currently the patch tries to be conservative and to change as few
    files as possible, in order to simplify backing out the patch.
    For example, no attempt is made to rip out the existing build
    mechanisms.  Such simplifications can wait for later in the beta
    cycle, when we're certain the patch will be left in, or they can
    wait for Python 2.2.

    The patch makes the following changes:

    * Makes some required changes to distutils/sysconfig (these will
      be checked in separately)

    * In the top-level Makefile.in, the "sharedmods" target simply
      runs "./python setup.py build", and "sharedinstall" runs
      "./python setup.py install".  The "clobber" target also deletes
      the build/ subdirectory where Distutils puts its output.

    * Modules/Setup.config.in only contains entries for the gc and thread
      modules; the readline, curses, and db modules are removed because
      it's now setup.py's job to handle them.

    * Modules/Setup.dist now contains entries for only 3 modules --
      _sre, posix, and strop.

    * The configure script builds setup.cfg from setup.cfg.in.  This
      is needed for two reasons: to make building in subdirectories
      work, and to get the configured installation prefix.

    * Adds setup.py to the top directory of the source tree.  setup.py
      is the largest piece of the puzzle, though not the most
      complicated.  setup.py contains a subclass of the BuildExt
      class, and extends it with a detect_modules() method that does
      the work of figuring out when modules can be compiled, and adding
      them to the 'exts' list.

Unresolved Issues

    Do we need to make it possible to disable the 3 hard-wired modules
    without manually hacking the Makefiles?  If yes, perhaps a
    configure switch is sufficient.

    The Distutils always compile modules as shared libraries.  How do
    we support compiling them statically into the resulting Python
    binary?

    [Answer: building a Python binary with the Distutils should be
    feasible, though no one has implemented it yet.  This should be
    done, though the main setup.py will probably need to implement
    its own BuildExt subclass anyway.]

    makesetup and the other contents of $(LIBPL)/config need to be
    preserved for compatibility with existing modules; for how many
    versions do we need to keep them around?

Copyright

    This document has been placed in the public domain.


PEP 230 -- Warning Framework

PEP: | 230
Title: | Warning Framework
Version: | $Revision: 1.5 $
Author: | guido@python.org (Guido van Rossum)
Status: | Draft
Type: | Standards Track
Python-Version: | 2.1
Status: | Incomplete
Post-History: | 05-Nov-2000

Abstract

    This PEP proposes a C and Python level API, as well as command
    line flags, to issue warning messages and control what happens to
    them.  This is mostly based on GvR's proposal posted to python-dev
    on 05-Nov-2000, with some ideas (such as using classes to
    categorize warnings) merged in from Paul Prescod's
    counter-proposal posted on the same date.  Also, an attempt to
    implement the proposal caused several small tweaks.

Motivation

    With Python 3000 looming, it is necessary to start issuing
    warnings about the use of obsolete or deprecated features, in
    addition to errors.  There are also lots of other reasons to be
    able to issue warnings, both from C and from Python code, both at
    compile time and at run time.

    Warnings aren't fatal, and thus it's possible that a program
    triggers the same warning many times during a single execution.
    It would be annoying if a program emitted an endless stream of
    identical warnings.  Therefore, a mechanism is needed that
    suppresses multiple identical warnings.

    It is also desirable to have user control over which warnings are
    printed.  While in general it is useful to see all warnings all
    the time, there may be times where it is impractical to fix the
    code right away in a production program.  In this case, there
    should be a way to suppress warnings.

    It is also useful to be able to suppress specific warnings during
    program development, e.g. when a warning is generated by a piece
    of 3rd party code that cannot be fixed right away, or when there
    is no way to fix the code (possibly a warning message is generated
    for a perfectly fine piece of code).  It would be unwise to offer
    to suppress all warnings in such cases: the developer would miss
    warnings about the rest of the code.

    On the other hand, there are also situations conceivable where
    some or all warnings are better treated as errors.  For example,
    it may be a local coding standard that a particular deprecated
    feature should not be used.  In order to enforce this, it is
    useful to be able to turn the warning about this particular
    feature into an error, raising an exception (without necessarily
    turning all warnings into errors).

    Therefore, I propose to introduce a flexible "warning filter"
    which can filter out warnings or change them into exceptions,
    based on:

    - Where in the code they are generated (per package, module, or
      function)

    - The warning category (warning categories are discussed below)

    - A specific warning message

    The warning filter must be controllable both from the command line
    and from Python code.

APIs For Issuing Warnings

    - To issue a warning from Python:

        import warnings
        warnings.warn(message[, category[, stacklevel]])

      The category argument, if given, must be a warning category
      class (see below); it defaults to warnings.UserWarning.  This
      may raise an exception if the particular warning issued is
      changed into an error by the warnings filter.  The stacklevel
      can be used by wrapper functions written in Python, like this:

      def deprecation(message):
          warn(message, DeprecationWarning, level=2)

      This makes the warning refer to the deprecation()'s caller,
      rather than to the source of deprecation() itself (since the
      latter would defeat the purpose of the warning message).

    - To issue a warning from C:

        int PyErr_Warn(PyObject *category, char *message);

      Return 0 normally, 1 if an exception is raised (either because
      the warning was transformed into an exception, or because of a
      malfunction in the implementation, such as running out of
      memory).  The category argument must be a warning category class
      (see below) or NULL, in which case it defaults to
      PyExc_RuntimeWarning.  When PyErr_Warn() function returns 1, the
      caller should do normal exception handling.

      The current C implementation of PyErr_Warn() imports the
      warnings module (implemented in Python) and calls its warn()
      function.  This minimizes the amount of C code that needs to be
      added to implement the warning feature.

      [XXX Open Issue: what about issuing warnings during lexing or
      parsing, which don't have the exception machinery available?]

Warnings Categories

    There are a number of built-in exceptions that represent warning
    categories.  This categorization is useful to be able to filter
    out groups of warnings.  The following warnings category classes
    are currently defined:

    - Warning -- this is the base class of all warning category
      classes and it itself a subclass of Exception

    - UserWarning -- the default category for warnings.warn()

    - DeprecationWarning -- base category for warnings about deprecated
      features

    - SyntaxWarning -- base category for warnings about dubious
      syntactic features

    - RuntimeWarning -- base category for warnings about dubious
      runtime features

    [XXX: Other warning categories may be proposed during the review
    period for this PEP.]

    These standard warning categories are available from C as
    PyExc_Warning, PyExc_UserWarning, etc.  From Python, they are
    available in the __builtin__ module, so no import is necessary.

    User code can define additional warning categories by subclassing
    one of the standard warning categories.  A warning category must
    always be a subclass of the Warning class.

The Warnings Filter

    The warnings filter control whether warnings are ignored,
    displayed, or turned into errors (raising an exception).

    There are three sides to the warnings filter:

    - The data structures used to efficiently determine the
      disposition of a particular warnings.warn() or PyErr_Warn()
      call.

    - The API to control the filter from Python source code.

    - The command line switches to control the filter.

    The warnings filter works in several stages.  It is optimized for
    the (expected to be common) case where the same warning is issued
    from the same place in the code over and over.

    First, the warning filter collects the module and line number
    where the warning is issued; this information is readily available
    through sys._getframe().

    Conceptually, the warnings filter maintains an ordered list of
    filter specifications; any specific warning is matched against
    each filter specification in the list in turn until a match is
    found; the match determines the disposition of the match.  Each
    entry is a tuple as follows:

      (category, message, module, lineno, action)

    - category is a class (a subclass of warnings.Warning) of which
      the warning category must be a subclass in order to match

    - message is a compiled regular expression that the warning
      message must match (the match is case-insensitive)

    - module is a compiled regular expression that the module name
      must match

    - lineno is an integer that the line number where the warning
      occurred must match, or 0 to match all line numbers

    - action is one of the following strings:

        - "error" -- turn matching warnings into exceptions

        - "ignore" -- never print matching warnings

        - "always" -- always print matching warnings

        - "default" -- print the first occurrence of matching warnings
          for each location where the warning is issued

        - "module" -- print the first occurrence of matching warnings
          for each module where the warning is issued

        - "once" -- print only the first occurrence of matching
          warnings

    Since the Warning class is derived from the built-in Exception
    class, to turn a warning into an error we simply raise
    category(message).

Warnings Output And Formatting Hooks

    When the warnings filter decides to issue a warning (but not when
    it decides to raise an exception), it passes the information about
    the function warnings.showwarning(message, category, filename, lineno).
    The default implementation of this function writes the warning text
    to sys.stderr, and shows the source line of the filename.  It has
    an optional 5th argument which can be used to specify a different
    file than sys.stderr.

    The formatting of warnings is done by a separate function,
    warnings.formatwarning(message, category, filename, lineno).  This
    returns a string (that may contain newlines and ends in a newline)
    that can be printed to get the identical effect of the
    showwarning() function.

API For Manipulating Warning Filters

      warnings.filterwarnings(message, category, module, lineno, action)

    This checks the types of the arguments, compiles the message and
    module regular expressions, and inserts them as a tuple in front
    of the warnings filter.

      warnings.resetwarnings()

    Reset the warnings filter to empty.

Command Line Syntax

    There should be command line options to specify the most common
    filtering actions, which I expect to include at least:

    - suppress all warnings

    - suppress a particular warning message everywhere

    - suppress all warnings in a particular module

    - turn all warnings into exceptions

    I propose the following command line option syntax:

    -Waction[:message[:category[:module[:lineno]]]]

    Where:

    - 'action' is an abbreviation of one of the allowed actions
      ("error", "default", "ignore", "always", "once", or "module")

    - 'message' is a message string; matches warnings whose message
      text is an initial substring of 'message' (matching is
      case-insensitive)

    - 'category' is an abbreviation of a standard warning category
      class name *or* a fully-qualified name for a user-defined
      warning category class of the form [package.]module.classname

    - 'module' is a module name (possibly package.module)

    - 'lineno' is an integral line number

    All parts except 'action' may be omitted, where an empty value
    after stripping whitespace is the same as an omitted value.

    The C code that parses the Python command line saves the body of
    all -W options in a list of strings, which is made available to
    the warnings module as sys.warnoptions.  The warnings module
    parses these when it is first imported.  Errors detected during
    the parsing of sys.warnoptions are not fatal; a message is written
    to sys.stderr and processing continues with the option.

    Examples:

    -Werror
        Turn all warnings into errors

    -Wall
        Show all warnings

    -Wignore
        Ignore all warnings

    -Wi:hello
        Ignore warnings whose message text starts with "hello"

    -We::Deprecation
        Turn deprecation warnings into errors

    -Wi:::spam:10
        Ignore all warnings on line 10 of module spam

    -Wi:::spam -Wd:::spam:10
        Ignore all warnings in module spam except on line 10

    -We::Deprecation -Wd::Deprecation:spam
        Turn deprecation warnings into errors except in module spam

Open Issues

    Some open issues off the top of my head:

    - What about issuing warnings during lexing or parsing, which
      don't have the exception machinery available?

    - The proposed command line syntax is a bit ugly (although the
      simple cases aren't so bad: -Werror, -Wignore, etc.).  Anybody
      got a better idea?

    - I'm a bit worried that the filter specifications are too
      complex.  Perhaps filtering only on category and module (not on
      message text and line number) would be enough?

    - There's a bit of confusion between module names and file names.
      The reporting uses file names, but the filter specification uses
      module names.  Maybe it should allow filenames as well?

    - I'm not at all convinced that packages are handled right.

    - Do we need more standard warning categories?  Fewer?

    - In order to minimize the start-up overhead, the warnings module
      is imported by the first call to PyErr_Warn().  It does the
      command line parsing for -W options upon import.  Therefore, it
      is possible that warning-free programs will not complain about
      invalid -W options.

Rejected Concerns

    Paul Prescod, Barry Warsaw and Fred Drake have brought up several
    additional concerns that I feel aren't critical.  I address them
    here (the concerns are paraphrased, not exactly their words):

    - Paul: warn() should be a built-in or a statement to make it easily
      available.

      Response: "from warnings import warn" is easy enough.

    - Paul: What if I have a speed-critical module that triggers
      warnings in an inner loop.  It should be possible to disable the
      overhead for detecting the warning (not just suppress the
      warning).

      Response: rewrite the inner loop to avoid triggering the
      warning.

    - Paul: What if I want to see the full context of a warning?

      Response: use -Werror to turn it into an exception.

    - Paul: I prefer ":*:*:" to ":::" for leaving parts of the warning
      spec out.

      Response: I don't.

    - Barry: It would be nice if lineno can be a range specification.

      Response: Too much complexity already.

    - Barry: I'd like to add my own warning action.  Maybe if `action'
      could be a callable as well as a string.  Then in my IDE, I
      could set that to "mygui.popupWarningsDialog".

      Response: For that purpose you would override
      warnings.showwarning().

    - Fred: why do the Warning category classes have to be in
      __builtin__?

      Response: that's the simplest implementation, given that the
      warning categories must be available in C before the first
      PyErr_Warn() call, which imports the warnings module.  I see no
      problem with making them available as built-ins.

Implementation

    Here's a prototype implementation:

  http://sourceforge.net/patch/?func=detailpatch&patch_id=102715&group_id=5470


PEP 231 -- __findattr__()

PEP: | 231
Title: | __findattr__()
Version: | $Revision: 1.4 $
Author: | barry@digicool.com (Barry A. Warsaw)
Python-Version: | 2.1
Status: | Draft
Created: | 30-Nov-2000
Post-History: |

Introduction

    This PEP describes an extension to instance attribute lookup and
    modification machinery, which allows pure-Python implementations
    of many interesting programming models.  This PEP tracks the
    status and ownership of this feature.  It contains a description
    of the feature and outlines changes necessary to support the
    feature.  This PEP summarizes discussions held in mailing list
    forums, and provides URLs for further information, where
    appropriate.  The CVS revision history of this file contains the
    definitive historical record.

Background

    The semantics for Python instances allow the programmer to
    customize some aspects of attribute lookup and attribute
    modification, through the special methods __getattr__() and
    __setattr__() [1].

    However, because of certain restrictions imposed by these methods,
    there are useful programming techniques that can not be written in
    Python alone, e.g. strict Java Bean-like[2] interfaces and Zope
    style acquisitions[3].  In the latter case, Zope solves this by
    including a C extension called ExtensionClass[5] which modifies
    the standard class semantics, and uses a metaclass hook in
    Python's class model called alternatively the "Don Beaudry Hook"
    or "Don Beaudry Hack"[6].

    While Zope's approach works, it has several disadvantages.  First,
    it requires a C extension.  Second it employs a very arcane, but
    truck-sized loophole in the Python machinery.  Third, it can be
    difficult for other programmers to use and understand (the
    metaclass has well-known brain exploding properties).  And fourth,
    because ExtensionClass instances aren't "real" Python instances,
    some aspects of the Python runtime system don't work with
    ExtensionClass instances.

    Proposals for fixing this problem have often been lumped under the
    rubric of fixing the "class/type dichotomy"; that is, eliminating
    the difference between built-in types and classes[7].  While a
    laudable goal itself, repairing this rift is not necessary in
    order to achieve the types of programming constructs described
    above.  This proposal provides an 80% solution with a minimum of
    modification to Python's class and instance objects.  It does
    nothing to address the type/class dichotomy.

Proposal

    This proposal adds a new special method called __findattr__() with
    the following semantics:

    * If defined in a class, it will be called on all instance
      attribute resolutions instead of __getattr__() and
      __setattr__().

    * __findattr__() is never called recursively.  That is, when a
      specific instance's __findattr__() is on the call stack, further
      attribute accesses for that instance will use the standard
      __getattr__() and __setattr__() methods.

    * __findattr__() is called for both attribute access (`getting')
      and attribute modification (`setting').  It is not called for
      attribute deletion.

    * When called for getting, it is passed a single argument (not
      counting `self'): the name of the attribute being accessed.

    * When called for setting, it is called with third argument, which
      is the value to set the attribute to.

    * __findattr__() methods have the same caching semantics as
      __getattr__() and __setattr__(); i.e. if they are present in the
      class at class definition time, they are used, but if they are
      subsequently added to a class later they are not.

Key Differences with the Existing Protocol

    __findattr__()'s semantics are different from the existing
    protocol in key ways:

    First, __getattr__() is never called if the attribute is found in
    the instance's __dict__.  This is done for efficiency reasons, and
    because otherwise, __setattr__() would have no way to get to the
    instance's attributes.

    Second, __setattr__() cannot use "normal" syntax for setting
    instance attributes, e.g. "self.name = foo" because that would
    cause recursive calls to __setattr__().

    __findattr__() is always called regardless of whether the
    attribute is in __dict__ or not, and a flag in the instance object
    prevents recursive calls to __findattr__().  This gives the class
    a chance to perform some action for every attribute access.  And
    because it is called for both gets and sets, it is easy to write
    similar policy for all attribute access.  Further, efficiency is
    not a problem because it is only paid when the extended mechanism
    is used.

Related Work

    PEP 213 [9] describes a different approach to hooking into
    attribute access and modification.  The semantics proposed in PEP
    213 can be implemented using the __findattr__() hook described
    here, with one caveat.  The current reference implementation of
    __findattr__() does not support hooking on attribute deletion.
    This could be added if it's found desirable.  See example below.

Examples

    One programming style that this proposal allows is a Java
    Bean-like interface to objects, where unadorned attribute access
    and modification is transparently mapped to a functional
    interface.  E.g.

        class Bean:
            def __init__(self, x):
                self.__myfoo = x

            def __findattr__(self, name, *args):
                if name.startswith('_'):
                    # Private names
                    if args: setattr(self, name, args[0])
                    else:    return getattr(self, name)
                else:
                    # Public names
                    if args: name = '_set_' + name
                    else:    name = '_get_' + name
                    return getattr(self, name)(*args)

            def _set_foo(self, x):
                self.__myfoo = x

            def _get_foo(self):
                return self.__myfoo

        b = Bean(3)
        print b.foo
        b.foo = 9
        print b.foo

    A second, more elaborate example is the implementation of both
    implicit and explicit acquisition in pure Python:

        import types

        class MethodWrapper:
            def __init__(self, container, method):
                self.__container = container
                self.__method = method

            def __call__(self, *args, **kws):
                return self.__method.im_func(self.__container, *args, **kws)

        class WrapperImplicit:
            def __init__(self, contained, container):
                self.__contained = contained
                self.__container = container

            def __repr__(self):
                return '<Wrapper: [%s | %s]>' % (self.__container,
                                                 self.__contained)

            def __findattr__(self, name, *args):
                # Some things are our own
                if name.startswith('_WrapperImplicit__'):
                    if args: return setattr(self, name, *args)
                    else:    return getattr(self, name)
                # setattr stores the name on the contained object directly
                if args:
                    return setattr(self.__contained, name, args[0])
                # Other special names
                if name == 'aq_parent':
                    return self.__container
                elif name == 'aq_self':
                    return self.__contained
                elif name == 'aq_base':
                    base = self.__contained
                    try:
                        while 1:
                            base = base.aq_self
                    except AttributeError:
                        return base
                # no acquisition for _ names
                if name.startswith('_'):
                    return getattr(self.__contained, name)
                # Everything else gets wrapped
                missing = []
                which = self.__contained
                obj = getattr(which, name, missing)
                if obj is missing:
                    which = self.__container
                    obj = getattr(which, name, missing)
                    if obj is missing:
                        raise AttributeError, name
                of = getattr(obj, '__of__', missing)
                if of is not missing:
                    return of(self)
                elif type(obj) == types.MethodType:
                    return MethodWrapper(self, obj)
                return obj

        class WrapperExplicit:
            def __init__(self, contained, container):
                self.__contained = contained
                self.__container = container

            def __repr__(self):
                return '<Wrapper: [%s | %s]>' % (self.__container,
                                                 self.__contained)

            def __findattr__(self, name, *args):
                # Some things are our own
                if name.startswith('_WrapperExplicit__'):
                    if args: return setattr(self, name, *args)
                    else:    return getattr(self, name)
                # setattr stores the name on the contained object directly
                if args:
                    return setattr(self.__contained, name, args[0])
                # Other special names
                if name == 'aq_parent':
                    return self.__container
                elif name == 'aq_self':
                    return self.__contained
                elif name == 'aq_base':
                    base = self.__contained
                    try:
                        while 1:
                            base = base.aq_self
                    except AttributeError:
                        return base
                elif name == 'aq_acquire':
                    return self.aq_acquire
                # explicit acquisition only
                obj = getattr(self.__contained, name)
                if type(obj) == types.MethodType:
                    return MethodWrapper(self, obj)
                return obj

            def aq_acquire(self, name):
                # Everything else gets wrapped
                missing = []
                which = self.__contained
                obj = getattr(which, name, missing)
                if obj is missing:
                    which = self.__container
                    obj = getattr(which, name, missing)
                    if obj is missing:
                        raise AttributeError, name
                of = getattr(obj, '__of__', missing)
                if of is not missing:
                    return of(self)
                elif type(obj) == types.MethodType:
                    return MethodWrapper(self, obj)
                return obj

        class Implicit:
            def __of__(self, container):
                return WrapperImplicit(self, container)

            def __findattr__(self, name, *args):
                # ignore setattrs
                if args:
                    return setattr(self, name, args[0])
                obj = getattr(self, name)
                missing = []
                of = getattr(obj, '__of__', missing)
                if of is not missing:
                    return of(self)
                return obj

        class Explicit(Implicit):
            def __of__(self, container):
                return WrapperExplicit(self, container)

        # tests
        class C(Implicit):
            color = 'red'

        class A(Implicit):
            def report(self):
                return self.color

        # simple implicit acquisition
        c = C()
        a = A()
        c.a = a
        assert c.a.report() == 'red'

        d = C()
        d.color = 'green'
        d.a = a
        assert d.a.report() == 'green'

        try:
            a.report()
        except AttributeError:
            pass
        else:
            assert 0, 'AttributeError expected'

        # special names
        assert c.a.aq_parent is c
        assert c.a.aq_self is a

        c.a.d = d
        assert c.a.d.aq_base is d
        assert c.a is not a

        # no acquisiton on _ names
        class E(Implicit):
            _color = 'purple'

        class F(Implicit):
            def report(self):
                return self._color

        e = E()
        f = F()
        e.f = f
        try:
            e.f.report()
        except AttributeError:
            pass
        else:
            assert 0, 'AttributeError expected'

        # explicit
        class G(Explicit):
            color = 'pink'

        class H(Explicit):
            def report(self):
                return self.aq_acquire('color')

            def barf(self):
                return self.color

        g = G()
        h = H()
        g.h = h
        assert g.h.report() == 'pink'

        i = G()
        i.color = 'cyan'
        i.h = h
        assert i.h.report() == 'cyan'

        try:
            g.i.barf()
        except AttributeError:
            pass
        else:
            assert 0, 'AttributeError expected'

    C++-like access control can also be accomplished, although less
    cleanly because of the difficulty of figuring out what method is
    being called from the runtime call stack:

        import sys
        import types

        PUBLIC = 0
        PROTECTED = 1
        PRIVATE = 2

        try:
            getframe = sys._getframe
        except ImportError:
            def getframe(n):
                try: raise Exception
                except Exception:
                    frame = sys.exc_info()[2].tb_frame
                while n > 0:
                    frame = frame.f_back
                    if frame is None:
                        raise ValueError, 'call stack is not deep enough'
                return frame

        class AccessViolation(Exception):
            pass

        class Access:
            def __findattr__(self, name, *args):
                methcache = self.__dict__.setdefault('__cache__', {})
                missing = []
                obj = getattr(self, name, missing)
                # if obj is missing we better be doing a setattr for
                # the first time
                if obj is not missing and type(obj) == types.MethodType:
                    # Digusting hack because there's no way to
                    # dynamically figure out what the method being
                    # called is from the stack frame.
                    methcache[obj.im_func.func_code] = obj.im_class
                #
                # What's the access permissions for this name?
                access, klass = getattr(self, '__access__', {}).get(
                    name, (PUBLIC, 0))
                if access is not PUBLIC:
                    # Now try to see which method is calling us
                    frame = getframe(0).f_back
                    if frame is None:
                        raise AccessViolation
                    # Get the class of the method that's accessing
                    # this attribute, by using the code object cache
                    if frame.f_code.co_name == '__init__':
                        # There aren't entries in the cache for ctors,
                        # because the calling mechanism doesn't go
                        # through __findattr__().  Are there other
                        # methods that might have the same behavior?
                        # Since we can't know who's __init__ we're in,
                        # for now we'll assume that only protected and
                        # public attrs can be accessed.
                        if access is PRIVATE:
                            raise AccessViolation
                    else:
                        methclass = self.__cache__.get(frame.f_code)
                        if not methclass:
                            raise AccessViolation
                        if access is PRIVATE and methclass is not klass:
                            raise AccessViolation
                        if access is PROTECTED and not issubclass(methclass,
                                                                  klass):
                            raise AccessViolation
                # If we got here, it must be okay to access the attribute
                if args:
                    return setattr(self, name, *args)
                return obj

        # tests
        class A(Access):
            def __init__(self, foo=0, name='A'):
                self._foo = foo
                # can't set private names in __init__
                self.__initprivate(name)

            def __initprivate(self, name):
                self._name = name

            def getfoo(self):
                return self._foo

            def setfoo(self, newfoo):
                self._foo = newfoo

            def getname(self):
                return self._name

        A.__access__ = {'_foo'      : (PROTECTED, A),
                        '_name'     : (PRIVATE, A),
                        '__dict__'  : (PRIVATE, A),
                        '__access__': (PRIVATE, A),
                        }

        class B(A):
            def setfoo(self, newfoo):
                self._foo = newfoo + 3

            def setname(self, name):
                self._name = name

        b = B(1)
        b.getfoo()

        a = A(1)
        assert a.getfoo() == 1
        a.setfoo(2)
        assert a.getfoo() == 2

        try:
            a._foo
        except AccessViolation:
            pass
        else:
            assert 0, 'AccessViolation expected'

        try:
            a._foo = 3
        except AccessViolation:
            pass
        else:
            assert 0, 'AccessViolation expected'

        try:
            a.__dict__['_foo']
        except AccessViolation:
            pass
        else:
            assert 0, 'AccessViolation expected'

        b = B()
        assert b.getfoo() == 0
        b.setfoo(2)
        assert b.getfoo() == 5
        try:
            b.setname('B')
        except AccessViolation:
            pass
        else:
            assert 0, 'AccessViolation expected'

        assert b.getname() == 'A'

    Here's an implementation of the attribute hook described in PEP
    213 (except that hooking on attribute deletion isn't supported by
    the current reference implementation).

        class Pep213:
            def __findattr__(self, name, *args):
                hookname = '__attr_%s__' % name
                if args:
                    op = 'set'
                else:
                    op = 'get'
                # XXX: op = 'del' currently not supported
                missing = []
                meth = getattr(self, hookname, missing)
                if meth is missing:
                    if op == 'set':
                        return setattr(self, name, *args)
                    else:
                        return getattr(self, name)
                else:
                    return meth(op, *args)

        def computation(i):
            print 'doing computation:', i
            return i + 3

        def rev_computation(i):
            print 'doing rev_computation:', i
            return i - 3

        class X(Pep213):
            def __init__(self, foo=0):
                self.__foo = foo

            def __attr_foo__(self, op, val=None):
                if op == 'get':
                    return computation(self.__foo)
                elif op == 'set':
                    self.__foo = rev_computation(val)
                # XXX: 'del' not yet supported

        x = X()
        fooval = x.foo
        print fooval
        x.foo = fooval + 5
        print x.foo
        # del x.foo

Reference Implementation

   The reference implementation, as a patch to the Python core, can be
   found at this URL:

   http://sourceforge.net/patch/?func=detailpatch&patch_id=102613&group_id=5470

References

    [1] http://www.python.org/doc/current/ref/attribute-access.html
    [2] http://www.javasoft.com/products/javabeans/
    [3] http://www.digicool.com/releases/ExtensionClass/Acquisition.html
    [5] http://www.digicool.com/releases/ExtensionClass
    [6] http://www.python.org/doc/essays/metaclasses/
    [7] http://www.foretec.com/python/workshops/1998-11/dd-ascher-sum.html
    [8] http://www.python.org/doc/howto/rexec/rexec.html
    [9] http://python.sourceforge.net/peps/pep-0213.html

Rejection

    There are serious problems with the recursion-protection feature.
    As described here it's not thread-safe, and a thread-safe solution
    has other problems.  In general, it's not clear how helpful the
    recursion-protection feature is; it makes it hard to write code
    that needs to be callable inside __findattr__ as well as outside
    it.  But without the recursion-protection, it's hard to implement
    __findattr__ at all (since __findattr__ would invoke itself
    recursively for every attribute it tries to access).  There seems
    to be no good solution here.

    It's also dubious how useful it is to support __findattr__ both
    for getting and for setting attributes -- __setattr__ gets called
    in all cases alrady.

    The examples can all be implemented using __getattr__ if care is
    taken not to store instance variables under their own names.

Copyright

    This document has been placed in the Public Domain.


PEP 232 -- Function Attributes

PEP: | 232
Title: | Function Attributes
Version: | $Revision: 1.10 $
Author: | barry@digicool.com (Barry A. Warsaw)
Status: | Final
Type: | Standards Track
Created: | 02-Dec-2000
Python-Version: | 2.1
Post-History: | 20-Feb-2001

Introduction

    This PEP describes an extension to Python, adding attribute
    dictionaries to functions and methods.  This PEP tracks the status
    and ownership of this feature.  It contains a description of the
    feature and outlines changes necessary to support the feature.
    This PEP summarizes discussions held in mailing list forums, and
    provides URLs for further information, where appropriate.  The CVS
    revision history of this file contains the definitive historical
    record.

Background

    Functions already have a number of attributes, some of which are
    writable, e.g. func_doc, a.k.a. func.__doc__.  func_doc has the
    interesting property that there is special syntax in function (and
    method) definitions for implicitly setting the attribute.  This
    convenience has been exploited over and over again, overloading
    docstrings with additional semantics.

    For example, John Aycock has written a system where docstrings are
    used to define parsing rules[1].  Zope's ZPublisher ORB[2] uses
    docstrings to signal "publishable" methods, i.e. methods that can
    be called through the web.

    The problem with this approach is that the overloaded semantics
    may conflict with each other.  For example, if we wanted to add a
    doctest unit test to a Zope method that should not be publishable
    through the web.

Proposal

    This proposal adds a new dictionary to function objects, called
    func_dict (a.k.a. __dict__).  This dictionary can be set and get
    using ordinary attribute set and get syntax.

    Methods also gain `getter' syntax, and they currently access the
    attribute through the dictionary of the underlying function
    object.  It is not possible to set attributes on bound or unbound
    methods, except by doing so explicitly on the underlying function
    object.  See the `Future Directions' discussion below for
    approaches in subsequent versions of Python.

    A function object's __dict__ can also be set, but only to a
    dictionary object or None (e.g. setting __dict__ to UserDict
    raises a TypeError).  Deleting a function's __dict__ attribute is
    equivalent to setting it to None.  If no function attributes have
    ever been set, the function's __dict__ will be None.

Examples

    Here are some examples of what you can do with this feature.

        def a():
        pass

    a.publish = 1
    a.unittest = '''...'''

    if a.publish:
        print a()

    if hasattr(a, 'unittest'):
        testframework.execute(a.unittest)

    class C:
        def a(self):
            'just a docstring'
            a.publish = 1

    c = C()
    if c.a.publish:
            publish(c.a())

Other Uses

    Paul Prescod enumerated a bunch of other uses:

    http://mail.python.org/pipermail/python-dev/2000-April/003364.html

Future Directions

    Here are a number of future directions to consider.  Any adoption
    of these ideas would require a new PEP, which referenced this one,
    and would have to be targeted at a Python version subsequent to
    the 2.1 release.

    - A previous version of this PEP allowed for both setter and
      getter of attributes on unbound methods, and only getter on
      bound methods.  A number of problems were discovered with this
      policy.

      Because method attributes were stored in the underlying
      function, this caused several potentially surprising results:

      class C:
          def a(self): pass

      c1 = C()
      c2 = C()
      c1.a.publish = 1
      # c2.a.publish would now be == 1 also!

      Because a change to `a' bound c1 also caused a change to `a'
      bound to c2, setting of attributes on bound methods was
      disallowed.  However, even allowing setting of attributes on
      unbound methods has its ambiguities:

      class D(C): pass
      class E(C): pass

      D.a.publish = 1
      # E.a.publish would now be == 1 also!

      For this reason, the current PEP disallows setting attributes on
      either bound or unbound methods, but does allow for getting
      attributes on either -- both return the attribute value on the
      underlying function object.

      A future PEP might propose to implement setting (bound or
      unbound) method attributes by setting attributes on the instance
      or class, using special naming conventions.  I.e.

      class C:
          def a(self): pass

      C.a.publish = 1
      C.__a_publish__ == 1 # true

      c = C()
      c.a.publish = 2
      c.__a_publish__ == 2 # true

      d = C()
      d.__a_publish__ == 1 # true

      Here, a lookup on the instance would look to the instance's
      dictionary first, followed by a lookup on the class's
      dictionary, and finally a lookup on the function object's
      dictionary.

    - Currently, Python supports function attributes only on Python
      functions (i.e. those that are written in Python, not those that
      are built-in).  Should it be worthwhile, a separate patch can be
      crafted that will add function attributes to built-ins.

    - __doc__ is the only function attribute that currently has
      syntactic support for conveniently setting.  It may be
      worthwhile to eventually enhance the language for supporting
      easy function attribute setting.  Here are some syntaxes
      suggested by PEP reviewers:

      def a {
          'publish' : 1,
          'unittest': '''...''',
          }
          (args):
          # ...

      def a(args):
          """The usual docstring."""
          {'publish' : 1,
           'unittest': '''...''',
           # etc.
           }

      def a(args) having (publish = 1):
          # see reference [3]
          pass

      The BDFL is currently against any such special syntactic support
      for setting arbitrary function attributes.  Any syntax proposals
      would have to be outlined in new PEPs.

Dissenting Opinion

    When this was discussed on the python-dev mailing list in April
    2000, a number of dissenting opinions were voiced.  For
    completeness, the discussion thread starts here:

    http://mail.python.org/pipermail/python-dev/2000-April/003361.html

    The dissenting arguments appear to fall under the following
    categories:

    - no clear purpose (what does it buy you?)
    - other ways to do it (e.g. mappings as class attributes)
    - useless until syntactic support is included

    Countering some of these arguments is the observation that with
    vanilla Python 2.0, __doc__ can in fact be set to any type of
    object, so some semblance of writable function attributes are
    already feasible.  But that approach is yet another corruption of
    __doc__.

    And while it is of course possible to add mappings to class
    objects (or in the case of function attributes, to the function's
    module), it is more difficult and less obvious how to extract the
    attribute values for inspection.

    Finally, it may be desirable to add syntactic support, much the
    same way that __doc__ syntactic support exists.  This can be
    considered separately from the ability to actually set and get
    function attributes.

Reference Implementation

    This PEP has been accepted and the implementation has been
    integrated into Python 2.1.

References

    [1] Aycock, "Compiling Little Languages in Python",
    http://www.foretec.com/python/workshops/1998-11/proceedings/papers/aycock-little/aycock-little.html

    [2] http://classic.zope.org:8080/Documentation/Reference/ORB

    [3] Hudson, Michael, SourceForge patch implementing this syntax,
    http://sourceforge.net/tracker/index.php?func=detail&aid=403441&group_id=5470&atid=305470

Copyright

    This document has been placed in the Public Domain.


PEP 233 -- Python Online Help

PEP: | 233
Title: | Python Online Help
Version: | $Revision: 1.2 $
Author: | paul@prescod.net, paulp@activestate.com (Paul Prescod)
Status: | Deferred
Type: | Standards Track
Created: | 11-Dec-2000
Python-Version: | 2.1
Post-History: |

Abstract

    This PEP describes a command-line driven online help facility for
    Python.  The facility should be able to build on existing
    documentation facilities such as the Python documentation and
    docstrings.  It should also be extensible for new types and
    modules.

Interactive use:

    Simply typing "help" describes the help function (through repr()
    overloading).

    "help" can also be used as a function:

    The function takes the following forms of input:

        help( "string" ) -- built-in topic or global
        help( <ob> ) -- docstring from object or type
        help( "doc:filename" ) -- filename from Python documentation

    If you ask for a global, it can be a fully-qualified name such as
    help("xml.dom").

    You can also use the facility from a command-line

    python --help if

    In either situation, the output does paging similar to the "more"
    command.

Implementation

    The help function is implemented in an onlinehelp module which is
    demand-loaded.

    There should be options for fetching help information from
    environments other than the command line through the onlinehelp
    module:

        onlinehelp.gethelp(object_or_string) -> string

    It should also be possible to override the help display function
    by assigning to onlinehelp.displayhelp(object_or_string).

    The module should be able to extract module information from
    either the HTML or LaTeX versions of the Python documentation.
    Links should be accommodated in a "lynx-like" manner.

    Over time, it should also be able to recognize when docstrings are
    in "special" syntaxes like structured text, HTML and LaTeX and
    decode them appropriately.

    A prototype implementation is available with the Python source
    distribution as nondist/sandbox/doctools/onlinehelp.py.

Built-in Topics

    help( "intro" )  - What is Python? Read this first!
    help( "keywords" )  - What are the keywords?
    help( "syntax" )  - What is the overall syntax?
    help( "operators" )  - What operators are available?
    help( "builtins" )  - What functions, types, etc. are built-in?
    help( "modules" )  - What modules are in the standard library?
    help( "copyright" )  - Who owns Python?
    help( "moreinfo" )  - Where is there more information?
    help( "changes" )  - What changed in Python 2.0?
    help( "extensions" )  - What extensions are installed?
    help( "faq" )  - What questions are frequently asked?
    help( "ack" )  - Who has done work on Python lately?

Security Issues

    This module will attempt to import modules with the same names as
    requested topics.  Don't use the modules if you are not confident
    that everything in your PYTHONPATH is from a trusted source.


PEP 234 -- Iterators

PEP: | 234
Title: | Iterators
Version: | $Revision: 1.15 $
Author: | ping@lfw.org (Ka-Ping Yee), guido@python.org (Guido van Rossum)
Status: | Draft
Type: | Standards Track
Python-Version: | 2.1
Created: | 30-Jan-2001
Post-History: | 30-Apr-2001

Abstract

    This document proposes an iteration interface that objects can
    provide to control the behaviour of 'for' loops.  Looping is
    customized by providing a method that produces an iterator object.
    The iterator provides a 'get next value' operation that produces
    the next item in the sequence each time it is called, raising an
    exception when no more items are available.

    In addition, specific iterators over the keys of a dictionary and
    over the lines of a file are proposed, and a proposal is made to
    allow spelling dict.has_key(key) as "key in dict".

    Note: this is an almost complete rewrite of this PEP by the second
    author, describing the actual implementation checked into the
    trunk of the Python 2.2 CVS tree.  It is still open for
    discussion.  Some of the more esoteric proposals in the original
    version of this PEP have been withdrawn for now; these may be the
    subject of a separate PEP in the future.

C API Specification

    A new exception is defined, StopIteration, which can be used to
    signal the end of an iteration.

    A new slot named tp_iter for requesting an iterator is added to
    the type object structure.  This should be a function of one
    PyObject * argument returning a PyObject *, or NULL.  To use this
    slot, a new C API function PyObject_GetIter() is added, with the
    same signature as the tp_iter slot function.

    Another new slot, named tp_iternext, is added to the type
    structure, for obtaining the next value in the iteration.  To use
    this slot, a new C API function PyIter_Next() is added.  The
    signature for both the slot and the API function is as follows,
    although the NULL return conditions differ:  the argument is a
    PyObject * and so is the return value.  When the return value is
    non-NULL, it is the next value in the iteration.  When it is NULL,
    then for the tp_iternext slot there are three possibilities:

    - No exception is set; this implies the end of the iteration.

    - The StopIteration exception (or a derived exception class) is
      set; this implies the end of the iteration.

    - Some other exception is set; this means that an error occurred
      that should be propagated normally.

    The higher-level PyIter_Next() function clears the StopIteration
    exception (or derived exception) when it occurs, so its NULL return
    conditions are simpler:

    - No exception is set; this means iteration has ended.

    - Some exception is set; this means an error occurred, and should
      be propagated normally.

    In addition to the tp_iternext slot, every iterator object must
    also implement a next() method, callable without arguments.  This
    should have the same semantics as the tp_iternext slot function,
    except that the only way to signal the end of the iteration is to
    raise StopIteration.  The iterator object should not care whether
    its tp_iternext slot function is called or its next() method, and
    the caller may mix calls arbitrarily.  (The next() method is for
    the benefit of Python code using iterators directly; the
    tp_iternext slot is added to make 'for' loops more efficient.)

    To ensure binary backwards compatibility, a new flag
    Py_TPFLAGS_HAVE_ITER is added to the set of flags in the tp_flags
    field, and to the default flags macro.  This flag must be tested
    before accessing the tp_iter or tp_iternext slots.  The macro
    PyIter_Check() tests whether an object has the appropriate flag
    set and has a non-NULL tp_iternext slot.  There is no such macro
    for the tp_iter slot (since the only place where this slot is
    referenced should be PyObject_GetIter()).

    (Note: the tp_iter slot can be present on any object; the
    tp_iternext slot should only be present on objects that act as
    iterators.)

    For backwards compatibility, the PyObject_GetIter() function
    implements fallback semantics when its argument is a sequence that
    does not implement a tp_iter function: a lightweight sequence
    iterator object is constructed in that case which iterates over
    the items of the sequence in the natural order.

    The Python bytecode generated for 'for' loops is changed to use
    new opcodes, GET_ITER and FOR_ITER, that use the iterator protocol
    rather than the sequence protocol to get the next value for the
    loop variable.  This makes it possible to use a 'for' loop to loop
    over non-sequence objects that support the tp_iter slot.  Other
    places where the interpreter loops over the values of a sequence
    should also be changed to use iterators.

    Iterators ought to implement the tp_iter slot as returning a
    reference to themselves; this is needed to make it possible to
    use an iterator (as opposed to a sequence) in a for loop.

Python API Specification

    The StopIteration exception is made visible as one of the
    standard exceptions.  It is derived from Exception.

    A new built-in function is defined, iter(), which can be called in
    two ways:

    - iter(obj) calls PyObject_GetIter(obj).

    - iter(callable, sentinel) returns a special kind of iterator that
      calls the callable to produce a new value, and compares the
      return value to the sentinel value.  If the return value equals
      the sentinel, this signals the end of the iteration and
      StopIteration is raised rather than returning normal; if the
      return value does not equal the sentinel, it is returned as the
      next value from the iterator.  If the callable raises an
      exception, this is propagated normally; in particular, the
      function is allowed to raise StopIteration as an alternative way
      to end the iteration.  (This functionality is available from the
      C API as PyCallIter_New(callable, sentinel).)

    Iterator objects returned by either form of iter() have a next()
    method.  This method either returns the next value in the
    iteration, or raises StopIteration (or a derived exception class)
    to signal the end of the iteration.  Any other exception should be
    considered to signify an error and should be propagated normally,
    not taken to mean the end of the iteration.

    Classes can define how they are iterated over by defining an
    __iter__() method; this should take no additional arguments and
    return a valid iterator object.  A class is a valid iterator
    object when it defines a next() method that behaves as described
    above.  A class that wants to be an iterator also ought to
    implement __iter__() returning itself.

Dictionary Iterators

    The following two proposals are somewhat controversial.  They are
    also independent from the main iterator implementation.  However,
    they are both very useful.

    - Dictionaries implement a sq_contains slot that implements the
      same test as the has_key() method.  This means that we can write

          if k in dict: ...

      which is equivalent to

          if dict.has_key(k): ...

    - Dictionaries implement a tp_iter slot that returns an efficient
      iterator that iterates over the keys of the dictionary.  During
      such an iteration, the dictionary should not be modified, except
      that setting the value for an existing key is allowed (deletions
      or additions are not, nor is the update() method).  This means
      that we can write

          for k in dict: ...

      which is equivalent to, but much faster than

          for k in dict.keys(): ...

      as long as the restriction on modifications to the dictionary
      (either by the loop or by another thread) are not violated.

    - Add methods to dictionaries that return different kinds of
      iterators explicitly:

          for key in dict.iterkeys(): ...

          for value in dict.itervalues(): ...

          for key, value in dict.iteritems(): ...

      This means that "for x in dict" is shorthand for "for x in
      dict.iterkeys()".

    If this proposal is accepted, it makes sense to recommend that
    other mappings, if they support iterators at all, should also
    iterate over the keys.  However, this should not be taken as an
    absolute rule; specific applications may have different
    requirements.

File Iterators

    The following proposal is not controversial, but should be
    considered a separate step after introducing the iterator
    framework described above.  It is useful because it provides us
    with a good answer to the complaint that the common idiom to
    iterate over the lines of a file is ugly and slow.

    - Files implement a tp_iter slot that is equivalent to
      iter(f.readline, "").  This means that we can write

          for line in file:
              ...

      as a shorthand for

          for line in iter(file.readline, ""):
              ...

      which is equivalent to, but faster than

          while 1:
              line = file.readline()
              if not line:
                  break
              ...

    This also shows that some iterators are destructive: they consume
    all the values and a second iterator cannot easily be created that
    iterates independently over the same values.  You could open the
    file for a second time, or seek() to the beginning, but these
    solutions don't work for all file types, e.g. they don't work when
    the open file object really represents a pipe or a stream socket.

Rationale

    If all the parts of the proposal are included, this addresses many
    concerns in a consistent and flexible fashion.  Among its chief
    virtues are the following four -- no, five -- no, six -- points:

    1. It provides an extensible iterator interface.

    2. It allows performance enhancements to list iteration.

    3. It allows big performance enhancements to dictionary iteration.

    4. It allows one to provide an interface for just iteration
       without pretending to provide random access to elements.

    5. It is backward-compatible with all existing user-defined
       classes and extension objects that emulate sequences and
       mappings, even mappings that only implement a subset of
       {__getitem__, keys, values, items}.

    6. It makes code iterating over non-sequence collections more
       concise and readable.

Open Issues

    The following questions are still open.

    - The name iter() is an abbreviation.  Alternatives proposed
      include iterate(), traverse(), but these appear too long.
      Python has a history of using abbrs for common builtins,
      e.g. repr(), str(), len().

    - Using the same name for two different operations (getting an
      iterator from an object and making an iterator for a function
      with an sentinel value) is somewhat ugly.  I haven't seen a
      better name for the second operation though, and since they both
      return an iterator, it's easy to remember.

    - Once a particular iterator object has raised StopIteration, will
      it also raise StopIteration on all subsequent next() calls?
      Some say that it would be useful to require this, others say
      that it is useful to leave this open to individual iterators.
      Note that this may require an additional state bit for some
      iterator implementations (e.g. function-wrapping iterators).

    - It has been proposed that a file object should be its own
      iterator, with a next() method returning the next line.  This
      has certain advantages, and makes it even clearer that this
      iterator is destructive.  The disadvantage is that this would
      make it even more painful to implement the "sticky
      StopIteration" feature proposed in the previous bullet.

    - Some folks have requested extensions of the iterator protocol,
      e.g. prev() to get the previous item, current() to get the
      current item again, finished() to test whether the iterator is
      finished, and maybe even others, like rewind(), __len__(),
      position().

      While some of these are useful, many of these cannot easily be
      implemented for all iterator types without adding arbitrary
      buffering, and sometimes they can't be implemented at all (or
      not reasonably).  E.g. anything to do with reversing directions
      can't be done when iterating over a file or function.  Maybe a
      separate PEP can be drafted to standardize the names for such
      operations when the are implementable.

    - There is still discussion about whether

          for x in dict: ...

      should assign x the successive keys, values, or items of the
      dictionary.  The symmetry between "if x in y" and "for x in y"
      suggests that it should iterate over keys.  This symmetry has been
      observed by many independently and has even been used to "explain"
      one using the other.  This is because for sequences, "if x in y"
      iterates over y comparing the iterated values to x.  If we adopt
      both of the above proposals, this will also hold for
      dictionaries.

      The argument against making "for x in dict" iterate over the keys
      comes mostly from a practicality point of view: scans of the
      standard library show that there are about as many uses of "for x
      in dict.items()" as there are of "for x in dict.keys()", with the
      items() version having a small majority.  Presumably many of the
      loops using keys() use the corresponding value anyway, by writing
      dict[x], so (the argument goes) by making both the key and value
      available, we could support the largest number of cases.  While
      this is true, I (Guido) find the correspondence between "for x in
      dict" and "if x in dict" too compelling to break, and there's not
      much overhead in having to write dict[x] to explicitly get the
      value.

      For fast iteration over items, use "for key, value in
      dict.iteritems()".  I've timed the difference between

          for key in dict: dict[key]

      and

          for key, value in dict.iteritems(): pass

      and found that the latter is only about 7% faster.

Resolved Issues

    The following topics have been decided by consensus or BDFL
    pronouncement.

    - Two alternative spellings for next() have been proposed but
      rejected: __next__(), because it corresponds to a type object
      slot (tp_iternext); and __call__(), because this is the only
      operation.

      Arguments against __next__(): while many iterators are used in
      for loops, it is expected that user code will also call next()
      directly, so having to write __next__() is ugly; also, a
      possible extension of the protocol would be to allow for prev(),
      current() and reset() operations; surely we don't want to use
      __prev__(), __current__(), __reset__().

      Arguments against __call__() (the original proposal): taken out
      of context, x() is not very readable, while x.next() is clear;
      there's a danger that every special-purpose object wants to use
      __call__() for its most common operation, causing more confusion
      than clarity.

    - Some folks have requested the ability to restart an iterator.
      This should be dealt with by calling iter() on a sequence
      repeatedly, not by the iterator protocol itself.

    - It has been questioned whether an exception to signal the end of
      the iteration isn't too expensive.  Several alternatives for the
      StopIteration exception have been proposed: a special value End
      to signal the end, a function end() to test whether the iterator
      is finished, even reusing the IndexError exception.

      - A special value has the problem that if a sequence ever
        contains that special value, a loop over that sequence will
        end prematurely without any warning.  If the experience with
        null-terminated C strings hasn't taught us the problems this
        can cause, imagine the trouble a Python introspection tool
        would have iterating over a list of all built-in names,
        assuming that the special End value was a built-in name!

      - Calling an end() function would require two calls per
        iteration.  Two calls is much more expensive than one call
        plus a test for an exception.  Especially the time-critical
        for loop can test very cheaply for an exception.

      - Reusing IndexError can cause confusion because it can be a
        genuine error, which would be masked by ending the loop
        prematurely.

    - Some have asked for a standard iterator type.  Presumably all
      iterators would have to be derived from this type.  But this is
      not the Python way: dictionaries are mappings because they
      support __getitem__() and a handful other operations, not
      because they are derived from an abstract mapping type.

    - Regarding "if key in dict": there is no doubt that the
      dict.has_keys(x) interpretation of "x in dict" is by far the
      most useful interpretation, probably the only useful one.  There
      has been resistance against this because "x in list" checks
      whether x is present among the values, while the proposal makes
      "x in dict" check whether x is present among the keys.  Given
      that the symmetry between lists and dictionaries is very weak,
      this argument does not have much weight.

Mailing Lists

    The iterator protocol has been discussed extensively in a mailing
    list on SourceForge:

        http://lists.sourceforge.net/lists/listinfo/python-iterators

    Initially, some of the discussion was carried out at Yahoo;
    archives are still accessible:

        http://groups.yahoo.com/group/python-iter

Copyright

    This document is in the public domain.


PEP 235 -- Import on Case-Insensitive Platforms

PEP: | 235
Title: | Import on Case-Insensitive Platforms
Version: | $Revision: 1.3 $
Author: | Tim Peters
Status: | Final
Type: | Standards Track
Python-Version: | 2.1
Post-History: | 16 February 2001

Note

    This is essentially a retroactive PEP: the issue came up too late
    in the 2.1 release process to solicit wide opinion before deciding
    what to do, and can't be put off until 2.2 without also delaying
    the Cygwin and MacOS X ports.

Motivation

    File systems vary across platforms in whether or not they preserve
    the case of filenames, and in whether or not the platform C
    library file-opening functions do or don't insist on
    case-sensitive matches:

                      case-preserving     case-destroying
                     +-------------------+------------------+
    case-sensitive   | most Unix flavors | brrrrrrrrrr      |
                     +-------------------+------------------+
    case-insensitive | Windows           | some unfortunate |
                     | MacOSX HFS+       | network schemes  |
                     | Cygwin            |                  |
                     |                   | OpenVMS          |
                     +-------------------+------------------+

    In the upper left box, if you create "fiLe" it's stored as "fiLe",
    and only open("fiLe") will open it (open("file") will not, nor
    will the 14 other variations on that theme).

    In the lower right box, if you create "fiLe", there's no telling
    what it's stored as -- but most likely as "FILE" -- and any of the
    16 obvious variations on open("FilE") will open it.

    The lower left box is a mix: creating "fiLe" stores "fiLe" in the
    platform directory, but you don't have to match case when opening
    it; any of the 16 obvious variations on open("FILe") work.

    NONE OF THAT IS CHANGING!  Python will continue to follow platform
    conventions w.r.t. whether case is preserved when creating a file,
    and w.r.t. whether open() requires a case-sensitive match.  In
    practice, you should always code as if matches were
    case-sensitive, else your program won't be portable.

    What's proposed is to change the semantics of Python "import"
    statements, and there *only* in the lower left box.

Current Lower-Left Semantics

    Support for MacOSX HFS+, and for Cygwin, is new in 2.1, so nothing
    is changing there.  What's changing is Windows behavior.  Here are
    the current rules for import on Windows:

    1. Despite that the filesystem is case-insensitive, Python insists
       on a case-sensitive match.  But not in the way the upper left
       box works: if you have two files, FiLe.py and file.py on
       sys.path, and do

           import file

       then if Python finds FiLe.py first, it raises a NameError.  It
       does *not* go on to find file.py; indeed, it's impossible to
       import any but the first case-insensitive match on sys.path,
       and then only if case matches exactly in the first
       case-insensitive match.

    2. An ugly exception: if the first case-insensitive match on
       sys.path is for a file whose name is entirely in upper case
       (FILE.PY or FILE.PYC or FILE.PYO), then the import silently
       grabs that, no matter what mixture of case was used in the
       import statement.  This is apparently to cater to miserable old
       filesystems that really fit in the lower right box.  But this
       exception is unique to Windows, for reasons that may or may not
       exist.

    3. And another exception: if the environment variable PYTHONCASEOK
       exists, Python silently grabs the first case-insensitive match
       of any kind.

    So these Windows rules are pretty complicated, and neither match
    the Unix rules nor provide semantics natural for the native
    filesystem.  That makes them hard to explain to Unix *or* Windows
    users.  Nevertheless, they've worked fine for years, and in
    isolation there's no compelling reason to change them.

    However, that was before the MacOSX HFS+ and Cygwin ports arrived.
    They also have case-preserving case-insensitive filesystems, but
    the people doing the ports despised the Windows rules.  Indeed, a
    patch to make HFS+ act like Unix for imports got past a reviewer
    and into the code base, which incidentally made Cygwin also act
    like Unix (but this met the unbounded approval of the Cygwin
    folks, so they sure didn't complain -- they had patches of their
    own pending to do this, but the reviewer for those balked).

    At a higher level, we want to keep Python consistent, by following
    the same rules on *all* platforms with case-preserving
    case-insensitive filesystems.

Proposed Semantics

    The proposed new semantics for the lower left box:

    A. If the PYTHONCASEOK environment variable exists, same as
       before: silently accept the first case-insensitive match of any
       kind; raise ImportError if none found.

    B. Else search sys.path for the first case-sensitive match; raise
       ImportError if none found.

    #B is the same rule as is used on Unix, so this will improve cross-
    platform portability.  That's good.  #B is also the rule the Mac
    and Cygwin folks want (and wanted enough to implement themselves,
    multiple times, which is a powerful argument in PythonLand).  It
    can't cause any existing non-exceptional Windows import to fail,
    because any existing non-exceptional Windows import finds a
    case-sensitive match first in the path -- and it still will.  An
    exceptional Windows import currently blows up with a NameError or
    ImportError, in which latter case it still will, or in which
    former case will continue searching, and either succeed or blow up
    with an ImportError.

    #A is needed to cater to case-destroying filesystems mounted on Windows,
    and *may* also be used by people so enamored of "natural" Windows
    behavior that they're willing to set an environment variable to
    get it.  I don't intend to implement #A for Unix too, but that's
    just because I'm not clear on how I *could* do so efficiently (I'm
    not going to slow imports under Unix just for theoretical purity).

    The potential damage is here: #2 (matching on ALLCAPS.PY) is
    proposed to be dropped.  Case-destroying filesystems are a
    vanishing breed, and support for them is ugly.  We're already
    supporting (and will continue to support) PYTHONCASEOK for their
    benefit, but they don't deserve multiple hacks in 2001.


PEP 236 -- Back to the __future__

PEP: | 236
Title: | Back to the __future__
Version: | $Revision: 1.6 $
Author: | Tim Peters
Python-Version: | 2.1
Status: | Active
Type: | Standards Track
Created: | 26-Feb-2001
Post-History: | 26-Feb-2001

Motivation

    From time to time, Python makes an incompatible change to the
    advertised semantics of core language constructs, or changes their
    accidental (implementation-dependent) behavior in some way.  While this
    is never done capriciously, and is always done with the aim of
    improving the language over the long term, over the short term it's
    contentious and disrupting.

    The "Guidelines for Language Evolution" PEP [1] suggests ways to ease
    the pain, and this PEP introduces some machinery in support of that.

    The "Statically Nested Scopes" PEP [2] is the first application, and
    will be used as an example here.

Intent

    [Note:  This is policy, and so should eventually move into PEP 5[1]]

    When an incompatible change to core language syntax or semantics is
    being made:

    1. The release C that introduces the change does not change the
       syntax or semantics by default.

    2. A future release R is identified in which the new syntax or semantics
       will be enforced.

    3. The mechanisms described in the "Warning Framework" PEP [3] are used
       to generate warnings, whenever possible, about constructs or
       operations whose meaning may[4] change in release R.

    4. The new future_statement (see below) can be explicitly included in a
       module M to request that the code in module M use the new syntax or
       semantics in the current release C.

    So old code continues to work by default, for at least one release,
    although it may start to generate new warning messages.  Migration to
    the new syntax or semantics can proceed during that time, using the
    future_statement to make modules containing it act as if the new syntax
    or semantics were already being enforced.

    Note that there is no need to involve the future_statement machinery
    in new features unless they can break existing code; fully backward-
    compatible additions can-- and should --be introduced without a
    corresponding future_statement.

Syntax

    A future_statement is simply a from/import statement using the reserved
    module name __future__:

        future_statement: "from" "__future__" "import" feature ["as" name]
                          ("," feature ["as" name])*

        feature: identifier
        name: identifier

    In addition, all future_statments must appear near the top of the
    module.  The only lines that can appear before a future_statement are:

    + The module docstring (if any).
    + Comments.
    + Blank lines.
    + Other future_statements.

    Example:
        """This is a module docstring."""

        # This is a comment, preceded by a blank line and followed by
        # a future_statement.
        from __future__ import nested_scopes

        from math import sin
        from __future__ import alabaster_weenoblobs  # compile-time error!
        # That was an error because preceded by a non-future_statement.

Semantics

    A future_statement is recognized and treated specially at compile time:
    changes to the semantics of core constructs are often implemented by
    generating different code.  It may even be the case that a new feature
    introduces new incompatible syntax (such as a new reserved word), in
    which case the compiler may need to parse the module differently.  Such
    decisions cannot be pushed off until runtime.

    For any given release, the compiler knows which feature names have been
    defined, and raises a compile-time error if a future_statement contains
    a feature not known to it[5].

    The direct runtime semantics are the same as for any import statement:
    there is a standard module __future__.py, described later, and it will
    be imported in the usual way at the time the future_statement is
    executed.

    The *interesting* runtime semantics depend on the specific feature(s)
    "imported" by the future_statement(s) appearing in the module.

    Note that there is nothing special about the statement:

        import __future__ [as name]

    That is not a future_statement; it's an ordinary import statement, with
    no special semantics or syntax restrictions.

Example

    Consider this code, in file scope.py:

        x = 42
        def f():
            x = 666
            def g():
                print "x is", x
            g()
        f()

    Under 2.0, it prints:

        x is 42

    Nested scopes[2] are being introduced in 2.1.  But under 2.1, it still
    prints

        x is 42

    and also generates a warning.

    In 2.2, and also in 2.1 *if* "from __future__ import nested_scopes" is
    included at the top of scope.py, it prints

        x is 666

Standard Module __future__.py

    Lib/__future__.py is a real module, and serves three purposes:

    1. To avoid confusing existing tools that analyze import statements and
       expect to find the modules they're importing.

    2. To ensure that future_statements run under releases prior to 2.1
       at least yield runtime exceptions (the import of __future__ will
       fail, because there was no module of that name prior to 2.1).

    3. To document when incompatible changes were introduced, and when they
       will be-- or were --made mandatory.  This is a form of executable
       documentation, and can be inspected programatically via importing
       __future__ and examining its contents.

    Each statment in __future__.py is of the form:

        FeatureName = "_Feature(" OptionalRelease "," MandatoryRelease ")"

    where, normally, OptionalRelease <  MandatoryRelease, and both are
    5-tuples of the same form as sys.version_info:

    (PY_MAJOR_VERSION, # the 2 in 2.1.0a3; an int
     PY_MINOR_VERSION, # the 1; an int
     PY_MICRO_VERSION, # the 0; an int
     PY_RELEASE_LEVEL, # "alpha", "beta", "candidate" or "final"; string
     PY_RELEASE_SERIAL # the 3; an int
    )

    OptionalRelease records the first release in which

        from __future__ import FeatureName

    was accepted.

    In the case of MandatoryReleases that have not yet occurred,
    MandatoryRelease predicts the release in which the feature will become
    part of the language.

    Else MandatoryRelease records when the feature became part of the
    language; in releases at or after that, modules no longer need

        from __future__ import FeatureName

    to use the feature in question, but may continue to use such imports.

    MandatoryRelease may also be None, meaning that a planned feature got
    dropped.

    Instances of class _Feature have two corresponding methods,
    .getOptionalRelease() and .getMandatoryRelease().

    No feature line will ever be deleted from __future__.py.

    Example line:

      nested_scopes = _Feature((2, 1, 0, "beta", 1), (2, 2, 0, "final", 0))

    This means that

        from __future__ import nested_scopes

    will work in all releases at or after 2.1b1, and that nested_scopes are
    intended to be enforced starting in release 2.2.

Paritally Resolved Problem:  Runtime Compilation

    Several Python features can compile code during a module's runtime:

    1. The exec statement.
    2. The execfile() function.
    3. The compile() function.
    4. The eval() function.
    5. The input() function.

    Since a module M containing a future_statement naming feature F
    explicitly requests that the current release act like a future release
    with respect to F, any code compiled dynamically from text passed to
    one of these from within M should probably also use the new syntax or
    semantics associated with F.  The 2.1 release does behave this way.

    This isn't always desired, though.  For example, doctest.testmod(M)
    compiles examples taken from strings in M, and those examples should
    use M's choices, not necessarily the doctest module's choices.  In the
    2.1 release, this isn't possible, and no scheme has yet been suggested
    for working around this.

    In any case, a future_statement appearing "near the top" (see Syntax
    above) of text compiled dynamically by an exec, execfile() or compile()
    applies to the code block generated, but has no further effect on the
    module that executes such an exec, execfile() or compile().  This
    can't be used to affect eval() or input(), however, because they only
    allow expression input, and a future_statement is not an expression.

Resolved Problem:  Native Interactive Shells

    There are two ways to get an interactive shell:

    1. By invoking Python from a command line without a script argument.

    2. By invoking Python from a command line with the -i switch and with a
       script argument.

    An interactive shell can be seen as an extreme case of runtime
    compilation (see above):  in effect, each statement typed at an
    interactive shell prompt runs a new instance of exec, compile() or
    execfile().  A future_statement typed at an interactive shell applies to
    the rest of the shell session's life, as if the future_statement had
    appeared at the top of a module.

Unresolved Problem:  Simulated Interactive Shells

    Interactive shells "built by hand" (by tools such as IDLE and the Emacs
    Python-mode) should behave like native interactive shells (see above).
    However, the machinery used internally by native interactive shells has
    not been exposed, and there isn't a clear way for tools building their
    own interactive shells to achieve the desired behavior.

Questions and Answers

    Q:  What about a "from __past__" version, to get back *old* behavior?

    A:  Outside the scope of this PEP.  Seems unlikely to the author,
        though.  Write a PEP if you want to pursue it.

    Q:  What about incompatibilites due to changes in the Python virtual
        machine?

    A:  Outside the scope of this PEP, although PEP 5[1] suggests a grace
        period there too, and the future_statement may also have a role to
        play there.

    Q:  What about incompatibilites due to changes in Python's C API?

    A:  Outside the scope of this PEP.

    Q:  I want to wrap future_statements in try/except blocks, so I can
        use different code depending on which version of Python I'm running.
        Why can't I?

    A:  Sorry!  try/except is a runtime feature; future_statements are
        primarily compile-time gimmicks, and your try/except happens long
        after the compiler is done.  That is, by the time you do
        try/except, the semantics in effect for the module are already a
        done deal.  Since the try/except wouldn't accomplish what it
        *looks* like it should accomplish, it's simply not allowed.  We
        also want to keep these special statements very easy to find and to
        recognize.

        Note that you *can* import __future__ directly, and use the
        information in it, along with sys.version_info, to figure out where
        the release you're running under stands in relation to a given
        feature's status.

     Q: Going back to the nested_scopes example, what if release 2.2 comes
        along and I still haven't changed my code?  How can I keep the 2.1
        behavior then?

     A: By continuing to use 2.1, and not moving to 2.2 until you do change
        your code.  The purpose of future_statement is to make life easier
        for people who keep keep current with the latest release in a timely
        fashion.  We don't hate you if you don't, but your problems are
        much harder to solve, and somebody with those problems will need to
        write a PEP addressing them.  future_statement is aimed at a
        different audience.

     Q: Overloading "import" sucks.  Why not introduce a new statement for
        this?

     A: Like maybe "lambda lambda nested_scopes"?  That is, unless we
        introduce a new keyword, we can't introduce an entirely new
        statement.  But if we introduce a new keyword, that in itself
        would break old code.  That would be too ironic to bear.  Yes,
        overloading "import" does suck, but not as energeticallly as the
        alternatives -- as is, future_statements are 100% backward
        compatible.

Copyright

    This document has been placed in the public domain.

References and Footnotes

    [1] http://python.sourceforge.net/peps/pep-0005.html

    [2] http://python.sourceforge.net/peps/pep-0227.html

    [3] http://python.sourceforge.net/peps/pep-0230.html

    [4] Note that this is "may" and not "will":  better safe than sorry.  Of
        course spurious warnings won't be generated when avoidable with
        reasonable cost.

    [5] This ensures that a future_statement run under a release prior to
        the first one in which a given feature is known (but >= 2.1) will
        raise a compile-time error rather than silently do a wrong thing.
        If transported to a release prior to 2.1, a runtime error will be
        raised because of the failure to import __future__ (no such module
        existed in the standard distribution before the 2.1 release, and
        the double underscores make it a reserved name).


PEP 237 -- Unifying Long Integers and Integers

PEP: | 237
Title: | Unifying Long Integers and Integers
Version: | $Revision: 1.4 $
Author: | pep@zadka.site.co.il (Moshe Zadka)
Status: | Draft
Type: | Standards Track
Created: | 11-Mar-2001
Python-Version: | 2.2
Post-History: | 16-Mar-2001

Abstract

    Python has both integers (machine word size integral) types, and
    long integers (unbounded integral) types.  When integers
    operations overflow the machine registers, they raise an error.
    This PEP proposes to do away with the distinction, and unify the
    types from the perspective of both the Python interpreter and the
    C API.

Rationale

    Having the machine word size exposed to the language hinders
    portability.  For examples Python source files and .pyc's are not
    portable because of this.  Many programs find a need to deal with
    larger numbers after the fact, and changing the algorithms later
    is not only bothersome, but hinders performance in the normal
    case.

Literals

    A trailing 'L' at the end of an integer literal will stop having
    any meaning, and will be eventually phased out.  This will be done
    using warnings when encountering such literals.  The warning will
    be off by default in Python 2.2, on for 12 months, which will
    probably mean Python 2.3 and 2.4, and then will no longer be
    supported.

Builtin Functions

    The function long() will call the function int(), issuing a
    warning.  The warning will be off in 2.2, and on for two revisions
    before removing the function.  A FAQ will be added to explain that
    a solutions for old modules are:

         long=int

    at the top of the module, or:

         import __builtin__
         __builtin__.long=int

    In site.py.

C API

    All PyLong_As* will call PyInt_As*.  If PyInt_As* does not exist,
    it will be added.  Similarly for PyLong_From*.  A similar path of
    warnings as for the Python builtins will be followed.

Overflows

    When an arithmetic operation on two numbers whose internal
    representation is as machine-level integers returns something
    whose internal representation is a bignum, a warning which is
    turned off by default will be issued.  This is only a debugging
    aid, and has no guaranteed semantics.

Implementation

    The PyInt type's slot for a C long will be turned into a

        union {
            long i;
            struct {
                unsigned long length;
                digit digits[1];
            } bignum;
        };

    Only the n-1 lower bits of the long have any meaning; the top bit
    is always set.  This distinguishes the union.  All PyInt functions
    will check this bit before deciding which types of operations to
    use.

Jython Issues

    Jython will have a PyInt interface which is implemented by both
    from PyFixNum and PyBigNum.

Open Issues

    What to do about sys.maxint?

    What to do about PyInt_AS_LONG failures?

    What do do about %u, %o, %x formatting operators?

    How to warn about << not cutting integers?

    Should the overflow warning be on a portable maximum size?

    Will unification of types and classes help with a more straightforward
    implementations?

    Define an C API that can be used to find out what the representation of an
    int is.

Copyright

    This document has been placed in the public domain.


PEP 238 -- Non-integer Division

PEP: | 238
Title: | Non-integer Division
Version: | $Revision: 1.3 $
Author: | pep@zadka.site.co.il (Moshe Zadka)
Status: | Draft
Type: | Standards Track
Created: | 11-Mar-2001
Python-Version: | 2.2
Post-History: | 16-Mar-2001

Abstract

    Dividing integers currently returns the floor of the quantities.
    This behavior is known as integer division, and is similar to what
    C and FORTRAN do.  This has the useful property that all
    operations on integers return integers, but it does tend to put a
    hump in the learning curve when new programmers are surprised that

        1/2 == 0

    This proposal shows a way to change this while keeping backward
    compatibility issues in mind.

Rationale

    The behavior of integer division is a major stumbling block found
    in user testing of Python.  This manages to trip up new
    programmers regularly and even causes the experienced programmer
    to make the occasional mistake.  The workarounds, like explicitly
    coercing one of the operands to float or use a non-integer
    literal, are very non-intuitive and lower the readability of the
    program.

// Operator

    A `//' operator which will be introduced, which will call the
    nb_intdivide or __intdiv__ slots.  This operator will be
    implemented in all the Python numeric types, and will have the
    semantics of

        a // b == floor(a/b)

    Except that the type of a//b will be the type a and b will be
    coerced into.  Specifically, if a and b are of the same type, a//b
    will be of that type too.

Changing the Semantics of the / Operator

    The nb_divide slot on integers (and long integers, if these are a
    separate type, but see PEP 237[1]) will issue a warning when given
    integers a and b such that

        a % b != 0

    The warning will be off by default in the 2.2 release, and on by
    default for in the next Python release, and will stay in effect
    for 24 months.  The next Python release after 24 months, it will
    implement

        (a/b) * b = a (more or less)

    The type of a/b will be either a float or a rational, depending on
    other PEPs[2, 3].

__future__

    A special opcode, FUTURE_DIV will be added that does the
    equivalent of:

        if type(a) in (types.IntType, types.LongType):
           if type(b) in (types.IntType, types.LongType):
               if a % b != 0:
                    return float(a)/b
        return a/b

    (or rational(a)/b, depending on whether 0.5 is rational or float).

    If "from __future__ import non_integer_division" is present in the
    module, until the IntType nb_divide is changed, the "/" operator
    is compiled to FUTURE_DIV.

Open Issues

    Should the // operator be renamed to "div"?

    Should the // be made into a function called "div"?

References

    [1] PEP 237, Unifying Long Integers and Integers, Zadka,
        http://python.sourceforge.net/peps/pep-0237.html

    [2] PEP 239, Adding a Rational Type to Python, Zadka,
        http://python.sourceforge.net/peps/pep-0239.html

    [3] PEP 240, Adding a Rational Literal to Python, Zadka,
        http://python.sourceforge.net/peps/pep-0240.html

Copyright

    This document has been placed in the public domain.


PEP 239 -- Adding a Rational Type to Python

PEP: | 239
Title: | Adding a Rational Type to Python
Version: | $Revision: 1.3 $
Author: | pep@zadka.site.co.il (Moshe Zadka)
Status: | Draft
Type: | Standards Track
Created: | 11-Mar-2001
Python-Version: | 2.2
Post-History: | 16-Mar-2001

Abstract

    Python has no numeric type with the semantics of an unboundedly
    precise rational number.  This proposal explains the semantics of
    such a type, and suggests builtin functions and literals to
    support such a type.  This PEP suggests no literals for rational
    numbers; that is left for another PEP[1].

Rationale

    While sometimes slower and more memory intensive (in general,
    unboundedly so) rational arithmetic captures more closely the
    mathematical ideal of numbers, and tends to have behavior which is
    less surprising to newbies.  Though many Python implementations of
    rational numbers have been written, none of these exist in the
    core, or are documented in any way.  This has made them much less
    accessible to people who are less Python-savvy.

RationalType

    There will be a new numeric type added called RationalType.  Its
    unary operators will do the obvious thing.  Binary operators will
    coerce integers and long integers to rationals, and rationals to
    floats and complexes.

    The following attributes will be supported: .numerator and
    .denominator.  The language definition will promise that

        r.denominator * r == r.numerator

    that the GCD of the numerator and the denominator is 1 and that
    the denominator is positive.

    The method r.trim(max_denominator) will return the closest
    rational s to r such that abs(s.denominator) <= max_denominator.

The rational() Builtin

    This function will have the signature rational(n, d=1).  n and d
    must both be integers, long integers or rationals.  A guarantee is
    made that

        rational(n, d) * d == n

References

    [1] PEP 240, Adding a Rational Literal to Python, Zadka,
        http://python.sourceforge.net/peps/pep-0240.html

Copyright

    This document has been placed in the public domain.


PEP 240 -- Adding a Rational Literal to Python

PEP: | 240
Title: | Adding a Rational Literal to Python
Version: | $Revision: 1.4 $
Author: | pep@zadka.site.co.il (Moshe Zadka)
Status: | Draft
Type: | Standards Track
Created: | 11-Mar-2001
Python-Version: | 2.2
Post-History: | 16-Mar-2001

Abstract

    A different PEP[1] suggests adding a builtin rational type to
    Python.  This PEP suggests changing the ddd.ddd float literal to a
    rational in Python, and modifying non-integer division to return
    it.

Rationale

    Rational numbers are useful for exact and unsurprising arithmetic.
    They give the correct results people have been taught in various
    math classes.  Making the "obvious" non-integer type one with more
    predictable semantics will surprise new programmers less then
    using floating point numbers. As quite a few posts on c.l.py and
    on tutor@python.org have shown, people often get bit by strange
    semantics of floating point numbers: for example, round(0.98, 2)
    still gives 0.97999999999999998.

Proposal

    Literals conforming to the regular expression '\d*.\d*' will be
    rational numbers.

Backwards Compatibility

    The only backwards compatible issue is the type of literals
    mentioned above.  The following migration is suggested:

    1. "from __future__ import rational_literals" will cause all such
       literals to be treated as rational numbers.

    2. Python 2.2 will have a warning, turned off by default, about
       such literals in the absence of a __future__ statement.  The
       warning message will contain information about the __future__
       statement, and indicate that to get floating point literals,
       they should be suffixed with "e0".

    3. Python 2.3 will have the warning turned on by default.  This
       warning will stay in place for 24 months, at which time the
       literals will be rationals and the warning will be removed.

Common Objections

    Rationals are slow and memory intensive!
    (Relax, I'm not taking floats away, I'm just adding two more characters.
    1e0 will still be a float)

References

    [1] PEP 239, Adding a Rational Type to Python, Zadka,
        http://python.sourceforge.net/peps/pep-0239.html

Copyright

    This document has been placed in the public domain.


PEP 241 -- Metadata for Python Software Packages

PEP: | 241
Title: | Metadata for Python Software Packages
Version: | $Revision: 1.8 $
Author: | A.M. Kuchling
Type: | Standards Track
Created: | 12-Mar-2001
Status: | Draft
Post-History: | 19-Mar-2001

Introduction

   This PEP describes a mechanism for adding metadata to Python
   packages.  It includes specifics of the field names, and their
   semantics and usage.

Including Metadata in Packages

    The Distutils 'sdist' command will be modified to extract the
    metadata fields from the arguments and write them to a file in the
    generated zipfile or tarball.  This file will be named PKG-INFO
    and will be placed in the top directory of the source
    distribution (where the README, INSTALL, and other files usually
    go).

    Developers may not provide their own PKG-INFO file.  The "sdist"
    command will, if it detects an existing PKG-INFO file, terminate
    with an appropriate error message.  This should prevent confusion
    caused by the PKG-INFO and setup.py files being out of sync.

    The PKG-INFO file format is a single set of RFC-822 headers
    parseable by the rfc822.py module.  The field names listed in the
    following section are used as the header names.  There's no
    extension mechanism in this simple format; the Catalog and Distutils
    SIGs will aim at getting a more flexible format ready for Python 2.2.

Fields

    This section specifies the names and semantics of each of the
    supported metadata fields.

    Fields marked with "(Multiple use)" may be specified multiple
    times in a single PKG-INFO file.  Other fields may only occur
    once in a PKG-INFO file.  Fields marked with "(optional)" are
    not required to appear in a valid PKG-INFO file, all other
    fields must be present.

    Metadata-Version

      Version of the file format; currently "1.0" is the only
      legal value here.

      Example:

           Metadata-Version: 1.0

    Name

      The name of the package.

      Example:

          Name: BeagleVote

    Version

      A string containing the package's version number.  This
      field should be parseable by one of the Version classes
      (StrictVersion or LooseVersion) in the distutils.version
      module.

      Example:

          Version: 1.0a2

    Platform (multiple use)

      A comma-separated list of platform specifications, summarizing
      the operating systems supported by the package.  The major
      supported platforms are listed below, but this list is
      necessarily incomplete.

            POSIX, MacOS, Windows, BeOS, PalmOS.

      Binary distributions will use the Supported-Platform field in
      their metadata to specify the OS and CPU for which the binary
      package was compiled.  The semantics of the Supported-Platform
      are not specified in this PEP.

      Example:

          Platform: POSIX, Windows

    Summary

      A one-line summary of what the package does.

      Example:

          Summary: A module for collecting votes from beagles.

    Description (optional)

      A longer description of the package that can run to several
      paragraphs.  (Software that deals with metadata should not
      assume any maximum size for this field, though one hopes that
      people won't include their instruction manual as the
      long-description.)

      Example:

          Description: This module collects votes from beagles
                       in order to determine their electoral wishes.
                       Do NOT try to use this module with basset hounds;
                       it makes them grumpy.

    Keywords (optional)

      A list of additional keywords to be used to assist searching
      for the package in a larger catalog.

      Example:

          Keywords: dog puppy voting election

    Home-page (optional)

      A string containing the URL for the package's home page.

      Example:

          Home-page: http://www.example.com/~cschultz/bvote/

    Author (optional)

      A string containing at a minimum the author's name.  Contact
      information can also be added, separating each line with
      newlines.

      Example:

          Author: C. Schultz
                  Universal Features Syndicate
                  Los Angeles, CA

    Author-email

      A string containing the author's e-mail address.  It can contain
      a name and e-mail address in the legal forms for a RFC-822
      'From:' header.  It's not optional because cataloging systems
      can use the e-mail portion of this field as a unique key
      representing the author.  A catalog might provide authors the
      ability to store their GPG key, personal home page, and other
      additional metadata *about the author*, and optionally the
      ability to associate several e-mail addresses with the same
      person.  Author-related metadata fields are not covered by this
      PEP.

      Example:

          Author-email: "C. Schultz" <cschultz@example.com>

    License

      A string selected from a short list of choices, specifying the
      license covering the package.  Some licenses result in the
      software being freely redistributable, so packagers and
      resellers can automatically know that they're free to
      redistribute the software.  Other licenses will require
      a careful reading by a human to determine the software can be
      repackaged and resold.

      The choices are:

        Artistic, BSD, DFSG, GNU GPL, GNU LGPL, "MIT/X11",
        Mozilla PL, "public domain", Python, Qt PL, Zope PL, unknown,
        nocommercial, nosell, nosource, shareware, other

      Definitions of some of the licenses are:

       DFSG           The license conforms to the Debian Free Software
                      Guidelines, but does not use one of the other
                      DFSG conforming licenses listed here.
                      More information is available at:
                      http://www.debian.org/social_contract#guidelines

       Python         Python 1.6 or higher license.  Version 1.5.2 and
                      earlier are under the MIT/X11 license.

       public domain  Software is public domain, not copyrighted.
       unknown        Status is not known
       nocommercial   Free private use but commercial use not permitted
       nosell         Free use but distribution for profit by arrangement
       nosource       Freely distributable but no source code
       shareware      Payment is requested if software is used
       other          General category for other non-DFSG licenses

      Some of these licenses can be interpreted to mean the software is
      freely redistributable.  The list of redistributable licenses is:

      Artistic, BSD, DFSG, GNU GPL, GNU LGPL, "MIT/X11",
      Mozilla PL, "public domain", Python, Qt PL, Zope PL,
      nosource, shareware

      Note that being redistributable does not mean a package
      qualifies as free software, 'nosource' and 'shareware' being
      examples.

      Example:

          License: MIT/X11

Acknowledgements

    Many changes and rewrites to this document were suggested by the
    readers of the Distutils SIG.  In particular, Sean Reifschneider
    often contributed actual text for inclusion in this PEP.

    The list of licenses was compiled using the SourceForge license
    list and the CTAN license list compiled by Graham Williams; Carey
    Evans also offered several useful suggestions on this list.

Copyright

    This document has been placed in the public domain.


PEP 242 -- Numeric Kinds

PEP: | 242
Title: | Numeric Kinds
Version: | $Revision: 1.3 $
Author: | paul@pfdubois.com (Paul F. Dubois)
Status: | Draft
Type: | Standards Track
Created: | 17-Mar-2001
Python-Version: | 2.2
Post-History: | 17-Apr-2001

Abstract

    This proposal gives the user optional control over the precision
    and range of numeric computations so that a computation can be
    written once and run anywhere with at least the desired precision
    and range.  It is backward compatible with existing code.  The
    meaning of decimal literals is clarified.

Rationale

    Currently it is impossible in every language except Fortran 90 to
    write a program in a portable way that uses floating point and
    gets roughly the same answer regardless of platform -- or refuses
    to compile if that is not possible.  Python currently has only one
    floating point type, equal to a C double in the C implementation.

    No type exists corresponding to single or quad floats.  It would
    complicate the language to try to introduce such types directly
    and their subsequent use would not be portable.  This proposal is
    similar to the Fortran 90 "kind" solution, adapted to the Python
    environment.  With this facility an entire calculation can be
    switched from one level of precision to another by changing a
    single line.  If the desired precision does not exist on a
    particular machine, the program will fail rather than get the
    wrong answer.  Since coding in this style would involve an early
    call to the routine that will fail, this is the next best thing to
    not compiling.

Supported Kinds of Ints and Floats

    Complex numbers are treated separately below, since Python can be
    built without them.

    Each Python compiler may define as many "kinds" of integer and
    floating point numbers as it likes, except that it must support at
    least two kinds of integer corresponding to the existing int and
    long, and must support at least one kind of floating point number,
    equivalent to the present float.

    The range and precision of the these required kinds are processor
    dependent, as at present, except for the "long integer" kind,
    which can hold an arbitrary integer.

    The built-in functions int(), long(), and float() convert inputs
    to these default kinds as they do at present.  (Note that a
    Unicode string is actually a different "kind" of string and that a
    sufficiently knowledgeable person might be able to expand this PEP
    to cover that case.)

    Within each type (integer, floating) the compiler supports a
    linearly-ordered set of kinds, with the ordering determined by the
    ability to hold numbers of an increased range and/or precision.

Kind Objects

    Two new standard functions are defined in a module named "kinds".
    They return callable objects called kind objects.  Each int or
    floating kind object f has the signature result = f(x), and each
    complex kind object has the signature result = f(x, y=0.).

    int_kind(n)
        For an integer argument n >= 1, return a callable object whose
        result is an integer kind that will hold an integer number in
        the open interval (-10**n,10**n).  The kind object accepts
        arguments that are integers including longs.  If n == 0,
        returns the kind object corresponding to the Python literal 0.

    float_kind(nd, n)
        For nd >= 0 and n >= 1, return a callable object whose result
        is a floating point kind that will hold a floating-point
        number with at least nd digits of precision and a base-10
        exponent in the closed interval [-n, n].  The kind object
        accepts arguments that are integer or float.

        If nd and n are both zero, returns the kind object
        corresponding to the Python literal 0.0.

    The compiler will return a kind object corresponding to the least
    of its available set of kinds for that type that has the desired
    properties.  If no kind with the desired qualities exists in a
    given implementation an OverflowError exception is thrown.  A kind
    function converts its argument to the target kind, but if the
    result does not fit in the target kind's range, an OverflowError
    exception is thrown.

    Besides their callable behavior, kind objects have attributes
    giving the traits of the kind in question.

    1. name is the name of the kind.  The standard kinds are called
       int, long, double.

    2. typecode is a single-letter string that would be appropriate
       for use with Numeric or module array to form an array of this
       kind.  The standard types' typecodes are 'i', 'O', 'd'
       respectively.

    3. Integer kinds have these additional attributes: MAX, equal to
       the maximum permissible integer of this kind, or None for the
       long kind.  MIN, equal to the most negative permissible integer
       of this kind, or None for the long kind.

    4. Float kinds have these additional attributes whose properties
       are equal to the corresponding value for the corresponding C
       type in the standard header file "float.h".  MAX, MIN, DIG,
       MANT_DIG, EPSILON, MAX_EXP, MAX_10_EXP, MIN_EXP, MIN_10_EXP,
       RADIX, ROUNDS (== FLT_RADIX, FLT_ROUNDS in float.h).  These
       values are of type integer except for MAX, MIN, and EPSILON,
       which are of the Python floating type to which the kind
       corresponds.

Attributes of Module kinds

    int_kinds is a list of the available integer kinds, sorted from lowest
              to highest kind.  By definition, int_kinds[-1] is the
              long kind.

    float_kinds is a list of the available floating point kinds, sorted
                from lowest to highest kind.

    default_int_kind is the kind object corresponding to the Python
                     literal 0

    default_long_kind is the kind object corresponding to the Python
                      literal 0L

    default_float_kind is the kind object corresponding to the Python
                       literal 0.0

Complex Numbers

    If supported, complex numbers have real and imaginary parts that
    are floating-point numbers with the same kind.  A Python compiler
    must support a complex analog of each floating point kind it
    supports, if it supports complex numbers at all.

    If complex numbers are supported, the following are available in
    module kinds:

    complex_kind(nd, n)
        Return a callable object whose result is a complex kind that
        will hold a complex number each of whose components (.real,
        .imag) is of kind float_kind(nd, n).  The kind object will
        accept one argument that is of any integer, real, or complex
        kind, or two arguments, each integer or real.

    complex_kinds is a list of the available complex kinds, sorted
                  from lowest to highest kind.

    default_complex_kind is the kind object corresponding to the
                         Python literal 0.0j.  The name of this kind
                         is doublecomplex, and its typecode is 'D'.

    Complex kind objects have these addition attributes:

    floatkind is the kind object of the corresponding float type.

Examples

    In module myprecision.py:

        import kinds
        tinyint = kinds.int_kind(1)
        single = kinds.float_kind(6, 90)
        double = kinds.float_kind(15, 300)
        csingle = kinds.complex_kind(6, 90)

    In the rest of my code:

        from myprecision import tinyint, single, double, csingle
        n = tinyint(3)
        x = double(1.e20)
        z = 1.2
        # builtin float gets you the default float kind, properties unknown
        w = x * float(x)
        # but in the following case we know w has kind "double".
        w = x * double(z)

        u = csingle(x + z * 1.0j)
        u2 = csingle(x+z, 1.0)

    Note how that entire code can then be changed to a higher
    precision by changing the arguments in myprecision.py.

    Comment: note that you aren't promised that single != double; but
    you are promised that double(1.e20) will hold a number with 15
    decimal digits of precision and a range up to 10**300 or that the
    float_kind call will fail.

Open Issues

    No open issues have been raised at this time.

Copyright

    This document has been placed in the public domain.


PEP 243 -- Module Repository Upload Mechanism

PEP: | 243
Title: | Module Repository Upload Mechanism
Version: | $Revision: 1.3 $
Author: | jafo-pep@tummy.com (Sean Reifschneider)
Discussions-To: | distutils-sig@python.org
Status: | Draft
Type: | Standards Track
Created: | 18-Mar-2001
Python-Version: | 2.1
Post-History: | 20-Mar-2001, 24-Mar-2001

Abstract

    For a module repository system (such as Perl's CPAN) to be
    successful, it must be as easy as possible for module authors to
    submit their work.  An obvious place for this submit to happen is
    in the Distutils tools after the distribution archive has been
    successfully created.  For example, after a module author has
    tested their software (verifying the results of "setup.py sdist"),
    they might type "setup.py sdist --submit".  This would flag
    Distutils to submit the source distribution to the archive server
    for inclusion and distribution to the mirrors.

    This PEP only deals with the mechanism for submitting the software
    distributions to the archive, and does not deal with the actual
    archive/catalog server.

Upload Process

    The upload will include the Distutils "PKG-INFO" meta-data
    information (as specified in PEP-241 [1]), the actual software
    distribution, and other optional information.  This information
    will be uploaded as a multi-part form encoded the same as a
    regular HTML file upload request.  This form is posted using
    ENCTYPE="multipart/form-data" encoding [2].

    The upload will be made to the host "modules.python.org" on port
    80/tcp (POST http://modules.python.org:80/swalowpost.cgi).  The form
    will consist of the following fields:

        distribution -- The file containing the module software (for
        example, a .tar.gz or .zip file).

        distmd5sum -- The MD5 hash of the uploaded distribution,
        encoded in ASCII representing the hexadecimal representation
        of the digest ("for byte in digest: s = s + ('%02x' %
        ord(byte))").

        pkginfo (optional) -- The file containing the distribution
        meta-data (as specified in PEP-241 [1]).  Note that if this is
        not included, the distribution file is expected to be in .tar
        format (gzipped and bzipped compreesed are allowed) or .zip
        format, with a "PKG-INFO" file in the top-level directory it
        extracts ("package-1.00/PKG-INFO").

        infomd5sum (required if pkginfo field is present) -- The MD5 hash
        of the uploaded meta-data, encoded in ASCII representing the
        hexadecimal representation of the digest ("for byte in digest:
        s = s + ('%02x' % ord(byte))").

        platform (optional) -- A string representing the target
        platform for this distribution.  This is only for binary
        distributions.  It is encoded as
        "<os_name>-<os_version>-<platform architecture>-<python
        version>".

        signature (optional) -- A OpenPGP-compatible signature [3] of
        the uploaded distribution as signed by the author.  This may
        be used by the cataloging system to automate acceptance of
        uploads.

        protocol_version -- A string indicating the protocol version that
        the client supports.  This document describes protocol version "1".

Return Data

    The status of the upload will be reported using HTTP non-standard
    ("X-*)" headers.  The "X-Swalow-Status" header may have the following
    values:

        SUCCESS -- Indicates that the upload has succeeded.

        FAILURE -- The upload is, for some reason, unable to be
        processed.

        TRYAGAIN -- The server is unable to accept the upload at this
        time, but the client should try again at a later time.
        Potential causes of this are resource shortages on the server,
        administrative down-time, etc...

    Optionally, there may be a "X-Swalow-Reason" header which includes a
    human-readable string which provides more detailed information about
    the "X-Swalow-Status".

    If there is no "X-Swalow-Status" header, or it does not contain one of
    the three strings above, it should be treated as a temporary failure.

    Example:

        >>> f = urllib.urlopen('http://modules.python.org:80/swalowpost.cgi')
        >>> s = f.headers['x-swalow-status']
        >>> s = s + ': ' + f.headers.get('x-swalow-reason', '<None>')
        >>> print s
        FAILURE: Required field "distribution" missing.

Sample Form

    The upload client must submit the page in the same form as
    Netscape Navigator version 4.76 for Linux produces when presented
    with the following form:

        <H1>Upload file</H1>
        <FORM NAME="fileupload" METHOD="POST" ACTION="swalowpost.cgi"
              ENCTYPE="multipart/form-data">
        <INPUT TYPE="file" NAME="distribution"><BR>
        <INPUT TYPE="text" NAME="distmd5sum"><BR>
        <INPUT TYPE="file" NAME="pkginfo"><BR>
        <INPUT TYPE="text" NAME="infomd5sum"><BR>
        <INPUT TYPE="text" NAME="platform"><BR>
        <INPUT TYPE="text" NAME="signature"><BR>
        <INPUT TYPE="hidden" NAME="protocol_version" VALUE="1"><BR>
        <INPUT TYPE="SUBMIT" VALUE="Upload">
        </FORM>

Platforms

    The following are valid os names:

        aix beos debian dos freebsd hpux mac macos mandrake netbsd
        openbsd qnx redhat solaris suse windows yellowdog

    The above include a number of different types of distributions of
    Linux.  Because of versioning issues these must be split out, and
    it is expected that when it makes sense for one system to use
    distributions made on other similar systems, the download client
    will make the distinction.

    Version is the official version string specified by the vendor for
    the particular release.  For example, "2000" and "nt" (Windows),
    "9.04" (HP-UX), "7.0" (RedHat, Mandrake).

    The following are valid architectures:

        alpha hppa ix86 powerpc sparc ultrasparc

Status

    I currently have a proof-of-concept client and server implemented.
    I plan to have the Distutils patches ready for the 2.1 release.
    Combined with Andrew's PEP-241 [1] for specifying distribution
    meta-data, I hope to have a platform which will allow us to gather
    real-world data for finalizing the catalog system for the 2.2
    release.

References

    [1] Metadata for Python Software Package, Kuchling,
        http://python.sourceforge.net/peps/pep-0241.html

    [2] RFC 1867, Form-based File Upload in HTML
        http://www.faqs.org/rfcs/rfc1867.html

    [3] RFC 2440, OpenPGP Message Format
        http://www.faqs.org/rfcs/rfc2440.html

Copyright

    This document has been placed in the public domain.


PEP 244 -- The `directive' statement

PEP: | 244
Title: | The `directive' statement
Version: | $Revision: 1.1 $
Author: | loewis@informatik.hu-berlin.de (Martin von Loewis)
Status: | Active
Type: | Standards Track
Created: | 20-Mar-2001
Python-Version: | 2.1
Post-History: |

Motivation

    From time to time, Python makes an incompatible change to the
    advertised semantics of core language constructs, or changes their
    accidental (implementation-dependent) behavior in some way.  While
    this is never done capriciously, and is always done with the aim
    of improving the language over the long term, over the short term
    it's contentious and disrupting.

    The "Guidelines for Language Evolution" PEP [1] suggests ways to
    ease the pain, and this PEP introduces some machinery in support
    of that.

    The "Statically Nested Scopes" PEP [2] is the first application,
    and will be used as an example here.

    When a new, potentially incompatible language feature is added,
    some modules and libraries may chose to use it, while others may
    not.  This specification introduces a syntax where a module author
    can denote whether a certain language feature is used in the
    module or not.

    In discussion of this PEP, readers commented that there are two
    kinds of "settable" language features:

    - those that are designed to eventually become the only option, at
      which time specifying use of them is not necessary anymore.  The
      features for which the syntax of the "Back to the __future__"
      PEP [3] was proposed fall into this category.  This PEP supports
      declaring such features, and supports phasing out the "old"
      meaning of constructs whose semantics has changed under the new
      feature.  However, it defines no policy as to what features must
      be phased out eventually.

    - those which are designed to stay optional forever, e.g. if they
      change some default setting in the interpreter.  An example for
      such settings might be the request to always emit line-number
      instructions for a certain module; no specific flags of that
      kind are proposed in this specification.

    Since a primary goal of this PEP is to support new language
    constructs without immediately breaking old libraries, special
    care was taken not to break old libraries by introducing the new
    syntax.

Syntax

    A directive_statement is a statement of the form

        directive_statement: 'directive' NAME [atom] [';'] NEWLINE

    The name in the directive indicates the kind of the directive; it
    defines whether the optional atom can be present, and whether
    there are further syntactical or semantical restrictions to the
    atom.  In addition, depending on the name of the directive,
    certain additional syntactical or semantical restrictions may be
    placed on the directive (e.g. placement of the directive in the
    module may be restricted to the top of the module).

    In the directive_statement, 'directive' is a new
    keyword. According to [1], this keyword is initially considered as
    a keyword only when used in a directive statement, see "Backwards
    Compatibility" below.

Semantics

    A directive statement instructs the Python interpreter to process
    a source file in a different way; the specific details of that
    processing depend on the directive name.  The optional atom is
    typically interpreted when the source code is processed; details
    of that interpretation depend on the directive.

Specific Directives: transitional

    If a syntactical or semantical change is added to Python which is
    incompatible, [1] mandates a transitional evolution of the
    language, where the new feature is initially available alongside
    with the old one.  Such a transition is possible by means of the
    transitional directive.

    In a transitional directive, the NAME is 'transitional'. The atom
    MUST be present, and it MUST be a NAME.  The possible values for
    that name are defined when the language change is defined.  One
    example for such a directive is

        directive transitional nested_scopes

    The transitional directive MUST occur at before any other
    statement in a module, except for the documentation string
    (i.e. it may appear as the second statement of a module only if
    the first statement is a STRING+).

Backwards Compatibility

    Introducing 'directive' as a new keyword might cause
    incompatibilities with existing code.  Following the guideline in
    [1], in the initial implementation of this specification,
    directive is a new keyword only if it was used in a valid
    directive_statement (i.e. if it appeared as the first non-string
    token in a module).

Unresolved Problems:  directive as the first identifier

    Using directive in a module as

    directive = 1

    (i.e. the name directive appears as the first thing in a module)
    will treat it as keyword, not as identifier. It would be possible
    to classify it as a NAME with an additional look-ahead token, but
    such look-ahead is not available in the Python tokenizer.

Questions and Answers

    Q: It looks like this PEP was written to allow definition of source
       code character sets.  Is that true?

    A: No.  Even though the directive facility can be extended to
       allow source code encodings, no specific directive is proposed.

    Q: Then why was this PEP written at all?

    A: It acts as a counter-proposal to [3], which proposes to
       overload the import statement with a new meaning.  This PEP
       allows to solve the problem in a more general way.

    Q: But isn't mixing source encodings and language changes like
       mixing apples and oranges?

    A: Perhaps.  To address the difference, the predefined
       "transitional" directive has been defined.

References and Footnotes

    [1] http://python.sourceforge.net/peps/pep-0005.html

    [2] http://python.sourceforge.net/peps/pep-0227.html

    [3] http://python.sourceforge.net/peps/pep-0236.html

Copyright

    This document has been placed in the public domain.


PEP 245 -- Python Interface Syntax

PEP: | 245
Title: | Python Interface Syntax
Version: | $Revision: 1.4 $
Author: | michel@digicool.com (Michel Pelletier)
Discussions-To: | http://www.zope.org/Wikis/Interfaces
Status: | Draft
Type: | Standards Track
Created: | 11-Jan-2001
Python-Version: | 2.2
Post-History: | 21-Mar-2001

Introduction

    This PEP describes a proposed syntax for creating interface
    objects in Python.

Overview

    In addition to thinking about adding a static type system to
    Python, the Types-SIG was also charged to devise an interface
    system for Python.  In December of 1998, Jim Fulton released a
    prototype interfaces system based on discussions from the SIG.
    Many of the issues and background information on this discussion
    and prototype can be found in the SIG archives[1].

    Around the end of 2000, Digital Creations began thinking about
    better component model designs for Zope[2].  Zope's future
    component model relies heavily on interface objects.  This led to
    further development of Jim's "Scarecrow" interfaces prototype.
    Starting with version 2.3, Zope comes with an Interface package as
    standard software.  Zope's Interface package is used as the
    reference implementation for this PEP.

    The syntax proposed by this PEP relies on syntax enhancements
    describe in PEP 232 [3] and describes an underlying framework
    which PEP 233 [4] could be based upon.  There is some work being
    done with regard to interface objects and Proxy objects, so for
    those optional parts of this PEP you may want to see[5].

The Problem

    Interfaces are important because they solve a number of problems
    that arise while developing software:

    - There are many implied interfaces in Python, commonly referred
      to as "protocols".  Currently determining those protocols is
      based on implementation introspection, but often that also
      fails.  For example, defining __getitem__ implies both a
      sequence and a mapping (the former with sequential, integer
      keys).  There is no way for the developer to be explict about
      which protocols the object intends to implement.

    - Python is limited, from the developer's point of view, by the
      split between types and classes.  When types are expected, the
      consumer uses code like 'type(foo) == type("")' to determine if
      'foo' is a string.  When instances of classes are expected, the
      consumer uses 'isinstance(foo, MyString)' to determine if 'foo'
      is an instance of the 'MyString' class.  There is no unified
      model for determining if an object can be used in a certain,
      valid way.

    - Python's dynamic typing is very flexible and powerful, but it
      does not have the advantage of static typed languages that
      provide type checking.  Static typed langauges provide you with
      much more type saftey, but are often overly verbose because
      objects can only be generalized by common subclassing and used
      specificly with casting (for example, in Java).

    There are also a number of documentation problems that interfaces
    try to solve.

    - Developers waste a lot of time looking at the source code of
      your system to figure out how objects work.

    - Developers who are new to your system may misunderstand how your
      objects work, causing, and possibly propagating, usage errors.

    - Because a lack of interfaces means usage is inferred from the
      source, developers may end up using methods and attributes that
      are meant for "internal use only".

    - Code inspection can be hard, and very discouraging to novice
      programmers trying to properly understand code written by gurus.

    - A lot of time is wasted when many people try very hard to
      understand obscurity (like undocumented software).  Effort spend
      up front documenting interfaces will save much of this time in
      the end.

    Interfaces try to solve these problems by providing a way for you
    to specify a contractual obligation for your object, documentation
    on how to use an object, and a built-in mechanism for discovering
    the contract and the documentation.

    Python has very useful introspection features.  It is well known
    that this makes exploring concepts in the interactive interpreter
    easier, because Python gives you the ability to look at all kinds
    of information about the objects: the type, doc strings, instance
    dictionaries, base classes, unbound methods and more.

    Many of these features are oriented toward introspecting, using
    and changing the implementation of software, and one of them ("doc
    strings") is oriented toward providing documentation.  This
    proposal describes an extension to this natural introspection
    framework that describes an object's interface.

Overview of the Interface Syntax

    For the most part, the syntax of interfaces is very much like the
    syntax of classes, but future needs, or needs brought up in
    discussion, may define new possibilities for interface syntax.

    A formal BNF description of the syntax is givena later in the PEP,
    for the purposes of illustration, here is an example of two
    different interfaces created with the proposed syntax:

        interface CountFishInterface:
            "Fish counting interface"

            def oneFish():
                "Increments the fish count by one"

            def twoFish():
                "Increments the fish count by two"

            def getFishCount():
                "Returns the fish count"

        interface ColorFishInterface:
            "Fish coloring interface"

            def redFish():
                "Sets the current fish color to red"

            def blueFish():
                "Sets the current fish color to blue"

            def getFishColor():
                "This returns the current fish color"

    This code, when evaluated, will create two interfaces called
    `CountFishInterface' and `ColorFishInterface'. These interfaces
    are defined by the `interface' statement.

    The prose documentation for the interfaces and their methods come
    from doc strings.  The method signature information comes from the
    signatures of the `def' statements.  Notice how there is no body
    for the def statements.  The interface does not implement a
    service to anything; it merely describes one.  Documentation
    strings on interfaces and interface methods are mandatory, a
    'pass' statement cannot be provided.  The interface equivalent of
    a pass statement is an empty doc string.

    You can also create interfaces that "extend" other interfaces.
    Here, you can see a new type of Interface that extends the
    CountFishInterface and ColorFishInterface:

        interface FishMarketInterface(CountFishInterface, ColorFishInterface):
            "This is the documentation for the FishMarketInterface"

            def getFishMonger():
                "Returns the fish monger you can interact with"

            def hireNewFishMonger(name):
                "Hire a new fish monger"

            def buySomeFish(quantity=1):
                "Buy some fish at the market"

    The FishMarketInteface extends upon the CountFishInterface and
    ColorfishInterface.

Interface Assertion

    The next step is to put classes and interfaces together by
    creating a concrete Python class that asserts that it implements
    an interface.  Here is an example FishMarket component that might
    do this:

        class FishError(Error):
            pass

        class FishMarket implements FishMarketInterface:
            number = 0
            color = None
            monger_name = 'Crusty Barnacles'

            def __init__(self, number, color):
                self.number = number
                self.color = color

            def oneFish(self):
                self.number += 1

            def twoFish(self):
                self.number += 2

            def redFish(self):
                self.color = 'red'

            def blueFish(self):
                self.color = 'blue'

            def getFishCount(self):
                return self.number

            def getFishColor(self):
                return self.color

            def getFishMonger(self):
                return self.monger_name

            def hireNewFishMonger(self, name):
                self.monger_name = name

            def buySomeFish(self, quantity=1):
                if quantity > self.count:
                    raise FishError("There's not enough fish")
                self.count -= quantity
                return quantity

    This new class, FishMarket defines a concrete class which
    implements the FishMarketInterface.  The object following the
    `implements' statement is called an "interface assertion".  An
    interface assertion can be either an interface object, or tuple of
    interface assertions.

    The interface assertion provided in a `class' statement like this
    is stored in the class's `__implements__' class attribute.  After
    interpreting the above example, you would have a class statement
    that can be examined like this with an 'implements' built-in
    function:

        >>> FishMarket
        <class FishMarket at 8140f50>
        >>> FishMarket.__implements__
        (<Interface FishMarketInterface at 81006f0>,)
        >>> f = FishMarket(6, 'red')
        >>> implements(f, FishMarketInterface)
        1
        >>>

    A class can realize more than one interface.  For example, say you
    had an interface called `ItemInterface' that described how an
    object worked as an item in a container object.  If you wanted to
    assert that FishMarket instances realized the ItemInterface
    interface as well as the FishMarketInterface, you can provide an
    interface assertion that contained a tuple of interface objects to
    the FishMarket class:

        class FishMarket implements FishMarketInterface, ItemInterface:
            # ...

    Interface assertions can also be used if you want to assert that
    one class implements an interface, and all of the interfaces that
    another class implements:

        class MyFishMarket implements FishMarketInterface, ItemInterface:
            # ...

        class YourFishMarket implements FooInterface, MyFishMarket.__implements__:
            # ...

    This new class YourFishMarket, asserts that it implements the
    FooInterface, as well as the interfaces implemented by the
    MyFishMarket class.

    It's worth going into a little bit more detail about interface
    assertions.  An interface assertion is either an interface object,
    or a tuple of interface assertions.  For example:

        FooInterface

        FooInterface, (BarInteface, BobInterface)

        FooInterface, (BarInterface, (BobInterface, MyClass.__implements__))

    Are all valid interface assertions.  When two interfaces define
    the same attributes, the order in which information is preferred
    in the assertion is from top-to-bottom, left-to-right.

    There are other interface proposals that, in the need for
    simplicity, have combined the notion of class and interface to
    provide simple interface enforcement.  Interface objects have a
    `deferred' method that returns a deferred class that implements
    this behavior:

        >>> FM = FishMarketInterface.deferred()
        >>> class MyFM(FM): pass

        >>> f = MyFM()
        >>> f.getFishMonger()
        Traceback (innermost last):
          File "<stdin>", line 1, in ?
        Interface.Exceptions.BrokenImplementation:
        An object has failed to implement interface FishMarketInterface

                The getFishMonger attribute was not provided.
        >>>

    This provides for a bit of passive interface enforcement by
    telling you what you forgot to do to implement that interface.

Formal Interface Syntax

    Python syntax is defined in a modified BNF grammer notation
    described in the Python Reference Manual [8].  This section
    describes the proposed interface syntax using this grammar:

        interfacedef:   "interface" interfacename [extends] ":" suite
        extends:        "(" [expression_list] ")"
        interfacename:  identifier

    An interface definition is an executable statement.  It first
    evaluates the extends list, if present.  Each item in the extends
    list should evaluate to an interface object.

    The interface's suite is then executed in a new execution frame
    (see the Python Reference Manual, section 4.1), using a newly
    created local namespace and the original global namespace.  When
    the interface's suite finishes execution, its execution frame is
    discarded but its local namespace is saved as interface elements.
    An interface object is then created using the extends list for the
    base interfaces and the saved interface elements.  The interface
    name is bound to this interface object in the original local
    namespace.

    This PEP also proposes an extension to Python's 'class' statement:

        classdef:    "class" classname [inheritance] [implements] ":" suite
        implements:  "implements" implist
        implist:     expression-list

        classname,
        inheritance,
        suite,
        expression-list:  see the Python Reference Manual

    Before a class' suite is executed, the 'inheritance' and
    'implements' statements are evaluated, if present.  The
    'inheritance' behavior is unchanged as defined in Section 7.6 of
    the Language Reference.

    The 'implements', if present, is evaluated after inheritance.
    This must evaluate to an interface specification, which is either
    an interface, or a tuple of interface specifications.  If a valid
    interface specification is present, the assertion is assigned to
    the class object's '__implements__' attribute, as a tuple.

    This PEP does not propose any changes to the syntax of function
    definitions or assignments.

Classes and Interfaces

    The example interfaces above do not describe any kind of behavior
    for their methods, they just describe an interface that a typical
    FishMarket object would realize.

    You may notice a similarity between interfaces extending from
    other interfaces and classes sub-classing from other classes.
    This is a similar concept.  However it is important to note that
    interfaces extend interfaces and classes subclass classes.  You
    cannot extend a class or subclass an interface.  Classes and
    interfaces are separate.

    The purpose of a class is to share the implementation of how an
    object works.  The purpose of an interface is to document how to
    work with an object, not how the object is implemented.  It is
    possible to have several different classes with very different
    implementations realize the same interface.

    It's also possible to implement one interface with many classes
    that mix in pieces the functionality of the interface or,
    conversely, it's possible to have one class implement many
    interfaces.  Because of this, interfaces and classes should not be
    confused or intermingled.

Interface-aware built-ins

    A useful extension to Python's list of built-in functions in the
    light of interface objects would be `implements()'.  This builtin
    would expect two arguments, an object and an interface, and return
    a true value if the object implements the interface, false
    otherwise.  For example:

        >>> interface FooInterface: pass
        >>> class Foo implements FooInterface: pass
        >>> f = Foo()
        >>> implements(f, FooInterface)
        1

    Currently, this functionality exists in the reference
    implementation as functions in the `Interface' package, requiring
    an "import Interface" to use it.  Its existence as a built-in
    would be purely for a convenience, and not necessary for using
    interfaces, and analogous to `isinstance()' for classes.

Backward Compatibility

    The proposed interface model does not introduce any backward
    compatibility issues in Python.  The proposed syntax, however,
    does.

    Any existing code that uses `interface' as an identifier will
    break.  There may be other kinds of backwards incompatibility that
    defining `interface' as a new keyword will introduce.  This
    extension to Python's syntax does not change any existing syntax
    in any backward incompatible way.

    The new `from __future__' Python syntax[6], and the new warning
    framework [7] is ideal for resolving this backward
    incompatibility.  To use interface syntax now, a developer could
    use the statement:

        from __future__ import interfaces

    In addition, any code that uses the keyword `interface' as an
    identifier will be issued a warning from Python.  After the
    appropriate period of time, the interface syntax would become
    standard, the above import statement would do nothing, and any
    identifiers named `interface' would raise an exception.  This
    period of time is proposed to be 24 months.

Summary of Proposed Changes to Python

    Adding new `interface' keyword and extending class syntax with
    `implements'.

    Extending class interface to include __implements__.

    Add 'implements(obj, interface)' built-in.

Risks

    This PEP proposes adding one new keyword to the Python language,
    `interface'.  This will break code.

Open Issues

    Goals

    Syntax

    Architecture

Dissenting Opinion

    This PEP has not yet been discussed on python-dev.

References

    [1] http://mail.python.org/pipermail/types-sig/1998-December/date.html

    [2] http://www.zope.org

    [3] PEP 232, Function Attributes, Warsaw
        http://python.sourceforge.net/peps/pep-0232.html

    [4] PEP 233, Python Online Help, Prescod
        http://python.sourceforge.net/peps/pep-0233.html

    [5] http://www.lemburg.com/files/python/mxProxy.html

    [6] PEP 236, Back to the __future__, Peters
        http://python.sourceforge.net/peps/pep-0236.html

    [7] PEP 230, Warning Framework, van Rossum
        http://python.sourceforge.net/peps/pep-0236.html

Copyright

    This document has been placed in the public domain.


PEP 246 -- Object Adaptation

PEP: | 246
Title: | Object Adaptation
Version: | $Revision: 1.2 $
Author: | cce@clarkevans.com (Clark C. Evans)
Status: | Draft
Type: | Standards Track
Created: | 21-Mar-2001
Python-Version: | 2.2
Post-History: | 29-Mar-2001

Abstract

    This proposal puts forth an extensible mechanism for the
    adaptation of an object to a context where a specific type, class,
    interface, or other protocol is expected.

    This proposal provides a built-in "adapt" function that, for any
    object X and protocol Y, can be used to ask the Python environment
    for a version of X complaint with Y.  Behind the scenes, the
    mechanism asks the object X: "Are you now, or do you know how to
    wrap yourself to provide, a supporter of protocol Y?".  And, if
    this request fails, the function then asks the protocol Y: "Does
    object X support you, or do you know how to wrap it to obtain such
    a supporter?"  This duality is important, because protocols can be
    developed after objects are, or vice-versa, and this PEP lets
    either case be supported non-invasively with regard to the
    pre-existing component[s].

    This proposal does not limit what a protocol is, what compliance
    to the protocol means, nor what a wrapper constitutes.  This
    mechanism leverages existing protocol categories such as the type
    system and class hierarchy and can be expanded to support future
    protocol categories such as the pending interface proposal [1] and
    signature based type-checking system [2].

Motivation

    Currently there is no standardized mechanism in Python for asking
    if an object supports a particular protocol.  Typically, existence
    of particular methods, particularly those that are built-in such
    as __getitem__, is used as an indicator of support for a
    particular protocol.  This technique works for protocols blessed
    by the BDFL (Benevolent Dictator for Life), such as the new
    enumerator proposal identified by a new built-in __iter__[9].
    However, this technique does not admit an infallible way to
    identify interfaces lacking a unique, built-in signature method.

    More so, there is no standardized way to obtain an adapter for an
    object.  Typically, with objects passed to a context expecting a
    particular protocol, either the object knows about the context and
    provides its own wrapper or the context knows about the object and
    wraps it appropriately.  The difficulty with these approaches is
    that such adaptations are one-offs, are not centralized in a
    single place of the users code, and are not executed with a common
    technique, etc.  This lack of standardization increases code
    duplication with the same adapter occurring in more than one place
    or it encourages classes to be re-written instead of adapted.  In
    either case, maintainability suffers.

    It would be very nice to have a standard function that can be
    called upon to verify an object's compliance with a particular
    protocol and provide for a wrapper if one is readily available --
    all without having to hunt through a library's documentation for
    the appropriate incantation.

Requirements

    When considering an objects compliance with a protocol, there are
    several cases to be examined:

    a) When the protocol is a type or class, and the object has
       exactly that type or is a member of the class.  In this case
       compliance is automatic.

    b) When the object knows about the protocol and either considers
       itself compliant or knows how to wrap itself appropriately.

    c) When the protocol knows about the object and either the object
       already complies or can be wrapped accordingly.

    d) When the protocol is a class, and the object is a member of a
       subclass.  This is distinct from the first case (a) above,
       since inheritance does not necessarily imply substitutability
       and must be handled carefully.

    e) When the context knows about the object and the protocol and
       knows how to adapt the object so that the required protocol is
       satisfied.  This could use an adapter registry or similar
       method.

    For this proposal's requirements, the first case should be come
    for free and the next three cases should be relatively relatively
    easy to accomplish.  This proposal does not address the last case,
    however it provides a base mechanism upon which such an approach
    could be developed.  Further, with only minor implementation
    changes, this proposal should be able to incorporate a new
    interface type or type checking system.

    The fourth case above is subtle.  A lack of substitutability can
    occur when a method restricts an argument's domain or raises an
    exception which a base class does not or extends the co-domain to
    include return values which the base class may never produce.
    While compliance based on class inheritance should be automatic,
    this proposal should allow an object to signal that it is not
    compliant with a base class protocol.

Specification

    This proposal introduces a new built-in function, adapt(), which
    is the basis for supporting these requirements.

    The adapt() function has three parameters:

    - `obj', the object to be adapted

    - `protocol', the protocol requested of the object

    - `alternate', an optional object to return if the object could
      not be adapted

    A successful result of the adapt() function returns either the
    object passed `obj' if the object is already compliant with the
    protocol, or a secondary object `wrapper', which provides a view
    of the object compliant with the protocol.  The definition of
    wrapper is explicitly vague and a wrapper is allowed to be a full
    object with its own state if necessary.  A failure to adapt the
    object to the protocol will raise a TypeError unless the alternate
    parameter is used, in this case the alternate argument is
    returned.

    To enable the first case listed in the requirements, the adapt()
    function first checks to see if the object's type or the object's
    class are identical to the protocol.  If so, then the adapt()
    function returns the object directly without further ado.

    To enable the second case, when the object knows about the
    protocol, the object must have a __conform__() method.  This
    optional method takes two arguments:

    - `self', the object being conformed

    - `protocol, the protocol requested

    The object may return itself through this method to indicate
    compliance.  Alternatively, the object also has the option of
    returning a wrapper object compliant with the protocol.  Finally,
    if the object cannot determine its compliance, it should either
    return None or raise a TypeError to enable the remaining
    mechanisms.

    To enable the third case, when the protocol knows about the
    object, the protocol must have an __adapt__() method.  This
    optional method takes two arguments:

    - `self', the protocol requested

    - `obj', the object being adapted

    If the protocol finds the object to be compliant, it can return
    obj directly.  Alternatively, the method may return a wrapper
    compliant with the protocol.  Finally, compliance cannot be
    determined, this method should either return None or raise a
    TypeError so other mechanisms can be tried.

    The fourth case, when the object's class is a sub-class of the
    protocol, is handled by the built-in adapt() function.  Under
    normal circumstances, if "isinstance(object, protocol)" then
    adapt() returns the object directly.  However, if the object is
    not substitutable, either the __conform__() or __adapt__() methods
    above may raise an adaptForceFailException to prevent this default
    behavior.

    Please note two important things.  First, this proposal does not
    preclude the addition of other protocols.  Second, this proposal
    does not preclude other possible cases where adapter pattern may
    hold, such as the context knowing the object and the protocol (the
    last case in the requirements).  In fact, this proposal opens the
    gate for these other mechanisms to be added.

Reference Implementation and Test Cases

    -----------------------------------------------------------------
    adapt.py
    -----------------------------------------------------------------
    import types

    adaptRaiseTypeException = "(raise a type exception on failure)"
    adaptForceFailException = "(forced failure of adapt)"

    # look to see if the object passes other protocols
    def _check(obj,protocol,default):
            return default

    def adapt(obj, protocol, alternate = adaptRaiseTypeException):

        # first check to see if object has the exact protocol
        if type(obj) is types.InstanceType and \
           obj.__class__ is protocol: return obj
        if type(obj) is protocol: return obj

        # next check other protocols for exact conformance
        # before calling __conform__ or __adapt__
        if _check(obj,protocol,0):
            return obj

        # procedure to execute on success
        def succeed(obj,retval,protocol,alternate):
            if _check(retval,protocol,1):
                return retval
            else:
                return fail(obj,alternate)

        # procedure to execute on failure
        def fail(obj,protocol,alternate):
            if alternate is adaptRaiseTypeException:
                raise TypeError("%s cannot be adapted to %s" \
                                 % (obj,protocol))
            return alternate

        # try to use the object's adapting mechanism
        conform = getattr(obj, '__conform__',None)
        if conform:
            try:
                retval = conform(protocol)
                if retval:
                    return succeed(obj,retval,protocol,alternate)
            except adaptForceFailException:
                return fail(obj,protocol,alternate)
            except TypeError: pass

        # try to use the protocol's adapting mechanism
        adapt = getattr(protocol, '__adapt__',None)
        if adapt:
            try:
                retval = adapt(obj)
                if retval:
                    return succeed(obj,retval,protocol,alternate)
            except adaptForceFailException:
                return fail(obj,protocol,alternate)
            except TypeError: pass

        # check to see if the object is an instance
        try:
            if isinstance(obj,protocol):
                return obj
        except TypeError: pass

        # no-adaptation-possible case
        return fail(obj,protocol,alternate)

    -----------------------------------------------------------------
    test.py
    -----------------------------------------------------------------
    import types
    from adapt import adaptForceFailException
    from adapt import adapt

    class KnightsWhoSayNi: pass

    class Eggs:  # an unrelated class/interface
        def eggs(self): print "eggs!"
        word = "Nee-womm"

    class Ham:  # used as an interface, no inhertance
        def ham(self): pass
        word = "Ping"

    class Spam: # a base class, inheritance used
        def spam(self): print "spam!"

    class EggsSpamAndHam (Spam,KnightsWhoSayNi):
        def ham(self): print "ham!"
        def __conform__(self,protocol):
            if protocol is Ham:
                # implements Ham's ham, but does not have a word
                return self
            if protocol is KnightsWhoSayNi:
                # we are no longer the Knights who say Ni!
                raise adaptForceFailException
            if protocol is Eggs:
                # Knows how to create the eggs!
                return Eggs()

    class SacredWord:
        class HasSecredWord:
            def __call__(self, obj):
                if getattr(obj,'word',None): return obj
        __adapt__= HasSecredWord()

    class Bing (Ham):
        def __conform__(self,protocol):
            raise adaptForceFailException

    def test():
        x = EggsSpamAndHam()
        adapt(x,Spam).spam()
        adapt(x,Eggs).eggs()
        adapt(x,Ham).ham()
        adapt(x,EggsSpamAndHam).ham()
        print adapt(Eggs(),SacredWord).word
        print adapt(Ham(),SacredWord).word
        pass
        if adapt(x,KnightsWhoSayNi,None): raise "IckyIcky"
        if not adapt(x,Spam,None): raise "Spam"
        if not adapt(x,Eggs,None): raise "Eggs"
        if not adapt(x,Ham,None): raise "Ham"
        if not adapt(x,EggsSpamAndHam,None): raise "EggsAndSpam"
        if     adapt(x,KnightsWhoSayNi,None): raise "NightsWhoSayNi"
        if     adapt(x,SacredWord,None): raise "SacredWord"
        try:
            adapt(x,SacredWord)
        except TypeError: pass
        else: raise "SacredWord"
        try:
            adapt(x,KnightsWhoSayNi)
        except TypeError: print "Ekky-ekky-ekky-ekky-z'Bang, " \
                                + "zoom-Boing, z'nourrrwringmm"
        else: raise "NightsWhoSayNi"
        pass
        b = Bing()
        if not adapt(b,Bing,None): raise "Not a Bing"
        if adapt(b,Ham,None): raise "Not a Ham!"
        if adapt(1,types.FloatType,None): raise "Not a float!"
        if adapt(b,types.FloatType,None): raise "Not a float!"
        if adapt(1,Ham,None):             raise "Not a Ham!"
        if not adapt(1,types.IntType,None): raise "Is an Int!"

    -----------------------------------------------------------------
    Expected Output
    -----------------------------------------------------------------
    >>> import test
    >>> test.test()
    spam!
    eggs!
    ham!
    ham!
    Nee-womm
    Ping
    Ekky-ekky-ekky-ekky-z'Bang, zoom-Boing, z'nourrrwringmm
    >>>

Relationship To Paul Prescod and Tim Hochberg's Type Assertion method

    Paul and Tim had proposed a type checking mechanism, where the
    Interface is passed an object to verify.  The example syntax Paul
    put forth recently [2] was:

        interface Interface
            def __check__(self,obj)

    For discussion purposes, here would be a protocol with __check__:

        class Interface:
            class Checker:
                def __call__(self, obj): pass  #check the object
            __check__= Checker()

    The built-in adapt() function could be augmented to use this
    checking mechanism updating the _check method as follows:

    # look to see if the object passes other protocols
    def _check(obj,protocol,default):
        check = getattr(protocol, '__check__',None)
        if check:
            try:
               if check(obj): return 1
            except TypeError: pass
            return 0
        else:
            return default

    In short, the work put forth by Paul and company is great, and
    there should be no problem preventing these two proposals from
    working together in harmony, if not be completely complementary.

Relationship to Python Interfaces [1] by Michel Pelletier

    The relationship to this proposal to Michel's proposal could also
    be complementary.  Following is how the _check method would be
    updated for this mechanism:

    # look to see if the object passes other protocols
    def _check(obj,protocol,default):
        if type(protocol) is types.InterfaceType:
            return implements(obj,protocol)
        return default

Relationship to Carlos Ribeiro's proxy technique [7] and [8]

    Carlos presented a technique where this method could return a
    proxy instead of self or a wrapper.  The advantage of this
    approach is that the internal details of the object are protected.
    This is very neat.  No changes are necessary to this proposal to
    support this usage as a standardized mechanism to obtain named
    proxies.

Relationship To Microsoft's Query Interface

    Although this proposal may sounds similar to Microsoft's
    QueryInterface, it differs by a number of aspects.

    First, it is bi-directional allowing the interface to be queried
    as well giving more dynamic abilities (more Pythonic).  Second,
    there is not a special "IUnknown" interface which can be used for
    object identity, although this could be proposed as one of those
    "special" blessed interface protocol identifiers.  Third, with
    QueryInterface, once an object supports a particular interface it
    must always there after support this interface; this proposal
    makes no such guarantee, although this may be added at a later
    time.  Fourth, implementations of Microsoft's QueryInterface must
    support a kind of equivalence relation.

    By reflexive they mean the querying an interface for itself must
    always succeed.  By symmetrical they mean that if one can
    successfully query an interface IA for a second interface IB, then
    one must also be able to successfully query the interface IB for
    IA.  And finally, by transitive they mean if one can successfully
    query IA for IB and one can successfully query IB for IC, then one
    must be able to successfully query IA for IC.  Ability to support
    this type of equivalence relation should be encouraged, but may
    not be possible.  Further research on this topic (by someone
    familiar with Microsoft COM) would be helpful in further
    determining how compatible this proposal is.

Question and Answer

    Q:  What benefit does this provide?

        The typical Python programmer is an integrator, someone who is
        connecting components from various vendors.  Often times the
        interfaces between these components require an intermediate
        adapter.  Usually the burden falls upon the programmer to
        study the interface exposed by one component and required by
        another, determine if they are directly compatible, or develop
        an adapter.  Sometimes a vendor may even include the
        appropriate adapter, but then searching for the adapter and
        figuring out how to deploy the adapter takes time.

        This technique enables vendors to work with each other
        directly by implementing __conform__ or __adapt__ as
        necessary.  This frees the integrator from making their own
        adapters.  In essence, this allows the components to have a
        simple dialogue among themselves.  The integrator simply
        connects one component to another, and if the types don't
        automatically match an adapting mechanism is built-in.

        For example, consider SAX1 and SAX2 interfaces, there is an
        adapter required to switch between them.  Normally the
        programmer must be aware of this; however, with this
        adaptation framework this is no longer the case.

    Q:  Why does this have to be built-in, can't it be standalone?

        Yes, it does work standalone.  However, if it is built-in, it
        has a greater chance of usage.  The value of this proposal is
        primarily in standardization.  Furthermore:

        0.  The mechanism is by its very nature a singleton.

        1.  If used frequently, it will be much faster as a built-in

        2.  It is extensible and unassuming.

        3.  A whole-program optimizing compiler could optimize it out
            in particular cases (ok, this one is far fetched)

    Q:  Why the verbs __conform__ and __adapt__?

        conform, verb intransitive
            1. To correspond in form or character; be similar.
            2. To act or be in accord or agreement; comply.
            3. To act in accordance with current customs or modes.

        adapt, verb transitive
            1. To make suitable to or fit for a specific use or
               situation.

        Source:  The American Heritage Dictionary of the English
                 Language, Third Edition

Backwards Compatibility

    There should be no problem with backwards compatibility unless
    someone had used __conform__ or __adapt__, but this seems
    unlikely.  Indeed this proposal, save an built-in adapt()
    function, could be tested without changes to the interpreter.

Credits

    This proposal was created in large part by the feedback of the
    talented individuals on both the main mailing list and also the
    type-sig list.  Specific contributors include (sorry if I missed
    someone).

    This proposal is based largely off the suggestions from Alex
    Martelli and Paul Prescod with significant feedback from Robin
    Thomas and borrowing ideas from Marcin 'Qrczak' Kowalczyk and
    Carlos Ribeiro.  Other contributors (via comments) include Michel
    Pelletier, Jeremy Hylton, Aahz Maruch, Fredrik Lundh, Rainer
    Deyke, Timothy Delaney, and Huaiyu Zhu

References and Footnotes

    [1] PEP 245, Python Interface Syntax, Pelletier
        http://www.python.org/peps/pep-0245.html

    [2] http://mail.python.org/pipermail/types-sig/2001-March/001223.html

    [3] http://www.zope.org/Members/michel/types-sig/TreasureTrove

    [4] http://mail.python.org/pipermail/types-sig/2001-March/001105.html

    [5] http://mail.python.org/pipermail/types-sig/2001-March/001206.html

    [6] http://mail.python.org/pipermail/types-sig/2001-March/001223.html

    [7] http://mail.python.org/pipermail/python-list/2001-March/035136.html

    [8] http://mail.python.org/pipermail/python-list/2001-March/035197.html

    [9] PEP 234, Iterators, Yee
        http://www.python.org/peps/pep-0234.txt

Copyright

    This document has been placed in the public domain.


PEP 247 -- API for Cryptographic Hash Functions

PEP: | 247
Title: | API for Cryptographic Hash Functions
Version: | $Revision: 1.2 $
Author: | amk1@bigfoot.com (A.M. Kuchling)
Status: | Draft
Type: | Informational
Created: | 23-Mar-2001
Post-History: |

Abstract

    There are several different modules available that implement
    cryptographic hashing algorithms such as MD5 or SHA.  This
    document specifies a standard API for such algorithms, to make it
    easier to switch between different implementations.

Specification

    All hashing modules should present the same interface.  Additional
    methods or variables can be added, but those described in this
    document should always be present.

    Hash function modules define one function:

    new([string], [key])

        Create a new hashing object and return it.  You can now feed
        arbitrary strings into the object using its update() method,
        and can ask for the hash value at any time by calling its
        digest() method.

        The 'string' parameter, if supplied, will be immediately hashed
        into the object's starting state.  For keyed hashes such as
        HMAC, 'key' is a string containing the starting key.

    Hash function modules define one variable:

    digest_size

        An integer value; the size of the digest produced by the
        hashing objects, measured in bytes.  You could also obtain
        this value by creating a sample object, and taking the length
        of the digest string it returns, but using digest_size is
        faster.

    The methods for hashing objects are always the following:

    clear()

        Reset this hashing object to its initial state, as if the
        object was created from new(key=<initially specified key>)
        (XXX what should this do for a keyed hash?  A proposed
        semantic follows:).  There is no way to supply a starting
        string as there is for the new() function, and there's no way
        to change the key specified for a keyed hash.

    copy()

        Return a separate copy of this hashing object.  An update to
        this copy won't affect the original object.

    digest()

        Return the hash value of this hashing object, as a string
        containing 8-bit data.  The object is not altered in any way
        by this function; you can continue updating the object after
        calling this function.

    update(arg)

        Update this hashing object with the string arg.

    For convenience, hashing objects also have a digest_size
    attribute, measuring the size of the resulting digest in bytes.
    This is identical to the module-level digest_size variable.

    Here's an example, using a module named 'MD5':

        >>> from Crypto.Hash import MD5
        >>> m = MD5.new()
        >>> m.update('abc')
        >>> m.digest()
        '\220\001P\230<\322O\260\326\226?@}(\341\177r'

    Or, more compactly:

        >>> MD5.new('abc').digest()
        '\220\001P\230<\322O\260\326\226?@}(\341\177r'

Copyright

    This document has been placed in the public domain.


PEP 248 -- Python Database API Specification v1.0

PEP: | 248
Title: | Python Database API Specification v1.0
Version: | $Revision: 1.2 $
Author: | db-sig@python.org (Python Database SIG)
Editor: | mal@lemburg.com (Marc-Andre Lemburg)
Status: | Draft
Replaced-By: | 249
Type: | Informational

Introduction

    This API has been defined to encourage similarity between the
    Python modules that are used to access databases.  By doing this,
    we hope to achieve a consistency leading to more easily understood
    modules, code that is generally more portable across databases,
    and a broader reach of database connectivity from Python.

    This interface specification consists of several items:

        * Module Interface
        * Connection Objects
        * Cursor Objects
        * DBI Helper Objects

    Comments and questions about this specification may be directed to
    the SIG on Tabular Databases in Python
    (http://www.python.org/sigs/db-sig).

    This specification document was last updated on: April 9, 1996.
    It will be known as Version 1.0 of this specification.

Module Interface

    The database interface modules should typically be named with
    something terminated by 'db'.  Existing examples are: 'oracledb',
    'informixdb', and 'pg95db'.  These modules should export several
    names:

        modulename(connection_string)

            Constructor for creating a connection to the database.
            Returns a Connection Object.

        error

            Exception raise for errors from the database module.

Connection Objects

    Connection Objects should respond to the following methods:

        close()

            Close the connection now (rather than whenever __del__ is
            called).  The connection will be unusable from this point
            forward; an exception will be raised if any operation is
            attempted with the connection.

        commit()

            Commit any pending transaction to the database.

        rollback()

            Roll the database back to the start of any pending
            transaction.

        cursor()

            Return a new Cursor Object.  An exception may be thrown if
            the database does not support a cursor concept.

        callproc([params])

            (Note: this method is not well-defined yet.)  Call a
            stored database procedure with the given (optional)
            parameters.  Returns the result of the stored procedure.

        (all Cursor Object attributes and methods)

            For databases that do not have cursors and for simple
            applications that do not require the complexity of a
            cursor, a Connection Object should respond to each of the
            attributes and methods of the Cursor Object.  Databases
            that have cursor can implement this by using an implicit,
            internal cursor.

Cursor Objects

    These objects represent a database cursor, which is used to manage
    the context of a fetch operation.

    Cursor Objects should respond to the following methods and
    attributes:

        arraysize

            This read/write attribute specifies the number of rows to
            fetch at a time with fetchmany().  This value is also used
            when inserting multiple rows at a time (passing a
            tuple/list of tuples/lists as the params value to
            execute()).  This attribute will default to a single row.

            Note that the arraysize is optional and is merely provided
            for higher performance database interactions.
            Implementations should observe it with respect to the
            fetchmany() method, but are free to interact with the
            database a single row at a time.

        description

            This read-only attribute is a tuple of 7-tuples.  Each
            7-tuple contains information describing each result
            column: (name, type_code, display_size, internal_size,
            precision, scale, null_ok). This attribute will be None
            for operations that do not return rows or if the cursor
            has not had an operation invoked via the execute() method
            yet.

            The 'type_code' is one of the 'dbi' values specified in
            the section below.

            Note: this is a bit in flux. Generally, the first two
            items of the 7-tuple will always be present; the others
            may be database specific.

        close()

            Close the cursor now (rather than whenever __del__ is
            called).  The cursor will be unusable from this point
            forward; an exception will be raised if any operation is
            attempted with the cursor.

        execute(operation [,params])

            Execute (prepare) a database operation (query or command).
            Parameters may be provided (as a sequence
            (e.g. tuple/list)) and will be bound to variables in the
            operation.  Variables are specified in a database-specific
            notation that is based on the index in the parameter tuple
            (position-based rather than name-based).

            The parameters may also be specified as a sequence of
            sequences (e.g. a list of tuples) to insert multiple rows
            in a single operation.

            A reference to the operation will be retained by the
            cursor.  If the same operation object is passed in again,
            then the cursor can optimize its behavior.  This is most
            effective for algorithms where the same operation is used,
            but different parameters are bound to it (many times).

            For maximum efficiency when reusing an operation, it is
            best to use the setinputsizes() method to specify the
            parameter types and sizes ahead of time.  It is legal for
            a parameter to not match the predefined information; the
            implementation should compensate, possibly with a loss of
            efficiency.

            Using SQL terminology, these are the possible result
            values from the execute() method:

                If the statement is DDL (e.g. CREATE TABLE), then 1 is
                returned.

                If the statement is DML (e.g. UPDATE or INSERT), then the
                number of rows affected is returned (0 or a positive
                integer).

                If the statement is DQL (e.g. SELECT), None is returned,
                indicating that the statement is not really complete until
                you use one of the  'fetch' methods.

        fetchone()

            Fetch the next row of a query result, returning a single
            tuple.

        fetchmany([size])

            Fetch the next set of rows of a query result, returning as
            a list of tuples. An empty list is returned when no more
            rows are available. The number of rows to fetch is
            specified by the parameter.  If it is None, then the
            cursor's arraysize determines the number of rows to be
            fetched.

            Note there are performance considerations involved with
            the size parameter.  For optimal performance, it is
            usually best to use the arraysize attribute.  If the size
            parameter is used, then it is best for it to retain the
            same value from one fetchmany() call to the next.

        fetchall()

            Fetch all rows of a query result, returning as a list of
            tuples.  Note that the cursor's arraysize attribute can
            affect the performance of this operation.

        setinputsizes(sizes)

            (Note: this method is not well-defined yet.)  This can be
            used before a call to 'execute()' to predefine memory
            areas for the operation's parameters.  sizes is specified
            as a tuple -- one item for each input parameter.  The item
            should be a Type object that corresponds to the input that
            will be used, or it should be an integer specifying the
            maximum length of a string parameter.  If the item is
            'None', then no predefined memory area will be reserved
            for that column (this is useful to avoid predefined areas
            for large inputs).

            This method would be used before the execute() method is
            invoked.

            Note that this method is optional and is merely provided
            for higher performance database interaction.
            Implementations are free to do nothing and users are free
            to not use it.

        setoutputsize(size [,col])

            (Note: this method is not well-defined yet.)

            Set a column buffer size for fetches of large columns
            (e.g. LONG).  The column is specified as an index into the
            result tuple.  Using a column of None will set the default
            size for all large columns in the cursor.

            This method would be used before the 'execute()' method is
            invoked.

            Note that this method is optional and is merely provided
            for higher performance database interaction.
            Implementations are free to do nothing and users are free
            to not use it.

DBI Helper Objects

    Many databases need to have the input in a particular format for
    binding to an operation's input parameters.  For example, if an
    input is destined for a DATE column, then it must be bound to the
    database in a particular string format.  Similar problems exist
    for "Row ID" columns or large binary items (e.g. blobs or RAW
    columns).  This presents problems for Python since the parameters
    to the 'execute()' method are untyped.  When the database module
    sees a Python string object, it doesn't know if it should be bound
    as a simple CHAR column, as a raw binary item, or as a DATE.

    To overcome this problem, the 'dbi' module was created.  This
    module specifies some basic database interface types for working
    with databases.  There are two classes: 'dbiDate' and 'dbiRaw'.
    These are simple container classes that wrap up a value.  When
    passed to the database modules, the module can then detect that
    the input parameter is intended as a DATE or a RAW.  For symmetry,
    the database modules will return DATE and RAW columns as instances
    of these classes.

    A Cursor Object's 'description' attribute returns information
    about each of the result columns of a query.  The 'type_code is
    defined to be one of five types exported by this module: 'STRING',
    'RAW', 'NUMBER', 'DATE', or 'ROWID'.

    The module exports the following names:

        dbiDate(value)

            This function constructs a 'dbiDate' instance that holds a
            date value.  The value should be specified as an integer
            number of seconds since the "epoch" (e.g. time.time()).

        dbiRaw(value)

            This function constructs a 'dbiRaw' instance that holds a
            raw (binary) value.  The value should be specified as a
            Python string.

        STRING

            This object is used to describe columns in a database that
            are string-based (e.g. CHAR).

        RAW

            This object is used to describe (large) binary columns in
            a database (e.g. LONG RAW, blobs).

        NUMBER

            This object is used to describe numeric columns in a
            database.

        DATE

            This object is used to describe date columns in a
            database.

        ROWID

            This object is used to describe the "Row ID" column in a
            database.

Acknowledgements

    Many thanks go to Andrew Kuchling who converted the Python
    Database API Specification 1.0 from the original HTML format into
    the PEP format.

Copyright

    This document has been placed in the Public Domain.


PEP 249 -- Python Database API Specification v2.0

PEP: | 249
Title: | Python Database API Specification v2.0
Version: | $Revision: 1.3 $
Author: | db-sig@python.org (Python Database SIG)
Editor: | mal@lemburg.com (Marc-Andre Lemburg)
Status: | Draft
Type: | Informational
Replaces: | 248

Introduction

    This API has been defined to encourage similarity between the
    Python modules that are used to access databases.  By doing this,
    we hope to achieve a consistency leading to more easily understood
    modules, code that is generally more portable across databases,
    and a broader reach of database connectivity from Python.

    The interface specification consists of several sections:

        * Module Interface
        * Connection Objects
        * Cursor Objects
        * DBI Helper Objects
        * Type Objects and Constructors
        * Implementation Hints
        * Major Changes from 1.0 to 2.0

    Comments and questions about this specification may be directed
    to the SIG for Database Interfacing with Python
    (db-sig@python.org).

    For more information on database interfacing with Python and
    available packages see the Database Topic
    Guide at http://www.python.org/topics/database/.

    This document describes the Python Database API Specification
    2.0.  The previous version 1.0 version is still available as
    reference, in PEP 248. Package writers are encouraged to use
    this version of the specification as basis for new interfaces.

Module Interface

    Access to the database is made available through connection
    objects. The module must provide the following constructor for
    these:

        connect(parameters...)

            Constructor for creating a connection to the database.
            Returns a Connection Object. It takes a number of
            parameters which are database dependent. [1]

    These module globals must be defined:

        apilevel

            String constant stating the supported DB API level.
            Currently only the strings '1.0' and '2.0' are allowed.

            If not given, a DB-API 1.0 level interface should be
            assumed.

        threadsafety

            Integer constant stating the level of thread safety the
            interface supports. Possible values are:

                0     Threads may not share the module.
                1     Threads may share the module, but not connections.
                2     Threads may share the module and connections.
                3     Threads may share the module, connections and
                      cursors.

            Sharing in the above context means that two threads may
            use a resource without wrapping it using a mutex semaphore
            to implement resource locking. Note that you cannot always
            make external resources thread safe by managing access
            using a mutex: the resource may rely on global variables
            or other external sources that are beyond your control.

        paramstyle

            String constant stating the type of parameter marker
            formatting expected by the interface. Possible values are
            [2]:

                'qmark'         Question mark style,
                                e.g. '...WHERE name=?'
                'numeric'       Numeric, positional style,
                                e.g. '...WHERE name=:1'
                'named'         Named style,
                                e.g. '...WHERE name=:name'
                'format'        ANSI C printf format codes,
                                e.g. '...WHERE name=%s'
                'pyformat'      Python extended format codes,
                                e.g. '...WHERE name=%(name)s'

    The module should make all error information available through
    these exceptions or subclasses thereof:

        Warning

            Exception raised for important warnings like data
            truncations while inserting, etc. It must be a subclass of
            the Python StandardError (defined in the module
            exceptions).

        Error

            Exception that is the base class of all other error
            exceptions. You can use this to catch all errors with one
            single 'except' statement. Warnings are not considered
            errors and thus should not use this class as base. It must
            be a subclass of the Python StandardError (defined in the
            module exceptions).

        InterfaceError

            Exception raised for errors that are related to the
            database interface rather than the database itself.  It
            must be a subclass of Error.

        DatabaseError

            Exception raised for errors that are related to the
            database.  It must be a subclass of Error.

        DataError

            Exception raised for errors that are due to problems with
            the processed data like division by zero, numeric value
            out of range, etc. It must be a subclass of DatabaseError.

        OperationalError

            Exception raised for errors that are related to the
            database's operation and not necessarily under the control
            of the programmer, e.g. an unexpected disconnect occurs,
            the data source name is not found, a transaction could not
            be processed, a memory allocation error occurred during
            processing, etc.  It must be a subclass of DatabaseError.

        IntegrityError

            Exception raised when the relational integrity of the
            database is affected, e.g. a foreign key check fails.  It
            must be a subclass of DatabaseError.

        InternalError

            Exception raised when the database encounters an internal
            error, e.g. the cursor is not valid anymore, the
            transaction is out of sync, etc.  It must be a subclass of
            DatabaseError.

        ProgrammingError

            Exception raised for programming errors, e.g. table not
            found or already exists, syntax error in the SQL
            statement, wrong number of parameters specified, etc.  It
            must be a subclass of DatabaseError.

        NotSupportedError

            Exception raised in case a method or database API was used
            which is not supported by the database, e.g. requesting a
            .rollback() on a connection that does not support
            transaction or has transactions turned off.  It must be a
            subclass of DatabaseError.

    This is the exception inheritance layout:

        StandardError
        |__Warning
        |__Error
           |__InterfaceError
              |__DatabaseError
                 |__DataError
                 |__OperationalError
                 |__IntegrityError
                 |__InternalError
                 |__ProgrammingError
                 |__NotSupportedError

    Note: The values of these exceptions are not defined. They should
    give the user a fairly good idea of what went wrong, though.

Connection Objects

    Connection Objects should respond to the following methods:

        close()

            Close the connection now (rather than whenever __del__ is
            called).  The connection will be unusable from this point
            forward; an Error (or subclass) exception will be raised
            if any operation is attempted with the connection. The
            same applies to all cursor objects trying to use the
            connection.  Note that closing a connection without
            committing the changes first will cause an implicit
            rollback to be performed.

        commit()

            Commit any pending transaction to the database. Note that
            if the database supports an auto-commit feature, this must
            be initially off. An interface method may be provided to
            turn it back on.

            Database modules that do not support transactions should
            implement this method with void functionality.

        rollback()

            This method is optional since not all databases provide
            transaction support. [3]

            In case a database does provide transactions this method
            causes the the database to roll back to the start of any
            pending transaction.  Closing a connection without
            committing the changes first will cause an implicit
            rollback to be performed.

        cursor()

            Return a new Cursor Object using the connection.  If the
            database does not provide a direct cursor concept, the
            module will have to emulate cursors using other means to
            the extent needed by this specification.  [4]

Cursor Objects

    These objects represent a database cursor, which is used to
    manage the context of a fetch operation. Cursors created from
    the same connection are not isolated, i.e., any changes
    done to the database by a cursor are immediately visible by the
    other cursors. Cursors created from different connections can
    or can not be isolated, depending on how the transaction support
    is implemented (see also the connection's rollback() and commit()
    methods.)

    Cursor Objects should respond to the following methods and
    attributes:

        description

            This read-only attribute is a sequence of 7-item
            sequences.  Each of these sequences contains information
            describing one result column: (name, type_code,
            display_size, internal_size, precision, scale,
            null_ok). The first two items (name and type_code) are
            mandatory, the other five are optional and must be set to
            None if meaningfull values are not provided.

            This attribute will be None for operations that
            do not return rows or if the cursor has not had an
            operation invoked via the executeXXX() method yet.

            The type_code can be interpreted by comparing it to the
            Type Objects specified in the section below.

        rowcount

            This read-only attribute specifies the number of rows that
            the last executeXXX() produced (for DQL statements like
            'select') or affected (for DML statements like 'update' or
            'insert').

            The attribute is -1 in case no executeXXX() has been
            performed on the cursor or the rowcount of the last
            operation is not determinable by the interface. [7]

        callproc(procname[,parameters])

            (This method is optional since not all databases provide
            stored procedures. [3])

            Call a stored database procedure with the given name. The
            sequence of parameters must contain one entry for each
            argument that the procedure expects. The result of the
            call is returned as modified copy of the input
            sequence. Input parameters are left untouched, output and
            input/output parameters replaced with possibly new values.

            The procedure may also provide a result set as
            output. This must then be made available through the
            standard fetchXXX() methods.

        close()

            Close the cursor now (rather than whenever __del__ is
            called).  The cursor will be unusable from this point
            forward; an Error (or subclass) exception will be raised
            if any operation is attempted with the cursor.

        execute(operation[,parameters])

            Prepare and execute a database operation (query or
            command).  Parameters may be provided as sequence or
            mapping and will be bound to variables in the operation.
            Variables are specified in a database-specific notation
            (see the module's paramstyle attribute for details). [5]

            A reference to the operation will be retained by the
            cursor.  If the same operation object is passed in again,
            then the cursor can optimize its behavior.  This is most
            effective for algorithms where the same operation is used,
            but different parameters are bound to it (many times).

            For maximum efficiency when reusing an operation, it is
            best to use the setinputsizes() method to specify the
            parameter types and sizes ahead of time.  It is legal for
            a parameter to not match the predefined information; the
            implementation should compensate, possibly with a loss of
            efficiency.

            The parameters may also be specified as list of tuples to
            e.g. insert multiple rows in a single operation, but this
            kind of usage is depreciated: executemany() should be used
            instead.

            Return values are not defined.

        executemany(operation,seq_of_parameters)

            Prepare a database operation (query or command) and then
            execute it against all parameter sequences or mappings
            found in the sequence seq_of_parameters.

            Modules are free to implement this method using multiple
            calls to the execute() method or by using array operations
            to have the database process the sequence as a whole in
            one call.

            The same comments as for execute() also apply accordingly
            to this method.

            Return values are not defined.

        fetchone()

            Fetch the next row of a query result set, returning a
            single sequence, or None when no more data is
            available. [6]

            An Error (or subclass) exception is raised if the previous
            call to executeXXX() did not produce any result set or no
            call was issued yet.

        fetchmany([size=cursor.arraysize])

            Fetch the next set of rows of a query result, returning a
            sequence of sequences (e.g. a list of tuples). An empty
            sequence is returned when no more rows are available.

            The number of rows to fetch per call is specified by the
            parameter.  If it is not given, the cursor's arraysize
            determines the number of rows to be fetched. The method
            should try to fetch as many rows as indicated by the size
            parameter. If this is not possible due to the specified
            number of rows not being available, fewer rows may be
            returned.

            An Error (or subclass) exception is raised if the previous
            call to executeXXX() did not produce any result set or no
            call was issued yet.

            Note there are performance considerations involved with
            the size parameter.  For optimal performance, it is
            usually best to use the arraysize attribute.  If the size
            parameter is used, then it is best for it to retain the
            same value from one fetchmany() call to the next.

        fetchall()

            Fetch all (remaining) rows of a query result, returning
            them as a sequence of sequences (e.g. a list of tuples).
            Note that the cursor's arraysize attribute can affect the
            performance of this operation.

            An Error (or subclass) exception is raised if the previous
            call to executeXXX() did not produce any result set or no
            call was issued yet.

          nextset()

            (This method is optional since not all databases support
            multiple result sets. [3])

            This method will make the cursor skip to the next
            available set, discarding any remaining rows from the
            current set.

            If there are no more sets, the method returns
            None. Otherwise, it returns a true value and subsequent
            calls to the fetch methods will return rows from the next
            result set.

            An Error (or subclass) exception is raised if the previous
            call to executeXXX() did not produce any result set or no
            call was issued yet.

        arraysize

            This read/write attribute specifies the number of rows to
            fetch at a time with fetchmany(). It defaults to 1 meaning
            to fetch a single row at a time.

            Implementations must observe this value with respect to
            the fetchmany() method, but are free to interact with the
            database a single row at a time. It may also be used in
            the implementation of executemany().

        setinputsizes(sizes)

            This can be used before a call to executeXXX() to
            predefine memory areas for the operation's parameters.

            sizes is specified as a sequence -- one item for each
            input parameter.  The item should be a Type Object that
            corresponds to the input that will be used, or it should
            be an integer specifying the maximum length of a string
            parameter.  If the item is None, then no predefined memory
            area will be reserved for that column (this is useful to
            avoid predefined areas for large inputs).

            This method would be used before the executeXXX() method
            is invoked.

            Implementations are free to have this method do nothing
            and users are free to not use it.

        setoutputsize(size[,column])

            Set a column buffer size for fetches of large columns
            (e.g. LONGs, BLOBs, etc.).  The column is specified as an
            index into the result sequence.  Not specifying the column
            will set the default size for all large columns in the
            cursor.

            This method would be used before the executeXXX() method
            is invoked.

            Implementations are free to have this method do nothing
            and users are free to not use it.

Type Objects and Constructors

    Many databases need to have the input in a particular format for
    binding to an operation's input parameters.  For example, if an
    input is destined for a DATE column, then it must be bound to the
    database in a particular string format.  Similar problems exist
    for "Row ID" columns or large binary items (e.g. blobs or RAW
    columns).  This presents problems for Python since the parameters
    to the executeXXX() method are untyped.  When the database module
    sees a Python string object, it doesn't know if it should be bound
    as a simple CHAR column, as a raw BINARY item, or as a DATE.

    To overcome this problem, a module must provide the constructors
    defined below to create objects that can hold special values.
    When passed to the cursor methods, the module can then detect the
    proper type of the input parameter and bind it accordingly.

    A Cursor Object's description attribute returns information about
    each of the result columns of a query.  The type_code must compare
    equal to one of Type Objects defined below. Type Objects may be
    equal to more than one type code (e.g. DATETIME could be equal to
    the type codes for date, time and timestamp columns; see the
    Implementation Hints below for details).

    The module exports the following constructors and singletons:

        Date(year,month,day)

            This function constructs an object holding a date value.

        Time(hour,minute,second)

            This function constructs an object holding a time value.

        Timestamp(year,month,day,hour,minute,second)

            This function constructs an object holding a time stamp
            value.

        DateFromTicks(ticks)

            This function constructs an object holding a date value
            from the given ticks value (number of seconds since the
            epoch; see the documentation of the standard Python time
            module for details).

        TimeFromTicks(ticks)

            This function constructs an object holding a time value
            from the given ticks value (number of seconds since the
            epoch; see the documentation of the standard Python time
            module for details).

        TimestampFromTicks(ticks)

            This function constructs an object holding a time stamp
            value from the given ticks value (number of seconds since
            the epoch; see the documentation of the standard Python
            time module for details).

        Binary(string)

            This function constructs an object capable of holding a
            binary (long) string value.

        STRING

            This type object is used to describe columns in a database
            that are string-based (e.g. CHAR).

        BINARY

            This type object is used to describe (long) binary columns
            in a database (e.g. LONG, RAW, BLOBs).

        NUMBER

            This type object is used to describe numeric columns in a
            database.

        DATETIME

            This type object is used to describe date/time columns in
            a database.

        ROWID

            This type object is used to describe the "Row ID" column
            in a database.

    SQL NULL values are represented by the Python None singleton on
    input and output.

    Note: Usage of Unix ticks for database interfacing can cause
    troubles because of the limited date range they cover.

Implementation Hints

    * The preferred object types for the date/time objects are those
      defined in the mxDateTime package. It provides all necessary
      constructors and methods both at Python and C level.

    * The preferred object type for Binary objects are the
      buffer types available in standard Python starting with
      version 1.5.2. Please see the Python documentation for
      details. For information about the the C interface have a
      look at Include/bufferobject.h and
      Objects/bufferobject.c in the Python source
      distribution.

    * Here is a sample implementation of the Unix ticks based
      constructors for date/time delegating work to the generic
      constructors:

        import time

        def DateFromTicks(ticks):
            return apply(Date,time.localtime(ticks)[:3])

        def TimeFromTicks(ticks):
            return apply(Time,time.localtime(ticks)[3:6])

        def TimestampFromTicks(ticks):
            return apply(Timestamp,time.localtime(ticks)[:6])

    * This Python class allows implementing the above type
      objects even though the description type code field yields
      multiple values for on type object:

        class DBAPITypeObject:
            def __init__(self,*values):
                self.values = values
            def __cmp__(self,other):
                if other in self.values:
                    return 0
                if other < self.values:
                    return 1
                else:
                    return -1

      The resulting type object compares equal to all values
      passed to the constructor.

    * Here is a snippet of Python code that implements the exception
      hierarchy defined above:

        import exceptions

        class Error(exceptions.StandardError):
            pass

        class Warning(exceptions.StandardError):
            pass

        class InterfaceError(Error):
            pass

        class DatabaseError(Error):
            pass

        class InternalError(DatabaseError):
            pass

        class OperationalError(DatabaseError):
            pass

        class ProgrammingError(DatabaseError):
            pass

        class IntegrityError(DatabaseError):
            pass

        class DataError(DatabaseError):
            pass

        class NotSupportedError(DatabaseError):
            pass

      In C you can use the PyErr_NewException(fullname,
      base, NULL) API to create the exception objects.

Major Changes from Version 1.0 to Version 2.0

    The Python Database API 2.0 introduces a few major changes
    compared to the 1.0 version. Because some of these changes will
    cause existing DB API 1.0 based scripts to break, the major
    version number was adjusted to reflect this change.

    These are the most important changes from 1.0 to 2.0:

        * The need for a separate dbi module was dropped and the
          functionality merged into the module interface itself.

        * New constructors and Type Objects were added for date/time
          values, the RAW Type Object was renamed to BINARY. The
          resulting set should cover all basic data types commonly
          found in modern SQL databases.

        * New constants (apilevel, threadlevel, paramstyle) and
          methods (executemany, nextset) were added to provide better
          database bindings.

        * The semantics of .callproc() needed to call stored
          procedures are now clearly defined.

        * The definition of the .execute() return value changed.
          Previously, the return value was based on the SQL statement
          type (which was hard to implement right) -- it is undefined
          now; use the more flexible .rowcount attribute
          instead. Modules are free to return the old style return
          values, but these are no longer mandated by the
          specification and should be considered database interface
          dependent.

        * Class based exceptions were incorporated into the
          specification.  Module implementors are free to extend the
          exception layout defined in this specification by
          subclassing the defined exception classes.

Open Issues

    Although the version 2.0 specification clarifies a lot of
    questions that were left open in the 1.0 version, there are still
    some remaining issues:

        * Define a useful return value for .nextset() for the case where
          a new result set is available.

        * Create a fixed point numeric type for use as loss-less
          monetary and decimal interchange format.

Footnotes

    [1] As a guideline the connection constructor parameters should be
        implemented as keyword parameters for more intuitive use and
        follow this order of parameters:

        dsn         Data source name as string
        user        User name as string (optional)
        password    Password as string (optional)
        host        Hostname (optional)
        database    Database name (optional)

        E.g. a connect could look like this:

        connect(dsn='myhost:MYDB',user='guido',password='234$')

    [2] Module implementors should prefer 'numeric', 'named' or
        'pyformat' over the other formats because these offer more
        clarity and flexibility.

    [3] If the database does not support the functionality required
        by the method, the interface should throw an exception in
        case the method is used.

        The preferred approach is to not implement the method and
        thus have Python generate an AttributeError in
        case the method is requested. This allows the programmer to
        check for database capabilities using the standard
        hasattr() function.

        For some dynamically configured interfaces it may not be
        appropriate to require dynamically making the method
        available. These interfaces should then raise a
        NotSupportedError to indicate the non-ability
        to perform the roll back when the method is invoked.

    [4] a database interface may choose to support named cursors by
        allowing a string argument to the method. This feature is
        not part of the specification, since it complicates
        semantics of the .fetchXXX() methods.

    [5] The module will use the __getitem__ method of the parameters
        object to map either positions (integers) or names (strings)
        to parameter values. This allows for both sequences and
        mappings to be used as input.

        The term "bound" refers to the process of binding an input
        value to a database execution buffer. In practical terms,
        this means that the input value is directly used as a value
        in the operation.  The client should not be required to
        "escape" the value so that it can be used -- the value
        should be equal to the actual database value.

    [6] Note that the interface may implement row fetching using
        arrays and other optimizations. It is not
        guaranteed that a call to this method will only move the
        associated cursor forward by one row.

    [7] The rowcount attribute may be coded in a way that updates
        its value dynamically. This can be useful for databases that
        return usable rowcount values only after the first call to
        a .fetchXXX() method.

Acknowledgements

    Many thanks go to Andrew Kuchling who converted the Python
    Database API Specification 2.0 from the original HTML format into
    the PEP format.

Copyright

    This document has been placed in the Public Domain.


PEP 250 -- Using site-packages on All Platforms

PEP: | 250
Title: | Using site-packages on All Platforms
Version $Revision: | 1.1 $
Author: | gustav@morpheus.demon.co.uk (Paul Moore)
Status: | Draft
Type: | Standards Track
Created: | 2001-03-30
Python-Version: | 2.2
Post-History: |

Abstract

    The standard Python distribution includes a directory
    Lib/site-packages, which is used on Unix platforms to hold
    locally-installed modules and packages.  The site.py module
    distributed with Python includes support for locating other
    modules in the site-packages directory.

    This PEP proposes that the site-packages directory should be used
    uniformly across all platforms for locally installed modules.

Motivation

    On Windows platforms, the default setting for sys.path does not
    include a directory suitable for users to install locally
    developed modules.  The "expected" location appears to be the
    directory containing the Python executable itself.  Including
    locally developed code in the same directory as installed
    executables is not good practice.

    Clearly, users can manipulate sys.path, either in a locally
    modified site.py, or in a suitable sitecustomize.py, or even via
    .pth files.  However, there should be a standard location for such
    files, rather than relying on every individual site having to set
    their own policy.

    In addition, with distutils becoming more prevalent as a means of
    distributing modules, the need for a standard install location for
    distributed modules will become more common.  It would be better
    to define such a standard now, rather than later when more
    distutils-based packages exist which will need rebuilding.

    It is relevant to note that prior to Python 2.1, the site-packages
    directory was not included in sys.path for Macintosh platforms.
    This has been changed in 2.1, and Macintosh includes sys.path now,
    leaving Windows as the only major platform with no site-specific
    modules directory.

Implementation

    The implementation of this feature is fairly trivial.  All that
    would be required is a change to site.py, to change the section
    setting sitedirs.  The Python 2.1 version has

        if os.sep == '/':
            sitedirs = [makepath(prefix,
                                 "lib",
                                 "python" + sys.version[:3],
                                 "site-packages"),
                        makepath(prefix, "lib", "site-python")]
        elif os.sep == ':':
            sitedirs = [makepath(prefix, "lib", "site-packages")]
        else:
            sitedirs = [prefix]

    A suitable change would be to simply replace the last 4 lines with

        else:
            sitedirs == [makepath(prefix, "lib", "site-packages")]

    Changes would also be required to distutils, in the sysconfig.py
    file.  It is worth noting that this file does not seem to have
    been updated in line with the change of policy on the Macintosh,
    as of this writing.

Notes

    1. It would be better if this change could be included in Python
       2.1, as changing something of this nature is better done
       sooner, rather than later, to reduce the backward-compatibility
       burden. This is extremely unlikely to happen at this late stage
       in the release cycle, however.

    2. This change does not preclude packages using the current
       location -- the change only adds a directory to sys.path, it
       does not remove anything.

    3. In the Windows distribution of Python 2.1 (beta 1), the
       Lib\site-packages directory has been removed.  It would need to
       be reinstated.

Copyright

    This document has been placed in the public domain.


PEP 251 -- Python 2.2 Release Schedule

PEP: | 251
Title: | Python 2.2 Release Schedule
Version: | $Revision: 1.1 $
Author: | Barry Warsaw
Status: | Incomplete
Type: | Informational
Created: | 17-Apr-2001
Python-Version: | 2.2
Post-History: |

Abstract

    This document describes the post-Python 2.1 development and
    release schedule.  The schedule primarily concerns itself with
    PEP-sized items.  Small bug fixes and changes will occur up until
    the first beta release.

Release Schedule

    Python 2.2 is in the very early planning stages, and Guido
    expressed a desire to release Python 2.2 in October of 2001.

    Tentative future release dates

    October-2001: 2.2 final release

Copyright

    This document has been placed in the public domain.


PEP 252 -- Making Types Look More Like Classes

PEP: | 252
Title: | Making Types Look More Like Classes
Version: | $Revision: 1.7 $
Author: | guido@python.org (Guido van Rossum)
Status: | Draft
Type: | Standards Track
Python-Version: | 2.2
Created: | 19-Apr-2001
Post-History: |

Abstract

    This PEP proposes changes to the introspection API for types that
    makes them look more like classes.  For example, type(x) will be
    equivalent to x.__class__ for most built-in types.  When C is
    x.__class__, x.meth(a) will be equivalent to C.meth(x, a), and
    C.__dict__ contains descriptors for x's methods and other
    attributes.

    The PEP also introduces a new approach to specifying attributes,
    using attribute descriptors, or descriptors for short.
    Descriptors unify and generalize several different common
    mechanisms used for describing attributes: a descriptor can
    describe a method, a typed field in the object structure, or a
    generalized attribute represented by getter and setter functions.

Introduction

    One of Python's oldest language warts is the difference between
    classes and types.  For example, you can't directly subclass the
    dictionary type, and the introspection interface for finding out
    what methods and instance variables an object has is different for
    types and for classes.

    Healing the class/type split is a big effort, because it affects
    many aspects of how Python is implemented.  This PEP concerns
    itself with making the introspection API for types look the same
    as that for classes.  Other PEPs will propose making classes look
    more like types, and subclassing from built-in types; these topics
    are not on the table for this PEP.

Introspection APIs

    Introspection concerns itself with finding out what attributes an
    object has.  Python's very general getattr/setattr API makes it
    impossible to guarantee that there always is a way to get a list
    of all attributes supported by a specific object, but in practice
    two conventions have appeared that together work for almost all
    objects.  I'll call them the class-based introspection API and the
    type-based introspection API; class API and type API for short.

    The class-based introspection API is used primarily for class
    instances; it is also used by Jim Fulton's ExtensionClasses.  It
    assumes that all data attributes of an object x are stored in the
    dictionary x.__dict__, and that all methods and class variables
    can be found by inspection of x's class, written as x.__class__.
    Classes have a __dict__ attribute, which yields a dictionary
    containing methods and class variables defined by the class
    itself, and a __bases__ attribute, which is a tuple of base
    classes that must be inspected recursively.  Some assumption here
    are:

    - attributes defined in the instance dict override attributes
      defined by the object's class;

    - attributes defined in a derived class override attributes
      defined in a base class;

    - attributes in an earlier base class (meaning occurring earlier
      in __bases__) override attributes in a later base class.

    (The last two rules together are often summarized as the
    left-to-right, depth-first rule for attribute search.)

    The type-based introspection API is supported in one form or
    another by most built-in objects.  It uses two special attributes,
    __members__ and __methods__.  The __methods__ attribute, if
    present, is a list of method names supported by the object.  The
    __members__ attribute, if present, is a list of data attribute
    names supported by the object.

    The type API is sometimes combined by a __dict__ that works the
    same was as for instances (e.g., for function objects in Python
    2.1, f.__dict__ contains f's dynamic attributes, while
    f.__members__ lists the names of f's statically defined
    attributes).

    Some caution must be exercised: some objects don't list theire
    "intrinsic" attributes (e.g. __dict__ and __doc__) in __members__,
    while others do; sometimes attribute names that occur both in
    __members__ or __methods__ and as keys in __dict__, in which case
    it's anybody's guess whether the value found in __dict__ is used
    or not.

    The type API has never been carefully specified.  It is part of
    Python folklore, and most third party extensions support it
    because they follow examples that support it.  Also, any type that
    uses Py_FindMethod() and/or PyMember_Get() in its tp_getattr
    handler supports it, because these two functions special-case the
    attribute names __methods__ and __members__, respectively.

    Jim Fulton's ExtensionClasses ignore the type API, and instead
    emulate the class API, which is more powerful.  In this PEP, I
    propose to phase out the type API in favor of supporting the class
    API for all types.

    One argument in favor of the class API is that it doesn't require
    you to create an instance in order to find out which attributes a
    type supports; this in turn is useful for documentation
    processors.  For example, the socket module exports the SocketType
    object, but this currently doesn't tell us what methods are
    defined on socket objects.  Using the class API, SocketType shows
    us exactly what the methods for socket objects are, and we can
    even extract their docstrings, without creating a socket.  (Since
    this is a C extension module, the source-scanning approach to
    docstring extraction isn't feasible in this case.)

Specification of the class-based introspection API

    Objects may have two kinds of attributes: static and dynamic.  The
    names and sometimes other properties of static attributes are
    knowable by inspection of the object's type or class, which is
    accessible through obj.__class__ or type(obj).  (I'm using type
    and class interchangeably, because that's the goal of the
    exercise.)

    (XXX static and dynamic are lousy names, because the "static"
    attributes may actually behave quite dynamically.)

    The names and values of dynamic properties are typically stored in
    a dictionary, and this dictionary is typically accessible as
    obj.__dict__.  The rest of this specification is more concerned
    with discovering the names and properties of static attributes
    than with dynamic attributes.

    Examples of dynamic attributes are instance variables of class
    instances, module attributes, etc.  Examples of static attributes
    are the methods of built-in objects like lists and dictionaries,
    and the attributes of frame and code objects (c.co_code,
    c.co_filename, etc.).  When an object with dynamic attributes
    exposes these through its __dict__ attribute, __dict__ is a static
    attribute.

    In the discussion below, I distinguish two kinds of objects:
    regular objects (e.g. lists, ints, functions) and meta-objects.
    Meta-objects are types and classes.  Meta-objects are also regular
    objects, but we're mostly interested in them because they are
    referenced by the __class__ attribute of regular objects (or by
    the __bases__ attribute of meta-objects).

    The class introspection API consists of the following elements:

    - the __class__ and __dict__ attributes on regular objects;

    - the __bases__ and __dict__ attributes on meta-objects;

    - precedence rules;

    - attribute descriptors.

    1. The __dict__ attribute on regular objects

       A regular object may have a __dict__ attribute.  If it does,
       this should be a mapping (not necessarily a dictionary)
       supporting at least __getitem__, keys(), and has_key().  This
       gives the dynamic attributes of the object.  The keys in the
       mapping give attribute names, and the corresponding values give
       their values.

       Typically, the value of an attribute with a given name is the
       same object as the value corresponding to that name as a key in
       the __dict__.  In othe words, obj.__dict__['spam'] is obj.spam.
       (But see the precedence rules below; a static attribute with
       the same name *may* override the dictionary item.)

    2. The __class__ attribute on regular objects

       A regular object may have a __class__ attributes.  If it does,
       this references a meta-object.  A meta-object can define static
       attributes for the regular object whose __class__ it is.

    3. The __dict__ attribute on meta-objects

       A meta-object may have a __dict__ attribute, of the same form
       as the __dict__ attribute for regular objects (mapping, etc).
       If it does, the keys of the meta-object's __dict__ are names of
       static attributes for the corresponding regular object.  The
       values are attribute descriptors; we'll explain these later.
       (An unbound method is a special case of an attribute
       descriptor.)

       Becase a meta-object is also a regular object, the items in a
       meta-object's __dict__ correspond to attributes of the
       meta-object; however, some transformation may be applied, and
       bases (see below) may define additional dynamic attributes.  In
       other words, mobj.spam is not always mobj.__dict__['spam'].
       (This rule contains a loophole because for classes, if
       C.__dict__['spam'] is a function, C.spam is an unbound method
       object.)

    4. The __bases__ attribute on meta-objects

       A meta-object may have a __bases__ attribute.  If it does, this
       should be a sequence (not necessarily a tuple) of other
       meta-objects, the bases.  An absent __bases__ is equivalent to
       an empty sequece of bases.  There must never be a cycle in the
       relationship between meta objects defined by __bases__
       attributes; in other words, the __bases__ attributes define an
       inheritance tree, where the root of the tree is the __class__
       attribute of a regular object, and the leaves of the trees are
       meta-objects without bases.  The __dict__ attributes of the
       meta-objects in the inheritance tree supply attribute
       descriptors for the regular object whose __class__ is at the
       top of the inheritance tree.

    5. Precedence rules

       When two meta-objects in the inheritance tree both define an
       attribute descriptor with the same name, the left-to-right
       depth-first rule applies.  (XXX define rigorously.)

       When a dynamic attribute (one defined in a regular object's
       __dict__) has the same name as a static attribute (one defined
       by a meta-object in the inheritance tree rooted at the regular
       object's __class__), the dynamic attribute *usually* wins, but
       for some attributes the meta-object may specify that the static
       attribute overrides the dynamic attribute.

       (We can't have a simples rule like "static overrides dynamic"
       or "dynamic overrides static", because some static attributes
       indeed override dynamic attributes, e.g. a key '__class__' in
       an instance's __dict__ is ignored in favor of the statically
       defined __class__ pointer, but on the other hand most keys in
       inst.__dict__ override attributes defined in inst.__class__.
       The mechanism whereby a meta-object can specify that a
       particular attribute has precedence is not yet specified.)

    6. Attribute descriptors

       This is where it gets interesting -- and messy.  Attribute
       descriptors (descriptors for short) are stored in the
       meta-object's __dict__, and have two uses: a descriptor can be
       used to get or set the corresponding attribute value on the
       (non-meta) object, and it has an additional interface that
       describes the attribute for documentation or introspection
       purposes.

       There is little prior art in Python for designing the
       descriptor's interface, neither for getting/setting the value
       nor for describing the attribute otherwise, except some trivial
       properties (e.g. it's reasonable to assume that __name__ and
       __doc__ should be the attribute's name and docstring).  I will
       propose such an API below.

       If an object found in the meta-object's __dict__ is not an
       attribute descriptor, backward compatibility dictates
       semantics.  This basically means that if it is a Python
       function or an unbound method, the attribute is a method;
       otherwise, it is the default value for a data attribute.
       Backwards compatibility also dictates that (in the absence of a
       __setattr__ method) it is legal to assign to an attribute of
       type method, and that this creates a data attribute shadowing
       the method for this particular instance.  However, these
       semantics are only required for backwards compatibility with
       regular classes.

    The introspection API is a read-only API.  We don't define the
    effect of assignment to any of the special attributes (__dict__,
    __class__ and __bases__), nor the effect of assignment to the
    items of a __dict__.  Generally, such assignments should be
    considered off-limits.  An extension of this PEP may define some
    semantics for some such assignments.  (Especially because
    currently instances support assignment to __class__ and __dict__,
    and classes support assignment to __bases__ and __dict__.)

Specification of the attribute descriptor API

    Attribute descriptors have the following attributes.  In the
    examples, x is an object, C is x.__class__, x.meth() is a method,
    and x.ivar is a data attribute or instance variable.

    - name: the original attribute name.  Note that because of
      aliasing and renaming, the attribute may be known under a
      different name, but this is the name under which it was born.
      Example: C.meth.name == 'meth'.

    - doc: the attribute's documentation string.

    - objclass: the class that declared this attribute.  The
      descriptor only applies to objects that are instances of this
      class (this includes instances of its subclasses).  Example:
      C.meth.objclass is C.

    - kind: either "method" or "data".  This distinguishes between
      methods and data attributes.  The primary operation on a method
      attribute is to call it.  The primary operations on a data
      attribute are to get and to set it.  Example: C.meth.kind ==
      'method'; C.ivar.kind == 'data'.

    - default: for optional data attributes, this gives a default or
      initial value.  XXX Python has two kinds of semantics for
      referencing "absent" attributes: this may raise an
      AttributeError, or it may produce a default value stored
      somewhere in the class.  There could be a flag that
      distinguishes between these two cases.  Also, there could be a
      flag that tells whether it's OK to delete an attribute (and what
      happens then -- a default value takes its place, or it's truly
      gone).

    - attrclass: for data attributes, this can be the class of the
      attribute value, or None.  If this is not None, the attribute
      value is restricted to being an instance of this class (or of a
      subclass thereof).  If this is None, the attribute value is not
      constrained.  For method attributes, this should normally be
      None (a class is not sufficient information to describe a method
      signature).  If and when optional static typing is added to
      Python, this the meaning of this attribute may change to
      describe the type of the attribute.

    - signature: for methods, an object that describes the signature
      of the method.  Signature objects will be described further
      below.

    - readonly: Boolean indicating whether assignment to this
      attribute is disallowed.  This is usually true for methods.
      Example: C.meth.readonly == 1; C.ivar.kind == 0.

    - get(): a function of one argument that retrieves the attribute
      value from an object.  Examples: C.ivar.get(x) ~~ x.ivar;
      C.meth.get(x) ~~ x.meth.

    - set(): a function of two arguments that sets the attribute value
      on the object.  If readonly is set, this method raises a
      TypeError exception.  Example: C.ivar.set(x, y) ~~ x.ivar = y.

    - call(): for method descriptors, this is a function of at least
      one argument that calls the method.  The first argument is the
      object whose method is called; the remaining arguments
      (including keyword arguments) are passed on to the method.
      Example: C.meth.call(x, 1, 2) ~~ x.meth(1, 2).

    - bind(): for method descriptiors, this is a function of one
      argument that returns a "bound method object".  This in turn can
      be called exactly like the method should be called (in fact this
      is what is returned for a bound method).  This is the same as
      get().  Example: C.meth.bind(x) ~~ x.meth.

    For convenience, __name__ and __doc__ are defined as aliases for
    name and doc.  Also for convenience, calling the descriptor can do
    one of three things:

    - Calling a method descriptor is the same as calling its call()
      method.  Example: C.meth(x, 1, 2) ~~ x.meth(1, 2).

    - Calling a data descriptor with one argument is the same as
      calling its get() method.  Example: C.ivar(x) ~~ x.ivar.

    - Calling a data descriptor with two arguments is the same as
      calling its set() method.  Example: C.ivar(x, y) ~~ x.ivar = y.

    Note that this specification does not define how to create
    specific attribute descriptors.  This is up to the individual
    attribute descriptor implementations, of which there may be many.

Specification of the signature object API

    XXX

Discussion

    XXX

Examples

    XXX

Backwards compatibility

    XXX

Compatibility of C API

    XXX

Warnings and Errors

    XXX

Implementation

    A partial implementation of this PEP is available from CVS as a
    branch named "descr-branch".  To experiment with this
    implementation, proceed to check out Python from CVS according to
    the instructions at http://sourceforge.net/cvs/?group_id=5470 but
    add the arguments "-r descr-branch" to the cvs checkout command.
    (You can also start with an existing checkout and do "cvs update
    -r descr-branch".)  For some examples of the features described
    here, see the file Lib/test/test_descr.py.

    Note: the code in this branch goes beyond this PEP; it is also
    on the way to implementing PEP 253 (Subtyping Built-in Types).

References

    XXX

Copyright

    This document has been placed in the public domain.


PEP 253 -- Subtyping Built-in Types

PEP: | 253
Title: | Subtyping Built-in Types
Version: | $Revision: 1.2 $
Author: | guido@python.org (Guido van Rossum)
Status: | Draft
Type: | Standards Track
Python-Version: | 2.2
Created: | 14-May-2001
Post-History: |

Abstract

    This PEP proposes ways for creating subtypes of existing built-in
    types, either in C or in Python.  The text is currently long and
    rambling; I'll go over it again later to make it shorter.

    Traditionally, types in Python have been created statically, by
    declaring a global variable of type PyTypeObject and initializing
    it with a static initializer.  The fields in the type object
    describe all aspects of a Python object that are relevant to the
    Python interpreter.  A few fields contain dimensional information
    (e.g. the basic allocation size of instances), others contain
    various flags, but most fields are pointers to functions to
    implement various kinds of behaviors.  A NULL pointer means that
    the type does not implement the specific behavior; in that case
    the system may provide a default behavior in that case or raise an
    exception when the behavior is invoked.  Some collections of
    functions pointers that are usually defined together are obtained
    indirectly via a pointer to an additional structure containing.

    While the details of initializing a PyTypeObject structure haven't
    been documented as such, they are easily glanced from the examples
    in the source code, and I am assuming that the reader is
    sufficiently familiar with the traditional way of creating new
    Python types in C.

    This PEP will introduce the following optional features to types:

    - create an instance of a type by calling it

    - create a subtype in C by specifying a base type pointer

    - create a subtype in Python using a class statement

    - multiple inheritance

    This PEP builds on PEP 252, which adds standard introspection to
    types; in particular, types are assumed to have e.g. a __hash__
    method when the type object defines the tp_hash slot.  PEP 252 also
    adds a dictionary to type objects which contains all methods.  At
    the Python level, this dictionary is read-only; at the C level, it
    is accessible directly (but modifying it is not recommended except
    as part of initialization).

Metatypes

    Inevitably the following discussion will come to mention metatypes
    (or metaclasses).  Metatypes are nothing new in Python: Python has
    always been able to talk about the type of a type:

    >>> a = 0
    >>> type(a)
    <type 'int'>
    >>> type(type(a))
    <type 'type'>
    >>> type(type(type(a)))
    <type 'type'>
    >>>

    In this example, type(a) is a "regular" type, and type(type(a)) is
    a metatype.  While as distributed all types have the same metatype
    (which is also its own metatype), this is not a requirement, and
    in fact a useful 3rd party extension (ExtensionClasses by Jim
    Fulton) creates an additional metatype.  A related feature is the
    "Don Beaudry hook", which says that if a metatype is callable, its
    instances (which are regular types) can be subclassed (really
    subtyped) using a Python class statement.  We will use this rule
    to support subtyping of built-in types, and in the process we will
    introduce some additional metatypes, and a "metametatype". (The
    metametatype is nothing unusual; Python's type system allows any
    number of metalevels.)

    Note that Python uses the concept of metatypes or metaclasses in a
    different way than Smalltalk.  In Smalltalk-80, there is a
    hierarchy of metaclasses that mirrors the hierarchy of regular
    classes, metaclasses map 1-1 to classes (except for some funny
    business at the root of the hierarchy), and each class statement
    creates both a regular class and its metaclass, putting class
    methods in the metaclass and instance methods in the regular
    class.

    Nice though this may be in the context of Smalltalk, it's not
    compatible with the traditional use of metatypes in Python, and I
    prefer to continue in the Python way.  This means that Python
    metatypes are typically written in C, and may be shared between
    many regular types. (It will be possible to subtype metatypes in
    Python, so it won't be absolutely necessary to write C in order to
    use metatypes; but the power of Python metatypes will be limited,
    e.g. Python code will never be allowed to allocate raw memory and
    initialize it at will.)

Instantiation by calling the type object

    Traditionally, for each type there is at least one C function that
    creates instances of the type.  This function has to take care of
    both allocating memory for the object and initializing that
    memory.  As of Python 2.0, it also has to interface with the
    garbage collection subsystem, if the type chooses to participate
    in garbage collection (which is optional, but strongly recommended
    for so-called "container" types: types that may contain arbitrary
    references to other objects, and hence may participate in
    reference cycles).

    If we're going to implement subtyping, we must separate allocation
    and initialization: typically, the most derived subtype is in
    charge of allocation (and hence deallocation!), but in most cases
    each base type's initializer (constructor) must still be called,
    from the "most base" type to the most derived type.

    But let's first get the interface for instantiation right.  If we
    call an object, the tp_call slot if its type gets invoked.  Thus,
    if we call a type, this invokes the tp_call slot of the type's
    type: in other words, the tp_call slot of the metatype.
    Traditionally this has been a NULL pointer, meaning that types
    can't be called.  Now we're adding a tp_call slot to the metatype,
    which makes all types "callable" in a trivial sense.  But
    obviously the metatype's tp_call implementation doesn't know how
    to initialize individual types.  So the type defines a new slot,
    tp_construct, which is invoked by the metatype's tp_call slot.  If
    the tp_construct slot is NULL, the metatype's tp_call issues a
    nice error message: the type isn't callable.

    We already know that tp_construct is responsible for initializing
    the object (this will be important for subtyping too).  Who should
    be responsible for allocation of the new object? Either the
    metatype's tp_call can allocate the object, or the type's
    tp_construct can allocate it.  The solution is copied from typical
    C++ implementations: if the metatype's tp_call allocates storage
    for the object it passes the storage as a pointer to the type's
    tp_construct; if the metatype's tp_call does not allocate storage,
    it passes a NULL pointer to the type's tp_call in which case the
    type allocates the storage itself.  This moves the policy decision
    to the metatype, and different metatypes may have different
    policies.  The mechanisms are fixed though: either the metatype's
    tp_call allocates storage, or the type's tp_construct allocates.

    The deallocation mechanism chosen should match the allocation
    mechanism: an allocation policy should prescribe both the
    allocation and deallocation mechanism.  And again, planning ahead
    for subtyping would be nice.  But the available mechanisms are
    different.  The deallocation function has always been part of the
    type structure, as tp_dealloc, which combines the
    "uninitialization" with deallocation.  This was good enough for
    the traditional situation, where it matched the combined
    allocation and initialization of the creation function.  But now
    imagine a type whose creation function uses a special free list
    for allocation.  It's deallocation function puts the object's
    memory back on the same free list.  But when allocation and
    creation are separate, the object may have been allocated from the
    regular heap, and it would be wrong (in some cases disastrous) if
    it were placed on the free list by the deallocation function.

    A solution would be for the tp_construct function to somehow mark
    whether the object was allocated from the special free list, so
    that the tp_dealloc function can choose the right deallocation
    method (assuming that the only two alternatives are a special free
    list or the regular heap).  A variant that doesn't require space
    for an allocation flag bit would be to have two type objects,
    identical in the contents of all their slots except for their
    deallocation slot.  But this requires that all type-checking code
    (e.g. the PyDict_Check()) recognizes both types.  We'll come back
    to this solution in the context of subtyping.  Another alternative
    is to require the metatype's tp_call to leave the allocation to
    the tp_construct method, by passing in a NULL pointer.  But this
    doesn't work once we allow subtyping.

    Eventually, when we add any form of subtyping, we'll have to
    separate deallocation from uninitialization.  The way to do this
    is to add a separate slot to the type object that does the
    uninitialization without the deallocation.  Fortunately, there is
    already such a slot: tp_clear, currently used by the garbage
    collection subsystem.  A simple rule makes this slot reusable as
    an uninitialization: for types that support separate allocation
    and initialization, tp_clear must be defined (even if the object
    doesn't support garbage collection) and it must DECREF all
    contained objects and FREE all other memory areas the object owns.
    It must also be reentrant: it must be possible to clear an already
    cleared object.  The easiest way to do this is to replace all
    pointers DECREFed or FREEd with NULL pointers.

Subtyping in C

    The simplest form of subtyping is subtyping in C.  It is the
    simplest form because we can require the C code to be aware of the
    various problems, and it's acceptable for C code that doesn't
    follow the rules to dump core; while for Python subtyping we would
    need to catch all errors before they become core dumps.

    The idea behind subtyping is very similar to that of single
    inheritance in C++.  A base type is described by a structure
    declaration plus a type object.  A derived type can extend the
    structure (but must leave the names, order and type of the fields
    of the base structure unchanged) and can override certain slots in
    the type object, leaving others the same.

    Not every type can serve as a base type.  The base type must
    support separation of allocation and initialization by having a
    tp_construct slot that can be called with a preallocated object,
    and it must support uninitialization without deallocation by
    having a tp_clear slot as described above.  The derived type must
    also export the structure declaration for its instances through a
    header file, as it is needed in order to derive a subtype.  The
    type object for the base type must also be exported.

    If the base type has a type-checking macro (e.g. PyDict_Check()),
    this macro may be changed to recognize subtypes.  This can be done
    by using the new PyObject_TypeCheck(object, type) macro, which
    calls a function that follows the base class links.  There are
    arguments for and against changing the type-checking macro in this
    way.  The argument for the change should be clear: it allows
    subtypes to be used in places where the base type is required,
    which is often the prime attraction of subtyping (as opposed to
    sharing implementation).  An argument against changing the
    type-checking macro could be that the type check is used
    frequently and a function call would slow things down too much
    (hard to believe); or one could fear that a subtype might break an
    invariant assumed by the support functions of the base type.
    Sometimes it would be wise to change the base type to remove this
    reliance; other times, it would be better to require that derived
    types (implemented in C) maintain the invariants.

    The derived type begins by declaring a type structure which
    contains the base type's structure.  For example, here's the type
    structure for a subtype of the built-in list type:

    typedef struct {
        PyListObject list;
        int state;
    } spamlistobject;

    Note that the base type structure field (here PyListObject) must
    be the first field in the structure; any following fields are
    extension fields.  Also note that the base type is not referenced
    via a pointer; the actual contents of its structure must be
    included! (The goal is for the memory lay out of the beginning of
    the subtype instance to be the same as that of the base type
    instance.)

    Next, the derived type must declare a type object and initialize
    it.  Most of the slots in the type object may be initialized to
    zero, which is a signal that the base type slot must be copied
    into it.  Some fields that must be initialized properly:

    - the object header must be filled in as usual; the type should be
      PyType_Type

    - the tp_basicsize field must be set to the size of the subtype
      instances

    - the tp_base field must be set to the address of the base type's
      type object

    - the tp_dealloc slot function must be a deallocation function for
      the subtype

    - the tp_flags field must be set to the usual Py_TPFLAGS_DEFAULT
      value

    - the tp_name field must be set (otherwise it will be inherited,
      which is wrong)

    Exception: if the subtype defines no additional fields in its
    structure (i.e., it only defines new behavior, no new data), the
    tp_basicsize and the tp_dealloc fields may be set to zero.  In
    order to complete the initialization of the type,
    PyType_InitDict() must be called.  This replaces zero slots in the
    subtype with the value of the corresponding base type slots.  It
    also fills in tp_dict, the type's dictionary; this is more a
    matter of PEP 252.

    The subtype's tp_dealloc slot deserves special attention.  It must
    uninitialize and deallocate the object in an orderly manner: first
    it must uninitialize the fields added by the extension type; then
    it must call the base type's tp_clear function; finally it must
    deallocate the memory of the object.  Usually, the base type's
    tp_clear function has no global name; it is permissible to call it
    via the base type's tp_clear slot, e.g. PyListType.tp_clear(obj).
    Only if it is known that the base type uses the same allocation
    method as the subtype and the subtype requires no uninitialization
    (e.g. it adds no data fields or all its data fields are numbers)
    is it permissible to leave tp_dealloc set to zero in the subtype's
    type object; it will be copied from the base type.

    A subtype is not usable until PyType_InitDict() is called for it;
    this is best done during module initialization, assuming the
    subtype belongs to a module.  An alternative for subtypes added to
    the Python core (which don't live in a particular module) would be
    to initialize the subtype in their constructor function.  It is
    allowed to call PyType_InitDict() more than once, the second and
    further calls have no effect.  In order to avoid unnecessary
    calls, a test for tp_dict==NULL can be made.

    If the subtype itself should be subtypable (usually desirable), it
    should follow the same rules are given above for base types: have
    a tp_construct that accepts a preallocated object and calls the
    base type's tp_construct, and have a tp_clear that calls the base
    type's tp_clear.

Subtyping in Python

    The next step is to allow subtyping of selected built-in types
    through a class statement in Python.  Limiting ourselves to single
    inheritance for now, here is what happens for a simple class
    statement:

    class C(B):
        var1 = 1
        def method1(self): pass
        # etc.

    The body of the class statement is executes in a fresh environment
    (basically, a new dictionary used as local namespace), and then C
    is created.  The following explains how C is created.

    Assume B is a type object.  Since type objects are objects, and
    every object has a type, B has a type.  B's type is accessible via
    type(B) or B.__class__ (the latter notation is new for types; it
    is introduced in PEP 252).  Let's say B's type is M (for
    Metatype).  The class statement will create a new type, C.  Since
    C will be a type object just like B, we view the creation of C as
    an instantiation of the metatype, M.  The information that needs
    to be provided for the creation of C is: its name (in this example
    the string "C"); the list of base classes (a singleton tuple
    containing B); and the results of executing the class body, in the
    form of a dictionary (e.g. {"var1": 1, "method1": <function...>,
    ...}).

    According to the Don Beaudry hook, the following call is made:

    C = M("C", (B,), dict)

    (where dict is the dictionary resulting from execution of the
    class body).  In other words, the metatype (M) is called.  Note
    that even though we currently require there to be exactly one base
    class, we still pass in a (singleton) sequence of base classes;
    this makes it possible to support multiple inheritance later (or
    for types with a different metaclass!) without changing this
    interface.

    Note that calling M requires that M itself has a type: the
    meta-metatype.  In the current implementation, I have introduced a
    new type object for this purpose, named turtle because of my
    fondness of the phrase "turtles all the way down".  However I now
    believe that it would be better if M were its own metatype, just
    like before.  This can be accomplished by making M's tp_call slot
    slightly more flexible.

    In any case, the work for creating C is done by M's tp_construct
    slot.  It allocates space for an "extended" type structure, which
    contains space for: the type object; the auxiliary structures
    (as_sequence etc.); the string object containing the type name (to
    ensure that this object isn't deallocated while the type object is
    still referencing it); and some more auxiliary storage (to be
    described later).  It initializes this storage to zeros except for
    a few crucial slots (e.g. tp_name is set to point to the type
    name) and then sets the tp_base slot to point to B.  Then
    PyType_InitDict() is called to inherit B's slots.  Finally, C's
    tp_dict slot is updated with the contents of the namespace
    dictionary (the third argument to the call to M).

Copyright

    This document has been placed in the public domain.


Table Of Contents

  • Index of Python Enhancement Proposals (PEPs)
  • PEP Purpose and Guidelines
  • Procedure for Adding New Modules
  • Guidelines for Handling Bug Reports
  • Deprecation of Standard Modules
  • Guidelines for Language Evolution
  • Bug Fix Releases
  • Feature Requests
  • Python 1.6 Release Schedule
  • Python 2.0 Release Schedule
  • Lockstep Iteration
  • List Comprehensions
  • Augmented Assignments
  • Range Literals
  • Weak References
  • 2.0 Batteries Included
  • Rich Comparisions
  • Reworking the Coercion Model
  • Multi-dimensional Arrays
  • Decoupling the Interpreter Loop
  • Adding New Linear Algebra Operators to Python
  • Loop Counter Iteration
  • Attribute Access Handlers
  • Extended Print Statement
  • String Interpolation
  • Docstring Format
  • Display Hook for Interactive Use
  • Adding a Built-In Set Object Type
  • Stackless Python
  • Coroutines, Generators, Continuations
  • Import As
  • Web Library Enhancements
  • Change the Meaning of \x Escapes
  • Attribute Docstrings
  • Elementwise/Objectwise Operators
  • Python 2.1 Release Schedule
  • Statically Nested Scopes
  • Reworking Python's Numeric Model
  • Using Distutils to Build Python
  • Warning Framework
  • __findattr__()
  • Function Attributes
  • Python Online Help
  • Iterators
  • Import on Case-Insensitive Platforms
  • Back to the __future__
  • Unifying Long Integers and Integers
  • Non-integer Division
  • Adding a Rational Type to Python
  • Adding a Rational Literal to Python
  • Metadata for Python Software Packages
  • Numeric Kinds
  • Module Repository Upload Mechanism
  • The `directive' statement
  • Python Interface Syntax
  • Object Adaptation
  • API for Cryptographic Hash Functions
  • Python Database API Specification v1.0
  • Python Database API Specification v2.0
  • Using site-packages on All Platforms
  • Python 2.2 Release Schedule
  • Making Types Look More Like Classes
  • Subtyping Built-in Types