PEP 263 -- Defining Python Source Code Encodings
Contents AbstractThis PEP proposes to introduce a syntax to declare the encoding of a Python source file. The encoding information is then used by the Python parser to interpret the file using the given encoding. Most notably this enhances the interpretation of Unicode literals in the source code and makes it possible to write Unicode literals using e.g. UTF-8 directly in an Unicode aware editor. ProblemIn Python 2.1, Unicode literals can only be written using the Latin-1 based encoding "unicode-escape". This makes the programming environment rather unfriendly to Python users who live and work in non-Latin-1 locales such as many of the Asian countries. Programmers can write their 8-bit strings using the favorite encoding, but are bound to the "unicode-escape" encoding for Unicode literals. Proposed SolutionI propose to make the Python source code encoding both visible and changeable on a per-source file basis by using a special comment at the top of the file to declare the encoding. To make Python aware of this encoding declaration a number of concept changes are necessary with respect to the handling of Python source code data. Defining the EncodingPython will default to ASCII as standard encoding if no other encoding hints are given. To define a source code encoding, a magic comment must be placed into the source files either as first or second line in the file, such as: # coding=<encoding name> or (using formats recognized by popular editors): #!/usr/bin/python # -*- coding: <encoding name> -*- or: #!/usr/bin/python # vim: set fileencoding=<encoding name> : More precisely, the first or second line must match the following regular expression: ^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+) The first group of this expression is then interpreted as encoding name. If the encoding is unknown to Python, an error is raised during compilation. There must not be any Python statement on the line that contains the encoding declaration. If the first line matches the second line is ignored. To aid with platforms such as Windows, which add Unicode BOM marks to the beginning of Unicode files, the UTF-8 signature \xef\xbb\xbf will be interpreted as 'utf-8' encoding as well (even if no magic encoding comment is given). If a source file uses both the UTF-8 BOM mark signature and a magic encoding comment, the only allowed encoding for the comment is 'utf-8'. Any other encoding will cause an error. ExamplesThese are some examples to clarify the different styles for defining the source code encoding at the top of a Python source file:
ConceptsThe PEP is based on the following concepts which would have to be implemented to enable usage of such a magic comment:
Note that Python identifiers are restricted to the ASCII subset of the encoding, and thus need no further conversion after step 4. ImplementationFor backwards-compatibility with existing code which currently uses non-ASCII in string literals without declaring an encoding, the implementation will be introduced in two phases:
The builtin compile() API will be enhanced to accept Unicode as input. 8-bit string input is subject to the standard procedure for encoding detection as described above. If a Unicode string with a coding declaration is passed to compile(), a SyntaxError will be raised. SUZUKI Hisao is working on a patch; see [2] for details. A patch implementing only phase 1 is available at [1]. PhasesImplementation of steps 1 and 2 above were completed in 2.3, except for changing the default encoding to "ascii". The default encoding was set to "ascii" in version 2.5. ScopeThis PEP intends to provide an upgrade path from the current (more-or-less) undefined source code encoding situation to a more robust and portable definition. References
History
CopyrightThis document has been placed in the public domain. |
|