ꦫꦣꦺꦤ꧀ꦄꦤ꧀ꦠꦱꦺꦤ

PATH: usr / share / doc / python-docs-2.7.5 / html / _sources / library

:mod:`tokenize` --- Tokenizer for Python source
===============================================

.. module:: tokenize
   :synopsis: Lexical scanner for Python source code.
.. moduleauthor:: Ka Ping Yee
.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>

**Source code:** :source:`Lib/tokenize.py`

--------------

The :mod:`tokenize` module provides a lexical scanner for Python source code,
implemented in Python.  The scanner in this module returns comments as tokens as
well, making it useful for implementing "pretty-printers," including colorizers
for on-screen displays.

To simplify token stream handling, all :ref:`operators` and :ref:`delimiters`
tokens are returned using the generic :data:`token.OP` token type.  The exact
type can be determined by checking the token ``string`` field on the
:term:`named tuple` returned from :func:`tokenize.tokenize` for the character
sequence that identifies a specific operator token.

The primary entry point is a :term:`generator`:

.. function:: generate_tokens(readline)

The :func:`generate_tokens` generator requires one argument, *readline*,
   which must be a callable object which provides the same interface as the
   :meth:`readline` method of built-in file objects (see section
   :ref:`bltin-file-objects`).  Each call to the function should return one line
   of input as a string. Alternately, *readline* may be a callable object that
   signals completion by raising :exc:`StopIteration`.

The generator produces 5-tuples with these members: the token type; the token
   string; a 2-tuple ``(srow, scol)`` of ints specifying the row and column
   where the token begins in the source; a 2-tuple ``(erow, ecol)`` of ints
   specifying the row and column where the token ends in the source; and the
   line on which the token was found.  The line passed (the last tuple item) is
   the *logical* line; continuation lines are included.

.. versionadded:: 2.2

An older entry point is retained for backward compatibility:

.. function:: tokenize(readline[, tokeneater])

The :func:`tokenize` function accepts two parameters: one representing the input
   stream, and one providing an output mechanism for :func:`tokenize`.

The first parameter, *readline*, must be a callable object which provides the
   same interface as the :meth:`readline` method of built-in file objects (see
   section :ref:`bltin-file-objects`).  Each call to the function should return one
   line of input as a string. Alternately, *readline* may be a callable object that
   signals completion by raising :exc:`StopIteration`.

.. versionchanged:: 2.5
      Added :exc:`StopIteration` support.

The second parameter, *tokeneater*, must also be a callable object.  It is
   called once for each token, with five arguments, corresponding to the tuples
   generated by :func:`generate_tokens`.

All constants from the :mod:`token` module are also exported from
:mod:`tokenize`, as are two additional token type values that might be passed to
the *tokeneater* function by :func:`tokenize`:

.. data:: COMMENT

Token value used to indicate a comment.

.. data:: NL

Token value used to indicate a non-terminating newline.  The NEWLINE token
   indicates the end of a logical line of Python code; NL tokens are generated when
   a logical line of code is continued over multiple physical lines.

Another function is provided to reverse the tokenization process. This is useful
for creating tools that tokenize a script, modify the token stream, and write
back the modified script.

.. function:: untokenize(iterable)

Converts tokens back into Python source code.  The *iterable* must return
   sequences with at least two elements, the token type and the token string.  Any
   additional sequence elements are ignored.

The reconstructed script is returned as a single string.  The result is
   guaranteed to tokenize back to match the input so that the conversion is
   lossless and round-trips are assured.  The guarantee applies only to the token
   type and token string as the spacing between tokens (column positions) may
   change.

.. versionadded:: 2.5

Example of a script re-writer that transforms float literals into Decimal
objects::

def decistmt(s):
       """Substitute Decimals for floats in a string of statements.

>>> from decimal import Decimal
       >>> s = 'print +21.3e-5*-.1234/81.7'
       >>> decistmt(s)
       "print +Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7')"

>>> exec(s)
       -3.21716034272e-007
       >>> exec(decistmt(s))
       -3.217160342717258261933904529E-7

"""
       result = []
       g = generate_tokens(StringIO(s).readline)   # tokenize the string
       for toknum, tokval, _, _, _  in g:
           if toknum == NUMBER and '.' in tokval:  # replace NUMBER tokens
               result.extend([
                   (NAME, 'Decimal'),
                   (OP, '('),
                   (STRING, repr(tokval)),
                   (OP, ')')
               ])
           else:
               result.append((toknum, tokval))
       return untokenize(result)

[+] ..
[-] fpformat.txt [edit]
[-] string.txt [edit]
[-] aetypes.txt [edit]
[-] textwrap.txt [edit]
[-] codecs.txt [edit]
[-] array.txt [edit]
[-] urllib2.txt [edit]
[-] pickle.txt [edit]
[-] 2to3.txt [edit]
[-] ossaudiodev.txt [edit]
[-] markup.txt [edit]
[-] sha.txt [edit]
[-] stat.txt [edit]
[-] itertools.txt [edit]
[-] numeric.txt [edit]
[-] gzip.txt [edit]
[-] pkgutil.txt [edit]
[-] subprocess.txt [edit]
[-] token.txt [edit]
[-] debug.txt [edit]
[-] dummy_thread.txt [edit]
[-] mimify.txt [edit]
[-] bastion.txt [edit]
[-] popen2.txt [edit]
[-] autogil.txt [edit]
[-] importlib.txt [edit]
[-] io.txt [edit]
[-] msilib.txt [edit]
[-] development.txt [edit]
[-] hotshot.txt [edit]
[-] constants.txt [edit]
[-] curses.ascii.txt [edit]
[-] os.txt [edit]
[-] easydialogs.txt [edit]
[-] mutex.txt [edit]
[-] md5.txt [edit]
[-] stdtypes.txt [edit]
[-] bz2.txt [edit]
[-] imgfile.txt [edit]
[-] aetools.txt [edit]
[-] internet.txt [edit]
[-] xml.dom.pulldom.txt [edit]
[-] cookie.txt [edit]
[-] sunau.txt [edit]
[-] archiving.txt [edit]
[-] modules.txt [edit]
[-] base64.txt [edit]
[-] ttk.txt [edit]
[-] tty.txt [edit]
[-] cookielib.txt [edit]
[-] pydoc.txt [edit]
[-] misc.txt [edit]
[-] fm.txt [edit]
[-] whichdb.txt [edit]
[-] imghdr.txt [edit]
[-] datatypes.txt [edit]
[-] dis.txt [edit]
[-] datetime.txt [edit]
[-] userdict.txt [edit]
[-] py_compile.txt [edit]
[-] imputil.txt [edit]
[-] xml.dom.minidom.txt [edit]
[-] multiprocessing.txt [edit]
[-] functools.txt [edit]
[-] audioop.txt [edit]
[-] language.txt [edit]
[-] miniaeframe.txt [edit]
[-] ssl.txt [edit]
[-] copy_reg.txt [edit]
[-] heapq.txt [edit]
[-] syslog.txt [edit]
[-] fileformats.txt [edit]
[-] simplexmlrpcserver.txt [edit]
[-] numbers.txt [edit]
[-] pipes.txt [edit]
[-] shutil.txt [edit]
[-] zlib.txt [edit]
[-] optparse.txt [edit]
[-] allos.txt [edit]
[-] custominterp.txt [edit]
[-] windows.txt [edit]
[-] dbm.txt [edit]
[-] select.txt [edit]
[-] time.txt [edit]
[-] index.txt [edit]
[-] undoc.txt [edit]
[-] plistlib.txt [edit]
[-] htmllib.txt [edit]
[-] warnings.txt [edit]
[-] functions.txt [edit]
[-] argparse.txt [edit]
[-] distutils.txt [edit]
[-] macpath.txt [edit]
[-] fnmatch.txt [edit]
[-] tabnanny.txt [edit]
[-] calendar.txt [edit]
[-] jpeg.txt [edit]
[-] zipimport.txt [edit]
[-] unix.txt [edit]
[-] chunk.txt [edit]
[-] smtplib.txt [edit]
[-] tokenize.txt [edit]
[-] asynchat.txt [edit]
[-] decimal.txt [edit]
[-] cd.txt [edit]
[-] pickletools.txt [edit]
[-] statvfs.txt [edit]
[-] gettext.txt [edit]
[-] sqlite3.txt [edit]
[-] imageop.txt [edit]
[-] dl.txt [edit]
[-] logging.handlers.txt [edit]
[-] zipfile.txt [edit]
[-] copy.txt [edit]
[-] sched.txt [edit]
[-] nis.txt [edit]
[-] timeit.txt [edit]
[-] weakref.txt [edit]
[-] mailbox.txt [edit]
[-] othergui.txt [edit]
[-] sgmllib.txt [edit]
[-] mimewriter.txt [edit]
[-] email.charset.txt [edit]
[-] mmap.txt [edit]
[-] xmlrpclib.txt [edit]
[-] robotparser.txt [edit]
[-] xml.sax.reader.txt [edit]
[-] hmac.txt [edit]
[-] collections.txt [edit]
[-] mimetools.txt [edit]
[-] aifc.txt [edit]
[-] modulefinder.txt [edit]
[-] socketserver.txt [edit]
[-] grp.txt [edit]
[-] getopt.txt [edit]
[-] __future__.txt [edit]
[-] binascii.txt [edit]
[-] symtable.txt [edit]
[-] crypt.txt [edit]
[-] email.encoders.txt [edit]
[-] imaplib.txt [edit]
[-] spwd.txt [edit]
[-] wsgiref.txt [edit]
[-] errno.txt [edit]
[-] email-examples.txt [edit]
[-] sun.txt [edit]
[-] doctest.txt [edit]
[-] resource.txt [edit]
[-] asyncore.txt [edit]
[-] curses.panel.txt [edit]
[-] imp.txt [edit]
[-] htmlparser.txt [edit]
[-] thread.txt [edit]
[-] posix.txt [edit]
[-] repr.txt [edit]
[-] macostools.txt [edit]
[-] colorsys.txt [edit]
[-] bdb.txt [edit]
[-] stringio.txt [edit]
[-] mm.txt [edit]
[-] cgi.txt [edit]
[-] math.txt [edit]
[-] httplib.txt [edit]
[-] urlparse.txt [edit]
[-] aepack.txt [edit]
[-] colorpicker.txt [edit]
[-] binhex.txt [edit]
[-] tk.txt [edit]
[-] xml.txt [edit]
[-] webbrowser.txt [edit]
[-] contextlib.txt [edit]
[-] types.txt [edit]
[-] shelve.txt [edit]
[-] platform.txt [edit]
[-] readline.txt [edit]
[-] locale.txt [edit]
[-] sets.txt [edit]
[-] macosa.txt [edit]
[-] bisect.txt [edit]
[-] email.parser.txt [edit]
[-] xml.sax.txt [edit]
[-] xml.etree.elementtree.txt [edit]
[-] netrc.txt [edit]
[-] new.txt [edit]
[-] bsddb.txt [edit]
[-] fl.txt [edit]
[-] curses.txt [edit]
[-] hashlib.txt [edit]
[-] tarfile.txt [edit]
[-] ftplib.txt [edit]
[-] telnetlib.txt [edit]
[-] macos.txt [edit]
[-] multifile.txt [edit]
[-] dbhash.txt [edit]
[-] netdata.txt [edit]
[-] fileinput.txt [edit]
[-] python.txt [edit]
[-] mac.txt [edit]
[-] filesys.txt [edit]
[-] scrolledtext.txt [edit]
[-] compiler.txt [edit]
[-] tkinter.txt [edit]
[-] random.txt [edit]
[-] mailcap.txt [edit]
[-] fpectl.txt [edit]
[-] fcntl.txt [edit]
[-] ipc.txt [edit]
[-] formatter.txt [edit]
[-] profile.txt [edit]
[-] xml.dom.txt [edit]
[-] email.iterators.txt [edit]
[-] logging.config.txt [edit]
[-] rlcompleter.txt [edit]
[-] idle.txt [edit]
[-] gdbm.txt [edit]
[-] ast.txt [edit]
[-] someos.txt [edit]
[-] cmd.txt [edit]
[-] re.txt [edit]
[-] docxmlrpcserver.txt [edit]
[-] pdb.txt [edit]
[-] rexec.txt [edit]
[-] commands.txt [edit]
[-] restricted.txt [edit]
[-] crypto.txt [edit]
[-] dumbdbm.txt [edit]
[-] email.txt [edit]
[-] linecache.txt [edit]
[-] os.path.txt [edit]
[-] cgitb.txt [edit]
[-] _winreg.txt [edit]
[-] carbon.txt [edit]
[-] turtle.txt [edit]
[-] xml.sax.handler.txt [edit]
[-] json.txt [edit]
[-] email.generator.txt [edit]
[-] struct.txt [edit]
[-] mhlib.txt [edit]
[-] uuid.txt [edit]
[-] threading.txt [edit]
[-] mimetypes.txt [edit]
[-] traceback.txt [edit]
[-] filecmp.txt [edit]
[-] sndhdr.txt [edit]
[-] logging.txt [edit]
[-] getpass.txt [edit]
[-] dummy_threading.txt [edit]
[-] abc.txt [edit]
[-] shlex.txt [edit]
[-] email.mime.txt [edit]
[-] fractions.txt [edit]
[-] email.header.txt [edit]
[-] marshal.txt [edit]
[-] symbol.txt [edit]
[-] wave.txt [edit]
[-] urllib.txt [edit]
[-] queue.txt [edit]
[-] operator.txt [edit]
[-] compileall.txt [edit]
[-] sysconfig.txt [edit]
[-] quopri.txt [edit]
[-] inspect.txt [edit]
[-] xdrlib.txt [edit]
[-] __builtin__.txt [edit]
[-] ic.txt [edit]
[-] signal.txt [edit]
[-] dircache.txt [edit]
[-] rfc822.txt [edit]
[-] glob.txt [edit]
[-] stringprep.txt [edit]
[-] tix.txt [edit]
[-] i18n.txt [edit]
[-] email.util.txt [edit]
[-] anydbm.txt [edit]
[-] code.txt [edit]
[-] __main__.txt [edit]
[-] difflib.txt [edit]
[-] unicodedata.txt [edit]
[-] sunaudio.txt [edit]
[-] msvcrt.txt [edit]
[-] site.txt [edit]
[-] codeop.txt [edit]
[-] uu.txt [edit]
[-] winsound.txt [edit]
[-] smtpd.txt [edit]
[-] al.txt [edit]
[-] user.txt [edit]
[-] poplib.txt [edit]
[-] nntplib.txt [edit]
[-] keyword.txt [edit]
[-] ctypes.txt [edit]
[-] socket.txt [edit]
[-] simplehttpserver.txt [edit]
[-] sys.txt [edit]
[-] pyexpat.txt [edit]
[-] configparser.txt [edit]
[-] persistence.txt [edit]
[-] trace.txt [edit]
[-] runpy.txt [edit]
[-] csv.txt [edit]
[-] test.txt [edit]
[-] framework.txt [edit]
[-] gc.txt [edit]
[-] cmath.txt [edit]
[-] exceptions.txt [edit]
[-] pyclbr.txt [edit]
[-] sgi.txt [edit]
[-] parser.txt [edit]
[-] future_builtins.txt [edit]
[-] pwd.txt [edit]
[-] strings.txt [edit]
[-] frameworks.txt [edit]
[-] email.message.txt [edit]
[-] cgihttpserver.txt [edit]
[-] intro.txt [edit]
[-] email.errors.txt [edit]
[-] pprint.txt [edit]
[-] pty.txt [edit]
[-] xml.sax.utils.txt [edit]
[-] posixfile.txt [edit]
[-] tempfile.txt [edit]
[-] termios.txt [edit]
[-] gl.txt [edit]
[-] gensuitemodule.txt [edit]
[-] atexit.txt [edit]
[-] basehttpserver.txt [edit]
[-] unittest.txt [edit]