pip._vendor.webencodings Namespace Reference

Namespaces
namespace	labels
namespace	mklabels
namespace	tests
namespace	x_user_defined

Data Structures
class	Encoding
class	IncrementalDecoder
class	IncrementalEncoder

Functions
	ascii_lower (string)
	lookup (label)
	_get_encoding (encoding_or_label)
	decode (input, fallback_encoding, errors='replace')
	_detect_bom (input)
	encode (input, encoding=UTF8, errors='strict')
	iter_decode (input, fallback_encoding, errors='replace')
	_iter_decode_generator (input, decoder)
	iter_encode (input, encoding=UTF8, errors='strict')
	_iter_encode_generator (input, encode)

Variables
str	VERSION = '0.5.1'
dict	PYTHON_NAMES
dict	CACHE = {}
	UTF8 = lookup('utf-8')
	_UTF16LE = lookup('utf-16le')
	_UTF16BE = lookup('utf-16be')

Detailed Description

    webencodings
    ~~~~~~~~~~~~

    This is a Python implementation of the `WHATWG Encoding standard
    <http://encoding.spec.whatwg.org/>`. See README for details.

    :copyright: Copyright 2012 by Simon Sapin
    :license: BSD, see LICENSE for details.

Function Documentation

_detect_bom()

_detect_bom (   input )

protected

Return (bom_encoding, input), with any BOM removed from the input.

Definition at line 161 of file __init__.py.

def _detect_bom(input):
    """Return (bom_encoding, input), with any BOM removed from the input."""
    if input.startswith(b'\xFF\xFE'):
        return _UTF16LE, input[2:]
    if input.startswith(b'\xFE\xFF'):
        return _UTF16BE, input[2:]
    if input.startswith(b'\xEF\xBB\xBF'):
        return UTF8, input[3:]
    return None, input
 
 

References i.

Referenced by pip._vendor.webencodings.decode(), and IncrementalDecoder.decode().

Here is the caller graph for this function:

_get_encoding()

_get_encoding (   encoding_or_label )

protected

Accept either an encoding object or label.

:param encoding: An :class:`Encoding` object or a label string.
:returns: An :class:`Encoding` object.
:raises: :exc:`~exceptions.LookupError` for an unknown label.

Definition at line 91 of file __init__.py.

def _get_encoding(encoding_or_label):
    """
    Accept either an encoding object or label.
 
    :param encoding: An :class:`Encoding` object or a label string.
    :returns: An :class:`Encoding` object.
    :raises: :exc:`~exceptions.LookupError` for an unknown label.
 
    """
    if hasattr(encoding_or_label, 'codec_info'):
        return encoding_or_label
 
    encoding = lookup(encoding_or_label)
    if encoding is None:
        raise LookupError('Unknown encoding label: %r' % encoding_or_label)
    return encoding
 
 

References i.

Referenced by IncrementalEncoder.__init__(), pip._vendor.webencodings.decode(), and pip._vendor.webencodings.encode().

Here is the caller graph for this function:

_iter_decode_generator()

_iter_decode_generator (   input,   decoder  )

protected

Return a generator that first yields the :obj:`Encoding`,
then yields output chukns as Unicode strings.

Definition at line 214 of file __init__.py.

def _iter_decode_generator(input, decoder):
    """Return a generator that first yields the :obj:`Encoding`,
    then yields output chukns as Unicode strings.
 
    """
    decode = decoder.decode
    input = iter(input)
    for chunck in input:
        output = decode(chunck)
        if output:
            assert decoder.encoding is not None
            yield decoder.encoding
            yield output
            break
    else:
        # Input exhausted without determining the encoding
        output = decode(b'', final=True)
        assert decoder.encoding is not None
        yield decoder.encoding
        if output:
            yield output
        return
 
    for chunck in input:
        output = decode(chunck)
        if output:
            yield output
    output = decode(b'', final=True)
    if output:
        yield output
 
 

References i.

Referenced by pip._vendor.webencodings.iter_decode().

Here is the caller graph for this function:

_iter_encode_generator()

_iter_encode_generator (   input,   encode  )

protected

Definition at line 262 of file __init__.py.

def _iter_encode_generator(input, encode):
    for chunck in input:
        output = encode(chunck)
        if output:
            yield output
    output = encode('', final=True)
    if output:
        yield output
 
 

Referenced by pip._vendor.webencodings.iter_encode().

Here is the caller graph for this function:

ascii_lower()

ascii_lower

(

string

)

Transform (only) ASCII letters to lower case: A-Z is mapped to a-z.

    :param string: An Unicode string.
    :returns: A new Unicode string.

    This is used for `ASCII case-insensitive
    <http://encoding.spec.whatwg.org/#ascii-case-insensitive>`_
    matching of encoding labels.
    The same matching is also used, among other things,
    for `CSS keywords <http://dev.w3.org/csswg/css-values/#keywords>`_.

    This is different from the :meth:`~py:str.lower` method of Unicode strings
    which also affect non-ASCII characters,
    sometimes mapping them into the ASCII range:

        >>> keyword = u'Bac\N{KELVIN SIGN}ground'
        >>> assert keyword.lower() == u'background'
        >>> assert ascii_lower(keyword) != keyword.lower()
        >>> assert ascii_lower(keyword) == u'bac\N{KELVIN SIGN}ground'

Definition at line 35 of file __init__.py.

def ascii_lower(string):
    r"""Transform (only) ASCII letters to lower case: A-Z is mapped to a-z.
 
    :param string: An Unicode string.
    :returns: A new Unicode string.
 
    This is used for `ASCII case-insensitive
    <http://encoding.spec.whatwg.org/#ascii-case-insensitive>`_
    matching of encoding labels.
    The same matching is also used, among other things,
    for `CSS keywords <http://dev.w3.org/csswg/css-values/#keywords>`_.
 
    This is different from the :meth:`~py:str.lower` method of Unicode strings
    which also affect non-ASCII characters,
    sometimes mapping them into the ASCII range:
 
        >>> keyword = u'Bac\N{KELVIN SIGN}ground'
        >>> assert keyword.lower() == u'background'
        >>> assert ascii_lower(keyword) != keyword.lower()
        >>> assert ascii_lower(keyword) == u'bac\N{KELVIN SIGN}ground'
 
    """
    # This turns out to be faster than unicode.translate()
    return string.encode('utf8').lower().decode('utf8')
 
 

References i.

Referenced by pip._vendor.webencodings.lookup().

Here is the caller graph for this function:

decode()

decode	(		input,
			fallback_encoding,
			errors = 'replace'
	)

Decode a single string.

:param input: A byte string
:param fallback_encoding:
    An :class:`Encoding` object or a label string.
    The encoding to use if :obj:`input` does note have a BOM.
:param errors: Type of error handling. See :func:`codecs.register`.
:raises: :exc:`~exceptions.LookupError` for an unknown encoding label.
:return:
    A ``(output, encoding)`` tuple of an Unicode string
    and an :obj:`Encoding`.

Definition at line 139 of file __init__.py.

def decode(input, fallback_encoding, errors='replace'):
    """
    Decode a single string.
 
    :param input: A byte string
    :param fallback_encoding:
        An :class:`Encoding` object or a label string.
        The encoding to use if :obj:`input` does note have a BOM.
    :param errors: Type of error handling. See :func:`codecs.register`.
    :raises: :exc:`~exceptions.LookupError` for an unknown encoding label.
    :return:
        A ``(output, encoding)`` tuple of an Unicode string
        and an :obj:`Encoding`.
 
    """
    # Fail early if `encoding` is an invalid label.
    fallback_encoding = _get_encoding(fallback_encoding)
    bom_encoding, input = _detect_bom(input)
    encoding = bom_encoding or fallback_encoding
    return encoding.codec_info.decode(input, errors)[0], encoding
 
 

References pip._vendor.webencodings._detect_bom(), pip._vendor.webencodings._get_encoding(), and i.

Here is the call graph for this function:

encode()

encode	(		input,
			encoding = UTF8,
			errors = 'strict'
	)

Encode a single string.

:param input: An Unicode string.
:param encoding: An :class:`Encoding` object or a label string.
:param errors: Type of error handling. See :func:`codecs.register`.
:raises: :exc:`~exceptions.LookupError` for an unknown encoding label.
:return: A byte string.

Definition at line 172 of file __init__.py.

def encode(input, encoding=UTF8, errors='strict'):
    """
    Encode a single string.
 
    :param input: An Unicode string.
    :param encoding: An :class:`Encoding` object or a label string.
    :param errors: Type of error handling. See :func:`codecs.register`.
    :raises: :exc:`~exceptions.LookupError` for an unknown encoding label.
    :return: A byte string.
 
    """
    return _get_encoding(encoding).codec_info.encode(input, errors)[0]
 
 

References pip._vendor.webencodings._get_encoding(), and i.

Here is the call graph for this function:

iter_decode()

iter_decode	(		input,
			fallback_encoding,
			errors = 'replace'
	)

"Pull"-based decoder.

:param input:
    An iterable of byte strings.

    The input is first consumed just enough to determine the encoding
    based on the precense of a BOM,
    then consumed on demand when the return value is.
:param fallback_encoding:
    An :class:`Encoding` object or a label string.
    The encoding to use if :obj:`input` does note have a BOM.
:param errors: Type of error handling. See :func:`codecs.register`.
:raises: :exc:`~exceptions.LookupError` for an unknown encoding label.
:returns:
    An ``(output, encoding)`` tuple.
    :obj:`output` is an iterable of Unicode strings,
    :obj:`encoding` is the :obj:`Encoding` that is being used.

Definition at line 186 of file __init__.py.

def iter_decode(input, fallback_encoding, errors='replace'):
    """
    "Pull"-based decoder.
 
    :param input:
        An iterable of byte strings.
 
        The input is first consumed just enough to determine the encoding
        based on the precense of a BOM,
        then consumed on demand when the return value is.
    :param fallback_encoding:
        An :class:`Encoding` object or a label string.
        The encoding to use if :obj:`input` does note have a BOM.
    :param errors: Type of error handling. See :func:`codecs.register`.
    :raises: :exc:`~exceptions.LookupError` for an unknown encoding label.
    :returns:
        An ``(output, encoding)`` tuple.
        :obj:`output` is an iterable of Unicode strings,
        :obj:`encoding` is the :obj:`Encoding` that is being used.
 
    """
 
    decoder = IncrementalDecoder(fallback_encoding, errors)
    generator = _iter_decode_generator(input, decoder)
    encoding = next(generator)
    return generator, encoding
 
 

References pip._vendor.webencodings._iter_decode_generator().

Here is the call graph for this function:

iter_encode()

iter_encode	(		input,
			encoding = UTF8,
			errors = 'strict'
	)

“Pull”-based encoder.

:param input: An iterable of Unicode strings.
:param encoding: An :class:`Encoding` object or a label string.
:param errors: Type of error handling. See :func:`codecs.register`.
:raises: :exc:`~exceptions.LookupError` for an unknown encoding label.
:returns: An iterable of byte strings.

Definition at line 246 of file __init__.py.

def iter_encode(input, encoding=UTF8, errors='strict'):
    """
    “Pull”-based encoder.
 
    :param input: An iterable of Unicode strings.
    :param encoding: An :class:`Encoding` object or a label string.
    :param errors: Type of error handling. See :func:`codecs.register`.
    :raises: :exc:`~exceptions.LookupError` for an unknown encoding label.
    :returns: An iterable of byte strings.
 
    """
    # Fail early if `encoding` is an invalid label.
    encode = IncrementalEncoder(encoding, errors).encode
    return _iter_encode_generator(input, encode)
 
 

References pip._vendor.webencodings._iter_encode_generator().

Here is the call graph for this function:

lookup()

lookup

(

label

)

Look for an encoding by its label.
This is the spec’s `get an encoding
<http://encoding.spec.whatwg.org/#concept-encoding-get>`_ algorithm.
Supported labels are listed there.

:param label: A string.
:returns:
    An :class:`Encoding` object, or :obj:`None` for an unknown label.

Definition at line 61 of file __init__.py.

def lookup(label):
    """
    Look for an encoding by its label.
    This is the spec’s `get an encoding
    <http://encoding.spec.whatwg.org/#concept-encoding-get>`_ algorithm.
    Supported labels are listed there.
 
    :param label: A string.
    :returns:
        An :class:`Encoding` object, or :obj:`None` for an unknown label.
 
    """
    # Only strip ASCII whitespace: U+0009, U+000A, U+000C, U+000D, and U+0020.
    label = ascii_lower(label.strip('\t\n\f\r '))
    name = LABELS.get(label)
    if name is None:
        return None
    encoding = CACHE.get(name)
    if encoding is None:
        if name == 'x-user-defined':
            from .x_user_defined import codec_info
        else:
            python_name = PYTHON_NAMES.get(name, name)
            # Any python_name value that gets to here should be valid.
            codec_info = codecs.lookup(python_name)
        encoding = Encoding(name, codec_info)
        CACHE[name] = encoding
    return encoding
 
 

References pip._vendor.webencodings.ascii_lower(), and i.

Here is the call graph for this function:

Variable Documentation

_UTF16BE

_UTF16BE = lookup('utf-16be')

protected

Definition at line 136 of file __init__.py.

_UTF16LE

_UTF16LE = lookup('utf-16le')

protected

Definition at line 135 of file __init__.py.

CACHE

dict CACHE = {}

Definition at line 32 of file __init__.py.

PYTHON_NAMES

dict PYTHON_NAMES

Initial value:

=  {
    'iso-8859-8-i': 'iso-8859-8',
    'x-mac-cyrillic': 'mac-cyrillic',
    'macintosh': 'mac-roman',
    'windows-874': 'cp874'}

Definition at line 26 of file __init__.py.

UTF8

UTF8 = lookup('utf-8')

Definition at line 133 of file __init__.py.

VERSION

str VERSION = '0.5.1'

Definition at line 22 of file __init__.py.