pip._internal.req.req_file Namespace Reference

Data Structures
class	OptionParsingError
class	ParsedLine
class	ParsedRequirement
class	RequirementsFileParser

Functions
Generator[ParsedRequirement, None, None]	parse_requirements (str filename, PipSession session, Optional["PackageFinder"] finder=None, Optional[optparse.Values] options=None, bool constraint=False)
ReqFileLines	preprocess (str content)
ParsedRequirement	handle_requirement_line (ParsedLine line, Optional[optparse.Values] options=None)
None	handle_option_line (Values opts, str filename, int lineno, Optional["PackageFinder"] finder=None, Optional[optparse.Values] options=None, Optional[PipSession] session=None)
Optional[ParsedRequirement]	handle_line (ParsedLine line, Optional[optparse.Values] options=None, Optional["PackageFinder"] finder=None, Optional[PipSession] session=None)
LineParser	get_line_parser (Optional["PackageFinder"] finder)
Tuple[str, str]	break_args_options (str line)
optparse.OptionParser	build_parser ()
ReqFileLines	join_lines (ReqFileLines lines_enum)
ReqFileLines	ignore_comments (ReqFileLines lines_enum)
ReqFileLines	expand_env_variables (ReqFileLines lines_enum)
Tuple[str, str]	get_file_content (str url, PipSession session)

Variables
	ReqFileLines = Iterable[Tuple[int, str]]
	LineParser = Callable[[str], Tuple[str, Values]]
	SCHEME_RE = re.compile(r"^(http|https|file):", re.I)
	COMMENT_RE = re.compile(r"(^|\s+)#.*$")
	ENV_VAR_RE = re.compile(r"(?P<var>\$\{(?P<name>[A-Z0-9_]+)\})")
list	SUPPORTED_OPTIONS
list	SUPPORTED_OPTIONS_REQ
list	SUPPORTED_OPTIONS_REQ_DEST = [str(o().dest) for o in SUPPORTED_OPTIONS_REQ]
	logger = logging.getLogger(__name__)

Detailed Description

Requirements file parsing

Function Documentation

break_args_options()

Tuple[str, str] break_args_options

(

str

line

)

Break up the line into an args and options string.  We only want to shlex
(and then optparse) the options, not the args.  args can contain markers
which are corrupted by shlex.

Definition at line 416 of file req_file.py.

def break_args_options(line: str) -> Tuple[str, str]:
    """Break up the line into an args and options string.  We only want to shlex
    (and then optparse) the options, not the args.  args can contain markers
    which are corrupted by shlex.
    """
    tokens = line.split(" ")
    args = []
    options = tokens[:]
    for token in tokens:
        if token.startswith("-") or token.startswith("--"):
            break
        else:
            args.append(token)
            options.pop(0)
    return " ".join(args), " ".join(options)
 
 

References i.

Referenced by pip._internal.req.req_file.get_line_parser().

Here is the caller graph for this function:

build_parser()

optparse.OptionParser build_parser

(

)

Return a parser for parsing requirement lines

Definition at line 438 of file req_file.py.

def build_parser() -> optparse.OptionParser:
    """
    Return a parser for parsing requirement lines
    """
    parser = optparse.OptionParser(add_help_option=False)
 
    option_factories = SUPPORTED_OPTIONS + SUPPORTED_OPTIONS_REQ
    for option_factory in option_factories:
        option = option_factory()
        parser.add_option(option)
 
    # By default optparse sys.exits on parsing errors. We want to wrap
    # that in our own exception.
    def parser_exit(self: Any, msg: str) -> "NoReturn":
        raise OptionParsingError(msg)
 
    # NOTE: mypy disallows assigning to a method
    #       https://github.com/python/mypy/issues/2427
    parser.exit = parser_exit  # type: ignore
 
    return parser
 
 

References i.

Referenced by pip._internal.req.req_file.get_line_parser().

Here is the caller graph for this function:

expand_env_variables()

ReqFileLines expand_env_variables

(

ReqFileLines

lines_enum

)

Replace all environment variables that can be retrieved via `os.getenv`.

The only allowed format for environment variables defined in the
requirement file is `${MY_VARIABLE_1}` to ensure two things:

1. Strings that contain a `$` aren't accidentally (partially) expanded.
2. Ensure consistency across platforms for requirement files.

These points are the result of a discussion on the `github pull
request #3514 <https://github.com/pypa/pip/pull/3514>`_.

Valid characters in variable names follow the `POSIX standard
<http://pubs.opengroup.org/onlinepubs/9699919799/>`_ and are limited
to uppercase letter, digits and the `_` (underscore).

Definition at line 503 of file req_file.py.

def expand_env_variables(lines_enum: ReqFileLines) -> ReqFileLines:
    """Replace all environment variables that can be retrieved via `os.getenv`.
 
    The only allowed format for environment variables defined in the
    requirement file is `${MY_VARIABLE_1}` to ensure two things:
 
    1. Strings that contain a `$` aren't accidentally (partially) expanded.
    2. Ensure consistency across platforms for requirement files.
 
    These points are the result of a discussion on the `github pull
    request #3514 <https://github.com/pypa/pip/pull/3514>`_.
 
    Valid characters in variable names follow the `POSIX standard
    <http://pubs.opengroup.org/onlinepubs/9699919799/>`_ and are limited
    to uppercase letter, digits and the `_` (underscore).
    """
    for line_number, line in lines_enum:
        for env_var, var_name in ENV_VAR_RE.findall(line):
            value = os.getenv(var_name)
            if not value:
                continue
 
            line = line.replace(env_var, value)
 
        yield line_number, line
 
 

References i.

Referenced by pip._internal.req.req_file.preprocess().

Here is the caller graph for this function:

get_file_content()

Tuple[str, str] get_file_content	(	str	url,
		PipSession	session
	)

Gets the content of a file; it may be a filename, file: URL, or
http: URL.  Returns (location, content).  Content is unicode.
Respects # -*- coding: declarations on the retrieved files.

:param url:         File path or url.
:param session:     PipSession instance.

Definition at line 530 of file req_file.py.

def get_file_content(url: str, session: PipSession) -> Tuple[str, str]:
    """Gets the content of a file; it may be a filename, file: URL, or
    http: URL.  Returns (location, content).  Content is unicode.
    Respects # -*- coding: declarations on the retrieved files.
 
    :param url:         File path or url.
    :param session:     PipSession instance.
    """
    scheme = get_url_scheme(url)
 
    # Pip has special support for file:// URLs (LocalFSAdapter).
    if scheme in ["http", "https", "file"]:
        resp = session.get(url)
        raise_for_status(resp)
        return resp.url, resp.text
 
    # Assume this is a bare path.
    try:
        with open(url, "rb") as f:
            content = auto_decode(f.read())
    except OSError as exc:
        raise InstallationError(f"Could not open requirements file: {exc}")
    return url, content

References i.

Referenced by RequirementsFileParser._parse_file().

Here is the caller graph for this function:

get_line_parser()

LineParser get_line_parser

(

Optional["PackageFinder"]

finder

)

Definition at line 392 of file req_file.py.

def get_line_parser(finder: Optional["PackageFinder"]) -> LineParser:
    def parse_line(line: str) -> Tuple[str, Values]:
        # Build new parser for each line since it accumulates appendable
        # options.
        parser = build_parser()
        defaults = parser.get_default_values()
        defaults.index_url = None
        if finder:
            defaults.format_control = finder.format_control
 
        args_str, options_str = break_args_options(line)
 
        try:
            options = shlex.split(options_str)
        except ValueError as e:
            raise OptionParsingError(f"Could not split options: {options_str}") from e
 
        opts, _ = parser.parse_args(options, defaults)
 
        return args_str, opts
 
    return parse_line
 
 

References pip._internal.req.req_file.break_args_options(), pip._internal.req.req_file.build_parser(), and i.

Referenced by pip._internal.req.req_file.parse_requirements().

Here is the call graph for this function:

Here is the caller graph for this function:

handle_line()

Optional[ParsedRequirement] handle_line	(	ParsedLine	line,
		Optional[optparse.Values]	options = None,
		Optional["PackageFinder"]	finder = None,
		Optional[PipSession]	session = None
	)

Handle a single parsed requirements line; This can result in
creating/yielding requirements, or updating the finder.

:param line:        The parsed line to be processed.
:param options:     CLI options.
:param finder:      The finder - updated by non-requirement lines.
:param session:     The session - updated by non-requirement lines.

Returns a ParsedRequirement object if the line is a requirement line,
otherwise returns None.

For lines that contain requirements, the only options that have an effect
are from SUPPORTED_OPTIONS_REQ, and they are scoped to the
requirement. Other options from SUPPORTED_OPTIONS may be present, but are
ignored.

For lines that do not contain requirements, the only options that have an
effect are from SUPPORTED_OPTIONS. Options from SUPPORTED_OPTIONS_REQ may
be present, but are ignored. These lines may contain multiple options
(although our docs imply only one is supported), and all our parsed and
affect the finder.

Definition at line 278 of file req_file.py.

) -> Optional[ParsedRequirement]:
    """Handle a single parsed requirements line; This can result in
    creating/yielding requirements, or updating the finder.
 
    :param line:        The parsed line to be processed.
    :param options:     CLI options.
    :param finder:      The finder - updated by non-requirement lines.
    :param session:     The session - updated by non-requirement lines.
 
    Returns a ParsedRequirement object if the line is a requirement line,
    otherwise returns None.
 
    For lines that contain requirements, the only options that have an effect
    are from SUPPORTED_OPTIONS_REQ, and they are scoped to the
    requirement. Other options from SUPPORTED_OPTIONS may be present, but are
    ignored.
 
    For lines that do not contain requirements, the only options that have an
    effect are from SUPPORTED_OPTIONS. Options from SUPPORTED_OPTIONS_REQ may
    be present, but are ignored. These lines may contain multiple options
    (although our docs imply only one is supported), and all our parsed and
    affect the finder.
    """
 
    if line.is_requirement:
        parsed_req = handle_requirement_line(line, options)
        return parsed_req
    else:
        handle_option_line(
            line.opts,
            line.filename,
            line.lineno,
            finder,
            options,
            session,
        )
        return None
 
 

References pip._internal.req.req_file.handle_option_line(), pip._internal.req.req_file.handle_requirement_line(), and i.

Referenced by pip._internal.req.req_file.parse_requirements().

Here is the call graph for this function:

Here is the caller graph for this function:

handle_option_line()

None handle_option_line	(	Values	opts,
		str	filename,
		int	lineno,
		Optional["PackageFinder"]	finder = None,
		Optional[optparse.Values]	options = None,
		Optional[PipSession]	session = None
	)

Definition at line 208 of file req_file.py.

) -> None:
    if opts.hashes:
        logger.warning(
            "%s line %s has --hash but no requirement, and will be ignored.",
            filename,
            lineno,
        )
 
    if options:
        # percolate options upward
        if opts.require_hashes:
            options.require_hashes = opts.require_hashes
        if opts.features_enabled:
            options.features_enabled.extend(
                f for f in opts.features_enabled if f not in options.features_enabled
            )
 
    # set finder options
    if finder:
        find_links = finder.find_links
        index_urls = finder.index_urls
        no_index = finder.search_scope.no_index
        if opts.no_index is True:
            no_index = True
            index_urls = []
        if opts.index_url and not no_index:
            index_urls = [opts.index_url]
        if opts.extra_index_urls and not no_index:
            index_urls.extend(opts.extra_index_urls)
        if opts.find_links:
            # FIXME: it would be nice to keep track of the source
            # of the find_links: support a find-links local path
            # relative to a requirements file.
            value = opts.find_links[0]
            req_dir = os.path.dirname(os.path.abspath(filename))
            relative_to_reqs_file = os.path.join(req_dir, value)
            if os.path.exists(relative_to_reqs_file):
                value = relative_to_reqs_file
            find_links.append(value)
 
        if session:
            # We need to update the auth urls in session
            session.update_index_urls(index_urls)
 
        search_scope = SearchScope(
            find_links=find_links,
            index_urls=index_urls,
            no_index=no_index,
        )
        finder.search_scope = search_scope
 
        if opts.pre:
            finder.set_allow_all_prereleases()
 
        if opts.prefer_binary:
            finder.set_prefer_binary()
 
        if session:
            for host in opts.trusted_hosts or []:
                source = f"line {lineno} of {filename}"
                session.add_trusted_host(host, source=source)
 
 

References i.

Referenced by pip._internal.req.req_file.handle_line().

Here is the caller graph for this function:

handle_requirement_line()

ParsedRequirement handle_requirement_line	(	ParsedLine	line,
		Optional[optparse.Values]	options = None
	)

Definition at line 168 of file req_file.py.

) -> ParsedRequirement:
    # preserve for the nested code path
    line_comes_from = "{} {} (line {})".format(
        "-c" if line.constraint else "-r",
        line.filename,
        line.lineno,
    )
 
    assert line.is_requirement
 
    if line.is_editable:
        # For editable requirements, we don't support per-requirement
        # options, so just return the parsed requirement.
        return ParsedRequirement(
            requirement=line.requirement,
            is_editable=line.is_editable,
            comes_from=line_comes_from,
            constraint=line.constraint,
        )
    else:
        # get the options that apply to requirements
        req_options = {}
        for dest in SUPPORTED_OPTIONS_REQ_DEST:
            if dest in line.opts.__dict__ and line.opts.__dict__[dest]:
                req_options[dest] = line.opts.__dict__[dest]
 
        line_source = f"line {line.lineno} of {line.filename}"
        return ParsedRequirement(
            requirement=line.requirement,
            is_editable=line.is_editable,
            comes_from=line_comes_from,
            constraint=line.constraint,
            options=req_options,
            line_source=line_source,
        )
 
 

References i.

Referenced by pip._internal.req.req_file.handle_line().

Here is the caller graph for this function:

ignore_comments()

ReqFileLines ignore_comments

(

ReqFileLines

lines_enum

)

Strips comments and filter empty lines.

Definition at line 492 of file req_file.py.

def ignore_comments(lines_enum: ReqFileLines) -> ReqFileLines:
    """
    Strips comments and filter empty lines.
    """
    for line_number, line in lines_enum:
        line = COMMENT_RE.sub("", line)
        line = line.strip()
        if line:
            yield line_number, line
 
 

References i.

Referenced by pip._internal.req.req_file.preprocess().

Here is the caller graph for this function:

join_lines()

ReqFileLines join_lines

(

ReqFileLines

lines_enum

)

Joins a line ending in '\' with the previous line (except when following
comments).  The joined line takes on the index of the first line.

Definition at line 461 of file req_file.py.

def join_lines(lines_enum: ReqFileLines) -> ReqFileLines:
    """Joins a line ending in '\' with the previous line (except when following
    comments).  The joined line takes on the index of the first line.
    """
    primary_line_number = None
    new_line: List[str] = []
    for line_number, line in lines_enum:
        if not line.endswith("\\") or COMMENT_RE.match(line):
            if COMMENT_RE.match(line):
                # this ensures comments are always matched later
                line = " " + line
            if new_line:
                new_line.append(line)
                assert primary_line_number is not None
                yield primary_line_number, "".join(new_line)
                new_line = []
            else:
                yield line_number, line
        else:
            if not new_line:
                primary_line_number = line_number
            new_line.append(line.strip("\\"))
 
    # last line contains \
    if new_line:
        assert primary_line_number is not None
        yield primary_line_number, "".join(new_line)
 
    # TODO: handle space after '\'.
 
 

References i.

Referenced by pip._internal.req.req_file.preprocess().

Here is the caller graph for this function:

parse_requirements()

Generator[ParsedRequirement, None, None] parse_requirements	(	str	filename,
		PipSession	session,
		Optional["PackageFinder"]	finder = None,
		Optional[optparse.Values]	options = None,
		bool	constraint = False
	)

Parse a requirements file and yield ParsedRequirement instances.

:param filename:    Path or url of requirements file.
:param session:     PipSession instance.
:param finder:      Instance of pip.index.PackageFinder.
:param options:     cli options.
:param constraint:  If true, parsing a constraint file rather than
    requirements file.

Definition at line 129 of file req_file.py.

) -> Generator[ParsedRequirement, None, None]:
    """Parse a requirements file and yield ParsedRequirement instances.
 
    :param filename:    Path or url of requirements file.
    :param session:     PipSession instance.
    :param finder:      Instance of pip.index.PackageFinder.
    :param options:     cli options.
    :param constraint:  If true, parsing a constraint file rather than
        requirements file.
    """
    line_parser = get_line_parser(finder)
    parser = RequirementsFileParser(session, line_parser)
 
    for parsed_line in parser.parse(filename, constraint):
        parsed_req = handle_line(
            parsed_line, options=options, finder=finder, session=session
        )
        if parsed_req is not None:
            yield parsed_req
 
 

References pip._internal.req.req_file.get_line_parser(), pip._internal.req.req_file.handle_line(), and i.

Here is the call graph for this function:

preprocess()

ReqFileLines preprocess

(

str

content

)

Split, filter, and join lines, and return a line iterator

:param content: the content of the requirements file

Definition at line 156 of file req_file.py.

def preprocess(content: str) -> ReqFileLines:
    """Split, filter, and join lines, and return a line iterator
 
    :param content: the content of the requirements file
    """
    lines_enum: ReqFileLines = enumerate(content.splitlines(), start=1)
    lines_enum = join_lines(lines_enum)
    lines_enum = ignore_comments(lines_enum)
    lines_enum = expand_env_variables(lines_enum)
    return lines_enum
 
 

References pip._internal.req.req_file.expand_env_variables(), i, pip._internal.req.req_file.ignore_comments(), and pip._internal.req.req_file.join_lines().

Referenced by RequirementsFileParser._parse_file().

Here is the call graph for this function:

Here is the caller graph for this function:

Variable Documentation

COMMENT_RE

COMMENT_RE = re.compile(r"(^|\s+)#.*$")

Definition at line 46 of file req_file.py.

ENV_VAR_RE

ENV_VAR_RE = re.compile(r"(?P<var>\$\{(?P<name>[A-Z0-9_]+)\})")

Definition at line 52 of file req_file.py.

LineParser

LineParser = Callable[[str], Tuple[str, Values]]

Definition at line 43 of file req_file.py.

logger

logger = logging.getLogger(__name__)

Definition at line 81 of file req_file.py.

ReqFileLines

ReqFileLines = Iterable[Tuple[int, str]]

Definition at line 41 of file req_file.py.

SCHEME_RE

SCHEME_RE = re.compile(r"^(http|https|file):", re.I)

Definition at line 45 of file req_file.py.

SUPPORTED_OPTIONS

list SUPPORTED_OPTIONS

Initial value:

=  [
    cmdoptions.index_url,
    cmdoptions.extra_index_url,
    cmdoptions.no_index,
    cmdoptions.constraints,
    cmdoptions.requirements,
    cmdoptions.editable,
    cmdoptions.find_links,
    cmdoptions.no_binary,
    cmdoptions.only_binary,
    cmdoptions.prefer_binary,
    cmdoptions.require_hashes,
    cmdoptions.pre,
    cmdoptions.trusted_host,
    cmdoptions.use_new_feature,
]

Definition at line 54 of file req_file.py.

SUPPORTED_OPTIONS_REQ

list SUPPORTED_OPTIONS_REQ

Initial value:

=  [
    cmdoptions.global_options,
    cmdoptions.hash,
    cmdoptions.config_settings,
]

Definition at line 72 of file req_file.py.

SUPPORTED_OPTIONS_REQ_DEST

list SUPPORTED_OPTIONS_REQ_DEST = [str(o().dest) for o in SUPPORTED_OPTIONS_REQ]

Definition at line 79 of file req_file.py.