Skip to content

Instantly share code, notes, and snippets.

@chrahunt

chrahunt/README.md

Last active September 10, 2019 13:15
Show Gist options
  • Save chrahunt/9800dbad6fbd4ebe998d5a9b447afcc3 to your computer and use it in GitHub Desktop.
Save chrahunt/9800dbad6fbd4ebe998d5a9b447afcc3 to your computer and use it in GitHub Desktop.
Download ZIP
Python installation candidate handling
Raw
README.md

states.yml lists some possible states we can consider for installable candidates and how we can create new artifacts that give us more information during the installation process.

Conventions:

  1. If no action applies, than it's a failure
  2. If no conditional state within an action applies, then it's a failure

One possible implementation may look like:

Represent a Candidate as a proxy object with an implementation like

class Candidate:
    @property
    def name(self):
        try:
            return self.obj.name
        except AttributeError:
            self._advance()
        return self.name

    @property
    def metadata(self):
        try:
            return self.obj.metadata
        except AttributeError:
            self._advance()
        return self.metadata

    def _advance(self):
        self.obj = next(self.obj)

where self.obj is an object of a type that represents one of the states above. Then using a property not provided by the current state advances it as-needed, doing any download or build as required.

Raw
states.yml
---
- name: Remote unnamed VCS
example: git+https://github.com/pypa/pip.git
actions:
- name: download
to_state: local unnamed vcs
provides: []
tags: [user-provided]
- name: Remote named VCS
example: git+https://github.com/pypa/pip.git#egg=pip[all]-19.2.3
actions:
- name: download
to_state: local named vcs
provides:
- name
- name: version
optional: True
source: "#egg=..."
- name: desired_extras
optional: True
source: "#egg=..."
tags: [user-provided]
- name: Remote editable named VCS
example: -e git+https://github.com/pypa/pip.git#egg=pip
actions:
- name: download
to_state: local editable named vcs
provides: [name]
tags: [user-provided]
- name: Remote sdist
example: https://files.pythonhosted.org/packages/00/9e/4c83a0950d8bdec0b4ca72afd2f9cea92d08eb7c1a768363f2ea458d08b4/pip-19.2.3.tar.gz
actions:
- name: download
to_state: local sdist
provides: [name, version]
tags: [user-provided]
notes:
- Can be directly provided or returned from an index query if the user gave 'pip'
- name: Remote wheel
example: https://files.pythonhosted.org/packages/30/db/9e38760b32e3e7f40cce46dd5fb107b8c73840df38f0046d8e6514e675a1/pip-19.2.3-py2.py3-none-any.whl
actions:
- name: download
to_state: local wheel
provides: [name, version, wheel_tag]
tags: [user-provided]
notes:
- Can be directly provided or returned from an index query if the user gave 'pip'
- name: Local non-editable directory
example: ./pip
actions:
- name: copy
to_state: unpacked sources
action: copy to temporary directory
notes:
- this is the current pip behavior
- name: build sdist in place
to_state: local sdist
notes:
- this is the proposed behavior, see discussion on https://github.com/pypa/pip/issues/2195
- transitioning directly to local sdist means doing either a PEP 517 build_sdist or legacy setup.py sdist
provides: []
tags: [user-provided]
- name: Local editable directory
example: -e ./pip
actions:
- name: query
to_states:
- name: local editable legacy project
condition: setup.py is present
provides: []
tags: [user-provided]
- name: Local wheel
example: dist/pip-19.2.3-py2.py3-none-any.whl
actions:
- name: unpack
to_state: unpacked wheel
provides: [name, version, wheel_tag]
tags: [user-provided]
- name: Local editable legacy project
actions:
- name: install
to_state: installed editable distribution
- name: Unpacked wheel
actions:
- name: install
to_state: installed overwritable distribution
provides: [name, version, metadata, dependencies]
tags: [not-user-provided]
- name: Local sdist
actions:
- name: unpack
to_state: unpacked sources
provides: [name, version]
tags: [user-provided]
- name: Local editable named VCS
example: -e git+file://$PWD/pip#egg=pip
actions:
- name: query
to_states:
- name: local editable legacy project
condition: if setup.py present
provides:
- name
- name: version
optional: True
- name: desired_extras
optional: True
tags: [user-provided]
- name: Local unnamed VCS
example: git+file://$PWD/pip
actions:
- name: query
to_states:
- name: local modern project
condition: if pyproject.toml present
- name: local legacy project
condition: if setup.py present
provides: []
notes:
- the reason for going directly to modern/legacy project (as opposed to
treating this as unpacked sources) is because there may be a subdirectory
indicated, so the behavior for determining the directory is different
tags: [user-provided]
- name: Local named VCS
example: git+file://$PWD/pip#egg=pip
actions:
- name: query
to_states:
- name: local modern project
condition: if pyproject.toml present
- name: local legacy project
condition: if setup.py present
provides:
- name
- name: version
optional: True
source: "#egg=..."
- name: desired_extras
optional: True
tags: [user-provided]
- name: Local archive
example: ./package.zip
actions:
- name: unpack
to_state: local non-editable directory
provides: []
tags: [user-provided]
- name: Remote archive
example: https://company.com/repo/package.zip
actions:
- name: download
to_state: local archive
provides: []
tags: [user-provided]
- name: Installed overridable distribution
notes:
- Could represent a globally-installed package when we're installing to
a user site or virtual environment.
- TODO
- name: Installed overwritable distribution
example: .venv/lib/pythonX.Y/site-packages/foo.dist-info (or egg-info)
provides: [name, version, metadata, uninstallable]
notes:
- extras are missing
- https://github.com/pypa/packaging-problems/issues/215#issuecomment-504569821
- name: Installed editable distribution
example: .venv/lib/pythonX.Y/site-packages/foo.egg-link
- name: Unpacked wheel
actions:
- name: install
action: copy files, generate
to_state: installed overwritable distribution
provides:
- name: name
source: .dist-info/METADATA
- name: version
source: .dist-info/METADATA
- name: metadata
source: .dist-info/METADATA
tags: [not-user-provided]
- name: Unpacked sources
example: /tmp/pip-<random>/pip
actions:
- name: query
to_states:
- name: local legacy project
condition: setup.py present
- name: local modern project
condition: pyproject.toml present
provides:
- name: name
optional: True
notes:
- if our previous state provided it (e.g. sdist)
- name: version
optional: True
notes:
- if our previous state provided it (e.g. sdist)
- name: desired_extras
optional: True
notes:
- if our previous state provided it (e.g. sdist)
notes:
- Should we depend on PKG-INFO? No. In the future we can use it if there’s
a spec that says it can be trusted to have all relevant details (including
dependencies)
tags: [not-user-provided]
- name: Local legacy project
example: /tmp/pip-<random>/pip (with setup.py)
actions:
- name: build
to_state: local wheel
when: "'wheel' is installed"
action: setup.py bdist_wheel
- name: install
to_state: installed overwritable distribution
action: setup.py install
provides:
- name: name
source: setup.py egg_info
- name: version
source: setup.py egg_info
- name: metadata
source: setup.py egg_info
notes:
- if name and version were also provided by the successor of this state,
then they would be validated against the ones returned by egg_info
tags: [not-user-provided]
- name: Local modern project
example: /tmp/pip-<random>/pip (with pyproject.toml)
actions:
- name: build
action: PEP 517 `build_wheel` hook
to_state: local wheel
provides:
- name: metadata
when: "`prepare_metadata_for_build_wheel` is implemented"
source: .dist-info generated by prepare_metadata_for_build_wheel
tags: [not-user-provided]
Raw
verify.py
import os
from contextlib import contextmanager
from pathlib import Path
import yaml
obj = yaml.safe_load(
Path(__file__).parent.joinpath('stages.yml').read_text(encoding='utf-8')
)
states_by_name = {}
for state in obj:
states_by_name[state['name'].lower()] = state
@contextmanager
def error_reporter(name):
try:
yield
except:
print(f"Error processing {name}")
raise
for state in obj:
name = state['name']
actions = state.get('actions', [])
with error_reporter(name):
for action in actions:
try:
to_states = [action['to_state']]
except KeyError:
to_states = [s['name'] for s in action['to_states']]
for to_state_name in to_states:
if to_state_name not in states_by_name:
@pradyunsg
Copy link 

- name: Unfetched remote archive
  example: https://company.com/repo/package.zip
  actions:
    download:
      result: Local archive
  provides: []

- name: Local legacy project
  example: /tmp/pip-<random>/pip (with setup.py)
  actions:
    build:
      if: `wheel` is installed
      do: setup.py bdist_wheel
      result: Local wheel
    install:
      if: `wheel` is NOT installed
      do: setup.py install
      result: Installed overwritable distribution
  provides: [name, version, metadata]

@pradyunsg
Copy link

Noting that "unpacked wheel" shows up twice here: 10th and 20th element.

@pradyunsg
Copy link

Alsoooo modern projects, can't be installed directly. They have to go through being built into a wheel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment