Venvs and packaging - Python

From Helpful
(Redirected from Conda)
Jump to navigation Jump to search

Virtual environments and packaging

The problem: when installs break other installs

Language agnostic packaging · App images

Python packaging · Ruby packaging · Rust packaging · R packaging


Python virtual environments

virtualenv (python2, python3)

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

(see also the similar but distinct conda virtual environments)


virtualenv allows you to create a directory that represents

a specific python interpreter
often used to settle it to a specific python version
a distinct set of packages
that (by default) apply on top of the system ones
setuptools and pip that will install into this environment, instead of into the system

This makes it useful for user-installed apps where you need supporting libraries, and where that shouldn't conflict with other things.


Example - creating

Assuming that

  • the default python is python2.6
  • you run virtualenv NAME

Then you'll now have at least:

  • ./NAME/lib/python2.6/distutils
  • ./NAME/lib/python2.6/site-packages
  • ./NAME/include
  • ./NAME/bin
    • ./NAME/bin/python
    • ./NAME/bin/python2.6
    • ./NAME/bin/easy_install (installs into this environment)
    • ./NAME/bin/activate (lets you use the environment in the shell -- must be sourced through bash)

...which is site-packages, setuptools, a copy of the interpreter that uses this environment, and a few other things (e.g. recently also pip, wheel).


Example - using

It's useful context that where import looks (see also [1]) - e.g. that sys.path is initialized with the script's (if run via a script hashbang) or the python executable's (if invoked directly) containing path (then site stuff, then then PYTHON_PATH)


There are various ways to use that resulting file tree:

  • run source NAME/bin/activate is most typical
prepends the path to that python binary, meaning running 'python' will be the one being run over others
  • run the python binary in there
activate_this = '/path/to/env/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))

See also

https://www.dabapps.com/blog/introduction-to-pip-and-virtualenv-python/


Reproducing the same set of packages elsewhere isn't a virtualenv feature, but but something you typically want to do.

This is often done via pip freeze and pip install -r





https://docs.python-guide.org/dev/virtualenvs/


"No module named 'virtualenv.seed.via_app_data'"

Seems to indicate conflicting versons of virtualenv (yes, the virtualenv module that has been around since python2(verify). It's still perfectly usable in python3).

https://github.com/pypa/virtualenv/issues/1875

venv (python3)

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

venv is a module and tool introduced in py3.3.


Much like virtualenv, but became standard library (and cleaned up some details)


https://docs.python.org/3/library/venv.html

virtualenv/venv and packaging

Finding what virtual environments you have lying around

pipenv

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Conceptually, pipenv is the the combination of...

pip
and the virtualenv concept,

...giving you installs into isolated projects.


You may care that

  • the project directories it sets up will not be as cluttered as in virualenv/venv.
  • It also considers code versioning, in that
    • Users need only care about the Pipfile in a directory
    • the software you install is stored elsewhere, and potentially shared
~/.local/share/virtualenvs/ (rather than in env/lib within each project, as with virtualenv/venv), which e.g. keeps things cleaner around code versioning.


You can start a new one like: 

mkdir myproj
cd myproj
pipenv --python 3.6   # create project in curdir.  Optional (`pipenv install` creates a Pipfile too), but this way you control py version
pipenv install numpy  # install software in this environment

The only thing it puts in this directory is a Pipfile (there is more, but it's hidden in your homedir, because it's potentially shared state)


You can then start a subshell, for the environment implied by the current directory, like: 

pipenv shell


See also:

Anaconda, miniconda, conda (python and more)

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

These amount to its own package manager and its own environments isolation

With the aims to

more easily reproduce environments,
more controllable versions of things,
more portability within win+lin+osx,
do more than pure python (or python at all)
(avoid having a compilation environment - see the last two points)

...and seems aimed somewhat at academia


anaconda is a download-a-few-gigabytes-of-the-most-common-stuff-up-front, base of common things you can select from.

which may be most or all that you'll use

miniconda is a 'start installing from scratch' variant of the same thing

it mainly bootstraps the repository system, and downloads everything on demand


conda is the package manager they share


Conda environments

A conda environment is a distinct installation of everything, including python itself (if it's in there, which it usuaolly is. But if not, you may get a system python, pyenv shim, or such).


Consider e.g.: 

$ which python3
/usr/bin/python3
$ conda env list
# conda environments:
#
base                  *  /home/me/miniconda3 
foo                      /home/me/miniconda3/envs/foo
$ conda activate base
(base) $ which python3
home/me/miniconda3/bin/python3

"activating" a conda environment just places it first in resolving its bin/, which includes the executables of conda packages installed into that environment.

Notes:

this also helps separate it from anything relying on your system python being on the PATH
and is why you generally wouldn't add the conda bin to your path directly


as to cleaning: https://stackoverflow.com/questions/56266229/is-it-safe-to-manually-delete-all-files-in-pkgs-folder-in-anaconda-python


Getting conda in your shell

That the above assumed that you can already run your own conda, but that is something you have to set up. There are two practical parts to that:

  • getting conda into your PATH
  • whether or not it should activate the a base conda environment by default
in a new install it will do this, because auto_activate_base is true

The thing that conda init hooks in does both.

During the install there's a question whether to do that. If you said no but want this later, get to your conda command and do conda init)}}

If you want it to just put conda in the path and not active the base environment, you'll want to: 

conda config --set auto_activate_base false



Workflow stuff - environments and dependency files

You could treat conda as one overall environment, but you probably want to isolate projects: 

conda create -n yourenvname python=x.x anaconda

difference between conda create and conda env create? 

source activate yourenvname


Note that conda environments are not really compatible with virtualenv or pipenv due to specific features.

So yeah, if you previously had the virtualenv idea in your project workflow to recreate environments elsewhere, you'll need to switch to conda for that. Consider e.g. 

conda env export > environment.yml

and 

conda env create -f environment.yml

(environment.yml is conda's conceptual equivalent of requirements.txt)


You can' use pip within a conda environment, and can hook pip installs into such conda environment YAML files[2]

https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/#Myth-#5:-conda-doesn't-work-with-virtualenv,-so-it's-useless-for-my-workflow


See also:

pyenv

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Lets you

  • install and select user-specific versions of python,
    • and packages
  • do virtual environments (optional)
so until you use them, things using the same python version will share their own dist-packages


Implementation-wise, it intercepts python commands (via PATH trickery), and determines which python executable version (...and related commands, like pip) should be, depending on context.

The way it does that is *nix-specific, so the windows implementation[3] is a little different.


This lets you

  • set preferred python version per user (before you first set this, you will probably be using system)
pyenv global
  • set preferred python version for current shell
pyenv shell
  • set preferred python version under directory
pyenv local


There is a special version name, system, meaning "whatever's on the path". Before you install any of your own, pyenv versions would only mention system.


All the python variants pyenv knows how to install: 

pyenv install -l

Installing one: 

pyenv install 3.9     # which might e.g. install 3.9.16



pyenv and virtual environments

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

If you also wanted virtual environments, this is something you do separately [4].

Consider 

pyenv virtualenv 3.9 NAME

This creates a new environment by that name (within $PYENV_ROOT, not the current directory), which you can now select with e.g. pyenv local.

It's good practice to name it clearly, and possibly include the python version it's based on.


I had questions

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

"Where does it install python versions? (and packages)"

Under $PYENV_ROOT/versions/

which in user setups (probably most) will be ~/.pyenv/versions/.
and which before installing anything is empty (you're still using system)

Note that it will add shims (which pretend to be the main executables) in /root/.pyenv/shims, which is directly in the PATH.


So if you pyenv install 3.9.2 and 3.9.16, python3.9 is a shim that resolves that.

And if none is considered activated, that shim will fail.


"Where does it store version/env preferences?"

pyenv global goes to ~/.python-version

pyenv local goes to .python-version in the directory you execute that.

pyenv shell goes to PYENV_VERSION environment variable


"What if there are multiple preferences set?"

More specific override more general - PYENV_VERSION over .python-version over ~.python-version and if nothing is set you get the system python. (verify)


"Does it pick up system-installed packages?"

Unless it ends up picking system, no.

It seems to be pyenv's position that this is a mistake, that the whole point of pyenv is to have your own, that cannot conflict with or break your system python install.


You can, in theory, symlink the system python under ~/.pyenv/versions/ but this might be dangerous around the uninstall command(verify).


https://github.com/pyenv/pyenv


pipx

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Seems geared specifically to answer "how can I make stuff usable in my terminal, while having them come from a venv-like thing rather than the system install?"

Notes:

  • Like other pip-likes, allows installs from github repos,
so can be quick ways to start using someone else's tools
  • does not apply to things that do _not_ have CLI commands.
e.g. libraries without them won't be installable with pipx(verify)


Side note on freezing

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

(note: this is unrelated to package managers freezing a package, which is basically just listing packages and their versions, usually to duplicate elsewhere)


Freezing means wrapping your code so that it does not depend on anything other than the frozen product.

This usually meaning a copy of the a python interpreter, all modules it depends on, and some duct tape to make that work independently elsewhere.


It often creates a relatively large directory, and doesn't really let you alter it later. It's analogous to app images.

The main reason to do this is to have a self-contained copy that should run anywhere (in particular, it does not rely on an installed version of python) so is nice for packaging a production version of your desktop app.



To do this yourself, you can read things like https://docs.python.org/2/faq/windows.html#how-can-i-embed-python-into-a-windows-application


It's easier to use other people's tools.

Options I've tried:

  • cx_freeze
lin, win, osx
http://cx-freeze.sourceforge.net/
  • PyInstaller
lin, win, osx
can pack into single file
http://pyinstaller.python-hosting.com/
See also http://bytes.com/forum/thread579554.html for some get-started introduction

Untried:

lin, win, osx
  • py2exe [6] (a distutils extension)
windows (only)
can pack into single file
inactive project now?
  • Python's freeze.py (*nix) (I don't seem to have it, though)
mac OSX (only)
  • Gordon McMillan's Installer (discontinued, developed on into PyInstaller)


See also:


TODO: read:




Installing and creating python packages

Doing package installs

tl;dr

  • for system installs
pip (or similar) will install into the same dist-utils your system package manager
system package manage should mix decently with pip installs
but it can gets confusing when you have one install things the other isn't aware of
So you might want to prefer using just one as much as possible
and it's a secondary reason that virtualenv installs keeps things clearer in more custom setups


  • for distinct stacks (dev, fragile apps)
consider virtualenv
consider pipenv, conda, and similar -- sometimes doing something virtualenv-like for you is often simpler/cleaner


  • creating and uploading packages
look at distribute (basically a nicer setuptools)


pip notes

Install

python -m pip

The advice to use python -m pip instead of pip comes mostly from it being more obvious which of the installed python versions you're referencing.

It's otherwise identical.

pip search is dead, long live the alternatives
✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.


pip search never did much more than a substring search in name and summary, but the API it relied on was always considered experimental, and there was a long-term (and possibly accidental) DDoS going on that made hosting costs high. So they shut that down (See https://github.com/pypa/pip/issues/5216 and https://status.python.org/incidents/grk0k7sz6zkp for details)


pip may be working on a local replacement based on pip index (verify), but in the meantime, alternatives include:

  • pip_search (seems to scrape pypi website)
pip install pip_search
pip_search scikit
  • pypisearch (seems to require py>=3.8, though)
git clone https://github.com/shidenko97/pypisearch & cd pypisearch & pip install .
python -m pypisearch scikit


Install from git
Can you update all packages?

It's not really set up for it.

There are a few hacks.


There is 

pip list --outdated

and there is a third party tool called pip-review that lets you interactively choose which of those to update: 

pip-review --interactive
User installs
  • pip run as a non-root user seems to act as if --user was specified(verify)

pip and dependencies

showing package dependencies

For installed packages, 

pip show spacy

...will show something like (some formatting added here): 

Name:         spacy
Version:      3.5.0
Summary:      Industrial-strength Natural Language Processing (NLP) in Python
Home-page:    https://spacy.io
Author:       Explosion
Author-email: contact@explosion.ai
License:      MIT
Location:     /usr/local/lib/python3.8/dist-packages
Requires:     catalogue, cymem, jinja2, langcodes, murmurhash, numpy, packaging, 
              pathy, preshed, pydantic, requests, setuptools, smart-open, 
              spacy-legacy, spacy-loggers, srsly, thinc, tqdm, typer, wasabi
Required-by:  collocater, en-core-web-lg, en-core-web-md, en-core-web-sm, 
              en-core-web-trf, nl-core-news-lg, nl-core-news-md, 
              nl-core-news-sm, spacy-experimental, spacy-fastlang, 
              spacy-transformers


Note that the required-by only list things that require it and you have installed, not all the possible things, so will vary between installations.

A user installed package will show a different location, e.g.: 

Name: jedi
Version: 0.18.1
Summary: An autocompletion tool for Python that can be used for text editors.
Home-page: https://github.com/davidhalter/jedi
Author: David Halter
Author-email: davidhalter88@gmail.com
License: MIT
Location: /home/me/.local/lib/python3.8/site-packages
Requires: parso
Required-by: ipython

Development

Reproducing the same set of packages elsewhere

One convention that has grown strong enough to almost be a de facto standard is to create a file, usually called requirements.txt, that contains the package-and-version specs for each library you want.

Each (non-comment) line is essentially arguments to a unique call to the pip CLI tool, and is parsed by pip; it is e.g. the pip documentation that notes that yes, you could add comments


So options include 

FooProject
FooProject >= 1.2
FooProject >= 1.2 --global-option="--no-user-cfg"


As this documentation mentions, the last is roughly equivalent to going into FooProject and running: 

python setup.py --no-user-cfg install


requirements.txt combine well with virtual environments

Say, if you just created a venv, you can now do: 

pip install -r requirements.txt

...and it will install everything this project needs.


Similarly, if you are currently within a venv, you can create a requirements.txt like 

pip freeze > requitements.txt
💤 This will, however, be overly specific. That is, it will probably list the precise version you have installed, such as:
 webencodings==0.5.1
 WebOb==1.8.7
 websocket-client==0.53.0
 websockets==10.4
 Werkzeug==2.2.2

and you might actually want to edit that to be more accepting, if you want it to accept updates from each library.


pipenv originated in part from trying to make things even simpler than those manual steps of creating and picking up requirements.txt

Editable installs
✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.



PyPI notes

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Install related errors

DBus error on python package installs

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it. 
No such interface “org.freedesktop.DBus.Properties” on object at path /org/freedesktop/secrets/collection/login

When you use something like pip, or something more complex like poetry or twine.


You'll probably see packages like keyring and secretstorage.

If you didn't actually need auth storage, then prepending 

PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring

to your command should be a good test whether keyring is the problem - and be a good temporary workaround.


https://unix.stackexchange.com/questions/684854/free-desktop-dbus-error-while-installing-any-package-using-pip



Creating python packages

Some context on python packaging

Packaging was initially a bit minimal, and pasted on, There was also the need to have convenience during development. This and more led to people alternatives and duct taping, and python's packaging history is a bit of a messy confusion.


💤 History we can now mostly forget about

We had

  • distutils (2000ish)
standard library
  • PEP-273 introduced zip imports (2001)
can be copied into place
is then mostly equivalent to having that thing unzipped in the same location
...with some footnotes related to import's internals.
  • PyPI (2003)
meant as a central repository
initially just a repository of links to zips elsewhere, which you would manually download, unpack, and either setup.py install (distutil stuff) (or sometimes just copy the contents to site-packages)
  • setuptools (2004)
introduced eggs
introduced easy_install (which these days is no longer used)
  • egg (2004, see previous point. Never put into a PEP)
eggs are zip imports that adhere to some extra details, mostly for packaging systems, e.g. making them easier to discover, their dependencies resolved, and installed.
there are some variants. A good readup involves the how and why of setuptools, pkg_resources, EasyInstall / pip, and more
Ideally, you can now skip eggs
  • distribute (2008)
fork of setuptools, so also provides setuptools
had a newer variant of easy_install (from distribute, so ~2008)
(how relevant is this one?)
  • distutils2 (~2010) - made useful contributions, apparently not interesting as its own thing[8]

More interestingly, though...


  • pip (2008)
intended to replace easy_install
more aware of depdendencies (verify)
can uninstall; easy_install could not
downsides:
cannot install eggs (seemingly because we wanted to replace eggs with wheels?(verify))
doesn't isolate different versions (verify)
limited to python - C dependencies are still ehhh
  • wheel format is introduced (2013; PEP-427, PEP 491 ) as replacement for egg format.
intended to be a cleaner, better defined thing, for just installs - see On wheels
  • PEP-518 introduced pyproject.toml (2016)
specifies what build tools you require to build a wheel
better defined than setup.cfg did(verify) and with whatever version of setuptools someone would have installed which you would have no control over
(and using TOML format to be easier than ini/configparser?


developers giving developers installs

developers giving users installs

specifying packages

setup.cfg notes
setup.py notes
pyproject.toml notes
On wheels
✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.


any of this and docker

Dependency specifiers

See also:

Creating packages

Manual

Flit

Tries to make it easier for you to publish to PyPI

https://flit.pypa.io/


PDM

https://pdm.fming.dev/


Hatchling / Hatch

Hatch is a project manager.

Hatchling it its build backend.


https://pypi.org/project/hatchling/


Poetry

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.
Retrieved from "https://helpful.knobs-dials.com/index.php?title=Venvs_and_packaging_-_Python&oldid=98671#Anaconda,_miniconda,_conda_(python_and_more)"