Poetry notes

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

These are some specific notes about everyday use of python's poetry.

To see it in a wider packagey-context, see Venvs_and_packaging_-_Python#Python_virtual_environments.

What and why

From a practical sense, poetry has a few useful things going for it:

• as a better dependency resolver than pip

it will reconsider dependencies, not just add things and hope for the best

• as an convenient manager of a virtualenv's libraries

it does the pip installs inside it based on its dependency resolving - often less typing to install and/or run

• as project manager, in the "easier to reproduce versions between devs, and build where necessary" sense

uses pyproject.toml (standard), and its own poetry.lock to let you lock to specific versions

• as packager in the "...at which point we basically create something we can upload to pypi" sense

you can ignore this when you are only using it to reproduce environments

• ...which also as a way to reproduce environments in clusters and such (though things like uv may be a little more suited)

TODO: some comparison

Start from scratch

If you have existing code without poetry and have poetry create a new (mostly a pyptoject.toml) there

poetry init

Or create a fresh project directory with pyproject.toml and some basic files your new python project might need (or that you might throw away)

poetry new projname

Either way, you'll have to answer some questions.

(If you have a project with pyptoject.toml, move on to #poetry_install)

Either way you now have a poetry environment (meaning project-like metadata).

That will soon also create-and-manage a virtualenv (fitting the description in that project metadata).

You can, for example,

• add a library to that virualenv

poetry add lib

• run a script with that virtualenv

poetry run python myscript.py

Useful questions

Managing and installing

"What is package mode?"

💤 For context

• there is a concept of 'root package' :

the project defined by pyproject itself is called the root package

while everything installed into it is considered dependencies (for that root package)

This is more than just a technicality, in that

• when you use poetry just to create an environment to run stuff in, you might care about only the dependencies, and never about the root package

package-mode = true means

you are saying the point of this project is to build a package (sdist or wheel), e.g. to

create a package for someone to install

upload a package

test that builds run everywhere - see the idea of reproducible builds

package-mode = false means

you are saying that the packaging-specific part of project config is unimportant, and are e.g.

or just to use poetry for dependency management

often to

just run someone else's unpackaged code

or further develop that code and run it (and worry about packaging later, or not at all by just suggesting patches to them)

There is one potentially important footnote to that:

If you want edits to your project source source to be immediately reflected inside the virtualenv you are creating, then you want the editable install of the root package that only package-mode = true will give you.

If you only care to poetry run, you're fine either way.

This relates to the fact that the virtualenv is made to stand entirely independently, whereas poetry run will reach inside your project.

"What does poetry install do?"

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

If package-mode = true

• installs dependencies (into virtualenv)
• installs root package (into virtualenv), editable-install style

If package-mode = false,

• installs dependencies (into virtualenv)
• root package is ignored

If package-mode=true but you run poetry install --no-root, you also get the latter (dependencies-only ignoring the root package)

--no-root used to be more valuable before package-mode was an explicit concept in poetry

now it's mostly useful for "someone else's pyproject came with package mode on but I quickly want to install only deps"

(Note that you may have reasons beyond install to change or leave package-mode)

See also https://python-poetry.org/docs/basic-usage/#operating-modes

"How do lockfiles relate?"

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

For context, you can think of lockfiles as containing a specific solution of the dependencies listed in the project. Most of the time, a lockfile is usually altered when you add, remove, update, or lock

Since poetry install is only meant to install what is listed, it wouldn't need to write it. lockfiles will get read by poetry install if they are there.

...except that covers only common cases. More precisely, it is possible for you to have edited pyproject manually and to not have run update as you should, and poetry still does the right thing. That ends up being:

• if lockfile is there there and in sync with the dependencies listed in pyproject,

installs the exact versions listed in the lockfile (and doesn't write lockfile)

• if lockfile is there there but not in sync with the dependencies listed in pyproject,

does deperency resolution, installs what it ended up on, writes lockfile

• If lockfile it isn't there

does deperency resolution, installs what it ended up on, writes lockfile

Note that none of this is affected by whether package-mode is on or off. (a missing or out-of-date lockfile will always get rewritten, whether package-mode is on or off)

"Can I have my project's root package be live in my virtualenv, editable-install style?"

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Running

"Where does it get python versions from?"

After the virtualenv is created, it gets the python from inside the virtualenv - that is, after all, part of the point of a virtualenv.

You probably meant "when creating the virtualenv, what pythons can you get it to use?"

It can pick up python from:

• active interpreter (be it system-global, homebrew, pyenv, active virtualenv)

• pyenv-installed (even if not active?(verify))

It won't install new python versions on request -- but if you have pyenv set up, you can do that yourself fairly easily.

• explicitly pointing it to one, e.g. poetry env use /path/to/a/python

• previously (poetry-created) existing virtualenvs(verify)

(You van even tell it to do things without a virtualenv (including installing into it - if it has the permissions(verify)), but you probably don't want to)

"What versions are availble to me?"

And you have.

If you primarily use pyenv to get these versions, then pyenv versions is most of what you care for.

Poetry doesn't really have a way to list all the pythons it can find, but you can just try and see if it succeeds (assuming this is within your requires-python range):

poetry env use 3.9
poetry env use 3.10
poetry env use 3.11

Also, you can probably get a decent idea of what it sees in its current environment (which, note, isn't all it could be made use), pyenv or otherwise, with something like:

which -a python3.6 python3.7 python3.8 python3.9 python3.10 python3.11 python3.12 python | sort

"So can I just pyenv install version and it'll be available?"

Basically yes, e.g.

pyenv install 3.9 3.10 3.11 3.12

(Keep in mind pyenv is per-user. You can see which you've pyenv-installed with pyenv versions)

"What does poetry run actually do?"

You can think of poetry run as

• find the suitable virtualenv,
• activate that virtualenv (which puts its bin/, i.e. python, pip, and such first on the PATH)
• run whatever command you asked for

💤 In fact, at the core it seems

little more than a subprocess.Popen

in which it mainly just changes the (inherited) environment to changing:

• PATH - so that the virtualenv's bin comes first so that use of python, pip inside should work
• PYTHON_PATH - so that module paths get prepended to sys.path
• VIRTUAL_ENV - helps clarify that you are using a virtualenv (because the above two )
  

...and yes, it does this even when the thing you run isn't python directly.

"poetry run myscript" versus "poetry run python myscript"

These are distinct, and this is a somewhat common gotcha.

When you run poetry run python /path/to/env/with/script.py

• poetry will wrap the environment (same as below)
• then run the virtualenv's python executable (because virtualenv activation put it first(verify)) with that script as argument

Whereas when you run poetry run /path/to/env/with/script.py

• poetry will wrap the environment (same as above)
• but the OS will determine which python executable gets used -- via the usual hashbang logic.

which need not be the python in the virtualenv- and you typically want it to be

As such, you usually want the first of those.

There is one exception to this

• if you have, in pyproject, registered a CLI name via tool.poetry.scripts, this will be installed into the virtualenv's bin/ as a thin wrapper script that basically hardcodes that specific virtualenv's python.

so it is now less typing to do

poetry run that-script

(it should still be equivalent to, say, poetry run python .venv/bin/that-script(verify))

"Can I run other scripts using this virtualenv?"

Yes.

That virtualenv is a fully standard one, poetry only helps manage its contents.

And does not have its hooks in running them - just provides a fewer-keystroke way of starting them (poetry run).

You can activate and use it as any other.

Note that it is sometimes less typing to do activate-then-single-script-run via:

poetry run python /path/to/env/with/script.py

"How does poetry find things?"

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Finding the project directory

From the current directory, start walking to parents.

The first directory that contains a pyproject.toml is considered a project root (so yes, it depends on cwd).

Finding the virtualenv

The virtualenv is found in association to the project directory, but there is some flexibility.

By default, all poetry-created virtualenvs are placed in the user directory of each user that uses that pyproject (that pyproject is only the instructions)

Optionally, the virtualenv can also be placed inside the project directory

It would not be picked up unless you ask to look for it.

(It is also not shared until a few more conditions are met, and you may not want that)

It may fall back on an already-existing virtualenv that happened to refer to this directory (matching the project name and local absolute path -- the projectname-hash-pyversion entries refer to the hash of the absolute path, so that the same project in different places cannot collide)

Note that in theory, you can tell poetry to work in any environment, including the system python (see virtualenvs.create false) but you usually don't want to do that.

"Do I get my own virtualenv, do / can I share it?"

By default, your own:

every distinct user each user can and will generate a virtualenv

which will be written to their own home directory.

Poetry treats the pyproject.toml and/or lockfile only as instructions on how to build that (and the default place that goes to makes each user unique impicitly)

Alternatively, poetry can be told to, instead, create a .venv/ inside the project directory.

See poetry config virtualenvs.in-project true.

Notes:

• Note that even if an .venv exists because someone else set virtualenvs.in-project, your poetry will not pick that up (and instead still work within your userdir) until you set virtualenvs.in-project too.(verify)

• virtualenvs.in-project is a global(-only) setting that only tells poetry to create new virtualenvs differently (and look in a new place).

In particular, it will not move an already-created virtualenv from your ~/.cache to the project directory. The closest you have is to delete the applicable virtualenv from your profile, then poetry install again to recreate it, now in the newly intended place. (to find which venv to remove, try poetry env info or, if not activated, poetry env list), choose the correct one, and then e.g. poetry env remove myproject-rGWZLGg-py3.10

Generally discouraged, though. Things like permissions and the odd absolute paths could break it.

"Can I point vscode/vscodium to use the same interpreter-and-virtualenv that poetry would?"

Yes.

You can point code/codium's at any python intepreter, including those inseide an existing virtualenv.

And in the end, the virtualenv that poetry creates is just a fully standard python virtualenv.

Most of the work here is figuring out how tell the code/codium extension that does the python running for you about specific virtualenvs.

Which means you might need to tell it that one exists, and then select it (both can be done via Select Interpreter (verify)).

Note that vscode might pick up a virtualenv that is in the project directory -- but by default, poetry puts it in your home directory instead. You can change that behaviour (see the previous item)., and if you mostly use poetry from this context, that can make sense.

"What is poetry shell"

Poetry shell would create a new process with the virtualenv activated.

It also tried to be more helpful across more shells and OSes, and while that is useful, it also meant it was not quite equivalent in ways that are hard to explain, and which led to some subtle bugs.

It's also largely redundant (with poetry run).

So poetry deprecated the shell command, because most things you need it for can be done with poetry run, or with manual activation. (so another separate thing with possibly-slightly-different ).

Questions about workings

is it venv or virtualenv

Both exist, and poetry specifically uses virtualenv because they are more compatible, and not venv.

It abbreviates it venv sometimes, though, which can be a little confusing.

Warnings and errors

"The current project's supported Python range is not compatible with some of the required packages"

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.