Poetry notes

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

These are some specific notes about everyday use of python's poetry.

To see it in a wider packagey-context, see Venvs_and_packaging_-_Python#Python_virtual_environments.

What and why

From a practical sense, poetry has a few useful things going for it:

• as a better dependency resolver than pip

it will reconsider dependencies, not just add things and hope for the best

• as an convenient manager of a virtualenv's libraries

it does the pip installs inside it based on its dependency resolving - often less typing to install and/or run

• as project manager, in the "easier to reproduce versions between devs, and build where necessary" sense

uses pyproject.toml (standard), and its own poetry.lock to let you lock to specific versions

• as packager in the "...at which point we basically create something we can upload to pypi" sense

you can ignore this when you are only using it to reproduce environments

• ...which also as a way to reproduce environments in clusters and such (though things like uv may be a little more suited)

TODO: some comparison

Start from scratch

If you have existing code without poetry and have poetry create a new (mostly a pyptoject.toml) there

poetry init

Or create a fresh project directory with pyproject.toml and some basic files your new python project might need (or that you might throw away)

poetry new projname

Either way, you'll have to answer some questions.

(If you have a project with pyptoject.toml, move on to #poetry_install)

Either way you now have a poetry environment (meaning project-like metadata).

That will soon also create-and-manage a virtualenv (fitting the description in that project metadata).

You can, for example,

• add a library to that virualenv

poetry add lib

• run a script with that virtualenv

poetry run python myscript.py

Useful questions

Managing and installing

"What is package mode?"

💤 For context

• there is a concept of 'root package' :

the project defined by pyproject itself is called the root package

while everything installed into it is considered dependencies (for that root package)

This is more than just a technicality, in that

• when you use poetry just to create an environment to run stuff in, you might care about only the dependencies, and never about the root package

package-mode = true means

you are saying the point of this project is to build a package (sdist or wheel), e.g. to

create a package for someone to install

upload a package

test that builds run everywhere - see the idea of reproducible builds

package-mode = false means

you are saying that the packaging-specific part of project config is unimportant, and are e.g.

or just to use poetry for dependency management

often to

just run someone else's unpackaged code

or further develop that code and run it (and worry about packaging later, or not at all by just suggesting patches to them)

There is one potentially important footnote to that:

If you want edits to your project source source to be immediately reflected inside the venv you are creating, then you want the editable install of the root package that only package-mode = true will give you.

If you only care to poetry run, you're fine either way.

This relates to the fact that the venv is made to stand entirely independently, whereas poetry run will reach inside your project.

"What does poetry install do?"

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

If package-mode = true

• installs dependencies (into venv)
• installs root package (into venv), editable-install style

If package-mode = false,

• installs dependencies (into venv)
• root package is ignored

If package-mode=true but you run poetry install --no-root, you also get the latter (dependencies-only ignoring the root package)

--no-root used to be more valuable before package-mode was an explicit concept in poetry

now it's mostly useful for "someone else's pyproject came with package mode on but I quickly want to install only deps"

(Note that you may have reasons beyond install to change or leave package-mode)

See also https://python-poetry.org/docs/basic-usage/#operating-modes

"How do lockfiles relate?"

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

For context, you can think of lockfiles as containing a specific solution of the dependencies listed in the project. Most of the time, a lockfile is usually altered when you add, remove, update, or lock

Since poetry install is only meant to install what is listed, it wouldn't need to write it. lockfiles will get read by poetry install if they are there.

...except that covers only common cases. More precisely, it is possible for you to have edited pyproject manually and to not have run update as you should, and poetry still does the right thing. That ends up being:

• if lockfile is there there and in sync with the dependencies listed in pyproject,

installs the exact versions listed in the lockfile (and doesn't write lockfile)

• if lockfile is there there but not in sync with the dependencies listed in pyproject,

does deperency resolution, installs what it ended up on, writes lockfile

• If lockfile it isn't there

does deperency resolution, installs what it ended up on, writes lockfile

Note that none of this is affected by whether package-mode is on or off. (a missing or out-of-date lockfile will always get rewritten, whether package-mode is on or off)

"Can I have my project's root package be live in my virtualenv, editable-install style?"

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Running

"Where does it get python versions from?"

After the virtualenv is created, it gets the python from inside the virtualenv (that's part of the point of a virtualenv).

When creating the virtualenv, it's roughly wherever poetry can find it, including:

• active interpreter (global, homebrew, pyenv, active virtualenv)
• pyenv-installed (even if not active?(verify))
• explicitly pointing it to one, e.g. poetry env use /path/to/a/python
• previously (poetry-created) existing virtualenvs(verify)

It won't install new python versions on request -- but if you have pyenv set up, you can do that yourself fairly easily.

You can tell it to do things without a virtualenv (including installing into it - if it has the permissions(verify)).

"What does poetry run actually do?"

You can think of poetry run as

finds the suitable virtualenv,

activating that virtualenv (which puts its bin/, i.e. python, pip, and such first on the PATH)

runs whatever command you asked for

In fact, at the core it seems

little more than a subprocess.Popen

in which it mainly just changes the (inherited) environment to changing:

• PATH - so that the virtualenv's bin comes first so that use of python, pip inside should work
• PYTHON_PATH - so that module paths get prepended to sys.path
• VIRTUAL_ENV - helps clarify that you are using a virtualenv (because the above two )

and yes, it does this even when the thing you run isn't python directly.

"Can I run other scripts using this virtualenv?"

Yes.

That virtualenv is a fully standard one, poetry only helps manage its contents.

You can activate and use it as any other.

Note that it is sometimes less typing to do activate-then-single-script-run via:

poetry run python /path/to/env/with/script.py

⚠ That said, a common-enough confusion is that

poetry run python /path/to/env/with/script.py means something very different from
poetry run /path/to/env/with/script.py

• In the first case, poetry will wrap the environment, then run the virtualenv's python executable (because activating put it first);
• In the second case, poetry will wrap the environment, but the system and the shebang that determine which python executable gets used -- which need not be the python in the virtualenv

...or even python at all (which in the case of wrapper scripts may even be what you want - but usually isn't).

"How does poetry find things?"

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Finding the project directory

From the current directory, start walking to parents.

The first directory that contains a pyproject.toml is considered a project root (so yes, it depends on cwd).

Finding the virtualenv

The virtualenv is found in association to the project directory, but there is some flexibility.

By default, all poetry-created venvs are placed in the user directory of each user that uses that pyproject (that pyproject is only the instructions)

Optionally, the virtualenv can also be placed inside the project directory

It would not be picked up unless you ask to look for it.

(It is also not shared until a few more conditions are met, and you may not want that)

It may fall back on an already-existing virtualenv that happened to refer to this directory (matching the project name and local absolute path -- the projectname-hash-pyversion entries refer to the hash of the absolute path, so that the same project in different places cannot collide)

Note that in theory, you can tell poetry to work in any environment, including the system python (see virtualenvs.create false) but you usually don't want to do that.

"Do I get my own virtualenv, do / can I share it?"

By default, your own:

every distinct user each user can and will generate a virtualenv

which will be written to their own home directory.

Poetry treats the pyproject.toml and/or lockfile only as instructions on how to build that (and the default place that goes to makes each user unique impicitly)

Alternatively, poetry can be told to, instead, create a .venv/ inside the project directory.

See poetry config virtualenvs.in-project true.

Notes:

• Note that even if an .venv exists because someone else set virtualenvs.in-project, your poetry will not pick that up (and instead still work within your userdir) until you set virtualenvs.in-project too.(verify)

• virtualenvs.in-project is a global(-only) setting that only tells poetry to create new virtualenvs differently (and look in a new place).

In particular, it will not move an already-created virtualenv from your ~/.cache to the project directory. The closest you have is to delete the applicable virtualenv from your profile, then poetry install again to recreate it, now in the newly intended place. (to find which venv to remove, try poetry env info or, if not activated, poetry env list), choose the correct one, and then e.g. poetry env remove myproject-rGWZLGg-py3.10

Generally discouraged, though. Things like permissions and the odd absolute paths could break it.

"Can I point vscode/vscodium to use the same interpreter-and-virtualenv that poetry would?"

Yes.

You can point code/codium's at any python intepreter, including those inseide an existing virtualenv.

And in the end, the virtualenv that poetry creates is just a fully standard python virtualenv.

Most of the work here is figuring out how tell the code/codium extension that does the python running for you about specific virtualenvs.

Which means you might need to tell it that one exists, and then select it (both can be done via Select Interpreter (verify)).

Note that vscode might pick up a virtualenv that is in the project directory -- but by default, poetry puts it in your home directory instead. You can change that behaviour (see the previous item)., and if you mostly use poetry from this context, that can make sense.

"What is poetry shell"

Poetry shell would create a new process with the virtualenv activated.

It also tried to be more helpful across more shells and OSes, and while that is useful, it also meant it was not quite equivalent in ways that are hard to explain, and which led to some subtle bugs.

It's also largely redundant (with poetry run).

So poetry deprecated the shell command, because most things you need it for can be done with poetry run, or with manual activation. (so another separate thing with possibly-slightly-different ).

Warnings and errors

"The current project's supported Python range is not compatible with some of the required packages"

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.