Does pyconfigure even make sense anymore?

The ./bootstrap.sh, ./configure and make steps are pretty unusual for a Python application. These are based on pyconfigure by Brandon Invergo.

I suspect we may be able to add a lot of clarity and remove a significant barrier to contribution by looking at an alternative, but I'm not fully across the background here.

These steps currently:

  • update git submodules (bootstrap.sh)
  • looks for a python interpreter and virtualenv package
  • make a virtualenv
  • installs python dependencies into the virtualenv (./bin/python setup.py develop --upgrade) - why does it reinstall things you already have a system site packages?
  • runs devtools/update_extlib.sh to install Bower and then bower install jquery etc.
  • build translations
  • builds texinfo docs (broken and we don't use anyway)

That's all pretty unusual for a Python application, and adds a lot of fairly dense build config/infrastructure.

The translations make sense, and it's nice that the system can find the python interpreter, but beyond that these tools aren't normally used to create virtualenvs and install Python/JS deps. That's a bit unusual.

Noting that the last release of pyconfigure was 0.2.3 in 2016. The pyconfigure website says that the project has been decommissioned.

Installing could be a compelling feature, if we're not installing as a Python package. But maybe a python package would be easier.

I wonder if it's worth chatting to Brandon Invergo about why it was decommissioned?

Assigned to
a month ago
25 days ago
No labels applied.

~sturm a month ago

This issue has become more urgent to address now due to autoconf version issues we're starting to see. See #12.

~sturm 25 days ago

I asked for an opinion on #pypa IRC about packaging applications:

  • sturm I'm working on packaging for the MediaGoblin project - which is a Python web app. I think it makes sense to distribute a PyPI package, but we have both translations to compile and some third-party JS libs to bundle. Conceptually, would we we pre-build the translations and JS bundle, then include them as data files for subsequent python -m build? There's no way to add extra steps in python -m build right? It's
  • sturm just for preparing the final distribution?
  • webknjaz You need to learn about PEP517 in-tree build backends — this will allow you to add some code when producing artifacts.
  • sturm webknjaz: great, thanks I'll go do some reading
  • webknjaz OTOH it's questionable that you really need to distribute your app. Let me check what that is
  • sturm webknjaz: I'd be very interested in your thoughts on whether a PyPI package is even a good idea. It seems like a grey area for applications.
  • webknjaz Yep. I think it heavily depends on the application and the intended use of it.
  • sturm webknjaz: any rules of thumb you'd suggest in deciding?
  • webknjaz Well, libs and frameworks go to PyPI, and they have open-ended direct (usually only direct) deps specified.
  • webknjaz For apps, it's different. If you expose some sort of a CLI (or runpy) executable, you could probably use PyPI as well.
  • webknjaz But normally apps need stricter env specification.
  • webknjaz Like exact pins of the whole dependency tree (including transitive deps).
  • sturm stricter? as in only targeting a particular platform?
  • sturm ah I see
  • sturm we've been trying to keep pins very loose to hopefully ease distro packaging
  • webknjaz Normally, this is a task for virtualenvs and requirements+constraints files, and the assumption that the users probably wouldn't attempt to install incompatible deps/packages into the said venv
  • sturm that's a good point though, your existing PyPI release could break at any moment if a new incompatible dependency was released and you hadn't fully pinned it down.
  • webknjaz That's why you mostly need to package a virtualenv (or something similar).
  • webknjaz And there's a lot of tools that solve this problem in different (although similar) ways
  • webknjaz There's zipapps (natively supported by the Python) interpreter
  • webknjaz And tools like shiv or pex for creating them
  • webknjaz Also, there's things like py2exe and similar, that produce OS-specific bundles even including a separate interpreter copy sometimes
  • sturm ok, thanks - I'll look into that
  • webknjaz FYI we maintain this documentation website https://packaging.python.org/
  • webknjaz You should be able to find more pointers there
  • sturm is the main hassle you'd expect for distributing an app on PyPI the problem of getting the many deps to align/remain compatible?
  • sturm or are there other issues too?
  • webknjaz Also, here's a few more specific pointers https://packaging.python.org/overview/#packaging-python-applications https://packaging.python.org/discussions/deploying-python-applications/
  • sturm great, thanks
  • webknjaz As for your last question: when you publish an open-ended thing, be prepared to continuously maintain/test it as the dependency tree will be changing over time (each new user install will attempt to download the latest deps, not those you tested against at the time of the publication)
  • webknjaz Also, you'll be shifting the responsibility of managing the transitive dep tree pins in the installation locations to the end users.
  • webknjaz I guess, if the end-users are pythonistas, it's not very problematic, just point them to pip-tools as an example solution. But you may end up having to educate folks who don't understand that they'd need to manage a proper venv. OTOH now that we've adopted pipx, it may ease some of the issues with attempting to install incompatible things.
  • sturm right, makes sense. Is there any particular reason that dropping all the transitive deps into setup.py/cfg would be a bad idea?
  • sturm (aside from that it's not generally done)
  • webknjaz Well, normally, the right place for this is requirements. And pip-tools is not able to generate them in setup.cfg/py
  • webknjaz But it may be fine if you tell the users to install with pipx
  • sturm webknjaz: thanks very much, that's all really helpful - appreciate your time :)
  • webknjaz Another trade-off with the pinned deps in the libs is that it would be hard to get security fixes somewhere deep in the tree. Every time you go this route, you will need to have a mechanism in place to make new releases.
  • webknjaz You're welcome :)
Register here or Log in to comment, or comment via email.