Repository navigation
Deprecate test command #1684
Description
I think there's at least some agreement in #931 that we want to remove the test command. I think we should start by raising deprecation warnings pointing people at tox, the same way we've done for the upload and register commands.
The most likely stumbling block that I see is that I think a huge number of people have created their own test command that invokes their preferred test runner, pytest or whatever. Ideally we'd want to get the deprecation warning to them as well. Hopefully most of them are using TestCommand as their base class, though if we want to get really aggressive about it we could try parsing sys.argv directly.
I think we need to warn in the following situations:
- If the
setup.py testcommand is executed - If
tests_requireis specified - If
aliases.testis specified insetup.cfg
It's likely that at least two of these will be specified, but I think two separate warnings would be useful.
Activity
jaraco commentedon Feb 14, 2019
The pytest-runner project does subclass the test command, and there are projects actively depending on it. I do agree we want to deprecate each, and probably both together.
di commentedon Feb 17, 2019
A bit off-topic, but do we have an overarching issue for deprecating/replacing the various incantations of
setup.py? If not, would probably be good to have this, as well as some documentation about migrating on https://packaging.python.org/.(I can create these issues if they don't already exist)
jaraco commentedon Mar 9, 2019
Not to my knowledge.
native-api commentedon Jul 2, 2019
Is there, really? The feeedback there is basically that
toxis a large dependency and is designed for a different use case thansetup.py test.jayvdb commentedon Jul 3, 2019
This is a great way to kill sdists. (why have sdists if they have the same contents as wheels) How quickly the lesson of sourceforge.net has been forgotten. Or more recently Oracle changing the Term of Use of the Java downloads. IMO setuptools should be moving in the opposite direction, automatically including in sdists the test modules found using
test_requireor identified using pytest dry-run.Anyways, unless setuptools also kills its plugin mechanism, someone will recreate the
testcommand as soon as it is removed fromsetuptools. Maybe that is a good thing. (But if the goal is only to get that stuff out of the setuptools repo because it is hindering setuptools, why not split it off to a separate project now, rather than pretend that tox is a drop in solution)jaraco commentedon Jul 3, 2019
Let me see if I can clarify/illustrate. There are two main concerns of
setup.py test:Tox happens to do both of these in a user-friendly, declarative way, which is what makes it a recommended replacement. The user-experience of "run tox" is comparable to "run setup.py test".
toxdoes bring more sophistication (and thus friction) to the process, which is why it's somewhat not an exact match.Users are welcome to seek out other solutions. I wrote pip-run as one such attempt. It has a different behavior in that it always installs dependencies, so doesn't have the second-run speed improvements seen with setup.py test.
You're welcome to consider that. The biggest issue with
setup.py testnow is that it relies oneasy_installandsetup_requires, both of which are superseded by pip and PEP 517. Another issue is thatsetup.py testhas no mechanism to specify the version of setuptools under which to run... dependencies need to be resolved beforesetup.pyis invoked. Any tools that attempts to use setuptools plugins to run tests will be forever be responsible to ensure the requisite version of setuptools is installed in advance (it's too late by the time setuptools is imported).I'm not sure I follow. Can you elaborate on what the risks are?
I don't think you want to bundle any dependencies, even those for tests, for the same reasons as you don't want to bundle dependencies (you don't want to have to re-release your sdists every time a dependency updates). You want a lightweight, declarative means to declare dependencies. If bundled dependencies are valuable, a higher-level tool should build on the fundamental building blocks (independent packages) to produce and publish those bundles (similar to how Dockerhub works to host materialized environments).
pganssle commentedon Jul 3, 2019
To be clear, this would change nothing about the distinction between sdists and wheels. I personally still recommend that you include your test code in your sdist, but not in the wheel. I also recommend that your
tox.iniand any other test configuration should also go into the sdist.python setup.py testhas raised an error indateutilfor a few years now, but it has not affected the content or utility of the source distributions.jayvdb commentedon Jul 3, 2019
The reason to include tests in sdists, or even create sdists, becomes less obvious if the existing declarative syntax to describe them causes a deprecation warning. This will obviously mean packages with existing working metadata which describes their tests to sdist recipients will need to remove that metadata to be compatible with the latest setuptools. And the latest setuptools wont run the tests any more ... ? So again, why would those package maintainers bother with including tests in the sdist.
Add to that there are members of PyPA who were already pushing projects to not include tests in sdists (because GitHub!), who will use this deprecation as additional justification for being more vocal and aggressive in their stance.
These are not direct outcomes of this issue. However, they are fairly predictable side-effects.
Great, but you are not the typical package maintainer. Unfortunately most sdists these days are not testable, usually missing test configuration or test data, until someone from a distro tries to get the sdist working, and then has to try to convince the package maintainer because the Python Packaging Guide says almost nothing about tests. Rather than trying to use @pypa resources and products to address that growing problem, it seems the developers are washing their hands of the problem, and write so many conflicting PEPs that replace each other before a significant segment of the user base has adopted the previous one usually because the implementations were all only partial completed before the authors started writing a different PEP. Compare with the Perl ecosystem, where running the tests is an automatic default part of the package installation experience, and buildbots can (and do!) run the tests of almost every package because they all use consistent metadata for finding and running the tests - packages unmodified for 10 years still build. Tox manages creating virtualenvs and running python(OS-agnostic) commands in them. It does not declaratively describe the test modules nor the test runner (I dont consider the
testenvcommands =to be declarative.) If setuptools isnt going to allow its metadata to describeunittesttest modules, maybepytestneeds to become the recommended test runner, as its metadata in setup.cfg/pytest.ini/etc can at least hold the globs needed to know which modules are test modules and which are not.graingert commentedon Jul 3, 2019
102 remaining items
Add Python 3.9, 3.10, 3.11 Support and Fix Compilation Issues (#194)
remove python setup.py test
remove python setup.py test
Remove pytest-runner and deprecated test setup (#453)
Remove use of deprecated 'setup.py test' (chardet#187)
Use pyproject.toml
Update `unittest` and `coverage` usage
meson-python migration + Python 3.12 support + 4 new query strategies (…
Remove use of deprecated 'setup.py test' (chardet#187)
Use more modern pytest invocation
devel/py-pytest-runner: Update version 6.0.0=>6.0.1
www/py-django-prometheus: Convert to PEP517
www/py-dj50-django-prometheus: Convert to PEP517
www/py-django-prometheus: Convert to PEP517
www/py-dj50-django-prometheus: Convert to PEP517