You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Build's test suite is broken because packaging 22 is normalizing the extras when producing strings. But it doesn't do this with dependency names. It seems like it should be consistent.
This is actually consistent in the face of https://peps.python.org/pep-0685/ requiring we normalize extras, but nothing requiring we normalize project names. We could potentially normalize project names as well, but that might break code that wants the original string.
The PEP requires the names to be normalized when comparing, it doesn't say they have to be changed to a normalized string by packaging, though, as far as I can tell.
Maybe this:
For tools writing core metadata, they MUST write out extra names in their normalized form.
This seems problematic in the short term, though, as previous versions don't consider these extra matching. I think it would be better to provide a transition period where extras can be written as specified, and comparison is normalized, before also normalizing the writing. Projects that historically have a a_b extra will suddenly break if the extra gets rewritten to a-b (which I've seen happen when projects were moving to hatchling, by the way - see wntrblm/nox#659 (comment) , where tox_to_nox was being rewritten to tox-to-nox, breaking all workflows with nox[tox_to_nox]` until comparisons are supported in pip and older pips are not in use. Hatchling has an emergency "allow-ambiguous-features" switch that we needed to use to make sure users were not broken).
The latest version of Requirement implements __eq__ but this isn't very useful if the components (the name and extras) aren't normalised for comparison:
In [1]: from packaging.requirements import Requirement
In [2]: Requirement('foo') == Requirement('foo')
Out[2]: True
In [3]: Requirement('foo') == Requirement('Foo')
Out[3]: False
In [4]: Requirement('foo[foo_bar]') == Requirement('foo[foo_bar]')
Out[4]: True
In [5]: Requirement('foo[foo_bar]') == Requirement('foo[foo-bar]')
Out[5]: False
Normalising the marker does appear to be required by the PEP. I think it'd be beneficial to review how requirements are represented and compared, e.g.:
__str__ should retain the original representation;
__eq__ should normalise all components;
a new property normalized_str could produce a normalised representation of the requirement - pipe it back into the constructor to get a requirement with all constituent members normalised.
I believe this change might be related to pypa/pip#11649 ? Specifically, the latest jupyterlab-server from six days ago is now requesting a normalized extra name jsonschema[format-nongpl] while packages built with older packaging are still requesting jsonschema[format_nongpl], causing a collision.
I'm pretty sure a PR would be welcome (and, unless @brettcannon feels differently, likely merged subject to regular code reviews). :)
I was thinking about this today and I think there's a risk with the change, but one I think we an live with (i.e. I would be fine with a PR that changes the string representation to preserve the original string). Essentially it sucks when comparisons are non-obvious; the string you see will not be the string that comparisons are being made against. So we just need to be prepared to deal with that usability concern.
Hopefully as PEP 685 gets adopted this will become less of an issue.
We could also add a method that gives you a normalised form of the requirement. Unlike versions, this isnât defined directly by the PEPs but we have clearly defined semantics for this?
I'm personally happy with making equality handle the normalization and not care about the string representation. And if we construct a tuple for everything in the end then we can reuse that tuple for hashing.
The originally reported issue hasn't been addressed either - extra markers are canonicalised when the requirement is stringified but names aren't (nor are their extras):
>>> str(Requirement('scikit_learn; extra == "foo_bar"'))
'scikit_learn; extra == "foo-bar"'
Would it be possible to have at least a separated method that implements this normalisation (not necessarily str)?
That would facilitate a lot when writing METADATA files...
For example nowadays, we can use str(...), but it does not help when trying to write the requirement with the normalised extra (as required by PEP 685):
$ pip-run'packaging==24.1'>>>frompackaging.requirementsimportRequirement>>>req=Requirement('mariadb>=1.0.1; python_version >= "3" and extra == "mariadb_connector"')
>>>str(req)
'mariadb>=1.0.1; python_version >= "3" and extra == "mariadb_connector"'
(Optionally, this hypothetical method could also decide to normalise the requirement name, but that is not mandated by any PEP yet I believe).
Activity
brettcannon commentedon Dec 13, 2022
This is actually consistent in the face of https://peps.python.org/pep-0685/ requiring we normalize extras, but nothing requiring we normalize project names. We could potentially normalize project names as well, but that might break code that wants the original string.
@pradyunsg opinions?
henryiii commentedon Dec 13, 2022
The PEP requires the names to be normalized when comparing, it doesn't say they have to be changed to a normalized string by packaging, though, as far as I can tell.
Maybe this:
This seems problematic in the short term, though, as previous versions don't consider these extra matching. I think it would be better to provide a transition period where extras can be written as specified, and comparison is normalized, before also normalizing the writing. Projects that historically have a
a_bextra will suddenly break if the extra gets rewritten to a-b (which I've seen happen when projects were moving to hatchling, by the way - see wntrblm/nox#659 (comment) , wheretox_to_noxwas being rewritten totox-to-nox, breaking all workflows with nox[tox_to_nox]` until comparisons are supported in pip and older pips are not in use. Hatchling has an emergency "allow-ambiguous-features" switch that we needed to use to make sure users were not broken).layday commentedon Dec 13, 2022
The latest version of
Requirementimplements__eq__but this isn't very useful if the components (the name and extras) aren't normalised for comparison:Normalising the marker does appear to be required by the PEP. I think it'd be beneficial to review how requirements are represented and compared, e.g.:
__str__should retain the original representation;__eq__should normalise all components;normalized_strcould produce a normalised representation of the requirement - pipe it back into the constructor to get a requirement with all constituent members normalised.henryiii commentedon Dec 14, 2022
I believe this change might be related to pypa/pip#11649 ? Specifically, the latest
jupyterlab-serverfrom six days ago is now requesting a normalized extra namejsonschema[format-nongpl]while packages built with older packaging are still requestingjsonschema[format_nongpl], causing a collision.pradyunsg commentedon Dec 17, 2022
I think we should preserve them when we load them and represent them as-is; but compare them with normalisation. Basically, what @layday said. :)
pradyunsg commentedon Dec 17, 2022
I'm pretty sure a PR would be welcome (and, unless @brettcannon feels differently, likely merged subject to regular code reviews). :)
brettcannon commentedon Dec 20, 2022
I was thinking about this today and I think there's a risk with the change, but one I think we an live with (i.e. I would be fine with a PR that changes the string representation to preserve the original string). Essentially it sucks when comparisons are non-obvious; the string you see will not be the string that comparisons are being made against. So we just need to be prepared to deal with that usability concern.
Hopefully as PEP 685 gets adopted this will become less of an issue.
pradyunsg commentedon Feb 13, 2023
We could also add a method that gives you a normalised form of the requirement. Unlike versions, this isnât defined directly by the PEPs but we have clearly defined semantics for this?
uranusjr commentedon Feb 13, 2023
Does this mean to normalise the name, extras, versions in specifiers? Maybe additionally sort the extras and specifiers?
pradyunsg commentedon Feb 13, 2023
Yes, pretty much. Get the requirement into a string that can be compared with the same semantics as a requirement equality.
[-]Stringifying change in 22.0[/-][+]Normalisation and Reqirement's `str()`[/+]5 remaining items
brettcannon commentedon Jun 1, 2023
I'm personally happy with making equality handle the normalization and not care about the string representation. And if we construct a tuple for everything in the end then we can reuse that tuple for hashing.
Canonicalize names for requirements comparison
astrojuanlu commentedon Jun 2, 2023
Opened a pull request #696 let me know what you think.
Note change from pkg_requirements
Note change from pkg_requirements
Note change from pkg_requirements
Note change from pkg_requirements
micropkg packaging improvements (#2614)
Canonicalize names for requirements comparison
Canonicalize names for requirements comparison
layday commentedon Jul 7, 2023
IIUC #696 only canonicalises the name - shouldn't extras be canonicalised as well for comparison?
The originally reported issue hasn't been addressed either - extra markers are canonicalised when the requirement is stringified but names aren't (nor are their extras):
Can we reopen this?
brettcannon commentedon Jul 7, 2023
@layday reopened.
abravalheri commentedon Oct 22, 2024
Would it be possible to have at least a separated method that implements this normalisation (not necessarily
str)?That would facilitate a lot when writing
METADATAfiles...For example nowadays, we can use
str(...), but it does not help when trying to write the requirement with the normalised extra (as required by PEP 685):(Optionally, this hypothetical method could also decide to normalise the requirement name, but that is not mandated by any PEP yet I believe).