Skip to content

Normalisation and Requirement's str() #644

Open

Description

@henryiii

If packaging doesn't normalise dependency names when stringifying dependencies, why does it normalise their extras?

Originally posted by @layday in pypa/build#550 (comment)

Build's test suite is broken because packaging 22 is normalizing the extras when producing strings. But it doesn't do this with dependency names. It seems like it should be consistent.

Activity

  1. brettcannon commented on Dec 13, 2022

    @brettcannon
    Member

    This is actually consistent in the face of https://peps.python.org/pep-0685/ requiring we normalize extras, but nothing requiring we normalize project names. We could potentially normalize project names as well, but that might break code that wants the original string.

    @pradyunsg opinions?

  2. henryiii commented on Dec 13, 2022

    @henryiii
    ContributorAuthor

    The PEP requires the names to be normalized when comparing, it doesn't say they have to be changed to a normalized string by packaging, though, as far as I can tell.

    Maybe this:

    For tools writing core metadata, they MUST write out extra names in their normalized form.

    This seems problematic in the short term, though, as previous versions don't consider these extra matching. I think it would be better to provide a transition period where extras can be written as specified, and comparison is normalized, before also normalizing the writing. Projects that historically have a a_b extra will suddenly break if the extra gets rewritten to a-b (which I've seen happen when projects were moving to hatchling, by the way - see wntrblm/nox#659 (comment) , where tox_to_nox was being rewritten to tox-to-nox, breaking all workflows with nox[tox_to_nox]` until comparisons are supported in pip and older pips are not in use. Hatchling has an emergency "allow-ambiguous-features" switch that we needed to use to make sure users were not broken).

  3. layday commented on Dec 13, 2022

    @layday
    Member

    The latest version of Requirement implements __eq__ but this isn't very useful if the components (the name and extras) aren't normalised for comparison:

    In [1]: from packaging.requirements import Requirement
    
    In [2]: Requirement('foo') == Requirement('foo')
    Out[2]: True
    
    In [3]: Requirement('foo') == Requirement('Foo')
    Out[3]: False
    
    In [4]: Requirement('foo[foo_bar]') == Requirement('foo[foo_bar]')
    Out[4]: True
    
    In [5]: Requirement('foo[foo_bar]') == Requirement('foo[foo-bar]')
    Out[5]: False
    

    Normalising the marker does appear to be required by the PEP. I think it'd be beneficial to review how requirements are represented and compared, e.g.:

    • __str__ should retain the original representation;
    • __eq__ should normalise all components;
    • a new property normalized_str could produce a normalised representation of the requirement - pipe it back into the constructor to get a requirement with all constituent members normalised.
  4. henryiii commented on Dec 14, 2022

    @henryiii
    ContributorAuthor

    I believe this change might be related to pypa/pip#11649 ? Specifically, the latest jupyterlab-server from six days ago is now requesting a normalized extra name jsonschema[format-nongpl] while packages built with older packaging are still requesting jsonschema[format_nongpl], causing a collision.

  5. pradyunsg commented on Dec 17, 2022

    @pradyunsg
    Member

    @pradyunsg opinions?

    I think we should preserve them when we load them and represent them as-is; but compare them with normalisation. Basically, what @layday said. :)

  6. pradyunsg commented on Dec 17, 2022

    @pradyunsg
    Member

    I'm pretty sure a PR would be welcome (and, unless @brettcannon feels differently, likely merged subject to regular code reviews). :)

  7. brettcannon commented on Dec 20, 2022

    @brettcannon
    Member

    I'm pretty sure a PR would be welcome (and, unless @brettcannon feels differently, likely merged subject to regular code reviews). :)

    I was thinking about this today and I think there's a risk with the change, but one I think we an live with (i.e. I would be fine with a PR that changes the string representation to preserve the original string). Essentially it sucks when comparisons are non-obvious; the string you see will not be the string that comparisons are being made against. So we just need to be prepared to deal with that usability concern.

    Hopefully as PEP 685 gets adopted this will become less of an issue.

  8. pradyunsg commented on Feb 13, 2023

    @pradyunsg
    Member

    We could also add a method that gives you a normalised form of the requirement. Unlike versions, this isn’t defined directly by the PEPs but we have clearly defined semantics for this?

  9. uranusjr commented on Feb 13, 2023

    @uranusjr
    Member

    but we have clearly defined semantics for this

    Does this mean to normalise the name, extras, versions in specifiers? Maybe additionally sort the extras and specifiers?

  10. pradyunsg commented on Feb 13, 2023

    @pradyunsg
    Member

    Yes, pretty much. Get the requirement into a string that can be compared with the same semantics as a requirement equality.

  11. changed the title [-]Stringifying change in 22.0[/-] [+]Normalisation and Reqirement's `str()`[/+] on Feb 13, 2023
  12. 5 remaining items

  13. brettcannon commented on Jun 1, 2023

    @brettcannon
    Member

    I'm personally happy with making equality handle the normalization and not care about the string representation. And if we construct a tuple for everything in the end then we can reuse that tuple for hashing.

  14. added a commit that references this issue on Jun 2, 2023
    53248fe
  15. astrojuanlu commented on Jun 2, 2023

    @astrojuanlu
    Contributor

    Opened a pull request #696 let me know what you think.

  16. added a commit that references this issue on Jun 26, 2023
    da9380a
  17. layday commented on Jul 7, 2023

    @layday
    Member

    IIUC #696 only canonicalises the name - shouldn't extras be canonicalised as well for comparison?

    >>> Requirement('foo[foo_bar]') == Requirement('foo[foo-bar]')
    False
    

    The originally reported issue hasn't been addressed either - extra markers are canonicalised when the requirement is stringified but names aren't (nor are their extras):

    >>> str(Requirement('scikit_learn; extra == "foo_bar"'))
    'scikit_learn; extra == "foo-bar"'
    

    Can we reopen this?

  18. brettcannon commented on Jul 7, 2023

    @brettcannon
    Member

    @layday reopened.

  19. abravalheri commented on Oct 22, 2024

    @abravalheri
    Contributor

    Would it be possible to have at least a separated method that implements this normalisation (not necessarily str)?

    That would facilitate a lot when writing METADATA files...

    For example nowadays, we can use str(...), but it does not help when trying to write the requirement with the normalised extra (as required by PEP 685):

    $ pip-run 'packaging==24.1'
    >>> from packaging.requirements import Requirement
    >>> req = Requirement('mariadb>=1.0.1; python_version >= "3" and extra == "mariadb_connector"')
    >>> str(req)
    'mariadb>=1.0.1; python_version >= "3" and extra == "mariadb_connector"'

    (Optionally, this hypothetical method could also decide to normalise the requirement name, but that is not mandated by any PEP yet I believe).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions