Skip to content

Failure output order of dictionary keys is alphabetical instead of insertion order #13503

Closed
#14050
daniell-olaitan/pytest
#1
Closed
Failure output order of dictionary keys is alphabetical instead of insertion order#13503
#14050
daniell-olaitan/pytest
#1
Assignees
ritvi-alagusankar

Description

@mcarans
mcarans
opened on Jun 9, 2025
  1. Create a file like this:
class TestDictOrder:
    def test_dict_order(self):
        a = {
            "Existing Hash": "",
            "Existing Modified": "",
            "Existing Size": "",
            "Existing Broken": "",
            "Set Broken": "N",
            "Head Status": "",
            "Head Error": "",
            "Get Status": "",
            "Get Error": "",
            "New ETag": "",
            "ETag Changed": "",
            "New Modified": "",
            "Modified Changed": "",
            "Modified Newer": "",
            "Modified Value": "",
            "New Size": "",
            "Size Changed": "",
            "New Hash": "",
            "Hash Changed": "",
            "Update": "N",
        }
        print(a)
        assert a == {}
  1. run pytest with -vv
  2. The keys in the output failure message are not in insertion order (the guaranteed order since Python 3.7) but are displayed alphabetically instead (presumably because that's easier to implement?). I think there should be an option to have the output order be insertion order (or it should be the standard output order) as that can be helpful with debugging large and nested dictionaries and is the order when you print the dictionary.

In the output from pytest below, "Left contains 20 more items:" and "Full diff:" are alphabetical instead of insertion order:

1 > pytest -vv test_dict_order.py 
=================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.13.3, pytest-8.4.0, pluggy-1.6.0 -- /home/mcarans/Code/VirtualEnvs/scratch/bin/python
cachedir: .pytest_cache
rootdir: /home/mcarans/Code/scratch
plugins: typeguard-4.4.2
collected 1 item                                                                                                                                                                                                                                                                          

test_dict_order.py::TestDictOrder::test_dict_order FAILED                                                                                                                                                                                                                           [100%]

======================================================================================================================================== FAILURES =========================================================================================================================================
______________________________________________________________________________________________________________________________ TestDictOrder.test_dict_order ______________________________________________________________________________________________________________________________

self = <test_dict_order.TestDictOrder object at 0x7adebb01f890>

    def test_dict_order(self):
        a = {
            "Existing Hash": "",
            "Existing Modified": "",
            "Existing Size": "",
            "Existing Broken": "",
            "Set Broken": "N",
            "Head Status": "",
            "Head Error": "",
            "Get Status": "",
            "Get Error": "",
            "New ETag": "",
            "ETag Changed": "",
            "New Modified": "",
            "Modified Changed": "",
            "Modified Newer": "",
            "Modified Value": "",
            "New Size": "",
            "Size Changed": "",
            "New Hash": "",
            "Hash Changed": "",
            "Update": "N",
        }
        print(a)
>       assert a == {}
E       AssertionError: assert {'Existing Hash': '', 'Existing Modified': '', 'Existing Size': '', 'Existing Broken': '', 'Set Broken': 'N', 'Head Status': '', 'Head Error': '', 'Get Status': '', 'Get Error': '', 'New ETag': '', 'ETag Changed': '', 'New Modified': '', 'Modified Changed': '', 'Modified Newer': '', 'Modified Value': '', 'New Size': '', 'Size Changed': '', 'New Hash': '', 'Hash Changed': '', 'Update': 'N'} == {}
E         
E         Left contains 20 more items:
E         {'ETag Changed': '',
E          'Existing Broken': '',
E          'Existing Hash': '',
E          'Existing Modified': '',
E          'Existing Size': '',
E          'Get Error': '',
E          'Get Status': '',
E          'Hash Changed': '',
E          'Head Error': '',
E          'Head Status': '',
E          'Modified Changed': '',
E          'Modified Newer': '',
E          'Modified Value': '',
E          'New ETag': '',
E          'New Hash': '',
E          'New Modified': '',
E          'New Size': '',
E          'Set Broken': 'N',
E          'Size Changed': '',
E          'Update': 'N'}
E         
E         Full diff:
E         - {}
E         + {
E         +     'ETag Changed': '',
E         +     'Existing Broken': '',
E         +     'Existing Hash': '',
E         +     'Existing Modified': '',
E         +     'Existing Size': '',
E         +     'Get Error': '',
E         +     'Get Status': '',
E         +     'Hash Changed': '',
E         +     'Head Error': '',
E         +     'Head Status': '',
E         +     'Modified Changed': '',
E         +     'Modified Newer': '',
E         +     'Modified Value': '',
E         +     'New ETag': '',
E         +     'New Hash': '',
E         +     'New Modified': '',
E         +     'New Size': '',
E         +     'Set Broken': 'N',
E         +     'Size Changed': '',
E         +     'Update': 'N',
E         + }

test_dict_order.py:26: AssertionError
---------------------------------------------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------------------------------------------
{'Existing Hash': '', 'Existing Modified': '', 'Existing Size': '', 'Existing Broken': '', 'Set Broken': 'N', 'Head Status': '', 'Head Error': '', 'Get Status': '', 'Get Error': '', 'New ETag': '', 'ETag Changed': '', 'New Modified': '', 'Modified Changed': '', 'Modified Newer': '', 'Modified Value': '', 'New Size': '', 'Size Changed': '', 'New Hash': '', 'Hash Changed': '', 'Update': 'N'}
================================================================================================================================= short test summary info =================================================================================================================================
FAILED test_dict_order.py::TestDictOrder::test_dict_order - AssertionError: assert {'Existing Hash': '', 'Existing Modified': '', 'Existing Size': '', 'Existing Broken': '', 'Set Broken': 'N', 'Head Status': '', 'Head Error': '', 'Get Status': '', 'Get Error': '', 'New ETag': '', 'ETag Changed': '', 'New Modified': '', 'Modified Changed': '', 'Modified Newer': '', 'Modified Value': '', 'New Size': '', 'Size Changed': '', 'New Hash': '', 'Hash Changed': '', 'Update': 'N'} == {}
  
  Left contains 20 more items:
  {'ETag Changed': '',
   'Existing Broken': '',
   'Existing Hash': '',
   'Existing Modified': '',
   'Existing Size': '',
   'Get Error': '',
   'Get Status': '',
   'Hash Changed': '',
   'Head Error': '',
   'Head Status': '',
   'Modified Changed': '',
   'Modified Newer': '',
   'Modified Value': '',
   'New ETag': '',
   'New Hash': '',
   'New Modified': '',
   'New Size': '',
   'Set Broken': 'N',
   'Size Changed': '',
   'Update': 'N'}
  
  Full diff:
  - {}
  + {
  +     'ETag Changed': '',
  +     'Existing Broken': '',
  +     'Existing Hash': '',
  +     'Existing Modified': '',
  +     'Existing Size': '',
  +     'Get Error': '',
  +     'Get Status': '',
  +     'Hash Changed': '',
  +     'Head Error': '',
  +     'Head Status': '',
  +     'Modified Changed': '',
  +     'Modified Newer': '',
  +     'Modified Value': '',
  +     'New ETag': '',
  +     'New Hash': '',
  +     'New Modified': '',
  +     'New Size': '',
  +     'Set Broken': 'N',
  +     'Size Changed': '',
  +     'Update': 'N',
  + }
==================================================================================================================================== 1 failed in 0.03s ====================================================================================================================================
[

Activity

  1. RonnyPfannschmidt commented on Jun 10, 2025

    @RonnyPfannschmidt
    RonnyPfannschmidt
    on Jun 10, 2025
    Member

    Pytest shows a order invariant difference between 2 dicts as dict equality doesn't require the order of items in a dictionary to match

    Its not clear what you sre asking for

  2. RonnyPfannschmidt
    added
    status: needs informationreporter needs to provide more information; can be closed after 2 or more weeks of inactivity
    on Jun 10, 2025

  3. mcarans commented on Jun 10, 2025

    @mcarans
    mcarans
    on Jun 10, 2025
    Author

    @RonnyPfannschmidt Sorry for the lack of clarity. I'm thinking only of the way the differences are displayed rather than changing dict equality.

    assert {"c": 3, "d": 4, "b": 2, "a": 1} == {"d": 4, "c": 3}

    Currently the output is:

      Common items:
  {'c': 3, 'd': 4}
  Left contains 2 more items:
  {'a': 1, 'b': 2}
  
  Full diff:
    {
  +     'a': 1,
  +     'b': 2,
        'c': 3,
        'd': 4,
    }

    I'm suggesting the output would follow the insertion order as far as possible and default to using the order of one of the sides for common items (I have chosen right hand side below) ie.:

      Common items:
  {'d': 4, 'c': 3}
  Left contains 2 more items:
  {'b': 2, 'a': 1}
  
  Full diff:
    {
        'd': 4,
        'c': 3,
  +     'b': 2,
  +     'a': 1,
    }

  4. RonnyPfannschmidt commented on Jun 10, 2025

    @RonnyPfannschmidt
    RonnyPfannschmidt
    on Jun 10, 2025
    Member

    So the request is for the display order to orient on the insert order

    Based on the code it might be a mistake that the full diff is printed

    In any case we need a deep diff to correctly implement this .

    Currently pformat is used to get diff compatible left/right

    Unless a correct deep diff we can apply here is avaliable im against implementing this in pytest

  5. mcarans commented on Jun 10, 2025

    @mcarans
    mcarans
    on Jun 10, 2025
    Author

    @RonnyPfannschmidt pformat has an option sort_dicts which defaults to True. It is described as "If True, dictionaries will be formatted with their keys sorted, otherwise they will be displayed in insertion order"

    Is that what is sorting the keys alphabetically in the pytest output?

    If not, I found a deep diff library called deepdiff. For the example above, it gives:

    pprint(DeepDiff(a,b))
{'dictionary_item_removed': ["root['b']", "root['a']"]}

  6. nicoddemus commented on Jun 12, 2025

    @nicoddemus
    nicoddemus
    on Jun 12, 2025
    Member

    Is that what is sorting the keys alphabetically in the pytest output?

    pformat is used here:

    pytest/src/_pytest/assertion/util.py

    Lines 498 to 541 in 9e9633d

    def _compare_eq_dict(
    left: Mapping[Any, Any],
    right: Mapping[Any, Any],
    highlighter: _HighlightFunc,
    verbose: int = 0,
    ) -> list[str]:
    explanation: list[str] = []
    set_left = set(left)
    set_right = set(right)
    common = set_left.intersection(set_right)
    same = {k: left[k] for k in common if left[k] == right[k]}
    if same and verbose < 2:
    explanation += [f"Omitting {len(same)} identical items, use -vv to show"]
    elif same:
    explanation += ["Common items:"]
    explanation += highlighter(pprint.pformat(same)).splitlines()
    diff = {k for k in common if left[k] != right[k]}
    if diff:
    explanation += ["Differing items:"]
    for k in diff:
    explanation += [
    highlighter(saferepr({k: left[k]}))
    + " != "
    + highlighter(saferepr({k: right[k]}))
    ]
    extra_left = set_left - set_right
    len_extra_left = len(extra_left)
    if len_extra_left:
    explanation.append(
    f"Left contains {len_extra_left} more item{'' if len_extra_left == 1 else 's'}:"
    )
    explanation.extend(
    highlighter(pprint.pformat({k: left[k] for k in extra_left})).splitlines()
    )
    extra_right = set_right - set_left
    len_extra_right = len(extra_right)
    if len_extra_right:
    explanation.append(
    f"Right contains {len_extra_right} more item{'' if len_extra_right == 1 else 's'}:"
    )
    explanation.extend(
    highlighter(pprint.pformat({k: right[k] for k in extra_right})).splitlines()
    )
    return explanation

    Looks like it would be just a matter of passing sort_dicts=False to those pformat calls.

  7. RonnyPfannschmidt commented on Jun 12, 2025

    @RonnyPfannschmidt
    RonnyPfannschmidt
    on Jun 12, 2025
    Member

    We will have to ensure compatible dict ordering in nested structures to ensure minimal diffs

    Order matching dicts in modern python make this easier but its still a caveat

  8. github-actions commented on Jun 27, 2025

    @github-actions
    github-actionsBot
    on Jun 27, 2025 – with GitHub Actions
    Contributor

    This issue is stale because it has the status: needs information label and requested follow-up information was not provided for 14 days.

  9. github-actions
    added
    stale
    on Jun 27, 2025

  10. mcarans commented on Jun 27, 2025

    @mcarans
    mcarans
    on Jun 27, 2025
    Author

    What information do I need to provide to resolve the stale label?

  11. Pierre-Sassoulas
    added
    good first issueeasy issue that is friendly to new contributor
    and removed
    status: needs informationreporter needs to provide more information; can be closed after 2 or more weeks of inactivity
    stale
    on Jun 27, 2025

  12. ritvi-alagusankar commented on Jul 1, 2025

    @ritvi-alagusankar
    ritvi-alagusankar
    on Jul 1, 2025

    Hi, I’d like to work on this issue as my first contribution to the pytest repo.

  13. RonnyPfannschmidt
    assigned
    ritvi-alagusankar
    on Jul 1, 2025

  14. 14 remaining items

  15. RafiB982 commented on Dec 26, 2025

    @RafiB982
    RafiB982
    on Dec 26, 2025

    @mcarans , PR ( #14066 ) created for this issue.

  16. joddeepesh-cloud commented on Dec 27, 2025

    @joddeepesh-cloud
    joddeepesh-cloud
    on Dec 27, 2025

    Hello there i want to take this issue and want to solve it i have understood the problem and have the solution for this .Can you please assign this task to me !!

  17. joddeepesh-cloud
    added a commit that references this issue on Dec 28, 2025

    Add changrlog for issue pytest-dev#13503

    e4c4538
  18. hrideymarwah15
    added 4 commits that reference this issue on Jan 16, 2026

    Fix: Preserve dict insertion order in assertion output

    b8c5717

    Add test for dict insertion order preservation

    1d2980d

    Add changelog entry for issue pytest-dev#13503

    047de26

    Add changelog entry for issue pytest-dev#13503

    80cb861

  19. The-Compiler commented on Jan 18, 2026

    @The-Compiler
    The-Compiler
    on Jan 18, 2026
    Member

    Removing the "good first issue" label here because apparently nobody checks (or cares?) if someone else already started work on the issue, so now we have 6 PRs (if I'm counting correctly) for the same thing.

  20. The-Compiler
    removed
    good first issueeasy issue that is friendly to new contributor
    on Jan 18, 2026
  21. daniell-olaitan
    added a commit that references this issue on Jan 20, 2026

    Preserve dictionary insertion order in assertion output

    e394a8a
  22. nicoddemus
    closed this as completedin #14050on Feb 14, 2026
  23. nicoddemus
    added a commit that references this issue on Feb 14, 2026

    Fix: assertrepr_compare respects dict insertion order (#14050)

    966edc9
  24. patchback
    added a commit that references this issue on Feb 14, 2026

    Fix: assertrepr_compare respects dict insertion order (#14050)

    1a81ee6
  25. nicoddemus
    added a commit that references this issue on Feb 14, 2026

    Fix: assertrepr_compare respects dict insertion order (#14050) (#14193)

    60fb59d
  26. bluetech
    marked Assertion diff output for dicts does not consistently follow insertion order #14079 as a duplicate of this issue on Apr 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions