Repository navigation
fix: to_arrow for unions whose option-type children are longer than the used length - #4398
Conversation
…he used length UnionArray._to_arrow passed index.max() + 1 to each child while option children built validity bytes at their own length. Slice each child to the length the union uses, and size the validity scatter to match. Fixes #4228
ianna
left a comment
There was a problem hiding this comment.
Re-reviewed after the merge of main, at commit c235d12795240f57c36cdc0c4a218acbbf414a40. No actionable findings; approving.
The change correctly aligns child lengths, scattered validity masks, and indices after copying repeated elements. The PR diff is unchanged from the earlier review, and the combined conversion path works correctly with main's nullable-list fix from #4407.
Validation on the updated revision with Python 3.12 and PyArrow 26:
- 165 tests passed, including the #4228 and #4222 regression suites, related conversion tests, and Arrow/Feather/Parquet property tests.
- The four Arrow property tests passed with 1,000 examples each and the
_has_issue_4228exclusion temporarily disabled. - 400 additional value comparisons passed combining sliced nullable lists and masked unions, with extension arrays and null counting enabled and disabled.
The earlier review also confirmed that all six #4228 regression cases fail with the original implementation.
Non-blocking follow-up: remove the _has_issue_4228 property-test exclusion so these cases remain exercised in CI.
I did not rerun the full suite or nightly Hypothesis profile. CI was still running when checked during re-review.
This review was prepared with assistance from OpenAI Codex.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
|
Exercise copied-child lengths both above and below the original index bound, with extension arrays enabled and disabled. AI assistance: OpenAI Codex.
ianna
left a comment
There was a problem hiding this comment.
@wolfgang-aura -- Thanks for contributing! I'm approving the PR. All tests pass, the patch coverage is 100%. The doc failure is Python 3.11.0 + Soup Sieve 3.0.1, unrelated to the union fix or new test. It will be fixed with a separate PR. Thanks!
|
The docs workflow should be fixed in #4421 and when it's merged this PR should be re-based. |
|
@all-contributors please add @wolfgang-aura for code. |
|
I've put up a pull request to add @wolfgang-aura! 🎉 |
|
The nightly test failure on 2026-11-05 is because of #4228 . The predicate |
|
The documentation preview is ready to be viewed at https://preview.awkward-array.org/PR4398/ |
…he used length (#4398)
Fixes #4228.
UnionArray._to_arrowconverts each child withlength = index.max() + 1, but option-type children build their validity bytes at their own length. When a child is longer than the part the union uses, the two lengths disagree. A valid layout like the one in the issue then raisesIndexError: boolean index did not match indexed array, and sliced unions such asak.Array([1, "x", None])[:0]raiseArrowInvalid: Null count exceeds array length.The change, in
src/awkward/contents/unionarray.py:this_lengthbefore the validity bytes are scattered, and size the scatter buffer to it;this_lengthto that copy's length;this_lengthbefore calling_to_arrowon it.Tests are in a new
tests/test_4228_to_arrow_union_option_child_length.py: the explicit layout from the issue, a skipped index under an option, and the four sliced unions from the follow-up comment. All six fail without the change and pass with it. The fulltests/suite gives 5404 passed, 99 skipped, 2 failed; the two failures aretest_1294test_dictionary_encoding[*-through_arrow-True], which fail the same way onmainwith pyarrow 25.0.0.The
_has_issue_4228skip intests/properties/operations/test_to_from_arrow.pyis left in place. It can probably go now; that is easier to judge with the property tests in view.AI disclosure: this change was written by an AI agent (Claude Code, claude-sonnet-5-5) and reviewed by a second agent run. The tests were then moved into a new file per AGENTS.md and the full suite was run before submitting.