Skip to content

fix: to_arrow for unions whose option-type children are longer than the used length - #4398

Merged
ianna merged 6 commits into
scikit-hep:mainfrom
wolfgang-aura:wolfgang-aura/union-to-arrow-child-length
Nov 5, 2026
Merged

ianna merged 6 commits into
scikit-hep:mainfrom
wolfgang-aura:wolfgang-aura/union-to-arrow-child-length

Conversation

@wolfgang-aura

Copy link
Copy Markdown
Contributor

Fixes #4228.

UnionArray._to_arrow converts each child with length = index.max() + 1, but option-type children build their validity bytes at their own length. When a child is longer than the part the union uses, the two lengths disagree. A valid layout like the one in the issue then raises IndexError: boolean index did not match indexed array, and sliced unions such as ak.Array([1, "x", None])[:0] raise ArrowInvalid: Null count exceeds array length.

The change, in src/awkward/contents/unionarray.py:

  • compute this_length before the validity bytes are scattered, and size the scatter buffer to it;
  • in the branch that copies the content to its own index, set this_length to that copy's length;
  • slice a child longer than this_length before calling _to_arrow on it.

Tests are in a new tests/test_4228_to_arrow_union_option_child_length.py: the explicit layout from the issue, a skipped index under an option, and the four sliced unions from the follow-up comment. All six fail without the change and pass with it. The full tests/ suite gives 5404 passed, 99 skipped, 2 failed; the two failures are test_1294 test_dictionary_encoding[*-through_arrow-True], which fail the same way on main with pyarrow 25.0.0.

The _has_issue_4228 skip in tests/properties/operations/test_to_from_arrow.py is left in place. It can probably go now; that is easier to judge with the property tests in view.

AI disclosure: this change was written by an AI agent (Claude Code, claude-sonnet-5-5) and reviewed by a second agent run. The tests were then moved into a new file per AGENTS.md and the full suite was run before submitting.

…he used length

UnionArray._to_arrow passed index.max() + 1 to each child while option
children built validity bytes at their own length. Slice each child to
the length the union uses, and size the validity scatter to match.

Fixes #4228
@github-actions github-actions Bot added the type/fix PR title type: fix (set automatically) label Oct 1, 2026
ianna
ianna previously approved these changes Oct 11, 2026

@ianna ianna left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed after the merge of main, at commit c235d12795240f57c36cdc0c4a218acbbf414a40. No actionable findings; approving.

The change correctly aligns child lengths, scattered validity masks, and indices after copying repeated elements. The PR diff is unchanged from the earlier review, and the combined conversion path works correctly with main's nullable-list fix from #4407.

Validation on the updated revision with Python 3.12 and PyArrow 26:

  • 165 tests passed, including the #4228 and #4222 regression suites, related conversion tests, and Arrow/Feather/Parquet property tests.
  • The four Arrow property tests passed with 1,000 examples each and the _has_issue_4228 exclusion temporarily disabled.
  • 400 additional value comparisons passed combining sliced nullable lists and masked unions, with extension arrays and null counting enabled and disabled.

The earlier review also confirmed that all six #4228 regression cases fail with the original implementation.

Non-blocking follow-up: remove the _has_issue_4228 property-test exclusion so these cases remain exercised in CI.

I did not rerun the full suite or nightly Hypothesis profile. CI was still running when checked during re-review.

This review was prepared with assistance from OpenAI Codex.

@codecov

codecov Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.62%. Comparing base (24e8900) to head (465ace8).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files
Files with missing lines Coverage Δ
src/awkward/contents/unionarray.py 88.75% <100.00%> (+1.42%) ⬆️

... and 3 files with indirect coverage changes

Exercise copied-child lengths both above and below the original index bound, with extension arrays enabled and disabled.

AI assistance: OpenAI Codex.

@ianna ianna left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wolfgang-aura -- Thanks for contributing! I'm approving the PR. All tests pass, the patch coverage is 100%. The doc failure is Python 3.11.0 + Soup Sieve 3.0.1, unrelated to the union fix or new test. It will be fixed with a separate PR. Thanks!

@ianna

ianna commented Oct 11, 2026

Copy link
Copy Markdown
Member

The docs workflow should be fixed in #4421 and when it's merged this PR should be re-based.

@ianna

ianna commented Oct 11, 2026

Copy link
Copy Markdown
Member

@all-contributors please add @wolfgang-aura for code.

@allcontributors

Copy link
Copy Markdown
Contributor

@ianna

I've put up a pull request to add @wolfgang-aura! 🎉

@ianna
ianna enabled auto-merge (squash) October 13, 2026 04:01
@TaiSakuma

Copy link
Copy Markdown
Member

The nightly test failure on 2026-11-05 is because of #4228 . The predicate _has_issue_4228() failed to exclude a special case. This PR fixes it.

@ianna
ianna merged commit 60bc14d into scikit-hep:main Nov 5, 2026
36 checks passed
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

The documentation preview is ready to be viewed at https://preview.awkward-array.org/PR4398/

ikrommyd referenced this pull request in CoffeaTeam/awkward-lite Nov 6, 2026

This branch was previously deployed

1 inactive deployment
docs — 465ace8a Deployed Nov 5, 2026 by ianna via Remove Branch Preview #110
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/fix PR title type: fix (set automatically)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

to_arrow raises IndexError for valid unions with option-type children

3 participants