Skip to content

fix: overflow-safe stats without needless copies - #4223

Merged
ianna merged 6 commits into
mainfrom
ianna/stats_overflow_fix
Jul 25, 2026
Merged

ianna merged 6 commits into
mainfrom
ianna/stats_overflow_fix

Conversation

@ianna

@ianna ianna commented Jul 22, 2026 •

Copy link
Copy Markdown
Member

Resolves #3525

Where the overflow actually is

ak.sum's reducer already promotes integer/bool leaves to the platform integer
(int64/uint64) accumulator — see _promote_integer_rank in
src/awkward/_reducers.py and the awkward_reduce_sum<result, input, offsets>
kernel templated separately on accumulator and input dtype. So:

  • ak.sum(int32_array) already accumulates in int64. A plain integer mean
    without weights
    (sum(x) / count) is overflow-safe up to int64 today and
    needs no cast.
  • The real overflow is the element-wise products computed in Python before
    the reduction: x * x, x * weight, x * x * weight in var, std,
    covar, corr, moment. int32 * int32 stays int32 and overflows
    per-element — long before int64 accumulation can help.

So the original #3527 PR's "cast the whole input to float64 up front, always" is
heavier than needed: it copies floating inputs that never overflow, and it casts
in a spot that isn't the only thing that matters.

The two objections on the #3527 PR, addressed

  1. "Binary floats can't represent decimals exactly." True, but that's an
    argument against floating-point in general — it applies equally to x / 2,
    which we already promote. The alternative here is silent integer overflow,
    which is strictly worse than rounding error. NumPy makes exactly this choice
    for mean/var/std.
  2. "The cast triggers device-to-host copies on GPU." A dtype cast via
    ak.values_astype runs on the array's own backend — cupy.ndarray.astype on
    the cuda backend. It stays on-device; there is no host round-trip. The cost on
    GPU is float64 memory (2× an int32 buffer) and lower float64 throughput,
    not a D2H transfer. The solution below only pays that cost for integral
    inputs
    and only once per call.

Solution

Two tiers. Tier 1 is a small, self-contained Python PR. Tier 2 is the fully
copy-free version and needs a reducer tweak -- a separate PR to follow.

Tier 1 — promote only integral leaves, once, on-backend

A shared helper (e.g. in src/awkward/_layout.py):

import awkward as ak
from awkward._nplikes.numpy_like import NumpyMetadata

np = NumpyMetadata.instance()


def _has_integral_leaf(layout) -> bool:
    """True if any NumpyArray leaf is bool/integer (kinds 'b', 'i', 'u')."""
    found = False

    def action(node, **kwargs):
        nonlocal found
        if node.is_numpy and node.dtype.kind in "biu":
            found = True
        return None

    ak._do.recursively_apply(layout, action, return_array=False)
    return found


def promote_integral_to_float64(array):
    """Cast bool/integer data to float64 to avoid overflow in summations,
    matching NumPy's reduction dtype.

    Floating and complex data is returned unchanged (no copy). The cast runs on
    the array's own backend, so on GPU it stays on device — no host transfer.
    """
    if _has_integral_leaf(array.layout):
        return ak.operations.values_astype(array, np.float64, highlevel=True)
    return array

Then, in each statistic that forms products, promote once and reuse the
result across every reduction. For ak.var (src/awkward/operations/ak_var.py),
the change is one line plus reuse:

    x = ctx.wrap(x_layout)
    weight = ctx.wrap(weight_layout, allow_other=True)

    # Overflow guard: integer squares/products overflow before reduction.
    # Promote integral leaves to float64 once (no-op + zero copy for floats).
    x = promote_integral_to_float64(x)
    if weight is not None:
        weight = promote_integral_to_float64(weight)

    ...
    # sumwx, sumwxx (and sumw) now reduce float64 data — one promotion, reused.

The actual scope is smaller than "every stat," because most paths are already
safe:

  • ak_var.py — needs it (sum(x*x)). ak_std.py inherits the fix for
    free: it just calls ak.var and takes sqrt.
  • ak_moment.py — needs it (sum(x**n)).
  • ak_mean.py — needs it only in the weighted branch (sum(x*weight)).
    The unweighted path (sum(x)/count) is already overflow-safe to int64 via
    the reducer's int64 accumulator, and the final division yields float64, so
    it is left copy-free.
  • ak_covar.py / ak_corr.py — need nothing. They center first,
    (x - xmean) * (y - ymean), and xmean is already float64 (it comes from
    ak.mean), so the products are float before any multiply. Their internal
    ak.mean call inherits the weighted-branch fix if integer weights are passed.

So the real change is three files: ak_var.py, ak_moment.py, and the weighted
branch of ak_mean.py — plus the shared helper.

If you additionally want unweighted integer mean to match NumPy's
float64-accumulator behavior (guarding int64 overflow too), use Tier 2 rather
than adding a copy.

Net effect vs. the #3527 PR: floating inputs are untouched (zero copy — the common
case in analysis), integral inputs are promoted exactly once instead of
implicitly re-cast inside each x*x / x*weight, and the cast provably stays on
the GPU backend.

@github-actions github-actions Bot added the type/fix PR title type: fix (set automatically) label Jul 22, 2026
@ikrommyd

Copy link
Copy Markdown
Member

I've been doing exactly the same in #3527 and you were arguing against it since it has to copy the whole array at once. Which is true, when it's an array of integers, a whole array of floats has to be allocated at once of the same size to do so. It's not like numpy's np.sum(..., dtype=np.float64) which does the casting while iterating inside the loop.

@ianna
ianna force-pushed the ianna/stats_overflow_fix branch from 7f3de4a to 3a2ed18 Compare July 22, 2026 18:57
@ianna

ianna commented Jul 22, 2026

Copy link
Copy Markdown
Member Author

I've been doing exactly the same in #3527 and you were arguing against it since it has to copy the whole array at once. Which is true, when it's an array of integers, a whole array of floats has to be allocated at once of the same size to do so. It's not like numpy's np.sum(..., dtype=np.float64) which does the casting while iterating inside the loop.

tier 2 PR to follow

@ianna
ianna requested a review from TaiSakuma July 22, 2026 19:00
@github-actions

Copy link
Copy Markdown
Contributor

The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR4223

@codecov

codecov Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@27a23cd). Learn more about missing BASE report.
⚠️ Report is 10 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
Files with missing lines Coverage Δ
src/awkward/_layout.py 92.41% <100.00%> (ø)
src/awkward/operations/ak_mean.py 94.44% <100.00%> (ø)
src/awkward/operations/ak_moment.py 100.00% <100.00%> (ø)
src/awkward/operations/ak_var.py 89.65% <100.00%> (ø)

@TaiSakuma TaiSakuma left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR casts bool/integer input to float64 before the element-wise products that overflow in ak.var/ak.std/ak.moment and weighted ak.mean (x*x, x**n, x*weight). Floating and complex input is passed through unchanged; integral input is cast once per call, through the array's own backend.

corr and covar are left unchanged: they subtract float64 means before multiplying, so no integer products are formed. The PR follows #3527 (closed without merging, stated Fixes #3525), which cast all input, including floating input, to float64 up front.

Suggestions

This branch resolves the nan reported in #3525 (first item under Verification below). The description does not state that resolving an open issue is a goal of this PR — the link to #3525 is implicit, through #3527 — so the issue would stay open after merge. If closing #3525 with this PR is intended:

  1. Add Resolves #3525 to the description; GitHub then closes the issue on merge.

  2. Add a test that confirms #3525 is resolved: the issue is an int64 overflow, while the new overflow tests use int32. For example (verified to match NumPy on this branch):

    data = np.array([3_000_000_000, 4_000_000_000, 5_000_000_000], dtype=np.int64)
    # on current main: the squares overflow int64, the variance is negative, and ak.std returns nan
    assert ak.var(ak.Array(data)) == pytest.approx(np.var(data.astype(np.float64)))

Verification performed

  • With the data.csv attached to #3525: current main gives ak.std = nan (variance -5.24e+16); this branch gives 458469074.0604588; NumPy gives 458469074.0604589 (the last-digit difference is explained under Notes).
  • The six new tests pass on this branch. Four fail on current main; test_float_input_unchanged passes there by design, and the jagged test's case is under Minor.
  • 151 existing stats, reducer, and named-axis tests across six files still pass.
  • The promotion works on typetracer layouts (ak.var/ak.mean/ak.moment on a typetracer int32 array all produce float64 types). No test covers this; a typetracer test would prevent regressions in the dask-awkward path.

Notes

  • No public API changes: no signatures or exported names change, and the new helper lives in the private awkward._layout module.
  • Output dtypes are unchanged (ak.var(int32) and weighted ak.mean(int32) already returned float64 on main); only values change, and only for bool/integer input.
  • The diff is additive (+115/−0); no existing tests or assertions change.
  • Relative to NumPy: this adopts NumPy's dtype policy, not its algorithm. Awkward computes E[x²] − E[x]² in one pass while NumPy computes the centered two-pass form, so results agree to within floating-point rounding, not bit-for-bit. Low-order digits can differ more for ill-conditioned data (large mean relative to spread).

Related: ak.linear_fit has the same overflow

ak.linear_fit forms uncentered products (x**2, x * y, x * weight) and is not covered here. Reproduced on this branch:

>>> x = ak.Array(np.array([100000, 200000, 300000], dtype=np.int32))
>>> y = ak.Array([1.0, 2.0, 3.0])
>>> ak.linear_fit(x, y).slope   # correct value: 1e-05
-1.7030124510406536e-06

The same promotion applied to x and y in ak_linear_fit._impl would cover it, here or in a follow-up; #3527's diff includes a version of that change.

Minor

  • Building the jagged array with int32 values (e.g., via ak.values_astype(array, np.int32)) would make test_var_jagged_integer_no_overflow fail without the fix; as written, its values enter as int64 and square to at most 2.5e11, so the test also passes on current main.
  • In _has_integral_leaf, the traversal continues after found = True; returning node once found would stop the descent early. It is a metadata-only walk, so this is optional.
  • assert np.dtype(np.asarray(result).dtype) == np.dtype(np.float64) double-wraps a dtype; np.asarray(result).dtype == np.dtype(np.float64) is enough.

🤖 Drafted with Claude Code

@ianna

ianna commented Jul 24, 2026

Copy link
Copy Markdown
Member Author

@TaiSakuma - Thanks for reviewing it! I'll open a follow-up PR for ak.linear_fit. The rest is addressed I think. Thanks!

@TaiSakuma TaiSakuma left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ianna, Thank you for the update. I approve the PR.

An approval comment generated by Claude Code is below.


Commit b38c0fa addresses all items from the earlier review; each was re-verified locally:

  • Resolves #3525 is in the description, so the issue closes on merge.
  • test_var_int64_no_overflow_issue_3525 covers the issue's int64 case and fails on current main.
  • test_var_jagged_integer_no_overflow now uses int32 data and fails on current main.
  • test_typetracer_promotion covers the typetracer path that dask-awkward relies on.
  • _has_integral_leaf stops descending at the first integral leaf, and the dtype assertion is simplified.

On this branch, all 8 tests in test_4223_descriptive_statistics_overflow.py pass. On current main, 6 fail and 2 pass (test_float_input_unchanged passes there by design; test_typetracer_promotion checks output types that main also produces). The 151 existing stats, reducer, and named-axis tests still pass, and the data.csv from #3525 gives 458469074.0604588 (NumPy: 458469074.0604589).

ak.linear_fit remains for the announced follow-up PR.

🤖 Generated with Claude Code

@ianna
ianna merged commit d18a867 into main Jul 25, 2026
40 checks passed
@ianna
ianna deleted the ianna/stats_overflow_fix branch July 25, 2026 12:31
ikrommyd added a commit to ikrommyd/awkward that referenced this pull request Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/fix PR title type: fix (set automatically)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ak.std returns nan wrongly when being applied on int array

3 participants