Repository navigation
fix: overflow-safe stats without needless copies - #4223
Conversation
|
I've been doing exactly the same in #3527 and you were arguing against it since it has to copy the whole array at once. Which is true, when it's an array of integers, a whole array of floats has to be allocated at once of the same size to do so. It's not like numpy's |
7f3de4a to
3a2ed18
Compare
tier 2 PR to follow |
|
The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR4223 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
|
TaiSakuma
left a comment
There was a problem hiding this comment.
This PR casts bool/integer input to float64 before the element-wise products that overflow in ak.var/ak.std/ak.moment and weighted ak.mean (x*x, x**n, x*weight). Floating and complex input is passed through unchanged; integral input is cast once per call, through the array's own backend.
corr and covar are left unchanged: they subtract float64 means before multiplying, so no integer products are formed. The PR follows #3527 (closed without merging, stated Fixes #3525), which cast all input, including floating input, to float64 up front.
Suggestions
This branch resolves the nan reported in #3525 (first item under Verification below). The description does not state that resolving an open issue is a goal of this PR — the link to #3525 is implicit, through #3527 — so the issue would stay open after merge. If closing #3525 with this PR is intended:
-
Add
Resolves #3525to the description; GitHub then closes the issue on merge. -
Add a test that confirms #3525 is resolved: the issue is an int64 overflow, while the new overflow tests use int32. For example (verified to match NumPy on this branch):
data = np.array([3_000_000_000, 4_000_000_000, 5_000_000_000], dtype=np.int64) # on current main: the squares overflow int64, the variance is negative, and ak.std returns nan assert ak.var(ak.Array(data)) == pytest.approx(np.var(data.astype(np.float64)))
Verification performed
- With the
data.csvattached to #3525: current main givesak.std = nan(variance-5.24e+16); this branch gives458469074.0604588; NumPy gives458469074.0604589(the last-digit difference is explained under Notes). - The six new tests pass on this branch. Four fail on current main;
test_float_input_unchangedpasses there by design, and the jagged test's case is under Minor. - 151 existing stats, reducer, and named-axis tests across six files still pass.
- The promotion works on typetracer layouts (
ak.var/ak.mean/ak.momenton a typetracer int32 array all producefloat64types). No test covers this; a typetracer test would prevent regressions in the dask-awkward path.
Notes
- No public API changes: no signatures or exported names change, and the new helper lives in the private
awkward._layoutmodule. - Output dtypes are unchanged (
ak.var(int32)and weightedak.mean(int32)already returnedfloat64on main); only values change, and only for bool/integer input. - The diff is additive (+115/−0); no existing tests or assertions change.
- Relative to NumPy: this adopts NumPy's dtype policy, not its algorithm. Awkward computes
E[x²] − E[x]²in one pass while NumPy computes the centered two-pass form, so results agree to within floating-point rounding, not bit-for-bit. Low-order digits can differ more for ill-conditioned data (large mean relative to spread).
Related: ak.linear_fit has the same overflow
ak.linear_fit forms uncentered products (x**2, x * y, x * weight) and is not covered here. Reproduced on this branch:
>>> x = ak.Array(np.array([100000, 200000, 300000], dtype=np.int32))
>>> y = ak.Array([1.0, 2.0, 3.0])
>>> ak.linear_fit(x, y).slope # correct value: 1e-05
-1.7030124510406536e-06The same promotion applied to x and y in ak_linear_fit._impl would cover it, here or in a follow-up; #3527's diff includes a version of that change.
Minor
- Building the jagged array with int32 values (e.g., via
ak.values_astype(array, np.int32)) would maketest_var_jagged_integer_no_overflowfail without the fix; as written, its values enter as int64 and square to at most2.5e11, so the test also passes on current main. - In
_has_integral_leaf, the traversal continues afterfound = True; returningnodeonce found would stop the descent early. It is a metadata-only walk, so this is optional. assert np.dtype(np.asarray(result).dtype) == np.dtype(np.float64)double-wraps a dtype;np.asarray(result).dtype == np.dtype(np.float64)is enough.
🤖 Drafted with Claude Code
|
@TaiSakuma - Thanks for reviewing it! I'll open a follow-up PR for |
TaiSakuma
left a comment
There was a problem hiding this comment.
@ianna, Thank you for the update. I approve the PR.
An approval comment generated by Claude Code is below.
Commit b38c0fa addresses all items from the earlier review; each was re-verified locally:
Resolves #3525is in the description, so the issue closes on merge.test_var_int64_no_overflow_issue_3525covers the issue's int64 case and fails on current main.test_var_jagged_integer_no_overflownow uses int32 data and fails on current main.test_typetracer_promotioncovers the typetracer path that dask-awkward relies on._has_integral_leafstops descending at the first integral leaf, and the dtype assertion is simplified.
On this branch, all 8 tests in test_4223_descriptive_statistics_overflow.py pass. On current main, 6 fail and 2 pass (test_float_input_unchanged passes there by design; test_typetracer_promotion checks output types that main also produces). The 151 existing stats, reducer, and named-axis tests still pass, and the data.csv from #3525 gives 458469074.0604588 (NumPy: 458469074.0604589).
ak.linear_fit remains for the announced follow-up PR.
🤖 Generated with Claude Code
Resolves #3525
Where the overflow actually is
ak.sum's reducer already promotes integer/bool leaves to the platform integer(
int64/uint64) accumulator — see_promote_integer_rankinsrc/awkward/_reducers.pyand theawkward_reduce_sum<result, input, offsets>kernel templated separately on accumulator and input dtype. So:
ak.sum(int32_array)already accumulates inint64. A plain integer meanwithout weights (
sum(x) / count) is overflow-safe up toint64today andneeds no cast.
the reduction:
x * x,x * weight,x * x * weightinvar,std,covar,corr,moment.int32 * int32staysint32and overflowsper-element — long before
int64accumulation can help.So the original #3527 PR's "cast the whole input to
float64up front, always" isheavier than needed: it copies floating inputs that never overflow, and it casts
in a spot that isn't the only thing that matters.
The two objections on the #3527 PR, addressed
argument against floating-point in general — it applies equally to
x / 2,which we already promote. The alternative here is silent integer overflow,
which is strictly worse than rounding error. NumPy makes exactly this choice
for
mean/var/std.ak.values_astyperuns on the array's own backend —cupy.ndarray.astypeonthe cuda backend. It stays on-device; there is no host round-trip. The cost on
GPU is
float64memory (2× anint32buffer) and lowerfloat64throughput,not a D2H transfer. The solution below only pays that cost for integral
inputs and only once per call.
Solution
Two tiers. Tier 1 is a small, self-contained Python PR. Tier 2 is the fully
copy-free version and needs a reducer tweak -- a separate PR to follow.
Tier 1 — promote only integral leaves, once, on-backend
A shared helper (e.g. in
src/awkward/_layout.py):Then, in each statistic that forms products, promote once and reuse the
result across every reduction. For
ak.var(src/awkward/operations/ak_var.py),the change is one line plus reuse:
The actual scope is smaller than "every stat," because most paths are already
safe:
ak_var.py— needs it (sum(x*x)).ak_std.pyinherits the fix forfree: it just calls
ak.varand takessqrt.ak_moment.py— needs it (sum(x**n)).ak_mean.py— needs it only in the weighted branch (sum(x*weight)).The unweighted path (
sum(x)/count) is already overflow-safe toint64viathe reducer's
int64accumulator, and the final division yieldsfloat64, soit is left copy-free.
ak_covar.py/ak_corr.py— need nothing. They center first,(x - xmean) * (y - ymean), andxmeanis alreadyfloat64(it comes fromak.mean), so the products are float before any multiply. Their internalak.meancall inherits the weighted-branch fix if integer weights are passed.So the real change is three files:
ak_var.py,ak_moment.py, and the weightedbranch of
ak_mean.py— plus the shared helper.If you additionally want unweighted integer
meanto match NumPy'sfloat64-accumulator behavior (guardingint64overflow too), use Tier 2 ratherthan adding a copy.
Net effect vs. the #3527 PR: floating inputs are untouched (zero copy — the common
case in analysis), integral inputs are promoted exactly once instead of
implicitly re-cast inside each
x*x/x*weight, and the cast provably stays onthe GPU backend.