Skip to content

fix: read float16, float128 and complex256 in type strings - #4390

Merged
ianna merged 5 commits into
scikit-hep:mainfrom
graphed-org:fix/datashape-all-numpy-dtypes
Oct 17, 2026
Merged

ianna merged 5 commits into
scikit-hep:mainfrom
graphed-org:fix/datashape-all-numpy-dtypes

Conversation

@lgray

@lgray lgray commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 AI text below 🤖

Bug

awkward prints type strings for float16 arrays that ak.types.from_datashape (and so ak.enforce_type(array, "<type string>")) cannot read back:

>>> import numpy as np, awkward as ak
>>> t = ak.from_numpy(np.zeros(2, np.float16)).type
>>> str(t)
'2 * float16'
>>> ak.types.from_datashape(str(t))
awkward.types._awkward_datashape_parser.UnexpectedToken: Unexpected token Token('$END', '') at line 1, column 5.

"float16", "var * float16", "3 * float16", "?float16" and "{x: float16}" all fail the same way on 2.14.0. The datashape grammar's DTYPE terminal was a hand-kept copy of the primitive list. It left out float16, and also float128/complex256 on platforms where numpy has them, although NumpyType (_primitive_to_dtype_dict) accepts all three.

Fix

  • Grammar (studies/type-parser-2/awkward_datashape_grammar.g, parser regenerated): DTYPE now lists every primitive that NumpyType knows on any platform. NumpyType still rejects the ones the local numpy lacks, so on macOS var * float128 raises TypeError: unrecognized primitive instead of a parse error. DTYPE, DATETIME64 and TIMEDELTA64 no longer match a prefix of a longer identifier, and named_key accepts those tokens as field names.
  • RecordType.__str__ also escapes record names float128/complex256 on platforms without them, because the grammar reserves those names on every platform.
  • show() picks the numpy formatter key the way numpy's arrayprint does (np.floating gives "longfloat" for np.longdouble and "float" otherwise; complex works the same way). Before this, float16 values, and np.longdouble/np.clongdouble on platforms without float128, fell through to str and ignored precision and formatter.

Why the lists cannot drift apart again without a test failing: the round-trip tests are parametrized over _primitive_to_dtype_dict, which is the table NumpyType validates against. If a dtype is added to that table but not to the grammar, the tests fail on every platform whose numpy has the dtype. float128/complex256 are also pinned on platforms that lack them: they must lex and then be rejected by NumpyType.

The change also fixes type strings that already failed on main:

  • R[int64: int64]: a named record with a field named after any dtype.
  • float32x[x: int64] and datetime64_1[int64]: a record name that starts with a dtype name.

Regenerating the parser

The repository does not document how to regenerate the parser. These are the steps I used. Only the DATA and MEMO lines of src/awkward/types/_awkward_datashape_parser.py are replaced, so the hand edits in that file are kept:

PYTHONHASHSEED=0 uv run --no-project --python 3.10 --with lark-parser==0.12.0 \
    python -m lark.tools.standalone studies/type-parser-2/awkward_datashape_grammar.g > /tmp/parser.py
# then copy the DATA = (...) and MEMO = (...) lines from /tmp/parser.py into the committed file

MEMO is byte-reproducible. DATA is not, because lark numbers the LALR states differently from run to run. The tables are equal once the states are renumbered canonically. Lark reports no LALR conflicts for the new grammar.

Tests

tests/test_4390_all_numpy_numeric_dtypes.py checks Type → str → from_datashape for every entry of _primitive_to_dtype_dict in 9 type contexts (bare, var *, 3 *, ?, option[var * ], record, tuple, union, named record). It also covers:

  • parameters= and the high-level forms;
  • ak.from_numpy(...).type and ak.enforce_type;
  • float128/complex256 on platforms with and without them;
  • record and field names that equal or start with a dtype name;
  • show() precision and formatter keys.

I ran the file against the unfixed main src (1f04d6f) and against this branch:

platform numpy float128 main this branch
macOS arm64, numpy 2.5.3 no 168 failed, 265 passed 433 passed
linux/amd64 (python:3.12, numpy 2.5.3) yes 192 failed, 267 passed 459 passed

On macOS the failures are the float16 round-trips, show() for float16/longdouble/clongdouble, float128/complex256 not lexing, and the dtype-named and dtype-prefixed record and field names. On linux the float128/complex256 round-trips fail too. The full tests/ suite passes on this branch on macOS (-n 8: 7036 passed, 56 skipped).

Found, but not in this PR

  • Eager reducers, sort and argsort raise KeyError(('awkward_reduce_sum', float16, ...)) for float16/float128/complex256. fix: float16/float128/complex256 reducers, sort, and argsort no longer raise KeyError (#4259) #4270 covers this. It would be a large duplicate, plus kernel-spec and awkward-cpp work.
  • ak.from_iter segfaults on np.clongdouble scalars (macOS arm64, measured rc 139). builder_fromiter has no complexfloating branch, and clongdouble.tolist() returns itself, so it recurses without end. The fix is about 3 C++ lines in awkward-cpp; this should be its own issue.
  • ForthMachine output x float16 is rejected as an unrecognized dtype. It needs an awkward-cpp change and a decision on a half-float C++ type. Small, but only ak.forth users reach it.
  • C++ bridges (cling value_type, from_rdataframe cpp_type_of, cuda dtype_to_ctype) have no float16/float128/complex256 entries. The long-double half is about 4 lines but needs a ROOT/cppyy test environment. float16 has no C++17 type.
  • datetime64[μs] is refused by the unit regexes in numpytype.py and in the grammar. This is not a numeric dtype. The fix is about 3 regex lines plus a parser regen.

The datashape grammar's DTYPE terminal omitted these NumpyType
primitives, so type strings that awkward prints (e.g. "var * float16")
could not be parsed back by ak.types.from_datashape or
ak.enforce_type. The grammar now lists every primitive NumpyType can
know on any platform; NumpyType still rejects the ones a platform's
numpy lacks. The parser tables are regenerated with lark-parser 0.12.0.
The formatter lookup only matched float32/float64 and, where numpy has
them, float128/complex256. float16 values, and np.longdouble or
np.clongdouble values on platforms without float128, fell through to
str, ignoring the precision and formatter options. Select the formatter
key the way numpy's arrayprint does: np.floating maps to "longfloat"
for np.longdouble and "float" otherwise, np.complexfloating to
"longcomplexfloat" for np.clongdouble and "complexfloat" otherwise.
The DTYPE, DATETIME64 and TIMEDELTA64 terminals matched a prefix of a
longer identifier, so type strings awkward prints for records named
e.g. "float32x" or "datetime64_1" failed to parse. Each terminal now
requires that no identifier character follows it. Parser tables are
regenerated with lark-parser 0.12.0.
Named-record keys accept dtype, datetime64 and timedelta64 tokens as
identifiers, and record names are escaped against every dtype name the
grammar reserves (float128/complex256 included on platforms without them).
@github-actions github-actions Bot added the type/fix PR title type: fix (set automatically) label Sep 25, 2026
@codecov

codecov Bot commented Sep 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.11%. Comparing base (3da610a) to head (c17ae5d).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
Files with missing lines Coverage Δ
src/awkward/_nplikes/numpy_like.py 100.00% <100.00%> (ø)
src/awkward/prettyprint.py 85.01% <100.00%> (-0.34%) ⬇️
src/awkward/types/_awkward_datashape_parser.py 48.17% <ø> (ø)
src/awkward/types/recordtype.py 78.52% <ø> (+2.01%) ⬆️

... and 1 file with indirect coverage changes

@github-actions

Copy link
Copy Markdown
Contributor

The documentation preview is ready to be viewed at https://preview.awkward-array.org/PR4390/

@ianna ianna left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lgray -- Thanks! Looks good to me.

@ianna
ianna merged commit 9679264 into scikit-hep:main Oct 17, 2026
40 of 41 checks passed
ikrommyd referenced this pull request in CoffeaTeam/awkward-lite Oct 19, 2026
ikrommyd referenced this pull request in CoffeaTeam/awkward-lite Nov 12, 2026
…plex numbers (#1)

* feat: add long double kernels for reducers, sorting and unique

* feat: run float16 reducers, sorting and unique through the float32 kernels and name the type in errors for complex sorting

* feat: parse float16, float128 and complex256 in type strings

* fix: stop ArrayBuilder from crashing on long double complex scalars and from rounding long double scalars

* fix: keep long double in ak.covar and ak.corr

* fix: apply float formatters to float16 everywhere and to long double where NumPy has no float128

* fix: raise a TypeError for long double forms in Awkward 1 pickles where NumPy has no float128

* feat: raise clear errors for float16 values and long double arrays in Numba

* feat: pass long double to and from C++ and raise a clear error for float16

* feat: raise clear errors when Arrow or JSON cannot hold a type and convert long double where it is double

* test: cover float16 and long double across reducers, sorting, type strings, builders, printing and conversions

* feat: find the unique values of complex numbers without an axis through NumPy

* fix: convert long double to the nearest double in ArrayBuilder and JSON instead of raising

* test: expect int64 counts and indexes on 32-bit platforms and cover complex unique, JSON and builder conversions

* fix: choose the JSON replacement strings from the converted long double values and update the license line of the new test files

* fix: pass a long double initial to the min and max kernels without rounding it to double

* fix: parse record and field names that start with or equal a dtype name (#4390)

* test: sort float16 references in float32 to avoid a NumPy bug on 32-bit Windows

* test: compare float16 sorts with NumPy's again now that 32-bit Windows runs Python 3.11

* fix: keep awkward_unique_ranges within its buffer when a list starts with NaN

* feat: add complex kernels for sorting and unique

* feat: sort complex numbers and find the unique complex numbers and date-times along an axis

* fix: find the unique values along an axis of non-contiguous arrays

* feat: convert complex NaN to None in ak.nan_to_none

* refactor: share the complex sort and argsort comparator through numeric-sort.h

* fix: read long double forms of Awkward 1 pickles by their item size

* refactor: choose the kernel dtype of unique values without an axis through _sorting_kernel_dtype

* fix: raise for long double values beyond the range of double when converting to JSON

This branch was previously deployed

1 inactive deployment
docs — c17ae5de Deployed Oct 17, 2026 by ianna via Remove Branch Preview #70
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/fix PR title type: fix (set automatically)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants