Repository navigation
fix: read float16, float128 and complex256 in type strings - #4390
Merged
ianna merged 5 commits intoOct 17, 2026
Merged
Conversation
The datashape grammar's DTYPE terminal omitted these NumpyType primitives, so type strings that awkward prints (e.g. "var * float16") could not be parsed back by ak.types.from_datashape or ak.enforce_type. The grammar now lists every primitive NumpyType can know on any platform; NumpyType still rejects the ones a platform's numpy lacks. The parser tables are regenerated with lark-parser 0.12.0.
The formatter lookup only matched float32/float64 and, where numpy has them, float128/complex256. float16 values, and np.longdouble or np.clongdouble values on platforms without float128, fell through to str, ignoring the precision and formatter options. Select the formatter key the way numpy's arrayprint does: np.floating maps to "longfloat" for np.longdouble and "float" otherwise, np.complexfloating to "longcomplexfloat" for np.clongdouble and "complexfloat" otherwise.
The DTYPE, DATETIME64 and TIMEDELTA64 terminals matched a prefix of a longer identifier, so type strings awkward prints for records named e.g. "float32x" or "datetime64_1" failed to parse. Each terminal now requires that no identifier character follows it. Parser tables are regenerated with lark-parser 0.12.0.
Named-record keys accept dtype, datetime64 and timedelta64 tokens as identifiers, and record names are escaped against every dtype name the grammar reserves (float128/complex256 included on platforms without them).
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
|
Contributor
|
The documentation preview is ready to be viewed at https://preview.awkward-array.org/PR4390/ |
ikrommyd
referenced
this pull request
in CoffeaTeam/awkward-lite
Oct 17, 2026
ikrommyd
referenced
this pull request
in CoffeaTeam/awkward-lite
Oct 19, 2026
This was referenced Nov 8, 2026
ikrommyd
referenced
this pull request
in CoffeaTeam/awkward-lite
Nov 12, 2026
…plex numbers (#1) * feat: add long double kernels for reducers, sorting and unique * feat: run float16 reducers, sorting and unique through the float32 kernels and name the type in errors for complex sorting * feat: parse float16, float128 and complex256 in type strings * fix: stop ArrayBuilder from crashing on long double complex scalars and from rounding long double scalars * fix: keep long double in ak.covar and ak.corr * fix: apply float formatters to float16 everywhere and to long double where NumPy has no float128 * fix: raise a TypeError for long double forms in Awkward 1 pickles where NumPy has no float128 * feat: raise clear errors for float16 values and long double arrays in Numba * feat: pass long double to and from C++ and raise a clear error for float16 * feat: raise clear errors when Arrow or JSON cannot hold a type and convert long double where it is double * test: cover float16 and long double across reducers, sorting, type strings, builders, printing and conversions * feat: find the unique values of complex numbers without an axis through NumPy * fix: convert long double to the nearest double in ArrayBuilder and JSON instead of raising * test: expect int64 counts and indexes on 32-bit platforms and cover complex unique, JSON and builder conversions * fix: choose the JSON replacement strings from the converted long double values and update the license line of the new test files * fix: pass a long double initial to the min and max kernels without rounding it to double * fix: parse record and field names that start with or equal a dtype name (#4390) * test: sort float16 references in float32 to avoid a NumPy bug on 32-bit Windows * test: compare float16 sorts with NumPy's again now that 32-bit Windows runs Python 3.11 * fix: keep awkward_unique_ranges within its buffer when a list starts with NaN * feat: add complex kernels for sorting and unique * feat: sort complex numbers and find the unique complex numbers and date-times along an axis * fix: find the unique values along an axis of non-contiguous arrays * feat: convert complex NaN to None in ak.nan_to_none * refactor: share the complex sort and argsort comparator through numeric-sort.h * fix: read long double forms of Awkward 1 pickles by their item size * refactor: choose the kernel dtype of unique values without an axis through _sorting_kernel_dtype * fix: raise for long double values beyond the range of double when converting to JSON
This was referenced Nov 12, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Bug
awkward prints type strings for float16 arrays that
ak.types.from_datashape(and soak.enforce_type(array, "<type string>")) cannot read back:"float16","var * float16","3 * float16","?float16"and"{x: float16}"all fail the same way on 2.14.0. The datashape grammar'sDTYPEterminal was a hand-kept copy of the primitive list. It left outfloat16, and alsofloat128/complex256on platforms where numpy has them, althoughNumpyType(_primitive_to_dtype_dict) accepts all three.Fix
studies/type-parser-2/awkward_datashape_grammar.g, parser regenerated):DTYPEnow lists every primitive thatNumpyTypeknows on any platform.NumpyTypestill rejects the ones the local numpy lacks, so on macOSvar * float128raisesTypeError: unrecognized primitiveinstead of a parse error.DTYPE,DATETIME64andTIMEDELTA64no longer match a prefix of a longer identifier, andnamed_keyaccepts those tokens as field names.RecordType.__str__also escapes record namesfloat128/complex256on platforms without them, because the grammar reserves those names on every platform.show()picks the numpy formatter key the way numpy'sarrayprintdoes (np.floatinggives"longfloat"fornp.longdoubleand"float"otherwise; complex works the same way). Before this, float16 values, andnp.longdouble/np.clongdoubleon platforms without float128, fell through tostrand ignoredprecisionandformatter.Why the lists cannot drift apart again without a test failing: the round-trip tests are parametrized over
_primitive_to_dtype_dict, which is the tableNumpyTypevalidates against. If a dtype is added to that table but not to the grammar, the tests fail on every platform whose numpy has the dtype.float128/complex256are also pinned on platforms that lack them: they must lex and then be rejected byNumpyType.The change also fixes type strings that already failed on
main:R[int64: int64]: a named record with a field named after any dtype.float32x[x: int64]anddatetime64_1[int64]: a record name that starts with a dtype name.Regenerating the parser
The repository does not document how to regenerate the parser. These are the steps I used. Only the
DATAandMEMOlines ofsrc/awkward/types/_awkward_datashape_parser.pyare replaced, so the hand edits in that file are kept:PYTHONHASHSEED=0 uv run --no-project --python 3.10 --with lark-parser==0.12.0 \ python -m lark.tools.standalone studies/type-parser-2/awkward_datashape_grammar.g > /tmp/parser.py # then copy the DATA = (...) and MEMO = (...) lines from /tmp/parser.py into the committed fileMEMOis byte-reproducible.DATAis not, because lark numbers the LALR states differently from run to run. The tables are equal once the states are renumbered canonically. Lark reports no LALR conflicts for the new grammar.Tests
tests/test_4390_all_numpy_numeric_dtypes.pychecks Type → str →from_datashapefor every entry of_primitive_to_dtype_dictin 9 type contexts (bare,var *,3 *,?,option[var * ], record, tuple, union, named record). It also covers:parameters=and the high-level forms;ak.from_numpy(...).typeandak.enforce_type;float128/complex256on platforms with and without them;show()precision and formatter keys.I ran the file against the unfixed
mainsrc (1f04d6f) and against this branch:mainOn macOS the failures are the float16 round-trips,
show()for float16/longdouble/clongdouble,float128/complex256not lexing, and the dtype-named and dtype-prefixed record and field names. On linux the float128/complex256 round-trips fail too. The fulltests/suite passes on this branch on macOS (-n 8: 7036 passed, 56 skipped).Found, but not in this PR
KeyError(('awkward_reduce_sum', float16, ...))for float16/float128/complex256. fix: float16/float128/complex256 reducers, sort, and argsort no longer raise KeyError (#4259) #4270 covers this. It would be a large duplicate, plus kernel-spec and awkward-cpp work.ak.from_itersegfaults onnp.clongdoublescalars (macOS arm64, measured rc 139).builder_fromiterhas nocomplexfloatingbranch, andclongdouble.tolist()returns itself, so it recurses without end. The fix is about 3 C++ lines in awkward-cpp; this should be its own issue.output x float16is rejected as an unrecognized dtype. It needs an awkward-cpp change and a decision on a half-float C++ type. Small, but onlyak.forthusers reach it.value_type,from_rdataframecpp_type_of, cudadtype_to_ctype) have no float16/float128/complex256 entries. The long-double half is about 4 lines but needs a ROOT/cppyy test environment. float16 has no C++17 type.datetime64[μs]is refused by the unit regexes innumpytype.pyand in the grammar. This is not a numeric dtype. The fix is about 3 regex lines plus a parser regen.