Crash report

What happened?

The following aborts on a debug build, and SystemError's otherwise:
(runs on linux as the rlimit is not portable, but any hard memory limit should do)

import resource 
MB_512 = 512 * 1024 * 1024
resource.setrlimit(resource.RLIMIT_AS, (MB_512, MB_512)) 
eval("A"* 420000000)

What seems to be happening is: _PyTokenizer_FromUTF8 calls _PyTokenizer_translate_newlines

Which does some allocations. If any of those allocations fail, the tok->done field is set to E_NOMEM:

BUT, then _PyTokenizer_FromUTF8 just sees if the translated version is returned, and if not, frees the tok and returns NULL without setting the PyErr_NoMemory exception state:

So the NULL bubbles up the chain and is returned to the eval() call, but there's no error set, so you get the abort coming out of: _Py_CheckFunctionResult

It seems to me that the solution should be setting PyErr_NoMemory if the token->done ends up with E_NOMEM in _PyTokenizer_FromUTF8, but I'm no expert here, and not sure if there are ramifications given that utf8_tokenizer.c doesn't do much pyobject stuff.

Note: there's another, much less likely allocation just below in _PyTokenizer_FromUTF8:

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.16.0a0 (heads/main:441af3a, May 21 2026, 15:18:12) [Clang 22.1.5 ]

Linked PRs

• gh-150207: type-crash(): make helpers.c in tokenizer throw a no mem error  #150210
• gh-150207: Raise MemoryError on tokenizer allocation failure instead of crashing #150275
• [3.15] gh-150207: Raise MemoryError on tokenizer allocation failure instead of crashing (GH-150275) #150996
• [3.14] gh-150207: Raise MemoryError on tokenizer allocation failure instead of crashing (GH-150275) #150997
• [3.13] gh-150207: Raise MemoryError on tokenizer allocation failure instead of crashing (GH-150275) #150998