Repository navigation
Artificial newlines show in the "line" attribute of tokens in the tokenizer module #105564
Description
Example:
from tokenize import generate_tokens
from io import StringIO
import pprint
pprint.pprint(list(generate_tokens(StringIO('a').readline)))
prints:
[TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a\n'),
TokenInfo(type=4 (NEWLINE), string='', start=(1, 1), end=(1, 2), line='a'),
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]
Activity
pythongh-105564: Don't include artificial newlines in the line attribâĶ
pablogsal commentedon Jun 9, 2023
CC: @lysnikolaou @isidentical @mgmacias95
pythongh-105564: Don't include artificial newlines in the line attribâĶ
pythongh-105564: Don't include artificial newlines in the line attribâĶ
pythongh-105564: Don't include artificial newlines in the line attribâĶ
gh-105564: Don't include artificial newlines in the line attribute ofâĶ
pythongh-105564: Don't include artificial newlines in the line attribâĶ
[3.12] gh-105564: Don't include artificial newlines in the line attriâĶ