Skip to content

BytesGenerator breaks UTF8 string #92081

Closed
Closed
BytesGenerator breaks UTF8 string#92081
Assignees
warsaw
Labels
topic-emailtype-bugAn unexpected behavior, bug, or error

Description

@Yuribtr
Yuribtr
opened on Apr 30, 2022

Hi!
I found an issue when sending emails with Cyrillic letters in Subject header. Some spaces at Subject header are trimmed when sent.

Example:
When sending email with below subject:
ÐĢÐēÐĩÐīÐūОÐŧÐĩÐ―ÐļÐĩ Ðū ÐŋŅ€ÐļÐ―ŅŅ‚ÐļÐļ Ðē Ņ€Ð°ÐąÐūŅ‚Ņƒ ÐūÐąŅ€Ð°Ņ‰ÐĩÐ―ÐļŅ

at SMTP server logs I see subject that differs from original:
ÐĢÐēÐĩÐīÐūОÐŧÐĩÐ―ÐļÐĩ Ðū ÐŋŅ€ÐļÐ―ŅŅ‚ÐļÐļÐē Ņ€Ð°ÐąÐūŅ‚Ņƒ ÐūÐąŅ€Ð°Ņ‰ÐĩÐ―ÐļŅ

  • As you can see, space between words "ÐŋŅ€ÐļÐ―ŅŅ‚ÐļÐļ Ðē" was stripped.

During research I've found that problem relates to small piece of code which encodes EmailMessage instance to byte string.
Python versions tested and problem confirmed: 3.8, 3.9, 3.10

Here is minimal reproducible example. Code can be used "as is", without any third party packages.

Minimal reproducible example 
import io
import email.generator
from email.message import EmailMessage
from email.header import decode_header


def encode_decode(subject: str):
    # preparing EmailMessage
    msg = EmailMessage()
    msg['Subject'] = subject

    # below code sample was taken from "send_message" function (lib/python3.8/smtplib.py)
    # this is the place where problem actually appears
    with io.BytesIO() as bytesmsg:
        g = email.generator.BytesGenerator(bytesmsg)
        g.flatten(msg, linesep='\r\n')
        flatmsg = bytesmsg.getvalue()

    # assembling string and cutting off beginning part ('Subject: ')
    result = ''
    for string, encoding in decode_header(flatmsg.decode()):
        result += string.decode(encoding=encoding or 'utf8')
    return result[9:]


if __name__ == '__main__':
    test_cases = [
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„',   # ok
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ ',   # ok
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„',   # ok
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„Ņ„',   # ok
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',   # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„Ņ„ Ņ„',   # ok
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„ Ņ„',   # ok
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„ Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        ' Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # ok
        ' Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # broken
        'Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„',  # ok
    ]
    for in_ in test_cases:
        out_ = encode_decode(in_)
        res = 'ok' if out_ == in_ else 'broken'
        print(f'In  | {in_}', f'Out | {out_}', f'Res | {res}\n', sep='\n')

Above code demonstrates inequality of input and output strings after encoding message with BytesGenerator. Please note that not all strings with Cyrillic letters are broken. Only those strings that have word with single Cyrillic char only are affected under some conditions.
Small additional list of string with explanations you can find below:

Additional strings with explanations 
# Example 1 - below string will be broken
'ÐĢÐēÐĩÐīÐūОÐŧÐĩÐ―ÐļÐĩ Ðū ÐŋŅ€ÐļÐ―ŅŅ‚ÐļÐļ Ðē Ņ€Ð°ÐąÐūŅ‚Ņƒ ÐūÐąŅ€Ð°Ņ‰ÐĩÐ―ÐļŅ ÐīÐŧŅ ÐŋÐūÐīКÐŧŅŽŅ‡ÐĩÐ―ÐļŅ ŅƒŅÐŧŅƒÐģÐļ',
# fixed version (removed one cyrillic UTF8 "Ðļ" from word "ÐŋŅ€ÐļÐ―ŅŅ‚ÐļÐļ")
'ÐĢÐēÐĩÐīÐūОÐŧÐĩÐ―ÐļÐĩ Ðū ÐŋŅ€ÐļÐ―ŅŅ‚Ðļ Ðē Ņ€Ð°ÐąÐūŅ‚Ņƒ ÐūÐąŅ€Ð°Ņ‰ÐĩÐ―ÐļŅ ÐīÐŧŅ ÐŋÐūÐīКÐŧŅŽŅ‡ÐĩÐ―ÐļŅ ŅƒŅÐŧŅƒÐģÐļ',
# fixed version (changed cyrillic UTF8 letter "Ðū" to ASCII letter "o")
'ÐĢÐēÐĩÐīÐūОÐŧÐĩÐ―ÐļÐĩ o ÐŋŅ€ÐļÐ―ŅŅ‚ÐļÐļ Ðē Ņ€Ð°ÐąÐūŅ‚Ņƒ ÐūÐąŅ€Ð°Ņ‰ÐĩÐ―ÐļŅ ÐīÐŧŅ ÐŋÐūÐīКÐŧŅŽŅ‡ÐĩÐ―ÐļŅ ŅƒŅÐŧŅƒÐģÐļ',
#
# Example 2 - below string will be broken
'ÐĢÐēÐĩÐīÐūОÐŧÐĩÐ―ÐļÐĩ ÐŋŅ€ÐļÐ―ŅŅ‚ÐļÐļ Ðē Ņ€Ð°ÐąÐūŅ‚Ņƒ ÐūÐąŅ€Ð°Ņ‰ÐĩÐ―ÐļŅ ÐīÐŧŅ ÐŋÐūÐīКÐŧŅŽŅ‡ÐĩÐ―ÐļŅ ŅƒŅÐŧŅƒÐģÐļ',
# fixed version (removed preposition "Ðē" that consist from single cyrillic UTF8 letter)
'ÐĢÐēÐĩÐīÐūОÐŧÐĩÐ―ÐļÐĩ ÐŋŅ€ÐļÐ―ŅŅ‚ÐļÐļ Ņ€Ð°ÐąÐūŅ‚Ņƒ ÐūÐąŅ€Ð°Ņ‰ÐĩÐ―ÐļŅ ÐīÐŧŅ ÐŋÐūÐīКÐŧŅŽŅ‡ÐĩÐ―ÐļŅ ŅƒŅÐŧŅƒÐģÐļ',
# fixed version (changed preposition "Ðē" with cyrillic UTF8 letter to ASCII letter "B")
'ÐĢÐēÐĩÐīÐūОÐŧÐĩÐ―ÐļÐĩ ÐŋŅ€ÐļÐ―ŅŅ‚ÐļÐļ B Ņ€Ð°ÐąÐūŅ‚Ņƒ ÐūÐąŅ€Ð°Ņ‰ÐĩÐ―ÐļŅ ÐīÐŧŅ ÐŋÐūÐīКÐŧŅŽŅ‡ÐĩÐ―ÐļŅ ŅƒŅÐŧŅƒÐģÐļ',

Linked PRs

Activity

  1. Yuribtr
    added
    type-bugAn unexpected behavior, bug, or error
    on Apr 30, 2022
  2. hugovk
    added
    topic-email
    on Apr 30, 2022

  3. mrabarnett commented on Apr 30, 2022

    @mrabarnett
    mrabarnett
    on Apr 30, 2022

    The problem has something to do with the maximum header length when the message is encoded to ASCII.

    Increasing the maximum header length (the default is 78) makes the problem disappear for the sample tests:

    g = email.generator.BytesGenerator(bytesmsg, maxheaderlen=160)

  4. Yuribtr commented on May 1, 2022

    @Yuribtr
    Yuribtr
    on May 1, 2022
    Author

    The problem has something to do with the maximum header length when the message is encoded to ASCII.

    Increasing the maximum header length (the default is 78) makes the problem disappear for the sample tests:

    g = email.generator.BytesGenerator(bytesmsg, maxheaderlen=160)

    Thanks for hint. As temporary workaround I set maxheaderlen to zero:
    g = email.generator.BytesGenerator(bytesmsg, maxheaderlen=0)

    UPD.
    For sending emails you should set max_line_length greater than 3. This is necessary in order to avoid ValueError at /Lib/email/quoprimime.py::body_encode() in some cases. I prefer to set max_line_length to maximum.

    msg = EmailMessage()
msg.policy = msg.policy.clone(max_line_length=sys.maxsize)

  5. abadger commented on May 3, 2022

    @abadger
    abadger
    on May 3, 2022
    Contributor

    Note: This problem will occur with Generator as well as BytesGenerator.

    What I believe is happening in the problem cases is that the Generator is creating two or more encoded-words. A space character present in the input is placed between the encoded words rather than becoming a part of the encoded words. When the decoder is run, the decoder discards the linear whitespace that occurs between the encoded words. This leads to the omission of the space in the round tripped data.

    Here's the output from the Generator for one of the problem cases:

    Subject: =?utf-8?b?0YTRhNGE0YTRhNGE0YTRhNGE0YTRhNGE0YTRhNGE0YTRhNGE0YTRhNGE?=
  =?utf-8?b?0YQg0YQ=?=

    You can see that there are two spaces (in addition to the \r\n) in between the two encoded blocks.

    After reading the examples in https://www.rfc-editor.org/rfc/rfc2047 I believe that the generator is at fault here. The space characters should be added to either the leading or trailing encoded-word rather than being emitted as a literal between the words.

  6. abadger commented on May 3, 2022

    @abadger
    abadger
    on May 3, 2022
    Contributor

    Still exploring precisely what is going on but I have been able to modify the code to fix all of the cases reported by forcing whitespace to be encoded more frequently in _header_value_parser.py::_refold_parse_tree() and preventing leading and trailing whitespace from being detected in _fold_as_ew() . I think my changes are too broad but I'll post them here in case something happens to me before I have a chance to finish diagnosing this:

    EDIT: Only leading whitespace needs to be undetected in _fold_as_ew()

    diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py
index 8a8fb8bc42..498a3c1e01 100644
--- a/Lib/email/_header_value_parser.py
+++ b/Lib/email/_header_value_parser.py
@@ -2781,6 +2781,8 @@ def _refold_parse_tree(parse_tree, *, policy):
         if part.token_type == 'ptext' and set(tstr) & SPECIALS:
             # Encode if tstr contains special characters.
             want_encoding = True
+        elif part.token_type == 'fws' and last_ew:
+            want_encoding = True
         try:
             tstr.encode(encoding)
             charset = encoding
@@ -2877,19 +2879,19 @@ def _fold_as_ew(to_encode, lines, maxlen, last_ew, ew_combine_allowed, charset):
         to_encode = str(
             get_unstructured(lines[-1][last_ew:] + to_encode))
         lines[-1] = lines[-1][:last_ew]
-    if to_encode[0] in WSP:
-        # We're joining this to non-encoded text, so don't encode
-        # the leading blank.
-        leading_wsp = to_encode[0]
-        to_encode = to_encode[1:]
-        if (len(lines[-1]) == maxlen):
-            lines.append(_steal_trailing_WSP_if_exists(lines))
-        lines[-1] += leading_wsp
+    #if to_encode[0] in WSP:
+    #    # We're joining this to non-encoded text, so don't encode
+    #    # the leading blank.
+    #    leading_wsp = to_encode[0]
+    #    to_encode = to_encode[1:]
+    #    if (len(lines[-1]) == maxlen):
+    #        lines.append(_steal_trailing_WSP_if_exists(lines))
+    #    lines[-1] += leading_wsp
     trailing_wsp = ''
     if to_encode[-1] in WSP:
         # Likewise for the trailing space.

  7. abadger commented on May 3, 2022

    @abadger
    abadger
    on May 3, 2022
    Contributor

    Closer..... This change in _fold_as_ew fixes all but two of the newly reported problems while all the unittests continue to pass:

    -    if to_encode[0] in WSP:
+    if last_ew is None and to_encode[0] in WSP:

    The two which are broken are: Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„ and Ņ„ Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„Ņ„ Ņ„ Ņ„

    I believe something like the above should make it into the final fix as the comment for this block says that it should only be invoked if we're joining this encoded-word to a non-encoded word but there's nothing in this condition which checks that the previous word was non-encoded.

    And elif to_encode[0]: might also do the right thing here.

  8. abadger
    added a commit that references this issue on May 4, 2022

    Fix for email.generator.Generator with whitespace between encoded words.

    eb1cdc1

  9. abadger commented on May 4, 2022

    @abadger
    abadger
    on May 4, 2022
    Contributor

    I've opened a PR that fixes the cases where we need spaces between encoded words.

    I've figured out that the remaining two problems start with a space. I'm searching the RFCs but so far haven't found anything that says whether the generator should handle this by including the initial space in the initial encoded word or if the decoder should handle it by displaying the space (or as a third option, that initial spaces in a Subject should be ignored).

  10. abadger
    added 2 commits that reference this issue on Apr 26, 2023

    Fix for email.generator.Generator with whitespace between encoded words.

    80f5cfa

    Fix for email.generator.Generator with whitespace between encoded words.

    69b205d

  11. abadger commented on Apr 27, 2023

    @abadger
    abadger
    on Apr 27, 2023
    Contributor

    I believe #92281 now fixes all the problems reported here. If anyone would like to test that the problems are resolved by it, that would be appreciated!

  12. abadger
    added a commit that references this issue on Jul 21, 2023

    Fix for email.generator.Generator with whitespace between encoded words.

    4104b8e
  13. warsaw
    added a commit that references this issue on May 20, 2024

    gh-92081: Fix for email.generator.Generator with whitespace between eâ€Ķ

    a6fdb31
  14. miss-islington
    added 2 commits that reference this issue on May 20, 2024

    pythongh-92081: Fix for email.generator.Generator with whitespace betâ€Ķ

    6f431e7

    pythongh-92081: Fix for email.generator.Generator with whitespace betâ€Ķ

    3a96e43
  15. warsaw
    added 2 commits that reference this issue on May 20, 2024

    [3.13] gh-92081: Fix for email.generator.Generator with whitespace beâ€Ķ

    054f1af

    [3.12] gh-92081: Fix for email.generator.Generator with whitespace beâ€Ķ

    ffe9ba0
  16. warsaw
    self-assigned this
    on May 20, 2024

  17. warsaw commented on May 20, 2024

    @warsaw
    warsaw
    on May 20, 2024
    Member

    Thanks for the fix @abadger

  18. warsaw
    closed this as completedon May 20, 2024
  19. estyxx
    added a commit that references this issue on Jul 17, 2024

    pythongh-92081: Fix for email.generator.Generator with whitespace betâ€Ķ

    fb8d57a
  20. bitdancer
    added a commit that references this issue on Feb 19, 2026

    gh-144156: Fix email header folding concatenating encoded words (#144692

    0f7cd55
  21. miss-islington
    added a commit that references this issue on Feb 19, 2026

    pythongh-144156: Fix email header folding concatenating encoded words (â€Ķ

    b7ec168
  22. bitdancer
    added 2 commits that reference this issue on Feb 19, 2026

    [3.14] gh-144156: Fix email header folding concatenating encoded words (

    1404a4d

    [3.13] gh-144156: Fix email header folding concatenating encoded words (

    a3c0a80
  23. brijkapadia
    added a commit that references this issue on Feb 28, 2026

    pythongh-144156: Fix email header folding concatenating encoded words (â€Ķ

    21ab5a3
  24. ljfp
    added a commit that references this issue on Apr 25, 2026

    pythongh-144156: Fix email header folding concatenating encoded words (â€Ķ

    94812d8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

topic-emailtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions