Repository navigation
email.message get_payload throws UnicodeEncodeError with some surrogate Unicode characters #94606
Description
email.message get_payload gets a UnicodeEncodeError if the message body contains a line that has either:
a Unicode surrogate code point that is valid for surrogateescape encoding (U-DC80 through U-DCFF) and a non ASCII UTF-8 character
OR
a Unicode surrogate character that is not valid for surrogateescape encoding
Here is a minimal code example with one of the cases commented out
from email import message_from_string
from email.message import EmailMessage
m = message_from_string("surrogate char \udcc3 and 8-bit utf-8 ÃŦ on same line")
# m = message_from_string("surrogate char \udfff does it by itself")
payload = m.get_payload(decode=True)
On my python 3.10.5 on macOS this produces:
Traceback (most recent call last):
File "/Users/sidney/tmp/./test5.py", line 8, in <module>
payload = m.get_payload(decode=True)
File "/usr/local/Cellar/python@3.10/3.10.5/Frameworks/Python.framework/Versions/3.10/lib/python3.10/email/message.py", line 264, in get_payload
bpayload = payload.encode('ascii', 'surrogateescape')
UnicodeEncodeError: 'ascii' codec can't encode character '\xeb' in position 33: ordinal not in range(128)
This was tested on python 3.10.5 on macOS, however I tracked it down based on a report in the wild that was running python 3.8 on Ubuntu 20.04 processing actual emails
Linked PRs
- gh-94606: Fix error when message with Unicode surrogate not surrogateescaped string #94641
- [3.12] gh-94606: Fix error when message with Unicode surrogate not surrogateescaped string (GH-94641)Â #112971
- [3.11] gh-94606: Fix error when message with Unicode surrogate not surrogateescaped string (GH-94641)Â #112972
Activity
sidney commentedon Jul 6, 2022
I tracked this down to code in email.message.py that has the following
This looks like it is supposed to check if payload is a string that was created by decode with surrogateescape, and if it was, then convert it back to bytes the same way. PEP 383 is clear that surrogateescape is only for round trip decoding then encoding. encode with surrogateescaape should never be called on strings not created with decode surrogateescape.
The problem is that utils._has_surrogates(payload) is only a fast heuristic when it is used to guess if payload contains a string that was produced by decoding with surrogateescape. The function actually flags strings that contain any Unicode surrogate characters. However, strings that were created by decode with surrogateescape will only have Unicode surrogate characters in the U-DC80 through U-DCFF range and remaining characters will only be 7 bit ASCII. If a string either has an non-ASCII UTF8 character in addition to a Unicode surrogate character, or has a Unicode surrogate character out of that range, utils._has_surrogates() will return true but the string encode will raise the exception.
This should be able to be fixed by catching the exception raised by the encode and proceeding to get the same result the code would have if utils._has_surrogates(payload) had returned false.
I'll submit a PR for that after testing.
pythongh-94606: Fix error when message has Unicode surrogate but is nâĶ
pythongh-94606: Condense unit tests for issue into one parameterized âĶ
gh-94606: Fix error when message with Unicode surrogate not surrogateâĶ
pythongh-94606: Fix error when message with Unicode surrogate not surâĶ
pythongh-94606: Fix error when message with Unicode surrogate not surâĶ
[3.11] gh-94606: Fix error when message with Unicode surrogate not suâĶ
[3.12] gh-94606: Fix error when message with Unicode surrogate not suâĶ
pythongh-94606: Fix error when message with Unicode surrogate not surâĶ
pythongh-94606: Fix error when message with Unicode surrogate not surâĶ