[MaxFreedomPollard]

Description

EPUBParser._html_to_markdown strips every line of the markdownify output and drops the blank ones (docreader/parser/epub_parser.py:257-259). Markdown relies on those blank lines and that indentation to separate blocks, so every EPUB chapter comes out with broken structure:

• consecutive paragraphs merge into one;
• a paragraph directly above a scene break (<hr/>, written as ---) becomes a setext <h2> heading;
• a paragraph right after a blockquote or a list item is pulled into it as a lazy continuation line;
• a line right after a table is read by GFM as another table row;
• nested lists flatten, and <pre> code loses its indentation and blank lines.

The repo's own fixture internal/infrastructure/docparser/anydoc/testdata/upstream/epub/book.epub shows it. Excerpt of EPUBParser().parse_into_text(...) on main:

A list of things:
1. first
2. second
* nested bullet
3. third
See Chapter Two for the table, or jump straight to the marked paragraph.
...
| Bolts | 12 |
| Nuts | 30 |
This paragraph carries an inline anchor and is not a heading.

Rendered as CommonMark + GFM tables, the nested bullet splits the ordered list in two, "See Chapter Two…" becomes part of item 3, and "This paragraph…" becomes a third table row. With this change:

A list of things:

1. first
2. second
   * nested bullet
3. third

See Chapter Two for the table, or jump straight to the marked paragraph.
...
| Bolts | 12 |
| Nuts | 30 |

This paragraph carries an inline anchor and is not a heading.

The missing blank lines also matter for chunking: the Go splitter's first separator is "\n\n" (internal/infrastructure/chunker/strategy.go:196), and the body of an EPUB chapter had none.

The MHTML parser had the same line stripping until #1743 replaced it with MHTMLParser._normalize_markdown, which static HTML uploads also use through MHTMLParser.html_to_markdown. #3978 noted that the EPUB parser keeps its own copy of the conversion. That copy now calls the same normalizer, which only collapses runs of blank lines, keeps indentation and two-space hard breaks, and leaves fenced code verbatim. The ZIP fallback (used when ebooklib cannot read the file) goes through the same helper, so both paths are fixed. Image rewriting, link unwrapping and the rest of the EPUB conversion are unchanged. One visible side effect: a leading ideographic space (U+3000), which some Chinese EPUBs put in the text as a paragraph indent, is no longer stripped, the same as in HTML and MHTML output.

EPUBs reach this parser through the builtin (DocReader) engine: by default when the app is built without anydoc, and whenever a knowledge base's parser rules select builtin for epub.

Type of Change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change
• 📚 Documentation update
• 🎨 Refactor
• ⚡ Performance improvement
• 🧪 Test
• 🔧 Configuration / Build / CI

Related Issue

No issue filed. Related: #1743 (same fix for MHTML), #3978 (notes the separate EPUB copy).

Testing

New test test_chapter_markdown_keeps_block_structure in docreader/tests/test_epub_parser.py parses a one-chapter EPUB with a scene break, a blockquote, a nested list, a <pre> block and a table, through both the ebooklib path and the ZIP fallback. Python 3.10.22 on macOS arm64, dependencies from uv sync --project docreader --locked --no-dev.

$ uv run --project docreader python -m unittest docreader.tests.test_epub_parser
Ran 13 tests in 0.145s
OK

$ uv run --project docreader python -m unittest docreader.tests.test_epub_parser docreader.tests.test_mhtml_parser docreader.tests.test_html_parser docreader.tests.test_parser_routing
Ran 57 tests in 0.370s
OK

$ uv run --project docreader python -m compileall -q docreader
(no output)

$ git diff --check origin/main...HEAD
(no output)

Restoring only epub_parser.py from main and keeping the new test gives the failure this fixes (both subtests fail the same way):

FAIL: test_chapter_markdown_keeps_block_structure (docreader.tests.test_epub_parser.EPUBParserTest) (use_fallback=False)
AssertionError: 'It was night.\n\n---\n\nMorning came.' not found in '## Structured EPUB\n\n\n\n## Chapter\n\nIt was night.\n---\nMorning came.\n> A quoted line.\nAfter the quote.\n* parent\n+ child\n```\ndef total(items):\nreturn sum(items)\n```\n| Name | Qty |\n| --- | --- |\n| Apple | 3 |\nAfter the table.'

Full DocReader suite (python -m unittest discover -s docreader/tests -p "test_*.py"): 389 tests, 12 failing (11 failures, 1 error), the same 12 as on main in this environment and unrelated to EPUB: 2 in test_ssrf (DNS lookups are blocked here), 1 in test_ssrf_proxy (Playwright WebKit not installed) and 9 in test_preprocess_runner (the sandbox stops the worker subprocess). The Go gate (make lint / make test) was not run; no Go file changed.

Checklist

• git diff --check origin/main...HEAD passes
• Changed source files are formatted
• Targeted tests for the changed packages/components pass
• Diff-scoped lint passes where applicable (for Go: golangci-lint run --new-from-rev=origin/main ./...)
• Full-repository checks were run, or any unrelated/environment-dependent failures are documented above
• Self-reviewed the code
• Added/updated tests covering the change
• Updated related documentation (README, website-docs/, Swagger annotations, etc.)
• Breaking changes are clearly called out in the description above

Screenshots / Recordings

None. No UI change.

🤖 Generated with Claude Code