The same piece of malformed HTML—why do two parsers produce different results?

This paper uses synthetic samples to demonstrate the processing method. The code has been verified using examples in an isolated environment; this does not guarantee effectiveness for all real files or products. Before processing important files, please retain the original copies and verify them against the boundaries and acceptance criteria outlined in this document. When you use Beautiful Soup to extract the main content of a webpage, the most insidious issue isn’t an error message—it’s “no error, but the content is missing.” An HTML snippet lacking closing tags, in html.parser may automatically close the paragraph according to the specification, replaced by lxml may, however, exhibit structural discrepancies due to differing repair strategies. This is not a code logic error but rather a consequence of the parser’s varying tolerance for “invalid tokens.” Beautiful Soup is merely a wrapper; the actual construction of the parse tree is handled by the underlying parser, and the official documentation explicitly warns that, if the document is invalid, different parsers will produce distinct trees. The fault-tolerance logic of the three parsers diverges Python's standard library comes with html.parser pursues zero dependencies and moderate speed, and its fault-tolerance rules adhere to the HTML specification. When encountering an unclosed <p> tag, which automatically closes the preceding paragraph without nesting it; however, for <div> Regarding unclosed container tags, it does not automatically self-close them, which may cause subsequent content to be incorrectly attributed to the parent element. For well-formed documents, it is a safe and lightweight choice; however, for poorly structured, incomplete web pages, its output, while conforming to the specification, may not align with visual expectations. lxml Based on the C-language library libxml2, it is the fastest, with fault tolerance falling between that of the standard library and browsers. It attempts to repair obvious structural errors, such as automatically closing certain block-level elements, but its repair logic adheres to the XML/HTML hybrid specification rather than modern browser rendering standards. This means that when handling HTML5’s new features or extremely malformed legacy pages, lxml The generated tree may differ from what you see in Chrome DevTools. html5lib is the only pure Python parser that strictly emulates browser parsing behavior. No matter how fragmented the input, it always generates a valid DOM tree compliant with the HTML5 specification, including automatic completion <head> 、 <body> and handling complex nested corrections. The trade-off is extremely slow performance and the introduction of additional Python dependencies. When your task is to restore the document’s true visual structure in the browser—such as preserving context for translation—this “slowness” often represents a necessary cost in accuracy. synthesis example: Ambiguity in unclosed tags To specifically demonstrate the differences, we construct a typical piece of corrupted HTML as synthesis example . Note that this code is intended solely for demonstrating parser behavior and does not represent real production data: <div class="content"> <p>第一段文字 <p>第二段文字 <span>高亮内容</div> This markup has two issues: two <p> tags are all unclosed, and <span> spans an implicit paragraph boundary. The correct semantic expectation is: two independent paragraphs plus an inline highlight, or the highlight belongs solely to the second paragraph. In html.parser Below, although the two <p> will be correctly recognized as a sibling (due to automatic closing in the specification), but <span> may be parsed as enclosing the second paragraph up to </div> ends. And at html5lib Below, the parser will process strictly according to the browser's algorithm. <span> the nested correction, which might confine it to the second paragraph or involve more complex reorganization. If you use .find('p').get_text() extracts text; the string contents returned by the two parsers may be identical, but .find('span') The parent node or sibling node relationships will be entirely different. boundary cases are equally worthy of caution. If the input is an empty string, plain text without tags, or an HTML fragment containing an XML declaration, html5lib automatically adds the complete document skeleton ( <html><head><body> ), whereas html.parser remains unchanged. If downstream code assumes that the root node is the target container, html5lib The output will then have an additional layer of nesting, causing the selector to fail. lock parser and version verification code The following Python example demonstrates how to compare parser outputs in memory and use assertions to verify the expected behavior. This code is for instructional demonstration and must be used when locked beautifulsoup4 、 lxml 、 html5lib runs in the specific version's environment , because parser upgrades may alter fault-tolerance details. The example does not rely on the network or external files; all inputs are synthetic data. from bs4 import BeautifulSoup # 合成示例：未闭合标签的损坏HTML BROKEN_HTML = '<div class="c"><p>A<p>B<span>X</div>' def parse_and_extract(html: str, parser: str) -> str: soup = BeautifulSoup(html, parser) # 提取div直接子级p标签的文本数量，反映结构差异 div = soup.find('div', class_='c') if not div: return "NO_DIV" p_tags = div.find_all('p', recursive=False) return f"P_COUNT:{len(p_tags)}" # 预期：html5lib应识别出2个平级p标签 result_html5lib = parse_and_extract(BROKEN_HTML, 'html5lib') assert result_html5lib == "P_COUNT:2", f"html5lib 预期2个p，实际: {result_html5lib}" # 预期：html.parser也应识别出2个p（规范自动闭合） # 注意：此处不断言相等，仅记录观察值供人工判断，因版本间可能存在细微差异 result_std = parse_and_extract(BROKEN_HTML, 'html.parser') print(f"html.parser 结果: {result_std}") # 验证空输入边界 empty_result = parse_and_extract("", 'html5lib') assert empty_result == "NO_DIV", "空输入应返回NO_DIV" The core of this code segment is not “tests passing,” but rather establishing a reproducible observational baseline. assert statement locks your access to html5lib The expectation of behavior; whereas for html.parser only prints the result, as its output may vary across versions. In production projects, the selected parser name and version number should be documented in a configuration file or code comments to prevent team members from obtaining different results due to environmental differences. Parser Selection Acceptance and Manual Review Checklist Observation of the phenomenon possible causes processing decision non-applicable scenario The same selector returns a different number of nodes in different environments parser is not fixed or the version is inconsistent locks beautifulsoup4 and the parser version shall uniformly use html5lib or lxml has confirmed that all inputs are strictly valid HTML extracted text contains script/style content parser failed to correctly recognize tags, or the BS4 version is too outdated switches to html5lib , or use BS4's Script / Stylesheet class filtering needs to retain the original script content for analysis table/list structure flattening loses hierarchy html.parser has insufficient fault tolerance for complex nesting is changed to lxml or html5lib , prioritize the latter only needs to extract plain text without preserving the structure parsing takes longer than the acceptable threshold html5lib Performance bottleneck Switch to using for large-scale tasks lxml , sampling uses html5lib validation single parsing of small documents output DOM contains redundant <html><body> wraps html5lib Automatically completes the document skeleton adjusts the selector from body begins, or post-processing strips requires a complete HTML5 document structure important distinction : The table above pertains to “code-level parsing behavior” and does not equate to end-to-end document conversion performance. Even if the parser generates a correct parse tree, subsequent steps such as serialization, style reconstruction, and bilingual alignment may still introduce discrepancies. Manual review is required in the following cases: when the source document contains a large number of custom tags or proprietary attributes; when HTML fragments originate from PDF/Word conversions rather than native web pages; or when business logic depends on a specific text order that is visually controlled by CSS rather than determined by the DOM structure. References and Next Steps Beautiful Soup Official Documentation: Differences between parsers Beautiful Soup Official Documentation: Installing a parser Next step: Select a primary parser for your project (recommended html5lib (used in high-fidelity scenarios), write its name and version number into the dependency lockfile; supplement the 3-5 actual boundary cases you have encountered using the synthesis example template provided in this paper, thereby establishing an internal team baseline document for parsing behavior; add parser-version checks to existing extraction scripts to prevent silent drift. uses this checklist for document translation The body text, links, code, and structure of the web document should be processed separately. Include the protected sections in the acceptance checklist, then evaluate the readability and visual presentation of the translated content. OmniDoc (product entry point) provides an entry point for translating multi-format documents, and the official website highlights its OCR, terminology database, and bilingual alignment features. For teams that prefer to start processing directly from files, this is a path worth evaluating with real-world samples. Please use de-identified files that you are authorized to upload, and verify the results against the checklist provided in this document; we do not claim that OmniDoc utilizes the open-source libraries or internal algorithms referenced herein, nor do we substitute examples for formal product performance testing. - 1 -