HTML to plain text: how to prevent two paragraphs from being stuck together

This paper uses synthetic samples to demonstrate the processing method. The code has been verified using examples in an isolated environment; this does not guarantee effectiveness for all real files or products. Before processing important files, please retain the original copies and verify them against the boundaries and acceptance criteria outlined in this document. When you use Beautiful Soup to extract web page content, the most common issue isn't extraction failure, but rather that the extracted text gets "stuck" together. For example, <p>Hello</p><p>World</p> is parsed into HelloWorld , or line breaks have been lost between list items. This usually happens because of a direct call to get_text() does not specify a delimiter strategy; by default, the parser seamlessly concatenates all NavigableString objects. The key to solving this issue lies in correctly combining the use of separator and strip parameter, rather than attempting to “guess” where line breaks should occur after extraction using regular expressions. The pitfalls of default behavior and parameter semantics get_text() method accepts two key parameters: separator and strip . If no argument is passed, it will directly concatenate all text nodes in the tag tree. For block-level elements (such as div , p , li ), this default behavior is disastrous, because line breaks and indentation in the HTML source often become whitespace strings or are ignored after parsing, leading to the loss of semantic boundaries. separator parameter defines the insertions between text nodes. Set it to '\n' is the most common approach, but it is not perfect. If the HTML structure is deeply nested, a simple global line break may result in numerous blank lines. strip The parameter controls whether to remove leading and trailing whitespace from each text fragment. When strip=True At that time, Beautiful Soup will first process each independent NavigableString. .strip() , and then concatenate using the separator. This means <p> Hello </p> will become a clean Hello , avoids \n Hello \n This kind of output containing dirty data. The two must be used together: using only separator without strip will retain meaningless indentation spaces; using only strip without separator, although each line becomes clean, the lines may still stick together. Synthesize DOM boundary test cases To validate the extraction logic, we need to construct a synthetic HTML sample containing typical boundary issues, rather than relying on unstable real-world web pages. Below is a specially designed test case that covers three scenarios: block-level element adhesion, inline element spacing, and extraneous whitespace. input sample (synthetic example): <div> <p>第一段文字</p> <p>第二段文字<span>内联补充</span></p> <ul> <li> 列表项A </li> <li>列表项B</li> </ul> </div> Expected correct output: 第一段文字 第二段文字 内联补充 列表项A 列表项B Note that, due to separator='\n' operates on all text nodes, while the separator acts on extracted text nodes and does not automatically insert double line breaks based on block-level tags. A clear line break must be present between paragraphs and list items, and any extraneous spaces before or after list items should be removed. “Second paragraph of text” and “inline supplement,” being in different tags and both being text nodes, will be separated by the separator. error/boundary input handling: If the input becomes <p>End</p><script>alert(1)</script><p>Start</p> , and script tags are not filtered; whether script nodes are included in the text-dependent parser and node type cannot treat all patterns as the same behavior. At this point, even if a separator is set, the output will still contain alert(1) . The correct approach is to invoke get_text First, by means of soup.find_all('script') removes irrelevant tags, or skips specific tag types during generator traversal. This reminds us that get_text The parameters only address formatting issues and do not handle content cleansing. Python Implementation and Assertion Validation The following code demonstrates how to construct the aforementioned sample in memory and perform deterministic verification. This example relies on beautifulsoup4 library (recommended 4.9.0+ version). If you need to test external files, first save the HTML as a UTF-8-encoded file and use open('file.html', encoding='utf-8') is read to avoid parsing exceptions caused by encoding errors. The code does not include network requests and is fully self-contained. from bs4 import BeautifulSoup # 合成示例：模拟常见的HTML文本边界问题 html_content = """ <div> <p>第一段文字</p> <p>第二段文字<span>内联补充</span></p> <ul> <li> 列表项A </li> <li>列表项B</li> </ul> </div> """ soup = BeautifulSoup(html_content, 'html.parser') # 关键配置：separator定义边界，strip清理碎片空白 text_result = soup.get_text(separator='\n', strip=True) # 过滤空行以获得干净的语义行 lines = [line for line in text_result.split('\n') if line] # 验收断言：确保边界清晰且无脏数据 assert lines[0] == '第一段文字', f"段落粘连或内容错误: {lines[0]}" assert lines[1] == '第二段文字', "块级元素间未正确分隔" assert lines[2] == '内联补充', "内联元素文本节点被错误合并" assert lines[3] == '列表项A', f"列表项前置空格未清除: '{lines[3]}'" assert lines[4] == '列表项B', "列表项间边界丢失" print("提取结果预览:") for i, line in enumerate(lines): print(f"{i}: [{line}]") The expected output of this code is five lines of clean text. If your environment produces additional blank lines, it indicates that there are pure whitespace NavigableString nodes in the DOM that have not been stripped; this is html.parser Its normal behavior can be filtered using a list comprehension after the split. Do not use this code directly on complex pages in production environments; it is intended solely to demonstrate the mechanics of parameter combinations. Acceptance Criteria and Manual Review Thresholds automated extraction can never achieve the visual fidelity of 100%. The table below outlines how to distinguish between “code that is repairable” and “requires manual intervention” in practical document processing. These judgments are based on observations of the DOM structure, not subjective impressions. explicitly specifies the scope of inapplicability: when HTML contains large amounts of text generated by CSS pseudo-elements, Canvas-rendered content, or Shadow DOM, Beautiful Soup cannot extract meaningful information. In such cases, debugging parameters should not be continued; instead, the data collection strategy should be changed, or missing information should be accepted. References and Next Steps Beautiful Soup Official Documentation: get_text() Next steps: In your project, create a synthetic test set containing 5–10 typical boundary cases, and run the aforementioned assertion script. If the pass rate falls short of expectations, verify whether preprocessing of the DOM—such as removing script/style elements or expanding specific tags—is required before extraction. For structural issues that cannot be resolved through parameter adjustments, document the specific HTML fragments to serve as benchmark samples for evaluating specialized document-processing tools. The specific handling decision for the exception observation What should be done should not infer inline node separator checks whether the sentence has been fragmented separator applies to text node table flattening preserves the row-and-column structure and is processed separately cannot rely solely on get_text dynamically rendered content explicitly captures the gap static parsing is not equivalent to browser rendering uses this checklist for document translation The body text, links, code, and structure of the web document should be processed separately. Include the protected sections in the acceptance checklist, then evaluate the readability and visual presentation of the translated page. OmniDoc (Product Entry) provides an entry point for translating multi-format documents, and the official website highlights its OCR, terminology database, and bilingual alignment features. For teams that wish to start processing directly from files, this is a path worth evaluating with real-world samples. Please use de-identified files that you are authorized to upload, and verify the results against the checklist provided in this document; we do not claim that OmniDoc utilizes the open-source libraries or internal algorithms referenced herein, nor do we substitute examples for product performance testing. - 1 -