阅读文本 / Reading text
This paper uses synthetic samples to demonstrate the processing method. The code has been verified using examples in an isolated environment; this does not guarantee effectiveness for all real-world files or products. Before processing important files, please retain the original copies and verify them against the boundaries and acceptance criteria outlined in this document.
Which problem does this paper address first?
The baseline for urljoin should be the original page URL; do not arbitrarily append a trailing slash to convert a file URL into a directory. This example handles only a[href] and img[src], without overriding CSS, srcset, or base tags; rewriting URLs is also not part of HTML sanitization.
self-contained synthetic example
dependencies are listed in the import list. The code constructs inputs in memory; runtime logs are provided separately and do not constitute a guarantee for all real-world files.
from urllib.parse import urljoin, urlsplit
from bs4 import BeautifulSoup
def rewrite(markup, page_url):
soup = BeautifulSoup(markup, "html.parser")
for tag, attr in [("a", "href"), ("img", "src")]:
for node in soup.find_all(tag):
value = node.get(attr)
if not value or value.startswith("#"):
continue
result = urljoin(page_url, value)
if urlsplit(result).scheme not in {"https", "http"}:
del node[attr]
else:
node[attr] = result
return soup
result = rewrite('<img src="logo.png"><a href="../guide/">Guide</a><a href="#top">Top</a><a href="javascript:alert(1)">Bad</a>', "https://example.com/docs/page.html")
assert result.img["src"] == "https://example.com/docs/logo.png"
assert result.find_all("a")[0]["href"] == "https://example.com/guide/"
assert result.find_all("a")[1]["href"] == "#top"
assert "href" not in result.find_all("a")[2].attrs
When enterprises migrate internal documents, wikis, or legacy sites to a new domain, the most common “supernatural phenomena” they encounter are broken images and incorrect link redirects. This is usually not due to resource loss, but rather
parsing context lost
. The browser parses relative paths (such as
images/logo.png
), strictly adhering to the RFC 3986 standard: prioritize searching for
<base href="...">
tag; if absent, calculate it based on the current document’s URL. If the exported static HTML neither injects the correct
<base>
tag, which has also been detached from the original directory structure, causes all relative references to point to the root directory of the new domain or to the current page’s path, resulting in resource 404.
urljoin and the semantic pitfalls of the base tag
urllib.parse.urljoin(base, url)
is the core tool for handling link rewriting, but its behavior differs from how browsers handle
<base>
There are subtle differences in tag parsing, which is also the root cause of many scripts that “work after being fixed.”
urljoin
defaults to
base
The argument is treated as a “file path” rather than a “directory,” unless it begins with
/
at the end. For example,
urljoin("https://new.com/docs", "img.png")
The result is
https://new.com/img.png
(docs is discarded as a filename), rather than the expected
https://new.com/docs/img.png
。
In contrast, HTML
<base>
Although tag parsing in browsers also relies on trailing slashes to distinguish files from directories, it exhibits slightly higher tolerance for non-standard input. In programmatic processing, we cannot assume the validity of input data. Therefore, when invoking
urljoin
Preprocessing must be performed first: checking the validity of the scheme and forcibly appending the directory suffix.
/
, filter out null values. Only standardized base URLs can serve as reliable inputs. Furthermore, for paths that are already absolute (starting with
/
at the beginning) or links containing the full protocol,
urljoin
will either return the original value directly or replace only the hostname; this feature makes it more secure than string concatenation, but it also requires developers to clearly distinguish between the boundaries of “rewrite” and “preserve.”
synthesis example and boundary condition handling
The following code demonstrates how to safely rewrite resource links in HTML.
Note: This is a synthetic teaching example
, designed to demonstrate the logic rather than a production‑grade implementation. The example uses in‑memory strings to simulate input, thereby avoiding file system operations and network requests; please verify the logic in a controlled environment.
Acceptance Criteria and Manual Review Boundary
automation scripts can resolve mechanical path-joining issues, but they cannot replace the judgment of business semantics. The following table lists common exceptions and their handling decisions, helping teams distinguish between scenarios that are “code-resolvable” and those that require human intervention:
Scope of Non-Applicability Statement
: This method applies only to static HTML attribute rewriting. For links generated at runtime by JavaScript, references within the Shadow DOM, and CSS
background-image
URLs in the content, or Canvas/WebGL texture paths, are inaccessible to BeautifulSoup. Such cases must be validated in a real browser environment, or subjected to end-to-end visual regression testing using Playwright/Puppeteer. Code demonstrations do not equate to delivery quality; final acceptance must be based on the rendering results produced by the user agent.
References and Original Manufacturer Sources
Beautiful Soup Official Documentation - Attributes: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#attributes
Python urllib.parse.urljoin documentation: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urljoin
RFC 3986 URI Generic Syntax: https://datatracker.ietf.org/doc/html/rfc3986
Next steps: Add assertion checks for rewritten links to your CI/CD pipeline; perform sample-based validation of resource accessibility on historically exported documents; if you frequently encounter issues with dynamic content links, assess the necessity of introducing a headless browser for visual regression testing. Remember that any automated fix should be regarded as a “supporting measure,” not a “silver bullet.”
The decision on handling the specific exception
observation
What should be done
should not infer
File-type page URL
retains the original baseline
without a trailing slash
fragment anchor
retains the citation on this page
is not required to be changed to the remote page
script protocol
Removing attributes in the example
complete HTML still requires security sanitization
uses this checklist for document translation
The body text, links, code, and structure of the web document should be processed separately. Include the protected sections in the acceptance checklist, then assess the post-translation readability and page display.
OmniDoc (Product Entry)
provides an entry point for translating multi-format documents, and the official website highlights its OCR, terminology database, and bilingual alignment features. For teams that wish to start processing directly from files, this is a path worth evaluating with real-world samples. Please use de-identified files that you are authorized to upload, and verify the results against the checklist provided in this document; we do not claim that OmniDoc utilizes the open-source libraries or internal algorithms referenced herein, nor do we substitute examples for product performance testing.
- 1 -
Updated 2026-10-08