applying a black box to a PDF is not anonymization: what checks are actually required for true deletion

This paper uses synthetic samples to demonstrate the processing method. The code has been verified using examples in an isolated environment; this does not guarantee effectiveness for all real files or products. Before processing important files, please retain the original copies and verify them against the boundaries and acceptance criteria outlined in this document. Why can sensitive words still be searched for even after a black box has been applied? draws a black rectangle over the ID number in the PDF, saves it, and sends it to a colleague; the recipient can still use Ctrl+F to search for the complete number. This is not a software bug, but rather a consequence of the PDF’s structural characteristics: add_redact_annot It merely superimposes an annotation object on top of the page content stream, leaving the original text, vector paths, and even the pixel data of embedded images intact in the file. True de-identification must be performed. apply_redactions() , this operation physically removes the content objects corresponding to the occluded region and rewrites the page stream with a fill color. Without this step, all visual occlusions are merely a “self-deceptive” layering game. Synthesis Example: Comparison Between Correct De-identification and Invalid Masking To illustrate the issue, we construct a fully memory-based synthetic PDF sample that does not rely on any external files. The example includes two scenarios: one where only red-box annotations are added without applying them, and another where both the addition and application processes are fully executed. correctly outputs the expected : After application-level masking, get_text() The returned string should no longer contain the target sensitive term “CONFIDENTIAL-2024”; meanwhile, the rendered Pixmap image in that area should be a solid white block, with no residual strokes or anti-aliased edges. Error/Boundary Input Handling : If only calling add_redact_annot while skipping apply_redactions , the text extraction results still contain the full sensitive words. This is the most common logical omission. Another edge case is when coordinates are calculated based on the rotated page, but PyMuPDF requires that all coordinate values used in write operations must be relative to the unrotated page. If a rectangle in the rotated coordinate system is passed in, the masking position will be offset, leading to partial exposure of sensitive information. In this situation, one should use page.derotation_matrix Perform coordinate transformation, or call before operation page.remove_rotation() 。 PyMuPDF Data Masking and Double-Verification Code The following Python example uses the PyMuPDF (fitz) library and must be run in a locked dependency environment (such as PyMuPDF>=1.23.0 ). The code constructs the PDF entirely in memory, without any network requests or file I/O, making it suitable for integration testing or unit validation within CI pipelines. Note: This script is intended solely for logical demonstration and is not a production-grade data masking tool; real documents must be prepared by the user and their compliance assessed independently. import fitz # PyMuPDF def create_sample_pdf_with_sensitive_text(): doc = fitz.open() page = doc.new_page() page.insert_text((72, 72), "Public info here.", fontsize=12) page.insert_text((72, 100), "CONFIDENTIAL-2024", fontsize=12) return doc def redact_and_verify(doc, target_text="CONFIDENTIAL-2024"): page = doc[0] rects = page.search_for(target_text) if not rects: raise ValueError(f"Target text '{target_text}' not found") # 添加脱敏注释（仅标记，未删除） for rect in rects: page.add_redact_annot(rect, fill=(1, 1, 1)) # white fill # 关键步骤：物理应用脱敏（images=2表示像素级擦除，graphics=1表示移除被覆盖的矢量图形） page.apply_redactions(images=2, graphics=1) # 验证1：文本层检查 extracted = page.get_text() assert target_text not in extracted, "Text layer still contains sensitive data" # 验证2：渲染层检查（可选，用于视觉确认） pix = page.get_pixmap(dpi=144) # 实际项目中可保存pix或比对像素值，此处仅作流程示意 return True # 合成示例执行 sample_doc = create_sample_pdf_with_sensitive_text() redact_and_verify(sample_doc) In the code above, assert statement ensures that text-layer cleanup takes effect. If the assertion fails, it indicates that desensitization was not actually performed or that the coordinates are incorrect. Although rendering validation is not quantitatively assessed in the code, it remains indispensable during manual review or automated visual regression testing—some font-rendering engines may retain glyph-outline caches even after text has been removed, making such issues undetectable through text extraction alone. Acceptance Criteria and Manual Review Boundary code passing assertions does not guarantee that the document has been securely delivered. The table below lists the observation points and decision criteria to be monitored during actual acceptance testing, distinguishing between “successful code demonstration” and “effective end-to-end data masking.” Test samples must be constructed by users themselves or processed through data masking; the direct use of production documents containing real sensitive information is prohibited: Observation of the phenomenon possible causes processing decision get_text() contains no sensitive words, but the rendered image has ghosting font subset not cleaned or vector path not overwritten reruns with the parameters images=2 and graphics=1 applied to apply_redactions desensitized region position offset page is rotated, and the coordinates have not been transformed uses derotation_matrix for correction or first remove_rotation Sensitive words in the table have not been fully obscured. search_for returns multiple fragmented rectangles manually merge rectangles or use find_tables to locate cell ranges metadata/bookmarks/attachments still contain sensitive information apply_redactions applies only to the page content stream additionally calls doc.set_metadata({}), cleans up the outline, and removes embedded files Scope of Non-Applicability : This method is not applicable to scanned PDFs (without a text layer), encrypted documents (which must be decrypted first), or dynamic content composed of complex form fields/XFA. In these cases, the content must first be processed via OCR or converted into static pages, and after de-identification, each page must be manually reviewed. The code demonstration only verifies “the disappearance of text strings” and cannot replace a comprehensive assessment of visual integrity, contextual semantic disruptions, or cross-object association leaks. Reference and Next Step PyMuPDF Page.apply_redactions Official Documentation: https://pymupdf.readthedocs.io/en/latest/page.html#Page.apply_redactions PyMuPDF coordinate system description: https://pymupdf.readthedocs.io/en/latest/page.html#coordinates Next steps: In your test environment, reproduce the above verification process using a composite PDF containing rotated pages, tables, and embedded images; document the impact of each combination of apply_redactions parameters on output file size and rendering quality; establish an internal data masking acceptance checklist, specifying which inspection items can be automated and which require manual approval. uses this checklist for document translation first ensures that the page order, source locations, and text integrity have been thoroughly checked, then submits the same de-identified sample to the translation process. After translation, it is still necessary to verify that the headings, body text, and figure captions correspond correctly; one should not rely solely on whether the text flows smoothly. OmniDoc (Product Entry) provides an entry point for translating multi-format documents, and the official website highlights its OCR, terminology database, and bilingual alignment features. For teams that wish to start processing directly from files, this is a path worth evaluating with real-world samples. Please use de-identified files that you are authorized to upload, and verify the results against the checklist provided in this document; we do not claim that OmniDoc utilizes the open-source libraries or internal algorithms referenced herein, nor do we substitute examples for product performance testing. - 1 -