When parsing PDFs in batch, why should each process open the file itself?

This paper uses synthetic samples to demonstrate the processing method. The code has been verified using examples in an isolated environment; this does not guarantee effectiveness for all real files or products. Before processing important files, please retain the original copies and verify them against the boundaries and acceptance criteria specified in the text. Which problem does this paper address first Multi-process tasks should pass serializable parameters, with each worker process independently opening the document rather than sharing Document or Page objects. The following tests segmental coverage and independently opened worker functions; to maintain portability and auditability, the example uses serial invocation and does not present it as a multi-process performance test. self-contained synthetic example dependencies are listed in the import section. The code constructs inputs in memory; runtime logs are provided separately and do not constitute a guarantee for all real-world files. import pymupdf def partition(total, chunk): if type(chunk) is not int or chunk <= 0: raise ValueError("positive chunk required") return [(start, min(start + chunk, total)) for start in range(0, total, chunk)] def worker(raw, start, end): with pymupdf.open(stream=raw, filetype="pdf") as doc: return [(i, doc[i].get_text()) for i in range(start, end)] with pymupdf.open() as doc: for index in range(5): doc.new_page().insert_text((50, 50), f"Page {index}") raw = doc.tobytes() tasks = partition(5, 2) assert tasks == [(0, 2), (2, 4), (4, 5)] results = [item for start, end in tasks for item in worker(raw, start, end)] assert [i for i, text in results] == list(range(5)) assert all(f"Page {i}" in text for i, text in results) assert partition(0, 2) == [] Why can't the Document object be passed to a child process? ’s root cause lies in Document It internally holds pointers to the underlying C structures and file handles. These resources are bound to the memory space of the process that created them; once passed across processes, the pointers become invalid, and the handles can no longer be accessed legally by another process. The correct approach is to pass only serializable parameters such as the filename (as a string) and page-number ranges, allowing each child process to invoke the function independently within its own memory space. pymupdf.open() . This is not a performance defect, because the child process, as an independent entity, would in any case require its own document instance to safely perform extraction or rendering operations. synthesis example: correct input and dangerous boundary correct input : Pass the tuple (segment_id, filename, start_page, end_page) . Where filename is a string path, start_page and end_page is an integer. Upon receipt, the child process opens the file itself, processes the specified page-number range, and returns a list of results. error/boundary input : If the passed-in page number range is (18, 25) , whereas the document actually contains only 20 pages; when the child process accesses page 21, it will trigger an index out-of-bounds error. Another common mistake is directly passing Document object, which will be interrupted due to a pickle failure. There is also a hidden scenario: multiple child processes simultaneously write results to the same output file; in the absence of synchronization mechanisms, this can lead to interleaved or corrupted content. The handling method is to immediately verify at the child process entry point whether the page number range is out of bounds. doc.page_count , and use separate target paths or memory buffers for all I/O operations. For non-serializable objects, they should be excluded from task parameters during the design phase, rather than relying on runtime exception handling. Complete Python Example and Dependency Instructions Note: pool.map 's chunksize The parameter is set to its default value 1 here, which is suitable for scenarios with large task granularity. If there are an extremely large number of pages and each page is processed very quickly, you can increase the chunksize to reduce IPC overhead; however, this example focuses on security rather than tuning. Acceptance Criteria and Manual Review Boundary Observation of the phenomenon possible causes processing decision pickle.PicklingError or the child process silently crashes passed non-serializable objects such as Document/Page only passes basic types such as strings, integers, and tuples section page numbers return an empty string The page range exceeds the actual number of pages, or the page is a pure image with no text layer validates page number boundaries; for pages consisting solely of images, OCR tagging is required (this is expected behavior) output file content is corrupted multi-process writing to the same file Each process writes to its own temporary file, and the main process merges them. memory usage grows linearly with the number of processes Each process loads the complete document accepts this overhead; if the document is too large, switch to paginated streaming processing This approach is not suitable for tasks that require cross-page contextual correlation (such as table spanning across pages or full-text index construction), because each process operates in isolation. When encountering such requirements, one should revert to single-process handling or devise a dedicated chunking-and-aggregation strategy. Furthermore, the code demonstration only verifies “safe parallel extraction”; this does not imply end-to-end usability—text quality, encoding correctness, and the degree of format preservation still require manual sampling and review. In particular, for scanned PDFs, plain-text extraction may fail entirely, necessitating the inclusion of an OCR step. Reference and Next Step uses this checklist for document translation first ensures that the page order, source locations, and text integrity have been thoroughly checked, then submits the same de-identified sample to the translation process. After translation, it is still necessary to verify that the headings, body text, and figure captions correspond correctly; one should not rely solely on whether the text flows smoothly. OmniDoc (Product Entry) provides an entry point for translating multi-format documents, and the official website highlights its OCR, terminology database, and bilingual alignment features. For teams that wish to start processing directly from files, this is a path worth evaluating with real-world samples. Please use de-identified files that you are authorized to upload, and verify the results against the checklist provided in this document; we do not claim that OmniDoc utilizes the open-source libraries or internal algorithms referenced herein, nor do we substitute examples for product performance testing. Original manufacturer interface based on: PyMuPDF document 。 - 1 -