stephen/137_Docing-Test
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

working version

Browse source
This commit is contained in:
stephen 2026-05-29 21:54:28 +10:00
parent 8bff1d7f1b
commit 97f96e9df8
10 changed files with 2894 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

BIN
.DS_Store vendored Normal file
View file

Binary file not shown.

16
.gitignore vendored
View file

@ -0,0 +1,16 @@
# Python virtual environments
.venv/
env/
venv/
ENV/
# uv specific paths (if using uv's project caching)
.uv/
# Python bytecode cache
__pycache__/
*.py[cod]
*$py.class
# Mac system files
.DS_Store.

1
.python-version Normal file
View file

@ -0,0 +1 @@
3.14

BIN
SpanishTextBook.pdf Normal file
View file

Binary file not shown.

318
doc/Notes.md
View file

@ -1,4 +1,4 @@
# Docing Testing
# Docling Testing
## Using Git on Forgejo Server
@ -50,4 +50,318 @@ stephenlohning@Scotty 137_Docing-Test % tree
└── images
3 directories, 3 files
```
```
## Check uv version and upgrade
- This I did in VS Code terminal
```zsh
stephenlohning@Scotty 137_Docing-Test % uv --version
uv 0.5.20 (1c17662b3 2025-01-15)
stephenlohning@Scotty 137_Docing-Test % uv self update
info: Checking for updates...
success: Upgraded uv from v0.5.20 to v0.11.17! https://github.com/astral-sh/uv/releases/tag/0.11.17
stephenlohning@Scotty 137_Docing-Test %
```
### setting up uv
```zsh
stephenlohning@Scotty 137_Docing-Test % uv init
Initialized project `137-docing-test`
stephenlohning@Scotty 137_Docing-Test %
```
### check what version of python are avainable
```zsh
tephenlohning@Scotty 137_Docing-Test % uv python list
cpython-3.15.0b1-macos-aarch64-none <download available>
cpython-3.15.0b1+freethreaded-macos-aarch64-none <download available>
cpython-3.14.5-macos-aarch64-none /opt/homebrew/bin/python3.14 -> ../Cellar/python@3.14/3.14.5/bin/python3.14
cpython-3.14.5-macos-aarch64-none /opt/homebrew/bin/python3 -> ../Cellar/python@3.14/3.14.5/bin/python3
cpython-3.14.5-macos-aarch64-none /Users/stephenlohning/.pyenv/shims/python3.14
cpython-3.14.5-macos-aarch64-none /Users/stephenlohning/.pyenv/shims/python3
cpython-3.14.5-macos-aarch64-none <download available>
cpython-3.14.5+freethreaded-macos-aarch64-none <download available>
cpython-3.13.13-macos-aarch64-none <download available>
cpython-3.13.13+freethreaded-macos-aarch64-none <download available>
cpython-3.13.11-macos-aarch64-none /opt/homebrew/bin/python3.1
more here
```
### create virtual environment
- uv venv --python 3.14
This creates .venv directory, a main.py and a pyproject.toml
```zsh
stephenlohning@Scotty 137_Docing-Test % uv venv --python 3.14
Using CPython 3.14.2 interpreter at: /Users/stephenlohning/.pyenv/versions/3.14.2/bin/python3.14
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate
stephenlohning@Scotty 137_Docing-Test % ls -la
total 448
drwxr-xr-x 11 stephenlohning staff 352 May 29 20:30 .
drwxr-xr-x@ 158 stephenlohning staff 5056 May 29 19:42 ..
drwxr-xr-x 12 stephenlohning staff 384 May 29 20:13 .git
-rw-r--r-- 1 stephenlohning staff 0 May 29 19:48 .gitignore
-rw-r--r-- 1 stephenlohning staff 5 May 29 20:28 .python-version
drwxr-xr-x 7 stephenlohning staff 224 May 29 20:30 .venv
-rw-r--r-- 1 stephenlohning staff 1989 May 29 20:12 Readme.md
-rw-r--r-- 1 stephenlohning staff 212930 May 29 20:12 Readme.pdf
drwxr-xr-x 5 stephenlohning staff 160 May 29 20:06 doc
-rw-r--r-- 1 stephenlohning staff 93 May 29 20:21 main.py
-rw-r--r-- 1 stephenlohning staff 161 May 29 20:21 pyproject.toml
```
#### pyproject.toml
```zsh
stephenlohning@Scotty 137_Docing-Test % cat pyproject.toml
[project]
name = "137-docing-test"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = []
stephenlohning@Scotty 137_Docing-Test %
```
### uv add docling
The command uv add docling does two distinct things simultaneously: it executes an installation via uv, and it targets a specific, popular open-source AI package called docling.
Running this command executes the following actions:
#### 1. What uv does behind the scenes
Because you are using uv's modern project management tool, this single command handles the entire dependency lifecycle:
- Resolves Dependencies: It looks up docling on PyPI, checks all the sub-packages it requires, and resolves any version conflicts in milliseconds.
- Updates your pyproject.toml: It automatically writes docling into your project's configuration file under the dependencies list.
- Updates your uv.lock file: It locks down the exact cryptographic versions of docling and its sub-dependencies to ensure reproducibility across different machines.
- Installs into .venv: It downloads the wheels and installs them directly into your project's local virtual environment (creating the environment first if it doesn't already exist).
#### Example
```zsh
stephenlohning@Scotty 137_Docing-Test % uv add docling
Resolved 122 packages in 2.09s
Built pylatexenc==2.10
Built antlr4-python3-runtime==4.9.3
Prepared 99 packages in 4.59s
Installed 99 packages in 551ms
+ accelerate==1.13.0
+ annotated-doc==0.0.4
+ annotated-types==0.7.0
+ antlr4-python3-runtime==4.9.3
+ anyio==4.13.0
+ attrs==26.1.0
+ beautifulsoup4==4.14.3
+ certifi==2026.5.20
+ charset-normalizer==3.4.7
+ click==8.4.1
+ colorlog==6.10.1
+ defusedxml==0.7.1
+ dill==0.4.1
+ docling==2.96.0
+ docling-core==2.77.1
+ docling-ibm-models==3.13.2
+ docling-parse==6.2.0
+ docling-slim==2.96.0
+ et-xmlfile==2.0.0
+ faker==40.19.1
+ filelock==3.29.0
+ filetype==1.2.0
+ fsspec==2026.4.0
+ h11==0.16.0
+ hf-xet==1.5.0
+ httpcore==1.0.9
+ httpx==0.28.1
+ huggingface-hub==1.17.0
+ idna==3.17
+ jinja2==3.1.6
+ jsonlines==4.0.0
+ jsonref==1.1.0
+ jsonschema==4.26.0
+ jsonschema-specifications==2025.9.1
+ latex2mathml==3.81.0
+ lxml==6.1.1
+ markdown-it-py==4.2.0
+ marko==2.2.3
+ markupsafe==3.0.3
+ mdurl==0.1.2
+ mpire==2.10.2
+ mpmath==1.3.0
+ multiprocess==0.70.19
+ networkx==3.6.1
+ numpy==2.4.6
+ omegaconf==2.3.0
+ opencv-python==4.13.0.92
+ openpyxl==3.1.5
+ packaging==26.2
+ pandas==3.0.3
+ pillow==12.2.0
+ pluggy==1.6.0
+ polyfactory==3.3.0
+ psutil==7.2.2
+ pyclipper==1.4.0
+ pydantic==2.13.4
+ pydantic-core==2.46.4
+ pydantic-settings==2.14.1
+ pygments==2.20.0
+ pylatexenc==2.10
+ pypdfium2==5.8.0
+ python-dateutil==2.9.0.post0
+ python-docx==1.2.0
+ python-dotenv==1.2.2
+ python-pptx==1.0.2
+ pyyaml==6.0.3
+ rapidocr==3.8.1
+ referencing==0.37.0
+ regex==2026.5.9
+ requests==2.34.2
+ rich==15.0.0
+ rpds-py==2026.5.1
+ rtree==1.4.1
+ safetensors==0.7.0
+ scipy==1.17.1
+ semchunk==3.2.5
+ setuptools==81.0.0
+ shapely==2.1.2
+ shellingham==1.5.4
+ six==1.17.0
+ soupsieve==2.8.4
+ sympy==1.14.0
+ tabulate==0.10.0
+ tokenizers==0.22.2
+ torch==2.12.0
+ torchvision==0.27.0
+ tqdm==4.67.3
+ transformers==5.9.0
+ tree-sitter==0.25.2
+ tree-sitter-c==0.24.2
+ tree-sitter-javascript==0.25.0
+ tree-sitter-python==0.25.0
+ tree-sitter-typescript==0.23.2
+ typer==0.21.2
+ typing-extensions==4.15.0
+ typing-inspection==0.4.2
+ urllib3==2.7.0
+ websockets==16.0
+ xlsxwriter==3.2.9
stephenlohning@Scotty 137_Docing-Test %
```
### Updated pyproject.toml
```zsh
cat pyproject.toml
[project]
name = "137-docing-test"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
"docling>=2.96.0",
]
stephenlohning@Scotty 137_Docing-Test %
```
### Update the main.py
```py
# main.py
from docling.document_converter import DocumentConverter
def main():
# Can be a local path (e.g., "docs/invoice.pdf") or a URL
# source = "https://arxiv.org/pdf/2408.09869"
source = ./SpanishTextBook.pdf
print("Initializing Docling pipeline...")
converter = DocumentConverter()
print("Processing document locally...")
result = converter.convert(source)
# Export cleanly to structured Markdown
markdown_output = result.document.export_to_markdown()
print("\n--- Parsing Successful! Sample Output: ---\n")
print(markdown_output[:500])
if __name__ == "__main__":
main()
```
#### Runing via uv
- uv run main.py
# We had multiple errors with Docling
## 1. The Python Version Compatibility Issue
### The Error
Initially, installing Docling failed due to deep dependency conflicts rooted in the Python environment version. The system was running Python 3.14 (an experimental/pre-release version).
```bash
Plaintext
Configuring project 137_Docing-Test...
× No solution found when resolving dependencies:
╰─▶ Because docling v2.14.0 depends on numpy >=1.26.4 and <2.0.0...
and no versions of numpy match >=1.26.4,<2.0.0 because python_version >= "3.14"
```
### The Root Cause
Core scientific computing libraries utilized by Docling—specifically numpy (restricted to <2.0.0 by underlying models) and scipy—did not yet have pre-compiled binary wheels available for Python 3.14. Building them from source failed due to strict C-extension requirements that were incompatible with the unreleased Python runtime.
The Fix
We utilized the uv package manager to explicitly pin and enforce a stable, production-grade Python runtime environment without altering your global system configuration:
```Bash
uv init --python 3.12
```
This cleanly isolated the project to Python 3.12, instantly unlocking access to fully compiled, highly optimized binary wheels for numpy, scipy, torch, and onnxruntime.
## 2. The Metal Performance Shaders (MPS) float64 Crash
The Error
Once the environment was stable, running the document converter triggered a hard runtime crash during the Stage layout phase when processing the first page:
```zsh
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.
Stage layout failed for run 1: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64.
```
### The Root Cause
Docling uses IBM's rt_detr_v2 (Real-Time DEtection TRansformer) vision model for document layout and element segmentation.
- By default, PyTorch detected the Apple Silicon M3 GPU hardware and automatically routed tensors to the MPS (Metal Performance Shaders) backend.
- Deep inside the Hugging Face transformers layout code, the model initialized a positional embedding layer using high-precision 64-bit floating-point numbers:
ω= torch.arange(pos_dim,dtype=torch.float64)
----------------------------------------
pos_dim
​
- Because Apple's Metal shading language natively operates on 16-bit (half) and 32-bit (float) precisions to maximize unified memory bandwidth, the MPS framework physically lacks hardware support for float64 operations, causing the runtime execution to immediately fault.
## The Fix
Standard environment variables (like device="cpu" passing configurations or CUDA_VISIBLE_DEVICES="") failed because the model's internal source code bypassed them to poll the hardware directly.
We resolved this by implementing a pre-emptive monkeypatch at the absolute top of main.py before any heavy machine learning frameworks could execute their initialization loops:
```Python
import torch
# Intercept and mock PyTorch's hardware discovery backend completely
torch.backends.mps.is_available = lambda: False
torch.backends.mps.is_built = lambda: False
# Force standard CPU allocation fallback
torch.set_default_device("cpu")
```
## The Result
By blinding PyTorch to the presence of the local Metal architecture, the layout engine cleanly and safely fell back to the standard CPU processing path. This bypassed the precision limitations completely and achieved a flawless, structurally complete 9-page Markdown compilation in 36.44 seconds.

BIN
doc/Notes.pdf
View file

Binary file not shown.

49
main.py Normal file
View file

@ -0,0 +1,49 @@
# main.py
import os
import sys
# 1. Block hardware-level multi-device discovery loops
os.environ["CUDA_VISIBLE_DEVICES"] = ""
import torch
# 2. Hard monkeypatch PyTorch's MPS detection before ANY machine learning imports occur
# This stops underlying huggingface models from forcing device="mps" internally
torch.backends.mps.is_available = lambda: False
torch.backends.mps.is_built = lambda: False
# Force standard CPU allocation for safety
torch.set_default_device("cpu")
import time
from docling.document_converter import DocumentConverter
def main():
source = "https://arxiv.org/pdf/2408.09869"
print("🚀 Initializing Docling engine with hardware-isolated CPU routing...")
start_time = time.time()
# This will now natively spin up on standard CPU threads
converter = DocumentConverter()
print("⏳ Processing document locally (Parsing layout, OCR, and tables)...")
result = converter.convert(source)
print("✨ Conversion finished! Compiling Markdown structure...")
markdown_output = result.document.export_to_markdown()
# Save output cleanly to project directory
output_filename = "parsed_paper.md"
with open(output_filename, "w", encoding="utf-8") as f:
f.write(markdown_output)
elapsed_time = time.time() - start_time
print(f"✅ Success! Processing complete in {elapsed_time:.2f} seconds.")
print(f"📁 Structured output saved cleanly to: {os.path.abspath(output_filename)}")
print("\n--- Preview of First 300 Characters: ---")
print(markdown_output[:300])
if __name__ == "__main__":
main()

248
parsed_paper.md Normal file
View file

@ -0,0 +1,248 @@
<!-- image -->
## Docling Technical Report
## Version 1.0
Christoph Auer Maksym Lysak Ahmed Nassar Michele Dolfi Nikolaos Livathinos Panos Vagenas Cesar Berrospi Ramis Matteo Omenetti Fabian Lindlbauer Kasper Dinkla Lokesh Mishra Yusik Kim Shubham Gupta Rafael Teixeira de Lima Valery Weber Lucas Morin Ingmar Meijer Viktor Kuropiatnyk Peter W. J. Staar
AI4K Group, IBM Research R¨ uschlikon, Switzerland
## Abstract
This technical report introduces Docling , an easy to use, self-contained, MITlicensed open-source package for PDF document conversion. It is powered by state-of-the-art specialized AI models for layout analysis (DocLayNet) and table structure recognition (TableFormer), and runs efficiently on commodity hardware in a small resource budget. The code interface allows for easy extensibility and addition of new features and models.
## 1 Introduction
Converting PDF documents back into a machine-processable format has been a major challenge for decades due to their huge variability in formats, weak standardization and printing-optimized characteristic, which discards most structural features and metadata. With the advent of LLMs and popular application patterns such as retrieval-augmented generation (RAG), leveraging the rich content embedded in PDFs has become ever more relevant. In the past decade, several powerful document understanding solutions have emerged on the market, most of which are commercial software, cloud offerings [3] and most recently, multi-modal vision-language models. As of today, only a handful of open-source tools cover PDF conversion, leaving a significant feature and quality gap to proprietary solutions.
With Docling , we open-source a very capable and efficient document conversion tool which builds on the powerful, specialized AI models and datasets for layout analysis and table structure recognition we developed and presented in the recent past [12, 13, 9]. Docling is designed as a simple, self-contained python library with permissive license, running entirely locally on commodity hardware. Its code architecture allows for easy extensibility and addition of new features and models.
Here is what Docling delivers today:
- Converts PDF documents to JSON or Markdown format, stable and lightning fast
- Understands detailed page layout, reading order, locates figures and recovers table structures
- Extracts metadata from the document, such as title, authors, references and language
- Optionally applies OCR, e.g. for scanned PDFs
- Can be configured to be optimal for batch-mode (i.e high throughput, low time-to-solution)
- or interactive mode (compromise on efficiency, low time-to-solution)
- Can leverage different accelerators (GPU, MPS, etc).
## 2 Getting Started
To use Docling, you can simply install the docling package from PyPI. Documentation and examples are available in our GitHub repository at github.com/DS4SD/docling. All required model assets 1 are downloaded to a local huggingface datasets cache on first use, unless you choose to pre-install the model assets in advance.
Docling provides an easy code interface to convert PDF documents from file system, URLs or binary streams, and retrieve the output in either JSON or Markdown format. For convenience, separate methods are offered to convert single documents or batches of documents. A basic usage example is illustrated below. Further examples are available in the Doclign code repository.
from docling.document\_converter import DocumentConverter
```
source = "https://arxiv.org/pdf/2206.01062" # PDF path or URL converter = DocumentConverter() result = converter.convert_single(source) print(result.render_as_markdown()) # output: "## DocLayNet: A Large Human -Annotated Dataset for Document -Layout Analysis [...]"
```
Optionally, you can configure custom pipeline features and runtime options, such as turning on or off features (e.g. OCR, table structure recognition), enforcing limits on the input document size, and defining the budget of CPU threads. Advanced usage examples and options are documented in the README file. Docling also provides a Dockerfile to demonstrate how to install and run it inside a container.
## 3 Processing pipeline
Docling implements a linear pipeline of operations, which execute sequentially on each given document (see Fig. 1). Each document is first parsed by a PDF backend, which retrieves the programmatic text tokens, consisting of string content and its coordinates on the page, and also renders a bitmap image of each page to support downstream operations. Then, the standard model pipeline applies a sequence of AI models independently on every page in the document to extract features and content, such as layout and table structures. Finally, the results from all pages are aggregated and passed through a post-processing stage, which augments metadata, detects the document language, infers reading-order and eventually assembles a typed document object which can be serialized to JSON or Markdown.
## 3.1 PDF backends
Two basic requirements to process PDF documents in our pipeline are a) to retrieve all text content and their geometric coordinates on each page and b) to render the visual representation of each page as it would appear in a PDF viewer. Both these requirements are encapsulated in Docling's PDF backend interface. While there are several open-source PDF parsing libraries available for python, we faced major obstacles with all of them for different reasons, among which were restrictive licensing (e.g. pymupdf [7]), poor speed or unrecoverable quality issues, such as merged text cells across far-apart text tokens or table columns (pypdfium, PyPDF) [15, 14].
[1 see huggingface.co/ds4sd/docling-models/](https://huggingface.co/ds4sd/docling-models/)
Figure 1: Sketch of Docling's default processing pipeline. The inner part of the model pipeline is easily customizable and extensible.
<!-- image -->
We therefore decided to provide multiple backend choices, and additionally open-source a custombuilt PDF parser, which is based on the low-level qpdf [4] library. It is made available in a separate package named docling-parse and powers the default PDF backend in Docling. As an alternative, we provide a PDF backend relying on pypdfium , which may be a safe backup choice in certain cases, e.g. if issues are seen with particular font encodings.
## 3.2 AI models
As part of Docling, we initially release two highly capable AI models to the open-source community, which have been developed and published recently by our team. The first model is a layout analysis model, an accurate object-detector for page elements [13]. The second model is TableFormer [12, 9], a state-of-the-art table structure recognition model. We provide the pre-trained weights (hosted on huggingface) and a separate package for the inference code as docling-ibm-models . Both models are also powering the open-access deepsearch-experience, our cloud-native service for knowledge exploration tasks.
## Layout Analysis Model
Our layout analysis model is an object-detector which predicts the bounding-boxes and classes of various elements on the image of a given page. Its architecture is derived from RT-DETR [16] and re-trained on DocLayNet [13], our popular human-annotated dataset for document-layout analysis, among other proprietary datasets. For inference, our implementation relies on the onnxruntime [5].
The Docling pipeline feeds page images at 72 dpi resolution, which can be processed on a single CPU with sub-second latency. All predicted bounding-box proposals for document elements are post-processed to remove overlapping proposals based on confidence and size, and then intersected with the text tokens in the PDF to group them into meaningful and complete units such as paragraphs, section titles, list items, captions, figures or tables.
## Table Structure Recognition
The TableFormer model [12], first published in 2022 and since refined with a custom structure token language [9], is a vision-transformer model for table structure recovery. It can predict the logical row and column structure of a given table based on an input image, and determine which table cells belong to column headers, row headers or the table body. Compared to earlier approaches, TableFormer handles many characteristics of tables, such as partial or no borderlines, empty cells, rows or columns, cell spans and hierarchy both on column-heading or row-heading level, tables with inconsistent indentation or alignment and other complexities. For inference, our implementation relies on PyTorch [2].
The Docling pipeline feeds all table objects detected in the layout analysis to the TableFormer model, by providing an image-crop of the table and the included text cells. TableFormer structure predictions are matched back to the PDF cells in post-processing to avoid expensive re-transcription text in the table image. Typical tables require between 2 and 6 seconds to be processed on a standard CPU, strongly depending on the amount of included table cells.
## OCR
Docling provides optional support for OCR, for example to cover scanned PDFs or content in bitmaps images embedded on a page. In our initial release, we rely on EasyOCR [1], a popular thirdparty OCR library with support for many languages. Docling, by default, feeds a high-resolution page image (216 dpi) to the OCR engine, to allow capturing small print detail in decent quality. While EasyOCR delivers reasonable transcription quality, we observe that it runs fairly slow on CPU (upwards of 30 seconds per page).
We are actively seeking collaboration from the open-source community to extend Docling with additional OCR backends and speed improvements.
## 3.3 Assembly
In the final pipeline stage, Docling assembles all prediction results produced on each page into a well-defined datatype that encapsulates a converted document, as defined in the auxiliary package docling-core . The generated document object is passed through a post-processing model which leverages several algorithms to augment features, such as detection of the document language, correcting the reading order, matching figures with captions and labelling metadata such as title, authors and references. The final output can then be serialized to JSON or transformed into a Markdown representation at the users request.
## 3.4 Extensibility
Docling provides a straight-forward interface to extend its capabilities, namely the model pipeline. A model pipeline constitutes the central part in the processing, following initial document parsing and preceding output assembly, and can be fully customized by sub-classing from an abstract baseclass ( BaseModelPipeline ) or cloning the default model pipeline. This effectively allows to fully customize the chain of models, add or replace models, and introduce additional pipeline configuration parameters. To use a custom model pipeline, the custom pipeline class to instantiate can be provided as an argument to the main document conversion methods. We invite everyone in the community to propose additional or alternative models and improvements.
Implementations of model classes must satisfy the python Callable interface. The \_\_call\_\_ method must accept an iterator over page objects, and produce another iterator over the page objects which were augmented with the additional features predicted by the model, by extending the provided PagePredictions data model accordingly.
## 4 Performance
In this section, we establish some reference numbers for the processing speed of Docling and the resource budget it requires. All tests in this section are run with default options on our standard test set distributed with Docling, which consists of three papers from arXiv and two IBM Redbooks, with a total of 225 pages. Measurements were taken using both available PDF backends on two different hardware systems: one MacBook Pro M3 Max, and one bare-metal server running Ubuntu 20.04 LTS on an Intel Xeon E5-2690 CPU. For reproducibility, we fixed the thread budget (through setting OMP NUM THREADS environment variable ) once to 4 (Docling default) and once to 16 (equal to full core count on the test hardware). All results are shown in Table 1.
If you need to run Docling in very low-resource environments, please consider configuring the pypdfium backend. While it is faster and more memory efficient than the default docling-parse backend, it will come at the expense of worse quality results, especially in table structure recovery.
Establishing GPU acceleration support for the AI models is currently work-in-progress and largely untested, but may work implicitly when CUDA is available and discovered by the onnxruntime and torch runtimes backing the Docling pipeline. We will deliver updates on this topic at in a future version of this report.
Table 1: Runtime characteristics of Docling with the standard model pipeline and settings, on our test dataset of 225 pages, on two different systems. OCR is disabled. We show the time-to-solution (TTS), computed throughput in pages per second, and the peak memory used (resident set size) for both the Docling-native PDF backend and for the pypdfium backend, using 4 and 16 threads.
| CPU | Thread budget | native backend | native backend | native backend | pypdfium backend | pypdfium backend | pypdfium backend |
|-------------------------|-----------------|------------------|------------------|------------------|--------------------|--------------------|--------------------|
| | | TTS | Pages/s | Mem | TTS | Pages/s | Mem |
| Apple M3 Max (16 cores) | 4 16 | 177 s 167 s | 1.27 1.34 | 6.20 GB | 103 s 92 s | 2.18 2.45 | 2.56 GB |
| Intel(R) Xeon E5-2690 | 4 16 | 375 s 244 s | 0.60 0.92 | 6.16 GB | 239 s 143 s | 0.94 1.57 | 2.42 GB |
## 5 Applications
Thanks to the high-quality, richly structured document conversion achieved by Docling, its output qualifies for numerous downstream applications. For example, Docling can provide a base for detailed enterprise document search, passage retrieval or classification use-cases, or support knowledge extraction pipelines, allowing specific treatment of different structures in the document, such as tables, figures, section structure or references. For popular generative AI application patterns, such as retrieval-augmented generation (RAG), we provide quackling , an open-source package which capitalizes on Docling's feature-rich document output to enable document-native optimized vector embedding and chunking. It plugs in seamlessly with LLM frameworks such as LlamaIndex [8]. Since Docling is fast, stable and cheap to run, it also makes for an excellent choice to build document-derived datasets. With its powerful table structure recognition, it provides significant benefit to automated knowledge-base construction [11, 10]. Docling is also integrated within the open IBM data prep kit [6], which implements scalable data transforms to build large-scale multi-modal training datasets.
## 6 Future work and contributions
Docling is designed to allow easy extension of the model library and pipelines. In the future, we plan to extend Docling with several more models, such as a figure-classifier model, an equationrecognition model, a code-recognition model and more. This will help improve the quality of conversion for specific types of content, as well as augment extracted document metadata with additional information. Further investment into testing and optimizing GPU acceleration as well as improving the Docling-native PDF backend are on our roadmap, too.
We encourage everyone to propose or implement additional features and models, and will gladly take your inputs and contributions under review . The codebase of Docling is open for use and contribution, under the MIT license agreement and in alignment with our contributing guidelines included in the Docling repository. If you use Docling in your projects, please consider citing this technical report.
## References
- [1] J. AI. Easyocr: Ready-to-use ocr with 80+ supported languages. https://github.com/ JaidedAI/EasyOCR , 2024. Version: 1.7.0.
- [2] J. Ansel, E. Yang, H. He, N. Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Berard, E. Burovski, G. Chauhan, A. Chourdia, W. Constable, A. Desmaison, Z. DeVito, E. Ellison, W. Feng, J. Gong, M. Gschwind, B. Hirsh, S. Huang, K. Kalambarkar, L. Kirsch, M. Lazos, M. Lezcano, Y. Liang, J. Liang, Y. Lu, C. Luk, B. Maher, Y. Pan, C. Puhrsch, M. Reso, M. Saroufim, M. Y. Siraichi, H. Suk, M. Suo, P. Tillet, E. Wang, X. Wang, W. Wen, S. Zhang, X. Zhao, K. Zhou, R. Zou, A. Mathews, G. Chanan, P. Wu, and S. Chintala. Pytorch 2: Faster
machine learning through dynamic python bytecode transformation and graph compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS '24) . ACM, 4 2024. doi: 10.1145/3620665.3640366. URL https://pytorch.org/assets/pytorch2-2.pdf .
- [3] C. Auer, M. Dolfi, A. Carvalho, C. B. Ramis, and P. W. Staar. Delivering document conversion as a cloud service with high throughput and responsiveness. In 2022 IEEE 15th International Conference on Cloud Computing (CLOUD) , pages 363-373. IEEE, 2022.
- [4] J. Berkenbilt. Qpdf: A content-preserving pdf document transformer, 2024. URL https: //github.com/qpdf/qpdf .
- [5] O. R. developers. Onnx runtime. https://onnxruntime.ai/ , 2024. Version: 1.18.1.
- [6] IBM. Data Prep Kit: a community project to democratize and accelerate unstructured data preparation for LLM app developers, 2024. URL https://github.com/IBM/ data-prep-kit .
- [7] [A. S. Inc. PyMuPDF, 2024. URL https://github.com/pymupdf/PyMuPDF .](https://github.com/pymupdf/PyMuPDF)
- [8] [J. Liu. LlamaIndex, 11 2022. URL https://github.com/jerryjliu/llama\_index .](https://github.com/jerryjliu/llama_index)
- [9] M. Lysak, A. Nassar, N. Livathinos, C. Auer, and P. Staar. Optimized Table Tokenization for Table Structure Recognition. In Document Analysis and Recognition - ICDAR 2023: 17th International Conference, San Jos´ e, CA, USA, August 21-26, 2023, Proceedings, Part II , pages 37-50, Berlin, Heidelberg, Aug. 2023. Springer-Verlag. ISBN 978-3-031-41678-1. doi: 10. 1007/978-3-031-41679-8 3. URL https://doi.org/10.1007/978-3-031-41679-8\_3 .
- [10] L. Mishra, S. Dhibi, Y. Kim, C. Berrospi Ramis, S. Gupta, M. Dolfi, and P. Staar. Statements: Universal information extraction from tables with large language models for ESG KPIs. In D. Stammbach, J. Ni, T. Schimanski, K. Dutia, A. Singh, J. Bingler, C. Christiaen, N. Kushwaha, V. Muccione, S. A. Vaghefi, and M. Leippold, editors, Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024) , pages 193-214, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.climatenlp-1.15 .
- [11] L. Morin, V. Weber, G. I. Meijer, F. Yu, and P. W. J. Staar. Patcid: an open-access dataset of chemical structures in patent documents. Nature Communications , 15(1):6532, August 2024. ISSN 2041-1723. doi: 10.1038/s41467-024-50779-y. URL https://doi.org/10.1038/ s41467-024-50779-y .
- [12] A. Nassar, N. Livathinos, M. Lysak, and P. Staar. Tableformer: Table structure understanding with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4614-4623, 2022.
- [13] B. Pfitzmann, C. Auer, M. Dolfi, A. S. Nassar, and P. Staar. Doclaynet: a large humanannotated dataset for document-layout segmentation. pages 3743-3751, 2022.
- [14] pypdf Maintainers. pypdf: A Pure-Python PDF Library, 2024. URL https://github.com/ py-pdf/pypdf .
- [15] P. Team. PyPDFium2: Python bindings for PDFium, 2024. URL https://github.com/ pypdfium2-team/pypdfium2 .
- [16] Y. Zhao, W. Lv, S. Xu, J. Wei, G. Wang, Q. Dang, Y. Liu, and J. Chen. Detrs beat yolos on real-time object detection, 2023.
## Appendix
In this section, we illustrate a few examples of Docling's output in Markdown and JSON.
<!-- image -->
1 INTRODUCTION
Despite the substantial improvements achieved with machine-learning (ML) approaches and deep neural networks in recent years, document conversion remains a challenging problem, as demonstrated by the numerous public competitions held on this topic [1-4]. The challenge originates from the huge variability in PDF documents regarding layout, language and formats (scanned, programmatic or a combination of both). Engineering a single ML model that can be applied on all types of documents and provides high-quality layout segmentation remains to this day extremely challenging [5]. To highlight the variability in document layouts, we show a few example documents from the DocLayNet dataset in Figure 1. Figure 2: Title page of the DocLayNet paper (arxiv.org/pdf/2206.01062) - left PDF, right rendered Markdown. If recognized, metadata such as authors are appearing first under the title. Text content inside figures is currently dropped, the caption is retained and linked to the figure in the JSON representation (not shown).
KDD '22, August 14-18, 2022, Washington, DC, USA Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S. Nassar, and Peter Staar
Table 2: Prediction performance (mAP@0.5-0.95) of object detection networks on DocLayNet test set. The MRCNN (Mask R-CNN) and FRCNN (Faster R-CNN) models with ResNet-50 or ResNet-101 backbone were trained based on the network architectures from the detectron2 model zoo (Mask R-CNN R50, R101-FPN 3x, Faster R-CNN R101-FPN 3x), with default configurations. The YOLO implementation utilized was YOLOv5x6 [13]. All models were initialised using pre-trained weights from the COCO 2017 dataset.
| | human | MRCNN R50 R101 | FRCNN R101 | YOLO v5x6 |
|--------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------|-------------------------------------------------------------|
| Caption Footnote Formula List-item Page-footer Page-header Picture Section-header Table Text Title All | 84-89 83-91 83-85 87-88 93-94 85-89 69-71 83-84 77-81 84-86 60-72 82-83 | 68.4 71.5 70.9 71.8 60.1 63.4 81.2 80.8 61.6 59.3 71.9 70.0 71.7 72.7 67.6 69.3 82.2 82.9 84.6 85.8 76.7 80.4 72.4 73.5 | 70.1 73.7 63.5 81.0 58.9 72.0 72.0 68.4 82.2 85.4 79.9 73.4 | 77.7 77.2 66.2 86.2 61.1 67.9 77.1 74.6 86.3 88.1 82.7 76.8 |
to avoid this at any cost in order to have clear, unbiased baseline numbers for human document-layout annotation. Third, we introduced the feature of snapping boxes around text segments to obtain a pixel-accurate annotation and again reduce time and effort. The CCS annotation tool automatically shrinks every user-drawn box to the minimum bounding-box around the enclosed text-cells for all purely text-based segments, which excludes only Table and Picture . For the latter, we instructed annotation staff to minimise inclusion of surrounding whitespace while including all graphical lines. A downside of snapping boxes to enclosed text cells is that some wrongly parsed PDF pages cannot be annotated correctly and need to be skipped. Fourth, we established a way to flag pages as rejected for cases where no valid annotation according to the label guidelines could be achieved. Example cases for this would be PDF pages that render incorrectly or contain layouts that are impossible to capture with non-overlapping rectangles. Such rejected pages are not contained in the final dataset. With all these measures in place, experienced annotation staff managed to annotate a single page in a typical timeframe of 20s to 60s, depending on its complexity.
## 5 EXPERIMENTS
The primary goal of DocLayNet is to obtain high-quality ML models capable of accurate document-layout analysis on a wide variety of challenging layouts. As discussed in Section 2, object detection models are currently the easiest to use, due to the standardisation of ground-truth data in COCO format [16] and the availability of general frameworks such as detectron2 [17]. Furthermore, baseline numbers in PubLayNet and DocBank were obtained using standard object detection models such as Mask R-CNN and Faster R-CNN. As such, we will relate to these object detection methods in this paper and leave the detailed evaluation of more recent methods mentioned in Section 2 for future work.
Figure 5: Prediction performance (mAP@0.5-0.95) of a Mask R-CNNnetworkwithResNet50backbonetrainedonincreasing fractions of the DocLayNet dataset. The learning curve flattens around the 80% mark, indicating that increasing the size of the DocLayNet dataset with similar data will not yield significantly better predictions.
<!-- image -->
In this section, we will present several aspects related to the performance of object detection models on DocLayNet. Similarly as in PubLayNet, we will evaluate the quality of their predictions using mean average precision (mAP) with 10 overlaps that range from 0.5 to 0.95 in steps of 0.05 (mAP@0.5-0.95). These scores are computed by leveraging the evaluation code provided by the COCO API [16].
## Baselines for Object Detection
In Table 2, we present baseline experiments (given in mAP) on Mask R-CNN [12], Faster R-CNN [11], and YOLOv5 [13]. Both training and evaluation were performed on RGB images with dimensions of 1025 × 1025 pixels. For training, we only used one annotation in case of redundantly annotated pages. As one can observe, the variation in mAP between the models is rather low, but overall between 6 and 10% lower than the mAP computed from the pairwise human annotations on triple-annotated pages. This gives a good indication that the DocLayNet dataset poses a worthwhile challenge for the research community to close the gap between human recognition and ML approaches. It is interesting to see that Mask R-CNN and Faster R-CNN produce very comparable mAP scores, indicating that pixel-based image segmentation derived from bounding-boxes does not help to obtain better predictions. On the other hand, the more recent Yolov5x model does very well and even out-performs humans on selected labels such as Text , Table and Picture . This is not entirely surprising, as Text , Table and Picture are abundant and the most visually distinctive in a document.
Table 2: Prediction perlormance (mAP@0.5-0.95) of object detection networks on DocLayNet test set. The MRCNN (Mask R-CNN) and FRCNN (Faster R-CNN) models with ResNet-50 or ResNet-101 beckbone were rained based on the network architecturesfrom the detectron2 model zoo (Mask R-CNN R50, R101-FPN 3x, Faster R-CNN R101-FPN 3x), with default configurations. The YOLO implementation utized was YOLOv5x6 [13]. All models were initialised using pre-trained weightsfrom the COCO 2017 dataset.
| | human | MRCNN | MRCNNFRCNN | | YOLO |
|----------------|---------|---------|--------------|------|--------|
| Caption | 84-89 | 68.4 | 71.5 | 70.1 | 77.7 |
| Footnote | 83-91 | 70.9 | 71.8 | 73.7 | 77.2 |
| Formula | 83-85 | 60.1 | 63.4 | 63.5 | 66.2 |
| List-item | 89-48 | 81.2 | 80.8 | 81.0 | 86.2 |
| Page-ooer | 93-94 | 61.6 | 59.3 | 58.9 | 61.1 |
| Page-header | 85-89 | 71.9 | 70.0 | 72.0 | 67.9 |
| Picture | 69-71 | 71.7 | 72.7 | 72.0 | 77.1 |
| Section-header | 83-84 | 67.6 | 69.3 | 68.4 | 74.6 |
| Table | 77-81 | 82.2 | 82.9 | 82.2 | 86.3 |
| x1 | 84-86 | 84.6 | 85.8 | 85.4 | 88.1 |
| Title | 60-72 | 76.7 | 80.4 | 79.9 | 82.7 |
| AlIl | 82-83 | 72.4 | 73.5 | 73.4 | 76.8 |
to avoid this at any cost in order to have clear, unbiased baseline numbers for human dcument-layout annotation. hird, we introduced the feature of snapping boxes around text segments to obtain a pixel-accurate annotation and again reduce time and effort. The CCS -pxa fjeund je o sjoo-xe posopu oug punoe xoq-upunoq wnwu o o xoq umep-jesnne syuus fgeogewone joo uogeqouue based segments, which excludes only Table and Picture For the later, we instructed annotation staff to minimise inclusion of surrounding whitespace while including all graphical lines. A downside of snapping boxes to enclosed text cellsis that some wrongly parsed PDF pages cannot be annotated correctly and need to be skipped. Fourth, we established a way to flag pages as rejected for cases where no valid annotation according to the label guidelines could be achieved. Example cases for this would be PDF pages that render incorrectly or contain layouts that are impossible to capture with non-overlapping rectangles. Such rejected pages are not contained in the final dataset. With all these measures in place, experienced annotation staff managed to annotate a single page in a typical timeframe of 20s to 60s, depending on its complexity.
## 5EXPERIMENTS
The primary goal of DocLayNet is to obtain high-quality ML models capable of accurate document-layout analysis on a wide variety of challenging layouts. As discussed in Section 2, object detection models are currently the easiest to use, due to the standardisation of ground-truth data in COCO format [16] and the availability of general frameworks such as detectron2 [17]. Furthemore, baseline numbers in PubLayNet and DocBank were obtained using standard object detection models such as Mask R-CNN and Faster R-CNN. As such, we will relate to these object detection methods in this
Figure 5: Prediction performance (mAP@0.5-0.95) of a Mask R-CNN network with ResNet50 backbone trained on increasing fractions of the DocLayNet dataset. The leaming curve flattens around the 80% mark, indicating that increasing the size of the DocLayNet dataset with simlar data ill not yield significantly better predictions.
paper and leave the detaled evaluation of more reoent methods mentioned in Section 2 for future work.
In this section, we will present several aspects related to the performance of object detection models on DocLayNet. Similarly as in 0 S'0 wo ofue pu sdeμano o (dvw) uospeud oene ueu Susn suogoped e po Agenb u aenje m om oeqnd 0.95 in steps of 0.05 (mAP@0.5-0.95). These scores are computed by leveraging the evaluation code provided by the COCO API[16]
## BaselinesforObjectDetection
In Table 2, we present baseline experiments (given in mAP) on Mask R-CNN [12], Faster R-CNN [11], and YOLOv5 [13]. Both training and evaluation were performed on RGB images with dimensions of 1025 × 1025 pbxels. For training, we only used one annotation in case of redundantly annotated pages. As one can observe, the variation in mAP between the models is rather low, but overallbetween 6 and 0 eug uogeopu poo6 e senj6 sL sobed popeoue-edjμ uo suogegoue ueunq esujed oug wo. pnduoo dvw oug ueg emo %01 DocLayNet dataset poses a worthwhile challenge for the research community to close the gap between human recognition and ML approaches. It is interesting to see that Mask R-CNN and Faster R-CNN produce very comparable mAP scores, indicating that pixelupoau auou ou puey saugo au uo suogopeud sogeq ueqo o djou lou sep soxoq-ugpunoq wou panuap uogeuoubes oteu poseq Yolov5x model does very welland even out-performs humans on selected labels such as Text, Table and Picture Thisis not entirely surprising, as Text , Table and Picture are abundant and the most visually distinctive in a document.
Figure 3: Page 6 of the DocLayNet paper. If recognized, metadata such as authors are appearing first under the title. Elements recognized as page headers or footers are suppressed in Markdown to deliver uninterrupted content in reading order. Tables are inserted in reading order. The paragraph in '5. Experiments' wrapping over the column end is broken up in two and interrupted by the table.
KDD '22, August 14-18, 2022, Washington, DC, USA
Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S. Nassar, and Peter Staar
Table 1: DocLayNet dataset overview. Along with the frequency of each class label, we present the relative occurrence (as %
<!-- image -->
of row 'Total') in the train, test and validation sets. The inter-annotator agreement is computed as the mAP@0.5-0.95 metric
we distributed the annotation workload and performed continuous
only. For phases three and four, a group of 40 dedicated annotators
quality controls. Phase one and two required a small team of experts
were assembled and supervised.
while coverage ensures that all meaningful items on a page can
to a document category, such as
be annotated. We refrained from class labels that are very specific
Abstract in the Scientific Articles
semantics of the text. Labels such as
category. We also avoided class labels that are tightly linked to the
Author Affiliation and
teria for documents were described in Section 3. A large effort went into ensuring that all documents are free to use. The data sources in DocBank, are often only distinguishable by discriminating on 3 https://arxiv.org/ Figure 4: Table 1 from the DocLayNet paper in the original PDF (A), as rendered Markdown (B) and in JSON representation (C). Spanning table cells, such as the multi-column header 'triple interannotator mAP@0.5-0.95 (%)', is repeated for each column in the Markdown representation (B), which guarantees that every data point can be traced back to row and column headings only by its grid coordinates in the table. In the JSON representation, the span information is reflected in the fields of each table cell (C).
, as seen
Phase 1: Data selection and preparation.
Our inclusion cri-

9
pyproject.toml Normal file
View file

@ -0,0 +1,9 @@
[project]
name = "137-docing-test"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
"docling>=2.96.0",
]

2255
uv.lock Normal file
View file

File diff suppressed because it is too large Load diff