Skip to content

Releases: mindee/doctr

Release list

v1.1.0

v1.1.0 Latest
Latest

Choose a tag to compare

View all tags
@felixdittrich92 felixdittrich92 released this 21 Aug 13:08
v1.1.0
2d8e244
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Note: docTR 1.1.0 requires python >= 3.11

Note

docTR is now a full document understanding toolkit. On top of text detection and recognition, v1.1.0 adds layout analysis, table structure recognition and a reading-order aware export to Markdown / AsciiDoc / HTML / XML - plus a CLI so you can run the whole pipeline without writing a single line of Python.

What's Changed

Breaking Changes 🛠

  • The minimum supported Python version was raised from 3.10 to 3.11
  • The opencv-python upper bound was relaxed to <6.0.0

New features

  • Layout analysis with the new LW-DETR architecture (lw_detr_s, lw_detr_m) by @felixdittrich92
  • Table structure recognition with TableCenterNet and a StarNet backbone by @felixdittrich92
  • Reading order module and new exporters (Markdown / AsciiDoc / HTML / XML) by @felixdittrich92
  • Vocab whitelisting to constrain the recognition model to a known character set by @felixdittrich92
  • doctr-cli to run end-to-end OCR from your shell by @simont2k
  • Layout masking (ignore_regions) to skip page furniture before detection/recognition by @felixdittrich92
  • preserve_original_coords to map boxes back to the original page after straighten_pages=True by @saad-rd11
  • New dataset classes (LayoutDataset, TableStructureDataset), new metrics (ObjectDetectionMetric, TableCellMetric) and new reference scripts for both tasks

✨ Layout Analysis ✨

Layout analysis localizes and classifies the visual elements of a page - not just the words. The new LW-DETR models are trained on 11 classes (Caption, Footnote, Formula, List-item, Page-footer, Page-header, Picture, Section-header, Table, Text, Title) and, unlike most layout detectors, also work on rotated and skewed documents.

layout_example

Left: words found by db_resnet50 + crnn_vgg16_bn. Right: the 18 regions returned by lw_detr_s on the same page.

Available architectures:

Architecture Input shape # params mAP@[.5:.95] AP@[.5] AP@[.75] sec/it (B: 1) - CPU
lw_detr_s (1024, 1024, 3) 15.1 M 66.89 82.83 73.75 0.5
lw_detr_m (1024, 1024, 3) 29.5 M soon soon soon 0.7

Standalone usage

import numpy as np
from doctr.models import layout_predictor
 
model = layout_predictor("lw_detr_s", pretrained=True)
dummy_img = (255 * np.random.rand(800, 600, 3)).astype(np.uint8)
out = model([dummy_img])
# out[0] -> {"class_names": ["Title", "Text", ...], "boxes": array(...), "scores": array(...)}

Like the detection predictor, it accepts assume_straight_pages, preserve_aspect_ratio and symmetric_pad:

from doctr.models import layout_predictor
 
predictor = layout_predictor(
    "lw_detr_s", pretrained=True, assume_straight_pages=False, preserve_aspect_ratio=True
)

Inside the OCR pipeline

Pass detect_layout=True to the ocr_predictor (or the kie_predictor) and the detected regions are attached to every page, exported alongside the rest of the page and rendered by .show():

from doctr.io import DocumentFile
from doctr.models import ocr_predictor
 
model = ocr_predictor(pretrained=True, detect_layout=True, layout_arch="lw_detr_s")
doc = DocumentFile.from_images("path/to/your/doc.jpg")
result = model(doc)
 
# Access the detected layout regions of the first page
for region in result.pages[0].layout:
    print(region.type, region.confidence, region.geometry)
# Title 0.97 ((0.13, 0.06), (0.87, 0.11))
# Text 0.95 ((0.11, 0.14), (0.89, 0.42))
# Table 0.93 ((0.12, 0.45), (0.88, 0.79))
 
# The layout is part of the exported representation
export = result.pages[0].export()
print(export["layout"])
 
# Overlay both text and layout regions (use display_layout=False to hide the regions)
result.pages[0].show()

Ignoring regions

masking_example

Layout regions can also be used to remove content before it ever reaches the detection and recognition models. Pass the class names you want to drop to ignore_regions — the corresponding areas are masked out (filled with black), which speeds up inference and keeps page furniture out of your results:

from doctr.models import ocr_predictor
 
# Headers, footers and pictures are masked out before detection & recognition
model = ocr_predictor(pretrained=True, ignore_regions=["Page-header", "Page-footer", "Picture"])

✨ Table Structure Recognition ✨

Table structure recognition parses a table into a machine-understandable representation: it localizes every cell (spatial structure) and recovers the rows and columns each cell spans (logical structure). The new TableCenterNet architecture is built on a lightweight StarNet backbone (7.1 M params).

table_example

The 20 cells predicted by tablecenternet with their logical (row, column) coordinates, and the dense grid to_grid() builds from them.

Architecture Input shape # params Recall Precision F1 Structure acc sec/it (B: 1)
tablecenternet (1024, 1024, 3) 7.1 M 82.31 96.01 88.64 77.53 0.5

Metrics are produced by references/table/evaluate.py using the new TableCellMetric: cell-detection Recall, Precision and F1 (cells matched above an IoU threshold of 0.5), and Structure acc, the share of matched cells whose logical (row/column) coordinates are correctly predicted.

Standalone usage

import numpy as np
from doctr.models import table_predictor
 
model = table_predictor("tablecenternet", pretrained=True)
table_crop = (255 * np.random.rand(800, 600, 3)).astype(np.uint8)
out = model([table_crop])
# out[0] -> {"cells": [{"geometry": ..., "score": ..., "row_start": 0, "row_end": 0,
#            "col_start": 0, "col_end": 0}, ...], "num_rows": ..., "num_cols": ...}

Inside the OCR pipeline

Passing detect_tables=True to the ocr_predictor wires layout detection and table structure recognition together: every region the layout model labels as a Table is cropped and passed to the table model (so detect_tables=True implicitly enables the layout model). Words whose center falls inside a detected cell are regrouped into a structured table available under page.tables - and removed from the regular blocks output, so the same text is never returned twice.

from doctr.io import DocumentFile
from doctr.models import ocr_predictor
 
model = ocr_predictor(pretrained=True, detect_tables=True)
doc = DocumentFile.from_images("invoice_with_table.png")
result = model(doc)
 
page = result.pages[0]
for i, table in enumerate(page.tables):
    print(f"Table {i} ({table.num_rows}x{table.num_cols}):")
    print(table.to_grid())  # dense list of lists -> loads straight into pandas
 
# The remaining (non-table) text is still available as usual
print(page.render())
# Table 0 (3x3):
# [['Item', 'Qty', 'Price'],
#  ['Coffee', '2', '7.00'],
#  ['Tea', '1', '3.50']]
 
import pandas as pd
grid = page.tables[0].to_grid()
df = pd.DataFrame(grid[1:], columns=grid[0])

Tables are included in Page.export() under the tables key, so they survive the JSON export as well.

✨ Reading order & new exporters ✨

markdown_example

Results can now be linearized in reading order and exported to Markdown, AsciiDoc, HTML or XML. The content is ordered column by column, the reading direction is inferred from the recognized text (e.g. right-to-left for Arabic or Hebrew documents), and - when the predictor runs with detect_layout=True - the layout regions are used to render headings, list items and recognized tables, and to place page furniture (headers, footers, footnotes).

from doctr.io import DocumentFile
from doctr.models import ocr_predictor
 
model = ocr_predictor(pretrained=True, detect_layout=True)
result = model(DocumentFile.from_pdf("path/to/your/doc.pdf"))
 
markdown_output = result.export_as_markdown()
asciidoc_output = result.export_as_asciidoc()
html_output = result.export_as_html()
xml_output = result.export_as_xml()          # hOCR
raw_text_output = result.render()            # same as result.export_as("text")
dict_output = result.export()                # same as result.export_as("json")

export_as is a convenience dispatcher over all formats:

result.export_as("markdown")  # or "md"
result.export_as("asciidoc")  # or "adoc"
result.export_as("html")
result.export_as("text")      # same as render()
result.export_as("json")      # same as export()
result.export_as("xml")       # same as export_as_xml()

Sample Markdown output:

# Invoice 2024-042
 
## Billing address
 
Mindee, 54 Rue de Paradis, 75010 Paris
 
| Item | Qty | Price |
| --- |...
Read more

Contributors

badranx, felixdittrich92, and 5 other contributors
Assets 6
Loading

v1.0.1

v1.0.1

Choose a tag to compare

View all tags
@felixdittrich92 felixdittrich92 released this 04 Feb 14:45
v1.0.1
6541a6e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Note: docTR 1.0.1 requires python >= 3.10

What's Changed

Bug Fixes

Miscellaneous

New Contributors

Full Changelog: v1.0.0...v1.0.1

Contributors

SteffenSH, felixdittrich92, and 2 other contributors
Loading

v1.0.0

v1.0.0

Choose a tag to compare

View all tags
@felixdittrich92 felixdittrich92 released this 09 Jul 11:39
v1.0.0
7dabbe1
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Note: docTR 1.0.0 requires python >= 3.10

What's Changed

Breaking Change

TensorFlow has been removed as a supported backend. docTR now comes with PyTorch as the default and only deep learning backend.

The installation options torch and tf have been removed. You can now install docTR simply with:

pip install python-doctr

This will install docTR with PyTorch support by default.

Training script filenames have been updated to remove backend-specific extensions. For example:

recognition/train_pytorch.py → recognition/train.py

New features

  • A new crnn_vgg16_bn checkpoint was added

What's Changed

Breaking Changes 🛠

Bug Fixes

  • [bug] Fix viptr onnx export issue by @felixT2K in #1966
  • [Fix] Correct condition for image dilation in orientation estimation by @Razlaw in #1971

Improvements

Miscellaneous

New Contributors

Full Changelog: v0.12.0...v1.0.0

Contributors

Razlaw, felixdittrich92, and 2 other contributors
Loading

v0.12.0

v0.12.0

Choose a tag to compare

View all tags
@felixdittrich92 felixdittrich92 released this 20 Jun 07:43
v0.12.0
97d4006
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Note: docTR 0.12.0 requires python >= 3.10
Note: docTR 0.12.0 requires either TensorFlow >= 2.15.0 or PyTorch >= 2.0.0

Warning

TensorFlow Backend Deprecation Notice

Using docTR with TensorFlow as a backend is deprecated and will be removed in the next major release (v1.0.0).
We recommend switching to the PyTorch backend, which is more actively maintained and supports the latest features and models.
Alternatively, you can use OnnxTR, which does not require TensorFlow or PyTorch.

This decision was made based on several considerations:

  • Allows better focus on improving the core library
  • Frees up resources to develop new features faster
  • Enables more targeted optimizations with PyTorch

Warning

This release is the last minor release supporting TensorFlow as backend

What's changed

New features

  • A new lightweight recognition model viptr_tiny was added
  • New built-in dataset added - COCO-Text V2
  • A new custom model loading interface
# NEW
model = vitstr_small(pretrained=False, pretrained_backbone=False)
model.from_pretrained("<PATH_TO>")  # local path or url to .pt or .h5

# Instead of depending on the backend
reco_params = torch.load('<path_to_pt>', map_location="cpu")
reco_model.load_state_dict(reco_params)
# Or with TensorFlow
reco_model.load_weights(..)

What's Changed

Breaking Changes 🛠

New Features

  • [datasets] COCO-Text V2 integration by @sarjil77 in #1888
  • [references] Recognition - Allow built-in datasets usage by @sarjil77 in #1904
  • [Feat] PyTorch - VIP backbone and VIPTR recognition module by @lkosh in #1912

Bug Fixes

Improvements

Miscellaneous

New Contributors

Full Changelog: v0.11.0...v0.12.0

Contributors

lachesis, ulfaslak, and 11 other contributors
Loading

v0.11.0

v0.11.0

Choose a tag to compare

View all tags
@felixdittrich92 felixdittrich92 released this 30 Jan 09:25
v0.11.0
1c9ce92
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Note: docTR 0.11.0 requires python >= 3.10
Note: docTR 0.11.0 requires either TensorFlow >= 2.15.0 or PyTorch >= 2.0.0

What's changed

New features

  • Added torch.compile support (PyTorch backend)
  • Improved model training logging
  • Created a small labeling tool designed for docTR (early stage) --> doctr-labeler

Compile your model

Compiling your PyTorch models with torch.compile optimizes the model by converting it to a graph representation and applying backends that can improve performance.
This process can make inference faster and reduce memory overhead during execution.

Further information can be found in the PyTorch documentation

import torch
from doctr.models import (
    ocr_predictor,
    vitstr_small,
    fast_base,
    mobilenet_v3_small_crop_orientation,
    mobilenet_v3_small_page_orientation,
    crop_orientation_predictor,
    page_orientation_predictor
)

# Compile the models
detection_model = torch.compile(
    fast_base(pretrained=True).eval()
)
recognition_model = torch.compile(
    vitstr_small(pretrained=True).eval()
)
crop_orientation_model = torch.compile(
    mobilenet_v3_small_crop_orientation(pretrained=True).eval()
)
page_orientation_model = torch.compile(
    mobilenet_v3_small_page_orientation(pretrained=True).eval()
)

predictor = models.ocr_predictor(
    detection_model, recognition_model, assume_straight_pages=False
)
# NOTE: Only required for non-straight pages (`assume_straight_pages=False`) and non-disabled orientation classification
# Set the orientation predictors
predictor.crop_orientation_predictor = crop_orientation_predictor(crop_orientation_model)
predictor.page_orientation_predictor = page_orientation_predictor(page_orientation_model)

compiled_out = predictor(doc)

What's Changed

New Features

Bug Fixes

Improvements

Miscellaneous

New Contributors

Full Changelog: v0.10.0...v0.11.0

Contributors

cmoscardi, Razlaw, and 8 other contributors
Loading

v0.10.0

v0.10.0

Choose a tag to compare

View all tags
@felixdittrich92 felixdittrich92 released this 21 Oct 08:37
v0.10.0
d5dbc73
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Note: docTR 0.10.0 requires python >= 3.9
Note: docTR 0.10.0 requires either TensorFlow >= 2.15.0 or PyTorch >= 2.0.0

What's Changed

Soft Breaking Changes (TensorFlow backend only) 🛠

  • Changed the saving format from /weights to .weights.h5

NOTE: Please update your custom trained models and HuggingFace hub uploaded models, this will be the last release supporting manual loading from /weights.

New features

Disable page orientation classification

  • If you deal with documents which contains only small rotations (~ -45 to 45 degrees), you can disable the page orientation classification to speed up the inference.
  • This will only have an effect with assume_straight_pages=False and/or straighten_pages=True and/or detect_orientation=True.
from doctr.models import ocr_predictor

model = ocr_predictor(pretrained=True, assume_straight_pages=False, disable_page_orientation=True)

Disable crop orientation classification

  • If you deal with documents which contains only horizontal text, you can disable the crop orientation classification to speed up the inference.
  • This will only have an effect with assume_straight_pages=False and/or straighten_pages=True.
from doctr.models import ocr_predictor

model = ocr_predictor(pretrained=True, assume_straight_pages=False, disable_crop_orientation=True)

Loading custom exported orientation classification models

You can now load your custom trained orientation models, the following snippet demonstrates how:

from doctr.io import DocumentFile
from doctr.models import ocr_predictor, mobilenet_v3_small_page_orientation, mobilenet_v3_small_crop_orientation
from doctr.models.classification.zoo import crop_orientation_predictor, page_orientation_predictor

custom_page_orientation_model = mobilenet_v3_small_page_orientation("<PATH_TO_CUSTOM_EXPORTED_ONNX_MODEL>")
custom_crop_orientation_model = mobilenet_v3_small_crop_orientation("<PATH_TO_CUSTOM_EXPORTED_ONNX_MODEL>"))

predictor = ocr_predictor(pretrained=True, assume_straight_pages=False, detect_orientation=True)

# Overwrite the default orientation models
predictor.crop_orientation_predictor = crop_orientation_predictor(custom_crop_orientation_model)
predictor.page_orientation_predictor = page_orientation_predictor(custom_page_orientation_model)

What's Changed

Breaking Changes 🛠

New Features

Bug Fixes

Improvements

Miscellaneous

New Contributors

Full Changelog: v0.9.0...v0.10.0

Contributors

milosacimovic, felixdittrich92, and 3 other contributors
Loading

v0.9.0

v0.9.0

Choose a tag to compare

View all tags
@felixdittrich92 felixdittrich92 released this 08 Aug 13:57
v0.9.0
894eafd
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Note: docTR 0.9.0 requires python >= 3.9
Note: docTR 0.9.0 requires either TensorFlow >= 2.11.0 or PyTorch >= 1.12.0.

What's Changed

Soft Breaking Changes 🛠

  • The default detection model changed from db_resnet50 to fast_base.
    NOTE: Can be reverted by passing the detection model predictor = ocr_predictor(det_arch="db_resnet50", pretrained=True)
  • The default value of resolve_blocks changed from True to False
    NOTE: Can be reverted by passing resolve_blocks=True to the ocr_predictor

New features

✨ Installation ✨

We have splitted docTR into some optional parts to make it a bit more lightweight and to exclude parts which are not required for inference.
Optional parts are:

  • visualization (to support .show())
  • html support (to support .from_url(...))
  • contribution module
# for TensorFlow without any optional dependencies
pip install "python-doctr[tf]"

# for PyTorch without any optional dependencies
pip install "python-doctr[torch]"

# Installs pytorch and all available optional parts
pip install "python-doctr[torch,viz,html,contib]"

✨ ONNX and OnnxTR ✨

We have build a standalone library to provide a super lightweight way to use existing docTR onnx exported models or your custom onces.

benefits:

  • kown docTR interface (ocr_predictor, etc.)
  • no PyTorch or TensorFlow required - build on top of onnxruntime
  • more lightweight package with faster inference latency and less required resources
  • 8-Bit quantized models for faster inference on CPU

Give it a try and check it out: OnnxTR
docTR docs: ONNX / OnnxTR

Screenshot from 2024-08-09 09-15-37

What's Changed

Breaking Changes 🛠

  • [models] Change default model to fast_base - soft breaking change by @felixdittrich92 in #1588
  • [misc] update README & fix mypy & change resolve blocks default by @felixT2K in #1686

New Features

Bug Fixes

Improvements

Miscellaneous

New Contributors

Full Changelog: v0.8.1...v0.9.0

Contributors

justinjosephmkj, felixdittrich92, and 5 other contributors
Loading

v0.8.1

v0.8.1

Choose a tag to compare

View all tags
@felixdittrich92 felixdittrich92 released this 04 Mar 14:50
v0.8.1
62d94ff
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Note: doctr 0.8.1 requires either TensorFlow >= 2.11.0 or PyTorch >= 1.12.0.

What's Changed

Loading

v0.8.0

v0.8.0

Choose a tag to compare

View all tags
@felixdittrich92 felixdittrich92 released this 28 Feb 13:13
v0.8.0
67d1087
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Note: doctr 0.8.0 requires either TensorFlow >= 2.11.0 or PyTorch >= 1.12.0.

What's Changed

Breaking Changes 🛠

  • db_resnet50_rotation (PyTorch) and linknet_resnet18_rotation (TensorFlow) are removed (All models can handle rotated documents now)
  • .show(doc) changed to .show()

New features

  • All models have pretrained checkpoints now by @odulcy-mindee
  • All detection models was retrained on rotated samples by @odulcy-mindee
  • Improved orientation detection for documents rotated between -90 and 90 degrees by @felixdittrich92
  • Conda deployment job & receipt added by @frgfm
  • Official docTR docker images are added by @odulcy-mindee => docker-images
  • New benchmarks and documentation improvements by @felixdittrich92
  • WildReceipt dataset added by @HamzaGbada
  • EarlyStopping callback added to all training scripts by @SkaarFacee
  • Hook mechanism added to ocr_predictor to maniplulate the detection predictions in the middle of the pipeline to your needs by @felixdittrich92
from doctr.model import ocr_predictor

class CustomHook:
    def __call__(self, loc_preds):
        # Manipulate the location predictions here
        # 1. The outpout structure needs to be the same as the input location predictions
        # 2. Be aware that the coordinates are relative and needs to be between 0 and 1
        return loc_preds

my_hook = CustomHook()

predictor = ocr_predictor(pretrained=True)
# Add a hook in the middle of the pipeline
predictor.add_hook(my_hook)
# You can also add multiple hooks which will be executed sequentially
for hook in [my_hook, my_hook, my_hook]:
    predictor.add_hook(hook)

What's Changed

Breaking Changes 🛠

New Features

Bug Fixes

Improvements

Miscellaneous

New Contributors

Full Changelog: v0.7.0...v0.8.0

Contributors

simonw, nh2, and 11 other contributors
Loading

v0.7.0

v0.7.0

Choose a tag to compare

View all tags
@felixdittrich92 felixdittrich92 released this 09 Sep 13:23
v0.7.0
75bddfc
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Note: doctr 0.7.0 requires either TensorFlow >= 2.11.0 or PyTorch >= 1.12.0.
Note: We will release the missing PyTorch checkpoints with 0.7.1

What's Changed

Breaking Changes 🛠

  • We changed the preserve_aspect_ratio parameter to True by default in #1279
    => To restore the old behaviour you can pass preserve_aspect_ratio=False to the predictor instance

New features

Add of the KIE predictor

The KIE predictor is a more flexible predictor compared to OCR as your detection model can detect multiple classes in a document. For example, you can have a detection model to detect just dates and adresses in a document.

The KIE predictor makes it possible to use detector with multiple classes with a recognition model and to have the whole pipeline already setup for you.

from doctr.io import DocumentFile
from doctr.models import kie_predictor

# Model
model = kie_predictor(det_arch='db_resnet50', reco_arch='crnn_vgg16_bn', pretrained=True)
# PDF
doc = DocumentFile.from_pdf("path/to/your/doc.pdf")
# Analyze
result = model(doc)

predictions = result.pages[0].predictions
for class_name in predictions.keys():
    list_predictions = predictions[class_name]
    for prediction in list_predictions:
        print(f"Prediction for {class_name}: {prediction}")

The KIE predictor results per page are in a dictionary format with each key representing a class name and it's value are the predictions for that class.

What's Changed

Breaking Changes 🛠

New Features

Bug Fixes

Improvements

Miscellaneous

New Contributors

Full Changelog: v0.6.0...v0.7.0

Contributors

venkatapathy, eltociear, and 11 other contributors
Loading