OpusFilter
OpusFilter is a tool for filtering and combining parallel corpora.
Features:
- Corpus preprocessing pipelines configured with YAML
- Simple downloading of parallel corpora from OPUS with OpusTools
- Implementations for many common text file operations on parallel files
- Memory-efficient processing of large files
- Implemented filters based e.g. on language identification, word aligment, n-gram language models, and multilingual sentence embeddings
- Extendable with your own filters written in Python
OpusFilter has been presented in ACL 2020 system demonstrations.
Installing
Install the latest release from PyPI:
pip install opusfilterorpip install opusfilter[all](include optional Python libraries)
Install from source:
pip install .orpython setup.py install
Troubleshooting
OpusFilter should generally work fine on Python 3.8 to 3.13. In the case of troubles, try installing the exact versions in requirements.txt:
pip install -r requirements.txt
Documentation
The complete OpusFilter documentation is available from helsinki-nlp.github.io/OpusFilter.
You can also build the documents from the source:
pip install -r docs/requirements.txtorpip install .[docs]sphinx-build docs docs-html
Changelog
A changelog is available in docs/CHANGELOG.md.
Citing
If you use OpusFilter in your research, please cite our ACL 2020 paper:
@inproceedings{aulamo-etal-2020-opusfilter,
title = "{O}pus{F}ilter: A Configurable Parallel Corpus Filtering Toolbox",
author = {Aulamo, Mikko and Virpioja, Sami and Tiedemann, J{\"o}rg},
booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
month = jul,
year = "2020",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.acl-demos.20",
doi = "10.18653/v1/2020.acl-demos.20",
pages = "150--156"
}
A full bibliography of papers cited in the documentation and code can be found from docs/references.bib.
Contributing
See docs/CONTRIBUTING.md.
Release files for opusfilter 3.3.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| opusfilter-3.3.1.tar.gz | 130.0 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| opusfilter-3.3.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 198.9 kB
Release files / opusfilter-3.3.1.tar.gz
| Download URL | opusfilter-3.3.1.tar.gz |
|---|---|
| Size | 130.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2e1ff1d8ebc624c4055a759b31f27a06386391368fcdf535c0474ebfc516c12b
|
|
BLAKE2b-256 checksum How to use checksums |
798575873617090e696d33aa2d098ad53191ce95bab5453e9d85fdb7522b0b2b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Release files / opusfilter-3.3.1-py3-none-any.whl
| Download URL | opusfilter-3.3.1-py3-none-any.whl |
|---|---|
| Size | 68.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
832e0369301f73ec66371f25c6362f0e6a4a490acc3b96617d63d3f63ea4429b
|
|
BLAKE2b-256 checksum How to use checksums |
8f79b5501a5d85176f3fa2309b147d87f6fa8220cfbd9a8a78a972e97460d4b7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Security for opusfilter 3.3.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
