kerchunk
Cloud-friendly access to archival data
Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF, HDF5, GRIB), allowing efficient access to the data from traditional file systems or cloud object storage. It also provides a flexible way to create virtual datasets from multiple files. It does this by extracting the byte ranges, compression information and other information about the data and storing this metadata in a new, separate object. This means that you can create a virtual aggregate dataset over potentially many source files, for efficient, parallel and cloud-friendly in-situ access without having to copy or translate the originals. It is a gateway to in-the-cloud massive data processing while the data providers still insist on using legacy formats for archival storage.
Why Kerchunk:
We provide the following things:
- completely serverless architecture
- metadata consolidation, so you can understand a many-file dataset (metadata plus physical storage) in a single read
- read from all of the storage backends supported by fsspec, including object storage (s3, gcs, abfs, alibaba), http, cloud user storage (dropbox, gdrive) and network protocols (ftp, ssh, hdfs, smb...)
- loading of various file types (currently netcdf4/HDF, grib2, tiff, fits, zarr), potentially heterogeneous within a single dataset, without a need to go via the specific driver (e.g., no need for h5py)
- asynchronous concurrent fetch of many data chunks in one go, amortizing the cost of latency
- parallel access with a library like zarr without any locks
- logical datasets viewing many (>~millions) data files, and direct access/subselection to them via coordinate indexing across an arbitrary number of dimensions
For further information, please see the documentation.
1 maintainer
Metadata
1 maintainer
Release files for kerchunk 0.2.10
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| kerchunk-0.2.10.tar.gz | 716.5 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| kerchunk-0.2.10-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 785.0 kB
Release files / kerchunk-0.2.10.tar.gz
| Download URL | kerchunk-0.2.10.tar.gz |
|---|---|
| Size | 716.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
aae63c0fe4ca2e97025f026578a0577545011fe6679751392c815e0d1d6bf954
|
|
BLAKE2b-256 checksum How to use checksums |
5ca0ebeca522912e68f360117404a6b2f740d4b0a343d98e9586377a2fd21567
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.11
|
Release files / kerchunk-0.2.10-py3-none-any.whl
| Download URL | kerchunk-0.2.10-py3-none-any.whl |
|---|---|
| Size | 68.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7fdaa77dae25c75d3ec9402c49208f37ae51d346ab082724e3e32608438f8c66
|
|
BLAKE2b-256 checksum How to use checksums |
64e43c356a9ea448a48caa5e44cd51293f7e896cd606f2ef86da96f5d61cc427
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.11
|
Security for kerchunk 0.2.10
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
