Skip to content
scikit-hepPublic

Latest commit

 

History

3,006 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

PyPI version Conda-Forge Python 3.10‒3.14 BSD-3 Clause License Build Test Property Tests

Scikit-HEP DOI Documentation

NSF-1836650 NSF-2103945 NSF-2121686 NSF-2323298

Awkward Array is a library for nested, variable-sized data, including arbitrary-length lists, records, mixed types, and missing data, using NumPy-like idioms.

Arrays are dynamically typed, but operations on them are compiled and fast. Their behavior coincides with NumPy when array dimensions are regular and generalizes when they're not.

Motivating example

Given an array of lists of objects with x, y fields (with nested lists in the y field),

import awkward as ak

array = ak.Array([
    [{"x": 1.1, "y": [1]}, {"x": 2.2, "y": [1, 2]}, {"x": 3.3, "y": [1, 2, 3]}],
    [],
    [{"x": 4.4, "y": [1, 2, 3, 4]}, {"x": 5.5, "y": [1, 2, 3, 4, 5]}]
])

the following slices out the y values, drops the first element from each inner list, and runs NumPy's np.square function on everything that is left:

output = np.square(array["y", ..., 1:])

The result is

[
    [[], [4], [4, 9]],
    [],
    [[4, 9, 16], [4, 9, 16, 25]]
]

The equivalent using only Python is

output = []
for sublist in array:
    tmp1 = []
    for record in sublist:
        tmp2 = []
        for number in record["y"][1:]:
            tmp2.append(np.square(number))
        tmp1.append(tmp2)
    output.append(tmp1)

The expression using Awkward Arrays is more concise, using idioms familiar from NumPy, and it also has NumPy-like performance. For a similar problem 10 million times larger than the one above (single-threaded on a 2.2 GHz processor),

  • the Awkward Array one-liner takes 1.5 seconds to run and uses 2.1 GB of memory,
  • the equivalent using Python lists and dicts takes 140 seconds to run and uses 22 GB of memory.

Awkward Array is even faster when used in Numba's JIT-compiled functions.

See the Getting started documentation on awkward-array.org for an introduction, including a no-install demo you can try in your web browser.

Getting help

Installation

Awkward Array can be installed from PyPI using pip:

pip install awkward

The awkward package is pure Python, and it will download the awkward-cpp compiled components as a dependency. If there is no awkward-cpp binary package (wheel) for your platform and Python version, pip will attempt to compile it from source (which has additional dependencies, such as a C++ compiler).

Awkward Array is also available on conda-forge:

conda install -c conda-forge awkward

Because of the two packages (awkward-cpp may be updated in GitHub but not on PyPI), pip install through git (pip install git+https://...) will not work. Instead, use the Installation for developers section below.

Installation for developers

Clone this repository recursively to get the header-only C++ dependencies. The repository is a uv workspace, so uv sync (or any uv run command) compiles awkward-cpp and installs both packages as editable installations:

git clone --recursive https://github.com/scikit-hep/awkward.git
cd awkward

uv sync

Without uv, compile and install awkward-cpp, then install awkward as an editable installation:

python -m pip install -v ./awkward-cpp
python -m pip install -e .

Tests can be run in parallel with pytest (prefix with uv run if you use uv):

python -m pytest -n auto tests

For more details, see CONTRIBUTING.md, or one of the links below.

Documentation, Release notes, Roadmap, Citations

The documentation is on awkward-array.org, including

The Release notes for each version are in the GitHub Releases tab.

The Roadmap, Plans, and Deprecation Schedule are in the GitHub Wiki.

To cite Awkward Array in a paper, see the "Cite this repository" drop-down menu on the top-right of the GitHub front page. The BibTeX is

@software{Pivarski_Awkward_Array_2018,
author = {Pivarski, Jim and Osborne, Ianna and Ifrim, Ioana and Schreiner, Henry and Hollands, Angus and Biswas, Anish and Das, Pratyush and Roy Choudhury, Santam and Smith, Nicholas and Goyal, Manasvi and Fackeldey, Peter and Krommydas, Iason and Rios-Tascon, Andres and Sakuma, Tai},
doi = {10.5281/zenodo.4341376},
month = {10},
title = {{Awkward Array}},
year = {2018}
}

Acknowledgements

Support for this work was provided by NSF cooperative agreement OAC-1836650 (IRIS-HEP 1), PHY-2323298 (IRIS-HEP 2), grant OAC-1450377 (DIANA/HEP), PHY-1520942 and PHY-2121686 (US-CMS LHC Ops), and OAC-2103945 (Awkward Array).

Awkward Array is a NumFOCUS Affiliated Project.

We also thank Erez Shinan and the developers of the Lark standalone parser, which is used to parse type strings as type objects.

“The Awkward Array developers” refers collectively to the project’s contributors, acknowledged in the table below and in .all-contributorsrc. The Git history records individual contributions.

Thanks especially to the gracious help of Awkward Array contributors (including the original repository).

Jim Pivarski
Jim Pivarski

💻 📖 🚇 🚧
Ianna Osborne
Ianna Osborne

💻
Pratyush Das
Pratyush Das

💻
Anish Biswas
Anish Biswas

💻
glass-ships
glass-ships

💻 ⚠️
Henry Schreiner
Henry Schreiner

💻 🚇
Nicholas Smith
Nicholas Smith

💻 ⚠️
Lindsey Gray
Lindsey Gray

💻 ⚠️
Ellipse0934
Ellipse0934

⚠️
Dmitry Kalinkin
Dmitry Kalinkin

🚇
Charles Escott
Charles Escott

💻
Mason Proffitt
Mason Proffitt

💻
Michael Hedges
Michael Hedges

💻
Jonas Rembser
Jonas Rembser

💻
Jaydeep Nandi
Jaydeep Nandi

💻
benkrikler
benkrikler

💻
bfis
bfis

💻
Doug Davis
Doug Davis

💻
Joosep Pata
Joosep Pata

🤔
Martin Durant
Martin Durant

🤔
Gordon Watts
Gordon Watts

🤔
Nikolai Hartmann
Nikolai Hartmann

💻
Simon Perkins
Simon Perkins

💻
.hard
.hard

💻 ⚠️
HenryDayHall
HenryDayHall

💻
Angus Hollands
Angus Hollands

⚠️ 💻
ioanaif
ioanaif

💻 ⚠️
Bernhard M. Wiedemann
Bernhard M. Wiedemann

🚧
Matthew Feickert
Matthew Feickert

🚧
Santam Roy Choudhury
Santam Roy Choudhury

⚠️
Jeroen Van Goey
Jeroen Van Goey

📖
Ahmad-AlSubaie
Ahmad-AlSubaie

💻
Manasvi Goyal
Manasvi Goyal

💻
Aryan Roy
Aryan Roy

💻
Saransh
Saransh

💻
Laurits Tani
Laurits Tani

📖
Daniel Savoiu
Daniel Savoiu

💻
Ray Bell
Ray Bell

📖
Andrea Zonca
Andrea Zonca

💻
Chris Burr
Chris Burr

🚇
Zoë Bilodeau
Zoë Bilodeau

💻
Raymond Ehlers
Raymond Ehlers

🚧
Markus Löning
Markus Löning

📖
Kush Kothari
Kush Kothari

💻 ⚠️
Jonas Rübenach
Jonas Rübenach

💻
Jerry Ling
Jerry Ling

📖 💻
Luis Antonio Obis Aparicio
Luis Antonio Obis Aparicio

💻
Topher Cawlfield
Topher Cawlfield

💻
Massimiliano Galli
Massimiliano Galli

💻
Peter Fackeldey
Peter Fackeldey

💻
Andres Rios Tascon
Andres Rios Tascon

💻
maxymnaumchyk
maxymnaumchyk

💻
Thomas A Caswell
Thomas A Caswell

🚧
Bas Nijholt
Bas Nijholt

🚧
Igor Vaiman
Igor Vaiman

💻
Havryliuk Artem
Havryliuk Artem

💻
Iason Krommydas
Iason Krommydas

💻 ⚠️
Nick
Nick

💻
Alexander Puck Neuwirth
Alexander Puck Neuwirth

💻
Nilesh Patra
Nilesh Patra

💻
Tai Sakuma
Tai Sakuma

💻 ⚠️ 🚇 📖 🚧
Dylan Modesitt
Dylan Modesitt

💻
James Brofos
James Brofos

💻
Xiao
Xiao

💻
Ashwin Srinath
Ashwin Srinath

💻
aashirvad
aashirvad

💻
albert-physics
albert-physics

💻
Sushanth Mantrala
Sushanth Mantrala

💻
kmohrman
kmohrman

💻
Soumoditya Pramanik
Soumoditya Pramanik

📖
Aaron  Jomy
Aaron Jomy

💻
Paramveer singh
Paramveer singh

💻
renta0426
renta0426

💻
Vincenzo Eduardo Padulano
Vincenzo Eduardo Padulano

💻

💻: code, 📖: documentation, 🚇: infrastructure, 🚧: maintenance, ⚠: tests and feedback, 🤔: foundational ideas.

Releases

Used by

Contributors

Languages