Will node.js implementation be a first class citizen?

[geertbrit]

To my surprise I just saw node.js being listed as a target for polars. I’m pretty sure this wasn’t the case a month ago.

Is the intent to support node.js as a first class citizen with feature parity to python/rust?

The node-ecosystem could use a proper arrow library, it’s lagging a lot so this would be most welcome :) This includes things like arrow <-> parquet, recordbatch predicate push-down etc., but I’m sure you’re aware.

[geertbrit]

@ritchie46 think this issue was closed on accident?

[universalmind303]

The node.js implementation is almost up to feature parity with the python/rust implementations.

A few things like the dynamic-groupby, and apply/map still need to be implemented in nodejs though.

[ritchie46]

With regard to features I think it almost is already by virtue of the efforts of @universalmind303 🙌 .

With regard to guarantees about the nodejs impementation this depends more on the community than the Rust and Python implementation do. Those are maintained by me. I am however no expert in TS/JS and don't have the capacity two maintain more than 2 APIs.

I think that as long as we keep @universalmind303 on board on the project it will be. ;)

[universalmind303]

Some of the features like to_arrow don't really make sense to implement on the node side, as node doesn't really have a stable arrow library. Some of the other ones are tricky to do in node, and I wanted to get a feel for how people are using the library before adding those ones.

I have been focusing my efforts on better interop with native node structures (buffer, typedarray, stream, etc..) with the hopes of reducing the overhead when bridging values.

If there are any specific features missing from node that are available in the other languages, feel free to open up a feature request!

[geertbrit]

Great to hear! Particularly interested in (some things might already be done) * a good arrow <-> parquet transform read/write api including cost-effective appends * just saw that time downsampling landed in python. Very interested in having that in Node * range pruning on arrow record-batches and the parquet equivalent * efficient wrapping output in async iterator * transparant querying over multiple partitioned files * zero-copy behavior, although I think Node just doesn’t have the primitives to do this * probably outside or scope of polars, but I’d love to see a robust Arrow Flight implementation server/client in node. One can dream right :)

…

Sent from my iPhone

On 15 Jan 2022, at 21:04, universalmind303 ***@***.***> wrote:  Some of the features like to_arrow don't really make sense to implement on the node side, as node doesn't really have a stable arrow library. Some of the other ones are tricky to do in node, and I wanted to get a feel for how people are using the library before adding those ones. I have been focusing my efforts on better interop with native node structures (buffer, typedarray, stream, etc..) with the hopes of reducing the overhead when bridging values. If there are any specific features missing from node that are available in the other languages, feel free to open up a feature request! — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.

[universalmind303]

• zero-copy behavior, although I think Node just doesn’t have the primitives to do this

this one is really tricky and ive been looking into it a lot., I have been looking at some ideas for this one. As of right now, only TypedArray, and Buffer support zero copy. The performance difference when using the zero copy is wild though. like microseconds vs seconds on really large datasets. In some scenarios, its actually faster to serialize the data, pass it over as a buffer, then deserialize it.

• efficient wrapping output in async iterator

Did you have any ideas on this one? What data structures would you expect to implement async iterator? I initially was thinking of only implementing it for the lazy api, but I'm open to ideas. Maybe create a sync + async implementation for all of the io operations?

[jorgecarleitao]

Are wasm's primitives compatible with std::vec? If yes, using them would enable zero copy with Polars

[universalmind303]

Are wasm's primitives compatible with std::vec? If yes, using them would enable zero copy with Polars

nodejs-polars does not use wasm. It uses the node api n-api for native bindings between the node engine and c structures.

But yes, most of the array like structures are compatible with a Vec<PolarsNumeric>, but once you introduce None values Vec<Option<PolarsNumeric>> it makes things much more difficult.

If the polars chunked arrays exposed a method to convert them to a [u8] instead of [T], it would be easier to pass them to node.

[geertbrit]

Did you have any ideas on this one? What data structures would you expect to implement async iterator? I initially was thinking of only implementing it for the lazy api, but I'm open to ideas. Maybe create a sync + async implementation for all of the io operations?

Primarily, iterating over table-rows and column-values. Seems trivial perhaps but having this work even if the table to iterate over is streamed in using record-batches, and/or is the result of a combination of multiple partitioned files on disk (parquet and/or arrow) it's not so trivial anymore I think.

To give an example, atm I have a couple of thousand arrow files representing timeseries of various financial instruments. 1 instrument per file. Sometimes I want to iterate over a combination of these instruments, so 'zip-merging' these various instruments where the end-result is also perfectly time-ordered. So multiple time-ordered files in -> 1 time-orded arrow table out (the sorting of course can be done much more efficiently then when input is not guaranteed time-ordered).

If in the above example the iterator could use the knowledge of min/max time per recordbatch, it could do the 'zip-merge' of these multiple instruments in streaming fashion -> outputting the merged record-batches as the output to async iterate over.

Excuse the braindump type of post but hope you get the gist

[ritchie46]

Did you have any ideas on this one? What data structures would you expect to implement async iterator? I initially was thinking of only implementing it for the lazy api, but I'm open to ideas. Maybe create a sync + async implementation for all of the io operations?

Primarily, iterating over table-rows and column-values. This becomes a bit more interesting I believe when iterator is backed by a stream of record batches, instead of needing to have the entire apache arrow table in-mem.

Polars lazy will have a streaming generator API. Maybe implementations can use async to hook into that API. But it should not be implemented in a frontend.

The idea is that all operations are implemnted in the rust lib and then bindings just expose.

[geertbrit]

Polars lazy will have a streaming generator API.

Bit as an aside, may I suggest carefully looking into the performance difference of implementing a custom async iterator instead of just using an async generator in Node? There's still (NodeJs 16.*) a pretty hefty penalty around the 'yield' construct in my experience. (~2x perf difference on larger datasets)

[universalmind303]

Primarily, iterating over table-rows and column-values. Seems trivial perhaps but having this work even if the table to iterate over is streamed in using record-batches, and/or is the result of a combination of multiple partitioned files on disk (parquet and/or arrow) it's not so trivial anymore I think.

Overall, i think my vision of the node api closely aligns with these thoughts on the usage of async iterators. The async iterators are really slow doing it in pure JS. I did some POC work on a pure JS async iterator, but it was slow, and complicated.

Until there is better support on the rust side, I dont think it is worth it to implement on the JS side.

Also, I created #2400 to track Parquet & IPC support.