What you can do if you have gpu friendly format is you send compressed data over PCI-E and then decompress on the gpu. Thus your overall throughput will increase since PCI-E bandwidth is the limiting factor of the overall system.
The default writer will decompress the values, however, right now you can implement your own write strategy that will avoid doing it. We plan on adding that as an option since it’s quite common.
You might be interested in https://github.com/spiraldb/fastlanes. It gives you fast bit packed vectors. It needs padding to 1024 elements though for the performance.
You don’t have to wait for the doors to close to be able to scan your ticket in London Underground. The gate will stay open and let you through. It’s a little bit awkward since you have to approach as you scan your ticket leading to your hand lagging behind
The thing we are trying to achieve is to be able to experiment and tune the way data is groupped on disk. Parquet has one way of laying data out, csv is another (though it's a text format so a bit moot), ORC is another, Lance has yet another different method. The file format itself stores how it's physically laid out on disk so you can tune and tweak physical layouts to match the specific storage needs of your system (this is the toolkit part where you can take vortex and use it to implement your own file format). Having said that we will have an implementation of file format that follows particular layout.
Fulcrum | Software Engineer | London or New York | ONSITE | Full-Time
Fulcrum is building next generation storage platform for diverse data of the future. We believe users will need to process non tabular and tabular data together and we need to develop new methods to support them.
We develop Vortex (our core storage primitive) in the open https://github.com/spiraldb/vortex and currently are looking to hire more people to our 5 person team to help build our product.