Agreed, I find this to be a super productive environment, because you get all of vscode's IDE plus the niceties of Jupyter and IPython.
I wrote a small vscode extension that builds upon this to automatically infer code blocks via indentation, so that you don't have to select them manually: [0]
I develop Lonboard [0], a Python library for plotting large geospatial data. If you have small data (~max 30,000 coordinates), leaflet-based Python libraries like folium and ipyleaflet can be fine, but because Lonboard uses deck.gl for GPU-accelerated rendering, it's 30-50x faster than leaflet for large datasets [1].
It can read from HTTP urls, but you'd need to manage signing the URLs yourself. On the writing side, it currently writes to an ArrayBuffer, which then you could upload to a server or save on the user's machine.
Arrow JS is just ArrayBuffers underneath. You do want to amortize some operations to avoid unnecessary conversions. I.e. Arrow JS stores strings as UTF-8, but native JS strings are UTF-16 I believe.
Arrow is especially powerful across the WASM <--> JS boundary! In fact, I wrote a library to interpret Arrow from Wasm memory into JS without any copies [0]. (Motivating blog post [1])
That's why I'm working on the GeoParquet spec [0]! It gives you both compression-by-default and super fast reads and writes! So it's usually as small as gzipped CSV, if not smaller, while being faster to read and write than GeoPackage.
Try using `GeoDataFrame.to_parquet` and `GeoPandas.read_parquet`
Your issue is that you're using the default (old) binding to GDAL, based on Fiona [0].
You need to use pyogrio [1], its vectorized counterpart, instead. Make sure you use `engine="pyogrio"` when calling `to_file` [2]. Fiona does a loop in Python, while pyogrio is exclusively compiled. So pyogrio is usually about 10-15x faster than fiona. Soon, in pyogrio version 0.8, it will be another ~2-4x faster than pyogrio is now [3].
Sorry, this is not true _at all_ for geospatial data.
A quick benchmark [0] shows that saving to GeoPackage, FlatGeobuf, and GeoParquet are roughly 10x faster than saving to CSV. Additionally, the CSV is much larger than any other format.
nodejs-polars is node-specific and uses native FFI. polars can be compiled to Wasm but doesn't yet have a js API out of the box.
As for the fastest way to serialize data to Pandas data to the browser, you should use Parquet; it's the fastest to write on the Python side and read on the JS side, while also being compressed. See https://github.com/kylebarron/parquet-wasm (full disclosure, I wrote this)
pmtiles are only the counterpart to the "overviews" part of COG. pmtiles doesn't give you analysis-ready data. For that, look to FlatGeobuf or GeoParquet (once it gets a spatial index in v1.1 of the spec)
I'm working on bringing GeoRust algorithms [0] (which include boolean operations) to the web via WebAssembly. See this blog post [1] for an intro. There's also separate work on binding GEOS to Wasm [2], but I'm more excited about GeoArrow in the long term because you can interpret Wasm geometries in JS without any copies across the boundary [3].
The line visuals at the bottom are not using Mapbox for the data rendering. Rather they're using the open source Kepler.gl [0], (a user-friendly wrapping of the deck.gl library [1]). These can use Mapbox for the underlying basemap, but the data rendering is done separately.
(This is easy to tell if you look at the page source. The map at the bottom is an embed from a static HTML kepler.gl map [2]). Disclaimer: I've contributed to deck.gl
[0]: https://news.ycombinator.com/item?id=47482185