I spent a while mixing this up with PiKVM and was having trouble understanding how any of it would fit in with that project. Made a lot more sense once I got over that.
Would the licensing of Grist allow for building widgets/components to use it in dashboard tools like Streamlit, Jupyter, or Holoviz's Panel? Looks cool!
I've been wishing that JetBrains and Posit (nee RStudio) would adopt the devcontainer standard, but then I just read this article [1] about how MSFT is using VS Code's weird mix of open-source and proprietary components to fracture the market and ensure that any competitors who try to build off of VS Code are at a permanent disadvantage. Now I'm having second thoughts.
Devcontainers are a good example of their strategy. VS Code's source code is open source, but the devcontainers extension is not [2] and alternative vscode-in-the-browser providers are not able to use the devcontainers extension as they're not allowed to use the official VS Code extension marketplace.
I'm kind of obsessed with this space and have spent way too long setting up VS Code and RStudio IDEs on my homelab.
One area I struggle with when thinking about building container-based development environments is the best way to avoid mixing in your IDE specific dependencies with your project dependencies. I think some of the commercial tools do this, but I haven't gotten in set up well in my homelab.
I just came across two articles by a former GitPod employer who moved to Coder (these are the two main providers of open-source VS Code in the browser solutions). They're both really interesting.
The first is on how effectively Microsoft has used VS Code to fracture the market place to their advantage by strategically open-sourcing parts of VS Code while keeping many of its best features proprietary (Pylance, the python language server is a good example of this). https://ghuntley.com/fracture/
The second article is about why he thinks Coder's strategy is more promising than GitPod's prompting him to go work for them. It's not as detailed, but it touches on some of the parts of container-based development environments that I've found overly limiting. https://ghuntley.com/integrate/
Conda isn’t perfect but takes on a lot of problems that pip doesn’t deal with at all. Regular conda is really slow these days but you can use mamba instead or just configure conda to use the libmamba solver and it’s much nicer.
The folks at prefix.dev seem to be building some pretty cool drop in replacements for conda too.
If you use conda there are extensions that can help with this by automatically registering any available conda environments that include ipykernel in your Jupyter Lab environment.
nb_conda_kernels is pretty reliable but not actively maintained. Gator from the mamba folks is new and still a bit rough around the edges but looks like it will be pretty slick eventually.
It would be great if DuckDB handled this itself, but it seems like to be competitive with Athena on really massive datasets, you need to have a metadata layer that is used to figure out which parquet files in S3 DuckDB actually needs to query and then potentially run those in parallel. This seems to be the architecture of Puffin (which I haven't personally tried using yet).
One possible thing to look into would be whether this dataset is partitioned too much. My understanding is that the recommended file size for individual parquet files is 512MB to 1GB, whereas here they are 50MB. It would be interesting to see the impact of the partitioning strategy on these benchmarks.
I think the issues of privilege are worth considering, but I think they’re not the only dimension worth thinking about.
Consider this hypothetical. You’ve got two equally wealthy kids who go to the same private school, take all the same classes, and take the same SAT prep class.
Kid 1 has a perfect GPA and a 90th percentile SAT score. Kid 2 has perfect SAT score and a 90th percentile GPA. If there’s only one spot, which kid should a college admissions office pick?
I see that it looks like it will be self-hostable and that the repo is forthcoming but I had to scroll a lot to see that.
Looks interesting though!