We went through a very similar migration. Had a Next.js landing page and a separate TanStack Router SPA - consolidated both into a single Vite + TanStack Start app. Same experience with build times and the architecture mismatch: our app is heavily client-side with real-time state, and fighting Next.js's server-first assumptions wasn't worth it. TanStack Router's type-safe routing and file-based route generation have been great.
I cannot agree more. The battery not even lasting a day is what prevents me from using it. When I want to go for a run with it, it's always out of battery.
Have you looked into Spark? There are managed Spark options on AWS/GCP (for example Databricks). Spark lets you do exactly what you are saying.
Define minimum/maximum number of nodes, the machine capacity (RAM/CPU) and let Spark handle the scaling for you.
It gives you a Jupyter-like runtime to work on possibly massive datasets. Spark is perhaps too much for what you're looking for. Kubernetes could possibly be used with Airflow/DBT possibly, for example for ETL/ELT pipelines.