Thanks for this, especially the Hetzner ToS warning, thats genuinely useful and im going to review my own setup because of it. On hosting in general the0 doesnt care where it runs, its docker compose on a laptop or helm on any k8s cluster, so where you host is really an operator decision. Mine is a cheap k3s node cause my bots trade on the order of seconds to minutes. If you are latency sensitive you can host closer to your broker but to be honest I wouldnt even use the0 for actual HFT, that world is colocated bare metal in the exchange datacenter with kernel bypass NICs and hand tuned C++, and the fastest shops do order execution in FPGAs with verilog, a different game entirely.
On point 4 a small correction, theres actually no container build at deploy time. The runtime images per language are prebuilt, the CLI builds/vendors your bot locally and ships a versioned bundle, so an update is just bundle upload + container restart from an already cached image. On a warm node thats seconds, much closer to your git pull loop than a docker build cycle. The genuinely slow step is on the CLI side (dependency vendoring and the upload) not the runtime.
I dont know but I usually get most of the bugs ironed out in backtesting and paper trading runs. Hot-patching a bot mid-trade to me is more of a risk problem than a tooling problem. If a bot of mine misbehaves with positions open I stop it, flatten at the broker and fix it properly before the next session. At that cadence deploy speed stops mattering and versioned bundles mean I can roll back to the last known good bot instantly. But for tight iteration during development I fully agree nothing beats an in-place restart. I actually prototyped a the0 dev mode for a faster local loop and shelved the PR cause I wasnt happy with it, might revisit it to be honest.
Intresting question truth be told I went for tried and tested docker since its something industry uses without "issues" (I say that lightly cause they are).
I have to say that you can definately get further with a WASM VM today. pyodide can run libraries like numpy/pandas and WASIX (WebAssembly System Interface Extended that adds missing syscalls, dynamic linking, threads, sockets, libffi) can get a full cpython with native libs running but those are specific curated packages. The idea behind the0 was to bring the exact environment, whatever pip wheel or C binding you used backtesting locally runs unchanged in the container. With WASM anything outside the curated package list needs a wasm-specific build to exist at all as wasm is a different architecture straying from x86/ARM.
On language support I actually went through my 7 runtimes for this. Rust is the only one where wasm is a first-class target today. C/C++ has been clang-only for a decade, GCC's own wasm backend literally got approved by the steering committee last month and its still missing exceptions and debug info. Python as above runs as curated interpreter builds. Node is the funny one, wasm VMs dont run JS at all, you embed a JS engine compiled to wasm (Javy, StarlingMonkey) and lose all the Node APIs and native addons. The .NET 8 WASI workload is labelled experimental by Microsoft. GHC's wasm backend arrived in 9.6 and is still young. And for scala 3 the closest thing is Scala.js's new WasmGC backend, but thats Scala.js semantics, you lose the JVM library ecosystem which is the whole reason to write a bot in Scala. So 1 out of 7.
I did have a look at what would close the gap here. container2wasm is a possible hopeful solution but it does this by embedding a CPU emulator (Bochs) and booting linux inside the module so theres a emulation tax here I think I havent tested it. Theres also runwasi that solves the opposite problem, it would let docker/k8s schedule wasm modules as if they were containers. A glimmer of hope, however it wouldn't change the fact that we would have to compile to wasm first (back to fiddling with pyodide and WASIX). So we could either recompile the ecosystem or emulate the CPU.
That said for simple pure-logic bots a wasm runtime alongside the containers could be nice someday, the isolation and cold starts are genuinely better. Worth doing an expiriment and protoype of this. But for running whatever code in whatever language today, docker gets 7 out of 7 cause containers are the ISA everything was already built for, its the industry standard for production workloads for a reason.
Some more detail on the internals for those interested:
The toughest part was getting the Go-written runtime to be able to run arbitrary code in multiple languages. The other tough part was getting the logs and state of the bots to be queryable and streamable in real time. One fun feature is that you can create custom dashboards in React for each bot and stream the logs into them, there is a React sdk that helps you do this. It was a challenge getting this to work and involved treating each bot as a two phased bundle deployment and hydrating the frontend with the state of the bot and metrics from logs (I was inspired by the way Datadog does this).
The way bots send information such as logs and state is done through a daemon process that runs alongside the bot in its container (again inspired by Datadog). The daemon handles what I call the dirty work, persistence of logs and state to the Object Store (MinIO), behind the scenes; the bot just sends information via standard logging and persistence calls from the SDK. Its decoupled, the bot can run without the daemon, but then you lose the ability to query the state of the bot and stream logs.
Stack: the runtime is written in Go and uses NATS for messaging, PostgreSQL, MongoDB and MinIO for persistence. The frontend is React and the CLI is Go. It runs locally through docker or in k8s via helm charts. The0 can theoretically be scaled to run thousands of bots on a k8s cluster (this is based on your own code and the resources you have available).
In terms of whats next... honestly most of the major work is done, the0 is stable and I use it to run my own bots. I think most of the work is in extending the MCP server to support more features and making it easier for AI agents to interact with the bots. The next step is to get some honest feedback on what people want to see in the0 and how they would use it.
Man I have had a lot of gripes with debezium... the setup alone in a kubernetes cluster with strimzi kafka connect... And dealing with the kafka connect errors and fixing replication slots... ehhh was painful. Does this tool make it any well easier.
One problem I expirienced in production systems with staging doubles is that sometimes we do database restores from production onto staging and that makes the sinks and sources go haywire... does ingestr handle this well?
Is support for kubernetes/helm anywhere in the pipeline
The persistent identity files are interesting but there's a cost
problem. A recent paper (arxiv 2602.11988 https://arxiv.org/html/2602.11988v1) found context files
increase inference cost by 20%+ with marginal performance gains;
LLM-generated ones actually decreased success rates slightly.
Four identity files per agent injected every session feels like
monkey patching coherence with context. Context isn't memory, it's
just more tokens. The hard unsolved problem is cross-session
learning without the bloat.
Curious if you've measured the token overhead of the identity
files vs the performance gain they provide.
Interesting. I've always thought the real solution to hallucinations
lies in neurosymbolic AI.
LLMs purely rely on statistical pattern matching with no grounding
in formal logic or symbolic reasoning. You can throw more compute
and data at the problem but you can't guarantee correctness ever.
The neurosymbolic approach combines neural networks for what they're
good at (language, pattern recognition) with symbolic systems for
what they're good at (formal reasoning, provable correctness). The
hallucination can't form in the first place because the symbolic
component enforces correctness at the reasoning level.
The Sovereign Engine sounds more like execution constraints;
Intercepting outputs after the fact rather than grounding the
reasoning process itself. That's still valuable but it's a
different problem. A determined attacker finds the edge case your
constraints don't cover.
Genuinely curious how it works under the hood is there a symbolic
reasoning layer or is the "determinism" coming from the constraint
layer alone?
On point 4 a small correction, theres actually no container build at deploy time. The runtime images per language are prebuilt, the CLI builds/vendors your bot locally and ships a versioned bundle, so an update is just bundle upload + container restart from an already cached image. On a warm node thats seconds, much closer to your git pull loop than a docker build cycle. The genuinely slow step is on the CLI side (dependency vendoring and the upload) not the runtime.
I dont know but I usually get most of the bugs ironed out in backtesting and paper trading runs. Hot-patching a bot mid-trade to me is more of a risk problem than a tooling problem. If a bot of mine misbehaves with positions open I stop it, flatten at the broker and fix it properly before the next session. At that cadence deploy speed stops mattering and versioned bundles mean I can roll back to the last known good bot instantly. But for tight iteration during development I fully agree nothing beats an in-place restart. I actually prototyped a the0 dev mode for a faster local loop and shelved the PR cause I wasnt happy with it, might revisit it to be honest.