Thanks for publishing this. I quickly skimmed the paper, I saw the impressive linear scaling as you scaled to 16 nodes. How long did it take to train the various models in wall clock time?
Any thoughts how to detect direct DML on the state table? Presumably allowing direct DML on the state table without the same on the Outbox table would lead to silent data corruption or lost updates.
Python packaging gets a lot of criticism. It's a meme. The thing is, it's actually improved dramatically over the years and continues to improve.
The problems it solves are very complex if one looks a little below the surface. It is solving different problems to the ecosystems that it's often compared to: golang, rust, java, js.
However if the transitive-closure in the dependencies contains an sdist package anywhere in the tree, written in a way that wasn't designed to be imported on all platforms (this is rare, but does happen), then it isn't possible to resolve cross platform.
The prevalence of wheels does alleviate this to some extent. I think it is one of the early design decisions that pip can only install/resolve for the current platform and this will be very difficult to unwind. There are movements slowly in a direction that may one day make it possible to statically resolve python packages and remove dynamic dependency metadata or builds happening during dependency resolution.
There are many open and commercial options for hyperparameter optimization.
We didn't select Argo for hyperparameter optimization. The situation is that because we already used Argo for our ML training and data processing jobs, we wanted to see if it could be extended for hyperparameter optimization. It actually can!
As mentioned briefly in the article, we try to use tools with minimal overlap. It's not always possible of course and sometimes new tools are necessary.
Yes, YAML does seem to be the "Cloud Native" tax. For all but the simplest of Workflows, it is necessary to use a templating language or one of the Argo SDKs.
We use jsonnet for building and manipulating YAML at the moment and are reasonably happy with it.
This is a big deal. Congratulations to all involved.
In Software, complexity naturally increases over time and dependencies and interactions between components become impossible to reason about. Eventually this complexity causes the Software to collapse under its own weight.
Truly reproducible builds (such as NixOS and Nixpkgs) provides us with islands of "determinism" which can be taken as true invariants. This enables us to build more Systems and Software on top of deterministic foundations that can be reproduced by others.
This reproducibility also enables powerful things like decentralized / distributed trust. Different third-parties can build the same software and compare the results. If they differ, it could indicate one of the sources has been compromised. See Trustix https://github.com/tweag/trustix
I think there are indeed many challenges when evaluating Compound AI Systems (http://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems...)
But evals in complex systems are the best we have at the moment. It’s a “best-practice” just like all the forms of testing in the “test pyramid” (https://martinfowler.com/articles/practical-test-pyramid.htm...)
Nothing is a silver bullet. Just hard won, ideally automated, integrated quality and verification checks, built deep into the system and SDLC.