Author here. I wrote this because "AI database" keeps getting used as one category when there are at least four architecturally different approaches, each with a different inference mechanism. Vector DBs do embedding similarity, ML-in-database is train-then-predict in SQL, LLM-augmented routes to an LLM per query, and predictive databases do Bayesian inference at query time with no model lifecycle.
Full disclosure: I'm the co-founder of Aito, which falls in the predictive database category. I tried to cover all four fairly, including where our approach falls short (latency scales with dataset size, smallest ecosystem of the four). Happy to discuss any of the categories or the taxonomy itself.
One thing that stood out while writing: the major cloud platforms (Databricks, BigQuery, Snowflake) now span multiple categories at once, which makes "AI database" as a single label pretty useless. Also, the BayesLite work from MIT CSAIL deserves way more attention than it gets. Their BQL syntax is really nice.
Hmm.. So, I'm correct, you are maintaining QA suites for 10k agents, their prompts, toolboxes and some scenarios.
That sounds like an absolutely massive scale to do QA over. Running the test suite must cost fortune and take ages.
As such, I don't think the data storage is that big problem. E.g. If you have 10-100 requests stored per agent, it's 100k-1M snapshots. Booktest normally stores the states in Git, but there is also some kind of DVC support. If you need to recreate all snapshots regularly, e.g. of you change some system wide properties often, that may became problem or not. Git and DVC can manage quite high scale. The Git PR reviews won't work though with e.g 1M changed files.
Our scale in mono repo was maybe 10k LLM snapshots in several hundred files, which worked technically. Recreating all evaluations was a bit slow (e.g 5-10 minutes) and merges often forced recreation of snapshots and re-reviews. This is ofc massively smaller scale than what you are having. We did use booktest for assistant, but it was just one assistant over dozens of use case flows.
I guess it could be somehow manageable, if you try to avoid test level manual review and only review some aggregared results in the tool, but I cannot really promise anything. It may be a worth a try.
One of the key techniques is snapshotting the LLM (or any HTTP) request. This means that if the inputs won't change, the LLM will not be called. This will also snapshot /cache LLM verifications steps.
This doesn't only saves costs, but it's main goal was to force determinism and save time. Limited changes may need only the new/changed tests to be rerun with LLM. CI typically don't have LLM API keys and only rerun against snapshots with zero costs and delays
All LLM operations tend to be notoriously slow, and at least on our side: we are often more interested of how our code interacts with the LLM. Having the LLM being fully snapshopshotted does iterating the code delightfully fast.
If you want do sampling, this can be implemented in the test code. Booktest is a bit like pytest in the sense, that the actual testing logic heavylifting is left for the the developer. Lot of LLM test suites are more opinionated, but also more intrusive in that sense
Booktest is build based of 2 decade-career in data science. It has been used to support RnD on numerous LLM, ML, NLP, information retrieval and also more traditional software engineering.
It was partly inspired by earlier examples (kudos for Ferenc), but especially real pains with how to assert ML QA with regression testing, transparency and iteration cycle speed.
So, in systems where correctness is fuzzy, evaluation is expensive, and changes have non-local effects, a failing test without diagnostics often raises more questions than it answers. This is a painful combination, if left unsolved.
Booktest is now on its 3rd or 4th iteration of the same idea, and as such it addresses most common needs and problems in this space.
It is a review-driven regression testing approach that captures system behavior as readable artifacts, so humans can see, review, and reason about regressions instead of fighting tooling.
This approach has been used in production for testing ML/NLP systems processing large volumes of data, and we’ve now open-sourced it.
I'm curious whether this matches others’ experience, and how people handle this today.
If I'm reading this correctly, a few themes PG touches here are:
1) loss of control when hiring a professional manager without intrusion to sub organization, because you rely on the manager provided information. If the manager is not straightforward, the sub organization may become a black box, where issues can go unseen
2) Lack of access to the frontline people and their understanding, which is opened by the Jobs like key people meeting across the org
3) Id also imagine that if you have a founder with deep domain knowledge, who has worked across all aspects of business, going fully hands off with the details, and replacing decision makers with more generic managers potentially from other industries, means that lot of expertise gets disconnected from the relevant decisions.
Ultimately the outcomes are all about the decisions and the decisions are all about understanding, which is all about the information. As such, it's not surprising if cutting the seasoned founder from both key mid level decisions and the firsthand information flows brings disadvantages.
I'd over all interpret the article to be about how hands on the founder should be about the different aspects of the business rather than the leadership, as implied elsewhere
Author here. The article seeks to answer the question "How to make ML / LMM development faster with higher quality?"
The long format article seeks address the pain points in current tooling (Jupyter Notebook, unittests) used to support intelligent application RnD. It proposes a new approach for tooling and development, which combines the benefits of notebooks (review driven process, caches) with benefits of unit testing (repeatiability, regression testing)
The tool has been succesfully used to support the developmvent topic models, analytics and GPT based functionality. Here's an example of how to create a simple test that both creates a snapshot for results that also snapshots the environment and (e.g. GPT) API calls so that the test interaction can be replayed e.g. in CI.
import booktest as bt
import os
import requests
import json
@bt.snapshot_env("HOST_NAME")
@bt.mock_missing_env({"API_KEY": "mock"})
@bt.snapshot_requests()
def test_requests_and_env(t: bt.TestCaseRun):
t.h1("request:")
host_name = os.environ["HOST_NAME"]
response = (
t.t(f"making post request to {host_name} in ").imsln(
lambda:
requests.post(
host_name,
json={
"message": "hello"
},
headers={
"X-Api-Key": os.environ["API_KEY"]
})))
t.h1("response:")
t.tln(json.dumps(response.json()["json"], indent=4))
Regarding this topic, I found the following paper on TNT (three nightmare traits) interesting and bringing some clarity to the issues/confusions you are mentioning:
The project feels transformative, because the ML part was implemented an RPA developer using predictive database queries. It wasn't that complex data science project, but it demonstrates that simple ML can be done by (RPA) developers with somewhat tight budgets and schedules
If you have questions, feel free to ask! We@Aito also love feedback!
I wonder if you are familiar with the predictive databases?
We at Aito.ai have gotten lot of interest from different RPA/no-node users and providers, and predictive database queries seem like a best intelligent automation.
It would be interesting to deeply integrate predictive functionality in your system, especially as it integrates a DB naturally. This could be used to offer predictive functionality from the plarform out of the box.
We feel that the project was transformative, because the ML project was done end-to-end by RPA developers without data science team help. One of the RPA developer comments about Aito was 'that it was easy to use', which is a big thing in the machine learning space and in the context of ML democratization.
I feel this is an interesting and a rather stake on the innovation / startup aspects.
I got reminded of this Rand's blog post after PG's blog post and its comments, and also after the why companies stop innovating article.
It seems that there is a contradiction between people who are very high on openness (the personality traits) and people, who are very low on openness and also high on risk aversion and orderliness (as pointed out in Rand's article).
I feel it's still less about good vd bad, but more about explorative vs optimizing & conforming tendencies as in Rands post. The same topic has been also discussed multiple times before, once with pioneers, settlers and town planners article.
While there are obviously risks in the conformism going wild / aggressive, in general it strikes me that Rands is right in the sense that you need both 'explorers' and 'optimizers' in any successful organization.
- yet it has excellent defaults for most situations
- it is full blown programming language,
- yet it's syntax is extremely specialized for the purpose and familiar at the same time (it's bash)
- it's basic assumptions/structure are extremely simple (timestamps, dependencies)
- yet it's extremely powerful and can take into account most situations.
It's be extremely Unix-style tool with very simple building blocks that combine together in extremely powerful way. Its fast to learn and easy to master and as such, the best kind of design.
Sometimes making a build with it is a small programming project, yet after using all kinds of built tools, I end up just wishing that I could use make instead.
This is a good point, and Aito's inference engine has lot of similarities with search engines. As an interesting, we can provide TF-IDF scored full text search functionality from the same indexes we are also using for inferences.
Still, while there are tons of similarities, I feel that the inferences engines are fundamentally different from search engines. The data structures are different, and I can see them diverging even more in the future. The algorithms and modes of operations are very different, even if there is some overlap.
From the user point of view, there is still a striking similarity between Aito and ElasticSearch. Both act now as auxiliary databases (all thought we would like to make Aito fully ACID with an SQL interface in future) and provide more search engine / inference engine-like functionality than full database functionality.