This blog post pints to the fact that you need information across scales to make really insightful products and software. You need to understand fundamental mechanisms, strengths, and risks of your software to know where to make bets next. You need to know about the “how” of your optimization system to know which customer asks to deny.
Using layers like the loops described here to abdicate your work is you decoupling from the joint market/engineering value you originally provided.
My dream is, instead of separating data storage, state machines, valid state constraints, and the logic that transitions between valid states, we can actually unify these into some kernel of app state. Honestly, Postgres already has a lot of these capabilities, but I don’t see an obvious story on the app or product level, providing provably correct sets of states that apps can transition between, and which they can automatically expose to clients in informative ways (this user can like this post, but not edit). It looks colored Petri net shaped to me, but I don’t yet see a simple app state paradigm in the same way that the database has obvious successful boundaries.
Given the atomization and layering of work, this has become much harder to truly judge. Ten years ago I was excited to join a customer feedback platform - what could be better than helping companies understand their customers and provide better services and products? You can probably see where this is going, but inevitably the tools were just used to better tweak product profitability and eliminate end customer surplus, to the customer company’s benefit. And they were used by the likes of draft kings et al along with the Starbucks and Nikes of the world. I hear people claim that, in capitalism, no one hands are clean, and I am inclined to agree.
It seems that we have forgotten how to distinguish between value and profit, and now celebrate the latter instead of the former. Currency enables ever broader and more niche markets, but the financialization of everything is the Faustian bargain; we gained niche hobbies but lost our souls?
I too am still investigating the space, but what's attractive to me about CPNs is that they can be both the specification and the implementation. How you describe the CPN in code matters, but I'm toying with a rust + SQL-macros version that makes describing invariants etc natural. My understanding is that for TLA+ you'd need to write the spec, and then write an implementation for it. This might be another path for "describe formally verifiable shape then agentic code it", but it smells to me a little like it wouldn't be doing as much work as it could. I think in this there's an opportunity to create a "state store" where the network topology and invariants ensure the consistency of the "marking" (e.g. state of the database here) and that its in a valid state based on the current network definition. You could say "well SQL databases already have check constraints", and we'd probably use those under the hood, but I am betting on the ergonomics of putting the constraints right next to the things/actions relevant to them.
It is tempting to think that we can delegate describing the mental model to AI, but it seems like all of this boils down to humans making bets, and it also seems like the fundamental bets engineers are making are about the formalisms that encode the product and make it valuable.
Notes are options. Sometimes options for taking action later, sometimes as reference for related action. The bet when writing a note is that its risk adjusted return for those avenues is higher than the opportunity cost.
I also have similar thoughts on turning writing into action and re-entrance, would be interested to hear your thoughts:
This has proven to work well for me, but I’m chafing with git and agentic coding abstractions and looking for a unifying concept. Agent of empires doesn’t feel quite right, but is in the right direction.
I’m curious why you don’t target an HDL, which seems like it should match very well to llm capabilities, and rely on existing layout solvers for describing the last physical layout step?
The answer I’ve seen is “just pass structs of functions around”, which is just one step more explicit than the implicit version we’re all use to, but honestly I kinda like it to free ourselves of all the ceremony around generics.
I think this is where it gets interesting. With partition dependency propagation, backfills are just “hey this range of partitions should exist”. Or, your “wants” partitions are probably still active, and you can just taint the existing partitions. This invalidates the existing partitions, so the wants trigger builds again, and existing consumers don’t see the tainted partitions as live. I think things actually get a lot simpler when you stop trying to reason about those data relationships manually!
I'm working on a partition-oriented declarative data build system. The inspiration comes from working with systems like Airflow and AWS step functions, where data orchestration is described explicitly, and the dependency relationships between input and produced data partitions is complex. Put simply, writing orchestration code for this case sucks - the goal of the project is to enable whole data platforms to be made up of jobs that declare their input and output partition deps, so that they can be automatically fulfilled, enabling kubernetes-like continuous reconciliation of desired partitions.
This means, instead of the answer to "how do we produce this output data" being "trigger and pray everything upstream is still working", we can answer with "the system was asked to produce this output data partition and its dependencies were automatically built for it". My hope is that this allows the interface with the system to instead be continuously telling it what partitions we want to exist, and letting it figure out the rest, instead of the byzantine DAGs that get built in airflow/etc.
This comes out of a big feeling that even more recent orchestrators like Prefect, Dagster, etc are still solving the wrong problem, and not internalizing the right complexity.
It sounds like you implicitly delegated many important design decisions to claude? In my experience it helps to first discuss architecture and core components of the problem with Claude, then either tell it what to do for the high leverage decorations, or provide it with the relevant motivating context to allow it to make the right decisions itself.
The hard part about documentation is that it requires you to have a component that can be comprehensibly and sufficiently documented. So much of the software written is seen as provisional, that even its authors think “well, we’ll document the v1”, not realizing that their prototype is just that.
Using layers like the loops described here to abdicate your work is you decoupling from the joint market/engineering value you originally provided.