It's likely smart enough. It just needs to be told to do it and provided the ability to introspect it. How close could foundational models get to building this harness if explicitly prompted to?
We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.
Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required.
Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.
Any non-digital system you describe "as Code" is not a source of truth, but a Source of Hope. The code describes the intended state, something has to reconcile it with reality.
This is the same as having it in unstructured documents. Which means the auditing is still required funny enough.
So yes, this could be done. I'd love to see what run in the CI/CD for a change. When someone works on the wrong thing, or breaks compliance IRL, how do you backport it into this? "Alice is a software engineer, and created this SaaS account with her email when the company was founded. The admin email can not be changed and she has admin even though another role should control that"
Open models are going to win long-term. Anthropics' own research has to use OSS models [0]. China is demonstrating how quickly companies can iterate on open models, allowing smaller teams access and augmentation to the abilities of a model without paying the training cost.
My personal prediction is that the US foundational model makers will OSS something close to N-1 for the next 1-3 iterations. The CAPEX for the foundational model creation is too high to justify OSS for the current generation. Unless the US Gov steps up and starts subsidizing power, or Stargate does 10x what it is planned right now.
N-1 model value depreciates insanely fast. Making an OSS release of them and allowing specialized use cases and novel developments allows potential value to be captured and integrated into future model designs. It's medium risk, as you may lose market share. But also high potential value, as the shared discoveries could substantially increase the velocity of next-gen development.
There will be a plethora of small OSS models. Iteration on the OSS releases is going to be biased towards local development, creating more capable and specialized models that work on smaller and smaller devices. In an agentic future, every different agent in a domain may have its own model. Distilled and customized for its use case without significant cost.
Everyone is racing to AGI/SGI. The models along the way are to capture market share and use data for training and evaluations. Once someone hits AGI/SGI, the consumer market is nice to have, but the real value is in novel developments in science, engineering, and every other aspect of the world.
This will not be true for future frameworks, though it is likely true for current ones.
Future frameworks will be designed for AI and enablement. There will be a reversal in convention-over-configuration. Explicit referencing and configuration allow models to make fewer assumptions with less training.
All current models are trained on good and bad examples of existing frameworks. This is why asking an LLM to “code like John Carmack” produces better code.. Future frameworks can quickly build out example documentation and provide it within the framework for AI tools to reference directly.
There's an awkward reckoning in open source software about inclusivity and protecting the long-term security of projects coming.
Authors from several countries were already suspicious, such as Iran. Anyone from Russia and China or unknown places are all potential risks now.
Combined with recent inclusive ideologies, it’s gonna cause hard conversations. There will be a furthering in segmenting the Internet. Why fight contributing to an open source project when you could fork it and contribute with your allies?
For true enemies, there’s no risk to licensing or copyright issues. You can merge changes from the original, no problem. China even falls into this as there’s a limited ability for US companies to litigate within the country.
People think the Network State is hot, but at the end of the day, the Internet still has borders.
Hashi never sold me on the integration of their products, which was my primary issue with not selecting them. Each is independently useful, and there is no nudge to combine them for a 1+1=3 feature set.
Kubernetes was the chasm. Owning the computing platform is the core of utilizing Vault and integrating it.
The primary issue was that there was never a "One Click" way to create an environment using Vagarent, Packer, Nomad, Vault, Waypoint, and Boundry for a local developer-to-prod setup. Because of this, everyone built bespoke, and each component was independently debated and selected. They could have standardized a pipeline and allowed new companies to get off the ground quickly. Existing companies could still pick and choose their pieces. On both, you sell support contracts.
I hope they do well at IBM. Their cloud services' strategy is creating a holistic platform. So, there is still a chance Hashi products will get the integration they deserve.
Others' concerns are valid; the separation of concerns makes infra changes safer and easier to understand. Infra-tooling is slow because of the inherent risk of managing stateful services.
Mixing the infra and application logic is the obvious path forward, though. Just as most applications don't need more than Rails and a single Postgres, most apps don't need customized infra. Simplifying the 80% unlocks cycles for more creative work. Suppose you are successful enough to have to define custom infra at some point. That's good for you. That's a great problem to have.
AI is going to standardize architectures going forward. Providing a simple set of tools in the same code as the logic makes the planner easier for Copilots and reduces the context windows. Terraform and application code have implicit dependencies. Colocating them lets you define explicit dependencies that are more understandable.
I agree here. Formal models have to become easier to create through.
Today's ecosystem requires advanced knowledge of system design and still coding abilities.
To democratize model generation we need a more iterative and understandable way of defining intented execution. The problem is this devolves into just coding the damn thing pretty quickly.
It learns the same way a human does by learning patterns. It is not illegal to comprehend how to accomplish tasks by reading other people's source code.
The arguments against my point always assume perfect memory of everything this model is consumed. This is the plagiarism position. In reality, some patterns are more common than others and generate a code that looks nearly identical. I can’t speak for the reasons for this, as I’m not familiar with all of the methods. However, I don’t assume that is the current working state or intent of Codex.
My view is the copilot is not stealing open source code. It is learning from it just as a human reader would. People's disguste is based on the assimilation of what they thought was a human trait being machine derived from their work.
The copilot service backed by an army of actual humans wouldn’t be a story at all. Nor would anyone be angry, if an individual offered coding skills as a service, and had gone through the exercise of learning great amount to open source software to do so.
No open source license was written with this in mind. Because previously learning was something only humans could do and no one had issue with sharing that knowledge. Until licenses take machine learning use into account I see no problems with Copilot.
Source cannot be open if you restrict any viewing of it.
This is the only way to scale your impact as an IC.
The goal isn't to always have to do the repetitive reviews. It's to mentor people up so they do them for you eventually.
When you life those juniors up enough for them to mentor the next iteration you start the flywheel. You get to mentor and review people's work who do things at a higher level, and they get to learn to mentor juniors who were in their position. Everyone, even you, get to move up.
What were the limitations that required you to move all customers to a shared system at once?
Could you have selected some workspaces with lower traffic to migrate first? That would have decreased the load on the primary, potentially speeding up the replication, which is a flywheel to enable more customers to migrate to shards.
We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.