Love the idea at the end of the article about trying to see if this style of prompt injection could be used to get the bots to submit better quality, and actually useful PRs.
If that could be done, open source maintainers might be able to effectively get free labor to continue to support open source while members of the community pay for the tokens to get that work done.
Would be interested to see if such an experiment could work. If so, it turns from being prompt injection to just being better instructions for contributors, human or AI.
Fair points. I guess I was asking if this is a new, or fundamentally different problem from pre-AI. I could be over-simplifying -- what do you think?
This makes me think of risk assessment in general. There's a tradeoff between risk and reward. More risk might mean more _potential_, but it's more potential for both benefit and ruin.
> ... software can, in a very real sense, become self-improving.
This is referring to the software the agent is working on, not the agent.
> This creates a continuous feedback loop.
This is referring to the feedback loop of the agent effectively compressing learnings from a previous chat session into documentation it can use to more effectively bootstrap future sessions, or sub-agents. This isn't about altering the agent, but instead about creating a feedback loop between the agent and the software it's working on to improve the ability for the agent to take on the next task, or delegate a sub-task to a sub-agent.
> "... the type of self-improvement we’re talking about is far more pragmatic and much less dangerous."
This is a statement about the agent playing a part in maintaining not just the code, but other artifacts around the code. Not about the agent self-improving, nor the agent altering itself.
> ... which I imagine would be important for a military control AI
I think this is a common, but incorrect assumption. What military commanders want (and what CEOs want, and what users want), is control and assistance. They don't want a system that can't be turned off if it means losing control.
It's a mistake to assume that people want an immortal force. I haven't met anyone who wants that (okay, that's decidedly anecdotal), and I haven't seen anyone online say, "We want an all-powerful, immortal system that we cannot control." Who are the people asking for this?
> ... it will do whatever it can to prevent it being turned off.
This statement pre-supposes that there's an existing sense of self-will or self-preservation in the systems. Beyond LLMs creating scary-looking text, I don't see evidence that current systems have any sense of will or a survival instinct.
Agree, this is the point the article makes. I don't think the article claims that it's the agent that is directly improved or altered, but that through the process of the agent self-maintaining its environment, then using that improvement to bootstrap its future self or sub-agents, that the agent _performance_ is holistically better.
> ... if the docs act like a summary of current state, you can just read it at the start and update it at the end of a session
Yeah, exactly. The documentation is effectively a compressed version of the code, saving agent context for a good cross-section of (a) the big picture, and (b) the details needed to implement a given change to the system.
Think we're all on the same page here, but maybe framing it differently.
For sure this is a real example, but it's also largely a permissions issue where users are combining self-modifying capability with unlimited, effectively full admin access.
Outside of AI, the combination of "a given actor can make their own decisions, and they have unlimited permissions/access -- what could possibly go wrong?" very predictable bad things happen.
Whether the actor in this case is a bot of a human, the permissions are the problem, not the actor, IMO.
Interesting idea. While waiting for my verdict I was asked the same question (and provided the same answer) over and over. Was expecting a bunch of different questions, rather than the same questions, repeated.
Most of my code is in Ruby, but I've been watching languages like Go, Elixir, and ideas around the Actor model for years, trying to take the best ideas from these and applying them in my own system.
Inspired by all of these examples, but desiring a dead-simple solution for the Ruby applications I maintain - I created nobject.
It's not quite RPC. It's not quite the Actor model. It's not quite lightweight processes/channels. It's the ability to instantiate an object in one process but then push it to another process, yet be able to use that remote object like it's still local.
The example code in the repo's README will get you up-an-running in minutes.
Not sure why AI agents wouldn't also be connected to an ad platform eventually. Google does this currently.
While the author says that they are bypassing Google, that's not most people, and Google's results are front-loaded with AI answers, so Google results are already giving you specific answers, hallucinations and all. Not sure why average users would long-term switch to not-Google if Google can give them the amount of AI assistance that want or don't want.
Many of the core challenges that are fundamentally about human factors and collaboration haven't changed, such as the conflict between business requirements, engineering approaches, and the desire for no-code solutions.
Ex.: People were writing about agile, no-code, and the challenges of reducing cost and complexity before we had the terminology for it and before a whole industry of consultants existed to explain it.
Tom's writeup inspired me to dig a little more. I unfortunately (for me) discovered a very simple way to solve the game on the first guess every time. My fault for reading the game's code, but I just can't look at the puzzle the same way anymore.
I liked the first half of the essay - some interesting nuggets of some possible ideas, particularly the point about schisms forming rather than continued collective action once a group becomes large enough.
The last half sounds a little bit too much like the bias towards markets and profit motives and whatnot without question and without critique. There are plenty of dysfunctional for-profit orgs that don't allocate resources well, in spite of overwhelming feedback, from the market and elsewhere. The majority of for-profit orgs also have a sales team that is out there pounding the pavement trying to drive sales: it's not all, "Build it and they will come."
I don't buy that a successful company needs to be a mission either. That's very common thinking in the startup world, but not even close to representative of the whole.
I do prefer PostgreSQL if I have a choice, but from the practicality standpoint that many people are hitting on, I'm okay with various design decisions (i.e. take a look at some of the flags for MySQL's `sql-mode` option over the years) being phased out via the normal (warn -> deprecate -> throw error -> remove) lifecycle that things like this often go through in software. Once a technology gets wide adoption, no matter how "flawed" or not earlier versions were, you start to prioritize stability and reliability over "correctness" at some point. This leads to the understandably practical approach to many bugs in many enterprise systems where the team supporting a tech stack learns to work around the rough edges, and might even depend on certain "weird" functionality because it's simply more practical in both the short and long term than not doing it.
None of the above means that I don't see MySQL as flawed in some ways. I'm in a group of developers that I suspect make up a sizable portion of the MySQL community who didn't choose MySQL, but must support it, if for no other reason than because we see ourselves as professionals, and that's what professional do: make the employer's application work reliably.
For applications that have already survived past the point of finding product/market fit, a wholesale conversion of DBMS is rarely worth it, and conversions of this type are costly/risky even if it is worth it. I do understand many of the benefits (real and theoretical) of PostgreSQL, and if I'm around at the moment when a project's DBMS is being selected I'm going to recommend _not_ MySQL, but at some level I'm also paid to make the application that my employer is running on top of their DBMS work reliably ... and the fact is even among people who get PostgreSQL - who prefer it, would choose it if they could - many of us are also pragmatic enough not to pull the rug out from under a running application for "reasons".
It can function like this, yes. There's a flag for the `sql_mode` that enables/disables this functionality, and I believe zero dates are disallowed by default in MySQL 8.0+ IIRC.
If that could be done, open source maintainers might be able to effectively get free labor to continue to support open source while members of the community pay for the tokens to get that work done.
Would be interested to see if such an experiment could work. If so, it turns from being prompt injection to just being better instructions for contributors, human or AI.