---
Just to be clear, I'm not saying they don't make mistakes. In fact I constantly scream into the void with the sheer amount of absolute stupidity of those models, however I would never say, using them for what I use, that they can only be used for simple and small use cases.
"rewrite 50% of what it generates"
See, I'll not claim they write good code. But have you considered maybe your standards are a little bit too high for the tool? I made like 15 tools already using AI for my use, most of them I barely needed to touch in the code. The code is not great, no, but it's not useless and that's what matter for me. You try and iteratively ask for the AI to do things.
If you want to ensure a higher degree of quality you can ask for tests and use techniques such as mutation testing to increase coverage, etc.
If you expect the same level of quality as you would write by hand, then you probably is better off... not using those tools.
I mean if I was rewriting 50% of the generations I get I would not be using them at all.
The second test scores 60%, the first was way higher.
And I specifically said ""unless you are saying "I could not grasp it immediately but later I was able to after understanding the point" I think you and your friends should see a neurologist"", to which this person did not responded.
I saw the tests, solved some, I suspect the variability here is more a question of methodology than an inherent problem for those people. I also never stated that my point depended on those people scoring 100% specifically on the tests, even if it is in fact extremely easy (and it is, the objective of this test is to literally make tests that most humans could easily beat but that would be hard for an AI) variability will still exist and people with different perceptions would skew the results, this is expected. "Significantly misrepresenting the numbers" is also a stretch, I only mentioned the numbers ONE time in my point, most of it was about that inherent nature (or at least, the intended nature) of the tests.
So on the edge, if he was not able to understand them at all, and this was not just a problem of grasping the problem, my point was that this would possibly indicate a neurological problem, or developmental, due to the nature of them. It's not a question of "you need to get all of them right", his point was that he was unable to understand them at all, that it confused them to an understanding level.
Your view is incomplete. It did lowered lifespans, but also increased them MUCH MORE than before. It was not "oh, it got worse, now it is recovering". It was "it got worse, now it is much better than before".
The amount of infant deaths it lowered, for example, is massive.
And it is true, those people did not got an iPhone and died, but this is also you saying this for them. You don't know all the specifics of history or all their motivations, the industrial revolution had a bloody story, but it's origins were also organic, it also had aspects of improvement. The world population grew almost 10x.
I don't think we are in a position to judge those past events to the lens you are posing.
Did you wrote it from scratch to compare?
There's an old motto devs use all the time, you know. Measure first, don't guess.
How do you know it would not have took you the same time or less to write the program if it was you? Or if for example, if you were using the AI to write the boilerplate for you while you focused on the core of coding? Or using it as a tab completor assistant instead of it being an agent?
Saying it saved you time is easy when you don't have the data to back it up, it's easier than thinking that maybe, maybe this was not that good of an use of your time.
700 lines of code is 2 weeks of work for a good developer?
My friend, I wrote 350 lines of executable code (excluding boilerplate) in a morning (4AM to like 9AM, maybe a bit more) to make a test with voxel octrees like yesterday.
There's no reason it would take "2 weeks of work for a good developer" to write 700. What takes times in those projects is the research, if you already have this fresh in your head it should not take more than 3 days to make something very simple but reasonable, and a week at max to make something good (not perfect, but good).
Maybe because (and I'm quoting that article) it is still lacking in what it should have that you managed to accomplish this task in "few days" instead of "a few weeks, maybe months".
Maybe the bottleneck was not your typing speed, but the [specific knowledge] to build that system. Because if you know something well enough, you can build it way faster, like rebuilding something from scratch, you will be faster as you already know the paths. In which case, my question would be: would not be writing this as fast, or maybe at least more secure and reasonable, if you had the complete knowledge of the system first.
Because contrary to LLMs, humans can actually improve and learn when they do things, and they don't whey they don't do things. Not knowing the code to the full extent is worth the time "gained" by using the LLM to write it?
I think it's very hard to estimate those other aspects of the thing.
Novelness is not a characteristic of interpolation, tho, it's about extrapolation. If you have plenty of clients and plenty of related stuff to the provider side, even if on on auth, then it could be considerably trivial for the LLM to interpolate on that field.
quoting my own previous response:
> Also, I mentioned mammals because those problems are of an order that mammals and even other animals would need to solve in reality for a diversity of cases. I'm not saying that they would literally be able to take the test and solve it, nor to understand this is a test, but that they would need to solve problems of similar nature in reality. Naturally this point has it's own limits, but it's not easily discarded as you tried to do.
---
> Have you seen them or shown them to average people? I’m sure the people who write them understand them but if you show these problems to average people in the street they are completely clueless.
I can show them to people on my family, I'll do it today and come back with the answer, it's the best way of testing that out.
No, I think I saw the graphs on someone's channel, but maybe I misinterpreted the results. But to be fair, my point never depended on 100% of the participants being right 100% of the questions, there are innumerous factors that could affect your performance on those tests, including the pressure. The AI also had access to lenient conventions, so it should be "fair" in this sense.
Either way, there's something fishy about this presentation, it says:
"ARC-AGI-1 WAS EASILY BRUTE-FORCIBLE", but when o3 initially "solved" most of it the co-founder or ARC-PRIZE said:
"Despite the significant cost per task, these numbers aren't just the result of applying brute force compute to the benchmark. OpenAI's new o3 model represents a significant leap forward in AI's ability to adapt to novel tasks. This is not merely incremental improvement, but a genuine breakthrough, marking a qualitative shift in AI capabilities compared to the prior limitations of LLMs. o3 is a system capable of adapting to tasks it has never encountered before, arguably approaching human-level performance in the ARC-AGI domain.", he was saying confidently that it would not be a result of brute-forcing the problems.
And it was not the first time,
"ARC-AGI-1 consists of 800 puzzle-like tasks, designed as grid-based visual reasoning problems. These tasks, trivial for humans but challenging for machines, typically provide only a small number of example input-output pairs (usually around three). This requires the test taker (human or AI) to deduce underlying rules through abstraction, inference, and prior knowledge rather than brute-force or extensive training."
Now they are saying ARC-AGI-2 is not bruteforcible, what is happening there? They didn't provided any reasoning for why one was bruteforcible and the other not, nor how they are so sure about that.
They "recognized" that it could be brute-forced before, but in a way less expressive manner, by explicitly stating it would need "unlimited resources and time" to solve. And they are using the non-bruteforceability in this presentation as a point for it.
---
Also, I mentioned mammals because those problems are of an order that mammals and even other animals would need to solve in reality for a diversity of cases. I'm not saying that they would literally be able to take the test and solve it, nor to understand this is a test, but that they would need to solve problems of similar nature in reality. Naturally this point has it's own limits, but it's not easily discarded as you tried to do.
You read at the same speed line-by-line your code when you are in your git client?
You are doing something wrong. I go line-by-line through my code like 7x faster than I would do it for someone's else code, because I know what I wrote, my own intentions, my flow of coding and all of those details. I can just look at it en passant, while with AI code I need to carefully review every single detail and the connection between them to approve it.
> It's not just time spent typing. Figuring out what needs to be typed can be both draining and time consuming. It's often (but not always) much easier to review someone else's solution to the problem than it is to solve it from scratch on your own.
This is EXTREMELY false. When you write the code you [remember] it, it's fresh in your head, you [know] what it is doing and exactly what it's supposed to do. This is why debugging a codebase you didn't wrote is harder than one you wrote, if a bug happens you know exactly the spots it could be happening at and you can easily go and check them.
Now you are just being silly with your comparisons. There is no analogy between those things:
* the difference between handwritting a book and typing is the extreme pain you would feel in your hands versus being able to write more in the same time without it
* the difference between typing and using your voice could be of a similar magnitude for someone with problems in their hands
* the difference between any of those writing methods and using an AI to do it for you, is that you are abstracting YOURSELF from the equation, not the method of writing. It's not analogous, not even from a mountain of distance far. You are not less "bottlenecked" because you don't need to write the thing yourself, you are just not producing it at all, it's more analogous to you guiding the hands of another person with vague instructions, using of their own expressivity to make your book for you, then claiming it was you who wrote it. It's not a bottleneck question, it was never a bottleneck question, and this is the case because code IS the writing, it IS the problem solving area where you need to put your mind to work, not writing a prompt, but coding in a specific and well defined formal syntax.
Amazing, because I realized I procrastinate MORE when using LLM to write code which I know I could write. And not only that, I feel I'm losing the ability to do the coding myself and solve the problems myself when delegating this to the AI.
Which is why no one should base their own decisions for life, like using or not using an LLM, on some random story from the internet.
I have a very simple question with like, 5 lines at best, that basically no model, neither reasoning or simpler could grasp. For obvious reasons I'm not disclosing it here (because I fear data contamination in the long run), but it basically breaks the "reasoning" of those things.
Unfortunately, I still can't try the o3-pro because the API version is not easily available, and I'm certainly not willing to pay for it in pro mode, but when it comes to the plus version (if it comes) I'll try. To this date, because of this question (and similar ones) I stand very unimpressed with those models, the marketing is a thousand times larger than reality, and I suspect people in general are surprisingly less capable of detecting intelligence than they think.
The normal o3 also managed to break 3 isolated installations of linux I was trying it with, a few days ago. The task was very simple, simply setup ubuntu with btrfs, timeshift and grub-btrfs and it managed to fail every single time (even when searching the web), so it was not impressive either.
"most people I show them too have issues understanding them, and in fact I had issues understanding them"
???
those benchmarks are so extremely simple they have basically 100% human approval rates, unless you are saying "I could not grasp it immediately but later I was able to after understanding the point" I think you and your friends should see a neurologist. And I'm not mocking you, I mean seriously, those are tasks extremely basic for any human brain and even some other mammals to do.
I don't think two wrongs make a right, tho. Looking down at formal education is not the same as embracing anti-intellectualism. And while I admit the software industry is absolutely full of bullshit anti-intellectualism, I also don't believe formal education is or should be the standard for the education, formalism is not automatically better than alternative means of learning.
The problem of anti-intellectualism in SE is just the consequence of the field being more "democratized". Or, to put in other words, the mass is stupid and the mass-man is stupidier and primitive.
https://github.com/mii-nipah/voxcpm-rs
--- Just to be clear, I'm not saying they don't make mistakes. In fact I constantly scream into the void with the sheer amount of absolute stupidity of those models, however I would never say, using them for what I use, that they can only be used for simple and small use cases.