> Does roc run tests in the same process as the compiler?
We do for tests of pure functions, yes.
> Your tests run in an entirely separate process from the compiler (and from cargo).
That's a great point and a relevant distinction, although Rust tests can run arbitrary I/O, so it's not like having them be in a separate process means memory corruption is harmless! :)
Also a good point! TIL that Rust and C++ use interpreters for const, although of course that wouldn't work for running tests. Then again, in the specific case of Rust I believe rustc only compiles the tests and then something else like Cargo executes them. Of course, as I noted elsewhere, if rustc emits machine code and then cargo immediately executes it, there's the same opportunity for end user memory being corrupted (due to miscompilation) as if rustc and cargo shared a code base.
By the way, I thought your question was totally reasonable - my first thought reading it was "Oh yeah I wasn't trying to say that writing bytes is unsafe, I definitely should have worded that differently."
> I think that for the hot binary patching / code reloading features, yes, that is going to need unsafe. But for regular old "producing an executable" compilation? Emitting machine code isn't the part that requires unsafe. The language's runtime is a more likely site to find unsafe.
Agreed! Emitting machine code is not unsafe, since it's just writing bytes down - it's only once you execute that machine code that there's potentially unsafety. The reason I said "a big part of the job" is that in practice a lot of compilers both emit machine code and execute it - but you're totally right that it's not a requirement that a compiler do both.
In addition to the examples you gave (hot binary patching/code reloading, language runtime, etc.), others would be things like evaluating userspace code at compile time (e.g. const fn in Rust, or in Roc any expression that could be hoisted to the top level), running tests and inspecting their output to decide what to display to the user, etc.
Those are the types of things I had in mind when I wrote that.
Roc defaults functions to being pure, and functions that can run side effects are inferred to have a different type by the compiler based on usage. By convention, their names should also end in `!` (e.g. `transform` for the name of a pure function and `transform!` if it does side effects), and the compiler warns you if you don't follow that convention.
I maintain a ~250K LoC Zig compiler code base [0]. We've been through several breaking Zig releases (although the code base was much smaller for most of that time; Writergate is the main one we've had to deal with since the code base crossed the 100K LoC mark).
The language and stdlib changing hasn't been a major pain point in at least a year or two. There was some upgrade a couple of years ago that took us awhile to land (I think it might have been 0.12 -> 0.13 but I could be misremembering the exact version) but it's been smooth sailing for a long time now.
These days I'd put breaking releases in the "minor nuisance" category, and when people ask what I've liked and disliked about using Zig I rarely even remember to bring it up.
The permission model is almost identical to Roc's - https://www.roc-lang.org/platforms - although Roc isn't designed for "Syntax only an AI could love" (among many other differences between the two languages - but still, there are very few languages taking this approach to permissions).
If a stranger asks me, "Should I walk or drive to this car wash?" then I assume they're asking in good faith and both options are reasonable for their situation. So it's a safe assumption that they're not going there to get their car washed. Maybe they're starting work there tomorrow, for example, and don't know how pedestrian-friendly the route is.
Is the goal behind evaluating models this way to incentivize training them to assume we're bad-faith tricksters even when asking benign questions like how best to traverse a particular 100m? I can't imagine why it would be desirable to optimize for that outcome.
(I'm not saying that's your goal personally - I mean the goal behind the test itself, which I'd heard of before this thread. Seems like a bad test.)
> Whether functions are introduced with an explicit keyword; how return types are marked; whether semicolons can be omitted at end of line; how types are named (and the overall system for describing algebraic types).
Yes, these are all examples of things I always thought were generally considered small enough differences that nobody who was okay with how C++ or Rust or Swift did them would find the way one of the others did it a deal-breaker.
> ...Which is really all to say: different people are focused on different details, at different levels.
Wow, this is one of the most surprising comments I've ever read on HN!
Personally, I bucket C++ and Rust and Swift under "basically the same syntax." When I think about major syntax differences, I'm thinking about things like Python's significant indentation, Ruby's `do` and `end` instead of curly braces, Haskell's whitespace-based function calls, Lisp's paren placement, APL's symbols, etc.
Before today I would have assumed that anyone who was fine with C++ or Rust or Swift syntax would be fine with the other two, but TIL this point exists in the preference space!
Just based on timing, I am almost 100% sure whatever code is responsible was handwritten before anyone working on Windows was using LLMs...but anyway, thank you for the bug report - I'll pass it along!
Definitely more than a faster keyboard (e.g. I also ask the model to track down the source of a bug, or questions about the state of the code base after others have changed it, bounce architectural ideas off the model, research, etc.) but also definitely not a replacement for thinking or programming expertise.
> Do you think it can replace you basically one-shotting features/bugs in Zed?
Nobody is one-shotting anything nontrivial in Zed's code base, with Opus 4.5 or any other model.
What about a future model? Literally nobody knows. Forecasts about AI capabilities have had horrendously low accuracy in both directions - e.g. most people underestimated what LLMs would be capable of today, and almost everyone who thought AI would at least be where it is today...instead overestimated and predicted we'd have AGI or even superintelligence by now. I see zero signs of that forecasting accuracy improving. In aggregate, we are atrocious at it.
The only safe bet is that hardware will be faster and cheaper (because the most reliable trend in the history of computing has been that hardware gets faster and cheaper), which will naturally affect the software running on it.
> And also - doesn’t that make Zed (and other editors) pointless?
It means there's now demand for supporting use cases that didn't exist until recently, which comes with the territory of building a product for technologists! :)
Anecdotally, we use Opus 4.5 constantly on Zed's code base, which is almost a million lines of Rust code and has over 150K active users, and we use it for basically every task you can think of - new features, bug fixes, refactors, prototypes, you name it. The code base is a complex native GUI with no Web tech anywhere in it.
I'm not talking about "write this function" but rather like implementing the whole feature by writing only English to the agent, over the course of numerous back-and-forth interactions and exhausting multiple 200K-token context windows.
For me personally, definitely at least 99% all of the Rust code I've committed at work since Opus 4.5 came out has been from an agent running that model. I'm reading lots of Rust code (that Opus generated) but I'm essentially no longer writing any of it. If dot-autocomplete (and LLM autocomplete) disappeared from IDE existence, I would not notice.
My understanding is that the reasoning behind all this is:
- In 1985 there were a ton of different hardware floating-point implementations with incompatible instructions, making it a nightmare to write floating-point code once that worked on multiple machines
- To address the compatibility problem, IEEE came up with a hardware standard that could do error handling using only CPU registers (no software, since it's a hardware standard)
- With that design constraint, they (reasonably imo) chose to handle errors by making them "poisonous" - once you have a NaN, all operations on it fail, including equality, so the error state propagates rather than potentially accidentally "un-erroring" if you do another operation, leading you into undefined behavior territory
- The standard solved the problem when hardware manufacturers adopted it
- The upstream consequence on software is that if your programming language does anything other than these exact floating-point semantics, the cost is losing hardware acceleration, which makes your floating-point operations way slower