That’s a fair point. I wouldn’t want to take machine god’s proclamations on faith either. I’d prefer it if the knowledge was always within our grasp. There is also a possible middle ground where we don’t understand the questions or the answers but we still benefit from the effects of the application of that knowledge. “Sufficiently advanced technology is indistinguishable from magic” and all that. Hopefully we can still judge the results based on the effects. Rejecting a call to genocide should be easy enough, but on the other hand the Native Americans couldn’t foresee what the smallpox blankets would do to them. Working from a position where you are the weaker side of a knowledge gap is a scary thought. It’s a rational fear and I think a lot of people will end up on that side of the divide if AI continues to advance.
My instinct is to agree with you. I believe that the drive to a deeper understanding of the problems is what helps us unlock new areas of study, and find opportunities to transfer techniques or bridge otherwise unconnected domains.
But let’s consider a hypothetical: what if an intuitive understanding of the true “boundaries” of mathematics (if such things exist) is beyond the capabilities of a human mind? If there truly is no way to simplify some proofs down from 200,000 line incomprehensible gibberish to something you could teach to a high schooler or undergraduate or even a PhD. Is the proof still worthless? Sure, at the moment, it might be. Finding such a proof and understanding the implications of it are different skills, the latter of which AI almost certainly does not possess at the moment. But there may come a time where the AI can view the bigger picture and make the leaps you described (say, an eka-Calculus from an eka-unit-circle). These leaps may be as unintelligible to us as the proof in OP is.
I guess the question is: assuming that we can’t make the proof beautiful enough to spark deeper human understanding, do we still want it if it sparks deeper AI understanding?
Personally I would hate to live in a universe where the boundaries of science are beyond intuitive human understanding, but I think it’s almost certainly the case. The idea that the rules are all within our grasp reeks of anthropocentrism to me. I would love for the universe to prove me wrong though. It’d be a pleasant, hilarious coincidence if they do fit within the boundaries of our understanding.
Been kicking around a similar idea in the back of my mind from the first moment "functional core / imperative shell" [0] and "sans-IO" [1] infected my brain. I've been chasing that high ever since.
Unfortunately, whenever I try to apply this pattern 100%, I hit all kinds of walls: language isn't expressive enough to support what I want; the amount of wiring/glue to support it becomes a burden; the resulting code is spaghetti because the "declaration of intent" lives too far from "implementation of the intent"; "oops I invented my Nth leaky DSL"; and so on and so on. Part of the problem is certainly my own capabilities as a developer as well.
I can't help but fantasize about the platonic ideal of a "perfect" system where all that nasty evil I/O is banished to the Shadow Realm and I can frolic in the Fields of Idempotency and Reproduciblity -- one of these days I'll bite the bullet and try Haskell.
Nowadays I aim for 80% "perfection", and only in the areas where it matters. In addition, instead of effects I rely more on "reduce complexity as much as possible", which is (frustratingly) much harder to put into practice than "use X library/pattern to solve all problems". Though if I can model the system as a state machine and proptest it [2], that usually gets me where I want to be.
Though my soul feels like I just woke up from a dream where I was perfectly content, and now I'm back in the real world with all of its imperfections [3].
---
As for your specific project, it heavily reminds me of the Crux [4] model, which is itself inspired by Elm [5]. Also Flawless [6]. I wish you the best of luck with it.
You're not wrong, but if I'm talking to a Chinese Room, I'm still going to use pronouns and all sorts of meatbag-specific language. It doesn't matter if there really is a real person on the other end or not -- it's easier for me to just default to the assumption that there is. Monkey brain gonna anthropomorphize.
On the other hand if I try to talk to Facebook, all he says in response is "200 OK".
> music is basically a greenfield for AI wherever you look
AIN'T THAT THE TRUTH.
My girlfriend is studying musicology and she has some physical disabilities that make it difficult for her to write things down sometimes. So I try to help her by writing some AI-powered TTS/OCR/etc. apps here and there. It becomes painfully obvious that music was never considered an important part of any AI training dataset, anywhere.
These days, I'm pleasantly surprised by how well Opus 4.8 understands/explains music theory (as you said). But ask him to transcribe/OCR/OMR some sheet music and he'll confidently give you the MusicXML/Lilypond equivalent of "2 + 2 = horse".
I really hope this ignored area will be swept up with the rest of the rising AI wave, but it's still criminally undervalued.
My gut reaction is "I wish they'd just get to the point". Tbf some people would probably react the same way to my issue thread on the bug that I hit [0].
This thread will become a typical "haha slop company made slop" but I've been bitten by a bug exactly like this before in a (pre-AI, artisan) OSS project. The maintainer there didn't properly account for DST when calculating last backup time, so the app started and never stopped writing/re-writing backups continuously.
Perhaps the framing shouldn't be "haha slop" but rather why doesn't the AI write better quality software than we do? To which the answer is obvious IMO -- even emergent properties can't elevate AI intelligence too far above the training dataset. So how do we get to superintelligent (or at least "not-wreck-your-NVMe-endurance-telligent") AI, if we, as a whole, are not smart enough ourselves?
Judge not the slop-bot, lest ye be judged yourself, engineer.
GP wasn’t asking how you gauge their tone. They were asking how would you classify the tone of the comment they were replying to. The implication being “I wasn’t the first with a condescending tone”.
Eventually even a system like that can be gamed, similarly to how Leetcode-maxxing and the like sprung up in response to typical SV interview questions. Studying for the job becomes studying for the test becomes studying for the pre-test test.
This year, some members of the Rust Programming Language Community Server on Discord set out to solve AoC in under 1ms. I'm pleased to announce that through the use of LUTs, SIMD, more-than-questionable `unsafe`, assertions, LLVM intrinsics, and even some inline ASM that goal has been reached (for real this time)!
*As of today, our solutions are able to solve all 49 problems in <1ms!*
I have obtained consent from all the top participants to post their solutions to a shared repo, for the community to review and learn from! *All solutions are now available at the linked GitHub repo!*
Our solutions have a total runtime of *988936ns*!
# Context/Caveats
- All submissions were run on the same hardware (Ryzen 5950X) to ensure consistency, with the same compiler flags and features available. This was on rustc nightly (updated throughout the course of the contest), and with CPU speed capped at 3400 MHz with boost clock disabled.
- AVX-512 was not available on the machine so none (?) of the solutions utilize that particular set of accelerated instructions, but there is plenty of other SIMD in use.
- All submissions were run against the same inputs to ensure consistency.
- Caching anything that has been fed with input was not allowed to prevent cheating and/or trivial solutions like `Map<Input, Output>`.
- For the same reason, inputs were not directly available to the participants, and were not provided at compile-time.
- Participants were allowed to use compile-time tricks in their answers. Due to limitations in the benchmark bot, the runtime of these optimizations could not be measured. This was considered acceptable as the compiled binaries were expected to otherwise work correctly for arbitrary inputs. This means that participants are allowed to use look-up tables (LUTs) in their answers, but those LUTs are expected to work for arbitrary inputs, not just specific ones.
- I/O is trivial, and was thus not measured as part of the benchmark. That is, participants were provided with an `&str` or `&[u8]` input (their choice) and expected to provide an `impl Display` as part of their result. Therefore, input parsing was measured.
If you are interested, join us in #advent-of-code-2024 on the Discord server for further discussion :)
# Further Reading
If you would like a more in-depth explanation of some of the optimization techniques used, I highly recommend you check out this article by ameo [0] (one of our participants). It covers the process they used to optimize their solution for Day 9 Part 2, and how they got it to the top of our leaderboard. The article provides incredible information on the process of both high-level and micro optimization.
# Credits:
- Thank you to the members of the `Rust Programming Language Community` and `Serenity-rs` Discord servers and everyone else who participated in the challenge!
- Thank you to Eric Wastl for hosting AoC every year!
- Thank you to Noxim [1] for writing the original version of our benchmark bot [2].
- Extra special thank you to yuyuko [3], bend-n [4], and giooschi [5] for their help in maintaining and improving our benchmark bot.
Incredibly detailed and fun read! I love love love seeing this kind of performance analysis. IMO one of the most fun and creative areas of programming is finding new avenues to squeeze out every extra nanosecond.
Reminds me of one of my favourite video essays -- "Everything is a Remix" [0]. The video and this article cover the same ideas albeit with different examples. Which is funny on a meta level -- the article could be called a remix of the video.
The video (if I recall correctly) goes a bit further, attacking patents/IP law as anti-creative.
Is this meaningfully different from Conductor (which they archived a while back)? Browsing through the code I see quite a few similarities. Plus the use of JSON as the workflow definition language.
Hydra not populating with cross compile builds is the bane of my existence.
I'm using `clang` from `pkgs.pkgsCross.musl64.llvmPackages_latest.stdenv` to cross-compile Rust binaries from ARM macos to `x86_64-unknown-linux-musl`. It _works_, but every time I update my `flake.nix` it rebuilds *the entire LLVM toolchain*. On an M2 air, that takes something like 4 hours. It's incredibly frustrating and makes me wary of updating my dependencies or my flake file.
The alternative is to switch to dockerized builds but:
1) That adds a fairly heavyweight requirement to the build process
2) All the headache of writing dockerfiles with careful cache layering
Sure! I mentioned a few orthogonal concepts that go well together, and each of the following examples has a different combination that they employ:
- the company that developed Madsim (RisingWave) [0] [1] is tries hardest to eliminate non-determinism with the broadest scope (stubbing out syscalls, etc.)
- sled [2] itself has an interesting combo of deterministic tests combined with quickcheck+failpoints test case auto-discovery
- Dropbox [3] uses a similar approach but they talk about it a bit more abstractly.
Sans-IO is more documented in Python [4], but str0m [5] and quinn-proto [6] are the best examples in Rust I’m aware of. Note that sans-IO is orthogonal to deterministic test frameworks, but it composes well with them.
With the disclaimer that anything I comment on this site is my opinion alone, and does not reflect the company I work at —— I do work at a rust shop that has utilized these techniques on some projects.
TigerBeetle is an amazing example and I’ve looked at it before! They are really the best example of this approach outside of FoundationDB I think.
I've been super interested in this field since finding out about it from the `sled` simulation guide [0] (which outlines how FoundationDB does what they do).
Currently bringing a similar kind of testing in to our workplace by writing our services to run on top of `madsim` [1]. This lets us continue writing async/await-style services in tokio but then (in tests) replace them with a deterministic executor that patches all sources of non-determinism (including dependencies that call out to the OS). It's pretty seamless.
The author of this article isn't joking when they say that the startup cost of this effort is monumental. Dealing with every possible source of non-determinism, re-writing services to be testable/sans-IO [2], etc. takes a lot of engineering effort.
Once the system is in place though, it's hard to describe just how confident you feel in your code. Combined with tools like quickcheck [3], you can test hundreds of thousands of subtle failure cases in I/O, event ordering, timeouts, dropped packets, filesystem failures, etc.
This kind of testing is an incredibly powerful tool to have in your toolbelt, if you have the patience and fortitude to invest in it.
As for Antithesis itself, it looks very very cool. Bringing the deterministic testing down the stack to below the OS is awesome. Should make it possible to test entire systems without wiring up a harness manually every time. Can’t wait to try it out!