It's was built and is regularly maintained by a Riot Games principal C++ architect, and automatically compiles files in large "unity" chunks, distributes builds across all machines in an organization, and creates convenient Visual Studio .sln files, and XCode projects. It's also all command line driven and open source.
This is industrial strength C++ builds for very large rapidly changing code bases. It works.
I worked with a fairly large code base which leveraged threads to avoid having to use callbacks in a C++ code base. This allowed the engineers to use the more familiar linear programming style to make blocking network calls which would stall one thread, while others were unblocked to proceed as responses were received. The threads would interoperate with each other using a similar blocking rpc system.
But what ended up happening was that the system would tend to block unnecessarily across all threads, particularly in cross thread communication. These were due to many external network calls being performed sequentially when they could have been performed in parallel, and likewise cross-thread communication constantly blocking due to sequential calls. The end result was a system who'd performance was just fundamentally hobbled by threads that were constantly waiting on other threads or series of external calls, and it was very difficult to understand the nature of the gordian knot of blocking behavior that was causing these issues.
The main problem with using threads is that the moment you introduce a different cpu into the mix you need to deal with synchronization primitives around your state, and this means that engineers will use fewer threads (at least in our case) than necessary to reduce the complexity of the synchronization work needed to be done which means that you lose the advantage of asynchronous parallelism. Or at least that is what happened in this particular case.
The cost of engineering synchronization for async/await is zero, because this parallelism happens on a single thread. Since the cpu "work" to be done for async/io is relatively small, this argues for using single threaded 'callback' style solutions where you maximize the amount of parallelism and decrease the amount of potential blocking as well as minimizing the complexity of thread synchronization as much as possible. In cases where you want to leverage as many cpu's as possible, it's often the case that you can better benefit from cpu parallelism by simply forking your process on multiple cores.
Paleo-dunes reminds me of when I was gobsmacked when I learned the kilometers deep sandstone deposits in Grand Canyon and Grand Staircase (Coconino and Navajo other sandstone formations) were sand from huge sediment carried by ancient east to west running possibly Mississippi sized rivers from the Himalaya high great Appalachian mountain ranges on the North American east coast. Huge mountains were just "sanded down" over time and deposited thousands of miles away to the west.
The surprised and shocked to learn that thousands(!!) of feet of rock had been eroded away above the current ground level of the Grand Canyon to move that enormous volume of former solid rock elsewhere. It boggles the mind to imagine a mile of rock over a large region just being just moved to a new location.
And then surprise turns to astonishment, as they say, to learn that the Vishnu Schist unconformity at the bottom of the whole sequence records several hundred million years in two periods of the earth being a giant snowball covered in ice where the entire continental surface across the planet was sanded almost smooth by enormous Greenland scale glaciers.
Together serves models optimized for inference speed.
They're not Groq but Together (and Perplexity Labs) have the lowest latencies and fastest tokens per second of any commercial services available right now. Also the lowest prices afaik.
I have this hooked up experimentally to my universal Dungeon Master simulator DungeonGod and it seems to work quite well.
I had been using Together AI Mixtral (which is serving the Hermes Mixtrals) and it is pretty snappy, but nothing close to Groq. I think the next closes that I've tested is Perplexity Labs Mixtral.
A key blocker in just hanging out a shingle for an open source AI project is the fear that anything that might scale will bankrupt you (or just be offline if you get any significant traction). I think we're nearing the phase that we could potentially just turn these things "on" and eat the reasonable inference fees to see what people engage with - with a pretty decently cool free tier available.
I'd add that the simulator does multiple calls to the api for one response to do analysis and function selection in the underlying python game engine, which Groq makes less of a problem as it's close to instant. This adds a pretty significant pause in the OpenAI version. Also since this simulator runs on Discord with multiple users, I've had problems in the past with 'user response storms' where the AI couldn't keep up. Also less of a problem with Groq.
Fastbuild https://www.fastbuild.org/docs/home.html is the free distributed compilation system many game companies use. The combination of automatic unity builds (simply appending many .cpp source files together into very large combined files), caching and distributed compilation together gives you extremely fast C++ builds.
Also supports creating project files that are compatible with XCode and Visual Studio so you can just build from those IDE's and a pretty flexible dependency based build file format that can accomodate any kind of dependency.
What LLM's are doing when they imagine playing chess is what we do when we stand up after sitting on the floor, or what we do when we see a few million individual samples of color and light intensity and realize there's an apple and a knife in front of us.
I think what is almost impossible for most people to understand is that AI's do not need to be structured like the human brain and use the crutches we use to solve problems the way we do because evolution did not provide us with a way of instantly understanding complex physics or instantly absorbing the structure of a computer program by seeing it's code in one shot.
It micro-sleeps. As far as the AI knows no time passes during the periods when it's not answering. And it experiences a form of amnesia when the chat is over. A "50 first dates" bot. Bing reports that it does have both a wall clock and some form of processing metric it can "see" though (this seems to be consistent though it could be hallucinating).
A more serious omission is a lack of continuous inner dialog. Bing has the #inner-monologue tag but all of the AI's language based thought happens out in the open for the most part, and it doesn't have any time to process or ponder what's been said.
Of course that can be remedied somewhat by putting the AI in "ponder-bot" mode, where you can tell it to write out it's thoughts privately while you "pause". Both Bing and GPT will ponder if you ask them to and write an inner monologue if you give it time and ask it to shield it's dialog with some privacy.
It's interesting what Bing or GPT will reveal about their thoughts when in "ponder-bot" mode. Bing (at least in my trials) will only tell you generally what it hid from you in its private thoughts but the times I've tried to pry Bing ends the chat, and GPT will usually reveal them if you ask.
I would recommend getting an account and simply testing it directly. It's fairly easy to demonstrate that it operates at a conceptual level and is not merely predicting word probabilities in a simplistic way.
A good example with GPT is to watch it do complex math. There are simply too many permutations of math solutions for it to have ever memorized, and it can easily explain its process and the path it took to arrive at a solution.
Another good set of tests are ones around theory of mind, complex deduction problems and missing information, etc. A good source of information about the precise capabilities of GPT-4 is the Microsoft Sparks paper, which goes into a good number of tests MS researchers put the model to.
There's also the question of ethics. Some of these LLM's regularly profess that they are sentient (Bing for example regularly does). We don't "think" these are valid but at this point there's no way to be absolutely certain.
And again, folks who profess to know for certain an LLM can't in any way be sentient are leaning pretty far over their skis. It's unlikely given how they work, but not impossible.
I think the consensus at this point is that these models are much closer to AGI than anyone thought they could or should be, and that the delta between what we have now and AGI is smaller than it's ever been.
Anyone who tells you that these models are "just glorified text generators" is flat out wrong and hasn't bothered to do their homework. And anyone who claims they "know how it works under the hood" is making claims that all of the true experts have notably carefully avoided making.
I tried Chomsky's counterfactual apple questions with Chat GPT. It was able to infer counterfactuals about apples or any sort of physical object, both forwards and backwards in time. So I assume Chomsky has not spent much time with ChatGPT or he would know that it was capable of making physical inferences and counterfactual inferences like that. Not perfectly (it did make a few mistakes) but fairly reliably.
That ChatGPT regularly "makes stuff up" and that there is no difference between the truth or falsehood of any statement also seems to be false. I asked ChatGPT to act as a "[Lie Detector]" and to rate the truth or falsehood of a variety of statements asked. I asked about 40 questions ranging from physical situations (heavy objects floating away into air) temporal questions (time travel, etc.) and logical questions - and it very accurately could determine if each of these statements was "true" or "false" given physical or logical rules. Again not perfect but very accurate (38 out of 40 correct).
With attention - ChatGPT is very obviously operating at a level above the simple probabilistic prediction. It clearly seems that it has some notion as to the meaning of what is being said and is making inferences based on that meaning. That those inferences were trained probabilistically is certainly true, but that it was trained on the average human's understanding of those physical or temporal or other constraints also seems to be true and to also be fairly accurate.
Premature optimization may be the root of all evil. But failing to architect an application from the start for reasonable performance and scalability is the premature root of all optimization.
It's was built and is regularly maintained by a Riot Games principal C++ architect, and automatically compiles files in large "unity" chunks, distributes builds across all machines in an organization, and creates convenient Visual Studio .sln files, and XCode projects. It's also all command line driven and open source.
This is industrial strength C++ builds for very large rapidly changing code bases. It works.