The interesting thing here isn't "how, logistically, is the Groq corporate entity able to raise more money?". That's straightforward.
Rather, the interesting thing and the topic of most of the article is "how, after Nvidia hired most of Groq's team and licensed all their IP, did Groq manage to convince investors to invest in the remaining corporate entity?"
Their AES implementation uses old-school 2-share boolean masking [1], which has been shown to be insecure since 2005 [2][3]. A modern implementation would use domain-oriented masking [4], like OpenTitan does. Pretty bad look for Crossbar.
Their AES implementation uses old-school 2-share boolean masking [1], which has been shown to be insecure since 2005 [2][3]. A modern implementation would use domain-oriented masking [4], like OpenTitan does. Pretty bad look for Crossbar.
One thing that people rarely discuss about book publishing is a change to US tax law in the late 70s that meant that publishers couldn't write down the value of unsold inventory, but could write off that inventory by destroying it.
That meant that poorly selling books were destroyed to realize a taxable loss, which killed the ability for books to slowly "pick up steam" over a year or two to eventually generate a profit for the publisher. If you didn't make a profit fast, the backlog got destroyed and the book lost its chance to make money.
Author here -- I don't disagree! I actually noted this in the article:
> Well, it turns out that LLMs are also pretty valuable when it comes to chips for lucrative markets -- but they won’t be doing most of the design work. LLM copilots for Verilog are, at best, mediocre. But leveraging an LLM to write small snippets of simple code can still save engineers time, and ultimately save their employers money.
I think designers getting 2x faster is probably optimistic, but I also could be wrong about that! Most of my chip design experience has been at smaller companies, with good documentation, where I've been focused on datapath architecture & design, so maybe I'm underestimating how much boilerplate the average engineer deals with.
Regardless, I don't think LLMs will be designing high-performance datapath or networking Verilog anytime soon.
I didn't get into this in the article, but one of the major challenges with achieving superhuman performance on Verilog is the lack of high-quality training data. Most professional-quality Verilog is closed source, so LLMs are generally much worse at writing Verilog than, say, Python. And even still, LLMs are pretty bad at Python!
If they're doing inference on edge devices, one challenge I see is protecting model weights. If you want to deploy a proprietary model on an edge AI chip, the weights can get stolen via side-channel attacks [1]. Obviously this isn't a concern for open models, but I doubt Apple would go the open models route.
I'm curious what the motivation is here -- unfortunately, the dev blog is all in Chinese and I can't read it. If it's mostly to show a proof-of-concept of LLMs on a FPGA, that's awesome!
But if this is targeting real-world applications, I'd have concerns about price-to-performance. High-level synthesis tools often result in fairly poor performance compared to writing Verilog or SystemVerilog. Also, AI-focused SoCs like the Nvidia Jetson usually offer better price-to-performance and performance-per-watt than FPGA systems like the KV260.
Potentially focusing on specialized transformer architectures with high sparsity or significant quantization could give FPGAs an advantage over AI chips, though.
Not to toot my own horn, but I wrote up a piece on open-source FPGA development recently going a bit deeper into some of these insights, and why AI might not be the best use-case for open-source FPGA applications: https://www.zach.be/p/how-to-build-a-commercial-open-source
I'm definitely not! I'm a hardware designer and I work with FPGAs all the time, for both work and for personal projects. Like with all things, there's a right tool for every job, and I think for modern DL algorithms like Transformers, GPUs and AI ASICs are the better tools. For rapid hard prototyping, or for implementing specialized architectures, FPGAs are far better.
The big challenge when it comes to using FPGAs for deep learning is pretty simple: all of that reprogrammability comes at a performance cost. If you're doing something highly specific that conventional GPUs are bad at, like genomics research [1] or high-frequency trading [2], the performance tradeoff is worth it. But for deep learning, GPUs and AI ASICs are highly optimized for most of these computations, and an FPGA won't offer huge performance increases.
The main advantage FPGAs offer is being able to take advantage of new model optimizations much earlier than ASIC implementations could. Those proposed ternary LLMs could potentially run much faster on FPGAs, because the hardware could be optimized for exclusively ternary ops. [3]
I think one major challenge they'll face is that their architecture is incredibly fast at running the ~10-100B parameter open-source models, but starts hitting scaling issues with state-of-the-art models. They need 10k+ chips for a GPT-4-class model, but their optical interconnect only supports a few hundred chips.
"Smart home" has really fallen flat across the board, from smart speakers to the Roomba. It would be great if Apple changed that, but I'm not optimistic. I just don't think people really want smart home devices all that much.
This doesn't appear to support any FPGAs other than their FPGA chiplet [1]. Also, like UncleOxidant said, the most complex parts of this toolchain are just existing open-source tools (Yosys and VPR).
This seems like a useful toolchain for ZeroASIC customers who are using their hardware, but not for FPGA enthusiasts more broadly.
There are meaningful size and performance challenges to implementing post quantum crypto, but I don't think "oh no, HTTPS handshakes are 20% slower!" is something we should actually worry about.
The real worry is all of the small, secure, embedded devices that literally don't have enough memory or compute to run these algorithms at all. Even the state-of-the-art hardware-accelerated implementations of PQC use a ton of memory and a ton of silicon area, to the point where they're untenable for lower end devices.
I'm really skeptical of Eteched. They make claims about how their chips are "optimized for transformers", as if Nvidia's Tensor Cores aren't optimized for those workloads too. Plus, if they really are only good at transformers, they'd get screwed over if SSMs or some other models become dominant. And their website has a "tokens per second" graph with no Y-axis, which is never a good look.
I think that if you want to build an AI chip company, you have to leverage some novel technology (processing-in-memory, photonics, etc) that counter-positions you against Nvidia. Trying to beat Nvidia at optimizing digital logic seems like a losing battle.
Rather, the interesting thing and the topic of most of the article is "how, after Nvidia hired most of Groq's team and licensed all their IP, did Groq manage to convince investors to invest in the remaining corporate entity?"