That's the beauty of side projects, you get to decide and you don't have to care about working with others.
Really yearning for crafted code that meets every one of your perfect coding style guide and preference with green checks across a huge set of static analysis tools and a large test suite, go for it!
Tired of conforming to your ultra strict style guide at work and code reviews that go in endless circles nit picking about use of spacing or whatever? Let loose and stop having to care/worry so much about the details.
On your side-project it's also ok to ignore best engineering practice, reinvent the wheel because you feel like it or make decisions based on what seems most interesting even if it's not a 'good' decision.
The critical thing is what the author says:
> always make sure that you're doing them for yourself, and for the right reasons
For me my side projects are generally something to have fun with and something to learn new things with. When you're finding it a slog or you feel like you've learnt what what you need to it's fine to just dump it.
Actually finishing something is of course nice and for beginners in particular there's a lot of value in going from that it's mostly there just some loose ends to tie off stage to the actually done stage but you don't have to always do this (or indeed just do it in some select cases).
A nice game, though the truth table lighting round is pretty punishing! Big contrast to the circuit building part where you can take your time. Personally I'd drop the time requirements from that quiz section.
> Also we found the formal waiver analysis tools to be very effective for waiving unreachable code, in case you aren't using those.
Yes we had used them just never got it slickly integrated into the verification dashboard. We had used this kind of analysis for internal sign off. You could generate the waivers manually and check them in but that suffers from the problem discussed above. Plus as OpenTitan was a cross company project you run into EDA licensing issues where not everyone has access to the same set of tools and a UNR flow could be running fine on one partner's infrastructure but isn't workable everywhere for multitude of reasons.
The ideal would be the nightly regression would do the UNR flow to generate the waivers and apply them when generating coverage but as ever there's only so much engineering time to go around and always other priorities.
Having spent several years working on OT I can tell you that most of the gaps are things that should be waived anyway. Getting waiver files reliably integrated into that flow has been problematic as those files are fragile, alter the RTL and they typically break as they refer to things by line number or expect a particular expression to be identical to when you did a waiver for it.
This has all been examined and the holes have been deemed unconcerning, yes ideally there'd be full waivers documenting this but as with any real life engineering project you can't do everything perfectly! There is internal documentation explaining the rationale for why the holes aren't a problem but it's not public.
I worked on OpenTitan for around 5 years at lowRISC. It certainly has its ups and downs but it's generated some great stuff and I'm very glad to see hit proper volume production like this. Whilst there's definitely open source chips out there and lots more using bits of open source that don't actually advertise this fact I believe this is the first chip with completely open RTL that's in a major production volume use case.
One of highlights working on OpenTitan was the amount of interest we got from the academic community. Work they did could actually get factored into the first generation silicon making it stronger. Ordinarily chips like that are kept deeply under wraps and the first time the wider security community can take a look at them development has long completed so anything they might find could only effect generation 2 or 3 of the device.
Academic collaboration also helped get ahead in post quantum crypto. This first generation chip has limited capabilities there but thanks to multiple academics using the design as a base for their own PQC work there was lots to draw on for future designs.
I'm no longer at lowRISC so I don't know where OpenTitan is going next but I look forward to finding out.
This is an interesting piece of hardware though when they go multi-chip for larger models the speed will no doubt suffer.
They'll also be severely limited on context length as it needs to sit in SRAM. Looks like the current one tops out at 6144 tokens which I presume is a whole chips worth. You'd also have to dedicate a chip to a whole user as there's likely only enough SRAM for one user's worth of context. I wonder how much time it takes them to swap users in/out? I wouldn't be surprised if this chip is severely underutilized (can't use it all when running decode as you have to run token by token with one users and then idle time as you swap users in/out).
Maybe a more realistic deployment would have chips for linear layers and chips for attention? You could batch users through the shared weight chips and then provision more or less attention chips as you want which would be per user (or shared amongst a small group 2-4 users).
> If copying user tokens was the bottle neck, batching would not achieve any speed up.
Reality is more complex. As context length grows your KV cache becomes large and will begin to dominate your total FLOPs (and hence bytes loaded). The issue with KV cache is you cannot batch it because only one user can use it, unlike static layer weights where you can reuse them across multiple users.
Emerging sparse attention techniques can greatly relieve this issue though the extent to which frontier labs deploy them is uncertain. Deepseek v3.2 uses sparse attention though I don't know off hand how much this reduces KV cache FLOPs and associated memory bandwidth.
Anyone else find reading things like this slightly exhausting?
I'm very much pro AI for coding there are clearly significant capabilities there but I'm still getting my head around how to best utilise it.
Posts like these make it sound like ruthlessly optimizing your workflow letting no possible efficiency go every single day is the only way to work now. This has always been possible and generally not a good idea to focus on exclusively. There's always been processes to optimise and automate and always a balance as to which to pursue.
Personally I am incorporating AI into my daily work but not getting too bogged down by it. I read about some of the latest ideas and techniques and choose carefully which I employ. Sometimes I'll try and AI workflow and then abandon it. I recently connected Claude up to draw.io with an MCP, it had some good capabilities but for the specific task I wanted it wasn't really getting it so doing it manually was the better choice to achieve what I wanted in good time.
The models themselves and coding harnesses are also evolving quickly complex workflows people may put together can quickly become pointless.
Only taken a quick skim but this looks like solid material!
RISC-V Vector is definitely tricky to get a handle on, especially if you just read the architecture documentation (which is to be expected really, good specification for an architecture isn't compatible with a useful beginners guide). I found I needed to look at some presentations given by various members of the vector working group to get a good grasp of the principles.
There's been precious little material beyond the specification and some now slightly old slide decks so this is a great contribution.
> However, Groq’s architecture relies on SRAM (Static RAM). Since SRAM is typically built in logic fabs (like TSMC) alongside the processors themselves, it theoretically shouldn't face the same supply chain crunch as HBM.
It's true SRAM comes with your logic, you get a TSMC N3 (or N6 or whatever) wafer, you got SRAM. Unfortunately SRAM just doesn't have the capacity you have to augment with DRAM which you see companies like D-Matrix and Cerebras doing. Perhaps you can use cheaper/more available LPDDR or GDDR (Nvidia have done this themselves with Rubin CPX) but that also has supply issues.
Note it's not really about parameter storage (which you can amortize over multiple users) it's KV cache storage which gets you and that scales with the user count.
Now Groq does appear to be going for a pure SRAM play but if the easily available pure SRAM thing comes at some multiple of the capital cost of the DRAM thing it's not a simple escape hatch from DRAM availability.
It's for a CS course at Stanford not a PyTorch boot camp. It seems reasonable to expect some level of academic rigour and need to learn and demonstrate understanding of the fundamentals. If researchers aren't learning the fundamentals in courses like these where are they learning them?
You've also missed the point of the article, if you're building novel model architectures you can't magic away the leakiness. You need to understand the back prop behaviours of the building blocks you use to achieve a good training run. Ignore these and what could be a good model architecture with some tweaks will either entirely fail to train or produce disappointing results.
Perhaps you're working at a level of bolting pre built models together or training existing architectures on new datasets but this course operates below that level to teach you how things actually work.
Karpathy's contribution to teaching around deep learning is just immense. He's got a mountain of fantastic material from short articles like this, longer writing like https://karpathy.github.io/2015/05/21/rnn-effectiveness/ (on recurrent neural networks) and all of the stuff on YouTube.
Plus his GitHub. The recently released nanochat https://github.com/karpathy/nanochat is fantastic. Having minimal, understandable and complete examples like that is invaluable for anyone who really wants to understand this stuff.
To me at least "not good after all" means their current latest hardware has issues which means it cannot replace Nvidia GPUs yet. This is a hard problem so not getting there yet doesn't imply bad engineering just a reflection of the scale of the challenge! It also doesn't imply that if this generation is a miss following generations couldn't be large win. Indeed I think it would be very foolish to assume that Alibaba or other Chinese firms cannot build devices that can challenge Nvidia here on the basis of current generation not being up to it yet. As you say they have a large market that's willing to wait for them to become good.
Plus it may not be true, this new Alibaba chip could turn out to be brilliant.
I'd say there's a mix of 'Chinese GPUs are not that good after all' and 'Nvidia doesn't have any magical secret sauce, and China could easily catch up' going on. Nvidia GPUs are indeed remarkable devices with a complex software stack that offers all kinds of possibilities that you cannot replicate over night (or over a year or two!)
However they've also got a fair amount of generality, anything you might want to do that involves huge amounts of matmuls and vector maths you can probably map to a GPU and do a half decent job of it. This is good for things like model research and exploration of training methods.
Once this is all developed you can cherry pick a few specific things to be good at and build your own GPU concentrating on making those specific things work well (such as inference and training on Transformer architectures) and catch up to Nvidia on those aspects even if you cannot beat or match a GPU on every possible task, however you don't care as you only want to do some specific things well.
This is still hard and model architectures and training approaches are continuously evolving. Simplify things too much and target some ultra specific things and you end up with some pretty useless hardware that won't allow you to develop next year's models, nor run this year's particularly well. You can just develop and run last year's models. So you need to hit a sweet spot between enough flexibility to keep up with developments but don't add so much you have to totally replicate what Nvidia have done.
Ultimately the 'secret sauce' is just years of development producing a very capable architecture that offers huge flexibility across differing workloads. You can short-cut that development by reducing flexibility or not caring your architecture is rubbish at certain things (hence no magical secret sauce). This is still hard and your first gen could suck quite a lot (hence not that good after all) but when you've got a strong desire for an alternative hardware source you can probably put up with a lot of short-term pain for the long-term pay off.
If I'm reading this right, glitching the I2C bus prevents the Secure Enclave from booting. It seems the device recovers from this itself 'Although the device recovered and remained operable', maybe the Secure Enclave reboots itself after seeing a fault in the I2C?
No evidence of any security issue is presented. Though it's certainly wanted to drum it as something major 'This is a high-severity, unpatchable design flaw'.