JohnBooty

il y a 3 joursdiscuss

    Giving a literal monkey the ability to press more 
    keys faster doesn't get a good joke from it.

To some extent, it's because making a joke isn't an iterative exercise with a verifiably correct outcome.

However, a lot of jobs are. Notably, much of software engineering.

JohnBooty

il y a 5 joursdiscuss

    guess you are arguing that the models _now_ will 
    be durably useful enough to commit the time to 
    creating the ASIC?

Objectively, the current frontier-ish models will be useful for some time. Imagine the zombie apocolypse hits, recedes, and you need to rebuild society. An offline copy of Fable or even Opus would be a nice thing to have.

Subjectively, it's hard to say if people will pay for "a model from 18 months ago, but REALLY FAST AND CHEAP"

The speed difference suggests some use cases that might narrow the performance gap. With a > 50x performance delta you have some headroom to play with.

You can do many many fast iterations of ye olde "Ralph loops." You can also jack up the reasoning/effort level. And you could probably do some combination of both, while still running really fast ie 10x the number of iterations at 2x reasoning/effort.

So I think a hypothetical "50x faster Opus 4.8, but burned into ASICs" could be pretty competitive against the frontier models from 2027, 2028, 2029, and maybe beyond?

JohnBooty

il y a 5 joursdiscuss

At a 50-100x speedup even a GPT-4o class model could perhaps compete with much newer models simply by thinking deeper, doing harness-controlled Ralph loops, etc. Sure, then it might be "only" ~2-5x faster, but, you wouldn't need to throw all the ASICs into the trash bin.

One could also imagine hybrid models, where part of the model is burned into ASICs and part of the model exists in VRAM/HBM2 so it can be updated.

I don't have enough low-level knowledge to evaluate the technical or economic feasibility of the above ideas, however.

JohnBooty

il y a 9 joursdiscuss

    My iPod would be the culmination of my friendships

Hey! You can bring this back in 2026! In a fit of random inspiration (possibly inspired by a glass of wine or two) I began a new practice.

If I'm hanging out with somebody who's passionate about music, I pull up Spotify (or Apple Music, whatever) and create a blank playlist. I hand them my phone and ask them to add some tracks for me.

Here's the trick that makes it work... I give them strict instructions not to pick something they think I'LL like. I just want some tracks or albums that THEY are passionate about.

I've had a lot of fun doing this! People are also generally touched that you want to know what kind of music they feel strongly about. It's a fun way to build connections.

JohnBooty

il y a 16 joursdiscuss

Not parent poster but: I probably spend at least 2/3 of my tokens on code review & QA. At least at my workplace, that's the culture.

JohnBooty

le mois dernierdiscuss

    because they're smarter than everyone else and
    there is nothing of value to be learned from others

Yeah. It's absolutely unreal how often this is seen in our industry.

Especially since everybody in the industry tends to be pretty smart.

When two people with intelligence within a single standard deviation of each other, each of them is going to have competencies and expertise the other does not. There are going to be specific skills where one truly is 10x or even 100x the other, but not too many efforts boil down to one specific narrow skill.

JohnBooty

le mois dernierdiscuss

Yeah, there’s been a lot of debate about this on r/localllama — will there be a steady supply of new free/open models in the future?

And if not, can we simply keep augmenting “stale” models with new knowledge to keep them useful?

I’m on the pessimistic side of things on both questions.

As for the second question, obviously stale models can be augmented to an extent but it’s nowhere near a substitute for new knowledge being fully baked directly into its training.

JohnBooty

le mois dernierdiscuss

    It should have flatly refused.

I disagree in the strongest possible terms.

I think the LLM should advise you of risk and lack of feasability but should otherwise answer the question, unless you're trying to do something plainly destructive to others e.g. weaponizing anthrax or something.

    A reasonable travel agent would have fired me as a customer.

Unless the LLM was actually acting as a travel agent -- booking the trip for you -- as opposed to merely advising you, this expectation feels off.

    unless you can somehow make it have judgment

It did have judgement. It told you what a bad idea it was.

I think this is a great example of the unrealistic expectations people have for LLMs. No sane and sensible person would treat any single source of knowledge as infallible, for any consequential decision.

(Certainly, of course, you don't have to look very far for examples of idiots being overly trustful of LLMs, or Google, or GPS, or Wikipedia, or whatever. It certainly does happen and yes, I've heard all these arguments before about other technologies besides LLM. Replace "LLM" in your post with any of those other terms, and I promise you somebody made literally the exact same argument in 2003 or 2009 or 2014 or whatever)

Any reasonable person would consult a second doctor, or at least other sources of knowledge, after the doctor advises them of some irreversible course of action. Because we don't even expect highly trained and intelligent medical professionals to be perfect.

And yet, we get angry at LLMs for not having perfect judgement, even though their creators are extremely literal about how they can make mistakes.

JohnBooty

le mois dernierdiscuss

Well, there's two issues really.

1. Store-bought tomatoes are nearly always bred for shippability and appearance over flavor 2. Store-bought tomatoes are picked when unripe, so they shelf-ripen during transit and at the store, which is highly inferior to ripening on the vine for flavor

For the first issue... you can buy heirloom tomato seeds at any major hardware or garden store in America.

For the second issue... even the typical tomato breeds will taste great if you grow them yourself and let them vine-ripen till they're ready to eat.

JohnBooty

le mois dernierdiscuss

Now, wait just a minute.

You presented an LLM with an obviously bonkers goal, the LLM told you it was a bad idea at multiple steps, and this is somehow... a shortcoming of the LLM?!?

You said it yourself: you needed to "bully" the LLM into even producing this plan.

Please, tell me what it should have done instead. Be very specific!

JohnBooty

le mois dernierdiscuss

At a high level, the processes are extremely similar in many (not all) ways.

They're obviously achieved in drastically different ways at a low enough level; LLMs obviously do not simulate neurons or any biological construct. (For the record, I'm absolutely not one of those people who thinks LLMs are "alive" or should be treated like they are)

Reminds me of the olllllld days of Pentium II's when people got N64 emulation working shockingly quickly using HLE techniques. If you weren't around for this, it was quite the shocker at the time. I think the analogy is doubly apt, because HLE emulation has some serious limitations... it gets you maybe 80% of the way there really fast, and for the remaining 20% you need to roll up your sleeves and do serious LLE.

https://en.wikipedia.org/wiki/UltraHLE

    It takes the prompt and continues it based on weights in 
    the training data. If there is no data it picks the most 
    likely thing (maybe made up). If there is it’ll mostly 
    add things from that data. Maybe it’ll make tool calls and 
    pull in data that way too but you can’t actually trust all 
    the details.

I'd like you to point out which bits of this are different from talking to humans. If you replace "training data" with "memories", this is pretty much exactly how things might go if you asked a friend (or perhaps a flaky travel agent) for travel advice.

Note that I'm not arguing that LLMs are particularly talented at this particular use case. I'm pointing out that humans are also pretty unreliable.

You're also doing that thing where you point out that LLMs can be unreliable (yes, they are) without acknowledging how flawed nearly every other source of information is: people, websites, etc. I'm not defending LLMs in that regard... I'm just saying it's not a differentiator.

JohnBooty

il y a 2 moisdiscuss

It’s really one of the most flabbergasting things about discussing LLMs with the naysayers.

There are a lot of extremely legitimate concerns, like the environmental impact and so on.

But I just laugh when they point out that LLMs are merely clever regurgitators of their previous inputs… as if this isn’t how we as humans operate nearly all of the time. People realllllllllly want to think they’re special snowflakes.

JohnBooty

il y a 2 moisdiscuss

I’ll grant that most non-local tomatoes have always been bad by definition, because they’re picked while green so they don’t rot before reaching the store.

Plum tomatoes absolutely did not used to be this bad, though. They are SO mealy now. Horrible. Beefsteaks are mealier as well. Those Campari tomatoes are pretty good year-round, though, I have to admit.

This is all in the NE USA, FWIW. I don’t know the tomato situation elsewhere.

    I doubt most people could even tell the difference 
    between two tomatoes of the same type and ripeness 
    if one came from the grocery store and the other from 
    a backyard garden.

Yeah, and I would run as fast as Usain Bolt if we woke up with the same body one day.

But that kind of the thing. They would almost never be the same ripeness because outside of local tomato season the tomatoes are picked while unripe, and then they “shelf-ripen” in transit because ethylene gas etc. That’s always been an issue, of course, and hasn’t changed over time.

The other issue is breeding - the continual breeding for appearance rather than flavor. Maybe we’re all imagining that one.

JohnBooty

il y a 2 moisdiscuss

Yes. I think convenience/utility explains a lot of these “depressingly homogenized experiences” far more than dopamine-seeking.

My life is very, very full. I do not have enough hours in the day, or years in my life, to fulfill all of my obligations and chase all of my dreams and interests. Not even close.

So I buy a lot of clothes from Old Navy, because they offer tall sizes that I need (surprisingly rare) and I honestly just have other things to do with my time. I’m aware there’s a whole world of interesting fashion out there, I just have 100 other things I want/need to spend my time on.

It’s the same with food, a lot of the time. Sometimes I just need a known quantity.

The restaurant chains know this, too. Sure… the commercials are all about satisfying your dopamine needs. But the way they actually run their operations is all about enforcing consistency. A Big Mac is supposed to taste the same everywhere. If you are a McDonalds franchisee, you can pick and choose which McDonalds products and promotions you sell (you can operate without selling french fries, if you’re crazy enough) but you absolutely cannot customize the ones you do sell.

(Yes, there are regional differences between McDonalds in different regions. Even within the US, there are some small differences due to regional suppliers and ingredient price/availability etc. However, these are very small differences and trust me, they really are laser-focused on consistency.)

JohnBooty

il y a 2 moisdiscuss

    The people are not fine with bad strawberries but they don't know good strawberries

You most definitely get this phenomenon with tomatoes. There’s little demand for actually good tomatoes, because most people don’t even know what a good tomato tastes like at this point.

This applies to countless things, but tomatoes are a prime example because they deteriorate so quickly once picked relative to other fruits I guess. So they have completely bred the flavor out of them in a quest to achieve something that looks good on a supermarket shelf.

JohnBooty

il y a 2 moisdiscuss

I have a friend who works in the flavor and fragrance industry and one of the things strawberry fragrance is used for is… (drum roll) actual strawberries.

Yep, a light spritz of strawberry scent on actual fucking strawberries apparently makes them more appealing.

JohnBooty

il y a 2 moisdiscuss

I’m maybe going to blow some fucking minds here — learning this certainly blew my own mind —- BUT

I have a friend who works in the “fragrance and flavor” industry. (Which is actually pretty fascinating, mostly in the sense that there are only about three major players, who kind of decide how everything in the world looks and tastes)

Annnnnnnnnnnd one of the things fake strawberry fragrance is user for is… strawberries. Like, actual supermarket strawberries. Some produce companies put fake scents onto real fruit so they, you know, smell more fruity.

Fuck this world.

JohnBooty

il y a 2 moisdiscuss

You're correct about some things but mostly wrong.

Yes, a Mac with 128GB+ will let you load some pretty big models.

However, you're still not going to be able to run them at usable speeds. Here are some M5 Max benchmarks on a Qwen 27B model w/ 290K context.... 12 tokens/sec output.

https://www.reddit.com/r/oMLX/comments/1swztoh/m5_max_128gb_...

And that's a 27B model. So yes, a M5 Max 128GB will let you load some pretty big models - can probably fit 120B in there with room left over for context. But the M5 Max still doesn't have the compute to make it practical, at least from an interactive usage standpoint - 120B dense model is going to be like an order of magnitude slower than 27B. You have to understand the computation going on here. LLMs are basically a huge many-to-many operation, and those operations themselves are pretty heavy.

So back to my previous post... you need three things. You need fast memory, you need a lot of it, and you need GPU compute with direct access to that fast memory. The M5 Max has like, 1.5 of the 3.

The M5 Ultra (if it ever exists) could kinda hit all 3, although actually getting your hands on one will be quite the lottery ticket.

   My understanding is this is the advantage that’s pushing huge Mac Studio demand.

This is true, but also, people who made this investment found that they're still not very usable for those HUGE models. Don't take my word for it though. Lots of benchmarks out there. r/localllama is pretty active too.

JohnBooty

il y a 2 moisdiscuss

I'm not the person you're replying to, but I wholeheartedly agree with them...

Quick background: doing AI inference requires three things. Lots of memory, lots of memory bandwidth, and of course plenty of compute that has access to that memory.

Quick reference: nVidia 5090 has 1,792 GB/sec bandwidth. 3090 gets about 1000 GB/sec. DGX Spark and AMD 395 whatever get about 275 GB/sec.

Apple M1 Max gets 400GB/sec, M5 Max gets 614GB/sec. Ultra variants get 2x that bandwidth, base variants get 1/2 that bandwidth. However... their compute is rather weak.

Right now, Apple's offerings are juuuuuust fast enough to run dense 27B models at usable speeds at like, 10% of the performance/watt of nVidia. They're world-leading general purpose CPUs but not killer GPUs.

By all accounts, these Windows PCs nVidia is touting seem to have DGX Spark like performance, which is less than impressive. Same with the upcoming AMD AI-oriented consumer stuff.

The other context here is that running your own AI at home is just starting to become feasible in terms of open model availability and the ability to run it at usable speeds. Many are interested in it for reasons of privacy, security, and cost certainty vs. buying tokens.

    Since Apple already sells unified memory systems, what 
    is the market opportunity you envision?

nVidia and AMD can't make their consumer offerings too good at AI, because that risks interfering with their higher-margin data center sales.

(And, let's face it. Even if nVidia did release a 6090 with 64-128GB of memory for an affordable price, consumers wouldn't get their hands on them anyway because people would just start filling data centers with them)

So.

Now you see Apple's opportunity, right? No data center sales to interfere with. No relationship with nVidia or AMD to worry about.

They could choose to make an absolute beast of a home AI machine. The M5 Ultra, if announced, might be that. It's admittedly a niche market, but people are already buying 64GB+ Macs faster than Apple can make them and they're fetching high prices on the used market as well.

The only real questions are if this market is even something Apple would find time to care about, and if they could secure enough DRAM to make a go at it. They are enormous obviously but they're feeling the RAM pinch just like everybody.

JohnBooty

il y a 2 moisdiscuss

    would you be dissatisfied by Opus-4.6-level open-weight 
    models, just because Opus 4.8 will be out?

Well, I see what you mean, but two big concepts...

1A. Models get stale pretty quickly w.r.t. new developments that occur past their cutoff date. "But you can just keep them current by linking them to never documentation, etc!" Well, no, you sorta can't -- at least not in perpetuity. Those search results fill up your context window real quick. So that gets unsustainable real quick.

1B. Even when your context has plenty of free space, the results you get from "here's a link to the documentation for this new framework that released after your cutoff date" absolutely pales to the results you get from knowledge that is fully baked into the trained model as opposed to your context window. For one thing, that documentation link you pasted into your context might link to... a dozen code examples. Whereas if that was baked into the model itself, the model might have been trained on many thousands of examples in Github etc.

2. It's also a reality that most professional engineers have to keep up with their peers and competitors. We can maybe say it shouldn't be that way, but it is. So if $SOME_NEW_MODEL is significantly better than 4.6... and my peers and or competitors are using it, then yeah I might but really feeling the need to match them. And I'm not even necessarily talking about some kind of cutthroat dog-eat-dog stack-ranked workplace.

These limitations aren't relevant for all use cases or careers but they're hiiiiiiiighly relevant for professional software engineering.

Submissions

Why Doesn't Apple Use the iPad as a “Touch Bar on Steroids” for the Mac?

comments