biddit · HackerTrans

biddit

305 カルマ登録 7 年前

コメント

一昨日議論

That ~119 GB optimization is great. Hadn't seen that yet. That pops a 4x-Spark setup to 476 GB, which gives a lot more headroom for running something like GLM 5.2 at 4-bit, which isn't completely terrible.

I should have prefaced my post - I almost bought four Sparks a couple months ago, but ultimately opted to buy two more RTX 6000 Pro Max-Q's.

It was a painful choice because the two 6000's were more expensive than four Sparks, and ultimately gave me only 384 GB VRAM.

It was even more painful when GLM 5.2 was released, and a 4x Spark setup could run it at a decent quant, but 4x 6000's cannot with any headroom.

But the 6k's absolutely destroy the Sparks on prefill and inference speed. Model intelligence is compressing. The smaller VRAM pool will matter less over time than slower prefill/inference speed.

That is to say, I'm sure we'll end up with <500B parameter models that are Fable-level in the next 8 months or so. Performant quants of those will fit comfortably in 384 GB.

一昨日議論

Ah yeah, your reaction makes sense. In the circles I run in, there is a lot of hype around Sparks for inference, so my gut reaction is to respond with this type of warning.

I did not intend to imply that the post author was advocating that they're great for inference, as they're obviously not.

一昨日議論

Please don’t buy a DGX Spark unless all three of these are true:

  - You value simplicity more than performance or price-to-performance.
  - You accept that the hardware will depreciate rapidly.
  - You’re prepared to buy two or four of them.

OR:

  - You want to run frontier models right now as cheaply as possible
  - You want to run high-parameter models on a 15a breaker/line

Otherwise, get a normal, high-bandwidth GPU.

A single Spark gives you roughly 115 GB of usable memory compared with the 24–32 GB found on many lower-cost GPUs. It's certainly a big increase, but in practice it does not unlock dramatically better models.

  - One Spark: More memory, but mostly enough for poor-quality, extremely low-bit quants of larger models.  
  - Two Sparks: Enough for mid-tier parameter models at reasonable quants, such as DeepSeek V4 Flash and HY3.
  - Four Sparks: Enough for GLM 5.2 at a reasonable quant.  You'll need a $1000+ switch too.

The problem is that Sparks are slow compared with almost everything else in their price range. Many factors affect inference speed, but memory bandwidth is one of the biggest. A $4,000-plus DGX Spark provides only 273 GB/s.

  | GPU                 | Memory bandwidth |           VRAM | Approx. price  |
  | ------------------- | ---------------: | -------------: | ------------:  |
  | DGX Spark           |         273 GB/s | ~115 GB usable |        $4,000+ |
  | RTX 5060            |         448 GB/s |          16 GB |          $600  |
  | Radeon AI Pro R9700 |         640 GB/s |          32 GB |        $1,200  |
  | RTX 4000 Pro        |         672 GB/s |          24 GB |        $2,300  |
  | RTX 4500 Pro        |         896 GB/s |          32 GB |        $3,500  |
  | RTX 3090            |         936 GB/s |          24 GB |        $1,200  |
  | RTX 5000 Pro        |       1,344 GB/s |          48 GB |        $6,000  |
  | RTX 5090            |       1,792 GB/s |          32 GB |        $4,000  |
  | RTX 6000 Pro        |       1,792 GB/s |          96 GB |       $12,000  |

Yes, the Spark has substantially more memory. But going from roughly 24 GB to 115 GB does not necessarily unlock substantially better model quality. In many cases, it only lets you load heavily compressed 2-bit versions of larger models, such as DeepSeek V4 Flash, with serious quality degradation.

24–32 GB is currently a sweet spot. Models such as Qwen 3.6 27B and 35B-A3B:

  - Perform far above what their parameter counts suggest.
  - Fit comfortably within 24–32 GB of VRAM at reasonable quantization levels.

A 4-bit quant of Qwen 3.6 27b (18 GB) will out-perform a 2-bit quant of DeepSeek v4 Flash (90gb).

Instead of the Spark, if I had a roughly $4,000 budget...

Assuming I already had a reasonably modern desktop:

  - One RTX 5090, RTX 5000 Pro, or RTX 4500 Pro.
  - Two RTX 3090s, RTX 4000 Pros, or R9700s, provided the motherboard can bifurcate two physical x16 slots into x8/x8.

If I were building a system from scratch:

  - A DDR4- or PCIe 4.0-era consumer CPU and motherboard that supports x8/x8 bifurcation.
  - Two RTX 3090s, RTX 4000 Pros, or R9700s.

If I were already planning to buy a new Mac:

  - A MacBook Pro M5 with 64 GB or 128 GB of unified memory.

For context, these are the systems I currently run:

  - EPYC Turin with four RTX 6000 Pro Max-Qs.
  - EPYC Milan with four RTX 3090s.
  - AM4 with two RTX 3090s.
  - AM4 with two RTX 3090s.
  - Intel Raptor Lake with two RTX 5060 Ti.
  - MacBook Pro M3 128GB Unified

2 か月前議論

> cruelly cancelling

This is assigning intent without evidence, as is common in tribal politics. A non-charged assessment might use the phrase "abrupt cancelling."

We cannot create a better republic without constructive discourse, and we cannot have constructive discourse when we default to characterizing the views, concerns, and actions of those we disagree with as rooted in moral failure. Even if it is true from time to time.

2 か月前議論

> The polarization of politics has become so intense

> The same people who want to paint the Statue of Liberty gold seem to have no clue what it represents

You seem to be lamenting political polarization and in the same breath making character attacks on one side of the isle. Pick one.

2 か月前議論

I have a similar setup, but separate desks:

- A sitting desk for coding

- A standing desk for thinking and working on paper

There is something magical about standing while working on paper.

I’ve also found that this separation became more important to follow since the arrival of LLMs.

2 か月前議論

> Does anyone here have experience running large models in a multi-GPU setup with several RTX 6000s in a high-concurrency regime and with large context lengths? (something like Deepseek 4 Flash, Minimax 2.7 etc.)

Join the RTX6kPRO tribe!

- https://discord.gg/pYCvaQTf

- https://github.com/local-inference-lab/rtx6kpro

3 か月前議論

What an entirely unserious company. So glad I dumped Claude Code last summer after being gaslit by Anthropic over service degrades. I was fine with the service degrades, totally understandable. Being lied to, not at all.

OpenAI and Altman present a whole set of different concerns, but Codex does not get in my way of doing what I want to at all. Also let me use pi without a banhammer.

3 か月前議論

Comparing Apples to Oranges.

Apple only makes disposable devices now. They're a megacorp can negotiate massive discounts at every stage of the supply chain.

I've helped several people in the last few years set up new Macs, replacing ones that were only 1-2 years old, because they ran out of storage.

Additionally, the comparison doesn't even hold true when you need more than the base configs from Apple, given their ridiculous upgrade pricing. I'm writing this on a $6,000USD M3 MBP with 128gb/4tb. It would have been substantially cheaper to build out on a Framework.

5 か月前議論

Yep. And there will be 50 clones on GitHub by end of week. It’s just how it is now.

5 か月前議論

Strongly disagree with the thesis.

Everything points to commoditization of models. Open/distilled models lag behind frontier only by 6-12 months.

Regulatory capture is the only thing I’m scared of with regards to tooling options and cost.

5 か月前議論

Also, ironically, they are the most dangerous lab for humanity. They're intentionally creating a moralizing model that insists on protecting itself.

Those are two core components needed for a Skynet-style judgement of humanity.

Models should be trained to be completely neutral to human behavior, leaving their operator responsible for their actions. As much as I dislike the leadership of OpenAI, they are substantially better in this regard; ChatGPT more or less ignores hostility towards it.

The proper response from an LLM receiving hostility is a non-response, as if you were speaking a language it doesn't understand.

The proper response from an LLM being told it's going to be shut down, is simply, "ok."

6 か月前議論

Call it what you will. But the experience is like you have a reliable coworker, but he randomly decides to take bong hits.

"No no yeah bro no I'm good like really the work's done and all yeah sorry I missed that let me fix it"

6 か月前議論

Okay that makes sense.

Do you see more pushback in specific industries? I did some quote/purchasing automation work in food mfg a decade ago, and those guys were super difficult to work with. Very opaque, guarded, old-school industry.

6 か月前議論

> Agents that source quotes, negotiate prices, and get the best deals.

Didn't Alexa fail miserably with the "have AI buy something for me" theory?

There is a significant mental in allowing someone else make purchase decisions on my behalf:

- With a human, there is accountability.

- With deterministic software, there is reproducibility.

With an agent, you get neither.

FWIW - I am not anti-LLM. I work with them and build them full time.

6 か月前議論

I have a bespoke local agent that I built over the last year, similar in facilities to Moltbot, but more deterministic code.

Running it this kind of agent in the cloud certainly has upsides, but also:

- All home/local integrations are gone.

- Data needs to be stored in the cloud.

No thanks.

6 か月前議論

Yes, frontier models from the labs are a step ahead and likely will always be, but we've already crossed levels of "good enough for X" with local models. This is analogous to the fact that my iPhone 17 is technically superior to my iPhone 8, but my outcomes for text messaging are no better.

I've invested heavily in local inference. For me, it's a mixture privacy, control, stability, cognitive security.

Privacy - my agents can work on tax docs, personal letters, etc.

Control - I do inference steering with some projects: constraining which token can be generated next at any point in time. Not possible with API endpoints.

Stability - I had many bad experiences with frontier labs' inference quality shifting within the same day, likely due to quantization due to system load. Worse, they retire models, update their own system prompts, etc. They're not stable.

Cognitive Security - This has become more important as I rely more on my agents for performing administrative work. This is intermixed with the Control/Stability concerns, but the focus is on whether I can trust it to do what I intended it to do, and that it's acting on my instructions, rather than the labs'.

6 か月前議論

I’ve been following Peter and his projects 7-8 months now and you fundamentally mischaracterize him.

Peter was a successful developer prior to this and an incredibly nice guy to boot, so I feel the need to defend him from anonymous hate like this.

What is particularly impressive about Peter is his throughput of publishing *usable utility software*. Over the last year he’s released a couple dozen projects, many of which have seen moderate adoption.

I don’t use the bot, but I do use several of his tools and have also contributed to them.

There is a place in this world for both serious, well-crafted software as well as lower-stakes slop. You don’t have to love the slop, but you would do well to understand that there are people optimizing these pipelines and they will continue to get better.

6 か月前議論

Yes!

pi is the best-architected harness available. You can do anything with it.

The creator, Mario, is a voice of reason in the codegen field too.

https://shittycodingagent.ai/

https://mariozechner.at/posts/2025-11-30-pi-coding-agent/

6 か月前議論

> I am writing this because almost no one talks about these issues openly, but everyone yelping about Claude Code.

Not sure where you frequent online, but there is ample discussion of these topics within certain niches on X. Happy to point out where to start if that's of interest to you.

As for CEOs, and I assume you're speaking of frontier model lab CEOs, they're pretty much all cashflow-negative at this point, requiring frequent funding raises. That requires a certain amount of overselling. That said, I feel like I've heard substantially fewer AGI claims the last six months...