"Everybody will need to do some work if he is to be contented ... a 15-hour week may put off the problem for a great while. For 3 hours a day is quite enough to satisfy the old Adam in most of us!" - Keynes, 1930
Though this was a 100-year prediction so we still got three and half to go!
Page 18 of the paper:
> As shown in Table 1, our approach outperforms other methods for both Llama-3.1-8B-Instruct and Ministral-7B-Instruct, achieving significantly higher average scores. We evaluate our method using 2.5-bit and 3.5-bit quantization during text generation. These non-integer bit precisions result from our strategy of splitting channels into outlier and non-outlier sets, and applying two
independent instances of TurboQuant to each, allocating higher bit precision to outliers. This outlier treatment strategy is consistent with prior work [63, 51] . For example, in our 2.5-bit setup, 32 outlier channels are quantized at 3 bits, while the remaining 96 channels use 2 bits, leading to an effective bit precision of (32 ×3 + 96×2)/128 = 2.5. For 3.5-bit quantization, a different ratio of
outliers and regular channels leads to a higher effective bit precision. Despite using fewer bits than competing techniques, TurboQuant maintains performance comparable to unquantized models
So they find channels / indicies-of-the-vector that are important and give them more bits (3 bits) than the rest (2 bits).
>Isn't the turbo codebook the irregularly spaced centroid grid?
yes I believe so. They mention it's informed by the concentration of measure and the uncorrelated/independent vectors after the initial conditioning rotation. I feel like it was informed by PolarQuant, but that may just be how I intuit what's going on (because thinking about this in polar coordinates makes more sense in my head). IOW, I think the irregular spacing is maybe informed by TurboQuant.
However they do say, slightly to the contrary: "We find optimal scalar quantizers for random variables with Beta distributions by solving a continuous 1-dimensional k-means problem using the Max-Lloyd algorithm."
Here's my attempt at a undergrad-level summary (corrections welcome!):
The core idea is to quantize KV cache, but do so in a way that destroys minimal information. In this case, it's similarly scores between vectors. The simplest way to do this is to change all the elements from 16bit of precision to, say, 4 bits (Scalar Quant.). These papers improve on it by realizing: almost all the energy (concentration of measure) is towards the equator of the hypersphere (normally distributed as 1/d; d=vector dimensionality). (The curse/blessing of hyper dimensionality strikes again.) So when we quantize the elements (think "latitudes", e.g. to the nearest degree) we destroy a lot of information because basically all the vectors were around the equator (so some latitudes have a lot of vectors and some have very few). The idea is to rotate the vectors away from the equator so they're more consistently distributed (to better preserve the entropy during quantization, which I guess was amitport's DRIVE idea). PolarQuant does a hyperpolar coordinate transform which superficially seems neat for preserving entropy because of this equator/polar framing (and ultimately unnecessary as shown by TurboQuant). They also realized there's a bias to the resulting vectors during similarity, so they wrote the QJL paper to fix the bias. And then the TurboQuant paper took PolarQuant + QJL, removed the hyperpolar coords, and added in some gross / highly-pragmatic extra bits for important channels (c.f. elements of the vectors) which is sort of a pathology of LLMs these days but it is what it is. Et voila, highly compressed KV Cache. If you're curious why you can randomly rotate the input, it's because all the vectors are rotated the same, so similarity works out. You could always un-rotate to get the original, but there's no need because the similarity on rotated/unrotated is the same if you compare apples to apples (with the QJL debiasing). Why was PolarQuant even published? Insu Han is solely on that paper and demanded/deserved credit/promotion, would be my guess. The blog post is chock-full of errors and confusions.
Looking to the Chinese market is insightful, IMO. There's one platform for a luxury sedan, and it gets ~200mi on EV mode (~100MPGe) and then ~400mi on gas. It works out to about 70mpg purely on gas. I'm not sure how it's so high, but I'm guessing a combination of low drag (Cd), efficient small turbocharged engine (you really only need enough power maintain high speed, not accelerate up to it), and lots of regen braking.
BYD and Geely have similar systems. Their ICE are around 47% thermal efficiency so like ~double what you'd expect in a pure ICE car + regen and other bonuses.
Yes true; good point. I think this is changing (e.g. regen braking for aux. power on passenger trains maybe eventually capacitors for traction drives in the future), but currently and ~almost all the time, this is correct and a good point.
Kind of. EREVs are what locomotives have been doing for a century (and to a lesser extent barges), which is called diesel-electric in that field. I agree the terminology is lacking, but EREVs are quite compelling (and their high market share in China supports consumer demand).
Hybrid:
* ICE must run during regular operation (except for ~very short distances at ~very slow speeds) -- this increases operational costs (oil changes, economy, engine designed for torque and wide RPM range).
* Complex drivetrain with wheels moved by electric motors and ICE, axles, etc.
* Generally 10-40 miles of EV range
EREV:
* Basically an EV with a short range, and whenever you want to charge the battery on the go (or use the waste heat from the ICE) it can use an efficient (Atkinson cycle) engine to do so. (Though american EREVs have used poorly suited engines for parts availability and enormous towing numbers)
* Generally 50-200 miles of EV range
* Think "EV for daily commute; ICE for road trips (and heating)"
IMO EREVs would've been a better development path than hybrids or pure EVs.[0] Immediately lower TCO in various interest rate environments via highly-flexible battery sizes, no cold or range anxiety issues, technically simple drive train and BTMS.
[0] I mean the Prius made a lot of technical strides given the battery technology/costs and familiarity the industry had with ICE at time. Tesla went full EV which is a very optimistic approach, and works well enough if you stick around the charging network, but the batteries are still expensive and heavy compared to a small ICE + tank.
It relies on an “unintuitive observation”[0] that you can run batches basically for free (up to a limit). So if you only run one inference, you batch it plus a lot of guesses and, if you guess right, can speed up the inference by the number of guesses. If you guess wrong, you're back to regular speed (and still fully correct).
I have no idea in practice. But for the thermodynamic limit of actually making a difference, any irreversible change requires heat to be generated, e.g. initializing to zero, truncating, or bitshifts with discarded information. In contrast, addition/subtraction/multiplication/bitshifts without over-/under- flow will not necessarily generate heat.
PS. you can also use mass-energy equivalence to extend this to calculate the lower limit of mass for a given quantity of information. TL;DR: The internet weighs 50g https://www.youtube.com/watch?v=WaUzu-iksi8
That is correct[0], as known from the iPad M4 analysis.
I will say SVT-AV1 has had some significant ARM64 performance improvements lately (~300% YoY, with bitrate savings at a given preset[1][2], so call it a 400% increase), so for many use-cases software AV1 encoding (rather than hardware encoding) is likely the preferred modality.
The exceptions, IMO, are concurrant gaming with streaming (niche on MacOS?) and video server transcoding. However, even these exceptions are tenuous: Because Apple Silicon doesn't play x86's logical core / boost clock games, and considering the huge multi-threaded performance of M4, I think streaming with SW encoding of AV1 is quite feasible (for single streams) for both streaming and transcoding. x86 needs a dedicated AV1 encoder more-so due to the single-threaded perf hit from running a multi-threaded background workload. And the bit-rate efficiency will be much better from SW encoding.
That said, latency will suffer and I would still appreciate a HW AV1 encoder.
I've been doing that with a Minisforum bd790i -- a Ryzen 7945HX which is a monster of a CPU (16 zen4 cores, with very high single threaded and multi threaded benchmarks[1]) with a paltry TDP. Minisforum even coined the term "Mobile on Desktop (MoDT)". It's a great platform that is utterly cheap for what you get (mobo w/ PCIe gen5 + CPU + heatsink for less than the price of an individual comparable CPU). Note: This CPU is usually in a laptop with a relatively underpowered cooling solution compared to the linked motherboard with a proper heatsink+fan. Hence the benchmarks have a very wide spread due to varying (usually laptop-based) cooling.
Pertinent to the conversation though, the BIOS is very much lacking, and supposedly software-based fan control is not implemented. That said, running the fans constantly at ~silent levels of rotation keeps the temps cool (you can even run the heatsink without a fan if you want).
https://www.llamaindex.ai/ is much better IMO, but it's definitely a case of boilerplate-y, well-supported incumbent vs smaller, better, less supported (e.g. Java vs Python in the 00s or something like that). Depends on your team and your needs.
I've seen quad 4090 builds, e.g. here[0]. What do you mean no more than two cards? Yes, power is definitely an issue with multiple 4090s, though you can limit the max power using `nvidia-smi`, which IME doesn't hurt (mem-bottlenecked) inference.
The tyranny of the marginal user reminds me of population ethics' The Repugnant Conclusion.[0] This is the conclusion of utilitarianism, where if you have N people each with 10 happiness, well then, it would be better to have 10N people with 1.1 happiness, or 100N people with 0.111 happiness, until you have infinite people with barely any happiness. Substitute profit for happiness, and you get the tyranny of the marginal user.
Perhaps the resolutions to the Repugnant Conclusion (Section 2, "Eight Ways of Dealing with the Repugnant Conclusion") can also be applied to the tyranny of the marginal user. Though to be honest, I find none of the resolutions wholly compelling.
I'm curious how 1Password can say passkeys cannot be stolen, and yet passkeys are sharable within 1Password. Wouldn't an attacker stealing my 1Password credentials (secret-key/user/pass) also get (cf. steal) my passkeys stored in 1Password?
Regardless, I'm excited for passkeys: passwords are already abstracted out with password managers, and now they're going a step further of not being revealable w/ passkeys. However losing the escape hatch of reading a password off my phone into an untrusted computer is quite the use-case loss.
I'm surprised some low-fee automatic solutions aren't recommended here. Robo-advisors can split your cash across multiple banks on the backend. Using WealthFront[0] as an example, their cash account offers 4.05% APY, and FDIC insurance on balances up to 2M using a ton of partner banks[1] (each with 250k insurance, or 500k for join accounts). It also can send checks, issues debit cards, etc.
Does anyone have any resources on how to do this in California or the USA in general? I've been meaning to dashcam my bike for a while, and considered making an raspi-based dash cam system for automatic video processing and automating paperwork. If the alleged traffic violators retaliate against me, my efforts won't be in vain. I'm tired of being hit by cars on my bike.
Though this was a 100-year prediction so we still got three and half to go!