> The two modified satellites will test hardware planned for Starlink V3 and will attempt to scan Starship’s heat shield and transmit imagery down to operators to test methods of analyzing Starship’s heat shield readiness for return to launch site on future missions
Hope we get to see those images. Would be awesome to see a 3rd person view of starship in space.
Apple silicon has been an unmitigated success so it makes sense they’d go with Ternus. On a related note Apple needs to add Ternus to their spell check dictionary
So let’s start with a really simple decoder transformer with a single layer and single attention head, and train it to predict the next token in a sequence of text. To predict the next token you need a few things: a query for the very last token in the sequence, and a key and value for every prior token. You take your query and compute a dot product with every prior key (two large vectors in, scaler attention score out). That scaler attention score first goes through softmax, and then becomes the weight you use to compute a weighted average of your values, new value goes through the mlp, mlp output is projected into the logits from which you sample your next token (that’s the general idea at least skipped a few steps).
The last query in the sequence will be new for every new token you predict, but the set of prior keys and values stay the same, ie keys and values are reusable. The key value cache gets bigger and bigger for each new token you add to the sequence, and that’s where compression comes in. You have to store the keys and values in vram, and you’d like to keep the size down by not storing the raw uncompressed tensors. To make this work well your compression needs two things: it needs to be fast so that you can compress and decompress on the fly, and it needs to play well with softmax attention. Prior attempts at compression usually suck at one or the other, either the speed to decompress is too slow and your token/s takes a hit, or you lose important precision and the model output quality suffers. The claim in the paper is that they’ve made progress on both.
The allegations of fraud made by the people invading my home and terrorizing my friends? I’ll take those with a grain of salt.
And for the record, I’m not afraid of ICE, never said I was. ICE is racially profiling people, arresting them without cause, and deporting them without due process. I happen to be white, so that doesn’t apply in my case. It’s also coincidentally the same reason I feel safe posting a comment like this online. Free speech is being chilled in communities that ice targets, and I feel responsibility to relay what I’m hearing.
ICE also just gunned down a US citizen in the street, and that should scare everyone.
I lived my whole life in the twin cities and have a lot of friends, US citizens, who are too scared to go out to eat right now because of the ICE raids. If that wasn’t the point it is certainly the effect. I applaud Walz and Frey, and I will be ranking Frey first next time he’s up for reelection. Something tells me though he will be on to bigger things than mayor of Minneapolis.
> About the second point, I've been under the impression that because LLMs are trained on average code, they infer that the bugs and architectural flaws are desirable
This is really only true about base models that haven’t undergone post training. The big difference between ChatGPT and GPT3 was OpenAI’s instruct fine tuning. Out of the box, language models behave how you describe. Ask them a question and half the time they generate a list of questions instead of an answer. The primary goal of post training is to coerce the model into a state in which it’s more likely to output things as if it were a helpful assistant. The simplest version is text at the start of your context window like: “the following is code was written by a meticulous senior engineer”. After a prompt like that the most likely next tokens will never be the models imitation of a sloppy code. Instruct fine tuning does the same thing but as permanent modifications to the weights of the model.
Humans don’t learn to write messy complex code. Messy, complex code is the default, writing clean code takes skill.
You’re assuming the LLM produces extra complexity because it’s mimicking human code. I think it’s more likely that LLMs output complex code because it requires less thought and planning, and LLMs are still bad at planning.
Image generation and image input are two totally different things. This is about feeding text into LLMs as images, it has nothing to do with image generation.
I don't know if this game was vibe coded, but it certainly could have been. Most notable thing about this post is probably that vibe coded games are good enough now to fool HN.
Same. There are really only two features I care about in a phone: a high refresh rates and weight. At 165 grams the iPhone air is by far the lightest 120hz phone apple has ever made. Second place is the iPhone 15 Pro at 187 grams. Getting ready to ditch my 15 pro.
Embedding models are not always trained with the rest of the model. That’s the whole idea behind VLLMs. First layer embeddings are so interchangeable you can literally feed in the output of other models using linear projection layers.
And like the other commenter said, you can absolutely feed single tokens through the model. Your point doesn’t make any sense though regardless. How about priming the model with “You’re a helpful assistant” just like everyone else does.
>> For each LLM, we extract static, token-level embeddings from its input embedding layer (the ‘E‘matrix). This choice aligns our analysis with the context-free nature of stimuli typical in human
categorization experiments, ensuring a comparable representational basis.
They're analyzing input embedding models, not LLMs. I'm not sure how the authors justify making claims about the inner workings of LLMs when they haven't actually computed a forward pass. The EMatrix is not an LLM, its a lookup table.
Just to highlight the ridiculousness of this research, no attention was computed! Not a single dot product between keys and queries. All of their conclusions are drawn from the output of an embedding lookup table.
The figure showing their alignment score correlated with model size is particularly egregious. Model size is meaningless when you never activate any model parameters. If Bert is outperforming Qwen and Gemma something is wrong with your methodology.
So instead of next token prediction its next event prediction. At some point this just loops around and we're back to teaching models to predict the next token in the sequence.
You’re thinking about this like the final layer of the model is all that exists. It’s highly likely reasoning is happening at a lower layer, in a different latent space that can’t natively be projected into logits.
My personal theory is that it’s an emergent property of many attention heads working together. If each attention head is a bird, reasoning would be the movement of the flock.
That’s true yeah. The model can do that because calculating latents is independent of next token prediction. You do a forward pass for each token in your sequence without the final projection to logits.
Attention computes a weighted average of all previous latents. So yes, it’s a new token as input to the forward pass, but after it feeds through an attention head it contains a little bit of every previous latent.
The dimensionality I suppose depends on the vocab size and your hidden dimension size, but that’s not really relevant. It’s a single linear projection to go from latents to logits.
Reasoning is definitely not happening in the linear projection to logits if that’s what you mean.
Hope we get to see those images. Would be awesome to see a 3rd person view of starship in space.