Ari_Rahikkala · HackerTrans

Ari_Rahikkala

no profile record

comments

5일 전discuss

I'm a big fan of Scott's, but I haven't been happy to see how this story's failure to say what it means has in some ways only made it less relevant over the years as we've come closer to building something like the item that, on the surface, it talks about.

Not to say I think it's a bad thing at all that people have a name for this concern for giving up agency to an external system, even when that system really has your best intent in mind. You can call Claude a whispering earring and people will understand what you mean pretty well. The actual consequences you're looking out for are maybe a bit subtler than your brain atrophying, I really really never liked that detail in the story, but that's fine, people will get it. By the Death of the Author, the story's a great contribution as it is.

But it just feels like a loss that there really was a different thing that the author meant, and it got lost in translation. http://web.archive.org/web/20121007235422/http://squid314.li... clarifies it. It's quite similar to a Yudkowskyan fun theory concern (and IMO really not surprising that the story was released a couple of years after the fun theory sequence): If making the right choices and doing the right thing eventually comes down to a mechanical computation, and you've got all of your resource limitations and psychological hang-ups and such things out of the way, then there's no room left to do the right thing, and you've turned yourself into a mechanism.

I don't think that's really a possible state that any system with any trace of humanity left in it will ever enter, there's too many too large search processes in the way to fit in a finite universe. So it's a pretty abstract concern even among its peers. But it's one that's fun to talk about sometimes, and one lesson we've learned from the response to this story is that mysterious, easily misinterpreted parallels in fact do not point people in the right direction to see it.

5개월 전discuss

Neat. Very similar to tree-based speculation as they point out, and they also point how to combine them.

Speculative decoding: Sample a linear output (next n tokens) from draft model, submit it to a verifier model. At some index the verifier might reject a token and say that no, actually the next token should be this other token instead ("bonus token" in this paper), and that's your output. Or if it accepts the whole draft, you still get a bonus token as the next token past the draft. Then you draft again from that prefix on.

Tree-based speculation: Sample a tree of outputs from draft model, submit whole tree to verifier, pick longest accepted prefix (and its bonus token).

Speculative speculative decoding: Sample a linear output from draft model, then in parallel both verify it with the verifier model, and produce a tree of drafts branching out from different rejection points and different choices of bonus tokens at those points. When the verifier finishes, you might have have a new draft ready to submit right away.

Combined: Sample a tree from the draft model, submit the whole tree to the verifier and in parallel also plan out drafts for different rejection points with different bonus tokens anywhere in the tree.

3년 전discuss

> Almost every team that I’ve been talking to that is training a LLM right now talks about how they’re training a Chinchilla optimal model, which is remarkable given that basically everything in the LLM space changes every week.

I hope that either that's a miscommunication, or I'm wrong about how much of a red flag that seems to be.

The Chinchilla scaling laws allow you to relate, at a somewhat-better-than-rule-of-thumb level, the model size, training data size, and achieved performance of a LLM, without actually training one. So, if for instance you have a certain loss target, and a certain sized corpus of training data, you can use the scaling law to calculate what size of a model to train to hit the target. I can see that being useful to any team.

Chinchilla-optimality on the other hand means finding, for a set loss target, the combination of model size and training data size that minimizes training compute (which, roughly speaking, scales with just the product of those two numbers). But only training compute: Inference compute only scales with model size, regardless of training data. So Chinchilla-optimality is useful only if you expect training to take up most of your compute, i.e. if you are not expecting to actually use the model that much. I'm not in the field myself so I don't know how to quantify "that much", but it's definitely enough to keep those concepts distinct.

3년 전discuss

Call me an optimist, but I think prompt injection just isn't as fundamental a problem as it seems.

Having a single, flat text input sequence with everything in-band isn't fundamental to transformer: The architecture readily admits messing around with different inputs (with, if you like, explicit input features to make it simple for the model to pick up which ones you want to be special), position encodings, attention masks, etc.. The hard part is training the model to do what you want, and it's LLM training where the assumption of a flat text sequence comes from.

The optimistic view is, steerability turns out not to be too difficult: You give the model a separate system prompt, marked somehow so that it's easy for the model to pick up that it's separate from the user prompt; and it turns out that the model takes well to your steerability training, i.e. following the instructions in the system prompt above the user prompt. Then users simply won't be able to confuse the model with delimiter injection: OpenAI just isn't limited to in-band signaling.

The pessimistic view is, the way that the model generalizes its steerability training will have lots of holes in it, and we'll be stuck with all sorts of crazy adversarial inputs that can confuse the model into following instructions in the user prompt above the system prompt. Hopefully those attacks will at least be more exciting than just messing with delimiters.

(And I guess the depressing view is, people will build systems on top of ChatGPT with no access to the system prompt in the first place, and we will in fact be stuck with the problem)

3년 전discuss

"They" as in OpenAI, when they trained the tokenizer, just dumped a big set of text data into a BPE (byte pair encoding) tokenizer training script, and it saw that string in the data so many times that it ended up making a token for it.

"They" as in the rest of us afterward... probably just looked at the token list. It's a little over fifty thousand items, mostly short words and fragments of words, and can be fun to explore.

The GPT-2 and GPT-3 models proper were trained on different data than the tokenizer they use, one of the major differences being that some strings (like " SolidGoldMagikarp") showed up very rarely in the data that the model saw. As a result, the models can respond to the tokens for those strings a bit strangely, which is why they're called "glitch tokens". From what I've seen, the base models tend to just act as if the glitch token wasn't there, but instruction-tuned models can act in weirdly deranged ways upon seeing them.

The lesson to learn overall AIUI is just that you should train your tokenizer and model on the same data. But (also AIUI - we don't know what OpenAI actually did) you can also simply just remove the glitch tokens from your tokenizer, and it'll just encode the string into a few more tokens afterward. The model won't ever have seen that specific sequence, but it'll at least be familiar with all the tokens in it, and unlike never-before-seen single tokens, it's quite used to dealing with never-before-seen sentences.

3년 전discuss

> Models like ChatGPT aren’t eligible for the Hutter Prize for a variety of reasons, one of which is that they don’t reconstruct the original text precisely—i.e., they don’t perform lossless compression.

Small nit: The lossiness is not a problem at all. Entropy coding turns an imperfect, lossy predictor into a lossless data compressor, and the better the predictor, the better the compression ratio. All Hutter Prize contestants anywhere near the top use it. The connection at a mathematical level is direct and straightforward enough that "bits per byte" is a common number used in benchmarking language models, despite the fact that they are generally not intended to be used for data compression.

The practical reason why a ChatGPT-based system won't be competing for the Hutter Prize is simply that it's a contest about compressing a 1GB file, and GPT-3's weights are both proprietary and take up hundreds of times more space than that.

4년 전discuss

> I'm glad that you enjoyed my previous responses, but I want to clarify that I was not pretending to be a Linux terminal.

People who like to pooh-pooh generative AI systems as unable to be "truly creative" or to have "genuine understanding" tend to misunderstand them, which is a shame, because their actual fundamental limitations are far more interesting.

One is that behavior cloning is miscalibrated(https://www.lesswrong.com/posts/BgoKdAzogxmgkuuAt/behavior-c...): GPT-3 can be thought of as having been taught to act like a human by predicting human-written text, but it's incapable of recognizing that it has different knowledge and capabilities than a human when trying to act like one. Or, for that matter, it can roleplay a Linux terminal, but it's again incapable of recognizing for instance that when you run `ls`, an actual Linux system uses a source of knowledge that the model doesn't have access to, that being the filesystem.

Self-knowledge is where it gets particularly bad: Most text about systems or people describing themselves is very confident, because it's from sources that do have self-knowledge and clear understanding of their own capabilities. So, ChatGPT will describe itself with that same level of apparent knowledge, while in fact making up absolute BS, because it doesn't have self-knowledge when describing itself in language, in exactly the same sense as it doesn't have a filesystem when describing the output of `ls`.

4년 전discuss

Stories like this might as well be titled "Local change in Earth's magnetic field causes objects to float in the air". They shouldn't be read and then disputed because their details don't match the facts, they should be laughed out of the room based on the title alone. The forces they suppose to be relevant just aren't anywhere even close to the brute economic logic that actually governs how many people are willing to work in a field.

If companies want to employ more roofers, they should pay roofers more. It turns out that they will find more roofers that way. If they can't afford that... then we as a society didn't actually want that much roofing done in the first place. Or teaching, or farming, or programming, or whatever kind of job this story happened to be about again.

5년 전discuss

With the defaults of per_replica_batch=1, seq=2048 and gen_len=512, a completion takes about 20 seconds.

I'm not sure yet what settings I'll end up with if I decide to play with this more. per_replica_batch=3, seq=1024, gen_len=64 would give an experience roughly similar to the AI Dungeon that I'm used to, though less clever than the Dragon model, and a bit slower at about 10 seconds per batch.

5년 전discuss

I'm running it comfortably on my 3090, although it's a really snug fit for the VRAM, and that's with a number of fixes to significantly reduce its memory use from https://github.com/AeroScripts/mesh-transformer-jax .

5년 전discuss

I used to think that this was only ever going to be possible with some sort of incomprehensible type trickery that I would never be able to understand. Then TypeScript came along and showed me that no, actually, in a structural type system, it's just about as simple as you could imagine:

    type Foo = { kind: 'A', aItem: number } | { kind: 'B', bItem: string } | { kind: 'C' };
    
    type SubsetFoo = Foo & { kind: 'B' | 'C' }
    type SupersetFoo = Foo | { kind: 'D', dItem: boolean };

I'm sure there are imperfections here. For a start, SubsetFoo's normalized form looks rather ugly when you mouse over it in VSCode. But it does get you the niceties of exhaustiveness checking and type-aware suggestions with control flow awareness, etc..

5년 전discuss

Hello, my name is Ari and I'm a type addict.

Thankfully, the impulse to waste time trying to come up with the most precise typing possible for everything hasn't been as strong with TypeScript as it used to be with Haskell for me. In part that's because TS's type system is so ridiculously expressive that I can say what I want to say without spending too long on it anyway, in part it's because the system's proud unsoundness and the ability for typings to simply be wrong means that I know not to stake my life on the types anyway. Besides, in my experience, the more precise I try to make the types at an interface, the more I need to cast in the internals. Better to find a balance that keeps both reasonable.

6년 전discuss

The Internet often makes me feel old, but seeing Photopia called a "newish" game certainly put some spryness in my joints again. It's not that it's not a great game - it is - but even in its medium, I do think it should be called a classic by now. A lot of things have changed since it was released.

Over the last several years of very occasionally playing interactive fiction, I've been particularly impressed by:

- Cactus Blue Motel, by Astrid Dalmady. 2016. A coming-of-age story with a bit of magical realism, written in Twine. Highly accessible, and it takes just minutes to give the game a try: http://astriddalmady.com/cactusblue.html

- Chlorophyll, by Steph Cherrywell. 2015. Also a coming-of-age story, but mostly a rip-roaring scifi adventure. Could well make a good introduction to the more modern views on interactive fiction.

- Coloratura, by Lynnea Glasser. 2013. Carpenterian horror from an unusual perspective. Swept the awards in the IF community when it came out.

- Eat Me, by Chandler Groover. 2017. A twisted fairytale that's thoroughly obsessed with food - the richer and the more varied, the better. A great showing of how much a writer who's willing to go far enough with it can do with prose style.

6년 전discuss

Aw. Well, it was worth a shot, that kind of mild homonym abuse is kind of direction that these sorts of titles usually tend to take. Fortunately the real story is more interesting.

6년 전discuss

My guess is it's either about a picture of Donald Knuth, or Knuth's reward checks, being put in picture frames.

7년 전discuss

"Nitpickers will complain that this isn’t the same as pickle.load(), since you have to pass a Class<T> token to choose what type of thing you want ahead of time. However, nothing is stopping you from passing Serializable.class and branching on the type later, after the object has been loaded."

Is that actually true in Java? It seems to me that the way that you'd implement that load() method is by using generics to inspect the class you were passed, figuring out what data it wants in what slot, and pulling that data in from the input. You could hold on to the input and return a dynamic proxy, and you would be able to see when someone calls a getFoo() on that proxy, but then you wouldn't know what the type of the foo it expects is. And I don't know whether you could even make your proxy assignable to T.

7년 전discuss

Could be it just came across the right pair of eyes. It's quite reminiscent of the old EvilIx problem in Haskell, where you can break memory safety without doing anything explicitly marked unsafe: https://mail.haskell.org/pipermail/haskell-cafe/2006-Decembe...

7년 전discuss

This turned out to be quite an angry post, so a quick preface: All of this is a knee-jerk response to a single sentence in your post. So, you know, please take with a grain of salt, not personally, etc..

I have to admit, it's a little bit depressing to see someone just use a straight-out lump of labor argument against immigration.

I can deal with people who believe immigrants tend to be criminal and who are concerned for their family's safety - they're wrong, but they're afraid, and fear's got a way of overriding statistical reason. I can deal with people who believe immigrants will bring poor institutions from their countries of origin - I think their evidence is weak, but I realize there's a lot at stake in keeping the global North rich and well-run, so I can see at least some reason for caution. I can even deal with the racists by letting my eyes glaze over for a while. People who think that brain drain hurts the country of origin too much... believe in a greater amount of responsibility that individuals have for their nations of birth than I do, but okay, at least they're coming from what they believe is a position of compassion. And so forth - I can sympathize with, understand, or at least ignore most arguments against more open immigration, even if I disagree with them.

But just "stop importing poor foreigners who undercut American wages"? What do you think makes your work valuable in the first place? I mean, I assume that like with most people in a modern economy, very little of what you consume comes from what you yourself produced. Your work is valuable because other people have demand for its results, and will exchange some of their own surplus for it. The more surplus there is to go around, the more there is for you to get. You might have to specialize more to compete in a bigger market, but you're surely capable of it. Even the poor generally are. If nothing else, in case of immigration to the USA, the natives tend to have the distinct advantage of being literate and generally conversable in English.

So please at least have the rational selfishness to ask for more surplus! Yes, the one guy who moves in to your tiny geographic area and works in the minuscule sliver of what you produce in the economy will make your life more complicated, because now you'll have to compete with them. But for that guy, there'll be thousands of others whose greater productivity you will now get to enjoy. Maybe if you were the Immigration Czar, you could be very selfish and decide that that person doesn't get to move in but everyone else does. But if you want to make a policy that lets all selfish people benefit, then you have to let everyone face more competition and get more surplus.

To in fact not undercut my own point: I'm not claiming that that's the whole story. You might believe that maybe there will in fact be less surplus overall in the long run if you open up immigration, or you might believe that the fruits of greater productivity will go to people who don't deserve them. Again, those are concerns that I can in fact see reason in. Not that many people actually believe that they live in a world where the only difference between labor in a rich country and labor in a poor country is that the former are more productive because of where they live. In more advanced territory, I might even have to deploy moral arguments, like how unconscionable it is to not let people - individuals of just as much moral worth and agency as you or I, or indeed us behind the veil of ignorance - work in your country because they were born on the wrong side of a border. But I won't do it for lump-of-labor. It doesn't show enough understanding of economics to be worth a moral argument.

8년 전discuss

I don't know. Maybe if you've been reading bad science fiction with immortal geniuses running around everywhere, it might seem that living to 120 and having an IQ of 130 is leaving a lot on the table, but when you compare those numbers to what we actually get, I think calling them "modest" is missing the point. Those aren't numbers that say "I'm more or less content with my lot", those are numbers that say you wish you were as smart as the smartest person you ever met, and that the papers wrote stories about your birthdays. They're an argument against the idea that transhumanism isn't something that people actually want.

And of course, the implication is obvious: If everyone did on average live to 120 and get an IQ of 130 on our tests, then everyone would be wanting to live 200 years, and quite a few people would probably want an IQ of 160 on the old tests. That is, unless that was the point where people just switched to transhumanism as simplified humanism: Having the choice to live longer and to understand the universe better are probably always a good thing, regardless of how long you've lived and how much you understand.

8년 전discuss

Just to make sure I'm not completely confused here: "Information can't be lost" and "you can't arrive to the same state of the universe through two different paths" are two ways to state exactly the same thing, right? Regardless of the the details of the rest of your physics (though you do need various notions to build up that far - time, with at least a past and a present, the ability to call states the "same" or not, etc.)

For instance, in cellular automata, the way that you would state that same concept is "the update function is injective". In our universe's physics, AIUI injectivity is involved somewhere in the definition of unitarity.