Hi! I’m a product engineer on the Claude.ai team. Claude.ai does support branching conversations. If you hover on a message, there should be an edit button, and once you edit the message, you can again hover on it, which will show you left/right arrows that will switch between the branches. Please let me know if you have any troubles with this!
I don’t want to die today, and tomorrow I’m not going to want to die, and the day after I’m not going to want to die, and so on. Therefore, I want to live forever — proof by induction on the natural numbers.
If a model hasn't been explicitly told (via some system prompt or something) about its weights, it won't know them. It would be akin to asking you how many neurons you had. How would you know?
It seems to me that RAG is really search, and search is generally a hard problem without an easy one size fits all solution. E.g., as people push retrieval further and further in the context of LLM generation, they're going to go further down the rabbit hole of how to build a good search system.
Is everyone currently reinventing search from first principles?
I just got engaged. When we looked at rings, the jeweler asked my fiancé if she wanted natural or synthetic, and she responded “I don’t want a blood diamond!!” Of course, mined diamonds aren’t blood diamonds, but her impression was still they were a little ickier.
The jeweler told me that one reason to get a natural diamond was that the prices of lab grown diamonds had been falling, whereas natural hasn’t as much, so the ring would hold more value. I told her that was exactly why we wanted to go with a lab grown diamond! This isn’t an investment — we aren’t planning to sell the ring.
Ultimately, for a price that didn’t break the bank, we got an absolutely gorgeous ring with diamonds larger and higher quality than we would have been able to afford with natural. Diamond rings may have started as something to resell in divorce, but for us (for my fiancé really), it was more about getting something that was beautiful, and if it didn’t cost as much, great! I suspect most Americans will feel similarly.
A city on its own generally has the power to change zoning and land use laws. There are limitations (some states give cities more or less power here) but generally cities do have the ability to make changes that would reduce noise and improve quality of life for their residents.
On the other hand, cities do not have the ability to redesign cars. A city can’t just choose to invent a quieter car!
I would love it if we can engineer cars and trucks to be quieter; I think though that a more feasible path for any city is to work on zoning and land use changes.
> All eligible entries must include either the word
"wallabywinter" or the word "yallabywinter" (the “eligible keywords”)
in one or more places as close as possible to the code.
If I'm training codegen models, why wouldn't I just exclude code that contains these keywords? Shouldn't you have secret keywords, that people have to register to you, but you don't make public until after the fact, in order to avoid this?
It's not that they're just charging per token -- the actual models are operating on a token level. The model sees things in terms of tokens, and in openai's case, these tokens are subword (pieces of words), not words themselves, not characters.
So the real question is, what is the benefit of modeling your tokens as subwords, rather than as characters or words?
I think there is a lot of nuance here, and I don't understand it all. But, some benefits:
* Words, at least in English, are composed of different pieces, like roots, prefixes, and stems. Modeling at the subword level more naturally aligns your model with this aspect of language. If I tokenize "warmest", I get "warm" and "est". So, the meaning of the token "est" can be learned by the model -- whereas if you modeled by words, the model would have to individually relearn this aspect of information for every word ending in "est".
* Modeling at the subword level makes your sequences a lot shorter than modeling at the character level, which should help with things like efficiency.
* Modeling at the subword level makes your vocabulary a lot bigger than just modeling at the character level, which I suspect helps the model, as it can assign the subwords themselves meaning. E.g., it can learn the meaning of the token "warm" on its own, rather than having to learn this meaning only through learning the relationship of the tokens "w" "a" "r" and "m".
Hope this helps! Would love for anyone else to chime in/add on/correct me.
This is not really true. The Chinchilla paper showed that a 4% difference in loss between Chinchilla and Gopher led Chinchilla to blow Gopher out of the water at most tasks, including 30x performance in physics.
Empirically, LLMs have shown to have emergent abilities appear at different loss levels. So, a 10% difference could really matter.
^^ this. For instance, LLMs are really good at turning natural language into SQL. And if you know SQL, you can read it and make sure it looks good. But, it's much faster and easier than writing SQL by hand.
Do you know of any advice or guides on how to use the chatgpt api? It's unclear to me when things should be a system message, and if not, if they should be user or assistant messages.
You say here that the simple message should be really simple. So, further instructions should be in a user message? Any idea what constitutes a "simple" system message?
The other day, I was trying to build a web scraper, but I kept getting an error that the data wasn't utf-8 encoded. I gave chatgpt the error along with the first 10 bytes (e.g. b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03') and it informed me that this was a gzip-encoded binary file, and then it wrote me working code to decompress the file.
This is extremely valuable and saved me hours/was the difference between me stopping or not.
There will surely be people that overhype this, but there is real value as well.
One tip to speed up how you're finding the closest matches -- openai's embedding vectors are normalized and all fall upon a sphere. So, the cosine similarity is equal to the dot product. Might speed things up a bit to use only the dot product.
I was also in STL when this happened. Had to drive 4 hours away to Evansville, IN in order to get a rental car. Just showed up here and they told me I was lucky to reserve it last night, they’re now turning away walk ups at the counter.
I mean, I don't really disagree with you. For sure, most all services have in the history of society been mostly, if not entirely, consumed by the rich. For most of society, most everyone has been poor. If you could afford fancy stonework on your house, if you could afford someone to make your bed, of course you were rich.
The past century has seen an explosion of wealth in the world, with many people coming out of extreme poverty, emergence of large middle class in the US, etc. So, the world has gotten a lot richer. But, what does it mean to be richer?
Some things seem relatively obvious -- people have bigger houses. They get to have things like fridges and vacuums. These are tangible goods, for which the cost has fallen in line with increasing specialization and automation.
I'm saying that I think it's interesting that while a richer world can purchase more goods, it can't necessarily purchase more services, since the cost of labor rises in a richer world. Basically, productivity has to increase for people to be able to purchase the services with the rising cost of labor.
I'm not really concerned with the things I mentioned going extinct -- it's not a huge deal if we don't have stonework on buildings in NYC, and indeed if we can build housing faster than before by way of automation, that's great. But many people have wondered "why can't we build beautiful buildings in NYC like they used to?" and I'm basically saying, I think rising cost of labor is the reason, and it's interesting that this goes hand in hand with becoming a richer society.
Let's say there is a masseuse. They use their labor to give massages. They are also self employed, so they own their own business. If the cost of labor rises, the masseuse makes more money from giving massages, but this necessarily means that the price of their massages is going up (the cost of labor and the price of the massage are the same thing). Okay, let's take this one step further.
Let's assume that there is a massage parlor, with masseuses who do not own the business, and that the owner of the business has some profit. Cost of labor goes up, and now the business owner no longer has any profit, and it is earned by the masseuses. Yay! However, what happens if/when cost of labor goes up further, and there is no more profit? Prices will rise.
> You keep assuming that business owners can and will just raise prices and the markets will bear it, but this is not always the case.
I think this is my point actually. Markets won't always bear it. Sometimes they will raise their prices and the markets won't bear it, and then the business will no longer exist, and society won't be able to pay for it. E.g., the business of a hotel that makes everyones beds every day, or the business that builds a beautiful NYC building adorned with hand carved stone across the entire facade.
Laborers get more of the profits of companies and in turn get wealthier. However, the costs of services in turn go up (due to rising cost of labor), so those laborers (society) cannot actually afford more services (provided by people). Workers can more easily purchase goods which benefit from increasing automation and technology.
So, these workers which are capturing more of the profit can afford to buy fancier kitchen appliances and cook themselves better meals. But, can they all then afford to go out to eat more -- when going out to eat is primarily an endeavor dependent on the labor of others (and thus it becomes more expensive)?
There is a take that consistently comes up in discussions like this (and indeed is prominent several times on this post), which says "the problem isn't a shortage of workers, it's that employers don't want to pay workers enough." This seems somewhat limited to me, and there is an alternate explanation I would like to explore.
It seems to me rising cost of labor, while good for the people being paid that rising cost, can actually be bad for society overall, because you some businesses and industries will just disappear when the cost of labor goes high enough. And sure, you can respond that those businesses shouldn't exist if they can't pay workers enough, and yeah, fine. But the result is that we just then don't have those businesses (making society worse off?).
For this hotel example -- maybe cost of labor continues to go up, and it's much harder to automate making a bed than vacuuming a floor, and in 10 years, we all generally stay at hotels where robots vacuum our floors every day, but no one ever makes your bed for you -- that becomes a relic of the past.
I live in NYC, so a more prominent example I think about frequently is how we can no longer afford beautiful stone masonry on all our buildings. When labor was cheaper 100 years ago, it was easier to pay someone to carve stone -- now, not so much.
I want workers to be paid a lot so they can live good lives, but it's worth thinking about what happens to society as cost of labor continues to go up, and why the cost of labor is going up in the first place (Baumol's Cost Disease? Land rent?).