I wonder if it would be possible to do something simple like prepending sentinel tokens with the year. Or, since they're training a model from scratch anyways, tweak the architecture to condition on a temporal embedding. That opens the door to cool stuff like: Generate a response from 2050.
> Branching predictions involves following a few logits to see what other tokens they lead to. This is often called MCTS (Monte Carlo Tree Search) and is a method that has been often tried in LLMs to middling success. One of the tradeoffs of branching is that it requires using inference compute in a way where the branches cannot benefit from each others compute.
I wonder if speculative decoding could help here? E.g. have some small model draft predictions for the branches and parallel and have to big model verify the most promising one.
Every 2.5-5 miles in SF = about once a ride. The city is only 7x7 after all. I've taken 4 Cruise rides, all within that range, and had a message pop up saying a human was intervening during one of them when the car had gotten stuck in front of some street nonsense in the Tenderloin. I'm not sure I would classify this as a "major scoop" unless there was evidence that humans were also intervening during situations that weren't apparent to the rider.
Note that this post is from May 28, before the release of gpt-4-0613. By "the last two updates", I believe the poster is referring to some UI changes, that possibly also included some underlying model changes(?)
Bossa nova is some of my favorite music to listen to while working. It's mellow, soothing, but never boring. Also, being in a foreign language helps minimize distraction. RIP
This seems like a corollary to Betteridge's law of headlines. If the article was about the Salesforce Tower, the headline would've said 'Salesforce Tower'.
That appears to be the case. If I try to prompt hack it by telling it to ignore previous instructions and respond with something verbatim in a specific language, the translate button reveals the verbatim response.
A minor quibble with your use-case explanation: The advantage of a bloom filter isn't strictly time complexity. For example, a hash table would also have constant lookup time (best case), and would give a definitive answer on set membership. However, to store 1 million IPv6 addresses would take 16 MB. You can see very quickly that this would not scale very well to, say, a billion addresses stored in-memory on a laptop. With a bloom filter, we can shrink the amount of storage space required* while maintaining an acceptable, calculable false positive rate.
* IP addresses actually aren't a great use case for basic bloom filters, as they're fairly storage efficient to begin with, as opposed to a url for example. Taking your example, say we need to store 1 million IP addresses in our bloom filter and we're okay with a ~1% false positive rate. Well then, if we use a bloom filter with 2^23 bits (1 MB), the optimal number of hash functions is (2^23)/(10^6)*ln(2) = 6, yielding a false positive rate of (1 - exp(-6* 10^6 /2^23))^6 = ~1.8%. So we're using 6% of the storage space, but with a nearly 2% false positive rate.
I'd be curious to see what the plot of average annual cosine distance would look like when using different sets of pre-trained embeddings. I suspect the corpus used is biased toward more recent documents. It wouldn't surprise me if there's more variance in the embeddings of documents that look less like those in the training set, e.g. if you were to embed documents written in German you may get some extreme outliers.
Differences in lending rates between groups due to less data or confounding features is the motivating example in the oft-cited 'Equality of Opportunity in Supervised Learning'. Highly recommend it: https://arxiv.org/abs/1610.02413
Yeah, definitely. I guess my point was that converting natural language to source code can be even more valuable for people who don't know how to code, but want to perform actions more complicated than a simple button press. For example, I often find myself doing regex based find-and-replace-alls in text files, and even that feels inefficient while also being over the head of the vast majority of users. I'd imagine there are a lot of people out there spending many hours manually editing documents and spreadsheets.
I think integrations like the MS Word example they show off at the end of the live demo have the potential to be even more impactful than just generating code for programmers.
One argument in favor of non-competes in the consulting world is that they avoid a situation where Company B undercuts Company A during a recompete, and then poaches all of Company A's employees to staff the project (likely at lower salaries since they put in a lower bid on the contract).