I think your per-axis std normalization is likely doing a big pile of the work —- it’s fairly well-known that “wrong” PCA, setting sigma=Id or just taking a square root, gives better embeddings than the un-normalized version. It would be worth showing a comparison to similarly-normalized PCA I think, if it’s not too hard?
I did a goodly chunk of vibe coding over the summer and I found that the best language for me was Rust implementations with Python bindings for interface. A few reasons:
- Everything about rust enforcing correctness catches lots of bugs
- Using a high-level API means I can easily hand-check things in a repl
- In addition to tests, I required a full “demo notebook” with any PR — I should be able to read through it and confirm that all the functionality I wanted has actually been implemented
If the philosophy is (and it should be) “loc is free”, it’s worth thinking about how we can make LLMs produce more loc to give us additional comfort with correctness. Language choice is very much a way.
Another completely viable solution (other than adding extra starch) I’ve found is to sprinkle a bit of sodium citrate (the sodium salt of citric acid, a common food additive and cheap on Amazon) over the cheese before adding to the pan. This improves the melting qualities of the cheese and avoids the starch issue altogether. You’re basically using pecorino velveeta.
A good way to understand why cosine similarity is so common in NLP is to think in terms of a keyword search. A bag-of-words vector represents a document as a sparse vector of its word counts; counting the number of occurrences of some set of query words is the dot product of the query vector with the document vector; normalizing for length gives you cosine similarity. If you have word embedding vectors instead of discrete words, you can think of the same game, just now the “count” of a word with another word is the similarity of the word embeddings instead of a 0/1. Finally, LLMs give sentence embeddings as weighted sums of contextual word vectors, so it’s all just fuzzy word counting again.
The key question here isn’t so much whether GPT-4 beats the actual human decks as much as what it had to fabricate to do so. The humans are probably constrained by things like “reality” and “what their business has done in it” while GPT-4 could make up anything it wanted. A fair comparison would be to humans given the same prompt and told to invent whatever facts they wanted.
Sure. Moonshine I know a lot about, relative to physics at least. Given that I now work somewhere that specializes in term embeddings, it kind of feels like I've spent half my life injecting stuff into vector spaces...
Hey, I'm a mathematician, not a physicist. We tend only to get as far as "oh yeah, here's all this pretty math we care about. I hear it has some applications to physics..."
Presumably the extra credit is that the binary Golay code is closely related to the Leech Lattice, and thus to the entire moonshine situation, which gets you to string theory. See:
Very company dependent. At Luminoso (http://luminoso.com), we live in Python, with varying doses of Javascript for frontend, very little R (but some of us know it), Java and C++ are pluses (for working with other people's software), and we have at least some Haskell in production (mostly for preprocessing). And plenty of ops tools for deployment. We have used a few neural networks frameworks and machine learning packages; definitely the Python machine learning ecosystem is big for us but we haven't found our One True neural network framework yet. Other companies (say in image recognition, Ditto Labs for instance) are definitely more CNN-oriented, yet others are certainly doing work in R.
If you have a company that is doing genuinely interesting AI work, it's likely that they are somehow on the forefront of research, pushing existing tools to do things that they can only barely do. If you find yourself in an AI role (typically only some roles actually do AI, of course -- systems need to stay up, great UIs need to be built, etc.), I would guess that you'll need to familiarize yourself with their particular toolsets and methods rather than assuming that there are universally correct things to go learn.
Indeed -- it's struck me recently how Wes was probably among the first (if not necessarily number one, if only because of ill-definedness) people to make computers smaller. Looking back at the sixty years since he began, it's been quite something.
He'd talk about computers for fun, at least. But try to convince him that the internet actually worked and you'd have a whole evening's conversation on your hands...
It's a reference to a story about Ramanujan, who once (remarkably) told G. H. Hardy that 1729 wasn't a boring number because it was the smallest number that could be expressed as the sum of two cubes in more than one way. Sometimes trivial but always erudite in the best way, that was Wes.
Everyone has stories, most better than me. I find my most vivid memory of experiencing him as an adult was that every time you spoke with him he'd have had some completely flabbergasting new life experience you'd never imagined. He was a guest on something like the second trip to China after Nixon, took seminars with Oppenheimer at Berkeley (he tended to call him Oppy), and may have been invited to help create an early ATM. The list goes on and on.
One Christmas, we were talking about how difficult it is to assess the accuracy of an algorithm when you only get a few tries because running it is expensive; he promptly told us about his experience with this problem, which he encountered driving plutonium nuggets across Richland, Washington. After the video from the ACM conference on personal workstations got published here, I learned for the first time he'd been fired from MIT three times for insubordination, and I never got a chance to ask him what the story there was...
Oh! And he actually made a physical turing machine good enough to do fairly sophisticated computations on. It ended up in my father's intro CS class at Princeton for many years.
And HN will of course appreciate one of his favorite one-liners:
Did you know that 49 is the lowest number that can be expressed as the square of 7 in only one way?
I'm his grandson, not his colleague, so far be it from me to say he was a great man, but he was a great grandfather and an inspiration. He taught me an enormous amount, from logic to just how hard it can be to get people to adopt new technologies -- he eschewed word processing, writing every document directly in PostScript even up to the present -- but most importantly he showed me how important it is to love what you do, and for that I owe him a ton of my happiness and career.
I couldn't possibly agree more in general, though of course I'd quibble with many of the specifics...
As to stodginess, I'd say there's a big difference between personal/small commercial lines and the big ticket enterprise-type stuff -- the underwriting process goes from something data-driven to something relationship-driven very quickly indeed. The bigger the commercial line, the more likely it's all about who throws the best yacht parties (reinsurance in particular suffers from this massively).
Crowdsourced insurance is indeed a terrible idea, though you could imagine 'web of trust' insurance that almost made sense -- say my ten thousand best friends and I know that we're all great actuarial risks, perhaps because we have some kind of information on which it's illegal to select (that we're all in the same gym, say). We could then try to write ourselves health insurance for cheap, because our plan would select only us gym members. You can sort of make it work, as long as you're prepared to make the regulators hate you.
Which is the real problem, of course -- most people buy insurance because they have to, not because they want to. Auto insurance that wouldn't pass muster with the police, or home insurance that wouldn't satisfy the bank holding your mortgage, doesn't solve the problem.
Do good problems exist? Sure. A web-based Managing General Agency, for instance, could do very well for itself, but the expressed ideas in this section are pretty terrible.
This is an impact of the default Unicode normalization, which is set to NFKC. This normalization is lossy for things like the ordinal indicator and trademark symbol; if you'd like to keep the ordinal indicator unchanged, use NFC normalization:
>>> print ftfy.fix_text(u'ordinal indicator º to o in addresses.')
ordinal indicator o to o in addresses.
>>> print ftfy.fix_text(u'ordinal indicator º to o in addresses.',normalization='NFC')
ordinal indicator º to o in addresses.